<jupyter_start><jupyter_text>data acquistion* Read and load data into python* check dataframe datatypes* get statistical summary of numrical data* provide a concise summary of dataframe<jupyter_code># import libraries required for data analysis and visualization
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np


from IPython.display import display

%matplotlib inline<jupyter_output><empty_output><jupyter_text>read data and check it<jupyter_code>autos_df = pd.read_csv("auto-mpg.data-original",delim_whitespace=True,header = None)
autos_df.head()
# set header_attributes for your dataframe
header = ['mpg','cylinders','displacement','horsepower','weight',
         'accelration','model_year','origin','car_name']
# set header to your dataframe
autos_df.columns = header
#check our dataframe now to make sure everything is going right
autos_df.head(50)
# check data dimensions
autos_df.shape
# get basic info about dataset
autos_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 406 entries, 0 to 405
Data columns (total 9 columns):
mpg             398 non-null float64
cylinders       406 non-null float64
displacement    406 non-null float64
horsepower      400 non-null float64
weight          406 non-null float64
accelration     406 non-null float64
model_year      406 non-null float64
origin          406 non-null float64
car_name        406 non-null object
dtypes: float64(8), object(1)
memory usage: 28.6+ KB<jupyter_text>working with missing data* identify missing data* deal with missing data* correct data format<jupyter_code># identifing missing values
missing_data = autos_df.isnull()
# itertate through whole set of data to count true and false values
for col in missing_data.columns.values.tolist():
    print(col)
    print(missing_data[col].value_counts())<jupyter_output>mpg
False    398
True       8
Name: mpg, dtype: int64
cylinders
False    406
Name: cylinders, dtype: int64
displacement
False    406
Name: displacement, dtype: int64
horsepower
False    400
True       6
Name: horsepower, dtype: int64
weight
False    406
Name: weight, dtype: int64
accelration
False    406
Name: accelration, dtype: int64
model_year
False    406
Name: model_year, dtype: int64
origin
False    406
Name: origin, dtype: int64
car_name
False    406
Name: car_name, dtype: int64<jupyter_text>dealing with missing value* replace horsepower missing values with the mean of horsepower* drop the data of label y (dependent variable) ("mpg")<jupyter_code># calculate the mean then replace it with nan values in horsepower column
horsepow_mean = autos_df["horsepower"].astype("float").mean(axis=0)
autos_df["horsepower"].replace(np.nan,horsepow_mean,inplace=True)
# check out dataset
autos_df.info()
# we get some info from previous operation that we have null values(missing values)
# these missing values in "mpg" column
# since mpg is our y that we want to predicate.we should drop the whole record 
autos_df.dropna(subset=['mpg'],axis=0,inplace=True)
autos_df.reset_index(drop=True,inplace=True)
#after drop the row in "mpg" missing values
# check out our dataset again
autos_df.info()
# check our dataset to see if we need to correct somedat format or not
autos_df.dtypes<jupyter_output><empty_output><jupyter_text>correct data format* cylinder feature is multi-value discrete variable* model_year (multi-value discrete)* origin (multi-value discrete)<jupyter_code># convert each of "cylinders","model_year" and "origin" to integer to correct data format
autos_df[['cylinders', 'model_year','origin']]= autos_df[['cylinders', 'model_year', 'origin']].astype("int")
# check out the dataset to make sure from the changes that we made
autos_df.dtypes<jupyter_output><empty_output><jupyter_text>Data Standardization* example     * transform "mpg" to L/100km<jupyter_code># adding new feature by transform "mpg" to L/100km
autos_df["L/100"] = 235/autos_df["mpg"]
# check out the dataframe
autos_df.head()
# get max and min value of each continous variable
cont_var_features = ["displacement","horsepower","weight","accelration"]
i = 1
for col in cont_var_features:
    print(str(i)+"-"+col)
    print("Minimum value: {0} and Maximum Value: {1}".
          format(min(autos_df[col]),max(autos_df[col])))
    print()
    i = i +1
#simple normalization for weight
autos_df['weight'] = autos_df['weight'].astype(int,copy=True)
# normalize weight by devide weight by max(weight)
autos_df['weight'] = autos_df['weight']/autos_df['weight'].max()
# check out "weight" column after we did weight normalization
print(autos_df['weight'].min())
print(autos_df['weight'].max())
# binning both of "horsepower" and "displacement" columns
# convert "displacement" and "horsepower" into integer type
print("Displacement Minimum Value: ",autos_df['displacement'].min())
print("Displacement Maximum Value: ",autos_df['displacement'].max())
print("Horsepower Minimum Value: ",autos_df['horsepower'].min())
print("Horsepower Maximum Value: ",autos_df['horsepower'].max())
# convert type of displacement and horsepower from float' to int
autos_df["displacement"] = autos_df['displacement'].astype(int,copy=True)
autos_df["horsepower"] = autos_df['horsepower'].astype(int, copy=True)

print("Displacement Minimum Value: ",autos_df['displacement'].min())
print("Displacement Maximum Value: ",autos_df['displacement'].max())
print("Horsepower Minimum Value: ",autos_df['horsepower'].min())
print("Horsepower Maximum Value: ",autos_df['horsepower'].max())
# display hisotogram for  "displacement" 
plt.hist(autos_df['displacement'])
plt.xlabel("Displacement")
plt.ylabel("Frequency")
plt.show()
# display hisotogram for  "horsepower"
plt.hist(autos_df["horsepower"])
plt.xlabel("horsepower")
plt.ylabel("Frequency")
plt.show()
#create bins for both of features displacement and horsepower
horse_bins = np.linspace(min(autos_df['horsepower']),max(autos_df['horsepower']),4)
disp_bins = np.linspace(min(autos_df['displacement']),max(autos_df['displacement']),4)
group_names = ["low","medium","high"]

autos_df["horsepower_binned"] = pd.cut(autos_df["horsepower"],horse_bins,labels=group_names,include_lowest=True)
autos_df["displacement_binned"] = pd.cut(autos_df["displacement"],disp_bins,labels=group_names,include_lowest=True)
# check the result
autos_df[["horsepower","horsepower_binned","displacement","displacement_binned"]].head()
# let's see the number of vehicles in displacement_binned
autos_df["displacement_binned"].value_counts()
# let's see the number of vehicles in  horsepower_binned
autos_df["horsepower_binned"].value_counts()
# diplay the bar graph horsepower_binned
plt.bar(group_names,autos_df["horsepower_binned"].value_counts())
plt.xlabel("Horsepower bins")
plt.ylabel("Frequency")
plt.title("horsepower distrbution")
plt.show()
# display the bar graph displacement_binned
plt.bar(group_names,autos_df["displacement_binned"].value_counts())
plt.xlabel("displacement bins")
plt.ylabel("Frequency")
plt.title("Displacement distrbution")
plt.show()
# check columns of our dataframe
autos_df.columns
#check our dataframe
autos_df.info()
# check car_name feature
autos_df["car_name"].value_counts()
autos_df.to_csv("clean_auto.csv",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11650485436893204, 'fraction_numerical': 0.017614424410540914, 'mean_word_length': 6.704059829059829, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '626284', 'license': 'Apache-2.0', 'n_tokens_mistral': 2471, 'n_tokens_neox': 2293, 'n_words': 634, 'path': 'auto-mpg/.ipynb_checkpoints/auto_data_preprocessing-checkpoint.ipynb', 'repo_name': 'Mohamed-3ql/RegressionProjects', 'size': '7210'}