<jupyter_start><jupyter_text>Table of content- Identify and handle missing values    - Identify missing values    - Deal with missing values    - Correct data format- Data standardization- Data Normalization (centering/scaling)- Binning- Indicator variable  Identify and handle missing values<jupyter_code>df.replace("?", np.nan, inplace=True)
df['normalized-losses'].isnull().value_counts()
# loop for knowing missing values in each columns
for col in df.columns:
    print(df[col].isnull().value_counts())
    print()<jupyter_output>False    205
Name: symboling, dtype: int64

False    164
True      41
Name: normalized-losses, dtype: int64

False    205
Name: make, dtype: int64

False    205
Name: fuel-type, dtype: int64

False    205
Name: aspiration, dtype: int64

False    203
True       2
Name: num-of-doors, dtype: int64

False    205
Name: body-style, dtype: int64

False    205
Name: drive-wheels, dtype: int64

False    205
Name: engine-location, dtype: int64

False    205
Name: wheel-base, dtype: int64

False    205
Name: length, dtype: int64

False    205
Name: width, dtype: int64

False    205
Name: height, dtype: int64

False    205
Name: curb-weight, dtype: int64

False    205
Name: engine-type, dtype: int64

False    205
Name: num-of-cylinders, dtype: int64

False    205
Name: engine-size, dtype: int64

False    205
Name: fuel-system, dtype: int64

False    201
True       4
Name: bore, dtype: int64

False    201
True       4
Name: stroke, dtype: int64

False    205
Name: compression-ratio, dtype: int64

F[...]<jupyter_text>Deal with missing dataHow to deal with missing data?    drop data        a. drop the whole row        b. drop the whole column        replace data        a. replace it by mean        b. replace it by frequency        c. replace it based on other functions<jupyter_code>mean_norm_loss = df['normalized-losses'].astype('float').mean()
df['normalized-losses'].replace(np.nan, mean_norm_loss, inplace=True)
mean_bore_loss = df['bore'].astype('float').mean()
df['bore'].replace(np.nan, mean_bore_loss, inplace=True)
mean_stroke_loss = df['stroke'].astype('float').mean()
df['stroke'].replace(np.nan, mean_stroke_loss, inplace=True)
mean_horsepower_loss = df['horsepower'].astype('float').mean()
df['horsepower'].replace(np.nan, mean_horsepower_loss, inplace=True)
mean_rpm_loss = df['peak-rpm'].astype('float').mean()
df['peak-rpm'].replace(np.nan, mean_rpm_loss, inplace=True)
df['num-of-doors'].value_counts()
df['num-of-doors'].replace(np.nan, 'four', inplace=True)
df['peak-rpm'] = df['peak-rpm'].astype('int')
df['horsepower'] = df['horsepower'].astype(int)
df['stroke'] = df['stroke'].astype(float)
df['bore'] = df['bore'].astype(float)
df['normalized-losses'] = df['normalized-losses'].astype(int)
df.info()
df.dropna(subset=['price'], inplace= True)
df.price = df.price.astype(int)
df.info()
df.to_csv('automobiles.csv')<jupyter_output><empty_output><jupyter_text>data standardization<jupyter_code>df['city-mpg'] # needs to be coverted
df['city-kml'] = 235/df['city-mpg']
df.head()<jupyter_output><empty_output><jupyter_text>data normalization or scaling<jupyter_code>df['peak-rpm'] = df['peak-rpm']/ df['peak-rpm'].max()
df['city-kml'] = df['city-kml'] / df['city-kml'].max()
df.head()<jupyter_output><empty_output><jupyter_text>binning<jupyter_code>group_names = ['low','medium','high']
bins = np.linspace(df.horsepower.min(), df.horsepower.max(),4 )
df['hp-binned'] = pd.cut(df.horsepower,bins, labels=group_names,include_lowest=True)
df['hp-binned'].value_counts()
df['hp-binned'].value_counts().plot(kind='bar')
df['price'].plot.hist(bins=5)<jupyter_output><empty_output><jupyter_text>indicator variable or dummy variable<jupyter_code>df['fuel-type'].unique()
dummy_var1 = pd.get_dummies(df['fuel-type'])
dummy_var2 = pd.get_dummies(df['make'])
# merge the dummy var in df and drop the orignal col
df = pd.concat([df, dummy_var1[:-1], dummy_var2[:-1]], axis=1)
df.drop(columns=['fuel-type','make'],inplace=True)
df.info()
df['num-of-doors'].nunique() # not neccesssary to convert to dummy if you have 2 values only
df['engine-type'].nunique()
df['num-of-doors'].replace('two',0,inplace=True)
df['num-of-doors'].replace('four',1,inplace=True)
df.head()
pd.get_dummies(df['engine-type'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15875613747954173, 'fraction_numerical': 0.02922609305588029, 'mean_word_length': 5.28193832599119, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '10961', 'license': 'MIT', 'n_tokens_mistral': 1628, 'n_tokens_neox': 1521, 'n_words': 338, 'path': 'data_wrangling.ipynb', 'repo_name': '73Shivam/Machine_Learning', 'size': '4277'}