<jupyter_start><jupyter_text>Import Data<jupyter_code>gdurltitanic = 'https://drive.google.com/file/d/15tUbXaTIktuLzNjTq6euUHqBpGNuz7u6/view?usp=sharing'
gdtitanic = 'https://drive.google.com/uc?export=download&id='+gdurltitanic.split('/')[-2]
gdurlhousing = 'https://drive.google.com/file/d/11bQwRc8TIPWbLXwnElIOf_NygTCwomwa/view?usp=sharing'
gdhousing = 'https://drive.google.com/uc?export=download&id='+gdurlhousing.split('/')[-2]
gdmercedes = 'https://drive.google.com/uc?export=download&id='+'https://drive.google.com/file/d/1cgfq8R82cLL1xJYbh9jA_qbHihThzYHg/view?usp=sharing'.split('/')[-2]
githousing = 'https://raw.githubusercontent.com/sjcorp/notebooks/master/datasets/housing/train.csv'<jupyter_output><empty_output><jupyter_text>Lifecycle of a Data Science Project1. Data Collection - Company Databases, 3rd Party APIs, Surveys2. Feature Engineering - Handling Missing Values3. Feature Engineering - Encoding4. Feature Engineering - Transformation5. Feature Selection6. Model Training  Handling Missing ValuesWhat are the different types of missing data?- MCAR: Missing completely at Random; in such cases disregarding those cases would not bias the inferences made.- MNAR: Missing not at Random- Missing at RandomTwo types of data could be missing:- Continuous Data- Categorical Data  MNARHere Cabin and Age have a lot of NAN values and they are not missing at random. Its because data was collected post the accident.<jupyter_code># Random Sample Imputation

import pandas as pd
df = pd.read_csv(gdtitanic)
df.head()
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>MCAR<jupyter_code># Cabin value missing for people that survived
import numpy as np
df['cabin_null']=np.where(df['Cabin'].isnull(),1,0)
df['cabin_null'].mean()
df.groupby(['Survived'])['cabin_null'].mean()<jupyter_output><empty_output><jupyter_text>Missing at Random- Men not quoting Salary- Women not quoting Age  Techniques of Handling Missing Values1. Mean/Median/Mode replacement2. Random Sample Imputation3. Capturing NAN Values with a New Feature4. End of Distribution Imputation5. Arbitrary Imputation6. Frequent Categories Imputation  A. Continuous Value Imputation  1. Mean Median Mode ImputationWhen?- Data is missing completely at RandomHow?- Replace the NAN with most feequent occurance of variableAdvantages:- Easy to implement- Robust for outliers- Fastest way to complete the datasetDisadvantages:- Distortion to the original variance<jupyter_code># Random Sample Imputation

import pandas as pd
df = pd.read_csv(gdtitanic, usecols = ['Age', 'Fare', 'Survived'])
df.isnull().mean()
def impute_nan(df, variable, median):
  df[variable+'_median'] = df[variable].fillna(median)
median = df.Age.median()
median
impute_nan(df,'Age',median)
df.head()
# Check if it has changed the normal distribution

print(df['Age'].std())
print(df['Age_median'].std())
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.figure()
ax = fig.add_subplot(111)
df['Age'].plot(kind='kde', ax = ax)
df['Age_median'].plot(kind='kde', ax=ax,color='red')
lines, lables = ax.get_legend_handles_labels()
ax.legend(lines, labels, loc='best')<jupyter_output><empty_output><jupyter_text>2. Random Sample ImputationWhen: Used when values are missing completely at RandomHow: Take random observation from the dataset to replace NaN ValuesPros:- Easy to implement- Less distortion in varianceCons:- In every situation, randomness wont work<jupyter_code># Random Sample Imputation

import pandas as pd
df = pd.read_csv(gdtitanic, usecols = ['Age', 'Fare', 'Survived'])
df.head()
df.isnull().sum()
df['Age'].isnull().sum()
df.isnull().mean()
df['Age'].dropna().sample()
df['Age'].dropna().sample(df['Age'].isnull().sum(),random_state = 0)
df[df['Age'].isnull()].index
def impute_nan(df, variable, median):
  df[variable+"_median"] = df[variable].fillna(median)
  df[variable+"_random"] = df[variable]
  ## It will fill up the NA value with the random sample
  random_sample = df[variable].dropna().sample(df[variable].isnull().sum(),random_state=0)
  ## pandas need to have some index in order to merge the dataset
  random_sample.index=df[df[variable].isnull()].index
  df.loc[df[variable].isnull(),variable+'_random']=random_sample
median = df.Age.median()
median
impute_nan(df,"Age",median)
df.head()
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>3. Capturing NAN Values with a New FeatureThis works well when data is not missing completely at random<jupyter_code>df
df = pd.read_csv(gdtitanic, usecols = ['Age', 'Fare', 'Survived'])
df.head()
import numpy as np
df['Age_NAN']=np.where(df['Age'].isnull(),1,0)
df.head()<jupyter_output><empty_output><jupyter_text>4. Arbitrary Value ImputationThis technique was derived from Kaggle CompetitionsIt inconsists of replacing NAN by an Arbitrary ValueArbitrary Values Properties:- It should be more frequently presentAdvantages:- Easy- Captures the importance of missing variablesDisadvantages- Distorts the original distribution of the variable- If missing variables are not important, it may mask the predictive power of the original variable by distorting its distribution- Hard to decide what value to use<jupyter_code>import pandas as pd
df = pd.read_csv(gdtitanic,usecols = ['Age', 'Fare', 'Survived'])
df.head()
def impute_nan(df,variable):
    df[variable+'_zero']=df[variable].fillna(0)
    df[variable+'_hundred']=df[variable].fillna(100)<jupyter_output><empty_output><jupyter_text>5. End of Distribution imputation<jupyter_code>df=pd.read_csv('gdtitanic.csv', usecols=['Age','Fare','Survived'])
df.head()
df.Age.hist(bins=50)
extreme=df.Age.mean()+3*df.Age.std()
import seaborn as sns
sns.boxplot('Age',data=df)
def impute_nan(df,variable,median,extreme):
    df[variable+"_end_distribution"]=df[variable].fillna(extreme)
    df[variable].fillna(median,inplace=True)
impute_nan(df,'Age',df.Age.median(),extreme)
df.head()
df['Age'].hist(bins=50)
df['Age_end_distribution'].hist(bins=50)
sns.boxplot('Age_end_distribution',data=df)<jupyter_output><empty_output><jupyter_text>B. Handling Missing Values for Categorical Variables  1. Frequent Category Imputation (Mode Imputation)Advantages:- Easy to implementDisadvantages:- Will distort the relationship between the most frequent values and output/dependent variable- Since we are using the most frequent labels, it may over-represent them, if there are many NANs<jupyter_code>import pandas as pd
df = pd.read_csv(gdhousing,usecols = ['BsmtQual','FireplaceQu','GarageType','SalePrice'])
df.head()
df.isnull().sum()
df.isnull().mean().sort_values(ascending=True)<jupyter_output><empty_output><jupyter_text>Since BsmtQual & GarageType have very few missing values, we can replace them with the most frequently occuring value<jupyter_code># Compute the frequency for each feature
df.groupby(['BsmtQual'])['BsmtQual'].count().sort_values(ascending=False).plot.bar()
df.groupby(['GarageType'])['GarageType'].count().sort_values(ascending=False).plot.bar()
df.groupby(['FireplaceQu'])['FireplaceQu'].count().sort_values(ascending=False).plot.bar()
df['GarageType'].value_counts().index[0]
df['GarageType'].mode()[0]
## Replacement Function
def impute_nan(df,variable):
    most_frequent_category=df[variable].value_counts().index[0]
    df[variable].fillna(most_frequent_category,inplace=True)
for features in ['BsmtQual','FireplaceQu','GarageType']:
    impute_nan(df,features)
df.isnull().mean()<jupyter_output><empty_output><jupyter_text>2. Adding a Variable to Capture NaNHere we add the 'most frequent category' to the NaNs but at the same time by creating the new category, we are capturing the important of missing values.<jupyter_code>import pandas as pd
df = pd.read_csv(gdhousing,usecols = ['BsmtQual','FireplaceQu','GarageType','SalePrice'])
df.head()
import numpy as np
df['BsmtQual_Var']=np.where(df['BsmtQual'].isnull(),1,0)
df.head()
frequent = df['BsmtQual'].mode()[0]
df['BsmtQual'].fillna(frequent,inplace=True)
df.head()
df['FireplaceQu_Var']=np.where(df['FireplaceQu'].isnull(),1,0)
frequent = df['FireplaceQu'].mode()[0]
df['FireplaceQu'].fillna(frequent,inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>If there are multiple frequent categories, we just replace NaN with a compteley new category<jupyter_code>import pandas as pd
df = pd.read_csv(gdhousing,usecols = ['BsmtQual','FireplaceQu','GarageType','SalePrice'])
print(df.head())
df.shape
def impute_nan(df, variable):
    df[variable+"newvar"]=np.where(df[variable].isnull(),"Missing",df[variable])
for features in ['BsmtQual','FireplaceQu','GarageType']:
    impute_nan(df,features)
df.head()
df = df.drop(['BsmtQual','FireplaceQu','GarageType'],axis=1)
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1379231210556512, 'fraction_numerical': 0.007687894434882387, 'mean_word_length': 7.380769230769231, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '47710', 'license': 'MIT', 'n_tokens_mistral': 3051, 'n_tokens_neox': 2890, 'n_words': 684, 'path': 'ml_fundamentals/01_ml_feature_engineering_missing_values.ipynb', 'repo_name': 'sjcorp/notebooks', 'size': '8715'}