<jupyter_start><jupyter_text>Frequent category imputationIn the previous section we discussed the imputation of missing values in numerical variables. Categorical variables may also contain missing values. The methods of imputation share the same principle as those for numerical values.Replacement of missing values by the most frequent category is the equivalent of mean/median imputation. It consists of replacing all occurrences of missing values (NA) within a variable by the most frequent label or category of the variable. AssumptionsReplacement of NA by the most frequent category has the assumption that the data are missing completely at random (MCAR). The rationale is that the value that should have been seen for the missing observation is most likely the most frequent value or category. Advantages- Easy to implement- Fast way of obtaining complete datasets Limitations- Distortion the relation of the most frequent label with other variables within the dataset- May lead to an over-representation of the most frequent label if there is a big number of NA =============================================================================== Real Life example:  Predicting Sale Price of HousesThe problem at hand aims to predict the final sale price of homes based on different explanatory variables describing aspects of residential homes. Predicting house prices is useful to identify fruitful investments, or to determine whether the price advertised for a house is over or underestimated, before making a buying judgment.=============================================================================In the following cells, I will demonstrate NA imputation by most frequent category using the House Price datasets from Kaggle.If you haven't downloaded the dataset yet, in the lecture "Guide to setting up your computer" in section 1, you can find the details on how to do so.<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
% matplotlib inline

# for regression problems
from sklearn.linear_model import LinearRegression, Ridge

# to split and standarize the datasets
from sklearn.model_selection import train_test_split

# to evaluate regression models
from sklearn.metrics import mean_squared_error

import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>House Price dataset<jupyter_code># let's load the dataset with a few columns for the demonstration
cols_to_use = ['BsmtQual', 'FireplaceQu', 'GarageType', 'SalePrice']

data = pd.read_csv('houseprice.csv', usecols=cols_to_use)

# let's inspect the percentage of missing values in each variable
data.isnull().mean().sort_values(ascending=True)<jupyter_output><empty_output><jupyter_text>The variables BsmtQual and GarageType contain only a few observations with NA (2.5 and 5 %, respectively), therefore, replacing those observations with the value of the most frequent category is a good approach. On the other hand, the variable FireplaceQu, contains missing data in  almost half of the observations. Therefore, replacing those with the most frequent category is most likely going to distort the relationship of the most frequent category with the sale price. In addition, if so many observations contain missing values, it has to be for a reason, so we would be better off capturing missingness somehow. In future lectures, I will show how to capture missingness in categorical variables.  Imputation importantImputation should be done over the training set, and then propagated to the test set. This means that the most frequent category has to be selected from the training set, and used to replace NA both in train and test sets.<jupyter_code># let's separate into training and testing set

X_train, X_test, y_train, y_test = train_test_split(data[['BsmtQual', 'FireplaceQu', 'GarageType']],
                                                    data.SalePrice, test_size=0.3,
                                                    random_state=0)
X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>Let's inspect the frequency of the labels in the different variables<jupyter_code>X_train.groupby(['BsmtQual'])['BsmtQual'].count().sort_values(ascending=False).plot.bar()
X_train.groupby(['FireplaceQu'])['FireplaceQu'].count().sort_values(ascending=False).plot.bar()
X_train.groupby(['GarageType'])['GarageType'].count().sort_values(ascending=False).plot.bar()<jupyter_output><empty_output><jupyter_text>The first 2 variables show 2 predominant categories (first 2 plots). The third variable shows one predominant category. By predominant, I mean category that contains the majority of the observations.<jupyter_code># with this command we capture the most frequent label (check output with plot above)
X_train.groupby(['GarageType'])['GarageType'].count().sort_values(ascending=False).index[0]
# let's create a variable to replace NA with the most frequent label
# both in train and test set

def impute_na(df_train, df_test, variable):
    most_frequent_category = df_train.groupby([variable])[variable].count().sort_values(ascending=False).index[0]
    df_train[variable].fillna(most_frequent_category, inplace=True)
    df_test[variable].fillna(most_frequent_category, inplace=True)
# and let's replace the NA
for variable in ['BsmtQual', 'FireplaceQu', 'GarageType',]:
    impute_na(X_train, X_test, variable)
# let's check that NA have been replaced by inspecting the percentage of null values
X_train.isnull().sum()
X_test.isnull().sum()
# let's transform the categories into numbers quickly and dirty so we can use them in scikit-learn
# the below function numbers the labels from 0 to n, n being the number of different labels 
#  within the variable

for col in ['BsmtQual', 'FireplaceQu', 'GarageType',]:
    labels_dict = {k:i for i, k in enumerate(X_train[col].unique(), 0)}
    X_train.loc[:, col] = X_train.loc[:, col].map(labels_dict )
    X_test.loc[:, col] = X_test.loc[:, col].map(labels_dict)
    
X_train.head()<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code># Let's evaluate the performance of Linear Regression

linreg = LinearRegression()
linreg.fit(X_train, y_train)
print('Train set')
pred = linreg.predict(X_train)
print('Linear Regression mse: {}'.format(mean_squared_error(y_train, pred)))
print('Test set')
pred = linreg.predict(X_test)
print('Linear Regression mse: {}'.format(mean_squared_error(y_test, pred)))
print()<jupyter_output>Train set
Linear Regression mse: 5508120196.474013
Test set
Linear Regression mse: 6364449797.852686<jupyter_text>As you can see, imputation by the most frequent category is a straightforward approach to complete datasets to use in machine learning. How did these feature pre-processing affect the original variable?I mentioned previously that the variable FireplaceQu contained a high percentage of missing data. Therefore, very likely, replacing those observations by the most frequent category will affect the distribution of House prices within that label. Let's go ahead and have a look at how this imputation method affects variables when the number of missing values is high.<jupyter_code># let's recreate a training set where data are missing for FireplpaceQu

# we do the split again over the original dataset, with the same seed, so that we obtain the same split

X_train, X_test, y_train, y_test = train_test_split(data[['BsmtQual', 'FireplaceQu', 'GarageType', 'SalePrice']],
                                                    data.SalePrice, test_size=0.3,
                                                    random_state=0)

# and now we check the missing values
X_train.isnull().mean()
# let's inspect the average house price per category /  label within the variable FireplaceQu

X_train.groupby(['FireplaceQu'])['SalePrice'].mean().sort_values(ascending=False).plot.bar()<jupyter_output><empty_output><jupyter_text>And then compare with the average  house price when this variable has missing observations.<jupyter_code>X_train[X_train.FireplaceQu.isnull()]['SalePrice'].mean()<jupyter_output><empty_output><jupyter_text>We can see that the average price for the most frequent category Gd is around 200,000, whereas the average price for houses with missing observations is 140,000.Let's plot the 2 distributions together for a clearer view of the distribution of prices.<jupyter_code>fig = plt.figure()
ax = fig.add_subplot(111)
X_train[X_train.FireplaceQu=='Gd']['SalePrice'].plot(kind='kde', ax=ax)
X_train[X_train.FireplaceQu.isnull()]['SalePrice'].plot(kind='kde', ax=ax, color='red')
lines, labels = ax.get_legend_handles_labels()
labels = ['Gd', 'Missing observations']
ax.legend(lines, labels, loc='best')<jupyter_output><empty_output><jupyter_text>When values are missing for FirePlace, the average house price is around USS 140k. On the other hand, the average house price for the most frequent label (Gd), is over USS 200k. Thus, we can anticipate that replacing the missing observations by the most frequent label Gd is likely going to take the average of the house price down for that label. And this may affect the performance of the algorithm.<jupyter_code># let's check how many observations we have per category
X_train.FireplaceQu.value_counts()
# let's create a copy of the dataframe
temp = X_train.copy()

# fill the missing values with 'Ex'
impute_na(X_train, X_test, 'FireplaceQu')

X_train.FireplaceQu.value_counts()<jupyter_output><empty_output><jupyter_text>The imputation by most frequent category takes the number of houses with the label 'Gd' from 255 in the original dataset, to 733! Therefore, this might have a big impact on the distribution of Sale Price for that category. Let's  have a look.<jupyter_code># let's put all the variables within one new dataframe so we can plot them within one plot
temp = pd.concat([temp[['FireplaceQu', 'SalePrice']], X_train.FireplaceQu], axis=1)
temp.columns = ['FireplaceQu', 'SalePrice', 'Fireplace_imputed']
temp.head()
fig = plt.figure()
ax = fig.add_subplot(111)
temp[temp.FireplaceQu=='Gd']['SalePrice'].plot(kind='kde', ax=ax)
temp[temp.FireplaceQu.isnull()]['SalePrice'].plot(kind='kde', ax=ax, color='red')
temp[temp.Fireplace_imputed=='Gd']['SalePrice'].plot(kind='kde', ax=ax, color='green')
lines, labels = ax.get_legend_handles_labels()
labels = ['Gd', 'Missing observations', 'Gd + missing']
ax.legend(lines, labels, loc='best')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0966967543691185, 'fraction_numerical': 0.007777991165738429, 'mean_word_length': 5.546197360150849, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '652760', 'license': 'MIT', 'n_tokens_mistral': 2943, 'n_tokens_neox': 2795, 'n_words': 1138, 'path': 'feature_engineering/06.1_Frequent_category_imputation.ipynb', 'repo_name': 'Uttam580/basic_ml', 'size': '10414'}