<jupyter_start><jupyter_text>House Prices dataset: Feature EngineeringIn the following cells, we will engineer / pre-process the variables of the House Price Dataset from Kaggle. We will engineer the variables so that we tackle:1. Missing values2. Temporal variables3. Non-Gaussian distributed variables4. Categorical variables: remove rare labels5. Categorical variables: convert strings to numbers5. Standarise the values of the variables to the same range Setting the seedIt is important to note that we are engineering variables and pre-processing data with the idea of deploying the model. Therefore, from now on, for each step that includes some element of randomness, it is extremely important that we **set the seed**. This way, we can obtain reproducibility between our development and production code. Import necessary libraries<jupyter_code>import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import MinMaxScaler

pd.set_option('display.max_columns',None)

import warnings
warnings.simplefilter(action='ignore')<jupyter_output><empty_output><jupyter_text>Load Dataset<jupyter_code>df = pd.read_csv('houseprice.csv')
print(df.shape)
df.head()<jupyter_output>(1460, 81)<jupyter_text>Split Data into Training & Test Set:<jupyter_code>X_train,X_test,y_train,y_test = train_test_split(df,df['SalePrice'],test_size=0.1,random_state=0)
X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>Missing Values: Categorical variables<jupyter_code># make a list of the categorical variables that contain missing values

mis_var = [
    var for var in df.columns
    if X_train[var].isnull().sum() > 0 and X_train[var].dtypes == 'O'
]

# print percentage of missing values per variable
X_train[mis_var].isnull().mean()
X_train[mis_var] = X_train[mis_var].fillna('Missing')
X_test[mis_var] = X_test[mis_var].fillna('Missing')
X_train[mis_var].isnull().sum()
[var for var in mis_var if X_test[var].isnull().sum() > 0]<jupyter_output><empty_output><jupyter_text>Numerical variablesTo deal with missing values in numerical variables, we will:- add a binary missing value indicator variable- and then replace the missing values in the original variable with the mode<jupyter_code># make a list with the numerical variables that contain missing values
mis_var_num = [
    var for var in df.columns
    if X_train[var].isnull().sum() > 0 and X_train[var].dtypes != 'O'
]

# print percentage of missing values per variable
X_train[mis_var_num].isnull().mean()
# replace engineer missing values as we described above

for var in mis_var_num:

    # calculate the mode using the train set
    mode_val = X_train[var].mode()[0]

    # add binary missing indicator (in train and test)
    X_train[var+'_na'] = np.where(X_train[var].isnull(), 1, 0)
    X_test[var+'_na'] = np.where(X_test[var].isnull(), 1, 0)

    # replace missing values by the mode
    # (in train and test)
    X_train[var] = X_train[var].fillna(mode_val)
    X_test[var] = X_test[var].fillna(mode_val)

# check that we have no more missing values in the engineered variables
X_train[mis_var_num].isnull().sum()
[var for var in mis_var_num if X_test[var].isnull().sum() > 0]
# check the binary missing indicator variables

X_train[['LotFrontage_na', 'MasVnrArea_na', 'GarageYrBlt_na']].head()<jupyter_output><empty_output><jupyter_text>Temporal Variables: Capture Elapsed Time<jupyter_code>def elapsed_years(df, var):
    # capture difference between the year variable
    # and the year in which the house was sold
    df[var] = df['YrSold'] - df[var]
    return df
for var in ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']:
    X_train = elapsed_years(X_train, var)
    X_test = elapsed_years(X_test, var)<jupyter_output><empty_output><jupyter_text>Numerical Variable Transformation:<jupyter_code>for var in ['LotFrontage', 'LotArea', '1stFlrSF', 'GrLivArea', 'SalePrice']:
    X_train[var] = np.log(X_train[var])
    X_test[var] = np.log(X_test[var])
# check that test set does not contain null values in the feature engineered variables
[var for var in ['LotFrontage', 'LotArea', '1stFlrSF',
                 'GrLivArea', 'SalePrice'] if X_test[var].isnull().sum() > 0]
[var for var in ['LotFrontage', 'LotArea', '1stFlrSF',
                 'GrLivArea', 'SalePrice'] if X_train[var].isnull().sum() > 0]<jupyter_output><empty_output><jupyter_text>Categorical variables Removing rare labelsFirst, we will group those categories within variables that are present in less than 1% of the observations. That is, all values of categorical variables that are shared by less than 1% of houses, well be replaced by the string "Rare".<jupyter_code>cat_vars = [var for var in X_train.columns if X_train[var].dtype == 'O']
def find_frequent_labels(df, var, rare_perc):
    
    # function finds the labels that are shared by more than
    # a certain % of the houses in the dataset

    df = df.copy()

    tmp = df.groupby(var)['SalePrice'].count() / len(df)

    return tmp[tmp > rare_perc].index


for var in cat_vars:
    
    # find the frequent categories
    frequent_ls = find_frequent_labels(X_train, var, 0.01)
    
    # replace rare categories by the string "Rare"
    X_train[var] = np.where(X_train[var].isin(
        frequent_ls), X_train[var], 'Rare')
    
    X_test[var] = np.where(X_test[var].isin(
        frequent_ls), X_test[var], 'Rare')<jupyter_output><empty_output><jupyter_text>Encoding Categorical Variables<jupyter_code>def replace_categories(train, test, var, target):

    # order the categories in a variable from that with the lowest
    # house sale price, to that with the highest
    ordered_labels = train.groupby([var])[target].mean().sort_values().index

    # create a dictionary of ordered categories to integer values
    ordinal_label = {k: i for i, k in enumerate(ordered_labels, 0)}

    # use the dictionary to replace the categorical strings by integers
    train[var] = train[var].map(ordinal_label)
    test[var] = test[var].map(ordinal_label)
for var in cat_vars:
    replace_categories(X_train, X_test, var, 'SalePrice')
[var for var in X_train.columns if X_train[var].isnull().sum() > 0]
[var for var in X_test.columns if X_test[var].isnull().sum() > 0]
def analyse_vars(df, var):
    
    # function plots median house sale price per encoded
    # category
    
    df = df.copy()
    df.groupby(var)['SalePrice'].median().plot.bar()
    plt.title(var)
    plt.ylabel('SalePrice')
    plt.show()
    
for var in cat_vars:
    analyse_vars(X_train, var)<jupyter_output><empty_output><jupyter_text>Feature ScalingFor use in linear models, features need to be either scaled or normalised.<jupyter_code>train_vars = [var for var in X_train.columns if var not in ['Id', 'SalePrice']]

# count number of variables
len(train_vars)
# create scaler
scaler = MinMaxScaler()

#  fit  the scaler to the train set
scaler.fit(X_train[train_vars]) 

# transform the train and test set
X_train[train_vars] = scaler.transform(X_train[train_vars])

X_test[train_vars] = scaler.transform(X_test[train_vars])
X_train.head()
X_train.to_csv('xtrain.csv', index=False)
X_test.to_csv('xtest.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1062215477996965, 'fraction_numerical': 0.005104152296868534, 'mean_word_length': 5.036636136552873, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '252557', 'license': 'MIT', 'n_tokens_mistral': 2434, 'n_tokens_neox': 2352, 'n_words': 707, 'path': 'Feature-Engineering.ipynb', 'repo_name': 'souparnabose99/Lasso-Regress-Prod-Code', 'size': '7249'}