<jupyter_start><jupyter_text>House Prices: Advanced Regression Techniques (Part 1 - Exploration)In this notebook we apply linear regression to some data from a Kaggle. The notebook is divided into two sections. First we perform some in-depth data exploration and pre-processing, next we build the actual models.https://www.kaggle.com/c/house-prices-advanced-regression-techniquesThis notebook is derived from other existing notebooks by other authors, like,- https://www.kaggle.com/serigne/stacked-regressions-top-4-on-leaderboardFirst we import some of the libraries we will need:<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib

import matplotlib.pyplot as plt
from scipy import stats
from scipy.stats import skew
from scipy.stats import norm
from scipy.stats.stats import pearsonr

%config InlineBackend.figure_format = 'retina' #set 'png' here when working on notebook
%matplotlib inline
all_data = pd.read_csv("HousePricesInputVariables.csv")
all_data.head()<jupyter_output><empty_output><jupyter_text>Missing ValuesLet's check how many missing values are in the data set and how can we deal with them.<jupyter_code>all_data_na = (all_data.isnull().sum() / len(all_data)) * 100
all_data_na = all_data_na.drop(all_data_na[all_data_na == 0].index).sort_values(ascending=False)[:30]
missing_data = pd.DataFrame({'Missing Ratio' :all_data_na})
missing_data.head(10)
f, ax = plt.subplots(figsize=(8,6))
plt.xticks(rotation='90')
sns.barplot(x=all_data_na.index[:10], y=all_data_na[:10])
plt.xlabel('Features', fontsize=15)
plt.ylabel('Percent of Missing Values', fontsize=15)
plt.title('Top Variables for % of Missing Values', fontsize=15)<jupyter_output><empty_output><jupyter_text>We note that some of the attributes have the vast majority of the values set to unknown. If we apply imputation without any knowledge about the meaning of what a missing value represent, we would end up with attributes set almost completely to the same value. So they would be almost useless. How can we deal with all these missing values? First, we need to ask the domain expert whether some missing values have a special meaning. We actually don't have a domain expert but we have the data description in which we find out that - Alley: Type of alley access to property, NA means "No alley access"- BsmtCond: Evaluates the general condition of the basement, NA means "No Basement"- BsmtExposure: Refers to walkout or garden level walls, NA means "No Basement"- BsmtFinType1: Rating of basement finished area, NA means "No Basement"- BsmtFinType2: Rating of basement finished area (if multiple types), NA means "No Basement"- FireplaceQu: Fireplace quality, NA means "No Fireplace"- Functional: data description says NA means typical- GarageType: Garage location, NA means "No Garage"- GarageFinish: Interior finish of the garage, NA means "No Garage"- GarageQual: Garage quality, NA means "No Garage"- GarageCond: Garage condition, NA means "No Garage"- PoolQC: Pool quality, NA means "No Pool"- Fence: Fence quality, NA means "No Fence"- MiscFeature: Miscellaneous feature not covered in other categories, NA means "None"Accordingly, we need to keep this information into account when dealing with the missing values of these variables.  Impute Categorical Values with Known MeaningLet's impute the missing values for these attributes<jupyter_code>all_data["Alley"] = all_data["Alley"].fillna("None")

# for all basement features a missing value means that there is no basement
for col in ('BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2'):
    all_data[col] = all_data[col].fillna('None')

all_data["FireplaceQu"] = all_data["FireplaceQu"].fillna("None")

all_data["Functional"] = all_data["Functional"].fillna("Typ")

for col in ('GarageType', 'GarageFinish', 'GarageQual', 'GarageCond'):
    all_data[col] = all_data[col].fillna('None')

all_data["PoolQC"] = all_data["PoolQC"].fillna("None")

all_data["Fence"] = all_data["Fence"].fillna("None")

all_data["MiscFeature"] = all_data["MiscFeature"].fillna("None")<jupyter_output><empty_output><jupyter_text>We can also deal with other numerical variables and use some heuristic to impute them. For instance, for __LotFrontage__, since the area of each street connected to the house property most likely have a similar area to other houses in its neighborhood , we can fill in missing values by the median LotFrontage of the neighborhood.<jupyter_code>all_data["LotFrontage"] = all_data.groupby("Neighborhood")["LotFrontage"].transform(
    lambda x: x.fillna(x.median()))<jupyter_output><empty_output><jupyter_text>We can also set the garage year, area and number of cars to zero for missing values since this means that there is no garage.<jupyter_code>for col in ('GarageYrBlt', 'GarageArea', 'GarageCars'):
    all_data[col] = all_data[col].fillna(0)<jupyter_output><empty_output><jupyter_text>And we can do the same for basement measures.<jupyter_code>for col in ('BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF','TotalBsmtSF', 'BsmtFullBath', 'BsmtHalfBath'):
    all_data[col] = all_data[col].fillna(0)<jupyter_output><empty_output><jupyter_text>When __MasVnrArea__ and __MasVnrType__ are missing, it most likely means no masonry veneer for these houses. We can fill 0 for the area and None for the type.<jupyter_code>all_data["MasVnrType"] = all_data["MasVnrType"].fillna("None")
all_data["MasVnrArea"] = all_data["MasVnrArea"].fillna(0)<jupyter_output><empty_output><jupyter_text>For __MSZoning__ (the general zoning classification), 'RL' is by far the most common value. So we can fill in missing values with 'RL'<jupyter_code>all_data['MSZoning'] = all_data['MSZoning'].fillna(all_data['MSZoning'].mode()[0])<jupyter_output><empty_output><jupyter_text>__Utilities__ has all records are "AllPub", except for one "NoSeWa" and 2 missing values. Since the house with 'NoSewa' is in the training set, this feature won't help to predict labels in the test set. We can then safely remove it.<jupyter_code>all_data = all_data.drop(['Utilities'], axis=1)<jupyter_output><empty_output><jupyter_text>__KitchenQual__ has only one missing value, and same as Electrical, we set it to the most frequent values (that is 'TA')<jupyter_code>all_data['KitchenQual'] = all_data['KitchenQual'].fillna(all_data['KitchenQual'].mode()[0])<jupyter_output><empty_output><jupyter_text>__Exterior1st__ and __Exterior2nd__ have only one missing value. We will use the mode (the most common value)<jupyter_code>all_data['Exterior1st'] = all_data['Exterior1st'].fillna(all_data['Exterior1st'].mode()[0])
all_data['Exterior2nd'] = all_data['Exterior2nd'].fillna(all_data['Exterior2nd'].mode()[0])<jupyter_output><empty_output><jupyter_text>For __SaleType__ we can use the most frequent value (the mode) which correspond to "WD"<jupyter_code>all_data['SaleType'] = all_data['SaleType'].fillna(all_data['SaleType'].mode()[0])<jupyter_output><empty_output><jupyter_text>For __MSSubClass__ a missing value most likely means No building class. We can replace missing values with None<jupyter_code>all_data['MSSubClass'] = all_data['MSSubClass'].fillna("None")<jupyter_output><empty_output><jupyter_text>__Electrical__ has one missing value. Since this feature has mostly 'SBrkr', we can set that for the missing value.<jupyter_code>all_data['Electrical'] = all_data['Electrical'].fillna(all_data['Electrical'].mode()[0])<jupyter_output><empty_output><jupyter_text>Anymore missing?<jupyter_code>all_data_na = (all_data.isnull().sum() / len(all_data)) * 100
all_data_na = all_data_na.drop(all_data_na[all_data_na == 0].index).sort_values(ascending=False)[:30]
missing_data = pd.DataFrame({'Missing Ratio' :all_data_na})
missing_data.head(10)<jupyter_output><empty_output><jupyter_text>No more missing values! What would have happened if we did not use or did not have the data description?  Distribution of Numerical VariablesWe now explore the distribution of numerical variables. As we did for the class, we will apply the log1p function to all the skewed numerical variables.<jupyter_code># take the numerical features
numeric_feats = all_data.dtypes[all_data.dtypes != "object"].index

# compute the skewness but only for non missing variables (we already imputed them but just in case ...)
skewed_feats = all_data[numeric_feats].apply(lambda x: skew(x.dropna()))
#We compute skewness, the higher this value the less symmetric is the distribution 
skewness = pd.DataFrame({"Variable":skewed_feats.index, "Skewness":skewed_feats.data})
# select the variables with a skewness above a certain threshold
skewness = skewness.sort_values('Skewness', ascending=[0])

f, ax = plt.subplots(figsize=(8,6))
plt.xticks(rotation='90')
sns.barplot(x=skewness['Variable'], y=skewness['Skewness'])
plt.ylim(0,25)
plt.xlabel('Numerical Variables', fontsize=15)
plt.ylabel('Skewness', fontsize=15)
plt.title('', fontsize=15)<jupyter_output><empty_output><jupyter_text>Usually if we have a skewed distribution is to use a logarithmic transformation(loglp) to all the variables with a skewness above a certain threshold (0.75). Then, replot the skewness of attributes. Note that to have a fair comparison the two plots should have the same scale.<jupyter_code>skewed_feats = skewed_feats[skewed_feats > 0.75]
all_data[skewed_feats.index] = np.log1p(all_data[skewed_feats.index])
# compute the skewness but only for non missing variables (we already imputed them but just in case ...)
skewed_feats = all_data[numeric_feats].apply(lambda x: skew(x.dropna()))
skewness_new = pd.DataFrame({"Variable":skewed_feats.index, "Skewness":skewed_feats.data})
# select the variables with a skewness above a certain threshold

skewness_new = skewness_new.sort_values('Skewness', ascending=[0])

f, ax = plt.subplots(figsize=(8,6))
plt.xticks(rotation='90')
sns.barplot(x=skewness_new['Variable'], y=skewness_new['Skewness'])
plt.ylim(0,25)
plt.xlabel('Numerical Variables', fontsize=15)
plt.ylabel('Skewness', fontsize=15)
plt.title('', fontsize=15)<jupyter_output>/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:3: FutureWarning: Series.data is deprecated and will be removed in a future version
  This is separate from the ipykernel package so we can avoid doing imports until<jupyter_text>We can see that it reduced the skewness, now it is more symetric !!  Correlation AnalysisWe can finally perform some correlation analysis.<jupyter_code>corrmat = all_data.corr()
plt.figure(figsize=(12,9))
sns.heatmap(corrmat, vmax=0.9, square=True);<jupyter_output><empty_output><jupyter_text>Check the cluster map. We can see from the dendrogram that there are 2 groups<jupyter_code>import seaborn as sns
plt.figure(figsize=(12,9))
sns.clustermap(corrmat, vmax=0.9, square=True);<jupyter_output><empty_output><jupyter_text>Or we can further analyze the distribution of some variables.  One Hot EncodingWe now generate the one hot encoding for all the categorical variables. Pandas has the function __get_dummies__ that generates the binary variables for all the categorical variables<jupyter_code>print("Number of Variables before OHE: "+str(all_data.shape[1]))
all_data = pd.get_dummies(all_data)
print("Number of Variables after OHE: "+str(all_data.shape[1]))<jupyter_output>Number of Variables after OHE: 300<jupyter_text>Saving the Preprocessed DataWe create the matrices to be used for computing the models and also save the cleaned data so that we can avoid repeating the process.<jupyter_code>all_data.to_csv('HousePricesInputVariablesCleaned.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10435461485079806, 'fraction_numerical': 0.009802220680083275, 'mean_word_length': 7.034146341463415, 'pattern_counts': {'":': 4, '<': 83, '<?xml version=': 0, '>': 84, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '427508', 'license': 'MIT', 'n_tokens_mistral': 3824, 'n_tokens_neox': 3651, 'n_words': 1136, 'path': 'HousePrices.ipynb', 'repo_name': 'andherrera/Notebooks_data_exploration', 'size': '11528'}