<jupyter_start><jupyter_text><jupyter_code># imports needed for the analysis

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import LabelEncoder

# to supress warnings
import warnings
warnings.filterwarnings('ignore')
# load dataset into a dataframe
df = pd.read_csv("/content/train.csv", encoding='iso-8859-1')
df.head()<jupyter_output><empty_output><jupyter_text>**Lets Perform EDA (Exploratory Data Analysis) first.**<jupyter_code># lets see the total rows and columns of dataset respectivey
df.shape
# lets see statistical summary of dataset
df.describe()
# lets see the data types of each column
df.info()
# separate numerical and categorical variables
numeric_vars = df.columns[df.dtypes != 'object']
categorical_vars = df.columns[df.dtypes == 'object']
# lets see numeric column names
df[numeric_vars]
# lets see categorical column names 
df[categorical_vars]
# lets see the total missing values in each column of type numeric(int and float)
df[numeric_vars].isnull().sum().sort_values(ascending=False)
# lets see the percentage wise missing values in each column of type numeric(int and float)
df[numeric_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100
# lets see the total missing values in each column of type categorical('object' type)
df[categorical_vars].isnull().sum().sort_values(ascending=False)
# lets see the total missing values in each column of type categorical('object' type)
df[categorical_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100<jupyter_output><empty_output><jupyter_text>As we can see, 
* In numerical columns, no columns has a missing value percantage of    greater than 85 % and hence no column dropping.

* In Categorical columns, 3 columns with missing value percentage of greater than 85 % i,e. PoolQC, MiscFeature, Alley have to be dropped.

But, as a treatment to existing missing values, lets see the approach for both numerical and categorical.<jupyter_code># lets drop 3 columns (PoolQC, MiscFeature, Alley) from dataframe
df = df.drop(['PoolQC', 'MiscFeature', 'Alley'], axis=1)
# dataframe now looks like
df.head()
# verify number of columns should be less than 3 than before (81)
df.shape
numeric_vars = df.columns[df.dtypes != 'object']
categorical_vars = df.columns[df.dtypes == 'object']
# lets see the percentage wise missing values in each column of type numeric(int and float)
df[numeric_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100
# lets see the total missing values in each column of type categorical('object' type)
df[categorical_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100<jupyter_output><empty_output><jupyter_text>***Missing value treatment for Numeric values ***

NOTE: we will be looking at outliers in each numeric column to deicde what to impute (i,e. imputing with mean if no outliers, else with mean or IQR based on domain context knowledge)
 1) Missing value treatment for numeric column "LotFrontage"<jupyter_code># lets see outliers in this column
plt.figure(figsize=(10,8))
sns.boxplot(x='LotFrontage', data=df)<jupyter_output><empty_output><jupyter_text>The above boxplot clearly tells us the column 'LotFrontage' has Outliers. So lets Impute Missing Values with IQR value.<jupyter_code># Computing IQR value for the column 'LotFrontage'
df['LotFrontage'].describe()
# IQR value = 21 i,e.(80-59) for 'LotFrontage Column'. lets impute it in place of missing values
# But, IQR is equal to minimun value in the coulumn, also standard deviation is high for this column(24). Considering 25%, 50% and 75% values and 
# the outliers, We can see imputing with Mean/Medain(since both are close enough 70 and 69 respectively) makes more sense than IQR. 
# So lets impute with MEAN(70.05)

impute_val = 70.05
def func(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['LotFrontage'] = df['LotFrontage'].astype(str)
df['LotFrontage'] = df['LotFrontage'].apply(func)
df['LotFrontage'] = df['LotFrontage'].astype(float)
df['LotFrontage'].head()
# lets check for missing values again in 'LotFrontage' column(Supposed to be Zero by now)
df['LotFrontage'].isnull().sum()<jupyter_output><empty_output><jupyter_text>In similar fashion, Lets perform Missing Value treatment for other numerical columns too. 2) Missing value treatment for numeric column "GarageYrBlt"<jupyter_code># check for outliers in this column
plt.figure(figsize=(10,8))
sns.boxplot(x='GarageYrBlt', data=df)
df['GarageYrBlt'].describe()
# seems no outliers in this column, hence imputing with mean value (1978.506164)
impute_val = df['GarageYrBlt'].mean()
def func1(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['GarageYrBlt'] = df['GarageYrBlt'].astype(str)
df['GarageYrBlt'] = df['GarageYrBlt'].apply(func1)
df['GarageYrBlt'] = df['GarageYrBlt'].astype(float)
# check for missing values in "GarageYrBlt" column again (Suppose to be zero)
df['GarageYrBlt'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Since Year should be an Integer, Converting 'GarageYrBlt' to Integer type<jupyter_code>df['GarageYrBlt'] = df['GarageYrBlt'].astype(int)
df['GarageYrBlt'].head()<jupyter_output><empty_output><jupyter_text>3) Missing value treatment for numeric column "MasVnrArea"<jupyter_code>df['MasVnrArea'].value_counts()<jupyter_output><empty_output><jupyter_text>By analysing Data dictionary, 'MasVnrArea' value = 0.0 is valid(not treates as missing) for 'MasVnrType' = None. Hence, we treat only 'NA' as Missing values in this 'MasVnrArea' column.<jupyter_code># check for outliers in this column
plt.figure(figsize=(10,8))
sns.boxplot(x='MasVnrArea', data=df)
# seems outliers are huge in this column, looking at below stats, imputing with IQR does not make sense since 25% (Q1) is zero.
# So, imputing with Mean
df['MasVnrArea'].describe()
impute_val = df['MasVnrArea'].mean()
def func2(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['MasVnrArea']= df['MasVnrArea'].astype(str)
df['MasVnrArea'] = df['MasVnrArea'].apply(func2)
df['MasVnrArea'] = df['MasVnrArea'].astype(float)
df['MasVnrArea'].head()<jupyter_output><empty_output><jupyter_text>Converting type to Integer for column "MasVnrArea" since all values in dataset are of Integer type<jupyter_code>df['MasVnrArea'] = df['MasVnrArea'].astype('int')
df['MasVnrArea'].isnull().sum() # suppose to be zero by now<jupyter_output><empty_output><jupyter_text>**Missing Value treatment for Categorical Variables** Fence            80.753425
FireplaceQu      47.260274
GarageCond        5.547945
GarageQual        5.547945
GarageFinish      5.547945
GarageType        5.547945
BsmtFinType2      2.602740
BsmtExposure      2.602740
BsmtFinType1      2.534247
BsmtQual          2.534247
BsmtCond          2.534247
MasVnrType        0.547945
Electrical        0.068493






 AS we know, We impute  by Mode (most frequently occuring value) in case of Categorical Variables. 1) Imputing for categorical Variable "Fence" By Data description, NA values in Fence column mean No fence for the house. It cannot be treated as a Missing value and hence cannot impute by mode value of the column.

At any cost, during predictions we need these rows saying no fence for arriving at valid insights, hence we keep this information in a different manner, other than NA.<jupyter_code>df['Fence'].describe()
impute_val = str('No_fence')
def func3(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['Fence'] = df['Fence'].astype(str)
df['Fence'] = df['Fence'].apply(func3)

df['Fence'].isnull().sum() # suppose to be zero
df['Fence'].value_counts()<jupyter_output><empty_output><jupyter_text>2) Imputing for categorical column "FireplaceQu"<jupyter_code>df['FireplaceQu'].value_counts()
impute_val = str('No_fireplace')
def func4(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['FireplaceQu'] = df['FireplaceQu'].astype(str)
df['FireplaceQu'] = df['FireplaceQu'].apply(func4)
df['FireplaceQu'].isnull().sum() # supposed to be zero by now
df['FireplaceQu'].value_counts()
df['FireplaceQu'].head()<jupyter_output><empty_output><jupyter_text>3) Imputing value for categorical variable "GarageCond"<jupyter_code>impute_val = str('No Garage')
def func5(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['GarageCond']= df['GarageCond'].astype(str)
df['GarageCond'] = df['GarageCond'].apply(func5)
df['GarageCond'].isnull().sum() # suppose to be zero by now
df['GarageCond'].value_counts()
df['GarageCond'].head()<jupyter_output><empty_output><jupyter_text>4) Imputing for categorical variable "GarageQual"<jupyter_code>impute_val = str('No Garage')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['GarageQual']= df['GarageQual'].astype(str)
df['GarageQual']= df['GarageQual'].apply(func6)
df['GarageQual'].isnull().sum()
df['GarageQual'].value_counts()
df['GarageQual'].head()<jupyter_output><empty_output><jupyter_text>5) Imputing for the categorical variable "GarageFinish"<jupyter_code>impute_val = str('No Garage')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['GarageFinish']= df['GarageFinish'].astype(str)
df['GarageFinish']= df['GarageFinish'].apply(func6)
df['GarageFinish'].isnull().sum()
df['GarageFinish'].value_counts()
df['GarageFinish'].head()<jupyter_output><empty_output><jupyter_text>6) Imputing for the categorical variable "GarageType"<jupyter_code>impute_val = str('No Garage')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['GarageType']= df['GarageType'].astype(str)
df['GarageType']= df['GarageType'].apply(func6)
df['GarageType'].isnull().sum()
df['GarageType'].value_counts()
df['GarageType'].head()<jupyter_output><empty_output><jupyter_text>7) Imputing for the categorical variable "BsmtFinType2"<jupyter_code>impute_val = str('No type2 Basement finish')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['BsmtFinType2']= df['BsmtFinType2'].astype(str)
df['BsmtFinType2']= df['BsmtFinType2'].apply(func6)
df['BsmtFinType2'].isnull().sum()
df['BsmtFinType2'].value_counts()
df['BsmtFinType2'].head()<jupyter_output><empty_output><jupyter_text>8) Imputing for the categorical varaible "BsmtExposure"<jupyter_code>impute_val = str('No Basement')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['BsmtExposure']= df['BsmtExposure'].astype(str)
df['BsmtExposure']= df['BsmtExposure'].apply(func6)
df['BsmtExposure'].isnull().sum()
df['BsmtExposure'].value_counts()
df['BsmtExposure'].head()<jupyter_output><empty_output><jupyter_text>9) Imputing for the categorical varaible "BsmtFinType1"<jupyter_code>impute_val = str('No Basement')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['BsmtFinType1']= df['BsmtFinType1'].astype(str)
df['BsmtFinType1']= df['BsmtFinType1'].apply(func6)
df['BsmtFinType1'].isnull().sum()
df['BsmtFinType1'].value_counts()
df['BsmtFinType1'].head()<jupyter_output><empty_output><jupyter_text>10) Imputing for the categorical varaible "BsmtCond"<jupyter_code>impute_val = str('No Basement')
def func6(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['BsmtCond']= df['BsmtCond'].astype(str)
df['BsmtCond']= df['BsmtCond'].apply(func6)
df['BsmtCond'].isnull().sum()
df['BsmtCond'].value_counts()
df['BsmtCond'].head()<jupyter_output><empty_output><jupyter_text>11)  Imputing for the categorical varaible "MasVnrType" By analysing Data Dictionary and dataset for this column, NA here refers to Missing Value, hence needs to be imputed by Mean.<jupyter_code>df['MasVnrType'].describe()
impute_val = str(df['MasVnrType'].mode())
def func7(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['MasVnrType']= df['MasVnrType'].astype(str)
df['MasVnrType']= df['MasVnrType'].apply(func7)
df['MasVnrType'].isnull().sum()
df['MasVnrType'].value_counts()
df['MasVnrType'].head()<jupyter_output><empty_output><jupyter_text>12) Imputing for the categorical varaible "Electrical" By analysing Data Dictionary and dataset for this column, NA here refers to Missing Value, hence needs to be imputed by Mean.<jupyter_code>impute_val = str(df['Electrical'].mode())
def func7(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['Electrical']= df['Electrical'].astype(str)
df['Electrical']= df['Electrical'].apply(func7)
df['Electrical'].isnull().sum()
df['Electrical'].value_counts()
df['Electrical'].head()<jupyter_output><empty_output><jupyter_text>13) Imputing for the categorical variable "BsmtQual"
 By analysing Data Dictionary, here NA does not refer to Missing values, hence imputing by string "No Basement" to preserve the information.<jupyter_code>impute_val = str('No Basement')
def func8(x):
  if x == 'nan':
    return impute_val
  else:
    return x

df['BsmtQual']= df['BsmtQual'].astype(str)
df['BsmtQual']= df['BsmtQual'].apply(func8)
df['BsmtQual'].isnull().sum()
df['BsmtQual'].value_counts()
df['BsmtQual'].head()
# lets see the percentage wise missing values in each column of type numeric(int and float)
df[numeric_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100
df.shape
# lets see the total missing values in each column of type categorical('object' type)
df[categorical_vars].isnull().sum().sort_values(ascending=False)/len(df) * 100<jupyter_output><empty_output><jupyter_text>Now, All the Missing values are being treated. and the types of each column have been verified with the Data Dictionary, which is good to go ahead. We have many cAtegorical Varibales, whose values are supposed to be in numeric for analyis. Hence converting all categorical variable values to numeric using "LabelEncoder".<jupyter_code>df['SaleCondition']
number = LabelEncoder()
df['SaleCondition'] = number.fit_transform(df['SaleCondition'].astype(str))
df['SaleCondition']
df.shape<jupyter_output><empty_output><jupyter_text>Similarly for rest of categorical variables.<jupyter_code>df['SaleType'] = number.fit_transform(df['SaleType'].astype(str))
df['ExterQual'] = number.fit_transform(df['ExterQual'].astype(str))
df['MasVnrType'] = number.fit_transform(df['MasVnrType'].astype(str))
df['Exterior2nd'] = number.fit_transform(df['Exterior2nd'].astype(str))
df['Exterior1st'] = number.fit_transform(df['Exterior1st'].astype(str))
df['RoofMatl'] = number.fit_transform(df['RoofMatl'].astype(str))
df['RoofStyle'] = number.fit_transform(df['RoofStyle'].astype(str))
df['HouseStyle'] = number.fit_transform(df['HouseStyle'].astype(str))
df['BldgType'] = number.fit_transform(df['BldgType'].astype(str))
df['Condition2'] = number.fit_transform(df['Condition2'].astype(str))
df['Condition1'] = number.fit_transform(df['Condition1'].astype(str))
df['Neighborhood'] = number.fit_transform(df['Neighborhood'].astype(str))
df['LandSlope'] = number.fit_transform(df['LandSlope'].astype(str))
df['LotConfig'] = number.fit_transform(df['LotConfig'].astype(str))
df['Utilities'] = number.fit_transform(df['Utilities'].astype(str))
df['LandContour'] = number.fit_transform(df['LandContour'].astype(str))
df['LotShape'] = number.fit_transform(df['LotShape'].astype(str))
df['Street'] = number.fit_transform(df['Street'].astype(str))
df['ExterCond'] = number.fit_transform(df['ExterCond'].astype(str))
df['Foundation'] = number.fit_transform(df['Foundation'].astype(str))
df['BsmtQual'] = number.fit_transform(df['BsmtQual'].astype(str))
df['Functional'] = number.fit_transform(df['Functional'].astype(str))
df['Fence'] = number.fit_transform(df['Fence'].astype(str))
df['PavedDrive'] = number.fit_transform(df['PavedDrive'].astype(str))
df['GarageCond'] = number.fit_transform(df['GarageCond'].astype(str))
df['GarageQual'] = number.fit_transform(df['GarageQual'].astype(str))
df['GarageFinish'] = number.fit_transform(df['GarageFinish'].astype(str))
df['GarageType'] = number.fit_transform(df['GarageType'].astype(str))
df['FireplaceQu'] = number.fit_transform(df['FireplaceQu'].astype(str))
df['KitchenQual'] = number.fit_transform(df['KitchenQual'].astype(str))
df['BsmtCond'] = number.fit_transform(df['SaleCondition'].astype(str))
df['Electrical'] = number.fit_transform(df['Electrical'].astype(str))
df['CentralAir'] = number.fit_transform(df['CentralAir'].astype(str))
df['HeatingQC'] = number.fit_transform(df['HeatingQC'].astype(str))
df['Heating'] = number.fit_transform(df['Heating'].astype(str))
df['BsmtFinType2'] = number.fit_transform(df['BsmtFinType2'].astype(str))
df['BsmtFinType1'] = number.fit_transform(df['BsmtFinType1'].astype(str))
df['BsmtExposure'] = number.fit_transform(df['BsmtExposure'].astype(str))
df['MSZoning'] = number.fit_transform(df['MSZoning'].astype(str))
df.shape<jupyter_output><empty_output><jupyter_text>Now lets check whether given variables are linearly related with the output variable. Lets check randomly, if any one of input variable is founf to be not linearly related with output variable, then fitting Linear Regression model does not sound good for valueable in-sights. So we have to go for Polynomial regression in such case. (others like Data transformation, no-linear regression may also help)<jupyter_code># scatter plot to visualise data
sns.scatterplot( data = df , x = 'YearBuilt' , y = 'SalePrice')
# scatter plot to visualise data
sns.scatterplot( data = df , x = 'SaleCondition' , y = 'SalePrice')<jupyter_output><empty_output><jupyter_text>Its evident that the predictor variables are not linearly dependent on output variable "Salesprice".<jupyter_code># Before to that, Lets Standardise the input values using Min-Max Scaler
scaler = MinMaxScaler()
df[df.columns] = scaler.fit_transform(df[df.columns])
df.shape<jupyter_output><empty_output><jupyter_text>Visualising again (Should look same as before Scaling)<jupyter_code># scatter plot to visualise data
sns.scatterplot( data = df , x = 'YearBuilt' , y = 'SalePrice')
# scatter plot to visualise data
sns.scatterplot( data = df , x = 'SaleCondition' , y = 'SalePrice')<jupyter_output><empty_output><jupyter_text>Yes. The Plots look exactly Same. Applying Polynomial Regression<jupyter_code># SPlitting the dataset into X and y
columns = df.columns.tolist()
input_vars = columns[:-1]
output_var = columns[-1]
X = df[input_vars].values.reshape(-1,1)
y = df[[output_var]]

# input variables
X
# output variable
y
poly = PolynomialFeatures(3) # Want to generate features with degree less than or equal to
                             # the specified degree
Y = poly.fit_transform(X) # Transform the variable X to 1, X, X^2, X^3
print(Y)
# Building the polynomial regression model with degree 5
# degree=5 # got this number through trial and this is the lowest value which fit the data perfectly
# polyreg5 = PolynomialFeatures(degree)
# X_poly5 = polyreg5.fit_transform(X) # Transform the variable X to 1, X, X^2, X^3, X^4, X^5
# linreg5 = LinearRegression()
# linreg5.fit(X_poly5, y)<jupyter_output><empty_output><jupyter_text>Ridge Regression<jupyter_code># Applying Ridge Regression with varying the hyperparameter 'lambda'

X_seq = np.linspace(X.min(),X.max(),300).reshape(-1,1) # values to be considered for predictor variable
lambdas = [0, 0.001, 0.01, 0.1, 1, 10, 100, 1000] # Higher the value of lambda, 
                                                  # more the regularization
for i in lambdas: # for each lambda we get different model coefficients
    degree = 5 # Degree for polynomial regression - chose 5 since this is the lowest number that gave a perfect fit
    # Creating degree 5 features
    ridgecoef = PolynomialFeatures(degree)
    # Transforming input features to polynomial features (1, x1, x2, x3, x4, x5)    
    X_poly = ridgecoef.fit_transform(X)
    ridgereg = Ridge(alpha = i) # Initialize the Ridge Regression model with a specific lambda
    ridgereg.fit(X_poly, y) # fit the model on the polynomial features
    
    # Plotting the ridge regression model for each lambda
    plt.figure()
    plt.scatter(X,y)
    plt.plot(X_seq,ridgereg.predict(ridgecoef.fit_transform(X_seq)),color="black") # Polynomial Regression
    plt.plot(X_seq,reg.predict(X_seq),color="red") # Linear Regression
    plt.title("Polynomial regression with degree "+str(degree) + " and lambda = " + str(i))
    plt.show()
    
    #Computing the r2 score
    y_pred = ridgereg.predict(ridgecoef.fit_transform(X))
    print("r2 score = " + str(r2_score(y, y_pred))) 
    print(ridgereg.coef_) # model coefficients<jupyter_output><empty_output><jupyter_text>Lasso Regression<jupyter_code># Applying Lasso Regression with varying the hyperparameter 'lambda'

lambdas = [0.001, 0.01, 0.1, 1, 10, 100, 1000]
for i in lambdas:
    degree = 5
    # Creating degree 5 features
    lassocoef = PolynomialFeatures(degree)
    # Transforming input features to polynomial features (1, x1, x2, x3, x4, x5)
    X_poly = lassocoef.fit_transform(X)
    lassoreg = Lasso(alpha = i)
    lassoreg.fit(X_poly, y)
    # Plotting the ridge model
    plt.figure()
    plt.scatter(X,y)
    plt.plot(X_seq,lassoreg.predict(lassocoef.fit_transform(X_seq)),color="black")
    plt.plot(X_seq,reg.predict(X_seq),color="red")
    plt.title("Polynomial regression with degree "+str(degree) + " and lambda = " + str(i))
    plt.show()
    # Compute R^2 
    y_pred = lassoreg.predict(lassocoef.fit_transform(X))
    print("r2 score = " + str(r2_score(y, y_pred)))
    print(lassoreg.coef_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13317191283292978, 'fraction_numerical': 0.015807026360272283, 'mean_word_length': 5.593373493975903, 'pattern_counts': {'":': 0, '<': 125, '<?xml version=': 0, '>': 125, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '471936', 'license': 'Apache-2.0', 'n_tokens_mistral': 7926, 'n_tokens_neox': 7508, 'n_words': 1839, 'path': 'house_price_ML2_Logistic_regression.ipynb', 'repo_name': 'santoshkumarvagga/API', 'size': '21889'}