<jupyter_start><jupyter_text>Defining The Question Specifying the Question Hass Consulting Company is a real estate leader with over 25 years of experience. We have been tasked to study the factors that affect housing prices using given information on real estate properties that was collected over the past few months. Understanding the Context HassConsult Limited was formed in 1992. It offers investment-grade developments all over the City. The developments they offer have been conceptualized by their development team to be high quality assets using a data-driven approach in planning and design, that gears property for high rental returns and an escalated rate of capital growth. Metric of Success We will create a model that will allow the company to accurately predict the sale of prices upon being provided with the predictor variables. Experimental Design Our analysis will take the below flow:1.)Defining the question2.)Data Understanding3.)Data Analysis4.)Modelling5.)Conclusion & Recommendation6.)Challenging the Solution Data Preparation Reading the Data<jupyter_code>#Importing the libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn import model_selection
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import ElasticNet
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from math import sqrt
from sklearn.model_selection import KFold
import statsmodels.formula.api as smf

import warnings
warnings.filterwarnings("ignore")
#Loading our dataset
df = pd.read_csv('/content/Independent Project Week 7 - house_data.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Data Cleaning<jupyter_code>#We'll check for appropriate datatypes
df.info()
#We'll check for the number of rows and columns. 
#There are 21613 rows and 20 columns
df.shape
#We'll check for null values
#There are no null/missing values
df.isnull().sum()
#Checking for duplicate records. There are only 3 records
df.duplicated().sum()
#We'll now check for the particular duplicate rows to see if we can drop or keep them. 
#We'll keep the records since they're of different IDs
duplicateRows = df[df.duplicated()]
duplicateRows
df.columns
df.drop(columns= ['id', 'lat', 'long','sqft_living15','sqft_lot15','grade','condition'], axis=1, inplace=True)
df.head(2)
df.columns
df.shape
df.boxplot()
#Usinfg IQR to check for detecting outliers
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
print(IQR)
#Loop method to check for outliers visually using a boxplot
col_names = ['price','bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'floors',
       'waterfront', 'view', 'sqft_above', 'sqft_basement', 'yr_built',
       'yr_renovated', 'zipcode']

fig, ax = plt.subplots(len(col_names), figsize= (12,40))

for i, col_val in enumerate(col_names):
  sns.boxplot(y=df[col_val], ax= ax[i])
  ax[i].set_title('Box plot - {}'.format(col_val), fontsize= 10)
  ax[i].set_xlabel(col_val, fontsize= 8)
plt.show()
#Removing outliers 
df_out = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]
print(df_out.shape)
df.boxplot()<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis Univariate Data Analysis<jupyter_code>df_out.columns
#Most bedrooms were 3 bedroomed properties
df_out['bedrooms'].value_counts().plot(kind='pie')
explode = [0,0.1,0,0]
plt.title('No. of bedrooms');
df_out['bathrooms'].value_counts().plot.bar()
#Getting summary statistics for the data
df_out.describe()
print('Skewness of Year: ' +str(df_out.skew()))<jupyter_output>Skewness of Year: price            0.825225
bedrooms         0.228537
bathrooms       -0.082451
sqft_living      0.583929
sqft_lot         0.587123
floors           0.730735
waterfront       0.000000
view             0.000000
sqft_above       0.975000
sqft_basement    1.365776
yr_built        -0.470344
yr_renovated     0.000000
zipcode          0.361447
dtype: float64<jupyter_text>From the above, with some variables like waterfront and view having a skewness of 0.0 and some like bathrooms having -0.08, generally the data is approximately symettrical.<jupyter_code>print('Kurtosis for the dataframe: ' +str(df_out.kurt()))<jupyter_output>Kurtosis for the dataframe: price            0.305416
bedrooms        -0.297672
bathrooms       -0.805122
sqft_living     -0.095054
sqft_lot         0.530456
floors          -0.294116
waterfront       0.000000
view             0.000000
sqft_above       0.361766
sqft_basement    0.623242
yr_built        -0.643761
yr_renovated     0.000000
zipcode         -0.910866
dtype: float64<jupyter_text>The distribution has a kurtosis of less than 3  it is platykurtic. Compared to a normal distribution, its tails are shorter and thinner, and often its central peak is lower and broader. Bivariate Data Analysis<jupyter_code>#Pearson's heatmap to check correlation between the variables
corr = df_out.corr() 
plt.figure(figsize = (20,10))
sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns, annot=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Multivariate Data Analysis<jupyter_code>df_out.columns
#Since the data is all numerical, we'll go straight to:
from sklearn.preprocessing import LabelEncoder
labelencoder = LabelEncoder()

df_enc = df_out.copy(deep=True)
df_enc['bedrooms']=labelencoder.fit_transform(df_enc['bedrooms'])
df_enc['bathrooms']=labelencoder.fit_transform(df_enc['bathrooms'])
df_enc['sqft_living']=labelencoder.fit_transform(df_enc['sqft_living'])
df_enc['sqft_lot']=labelencoder.fit_transform(df_enc['sqft_lot'])
df_enc['floors']=labelencoder.fit_transform(df_enc['floors'])
df_enc['waterfront']=labelencoder.fit_transform(df_enc['waterfront'])
df_enc['view']=labelencoder.fit_transform(df_enc['view'])
df_enc['zipcode']=labelencoder.fit_transform(df_enc['zipcode'])
df_enc['yr_built']=labelencoder.fit_transform(df_enc['yr_built'])
df_enc['yr_renovated']=labelencoder.fit_transform(df_enc['yr_renovated'])
df_enc['sqft_basement']=labelencoder.fit_transform(df_enc['sqft_basement'])
df_enc.head()<jupyter_output><empty_output><jupyter_text>Linear Discriminant Analysis<jupyter_code>#Separating the target label (price) from the other features

feat = df_enc.iloc[:,1:14]
price_label = df_enc['price']

price_label
#Subject our frame to LDA model

#import LDA method from sklearn library
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

#creating an instance of LDA

lda = LDA()

lda_ = lda.fit(feat, price_label)
lda_feat = lda_.transform(feat)

#Displaying coefficients of the reducted columns
lda_.coef_
#The coefficients are in an array, we'll need to create a dataframe to be able to extract the column names
new_df = pd.DataFrame(index=feat.columns.values, data=lda_.coef_[0].T)
new_df.sort_values(0, ascending=False)<jupyter_output><empty_output><jupyter_text>**LDA Recommendation**From the analysis we can see that the number of floors, square foot of living arear, number of bathrooms and zipcode highly influence the price of properties. Modelling<jupyter_code>df_out.columns<jupyter_output><empty_output><jupyter_text>Multiple Linear Regression<jupyter_code>#Preparing the Data
#Dividing the data into attributes and labels as we did previously.
X = df_out[['bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot', 'floors',
       'waterfront', 'view', 'sqft_above', 'sqft_basement', 'yr_built',
       'yr_renovated', 'zipcode']]
y = df_out['price']

X
#Dividing our data into training and test sets

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
print(X_train.shape); print(X_test.shape)
#Training the Algorithm

#We will first instantiate the algorithm then fit the model into the training set

regressor = LinearRegression()
regressor.fit(X_train, y_train)
#In case of multivariable linear regression, the regression model has to 
#find the most optimal coefficients for all the attributes. 
#To see what coefficients our regression model has chosen, execute the following

coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])
coeff_df

# Making Predictions

# To make pre-dictions on the test data, execute the following

y_pred = regressor.predict(X_test)
# To compare the actual output values for X_test with the predicted values

df2 = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})
df2

# Evaluating the Algorithm

from sklearn import metrics
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
print(r2_score(y_test, y_pred))
#Here we'll make predictions on both the training set & test set and print evaluation metrics RMSE & R2 on both
y_pred_train= regressor.predict(X_train)
print(np.sqrt(mean_squared_error(y_train,y_pred_train)))
print(r2_score(y_train,y_pred_train ))

y_pred_test= regressor.predict(X_test)
print(np.sqrt(mean_squared_error(y_test,y_pred_test))) 
print(r2_score(y_test, y_pred_test))<jupyter_output>144896.17259155487
0.4044039235346619
140896.3625960609
0.4237832595873785<jupyter_text>From the above, the R2 is 40 % on the train data and 42% on the test data. The model doesn't have a good performance and this could be because of presence of outliers that were left. The above model has a below average performance. A good model should be in the range of 0.7-1 where 1 is a perfect fit. Quantile Regression<jupyter_code>#Creating a dataframe for the two variables(independent & dependent), I chose zipcode as it has a major influence on the price of a property
data = df_out[['price','zipcode']].copy()
data.head()
#Finding the regression coefficients for the conditioned median, 0.5 quantile using the Least Absolute Deviation Model
mod = smf.quantreg("zipcode ~ price", data)
res = mod.fit(q=0.5)
print(res.summary())
#Visualizing the Results
#We estimate the quantile regression model for many quantiles between .05 and .95, and compare best fit line from each of these models to Ordinary Least Squares results.
quantiles = np.arange(0.05, 0.96, 0.1)


def fit_model(q):
    res = mod.fit(q=q)
    return [q, res.params["Intercept"], res.params["price"]] + res.conf_int().loc[
        "price"
    ].tolist()


models = [fit_model(x) for x in quantiles]
models = pd.DataFrame(models, columns=['qt', 'intercept', 'x_coef', 'cf_lower_bound', 'cf_upper_bound'])

ols = smf.ols("zipcode ~ price", data).fit()
ols_ci = ols.conf_int().loc["price"].tolist()
ols = dict(
    a=ols.params["Intercept"], b=ols.params["price"], lb=ols_ci[0], ub=ols_ci[1]
)

print(models)
print(ols)
#Plot the changes in the quantile coeffiecients
plt.figure(6)
models.plot(x = 'qt', y = ['x_coef','cf_lower_bound', 'cf_upper_bound'], 
	title = 'Slope for different quantiles', kind ='line', style = ['b-','r--','g--'])
plt.show()<jupyter_output><empty_output><jupyter_text>Linear regression works by selecting coefficients for each independent variable that minimizes a loss function. If the coefficients are too large, it can lead to model over-fitting on the training dataset and it will not generalize well on the unseen data. Next, we'll do regularization which penalizes large coefficients. Below are regularization algorithms used. Ridge Regression Here the modification of the loss function in order to minimize the complexity of the model is done by adding a penalty parameter that is equivalent to the square of the magnitude of the coefficients.<jupyter_code>rr = Ridge(alpha=0.01)
rr.fit(X_train, y_train) 
pred_train_rr= rr.predict(X_train)
print(np.sqrt(mean_squared_error(y_train,pred_train_rr)))
print(r2_score(y_train, pred_train_rr))

pred_test_rr= rr.predict(X_test)
print(np.sqrt(mean_squared_error(y_test,pred_test_rr))) 
print(r2_score(y_test, pred_test_rr))<jupyter_output>144896.17259159865
0.4044039235343021
140896.36729038015
0.423783221191137<jupyter_text>Lasso Regression In Lasso, the loss function is modified to minimize the complexity of the model by limiting the sum of the absolute values of the model coefficient<jupyter_code>model_lasso = Lasso(alpha=0.01)
model_lasso.fit(X_train, y_train) 
pred_train_lasso= model_lasso.predict(X_train)
print(np.sqrt(mean_squared_error(y_train,pred_train_lasso)))
print(r2_score(y_train, pred_train_lasso))

pred_test_lasso= model_lasso.predict(X_test)
print(np.sqrt(mean_squared_error(y_test,pred_test_lasso))) 
print(r2_score(y_test, pred_test_lasso))<jupyter_output>144896.1725915603
0.4044039235346173
140896.36435386405
0.42378324520978083<jupyter_text>Elastic Net Regression Elastic Net Regression combines the properties of both Ridge and Lasso regression. Here, the R2 is the least of all models but by a very negligible margin. Hence it's the worst performing model<jupyter_code>model_enet = ElasticNet(alpha = 0.01)
model_enet.fit(X_train, y_train) 
pred_train_enet= model_enet.predict(X_train)
print(np.sqrt(mean_squared_error(y_train,pred_train_enet)))
print(r2_score(y_train, pred_train_enet))

pred_test_enet= model_enet.predict(X_test)
print(np.sqrt(mean_squared_error(y_test,pred_test_enet)))
print(r2_score(y_test, pred_test_enet))<jupyter_output>144897.82192961586
0.4043903642411174
140927.02250231648
0.42353245576086784<jupyter_text>Cross Validating the Model<jupyter_code>folds = KFold(n_splits=5)

# note that if you have a KFold object, you can figure out how many folds you set up 
# for it using get_n_splits
print('we are using ' +str(folds.get_n_splits(X)) + ' folds')

# We now create and assess 5 models based on the folds we created.
RMSES = [] # We will use this array to keep track of the RSME of each model
count = 1 # This will just help 
for train_index, test_index in folds.split(X):
  print('\nTraining model ' + str(count))
  
  # set up the train and test based on the split determined by KFold
  # With 5 folds, we will end up with 80% of our data in the training set, and 20% in the test set, just as above
  X_train, X_test = X[train_index], X[test_index]
  y_train, y_test = y[train_index], y[test_index]
  
  # fit a model accordingly
  regressor = LinearRegression()  
  regressor.fit(X_train, y_train)
  
  # assess the accuraccy of the model
  y_pred = regressor.predict(X_test)
  
  rmse_value =  np.sqrt(metrics.mean_squared_error(y_test, y_pred))
  RMSES.append(rmse_value)
  
  print('Model ' + str(count) + ' Root Mean Squared Error:',rmse_value)
  count = count + 1<jupyter_output>we are using 5 folds

Training model 1<jupyter_text>The RMSE for our first and fifth model is fairly larger than the other models while the third model's RMSE is way smaller compared to the other models and this could be because of the presence of outliers which the quantile method was not able to remove completely.The below average RMSE  is close to the fourth and second model hence we could go with the fourth model as being most fitting.<jupyter_code>np.mean(RMSES)
from sklearn.model_selection import LeaveOneOut

folds = LeaveOneOut()

# note that if you have a KFold object, you can figure out how many folds you set up 
# for it using get_n_splits
print('we are using ' +str(folds.get_n_splits(X)) + ' folds')

# We now create and assess 5 models based on the folds we created.
RMSES = [] # We will use this array to keep track of the RSME of each model
count = 0 # This will just help 
for train_index, test_index in folds.split(X):
  
  # set up the train and test based on the split determined by KFold
  # With 5 folds, we will end up with 80% of our data in the training set, and 20% in the test set, just as above
 X_train, X_test = X[train_index], X[test_index]
 y_train, y_test = y[train_index], y[test_index]
  
  # fit a model accordingly
regressor = LinearRegression()  
regressor.fit(X_train, y_train)
  
  # assess the accuraccy of the model
y_pred = regressor.predict(X_test)
  
rmse_value =  np.sqrt(metrics.mean_squared_error(y_test, y_pred))
RMSES.append(rmse_value)
  
count = count + 1
print('trained ' + str(count) + ' models' )
np.mean(RMSES)<jupyter_output>we are using 15787 folds
{'fraction_non_alphanumeric': 0.09287812041116006, 'fraction_numerical': 0.03713901125795399, 'mean_word_length': 5.652421652421652, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '551284', 'license': 'MIT', 'n_tokens_mistral': 5883, 'n_tokens_neox': 5352, 'n_words': 1672, 'path': 'HASS_CONSULTING_PROJECT.ipynb', 'repo_name': 'vngeno/HASS-CONSULTING-PROJECT', 'size': '16344'}