<jupyter_start><jupyter_text>"House Price Prediction" <NAME>- Final Project Objective:1. Predict the price for a given set of variables.2. Find the dependent variable for housing price and how dependent they are. Depending on the dependency, a buyer on budget can be made aware on the variables to compromise on. Dataset Courtesy - kaggleDataset contains 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa that may influences final price of each home. Import the library<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import scipy
from sklearn import linear_model  
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.preprocessing import PolynomialFeatures
from yellowbrick.target import FeatureCorrelation
from sklearn.preprocessing import scale
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectKBest, f_regression<jupyter_output><empty_output><jupyter_text>Use pandas read_csv function to import the data into a dataframe variable<jupyter_code>df=pd.read_csv('Dataset.csv')<jupyter_output><empty_output><jupyter_text>Data doesnot have missing data entry. So cleaning won't be necessary. Summary statistics<jupyter_code>##collapse_output
df.info()
df.describe(include='all')
# rows and columns of the dataframe
df.shape
#hide
# First 2 rows of the dataframe
df.head(2)<jupyter_output><empty_output><jupyter_text>Sale Price of homes<jupyter_code>#Histogram
df.plot(kind='hist',y='SalePrice')
#Mean price
Col1 = df.loc[:,'SalePrice']
print("Mean of SalePrice is",Col1.mean())
#Median price
print("Median of SalePrice is",Col1.median())
df.plot(kind='hist',y='SalePrice')
plt.axvline(x=Col1.mean(),color="red", label='Mean')
plt.axvline(x=Col1.median(),color="yellow",label='Median')
plt.legend(loc='best')
plt.show()
#Mode price
print("Mode of Sale Price is ",Col1.mode())
#MAD
print("Mean Absolute Deviation(MAD) of SalePrice is",Col1.mad())
#STD
print("Standard Deviation(STD) of SalePrice is",Col1.std())
#interquartile range (the difference of the 75% and 25% quantile)
print("75% quantile is",Col1.quantile(0.75))
print("25% quantile is",Col1.quantile(0.25))
print("Interquartile range (IQR) of SalePrice is",Col1.quantile(0.75) - Col1.quantile(0.25))
#Box plot of Sale Price
Col1.plot(kind='box',y='SalePrice');<jupyter_output><empty_output><jupyter_text>Plotting OverallCondition of the homes in AMes.<jupyter_code>OC = df.loc[:,'OverallCond']
c1=c2=c3=c4=c5=c6=c7=c8=c9=c10=0   
for i in range(0,len(OC)):
    if OC[i]==1:
       c1+=1
    elif OC[i]==2:
       c2+=1
    elif OC[i]==3:
       c3+=1
    elif OC[i]==4:
       c4+=1
    elif OC[i]==5:
       c5+=1
    elif OC[i]==6:
       c6+=1
    elif OC[i]==7:
       c7+=1
    elif OC[i]==8:
       c8+=1
    elif OC[i]==9:
       c9+=1
    elif OC[i]==10:
       c10+=1       
a = pd.DataFrame({'OverallCond': [c1,c2,c3,c4,c5,c6,c7,c8,c9,c10 ]}, index=['Very Poor',
                                                                          'Poor',
                                                                          'Fair',
                                                                          'Below Average',
                                                                          'Average',
                                                                          'Above Average',
                                                                          'Good',
                                                                         'Very Good',
                                                                          'Excellent',
                                                                         'Very Excellent'])<jupyter_output><empty_output><jupyter_text>Pie plot of Overall condition of the house<jupyter_code>plot = a.plot.pie(y='OverallCond', figsize=(10, 10))
#Boxplot of OverallCondition Vs SalesPrice
ax = df.boxplot(by='OverallCond', column='SalePrice',
                           figsize=(10, 10),rot=90)
#Scatterplot of OverallCondition Vs SalesPrice
df.plot.scatter(x='OverallCond',y='SalePrice');
#Hexagonal binning plot of Overall Condition Vs Sale Price
df.plot.hexbin(x='OverallCond',y='SalePrice');
Bldty = df.loc[:,'BldgType']
c1=c2=c3=c4=c5=c6=c7=c8=c9=c10=0   
for i in range(0,len(Bldty)):
    if Bldty[i]=='1Fam':
       c1+=1
    elif Bldty[i]=='2fmCon':
       c2+=1
    elif Bldty[i]=='Duplex':
       c3+=1
    elif Bldty[i]=='TwnhsE':
       c4+=1
    elif Bldty[i]=='Twnhs':
       c5+=1
b = pd.DataFrame({'BldgType': [c1,c2,c3,c4,c5]}, index=['Single-family Detached',
                                                            'Two-family Conversion; originally built as one-family dwelling',
                                                            'Duplex',
                                                            'Townhouse End Unit',
                                                            'Townhouse Inside Unit'])
    
# Types of Building Type       
plot = b.plot.pie(y='BldgType', figsize=(10, 10))
#Boxplot of Building Type Vs SalePrice
ax = df.boxplot(by='BldgType', column='SalePrice',
                           figsize=(10, 10),rot=90)
#Scatterplot of Building Plot Vs SalePrice
df.plot.scatter(x='BldgType',y='SalePrice');
Corr1 =df.MSSubClass.corr(df.SalePrice)
#Corr2 =df.MSZoning.corr(df.SalePrice)
Corr3 =df.LotFrontage.corr(df.SalePrice)
Corr4 =df.LotArea.corr(df.SalePrice)
#Corr5 =df.Street.corr(df.SalePrice)
#Corr6 =df.Alley.corr(df.SalePrice)
#Corr7 =df.LotShape.corr(df.SalePrice)
#Corr8 =df.LandContour.corr(df.SalePrice)
#Corr9 =df.Utilities.corr(df.SalePrice)
Corr17 =df.OverallQual.corr(df.SalePrice)
Corr18 =df.OverallCond.corr(df.SalePrice)
print("Correlation of Type of dwelling with SalePrice is",Corr1)
#print("Correlation of General Zone Classofication with SalePrice is",Corr2)
print("Correlation of Linear feet of street connected to property with SalePrice is",Corr3)
print("Correlation of Lot size in square feet with SalePrice is",Corr4)
#print("Correlation of Type of road access to property with SalePrice is",Corr5)
#print("Correlation of Type of alley access to property with SalePrice is",Corr6)
#print("Correlation of General shape of property with SalePrice is",Corr7)
#print("Correlation of Flatness of the property with SalePrice is",Corr8)
#print("Correlation of Type of utilities available with SalePrice is",Corr9)
#print("Correlation of Lot configuration with SalePrice is",Corr10)
print("Correlation of Quality with SalePrice is",Corr17)
print("Correlation of Overall Condition with SalePrice is",Corr18)<jupyter_output>Correlation of Type of dwelling with SalePrice is -0.08428413512659524
Correlation of Linear feet of street connected to property with SalePrice is 0.3517990965706781
Correlation of Lot size in square feet with SalePrice is 0.2638433538714056
Correlation of Quality with SalePrice is 0.7909816005838052
Correlation of Overall Condition with SalePrice is -0.07785589404867806<jupyter_text>Overall quality of the home positively affects the Sale proce whereas Overall Condition of the home do not positively affect the Sale Price.<jupyter_code>#Boxplot of OverallQuality Vs SalesPrice
ax = df.boxplot(by='OverallQual', column='SalePrice',
                           figsize=(10, 10),rot=90)
#Scatterplot of Overall Quality Vs SalePrice
df.plot.scatter(x='OverallQual',y='SalePrice');
#Hexagonal binning plot of Overall Quality Vs Sale Price
df.plot.hexbin(x='OverallQual',y='SalePrice');<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code># Split the input data into a training set and test set. We use 80% of data for training and 20% for testing.
OverallQual_train=df.loc[:,'OverallQual']
OverallQual_test = OverallQual_train[-292:]
OverallQual_train= OverallQual_train[:-292]
OverallQual_train = OverallQual_train[:,np.newaxis]
OverallQual_test = OverallQual_test[:,np.newaxis]
print("Train Data size",OverallQual_train.shape)
print("Test Data size",OverallQual_test.shape)
print(OverallQual_train)
# Split the output data/ target into a training set and test set. We use 80% of data for training and 20% for testing.
SalePrice_train = Col1[:-292]
SalePrice_test = Col1[-292:]
SalePrice_train
regr = linear_model.LinearRegression()
regr.fit(OverallQual_train,SalePrice_train)
SalePrice_pred = regr.predict(OverallQual_test)
# The coefficients
print("Coefficients: \n", regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(SalePrice_test,SalePrice_pred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(SalePrice_test,SalePrice_pred))<jupyter_output>Coefficients: 
 [45411.99877916]
Mean squared error: 2630182608.95
R squared value: 0.61<jupyter_text>Since R squared value is 0.61, there is a good correlation between Overall Quality and Sale Price.<jupyter_code># Plot the linear fit together with the training data
plt.scatter(OverallQual_train,SalePrice_train, color="black")
plt.plot(OverallQual_test,SalePrice_pred, color="blue", linewidth=3,label='Linear fit')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Since Sales Price is a positive value, linear fit value which is -ve does not appear in the figure. Polynomial Regression<jupyter_code>poly = PolynomialFeatures(degree=3, include_bias=False)
poly.fit(OverallQual_train,SalePrice_train)
poly_linreg_model = linear_model.LinearRegression()
polybmi_train = poly.fit_transform(OverallQual_train)
poly_linreg_model.fit(polybmi_train,SalePrice_train)
polybmi_test = poly.fit_transform(OverallQual_test)
ypred = poly_linreg_model.predict(polybmi_test)
plt.plot(OverallQual_test, ypred, label="Polynomial Fit")
plt.scatter(OverallQual_train,SalePrice_train, color="black")
plt.legend()
plt.show()
# The coefficients
print("Coefficients: \n", poly_linreg_model.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(SalePrice_test,ypred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(SalePrice_test,ypred))
# Split the input data into a training set and test set. We use 80% of data for training and 20% for testing.
GrLivArea_train=df.loc[:,'GrLivArea']
GrLivArea_test = GrLivArea_train[-292:]
GrLivArea_train= GrLivArea_train[:-292]
GrLivArea_train = GrLivArea_train[:,np.newaxis]
GrLivArea_test = GrLivArea_test[:,np.newaxis]
print("Train Data size",GrLivArea_train.shape)
print("Test Data size",GrLivArea_test.shape)
print(GrLivArea_train)
regr.fit(GrLivArea_train,SalePrice_train)
SalePrice_pred = regr.predict(GrLivArea_test)
# The coefficients
print("Coefficients: \n", regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(SalePrice_test,SalePrice_pred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(SalePrice_test,SalePrice_pred))<jupyter_output>Coefficients: 
 [113.84071513]
Mean squared error: 4235182093.21
R squared value: 0.37<jupyter_text>R squared value is 0.37. Correlation is much less than Overall Quality.<jupyter_code># Plot the linear fit together with the training data
plt.scatter(GrLivArea_train,SalePrice_train, color="black")
plt.plot(GrLivArea_test,SalePrice_pred, color="blue", linewidth=3,label='Linear Fit')
plt.legend()
plt.show()
poly.fit(GrLivArea_train,SalePrice_train)
poly_linreg_model = linear_model.LinearRegression()
polybmi_train = poly.fit_transform(GrLivArea_train)
poly_linreg_model.fit(polybmi_train,SalePrice_train)
polybmi_test = poly.fit_transform(GrLivArea_test)
ypred = poly_linreg_model.predict(polybmi_test)
plt.plot(GrLivArea_test, ypred, label="Polynomial Fit")
plt.scatter(GrLivArea_train,SalePrice_train, color="black")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Multiple Regression We need to normalise the data as different variables will have different range of values<jupyter_code>features = df.drop('SalePrice', axis=1)
features
features.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1460 entries, 0 to 1459
Data columns (total 80 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   Id             1460 non-null   int64  
 1   MSSubClass     1460 non-null   int64  
 2   MSZoning       1460 non-null   object 
 3   LotFrontage    1201 non-null   float64
 4   LotArea        1460 non-null   int64  
 5   Street         1460 non-null   object 
 6   Alley          91 non-null     object 
 7   LotShape       1460 non-null   object 
 8   LandContour    1460 non-null   object 
 9   Utilities      1460 non-null   object 
 10  LotConfig      1460 non-null   object 
 11  LandSlope      1460 non-null   object 
 12  Neighborhood   1460 non-null   object 
 13  Condition1     1460 non-null   object 
 14  Condition2     1460 non-null   object 
 15  BldgType       1460 non-null   object 
 16  HouseStyle     1460 non-null   object 
 17  OverallQual    1460 non-null   int64  
 18  OverallC[...]<jupyter_text>Dropping categorical value features<jupyter_code>features = features.select_dtypes(include='number').copy()
features<jupyter_output><empty_output><jupyter_text>Dropping ID number and columns with NA values. Dropping MSSubClass because the numbers do not provide any meaningful value<jupyter_code>features = features.drop(['MSSubClass','Id','LotFrontage','MasVnrArea','GarageYrBlt'],axis=1)
features
visualizer = FeatureCorrelation(labels = list(features.columns))
visualizer.fit(features, Col1)
visualizer.poof()<jupyter_output><empty_output><jupyter_text>K-Best<jupyter_code>Top10 = SelectKBest(f_regression, k=10).fit(features, Col1)
Topcolumns = features.columns[Top10.get_support()]
Topcolumns
myfeatures = features[Topcolumns]
myfeatures.describe()
X = pd.DataFrame(data=scale(myfeatures), columns=myfeatures.columns)
y = Col1
X.describe()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =.2)
regr.fit(X_train, y_train)
y_pred = regr.predict(X_test)
p = pd.DataFrame({'test': y_test, 'Predicted': y_pred})
p.head(292)
# The coefficients
print("Coefficients: \n", regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y_test,y_pred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(y_test,y_pred))
score = regr.score(X_train, y_train)
print('Score: {}'.format(score))
stddevscore = np.sqrt(mean_squared_error(y_test, y_pred))
print('Standard deviation score: {}'.format(stddevscore))<jupyter_output>Coefficients: 
 [27734.03491028  8605.85372476  6025.07061821  7306.65090344
  5986.02059705 23210.33194148 -3750.71537459  1751.70094373
  8551.36667026  3278.52093316]
Mean squared error: 1203578503.15
R squared value: 0.82
Score: 0.7592054817332723
Standard deviation score: 34692.62894549763<jupyter_text>Taking top 4 variables<jupyter_code>Top4 = SelectKBest(f_regression, k=4).fit(features, Col1)
Topcolumns = features.columns[Top4.get_support()]
Topcolumns<jupyter_output><empty_output><jupyter_text>Top Qualities which affect the Sale Price are Overall Quality, Above Ground Living Area square feet, Size of Garage in Car capacity and size of Garage in sq feet.<jupyter_code>myfeatures = features[Topcolumns]
X = pd.DataFrame(data=scale(myfeatures), columns=myfeatures.columns)
y = Col1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =.2)
regr.fit(X_train, y_train)
y_pred = regr.predict(X_test)
p = pd.DataFrame({'test': y_test, 'Predicted': y_pred})
# The coefficients
print("Coefficients: \n", regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y_test,y_pred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(y_test,y_pred))
score = regr.score(X_train, y_train)
print('Score: {}'.format(score))
stddevscore = np.sqrt(mean_squared_error(y_test, y_pred))
print('Standard deviation score: {}'.format(stddevscore))<jupyter_output>Coefficients: 
 [36221.603516   24975.32612471  9738.34288192  9307.78973081]
Mean squared error: 1555170713.55
R squared value: 0.75
Score: 0.7402311619898319
Standard deviation score: 39435.652822669<jupyter_text>Taking all variables<jupyter_code>Top32 = SelectKBest(f_regression, k=32).fit(features, Col1)
Topcolumns = features.columns[Top32.get_support()]
Topcolumns
myfeatures = features[Topcolumns]
X = pd.DataFrame(data=scale(myfeatures), columns=myfeatures.columns)
y = Col1
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =.2)
regr.fit(X_train, y_train)
y_pred = regr.predict(X_test)
p = pd.DataFrame({'test': y_test, 'Predicted': y_pred})
# The coefficients
print("Coefficients: \n", regr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y_test,y_pred))
# R squared value/ coefficient of determination: 1 is perfect prediction
print("R squared value: %.2f" % r2_score(y_test,y_pred))
score = regr.score(X_train, y_train)
print('Score: {}'.format(score))
stddevscore = np.sqrt(mean_squared_error(y_test, y_pred))
print('Standard deviation score: {}'.format(stddevscore))<jupyter_output>Coefficients: 
 [ 4.67848840e+03  2.46343853e+04  5.45552763e+03  9.65622187e+03
  2.97757797e+03 -2.13820341e+16 -7.56270255e+15 -2.07148736e+16
  2.05666553e+16  2.23382359e+15  2.52239640e+15  2.80959212e+14
 -3.03638828e+15  4.30870251e+03 -4.80046179e+00  1.31373033e+03
 -6.77493707e+01 -6.13454954e+03 -5.12031254e+03  7.71832836e+03
  2.20558250e+03  9.00855958e+03  5.31905245e+02  4.16506219e+03
 -9.85024966e+02  3.98409167e+02 -6.24505288e+02  4.22248173e+03
 -1.51831728e+03 -3.79313585e+02  2.32017611e+02 -1.33590800e+03]
Mean squared error: 1203221847.73
R squared value: 0.81
Score: 0.8004505379709943
Standard deviation score: 34687.488345717866<jupyter_text>Prediction was better when all the 33 variables were taken into consideration. Taking top 10 variables also showed similar results. Taking just 4 variables show a worse result.<jupyter_code>plt.plot(y_pred,label="Predicted value")
plt.plot(np.array(y_test),label="Test value")
plt.legend(loc='best')
plt.show()<jupyter_output><empty_output><jupyter_text>From the figure, we can see that test values and Predicted values are similar. Let's do a T- test to check if there ia statistically significant difference between test values and predicted values<jupyter_code>scipy.stats.ttest_ind(y_test, y_pred)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10887096774193548, 'fraction_numerical': 0.06494623655913978, 'mean_word_length': 4.338978185993112, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 35, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '635384', 'license': 'Apache-2.0', 'n_tokens_mistral': 7074, 'n_tokens_neox': 6202, 'n_words': 1519, 'path': '_notebooks/2022-3-16-House Price Prediction.ipynb', 'repo_name': 'AnjuAnnPaul/Demo', 'size': '18607'}