<jupyter_start><jupyter_text>Final Project SubmissionPlease fill out:* Student name: <NAME>* Student pace:Full time* Scheduled project review date/time: 11/7/2019* Instructor name: Rafael* Blog post URL:  Introduction Investigating the factors that drive housing prices  Loading up libraries<jupyter_code># Importing libraries
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns
import numpy as np
%matplotlib inline
inline_rc = dict(mpl.rcParams)

plt.style.use('seaborn')

import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy.stats import normaltest as normtest # D'Agostino and Pearson's omnibus test

from collections import Counter
from sklearn.preprocessing import RobustScaler
from sklearn.preprocessing import MinMaxScaler
%ls
# import and getting basic info about the data
df = pd.read_csv('kc_house_data.csv')
display(df.head())
#df.info()
# Ignore pink warnings 
import warnings
warnings.filterwarnings('ignore')

# Allow for a large # of columns
pd.set_option('display.max_columns',0)
# pd.set_option('display.max_rows','')
#getting basic info about the data
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 21597 entries, 0 to 21596
Data columns (total 21 columns):
id               21597 non-null int64
date             21597 non-null object
price            21597 non-null float64
bedrooms         21597 non-null int64
bathrooms        21597 non-null float64
sqft_living      21597 non-null int64
sqft_lot         21597 non-null int64
floors           21597 non-null float64
waterfront       19221 non-null float64
view             21534 non-null float64
condition        21597 non-null int64
grade            21597 non-null int64
sqft_above       21597 non-null int64
sqft_basement    21597 non-null object
yr_built         21597 non-null int64
yr_renovated     17755 non-null float64
zipcode          21597 non-null int64
lat              21597 non-null float64
long             21597 non-null float64
sqft_living15    21597 non-null int64
sqft_lot15       21597 non-null int64
dtypes: float64(8), int64(11), object(2)
memory usage: 3.5+ MB<jupyter_text>My data contains 21,597 entries. From the info, you can tell that we have both numerical and categorical data. The 3rd column  is what we want to predict (Price)  Scrub<jupyter_code># getting the 5 number summary of the the dataset 
df.describe().round(5)
# 'id', and 'date' colunms are not needed, so i will drop them right off the back
#dropping columns id and date
data = df.drop(['id', 'date'], axis = 1)
data.head()
#Visualizing the data
sns.pairplot(data, x_vars=["sqft_above", "sqft_living", "sqft_lot", "sqft_basement"], y_vars=["price"], hue = "bedrooms")
#checking to see which house is the most common(bedroom wise)
data['bedrooms'].value_counts().plot(kind='bar')
plt.title('Number of Bedrooms')
plt.xlabel('Bedrooms')
plt.ylabel('Count')
sns.despine
#Visualizing the data
sns.pairplot(data, 
             x_vars=["lat", "long", "bathrooms", "grade"], 
             y_vars=["price"],
             hue = "view")
             
# Using violin plot to plot View feature
sns.violinplot(data=df, x = 'view', y = 'price')
# Using violin plot to plot floors feature
sns.violinplot(data=df, x = 'floors', y = 'price')
# Using violin plot to plot waterfront feature
sns.violinplot(data=df, x = 'waterfront', y = 'price')
#checking the shape of the dataset
data.shape
#finding Null values
data.isna().any()
#Number of NaNs the in KC house dataset by column
data.isna().sum()
# find duplicates
duplicates = data[data.duplicated()]
print(len(duplicates))
# dropping duplicate values 
data.drop_duplicates(keep='first',inplace=True) 
#check for duplicates
duplicates = data[data.duplicated()]
print(len(duplicates))
# checking for missing values with placeholders
for col in data.columns:
    print(col, '\n', df[col].value_counts(normalize=True).head(10), '\n\n')<jupyter_output>price 
 350000.0    0.007964
450000.0    0.007964
550000.0    0.007362
500000.0    0.007038
425000.0    0.006945
325000.0    0.006853
400000.0    0.006714
375000.0    0.006390
300000.0    0.006158
525000.0    0.006066
Name: price, dtype: float64 


bedrooms 
 3     0.454878
4     0.318655
2     0.127796
5     0.074131
6     0.012594
1     0.009075
7     0.001760
8     0.000602
9     0.000278
10    0.000139
Name: bedrooms, dtype: float64 


bathrooms 
 2.50    0.248970
1.00    0.178312
1.75    0.141131
2.25    0.094782
2.00    0.089364
1.50    0.066907
2.75    0.054869
3.00    0.034866
3.50    0.033847
3.25    0.027272
Name: bathrooms, dtype: float64 


sqft_living 
 1300    0.006390
1400    0.006251
1440    0.006158
1660    0.005973
1010    0.005973
1800    0.005973
1820    0.005927
1480    0.005788
1720    0.005788
1540    0.005742
Name: sqft_living, dtype: float64 


sqft_lot 
 5000    0.016576
6000    0.013428
4000    0.011622
7200    0.010187
7500    0.005510
4800    0.005510
4500 [...]<jupyter_text>sqft_basement has a ? for roughly 2%<jupyter_code># replace ? with nan
data.replace(['?',], np.nan, inplace=True)
#checking to make sure ? is replaced with nan
for col in data.columns:
    print(col, '\n', data[col].value_counts(normalize=True).head(10), '\n\n')
df = data.drop(['waterfront', 'view', 'yr_renovated'], axis=1)
df.head()
# Dropping Rows That Contain Null Values and checking for null values
data = df.dropna()
data.isna().sum()
#checking the shape of my data
data.shape
#checking the data
data.head(10)<jupyter_output><empty_output><jupyter_text>multicollinearity check<jupyter_code>#checking for multicolinearity of features
data.corr()
# checking for correlation greater than .75
abs(data.corr()) > 0.75
# Generating Heatmap
corrmat = data.corr()
k = 18 #number of variables for heatmap
cols = corrmat.nlargest(k, 'price')['price'].index
cm = np.corrcoef(data[cols].values.T)
sns.set(font_scale=1.25)
f, ax = plt.subplots(figsize=(12, 9))
hm = sns.heatmap(cm, cbar=True, annot=True, square=True, fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values)
plt.show()<jupyter_output><empty_output><jupyter_text>after looking at heat map i removed sqft-living also removed lat, long, zipcode, sqft_basement since i dont plan to use them.<jupyter_code>#dropping sqft_living due to high correlation, and lat, long, zipcode, sqft_basement
data = data.drop(['sqft_living', 'lat', 'long', 'zipcode', 'sqft_basement'], axis=1)
data.head(12)<jupyter_output><empty_output><jupyter_text>Explore<jupyter_code>#checking for distribution
data.hist(figsize=(20,20));
# bin the year built
bins_yr_built = pd.cut(x =data['yr_built'], bins=[1889, 1930,1960,1990,2015] )
# Using pd.cut() returns unordered categories. Transform this to ordered categories 
bins_yr_built =  bins_yr_built.cat.as_ordered()
bins_yr_built.head()
 #Inspect the result
bins_yr_built.value_counts().plot(kind='bar')
#inserting binned year built into dataset
data['yr_built'] = bins_yr_built
data['yr_built'] = data['yr_built'].cat.codes
data.head(10)<jupyter_output><empty_output><jupyter_text>Scaling<jupyter_code># scaling 
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

cols = data.drop('price', axis=1).columns

#data_clean = pd.DataFrame(data['price'])
for col in cols:
    var =scaler.fit_transform(data[col].values.reshape(-1, 1))
    data[col]= var.flatten()
data.describe().round(3)
    
#getting data betweem -3 and 3 tot take care of the outliers
df = data.drop('price', axis=1)[np.abs(data) < 3]
df['price'] = data['price']
df.head(30)
# dropping the nan values
df = df.dropna()
#checking the data
df.head()
df.describe().round(3)
#selecting data excluding price
data_pred = df.iloc[:,:10]
data_pred.head()
#checking histogram
data_pred.hist(figsize = [10,12]);
# dropping sqft_lot since i dont plan to use it in the
#data_pred = data_pred.drop('sqft_lot', axis=1)<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>price = df['price']
data_ols = pd.concat([price, data_pred], axis=1)
data_ols.head(10)
# importing statsmodels
import statsmodels.api as sm
from statsmodels.formula.api import ols
#running model formula
outcome = 'price'
predictors = data_ols.drop(['price', 'condition'], axis=1)
pred_sum = "+".join(predictors.columns)
formula = outcome + "~" + pred_sum
# getting the summy of the model
model = ols(formula= formula, data=data_ols).fit()
model.summary()
#calling the parameters
coef = model.params
coef<jupyter_output><empty_output><jupyter_text>Linear regression using Scikit learn<jupyter_code>from sklearn.linear_model import LinearRegression
y = data_ols['price']
linreg = LinearRegression()
linreg.fit(predictors, y)
# coefficients
linreg.coef_
# intercept
linreg.intercept_<jupyter_output><empty_output><jupyter_text>scatter plot and regression line for the model<jupyter_code>Sqft_living15 = data_ols['sqft_living15'].values
Price = data_ols['price'].values
#regression line and scatter plot for price vs sqft_living15
Sqft_living15 = data_ols['sqft_living15'].values
Price = data_ols['price'].values

sns.regplot(x= Sqft_living15, y= Price)

plt.title("Price vs Sqft Living for neigboring houses")
plt.ylabel('Price')
plt.xlabel("sqft_living15")
#regression line and scatter plot for price vs sqft_above
Sqft_above = data_ols['sqft_above'].values
Price = data_ols['price'].values

sns.regplot(x= Sqft_above, y= Price)

plt.title("Price vs Sqft above ground")
plt.ylabel('Price')
plt.xlabel("sqft above ground")

#regression line and scatter plot for price vs grade
Grade = data_ols['grade'].values
Price = data_ols['price'].values

sns.regplot(x= Grade, y= Price)

plt.title("Price vs Grade")
plt.ylabel('Price')
plt.xlabel("Grade")
#scatter plot for price vs bethrooms
Bathrooms = data_ols['bathrooms'].values
Price = data_ols['price'].values

sns.regplot(x= Bathrooms, y= Price)

plt.title("Price vs Bathrooms")
plt.ylabel('Price')
plt.xlabel("bathrooms")<jupyter_output><empty_output><jupyter_text>QQplot<jupyter_code>resid1 = model.resid
fig = sm.graphics.qqplot(resid1, dist=stats.norm, line='45', fit=True)
# From the QQplot it is obvious that there is lighter mass on the top left end tail indicationg outliers
# From the model the features that affect price significatly are sqft_living15, sqft_above, bathrooms, condition and grade.
# as these features increase so does the prices of the house
#Conclusion is that for the prices to increase we will need to renovate the house by increasing the square feet living of the house and encourage neighbors to do same
#increase in the number of bathrooms will also increase the price of the house<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11138310893512852, 'fraction_numerical': 0.07447509650692025, 'mean_word_length': 4.69544235924933, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '158713', 'license': 'RSA-MD', 'n_tokens_mistral': 4208, 'n_tokens_neox': 3638, 'n_words': 1057, 'path': 'student.ipynb', 'repo_name': 'Dantarshi/dsc-v2-mod1-final-project-online-ds-ft-100719', 'size': '10626'}