<jupyter_start><jupyter_text>HASS CONSULTING COMPANY REAL ESTATE SALE PRICE PREDICTION  Defining the Question As a Data Scientist, you work for Hass Consulting Company which is a real estate leader with over 25 years of experience. You have been tasked to study the factors that affect housing prices using the given information on real estate properties that was collected over the past few months. Later onwards, create a model that would allow the company to accurately predict the sale of prices upon being provided with the predictor variables.   Specifying the Question 1. **Study the factors that affect housing prices using information on real estate properties**2. **Model that would allow the company accurately predict the sale of prices**  Defining the metric of success 1. Optimal and reliable prediction Model of the sale of prices2.  80% Accuracy score of the four models that I will use in prediction3. With the least RMSE score achievable  Understanding the Context Hass Consulting Company  is a real estate leader with over 25 years of experience.As a data scientist, I have been tasked to study the factors that affect housing prices using the given information on real estate properties that was collected over the past few months. Later onwards, create a model that would allow the company to accurately predict the sale of prices upon being provided with the predictor variables.   Experimental design taken 1. Perform EDA which involves univariate, bivariate and multivariate analysis2. Check for multicollinearity and plot residual plots3. Build regression models i.ei. Multiple Linear Regressionii. Quantile Regressioniii. Ridge Regressioniv. Lasso Regressionv. Elastic Net RegressionWhile performing your regression analysis, you will be required to perform modeling using the given regression techniques then evaluate their performance. You will be then required to provide your observations and recommendation on the suitability of each of the tested models on their appropriateness of solving the given problem. 4. Challenge your solution by providing insights on how you can make improvements in model improvement.  Appropriateness of Data One  datasets is provided by the consulting company:The dataset and glossary to use for this project can be found here Datasets  Reading the Data  Importing Relevant libraries<jupyter_code>import pandas_profiling as pp
#loading the libraries
import pandas as pd
import numpy as np
import seaborn as sb
from sklearn.preprocessing import LabelEncoder
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split,GridSearchCV,RandomizedSearchCV,KFold,StratifiedKFold,cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.metrics import mean_squared_error,r2_score,accuracy_score,f1_score,classification_report
from sklearn.metrics import confusion_matrix
from sklearn import metrics
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>Loading the housing data and its dictionary<jupyter_code>housing=pd.read_csv("/content/Independent Project Week 7 - house_data.csv")
dictionary=pd.read_excel("/content/dictionary.xlsx")
## Previewing the first five rows
housing.head()
##previewing the last five rows 
housing.tail()
##previewing my dictionary 
dictionary<jupyter_output><empty_output><jupyter_text>Data Exploration<jupyter_code>## Previewing the size of our housing dataframe
housing.shape
###there are 21613 rows and 20 columns.
##previwing our column data types
housing.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 21613 entries, 0 to 21612
Data columns (total 20 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   id             21613 non-null  int64  
 1   price          21613 non-null  float64
 2   bedrooms       21613 non-null  int64  
 3   bathrooms      21613 non-null  float64
 4   sqft_living    21613 non-null  int64  
 5   sqft_lot       21613 non-null  int64  
 6   floors         21613 non-null  float64
 7   waterfront     21613 non-null  int64  
 8   view           21613 non-null  int64  
 9   condition      21613 non-null  int64  
 10  grade          21613 non-null  int64  
 11  sqft_above     21613 non-null  int64  
 12  sqft_basement  21613 non-null  int64  
 13  yr_built       21613 non-null  int64  
 14  yr_renovated   21613 non-null  int64  
 15  zipcode        21613 non-null  int64  
 16  lat            21613 non-null  float64
 17  long           21613 non-null  float64
 18  sqft_l[...]<jupyter_text>Data Cleaning<jupyter_code>##Before doing any data cleaning,,,its a good practice to make a copy of the dataset so that when you make any mistake you can just revert to the original copy
housing_new=housing.copy()
housing_new.head()
##checking for null values
housing_new.isnull().sum()
##Bam!! Our dataframe does not have missing value 
## checking for duplicates
housing_new.duplicated().value_counts()
##With only 3 duplicates, that wont affect our model so we drop the duplicated value..double bam!!


housing_new.drop_duplicates(keep='first',inplace = True)
print("housing", housing_new.shape)
housing_new.duplicated().sum()
## 
## checking for outliers
columns_dict={ 'id':1, 'price':2, 'bedrooms':3, 'bathrooms':4, 'sqft_living':5, 'sqft_lot':6,
       'floors':7, 'waterfront':8, 'view':9, 'condition':10, 'grade':11, 'sqft_above':12,
       'sqft_basement':13, 'yr_built':14, 'yr_renovated':15, 'zipcode':16, 'lat':17, 'long':18,
       'sqft_living15':19, 'sqft_lot15':20}
plt.figure(figsize=(20,30))
# make a boxplot for each numerical column
for variable,i in columns_dict.items():
  plt.subplot(5,4,i)
  plt.boxplot(housing_new[variable])
  plt.title(variable)
plt.show()
##There are outliers in our dataframe but we wont drop them as they play an important role in model prediction
## checking for anomalies in the housing dataset
housing_new.nunique()<jupyter_output><empty_output><jupyter_text>Univariate Analysis<jupyter_code>housing_new.head()
dictionary
housing_new['price'].describe()
#Lets start off by focusing on the price  since its the target variable. 
import seaborn as sns
sns.set(font_scale = 2)
%pylab


figsize(8, 8)
import matplotlib.pyplot as plt
%matplotlib inline

# Set default font size
plt.rcParams['font.size'] = 24
from IPython.core.pylabtools import figsize



# Histogram of the Energy Star Score
plt.style.use('seaborn')
plt.hist(housing_new['price'], color = 'black');
plt.xlabel('Price of a house'); plt.ylabel('Factors affecting price'); 
plt.title('Price Distribution');
housing_new['price'].sort_values(ascending=False).head()
####the highest price of a house is 7700000.0.
##Finding the summary statistic of the housing dataframe
housing_new.describe()

housing_new.head()

## Bivariate Analysis
##a scatter plot between the price  and clearfell bathrooms

plt.scatter(housing_new['bedrooms'], housing_new['price'])

#Giving our scatterplot a title
plt.title('Price vs Bedrooms')

# Labelling our x axis
plt.xlabel('Price of a house')

#Labelling our y axis
plt.ylabel('Number of bedrooms')

# Displaying our scatterplot
plt.show()

housing_new.plot(x='price', y='floors')

##boxplot
sns.boxplot(x="grade", y="price", data=housing_new)
plt.show()<jupyter_output><empty_output><jupyter_text>**Multivariate Analysis**------<jupyter_code>housing_new.head()
##finding the correlation matrix of our features
housing_new.drop(['id', 'zipcode'], axis=1,inplace=True)

housing_new.columns
### correlation 
fig,ax=plt.subplots(figsize=(16,15))
ax=sns.heatmap(housing_new.drop(['price'], axis=1).corr(),annot=False,cmap='mako')<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code># Find all correlations and sort 
correlations_data = housing_new.corr()['price'].sort_values()

# Print the most negative correlations
print(correlations_data.head(10), '\n')

# Print the most positive correlations
print(correlations_data.tail(10))
features = housing_new.copy()<jupyter_output><empty_output><jupyter_text>All features aren't equal. It is all about selecting a small subset of features from a large pool of features<jupyter_code>def remove_collinear_features(x, threshold):
    '''
    Objective:
        Remove collinear features in a dataframe with a correlation coefficient
        greater than the threshold. Removing collinear features can help a model
        to generalize and improves the interpretability of the model.
        
    Inputs: 
        threshold: any features with correlations greater than this value are removed
    
    Output: 
        dataframe that contains only the non-highly-collinear features
    '''
    
    # Dont want to remove correlations between price
    y = x['price']
    x = x.drop(columns = ['price'])
    
    # Calculate the correlation matrix
    corr_matrix = x.corr()
    iters = range(len(corr_matrix.columns) - 1)
    drop_cols = []

    # Iterate through the correlation matrix and compare correlations
    for i in iters:
        for j in range(i):
            item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
            col = item.columns
            row = item.index
            val = abs(item.values)
            
            # If correlation exceeds the threshold
            if val >= threshold:
                # Print the correlated features and the correlation value
                # print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
                drop_cols.append(col.values[0])

    # Drop one of each pair of correlated columns
    drops = set(drop_cols)
    x = x.drop(columns = drops)
    x = x.drop(columns = ['long', 
                          'condition ',
                          'yr_built',
                          'sqft_lot15'])
    
    # Add the score back in to the data
    x['score'] = y
               
    return x
# Remove the collinear features above a specified correlation coefficient of 0.3
features = remove_collinear_features(features, 0.3);

features=housing_new[['lat','bedrooms','sqft_basement','view','bathrooms','sqft_living15','sqft_above','grade','price']]
features.head()<jupyter_output><empty_output><jupyter_text>Machine Learning Multiple Linear Regression<jupyter_code>features.head()
X=features[['lat','bedrooms','sqft_basement','view', 'bathrooms','bathrooms','sqft_living15', 'sqft_above','sqft_above','grade'	]]
y=features['price']
# Use 30% of the data as test
# The rest will be our training set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
regressor = LinearRegression()

regressor.fit(X_train, y_train)
##printing coefficients
print(regressor.intercept_)

#display the coefficient
print(regressor.coef_)

# Pairing the five features with their coefficients
list(zip(X, regressor.coef_))

#Making predictions and plotting residuals
y_pred = regressor.predict(X_test)
residuals = y_pred - y_test

# Plotting residuals

plt.scatter(y_pred, residuals, color='blue')
plt.title('Price Residuals')
plt.ylabel('Residuals')
plt.xlabel('Price')
plt.axhline(y= residuals.mean(), color='blue', linewidth=1)
plt.show()
#Importing Libraries
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.linear_model import LinearRegression
import scipy as sc
from sklearn.preprocessing import StandardScaler
# Assessing Heteroskedasticity using barlett's test

test_result, p_value = sc.stats.bartlett(y_pred, residuals)

print("P-Value is ;",p_value )
print('Baseline MSE:', metrics.mean_squared_error(y_test, y_pred))
print("Mean Price is: ", features['price'].mean())
print('Baseline RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

#Printing the coefficients
print("++++++++++++++++++++++++++++++++")
coef_dict_baseline = {}
for coef, feat in zip(regressor.coef_,X.columns):
    coef_dict_baseline[feat] = coef
coef_dict_baseline<jupyter_output>P-Value is ; 1.399622078711102e-100
Baseline MSE: 50153134435.958305
Mean Price is:  540178.9448403517
Baseline RMSE: 223948.95497849127
++++++++++++++++++++++++++++++++<jupyter_text>1. P value is less than 0.05 thus presence of heteroskedasticity 2. our model has  RMSE which is greater than >10% of Mean Price, hence model is performing poorly and should be examined more3. Residuals are not centered around the mean thus our model is not perfoming well Quantile Regression<jupyter_code>#Importing the Library

import statsmodels.regression.quantile_regression as Q_reg
features.head()
import statsmodels.api as sm


#Creating X and y variables

X = features.iloc[:,7:8]
y = features.price


#Splitting our data and using 70% for training

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=3)

#Creating a function for Quantile Regression and an array with the 3 different quantiles

quantiles = [0.25, 0.5, 0.75]

# Get the model, and the cofficients in (a) - (b)
#Splitting our data and using 70% for training

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=3)

#Creating a function for Quantile Regression and an array with the 3 different quantiles

quantiles = [0.25, 0.5, 0.75]
# prepare the quantile that we want to work with
#quantile reg
import statsmodels.formula.api as smf

# Dividing our data into training and test sets
# ---
# 
#from sklearn.model_selection import train_test_split
#x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=10)

# Finding the regression coefficients for the conditioned median, 0.5 quantile
#
mod = smf.quantreg('y ~ X', features)
res = mod.fit(q=.5)

# Then print out the summary of our model
#
print(res.summary())
    

residual1 = mod.fit(q = 0.25)
residual3 = mod.fit(q = 0.5)
residual4 = mod.fit(q = 0.75)

# display the results 
print("25th Percentile: ", residual1.summary(), '\n++++++++++++++++++++++++++\n')
print("\n50th Percentile: ", residual3.summary(), '\n___________________________\n')
print("\n75th Percentile: ", residual4.summary())
##algorithm evaluation

from sklearn import metrics
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

#R2 score
from sklearn.metrics import r2_score
q = r2_score(y_test, y_pred)
print(q)
###prediction accuracy of 62.48% is not quite bad<jupyter_output>-0.6247493515454394<jupyter_text>**Lasso Regression**<jupyter_code>#Creating X and y variables
from sklearn import linear_model

X = features.iloc[:,7:8]
y = features.price

# Creating lasso object

lasso = linear_model.Lasso() 

# Fitting the lasso model

lasso.fit(X, y) 

#Splitting into 70% for training

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=30)

#specifying parameters
parameters = {'alpha': np.concatenate((np.arange(0.1,2,0.1), np.arange(2, 5, 0.5), np.arange(5, 25, 1)))}

gridlasso = GridSearchCV(lasso, parameters, scoring ='r2')
# Fitting Lasso model 
import warnings
warnings.filterwarnings("ignore")
gridlasso.fit(X_train, y_train)

#Predicting and printing the metrics

print("lasso score:", gridlasso.score(X_test, y_test))
print("lasso MSE:", mean_squared_error(y_test, gridlasso.predict(X_test)))
print("Mean Price is: ", features['price'].mean())
print("lasso RMSE:", np.sqrt(mean_squared_error(y_test, gridlasso.predict(X_test))))

#Printing the coefficients
print("#################")
coef_dict_baseline = {}
for coef, feat in zip(lasso.coef_,X.columns):
    coef_dict_baseline[feat] = coef
coef_dict_baseline
###RMSE scores have reduced meaning there is an improvement in our model<jupyter_output>lasso score: 0.43269375142413924
lasso MSE: 72499913662.59315
Mean Price is:  540178.9448403517
lasso RMSE: 269258.08003213786
#################<jupyter_text>Ridge Regression<jupyter_code>#Importing Libraries for Regression
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
X = features.iloc[:,7:8]
y = features.price
ridge = Ridge(normalize=True)
search = GridSearchCV(estimator=ridge,param_grid={'alpha':np.logspace(-5,2,8)},scoring='neg_mean_squared_error',n_jobs=1,refit=True,cv=10)
#Ridge Regression Model
#L2 or alpha is the name of the hyperparameter used in ridge regression. 
#It requires the use of a grid
#In ridge model we indicat normalization in order to get better estimates
#Refit set to true means to adjust the parameters while modeling 
#cv is the number of folds to develop for the cross-validation
#logspace means log of -5 to 2, but we only get 8 values from within that range evenly spread out
#Fitting the Ridge model and identifying the best parameters and scores for the model
search.fit(X,y)
search.best_params_
{'alpha': 0.01}
print(abs(search.best_score_) )
print(search.best_params_)
# We can confirm this by fitting our model with the ridge information and finding the RMSE below
#
ridge = Ridge(normalize=True,alpha=0.01)
ridge.fit(X,y)


y_pred = ridge.predict(X)
print("Ridge score:", ridge.score(X, y))
print("Mean Price is: ", features['price'].mean())
print('Ridge MSE:', metrics.mean_squared_error(y, y_pred))
print('Ridge RMSE:', np.sqrt(metrics.mean_squared_error(y, y_pred)))

#Printing the coefficients
print("#################")
coef_dict_baseline = {}
for coef, feat in zip(ridge.coef_,X.columns):
    coef_dict_baseline[feat] = coef
coef_dict_baseline
###RMSE has further reduced meaning that our model has increased,,,lets explore the elastic net regression and see<jupyter_output>75020197618.61433
{'alpha': 0.001}
Ridge score: 0.44548380060051784
Mean Price is:  540178.9448403517
Ridge MSE: 74841605123.73648
Ridge RMSE: 273571.93774898857
#################<jupyter_text>***Elastic Net Regression***<jupyter_code>from sklearn.linear_model import ElasticNet
X = features.iloc[:,7:8]
y = features.price
# Normalising Elastic Net Regression and creating our GridSearch

elastic = ElasticNet(normalize=True)
search = GridSearchCV(estimator=elastic,param_grid={'alpha':np.logspace(-5,2,8),'l1_ratio':[.2,.4,.6,.8]},scoring='neg_mean_squared_error',n_jobs=1,refit=True,cv=10)
# Fitting our model and displaying the best parameters and best results
# 
search.fit(X,y)
search.best_params_
abs(search.best_score_)
print(search.best_params_)
print(abs(search.best_score_))
#The best hyperparameters was an alpha set to 0.00001 and a l1_ratio of 0.8. 

elastic = ElasticNet(normalize=True,alpha=0.00001,l1_ratio=0.8)
elastic.fit(X,y)

#Printing the metrics after prediction

y_pred = elastic.predict(X)

print("Elastic score:", elastic.score(X, y))
print("Mean Price is: ", features['price'].mean())
print('Elastic MSE:', metrics.mean_squared_error(y, y_pred))
print('Elastic RMSE:', np.sqrt(metrics.mean_squared_error(y, y_pred)))

#Printing the coefficients
print("#################")
coef_dict_baseline = {}
for coef, feat in zip(elastic.coef_,X.columns):
    coef_dict_baseline[feat] = coef
coef_dict_baseline
###RMSE has increased indicating a slight misaccuraacy in our model...Ridge regression should therefore be used<jupyter_output>{'alpha': 1e-05, 'l1_ratio': 0.8}
75086604620.3524
Elastic score: 0.4447627733449378
Mean Price is:  540178.9448403517
Elastic MSE: 74938920291.81274
Elastic RMSE: 273749.7402588955
#################<jupyter_text>Challenging  the Solution Follow up questions<jupyter_code>#We had the right data
##Other models should be accessed to ensure a better prediction
## We should more data that is not collinear to do our prediction<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10440950349685035, 'fraction_numerical': 0.03749813997321561, 'mean_word_length': 5.065583634175692, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 53, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '460259', 'license': 'MIT', 'n_tokens_mistral': 6950, 'n_tokens_neox': 6239, 'n_words': 1991, 'path': 'Hass_Consulting_Company_supervised_learning_house_price_sale_prediction (1).ipynb', 'repo_name': 'Obura98/Hass_Consulting_Company_supervised_learning_house_price_sale_prediction', 'size': '20161'}