<jupyter_start><jupyter_text>***Authors: <NAME>, <NAME>, <NAME>***\***Date: 3/6/2020***  **Multiple Regression Analysis of Housing Sale Prices in 2019** **Data Source: https://info.kingcounty.gov/assessor/DataDownload/default.aspx**  Methodology1. Data Acquisition    * Get Property sales and assesments from the King County Department of Asssesments website    * Merge the Sale information with the house information    * Save the Merged table to the repo2. Data Cleaning    *     * 3. Creation of a model    * Start with a base linear regression model    * Expand on model to include more explanatory variables4. Analysis of Model    * Use metrics to find the best model  Import Necessary Packages<jupyter_code>import math
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

import os, sys
module_path = os.path.abspath(os.path.join(os.pardir, os.pardir))
if module_path not in sys.path:
    sys.path.append(module_path)
from src.make_data_table import *
from src.stat_helpers import *
from src.stats_maker import *
from src.wrangle_helpers import *<jupyter_output><empty_output><jupyter_text>Data AcquisitionThis will get the data from the internet, unzip the files, only get the 2019 data, merge the files then saves it to the repo as well as deleting the downloaded files.\It has already been run and is save to the repo under the data folder.<jupyter_code>#build_data_file()<jupyter_output>New files downloaded: {'EXTR_Parcel.csv', 'EXTR_ResBldg.csv', 'EXTR_RPSale.csv'}
PINs added to tables
Tables merged and saved to: /Users/jarodc33/Mod_Projects/mod_2/king_county_housing/data/housing2019.csv
All downloaded files have been deleted.
Data is now accessible from the data folder.<jupyter_text>Data Cleaning The load and wrangle function will obtain the data and will add some extra columns.\The modified dataframe makes certain columns into integers and has added columns that are useful for analysis.<jupyter_code>data = load_and_wrangle_data('../../data/housing2019.csv')[1]<jupyter_output><empty_output><jupyter_text>Creation of a Model Base Model First in order to make a base model we started by looking at a correlation heatmap of variable compared to Sale Price. Then we chose to use the SqFtTotLiving as a comparison for our first regression model.We then wanted to first plot this relationship to see if it seems at all slightly linear.<jupyter_code>plt.scatter(data['SqFtTotLiving'], data['SalePrice'], alpha = 0.1)
plt.xlabel('SqFtTotLiving')
plt.ylabel('SalePrice');<jupyter_output><empty_output><jupyter_text>It looks as though there is a correlation so we will run a linear regression on these two variables.<jupyter_code>base_model = get_model(data, ['SqFtTotLiving'], dependent = 'SalePrice')
base_model.summary()<jupyter_output><empty_output><jupyter_text>There is an R-squared of about 0.34. Our p values for the intercept and coefficients are low enough to say that theyt are statistically significant.It shows that for each square foot off living space the sale price is expected to go up $304  Evaluation of Model We used 4 measures to validate each model:1. Rainbow Statistic: This tests if the independant variable linearly predict the dependant (Lower statistic or higher p-value  is better)2. Jarque-Bera (JB): Tells whether the residuals are normally distributed (Lower statistic or higher p-value is better)3. Homoscedasticity: Indicates if the variances are homogeneous (Higher p-value is better)4. Variance Inflation Factor: Measures multicollinearity. Lower VIF indicated the independant variables do not correlate with eachotherIn order to generate this we use a method in stats_makerThe JB can be found in the model Summary<jupyter_code>stats = stats_from_columns(data, ['SqFtTotLiving'], dependent = 'SalePrice')
print('Rainbow p-value: ' + str(stats['rainbow']['p-value']))
print('JB: 229982')
print('Homoscedasticity F-statistic: 15470')<jupyter_output>Rainbow p-value: 0.013161169064821551
JB: 229982
Homoscedasticity F-statistic: 15470<jupyter_text>Currently our model violates all of our measure's assumptions. We can also visualize the residuals to get an idea of what they look like. We take te logarithm because the non-logarithm residuals do not give a good sense of their distribution<jupyter_code>log_df = data.copy()
log_df['SalePrice'] = log_df['SalePrice'].apply(lambda x: math.log(x, 10))

y = log_df['SalePrice']
y_hat = get_model(log_df, ['SqFtTotLiving']).predict()
plt.scatter(x=y_hat, y=y-y_hat, color="blue", alpha=0.2)
plt.xlabel('Predicted log(SalePrice)')
plt.ylabel('Residual log(SalePrice)');<jupyter_output><empty_output><jupyter_text>As we can see our model is greatly over predicting saleprice at 105.6 to 106  Obtaining our Best model In order to improve our model more independant variables were added to the relationship.SqFtTotLiving was replaced with BldgGrade because it BldgGrade was more predictive and also reduced the collinearity between variables. They could not be included together because their collinearity was above our VIF threshold of 5 when used together.A list of columns were used to generate our current best model as seen bellow.<jupyter_code>columns = ['BldgGrade', 'TidelandShoreland', 'SqFtOpenPorch', 'SqFtDeck', 'Area', 'ViewScore','YrRenovated']
best_model_stats = stats_from_columns(data, columns, dependent = 'SalePrice')
best_model_stats['ols'].summary()
print('Rainbow p-value: ' + str(best_model_stats['rainbow']['p-value']))
print('JB: 283186')
print('Homoscedasticity statistic: 3258')
best_model_stats['vif']<jupyter_output>Rainbow p-value: 0.35401682824193115
JB: 283186
Homoscedasticity statistic: 3258<jupyter_text>The R-squared of our model increased to 0.44, which is an improvement from the original model and previous nmodels. The Rainbow p-value has increased and it no longer violates the assumption or linearity. The JB however has increased but the Homoscedasticity F-statistic has greatly decreased! The variance inflation factor between the variable in under 5 and there for we can say that the multicolinearity between the variables is not high. The residuales were once again plotted in order to visualize.Logarithms were applied as before in order to visualize.<jupyter_code>log_df = data.copy()
log_df['SalePrice'] = log_df['SalePrice'].apply(lambda x: math.log(x, 10))

y = log_df['SalePrice']
y_hat = get_model(log_df, columns).predict()
plt.scatter(x=y_hat, y=y-y_hat, color="blue", alpha=0.2)
plt.xlabel('Predicted log(SalePrice)')
plt.ylabel('Residual log(SalePrice)');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08338423946243127, 'fraction_numerical': 0.021686010995723884, 'mean_word_length': 6.00427807486631, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '23639', 'license': 'MIT', 'n_tokens_mistral': 2123, 'n_tokens_neox': 1934, 'n_words': 740, 'path': 'notebooks/report/Final_Report.ipynb', 'repo_name': 'sam-thurman/king_county_housing', 'size': '6568'}