<jupyter_start><jupyter_text>Assignment 3: Evaluating Regression Models - <NAME>   Purpose Our client -- Happy Real Estate Brokers -- wants to utilize machine learning models to complement their traditional valuation methods. Their goal is to assess the market value of residential real estate. This report will serve as an overview of the entire process (e.g. exploratory data analysis, model validation, etc.) taken to fulfill our client's needs. We will conclude the report with a final recommendation describing which machine learning model we believe will best allow them to assess the market value of residential real estate.  This report will be broken up into Sections that cover the specific methodology that went into arriving at the final recommendation. The report will include and/or cover, at minimum, the following specific items and/or tasks:* Utilize the Boston Housing Study data to train each machine learning algorithm* Use all the explanatory variables in the data except "neighborhood" and use all 506 tract observations * The response variable will be the median value of homes in thousands of 1970s dollars* Develop at least two regression methods - linear, ridge, lasso and/or elastic net * Use K-fold cross-validation to calculate root mean-squared error scores to evaluate models* Provide final management recommendation   Section 1 - Exploratory Data Analysis   The first step in building a machine learning model is to gain a firm understanding of the data that will be used to train the model. Section 1 covers all the work that was completed as part of the Exploratory Data Analysis (EDA) process along with comments explaining code and output wherever necessary.<jupyter_code>#Load all the necessary packages we need to complete the exercise

import pandas as pd
import numpy as np
import matplotlib as plt
from pandas.plotting import scatter_matrix
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDRegressor
from sklearn.linear_model import Lasso
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_error
import nbconvert
#Read in Boston Housing Study data 
boston_input = pd.read_csv("boston.csv")
#Drop neighborhood attribute since it will not be used to make predictions  
boston_input = boston_input.drop("neighborhood", 1)
#This method allows us to further understand the type of data each of our 
#attributes holds and to see if there are missing values
boston_input.info()
#Lets plot the distribution of all the attributes
#This provides us with visual insight of our attributes
%matplotlib inline
boston_input.hist(bins = 40, figsize=(30,25));
#Create correlation matrix to see the linear correlation between all the attributes 
#and the target variable - mv 
corr_matrix = boston_input.corr()
corr_matrix["mv"].sort_values(ascending=False)
#Scatter plot matrix for select attributes was created to visualize correlation
attributes = ["mv","rooms", "zn", "dis", "lstat", "ptratio", "indus"]
scatter_matrix(boston_input[attributes], figsize=(16,12));<jupyter_output><empty_output><jupyter_text>Key Takeaways (nonexhaustive)* Based on correlation matrix, it looks like the rooms and lstat features have the highest linear correlation with the label - mv; this is further validated by looking at the scatterplot matrix * There seems to be more negatively correlated features with mv than positively correlated ones * mv appears to follow a normal distribution and most of the homes seem to have a median value between 20,000 - 25,000 (USD)* Data set does not have any missing values although it does appear there was a cap placed on the mv variable* Majority of homes are not by the Charles River (chas is the only dummy variable in the data set) * Appears as though feature scaling might be appropriate given the different scales   Section 2 - Data Preparation   Upon completing the initial EDA process, we got a better understanding of the features that make up the data set. The next section in this exercise is to prepare the data that will be used to train the machine learning algorithms.  Typically, this step involves imputing missing values, determining how to deal with outliers, and fixing any erroneous values. Also, variable transformation and feature scaling must be considered as well during this step.<jupyter_code>#Make a copy of the original data set
boston = boston_input.copy()
#Utilize function to split the data set into a training set and test set
#The parameters in the section dictate the size of the test set and a random 
#generator seed so that the data is split using the same shuffled indices
train_set, test_set = train_test_split(boston, test_size = 0.2, random_state = 10)
#At this point, several decisions need to be made in terms of transformations
#Since there are no missing values, we will not need to address that aspect
#Split the labels away from the training set
boston_train = train_set.drop("mv", axis =1)
boston_train_labels = train_set["mv"].copy()
#We decided that it was important to standardize the features (excluding mv)
#which will be accomplished by using StandardScaler 
scaler = StandardScaler()
boston_train_scaled = scaler.fit_transform(boston_train)<jupyter_output><empty_output><jupyter_text>Key Takeaways* This section focused on splitting the Boston data set into training and testing subsets. We decided to allocate 20% of the Boston data set to be used as the test data set* The only transformation that was made was to standardize the predictor variables because they generally work better with machine learning algorithms  Section 3 - Build Models  Now we will build multiple machine learning models based on the updated training data. We will continue to utilize Scikit-Learn in this section. Specifically, we will utilize different algorithms to train our models.Since this is a supervised learning exercise, we will have the labels separated as well.  Each algorithm arrives at the y-intercept and coefficients differently. For example, the Ridge regression tries to keep the weights as small as possible while the Lasso regression (another regularized version of linear regression) tries to completely eliminate the weights of the least important features.<jupyter_code>#Build Linear Regression Model using SVD
#Print out the y-intercept and coefficients 
lin_reg = LinearRegression()
lin_reg.fit(boston_train_scaled, boston_train_labels)
lin_reg.intercept_, lin_reg.coef_
#Build Linear Regression Model using Stochastic Gradient Descent (SGD)
#Print out the y-intercept and coefficients
lin_sgd_reg = SGDRegressor(max_iter = 50, penalty = None, eta0=0.1)
lin_sgd_reg.fit(boston_train_scaled, boston_train_labels.ravel())
lin_sgd_reg.intercept_, lin_sgd_reg.coef_
#Build Lasso Regression Model 
#Print out the y-intercept and coefficients 
lasso_reg = Lasso(alpha=0.1)
lasso_reg.fit(boston_train_scaled, boston_train_labels)
lasso_reg.intercept_, lasso_reg.coef_
#Build Ridge Regression Model using closed-form equation
#Print out the y-intercept and coefficients 
ridge_reg = Ridge(alpha=1, solver="cholesky")
ridge_reg.fit(boston_train_scaled, boston_train_labels.ravel())
ridge_reg.intercept_, ridge_reg.coef_<jupyter_output><empty_output><jupyter_text>Key Takeaways* This section was straight-forward in the sense that Scikit-Learn was utilized to create all the models* The standardized training data set was used along with the non-standardized labels to train the models* From purely "eyeballing" the intercept and coefficients, notice that they are all similar; Note: that reviewing the p-values and model statistical signficance is not of high importance to us during this exercise since our main goal is predictive accuracy * One item that stands out is that by purely using the y-intercept, all of the models would predict the average median value of a house in Boston to be around 22,000 (USD) which is slightly lower than the mv mean outputted during EDA section* There are more negative coeffecients which makes sense given the correlation matrix that was outputted during the EDA section* At this step, we simply trained various models but have not decided which one yet is our "best" one   Section 4 - Model Validation  This section will cover the cross-validation steps utilized to measure the performance of each model. Evaluating each model based on an objective criteria is necessary to understand how well each model can generalize (i.e. overfitting/underfitting) new data. Predictive accuracy is of upmost importance because our client will be utilizing our recommendation to assess the market value of residential real estate.<jupyter_code>#Use each model to make predictions 
linreg_predictions = lin_reg.predict(boston_train_scaled)
linsgd_predictions = lin_sgd_reg.predict(boston_train_scaled)
lasso_predictions = lasso_reg.predict(boston_train_scaled)
ridge_predictions = ridge_reg.predict(boston_train_scaled)
#Print out the Root Mean Square Error for each model's predictions
linreg_mse = mean_squared_error(boston_train_labels, linreg_predictions)
linsgd_mse = mean_squared_error(boston_train_labels, linsgd_predictions)
lasso_mse = mean_squared_error(boston_train_labels, lasso_predictions)
ridge_mse = mean_squared_error(boston_train_labels, ridge_predictions)

linreg_rmse = np.sqrt(linreg_mse)
print("linreg_rmse:", linreg_rmse,"\n")
linsgd_rmse = np.sqrt(linsgd_mse)
print("linsgd_rmse:", linsgd_rmse, "\n")
lasso_rmse = np.sqrt(lasso_mse)
print("lasso_rmse:", lasso_rmse, "\n")
ridge_rmse = np.sqrt(ridge_mse)
print("ridge_rmse:", ridge_rmse, "\n")
#Utilize K-fold cross-validation to calculate RMSE scores
linreg_scores = cross_val_score(lin_reg, boston_train_scaled, boston_train_labels, 
                                scoring="neg_mean_squared_error", cv=4)
linsgd_scores = cross_val_score(lin_sgd_reg, boston_train_scaled, 
                                boston_train_labels, 
                                scoring="neg_mean_squared_error", cv=4)
lasso_scores = cross_val_score(lasso_reg, boston_train_scaled, boston_train_labels, 
                               scoring="neg_mean_squared_error", cv=4)
ridge_scores = cross_val_score(ridge_reg, boston_train_scaled, boston_train_labels, 
                               scoring="neg_mean_squared_error", cv=4)

linreg_rmse_scores = np.sqrt(-linreg_scores)
linsgd_rmse_scores = np.sqrt(-linsgd_scores)
lasso_rmse_scores = np.sqrt(-lasso_scores)
ridge_rmse_scores = np.sqrt(-ridge_scores)
#Create function to display the RMSE score statistics for each model 
def display_scores(scores):
    print ("Scores:", scores)
    print ("Mean:", scores.mean())
    print ("Standard Deviation:", scores.std())
#Use function to print out descriptive info about the results of the K-fold 
#cross-validation for each model
print("Linear Regression w/ SVD RMSE")
display_scores(linreg_rmse_scores)
print("\n")
print("Linear Regression w/ SGD RMSE")
display_scores(linsgd_rmse_scores)
print("\n")
print("Lasso Regression RMSE")
display_scores(lasso_rmse_scores)
print("\n")
print("Ridge Regression w/ closed-form RMSE")
display_scores(ridge_rmse_scores)
#Evaluate two of our best models on the test set - linreg and ridge_sgd_reg
#Split the test set into just the features and label
boston_test = test_set.drop("mv", axis =1)
boston_test_labels = test_set["mv"].copy()
#Transform boston_test using the same approach as the boston_train
boston_test_scaled = scaler.fit_transform(boston_test)
#Save the model parameters from the two best trained models 
linreg_model = lin_reg.fit(boston_train_scaled, boston_train_labels)
ridge_model = ridge_reg.fit(boston_train_scaled, boston_train_labels.ravel())
#Make final predictions using each model
linreg_final_predictions = linreg_model.predict(boston_test_scaled)
ridge_final_predictions = ridge_model.predict(boston_test_scaled)
#Compare RMSE scores
linreg_final_mse = mean_squared_error(boston_test_labels, linreg_final_predictions)
linreg_final_rmse = np.sqrt(linreg_final_mse)
print("linreg_final_rmse:", linreg_final_rmse, "\n")
ridge_final_mse = mean_squared_error(boston_test_labels, ridge_final_predictions)
ridge_final_rmse = np.sqrt(ridge_final_mse)
print("ridge_final_rmse:", ridge_final_rmse, "\n")<jupyter_output>linreg_final_rmse: 6.72101643015 

ridge_final_rmse: 6.7190192134
{'fraction_non_alphanumeric': 0.055796751889371284, 'fraction_numerical': 0.006592699790963177, 'mean_word_length': 5.680451127819549, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '433783', 'license': 'Unlicense', 'n_tokens_mistral': 3561, 'n_tokens_neox': 3362, 'n_words': 1436, 'path': 'Assignment3_OscarHernandez.ipynb', 'repo_name': 'oschernan/Practical-Machine-Learning', 'size': '12447'}