<jupyter_start><jupyter_text>Regression Week 2: Multiple Regression (Interpretation) In this notebook, we will use data on house sales in King County, Seatle to predict prices using multiple regression. The goal of this notebook is to explore multiple regression and feature engineering. You will:* Use DataFrames to do some feature engineering* Use sklearn to compute the regression weights (coefficients/parameters)* Given the regression weights, predictors and outcome write a function to compute the Residual Sum of Squares (RSS)* Look at coefficients and interpret their meanings* Evaluate multiple models via RSS  Importing Libraries<jupyter_code>import os
import zipfile
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Unzipping files with house sales dataDataset is from house sales in King County, the region where the city of Seattle, WA is located.<jupyter_code># Put files in current direction into a list
files_list = [f for f in os.listdir('.') if os.path.isfile(f)]
# Filenames of unzipped files
unzip_files = ['kc_house_train_data.csv','kc_house_test_data.csv', 'kc_house_data.csv']
# If upzipped file not in files_list, unzip the file
for filename in unzip_files:
    if filename not in files_list:
        zip_file = filename + '.zip'
        unzipping = zipfile.ZipFile(zip_file)
        unzipping.extractall()
        unzipping.close<jupyter_output><empty_output><jupyter_text>Loading Sales data, Sales Training data, and Sales Test data<jupyter_code># Dictionary with the correct dtypes for the DataFrame columns
dtype_dict = {'bathrooms':float, 'waterfront':int, 'sqft_above':int, 
              'sqft_living15':float, 'grade':int, 'yr_renovated':int, 
              'price':float, 'bedrooms':float, 'zipcode':str, 
              'long':float, 'sqft_lot15':float, 'sqft_living':float, 
              'floors':str, 'condition':int, 'lat':float, 'date':str, 
              'sqft_basement':int, 'yr_built':int, 'id':str, 'sqft_lot':int, 'view':int}
# Loading sales data, sales training data, and test_data into DataFrames
sales = pd.read_csv('kc_house_data.csv', dtype = dtype_dict)
train_data = pd.read_csv('kc_house_train_data.csv', dtype = dtype_dict)
test_data = pd.read_csv('kc_house_test_data.csv', dtype = dtype_dict)
# Looking at head of training data DataFrame
train_data.head()<jupyter_output><empty_output><jupyter_text>Learning a multiple regression model Now, learn a multiple regression model predicting 'price' based on the following features:example_features = ['sqft_living', 'bedrooms', 'bathrooms'] on training data. First, let's plot the data for these features.<jupyter_code>plt.figure(figsize=(8,6))
plt.plot(train_data['sqft_living'], train_data['price'],'.')
plt.xlabel('Living Area (ft^2)', fontsize=16)
plt.ylabel('House Price ($)', fontsize=16)
plt.show()
plt.figure(figsize=(12,8))

plt.subplot(1, 2, 1)
plt.plot(train_data['bedrooms'], train_data['price'],'.')
plt.xlabel('# Bedrooms', fontsize=16)
plt.ylabel('House Price ($)', fontsize=16)

plt.subplot(1, 2, 2)
plt.plot(train_data['bathrooms'], train_data['price'],'.')
plt.xlabel('# Bathrooms', fontsize=16)
plt.ylabel('House Price ($)', fontsize=16)

plt.show()<jupyter_output><empty_output><jupyter_text>Now, creating a list of the features we are interested in, the feature matrix, and the output vector.<jupyter_code>example_features = ['sqft_living', 'bedrooms', 'bathrooms']
X_multi_lin_reg = train_data[example_features]
y_multi_lin_reg = train_data['price']<jupyter_output><empty_output><jupyter_text>Creating a Linear Regression Object for Sklearn library and using the feature matrix and output vector to perform linear regression.<jupyter_code>example_model = LinearRegression()
example_model.fit(X_multi_lin_reg, y_multi_lin_reg)<jupyter_output><empty_output><jupyter_text>Now that we have fitted the model we can extract the regression weights (coefficients):<jupyter_code># printing the intercept and coefficients
print example_model.intercept_
print example_model.coef_
# Putting the intercept and weights from the multiple linear regression into a Series
example_weight_summary = pd.Series( [example_model.intercept_] + list(example_model.coef_),
                                  index = ['intercept'] + example_features )
print example_weight_summary<jupyter_output>intercept      87912.865815
sqft_living      315.406691
bedrooms      -65081.887116
bathrooms       6942.165986
dtype: float64<jupyter_text>Making PredictionsRecall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:<jupyter_code>example_predictions = example_model.predict(X_multi_lin_reg)
print example_predictions[0] # should be close to 271789.505878<jupyter_output>271789.26538<jupyter_text>Compute RSS Now that we can make predictions given the model, let's write a function to compute the RSS of the model.<jupyter_code>def get_residual_sum_of_squares(model, data, outcome):
    # - data holds the data points with the features (columns) we are interested in performing a linear regression fit
    # - model holds the linear regression model obtained from fitting to the data
    # - outcome is the y, the observed house price for each data point
    
    # By using the model and applying predict on the data, we return a numpy array which holds
    # the predicted outcome (house price) from the linear regression model
    model_predictions = model.predict(data)

    # Computing the residuals between the predicted house price and the actual house price for each data point
    residuals = outcome - model_predictions

    # To get RSS, square the residuals and add them up
    RSS = sum(residuals*residuals)

    return(RSS)<jupyter_output><empty_output><jupyter_text>Create some new features Although we often think of multiple regression as including multiple different features (e.g.  of bedrooms, squarefeet, and  of bathrooms), we can also consider transformations of existing features e.g. the log of the squarefeet or even "interaction" features such as the product of bedrooms and bathrooms. Create the following 4 new features as column in both TEST and TRAIN data:* bedrooms_squared = bedrooms\*bedrooms* bed_bath_rooms = bedrooms\*bathrooms* log_sqft_living = log(sqft_living)* lat_plus_long = lat + long<jupyter_code># Creating new 'bedrooms_squared' feature
train_data['bedrooms_squared'] = train_data['bedrooms']*train_data['bedrooms']
test_data['bedrooms_squared'] = test_data['bedrooms']*test_data['bedrooms']
# Creating new 'bed_bath_rooms' feature
train_data['bed_bath_rooms'] = train_data['bedrooms']*train_data['bathrooms']
test_data['bed_bath_rooms'] = test_data['bedrooms']*test_data['bathrooms']
# Creating new 'log_sqft_living' feature
train_data['log_sqft_living'] = np.log(train_data['sqft_living'])
test_data['log_sqft_living'] = np.log(test_data['sqft_living'])
# Creating new 'lat_plus_long' feature
train_data['lat_plus_long'] = train_data['lat'] + train_data['long']
test_data['lat_plus_long'] = test_data['lat'] + test_data['long']
# Displaying head of train_data DataFrame and test_data DataFrame to verify that new features are present
train_data.head()
test_data.head()<jupyter_output><empty_output><jupyter_text>* Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.* bedrooms times bathrooms gives what's called an "interaction" feature. It is large when *both* of them are large.* Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.* Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why) **Quiz Question: What is the mean (arithmetic average) value of your 4 new features on TEST data? (round to 2 digits)**<jupyter_code>print "Mean of Test data 'bedrooms_squared' feature: %.2f " % np.mean(test_data['bedrooms_squared'].values)
print "Mean of Test data 'bed_bath_rooms' feature: %.2f " % np.mean(test_data['bed_bath_rooms'].values)
print "Mean of Test data 'log_sqft_living' feature: %.2f " % np.mean(test_data['log_sqft_living'].values)
print "Mean of Test data 'lat_plus_long' feature: %.2f " % np.mean(test_data['lat_plus_long'].values)<jupyter_output>Mean of Test data 'bedrooms_squared' feature: 12.45 
Mean of Test data 'bed_bath_rooms' feature: 7.50 
Mean of Test data 'log_sqft_living' feature: 7.55 
Mean of Test data 'lat_plus_long' feature: -74.65<jupyter_text>Learning Multiple Models Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:* Model 1: squarefeet,  bedrooms,  bathrooms, latitude & longitude* Model 2: add bedrooms\*bathrooms* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude<jupyter_code>model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']
model_2_features = model_1_features + ['bed_bath_rooms']
model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']<jupyter_output><empty_output><jupyter_text>Now that you have the features, learn the weights for the three different models for predicting target = 'price' and look at the value of the weights/coefficients:<jupyter_code># Creating a LinearRegression Object for Model 1 and learning the multiple linear regression model
model_1 = LinearRegression()
model_1.fit(train_data[model_1_features], train_data['price'])
# Creating a LinearRegression Object for Model 2 and learning the multiple linear regression model
model_2 = LinearRegression()
model_2.fit(train_data[model_2_features], train_data['price'])
# Creating a LinearRegression Object for Model 3 and learning the multiple linear regression model
model_3 = LinearRegression()
model_3.fit(train_data[model_3_features], train_data['price'])<jupyter_output><empty_output><jupyter_text>** Now, Examine/extract each model's coefficients: **<jupyter_code># Putting the Model 1 intercept and weights from the multiple linear regression for the 3 models into a Series
model_1_summary = pd.Series( [model_1.intercept_] + list(model_1.coef_),
                                  index = ['intercept'] + model_1_features , name='Model 1 Coefficients' )
print model_1_summary
# Putting the Model 2 intercept and weights from the multiple linear regression for the 3 models into a Series
model_2_summary = pd.Series( [model_2.intercept_] + list(model_2.coef_),
                                  index = ['intercept'] + model_2_features , name='Model 2 Coefficients' )
print model_2_summary
# Putting the Model 3 intercept and weights from the multiple linear regression for the 3 models into a Series
model_3_summary = pd.Series( [model_3.intercept_] + list(model_3.coef_),
                                  index = ['intercept'] + model_3_features , name='Model 3 Coefficients' )
print model_3_summary<jupyter_output>intercept          -62036084.986098
sqft_living              529.422820
bedrooms               34514.229578
bathrooms              67060.781319
lat                   534085.610867
long                 -406750.710861
bed_bath_rooms         -8570.504395
bedrooms_squared       -6788.586670
log_sqft_living      -561831.484076
lat_plus_long         127334.900006
Name: Model 3 Coefficients, dtype: float64<jupyter_text>**Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?**<jupyter_code>print "Positive: ", model_1_summary['bathrooms']<jupyter_output>Positive:  15706.7420827<jupyter_text>**Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?**<jupyter_code>print "Negative: ", model_2_summary['bathrooms']<jupyter_output>Negative:  -71461.3082928<jupyter_text>Think about what this means: In model 2, the new 'bed_bath_rooms' feature causes the house price to be over-estimated. Thus, the 'bathrooms' turns to negative to better agree with the observed values for the house prices.  Comparing multiple modelsNow that you've learned three models and extracted the model weights we want to evaluate which model is best. First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models.<jupyter_code># Compute the RSS on TRAINING data for each of the three models and record the values:
rss_model_1_train = get_residual_sum_of_squares(model_1, train_data[model_1_features], train_data['price'])
rss_model_2_train = get_residual_sum_of_squares(model_2, train_data[model_2_features], train_data['price'])
rss_model_3_train = get_residual_sum_of_squares(model_3, train_data[model_3_features], train_data['price'])
print "RSS for Model 1 Training Data: ", rss_model_1_train
print "RSS for Model 2 Training Data: ", rss_model_2_train
print "RSS for Model 3 Training Data: ", rss_model_3_train<jupyter_output>RSS for Model 1 Training Data:  9.6787996305e+14
RSS for Model 2 Training Data:  9.58419635074e+14
RSS for Model 3 Training Data:  9.0343645505e+14<jupyter_text>**Quiz Question: Which model (1, 2 or 3) has lowest RSS on TRAINING Data?** Is this what you expected? Model 3 has the lowest RSS on the Training Data. This is expected since Model 3 has the most features. Now compute the RSS on on TEST data for each of the three models.<jupyter_code># Compute the RSS on TESTING data for each of the three models and record the values:
rss_model_1_test = get_residual_sum_of_squares(model_1, test_data[model_1_features], test_data['price'])
rss_model_2_test = get_residual_sum_of_squares(model_2, test_data[model_2_features], test_data['price'])
rss_model_3_test = get_residual_sum_of_squares(model_3, test_data[model_3_features], test_data['price'])
print "RSS for Model 1 Test Data: ", rss_model_1_test
print "RSS for Model 2 Test Data: ", rss_model_2_test
print "RSS for Model 3 T Data: ", rss_model_3_test<jupyter_output>RSS for Model 1 Test Data:  2.25500469795e+14
RSS for Model 2 Test Data:  2.23377462976e+14
RSS for Model 3 T Data:  2.59236319207e+14
{'fraction_non_alphanumeric': 0.08680266444629475, 'fraction_numerical': 0.030391340549542047, 'mean_word_length': 5.563296903460838, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '523738', 'license': 'MIT', 'n_tokens_mistral': 4798, 'n_tokens_neox': 4348, 'n_words': 1467, 'path': 'Week_2_Multiple_Regression/assign-1_sklearn_mulp_regre.ipynb', 'repo_name': 'Santana9937/Coursera_ML_sSpecialization', 'size': '14412'}