<jupyter_start><jupyter_text>_Lambda School Data Science, Unit 2_  Linear Models  Sprint Challenge: Predict home prices in Ames, Iowa 🏠💸To demonstrate mastery on your Sprint Challenge, do all the required, numbered instructions in this notebook.To earn a score of "3", also do all the stretch goals.For your Sprint Challenge, you'll use historical housing data. There's a data dictionary at the bottom of the notebook. Run this code cell to load the dataset:<jupyter_code>import pandas as pd
URL = 'https://drive.google.com/uc?export=download&id=1522WlEW6HFss36roD_Cd9nybqSuiVcCK'
df = pd.read_csv(URL)
assert df.shape == (2904, 47)
'''

I should have done some exploratory analysis on some or all features.
  Look for outliers & correlation plots in the profile_report 
  Or individual feature plotting or describe methods
  
For categorical variables, since encoding splits the feature into several you 
  can't really do a correlation plot/value for the encoded SET of that 
  categorical feature, so best you can do is check category distribution.

You might be able to do chi-square tests and other tests that we will dig into
  like SHAP and Feature importance plots.
  
Otherwise, you can compare how MAE responds to each feature. But that's more
  dependent effect.
 
'''<jupyter_output><empty_output><jupyter_text>1. Do train/test splitTrain on houses sold in the years 2006 - 2008. Test on houses sold in the years 2009 - 2010.<jupyter_code># Check the "Yr_Sold" feature
df['Yr_Sold'].value_counts()
# Split the dataframe into train/test
train = df[ (df['Yr_Sold'] >= 2006) & (df['Yr_Sold'] <= 2008) ]
test = df[df['Yr_Sold'] >= 2009]
assert train.shape[0] + test.shape[0] == df.shape[0]

#OR do with query method:
#  df.query('Yr_Sold <= 2008')<jupyter_output><empty_output><jupyter_text>2. Begin with baselinesWhat is the Mean Absolute Error and R^2 score for a mean baseline?<jupyter_code>#Use mean as Baseline predictor

from sklearn.metrics import mean_absolute_error, r2_score, mean_squared_error
import numpy as np
import math

target = 'SalePrice'
y_train = train[target]
y_test = test[target]
y_pred = np.full_like(y_test,fill_value=y_train.mean())
                              # or can use [train['price'].mean()]*len(y_test)

#print(len(y_test), len(y_pred))
#print(y_pred)

print('Mean Baseline:')
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')<jupyter_output>Mean Baseline:
MAE: 58276.60467479675
R^2 score: -0.001791240523888682<jupyter_text>3. Arrange data into X features matrix and y target vectorSelect at least one numeric feature and at least one categorical feature.<jupyter_code># Pick numeric feature 
df['Bedroom_AbvGr'].value_counts()
# Pick categorical feature (preferably not high cardinality)
df['Bldg_Type'].value_counts()
from sklearn.preprocessing import StandardScaler

features = [
    'Bedroom_AbvGr',
    'Bldg_Type',
]

target = 'SalePrice'

X_train = train[features]
y_train = train[target]

X_test = test[features]
y_test  = test[target]<jupyter_output><empty_output><jupyter_text>4. Do one-hot encodingEncode your categorical feature(s).<jupyter_code>#!pip install category_encoders
X_train.shape, y_train.shape
X_test.shape
# Do one-hot encoding

import category_encoders as ce

encoder = ce.OneHotEncoder(use_cat_names=True)
X_train = encoder.fit_transform(X_train)
X_test = encoder.transform(X_test)

X_train.head()<jupyter_output><empty_output><jupyter_text>5. Use scikit-learn for Linear RegressionFit your model.<jupyter_code># Scale data so that Linear Regression equally weights all dimensions 

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
# Data has been transformed to mean of 0, and standard deviation of 1

X_train
#Obtain Linear Regression Fit

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(X_train, y_train)
#Apply Linear Regression Fit

y_pred = model.predict(X_test)<jupyter_output><empty_output><jupyter_text>6. Use regression metrics: MAE, R^2What is the Mean Absolute Error and R^2 score for your model? ***Beat the mean baseline!***<jupyter_code>print('Linear Regression:')
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')<jupyter_output>Linear Regression:
MAE: 56157.14384302609
R^2 score: 0.04159717292162646<jupyter_text>7. Get coefficients of a linear modelPrint or plot the coefficients for the features in your model.<jupyter_code>model.intercept_
model.coef_
# Let's at least look at ONE of the dimensional components of fit

x = df['Bedroom_AbvGr']
y = df['SalePrice']

import matplotlib.pyplot as plt

def plot_preds(x, y, y_pred):
  plt.scatter(x, y, label='y_true')
  plt.plot(x,y_pred, label='y_pred')
  plt.legend()
  #mae = mean_absolute_error(y,y_pred)
  #r2 = r2_score(y,y_pred)
  #print(f'Mean Absolute Error: {mae}')
  #print(f'R^2 score: {r2}')
m = model.coef_[0]
b = model.intercept_

y_pred = m*x + b
#Lines above are the same as y_pred = model.predict(X)

plot_preds(x, y, y_pred)
#SOLUTION (ALEX)

features = ['Gr_Liv_Area', 'Overall_Qual', 'Neighborhood']
X_train = train[features]
X_test = test[features]

encoder = ce.OneHotEncoder(use_cat_names=True)
X_train_encoded = encoder.fit_transform(X_train)
X_test_encoded = encoder.fit_transform(X_train)    #X_test encoded on X_train because X_test 
                                                   #misses two values because of lower statistics

X_train_encoded.head()
#SOLUTION (ALEX)
X_test_encoded.head(5)
#SOLUTION (ALEX)
model.fit(X_train_encoded, y_train)
model.predict(X_test_encoded)
#SOLUTION (ALEX)
print('Intercept', model.intercept_)
print(model.coef_)
coefficients = pd.Series(model.coef_, X_train_encoded.columns)
print(coefficients.to_string())
#Coefficient value corresponds to the amount of contribution of final prediction
# though, I'm a little skeptical because I don't think we scaled the data beforehand

%matplotlib inline
coefficients.sort_values().plot.barh(color='grey', figsize=(10,10));<jupyter_output><empty_output><jupyter_text>Stretch Goals- Try at least 3 different feature combinations.- Plot exploratory visualizations.- Get regression metrics RMSE, MAE, and $R^2$, for both the train and test data.<jupyter_code># 1st feature combination - Use 2 features to 3d plot regression
features = [
    'Overall_Qual',
    '1st_Flr_SF',
]

target = 'SalePrice'

X_train = train[features]
y_train = train[target]

X_test = test[features]
y_test  = test[target]

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

pd.Series(model.coef_,features)

print(f"{len(features)} numeric features...")
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')
print(f'Root Mean Squared Error: {math.sqrt(mean_squared_error(y_test, y_pred))}')
# 1st feature combination - check effect of scalar normalization
# Apparently, linear regression yields identical result with scaled data

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"{len(features)} numeric features...")
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')
print(f'Root Mean Squared Error: {math.sqrt(mean_squared_error(y_test, y_pred))}')
#3d plot of fit model
def configure_plotly_browser_state():
    import IPython
    display(IPython.core.display.HTML('''
        <script src="/static/components/requirejs/require.js"></script>
        <script>
          requirejs.config({
            paths: {
              base: '/static/base',
              plotly: 'https://cdn.plot.ly/plotly-latest.min.js?noext',
            },
          });
        </script>
        '''))

import itertools
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)

def viz3D(fitted_model, X, features, target='', num=100):
    """
    Visualize model predictions in 3D, for regression model fit on 2 features
    
    Parameters
    ----------
    fitted_model : scikit-learn model, already fitted
    X : pandas dataframe, which was used to fit model
    features : list of strings, column names of the 2 features
    target : string, name of target
    num : int, number of grid points for each feature
    
    References
    ----------
    https://plot.ly/python/3d-charts/
    """
    feature1, feature2 = features
    min1, max1 = X[feature1].min(), X[feature1].max()
    min2, max2 = X[feature2].min(), X[feature2].max()
    x1 = np.linspace(min1, max1, num)
    x2 = np.linspace(min2, max2, num)
    combos = list(itertools.product(x1, x2))
    Z = fitted_model.predict(combos).reshape(num, num)
    
    configure_plotly_browser_state()
    data = [go.Surface(x=x1, y=x2, z=Z)]
    layout = go.Layout(
        scene={'xaxis': {'title': feature1, 'range': [min1,max1], 'showticklabels': True}, 
               'yaxis': {'title': feature2, 'range': [min2,max2], 'showticklabels': True}, 
               'zaxis': {'title': target, 'showticklabels': True}}, 
    )
    fig = go.Figure(data=data, layout=layout)
    iplot(fig)
    
 
viz3D(model, X, features, target)
# 2nd feature combination - Add another numeric feature
features = [
    'Overall_Qual',
    '1st_Flr_SF',
    'Gr_Liv_Area'
]

target = 'SalePrice'

X_train = train[features]
y_train = train[target]

X_test = test[features]
y_test  = test[target]

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

pd.Series(model.coef_,features)

print(f"{len(features)} numeric features...")
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')
print(f'Root Mean Squared Error: {math.sqrt(mean_squared_error(y_test, y_pred))}')
# 3rd feature combination - Include all numeric features

features = df.select_dtypes('number').columns.drop('SalePrice').tolist()
target = 'SalePrice'

X_train = train[features]
y_train = train[target]

X_test = test[features]
y_test  = test[target]

model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

pd.Series(model.coef_,features)

print(f"{len(features)} numeric features...")
print(f"MAE: {mean_absolute_error(y_test, y_pred)}")
print(f'R^2 score: {r2_score(y_test,y_pred)}')
print(f'Root Mean Squared Error: {math.sqrt(mean_squared_error(y_test, y_pred))}')
# Check profile report to see which numeric features correlate most with target
!pip install --upgrade pandas-profiling
import pandas_profiling
df.profile_report()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11650938989068485, 'fraction_numerical': 0.01896664486592544, 'mean_word_length': 4.976549413735343, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '577933', 'license': 'MIT', 'n_tokens_mistral': 3887, 'n_tokens_neox': 3643, 'n_words': 971, 'path': 'DS_Sprint_Challenge_5_Linear_Models.ipynb', 'repo_name': 'martinclehman/DS-Unit-2-Linear-Models', 'size': '10703'}