<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Model Validation: The Bias-Variance Tradeoff and the Train-Test Split2&nbsp;&nbsp;Objectives3&nbsp;&nbsp;Motivation4&nbsp;&nbsp;The Bias-Variance Tradeoff4.1&nbsp;&nbsp;A Model Example4.1.1&nbsp;&nbsp;Model A4.1.2&nbsp;&nbsp;Model B4.2&nbsp;&nbsp;High-Bias vs High-Variance4.2.1&nbsp;&nbsp;Bias4.2.2&nbsp;&nbsp;Variance4.2.3&nbsp;&nbsp;Balancing Bias and Variance4.3&nbsp;&nbsp;Different Models for King County Housing Data4.4&nbsp;&nbsp;🧠 Knowledge Check5&nbsp;&nbsp;Train-Test Split5.1&nbsp;&nbsp;Is the Model Overfitting or Underfitting?5.2&nbsp;&nbsp;Exercise: Name that Model!5.3&nbsp;&nbsp;Should You Ever Fit on Your Test Set?5.4&nbsp;&nbsp;Train-Test Split Our Earlier Example5.5&nbsp;&nbsp;Now check performance on test data5.6&nbsp;&nbsp;🧠 Knowledge Check5.7&nbsp;&nbsp;Same Procedure with a Polynomial Model5.8&nbsp;&nbsp;Exercise6&nbsp;&nbsp;k-Fold Cross-Validation: Even More Rigorous Validation6.1&nbsp;&nbsp;Example6.1.1&nbsp;&nbsp;Simple Model6.1.2&nbsp;&nbsp;More Complex Model6.1.3&nbsp;&nbsp;Medium-Complexity Model6.1.4&nbsp;&nbsp;Checking Our Models Against the Holdout Test Set6.1.4.1&nbsp;&nbsp;Testing Other Models<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_validate<jupyter_output><empty_output><jupyter_text>Model Validation: The Bias-Variance Tradeoff and the Train-Test Split  Objectives - **Explain** the bias-variance tradeoff and the correlative notions of underfit and overfit models- **Describe** a train-test split and **explain** its purpose in the context of predictive statistics / machine learning- **Explain** the algorithm of cross-validation  Motivation At this point, we have seen different ways to create models from our data through different linear regression techniques. That's great but just like a student practicing for a big end-of-the-year test, we want to make sure our _models_ are ready to predict on data it hasn't seen yet. We want to know if the model we made is ready to make predictions for data in the "wild". Usually, when our model is ready to be used in the "real world" we refer to this as putting our model into **production** or **deploying** our model. The data it will use to make predictions will be data its never seen before. Similar to parents sending a once teenager into the world to make it on their own, we want to make sure our model is ready for the world of new data! But you might be thinking, how do I make sure my model I've been cultivating with my own data is ready? This is where we ***model validation*** techniques to ensure our model can generalize to data it hasn't directly seen before. Going back to our analogy of a teenager ready to leave the home, we want our teenager (model) be informed so it's not naive but also flexible enough to adjust to new situations.We'll go over how to ensure our model is ready, but first we have to discuss how our model can make errors in the context of the **bias-variance tradeoff**.  The Bias-Variance Tradeoff  A Model Example Typically we'll talk about a model as how _complex_ it is in making predictions. Let's take a look at this data with just one feature and a target: ![](https://camo.githubusercontent.com/36a1cb13983f39fc58ecdfffb415b2258e73ba1c/68747470733a2f2f6769746875622e636f6d2f6c6561726e2d636f2d73747564656e74732f6473632d322d32342d30372d626961732d76617269616e63652d74726164652d6f66662d6f6e6c696e652d64732d73702d3030302f7261772f6d61737465722f696e6465785f66696c65732f696e6465785f375f312e706e67) We can probably picture how a good model will fit to this data. Let's look at a couple models and discuss how they're making mistakes.  Model A ![](img/model_simple.png) What do we observe here? How would you describe where the model is failing?  Model B ![](img/model_complex.png) What do we observe here? How would you describe where the model is failing?  High-Bias vs High-Variance We can break up how the model makes mistakes (the error) by saying there are three parts:- Error inherent of the data (noise): **irreducible error**- Error from being not capturing patterns (too simple): **bias**- Error from using patterns in the data but don't generalize well (too complex): **variance** We can summarize this in an equation for the _mean squared error_ (MSE):$MSE = Bias(\hat{y})^2 + Var(\hat{y}) + \sigma^2$ ![optimal](img/optimal_bias_variance.png)http://scott.fortmann-roe.com/docs/BiasVariance.html  Bias **High-bias** algorithms tend to be less complex, with simple or rigid underlying structure. ![](img/noisy-sine-linear.png) + They train models that are consistent, but inaccurate on average.+ These include linear or parametric algorithms such as regression and naive Bayes.+ The following sorts of difficulties could lead to high bias:  - We did not include the correct predictors  - We did not take interactions into account  - We missed a non-linear (polynomial) relationship       High-bias models are generally **underfit**: The models have not picked up enough of the signal in the data. And so even though they may be consistent, they don't perform particularly well on the initial data, and so they will be consistently inaccurate.  Variance On the other hand, **high-variance** algorithms tend to be more complex, with flexible underlying structure. ![](img/noisy-sine-decision-tree.png) + They train models that are accurate on average, but inconsistent.+ These include non-linear or non-parametric algorithms such as decision trees and nearest-neighbor models.+ The following sorts of difficulties could lead to high variance:  - We included an unreasonably large number of predictors;  - We created new features by squaring and cubing each feature. High variance models are **overfit**: The models have picked up on the noise as well as the signal in the data. And so even though they may perform well on the initial data, they will be inconsistently accurate on new data.  Balancing Bias and Variance While we build our models, we have to keep this relationship in mind.  If we build complex models, we risk overfitting our models.  Their predictions will vary greatly when introduced to new data.  If our models are too simple, the predictions as a whole will be inaccurate.    ![](img/noisy-sine-third-order-polynomial.png) The goal is to build a model with enough complexity to be accurate, but not too much complexity to be erratic. ![](img/target.png)  Different Models for King County Housing Data<jupyter_code># For consistent randomness
np.random.seed(42)
df = pd.read_csv('data/king_county.csv', index_col='id')
display(df.shape)
df.head()<jupyter_output><empty_output><jupyter_text>Let's take a random data point that we can use for prediction. Note the trained model will never see this point until after it's done training<jupyter_code>my_sample = df.sample(1)
my_sample_price = my_sample.iloc[0,0]
my_sample_pt = my_sample.drop('price', axis=1)

# Show my random sample off
print(f'Price of sample: ${my_sample_price}')
my_sample_pt<jupyter_output><empty_output><jupyter_text>Now let's generate random subsets of our data now to train different models and see how well they do in predicting  that random point<jupyter_code>def train_lr_randomly(data, sample_pt=None, ntimes=100):
    '''
    Takes in features & targets from `data` to train a linear regression with a
    random sample `ntimes`. It then returns a list of R2 scores, RMSEs, and the 
    predictions from a provided data point of features `sample_pt`.
    '''
    # To save all of our predictions
    r2 = []
    rmse = []
    # Only return predictions if there is something to predict (sample_pt given)
    point_preds = [] if (sample_pt is not None) else None

    # We'll repeat this little experiment to see how the model does
    for i in range(ntimes):
        # Creating a random sample of data to train on
        df_sample = data.sample(5000, replace=True)
        y = df_sample.price
        X = df_sample.drop('price', axis=1)

        # Our linear regression model about to be trained
        lr = LinearRegression()
        lr.fit(X, y)

        # Making predictions & evaluating on the data we used to train the model
        y_hat = lr.predict(X)
        rmse.append(np.sqrt(mean_squared_error(y, y_hat)))
        r2.append(lr.score(X, y))

        # Making a prediction on the one point the model definitely never saw
        if sample_pt is not None:
            y_hat_pt = lr.predict(sample_pt)
            # Getting just the single point to add into list
            point_preds.append(y_hat_pt[0])
    
    return r2, rmse, point_preds
# Run 100 linear regression trainings on some random data from df and compare
# it with the random sample point
r2_simple, rmse_simple, pt_preds_simple = train_lr_randomly(
                                                        data=df, 
                                                        sample_pt=my_sample_pt,
                                                        ntimes=100                            
)
ax = sns.boxplot(x=pt_preds_simple);
ax = sns.swarmplot(x=pt_preds_simple, color='orange', ax=ax)
ax.set_title(f'Predicting Sample Pt Price: ${my_sample_price:,.2f} (Simple LR)');<jupyter_output><empty_output><jupyter_text>Now let's see how we do with a more complex model (using polynomial features)<jupyter_code>df = pd.read_csv('data/king_county.csv', index_col='id')

pf = PolynomialFeatures(2)

df_poly = pd.DataFrame(pf.fit_transform(df.drop('price', axis=1)))
df_poly.index = df.index
df_poly['price'] = df['price']

cols = list(df_poly)

# move the column to head of list using index, pop and insert
cols.insert(0, cols.pop(cols.index('price')))

df_poly = df_poly.loc[:, cols]

df_poly.head(10)
# Let's make sure we have another sample point to test our models
my_sample_pt_poly = pf.transform(my_sample_pt)

# Show my random sample off
print(f'Price of sample: ${my_sample_price}')
my_sample_pt_poly.shape
# Run 100 linear regression trainings on some random polynomial features data
# from df_poly and compare it with the random sample point
r2_poly, rmse_poly, pt_preds_poly = train_lr_randomly(
                                                data=df_poly, 
                                                sample_pt=my_sample_pt_poly,
                                                ntimes=100                            
)
f, (ax1,ax2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(6,4));

# Simple model
ax = sns.boxplot(x=pt_preds_simple, ax=ax1);
ax = sns.swarmplot(x=pt_preds_simple, color='orange', ax=ax)
ax.set_title(f'Predicting Sample Pt Price: ${my_sample_price:,.2f} (Simple LR)');

# Complex model
ax = sns.boxplot(x=pt_preds_poly, ax=ax2);
ax = sns.swarmplot(x=pt_preds_poly, color='orange', ax=ax)
ax.set_title(f'Predicting Sample Pt Price: ${my_sample_price:,.2f} (Complex LR)');

# Makes spacing work better
f.tight_layout()
print('How it did on its own training data:')
print(f'\t- simple mean {np.mean(rmse_simple)}')
print(f'\t- complex mean {np.mean(rmse_poly)}')
print('\n')

print('How it did on data it never saw:')
print(f'\t- simple variance {np.std(pt_preds_simple)}')
print(f'\t- complex variance {np.std(pt_preds_poly)}')<jupyter_output><empty_output><jupyter_text>🧠 Knowledge Check ![which_model](img/which_model_is_better_2.png)  Train-Test Split It is hard to know if your model is too simple or complex by just using it on training data.We can _hold out_ part of our training sample, use it as a test sample, and then use it to monitor our prediction error.This allows us to evaluate whether our model has the right balance of bias/variance.  * **training set** —a subset to train a model.* **test set**—a subset to test the trained model.<jupyter_code>df = pd.read_csv('data/king_county.csv', index_col='id')

y = df.price
X = df[['bedrooms', 'sqft_living']]

X_train, X_test, y_train, y_test = train_test_split(X, 
                                                    y,
                                                    test_size=None,
                                                    random_state=42
)
display(X_train.head())
display(X_test.head())
print(X_train.shape)
print(X_test.shape)

print(X_train.shape[0] == y_train.shape[0])
print(X_test.shape[0] == y_test.shape[0])<jupyter_output><empty_output><jupyter_text>Is the Model Overfitting or Underfitting? If our model is not performing well on the training  data, we are probably underfitting it.   To know if our  model is overfitting the data, we need  to test our model on unseen data. We then measure our performance on the unseen data. If the model performs significantly worse on the  unseen data, it is probably  overfitting the data.   Exercise: Name that Model! Consider the following scenarios and describe them according to bias and variance. There are four possibilities:- a. The model has low bias and high variance.- b. The model has high bias and low variance.- c. The model has both low bias and low variance.- d. The model has both high bias and high variance. **Scenario 1**: The model has a low RMSE on training and a low RMSE on test.     Answer        c. The model has both low bias and low variance.     **Scenario 2**: The model has a high $R^2$ on the training set, but a low $R^2$ on the test.     Answer        a. The model has low bias and high variance.     **Scenario 3**: The model performs well on data it is fit on and well on data it has not seen.     Answer        c. The model has both low bias and low variance.       **Scenario 4**: The model has a low $R^2$ on training but high on the test set.     Answer        d. The model has both high bias and high variance.     **Scenario 5**: The model leaves out many of the meaningful predictors, but is consistent across samples.     Answer        b. The model has high bias and low variance.     **Scenario 6**: The model is highly sensitive to random noise in the training set.     Answer        a. The model has low bias and high variance.      Should You Ever Fit on Your Test Set?   ![no](https://media.giphy.com/media/d10dMmzqCYqQ0/giphy.gif) **Never fit on test data.** If you are seeing surprisingly good results on your evaluation metrics, it might be a sign that you are accidentally training on the test set.  Train-Test Split Our Earlier Example Let's go back to our KC housing data without the polynomial transformation.<jupyter_code>df = pd.read_csv('data/king_county.csv', index_col='id')
df.head()<jupyter_output><empty_output><jupyter_text>Now, we create a train-test split via the `sklearn.model_selection` package.<jupyter_code>y = df.price
X = df[['bedrooms', 'sqft_living']]

# Here is the convention for a traditional train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=42)
# Instanstiate your linear regression object
lr = LinearRegression()
# fit the model on the training set
lr.fit(X_train, y_train)
# Check the R^2 of the training data
lr.score(X_train, y_train)
lr.coef_<jupyter_output><empty_output><jupyter_text>A .506 R-squared reflects a model that explains about half of the total variance in the data.   Now check performance on test data Next, we test how well the model performs on the unseen test data. Remember, we do not fit the model again. The model has calculated the optimal parameters learning from the training set.<jupyter_code>lr.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>🧠 Knowledge Check How would you describe the bias of the model based on the above training $R^2$?     Answer        The difference between the train and test scores is low.     What does that indicate about variance?  Same Procedure with a Polynomial Model<jupyter_code>df = pd.read_csv('data/king_county.csv', index_col='id')
df.head()
poly_2 = PolynomialFeatures(4)

X_poly = pd.DataFrame(
            poly_2.fit_transform(df.drop('price', axis=1))
                      )

y = df.price
X_poly.head()
X_train, X_test, y_train, y_test = train_test_split(X_poly, y,
                                                    test_size=0.2,
                                                    random_state=42)
lr_poly = LinearRegression()

# Always fit on the training set
lr_poly.fit(X_train, y_train)

lr_poly.score(X_train, y_train)
lr_poly.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Exercise [This post about scaling and data leakage](https://datascience.stackexchange.com/questions/38395/standardscaler-before-and-after-splitting-data) explains that if you are going to scale your data, you should only train your scaler on the training data to prevent data leakage.  Perform the same train-test split as shown above for the simple model, but now scale your data appropriately.  The $R^2$ for both train and test should be the same as before.<jupyter_code>np.random.seed(42)

y = df.price
X = df[['bedrooms', 'sqft_living']]

# Train test split with random_state=42 and test_size=0.2

# Create (reasonable) polynomial features

# Scale appropriately

# fit and score the model (checkout the test set if there is time)<jupyter_output><empty_output><jupyter_text>k-Fold Cross-Validation: Even More Rigorous Validation   Our goal of using a test set is to simulate what happens when our model attempts predictions on data it's never seen before. But there's always a chance our model will by chance perform well on the test set. This is where we could use a more rigorous validation method and so we turn to **k-fold cross-validation**. ![kfolds](img/k_folds.png)[image via sklearn](https://scikit-learn.org/stable/modules/cross_validation.html) In this process, we split the dataset into a train set and holdout test sets like usual by performing a shuffling train-test split on the train set.  We then do $k$-number of _folds_ of the training data. This means we divide the training set into different sections or folds. We then take turns on using each fold as a **validation set** (or **dev set**) and train on the larger fraction. Then we calculate a validation score from the validation set the model has never seen. We repeat this process until each fold has served as a validation set. This process allows us to try out training our model and check to see if it is likely to overfit or underfit without touching the holdout test data set. If we think the model is looking good according to our cross-validation using the training data, we retrain the model using all of the training data. Then we can do one final evaluation using the test data. It's important that we hold onto our test data until the end and refrain from making adjustments to the model based on the test results.  Example<jupyter_code>X = df.drop('price', axis=1)
y = df.price
# Let's create our holdout test
X_train, X_test, y_train, y_test = train_test_split(
                                                X,
                                                y,
                                                test_size=0.2,
                                                random_state=42
)<jupyter_output><empty_output><jupyter_text>Simple Model<jupyter_code>model_simple = LinearRegression()
scores_simple = cross_validate(
                    model_simple, X_train, y_train, cv=5, 
                    return_train_score=True
)
scores_simple
# Mean train r_2
np.mean(scores_simple['train_score']), np.std(scores_simple['train_score'])
# Mean test r_2
np.mean(scores_simple['test_score']), np.std(scores_simple['test_score'])
# Fit on all the training data
model_simple.fit(X_train, y_train)
model_simple.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>More Complex Model<jupyter_code># Test out our polynomial model
poly_3 = PolynomialFeatures(3)
X_poly3 = poly_3.fit_transform(X_train)
model_poly3 = LinearRegression()
scores_complex3 = cross_validate(
                        model_poly3, X_poly3, y_train, cv=5, 
                        return_train_score=True
)
scores_complex3
# Mean train r_2
np.mean(scores_complex3['train_score']), np.std(scores_complex3['train_score']) 
# Mean test r_2
np.mean(scores_complex3['test_score']), np.std(scores_complex3['test_score'])
# Fit on all the training data
model_poly3.fit(X_poly3, y_train)
model_poly3.score(X_poly3, y_train)<jupyter_output><empty_output><jupyter_text>Medium-Complexity Model<jupyter_code># Test out our polynomial model
poly_2 = PolynomialFeatures(2)
X_poly2 = poly_2.fit_transform(X_train)
model_poly2 = LinearRegression()
scores_complex2 = cross_validate(
                        model_poly2, X_poly2, y_train, cv=5, 
                        return_train_score=True
)
scores_complex2
# Mean train r_2
np.mean(scores_complex2['train_score']), np.std(scores_complex2['train_score']) 
# Mean test r_2
np.mean(scores_complex2['test_score']), np.std(scores_complex2['test_score'])
model_poly2.fit(X_poly2, y_train)
model_poly2.score(X_poly2, y_train)<jupyter_output><empty_output><jupyter_text>Checking Our Models Against the Holdout Test Set Once we have an acceptable model, we train our model on the entire training set and score on the test to validate.<jupyter_code>best_model = model_poly2
# Remember we have to transform X_test in the same way
best_model.score(
    poly_2.transform(X_test),
    y_test
)<jupyter_output><empty_output><jupyter_text>Testing Other Models<jupyter_code># Simple model
model_simple.score(
    X_test,
    y_test
)
# Remember we have to transform X_test in the same way for this model
model_poly3.score(
    poly_3.transform(X_test),
    y_test
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0797065123273937, 'fraction_numerical': 0.0202342432666454, 'mean_word_length': 3.8484312859036676, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '85022', 'license': 'MIT', 'n_tokens_mistral': 6695, 'n_tokens_neox': 6297, 'n_words': 2485, 'path': 'Phase_2/ds-model_validation-nbz32-main/model_validation.ipynb', 'repo_name': 'BenJMcCarty/ds-east-042621-lectures', 'size': '21943'}