<jupyter_start><jupyter_text>Cross-Validation Lesson_Authors: <NAME> (SF), <NAME> (DC), <NAME> (SF), <NAME>(AUS)_--- Learning Objectives- **Describe** train/test split and cross-validation.- **Explain** how these validation techniques differ and why we want to use them.- **Split** data into testing and training sets using both train/test split and cross-validation and **apply** both techniques to score a model.  Overfitting and Underfitting--- ![](http://tomrobertshaw.net/img/2015/12/overfitting.jpg) **What's wrong with the first model?**- The underfit model falls short of capturing the complexity of the "true model" of the data.**What's wrong with the third model?**- The overfit model is too complex and is modeling random noise in the data.**The middle model is a good compromise.**- It approximates the complexity of the true model and does not model random noise in our sample as true relationships.  Importing libraries---We'll need the following libraries for today's lesson:- `pandas`- `numpy`- `seaborn`- we'll need a `LinearRegression` from `sklearn.linear_model`- from `sklearn.model_selection` we'll need `train_test_split`, and `cross_val_score`<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split, cross_val_score<jupyter_output><empty_output><jupyter_text>Load the Data---Today's dataset (`Advertising.csv`) is from the [ISLR website](https://www.statlearning.com/). Drop `Unnamed: 0` once you've loaded the csv into a `DataFrame`.<jupyter_code>df = pd.read_csv('datasets/Advertising.csv')
df.head()
df.drop(columns='Unnamed: 0',axis=1,inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Data cleaning---Check the following in the cells below:1. Do we have any null values?2. Do we have all numerical columns?<jupyter_code># Check for nulls
df.isnull().sum()
# Check column data types
df.info()
df.dtypes<jupyter_output><empty_output><jupyter_text>EDA: Correlation Matrix Heatmap---Heatmaps are an effective way to visually examine the correlational structure of your predictors.<jupyter_code>sns.heatmap(df.corr(),annot=True,vmin=-1,vmax=1,cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>EDA: Pair Plot---Let's visualize our correlations from another angle. Use seaborn's `.pairplot()` method to create scatterplots for each of our columns.<jupyter_code>sns.pairplot(df)<jupyter_output><empty_output><jupyter_text>Create our feature matrix (`X`) and target vector (`y`)---The following columns will be our features:- `TV`- `radio`- `newspaper`The `sales` column is our label: the column we're trying to predict.In the cell below, create your `X` and `y` variables.<jupyter_code>features = ['TV','radio','newspaper']
X=df[features]
y=df['sales']<jupyter_output><empty_output><jupyter_text>Train/Test Split and Model Validation---So far we've focused on fitting the best model to our data. But is this the best model for our sample data or the best model overall? How do we know?In practice we need to validate our model's ability to generalize to new data. One popular method for performing model validation is by splitting our data into subsets: data on which we *train* our model and data on which we *test* our model.The most basic type of "hold-out" validation is called **train/test split**. We split our data into two pieces:> **"A Training Set":** The subset of the data on which we fit our model.> **"A Testing Set":** The subset of the data on which we evaluate the quality of our predictions.**Train/Test Split Benefits:**- Testing data can represent "future" data; for prediction-oriented models, it's critical to ensure that a model that is performing well on current data will likely perform well on future data.- It can help diagnose and avoid overfitting via model tuning.- It can improve the quality of our predictions.  Scikit-Learn's `train_test_split` Function---Performing train/test splits using scikit-learn is easy — load the `train_test_split` function:```pythonfrom sklearn.model_selection import train_test_split```**Arguments**:- *Arrays*: Any number of arrays/matrices to split up into training and testing sets (they should be the same length).- `test_size`: An integer representing the exact size of the testing subset or a float for a percentage.- `train_size`: Alternatively, you can specify the training size.- `stratify`: Supply a vector to stratify the splitting (by more important classification tasks).- `random_state`: a numerical seed for randomly splitting your data with reproducibility**Perform a split of our `X` and `y`.**<jupyter_code>X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=2)<jupyter_output><empty_output><jupyter_text>Linear Regression model---In the cells below:1. create a `LinearRegression` model2. fit it to your **training data** (`X_train`, `y_train`)<jupyter_code>lr = LinearRegression()
lr.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Model Evaluation---We need two scores to determine if our model is performing well, or experiencing high bias/variance. Use the `.score()` method on **both** the training set (`X_train`, `y_train`) and the test set (`X_test`, `y_test`).```pythonmodel.score(X_train, y_train)```<jupyter_code># Train score
lr.score(X_train, y_train)
# Test score
lr.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>K-Fold Cross-Validation---K-fold cross-validation takes the idea of a single train/test split and expands it to *multiple tests* across different train/test splits of your data.**K-fold cross-validation builds K models — one for each train/test pair — and evaluates those models on each respective test set.** K-Fold Cross-Validation Visually---Special Cases:- **When K=2**: This is equivalent to performing ***two*** mirror image 50-50 train/test splits.- **When K=number of rows**: This is known as "leave-one-out cross-validation," or LOOCV. A model is built on all but one row and tested on the single excluded observation.  K-Fold Example---To understand how cross validation is done, we'll manually code an example. We'll choose `k` to be 5.<jupyter_code>X_1 = X_train.values[:30, :]
y_1 = y_train.values[:30]

X_2 = X_train.values[30:60, :]
y_2 = y_train.values[30:60]

X_3 = X_train.values[60:90, :]
y_3 = y_train.values[60:90]

X_4 = X_train.values[90:120, :]
y_4 = y_train.values[90:120]

X_5 = X_train.values[120:, :]
y_5 = y_train.values[120:]<jupyter_output><empty_output><jupyter_text>**First Test Score: `X_1`, `y_1`**<jupyter_code>X_fold = np.concatenate((X_2, X_3, X_4, X_5), axis=0)
y_fold = np.concatenate((y_2, y_3, y_4, y_5), axis=0)

lr_fold = LinearRegression()
lr_fold.fit(X_fold, y_fold)
score_1 = lr_fold.score(X_1, y_1)
score_1<jupyter_output><empty_output><jupyter_text>**Second Test Score: `X_2`, `y_2`**<jupyter_code>X_fold = np.concatenate((X_1, X_3, X_4, X_5), axis=0)
y_fold = np.concatenate((y_1, y_3, y_4, y_5), axis=0)

lr_fold = LinearRegression()
lr_fold.fit(X_fold, y_fold)
score_2 = lr_fold.score(X_2, y_2)
score_2<jupyter_output><empty_output><jupyter_text>**Third Test Score: `X_3`, `y_3`**<jupyter_code>X_fold = np.concatenate((X_1, X_2, X_4, X_5), axis=0)
y_fold = np.concatenate((y_1, y_2, y_4, y_5), axis=0)

lr_fold = LinearRegression()
lr_fold.fit(X_fold, y_fold)
score_3 = lr_fold.score(X_3, y_3)
score_3<jupyter_output><empty_output><jupyter_text>**Fourth Test Score: `X_4`, `y_4`**<jupyter_code>X_fold = np.concatenate((X_1, X_2, X_3, X_5), axis=0)
y_fold = np.concatenate((y_1, y_2, y_3, y_5), axis=0)

lr_fold = LinearRegression()
lr_fold.fit(X_fold, y_fold)
score_4 = lr_fold.score(X_4, y_4)
score_4<jupyter_output><empty_output><jupyter_text>**Fifth Test Score: `X_5`, `y_5`**<jupyter_code>X_fold = np.concatenate((X_1, X_2, X_3, X_4), axis=0)
y_fold = np.concatenate((y_1, y_2, y_3, y_4), axis=0)

lr_fold = LinearRegression()
lr_fold.fit(X_fold, y_fold)
score_5 = lr_fold.score(X_5, y_5)
score_5<jupyter_output><empty_output><jupyter_text>**Average Them Together**The average of our five scores is colloquially known as the **cross val score**.The cross val score is more conservative than your test score, because it's an average of five test scores.<jupyter_code>scores = np.array([score_1,score_2,score_3,score_4,score_5])
scores
scores.mean()<jupyter_output><empty_output><jupyter_text>Or you can use the `cross_val_score` function from `sklearn`.<jupyter_code>scores = cross_val_score(lr,X_train,y_train,cv = 5)
scores.mean()<jupyter_output><empty_output><jupyter_text>Putting it all together---In order to evaluate your model, you'll want three scores:- Train- Test- Cross ValIf Test and Cross Val are similar, then you have representative test set. If they diverge, then you probably have a large sampling error.<jupyter_code># Train/Test Split
X_train,X_test,y_train,y_test = train_test_split(X,y,random_state=42)
# Instantiate/Fit Model
lr = LinearRegression()
lr.fit(X_train,y_train)
# Train Score
lr.score(X_train,y_train)
# Test Score
lr.score(X_test,y_test)
# Cross Val Score
cross_val_score(lr,X_train,y_train,cv = 5).mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11890710382513661, 'fraction_numerical': 0.017158469945355193, 'mean_word_length': 6.222573007103394, 'pattern_counts': {'":': 2, '<': 77, '<?xml version=': 0, '>': 79, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '117347', 'license': 'Apache-2.0', 'n_tokens_mistral': 3204, 'n_tokens_neox': 3055, 'n_words': 971, 'path': 'Notebook/Lesson-cross-validation-train-test-split/starter-code.ipynb', 'repo_name': 'sgdataguru/ITE_Machine_Learning_Workshop', 'size': '9181'}