<jupyter_start><jupyter_text>Model Validation Assignment  We'll look at 4 methods of model validation- Performance estimation  - 2-way holdout method (**train/test split**)  - (Repeated) k-fold **cross-validation without independent test set** - Model selection (hyperparameter optimization) and performance estimation ← ***We usually want to do this***  - 3-way holdout method (**train/validation/test split**)  - (Repeated) k-fold **cross-validation with independent test set**  Source: https://sebastianraschka.com/blog/2018/model-evaluation-selection-part4.html  ASSIGNMENT options- Replicate the lesson code. [Do it "the hard way" or with the "Benjamin Franklin method."](https://docs.google.com/document/d/1ubOw9B3Hfip27hF2ZFnW3a3z9xAgrUDRReOEo-FHCVs/edit)- Apply the lesson to other datasets you've worked with before, and compare results.- Choose how to split the Bank Marketing dataset. Train and validate baseline models.- Get weather data for your own area and calculate both baselines.  _"One (persistence) predicts that the weather tomorrow is going to be whatever it was today. The other (climatology) predicts whatever the average historical weather has been on this day from prior years."_ What is the mean absolute error for each baseline? What if you average the two together? - When would this notebook's pipelines fail? How could you fix them? Add more [preprocessing](https://scikit-learn.org/stable/modules/preprocessing.html) and [imputation](https://scikit-learn.org/stable/modules/impute.html) to your [pipelines](https://scikit-learn.org/stable/modules/compose.html) with scikit-learn.- [This example from scikit-learn documentation](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html) demonstrates its improved `OneHotEncoder` and new `ColumnTransformer` objects, which can replace functionality from third-party libraries like category_encoders and sklearn-pandas. Adapt this example, which uses Titanic data, to work with another dataset.  Bank Marketing - shuffled or split by time?<jupyter_code>!wget https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip
!unzip bank-additional.zip.1
%cd bank-additional
import pandas as pd
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
import category_encoders as ce

bank = pd.read_csv('bank-additional-full.csv', sep=';')

X = bank.drop(columns='y')
y = bank['y'] == 'yes' #<--- turning yes & no to boolean
bank.head()<jupyter_output><empty_output><jupyter_text>Shuffled split sklearn has a `train_test_split` method, but no `train_validation_test_split`Here we write our own!:<jupyter_code>def train_validation_test_split(
    X, y, train_size=0.8, val_size=0.1, test_size=0.1,
    random_state=None, shuffle=True):
    
    assert train_size + val_size + test_size ==1
    
    X_train_val, X_test, y_train_val, y_test = train_test_split(
        X, y, test_size=test_size, random_state=random_state, shuffle=shuffle)
    
    X_train, X_val, y_train, y_val = train_test_split(
        X_train_val, y_train_val, test_size=val_size/(train_size + val_size),
        random_state=random_state, shuffle=shuffle)
    
    return X_train, X_val, X_test, y_train, y_val, y_test
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(X, y, shuffle=True)
[array.shape for array in (X_train, X_val, X_test, y_train, y_val, y_test)]
# checking what proportion of the time y is 1/true/yes
y_train.mean(), y_val.mean(), y_test.mean()
# sklearn expects no nulls
def no_nulls(df):
    return not any(df.isnull().sum())

no_nulls(X_train)
# creating pipeline that encodes, scales & applies Logistic Regression
pipeline = make_pipeline(
    ce.OneHotEncoder(use_cat_names=True),
    StandardScaler(),
    LogisticRegression(solver='lbfgs'))
# applying pipeline NOT USING TEST SET
# HIDE TEST SET until you have a final model
# Using the test set repeatedly will cause model to OVERFIT the data
from sklearn.metrics import accuracy_score

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_val)
accuracy_score(y_val,y_pred)<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/preprocessing/data.py:625: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  return self.partial_fit(X, y)
/usr/local/lib/python3.6/dist-packages/sklearn/base.py:465: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  return self.fit(X, y, **fit_params).transform(X)
/usr/local/lib/python3.6/dist-packages/sklearn/pipeline.py:331: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  Xt = transform.transform(Xt)<jupyter_text>Split by time<jupyter_code># No shuffling - so selected as displayed in df (by date)
X_train, X_val, X_test, y_train, y_val, y_test = train_validation_test_split(X, y, shuffle=False)
[array.shape for array in (X_train, X_val, X_test, y_train, y_val, y_test)]
# here we'll see if things have changed overtime
y_train.mean(), y_val.mean(), y_test.mean()
# applying pipeline WITHOUT TEST SET
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_val)
accuracy_score(y_val, y_pred)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09562043795620438, 'fraction_numerical': 0.01167883211678832, 'mean_word_length': 5.540572792362768, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 12, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '614061', 'license': 'MIT', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1721, 'n_words': 520, 'path': 'module-1-begin-modeling-process/LS_DS_241_Begin_modeling_process_Assignment.ipynb', 'repo_name': 'carlos-gutier/DS-Unit-2-Sprint-4-Model-Validation', 'size': '5480'}