<jupyter_start><jupyter_text>Module 4: Model TrainingAt last, it's time to build our models! Remember, professional data scientists actually spend the bulk of their time on the 3 steps leading up to this one: 1. Exploratory Analysis2. Data Cleaning3. Feature EngineeringAgain, that's because **better data beats fancier algorithms**. In this module...First, we'll load our analytical base table from Module 2. Then, we'll go through the essential modeling steps:1. [Split your dataset](split)2. [Build model pipelines](pipelines)3. [Declare hyperparameters to tune](hyperparameters)4. [Fit and tune models with cross-validation](fit-tune)5. [Evaluate metrics](evaluate)6. [Area under ROC curve](auroc)Finally, we'll save the best model to use in the next module. Since we've seen these steps in Project 2 already, we'll try to speed through most of them. However, we'll spend a little extra time on the last 2 steps because classification tasks require new metrics.  First, let's import libraries, recruit models, and load the ABT.First, let's import the libraries that we'll need.<jupyter_code># print_function for compatibility with Python 3
from __future__ import print_function
print('Print function ready to serve.')

# NumPy for numerical computing
import numpy as np

# Pandas for DataFrames
import pandas as pd
pd.set_option('display.max_columns', 100) 

# Matplotlib for visualization
from matplotlib import pyplot as plt

# display plots in the notebook
%matplotlib inline

# Seaborn for easier visualization
import seaborn as sns

# Scikit-Learn for Modeling
import sklearn

# Pickle for saving model files
import pickle<jupyter_output>Print function ready to serve.<jupyter_text>Next, let's import the algorithms we introduced in Module 3.<jupyter_code># Import Logistic Regression
from sklearn.linear_model import LogisticRegression

# Import RandomForestClassifier and GradientBoostingClassifer
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier<jupyter_output><empty_output><jupyter_text>Next, let's import the Scikit-Learn functions and helpers we'll need for this module.* Remember, it's OK to just add more imports here as you need them and re-run this cell (it won't change anything else you've done).<jupyter_code># Function for splitting training and test set
from sklearn.model_selection import train_test_split

# Function for creating model pipelines
from sklearn.pipeline import make_pipeline

# For standardization
from sklearn.preprocessing import StandardScaler

# Helper for cross-validation
from sklearn.model_selection import GridSearchCV

# Classification metrics (added later)
from sklearn.metrics import roc_curve, auc<jupyter_output><empty_output><jupyter_text>Finally, let's read the analytical base table you saved at the end of Module 2.<jupyter_code># Load analytical base table from Module 2
df = pd.read_csv('analytical_base_table.csv')<jupyter_output><empty_output><jupyter_text>1. Split your datasetJust as we did in Project 2, let's start by splitting our data into separate training and test sets. **First, separate the dataframe into separate objects for the target variable, y, and the input features, X.**<jupyter_code>df.columns
# Create separate object for target variable
y = df.status

# Create separate object for input features
X = df.drop('status', axis=1)<jupyter_output><empty_output><jupyter_text>**After you've imported the train_test_split() function, split X and y into training and test sets.*** Pass in the argument test_size=0.2 to set aside 20% of our observations for the test set.* Pass in random_state=1234 to set the random state for replicable results.* **Important:** Also pass in the argument stratify=df.status in order to make sure the target variable's classes are balanced in each subset of data! This is **stratified random sampling**.* Then, print the number of observations in each subset to check that it was done correctly.<jupyter_code># Split X and y into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1234, stratify=df.status)

# Print number of observations in X_train, X_test, y_train, and y_test
print(len(X_train), len(X_test), len(y_train), len(y_test))<jupyter_output>11254 2814 11254 2814<jupyter_text>Back to Contents  2. Build model pipelinesNext, let's set up preprocessing pipelines for each of our algorithms.**Create a single pipeline dictionary with pipelines for each algorithm**.* Use the keys:    * 'l1' for $L_1$-regularized logistic regression    * 'l2' for $L_2$-regularized logistic regression    * 'rf' for random forest    * 'gb' for gradient boosted tree.* Each pipeline should standardize the data first.* Remember to set random_state=123 for each algorithm to ensure replicable results.<jupyter_code># Pipeline dictionary
pipelines = {
    'l1' : make_pipeline(StandardScaler(), LogisticRegression(penalty='l1', random_state=123)),
    'l2' : make_pipeline(StandardScaler(), LogisticRegression(penalty='l2', random_state=123)),
    'rf' : make_pipeline(StandardScaler(), RandomForestClassifier(random_state=123)),
    'gb' : make_pipeline(StandardScaler(), GradientBoostingClassifier(random_state=123))
}<jupyter_output><empty_output><jupyter_text>Back to Contents  3. Declare hyperparameters to tuneNext, let's declare hyperparameters to tune.**First, list the tunable hyperparameters of your $L_1$-regularized logistic regression pipeline.**<jupyter_code># List tuneable hyperparameters of our Logistic pipeline
pipelines['l1'].get_params()<jupyter_output><empty_output><jupyter_text>Let's declare the **hyperparameter grids** to tune.<jupyter_code># Logistic Regression hyperparameters
l1_hyperparameters = {
    'logisticregression__C' : np.linspace(1e-3, 1e3, 10),
}
l2_hyperparameters = {
    'logisticregression__C' : np.linspace(1e-3, 1e3, 10),
}<jupyter_output><empty_output><jupyter_text>**Declare the hyperparameter grid for the random forest.**<jupyter_code># Random Forest hyperparameters
rf_hyperparameters = {
    'randomforestclassifier__n_estimators': [100, 200],
    'randomforestclassifier__max_features': ['auto', 'sqrt', 0.33]
}<jupyter_output><empty_output><jupyter_text>**Declare the hyperparameter grid for the boosted tree.**<jupyter_code># Boosted Tree hyperparameters
gb_hyperparameters = {
    'gradientboostingclassifier__n_estimators': [100, 200],
    'gradientboostingclassifier__learning_rate': [0.05, 0.1, 0.2],
    'gradientboostingclassifier__max_depth': [1, 3, 5]
}<jupyter_output><empty_output><jupyter_text>**Create a hyperparameters dictionary**.* Use the same keys as in the pipelines dictionary.<jupyter_code># Create hyperparameters dictionary
hyperparameters = {
    'l1' : l1_hyperparameters,
    'l2' : l2_hyperparameters,
    'rf' : rf_hyperparameters,
    'gb' : gb_hyperparameters
}<jupyter_output><empty_output><jupyter_text>Back to Contents  4. Fit and tune models with cross-validationNow that we have our pipelines and hyperparameters dictionaries declared, we're ready to tune our models with **cross-validation**.**Create a fitted_models dictionary that includes models that have been tuned using cross-validation.*** The keys should be the same as those in the pipelines and hyperparameters dictionaries. * The values should be GridSearchCV objects that have been fitted to X_train and y_train.* After fitting each model, print '{name} has been fitted.' just to track the progress.* (Optionally) You can set n_jobs=-1 to use as many cores as available on your computer.This step can take a few minutes, so please be patient.<jupyter_code># Create empty dictionary called fitted_models
fitted_models = {}

# Loop through model pipelines, tuning each one and saving it to fitted_models
for name, pipeline in pipelines.items():
    # Create cross-validation object from pipeline and hyperparameters
    model = GridSearchCV(pipeline, hyperparameters[name], cv=10, n_jobs=-1)
    
    # Fit model on X_train, y_train
    model.fit(X_train, y_train)
    
    # Store model in fitted_models[name] 
    fitted_models[name] = model
    
    # Print '{name} has been fitted'
    print(name, 'has been fitted.')<jupyter_output>l1 has been fitted.
l2 has been fitted.
rf has been fitted.
gb has been fitted.<jupyter_text>5. Evaluate metricsFinally, it's time to evaluate our models and pick the best one.**First, display the best\_score_ attribute for each fitted model.**<jupyter_code># Display best_score_ for each fitted model
for name, model in fitted_models.items():
    print(name, model.best_score_)<jupyter_output>l1 0.849386884663
l2 0.849386884663
rf 0.979651679403
gb 0.975208814644<jupyter_text>If you get different numbers, check to see that you've set the random_state= correctly for each of the models.Back to Contents  6. Area under ROC curve**Area under ROC curve** is the most reliable metric for classification tasks.**Add this import to the top of your Companion Workbook (good practice for keeping all your imports in one place.** * Remember to run the entire code cell afterward.<jupyter_code># Classification metrics
from sklearn.metrics import roc_curve, auc<jupyter_output><empty_output><jupyter_text>Before presenting the idea of an ROC curve, we must first discuss what a **confusion matrix** is. Let's see an example using our $L_1$-regularized logistic regression. * First, let's use .predict() to get the predicted classes directly.<jupyter_code># Predict classes using L1-regularized logistic regression 
pred = fitted_models['l1'].predict(X_test)

# Display first 5 predictions
pred[:10]<jupyter_output><empty_output><jupyter_text>* Next, let's display the confusion matrix:<jupyter_code># Import confusion_matrix
from sklearn.metrics import confusion_matrix

# Display confusion matrix for y_test and pred
print( confusion_matrix(y_test, pred))<jupyter_output>[[1969  172]
 [ 221  452]]<jupyter_text>Remember, we can predict a **probability** for each class using .predict_proba(), instead of the class directly.For example:<jupyter_code># Predict PROBABILITIES using L1-regularized logistic regression
pred = fitted_models['l1'].predict_proba(X_test)

# Get just the prediction for the positive class (1)
pred = [p[1] for p in pred]

# Display first 5 predictions
pred[:10]<jupyter_output><empty_output><jupyter_text>Let's calculate the ROC curve using the roc_curve() function that we imported earlier.<jupyter_code># Calculate ROC curve from y_test and pred
fpr, tpr, thresholds = roc_curve(y_test, pred)<jupyter_output><empty_output><jupyter_text>We can throw these into a DataFrame for convenience and look at the last 10:<jupyter_code># Store fpr, tpr, thresholds in DataFrame and display last 10
pd.DataFrame({'FPR' : fpr, 'TPR' : tpr, 'Thresholds' : thresholds}).tail(10)<jupyter_output><empty_output><jupyter_text>As you can see, as you decrease the threshold, both the false positive rate **and** the true positive rate increase.We can plot the entire curve.<jupyter_code># Initialize figure
fig = plt.figure(figsize=(8,8))
plt.title('Receiver Operating Characteristic')

# Plot ROC curve
plt.plot(fpr, tpr, label='l1')
plt.legend(loc='lower right')

# Diagonal 45 degree line
plt.plot([0,1],[0,1],'k--')

# Axes limits and labels
plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output><jupyter_text>To calculate AUROC, use the auc() function we imported earlier in conjunction with the roc_curve() function.<jupyter_code># Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_test, pred)

# Calculate AUROC
print( auc(fpr, tpr) )<jupyter_output>0.901539531388<jupyter_text>Now that we've taken a detour to dive into some of the intuition behind AUROC, let's calculate it for each of our fitted models on the test set.**Using a for loop, print the performance of each model in fitted_models on the test set.*** Print the auc of the roc_curve.* Label the output with the name of the algorithm. For example:rf 0.991520189216<jupyter_code># Code here
for name, model in fitted_models.items():
    pred = model.predict_proba(X_test)
    pred = [p[1] for p in pred]
    
    fpr, tpr, thresholds = roc_curve(y_test, pred)
    print( name, auc(fpr, tpr))<jupyter_output>l1 0.901539531388
l2 0.901538837374
rf 0.991520189216
gb 0.988680283685<jupyter_text>**Finally, save the winning Pipeline object into a pickle file.*** Just save the Pipeline, not the GridSearchCV object.* **Hint:** Remember the .best\_estimator_ attribute?<jupyter_code>fitted_models['rf'].best_estimator_
# Save winning model as final_model.pkl
with open('final_model.pkl', 'wb') as f:
    pickle.dump(fitted_models['rf'].best_estimator_, f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.086549799544061, 'fraction_numerical': 0.02507664491785237, 'mean_word_length': 5.932970027247956, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 90, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '24433', 'license': 'MIT', 'n_tokens_mistral': 4088, 'n_tokens_neox': 3760, 'n_words': 1395, 'path': 'Module 4 - Model Training.ipynb', 'repo_name': 'nmlook/EDS-Project-3-Workbook-Bundle', 'size': '12721'}