<jupyter_start><jupyter_text>Bagging Exercises![Bagging.png](Assets/Bagging.png)  Learning Objectives- Associate concepts of bootstrapping and aggregating (commonly known as “bagging”) to reduce variance- Apply Random Forest algorithm that further reduces the correlation seen in bagging models- Apply Intel® Extension for Scikit-learn* to leverage underlying compute capabilities of hardware scikit-learn* Frameworks provide structure that Data Scientists use to build code. Frameworks are more than just libraries, because in addition to callable code, frameworks influence how code is written. A main virtue of using an optimized framework is that code runs faster. Code that runs faster is just generally more convenient but when we begin looking at applied data science and AI models, we can see more material benefits. Here you will see how optimization, particularly hyperparameter optimization can benefit more than just speed. These exercises will demonstrate how to apply **the Intel® Extension for Scikit-learn*,** a seamless way to speed up your Scikit-learn application. The acceleration is achieved through the use of the Intel® oneAPI Data Analytics Library (oneDAL). Patching is the term used to extend scikit-learn with Intel optimizations and makes it a well-suited machine learning framework for dealing with real-life problems. To get optimized versions of many Scikit-learn algorithms using a patch() approach consisting of adding these lines of code PRIOR to importing sklearn: - **from sklearnex import patch_sklearn**- **patch_sklearn()** This exercise relies on installation of  Intel® Extension for Scikit-learn*If you have not already done so, follow the instructions from Week 1 for instructions  IntroductionWe will be using the customer churn data from the telecom industry that we used in week 1 for this week's exercises. The data file is called `Orange_Telecom_Churn_Data.csv`.<jupyter_code>from __future__ import print_function
import os
data_path = [ 'data']

from sklearnex import patch_sklearn
patch_sklearn()

from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score
from sklearn.metrics import f1_score, roc_auc_score
from sklearn.metrics import roc_curve, precision_recall_curve, confusion_matrix<jupyter_output><empty_output><jupyter_text>Question 1* Import the customer churn data, which is found in the file `Orange_Telecom_Churn_Data.csv`. * Remove any columns that are likely not to be used for prediction.* Encode data types as appropriate.<jupyter_code>import pandas as pd
import numpy as np

filepath = os.sep.join(data_path + ['Orange_Telecom_Churn_Data.csv'])
data = pd.read_csv(filepath, sep=',')
data.head().T<jupyter_output><empty_output><jupyter_text>The prediction should definitely not include phone number. And it is unlikely that area code or state would be desired, unless there is some reason to assume the model has a very specific geographic factor.<jupyter_code>data.drop(['state', 'phone_number', 'area_code'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Examine the data types. Everything looks OK other than `int_plan` and `voice_mail_plan` need to be boolean encoded.<jupyter_code>data.dtypes
for col in ['intl_plan', 'voice_mail_plan']:
    data[col] = data[col].replace('yes','True').replace('no','False').astype(np.bool)
    
    
data[['intl_plan', 'voice_mail_plan']].dtypes<jupyter_output><empty_output><jupyter_text>Question 2* Examine distribution of the predicted variable (`churned`).* Split the data into train and test sets. Decide if a stratified split should be used or not based on the distribution.* Examine the distribution of the predictor variable in the train and test data.<jupyter_code># Data are skewed at ~85% towards non-churned customers
# This will be important to remember when model building

data.churned.value_counts()
data.churned.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Given the skew in the predictor variable, let's split the data with the *churned* values being stratified.<jupyter_code>feature_cols = [x for x in data.columns if x != 'churned']


# Split the data into two parts with 1500 points in the test data
# This creates a generator
strat_shuff_split = StratifiedShuffleSplit(n_splits=1, test_size=1500, random_state=42)

# Get the index values from the generator
train_idx, test_idx = next(strat_shuff_split.split(data[feature_cols], data['churned']))

# Create the data sets
X_train = data.loc[train_idx, feature_cols]
y_train = data.loc[train_idx, 'churned']

X_test = data.loc[test_idx, feature_cols]
y_test = data.loc[test_idx, 'churned']
y_train.value_counts(normalize=True)
y_test.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Question 3* Fit random forest models with a range of tree numbers and evaluate the out-of-bag error for each of these models.* Plot the resulting oob errors as a function of the number of trees.*Hint:* since the only thing changing is the number of trees, the `warm_start` flag can be used so that the model just adds more trees to the existing model each time. Use the `set_params` method to update the number of trees.<jupyter_code># Suppress warnings about too few trees from the early models

import warnings
warnings.filterwarnings("ignore", category=UserWarning)
warnings.filterwarnings("ignore", category=RuntimeWarning)
# Initialize the random forest estimator
# Note that the number of trees is not setup here
RF = RandomForestClassifier(oob_score=True, 
                            random_state=42, 
                            warm_start=True,
                            n_jobs=-1)

oob_list = list()

# Iterate through all of the possibilities for 
# number of trees
for n_trees in [15, 20, 30, 40, 50, 100, 150, 200, 300, 400]:
    
    # Use this to set the number of trees
    RF.set_params(n_estimators=n_trees)

    # Fit the model
    RF.fit(X_train, y_train)

    # Get the oob error
    oob_error = 1 - RF.oob_score_
    
    # Store it
    oob_list.append(pd.Series({'n_trees': n_trees, 'oob': oob_error}))

rf_oob_df = pd.concat(oob_list, axis=1).T.set_index('n_trees')

rf_oob_df<jupyter_output><empty_output><jupyter_text>The error looks like it has stabilized around 100-150 trees.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_context('talk')
sns.set_palette('dark')
sns.set_style('white')

ax = rf_oob_df.plot(legend=False, marker='o')
ax.set(ylabel='out-of-bag error');<jupyter_output><empty_output><jupyter_text>Question 4* Repeat question 3 using extra randomized trees (`ExtraTreesClassifier`). Note that the `bootstrap` parameter will have to be set to `True` for this model.* Compare the out-of-bag errors for the two different types of models.<jupyter_code># Initialize the random forest estimator
# Note that the number of trees is not setup here
EF = ExtraTreesClassifier(oob_score=True, 
                          random_state=42, 
                          warm_start=True,
                          bootstrap=True,
                          n_jobs=-1)

oob_list = list()

# Iterate through all of the possibilities for 
# number of trees
for n_trees in [15, 20, 30, 40, 50, 100, 150, 200, 300, 400]:
    
    # Use this to set the number of trees
    EF.set_params(n_estimators=n_trees)
    EF.fit(X_train, y_train)

    # oob error
    oob_error = 1 - EF.oob_score_
    oob_list.append(pd.Series({'n_trees': n_trees, 'oob': oob_error}))

et_oob_df = pd.concat(oob_list, axis=1).T.set_index('n_trees')

et_oob_df<jupyter_output><empty_output><jupyter_text>Combine the two dataframes into a single one for easier plotting.<jupyter_code>oob_df = pd.concat([rf_oob_df.rename(columns={'oob':'RandomForest'}),
                    et_oob_df.rename(columns={'oob':'ExtraTrees'})], axis=1)

oob_df<jupyter_output><empty_output><jupyter_text>The random forest model performs consistently better than the extra randomized trees.<jupyter_code>sns.set_context('talk')
sns.set_palette('dark')
sns.set_style('white')

ax = oob_df.plot(marker='o')
ax.set(ylabel='out-of-bag error');<jupyter_output><empty_output><jupyter_text>Question 5* Select one of the models that performs well and calculate error metrics and a confusion matrix on the test data set. * Given the distribution of the predicted class, which metric is most important? Which could be deceiving?<jupyter_code># Random forest with 100 estimators
model = RF.set_params(n_estimators=100)
y_test = y_test.to_numpy()
y_pred = model.predict(X_test)<jupyter_output><empty_output><jupyter_text>Unsurprisingly, recall is rather poor for the customers who churned (True) class since they are quite small. We are doing better than random guessing, though, as the accuracy is 0.932 (vs 0.85 for random guessing).<jupyter_code>cr = classification_report(y_test, y_pred)
print(cr)
score_df = pd.DataFrame({'accuracy': accuracy_score(y_test, y_pred),
                         'precision': precision_score(y_test, y_pred),
                         'recall': recall_score(y_test, y_pred),
                         'f1': f1_score(y_test, y_pred),
                         'auc': roc_auc_score(y_test.astype(int), y_pred.astype(int))},
                         index=pd.Index([0]))

print(score_df)<jupyter_output><empty_output><jupyter_text>Question 6* Print or visualize the confusion matrix.* Plot the ROC-AUC and precision-recall curves.* Plot the feature importances.<jupyter_code>sns.set_context('talk')
cm = confusion_matrix(y_test, y_pred)
ax = sns.heatmap(cm, annot=True, fmt='d')

labels = ['False', 'True']
ax.set_xticklabels(labels);
ax.set_yticklabels(labels[::-1]);
ax.set_ylabel('Prediction');
ax.set_xlabel('Ground Truth')<jupyter_output><empty_output><jupyter_text>The ROC-AUC and precision-recall curves.<jupyter_code>sns.set_context('talk')

fig, axList = plt.subplots(ncols=2)
fig.set_size_inches(11, 5)

# Get the probabilities for each of the two categories
y_prob = model.predict_proba(X_test)

# Plot the ROC-AUC curve
ax = axList[0]

fpr, tpr, thresholds = roc_curve(y_test, y_prob[:,1])
ax.plot(fpr, tpr)
# It is customary to draw a diagonal dotted line in ROC plots.
# This is to indicate completely random prediction. Deviation from this
# dotted line towards the upper left corner signifies the power of the model.
ax.plot([0, 1], [0, 1], ls='--', color='black', lw=.3)
ax.set(xlabel='False Positive Rate',
       ylabel='True Positive Rate',
       xlim=[-.01, 1.01], ylim=[-.01, 1.01],
       title='ROC curve')
ax.grid(True)

# Plot the precision-recall curve
ax = axList[1]

precision, recall, _ = precision_recall_curve(y_test, y_prob[:,1])
ax.plot(recall, precision)
ax.set(xlabel='Recall', ylabel='Precision',
       xlim=[-.01, 1.01], ylim=[-.01, 1.01],
       title='Precision-Recall curve')
ax.grid(True)

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>The feature importances. Total daily cost is the biggest predictor of customer churn.<jupyter_code>feature_imp = pd.Series(model.feature_importances_, index=feature_cols).sort_values(ascending=False)

ax = feature_imp.plot(kind='bar')
ax.set(ylabel='Relative Importance');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0925583443416997, 'fraction_numerical': 0.012417437252311756, 'mean_word_length': 4.958027282266527, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '328393', 'license': 'MIT', 'n_tokens_mistral': 3686, 'n_tokens_neox': 3474, 'n_words': 1143, 'path': 'AI-and-Analytics/Jupyter/Introduction_to_Machine_Learning/08-Bagging/Bagging_Exercises-ANSWERS.ipynb', 'repo_name': 'krisrak/oneAPI-samples', 'size': '11355'}