<jupyter_start><jupyter_text>Decision Trees Exercises![DecisionTree.png](Assets/DecisionTree.png)  Learning Objectives- Recognize Decision trees and how to use them for classification problems- Recognize how to identify the best split and the factors for splitting- Explain strengths and weaknesses of decision trees- Explain how regression trees help with classifying continuous values- Describe motivation for choosing Random Forest Classifier over Decision Trees- Apply Intel® Extension for Scikit-learn* to leverage underlying compute capabilities of hardware for Random Forest Classifier scikit-learn* Frameworks provide structure that Data Scientists use to build code. Frameworks are more than just libraries, because in addition to callable code, frameworks influence how code is written. A main virtue of using an optimized framework is that code runs faster. Code that runs faster is just generally more convenient but when we begin looking at applied data science and AI models, we can see more material benefits. Here you will see how optimization, particularly hyperparameter optimization can benefit more than just speed. These exercises will demonstrate how to apply **the Intel® Extension for Scikit-learn*,** a seamless way to speed up your Scikit-learn application. The acceleration is achieved through the use of the Intel® oneAPI Data Analytics Library (oneDAL). Patching is the term used to extend scikit-learn with Intel optimizations and makes it a well-suited machine learning framework for dealing with real-life problems. To get optimized versions of many Scikit-learn algorithms using a patch() approach consisting of adding these lines of code PRIOR to importing sklearn: - **from sklearnex import patch_sklearn**- **patch_sklearn()** This exercise relies on installation of  Intel® Extension for Scikit-learn*If you have not already done so, follow the instructions from Week 1 for instructions  IntroductionWe will be using the wine quality data set for these exercises. This data set contains various chemical properties of wine, such as acidity, sugar, pH, and alcohol. It also contains a quality metric (3-9, with highest being better) and a color (red or white). The name of the file is `Wine_Quality_Data.csv`.<jupyter_code>from __future__ import print_function
import os
data_path = ['data']

# This listener will import Intel Extnsions for Scikit-learn optimized versions 
# for any applicable imports from scikit-learn once this patch has been run

from sklearnex import patch_sklearn
patch_sklearn()

from io import StringIO
from IPython.display import Image, display

from sklearn.tree import export_graphviz
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.tree import DecisionTreeClassifier<jupyter_output><empty_output><jupyter_text>Question 1* Import the data and examine the features.* We will be using all of them to predict `color` (white or red), but the colors feature will need to be integer encoded.<jupyter_code>import pandas as pd
import numpy as np

filepath = os.sep.join(data_path + ['Wine_Quality_Data.csv'])
data = pd.read_csv(filepath, sep=',')
data.head()
data.dtypes<jupyter_output><empty_output><jupyter_text>Convert the color feature to an integer. This is a quick way to do it using Pandas.<jupyter_code>data['color'] = data.color.replace('white',0).replace('red',1).astype(int)<jupyter_output><empty_output><jupyter_text>Question 2* Use `StratifiedShuffleSplit` to split data into train and test sets that are stratified by wine quality. If possible, preserve the indices of the split for question 5 below.* Check the percent composition of each quality level for both the train and test data sets.<jupyter_code># All data columns except for color
feature_cols = [x for x in data.columns if x not in 'color']
# Split the data into two parts with 1000 points in the test data
# This creates a generator
strat_shuff_split = StratifiedShuffleSplit(n_splits=1, test_size=1000, random_state=42)

# Get the index values from the generator
train_idx, test_idx = next(strat_shuff_split.split(data[feature_cols], data['color']))

# Create the data sets
X_train = data.loc[train_idx, feature_cols]
y_train = data.loc[train_idx, 'color']

X_test = data.loc[test_idx, feature_cols]
y_test = data.loc[test_idx, 'color']<jupyter_output><empty_output><jupyter_text>Now check the percent composition of each quality level in the train and test data sets. The data set is mostly white wine, as can be seen below.<jupyter_code>y_train.value_counts(normalize=True).sort_index()
y_test.value_counts(normalize=True).sort_index()<jupyter_output><empty_output><jupyter_text>Question 3* Fit a decision tree classifier with no set limits on maximum depth, features, or leaves.* Determine how many nodes are present and what the depth of this (very large) tree is.* Using this tree, measure the prediction error in the train and test data sets. What do you think is going on here based on the differences in prediction error?<jupyter_code>dt = DecisionTreeClassifier(random_state=42)
dt = dt.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>The number of nodes and the maximum actual depth.<jupyter_code>dt.tree_.node_count, dt.tree_.max_depth<jupyter_output><empty_output><jupyter_text>A function to return error metrics.<jupyter_code>def measure_error(y_true, y_pred, label):
    return pd.Series({'accuracy':accuracy_score(y_true, y_pred),
                      'precision': precision_score(y_true, y_pred),
                      'recall': recall_score(y_true, y_pred),
                      'f1': f1_score(y_true, y_pred)},
                      name=label)<jupyter_output><empty_output><jupyter_text>The decision tree predicts a little better on the training data than the test data, which is consistent with (mild)  overfitting. Also notice the perfect recall score for the training data. In many instances, this prediction difference is even greater than that seen here.<jupyter_code># The error on the training and test data sets
y_train_pred = dt.predict(X_train)
y_test_pred = dt.predict(X_test)

train_test_full_error = pd.concat([measure_error(y_train, y_train_pred, 'train'),
                              measure_error(y_test, y_test_pred, 'test')],
                              axis=1)

train_test_full_error<jupyter_output><empty_output><jupyter_text>Question 4* Replace Decision Tree in the cell above with Random Forest algorithm* from sklearn.ensemble import RandomForestClassifier * patch sklearn to apply fast version from Intel Extensions for Sckit-learn** Instantiate RandomForestClassifier(random_state=42)* Examine feature importance: dict(zip(X_train.columns, dt.feature_importances_))* Compare Metrics of Decision Tree to Random Forest<jupyter_code>from sklearnex import patch_sklearn
patch_sklearn()

from sklearn.ensemble import RandomForestClassifier

dt = RandomForestClassifier(random_state=42)
dt = dt.fit(X_train, y_train)

dict(zip(X_train.columns, dt.feature_importances_))
def measure_error(y_true, y_pred, label):
    return pd.Series({'accuracy':accuracy_score(y_true, y_pred),
                      'precision': precision_score(y_true, y_pred),
                      'recall': recall_score(y_true, y_pred),
                      'f1': f1_score(y_true, y_pred)},
                      name=label)
# The error on the training and test data sets
y_train_pred = dt.predict(X_train)
y_test_pred = dt.predict(X_test)

train_test_full_error = pd.concat([measure_error(y_train, y_train_pred, 'train'),
                              measure_error(y_test, y_test_pred, 'test')],
                              axis=1)

train_test_full_error<jupyter_output><empty_output><jupyter_text>Question 5* Using grid search with cross validation, find a decision tree that performs well on the test data set. Use a different variable name for this decision tree model than in question 3 so that both can be used in question 6.* Determine the number of nodes and the depth of this tree.* Measure the errors on the training and test sets as before and compare them to those from the tree in question 3.<jupyter_code>dt = DecisionTreeClassifier(random_state=42)
dt = dt.fit(X_train, y_train)
param_grid = {'max_depth':range(1, dt.get_depth() + 1, 2),
              'max_features': range(1, len(dt.feature_importances_)+1)}

GR = GridSearchCV(DecisionTreeClassifier(random_state=42),
                  param_grid=param_grid,
                  scoring='accuracy',
                  n_jobs=-1)

GR = GR.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>The number of nodes and the maximum depth of the tree.<jupyter_code>GR.best_estimator_.tree_.node_count, GR.best_estimator_.tree_.max_depth<jupyter_output><empty_output><jupyter_text>These test errors are a little better than the previous ones. So it would seem the previous example overfit the data, but only slightly so.<jupyter_code>y_train_pred_gr = GR.predict(X_train)
y_test_pred_gr = GR.predict(X_test)

train_test_gr_error = pd.concat([measure_error(y_train, y_train_pred_gr, 'train'),
                                 measure_error(y_test, y_test_pred_gr, 'test')],
                                axis=1)
train_test_gr_error<jupyter_output><empty_output><jupyter_text>Question 6* Re-split the data into `X` and `y` parts, this time with `residual_sugar` being the predicted (`y`) data. *Note:* if the indices were preserved from the `StratifiedShuffleSplit` output in question 2, they can be used again to split the data.* Using grid search with cross validation, find a decision tree **regression** model that performs well on the test data set.* Measure the errors on the training and test sets using mean squared error.* Make a plot of actual *vs* predicted residual sugar.<jupyter_code>feature_cols = [x for x in data.columns if x != 'residual_sugar']

# Create the data sets
X_train = data.loc[train_idx, feature_cols]
y_train = data.loc[train_idx, 'residual_sugar']

X_test = data.loc[test_idx, feature_cols]
y_test = data.loc[test_idx, 'residual_sugar']
dr = DecisionTreeRegressor().fit(X_train, y_train)

param_grid = {'max_depth':range(1, dr.tree_.max_depth+1, 2),
              'max_features': range(1, len(dr.feature_importances_)+1)}

GR_sugar = GridSearchCV(DecisionTreeRegressor(random_state=42),
                     param_grid=param_grid,
                     scoring='neg_mean_squared_error',
                      n_jobs=-1)

GR_sugar = GR_sugar.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>The number of nodes and the maximum depth of the tree. This tree has lots of nodes, which is not so surprising given the continuous data.<jupyter_code>GR_sugar.best_estimator_.tree_.node_count, GR_sugar.best_estimator_.tree_.max_depth<jupyter_output><empty_output><jupyter_text>The error on train and test data sets. Since this is continuous, we will use mean squared error.<jupyter_code>y_train_pred_gr_sugar = GR_sugar.predict(X_train)
y_test_pred_gr_sugar  = GR_sugar.predict(X_test)

train_test_gr_sugar_error = pd.Series({'train': mean_squared_error(y_train, y_train_pred_gr_sugar),
                                         'test':  mean_squared_error(y_test, y_test_pred_gr_sugar)},
                                          name='MSE').to_frame().T

train_test_gr_sugar_error<jupyter_output><empty_output><jupyter_text>A plot of actual vs predicted residual sugar.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

sns.set_context('notebook')
sns.set_style('white')
sns.set_palette('dark')

%matplotlib inline
fig = plt.figure(figsize=(6,6))
ax = plt.axes()

ph_test_predict = pd.DataFrame({'test':y_test.values,
                                'predict': y_test_pred_gr_sugar}).set_index('test').sort_index()

ph_test_predict.plot(marker='o', ls='', ax=ax)
ax.set(xlabel='Test', ylabel='Predict', xlim=(0,35), ylim=(0,35));<jupyter_output><empty_output><jupyter_text>Question 7 *(Optional)*This question is optional as it requires an additional command line program (GraphViz) and Python library (PyDotPlus). GraphViz can be installed with a package manager on Linux and Mac. For PyDotPlus, either `pip` or `conda` (`conda install -c conda-forge pydotplus`) can be used to install the library.Once these programs are installed:* Create a visualization of the decision tree from question 3, where wine color was predicted and the number of features and/or splits are not limited.* Create a visualization of the decision tree from question 4, where wine color was predicted but a grid search was used to find the optimal depth and number of features.The decision tree from question 5 will likely have too many nodes to visualize.<jupyter_code>try:
    import pydotplus
    pydotplus_installed = True
    
except:
    print('PyDotPlus must be installed to execute the remainder of the cells associated with this question.')
    print('Please see the instructions for this question for details.')
    pydotplus_installed = False<jupyter_output><empty_output><jupyter_text>The tree from question 3.<jupyter_code>if pydotplus_installed:
    
    # Create an output destination for the file
    dot_data = StringIO()

    export_graphviz(dt, out_file=dot_data, filled=True)
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())

    # View the tree image
    filename = 'wine_tree.png'
    graph.write_png(filename)
    img = Image(filename=filename)
    display(img)
    
else:
    print('This cell not executed because PyDotPlus could not be loaded.')<jupyter_output><empty_output><jupyter_text>The tree fit with cross validation from question 4. This tree is much shallower than the previous one.<jupyter_code>if pydotplus_installed:
    
    # Create an output destination for the file
    dot_data = StringIO()

    export_graphviz(GR.best_estimator_, out_file=dot_data, filled=True)
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())

    # View the tree image
    filename = 'wine_tree_prune.png'
    graph.write_png(filename)
    img = Image(filename=filename) 
    display(img)
    
else:
    print('This cell not executed because PyDotPlus could not be loaded.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07628062360801782, 'fraction_numerical': 0.005080734966592428, 'mean_word_length': 4.729266347687401, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '530162', 'license': 'MIT', 'n_tokens_mistral': 4292, 'n_tokens_neox': 4143, 'n_words': 1448, 'path': 'AI-and-Analytics/Jupyter/Introduction_to_Machine_Learning/07-Decision_Trees/Decision_Trees_Exercises-ANSWERS.ipynb', 'repo_name': 'krisrak/oneAPI-samples', 'size': '14368'}