<jupyter_start><jupyter_text>ML Course, Bogotá, Colombia  (&copy; <NAME>; June 2019)<jupyter_code>%run ../talktools.py<jupyter_output><empty_output><jupyter_text>Building Trees Rigorously (Node Splitting Criteria)  Collections of Trees ("Decision Forests", "Random Forests")  Classifying NIST Handwritten DigitsWe will try to classify handwritten digits (0-9) from their raw pixelated images.Each image is 8 $\times$ 8 pixels.  We will not do any feature extraction and instead classify based on the intensity values for each pixel.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, metrics
# import NIST digits data set (1797 8x8 images)
digits = datasets.load_digits()

f,axs = plt.subplots(1,10,figsize=(16, 6))
objarr = np.empty_like(axs)
for n, ax in enumerate(axs.flat):
    objarr.flat[n] = ax.imshow(digits['images'][n], cmap='gray', interpolation='nearest')
    ax.get_xaxis().set_ticks([])
    ax.get_yaxis().set_ticks([])<jupyter_output><empty_output><jupyter_text>$\textbf{Now we split the data into a training and testing set.}$$\textbf{We will only fit the classifier on the training set and use the testing set to evaluate performance.}$<jupyter_code># take the first 500 as training
train = 500
Xtr = digits['data'][:train]
Ytr = digits['target'][:train]
print("training size: " + str(len(Ytr)))

# testing set
Xte = digits['data'][train:]
Yte = digits['target'][train:]
print("testing size: " + str(len(Yte)))<jupyter_output><empty_output><jupyter_text>Fit a Random Forest Classifier<jupyter_code># Create a classifier: 
from sklearn.ensemble import RandomForestClassifier

# instantiate classifier object
classifier = RandomForestClassifier(n_estimators=50)

# fit the classification model on training set
classifier.fit(Xtr, Ytr)

# make predictions for testing set
pred_rf = classifier.predict(Xte) 

print("True Class / Predicted class")
print(np.vstack((Yte[0:10],pred_rf[0:10])))<jupyter_output><empty_output><jupyter_text>http://cs.stanford.edu/people/karpathy/svmjs/demo/demoforest.html  Error EstimationQ: What evaluation metrics are available?Loss Functions- `metrics.zero_one(y_true, y_pred)`Zero-One classification loss- `metrics.hinge_loss(y_true, pred_decision[, ...])`Cumulated hinge loss (non-regularized).- `metrics.mean_square_error(y_true, y_pred)`Mean square error regression lossScore Functions- `metrics.zero_one_loss(y_true, y_pred)`Zero-One classification score- `metrics.auc(x, y)`Compute Area Under the Curve (AUC)- `metrics.precision_score(y_true, y_pred[, ...])`Compute the precision- `metrics.recall_score(y_true, y_pred[, pos_label])`Compute the recall- `metrics.fbeta_score(y_true, y_pred, beta[, ...])`Compute fbeta score- `metrics.f1_score(y_true, y_pred[, pos_label])`Compute f1 scoreEvaluation Plots- `metrics.confusion_matrix(y_true, y_pred[, ...])` Compute confusion matrix to evaluate the accuracy of a classification- `metrics.roc_curve(y_true, y_score)` Compute Receiver operating characteristic (ROC)- `metrics.precision_recall_curve(y_true, ...)` Compute precision-recall pairs for different probability thresholds<jupyter_code># compute zero-one loss / score & confusion matrix
from sklearn import metrics

rf_01 = metrics.zero_one_loss(Yte, pred_rf) # zero-one loss
rf_01_score = metrics.accuracy_score(Yte, pred_rf) # zero-one score
rf_confmat = metrics.confusion_matrix(Yte, pred_rf) # conf mat

print("Zero-One Loss: " + str(rf_01))
print("Zero-One Score: " + str(rf_01_score))
print("Confusion Matrix:")
print("[i, j] is the # of objects truly in group i but predicted to be in group j")
print(rf_confmat)
import seaborn as sns
sns.heatmap(rf_confmat, annot=True,  fmt='', 
            xticklabels=[str(x) for x in range(10)], yticklabels=[str(x) for x in range(10)])
# plot some digits that we get wrong
wrong = np.where(pred_rf != Yte)[0][:9]

f,axs = plt.subplots(3,3,figsize=(7, 7))
objarr = np.empty_like(axs)

for n, ax in enumerate(axs.flat):
    objarr.flat[n] = ax.imshow(np.reshape(Xte[wrong[n]],(8,8)).astype(int),
                              cmap='gray_r', interpolation='nearest')
    ax.grid(False)
    ax.xaxis.set_visible(False)
    ax.yaxis.set_visible(False)
    ax.set_title("True = " + str(int(Yte[wrong[n]])) +". Pred = " + str(int(pred_rf[wrong[n]])))
# compute precision and recall
# Note: precision & recall are for 2-class; multi-class returns weighted avg. prec/recall

rf_precision = metrics.precision_score(Yte, pred_rf,average="weighted") # TP / (TP + FP)
rf_recall = metrics.recall_score(Yte, pred_rf,average="weighted") # TP / (TP + FN)

print("Avg. Precision: ",rf_precision)
print("Avg. Recall: ", rf_recall)
#  ROC curve, AUC for RF classifier using digit = 1
digit = 1
Yte_1 = list(map(lambda x: x == digit and 1. or 0.,Yte)) # does Y = digit

pred_rf_prob = classifier.predict_proba(Xte) 

pred_rf_prob_1 = pred_rf_prob[:,digit]
fpr, tpr, thresholds = metrics.roc_curve(Yte_1, pred_rf_prob_1)

f, ax = plt.subplots(1,1,figsize=(5, 5))
ax.plot(fpr,tpr,'b-',linewidth=3)
ax.set_xlim([0.,0.3])
ax.set_ylim([0.6,1.0005])
ax.set_xlabel("False Positive Rate",size=15)
ax.set_ylabel("True Positive Rate",size=15)
ax.set_title("ROC Curve for NIST digit={} RF Classifier".format(digit),size=22)
print("AUC for digit={}: ".format(digit) + str(metrics.auc(fpr,tpr)))<jupyter_output><empty_output><jupyter_text>$\textbf{Tuning the Classifier}$  ======================================================For now we have used an RF classifier with the choice of `n_estimators` and the default parameters.Q: How do I choose which model and (hyper) parameters to use? - KNN with what  of neighbors? - SVM which what kernel & bandwidth? - RF with how many estimators and which max_features? - GP with what kernel & bandwidth? **Solution: use `grid_search.GridSearchCV`**:`grid_search.GridSearchCV(estimator, param_grid, loss_func, n_jobs, cv=None)`Computes cv-fold cross-validated loss_func (or score_func) of estimator over a param_grid on n_jobs cores, and returns the best model!Let's see how we can rigorously find the optimal model using cross-validation and grid search.<jupyter_code># Find the best Random Forest classifier

from sklearn import model_selection
from sklearn.ensemble import RandomForestClassifier

import logging
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')


# explore 3 different forest sizes and 3 choices of mtry
parameters = {'n_estimators':[20,50,100],  'max_features':[8,10,'auto'], 
             'criterion': ['gini','entropy']}
rf_tune = model_selection.GridSearchCV(RandomForestClassifier(), parameters, 
                                   n_jobs = -1, cv = 5,verbose=1)
rf_opt = rf_tune.fit(Xtr, Ytr)

print("Best zero-one score: " + str(rf_opt.best_score_) + "\n")
print("Optimal Model:\n" + str(rf_opt.best_estimator_))
rf_opt.best_params_<jupyter_output><empty_output><jupyter_text>Towards interpretabilityWhich features are important in my model?<jupyter_code>%matplotlib inline
feature_importances = rf_opt.best_estimator_.feature_importances_
feature_importances = feature_importances.reshape(8,8)
print(feature_importances.shape)
sns.heatmap(feature_importances, cmap=plt.cm.viridis)<jupyter_output><empty_output><jupyter_text>**sklearn FRE**https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.htmlFeature ranking with recursive feature elimination.Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.<jupyter_code>from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets

digits = datasets.load_digits()
X = digits['data']
Y = digits['target']

clf = RandomForestClassifier(n_estimators=100)
rfe = RFE(estimator=clf, n_features_to_select=1, step=1)
rfe.fit(X, Y)
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns 

ranking = rfe.ranking_.reshape(digits.images[0].shape)
sns.heatmap(ranking, cmap=plt.cm.viridis_r)<jupyter_output><empty_output><jupyter_text>** LightGBM **A fast gradient boosting framework that uses tree-based learning algorithm, generally useful for "big" datasets (say 10k rows).<jupyter_code># on a mac: brew install libomp
# !conda install -c conda-forge lightgbm graphviz -y
import lightgbm
import pandas as pd
import lightgbm as lgb

import matplotlib.pyplot as plt

%matplotlib inline

try:
    # To enable interactive mode you should install ipywidgets
    # https://github.com/jupyter-widgets/ipywidgets
    from ipywidgets import interact, SelectMultiple
    INTERACTIVE = True
except ImportError:
    INTERACTIVE = False
!wget https://github.com/microsoft/LightGBM/blob/master/examples/regression/regression.train?raw=true
!wget https://github.com/microsoft/LightGBM/blob/master/examples/regression/regression.test?raw=true
df_train = pd.read_csv('regression.train?raw=true', header=None, sep='\t')
df_test = pd.read_csv('regression.test?raw=true', header=None, sep='\t')

y_train = df_train[0]
y_test = df_test[0]
X_train = df_train.drop(0, axis=1)
X_test = df_test.drop(0, axis=1)
print(len(y_train), X_train.shape)<jupyter_output><empty_output><jupyter_text>Create Dataset object for LightGBM<jupyter_code>lgb_train = lgb.Dataset(X_train, y_train)
lgb_test = lgb.Dataset(X_test, y_test, reference=lgb_train)<jupyter_output><empty_output><jupyter_text>Configuration dictionary<jupyter_code>params = {
    'num_leaves': 5,
    'metric': ['l1', 'l2'],
    'verbose': -1
}
evals_result = {}  # to record eval results for plotting
gbm = lgb.train(params,
                lgb_train,
                num_boost_round=100,
                valid_sets=[lgb_train, lgb_test],
                feature_name=['f' + str(i + 1) for i in range(X_train.shape[-1])],
                categorical_feature=[21],
                evals_result=evals_result,
                verbose_eval=10)
def render_metric(metric_name):
    ax = lgb.plot_metric(evals_result, metric=metric_name, figsize=(10, 5))
    plt.show()
if INTERACTIVE:
    # create widget to switch between metrics
    interact(render_metric, metric_name=params['metric'])
else:
    render_metric(params['metric'][0])<jupyter_output><empty_output><jupyter_text>Plot feature importances<jupyter_code>def render_plot_importance(importance_type, max_features=10,
                           ignore_zero=True, precision=4):
    ax = lgb.plot_importance(gbm, importance_type=importance_type,
                             max_num_features=max_features,
                             ignore_zero=ignore_zero, figsize=(12, 8),
                             precision=precision)
    plt.show()
if INTERACTIVE:
    # create widget for interactive feature importance plot
    interact(render_plot_importance,
             importance_type=['split', 'gain'],
             max_features=(1, X_train.shape[-1]),
             precision=(0, 10))
else:
    render_plot_importance(importance_type='split')
def render_tree(tree_index, show_info, precision=4):
    show_info = None if 'None' in show_info else show_info
    return lgb.create_tree_digraph(gbm, tree_index=tree_index,
                                   show_info=show_info, precision=precision)
#!conda install graphviz pydot -y
if INTERACTIVE:
    # create widget to switch between trees and control info in nodes
    interact(render_tree,
             tree_index=(0, gbm.num_trees() - 1),
             show_info=SelectMultiple(  # allow multiple values to be selected
                 options=['None',
                          'split_gain',
                          'internal_value',
                          'internal_count',
                          'leaf_count'],
                 value=['None']),
             precision=(0, 10))
    tree = None
else:
    tree = render_tree(84, ['None'])
tree<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11806508923524187, 'fraction_numerical': 0.012355648873455544, 'mean_word_length': 4.733333333333333, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '536175', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4218, 'n_tokens_neox': 4006, 'n_words': 1013, 'path': 'Lectures/3_SupervisedLearning/02_Classification_II.ipynb', 'repo_name': 'julianfbaquero/LML', 'size': '12387'}