<jupyter_start><jupyter_text>Chapter 3 -  Classification Multiclass Classification<jupyter_code>import pickle

import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

from sklearn.linear_model import SGDClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score, cross_val_predict
from sklearn.multiclass import OneVsOneClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (precision_score, 
                             recall_score, 
                             classification_report, 
                             confusion_matrix, f1_score, 
                             precision_recall_curve, roc_curve, roc_auc_score)

def load(fname):
    mnist = None
    try:
        with open(fname, 'rb') as f:
            mnist = pickle.load(f)
            return mnist
    except FileNotFoundError:
        from sklearn.datasets import fetch_openml
        mnist = fetch_openml('mnist_784', version=1, cache=True)
        with open(fname, 'wb') as f:
            mnist = pickle.dump(mnist, f)
        return mnist<jupyter_output><empty_output><jupyter_text>In the multiclass classifier problem, the target number of classes extends beyond just 2. There are some classifiers that can handle more than 2 classes while some are strictly binary classifiers. However, we can still use binary classifiers for multiclass problems.In the MNIST problem where there are 10 classes, we can create a system of multiple binary classifiers (e.g. a 0-detector, a 1-detector etc.). To classify a new image, run through every binary classifier and select the best score. This is known as the one-versus-all (OVA) strategy. The number of classifiers is the same as the number of classes. With 10 classes, we will have 10 classifiers.Another way is to train a 0v1, 0v2, 1v2 etc. classifier and select the class that has won the most duels / the class with the most number of assignments for each of these classifiers. This is called the one-versus-one (OVO) strategy. The number of classifiers to train here is $\frac{N(N-1)}{2}$ classifiers.<jupyter_code># Ingest
mnist_data = load('mnist.data.pkl')
X, y = mnist_data['data'], mnist_data['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=0)<jupyter_output><empty_output><jupyter_text>In `sklearn`, in a case where $k>2$, the OVA strategy is used by default.<jupyter_code># Train a SGD classifier with 10 classes
clf = SGDClassifier(random_state=0)
clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Notes: Training took about 2 minutes. We can see the results for each of the classifiers by using `decision_function`.<jupyter_code># For evaluation
i = 5
Xti = [X_train[i]]
# Get the prediction result
print(clf.predict(Xti))
print()
# Get the decision scores for each class
print(list([m.round(2) for m in clf.decision_function(Xti)[0]]))
print()
for kpred1, kpred2 in list(zip(range(10), clf.decision_function(Xti)[0])):
    print('{} : {:.2f}'.format(str(kpred1), kpred2))
print()
print(y_test[i])<jupyter_output>['9']

[-9791.77, -79739.28, -10462.73, -20837.1, -7980.04, -10559.4, -21397.66, -5991.81, -6365.83, -2799.92]

0 : -9791.77
1 : -79739.28
2 : -10462.73
3 : -20837.10
4 : -7980.04
5 : -10559.40
6 : -21397.66
7 : -5991.81
8 : -6365.83
9 : -2799.92

9<jupyter_text>`sklearn` has a tool for OVO classification.<jupyter_code>ovo_clf = OneVsOneClassifier(SGDClassifier(random_state=0))
ovo_clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Note: Training took about 30 seconds<jupyter_code># Find the number of classifiers / duels
print(len(ovo_clf.estimators_))
# For evaluation
i = 5
Xti = [X_train[i]]
# Get the prediction result
print(ovo_clf.predict(Xti))
print(ovo_clf.decision_function(Xti)[0])
# Get the decision scores for each class
print(y_test[i])<jupyter_output>['9']
[ 6.33333312 -0.33333325  6.33333302  1.66666704  7.33333288  1.66666786
  3.66666757  6.33333286  3.33333191  9.33333302]
9<jupyter_text>`RandomForestClassifier` can simply perform the classification to multiple classes without using the OVO or OVA strategy. For a new sample, we can find the probability that an observation belongs in each class. The class with the highest probability is the predicted class.<jupyter_code>forest_clf = RandomForestClassifier(random_state=0)
forest_clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Note: Training took about 1 minute<jupyter_code># For evaluation
i = 5
Xti = [X_train[i]]
# Get the prediction result
print(forest_clf.predict(Xti))
# Get the probabilities for the observation existing in each class
for kpred1, kpred2 in list(zip(range(10), forest_clf.predict_proba(Xti)[0])):
    print('{} : {:.2f}'.format(str(kpred1), kpred2))
print(y_test[i])<jupyter_output>['9']
0 : 0.01
1 : 0.00
2 : 0.04
3 : 0.01
4 : 0.03
5 : 0.02
6 : 0.00
7 : 0.03
8 : 0.02
9 : 0.84
9<jupyter_text>Performance evaluation for each of the models:<jupyter_code># Evaluation: SGD
print(cross_val_score(clf, X_train[:10000], y_train[:10000], cv=3, scoring='accuracy'))
# Note: Took about 3 minutes for all observations

# Evaluation: OVO SGD
print(cross_val_score(ovo_clf, X_train[:10000], y_train[:10000], cv=3, scoring='accuracy'))
# Note: Took about 1 minute for all observations

# Evaluation: Random Forest
print(cross_val_score(forest_clf, X_train[:10000], y_train[:10000], cv=3, scoring='accuracy'))
# Note: Took about 2 minutes for all observations<jupyter_output>[0.85632873 0.86558656 0.85808581]
[0.89592082 0.89978998 0.90639064]
[0.9355129  0.94389439 0.94449445]<jupyter_text>Conclusion: Random Forest has a lot more success than other algorithms.Other ways we can improve the algorithm is to perform scaling before doing model fitting. This does improve the SGD classifier score<jupyter_code># Scale before testing
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))

# Evaluation: SGD
print(cross_val_score(clf, X_train_scaled[:10000], y_train[:10000], cv=3, scoring='accuracy'))
# Note: Cross Validation took 13 minutes<jupyter_output>/Users/bryanlim/.pyenv/versions/3.7.2/envs/botanic/lib/python3.7/site-packages/sklearn/linear_model/_stochastic_gradient.py:557: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
  ConvergenceWarning)<jupyter_text>Error AnalysisOne way to evalute the model is to see the errors that it is making. In the 10 digit classification problem, we can observe the confusion matrix.<jupyter_code>y_train_predict = cross_val_predict(clf, X_train_scaled, y_train, cv=3)
# Note: Training & prediction took 10 minutes
# Compute the confusion matrix
conf_mx = confusion_matrix(y_train, y_train_predict)
print(conf_mx)
# A way to visualise the confusion matrix is to use matshow.
plt.matshow(conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>Notice that the squares that are not on the main diagnonal are dark. To focus on them, divide each value in the confusion matrix by the number of images in the corresponding class.<jupyter_code>row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx/row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09860867215993516, 'fraction_numerical': 0.06483857895447792, 'mean_word_length': 5.039151712887439, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '184874', 'license': 'MIT', 'n_tokens_mistral': 2821, 'n_tokens_neox': 2485, 'n_words': 748, 'path': 'chap03/03-textbook-classification-03.ipynb', 'repo_name': 'bryanblackbee/topic__hands-on-machine-learning', 'size': '7403'}