<jupyter_start><jupyter_text>MNIST handwritten digits classification with an ensemble of classifiers In this notebook, we'll use a [classifier emsemble](https://scikit-learn.org/stable/modules/ensemble.htmlvoting-classifier) to classify MNIST digits using scikit-learn (version 0.20 or later required).First, the needed imports.<jupyter_code>%matplotlib inline

from pml_utils import get_mnist, show_failures

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn import __version__
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

from distutils.version import LooseVersion as LV
assert(LV(__version__) >= LV("0.20")), "Version >= 0.20 of sklearn is required."<jupyter_output><empty_output><jupyter_text>Then we load the MNIST data. First time we need to download the data, which can take a while.<jupyter_code>X_train, y_train, X_test, y_test = get_mnist('MNIST')

print('MNIST data loaded: train:',len(X_train),'test:',len(X_test))
print('X_train:', X_train.shape)
print('y_train:', y_train.shape)
print('X_test', X_test.shape)
print('y_test', y_test.shape)<jupyter_output><empty_output><jupyter_text>The training data (`X_train`) is a matrix of size (60000, 784), i.e. it consists of 60000 digits expressed as 784 sized vectors (28x28 images flattened to 1D). `y_train` is a 60000-dimensional vector containing the correct classes ("0", "1", ..., "9") for each training digit.  Individual classifiersLet's first define and train a set of different classifiers.  SGDClassifier<jupyter_code>%%time

clf_sgd = SGDClassifier()
print(clf_sgd.fit(X_train, y_train))
pred_sgd = clf_sgd.predict(X_test)
print('Predicted', len(pred_sgd), 'digits with accuracy:', accuracy_score(y_test, pred_sgd))<jupyter_output><empty_output><jupyter_text>Decision tree<jupyter_code>%%time

clf_dt = DecisionTreeClassifier()
print(clf_dt.fit(X_train, y_train))
pred_dt = clf_dt.predict(X_test)
print('Predicted', len(pred_dt), 'digits with accuracy:', accuracy_score(y_test, pred_dt))<jupyter_output><empty_output><jupyter_text>Bernoulli naive Bayes<jupyter_code>%%time

clf_bnb = BernoulliNB(binarize=128.)
print(clf_bnb.fit(X_train, y_train))
pred_bnb = clf_bnb.predict(X_test)
print('Predicted', len(pred_bnb), 'digits with accuracy:', accuracy_score(y_test, pred_bnb))<jupyter_output><empty_output><jupyter_text>Ensemble classifierThe goal of ensemble methods is to combine the predictions of several base classifiers to improve generalizability and robustness. LearningWe use [`VotingClassifier`](https://scikit-learn.org/stable/modules/ensemble.htmlvoting-classifier) to combine the results of the individual classifiers.The default mode is to use majority (`"hard"`) voting, where each classifier gets a vote and the final prediction is the class that gets the majority of the votes.Another option is to use the average of the predicted probabilities (`"soft"` voting), which however requires that all used individual classifiers are able to predict class probabilities.<jupyter_code>%%time

clf_vote = VotingClassifier(estimators=[('sgd', clf_sgd),
                                        ('dt', clf_dt),
                                        ('bnb', clf_bnb)],
                            voting='hard')
clf_vote.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>InferenceThe classification accuracy of the ensemble classifier:<jupyter_code>pred_vote = clf_vote.predict(X_test)
print('Predicted', len(pred_vote), 'digits with accuracy:', accuracy_score(y_test, pred_vote))<jupyter_output><empty_output><jupyter_text>Confusion matrixWe can compute the confusion matrix to see which digits get mixed the most:<jupyter_code>labels=[str(i) for i in range(10)]
print('Confusion matrix (rows: true classes; columns: predicted classes):'); print()
cm=confusion_matrix(y_test, pred_vote, labels=labels)
print(cm); print()<jupyter_output><empty_output><jupyter_text>Accuracy, precision and recallClassification accuracy for each class:<jupyter_code>for i,j in enumerate(cm.diagonal()/cm.sum(axis=1)): print("%d: %.4f" % (i,j))<jupyter_output><empty_output><jupyter_text>Precision and recall for each class:<jupyter_code>print(classification_report(y_test, pred_vote, labels=labels))<jupyter_output><empty_output><jupyter_text>Failure analysisWe can also do some failure analysis.  Let's check the 10 first wrongly predicted digits.<jupyter_code>show_failures(pred_vote, y_test, X_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10236886632825719, 'fraction_numerical': 0.00994077834179357, 'mean_word_length': 6.855481727574751, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '376115', 'license': 'MIT', 'n_tokens_mistral': 1583, 'n_tokens_neox': 1502, 'n_words': 401, 'path': 'notebooks/sklearn-mnist-ens.ipynb', 'repo_name': 'CSCfi/machine-learning-scripts', 'size': '4728'}