<jupyter_start><jupyter_text>Recognizing hand-written digitsAn example showing how the scikit-learn can be used to recognize images of hand-written digits.This example is commented in the [tutorial section of the user manual](http://scikit-learn.org/stable/tutorial/basic/tutorial.htmlintroduction).<jupyter_code>from sklearn import datasets, metrics
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.externals import joblib

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>The digits datasetThe data that we are interested in is a set of 1797 images of digits, each one made of $8\times8$ pixels.<jupyter_code>digits = datasets.load_digits()<jupyter_output><empty_output><jupyter_text>Let's have a look at the first 4 images, stored in the `images` attribute of thedataset. Note that each image must have the same size. For theseimages, we know which digit they represent: it is given in the `target` ofthe dataset.<jupyter_code>images_and_labels = list(zip(digits.images, digits.target))
for index, (image, label) in enumerate(images_and_labels[:8]):
    plt.subplot(2, 4, index + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Training: %i' % label)<jupyter_output><empty_output><jupyter_text>To apply a classifier on this data, we need to flatten the image, toturn the data in a (samples, feature) matrix:<jupyter_code>n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))<jupyter_output><empty_output><jupyter_text>For example, the data corresponding to the image of the "0" above is:<jupyter_code>data[0]<jupyter_output><empty_output><jupyter_text>Each pixel is a 4-bit value, from 0 (white) to 15 (black).  Cross-validation: evaluating network performanceLearning the parameters of the MLP and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data. This situation is called [**overfitting**](https://en.wikipedia.org/wiki/Overfitting). To avoid it, it is common practice  to hold out part of the available data as a test set `[X_test, y_test]`. In scikit-learn a random split into training and test sets can be quickly computed with the [`train_test_split`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.htmlsklearn.model_selection.train_test_split) helper function.<jupyter_code>X, y = data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4)<jupyter_output><empty_output><jupyter_text>Data scalingStandardization of datasets is a common requirement for neural networks; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.In scikit-learn, the preprocessing module provides a utility class [StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.htmlsklearn.preprocessing.StandardScaler) that computes the mean and standard deviation on a training set so as to be able to later reapply the same transformation on the testing set.<jupyter_code># Don't cheat - fit only on training data
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
# apply same transformation to test data
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Build the modelCreate a [MLP object](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html) with the following arguments:* [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) solver (standard technique in [backpropagation](https://en.wikipedia.org/wiki/Backpropagation))* one hidden layer with 5 neurons* 4000 iterations maximumThe rest of the arguments are set to their default values (see documentation).<jupyter_code>net = MLPClassifier(solver='sgd',\
                    hidden_layer_sizes=(5, ),\
                    max_iter=4000)<jupyter_output><empty_output><jupyter_text>Train the networkThe `fit` function automatically iterates until convergence or the maximum number of iterations is reached, so you only need to execute the following cell once.<jupyter_code>net.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Analysis of the networkIn scikit-learn there are some functions for measuring the accuracy of the classification. Classification reportThe [classification_report](http://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-report) function builds a text report showing the main classification metrics:* [precision](https://en.wikipedia.org/wiki/Precision_and_recall): ratio $\frac{tp}{tp + fp}$ where $tp$ is the number of true positives and $fp$ the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.* [recall](https://en.wikipedia.org/wiki/Precision_and_recall): ratio $\frac{tp}{tp + fn}$ where $tp$ is the number of true positives and $fn$ the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.* [f1-score](https://en.wikipedia.org/wiki/F1_score): a weighted average of the precision and recall, where an $F_1$ score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the $F_1$ score are equal. The formula for the $F_1$ score is: $F_1 = 2\cdot\frac{precision \cdot recall}{precision + recall}$* support: the support is the number of occurrences of each class<jupyter_code>expected = y_test
predicted = net.predict(X_test)
print(metrics.classification_report(expected, predicted))<jupyter_output><empty_output><jupyter_text>Confusion matrixThe [`confusion_matrix`](http://scikit-learn.org/stable/modules/model_evaluation.htmlconfusion-matrix) function evaluates classification accuracy by computing the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix).By definition, a confusion matrix $C$ is such that $C_{i, j}$ is equal to the number of observations known to be in group $i$ but predicted to be in group $j$. Thus an optimal classification result would produce a diagonal matrix.<jupyter_code>print(metrics.confusion_matrix(expected, predicted))<jupyter_output><empty_output><jupyter_text>Training curveNumber of iterations during training:<jupyter_code>net.n_iter_<jupyter_output><empty_output><jupyter_text>Loss curve: (currently, [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.htmlsklearn.neural_network.MLPClassifier) supports only the [Cross-Entropy loss function](https://en.wikipedia.org/wiki/Cross_entropyCross-entropy_error_function_and_logistic_regression))<jupyter_code>plt.plot(net.loss_curve_);
plt.xlabel('Iterations');
plt.ylabel('Loss');<jupyter_output><empty_output><jupyter_text>Samples of predictions<jupyter_code>images_and_predictions = list(zip(X_test, predicted))
for index, (image, prediction) in enumerate(images_and_predictions[:8]):
    plt.subplot(2, 4, index+1)
    plt.axis('off')
    plt.imshow(image.reshape((8,8)), cmap=plt.cm.gray_r, interpolation='nearest')
    plt.title('Target: %i\nPrediction: %i' % (y_test[index],prediction))<jupyter_output><empty_output><jupyter_text>Save the dataset, scaler and the network into files.<jupyter_code>joblib.dump(scaler, 'digits_scaler.pkl') 
joblib.dump(net, 'digits_net.pkl')<jupyter_output><empty_output><jupyter_text>You can check the files of your scaler and network with [this notebook](mlp_digits_check.ipynb).  Can the network do better?Re-train the network several times with the same dataset, and check what is the highest value for the percentage of correct classification. You may increase the number of hidden neurons, or even use two hidden layers.  Shuffling the datasetYou may have noticed in the previous example, that splitting the dataset differently may lead to slightly different classification results.In order to better measure such results, you are going to test your network with 100 different test sets extracted from the data set. For each iteration, you will compute the score of the **same** network with a **different** test set, and store the values in a list. Finally, you will compute the mean and standard deviation from the values in the list. Create a new notebook with the following statements:* Load your scaler and network files* Load the digits dataset* Iterate 100 times:  * Generate a new test set  * Compute the net score* Compute the mean and standard deviation of the values* Print the result<jupyter_code>import numpy as np
score = []
for _ in range(100):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4)
    X_test = scaler.transform(X_test)
    score.append(net.score(X_test, y_test)*100)
print('Score: mean = %.2f, std = %.2f' % (np.mean(score), np.std(score)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0889493739793141, 'fraction_numerical': 0.006423516603157322, 'mean_word_length': 6.216025137470542, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '127666', 'license': 'MIT', 'n_tokens_mistral': 2744, 'n_tokens_neox': 2621, 'n_words': 915, 'path': 'MLP/mlp_digits.ipynb', 'repo_name': 'ecervera/nn-nb', 'size': '9185'}