<jupyter_start><jupyter_text>MNIST handwritten digits classification with MLPsIn this notebook, we'll train a multi-layer perceptron model to classify MNIST digits using **Keras** (version $\ge$ 2 required). First, the needed imports. Keras tells us which backend (Theano, Tensorflow, CNTK) it will be using.<jupyter_code>%matplotlib inline

import sys
sys.path.append('../')
from pml_utils import get_mnist

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.utils import np_utils
from keras import backend as K

from distutils.version import LooseVersion as LV
from keras import __version__

from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

print('Using Keras version:', __version__, 'backend:', K.backend())
assert(LV(__version__) >= LV("2.0.0"))<jupyter_output><empty_output><jupyter_text>If we are using TensorFlow as the backend, we can check whether we have GPUs available:<jupyter_code>if K.backend() == "tensorflow":
    from tensorflow.python.client import device_lib
    print(device_lib.list_local_devices())<jupyter_output><empty_output><jupyter_text>MNIST data setNext we'll load the MNIST handwritten digits data set.  First time we may have to download the data, which can take a while.<jupyter_code>X_train, y_train, X_test, y_test = get_mnist('../MNIST', False, False)

nb_classes = 10

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

# one-hot encoding:
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

print()
print('MNIST data loaded: train:',len(X_train),'test:',len(X_test))
print('X_train:', X_train.shape, X_train.dtype)
print('y_train:', y_train.shape, y_train.dtype)
print('Y_train:', Y_train.shape, Y_train.dtype)<jupyter_output><empty_output><jupyter_text>The training data (`X_train`) is a 3rd-order tensor of size (60000, 28, 28), i.e. it consists of 60000 images of size 28x28 pixels. `y_train` is a 60000-dimensional vector containing the correct classes ("0", "1", ..., "9") for each training sample, and `Y_train` is a [one-hot](https://en.wikipedia.org/wiki/One-hot) encoding of `y_train`.Let's take a closer look. Here are the first 10 training digits:<jupyter_code>pltsize=1
plt.figure(figsize=(10*pltsize, pltsize))

for i in range(10):
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(X_train[i,:,:], cmap="gray")
    plt.title('Class: '+str(y_train[i]))
    print('Training sample',i,': class:',y_train[i], ', one-hot encoded:', Y_train[i])<jupyter_output><empty_output><jupyter_text>Multi-layer perceptron (MLP) network  InitializationLet's now create an MLP model that has multiple layers, non-linear activation functions, and dropout layers.  `Dropout()` randomly sets a fraction of inputs to zero during training, which is one approach to regularization and can sometimes help to prevent overfitting.There are two options below, a simple and a bit more complex model.  Select either one.The output of the last layer needs to be a softmaxed 10-dimensional vector to match the groundtruth (`Y_train`). Finally, we again `compile()` the model, this time using [*Adam*](https://keras.io/optimizers/adam) as the optimizer.<jupyter_code># Model initialization:
model = Sequential()

# A simple model:
model.add(Dense(units=20, input_dim=28*28))
model.add(Activation('relu'))

# A bit more complex model:
#model.add(Dense(units=50, input_dim=28*28))
#model.add(Activation('relu'))
#model.add(Dropout(0.2))

#model.add(Dense(units=50))
#model.add(Activation('relu'))
#model.add(Dropout(0.2))

# The last layer needs to be like this:
model.add(Dense(units=10, activation='softmax'))

model.compile(loss='categorical_crossentropy', 
              optimizer='adam', 
              metrics=['accuracy'])
print(model.summary())
SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>Learning<jupyter_code>%%time
epochs = 10 # one epoch with simple model takes about 4 seconds

history = model.fit(X_train.reshape((-1,28*28)), 
                    Y_train, 
                    epochs=epochs, 
                    batch_size=32,
                    verbose=2)
plt.figure(figsize=(5,3))
plt.plot(history.epoch,history.history['loss'])
plt.title('loss')

plt.figure(figsize=(5,3))
plt.plot(history.epoch,history.history['acc'])
plt.title('accuracy');<jupyter_output><empty_output><jupyter_text>InferenceAccuracy for test data.<jupyter_code>%%time
scores = model.evaluate(X_test.reshape((-1,28*28)), Y_test, verbose=2)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))<jupyter_output><empty_output><jupyter_text>We can take a closer look at the results, using the `show_failures()` function.<jupyter_code>def show_failures(predictions, trueclass=None, predictedclass=None, maxtoshow=10):
    rounded = np.argmax(predictions, axis=1)
    errors = rounded!=y_test
    print('Showing max', maxtoshow, 'first failures. '
          'The predicted class is shown first and the correct class in parenthesis.')
    ii = 0
    plt.figure(figsize=(maxtoshow, 1))
    for i in range(X_test.shape[0]):
        if ii>=maxtoshow:
            break
        if errors[i]:
            if trueclass is not None and y_test[i] != trueclass:
                continue
            if predictedclass is not None and rounded[i] != predictedclass:
                continue
            plt.subplot(1, maxtoshow, ii+1)
            plt.axis('off')
            plt.imshow(X_test[i,:,:], cmap="gray")
            plt.title("%d (%d)" % (rounded[i], y_test[i]))
            ii = ii + 1<jupyter_output><empty_output><jupyter_text>Here are the first 10 test digits the MLP classified to a wrong class:<jupyter_code>predictions = model.predict(X_test.reshape((-1,28*28)))

show_failures(predictions)<jupyter_output><empty_output><jupyter_text>We can use `show_failures()` to inspect failures in more detail. For example, here are failures in which the true class was "6":<jupyter_code>show_failures(predictions, trueclass=6)<jupyter_output><empty_output><jupyter_text>We can also compute the confusion matrix to see which digits get mixed the most, and look at classification accuracies separately for each class:<jupyter_code>from sklearn.metrics import confusion_matrix

print('Confusion matrix (rows: true classes; columns: predicted classes):'); print()
cm=confusion_matrix(y_test, np.argmax(predictions, axis=1), labels=list(range(10)))
print(cm); print()

print('Classification accuracy for each class:'); print()
for i,j in enumerate(cm.diagonal()/cm.sum(axis=1)): print("%d: %.4f" % (i,j))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12724859923326454, 'fraction_numerical': 0.01857859038631672, 'mean_word_length': 5.099820143884892, 'pattern_counts': {'":': 2, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '604718', 'license': 'MIT', 'n_tokens_mistral': 2367, 'n_tokens_neox': 2238, 'n_words': 600, 'path': 'notebooks/keras-mnist-mlp.ipynb', 'repo_name': 'CSCfi/machine-learning-scripts', 'size': '6782'}