<jupyter_start><jupyter_text>MNIST handwritten digits classification with MLPsIn this notebook, we'll train a multi-layer perceptron model to classify MNIST digits using **Tensorflow** (version $\ge$ 2.0 required) with the **Keras API**.First, the needed imports.<jupyter_code>%matplotlib inline

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout, Flatten
import tensorflow.keras.utils as utils
import tensorflow_datasets as tfds
tfds.disable_progress_bar()

from distutils.version import LooseVersion as LV

from IPython.display import SVG, display

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

import tensorflow as tf
print('Using Tensorflow version: {}, and Keras version: {}.'.format(tf.__version__, tf.keras.__version__))
assert(LV(tf.__version__) >= LV("2.0.0"))<jupyter_output><empty_output><jupyter_text>Let's check if we have GPU available.<jupyter_code>if tf.test.is_gpu_available():
    from tensorflow.python.client import device_lib
    for d in device_lib.list_local_devices():
        if d.device_type == 'GPU':
            print('GPU', d.physical_device_desc)
else:
    print('No GPU, using CPU instead.')<jupyter_output><empty_output><jupyter_text>MNIST data setNext we'll load the MNIST handwritten digits data set.  First time we may have to download the data, which can take a while. Altenative: Fashion-MNISTAlternatively, MNIST can be replaced with Fashion-MNIST, which can be used as drop-in replacement for MNIST.   Fashion-MNIST contains images of 10 fashion categories:Label|Description|Label|Description--- | --- |--- | ---0|T-shirt/top|5|Sandal1|Trouser|6|Shirt2|Pullover|7|Sneaker3|Dress|8|Bag4|Coat|9|Ankle boot<jupyter_code>nb_classes = 10
batch_size = 32

datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)
mnist_train, mnist_test = datasets['train'], datasets['test']

print(info)

def scale(image, label):
  image = tf.cast(image, tf.float32)
  image /= 255.0

  return image, label #tf.one_hot(label, nb_classes)

train_dataset = mnist_train.map(scale).shuffle(info.splits['train'].num_examples).batch(batch_size)
test_dataset = mnist_test.map(scale).batch(batch_size)<jupyter_output><empty_output><jupyter_text>The training data (`X_train`) is a 3rd-order tensor of size (60000, 28, 28), i.e. it consists of 60000 images of size 28x28 pixels. `y_train` is a 60000-dimensional vector containing the correct classes ("0", "1", ..., "9") for each training sample, and `Y_train` is a [one-hot](https://en.wikipedia.org/wiki/One-hot) encoding of `y_train`.Let's take a closer look. Here are the first 10 training digits (or fashion items for Fashion-MNIST):<jupyter_code>pltsize=1
plt.figure(figsize=(10*pltsize, pltsize))

for i, data in enumerate(mnist_train.take(10)):
    image, label = data
    label = label.numpy()
    plt.subplot(1,10,i+1)
    plt.axis('off')
    plt.imshow(tf.reshape(image, (28, 28)), cmap="gray_r")
    plt.title('Class: '+str(label))<jupyter_output><empty_output><jupyter_text>Linear model InitializationLet's begin with a simple linear model.  We first initialize the model with `Sequential()`.  Then we add a `Dense` layer that has 28*28=784 input nodes (one for each pixel in the input image) and 10 output nodes. The `Dense` layer connects each input to each output with some weight parameter. Finally, we select *categorical crossentropy* as the loss function, select [*stochastic gradient descent*](https://keras.io/optimizers/sgd) as the optimizer, add *accuracy* to the list of metrics to be evaluated, and `compile()` the model. Note there are [several different options](https://keras.io/optimizers/) for the optimizer in Keras that we could use instead of *sgd*.<jupyter_code>linmodel = Sequential()
linmodel.add(Flatten(input_shape=(28, 28, 1)))
linmodel.add(Dense(units=10, activation='softmax'))

linmodel.compile(loss='sparse_categorical_crossentropy', 
                 optimizer='sgd', 
                 metrics=['accuracy'])
print(linmodel.summary())<jupyter_output><empty_output><jupyter_text>The summary shows that there are 7850 parameters in our model, as the weight matrix is of size 785x10 (not 784, as there's an additional bias term).We can also draw a fancier graph of our model.<jupyter_code>utils.plot_model(linmodel, show_shapes=True)<jupyter_output><empty_output><jupyter_text>LearningNow we are ready to train our first model.  An *epoch* means one pass through the whole training data. The `reshape()` function flattens our 28x28 images into vectors of length 784.  (This means we are not using any information about the spatial neighborhood relations of pixels.  This setup is known as the *permutation invariant MNIST*.)  You can run code below multiple times and it will continue the training process from where it left off.  If you want to start from scratch, re-initialize the model using the code a few cells ago.<jupyter_code>%%time
epochs = 10 # one epoch takes about 6 seconds

linhistory = linmodel.fit(train_dataset, epochs=epochs, verbose=2)<jupyter_output><empty_output><jupyter_text>Let's now see how the training progressed. * *Loss* is a function of the difference of the network output and the target values.  We are minimizing the loss function during training so it should decrease over time.* *Accuracy* is the classification accuracy for the training data.  It gives some indication of the real accuracy of the model but cannot be fully trusted, as it may have overfitted and just memorizes the training data.<jupyter_code>plt.figure(figsize=(5,3))
plt.plot(linhistory.epoch,linhistory.history['loss'])
plt.title('loss')

plt.figure(figsize=(5,3))
plt.plot(linhistory.epoch,linhistory.history['accuracy'])
plt.title('accuracy');<jupyter_output><empty_output><jupyter_text>InferenceFor a better measure of the quality of the model, let's see the model accuracy for the test data.<jupyter_code>linscores = linmodel.evaluate(test_dataset, verbose=2)
print("%s: %.2f%%" % (linmodel.metrics_names[1], linscores[1]*100))<jupyter_output><empty_output><jupyter_text>We can now take a closer look on the results.Let's define a helper function to show the failure cases of our classifier.<jupyter_code>def show_failures(predictions, trueclass=None, predictedclass=None, maxtoshow=10):
    rounded = np.argmax(predictions, axis=1)
    errors = rounded!=y_test
    print('Showing max', maxtoshow, 'first failures. '
          'The predicted class is shown first and the correct class in parenthesis.')
    ii = 0
    plt.figure(figsize=(maxtoshow, 1))
    for i in range(X_test.shape[0]):
        if ii>=maxtoshow:
            break
        if errors[i]:
            if trueclass is not None and y_test[i] != trueclass:
                continue
            if predictedclass is not None and rounded[i] != predictedclass:
                continue
            plt.subplot(1, maxtoshow, ii+1)
            plt.axis('off')
            plt.imshow(X_test[i,:,:], cmap="gray_r")
            plt.title("%d (%d)" % (rounded[i], y_test[i]))
            ii = ii + 1<jupyter_output><empty_output><jupyter_text>Here are the first 10 test digits the linear model classified to a wrong class:<jupyter_code>linpredictions = linmodel.predict(X_test.reshape((-1,28*28)))

show_failures(linpredictions)<jupyter_output><empty_output><jupyter_text>Multi-layer perceptron (MLP) network Activation functionsLet's start by plotting some common activation functions for neural networks. `'relu'` stands for rectified linear unit, $y=\max(0,x)$, a very simple non-linearity we will be using in our MLP network below.<jupyter_code>x = np.arange(-4,4,.01)
plt.figure()
plt.plot(x, np.maximum(x,0), label='relu')
plt.plot(x, 1/(1+np.exp(-x)), label='sigmoid')
plt.plot(x, np.tanh(x), label='tanh')
plt.axis([-4, 4, -1.1, 1.5])
plt.title('Activation functions')
plt.legend(loc='best');<jupyter_output><empty_output><jupyter_text>InitializationLet's now create a more complex MLP model that has multiple layers, non-linear activation functions, and dropout layers.  `Dropout()` randomly sets a fraction of inputs to zero during training, which is one approach to regularization and can sometimes help to prevent overfitting.There are two options below, a simple and a bit more complex model.  Select either one.The output of the last layer needs to be a softmaxed 10-dimensional vector to match the groundtruth (`Y_train`). Finally, we again `compile()` the model, this time using [*Adam*](https://keras.io/optimizers/adam) as the optimizer.<jupyter_code># Model initialization:
model = Sequential()

# A simple model:
model.add(Dense(units=20, input_dim=28*28))
model.add(Activation('relu'))

# A bit more complex model:
#model.add(Dense(units=50, input_dim=28*28))
#model.add(Activation('relu'))
#model.add(Dropout(0.2))

#model.add(Dense(units=50))
#model.add(Activation('relu'))
#model.add(Dropout(0.2))

# The last layer needs to be like this:
model.add(Dense(units=10, activation='softmax'))

model.compile(loss='categorical_crossentropy', 
              optimizer='adam', 
              metrics=['accuracy'])
print(model.summary())
utils.plot_model(model, show_shapes=True)<jupyter_output><empty_output><jupyter_text>Learning<jupyter_code>%%time
epochs = 10 # one epoch with simple model takes about 4 seconds

history = model.fit(X_train.reshape((-1,28*28)), 
                    Y_train, 
                    epochs=epochs, 
                    batch_size=32,
                    verbose=2)
plt.figure(figsize=(5,3))
plt.plot(history.epoch,history.history['loss'])
plt.title('loss')

plt.figure(figsize=(5,3))
plt.plot(history.epoch,history.history['accuracy'])
plt.title('accuracy');<jupyter_output><empty_output><jupyter_text>InferenceAccuracy for test data.  The model should be somewhat better than the linear model.<jupyter_code>%%time
scores = model.evaluate(X_test.reshape((-1,28*28)), Y_test, verbose=2)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))<jupyter_output><empty_output><jupyter_text>We can again take a closer look on the results, using the `show_failures()` function defined earlier.Here are the first 10 test digits the MLP classified to a wrong class:<jupyter_code>predictions = model.predict(X_test.reshape((-1,28*28)))

show_failures(predictions)<jupyter_output><empty_output><jupyter_text>We can use `show_failures()` to inspect failures in more detail. For example, here are failures in which the true class was "6":<jupyter_code>show_failures(predictions, trueclass=6)<jupyter_output><empty_output><jupyter_text>We can also compute the confusion matrix to see which digits get mixed the most, and look at classification accuracies separately for each class:<jupyter_code>from sklearn.metrics import confusion_matrix

print('Confusion matrix (rows: true classes; columns: predicted classes):'); print()
cm=confusion_matrix(y_test, np.argmax(predictions, axis=1), labels=list(range(10)))
print(cm); print()

print('Classification accuracy for each class:'); print()
for i,j in enumerate(cm.diagonal()/cm.sum(axis=1)): print("%d: %.4f" % (i,j))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11513897634253846, 'fraction_numerical': 0.01906989295673293, 'mean_word_length': 5.260135135135135, 'pattern_counts': {'":': 1, '<': 73, '<?xml version=': 0, '>': 75, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '586236', 'license': 'MIT', 'n_tokens_mistral': 3701, 'n_tokens_neox': 3471, 'n_words': 1060, 'path': 'notebooks/tf2-mnist-mlp-datasets.ipynb', 'repo_name': 'CSCfi/machine-learning-scripts', 'size': '11117'}