<jupyter_start><jupyter_text>Machine Learning Lab \7: Handwritten Digit Recognition Textbook is available @ [https://www.github.com/a-mhamdi/isetbz](https://www.github.com/a-mhamdi/isetbz)--- Let's begin with importing all the required librairies.<jupyter_code>import os

import numpy as np
from matplotlib import pyplot as plt

plt.rcParams['figure.dpi'] = 300

import tensorflow as tf

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers.convolutional import Conv2D, MaxPooling2D

from keras.utils import np_utils
from keras.models import model_from_json

from PIL import Image
import keras.backend.tensorflow_backend as tfback
from keras import backend as K<jupyter_output><empty_output><jupyter_text>Check the current installed version of `tensorflow` and `keras`.<jupyter_code>print("tf.__version__ is", tf.__version__)
print("tf.keras.__version__ is:", tf.keras.__version__)<jupyter_output>tf.__version__ is 2.2.0
tf.keras.__version__ is: 2.3.0-tf<jupyter_text>The following method allows to get a list of available **GPU** devices, formatted as strings.<jupyter_code>def _get_available_gpus():
    #global _LOCAL_DEVICES
    if tfback._LOCAL_DEVICES is None:
        devices = tf.config.list_logical_devices()
        tfback._LOCAL_DEVICES = [x.name for x in devices]
    return [x for x in tfback._LOCAL_DEVICES if 'device:gpu' in x.lower()]
tfback._get_available_gpus = _get_available_gpus
# K.image_data_format() == 'channels_first'
# K.set_image_dim_ordering('tf')
K.set_image_data_format('channels_last') # tf: TensorFlow, th: Theano<jupyter_output><empty_output><jupyter_text>Fix random seed for reproducibility.<jupyter_code>np.random.seed(0)<jupyter_output><empty_output><jupyter_text>Load and normalize the data.<jupyter_code>(X_train, y_train), (X_test, y_test) = mnist.load_data()
'''
img_idx = np.random.randint(0, high=X_test.shape[0])
plt.imshow(X_train[img_idx, :, :], cmap=plt.cm.gray_r, interpolation="nearest")
plt.show()
print("The output is {}.".format(y_train[img_idx]))
'''
num_samples_train = np.random.randint(0, high=X_train.shape[0], size=20000)
X_train = X_train[num_samples_train, :, :]
y_train =y_train[num_samples_train]
num_samples_test = np.random.randint(0, high=X_test.shape[0], size=4000)
X_test = X_test[num_samples_test, :, :]
y_test =y_test[num_samples_test]<jupyter_output><empty_output><jupyter_text>Reshape the inputs.<jupyter_code>X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')<jupyter_output><empty_output><jupyter_text>Normalize the inputs from 0 $\to$ 255 to 0 $\to$ 1.<jupyter_code>X_train = X_train/255
X_test = X_test/255<jupyter_output><empty_output><jupyter_text>One hot encode the outputs.<jupyter_code>y_train = np_utils.to_categorical(y_train)
y_test = np_utils.to_categorical(y_test)<jupyter_output><empty_output><jupyter_text>Number of classes is 10.<jupyter_code>num_classes = y_test.shape[1]<jupyter_output><empty_output><jupyter_text>It is time now to define and build the model.<jupyter_code>my_model = Sequential()
my_model.add(Conv2D(16, (5,5), input_shape=(28,28,1), activation='relu'))
my_model.add(MaxPooling2D(pool_size=(2,2)))
my_model.add(Conv2D(32, (3,3), activation='relu'))
my_model.add(MaxPooling2D(pool_size=(2,2)))
my_model.add(Dropout(0.2))
my_model.add(Flatten())
# Fully Connected NN
my_model.add(Dense(128, activation='relu'))
my_model.add(Dense(50, activation='relu'))
my_model.add(Dense(num_classes, activation='softmax'))
my_model.summary()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 24, 24, 16)        416       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 12, 12, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 10, 10, 32)        4640      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 5, 5, 32)          0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 5, 5, 32)          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 800)               0         
______________________________________________________[...]<jupyter_text>- [List of losses](https://keras.io/losses/)- [List of optimizers](https://keras.io/optimizers/)- [List of metrics](https://keras.io/metrics/)<jupyter_code>my_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Fit the model.<jupyter_code>r = my_model.fit(x=X_train, y=y_train, validation_data=(X_test, y_test), epochs=5, batch_size=100)

print("Returned:", r)<jupyter_output>Train on 20000 samples, validate on 4000 samples
Epoch 1/5
20000/20000 [==============================] - 5s 264us/step - loss: 0.5183 - accuracy: 0.8475 - val_loss: 0.1267 - val_accuracy: 0.9680
Epoch 2/5
20000/20000 [==============================] - 4s 200us/step - loss: 0.1278 - accuracy: 0.9627 - val_loss: 0.0794 - val_accuracy: 0.9745
Epoch 3/5
20000/20000 [==============================] - 5s 232us/step - loss: 0.0858 - accuracy: 0.9746 - val_loss: 0.0746 - val_accuracy: 0.9722
Epoch 4/5
20000/20000 [==============================] - 5s 242us/step - loss: 0.0660 - accuracy: 0.9789 - val_loss: 0.0632 - val_accuracy: 0.9808
Epoch 5/5
20000/20000 [==============================] - 5s 259us/step - loss: 0.0541 - accuracy: 0.9822 - val_loss: 0.0569 - val_accuracy: 0.9810
Returned: <keras.callbacks.callbacks.History object at 0x7f539c4b1640><jupyter_text>By printing the available keys, we should see: `dict_keys(['val_loss', 'val_accuracy', 'loss', 'accuracy'])`.<jupyter_code>print(r.history.keys())
# Losses
plt.subplot(1, 2, 1)
plt.plot(r.history['loss'], label='Tr. Loss')
plt.plot(r.history['val_loss'], label='Val. Loss')
plt.grid()
plt.legend()

# Accuracies
plt.subplot(1, 2, 2)
plt.plot(r.history['accuracy'], label='Tr. Accuracy')
plt.plot(r.history['val_accuracy'], label='Val. Accuracy')
plt.grid()
plt.legend()<jupyter_output><empty_output><jupyter_text>Evaluate the model.<jupyter_code>scores = my_model.evaluate(X_test, y_test, verbose=0)
print("CNN error: {}%".format(100*(1-scores[1])))<jupyter_output>CNN error: 1.8999993801116943%<jupyter_text>Test the model.<jupyter_code>img = Image.open('to-test/7.png');
img = img.convert('L')
img = img.resize((28, 28))

array_img = (np.array(img))/255
in_data = array_img.reshape((1, 28, 28, 1)).astype('float32')

plt.imshow(array_img*255, cmap=plt.cm.gray_r, interpolation="nearest")
plt.axis('Off')
y_pred = my_model.predict(in_data)
_, idx = np.where(y_pred == np.max(y_pred))
print("Result is {}. Probability is {}%.".format(int(idx), 100*y_pred[0, int(idx)]))<jupyter_output>Result is 8. Probability is 55.90149760246277%.
{'fraction_non_alphanumeric': 0.16300722623679822, 'fraction_numerical': 0.0575319622012229, 'mean_word_length': 6.09064039408867, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '236005', 'license': 'MIT', 'n_tokens_mistral': 2849, 'n_tokens_neox': 2530, 'n_words': 476, 'path': 'Python/ml/mnist-cnn.ipynb', 'repo_name': 'a-mhamdi/eg-build', 'size': '7196'}