<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Imports2&nbsp;&nbsp;The Data2.1&nbsp;&nbsp;What does the imported data set look like?2.1.1&nbsp;&nbsp;X2.1.2&nbsp;&nbsp;y2.2&nbsp;&nbsp;What do the digit images look like?2.3&nbsp;&nbsp;Split the data into a training and test set.2.4&nbsp;&nbsp;The y labels need to be converted to binary class matrices3&nbsp;&nbsp;Experimenting with a basic Deep Neural Network for image analysis using the MNIST digit dataset3.1&nbsp;&nbsp;Experiment 1 - Using a basic Keras Sequential Model3.1.1&nbsp;&nbsp;Experiment 1 results3.2&nbsp;&nbsp;Experiment 2 - What effect does scaling the data have?3.3&nbsp;&nbsp;Add a second hidden layer3.4&nbsp;&nbsp;What effect does adding a batch size have?3.4.1&nbsp;&nbsp;A small batch size, smooths the accuracy curve for the test samples3.5&nbsp;&nbsp;Hands on book recommends multiple layers of the same size with no taper3.5.1&nbsp;&nbsp;Summary4&nbsp;&nbsp;Improved Keras DNN model4.1&nbsp;&nbsp;Steps to avoid vanishing/ exploding gradients4.1.1&nbsp;&nbsp;Usage of initializers4.1.2&nbsp;&nbsp;Activation Functions4.2&nbsp;&nbsp;Regularisation4.2.1&nbsp;&nbsp;Early Stopping4.2.2&nbsp;&nbsp;Learning Rate reduction4.2.3&nbsp;&nbsp;Adding Dropout Layers4.2.4&nbsp;&nbsp;Addng more epochs to the dropout model4.2.5&nbsp;&nbsp;Regularisation conclusion4.2.6&nbsp;&nbsp;What happens if the number of dropout layers is reduced?5&nbsp;&nbsp;Convolutional Neural Network Model5.1&nbsp;&nbsp;Reshape the data into 28 x 28 images5.2&nbsp;&nbsp;Basic CNN model5.3&nbsp;&nbsp;Can a more complex CNN do better?5.4&nbsp;&nbsp;99.2% accuracy amazing ! - what digits is the model getting wrong?5.5&nbsp;&nbsp;Results summary6&nbsp;&nbsp;Overall Conclusions  Exploring Neural Networks for use with images; using the MNIST Digits data set  These experiments aimed to explore the performance of different types of Neural Networks and how to set up their hyperparameters. What is the ideal solution to analysing the standard MNIST digits data set?The MNIST digits data set http://yann.lecun.com/exdb/mnist/ of handwritten digits, has 70000 examples. The digits have already been size-normalized and centered in a fixed-size image of 28 x 28 pixels.  The data set was chosen for these experiments as it requires minimal preprocessing and formating. Therefore the focus can be on the performance different types of Neural Networks and the effect of hyperparameter tuning.     Imports<jupyter_code>%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
from keras.layers import Dense
from keras.models import Sequential
from keras.utils import to_categorical
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings(action='once') # supress the repition of warnings<jupyter_output><empty_output><jupyter_text>The Data  For full details of the data set used here see https://www.openml.org/d/554.<jupyter_code># Load data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)<jupyter_output><empty_output><jupyter_text>What does the imported data set look like?   The data was received as two arrays loaded onto the variables X an y. X contains our image data and y contains the label to describe the image and is the target we will try to predict.  X  The shape command below shows that there are 70000 images, which fits with the data description above. The 784 columns is due to the data having one column for each pixel. i.e. 28 x 28 = 784. The data is in the numpy array format already rather than a dataframe and the values range from 0 to 255 for the pixel intensity.<jupyter_code>print(' X shape:', X.shape, '\n X type:', type(X), '\n Data:\n',
      X[:5], '\n X Max:', X.max(), '\n X Min', X.min())<jupyter_output>X shape: (70000, 784) 
 X type: <class 'numpy.ndarray'> 
 Data:
 [[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]] 
 X Max: 255.0 
 X Min 0.0<jupyter_text>y  y contains our target label and only has a single column with the labels 0 to 9. Notice that the labels are text strings rather than numbers.<jupyter_code>print(' y shape:', y.shape, '\n y type:', type(y), '\n Data:\n',
      y[:5], '\n y Max:', y.max(), '\n y Min', y.min())<jupyter_output>y shape: (70000,) 
 y type: <class 'numpy.ndarray'> 
 Data:
 ['5' '0' '4' '1' '9'] 
 y Max: 9 
 y Min 0<jupyter_text>What do the digit images look like?  Lets have a look at the first 10 digits<jupyter_code>for i in range(10):
    a_digit = X[i].reshape(28, 28) # put the 784 columns into a 28 x 28 format
    plt.imshow(a_digit, cmap=matplotlib.cm.binary)
    print('The digit label', y[i])
    plt.show()<jupyter_output>The digit label 5<jupyter_text>Split the data into a training and test set.  In order to test the performance of these models the images need to be split into a training and test set (Validation set). We will keep back 25% of the image to evaluate the model's performance. The term validation set is often used for a test split of the training data when there is an unseen test dataset such as in data science competitions. Here the term will be used to mean the 25% test data held back from the training data.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y , 
                                                    test_size=0.25, 
                                                    random_state=42, 
                                                    shuffle=True)<jupyter_output><empty_output><jupyter_text>The y labels need to be converted to binary class matrices  i.e. a value of 1 or 0 for each label. To do this we will use the Keras function to_categorical<jupyter_code>num_classes = 10
y_train = to_categorical(y_train, num_classes)
y_test = to_categorical(y_test, num_classes)<jupyter_output><empty_output><jupyter_text>Experimenting with a basic Deep Neural Network for image analysis using the MNIST digit dataset  Experiment 1 - Using a basic Keras Sequential Model  First we will use a model with a single Dense layer and an output layer to keep things as simple as possible and create a baseline to work from. For more information on the Sequential model see https://keras.io/getting-started/sequential-model-guide/<jupyter_code># use Keras to build the model
model = Sequential()

# layer 1
model.add(Dense(300, input_dim=784,  activation='relu'))

# output layer
model.add(Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model_his = model.fit(X_train, y_train,
                      epochs=10,
                      validation_data=(X_test, y_test)
                     )
# create a function to plot the results and use later in the code
def plot_results(acc, val_acc, fig_no):
    # Plot train vs test accuracy per epoch
    plt.figure()

    # Use the history metrics
    plt.plot(acc)
    plt.plot(val_acc)

    # add some labels
    plt.title('Model accuracy')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['Train data', 'Validation data'])
    plt.ylim(0, 1)  # set to highest possible value to

    # the highest accuracy score
    print('\n Fig {}. The highest test score:{:.3f} | The highest training score {:.3f}'.format(
        fig_no, max(val_acc), max(acc)))
    plt.show()
    # keep a record of the results
    return (max(acc), max(val_acc))

exp1 = plot_results(model_his.history['acc'], model_his.history['val_acc'], 1)<jupyter_output>Fig 1. The highest test score:0.545 | The highest training score 0.534<jupyter_text>Experiment 1 results  The model performed poorly with accuracy ranging from 0.35 to 0.55 depending on the run. If you try repeating the run you will see that the performance is not very stable. The model was not overfitted with the training accuracy close to the validation accuracy  Experiment 2 - What effect does scaling the data have?In general, scaling improves the results of machine learning models. With neural networks, it is generally accepted that scaling the data will improve the convergence speed and the accuracy of the model. There are three approaches to scaling the data.  a) $(X - X.min()) / (X.max() - X.min())$   which produces values from 0 to 1  b) $2*(X - X.min()) / (X.max() - X.min()) - 1 $  which produces values from -1 to 1  c) $(X - X.mean()) / X.std()$   which centres the values around the meanThe use of centring may also help to prevent vanishing and exploding gradients within the model. However, this data is already quite uniform with no outliers and the X values ranging from 0 to 255. Will any of these scaling methods improve the results of our simple model?<jupyter_code># X min is zero so the equations above can be simplified
Xa = X / X.max()
Xb = 2 * (X / X.max()) -1
Xc = (X - X.mean()) / X.std()
# repeat everything as before for each of the scaling methods
exp = 2  # track which experiment we are on


def scale_and_model(X, y, exp):
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        test_size=0.25,
                                                        random_state=42,
                                                        shuffle=True)
    num_classes = 10
    y_train = to_categorical(y_train, num_classes)
    y_test = to_categorical(y_test, num_classes)

    # use Keras to build the model
    model = Sequential()
    # layer 1
    model.add(Dense(300, input_dim=784,  activation='relu'))
    # output layer
    model.add(Dense(10, activation='softmax'))
    # Compile the model
    model.compile(optimizer='adam', loss='categorical_crossentropy',
                  metrics=['accuracy'])
    # Train the model
    model_H = model.fit(X_train, y_train,
                        epochs=10,
                        verbose=0,  # turn off the output of each epoch
                        validation_data=(X_test, y_test)
                        )

    plot_results(model_H.history['acc'], model_H.history['val_acc'], exp)
exp2 = scale_and_model(Xa, y, 2)
exp3 = scale_and_model(Xb, y, 3)
exp4 = scale_and_model(Xc, y, 4)<jupyter_output>Fig 4. The highest test score:0.976 | The highest training score 0.993<jupyter_text>Add a second hidden layer<jupyter_code>model = Sequential()

# layer 1
model.add(Dense(300, input_dim=784,  activation='relu'))

# layer 2
model.add(Dense(100,  activation='relu'))

# output layer 
model.add(Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

#Train the model
model_his = model.fit(X_train, y_train, 
                      epochs=10, 
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()

# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])

# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\n The highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>What effect does adding a batch size have?<jupyter_code>model = Sequential()

# layer 1
model.add(Dense(300, input_dim=784,  activation='relu'))

# layer 2
model.add(Dense(100,  activation='relu'))

# output layer 
model.add(Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=10, 
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()

# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])

# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>A small batch size, smooths the accuracy curve for the test samples  12 worked better than 50 and 128 was less smooth than 50 all values were smoother than using no batch size at all. The smaller size takes longer to complete each epoch - 35s for 12.  Hands on book recommends multiple layers of the same size with no taper<jupyter_code>model = Sequential()

# layer 1
model.add(Dense(300, input_dim=784,  activation='relu'))

# layer 2
model.add(Dense(300,  activation='relu'))

# layer 3
model.add(Dense(300,  activation='relu'))

# layer 4
model.add(Dense(300,  activation='relu'))

# output layer 
model.add(Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=10, 
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()

# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])

# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Summary  Looks like adding more hidden layers only marginally improves the model and batch size is a useful tool.   Improved Keras DNN model     Steps to avoid vanishing/ exploding gradients   Usage of initializers  The default option is Xavier initialisation, Gerlon in Hands On recommends He initialisation with relu activation<jupyter_code>model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
# layer 2
model.add(Dense(300,  activation='relu'))
# layer 3
model.add(Dense(300,  activation='relu'))
# layer 4
model.add(Dense(300,  activation='relu'))
# output layer 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=10, 
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Activation FunctionsUsed to avoid dying neurons stuck at zero output In general ELU > Leaky ReLU > ReLU > tanh > logistic  ELU is slighter slower during runtime<jupyter_code>model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
# layer 2
model.add(Dense(300,  activation='elu'))
# layer 3
model.add(Dense(300,  activation='elu'))
# layer 4
model.add(Dense(300,  activation='elu'))
# output layer 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=10, 
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Regularisation    Early Stopping<jupyter_code>from keras.callbacks import EarlyStopping

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=3)

model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
# layer 2
model.add(Dense(300,  activation='elu'))
# layer 3
model.add(Dense(300,  activation='elu'))
# layer 4
model.add(Dense(300,  activation='elu'))
# output layer 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=30, 
                      callbacks=[es],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Learning Rate reduction<jupyter_code>from keras.callbacks import ReduceLROnPlateau

red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=5)

model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
# layer 2
model.add(Dense(300,  activation='elu'))
# layer 3
model.add(Dense(300,  activation='elu'))
# layer 4
model.add(Dense(300,  activation='elu'))
# output layer 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=30, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Adding Dropout Layers  Add a dropout layer between each hidden layer and see what happens<jupyter_code>from keras.layers import Dropout

red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=5)

model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.5)) # 0.5 probability of a neuron being dropped
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5)) 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=30, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Addng more epochs to the dropout model  Adding the dropout layers slowed the training of the model, however it dramatically reduced over fitting and looked like it would continue to learn with additional epochs. So many dropout layers may be overkill?<jupyter_code>red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=8)

model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5)) 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=50, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Regularisation conclusion  * The reduced learning rate function had a dramatic effect on improving the model accuracy  * The dropout layers improved the over fitting to the training data  What happens if the number of dropout layers is reduced?<jupyter_code>red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=8)

model = Sequential()
# layer 1
model.add(Dense(300, input_dim=784,  activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
# model.add(Dropout(0.5))
model.add(Dense(300,  activation='elu'))
# model.add(Dropout(0.5)) 
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=50, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Convolutional Neural Network Model   Reshape the data into 28 x 28 images  The data used above was processed as single rows of 784 columns for the CNN we need to processes it as an image. Therefore it needs reshaping into 70000 28 x 28 arrays with a single channel for the greyscale values<jupyter_code># reshape to be [samples][width][height][channels]
Xr = X.reshape(X.shape[0], 28, 28, 1)

# normalise the data divide by the highest value 
Xr /= 255
# check it looks right
# Xr[0]
# convert class vectors to binary class matrices
num_classes = 10
y = to_categorical(y, num_classes)
# split the data 
X_train, X_test, y_train, y_test = train_test_split(Xr, y , 
                                                    test_size=0.25, 
                                                    random_state=42, 
                                                    shuffle=True)<jupyter_output><empty_output><jupyter_text>Basic CNN model<jupyter_code>from keras.layers import Conv2D, MaxPooling2D, Flatten

red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=8)

model = Sequential()

model.add(Conv2D(64, input_shape=(28, 28, 1) , kernel_size=(3,3), activation='elu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(300, activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=50, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>Can a more complex CNN do better?  This model next model is based on a shortened ResNet style model with less convolutional layers and without the skipping layers<jupyter_code>red_lr = ReduceLROnPlateau(monitor='val_acc', patience=2, verbose=1, mode='max' )

es = EarlyStopping(monitor='val_acc', mode='max', verbose=1, patience=8)

model = Sequential()

model.add(Conv2D(64, input_shape=(28, 28, 1) , kernel_size=(7,7), activation='elu'))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Conv2D(64, kernel_size=(3,3), activation='elu'))
model.add(Conv2D(64, kernel_size=(3,3), activation='elu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(300, activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(300, activation='elu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))
# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
#Train the model
model_his = model.fit(X_train, y_train, 
                      batch_size=50, # 50, 128, 12
                      epochs=50, 
                      callbacks=[es, red_lr],
                      validation_data=(X_test, y_test) 
                      )

# Plot train vs test accuracy per epoch
plt.figure()
# Use the history metrics
plt.plot(model_his.history['acc'])
plt.plot(model_his.history['val_acc'])
# Make it pretty
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'])
# the highest accuracy score for the adding another hidden layer
print('\nThe highest test score', max(model_his.history['val_acc']).round(3))
plt.show()<jupyter_output><empty_output><jupyter_text>99.2% accuracy amazing ! - what digits is the model getting wrong?<jupyter_code># fit the model
model.fit(X_train, y_train, 
                      batch_size=50,
                      epochs=15)
# predict X_test
preds = model.predict(X_test)
# convert back from a categorical array to an array of the digits predicted and expected
num_preds = np.argmax(preds, axis=1)

num_y_test = np.argmax(y_test, axis=1)
# check the accuracy is the same as before
from sklearn.metrics import accuracy_score

accuracy_score(num_preds, num_y_test)
# which numbers did the model get wrong?
# create a df to hold the results
result = pd.DataFrame({'preds' : num_preds, 'y' : num_y_test})
# label th inncorrect results
result['correct'] = result['preds'] == result['y']
# select the incorrect results
wrong = result[result['correct'] == False]
# which numbers did the model get wrong?
wrong['y'].value_counts()<jupyter_output><empty_output><jupyter_text>There isn't a single number the model gets wrong, it performs better on 0's ,1's and 6's.Lets have a look at some of the misidentified numbers.<jupyter_code>wrong.head(9)
w_list = wrong[:18].index
plt.figure(figsize=(9, 12))
for i, v in enumerate(w_list):
    plt.subplot(6,3,i+1)
    plt.imshow(X_test[v][:,:,0], cmap=matplotlib.cm.binary)
    plt.title('Pred: {} Act: {}'.format(wrong['preds'][v], wrong['y'][v]))
plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11937232717109472, 'fraction_numerical': 0.026804054281577262, 'mean_word_length': 4.006405933917734, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 98, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '653803', 'license': 'MIT', 'n_tokens_mistral': 10075, 'n_tokens_neox': 9249, 'n_words': 2772, 'path': 'Exploring Neural Networks for use with images; using the MNIST Digits data set.ipynb', 'repo_name': 'ant358/Handwritten-digits', 'size': '29697'}