<jupyter_start><jupyter_text>Homework 4 Defeating 0.99% Error in MNIST Evaluation---Begin by downloading the data and splitting it into test and train datasets. The data needs to be reshapped as a single channel grayscale image, and the normalized so that the 'pixel' values all range from 0 to 1. The input images are also flattened in order to pass the data to the MLP input layer.<jupyter_code>from tensorflow.keras.utils import to_categorical
from tensorflow.keras.datasets import mnist
import numpy as np
from sklearn.model_selection import KFold
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Dense
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

# Download and reshape the data.
(x_train, y_train), (x_test, y_test) = mnist.load_data()
num_pixels = x_train.shape[1] * x_train.shape[2]
x_train = x_train.reshape(x_train.shape[0], 28, 28, 1).astype('float32')
x_test = x_test.reshape(x_test.shape[0], 28, 28, 1).astype('float32')
x_train_flat = x_train.reshape(x_train.shape[0], num_pixels).astype('float32')
x_test_flat = x_test.reshape(x_test.shape[0], num_pixels).astype('float32')

# Convert to 'one-hot' encoding.
y_test = to_categorical(y_test)
y_train = to_categorical(y_train)

# Normalize pixels.
x_train_flat = x_train_flat / 255.0
x_test_flat = x_test_flat / 255.0
x_train = x_train / 255.0
x_test = x_test / 255.0
num_classes = y_test.shape[1]<jupyter_output><empty_output><jupyter_text>Part 1---The baseline is defined and evaluated. The baseline produces an error of 1.85%.<jupyter_code># define a simple CNN model
def baseline_model():
  # create model
	model = Sequential()
	model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
	model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
	# Compile model
	model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

# build the model
model = baseline_model()
# Fit the model
model.fit(x_train_flat, y_train, validation_data=(x_test_flat, y_test), epochs=10, batch_size=200, verbose=2)
# Final evaluation of the model
scores = model.evaluate(x_test_flat, y_test, verbose=0)
print("CNN Error: %.2f%%" % (100-scores[1]*100))<jupyter_output>Epoch 1/10
300/300 - 1s - loss: 0.2796 - accuracy: 0.9205 - val_loss: 0.1375 - val_accuracy: 0.9606
Epoch 2/10
300/300 - 1s - loss: 0.1121 - accuracy: 0.9670 - val_loss: 0.0976 - val_accuracy: 0.9704
Epoch 3/10
300/300 - 1s - loss: 0.0716 - accuracy: 0.9790 - val_loss: 0.0746 - val_accuracy: 0.9763
Epoch 4/10
300/300 - 1s - loss: 0.0502 - accuracy: 0.9853 - val_loss: 0.0685 - val_accuracy: 0.9788
Epoch 5/10
300/300 - 1s - loss: 0.0360 - accuracy: 0.9898 - val_loss: 0.0677 - val_accuracy: 0.9791
Epoch 6/10
300/300 - 1s - loss: 0.0272 - accuracy: 0.9929 - val_loss: 0.0658 - val_accuracy: 0.9786
Epoch 7/10
300/300 - 1s - loss: 0.0206 - accuracy: 0.9948 - val_loss: 0.0684 - val_accuracy: 0.9790
Epoch 8/10
300/300 - 1s - loss: 0.0146 - accuracy: 0.9969 - val_loss: 0.0587 - val_accuracy: 0.9820
Epoch 9/10
300/300 - 1s - loss: 0.0099 - accuracy: 0.9980 - val_loss: 0.0602 - val_accuracy: 0.9815
Epoch 10/10
300/300 - 1s - loss: 0.0077 - accuracy: 0.9987 - val_loss: 0.0612 - val_accuracy: 0.9815[...]<jupyter_text>A new model is defined. This model is wider, deeper, and has Dropout layers for regularization. The ImageDataGenerator class is also used to augment the input data. A wrapper class flattens the output of the data augmentation in order to make it compatable with the MLP network.<jupyter_code># define a simple CNN model
def new_model():
  # create model
  model = Sequential()
  model.add(Dense(num_pixels*3, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
  model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
  model.add(Dense(num_classes, activation='softmax'))
  # Compile model
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  return model
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import NumpyArrayIterator
from tensorflow.keras.callbacks import EarlyStopping

dataflow = ImageDataGenerator(
    width_shift_range=0.1,
    height_shift_range=0.1,
    rotation_range=10
    )
dataflow.fit(x_train)

batch_size=200
class Wrapper(NumpyArrayIterator):
  def __init__(self, source):
    self.source = source

  def on_epoch_end(self):
    self.source.on_epoch_end()

  def __getitem__(self, idx):
    original = self.source.__getitem__(idx)
    x_f = original[0].reshape(batch_size, num_pixels).astype('float32')
    return (x_f, original[1])

  def __len__(self):
    return self.source.__len__()

  def __iter__(self):
    return self.source.__iter__()

# build the model
model = new_model()
flowSource = Wrapper(dataflow.flow(x_train, y_train, batch_size=batch_size))
# Fit the model
model.fit(flowSource, validation_data=(x_test_flat, y_test), epochs=10, steps_per_epoch=len(x_train) / batch_size, batch_size=batch_size, verbose=2)
# Final evaluation of the model
scores = model.evaluate(x_test_flat, y_test, verbose=0)
print("CNN Error: %.2f%%" % (100-scores[1]*100))
print(model.summary())<jupyter_output>Model: "sequential_61"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_156 (Dense)            (None, 2352)              1846320   
_________________________________________________________________
dense_157 (Dense)            (None, 784)               1844752   
_________________________________________________________________
dense_158 (Dense)            (None, 10)                7850      
=================================================================
Total params: 3,698,922
Trainable params: 3,698,922
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>By augmenting the input data and expanding the MLP net, the model produces an error of 1.47%--- Part 2In order to define a CNN model, a few helper methods are defined to evaluate the model and perform a grid search.<jupyter_code># Define CV split
kfold = KFold(n_splits=5, shuffle=True)

# Define helper functions.
def eval_model(train_index, test_index, build_fn, *args):
  model = build_fn(*args)
  print(model.summary)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  return score

def perform_grid_search(build_fn, param_grid, x, y, n_jobs=1, cv=None, **sk_params):
  estimator = KerasClassifier(build_fn=build_fn, **sk_params)
  grid_search = GridSearchCV(estimator, param_grid, n_jobs=n_jobs, cv=cv)
  grid_search.fit(x, y)

  means = grid_search.cv_results_['mean_test_score']
  stds = grid_search.cv_results_['std_test_score']
  for mean, std, params in zip(means, stds, grid_search.cv_results_['params']):
      print("%0.3f (+/-%0.03f) for %r"
              % (mean, std * 2, params))<jupyter_output><empty_output><jupyter_text>Starting with a single CNN layer, the possible number of Conv2D layers are evaluated. The padding is chosen as 'same' considering that the data around the edges of the images does not really inuence the important pixel data representing the hand-written digit.<jupyter_code>def determine_number_of_layers(layers, size, kernel_size, pool_size):
  model = Sequential()
  model.add(Conv2D(size, kernel_size, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
  for _ in range(1,layers):
    model.add(Conv2D(size, kernel_size, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  model.add(Flatten())
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  return model

scores = {}
scores['Test #1'] = []
scores['Test #2'] = []
scores['Test #3'] = []

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(1,32,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #1'].append(score)

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(2,32,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #2'].append(score)

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(3,32,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #3'].append(score)

for s in scores:
  print('{} Test Accuracy: {:.2%}'.format(s, np.mean([acc[1] for acc in scores[s]])))<jupyter_output>Epoch 1/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.1923 - accuracy: 0.9454
Epoch 2/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0719 - accuracy: 0.9789
Epoch 3/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0505 - accuracy: 0.9852
Epoch 4/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0366 - accuracy: 0.9886
Epoch 5/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0290 - accuracy: 0.9909
Epoch 6/10
1500/1500 [==============================] - 4s 2ms/step - loss: 0.0213 - accuracy: 0.9931
Epoch 7/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0176 - accuracy: 0.9943
Epoch 8/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0116 - accuracy: 0.9966
Epoch 9/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0099 - accuracy: 0.9967
Epoch 10/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.007[...]<jupyter_text>From the results of the experiment, three Conv2D layers performed the best. The greatest amount of improvement though came from switching from one to two Conv2D layers. Two layers are chosen since the increase in performance with three is negligible, and two layers will help keep the number of parameters constrained.---The next experiment tests for the best number of (Conv2D + MaxPooling2D) layers.<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

def determine_number_of_layers(cnn_layers, kernel_size, pool_size):
  size = 32
  model = Sequential()
  model.add(Conv2D(size, kernel_size, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
  model.add(Conv2D(size, kernel_size, activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  for i in range(1,cnn_layers):
    model.add(Conv2D(size*(i+1), kernel_size, activation = 'relu', padding = 'same'))
    model.add(Conv2D(size*(i+1), kernel_size, activation = 'relu', padding = 'same'))
    model.add(MaxPooling2D((pool_size, pool_size)))

  model.add(Flatten())
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  return model

scores = {}
scores['Test #1'] = []
scores['Test #2'] = []
scores['Test #3'] = []

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(1,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #1'].append(score)

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(2,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #2'].append(score)

for train_index, test_index in kfold.split(x_train,y_train): 
  model = determine_number_of_layers(3,3,1)
  model.fit(x_train[train_index], y_train[train_index], epochs=10)
  score = model.evaluate(x_train[test_index], y_train[test_index])
  scores['Test #3'].append(score)

for s in scores:
  print('{} Test Accuracy: {:.2%}'.format(s, np.mean([acc[1] for acc in scores[s]])))<jupyter_output>Epoch 1/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.1424 - accuracy: 0.9566
Epoch 2/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0499 - accuracy: 0.9851
Epoch 3/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0333 - accuracy: 0.9895
Epoch 4/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0217 - accuracy: 0.9923
Epoch 5/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0154 - accuracy: 0.9950
Epoch 6/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0125 - accuracy: 0.9958
Epoch 7/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0100 - accuracy: 0.9966
Epoch 8/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0065 - accuracy: 0.9977
Epoch 9/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.0051 - accuracy: 0.9982
Epoch 10/10
1500/1500 [==============================] - 4s 3ms/step - loss: 0.005[...]<jupyter_text>From the three experiments, its clear that ther performance increases gradually when more layers are added. Two layers are selected to strike a balance between performance and model size, but the may be a hyperparameter that can be optimized further if needed.---In the next experiement, a dense layer is added between the CNN layers and the output.<jupyter_code>scores = []

def determine_dense_size(dense_layer, kernel_size=3, pool_size=1):
  size = 32
  model = Sequential()
  model.add(Conv2D(size, kernel_size, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
  model.add(Conv2D(size, kernel_size, activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  model.add(Conv2D(size*2, kernel_size, activation = 'relu', padding = 'same'))
  model.add(Conv2D(size*2, kernel_size, activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  model.add(Flatten())
  model.add(Dense(dense_layer, activation='relu'))
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  return model

params = [128,256,512]

for p in params:
  for train_index, test_index in kfold.split(x_train,y_train):
    score = eval_model(train_index, test_index, determine_dense_size, p)
    scores.append(score)

for i, s in zip(range(len(scores)),scores):
  print('Test {} Test Accuracy: {:.2%}'.format(i+1, s[1]))<jupyter_output>Epoch 1/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.1137 - accuracy: 0.9656
Epoch 2/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0417 - accuracy: 0.9870
Epoch 3/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0266 - accuracy: 0.9916
Epoch 4/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0196 - accuracy: 0.9934
Epoch 5/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0125 - accuracy: 0.9960
Epoch 6/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0107 - accuracy: 0.9965
Epoch 7/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0081 - accuracy: 0.9972
Epoch 8/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0088 - accuracy: 0.9976
Epoch 9/10
1500/1500 [==============================] - 10s 7ms/step - loss: 0.0055 - accuracy: 0.9984
Epoch 10/10
1500/1500 [==============================] - 10s 7ms/step - l[...]<jupyter_text>From this test, even the smallest dense layer provided a small increase in performance. Varying the size of the layer did not seem to impact the results, so the smallest tested layer is chosen.---The next experiment varies the CNN kernel size.<jupyter_code>def determine_kernel_size(kernel_size=3, pool_size=1):
  size = 32
  model = Sequential()
  model.add(Conv2D(size, kernel_size, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
  model.add(Conv2D(size, kernel_size, activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  model.add(Conv2D(size*2, kernel_size, activation = 'relu', padding = 'same'))
  model.add(Conv2D(size*2, kernel_size, activation = 'relu', padding = 'same'))
  model.add(MaxPooling2D((pool_size, pool_size)))
  model.add(Flatten())
  model.add(Dense(128, activation='relu'))
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
  return model
scores = {}

model = determine_kernel_size(kernel_size=3)
print(model.summary)
model.fit(x_train, y_train, epochs=10)
score = model.evaluate(x_test, y_test)
scores['Test #1'] = score

model = determine_kernel_size(kernel_size=5)
print(model.summary)
model.fit(x_train, y_train, epochs=10)
score = model.evaluate(x_test, y_test)
scores['Test #2'] = score

for s in scores:
  print('{} Accuracy: {:.2%}'.format(s, scores[s][1]))<jupyter_output><bound method Model.summary of <tensorflow.python.keras.engine.sequential.Sequential object at 0x7fa784709ef0>>
Epoch 1/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.1087 - accuracy: 0.9672
Epoch 2/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0376 - accuracy: 0.9883
Epoch 3/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0234 - accuracy: 0.9930
Epoch 4/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0144 - accuracy: 0.9954
Epoch 5/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0121 - accuracy: 0.9965
Epoch 6/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0087 - accuracy: 0.9974
Epoch 7/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0082 - accuracy: 0.9973
Epoch 8/10
1875/1875 [==============================] - 13s 7ms/step - loss: 0.0058 - accuracy: 0.9981
Epoch 9/10
1875/1875 [==============================] - 13s 7ms/[...]<jupyter_text>From these results, there isn't a big difference from changing the kernel size so (3,3) is chosen.---With the chosen hyperparameters, the image data is augmented in order to train the final model.<jupyter_code>from tensorflow.keras.preprocessing.image import ImageDataGenerator

dataflow = ImageDataGenerator(
    width_shift_range=0.1,
    height_shift_range=0.1,
    rotation_range=10
    )
dataflow.fit(x_train)

model = Sequential()
model.add(Conv2D(32, 3, input_shape=(28, 28, 1), activation = 'relu', padding = 'same'))
model.add(Conv2D(32, 3, activation = 'relu', padding = 'same'))
model.add(MaxPooling2D())
model.add(Conv2D(64, 3, activation = 'relu', padding = 'same'))
model.add(Conv2D(64, 3, activation = 'relu', padding = 'same'))
model.add(MaxPooling2D())
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
print(model.summary())
model.fit(
    dataflow.flow(x_train, y_train, batch_size=86),
    steps_per_epoch=len(x_train) / 86, 
    epochs=10,  
    batch_size=86,
    validation_data=(x_test, y_test))

# Final evaluation of the model
scores = model.evaluate(x_test, y_test, verbose=0)
print("CNN Error: %.2f%%" % (100-scores[1]*100))<jupyter_output>Model: "sequential_67"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_28 (Conv2D)           (None, 28, 28, 32)        320       
_________________________________________________________________
conv2d_29 (Conv2D)           (None, 28, 28, 32)        9248      
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_30 (Conv2D)           (None, 14, 14, 64)        18496     
_________________________________________________________________
conv2d_31 (Conv2D)           (None, 14, 14, 64)        36928     
_________________________________________________________________
max_pooling2d_19 (MaxPooling (None, 7, 7, 64)          0         
_____________________________________________________[...]<jupyter_text>In the final evaluation with the augmented data, the error reaches 0.51%.--- Final Evaluation Error: 0.51%<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1835025499461938, 'fraction_numerical': 0.07720020586721564, 'mean_word_length': 5.150791366906475, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '349165', 'license': 'MIT', 'n_tokens_mistral': 8281, 'n_tokens_neox': 7054, 'n_words': 1755, 'path': 'HW4/mnist_challenge.ipynb', 'repo_name': 'robjmenna/RBE595_Homework', 'size': '21373'}