<jupyter_start><jupyter_text>Digit Recognizer using CNN  Introduction - MNIST ("Modified National Institute of Standards and Technology") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms.- In this Project, our goal is to correctly identify digits from a dataset of tens of thousands of handwritten images.  Dataset Description - MNIST is a simple computer vision dataset. It consists of 28x28 pixel images of handwritten digits.- Every MNIST data point, every image, can be thought of as an array of numbers describing how dark each pixel is. Since each image has 28 by 28 pixels, we get a 28x28 array. - We can flatten each array into a 28∗28 = 784 dimensional vector. Each component of the vector is a value between zero and one describing the intensity of the pixel. Thus, we generally think of MNIST as being a collection of 784-dimensional vectors.  - Not all vectors in this 784-dimensional space are MNIST digits. Typical points in this space are very different! To get a sense of what a typical point looks like, we can randomly pick a few points and examine them. In a random point – a random 28x28 image – each pixel is randomly black, white or some shade of gray. The result is that random points look like noise.  Context - Images like MNIST digits are very rare. While the MNIST data points are embedded in 784-dimensional space, they live in a very small subspace. - One popular theory among machine learning researchers is the manifold hypothesis: MNIST is a low dimensional manifold, sweeping and curving through its high-dimensional embedding space. - Another hypothesis, more associated with topological data analysis, is that data like MNIST consists of blobs with tentacle-like protrusions sticking out into the surrounding space.  But no one really knows, so lets explore!  1. Importing Dependencies<jupyter_code># Basic Libraries
import numpy as np
import pandas as pd

# Plotting Libraries
import matplotlib.pyplot as plt
import seaborn as sns

# Tensorflow
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Input, Add, Activation, Dropout, Flatten, Dense
from tensorflow.keras.layers import Convolution2D, MaxPool2D, AveragePooling2D, ZeroPadding2D, Conv2D
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.regularizers import l2
from tensorflow.keras.callbacks import LearningRateScheduler
from tensorflow.keras import backend as k

from tensorflow.keras.datasets import mnist

from keras.utils import np_utils

# Other Files
import pickle
from os import listdir
from os.path import isfile, join
import os
import time
# this function is used to update the plots for each epoch and error
def plt_dynamic(x, vy, ty, ax, colors = ['b']):
    ax.plot(x, vy, 'b', label = "Validation Loss")
    ax.plot(x, ty, 'r', label = "Train Loss")
    plt.legend()
    plt.grid()
    fig.canvas.draw()<jupyter_output><empty_output><jupyter_text>2. Bringing the data to a format acceptable by the CNN<jupyter_code># input image dimensions
img_rows, img_cols = 28, 28

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if k.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
    
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)
# Dumping the data generated into seperate files
with open('x_train.pickle', 'wb') as handle:
    pickle.dump(x_train, handle, protocol = pickle.HIGHEST_PROTOCOL)
with open('y_train.pickle', 'wb') as handle:
    pickle.dump(y_train, handle, protocol = pickle.HIGHEST_PROTOCOL)
with open('x_test.pickle', 'wb') as handle:
    pickle.dump(x_test, handle, protocol = pickle.HIGHEST_PROTOCOL)
with open('y_test.pickle', 'wb') as handle:
    pickle.dump(y_test, handle, protocol = pickle.HIGHEST_PROTOCOL)
# Training and validation files

files = ['x_train.pickle', 'y_train.pickle',
         'x_test.pickle', 'y_test.pickle']

# Load training samples
with open(files[0], 'rb') as img_file:
    x_train = pickle.load(img_file, encoding='bytes')
    
# Load training labels
with open(files[1], 'rb') as lb_file:
    y_train = pickle.load(lb_file, encoding='bytes')
    
# Load validation samples
with open(files[2], 'rb') as img_file_v:
    x_test = pickle.load(img_file_v, encoding='bytes')
    
# Load validation labels
with open(files[3], 'rb') as lb_file_v:
    y_test = pickle.load(lb_file_v, encoding='bytes')<jupyter_output><empty_output><jupyter_text>3. Data Preprocessing<jupyter_code>num_classes = 10

# Transofrm the Features to a float32 type
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')

# Normalizing the input features
x_train /= 255
x_test /= 255

# convert class vectors to binary class matrices
y_train = np_utils.to_categorical(y_train, num_classes)
y_test = np_utils.to_categorical(y_test, num_classes)
print('Shape of Training Data   :', x_train.shape)
print('Shape of Test Data       :', x_test.shape)
print('Shape of Training Lables :', y_train.shape)
print('Shape of Test Lables     :', y_test.shape)<jupyter_output>Shape of Training Data   : (60000, 28, 28, 1)
Shape of Test Data       : (10000, 28, 28, 1)
Shape of Training Lables : (60000, 10)
Shape of Test Lables     : (10000, 10)<jupyter_text>4. Building the model  1. Base Model (2 Layer Architecture) 1. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu2. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu3. Max Pool layer with size 2×24. Dropout5. Flatten layer6. Fully connected layer with 256 units and a rectifier activation function as relu7. Fully connected output layer with 10 units and a softmax activation function8. Compiling the Model with adadelta optimizer and metrices as accuracy<jupyter_code># Initialising the Model_1
Model_1 = Sequential()

# Convolution
Model_1.add(Conv2D(filters = 32, kernel_size = (3,3), input_shape = input_shape, activation = 'relu'))
# Convolution
Model_1.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu'))
# Max Pooling
Model_1.add(MaxPool2D(pool_size = (2,2)))
# Dropout
Model_1.add(Dropout(0.25))

# Flattening
Model_1.add(Flatten())

# Full Connection
Model_1.add(Dense(256, activation = 'relu'))
# Dropout
Model_1.add(Dropout(0.25))

# Output Layer
Model_1.add(Dense(num_classes, activation = 'softmax'))

# Compiling Model_1
Model_1.compile(loss = 'categorical_crossentropy', optimizer = 'adadelta', metrics = ['accuracy'])
Model_1.summary()
Model_1_result = Model_1.fit(x_train, y_train, epochs = 15, validation_data = (x_test, y_test), workers = -1, 
                             batch_size = 256)
Model_1_evaluation = Model_1.evaluate(x_test, y_test)
print('Test Loss is : {} and the Test Accuracy is: {}'.format(Model_1_evaluation[0], Model_1_evaluation[1]))
# Plotting Train & Validation Loss
fig,ax = plt.subplots(1,1)
ax.set_xlabel('epoch') 
ax.set_ylabel('Categorical Crossentropy Loss')

# list of epoch numbers
x = list(range(1, 16))
vy = Model_1_result.history['val_loss']
ty = Model_1_result.history['loss']

# Calling Function
plt_dynamic(x, vy, ty, ax)<jupyter_output><empty_output><jupyter_text>Model 1 : Conclusions 1. Using our base model we are able to get of only 11.34% training accuracy and 11.35% test accuracy, which is very bad. 2. So we have options like we should increase the Hidden Layers, add Drop outs, add Batch Normalizations3. So lets regularize the model. In deep learning drop out is very good form of regularization. Dropout is a regularization method that approximates training a large number of neural networks with different architectures in parallel.4. Drop out literally means we gonna drop some of the neural units randomly from our network, which forces the next layer to learn the patterns again. In this way our model will become more robust and generalizes well.5. Lets try to add drop out after each CNN layer + MaxPool Layer.  2. Model 2 with Drop outs (3 LAYER ARCHITECTURE) 1. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu2. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu3. Max Pool layer with size 2×24. Dropout set to 25%5. Convolutional input layer, 128 feature maps with a size of 3×3, a rectifier activation function as relu6. Max Pool layer with size 2×27. Dropout set to 50%8. Flatten layer9. Fully connected layer with 256 units and a rectifier activation function as relu10. Dropout set to 50%11. Fully connected output layer with 10 units and a softmax activation function12. Compiling the Model with adam optimizer and metrices as accuracy<jupyter_code># Initialising the Model_2
Model_2 = Sequential()

# Convolution
Model_2.add(Conv2D(filters = 32, kernel_size = (3,3), input_shape = input_shape, activation = 'relu'))
# Convolution
Model_2.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu'))
# Max Pooling
Model_2.add(MaxPool2D(pool_size = (2,2)))
# Adding Dropouts
Model_2.add(Dropout(0.25))

# Adding 2nd Convolutional Layer
Model_2.add(Conv2D(filters = 128, kernel_size = (3,3), activation = 'relu'))
# Max Pooling
Model_2.add(MaxPool2D(pool_size = (2,2)))
# Adding Dropouts
Model_2.add(Dropout(0.5))

# Flattening
Model_2.add(Flatten())

# Full Connection
Model_2.add(Dense(256, activation = 'relu'))
# Adding Dropouts
Model_2.add(Dropout(0.50))

# Output Layer
Model_2.add(Dense(num_classes, activation = 'softmax'))

# Compiling Model_1
Model_2.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
Model_2.summary()
Model_2_result = Model_2.fit(x_train, y_train, epochs = 12, validation_data = (x_test, y_test), batch_size = 256, 
                             workers = -1)
Model_2_evaluation = Model_2.evaluate(x_test, y_test)
print('Test Loss is : {} and the Test Accuracy is: {}'.format(Model_2_evaluation[0], Model_2_evaluation[1]))
# Plotting Train & Validation Loss
fig,ax = plt.subplots(1,1)
ax.set_xlabel('epoch') 
ax.set_ylabel('Categorical Crossentropy Loss')

# list of epoch numbers
x = list(range(1, 13))
vy = Model_2_result.history['val_loss']
ty = Model_2_result.history['loss']

# Calling Function
plt_dynamic(x, vy, ty, ax)<jupyter_output><empty_output><jupyter_text>Model 2: Conclusions  1. Using our base model we are able to get of only 97.36% training accuracy and 98.58% test accuracy, which is really good than the previous model. 2. Lets try to improve our model by adding the Hidden Layers, add Drop outs, add Batch Normalizations  3. Model 3 with Drop outs, More Layers, Batch Normalization & More Filters (5 Layer Architecture) 1. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu2. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu3. Max Pool layer with size 2×24. Batch Normalization5. Dropout set to 40%6. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu7. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu8. Max Pool layer with size 2×29. Batch Normalization10. Dropout set to 40%11. Convolutional input layer, 128 feature maps with a size of 3×3, a rectifier activation function as relu12. Batch Normalization13. Dropout set to 50%14. Flatten layer15. Fully connected layer with 256 units and a rectifier activation function as relu    Dropout set to 50%16. Fully connected layer with 128 units and a rectifier activation function as relu17. Dropout set to 40%18. Fully connected output layer with 10 units and a softmax activation function19. Compiling the Model with adam optimizer and metrices as accuracy<jupyter_code># Initialising the Model_3
Model_3 = Sequential()

# Convolution
Model_3.add(Conv2D(filters = 32, kernel_size = (3,3), input_shape = input_shape, activation = 'relu'))
# Convolution
Model_3.add(Conv2D(filters = 32, kernel_size = (3,3), activation = 'relu'))
# Max Pooling
Model_3.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_3.add(BatchNormalization())
# Adding Dropouts
Model_3.add(Dropout(0.40))

# Convolutional Layer
Model_3.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu'))
# Convolutional Layer
Model_3.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu'))
# Max Pooling
Model_3.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_3.add(BatchNormalization())
# Adding Dropouts
Model_3.add(Dropout(0.40))

# Convolution
Model_3.add(Conv2D(128, kernel_size=3, activation = 'relu'))
# Batch Normalization
Model_3.add(BatchNormalization())
# Adding Dropouts
Model_3.add(Dropout(0.50))

# Flattening
Model_3.add(Flatten())

# Full Connection
Model_3.add(Dense(256, activation = 'relu'))
# Adding Dropouts
Model_3.add(Dropout(0.50))
# Full Connection
Model_3.add(Dense(128, activation = "relu"))
# Adding Dropouts
Model_3.add(Dropout(0.4))

# Output Layer
Model_3.add(Dense(num_classes, activation = 'softmax'))

# Compiling Model_3
Model_3.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
Model_3.summary()
Model_3_result = Model_3.fit(x_train, y_train, epochs = 12, validation_data = (x_test, y_test), batch_size = 256,
                             workers = -1)
Model_3_evaluation = Model_3.evaluate(x_test, y_test)
print('Test Loss is : {} and the Test Accuracy is: {}'.format(Model_3_evaluation[0], Model_3_evaluation[1]))
# Plotting Train & Validation Loss
fig,ax = plt.subplots(1,1)
ax.set_xlabel('epoch') 
ax.set_ylabel('Categorical Crossentropy Loss')

# list of epoch numbers
x = list(range(1, 13))
vy = Model_3_result.history['val_loss']
ty = Model_3_result.history['loss']

# Calling Function
plt_dynamic(x, vy, ty, ax)<jupyter_output><empty_output><jupyter_text>Model 3. Conclusions:  - The Model is overfitting as the Train & Test accuracy has great difference. Lets reduce number of epochs to 10.- Lets try to improve our model by adding the Hidden Layers, add Drop outs, Batch Normalizations, padding and weight initializations  Model 4: with Increased Dropouts, More Layers, More Filters, Padding, Using He Kernel Initializer and using Batch Normalization (7 Layer Architecture) 1. Convolutional input layer, 16 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same2. Convolutional input layer, 16 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same3. Max Pool layer with size 2×24. Batch Normalization5. Dropout set to 25%6. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same7. Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function as relu8. Max Pool layer with size 2×29. Batch Normalization10. Dropout set to 30%11. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same12. Convolutional input layer, 64 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same13. Max Pool layer with size 2×214. Batch Normalization15. Dropout set to 30%16. Convolutional input layer, 128 feature maps with a size of 3×3, a rectifier activation function as relu, initialization as He, padding same17. Max Pool layer with size 2×218. Batch Normalization19. Dropout set to 50%20. Flatten layer21. Fully connected layer with 256 units and a rectifier activation function as relu    Dropout set to 50%22. Fully connected layer with 128 units and a rectifier activation function as relu23. Dropout set to 40%24. Fully connected output layer with 10 units and a softmax activation function25. Compiling the Model with adam optimizer and metrices as accuracy<jupyter_code># Initialising the Model_4
Model_4 = Sequential()

# Convolution
Model_4.add(Conv2D(filters = 16, kernel_size = (3,3), input_shape = input_shape, activation = 'relu', padding = 'same',
                   kernel_initializer = 'he_uniform'))
# Convolution
Model_4.add(Conv2D(filters = 16, kernel_size = (3,3), activation = 'relu', kernel_initializer = 'he_uniform', 
                   padding='same'))
# Max Pooling
Model_4.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_4.add(BatchNormalization())
# Adding Dropouts
Model_4.add(Dropout(0.25))


# Convolution
Model_4.add(Conv2D(filters = 32, kernel_size = (3,3), activation = 'relu', kernel_initializer='he_uniform', 
                   padding='same'))
# Convolution
Model_4.add(Conv2D(filters = 32, kernel_size = (3,3), activation = 'relu', kernel_initializer='he_uniform', 
                   padding='same'))
# Max Pooling
Model_4.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_4.add(BatchNormalization())
# Adding Dropouts
Model_4.add(Dropout(0.30))


# Convolution
Model_4.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu', kernel_initializer='he_uniform', 
                   padding='same'))
# Convolution
Model_4.add(Conv2D(filters = 64, kernel_size = (3,3), activation = 'relu', kernel_initializer='he_uniform', 
                   padding='same'))
# Max Pooling
Model_4.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_4.add(BatchNormalization())
# Adding Dropouts
Model_4.add(Dropout(0.40))


# Convolution
Model_4.add(Conv2D(filters = 128, kernel_size = (3,3), activation = 'relu', kernel_initializer='he_uniform', 
                   padding='same'))
# Max Pooling
Model_4.add(MaxPool2D(pool_size = (2,2)))
# Batch Normalization
Model_4.add(BatchNormalization())
# Adding Dropouts
Model_4.add(Dropout(0.50))

# Flattening
Model_4.add(Flatten())

# Full Connection
Model_4.add(Dense(256, activation = 'relu', kernel_initializer='he_uniform'))
# Adding Dropouts
Model_4.add(Dropout(0.50))
# Full Connection
Model_4.add(Dense(128, activation = "relu"))


# Output Layer
Model_4.add(Dense(num_classes, activation = 'softmax'))

# Compiling Model_4
Model_4.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
Model_4.summary()
Model_4_result = Model_4.fit(x_train, y_train, epochs = 10, validation_data = (x_test, y_test), batch_size = 256,
                             workers = -1)
Model_4_evaluation = Model_4.evaluate(x_test, y_test)
print('Test Loss is : {} and the Test Accuracy is: {}'.format(Model_4_evaluation[0], Model_4_evaluation[1]))
# Plotting Train & Validation Loss
fig,ax = plt.subplots(1,1)
ax.set_xlabel('epoch') 
ax.set_ylabel('Categorical Crossentropy Loss')

# list of epoch numbers
x = list(range(1, 11))
vy = Model_4_result.history['val_loss']
ty = Model_4_result.history['loss']

# Calling Function
plt_dynamic(x, vy, ty, ax)
from prettytable import PrettyTable

pretty = PrettyTable()

pretty.field_names = ['Sr.No','MODEL','train_accuracy','test_accuracy','train_loss','test_loss']

pretty.add_row(['1','2-LAYER WITH KERNEL SIZE ','0.1124','0.1134','2.3022','2.3021'])
pretty.add_row(['2','3-LAYER WITH KERNEL SIZE ','0.9736','0.9859','0.0877','0.0425'])
pretty.add_row(['3','5-LAYER WITH KERNEL SIZE ','0.9799','0.3836','0.0717','6.1720'])
pretty.add_row(['4','7-LAYER WITH KERNEL SIZE ','0.9797','0.9844','0.0697','0.0514'])


print(pretty)<jupyter_output>+-------+---------------------------+----------------+---------------+------------+-----------+
| Sr.No |           MODEL           | train_accuracy | test_accuracy | train_loss | test_loss |
+-------+---------------------------+----------------+---------------+------------+-----------+
|   1   | 2-LAYER WITH KERNEL SIZE  |     0.1124     |     0.1134    |   2.3022   |   2.3021  |
|   2   | 3-LAYER WITH KERNEL SIZE  |     0.9736     |     0.9859    |   0.0877   |   0.0425  |
|   3   | 5-LAYER WITH KERNEL SIZE  |     0.9799     |     0.3836    |   0.0717   |   6.1720  |
|   4   | 7-LAYER WITH KERNEL SIZE  |     0.9797     |     0.9844    |   0.0697   |   0.0514  |
+-------+---------------------------+----------------+---------------+------------+-----------+
{'fraction_non_alphanumeric': 0.11229635376260667, 'fraction_numerical': 0.04383242823894492, 'mean_word_length': 4.494139584443261, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '540736', 'license': 'MIT', 'n_tokens_mistral': 7073, 'n_tokens_neox': 6498, 'n_words': 2215, 'path': 'Digit Recognizer using CNN.ipynb', 'repo_name': 'sarthak-kasturiwale/Digit-Recognition-using-MNIST-Data-', 'size': '20624'}