<jupyter_start><jupyter_text>Machine Learning Engineer Nanodegree   Capstone Project: Building a Handwritten Digit Recognizer  IntroductionIn this project, we are going to build a Deep Neural Network(DNN) that can recognize the digit handwritten in an image. The DNN is trained over MNIST dataset containing handwritten digits and their respective numerical digit.   Requirements- [Anaconda2](https://www.continuum.io/downloads)- [Tensorflow](https://www.tensorflow.org/) or [Theano](http://www.deeplearning.net/software/theano/)- [Keras](https://keras.io/)- [OpenCV](http://opencv.org/)- (Optional) A GPU with Cuda support is highly recommended for fast computation. Install respective graphics drivers, cuda toolkit, cudnn, [tensorflow-gpu](https://www.tensorflow.org/get_started/os_setup) version  Data Exploration Run the below code to import the required python libraries and initialize variables<jupyter_code>from __future__ import print_function
from numpy import random
from keras.datasets import mnist # helps in loading the MNIST dataset
from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras import backend as K
from keras.models import load_model
from keras.utils.np_utils import probas_to_classes
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import time
import cv2
#to plot inside the notebook itself
%matplotlib inline 

# to be able to reproduce the same randomness
random.seed(42)  

# No of rows and columns in the image
img_rows = 28
img_cols = 28

#No of output classes (0-9)
nb_classes = 10<jupyter_output>Using TensorFlow backend.<jupyter_text>Load the DatasetKeras provides api to download and load the mnist dataset in a single line. Run the below line to download and load the dataset as training and testing set.<jupyter_code>(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Show the results of the split
print("Training set has {} samples.".format(X_train.shape[0]))
print("Testing set has {} samples.".format(X_test.shape[0]))
print("\n")
# Show the number of rows and columns 
print("Row pixels in each image : {}.".format(X_train.shape[1]))
print("Column pixels in each image : {}.".format(X_train.shape[2]))
print("\n")
print("Successfully Downloaded and Loaded the dataset")<jupyter_output>Training set has 60000 samples.
Testing set has 10000 samples.


Row pixels in each image : 28.
Column pixels in each image : 28.


Successfully Downloaded and Loaded the dataset<jupyter_text>Visualize an Image sampleLet's plot the first image sample in the training set using matplotlib.<jupyter_code># Show the handwritten image
plt.imshow(X_train[0], cmap=cm.binary)<jupyter_output><empty_output><jupyter_text>Preprocess the Data Keras can use either tensorflow or theano. While they both represent image in different format. Theano uses the format (no. of color channels, no. of rows, no. of cols) and tensorflow uses (no. of rows, no. of cols, no. of color channels). Thus a wrapper is needed when switching backends. Run the below code to ensure that the image is in the correct ordering for the respective background.<jupyter_code>if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)
print("The data is reshaped to the respective format!" , input_shape)<jupyter_output>The data is reshaped to the respective format! (28, 28, 1)<jupyter_text>The MNIST dataset contains grayscale images where the color channel value varies from 0 to 255. In order to reduce the computational load and training difficulty, we will map the values from 0 - 255 to 0 - 1 by dividing each pixel values by 255. Run the below code to do this.<jupyter_code>X_train = X_train.astype('float32') #converted to float so that it can hold floating values between 0-1
X_test = X_test.astype('float32') #converted to float so that it can hold floating values between 0-1
X_train /= 255
X_test /= 255<jupyter_output><empty_output><jupyter_text>The target labels y_train,y_test are in the form of numerical integers(0-9), we need to convert them to binary form in order for the neural network to perform mapping from input to output correctly and efficiently. Run the below code to do this.<jupyter_code>print("In Integer form : ", y_train,y_test)

Y_train = np_utils.to_categorical(y_train, nb_classes) #converted to their binary forms
Y_test = np_utils.to_categorical(y_test, nb_classes) #converted to their binary forms

print("In Binary form : ", Y_train,Y_test)
print("Preprocessing of Data is Done Successfully...")<jupyter_output>Preprocessing of Data is Done Successfully...<jupyter_text>Implementing the Deep Neural Network (DNN)  Build the ModelRun the code below to build the Deep Neural Network<jupyter_code>pool_size = (2, 2)
kernel_size = (3, 3)

model = Sequential()

model.add(Convolution2D(32, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

print("Successfully built the DNN Model!")<jupyter_output>Successfully built the DNN Model!<jupyter_text>Compile the ModelThe model is compiled with 'categorical_crossentropy' as loss funtion, 'adadelta' as optimizer and 'accuracy' as metrics.<jupyter_code>model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy']
             )
print("Model Compilation completed!")<jupyter_output>Model Compilation completed!<jupyter_text>Visualize the Model<jupyter_code>from IPython.display import SVG
from keras.utils.visualize_util import model_to_dot

SVG(model_to_dot(model).create(prog='dot', format='svg'))<jupyter_output>Warning: Could not load "/home/joel/anaconda2/lib/graphviz/libgvplugin_pango.so.6" - file not found<jupyter_text>Train the DNN ModelNow let's the train the DNN Model built above with mnist training dataset(X_train,Y_train).<jupyter_code>batch_size = 128
nb_epoch=10 

start = time.time()
model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=1,validation_data=(X_test, Y_test))
done = time.time()

elapsed = (done - start)/60

print("Model trained Successfully : Took - {} mins!".format(elapsed))<jupyter_output>Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 21s - loss: 0.8680 - acc: 0.7994 - val_loss: 0.3598 - val_acc: 0.9004
Epoch 2/10
60000/60000 [==============================] - 21s - loss: 0.3307 - acc: 0.9053 - val_loss: 0.2823 - val_acc: 0.9215
Epoch 3/10
60000/60000 [==============================] - 21s - loss: 0.2763 - acc: 0.9201 - val_loss: 0.2403 - val_acc: 0.9320
Epoch 4/10
60000/60000 [==============================] - 21s - loss: 0.2447 - acc: 0.9296 - val_loss: 0.2200 - val_acc: 0.9362
Epoch 5/10
60000/60000 [==============================] - 21s - loss: 0.2202 - acc: 0.9358 - val_loss: 0.1957 - val_acc: 0.9431
Epoch 6/10
60000/60000 [==============================] - 21s - loss: 0.2005 - acc: 0.9417 - val_loss: 0.1801 - val_acc: 0.9474
Epoch 7/10
60000/60000 [==============================] - 21s - loss: 0.1840 - acc: 0.9464 - val_loss: 0.1654 - val_acc: 0.9512
Epoch 8/10
60000/60000 [==============================[...]<jupyter_text>In the neural network terminology:- **one epoch** - one forward pass and one backward pass of all the training examples- **batch size** - the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.Reference : https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network/164875164875  Evaluate the DNN ModelNow let's the test the DNN Model built above with mnist test dataset(X_test,Y_test).<jupyter_code>score = model.evaluate(X_test, Y_test, verbose=0)
print('Test Loss Value:', score[0])
print('Test Accuracy Value:', score[1])<jupyter_output>Test Loss Value: 0.139938012361
Test Accuracy Value: 0.9582<jupyter_text>Refinement of the Deep Neural Network(DNN)  Build a Refined ModelLet's refine the above model by making it much deeper by adding extra convolutional, dense, activation dropout layer and changing the optimizer to adam.<jupyter_code>pool_size = (2, 2)
kernel_size = (3, 3)

rmodel = Sequential()

rmodel.add(Convolution2D(32, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
rmodel.add(Activation('relu'))
rmodel.add(Convolution2D(64, kernel_size[0], kernel_size[1]))
rmodel.add(Activation('relu'))
rmodel.add(MaxPooling2D(pool_size=pool_size))
rmodel.add(Dropout(0.25))

rmodel.add(Flatten())
rmodel.add(Dense(128))
rmodel.add(Activation('relu'))
rmodel.add(Dropout(0.5))
rmodel.add(Dense(nb_classes))
rmodel.add(Activation('softmax'))

print("Successfully built the Refined DNN Model!")<jupyter_output>Successfully built the Refined DNN Model!<jupyter_text>Compile the Refined ModelThe refined model is compiled with 'categorical_crossentropy' as loss funtion, 'adam' as optimizer and 'accuracy' as metrics.<jupyter_code>rmodel.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
print("Refined Model Compilation completed!")<jupyter_output>Refined Model Compilation completed!<jupyter_text>Visualize the Refined Model<jupyter_code>from IPython.display import SVG
from keras.utils.visualize_util import model_to_dot

SVG(model_to_dot(rmodel).create(prog='dot', format='svg'))<jupyter_output>Warning: Could not load "/home/joel/anaconda2/lib/graphviz/libgvplugin_pango.so.6" - file not found<jupyter_text>Train the Refined DNN ModelNow let's the train the DNN Model built above with mnist training dataset(X_train,Y_train).<jupyter_code>batch_size = 128
nb_epoch=10 

start = time.time()
rmodel.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=1, validation_data=(X_test, Y_test))
done = time.time()

elapsed = (done - start)/60

print("Refined Model trained Successfully : Took - {} mins!".format(elapsed))<jupyter_output>Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 83s - loss: 0.2688 - acc: 0.9181 - val_loss: 0.0579 - val_acc: 0.9823
Epoch 2/10
60000/60000 [==============================] - 75s - loss: 0.0890 - acc: 0.9728 - val_loss: 0.0374 - val_acc: 0.9868
Epoch 3/10
60000/60000 [==============================] - 89s - loss: 0.0681 - acc: 0.9794 - val_loss: 0.0344 - val_acc: 0.9897
Epoch 4/10
60000/60000 [==============================] - 88s - loss: 0.0567 - acc: 0.9826 - val_loss: 0.0281 - val_acc: 0.9906
Epoch 5/10
60000/60000 [==============================] - 88s - loss: 0.0473 - acc: 0.9851 - val_loss: 0.0310 - val_acc: 0.9899
Epoch 6/10
60000/60000 [==============================] - 87s - loss: 0.0401 - acc: 0.9874 - val_loss: 0.0255 - val_acc: 0.9919
Epoch 7/10
60000/60000 [==============================] - 89s - loss: 0.0373 - acc: 0.9880 - val_loss: 0.0310 - val_acc: 0.9906
Epoch 8/10
60000/60000 [==============================[...]<jupyter_text>Evaluate the Refined DNN ModelNow let's the test the DNN Model built above with mnist test dataset(X_test,Y_test).<jupyter_code>score = rmodel.evaluate(X_test, Y_test, verbose=0)
print('Test Loss Value:', score[0])
print('Test Accuracy Value:', score[1])<jupyter_output>Test Loss Value: 0.0373758753725
Test Accuracy Value: 0.9906<jupyter_text>Test the Refined Model with Real DataLets load images of an handwritten digit and convert it to the required format and then predict what digit is written in it.   Load Images from FolderUsing opencv library, let's load images in a folder. Make sure to have the digit written in a white background(not transparent) similiar to the mnist dataset.**Hint** If having images with transparent background, you can convert it into white background using the command(Ubuntu) *$convert -flatten input.png output.png*<jupyter_code>import os, os.path

imgs = []
path = "/home/joel/PROJECTS/Udacity-MLND/Udacity-MLND-Capstone-Handwritting-Digit-Recognition/images"
count=0
for f in os.listdir(path):
    imgs.append(cv2.imread(os.path.join(path,f)))
    count+=1
print("Successfully loaded {} images".format(count))<jupyter_output>Successfully loaded 11 images<jupyter_text>Preprocess the Loaded ImageUsing numpy, opencv let's preprocess the image into the format required by the model to predict<jupyter_code>X_pred = []

for img in imgs:
    # Convert the color image to rgb
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    # Invert black and white color(since in opencv white is considered 255 and black 0 but we need vice versa in order to match with the dataset)
    invert_gray = (255-gray)
    # Resize the image to 28,28 pixels as per the mnist dataset format
    resized = cv2.resize(invert_gray, (28, 28))

    # Convert the image format from (28,28) to (28,28,1) in order for the model to recognize
    resized = np.asarray(resized)
    resized.shape+=1,
    
    #scale the color channel from 0-255 to 0-1
    resized/=255
    
    X_pred.append(resized)
    
X_pred = np.asarray(X_pred)
print(X_pred.shape)<jupyter_output>(11, 28, 28, 1)<jupyter_text>Predict the digit in the ImagesLet's predict the number in the images using the refined model<jupyter_code># Predict the output
proba = rmodel.predict(X_pred)
# Convert the predicted output to respective integer number
answers = probas_to_classes(proba)

#plot the image and the predicted number
i=0
for img in imgs:
    plt.figure()
    plt.imshow(img, cmap=cm.binary)
    plt.suptitle("The predicted digit is : " + str(answers[i]))
    i+=1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.133094535749403, 'fraction_numerical': 0.05492344430397528, 'mean_word_length': 5.0232656514382406, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '591998', 'license': 'MIT', 'n_tokens_mistral': 5236, 'n_tokens_neox': 4627, 'n_words': 1426, 'path': 'handwritten_digit_recognizer.ipynb', 'repo_name': 'joels94/Udacity-MLND-Capstone-Handwritting-Digit-Recognition', 'size': '14238'}