<jupyter_start><jupyter_text>Neural Network - Street View House Number (SVHN) DescriptionThe goal of this hands-on project is to build a classifier model from scratch to understand the nitty gritties of building and training a model and further to understand the advantages of neural networks. First we will implement a simple KNN classifier and later implement a Neural Network to classify the images in the SVHN dataset. We will compare the computational efficiency and accuracy between the traditional method (k-Nearest Neighbor) and neural networks. Dataset InformationThe **Street View House Number (SVHN)** dataset is a real-world image dataset for developing machine learning and object recognition algorithms with minimal requirement on data formatting but comes from a significantly harder, unsolved, real world problem (recognizing digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images.SVHN identification is a difficult problem. The main challenges are often decomposed into three subproblems: (i) finding the number on the house, (ii) segmenting the individual digits, and (iii) classifying the digits. Each of these subproblems is difficult, for different reasons. Localization is a challenge because numbers are hard to distinguish from other shapes; segmentation is difficult because the digits are often close together (and sometimes even connected); and classification is difficult because there is a massive variety in the typefaces, textures and colours present in house numbers.We'll be working on the 3rd format of the problem which is to classify 32x32 images centered around a single character (many of the images do contain some distractors at the sides).--- Imports and Configurations<jupyter_code># Utilities
import sys
import h5py
import warnings
from time import time

# Numerical calculation
import numpy as np

# Data handling
import pandas as pd

# Data Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Conventional ML Model
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Neural Network
from tensorflow.keras import utils, losses, optimizers, callbacks
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dropout, ReLU, Dense, BatchNormalization
# Configure for any default setting of any library
np.random.seed(5)
warnings.filterwarnings('ignore')
%matplotlib inline
pd.set_option('display.float_format', lambda x: '%.3f' % x)<jupyter_output><empty_output><jupyter_text>Load and Visualize the DataThis notebook is made generic to run in both Google Colab and in local workstation. Load the h5 file and visualize the data in it.<jupyter_code>svhn_file = 'SVHN_single_grey1.h5'
if 'google.colab' in sys.modules:    
    # Google Colab lib
    from google.colab import drive
    # Mount the drive
    drive.mount('/content/drive/')
    # Open the drive file as readonly
    svhn = h5py.File('/content/drive/My Drive/Colab Notebooks/DLCP/' + svhn_file, 'r')
else:
    # Open the local file as readonly
    svhn = h5py.File(svhn_file, 'r')
# Load the training, validation and test sets
X_train_o = svhn['X_train'][:]
X_test_o = svhn['X_test'][:]
X_val_o = svhn['X_val'][:]
y_train_o = svhn['y_train'][:]
y_test_o = svhn['y_test'][:]
y_val_o = svhn['y_val'][:]

print('Shape of different datasets are:')
print('X_train :', X_train_o.shape)
print('X_test  :', X_test_o.shape)
print('X_val   :', X_val_o.shape)
print('y_train :', y_train_o.shape)
print('y_test  :', y_test_o.shape)
print('y_val   :', y_val_o.shape)
# visualizing the first 15 images in the dataset and their labels
print('Labels corresponding to below images: %s' % (y_train_o[0:15]))
plt.figure(figsize=(15, 1))
for i in range(15):
    plt.subplot(1, 15, i+1)
    plt.imshow(X_train_o[i], cmap="gray")
    plt.axis('off')<jupyter_output>Labels corresponding to below images: [2 6 7 4 4 0 3 0 7 3 1 0 1 3 1]<jupyter_text>Preprocess the Image The images are gray-scaled images with 32x32 pixels each of 1 byte in size. So the values containing in each pixel ranges from $0$ to $2^8-1 = 255$. In order to feed them into Neural Network we must convert the input matrix to 1-D vector. Hence as part of this preprocessing step, we will do the following- Flatteing- Normalization<jupyter_code># Flatten the X_* datasets
X_train_f = X_train_o.reshape(X_train_o.shape[0], X_train_o.shape[1] * X_train_o.shape[2]).astype('float32')
X_test_f = X_test_o.reshape(X_test_o.shape[0], X_test_o.shape[1] * X_test_o.shape[2]).astype('float32')
X_val_f = X_val_o.reshape(X_val_o.shape[0], X_val_o.shape[1] * X_val_o.shape[2]).astype('float32')

print('Shape after flattening:')
print('X_train :', X_train_f.shape)
print('X_test  :', X_test_f.shape)
print('X_val   :', X_val_f.shape)
# Normalize the inputs from 0-255 to 0-1
X_train = X_train_f / 255.0
X_test = X_test_f / 255.0
X_val = X_val_f / 255.0

print(X_train)<jupyter_output>[[0.12968785 0.11866706 0.10530196 ... 0.19477727 0.19942354 0.20799099]
 [0.34101608 0.3414451  0.34656274 ... 0.29498273 0.30054745 0.31092745]
 [0.48284313 0.49356118 0.47872823 ... 0.36837214 0.41093138 0.44637334]
 ...
 [0.6042902  0.61100274 0.6149663  ... 0.52993256 0.53592    0.5398412 ]
 [0.8137075  0.8176286  0.8353796  ... 0.83974785 0.8319055  0.820142  ]
 [0.37285647 0.37677765 0.39246237 ... 0.5414914  0.5509529  0.5509529 ]]<jupyter_text>Understand the target classesLet's understand diversity, datatypes and distribution of the target classes.<jupyter_code># Get unique values in target sets
np.unique(y_train_o)
np.unique(y_test_o)
np.unique(y_val_o)<jupyter_output><empty_output><jupyter_text>All datasets contain unsigned int8 type target class values 0-9. Let's convert them into categorical and check their distribution.<jupyter_code>fig, axes = plt.subplots(1, 3, figsize=(16,5))
sns.distplot(y_train_o, ax=axes[0], axlabel='Training Set')
sns.distplot(y_test_o, ax=axes[1], axlabel='Test Set')
sns.distplot(y_val_o, ax=axes[2], axlabel='Validation Set')
# Apply One-Hot-Encoding
y_train = utils.to_categorical(y_train_o)
y_test = utils.to_categorical(y_test_o)
y_val = utils.to_categorical(y_val_o)

print('Shape of different target sets are:')
print('y_train :', y_train.shape)
print('y_test  :', y_test.shape)
print('y_val   :', y_val.shape)<jupyter_output>Shape of different target sets are:
y_train : (42000, 10)
y_test  : (18000, 10)
y_val   : (60000, 10)<jupyter_text>k-Nearest Neighbor ModelWe will now consider a subset of the data for KNN classification. K Nearest neighbours classification is a traditional ML model building. We will analyze it is pertinence in the context of OCR.<jupyter_code># Take a subset of the original dataset to estimate for the best k
X_train_sub = X_train[:3000]
y_train_sub = y_train_o[:3000]
X_test_sub = X_test[:1000]
y_test_sub = y_test_o[:1000]
# Find accuracy scores for 30 iterations
st = time()
acc_scores = []
for k in range(1,31):
    knn = KNeighborsClassifier(n_neighbors=k , weights='uniform', metric='euclidean')
    knn.fit(X_train_sub, y_train_sub)
    acc_scores.append(accuracy_score(y_test_sub, knn.predict(X_test_sub)) * 100)
    
print('Time taken:', time()-st)
print('Scores out of 30 iterations:', np.round(acc_scores, 1))
# Plot the accuracy scores
plt.figure(figsize=(15,5))
plt.plot(range(1,31), acc_scores , '-go')
_ = plt.xticks(range(1,31))
plt.axvline(np.argmax(acc_scores) + 1)<jupyter_output><empty_output><jupyter_text>The accuracy score seems to be the best at 35.8% for k value of 16. Let's set the k value to 16 and train a model with the entire training dataset.<jupyter_code># Create the classifier model using the best k found among 30 iterations
st = time()
clf = KNeighborsClassifier(n_neighbors= np.argmax(acc_scores) + 1)
clf.fit(X_train, y_train_o)

predictions = clf.predict(X_test)
print('Time taken:', time()-st)
# Classification Report
print("EVALUATION ON TESTING DATA")
print(classification_report(y_test_o, predictions))
# Visualize the Confusion Matrix
plt.figure(figsize=(12,6))
sns.heatmap(confusion_matrix(y_test_o, predictions), annot=True, cmap="PRGn")<jupyter_output><empty_output><jupyter_text>**Observations:**- Accuracy of 53% shows KNN performed very poorly.- The process with all 42000 images in the training data and 18000 images in test data took more than 30 minutes to execute. This is one of the obvious reasons the traditional classification models are not used for OCR. Let's build a **Deep Neural Network** classifier and compare the performance. Neural Network ModelA Neural Network, unlike traditional ML algorithms, is a series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates. Neural networks can adapt to changing input; so the network generates the best possible result without needing to redesign the output criteria.Let's define a generic method to architect the network based on a layers array.<jupyter_code># Generic method to Build the Neural Network
def NNModel(layers):
    # Create model
    network = Sequential()
    # Add the layers
    if layers:
        for layer in layers:
            network.add(layer['layer_type'](**layer['params']))
    return network<jupyter_output><empty_output><jupyter_text>Initialize Tensorboard callback for visualization of training<jupyter_code># Setting the directory to store the log file
# tb_callback = callbacks.TensorBoard(log_dir='.logs/{}'.format(time()), 
#                                     histogram_freq=0, 
#                                     write_graph=True, 
#                                     write_images=True)<jupyter_output><empty_output><jupyter_text>Initialize Early Stopping and Model Checkpoint callbacks<jupyter_code># Stop if the val_loss is not changing even 0.001 for more than 10 continious epochs
es_callback = callbacks.EarlyStopping(monitor='val_loss', 
                                      min_delta=0.001, 
                                      patience=10)

# Saving the best weights - Saves the weights when the val_loss achieves a new low value
# mc_callback = callbacks.ModelCheckpoint('SVHN_checkpoint_{epoch:02d}_loss{val_loss:.4f}.h5',
#                                         monitor='val_loss',
#                                         verbose=1,
#                                         save_best_only=True,
#                                         mode='auto')<jupyter_output><empty_output><jupyter_text>Iteration - 1Consists of Layers:- Dense -> 256/ReLU- Dense -> 64/ReLU- Dense -> 32/ReLU- Dense -> 10/Softmax (Output)Learning Rate: 0.005Momentum: 0.02Epochs: 30Batch Size: 200Optimizer: SGDLoss: Categorical CrossentropyDefine the Layers<jupyter_code># Define the layers for Iteration - 1
layers = [
    {
        'layer_type': Dense,
        'params': {
            'units': 256,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 64,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 64,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': len(np.unique(y_train_o)),
            'activation': 'softmax'
        }
    }
]

# Build network architecture
model1 = NNModel(layers)<jupyter_output>WARNING:tensorflow:From C:\Users\aksha\Anaconda3\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor<jupyter_text>Set Optimizer and Loss function for training<jupyter_code>learning_rate = 5e-3
momentum = 2e-2

# Optimizer = SGD, Loss function = Categorical cross entropy
model1.compile(optimizer = optimizers.SGD(learning_rate, momentum), 
              loss = losses.categorical_crossentropy, 
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Train the network<jupyter_code>epochs = 30
batch_size = 200

# Start training the network
st = time()
model1.fit(X_train, y_train, 
          validation_data=(X_val, y_val), 
          epochs=epochs, 
          batch_size=batch_size, 
          verbose=2,
          callbacks=[es_callback])
print('\nTotal time taken:', time()-st)
# Print the Summary of the Network
model1.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                multiple                  262400    
_________________________________________________________________
dense_1 (Dense)              multiple                  16448     
_________________________________________________________________
dense_2 (Dense)              multiple                  4160      
_________________________________________________________________
dense_3 (Dense)              multiple                  650       
=================================================================
Total params: 283,658
Trainable params: 283,658
Non-trainable params: 0
_________________________________________________________________<jupyter_text>**Observations:**- This is a 4 layer neural network without any regularization techniques applied.- Though the score is better than traditional ML algorithm (k-NN), it is still **not an acceptable performance**.- The network **needs to be optimized** in order to better learn the patterns in the dataset.- So let's add Batch Norm & Dropout layers and use Adam optimizer in the next iteration. Iteration - 2Consists of layers:- BatchNormalization -> (1024,)- Dropout -> 0.25- Dense -> 256/ReLU- BatchNormalization -> (256,)- Dense -> 64/ReLU- Dropout -> 0.2- Dense -> 32/ReLU- Dense -> 10/Softmax (Output)Learning Rate: 0.001Epochs: 30Batch Size: 150Optimizer: AdamLoss: Categorical CrossentropyDefine the Layers<jupyter_code># Define the layers for Iteration - 2
layers = [
    {
        'layer_type': BatchNormalization,
        'params': {
            'input_shape': (1024,)
        }
    },{
        'layer_type': Dropout,
        'params': {
            'rate': 0.25
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 256,
            'activation': 'relu'
        }
    },{
        'layer_type': BatchNormalization,
        'params': {
            'input_shape': (256,)
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 64,
            'activation': 'relu'
        }
    },{
        'layer_type': Dropout,
        'params': {
            'rate': 0.2
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 32,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': len(np.unique(y_train_o)),
            'activation': 'softmax'
        }
    }
]

# Build network architecture
model2 = NNModel(layers)<jupyter_output><empty_output><jupyter_text>Set Optimizer and Loss function for training<jupyter_code>learning_rate = 1e-3

# Optimizer = Adam, Loss function = Categorical cross entropy
model2.compile(optimizer = optimizers.Adam(lr=learning_rate),
              loss = losses.categorical_crossentropy, 
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Train the network<jupyter_code>epochs = 30
batch_size = 150

# Start training the network
st = time()
model2.fit(X_train, y_train, 
          validation_data=(X_val, y_val),
          epochs=epochs, 
          batch_size=batch_size, 
          verbose=2,
          callbacks=[es_callback])
print('\nTotal time taken:', time()-st)
# Print the Summary of the Network
model2.summary()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization (BatchNo (None, 1024)              4096      
_________________________________________________________________
dropout (Dropout)            (None, 1024)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 256)               262400    
_________________________________________________________________
batch_normalization_1 (Batch (None, 256)               1024      
_________________________________________________________________
dense_5 (Dense)              (None, 64)                16448     
_________________________________________________________________
dropout_1 (Dropout)          (None, 64)                0         
______________________________________________________[...]<jupyter_text>**Observations:**- Adding the batch norm layer and dropout layer increased the performance significantly.- However, the difference in the training accuracy and validation accuracy indicates the network is somewhat overfit.- Hence, let's drop 1 batch norm and 1 dropout layer in next iteration.- Also increase the epoch size to 50 in next iteration and see how it performs. Iteration - 3Consists of layers:- BatchNormalization -> (1024,)- Dense -> 256/ReLU- Dense -> 64/ReLU- Dropout -> 0.2- Dense -> 32/ReLU- Dense -> 10/Softmax (Output)Learning Rate: 0.001Epochs: 50Batch Size: 150Optimizer: AdamLoss: Categorical CrossentropyDefine the Layers<jupyter_code># Define the layers for Iteration - 3
layers = [
    {
        'layer_type': BatchNormalization,
        'params': {
            'input_shape': (1024,)
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 256,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 64,
            'activation': 'relu'
        }
    },{
        'layer_type': Dropout,
        'params': {
            'rate': 0.2
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': 32,
            'activation': 'relu'
        }
    },{
        'layer_type': Dense,
        'params': {
            'units': len(np.unique(y_train_o)),
            'activation': 'softmax'
        }
    }
]

# Build network architecture
model3 = NNModel(layers)<jupyter_output><empty_output><jupyter_text>Set Optimizer and Loss function for training<jupyter_code>learning_rate = 1e-3

# Optimizer = Adam, Loss function = Categorical cross entropy
model3.compile(optimizer = optimizers.Adam(lr=learning_rate), 
              loss = losses.categorical_crossentropy, 
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Train the network<jupyter_code>epochs = 50
batch_size = 150
# Start training the network
st = time()
model3.fit(X_train, y_train, 
          validation_data=(X_val, y_val), 
          epochs=epochs, 
          batch_size=batch_size, 
          verbose=2,
          callbacks=[es_callback])
print('\nTotal time taken:', time()-st)
# Print the Summary of the Network
model3.summary()<jupyter_output>Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
batch_normalization_2 (Batch (None, 1024)              4096      
_________________________________________________________________
dense_8 (Dense)              (None, 256)               262400    
_________________________________________________________________
dense_9 (Dense)              (None, 64)                16448     
_________________________________________________________________
dropout_2 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_10 (Dense)             (None, 32)                2080      
_________________________________________________________________
dense_11 (Dense)             (None, 10)                330       
======================================================[...]<jupyter_text>**Observations:**- This network consists of 4 dense layers along with one batch norm layer and one dropout layer.- The performance of this network is also very good with 90% train accuracy and 91% of validation accuracy.It is noticed that the network in iteration 3 performed very well. Hence we will consider this model3 to evaluate the SVHN dataset. Evaluate the Neural NetworkEvaluate trained model on test data<jupyter_code># Testing the model on test set
predictions = model3.predict_classes(X_test)
score = model3.evaluate(X_test, y_test)
print('Test loss      :', score[0])
print('Test accuracy  :', score[1])
# Classification Report
print("EVALUATION ON TESTING DATA")
print(classification_report(y_test_o, predictions))
# Visualize the Confusion Matrix
plt.figure(figsize=(12,6))
sns.heatmap(confusion_matrix(y_test_o, predictions), annot=True, cmap="gist_heat_r")<jupyter_output><empty_output><jupyter_text>Visualizing some predictions<jupyter_code># visualizing the first 15 images in the dataset and their labels
print('Actual labels corresponding to below images: %s' % (y_test_o[0:15]))
plt.figure(figsize=(15, 1))
for i in range(15):
    plt.subplot(1, 15, i+1)
    plt.imshow(X_test_o[i], cmap="gray")
    plt.axis('off')
    
print('Predicted labels corresponding to below images: %s' % (predictions[0:15]))<jupyter_output>Actual labels corresponding to below images: [1 7 2 9 0 9 1 8 4 4 8 9 1 4 3]
Predicted labels corresponding to below images: [0 7 2 9 0 9 1 8 1 4 8 5 1 4 3]<jupyter_text>Saving the Model<jupyter_code># Set the paths to store the model and weights
model3.save('SVHN_DeepNN_model.h5')
model3.save_weights('SVHN_DeepNN_weights.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10691511012618155, 'fraction_numerical': 0.03925647867577224, 'mean_word_length': 3.5827979274611397, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 99, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '448021', 'license': 'Apache-2.0', 'n_tokens_mistral': 7134, 'n_tokens_neox': 6350, 'n_words': 1943, 'path': 'CaseStudies/10-DeepLearning.NeuralNetwork-SVHN/SVHN_Deep.Neural.Network.ipynb', 'repo_name': 'averma111/AIML-PGP', 'size': '22111'}