<jupyter_start><jupyter_text>"A Basic Neural Network: Differentiate Hand-Written Digits"- badges: true- author: <NAME>  Key Objectives:- Building a neural network that differentiates two hand-written digits 3 and 8.- Comparing the results of this Neural Network (NN) to that of a Logistic Regression (LR) model. Requirements:- 'Kudzu' : A neural network library that was designed during our course by [Univ.AI](www.univ.ai). - MNIST DatabaseIf MNIST is not installed, use the command `!pip install mnist` given below.It can be run both from the command line and Jupyter Notebook.<jupyter_code>!pip install mnist<jupyter_output>Collecting mnist
  Downloading mnist-0.2.2-py2.py3-none-any.whl (3.5 kB)<jupyter_text>Importing necessary libraries<jupyter_code>%load_ext autoreload
%autoreload 2

%matplotlib inline
import matplotlib.pyplot as plt

import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Preparing the Data<jupyter_code>import mnist
train_images = mnist.train_images()
train_labels = mnist.train_labels()
train_images.shape, train_labels.shape
test_images = mnist.test_images()
test_labels = mnist.test_labels()
test_images.shape, test_labels.shape
image_index = 7776 # You may select anything up to 60,000
print(train_labels[image_index]) 
plt.imshow(train_images[image_index], cmap='Greys')<jupyter_output>2<jupyter_text>Filter data to get 3 and 8 out<jupyter_code>train_filter = np.where((train_labels == 3 ) | (train_labels == 8))
test_filter = np.where((test_labels == 3) | (test_labels == 8))
X_train, y_train = train_images[train_filter], train_labels[train_filter]
X_test, y_test = test_images[test_filter], test_labels[test_filter]<jupyter_output><empty_output><jupyter_text>We normalize the pixel values in the 0 to 1 range<jupyter_code>X_train = X_train/255.
X_test = X_test/255.<jupyter_output><empty_output><jupyter_text>Setup the labels as 1 (when the digit is 3) and 0 (when the digit is 8)<jupyter_code>y_train = 1*(y_train==3)
y_test = 1*(y_test==3)
X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>Reshape the input data to create a linear array<jupyter_code>X_train = X_train.reshape(X_train.shape[0], -1)
X_test = X_test.reshape(X_test.shape[0], -1)
X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>Importing appropriate functions from 'Kudzu'<jupyter_code>from kudzu.layer import Sigmoid
from kudzu.layer import Relu
from kudzu.layer import Affine, Sigmoid

from kudzu.model import Model
from kudzu.train import Learner
from kudzu.optim import GD
from kudzu.data import Data, Dataloader, Sampler

from kudzu.callbacks import AccCallback
from kudzu.callbacks import ClfCallback

from kudzu.loss import MSE<jupyter_output><empty_output><jupyter_text>Let us create a `Config` class, to store important parameters. This class essentially plays the role of a dictionary.<jupyter_code>class Config:
    pass
config = Config()
config.lr = 0.001
config.num_epochs = 251
config.bs = 50<jupyter_output><empty_output><jupyter_text>Initializing data to the variables<jupyter_code>data = Data(X_train, y_train.reshape(-1,1))
sampler = Sampler(data, config.bs, shuffle=True)

dl = Dataloader(data, sampler)

opt = GD(config.lr)
loss = MSE()
training_xdata = X_train
testing_xdata = X_test
training_ydata = y_train.reshape(-1,1)
testing_ydata = y_test.reshape(-1,1)<jupyter_output><empty_output><jupyter_text>Running Models with the Training dataDetails about the network layers:- A first affine layer has 784 inputs and does 100 affine transforms. These are followed by a Relu- A second affine layer has 100 inputs from the 100 activations of the past layer, and does 100 affine transforms. These are followed by a Relu- A third affine layer has 100 activations and does 2 affine transformations to create an embedding for visualization. There is no non-linearity here.- A final "logistic regression" which has an affine transform from 2 inputs to 1 output, which is squeezed through a sigmoid.Help taken from Anshuman's Notebook.<jupyter_code># layers for the Neural Network
layers = [Affine("first", 784, 100), Relu("first"), Affine("second", 100, 100), Relu("second"), Affine("third", 100, 2), Affine("final", 2, 1), Sigmoid("final")]
model_nn = Model(layers)

# layers for the Logistic Regression
layers_lr = [Affine("logits", 784, 1), Sigmoid("sigmoid")]
model_lr = Model(layers_lr)
# suffix _nn stands for Neural Network.
learner_nn = Learner(loss, model_nn, opt, config.num_epochs)
acc_nn = ClfCallback(learner_nn, config.bs, training_xdata , testing_xdata, training_ydata, testing_ydata)
learner_nn.set_callbacks([acc_nn])
print("====== Neural Network ======")
learner_nn.train_loop(dl)<jupyter_output>====== Neural Network ======<jupyter_text>Logistic Regression based Implementation.<jupyter_code>learner_lr = Learner(loss, model_lr, opt, config.num_epochs)
acc_lr = ClfCallback(learner_lr, config.bs, training_xdata , testing_xdata, training_ydata, testing_ydata)
learner_lr.set_callbacks([acc_lr])
print("====== Logistic Regression ======")
learner_lr.train_loop(dl)<jupyter_output>====== Logistic Regression ======
Epoch 0, Loss 0.2603
Training Accuracy: 0.6203, Testing Accuracy: 0.6195<jupyter_text>Comparing results of NN and LR<jupyter_code>plt.figure(figsize=(15,10))

# Neural Network plots
plt.plot(acc_nn.accuracies, 'r-', label = "Training Accuracies - NN")
plt.plot(acc_nn.test_accuracies, 'g-', label = "Testing Accuracies - NN")

# Logistic Regression plots
plt.plot(acc_lr.accuracies, 'k-', label = "Training Accuracies - LR")
plt.plot(acc_lr.test_accuracies, 'b-', label = "Testing Accuracies - LR")

plt.ylim(0.8, 1)

plt.legend()<jupyter_output><empty_output><jupyter_text>From the plot, we can observe the following:- Neural Network achieves higher accuracy than the Logistic Regression model.- This apparently, is because of overfitting, i.e. NN captures more noise than data.- Testing accuracy of NN drops below the Training accuracy at higher epochs. This explains the over-fitting on training data.- Logistic Regression gives a reliable accuracy, without the above mentioned problem.  Moving till the last but one layer (excluding it). Plotting the outputs of this layer of the NN.<jupyter_code>model_new = Model(layers[:-2])
plot_testing = model_new(testing_xdata)
plt.figure(figsize=(8,7))
plt.scatter(plot_testing[:,0], plot_testing[:,1], alpha = 0.1, c = y_test.ravel());
plt.title('Outputs')<jupyter_output><empty_output><jupyter_text>Plotting probability contours<jupyter_code>model_prob = Model(layers[-2:]) 
# Adjust the x and y ranges according to the above generated plot.
x_range = np.linspace(-4, 1, 100) 
y_range = np.linspace(-6, 6, 100) 
x_grid, y_grid = np.meshgrid(x_range, y_range) # x_grid and y_grig are of size 100 X 100

# converting x_grid and y_grid to continuous arrays
x_grid_flat = np.ravel(x_grid)
y_grid_flat = np.ravel(y_grid)

# The last layer of the current model takes two columns as input. Hence transpose of np.vstack() is required.
X = np.vstack((x_grid_flat, y_grid_flat)).T

# x_grid and y_grid are of size 100 x 100
probability_contour = model_prob(X).reshape(100,100) 
plt.figure(figsize=(10,9))
plt.scatter(plot_testing[:,0], plot_testing[:,1], alpha = 0.1, c = y_test.ravel())
contours = plt.contour(x_grid,y_grid,probability_contour)
plt.title('Probability Contours')
plt.clabel(contours, inline = True );<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10834909483923263, 'fraction_numerical': 0.022966765738989463, 'mean_word_length': 6.455186304128902, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '28868', 'license': 'Apache-2.0', 'n_tokens_mistral': 2709, 'n_tokens_neox': 2498, 'n_words': 719, 'path': '_notebooks/2020-08-11-NeuralNetwork.ipynb', 'repo_name': 'saimanvitha02/blog', 'size': '7410'}