<jupyter_start><jupyter_text>MNIST Example for MLP with momentum and batchnormThe purpose of this demo is to help you learn about neural networks and explore how changing the architecture of a neural network impacts the performance of the network.  You are free to change model acrhitecture, add depth or select any other hidden layers sizes.If you have any suggestions or find errors, please, don't be hesitate to text me up @ <EMAIL><jupyter_code>import mlp
import activations as ac
from random import randint
import mnist
import numpy as np
import time
from matplotlib import pyplot as plt
import os
import glob<jupyter_output><empty_output><jupyter_text>Initializing parameters<jupyter_code>np.random.seed(11785)

#initialize neural parameters
learning_rate = 0.004
momentum = 0.996 
num_bn_layers= 1
mini_batch_size = 10
epochs = 40

# random weight init
def weight_init(x, y):
    return np.random.randn(x, y)

# zero bias init
def bias_init(x):
    return np.zeros((1, x))

# initialize training, validation and testing data
train, val, test = mnist.load_mnist()<jupyter_output><empty_output><jupyter_text>Creating multi-layer perceptron class<jupyter_code>net = mlp.MLP(784, 10, [64, 32], [ac.Sigmoid(), ac.Sigmoid(), ac.Sigmoid()], weight_init, bias_init, ac.SoftmaxCrossEntropy(), learning_rate,
          momentum, num_bn_layers)<jupyter_output><empty_output><jupyter_text>Let's view random MNIST dataIf MNIST dataset doesn't exist in "data" folder, it will be downloaded from http://deeplearning.net/data/mnist/mnist.pkl.gz<jupyter_code>m = train[0].shape[0]
for i in range(36):
    ax = plt.subplot(6, 6, i+1)
    ax.axis('off')
    x_draw = train[0][randint(0, m)]
    x_draw = np.reshape(x_draw, (-1, 28))
    
    plt.imshow(x_draw, cmap='Greys')<jupyter_output><empty_output><jupyter_text>Training MLP<jupyter_code>%%time
net.fit(train, val, epochs, mini_batch_size)<jupyter_output>Epoch 1, accuracy 84.78 %.
Epoch 2, accuracy 89.78 %.
Epoch 3, accuracy 91.02 %.
Epoch 4, accuracy 91.74 %.
Epoch 5, accuracy 92.49000000000001 %.
Epoch 6, accuracy 92.78999999999999 %.
Epoch 7, accuracy 92.99 %.
Epoch 8, accuracy 93.2 %.
Epoch 9, accuracy 93.44 %.
Epoch 10, accuracy 93.69 %.
Epoch 11, accuracy 93.89999999999999 %.
Epoch 12, accuracy 93.91000000000001 %.
Epoch 13, accuracy 93.93 %.
Epoch 14, accuracy 94.19 %.
Epoch 15, accuracy 94.32000000000001 %.
Epoch 16, accuracy 94.27 %.
Epoch 17, accuracy 94.31 %.
Epoch 18, accuracy 94.49 %.
Epoch 19, accuracy 94.6 %.
Epoch 20, accuracy 94.65 %.
Epoch 21, accuracy 94.74000000000001 %.
Epoch 22, accuracy 94.67999999999999 %.
Epoch 23, accuracy 94.87 %.
Epoch 24, accuracy 94.93 %.
Epoch 25, accuracy 94.97 %.
Epoch 26, accuracy 94.89999999999999 %.
Epoch 27, accuracy 94.78 %.
Epoch 28, accuracy 94.91000000000001 %.
Epoch 29, accuracy 94.86 %.
Epoch 30, accuracy 94.62 %.
Epoch 31, accuracy 94.58 %.
Epoch 32, accuracy 94.59 %.
Epoch 3[...]<jupyter_text>Training loss<jupyter_code>plt.plot(net.training_loss)
plt.title('training loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['training'], loc='best')
plt.show()<jupyter_output><empty_output><jupyter_text>Validation accuracy curve<jupyter_code>plt.plot(net.validation_acc)
plt.title('validation accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['validation acc'], loc='best')
plt.show()
#save the model
net.save()<jupyter_output><empty_output><jupyter_text>Loading last trained model<jupyter_code>path = os.path.join(os.curdir, 'models/*')

files = sorted(
    glob.iglob(path), key=os.path.getctime, reverse=True) 

#load latest model
print(files[0])
net.load(files[0])<jupyter_output>./models/1564657792.3618639_nn_model.pkl<jupyter_text>Let's check accuracy according to the test samples<jupyter_code>#testing neural network
test_acc = net.validate(test) * 100.0
print("Test Accuracy: " + str(test_acc) + "%")<jupyter_output>Test Accuracy: 94.93%
{'fraction_non_alphanumeric': 0.12365038560411311, 'fraction_numerical': 0.09177377892030848, 'mean_word_length': 5.326829268292683, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 9, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '589538', 'license': 'MIT', 'n_tokens_mistral': 1667, 'n_tokens_neox': 1358, 'n_words': 377, 'path': 'demo.ipynb', 'repo_name': 'alm4z/mlp-mnist', 'size': '3902'}