<jupyter_start><jupyter_text>Deep Learning using Rectified Linear Units=== OverviewIn this notebook, we explore the performance of a neural net with varying activation functions on an image classification task. We load our dependencies.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import tensorflow as tf

from neural_net import NeuralNet, loss_fn, train_step, train<jupyter_output><empty_output><jupyter_text>We set up the GPU memory growth.<jupyter_code>tf.config.experimental.set_memory_growth(tf.config.experimental.list_physical_devices('GPU')[0], True)<jupyter_output><empty_output><jupyter_text>We set the random seeds for reproducibility.<jupyter_code>np.random.seed(42)
tf.random.set_seed(42)<jupyter_output><empty_output><jupyter_text>We set the batch size, the number of epochs, and the number of units per layer.<jupyter_code>batch_size = 256
epochs = 300
neurons = [512, 512]<jupyter_output><empty_output><jupyter_text>Data Preparation We load the MNIST dataset.<jupyter_code>(train_features, train_labels), (test_features, test_labels) = tf.keras.datasets.mnist.load_data()<jupyter_output><empty_output><jupyter_text>We scale the images.<jupyter_code>train_features = train_features.astype('float32').reshape(-1, 784) / 255.
test_features = test_features.astype('float32').reshape(-1, 784) / 255.<jupyter_output><empty_output><jupyter_text>We one-hot encode labels.<jupyter_code>train_labels = tf.one_hot(train_labels, 10)
test_labels = tf.one_hot(test_labels, 10)<jupyter_output><empty_output><jupyter_text>We create a `tf.data.Dataset` object for the training dataset.<jupyter_code>dataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels))
dataset = dataset.prefetch(4096).shuffle(4096).batch(batch_size, True)<jupyter_output><empty_output><jupyter_text>Model We use a `NeuralNet` class that consists of two hidden layers.  Logistic-based ModelWe define a 2-layer NN with Logistic activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.sigmoid,
    initializer='glorot_uniform',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>logistic_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 255.48536682128906, mean accuracy = 0.7616519927978516
epoch 100/300 : mean loss = 133.155029296875, mean accuracy = 0.8528144955635071
epoch 150/300 : mean loss = 102.96658325195312, mean accuracy = 0.8795906901359558
epoch 200/300 : mean loss = 90.62866973876953, mean accuracy = 0.890808641910553
epoch 250/300 : mean loss = 83.97627258300781, mean accuracy = 0.8976863026618958
epoch 300/300 : mean loss = 79.65133666992188, mean accuracy = 0.9025607705116272<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9057000279426575<jupyter_text>TanH-based ModelWe define a 2-layer NN with Hyperbolic Tangent activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.tanh,
    initializer='glorot_uniform',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>tanh_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 72.74628448486328, mean accuracy = 0.9097222089767456
epoch 100/300 : mean loss = 60.41146469116211, mean accuracy = 0.9266993999481201
epoch 150/300 : mean loss = 51.40932846069336, mean accuracy = 0.9381677508354187
epoch 200/300 : mean loss = 43.79719161987305, mean accuracy = 0.9476996660232544
epoch 250/300 : mean loss = 37.72518539428711, mean accuracy = 0.954543948173523
epoch 300/300 : mean loss = 32.769771575927734, mean accuracy = 0.9600861668586731<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9578999876976013<jupyter_text>ReLU-based ModelWe define a 2-layer NN with ReLU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.relu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>relu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 59.68490982055664, mean accuracy = 0.9267327785491943
epoch 100/300 : mean loss = 40.60714340209961, mean accuracy = 0.9511218070983887
epoch 150/300 : mean loss = 30.836206436157227, mean accuracy = 0.9627403616905212
epoch 200/300 : mean loss = 24.881916046142578, mean accuracy = 0.9702523946762085
epoch 250/300 : mean loss = 20.494564056396484, mean accuracy = 0.9753772616386414
epoch 300/300 : mean loss = 17.315397262573242, mean accuracy = 0.9797676205635071<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9735999703407288<jupyter_text>Leaky ReLU-based ModelWe define a 2-layer NN with Leaky ReLU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.leaky_relu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>lrelu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 62.70967483520508, mean accuracy = 0.9226095080375671
epoch 100/300 : mean loss = 44.980323791503906, mean accuracy = 0.945846676826477
epoch 150/300 : mean loss = 35.20400619506836, mean accuracy = 0.9576489329338074
epoch 200/300 : mean loss = 28.809045791625977, mean accuracy = 0.9662293195724487
epoch 250/300 : mean loss = 24.211057662963867, mean accuracy = 0.9711705446243286
epoch 300/300 : mean loss = 20.69866180419922, mean accuracy = 0.9753438830375671<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9697999954223633<jupyter_text>Softplus-based ModelWe define a 2-layer NN with Softplus activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.softplus,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>softplus_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 77.94357299804688, mean accuracy = 0.9036625027656555
epoch 100/300 : mean loss = 67.4006118774414, mean accuracy = 0.9175513982772827
epoch 150/300 : mean loss = 61.62861251831055, mean accuracy = 0.9262486696243286
epoch 200/300 : mean loss = 56.37195587158203, mean accuracy = 0.93287593126297
epoch 250/300 : mean loss = 51.185768127441406, mean accuracy = 0.9392527937889099
epoch 300/300 : mean loss = 46.235389709472656, mean accuracy = 0.9446447491645813<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9427000284194946<jupyter_text>ELU-based ModelWe define a 2-layer NN with ELU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.elu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>elu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 66.81639862060547, mean accuracy = 0.9182191491127014
epoch 100/300 : mean loss = 51.78060531616211, mean accuracy = 0.9379173517227173
epoch 150/300 : mean loss = 41.613834381103516, mean accuracy = 0.9501702785491943
epoch 200/300 : mean loss = 34.51963806152344, mean accuracy = 0.9582999348640442
epoch 250/300 : mean loss = 29.249319076538086, mean accuracy = 0.9649605751037598
epoch 300/300 : mean loss = 25.471235275268555, mean accuracy = 0.9700520634651184<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.964900016784668<jupyter_text>Training PerformanceWe lay down the training performance of each model.<jupyter_code>sns.set_style('dark', {'grid.linestyle': '--'})
plt.figure(figsize=(16, 8))
plt.rcParams.update({'font.size': 14})

plt.subplot(121)
plt.plot(range(len(logistic_performance[0])), logistic_performance[0], label='logistic')
plt.plot(range(len(tanh_performance[0])), tanh_performance[0], label='tanh')
plt.plot(range(len(relu_performance[0])), relu_performance[0], label='relu')
plt.plot(range(len(lrelu_performance[0])), lrelu_performance[0], label='leaky_relu')
plt.plot(range(len(softplus_performance[0])), softplus_performance[0], label='softplus')
plt.plot(range(len(elu_performance[0])), elu_performance[0], label='elu')
plt.xlabel('epochs')
plt.ylabel('train accuracy')
plt.legend(loc='lower right')
plt.grid()

plt.subplot(122)
plt.plot(range(len(logistic_performance[1])), logistic_performance[1], label='logistic')
plt.plot(range(len(tanh_performance[1])), tanh_performance[1], label='tanh')
plt.plot(range(len(relu_performance[1])), relu_performance[1], label='relu')
plt.plot(range(len(lrelu_performance[1])), lrelu_performance[1], label='leaky_relu')
plt.plot(range(len(softplus_performance[1])), softplus_performance[1], label='softplus')
plt.plot(range(len(elu_performance[1])), elu_performance[1], label='elu')
plt.xlabel('epochs')
plt.ylabel('train loss')
plt.legend(loc='upper right')
plt.grid()

plt.suptitle('Neural Nets on MNIST with mini-batch size of 256')
plt.savefig('mnist_experiments_256.png', dpi=300)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11478599221789883, 'fraction_numerical': 0.13923194045000845, 'mean_word_length': 7.674247982391782, 'pattern_counts': {'":': 0, '<': 145, '<?xml version=': 0, '>': 145, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '391614', 'license': 'Apache-2.0', 'n_tokens_mistral': 5342, 'n_tokens_neox': 4216, 'n_words': 801, 'path': 'notebooks/mnist/mnist-experiments-256.ipynb', 'repo_name': 'AFAgarap/relu-classifier', 'size': '11822'}