<jupyter_start><jupyter_text>Deep Learning using Rectified Linear Units=== OverviewIn this notebook, we explore the performance of a neural net with varying activation functions on an image classification task. We load our dependencies.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import tensorflow as tf

from neural_net import NeuralNet, loss_fn, train_step, train<jupyter_output><empty_output><jupyter_text>We set up the GPU memory growth.<jupyter_code>tf.config.experimental.set_memory_growth(tf.config.experimental.list_physical_devices('GPU')[0], True)<jupyter_output><empty_output><jupyter_text>We set the random seeds for reproducibility.<jupyter_code>np.random.seed(42)
tf.random.set_seed(42)<jupyter_output><empty_output><jupyter_text>We set the batch size, the number of epochs, and the number of units per layer.<jupyter_code>batch_size = 512
epochs = 300
neurons = [512, 512]<jupyter_output><empty_output><jupyter_text>Data Preparation We load the MNIST dataset.<jupyter_code>(train_features, train_labels), (test_features, test_labels) = tf.keras.datasets.mnist.load_data()<jupyter_output><empty_output><jupyter_text>We scale the images.<jupyter_code>train_features = train_features.astype('float32').reshape(-1, 784) / 255.
test_features = test_features.astype('float32').reshape(-1, 784) / 255.<jupyter_output><empty_output><jupyter_text>We one-hot encode labels.<jupyter_code>train_labels = tf.one_hot(train_labels, 10)
test_labels = tf.one_hot(test_labels, 10)<jupyter_output><empty_output><jupyter_text>We create a `tf.data.Dataset` object for the training dataset.<jupyter_code>dataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels))
dataset = dataset.prefetch(4096).shuffle(4096).batch(batch_size, True)<jupyter_output><empty_output><jupyter_text>Model We use a `NeuralNet` class that consists of two hidden layers.  Logistic-based ModelWe define a 2-layer NN with Logistic activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.sigmoid,
    initializer='glorot_uniform',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>logistic_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 219.92889404296875, mean accuracy = 0.6262186169624329
epoch 100/300 : mean loss = 125.63848114013672, mean accuracy = 0.7658587098121643
epoch 150/300 : mean loss = 83.90592193603516, mean accuracy = 0.8242354393005371
epoch 200/300 : mean loss = 65.95451354980469, mean accuracy = 0.8540331125259399
epoch 250/300 : mean loss = 56.727027893066406, mean accuracy = 0.870559573173523
epoch 300/300 : mean loss = 51.20519256591797, mean accuracy = 0.8803919553756714<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.8871999979019165<jupyter_text>TanH-based ModelWe define a 2-layer NN with Hyperbolic Tangent activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.tanh,
    initializer='glorot_uniform',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>tanh_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 41.1085090637207, mean accuracy = 0.8992888331413269
epoch 100/300 : mean loss = 34.88267517089844, mean accuracy = 0.9150474071502686
epoch 150/300 : mean loss = 31.67683982849121, mean accuracy = 0.92311030626297
epoch 200/300 : mean loss = 29.21517562866211, mean accuracy = 0.9298544526100159
epoch 250/300 : mean loss = 27.132604598999023, mean accuracy = 0.9344785213470459
epoch 300/300 : mean loss = 25.023637771606445, mean accuracy = 0.939987301826477<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9387000203132629<jupyter_text>ReLU-based ModelWe define a 2-layer NN with ReLU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.relu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>relu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 37.917972564697266, mean accuracy = 0.9058660268783569
epoch 100/300 : mean loss = 28.591215133666992, mean accuracy = 0.9302717447280884
epoch 150/300 : mean loss = 23.436594009399414, mean accuracy = 0.943175733089447
epoch 200/300 : mean loss = 19.79465103149414, mean accuracy = 0.9521400928497314
epoch 250/300 : mean loss = 17.128971099853516, mean accuracy = 0.9587673544883728
epoch 300/300 : mean loss = 15.156322479248047, mean accuracy = 0.9637587070465088<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9607999920845032<jupyter_text>Leaky ReLU-based ModelWe define a 2-layer NN with Leaky ReLU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.leaky_relu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>lrelu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 38.26172637939453, mean accuracy = 0.9059996008872986
epoch 100/300 : mean loss = 29.883651733398438, mean accuracy = 0.927200198173523
epoch 150/300 : mean loss = 25.318309783935547, mean accuracy = 0.9388020634651184
epoch 200/300 : mean loss = 22.130830764770508, mean accuracy = 0.9464309811592102
epoch 250/300 : mean loss = 19.446626663208008, mean accuracy = 0.9531416893005371
epoch 300/300 : mean loss = 17.38770294189453, mean accuracy = 0.958650529384613<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9553999900817871<jupyter_text>Softplus-based ModelWe define a 2-layer NN with Softplus activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.softplus,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>softplus_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 44.91297149658203, mean accuracy = 0.8903579115867615
epoch 100/300 : mean loss = 37.71559143066406, mean accuracy = 0.9074686169624329
epoch 150/300 : mean loss = 34.804866790771484, mean accuracy = 0.9156650900840759
epoch 200/300 : mean loss = 32.73615264892578, mean accuracy = 0.9212907552719116
epoch 250/300 : mean loss = 31.295867919921875, mean accuracy = 0.9249966740608215
epoch 300/300 : mean loss = 29.9864444732666, mean accuracy = 0.9279513955116272<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9282000064849854<jupyter_text>ELU-based ModelWe define a 2-layer NN with ELU activation function.<jupyter_code>model = NeuralNet(
    units=neurons,
    activation=tf.nn.elu,
    initializer='he_normal',
    num_classes=10
)<jupyter_output><empty_output><jupyter_text>We train our model for 300 epochs.<jupyter_code>elu_performance = train(model, loss_fn, dataset, epochs=epochs)<jupyter_output>epoch 50/300 : mean loss = 38.46513748168945, mean accuracy = 0.9060162901878357
epoch 100/300 : mean loss = 31.891250610351562, mean accuracy = 0.9221253991127014
epoch 150/300 : mean loss = 27.93938636779785, mean accuracy = 0.9324585795402527
epoch 200/300 : mean loss = 24.76047706604004, mean accuracy = 0.9414229393005371
epoch 250/300 : mean loss = 22.108373641967773, mean accuracy = 0.9475494027137756
epoch 300/300 : mean loss = 19.78851890563965, mean accuracy = 0.9525073170661926<jupyter_text>We produce predictions for the test data.<jupyter_code>predictions = tf.nn.softmax(model(test_features))<jupyter_output><empty_output><jupyter_text>We compute the test accuracy.<jupyter_code>accuracy = tf.metrics.Accuracy()
test_accuracy = accuracy(tf.argmax(predictions, 1), tf.argmax(test_labels, 1))<jupyter_output><empty_output><jupyter_text>Check for the test accuracy.<jupyter_code>print('Test accuracy : {}'.format(test_accuracy.numpy()))<jupyter_output>Test accuracy : 0.9502000212669373<jupyter_text>Training PerformanceWe lay down the training performance of each model.<jupyter_code>sns.set_style('dark', {'grid.linestyle': '--'})
plt.figure(figsize=(16, 8))
plt.rcParams.update({'font.size': 14})

plt.subplot(121)
plt.plot(range(len(logistic_performance[0])), logistic_performance[0], label='logistic')
plt.plot(range(len(tanh_performance[0])), tanh_performance[0], label='tanh')
plt.plot(range(len(relu_performance[0])), relu_performance[0], label='relu')
plt.plot(range(len(lrelu_performance[0])), lrelu_performance[0], label='leaky_relu')
plt.plot(range(len(softplus_performance[0])), softplus_performance[0], label='softplus')
plt.plot(range(len(elu_performance[0])), elu_performance[0], label='elu')
plt.xlabel('epochs')
plt.ylabel('train accuracy')
plt.legend(loc='lower right')
plt.grid()

plt.subplot(122)
plt.plot(range(len(logistic_performance[1])), logistic_performance[1], label='logistic')
plt.plot(range(len(tanh_performance[1])), tanh_performance[1], label='tanh')
plt.plot(range(len(relu_performance[1])), relu_performance[1], label='relu')
plt.plot(range(len(lrelu_performance[1])), lrelu_performance[1], label='leaky_relu')
plt.plot(range(len(softplus_performance[1])), softplus_performance[1], label='softplus')
plt.plot(range(len(elu_performance[1])), elu_performance[1], label='elu')
plt.xlabel('epochs')
plt.ylabel('train loss')
plt.legend(loc='upper right')
plt.grid()

plt.suptitle('Neural Nets on Fashion-MNIST with mini-batch size of 512')
plt.savefig('fmnist_experiments_512.png', dpi=300)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1147637961632722, 'fraction_numerical': 0.13927152877545845, 'mean_word_length': 7.6759530791788855, 'pattern_counts': {'":': 0, '<': 145, '<?xml version=': 0, '>': 145, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '269962', 'license': 'Apache-2.0', 'n_tokens_mistral': 5348, 'n_tokens_neox': 4215, 'n_words': 801, 'path': 'notebooks/fashion-mnist/fmnist-experiments-512.ipynb', 'repo_name': 'AFAgarap/relu-classifier', 'size': '11833'}