<jupyter_start><jupyter_text>Activation and Loss functions To complete this assignment, create a Jupyter notebook containing your solutions to the following tasks:1. In this task, you'll implement several ANN models with different activation functions. Specifically:  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using tanh activation function for each layer.  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using sigmoid activation function for each layer.  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using ReLU activation function for each layer.  1. Compare the result of each model with each other. Which activation function did perform better?1. In this task, you'll implement the ANN models specified below with the hinge loss function as their loss functions:  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using tanh activation function for each layer.  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using sigmoid activation function for each layer.  1. Implement a three layer ANN model with 128, 64 and 10 neurons in the layers using ReLU activation function for each layer.  1. Compare the result of each model with the result of the same model from the previous task. Which loss function did perform better? Lib imports and data load<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential 
from tensorflow.keras.layers import Dense
(X_train, y_train), (X_test, y_test) = mnist.load_data()

input_dim = 784  # 28*28
output_dim = nb_classes = 10
batch_size = 128

X_train = X_train.reshape(60000, input_dim)
X_test = X_test.reshape(10000, input_dim)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

Y_train = to_categorical(y_train, nb_classes)
Y_test = to_categorical(y_test, nb_classes)<jupyter_output><empty_output><jupyter_text>Model Set 1 - Loss:'categorical_crossentropy'<jupyter_code>for actv_func in ['tanh', 'sigmoid', 'relu']:
    model = Sequential()
    model.add(Dense(128, input_shape=(input_dim,), activation=actv_func))
    model.add(Dense(64, activation=actv_func))
    model.add(Dense(output_dim, activation='softmax'))

    mtrcs = ['accuracy']
    loss_fn = 'categorical_crossentropy'
    model.compile(optimizer='adam', loss=loss_fn, metrics=mtrcs)
    model.fit(X_train, Y_train, batch_size=batch_size, epochs=20, verbose=0)

    train_score = model.evaluate(X_train, Y_train, verbose=0)
    test_score = model.evaluate(X_test, Y_test, verbose=0)

    scoredf = pd.DataFrame(index=[f"loss-function: {loss_fn}"] + ["metrics: " + i for i in mtrcs], 
                           columns=['train-score', 'test-score'])
    scoredf['train-score'] = train_score
    scoredf['test-score'] = test_score
    display(f"activation function: {actv_func}",
            scoredf)<jupyter_output><empty_output><jupyter_text>tanh performs the best with the highest accuracy and lowest loss score for both the training and validation data.  Model Set 2 - Loss:'hinge'<jupyter_code>for actv_func in ['tanh', 'sigmoid', 'relu']:
    model = Sequential()
    model.add(Dense(128, input_shape=(input_dim,), activation=actv_func))
    model.add(Dense(64, activation=actv_func))
    model.add(Dense(output_dim, activation='softmax'))

    mtrcs = ['accuracy']
    loss_fn = 'hinge'
    model.compile(optimizer='adam', loss=loss_fn, metrics=mtrcs)
    model.fit(X_train, Y_train, batch_size=batch_size, epochs=20, verbose=0)

    train_score = model.evaluate(X_train, Y_train, verbose=0)
    test_score = model.evaluate(X_test, Y_test, verbose=0)

    scoredf = pd.DataFrame(index=[f"loss-function: {loss_fn}"] + ["metrics: " + i for i in mtrcs], 
                           columns=['train-score', 'test-score'])
    scoredf['train-score'] = train_score
    scoredf['test-score'] = test_score
    display(f"activation function: {actv_func}",
            scoredf)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.095488183337312, 'fraction_numerical': 0.02530436858438768, 'mean_word_length': 4.662162162162162, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '117646', 'license': 'MIT', 'n_tokens_mistral': 1422, 'n_tokens_neox': 1309, 'n_words': 425, 'path': 'Assignment-4.ipynb', 'repo_name': 'shobes572/DSI_Module-33', 'size': '4189'}