<jupyter_start><jupyter_text>Neural Network Classifier with Mini-Batch Gradient Descent Author: <NAME>, Data Scientist at GfK.[LinkedIn profile](https://www.linkedin.com/in/ivan-bongiorni-b8a583164/)This model is a **feed forward Neural Network Classifier** based on the classical **spam dataset**.It represents a tutorial on basic TensorFlow 2.0 (alpha). The 1.x version of the same model can be found in [this Notebook](https://github.com/IvanBongiorni/TensorFlow_Tutorial/blob/master/TensorFlow_2_Classification_MiniBatchGD.ipynb).Summary:0. Why Mini-Batch Gradient Descent?1. Import data + dataprep,2. Neural Network architecture,3. Implementation of full-Batch Gradient Descent,4. Visualization  0. Why Mini-Batch Gradient Descent?**Mini-Batch Gradient Descent** is of fundamental importance for Deep Learning.It consists in feeding, at each iteration of the Gradient Descent, only a subset of (i.e.: a mini-batch) of the training data. At each iteration, mini-batch gradient descent will take a different mini-batch of data, and train just on that one.This technique presents at least three advantages:- It prevents the optimization algorithm to end up stuck into some local optima of the loss function.- It allows for a significant drop in computational times. That is because the optimizer trains the network just on a mini-batch of data, requiring much less effort. Often Data Scientists are dealing with massive amounts of data, and training a model on the whole train dataset at each epoch might simply be impossible.- It is the mandatory technique for [online learning](https://en.wikipedia.org/wiki/Online_machine_learning).<jupyter_code>import tensorflow as tf

print(tf.__version__)
import numpy as np
import pandas as pd
pd.options.display.max_columns = None  # this prints all columns

from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>1. Import data + dataprepThe **spam classification dataset**, [available from the UCI ML repository](https://archive.ics.uci.edu/ml/datasets/spambase) contains an already preprocessed collection of email data. Each datapoint corresponds to an email, and is classified as either "spam" or "not spam".<jupyter_code># load the data directly from the repository into pandas

df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/spambase/spambase.data", header=None)
df.shape
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Explore the target variableLet's explore the dependent variable. Before feeding the data in a Neural Network, it will require one-hot encoding.<jupyter_code># Description of the classification variable
df.iloc[:,57].describe()
print(set(df.iloc[:,57]))
print('no. classes: ' + str(len(set(df[df.columns[1]]))))
# One-hot encoding

classification = pd.get_dummies(df.iloc[:,57])
print(classification.shape)<jupyter_output>(4601, 2)<jupyter_text>I obtained a target dataframe, called `classification`, that contains the one-hot encoded version of my dependent binary variable. Now I can isolate the explanatory variables in my dataframe `df`. In order to do that I'll drop columns 0 and 1: the first is an index, the second is the target variable.<jupyter_code>df = df.drop(df.columns[57], axis=1)   # drop target column from explanatory df
# Now the shape is:
df.shape<jupyter_output><empty_output><jupyter_text>Before training the model, I turn both explanatory and target data into numpy objects:<jupyter_code>df = df.values.astype(np.float64)
classification = classification.values.astype(np.float64)<jupyter_output><empty_output><jupyter_text>Train-Test splitIn an actual ML job, you would split your dataset in **Train**, **Validation** and **Test** sets. However, this is just an example on how to implement and run a Neural Network, so I'll skip that part and will split the data in train and test only.<jupyter_code>## TRAIN-TEST SPLIT
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df, classification, test_size=0.25, random_state=173)

print('X_train shape: ' + str(X_train.shape) + '\ny_train shape: ' + str(y_train.shape))
print('\nX_test shape: ' + str(X_test.shape) + '\ny_test shape: ' + str(y_test.shape))<jupyter_output>X_train shape: (3450, 57)
y_train shape: (3450, 2)

X_test shape: (1151, 57)
y_test shape: (1151, 2)<jupyter_text>Scaling the variables must happen after the train-test split. That is because the test set must be scaled using the parameters of the training set: in real world cases you don't know what data you'll get from training, therefore this is the only way to truly understand the performance of the model.<jupyter_code># Scale the variables using Z-scores
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>2. Neural Network architectureSince the network is not very deep, and the number of parameters is relatively small, I can employ more "demanding" (and performing) activation functions. In this case, I chose **ELU** (**Exponential Linear Unit**) activations. A **softmax** function is then applied at the end, so that the attribution of classes ("spam"/"not-spam") is shrinked into probabilities.Additionally, I apply **dropout** in order to prevent overfitting.<jupyter_code>from tensorflow.keras.layers import Dense, BatchNormalization, Dropout
from tensorflow.keras.activations import elu, softmax

# Architecture
n_input = X_train.shape[1]
n_hidden1 = 30
n_hidden2 = 30
n_hidden3 = 15
n_output = y_train.shape[1]

# set dropout probability
dropout_prob = 0.1<jupyter_output><empty_output><jupyter_text>The definition of a model in TensorFlow 2.0 follows the syntax of Keras' `Sequential()` models.Each layer is defined by the `Dense()` function, taking as inputs: the previous layer, the number of nodes, and the activation function (it can actually take a lor of additional arguments, but I'll not review them here). The input layer also requires a definition of the input data shape.<jupyter_code>model = tf.keras.models.Sequential([
    Dense(n_input, input_shape = (n_input,), activation = elu),   # Input layer
    
    Dense(n_hidden1, activation = elu), # hidden layer 1
    Dropout(dropout_prob), 
    
    Dense(n_hidden2, activation = elu), # hidden layer 2
    Dropout(dropout_prob), 
    
    Dense(n_hidden3, activation = elu), # hidden layer 3
    Dropout(dropout_prob), 
    
    Dense(n_output, activation = softmax)  # Output layer
])
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 57)                3306      
_________________________________________________________________
dense_1 (Dense)              (None, 30)                1740      
_________________________________________________________________
dropout (Dropout)            (None, 30)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 30)                930       
_________________________________________________________________
dropout_1 (Dropout)          (None, 30)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 15)                465       
________________________________________________________[...]<jupyter_text>3. Implementation of Mini-Batch Gradient DescentIn order to feed mini-batches of data into the Network at each iteration, I will write a function that takes a different subset of the main dataset at each epoch.<jupyter_code>@tf.function
def fetch_batch(X, y, batch_size, epoch):
    '''
    Extracts a mini-batches of given size from a dataset.
    WARNING: Requires a batch size that is multiple of the number of training epochs
    
    Returns: Train and Test Mini-Batches
    '''
    start = epoch*batch_size  # starting row of next mini-batch
        
    X_batch = X[start:start+batch_size, :]
    y_batch = y[start:start+batch_size, :]
    
    return X_batch, y_batch<jupyter_output><empty_output><jupyter_text>On the @tf.function decoratorIt's worth to spend few words on the **decorator @tf.function**. The decorator transform plain Python functions into TensorFlow's **graph code** (yeah, the underlying TensorFlow code is still in graph mode as in 1.x, you just can't see it now). This has the advantage to make it compatible with the module's functinos, and the make their runtime much faster. Now I'll define the usual hyperparameters of the model:<jupyter_code># Loss: Binary cross-entropy is specific for binary classification tasks
bce_loss = tf.keras.losses.BinaryCrossentropy()

# Binary Accuracy (expressed in the [0,1] interval)
accuracy = tf.keras.metrics.BinaryAccuracy()

# Adam Optimizer (what you'll need 99.99% of the time)
optimizer = tf.optimizers.Adam(learning_rate = 0.0005)<jupyter_output><empty_output><jupyter_text>As [in the previous Notebook](https://github.com/IvanBongiorni/TensorFlow2.0_Tutorial/blob/master/TensorFlow2.0_01_basic_Classifier.ipynb), I'll use TensorFlow's **eager execution method**. Please refer to it for a more detailed explanation. The fundamental element of training a Network in eager execution is represented by `tf.GradientTape()`. This object calculates and stores the gradient of the loss function at each iteration of the training operation. Once you generate a GradientTape, you can call the `.gradient()` argument to get the actual gradient (i.e. the first derivative of the loss function). Later, you feed this values into an `optimizer` using the `.apply_gradients` argument that updates the Network's trainable variables (the very act of "learning").<jupyter_code>from sklearn.utils import shuffle

X_train, y_train = shuffle(X_train, y_train, random_state=123)
batch_size = 50

n_epochs = len(X_train) // batch_size

print('No. of epochs: ' + str(n_epochs))
###  TRAINING

loss_history = []
accuracy_history = []


# I set a number of epochs that correspond to the number of mini-batches
for epoch in range(n_epochs):
    
    # load the Mini-Batches
    X_batch, y_batch = fetch_batch(X_train, y_train, batch_size, epoch)
    
    with tf.GradientTape() as tape:
        current_loss = bce_loss(model(X_batch), y_batch)  # take current loss
    
    gradients = tape.gradient(current_loss, model.trainable_variables)    # get the gradient of the loss function
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))  # update the weights
    
    loss_history.append(current_loss.numpy())   # save current loss in its history vector
    
    accuracy.update_state(y_batch, model(X_batch))  # compute current accuracy
    current_accuracy = accuracy.result().numpy()  # save its result as numpy object
    accuracy_history.append(current_accuracy)
    
    # To monitor progress, print loss and accuracy scores every 10 epochs
    if (epoch+1) % 10 == 0:
        print(str(epoch+1) + '.\tTraining Loss: ' + str(current_loss.numpy()) + ',\tAccuracy: ' + str(current_accuracy))
    
    accuracy.reset_states()  # reset the state of accuracy object for next iteration
#
print('\nTraining complete.')<jupyter_output>10.	Training Loss: 4.979808330535889,	Accuracy: 0.84
20.	Training Loss: 4.81271505355835,	Accuracy: 0.78
30.	Training Loss: 3.1986169815063477,	Accuracy: 0.88
40.	Training Loss: 2.1304428577423096,	Accuracy: 0.92
50.	Training Loss: 2.0643529891967773,	Accuracy: 0.94
60.	Training Loss: 2.1822969913482666,	Accuracy: 0.92

Training complete.<jupyter_text>4. VisualizationOnce the training is done, let's check the model's improvement visually:<jupyter_code># using plt.subplot I can show paired loss and accuracy

plt.figure(figsize = (15, 4))  # adjust figures size
plt.subplots_adjust(wspace=0.2)  # adjust distance

# loss plot
plt.subplot(1, 2, 1)
plt.plot(loss_history)
plt.title('Loss History')
plt.xlabel('epochs')
plt.ylabel('Binary Cross-Entropy')

# accuracy plot
plt.subplot(1, 2, 2)
plt.plot(accuracy_history)
plt.title('Accuracy History')
plt.xlabel('epochs')
plt.ylabel('Accuracy')

plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, the main difference from full-Batch Gradient Descent is that the loss and accuracy histories look much more noisy. This was predictable, since at each iteration the model is trained only on a very small amount of data.At this point, Let's make a prediction on the test set.<jupyter_code>prediction = model.predict(X_test)    # (Keras syntax)
# outputs a vector that contains the value of the class with the highest predicted probability
prediction = np.argmax(prediction, axis=1)

# so I do it also for the test data - i.e.: reverse one-hot encoding and get a vector of 0-1 values
testdata = np.argmax(y_test, axis=1)

# Now I can plot the Confusion Matrix
from sklearn.metrics import confusion_matrix

CM = confusion_matrix(prediction, testdata)
print(CM)<jupyter_output>[[665  65]
 [ 37 384]]<jupyter_text>The Confusion Matrix looks very good: oservations on the matrix diagonal are correct predictions. Luckily, only two datapoints have been misclassified.<jupyter_code># Accuracy = sum of the diagonal / sum of the whole matrix

print('Test Accuracy: ' + str(np.sum(np.diag(CM)) / np.sum(CM)))<jupyter_output>Test Accuracy: 0.9113814074717637<jupyter_text>I like to visualize Confusion Matrices using a heatmap. IMHO, `seaborn` is the best Python library for that:<jupyter_code>import seaborn

seaborn.heatmap(CM, annot=True)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09039959240119368, 'fraction_numerical': 0.023582502365528786, 'mean_word_length': 4.844321565291366, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '640367', 'license': 'MIT', 'n_tokens_mistral': 4273, 'n_tokens_neox': 3954, 'n_words': 1432, 'path': 'TensorFlow2.0__00.02_MiniBatch_Gradient_Descent.ipynb', 'repo_name': 'Gautamtverma/TensorFlow2.0_Notebooks', 'size': '13747'}