<jupyter_start><jupyter_text>Classifying spectra using a *Fully Connected Neural Network* (FNN)1. **Defining the input parameters**2. **Loading the training and testing data set**3. **Set up the architecture for the neural network**4. **Train the network**     4.1. **Plot of the loss and accuracy curve for NN1**5. **Evaluate the result on the testing data set****Data:** 25th Nov. 2019 **Author**: <NAME> **Guidance from**: Schäfer <NAME>**Script adapted from**: <NAME><jupyter_code>import numpy as np
import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 20})
from sklearn.metrics import confusion_matrix
import pickle<jupyter_output><empty_output><jupyter_text>To install **`tensorflow`** visit [here](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html) for instructions.<jupyter_code>import tensorflow as tf
import keras
from keras.models import Sequential, load_model
from keras.layers import (Dense, Conv1D, MaxPooling1D,
                          Dropout, Flatten, BatchNormalization)
from keras import optimizers
from keras import regularizers<jupyter_output><empty_output><jupyter_text>1. Defining the input parameters<jupyter_code># code parameters used further
epochs = 20
batch_size = 40 
momentum = 0.0
learning_rate=0.0001
verbrose = 2    # degree of info. outputted during training

# directory to save model
save_model_dir = 'Algos/NN_folder/NN1'

# do you want to reuse a previously trained model?
reuse = False<jupyter_output><empty_output><jupyter_text>2. Loading the training and testing data sets<jupyter_code>def loadTrainingTestingData(data_names):
    X_train = np.load("Data_files/"+data_names[0]+".npy", mmap_mode='r')
    Y_train = np.load("Data_files/"+data_names[1]+".npy", mmap_mode='r')
    X_test = np.load("Data_files/"+data_names[2]+".npy", mmap_mode='r')
    Y_test = np.load("Data_files/"+data_names[3]+".npy", mmap_mode='r')
    return X_train, Y_train, X_test, Y_test
# reading the data
data_names = ['X_train', 'Y_train', 'X_test', 'Y_test']

X_train, Y_train, X_test, Y_test = loadTrainingTestingData(data_names)
data_sets = [X_train, Y_train, X_test, Y_test]

# converts the vectors of intergers to a binary matrix
Y_train = keras.utils.to_categorical(Y_train)
Y_test = keras.utils.to_categorical(Y_test)
data_sets = [X_train, Y_train, X_test, Y_test]

# print shape of training and testing data sets
for i in range(len(data_names)):
               print(data_names[i],'=', np.shape(data_sets[i]))<jupyter_output>X_train = (553301, 4317)
Y_train = (553301, 3)
X_test = (237130, 4317)
Y_test = (237130, 3)<jupyter_text>3. Setting up the architecture for the neural networkFigure shows the typical architecture of the fully connected neural network used in  the succeeding cell block. In our case, we have 1 input layer with 90 neurons, 1 hidden layer with 30 neurons and 1 output layer with 3 neurons.Training the data set implies that we are trying to optimize the loss function. * *Optimizer*: The optimizer chosen here is the Stochastic gradient descent optimizer (SGD); the list of other optimizers can be found [here](https://keras.io/optimizers/). * *Loss function*: Here, 'Cross entropy' is a method to define this loss function.<jupyter_code>def reuseModel(save_model_dir):
    """
    Function loads a previously trained model
    """
    with open(save_model_dir+'_history.pkl', 'rb') as filehandler:
            history = pickle.load(filehandler)
    return load_model(save_model_dir+'.h5')

def buildCompileModel(learning_rate, momentum):
    """
    Function to build the model architecture, set optimizers and compile the model 
    """
    model = Sequential()
    # 1 input layer, 1 hidden layer, and 1 output layer
    model.add(Dense(90, activation='relu', input_shape=(4317,)))
    model.add(Dense(30, activation='relu'))
    model.add(Dense(3, activation='softmax'))

    # optimizer 
    sgd = optimizers.SGD(lr=learning_rate, momentum=momentum, nesterov=False)

    # compilation for a multi-category classification problem
    model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])
    return model
if reuse:
    reuseModel(save_model_dir)
    load_model(save_model_dir+'.h5')
else:
    model = buildCompileModel(learning_rate, momentum)<jupyter_output><empty_output><jupyter_text>4. Train the networkThe **`model.fit()`** fits the model to the data for the desired number of **`epochs`** (a pass through the entire training set), **`batch_size`** (number of training sample propogated at once through the network), and **`verbrose`** (degree of information printed out while training).<jupyter_code>model_history = model.fit(X_train, Y_train,epochs=epochs, batch_size=batch_size,validation_data=(X_test, Y_test),verbose=verbrose)<jupyter_output>Train on 553301 samples, validate on 237130 samples
Epoch 1/20
 - 37s - loss: 0.5317 - accuracy: 0.8340 - val_loss: 0.4026 - val_accuracy: 0.8835
Epoch 2/20
 - 32s - loss: 0.3813 - accuracy: 0.8921 - val_loss: 0.3470 - val_accuracy: 0.8986
Epoch 3/20
 - 33s - loss: 0.3317 - accuracy: 0.9067 - val_loss: 0.3095 - val_accuracy: 0.9124
Epoch 4/20
 - 33s - loss: 0.3104 - accuracy: 0.9137 - val_loss: 0.2931 - val_accuracy: 0.9168
Epoch 5/20
 - 33s - loss: 0.2923 - accuracy: 0.9185 - val_loss: 0.3025 - val_accuracy: 0.9201
Epoch 6/20
 - 33s - loss: 0.2899 - accuracy: 0.9222 - val_loss: 0.2640 - val_accuracy: 0.9239
Epoch 7/20
 - 33s - loss: 0.2765 - accuracy: 0.9241 - val_loss: 0.2540 - val_accuracy: 0.9295
Epoch 8/20
 - 33s - loss: 0.2567 - accuracy: 0.9282 - val_loss: 0.2479 - val_accuracy: 0.9300
Epoch 9/20
 - 33s - loss: 0.2612 - accuracy: 0.9290 - val_loss: 0.2448 - val_accuracy: 0.9284
Epoch 10/20
 - 33s - loss: 0.2481 - accuracy: 0.9319 - val_loss: 0.2344 - val_accuracy: 0.9323
Epoch 1[...]<jupyter_text>4.1 Plot of the loss and accuracy curve for NN1<jupyter_code>def setLabels(ax, xlabel, ylabel, ticks, legend):
    """
    Function sets the axes labels, legent size, and ticks
    """
    ax.set_xlabel(xlabel, fontsize=20)
    ax.set_ylabel(ylabel, fontsize=20)
    if legend:
        ax.legend(loc='center right', fontsize=18)
    ax.grid()
    if ticks:
        ax.tick_params(which='both', labelsize=18)
    return

def plotLoss(ax, loss_Accuracy, labels):
    """
    Function plots the loss curve for training run of x epochs
    """
    ax.plot(history[loss_Accuracy[0]], '#00a15b', lw=2, label=labels[0])
    ax.plot(history[loss_Accuracy[1]], '#fc5400', lw=2, label=labels[1])
    ax.set_title('Loss curve for CNN\n')
    setLabels(ax, 'Epoch', r'$\mathcal{L}\ \equiv$ Categorical cross entropy', ticks=True, legend=True)
    return

def plotAccuracy(ax, loss_Accuracy, labels):
    """
    Function plots the accuracy curve for training run of x epochs
    """
    ax.plot(history[loss_Accuracy[2]],  '#00a15b', lw=2, label=labels[0])
    ax.plot(history[loss_Accuracy[3]], '#fc5400', lw=2, label=labels[1])
    ax.set_title('Accuracy curve for CNN\n')
    setLabels(ax, 'Epoch', 'Accuracy', ticks=True, legend=True)
    return<jupyter_output><empty_output><jupyter_text>The loss and accuracy curves are plotted for NN1 for epochs.<jupyter_code># logs the history of the run
history = model_history.history

# plots the history
fig, ax = plt.subplots(1,2, figsize=(14,6))
labels = ['training', 'validation']
loss_Accuracy = ['loss', 'val_loss', 'accuracy', 'val_accuracy']

plotLoss(ax[0], loss_Accuracy, labels)
plotAccuracy(ax[1], loss_Accuracy, labels)

fig.savefig('Images/LandA_RNN.png')    
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>5. Evaluate the result on the testing data setOnce we have trained the model to make predictions on the data, we evaluate it's performance on the test data set.<jupyter_code>def saveModel(history, save_model_dir):
    """
    Function to save the model and it's history
    """
    model.save(save_model_dir+'.h5')
    
    with open(save_model_dir+'_history.pkl', 'wb') as filehandler:
        pickle.dump(history, filehandler)
    print('\nModel saved')
    return
# predictions made on the test data set, computes the confusion matrix

score = model.evaluate(X_test, Y_test,verbose=0)
print('\nLoss = %.2f percent, Accuracy = %.2f percent'%(score[0]*100, score[1]*100))

saveModel(history, save_model_dir)<jupyter_output>Loss = 19.03 percent, Accuracy = 94.48 percent

Model saved<jupyter_text>5.1 Computes the confusion matrixIt is a table that describes the performance of the classifier when compared to the test set data.<jupyter_code>def computeConfusionMatrix(model, X_test, Y_test):
    """
    Function prints the confusion matrix for the trained classifier/model
    @param model :: trained model
    @param X,Y_test :: array containing the X,Y test data sets
    """
    # need to recover the Y_pred from the one-hot encoded array
    X_pred = model.predict(X_test)
    Y_pred = np.argmax(X_pred, axis=1)
    
    # calculates the matrix by comparison
    confusion_mat = confusion_matrix(Y_test.argmax(axis=1), Y_pred)
    normalization = np.sum(confusion_mat, axis=1, keepdims=True)
    
    confusion_mat = confusion_mat/normalization
    print(confusion_mat)
    return confusion_mat
confusion_mat = computeConfusionMatrix(model, X_test, Y_test)<jupyter_output>[[0.88374532 0.11149397 0.00476072]
 [0.01759276 0.97880887 0.00359837]
 [0.02205882 0.09662095 0.88132022]]<jupyter_text>6. Find optimal values for model parametersThe following snippet of code aims to compute the optimal values for the model parameters:* **Batch size**: size at which data is sent into the network* **Epoch**: a complete iteration over all batches* **Learning rate**: rate which the weights are updated during training* **Momentum**: amount that smooths the progressions of the learning algorithmThe metric for evaluating the optimal values are the loss and accuracy percentages for every epoch of training and validation.Note: `l&a` $\equiv$ loss and accuracy, `t&v` $\equiv$ training and validation.<jupyter_code># among these values for the parameters the "optimal" combination is chosen
batch_sz_arr = [40, 30, 50]
epochs_arr = [20, 30, 40]
learn_rt_arr = [0.001, 0.01, 0.1, 0.0001]
momentum_arr = [0., 0.5, 0.9]<jupyter_output><empty_output><jupyter_text>**Notes on choice of parameters:*** For batch-size 40, the best loss and accuracy curve was observed.<jupyter_code>def computeOptimalParams(batch_size, epochs, learning_rate, momentum, data_set): 
    """
    Function computes the loss and accuracy for the given set of input parameters 
    @batch_size, epoch, learning_rate, momentum
    
    Returns:
    @loss_arr :: loss for every epoch for training and validation cases
    @accuracy_arr :: accuracy for every epoch for the training and validation cases
    """
    # builds and compiles the model
    model = buildCompileModel(learning_rate, momentum)
    
    # trains the model
    model_history = model.fit(data_sets[0], data_sets[1], epochs=epochs, batch_size=batch_size, validation_data=(data_sets[2],data_sets[3]), verbose=verbrose)
    print('Model trained')
    
    # logs the history of the run with the train, test curves
    history = model_history.history
    loss_arr = [history['loss'], history['val_loss']]
    accuracy_arr = [history['accuracy'], history['val_accuracy']]
    return loss_arr, accuracy_arr

def appendLossAccValues(loss_accuracy, loss_arr, accuracy_arr):
    """
    Function that appends the values for the 'loss & accuracy' (l&a) for 'train & validation' (t&v)for every epoch
    @Input loss_acc :: contains 4-columns with the l&a values for t&v
    @loss_arr :: 2-columns with loss values for t&v
    @accuracy_arr :: 2-columns with accuracy values for t&v
    
    @Returns :: 4-column updated `loss_accuracy` array
    """
    # append training values
    loss_accuracy[0] = np.append(loss_accuracy[0], [loss_arr[0]], axis=0)
    loss_accuracy[1] = np.append(loss_accuracy[1], [accuracy_arr[0]], axis=0)
    
    # append validation values
    loss_accuracy[2] = np.append(loss_accuracy[2], [loss_arr[1]], axis=0)
    loss_accuracy[3] = np.append(loss_accuracy[3], [accuracy_arr[1]], axis=0)
    
    return loss_accuracy

def prepareArrrays(epochs):
    """
    Function to prepare arrays of size = # of epochs
    """
    loss_train, accuracy_train = np.empty((0, epochs)), np.empty((0, epochs))
    loss_valid, accuracy_valid = np.empty((0, epochs)), np.empty((0, epochs))
    return [loss_train, accuracy_train, loss_valid, accuracy_valid]<jupyter_output><empty_output><jupyter_text>6.1 Varying batch-size<jupyter_code># preparing empty arrays
loss_accu_bs = prepareArrrays(epochs_arr[0])

for batch_sz in batch_sz_arr:
    # computing l&a for every 'batch_sz' in the 'batch_sz_arr'
    loss_arr, accuracy_arr = computeOptimalParams(batch_sz, epochs_arr[0], learn_rt_arr[0], momentum_arr[0], data_sets)
      
    # appends loss and accuracy values for every t&v cases
    loss_accu_bs = appendLossAccValues(loss_accu_bs, loss_arr, accuracy_arr)<jupyter_output><empty_output><jupyter_text>6.1.1 Plotting the loss and accuracy curves for various batch-sizes<jupyter_code>def plotCurves(list_idx, ax, loss_accu_bs_arr, line_style, color, param_arr, ylabel, legend_idx, type_):
    """
    Function plots the loss curves for t&v for varying set of parameters
    @list_idx :: int array whose length = # of batch-size params tried
    @line_style, param_arr, ylabel :: plotting accesories, and labels for plot
    @legend_idx :: selects the legend label
    @loss_accu_bs_arr :: array with the t&v curves for every set of trial parameters    
    """
    labels = ['batch size=%d', 'learning rate=%0.4f', 'momentum=%0.1f', 'epochs=%d']
    for i in list_idx:
        if type_ == 'train':
            ax.plot(loss_accu_bs_arr[i],  color, linestyle=line_style[i], lw=2, label=labels[legend_idx]%param_arr[i])
            setLabels(ax, 'Epoch', ylabel+' (%)', ticks=True, legend=True)
            ax.legend(loc='best')
        elif type_ == 'test':
            ax.plot(loss_accu_bs_arr[i],  color, linestyle=line_style[i], lw=2, label=param_arr[i])
            setLabels(ax, 'Epoch', ylabel+' (%)', ticks=True, legend=False)   
    return

def varyingParamsPlot(idx, ax, loss_accu, param_arr, colr, legend_idx):
    """
    Function plots the loss and accuracy curves  (both t&v) for every element of param_arr
    @loss_accu :: 3-D array containing the l&a values for every element of param_arr
    @param_arr :: paramter array used to see the effect on l&a curves
    @idx :: index array == len(param_arr)
    """
    # plot-accesories
    line_style = ['-', '--', ':', '-.']
    
    # t&v -loss curves
    plotCurves(idx, ax[0], loss_accu[0], line_style, colr,param_arr,'Loss',legend_idx,'train')
    plotCurves(idx, ax[0], loss_accu[2],line_style, '#6c6d70',param_arr,'Loss',legend_idx,'test')

    # t&v - accuracy curves
    plotCurves(idx, ax[1], loss_accu[1],line_style, colr, param_arr,'Accuracy',legend_idx,'train')
    plotCurves(idx, ax[1], loss_accu[3],line_style, '#6c6d70', param_arr,'Accuracy',legend_idx,'test')
    return
fig, ax = plt.subplots(1,2, figsize=(15,6))

# len(idx) = number of attempts to vary parameters
idx = [0, 1, 2]

# plotting the effect of varying batch-size
varyingParamsPlot(idx, ax, loss_accu_bs, batch_sz_arr, 'r', legend_idx=0)

fig.suptitle('Plot showing the effect of varying batch size: RNN')
fig.savefig('Images/Varying_bs_RNN.png')
np.save('Data_files/Varying_bs_RNN.npy',loss_accu_bs)<jupyter_output><empty_output><jupyter_text>6.2 Varying learning rate<jupyter_code># preparing empty arrays
loss_accu_lr = prepareArrrays(epochs_arr[0])

for learn_rt in learn_rt_arr:
    # computing l&a for every 'learn_rt' in the 'learn_rt_arr'
    loss_arr, accuracy_arr = computeOptimalParams(batch_sz_arr[0], epochs_arr[0], learn_rt, momentum_arr[0], data_sets)
      
    # appends loss and accuracy values for every t&v cases
    loss_accu_lr = appendLossAccValues(loss_accu_lr, loss_arr, accuracy_arr)<jupyter_output><empty_output><jupyter_text>6.2.1 Plotting the loss and accuracy curves for various learning-rates<jupyter_code>fig, ax = plt.subplots(1,2, figsize=(15,6))

# len(idx) = number of attempts to vary parameters
idx = [0, 3]

# plotting the effect of varying batch-size
varyingParamsPlot(idx, ax, loss_accu_lr, learn_rt_arr, '#00008b', legend_idx=1)

fig.suptitle('Plot showing the effect of varying learning rate')
fig.savefig('Images/Varying_lr_RNN.png')
np.save('Data_files/Varying_lr_RNN.npy',loss_accu_lr)<jupyter_output><empty_output><jupyter_text>6.3 Varying momentum<jupyter_code># preparing empty arrays
loss_accu_m = prepareArrrays(epochs_arr[0])

for momentum in momentum_arr:
    # computing l&a for every 'momentum' in the 'momentum_arr'
    loss_arr, accuracy_arr = computeOptimalParams(batch_sz_arr[0], epochs_arr[0], learn_rt_arr[0], momentum, data_sets)
      
    # appends loss and accuracy values for every t&v cases
    loss_accu_m = appendLossAccValues(loss_accu_m, loss_arr, accuracy_arr)<jupyter_output><empty_output><jupyter_text>6.2.1 Plotting the loss and accuracy curves for various momenta<jupyter_code>np.save('Data_files/Varying_m_RNN.npy',loss_accu_m)<jupyter_output><empty_output><jupyter_text>6.3.1 Plotting the loss and accuracy curves for various learning-rates<jupyter_code>fig, ax = plt.subplots(1,2, figsize=(15,6))

# len(idx) = number of attempts to vary parameters
idx = [0, 1]

# plotting the effect of varying batch-size
varyingParamsPlot(idx, ax, loss_accu_m, momentum_arr,'#05f29b', legend_idx=2)

fig.suptitle('Plot showing the effect of varying momentum: RNN')
fig.savefig('Images/Varying_m_RNN.png')<jupyter_output><empty_output><jupyter_text>6.4 Varying epochs<jupyter_code>s = ["one","two", "three"]

for i,epochs in enumerate(epochs_arr):
    # preparing empty arrays
    exec("loss_accu_e%s = prepareArrrays(epochs_arr[0])"%(s[i]))
    
    # computing l&a for every 'epochs' in the 'epochs_arr'
    loss_arr, accuracy_arr = computeOptimalParams(batch_sz_arr[0], epochs, learn_rt_arr[0], momentum_arr[0], data_sets)
      
    # appends loss and accuracy values for every t&v cases
    exec("loss_accu_e%s=[loss_arr, accuracy_arr]"%(s[i]))
np.save('Data_files/Varying_ei_RNN.npy',loss_accu_eone)
np.save('Data_files/Varying_eii_RNN.npy',loss_accu_etwo)
np.save('Data_files/Varying_eiii_RNN.npy',loss_accu_ethree)
def plotEpochs(ax, loss_accu_e, line_style, epochs_arr): 
    """
    Function plots the l&a curves for various epoch values
    """
    # t&v - loss curves
    ax[0].plot(loss_accu_e[0][0], '#fc5603', linestyle=line_style, lw=2,label='epochs=%d'%epochs_arr)
    ax[0].plot(loss_accu_e[0][1],'#6c6d70',linestyle=line_style, lw=2)

    # t&v - accuracy curves
    ax[1].plot(loss_accu_e[1][0],'#fc5603',linestyle=line_style, lw=2,label='epochs=%d'%epochs_arr)
    ax[1].plot(loss_accu_e[1][1], '#6c6d70',linestyle=line_style, lw=2)    
    
    # sets labels for the two plots
    setLabels(ax[0], 'Epoch', 'Loss', ticks=True, legend=True)
    setLabels(ax[1], 'Epoch', 'Accuracy', ticks=True, legend=True)
    return

def readEpochsData():
    """
    Function reading the data of the trained models, for all attempted epochs
    """
    la_i = np.load('Data_files/Varying_ei_CNN.npy', mmap_mode='r')
    la_ii = np.load('Data_files/Varying_eii_CNN.npy', mmap_mode='r')
    la_iii = np.load('Data_files/Varying_eiii_CNN.npy', mmap_mode='r')
    return [la_i, la_ii, la_iii]
fig, ax = plt.subplots(1,2, figsize=(15,6))

loss_accu_e = readEpochsData()
line_style = ['-', '--', ':', '-.']
    
for i in [0,1,2]:    
    plotEpochs(ax, loss_accu_e[i], line_style[i], epochs_arr[i])
    
fig.suptitle('Plot showing the effect of varying epcohs: RNN')
fig.savefig('Images/Varying_e_RNN.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1216093117408907, 'fraction_numerical': 0.035931174089068825, 'mean_word_length': 4.813768755516328, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '604368', 'license': 'MIT', 'n_tokens_mistral': 7353, 'n_tokens_neox': 6853, 'n_words': 1776, 'path': '04_Classifying_spectra_using_FNN.ipynb', 'repo_name': 'SoumyaShreeram/MLGravitationalLensing', 'size': '19787'}