<jupyter_start><jupyter_text>Recurrent Neural Network (RNN) for NLP--- Outline of Document1. RNN Model with LSTM2. Model Cross Validation3. Evaluation Metrics4. Pros and Cons of Model  RNN Model with LSTM  Download Required Libraries<jupyter_code>!pip install tensorflow
!pip install imblearn
!pip install keras==2.6.0<jupyter_output>Requirement already satisfied: tensorflow in c:\python38\lib\site-packages (2.6.0)
Requirement already satisfied: absl-py~=0.10 in c:\python38\lib\site-packages (from tensorflow) (0.14.0)
Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\python38\lib\site-packages (from tensorflow) (1.1.2)
Requirement already satisfied: numpy~=1.19.2 in c:\python38\lib\site-packages (from tensorflow) (1.19.5)
Requirement already satisfied: six~=1.15.0 in c:\python38\lib\site-packages (from tensorflow) (1.15.0)
Requirement already satisfied: gast==0.4.0 in c:\python38\lib\site-packages (from tensorflow) (0.4.0)
Requirement already satisfied: opt-einsum~=3.3.0 in c:\python38\lib\site-packages (from tensorflow) (3.3.0)
Requirement already satisfied: tensorflow-estimator~=2.6 in c:\python38\lib\site-packages (from tensorflow) (2.6.0)
Requirement already satisfied: tensorboard~=2.6 in c:\python38\lib\site-packages (from tensorflow) (2.6.0)
Requirement already satisfied: protobuf>=3.9.2 in c:\p[...]<jupyter_text>Import Required Libraries<jupyter_code>import pandas as pd
import numpy as np
import csv, datetime, time, json
import matplotlib.pyplot as plt
import seaborn as sns
import pickle
from zipfile import ZipFile
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Model
from keras.layers import Input, TimeDistributed, Dense, Lambda, concatenate, Dropout, BatchNormalization, LSTM
from keras.layers.merge import Concatenate
from keras.layers.embeddings import Embedding
from keras.utils.vis_utils import plot_model
from keras.regularizers import l2
from keras import backend as K
from keras.callbacks import Callback, ModelCheckpoint
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from imblearn.under_sampling import RandomUnderSampler<jupyter_output><empty_output><jupyter_text>Global Constants<jupyter_code>MAX_QN_LEN = 219
MAX_WORDS = 20000
EMBED_DIM = 300 #GloVe is 300dim
MAX_SEQUENCE = 25
TRAIN_TEST = 0.1
VALIDATION = 0.1
SEED = 42
NUM_EPOCHS = 50
DROPOUT = 0.1
BATCH_SIZE = 32  #Is the default value
REGULARIZER = 0.01
OPTIMIZER = 'adam'

BASE_MODEL_WEIGHTS_FILE = "question_pairs_weights.h5" #Will be saved to a new file if this file didnt exist before
MODEL_WEIGHTS_FILE_MLSTM = "question_pairs_weights_MLSTM.h5"
MODEL_WEIGHTS_FILE_MLSTM_FEATURE = "question_pairs_weights_MLSTM_w_Features.h5"<jupyter_output><empty_output><jupyter_text>Processing Data<jupyter_code>%cd "C:\Users\luciu\Desktop\NUS Lecture Notes\Y3S1\CS3244\Project"
train_set = pd.read_csv('train_set_with_features.csv', index_col=[0])
train_set.head(5)
Y_labels = train_set["is_duplicate"]
X_features = train_set.drop("is_duplicate", axis=1)

X_train, X_test, y_train, y_test = train_test_split(X_features, Y_labels, test_size=TRAIN_TEST, random_state=SEED)

#print(y_train_full.value_counts())

#undersample = RandomUnderSampler(sampling_strategy='majority')
# undersample = RandomUnderSampler(replacement=True)
# X_train, y_train = undersample.fit_resample(X_train_full, y_train_full)

print(y_train.value_counts())
X_train_vector = X_train[['question1', 'question2']]
X_train_vector['question2'] = X_train_vector['question2'].astype(str)
X_train_vector.head(5)
X_train_features = X_train.drop(['question1', 'question2', 'qid1', 'qid2', 'id'], axis=1)
X_train_features.head(5)
X_test_vector = X_test[['question1', 'question2']]
X_test_vector['question2'] = X_test_vector['question2'].astype(str)
X_test_vector.head(5)
X_test_features = X_test.drop(['question1', 'question2', 'qid1', 'qid2', 'id'], axis=1)
X_test_features.head(5)
X_test_features.columns<jupyter_output><empty_output><jupyter_text>Tokenizer creates a keras.tokenizer object which will convert the list of strings that a user pushes in into a list of [tokens]. Based on MAX_NUMBER_OF_WORDS, the tokenizer will cut the number of tokens to that size if needed. In this case, since I put 20k as the limit, we will not be cutting anything. text_to_sequence fits the list of sentences into list of tokens. word_index provides the total number of words in this object<jupyter_code>questions = X_train_vector['question1'].tolist() + X_train_vector['question2'].tolist()
tokenizer = Tokenizer(num_words=MAX_WORDS)
tokenizer.fit_on_texts(questions)
#Training Set
question1_train = tokenizer.texts_to_sequences(X_train_vector['question1'].tolist())
question2_train = tokenizer.texts_to_sequences(X_train_vector['question2'].tolist())

#Testing Set
question1_test = tokenizer.texts_to_sequences(X_test_vector['question1'].tolist())
question2_test = tokenizer.texts_to_sequences(X_test_vector['question2'].tolist())

word_index = tokenizer.word_index
print("Words in index: %d" % len(word_index))
import pickle

# saving
with open('tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)<jupyter_output><empty_output><jupyter_text>glove.840b.300d.txt is a 5gb pretrained word vector file by GloVe. The format of the vectors in the txt file should be tuples of word and value. eg. (Dog, 0.12345) (Cat, 0.12000) etc. The code below splits the strings into tuples then forms a dictionary of dict[word] = value for every word in the pretrained model<jupyter_code>embeddings_index = {}
with open('glove.840b.300d.txt', encoding='utf-8') as f:
  for line in f:
    values = line.split(' ')
    word = values[0]
    embedding = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = embedding
print('Word embeddings: %d' % len(embeddings_index))<jupyter_output>Word embeddings: 2196017<jupyter_text>In the function below, The number of words is shrunk to the min of MAX_NUMBER_OF_WORDS vs len(word_index). In this case, the number of words = MAX_NUMBER OF WORDS = 20kFor each word in word_index, we take the first 20k words from the our pretrained corpusand try to tag each word from word_index (Which are words from our questions)to the corpus.This seems funny since we will expect to get gaps in the matrix whenthe word does not map to any vector in the first 20k of the corpus.Hence, when the embedding is done, we print the Null word embeddings (indicating the gap) and we see that it is only 435/20000 which can be accepted.More tuning can be done here to increase/decrease the number of words<jupyter_code>number_of_words = min(MAX_WORDS, len(word_index))
word_embedding_matrix = np.zeros((number_of_words + 1, EMBED_DIM))
for word, i in word_index.items():
  if i > MAX_WORDS:
    continue
  embedding_vector = embeddings_index.get(word)
  if embedding_vector is not None:
    word_embedding_matrix[i] = embedding_vector
  
print('Null word embeddings: %d' % np.sum(np.sum(word_embedding_matrix, axis=1) == 0))
np.savetxt("word_embedding.csv", word_embedding_matrix, delimiter=",")<jupyter_output><empty_output><jupyter_text>For LSTM, the length of our q1 and q2 has to be the same. Therefore, we pad them to an equal length of MAX_SEQUENCE_LENGTH which has been set to 25<jupyter_code>q1_train_data = pad_sequences(question1_train, maxlen=MAX_SEQUENCE)
q2_train_data = pad_sequences(question2_train, maxlen=MAX_SEQUENCE)

q1_test_data =  pad_sequences(question1_test, maxlen=MAX_SEQUENCE)
q2_test_data =  pad_sequences(question2_test, maxlen=MAX_SEQUENCE)

print('Shape of question1 train data tensor:', q1_train_data.shape)
print('Shape of question2 train data tensor:', q2_train_data.shape)

print('Shape of question1 test data tensor:', q1_test_data.shape)
print('Shape of question2 test data tensor:', q1_test_data.shape)

print('Shape of label tensor:', Y_labels.shape)<jupyter_output>Shape of question1 train data tensor: (363839, 25)
Shape of question2 train data tensor: (363839, 25)
Shape of question1 test data tensor: (40427, 25)
Shape of question2 test data tensor: (40427, 25)
Shape of label tensor: (404266,)<jupyter_text>Baseline Model  Training the model<jupyter_code>question1_train = Input(shape=(MAX_SEQUENCE,)) 
question2_train = Input(shape=(MAX_SEQUENCE,))  
question1_test = Input(shape=(MAX_SEQUENCE,))
question2_test = Input(shape=(MAX_SEQUENCE,))<jupyter_output><empty_output><jupyter_text>In the two functions below, several Keras functions are used. Embedding(  1. input_dim = (size of vocab list (20k),  2. output_dim = (Length of vector of each word (300),  3. weights = The pretrained matrix that we are using,  4. input_length = Our padded length (25),  5. trainable = False (prevents our pretrained weights from being updated)  )This creates an Embedding object which should take in a sequences of integers [basically list of integers (index of the words)], ensuring that the sequences have the same length. The embedding layer then maps the integer inputs to the vectors found in the embedding matrix in the weights. This creates a 3D matrix of (samples, sequence_length, embedding_dim)Dense(  1. units = Output Dimensions (300)  2. activation = 'relu' refers to rectified linear unit activation function which helps to speed up training and do some stuff (IDK I will find out soon))This creates an Output object that is shrunk to the matrix size of units, which in this case is EMBEDDING_DIMENSIONS which is 300 (No change)TimeDistributed(1. Layer (which for now is Dense(Embedding))2. q1 (which will be the inputs to feed into this layers for data preparation)TimeDistributed will feed the inputs of q1 or q2 into the layer which prepares each input with the given layer (Dense(Embedding)). In other words. TimeDistributed Layer will apply the same layer to several inputs and each input will produce 1 output. K (keras.backend)1. x is a tensor or variable2. axis = 1 refers to the column of the matrix K is keras backend (imported as K) which provides a list of ultilites such as Max, Min etc.In the lambda function, K.max converts each tensor (matrix) into the column of the maximum x value.(Why is there a need to change to Max of columns though)Lambda1. lambda = The conversion from x to max of x2. output dimensions = expected output shapeThis is a layer to implement the lambda conversion of x to max of x. (using K.max).<jupyter_code>q1 = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(question1_train) 
q1 = TimeDistributed(Dense(EMBED_DIM, activation='relu'))(q1) 
q1 = Lambda(lambda x: K.max(x, axis=1), output_shape=(EMBED_DIM,))(q1) 
q2 = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(question2_train)
q2 = TimeDistributed(Dense(EMBED_DIM, activation='relu'))(q2)
q2 = Lambda(lambda x: K.max(x, axis=1), output_shape=(EMBED_DIM,))(q2)<jupyter_output><empty_output><jupyter_text>Dense (See above)Dropout1. droput rate = 0.1 (DROPOUT Global constant)Dropout Layer randomly sets input units to 0 with a frequency of dropout rate during each step. This is to randomly drop the noise which will help to prevent overfitting. BatchNormalization1. No argumentsBatchNormalization applies a transofrmation that maintains the mean output close to 0 and output standard deviation close to 1. This will help to solve internal covariate shift which happens when training our neural network which results in the change in weights of each neuron. The change in weights will cause the distribution of input data for each layer to be different in each iteration which will slow down learning.Normalizing the weights will help to keep the mean output to 0 and standard deviation close to 1 which will reduce covariate shift. Apply 4 layers of training<jupyter_code>inputs = concatenate([q1, q2])
inputs = Dense(70, activation='relu')(inputs)     #First
inputs = Dropout(DROPOUT)(inputs)
inputs = BatchNormalization()(inputs)
inputs = Dense(150, activation='relu')(inputs)     #Second
inputs = Dropout(DROPOUT)(inputs)
inputs = BatchNormalization()(inputs)
inputs = Dense(70, activation='relu')(inputs)     #Third
inputs = Dropout(DROPOUT)(inputs)
inputs = BatchNormalization()(inputs)
inputs = Dense(35, activation='relu')(inputs)     #Fourth
inputs = Dropout(DROPOUT)(inputs)
inputs = BatchNormalization()(inputs)<jupyter_output><empty_output><jupyter_text>Create a compressed output of each tensor as single value output using the Dense layer(1...) which means 1 dimension eg 1 valueSigmoid is used for logistic regression to linearly classify the output.<jupyter_code>final_output = Dense(1, activation='sigmoid')(inputs)
model = Model(inputs=[question1_train, question2_train], outputs=final_output)
model.compile(loss='binary_crossentropy', optimizer=OPTIMIZER, metrics=['accuracy'])
model.summary()<jupyter_output>Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 25)]         0                                            
__________________________________________________________________________________________________
input_2 (InputLayer)            [(None, 25)]         0                                            
__________________________________________________________________________________________________
embedding (Embedding)           (None, 25, 300)      6000300     input_1[0][0]                    
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 25, 300)      6000300     input_2[0][0]                [...]<jupyter_text>Using the prepared layers, now feed in the training questions and their original labels.<jupyter_code>print("Starting training at", datetime.datetime.now())
t0 = time.time()
callbacks = [ModelCheckpoint(MODEL_WEIGHTS_FILE, monitor='val_accuracy', save_best_only=True)]
history = model.fit([q1_train_data, q2_train_data], 
                    y_train, 
                    epochs=NUM_EPOCHS,
                    validation_split=VALIDATION,
                    verbose=1,
                    batch_size=BATCH_SIZE,
                    callbacks=callbacks)

t1 = time.time()
print("Training ended at", datetime.datetime.now())
print("Minutes elapsed: %f" % ((t1 - t0) / 60.))
max_val_acc, idx = max((val, idx) for (idx, val) in enumerate(history.history['val_accuracy']))
print('Maximum validation accuracy = {0:.4f} (epoch {1:d})'.format(max_val_acc, idx+1))<jupyter_output><empty_output><jupyter_text>Testing the model<jupyter_code>model.load_weights(BASE_MODEL_WEIGHTS_FILE)
loss, accuracy = model.evaluate([q1_test_data, q2_test_data], y_test, verbose=0)
print('Test loss = {0:.4f}, test accuracy = {1:.4f}'.format(loss, accuracy))<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>print(history.history['accuracy'])
print(history.history['loss'])
print(history.history['val_accuracy'])
print(history.history['val_loss'])
plt.plot(history.history['loss'], color ='blue');
plt.plot(history.history['val_loss'], color='red');
plt.title('Training Loss Vs Validation Loss');
plt.xlabel('Epochs');
plt.ylabel('Loss');
plt.plot(history.history['accuracy'], color ='blue');
plt.plot(history.history['val_accuracy'], color='red');
plt.title('Training Accuracy Vs Validation Accuracy');
plt.xlabel('Epochs');
plt.ylabel('Accuracy');
incorrects = model.predict([q1_test_data, q2_test_data], verbose=1)
incorrects[incorrects > 0.5] = 1
incorrects[incorrects <= 0.5] = 0
flattened_incorrect = incorrects.flatten()
heatmap_data = confusion_matrix(y_test, flattened_incorrect)
group_names = ["True Neg","False Pos","False Neg","True Pos"]
group_counts = ["{0}".format(value) for value in heatmap_data.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in heatmap_data.flatten()/np.sum(heatmap_data)]
labels = [f"{group_name}\n{group_count}\n{group_percentage}" for group_name, group_count, group_percentage in zip(group_names,group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)

sns.heatmap(heatmap_data, annot=labels, fmt="", cmap='Blues')
tn, fp, fn, tp = confusion_matrix(y_test, flattened_incorrect).ravel()
Precision = tp / (tp + fp)
Accuracy = (tp + tn) / (tp + tn + fp + fn)
Recall = tp / (tp + fn)
F1 = 2 / ((1/Recall) + (1/Precision))

print("Precision: " + str(Precision) + "\n" + "Accuracy: " + str(Accuracy) + "\n" + "Recall: " + str(Recall) + "\n" + "F1: " + str(F1))
model.save("BASELINE_MODEL.h5")<jupyter_output><empty_output><jupyter_text>Using Siamnese LSTM with Manhatten Distance  Training the Model<jupyter_code># NEW MODEL TRYING TO USE SIAMNESE LSTM
def exponent_neg_manhattan_distance(left, right):
    return K.exp(-K.sum(K.abs(left-right), axis=1, keepdims=True))


q1_input = Input(shape=(MAX_SEQUENCE,))
q2_input = Input(shape=(MAX_SEQUENCE,))

LSTM_DIM = 128
LSTM_DROPOUT = 0.10
LSTM_REGULARIZATION = 0.00

q1_LSTM = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(q1_input)
q2_LSTM = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(q2_input)

manhatten_LSTM = LSTM(LSTM_DIM, kernel_regularizer=l2(LSTM_REGULARIZATION), dropout=LSTM_DROPOUT, recurrent_dropout=LSTM_DROPOUT)
q1_LSTM_Output = manhatten_LSTM(q1_LSTM)
q2_LSTM_Output = manhatten_LSTM(q2_LSTM)

malstm_distance = Lambda(function=lambda x: exponent_neg_manhattan_distance(x[0], x[1]),output_shape=lambda x: (x[0][0], 1))([q1_LSTM_Output, q2_LSTM_Output])


malstm = Model([q1_input, q2_input], [malstm_distance])
malstm.compile(loss='mean_squared_error', optimizer=OPTIMIZER, metrics=['accuracy'])
malstm.summary()
print("Starting training at", datetime.datetime.now())
t0 = time.time()

callbacks_MLSTM = [ModelCheckpoint(MODEL_WEIGHTS_FILE_MLSTM, monitor='val_accuracy', save_best_only=True)]
malstm_trained = malstm.fit([q1_train_data, q2_train_data],
                            y_train,
                            batch_size=BATCH_SIZE,
                            epochs=NUM_EPOCHS,
                            validation_split=VALIDATION,
                            verbose = 1,
                            callbacks = callbacks_MLSTM)

t1 = time.time()
print("Training ended at", datetime.datetime.now())
print("Minutes elapsed: %f" % ((t1 - t0) / 60.))
max_val_acc, idx = max((val, idx) for (idx, val) in enumerate(malstm_trained.history['val_accuracy']))
print('Maximum validation accuracy = {0:.4f} (epoch {1:d})'.format(max_val_acc, idx+1))<jupyter_output><empty_output><jupyter_text>Testing the Model<jupyter_code>malstm.load_weights(MODEL_WEIGHTS_FILE_MLSTM)
loss, accuracy = malstm.evaluate([q1_test_data, q2_test_data], y_test, verbose=1)
print('Test loss = {0:.4f}, test accuracy = {1:.4f}'.format(loss, accuracy))<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>plt.plot(malstm_trained.history['loss'], color ='blue');
plt.plot(malstm_trained.history['val_loss'], color='red');
plt.title('Training Loss Vs Validation Loss');
plt.xlabel('Epochs');
plt.ylabel('Loss');
plt.plot(malstm_trained.history['accuracy'], color ='blue');
plt.plot(malstm_trained.history['val_accuracy'], color='red');
plt.title('Training Accuracy Vs Validation Accuracy');
plt.xlabel('Epochs');
plt.ylabel('Accuracy');
incorrects = malstm.predict([q1_test_data, q2_test_data], verbose=1)
incorrects[incorrects > 0.5] = 1
incorrects[incorrects <= 0.5] = 0
flattened_incorrect = incorrects.flatten()
heatmap_data = confusion_matrix(y_test, flattened_incorrect)
group_names = ["True Neg","False Pos","False Neg","True Pos"]
group_counts = ["{0}".format(value) for value in heatmap_data.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in heatmap_data.flatten()/np.sum(heatmap_data)]
labels = [f"{name}\n{count}\n{percentage}" for name, count, percentage in zip(group_names,group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)

sns.heatmap(heatmap_data, annot=labels, fmt="", cmap='Blues')
tn, fp, fn, tp = confusion_matrix(y_test, flattened_incorrect).ravel()
Precision = tp / (tp + fp)
Accuracy = (tp + tn) / (tp + tn + fp + fn)
Recall = tp / (tp + fn)
F1 = 2 / ((1/Recall) + (1/Precision))

print(f"Precision: {Precision} \nAccuracy: {Accuracy} \nRecall: {Recall} \nF1: {F1}")
malstm.save("MALTSM_No_Features.h5")<jupyter_output><empty_output><jupyter_text>Using Siamnese LSTM with Features Training the Model<jupyter_code>def exponent_neg_manhattan_distance(left, right):
    return K.exp(-K.sum(K.abs(left-right), axis=1, keepdims=True))


q1_input = Input(shape=(MAX_SEQUENCE,))  #25-dim vectors 
q2_input = Input(shape=(MAX_SEQUENCE,))  #25-dim vectors

LSTM_DIM = 128
LSTM_DROPOUT = 0.10
LSTM_REGULARIZATION = 0.0001

q1_LSTM = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(q1_input)  #Creates an Embedding Object and feeds question1 into it
q2_LSTM = Embedding(number_of_words + 1,
               EMBED_DIM,
               weights=[word_embedding_matrix],
               input_length=MAX_SEQUENCE,
               trainable=False)(q2_input)


manhatten_LSTM = LSTM(LSTM_DIM, kernel_regularizer=l2(LSTM_REGULARIZATION), dropout=LSTM_DROPOUT, recurrent_dropout=LSTM_DROPOUT)
q1_LSTM_Output = manhatten_LSTM(q1_LSTM)
q2_LSTM_Output = manhatten_LSTM(q2_LSTM)

malstm_distance = Lambda(function=lambda x: exponent_neg_manhattan_distance(x[0], x[1]),output_shape=lambda x: (x[0][0], 1))([q1_LSTM_Output, q2_LSTM_Output])  #What is x[0][0]??

no_of_features = len(X_train_features.columns)
feature_input = Input(shape=(no_of_features,))
feature_layer = Dense(70, activation='relu')(feature_input)

concat_layer = Concatenate()([q1_LSTM_Output, q2_LSTM_Output, feature_layer])
concat_layer = Dropout(DROPOUT)(concat_layer)
concat_layer = BatchNormalization()(concat_layer)

concat_layer = Dense(150, kernel_regularizer=l2(REGULARIZER), activation='relu')(concat_layer)
concat_layer = Dropout(DROPOUT)(concat_layer)
concat_layer = BatchNormalization()(concat_layer)

concat_layer = Dense(70, kernel_regularizer=l2(REGULARIZER), activation='relu')(concat_layer)
concat_layer = Dropout(DROPOUT)(concat_layer)
concat_layer = BatchNormalization()(concat_layer)

concat_layer = Dense(35, kernel_regularizer=l2(REGULARIZER), activation='relu')(concat_layer)
concat_layer = Dropout(DROPOUT)(concat_layer)
concat_layer = BatchNormalization()(concat_layer)

output = Dense(1, activation='sigmoid')(concat_layer)

malstm_w_feature = Model([q1_input, q2_input, feature_input], [output])
malstm_w_feature.compile(loss='binary_crossentropy', optimizer=OPTIMIZER, metrics=['accuracy'])
malstm_w_feature.summary()
print("Starting training at", datetime.datetime.now())
t0 = time.time()

callbacks_MLSTM_feature = [ModelCheckpoint(MODEL_WEIGHTS_FILE_MLSTM_FEATURE, monitor='val_accuracy', save_best_only=True)]
malstm_w_feature_trained = malstm_w_feature.fit([q1_train_data, q2_train_data, X_train_features],
                            y_train,
                            batch_size=BATCH_SIZE,
                            epochs=NUM_EPOCHS,
                            validation_split=VALIDATION,
                            verbose = 1,
                            callbacks = callbacks_MLSTM_feature)

t1 = time.time()
print("Training ended at", datetime.datetime.now())
print("Minutes elapsed: %f" % ((t1 - t0) / 60.))
max_val_acc, idx = max((val, idx) for (idx, val) in enumerate(malstm_w_feature_trained.history['val_accuracy']))
print('Maximum validation accuracy = {0:.4f} (epoch {1:d})'.format(max_val_acc, idx+1))<jupyter_output><empty_output><jupyter_text>Testing the Model<jupyter_code>malstm_w_feature.load_weights(MODEL_WEIGHTS_FILE_MLSTM_FEATURE)
loss, accuracy = malstm_w_feature.evaluate([q1_test_data, q2_test_data, X_test_features], y_test, verbose=1)
print('Test loss = {0:.4f}, test accuracy = {1:.4f}'.format(loss, accuracy))<jupyter_output><empty_output><jupyter_text>Evaluation Metrics<jupyter_code>incorrects = malstm_w_feature.predict([q1_test_data, q2_test_data, X_test_features], verbose=1)
incorrects[incorrects > 0.5] = 1
incorrects[incorrects <= 0.5] = 0
flattened_incorrect = incorrects.flatten()
heatmap_data = confusion_matrix(y_test, flattened_incorrect)
group_names = ["True Neg","False Pos","False Neg","True Pos"]
group_counts = ["{0}".format(value) for value in heatmap_data.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in heatmap_data.flatten()/np.sum(heatmap_data)]
labels = [f"{name}\n{count}\n{percentage}" for name, count, percentage in zip(group_names,group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)

sns.heatmap(heatmap_data, annot=labels, fmt="", cmap='Blues')
tn, fp, fn, tp = confusion_matrix(y_test, flattened_incorrect).ravel()
Precision = tp / (tp + fp)
Accuracy = (tp + tn) / (tp + tn + fp + fn)
Recall = tp / (tp + fn)
F1 = 2 / ((1/Recall) + (1/Precision))

print(f"Precision: {Precision} \nAccuracy: {Accuracy} \nRecall: {Recall} \nF1: {F1}")
malstm_w_feature.save("MALTSM_w_features.h5")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11582293201023335, 'fraction_numerical': 0.02337390495387239, 'mean_word_length': 5.475652610441767, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '471508', 'license': 'MIT', 'n_tokens_mistral': 8985, 'n_tokens_neox': 8512, 'n_words': 2068, 'path': 'Source Codes/Recurrent Neural Network.ipynb', 'repo_name': 'luajunan/CS3244-Quora-Semantics-Project', 'size': '25798'}