<jupyter_start><jupyter_text>MLP Classification with TREC DatasetWe will build a text classification model using MLP model on the TREC Dataset.  Load the library<jupyter_code>import tensorflow as tf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import re
import nltk
import random
from nltk.corpus import stopwords, twitter_samples
# from nltk.tokenize import TweetTokenizer
from sklearn.model_selection import KFold
from nltk.stem import PorterStemmer
from string import punctuation
from sklearn.preprocessing import OneHotEncoder
from tensorflow.keras.preprocessing.text import Tokenizer
import time

%config IPCompleter.greedy=True
%config IPCompleter.use_jedi=False
# nltk.download('twitter_samples')<jupyter_output><empty_output><jupyter_text>Load the Dataset<jupyter_code>corpus = pd.read_pickle('../../../0_data/TREC/TREC.pkl')
corpus.label = corpus.label.astype(int)
print(corpus.shape)
corpus
corpus.info()
corpus.groupby( by=['split','label']).count()
corpus.groupby(by='split').count()
# Separate the sentences and the labels
# Separate the sentences and the labels for training and testing
train_x = list(corpus[corpus.split=='train'].sentence)
train_y = np.array(corpus[corpus.split=='train'].label)
print(len(train_x))
print(len(train_y))

test_x = list(corpus[corpus.split=='test'].sentence)
test_y = np.array(corpus[corpus.split=='test'].label)
print(len(test_x))
print(len(test_y))<jupyter_output>5452
5452
500
500<jupyter_text>Data PreprocessingPreparing data for word embedding, especially for pre-trained word embedding like Word2Vec or GloVe, __don't use standard preprocessing steps like stemming or stopword removal__. Compared to our approach on cleaning the text when doing word count based feature extraction (e.g. TFIDF) such as removing stopwords, stemming etc, now we will keep these words as we do not want to lose such information that might help the model learn better.__<NAME>__, one of the developers of Word2Vec, in _word2vec-toolkit: google groups thread., 2015_, suggests only very minimal text cleaning is required when learning a word embedding model. Sometimes, it's good to disconnectIn short, what we will do is:- Puntuations removal- Lower the letter case- TokenizationThe process above will be handled by __Tokenizer__ class in TensorFlow- One way to choose the maximum sequence length is to just pick the length of the longest sentence in the training set. Develop VocabularyA part of preparing text for text classification involves defining and tailoring the vocabulary of words supported by the model. **We can do this by loading all of the documents in the dataset and building a set of words.**The larger the vocabulary, the more sparse the representation of each word or document. So, we may decide to support all of these words, or perhaps discard some. The final chosen vocabulary can then be saved to a file for later use, such as filtering words in new documents in the future.<jupyter_code># Define a function to compute the max length of sequence
def max_length(sequences):
    '''
    input:
        sequences: a 2D list of integer sequences
    output:
        max_length: the max length of the sequences
    '''
    max_length = 0
    for i, seq in enumerate(sequences):
        length = len(seq)
        if max_length < length:
            max_length = length
    return max_length
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"

# Separate the sentences and the labels
train_x = list(corpus[corpus.split=='train'].sentence)
train_y = np.array(corpus[corpus.split=='train'].label)
test_x = list(corpus[corpus.split=='test'].sentence)
test_y = np.array(corpus[corpus.split=='test'].label)

# Cleaning and Tokenization
tokenizer = Tokenizer(oov_token=oov_tok)
tokenizer.fit_on_texts(train_x)

print("Example of sentence: ", train_x[4])

# Turn the text into sequence
training_sequences = tokenizer.texts_to_sequences(train_x)
max_len = max_length(training_sequences)

print('Into a sequence of int:', training_sequences[4])

# Pad the sequence to have the same size
training_padded = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
print('Into a padded sequence:', training_padded[4])
# See the first 10 words in the vocabulary

word_index = tokenizer.word_index
for i, word in enumerate(word_index):
    print(word, word_index.get(word))
    if i==9:
        break
vocab_size = len(word_index)+1
print(vocab_size)<jupyter_output><UNK> 1
the 2
what 3
is 4
of 5
in 6
a 7
how 8
's 9
was 10
8461<jupyter_text>Model 1: Embedding RandomA __standard model__ for document classification is to use (quoted from __<NAME>__, the author of [machinelearningmastery.com](https://machinelearningmastery.com)):>- Word Embedding: A distributed representation of words where different words that have a similar meaning (based on their usage) also have a similar representation.>- Convolutional Model: A feature extraction model that learns to extract salient features from documents represented using a word embedding.>- Fully Connected Model: The interpretation of extracted features in terms of a predictive output.Therefore, the model is comprised of the following elements:- __Input layer__ that defines the length of input sequences.- __Embedding layer__ set to the size of the vocabulary and 100-dimensional real-valued representations.- __Conv1D layer__ with 32 filters and a kernel size set to the number of words to read at once.- __MaxPooling1D layer__ to consolidate the output from the convolutional layer.- __Flatten layer__ to reduce the three-dimensional output to two dimensional for concatenation.The CNN model is inspired by __Yoon Kim__ paper in his study on the use of Word Embedding + CNN for text classification. The hyperparameters we use based on his study are as follows:- Transfer function: rectified linear.- Kernel sizes: 1-8.- Number of filters: 100.- Dropout rate: 0.5.- L2 Constraint: 3.- Batch Size: 50.- Update Rule: AdamWe will perform the best parameter using __grid search__ and 10-fold cross validation.  CNN ModelNow, we will build Convolutional Neural Network (CNN) models to classify encoded documents as either positive or negative.The model takes inspiration from `CNN for Sentence Classification` by *<NAME>*.Now, we will define our CNN model as follows:- One Conv layer with 100 filters, kernel size 5, and relu activation function;- One MaxPool layer with pool size = 2;- One Dropout layer after flattened;- Optimizer: Adam (The best learning algorithm so far)- Loss function: binary cross-entropy (suited for binary classification problem)**Note**: - The whole purpose of dropout layers is to tackle the problem of over-fitting and to introduce generalization to the model. Hence it is advisable to keep dropout parameter near 0.5 in hidden layers. - https://missinglink.ai/guides/keras/keras-conv1d-working-1d-convolutional-neural-networks-keras/<jupyter_code>from tensorflow.keras import regularizers
from tensorflow.keras.constraints import MaxNorm

def define_model(filters = 100, kernel_size = 3, activation='relu', input_dim = None, output_dim=300, max_length = None ):
    
    model = tf.keras.models.Sequential([
        tf.keras.layers.Embedding(input_dim=vocab_size, 
                                  output_dim=output_dim, 
                                  input_length=max_length, 
                                  input_shape=(max_length, )),
        
        tf.keras.layers.Conv1D(filters=filters, kernel_size = kernel_size, activation = activation, 
                               # set 'axis' value to the first and second axis of conv1D weights (rows, cols)
                               kernel_constraint= MaxNorm( max_value=3, axis=[0,1])),
        
        tf.keras.layers.MaxPool1D(2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(10, activation=activation, 
                              # set axis to 0 to constrain each weight vector of length (input_dim,) in dense layer
                              kernel_constraint = MaxNorm( max_value=3, axis=0)),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(units=6, activation='softmax')
    ])
    
    model.compile( loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
#     model.summary()
    return model
model_0 = define_model( input_dim=1000, max_length=100)
model_0.summary()
class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') > 0.93):
            print("\nReached 93% accuracy so cancelling training!")
            self.model.stop_training=True


callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=20, verbose=2, 
                                             mode='auto', restore_best_weights=True)
# callbacks = myCallback()<jupyter_output><empty_output><jupyter_text>Train and Test the Model<jupyter_code># Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu', 'tanh']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6]

columns = ['Activation', 'Filters', 'Acc']
record = pd.DataFrame(columns = columns)

# Separate the sentences and the labels
train_x = list(corpus[corpus.split=='train'].sentence)
train_y = np.array(corpus[corpus.split=='train'].label)
test_x = list(corpus[corpus.split=='test'].sentence)
test_y = np.array(corpus[corpus.split=='test'].label)

for activation in activations:

    for kernel_size in kernel_sizes:

        # encode data using
        # Cleaning and Tokenization
        tokenizer = Tokenizer(oov_token=oov_tok)
        tokenizer.fit_on_texts(train_x)

        # Turn the text into sequence
        training_sequences = tokenizer.texts_to_sequences(train_x)
        test_sequences = tokenizer.texts_to_sequences(test_x)

        max_len = max_length(training_sequences)

        # Pad the sequence to have the same size
        Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
        Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

        word_index = tokenizer.word_index
        vocab_size = len(word_index)+1

        # Define the input shape
        model = define_model(filters, kernel_size, activation, input_dim=vocab_size, max_length=max_len)

        # Train the model
        model.fit(Xtrain, train_y, batch_size=50, epochs=60, verbose=2, 
                  callbacks=[callbacks], validation_data=(Xtest, test_y))

        # evaluate the model
        loss, acc = model.evaluate(Xtest, test_y, verbose=0)
        print('Test Accuracy: {}'.format(acc*100))

        parameters = [activation, kernel_size]
        entries = parameters + [acc]

        temp = pd.DataFrame([entries], columns=columns)
        record = record.append(temp, ignore_index=True)
        print()
        print(record)
        print()<jupyter_output>Epoch 1/60
110/110 - 4s - loss: 1.5717 - accuracy: 0.3254 - val_loss: 1.2709 - val_accuracy: 0.5900
Epoch 2/60
110/110 - 4s - loss: 1.1928 - accuracy: 0.5072 - val_loss: 0.9443 - val_accuracy: 0.6980
Epoch 3/60
110/110 - 4s - loss: 0.9403 - accuracy: 0.6286 - val_loss: 0.7039 - val_accuracy: 0.8140
Epoch 4/60
110/110 - 4s - loss: 0.7102 - accuracy: 0.7208 - val_loss: 0.5478 - val_accuracy: 0.8620
Epoch 5/60
110/110 - 4s - loss: 0.5936 - accuracy: 0.7619 - val_loss: 0.5087 - val_accuracy: 0.8600
Epoch 6/60
110/110 - 4s - loss: 0.5077 - accuracy: 0.7876 - val_loss: 0.4671 - val_accuracy: 0.8560
Epoch 7/60
110/110 - 4s - loss: 0.4710 - accuracy: 0.7997 - val_loss: 0.4799 - val_accuracy: 0.8740
Epoch 8/60
110/110 - 4s - loss: 0.4430 - accuracy: 0.8138 - val_loss: 0.5056 - val_accuracy: 0.8800
Epoch 9/60
110/110 - 4s - loss: 0.4102 - accuracy: 0.8278 - val_loss: 0.4895 - val_accuracy: 0.8580
Epoch 10/60
110/110 - 4s - loss: 0.4020 - accuracy: 0.8245 - val_loss: 0.5055 - val_accuracy: 0.8500[...]<jupyter_text>Summary<jupyter_code>record.sort_values(by='Acc', ascending=False)
report = record.sort_values(by='Acc', ascending=False)
report = report.to_excel('CNN_TREC.xlsx', sheet_name='random')<jupyter_output><empty_output><jupyter_text>Model 2: Word2Vec Static __Using and updating pre-trained embeddings__* In this part, we will create an Embedding layer in Tensorflow Keras using a pre-trained word embedding called Word2Vec 300-d tht has been trained 100 bilion words from Google News.* In this part,  we will leave the embeddings fixed instead of updating them (dynamic). 1. __Load `Word2Vec` Pre-trained Word Embedding__<jupyter_code>from gensim.models import KeyedVectors
word2vec = KeyedVectors.load_word2vec_format('../GoogleNews-vectors-negative300.bin', binary=True)
# Access the dense vector value for the word 'handsome'
# word2vec.word_vec('handsome') # 0.11376953
word2vec.word_vec('cool') # 1.64062500e-01<jupyter_output><empty_output><jupyter_text>2. __Check number of training words present in Word2Vec__<jupyter_code>def training_words_in_word2vector(word_to_vec_map, word_to_index):
    '''
    input:
        word_to_vec_map: a word2vec GoogleNews-vectors-negative300.bin model loaded using gensim.models
        word_to_index: word to index mapping from training set
    '''
    
    vocab_size = len(word_to_index) + 1
    count = 0
    # Set each row "idx" of the embedding matrix to be 
    # the word vector representation of the idx'th word of the vocabulary
    for word, idx in word_to_index.items():
        if word in word_to_vec_map:
            count+=1
            
    return print('Found {} words present from {} training vocabulary in the set of pre-trained word vector'.format(count, vocab_size))
# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)

# Cleaning and Tokenization
tokenizer = Tokenizer(oov_token=oov_tok)
tokenizer.fit_on_texts(sentences)

word_index = tokenizer.word_index
training_words_in_word2vector(word2vec, word_index)<jupyter_output>Found 7526 words present from 8761 training vocabulary in the set of pre-trained word vector<jupyter_text>2. __Define a `pretrained_embedding_layer` function__<jupyter_code>emb_mean = word2vec.vectors.mean()
emb_std = word2vec.vectors.std()
print('emb_mean: ', emb_mean)
print('emb_std: ', emb_std)
from tensorflow.keras.layers import Embedding

def pretrained_embedding_matrix(word_to_vec_map, word_to_index, emb_mean, emb_std):
    '''
    input:
        word_to_vec_map: a word2vec GoogleNews-vectors-negative300.bin model loaded using gensim.models
        word_to_index: word to index mapping from training set
    '''
    np.random.seed(2021)
    
    # adding 1 to fit Keras embedding (requirement)
    vocab_size = len(word_to_index) + 1
    # define dimensionality of your pre-trained word vectors (= 300)
    emb_dim = word_to_vec_map.word_vec('handsome').shape[0]
    
    # initialize the matrix with generic normal distribution values
    embed_matrix = np.random.normal(emb_mean, emb_std, (vocab_size, emb_dim))
    
    # Set each row "idx" of the embedding matrix to be 
    # the word vector representation of the idx'th word of the vocabulary
    for word, idx in word_to_index.items():
        if word in word_to_vec_map:
            embed_matrix[idx] = word_to_vec_map.get_vector(word)
            
    return embed_matrix
# Test the function
w_2_i = {'<UNK>': 1, 'handsome': 2, 'cool': 3, 'shit': 4 }
em_matrix = pretrained_embedding_matrix(word2vec, w_2_i, emb_mean, emb_std)
em_matrix<jupyter_output><empty_output><jupyter_text>CNN Model<jupyter_code>from tensorflow.keras import regularizers
from tensorflow.keras.constraints import MaxNorm

def define_model_2(filters = 100, kernel_size = 3, activation='relu', 
                 input_dim = None, output_dim=300, max_length = None, emb_matrix = None):
    
    model = tf.keras.models.Sequential([
        tf.keras.layers.Embedding(input_dim=input_dim, 
                                  output_dim=output_dim, 
                                  input_length=max_length, 
                                  input_shape=(max_length, ),
                                  # Assign the embedding weight with word2vec embedding marix
                                  weights = [emb_matrix],
                                  # Set the weight to be not trainable (static)
                                  trainable = False),
        
        tf.keras.layers.Conv1D(filters=filters, kernel_size = kernel_size, activation = activation, 
                               # set 'axis' value to the first and second axis of conv1D weights (rows, cols)
                               kernel_constraint= MaxNorm( max_value=3, axis=[0,1])),
        
        tf.keras.layers.MaxPool1D(2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(10, activation=activation, 
                              # set axis to 0 to constrain each weight vector of length (input_dim,) in dense layer
                              kernel_constraint = MaxNorm( max_value=3, axis=0)),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(units=6, activation='softmax')
    ])
    
    model.compile( loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
#     model.summary()
    return model
model_0 = define_model_2( input_dim=1000, max_length=100, emb_matrix=np.random.rand(1000, 300))
model_0.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 100, 300)          300000    
_________________________________________________________________
conv1d (Conv1D)              (None, 98, 100)           90100     
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 49, 100)           0         
_________________________________________________________________
flatten (Flatten)            (None, 4900)              0         
_________________________________________________________________
dropout (Dropout)            (None, 4900)              0         
_________________________________________________________________
dense (Dense)                (None, 10)                49010     
________________________________________________________[...]<jupyter_text>Train and Test the Model<jupyter_code>class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') >= 0.9):
            print("\nReached 90% accuracy so cancelling training!")
            self.model.stop_training=True

callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=30, verbose=2, 
                                             mode='auto', restore_best_weights=True)
# Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6]
emb_mean = emb_mean
emb_std = emb_std

columns = ['Activation', 'Filters', 'Acc']
record2 = pd.DataFrame(columns = columns)

# Separate the sentences and the labels
train_x = list(corpus[corpus.split=='train'].sentence)
train_y = np.array(corpus[corpus.split=='train'].label)
test_x = list(corpus[corpus.split=='test'].sentence)
test_y = np.array(corpus[corpus.split=='test'].label)

for activation in activations:
    for kernel_size in kernel_sizes:
            
        # encode data using
        # Cleaning and Tokenization
        tokenizer = Tokenizer(oov_token=oov_tok)
        tokenizer.fit_on_texts(train_x)

        # Turn the text into sequence
        training_sequences = tokenizer.texts_to_sequences(train_x)
        test_sequences = tokenizer.texts_to_sequences(test_x)

        max_len = max_length(training_sequences)

        # Pad the sequence to have the same size
        Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
        Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

        word_index = tokenizer.word_index
        vocab_size = len(word_index)+1
        
        emb_matrix = pretrained_embedding_matrix(word2vec, word_index, emb_mean, emb_std)
        
        # Define the input shape
        model = define_model_2(filters, kernel_size, activation, input_dim=vocab_size, 
                              max_length=max_len, emb_matrix=emb_matrix)

        # Train the model
        model.fit(Xtrain, train_y, batch_size=50, epochs=200, verbose=2, 
                  callbacks=[callbacks], validation_data=(Xtest, test_y))

        # evaluate the model
        loss, acc = model.evaluate(Xtest, test_y, verbose=0)
        print('Test Accuracy: {}'.format(acc*100))
            
        parameters = [activation, kernel_size]
        entries = parameters + [acc*100]

        temp = pd.DataFrame([entries], columns=columns)
        record2 = record2.append(temp, ignore_index=True)
        print()
        print(record2)
        print()<jupyter_output>Epoch 1/200
110/110 - 2s - loss: 1.7617 - accuracy: 0.2284 - val_loss: 1.7203 - val_accuracy: 0.2400
Epoch 2/200
110/110 - 1s - loss: 1.6113 - accuracy: 0.3096 - val_loss: 1.4970 - val_accuracy: 0.3200
Epoch 3/200
110/110 - 1s - loss: 1.3879 - accuracy: 0.4320 - val_loss: 1.2308 - val_accuracy: 0.6540
Epoch 4/200
110/110 - 1s - loss: 1.1771 - accuracy: 0.5367 - val_loss: 0.9947 - val_accuracy: 0.6880
Epoch 5/200
110/110 - 1s - loss: 1.0652 - accuracy: 0.5776 - val_loss: 0.8678 - val_accuracy: 0.7420
Epoch 6/200
110/110 - 1s - loss: 1.0141 - accuracy: 0.5930 - val_loss: 0.7924 - val_accuracy: 0.7840
Epoch 7/200
110/110 - 1s - loss: 0.9387 - accuracy: 0.6288 - val_loss: 0.7072 - val_accuracy: 0.8240
Epoch 8/200
110/110 - 1s - loss: 0.9013 - accuracy: 0.6344 - val_loss: 0.6719 - val_accuracy: 0.8140
Epoch 9/200
110/110 - 1s - loss: 0.8645 - accuracy: 0.6568 - val_loss: 0.6207 - val_accuracy: 0.8400
Epoch 10/200
110/110 - 1s - loss: 0.8422 - accuracy: 0.6621 - val_loss: 0.5932 - val_accura[...]<jupyter_text>Summary<jupyter_code>record2.sort_values(by='Acc', ascending=False)
report = record2.sort_values(by='Acc', ascending=False)
report = report.to_excel('CNN_TREC_2.xlsx', sheet_name='static')<jupyter_output><empty_output><jupyter_text>Model 3: Word2Vec Dynamic * In this part,  we will fine tune the embeddings while training (dynamic).  CNN Model<jupyter_code>from tensorflow.keras import regularizers
from tensorflow.keras.constraints import MaxNorm

def define_model_3(filters = 100, kernel_size = 3, activation='relu', 
                 input_dim = None, output_dim=300, max_length = None, emb_matrix = None):
    
    model = tf.keras.models.Sequential([
        tf.keras.layers.Embedding(input_dim=input_dim, 
                                  output_dim=output_dim, 
                                  input_length=max_length, 
                                  input_shape=(max_length, ),
                                  # Assign the embedding weight with word2vec embedding marix
                                  weights = [emb_matrix],
                                  # Set the weight to be not trainable (static)
                                  trainable = True),
        
        tf.keras.layers.Conv1D(filters=filters, kernel_size = kernel_size, activation = activation, 
                               # set 'axis' value to the first and second axis of conv1D weights (rows, cols)
                               kernel_constraint= MaxNorm( max_value=3, axis=[0,1])),
        
        tf.keras.layers.MaxPool1D(2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(10, activation=activation, 
                              # set axis to 0 to constrain each weight vector of length (input_dim,) in dense layer
                              kernel_constraint = MaxNorm( max_value=3, axis=0)),
        tf.keras.layers.Dropout(0.5),
        tf.keras.layers.Dense(units=6, activation='softmax')
    ])
    
    model.compile( loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
#     model.summary()
    return model
model_0 = define_model_3( input_dim=1000, max_length=100, emb_matrix=np.random.rand(1000, 300))
model_0.summary()<jupyter_output>Model: "sequential_12"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_12 (Embedding)     (None, 100, 300)          300000    
_________________________________________________________________
conv1d_12 (Conv1D)           (None, 98, 100)           90100     
_________________________________________________________________
max_pooling1d_12 (MaxPooling (None, 49, 100)           0         
_________________________________________________________________
flatten_12 (Flatten)         (None, 4900)              0         
_________________________________________________________________
dropout_24 (Dropout)         (None, 4900)              0         
_________________________________________________________________
dense_24 (Dense)             (None, 10)                49010     
_____________________________________________________[...]<jupyter_text>Train and Test the Model<jupyter_code>class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') >= 0.9):
            print("\nReached 90% accuracy so cancelling training!")
            self.model.stop_training=True

callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=30, verbose=2, 
                                             mode='auto', restore_best_weights=True)
# Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6, 7, 8]
emb_mean = emb_mean
emb_std = emb_std

columns = ['Activation', 'Filters', 'Acc']
record3 = pd.DataFrame(columns = columns)

# Separate the sentences and the labels
train_x = list(corpus[corpus.split=='train'].sentence)
train_y = np.array(corpus[corpus.split=='train'].label)
test_x = list(corpus[corpus.split=='test'].sentence)
test_y = np.array(corpus[corpus.split=='test'].label)

for activation in activations:
    for kernel_size in kernel_sizes:
            
        # encode data using
        # Cleaning and Tokenization
        tokenizer = Tokenizer(oov_token=oov_tok)
        tokenizer.fit_on_texts(train_x)

        # Turn the text into sequence
        training_sequences = tokenizer.texts_to_sequences(train_x)
        test_sequences = tokenizer.texts_to_sequences(test_x)

        max_len = max_length(training_sequences)

        # Pad the sequence to have the same size
        Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
        Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

        word_index = tokenizer.word_index
        vocab_size = len(word_index)+1
        
        emb_matrix = pretrained_embedding_matrix(word2vec, word_index, emb_mean, emb_std)
        
        # Define the input shape
        model = define_model_3(filters, kernel_size, activation, input_dim=vocab_size, 
                              max_length=max_len, emb_matrix=emb_matrix)

        # Train the model
        model.fit(Xtrain, train_y, batch_size=50, epochs=200, verbose=2, 
                  callbacks=[callbacks], validation_data=(Xtest, test_y))

        # evaluate the model
        loss, acc = model.evaluate(Xtest, test_y, verbose=0)
        print('Test Accuracy: {}'.format(acc*100))
            
        parameters = [activation, kernel_size]
        entries = parameters + [acc*100]

        temp = pd.DataFrame([entries], columns=columns)
        record3 = record3.append(temp, ignore_index=True)
        print()
        print(record3)
        print()<jupyter_output>Epoch 1/200
110/110 - 5s - loss: 1.6516 - accuracy: 0.2797 - val_loss: 1.3393 - val_accuracy: 0.6640
Epoch 2/200
110/110 - 4s - loss: 1.2424 - accuracy: 0.4949 - val_loss: 0.9631 - val_accuracy: 0.7620
Epoch 3/200
110/110 - 4s - loss: 0.9779 - accuracy: 0.6056 - val_loss: 0.7081 - val_accuracy: 0.8600
Epoch 4/200
110/110 - 4s - loss: 0.7964 - accuracy: 0.6920 - val_loss: 0.5762 - val_accuracy: 0.8520
Epoch 5/200
110/110 - 4s - loss: 0.6811 - accuracy: 0.7309 - val_loss: 0.5423 - val_accuracy: 0.8680
Epoch 6/200
110/110 - 4s - loss: 0.6048 - accuracy: 0.7652 - val_loss: 0.4952 - val_accuracy: 0.8820
Epoch 7/200
110/110 - 4s - loss: 0.5592 - accuracy: 0.7753 - val_loss: 0.4726 - val_accuracy: 0.8840
Epoch 8/200
110/110 - 4s - loss: 0.5132 - accuracy: 0.7999 - val_loss: 0.4633 - val_accuracy: 0.8860
Epoch 9/200
110/110 - 5s - loss: 0.4846 - accuracy: 0.8065 - val_loss: 0.4549 - val_accuracy: 0.8780
Epoch 10/200
110/110 - 5s - loss: 0.4617 - accuracy: 0.8133 - val_loss: 0.4648 - val_accura[...]<jupyter_text>Summary<jupyter_code>record3.sort_values(by='Acc', ascending=False)
report = record3.sort_values(by='Acc', ascending=False)
report = report.to_excel('CNN_TREC_3.xlsx', sheet_name='dynamic')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09857462362719899, 'fraction_numerical': 0.04716760690322796, 'mean_word_length': 3.265698419478855, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 68, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '289511', 'license': 'MIT', 'n_tokens_mistral': 10148, 'n_tokens_neox': 8998, 'n_words': 2473, 'path': '1_CNN/CNN_TREC.ipynb', 'repo_name': 'jrderek/Text-Classification-', 'size': '29974'}