<jupyter_start><jupyter_text>Hybrid (CNN+GRU) Classification with CR DatasetWe will build a text classification model using CNN and GRU model on the Customer Reviews Dataset. Since there is no standard train/test split for this dataset, we will use 10-Fold Cross Validation (CV). The CNN model is inspired by __Yoon Kim__ paper in his study on the use of Word Embedding + CNN for text classification. The hyperparameters we use based on his study are as follows:- Transfer function: rectified linear.- Kernel sizes: 3, 4, 5.- Number of filters: 100.- Dropout rate: 0.5.- Weight regularization (L2): 3.- Batch Size: 50.- Update Rule: Adam Load the library<jupyter_code>import tensorflow as tf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import re
import nltk
import random
# from nltk.tokenize import TweetTokenizer
from sklearn.model_selection import KFold

%config IPCompleter.greedy=True
%config IPCompleter.use_jedi=False
# nltk.download('twitter_samples')
tf.config.list_physical_devices('GPU')<jupyter_output><empty_output><jupyter_text>Load the Dataset<jupyter_code>corpus = pd.read_pickle('../../../0_data/MR/MR.pkl')
corpus.label = corpus.label.astype(int)
print(corpus.shape)
corpus
corpus.info()
corpus.groupby( by='label').count()
# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)
sentences[0]<jupyter_output><empty_output><jupyter_text>Data PreprocessingPreparing data for word embedding, especially for pre-trained word embedding like Word2Vec or GloVe, __don't use standard preprocessing steps like stemming or stopword removal__. Compared to our approach on cleaning the text when doing word count based feature extraction (e.g. TFIDF) such as removing stopwords, stemming etc, now we will keep these words as we do not want to lose such information that might help the model learn better.__<NAME>__, one of the developers of Word2Vec, in _word2vec-toolkit: google groups thread., 2015_, suggests only very minimal text cleaning is required when learning a word embedding model. Sometimes, it's good to disconnectIn short, what we will do is:- Puntuations removal- Lower the letter case- TokenizationThe process above will be handled by __Tokenizer__ class in TensorFlow- One way to choose the maximum sequence length is to just pick the length of the longest sentence in the training set.<jupyter_code># Define a function to compute the max length of sequence
def max_length(sequences):
    '''
    input:
        sequences: a 2D list of integer sequences
    output:
        max_length: the max length of the sequences
    '''
    max_length = 0
    for i, seq in enumerate(sequences):
        length = len(seq)
        if max_length < length:
            max_length = length
    return max_length
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"

print("Example of sentence: ", sentences[4])

# Cleaning and Tokenization
tokenizer = Tokenizer(oov_token=oov_tok)
tokenizer.fit_on_texts(sentences)

# Turn the text into sequence
training_sequences = tokenizer.texts_to_sequences(sentences)
max_len = max_length(training_sequences)

print('Into a sequence of int:', training_sequences[4])

# Pad the sequence to have the same size
training_padded = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
print('Into a padded sequence:', training_padded[4])
word_index = tokenizer.word_index
# See the first 10 words in the vocabulary
for i, word in enumerate(word_index):
    print(word, word_index.get(word))
    if i==9:
        break
vocab_size = len(word_index)+1
print(vocab_size)<jupyter_output><UNK> 1
the 2
a 3
and 4
of 5
to 6
is 7
's 8
it 9
in 10
18760<jupyter_text>Model 1: Embedding RandomA __standard model__ for document classification is to use (quoted from __<NAME>__, the author of [machinelearningmastery.com](https://machinelearningmastery.com)):>- Word Embedding: A distributed representation of words where different words that have a similar meaning (based on their usage) also have a similar representation.>- Convolutional Model: A feature extraction model that learns to extract salient features from documents represented using a word embedding.>- Fully Connected Model: The interpretation of extracted features in terms of a predictive output.Therefore, the model is comprised of the following elements:- __Input layer__ that defines the length of input sequences.- __Embedding layer__ set to the size of the vocabulary and 100-dimensional real-valued representations.- __Conv1D layer__ with 32 filters and a kernel size set to the number of words to read at once.- __MaxPooling1D layer__ to consolidate the output from the convolutional layer.- __Flatten layer__ to reduce the three-dimensional output to two dimensional for concatenation.The CNN model is inspired by __<NAME>__ paper in his study on the use of Word Embedding + CNN for text classification. The hyperparameters we use based on his study are as follows:- Transfer function: rectified linear.- Kernel sizes: 3, 4, 5.- Number of filters: 100.- Dropout rate: 0.5.- Weight regularization (L2): 3.- Batch Size: 50.- Update Rule: AdamWe will perform the best parameter using __grid search__ and 10-fold cross validation.  CNN-GRU ModelNow, we will build Convolutional Neural Network (CNN) models to classify encoded documents as either positive or negative.The model takes inspiration from `CNN for Sentence Classification` by *<NAME>*.Now, we will define our CNN model as follows:- One Conv layer with 100 filters, kernel size 5, and relu activation function;- One MaxPool layer with pool size = 2;- One Dropout layer after flattened;- Optimizer: Adam (The best learning algorithm so far)- Loss function: binary cross-entropy (suited for binary classification problem)**Note**: - The whole purpose of dropout layers is to tackle the problem of over-fitting and to introduce generalization to the model. Hence it is advisable to keep dropout parameter near 0.5 in hidden layers. - https://missinglink.ai/guides/keras/keras-conv1d-working-1d-convolutional-neural-networks-keras/<jupyter_code>from tensorflow.keras.utils import plot_model
from tensorflow.keras.constraints import MaxNorm
from tensorflow.keras.layers import Input, Embedding, Conv1D, Dropout, MaxPool1D, Flatten, Dense, Bidirectional, GRU
from tensorflow.keras.models import Model
from tensorflow.keras.layers import concatenate
def define_model(filters = 100, kernel_size = 3, activation='relu', input_dim = None, output_dim=300, max_length = None ):
  
    # Channel 1
    input1 = Input(shape=(max_length,))
    embeddding1 = Embedding(input_dim=input_dim, output_dim=output_dim, input_length=max_length)(input1)
    conv1 = Conv1D(filters=filters, kernel_size=kernel_size, activation='relu', 
                   kernel_constraint= MaxNorm( max_value=3, axis=[0,1]))(embeddding1)
    pool1 = MaxPool1D(pool_size=2, strides=2)(conv1)
    flat1 = Flatten()(pool1)
    drop1 = Dropout(0.5)(flat1)
    dense1 = Dense(10, activation='relu')(drop1)
    drop1 = Dropout(0.5)(dense1)
    out1 = Dense(1, activation='sigmoid')(drop1)
    
    # Channel 2
    input2 = Input(shape=(max_length,))
    embeddding2 = Embedding(input_dim=input_dim, output_dim=output_dim, input_length=max_length, mask_zero=True)(input2)
    gru2 = Bidirectional(GRU(64))(embeddding2)
    drop2 = Dropout(0.5)(gru2)
    out2 = Dense(1, activation='sigmoid')(drop2)
    
    # Merge
    merged = concatenate([out1, out2])
    
    # Interpretation
    outputs = Dense(1, activation='sigmoid')(merged)
    model = Model(inputs=[input1, input2], outputs=outputs)
    
    # Compile
    model.compile( loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model
model_0 = define_model( input_dim=1000, max_length=100)
model_0.summary()
class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') > 0.93):
            print("\nReached 93% accuracy so cancelling training!")
            self.model.stop_training=True


callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=7, verbose=2, 
                                             mode='auto', restore_best_weights=True)<jupyter_output><empty_output><jupyter_text>Train and Test the Model<jupyter_code># Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6]

columns = ['Activation', 'Filters', 'acc1', 'acc2', 'acc3', 'acc4', 'acc5', 'acc6', 'acc7', 'acc8', 'acc9', 'acc10', 'AVG']
record = pd.DataFrame(columns = columns)

# prepare cross validation with 10 splits and shuffle = True
kfold = KFold(10, True)

# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)

for activation in activations:
    for kernel_size in kernel_sizes:
        # kfold.split() will return set indices for each split
        acc_list = []
        for train, test in kfold.split(sentences):
            
            train_x, test_x = [], []
            train_y, test_y = [], []
            
            for i in train:
                train_x.append(sentences[i])
                train_y.append(labels[i])

            for i in test:
                test_x.append(sentences[i])
                test_y.append(labels[i])

            # Turn the labels into a numpy array
            train_y = np.array(train_y)
            test_y = np.array(test_y)

            # encode data using
            # Cleaning and Tokenization
            tokenizer = Tokenizer(oov_token=oov_tok)
            tokenizer.fit_on_texts(train_x)

            # Turn the text into sequence
            training_sequences = tokenizer.texts_to_sequences(train_x)
            test_sequences = tokenizer.texts_to_sequences(test_x)

            max_len = max_length(training_sequences)

            # Pad the sequence to have the same size
            Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
            Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

            word_index = tokenizer.word_index
            vocab_size = len(word_index)+1
            
            # Define the input shape
            model = define_model(filters, kernel_size, activation, input_dim=vocab_size, max_length=max_len)

            # Train the model and initialize test accuracy with 0
            acc = 0
            while(acc<0.7):
                
                # Train the model
                model.fit(x=[Xtrain, Xtrain], y = train_y, batch_size=50, epochs=100, verbose=1, 
                          callbacks=[callbacks], validation_data=([Xtest, Xtest], test_y))

                # evaluate the model
                loss, acc = model.evaluate([Xtest, Xtest], test_y, verbose=0)
                print('Test Accuracy: {}'.format(acc*100))
                
                if (acc<0.7):
                    print('The model suffered from local minimum. Retrain the model!')
                    model = define_model(filters, kernel_size, activation, input_dim=vocab_size, max_length=max_len)
                    
                else:
                    print('Done!')

            # evaluate the model
            loss, acc = model.evaluate([Xtest, Xtest], test_y, verbose=0)
            print('Test Accuracy: {}'.format(acc*100))

            acc_list.append(acc*100)
            
        mean_acc = np.array(acc_list).mean()
        parameters = [activation, kernel_size]
        entries = parameters + acc_list + [mean_acc]

        temp = pd.DataFrame([entries], columns=columns)
        record = record.append(temp, ignore_index=True)
        print()
        print(record)
        print()<jupyter_output>C:\Users\<NAME>\Anaconda3\envs\tf-gpu\lib\site-packages\sklearn\utils\validation.py:72: FutureWarning: Pass shuffle=True as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)<jupyter_text>Summary<jupyter_code>record.sort_values(by='AVG', ascending=False)
record[['Activation', 'AVG']].groupby(by='Activation').max().sort_values(by='AVG', ascending=False)
report = record.sort_values(by='AVG', ascending=False)
report = report.to_excel('HYBRID_MR.xlsx', sheet_name='random')<jupyter_output><empty_output><jupyter_text>Model 2: Word2Vec Static __Using and updating pre-trained embeddings__* In this part, we will create an Embedding layer in Tensorflow Keras using a pre-trained word embedding called Word2Vec 300-d tht has been trained 100 bilion words from Google News.* In this part,  we will leave the embeddings fixed instead of updating them (dynamic). 1. __Load `Word2Vec` Pre-trained Word Embedding__<jupyter_code>from gensim.models import KeyedVectors
word2vec = KeyedVectors.load_word2vec_format('../GoogleNews-vectors-negative300.bin', binary=True)
# Access the dense vector value for the word 'handsome'
# word2vec.word_vec('handsome') # 0.11376953
word2vec.word_vec('cool') # 1.64062500e-01<jupyter_output><empty_output><jupyter_text>2. __Check number of training words present in Word2Vec__<jupyter_code>def training_words_in_word2vector(word_to_vec_map, word_to_index):
    '''
    input:
        word_to_vec_map: a word2vec GoogleNews-vectors-negative300.bin model loaded using gensim.models
        word_to_index: word to index mapping from training set
    '''
    
    vocab_size = len(word_to_index) + 1
    count = 0
    # Set each row "idx" of the embedding matrix to be 
    # the word vector representation of the idx'th word of the vocabulary
    for word, idx in word_to_index.items():
        if word in word_to_vec_map:
            count+=1
            
    return print('Found {} words present from {} training vocabulary in the set of pre-trained word vector'.format(count, vocab_size))
# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)

# Cleaning and Tokenization
tokenizer = Tokenizer(oov_token=oov_tok)
tokenizer.fit_on_texts(sentences)

word_index = tokenizer.word_index
training_words_in_word2vector(word2vec, word_index)<jupyter_output>Found 16448 words present from 18760 training vocabulary in the set of pre-trained word vector<jupyter_text>2. __Define a `pretrained_embedding_layer` function__<jupyter_code>emb_mean = word2vec.vectors.mean()
emb_std = word2vec.vectors.std()
print('emb_mean: ', emb_mean)
print('emb_std: ', emb_std)
from tensorflow.keras.layers import Embedding

def pretrained_embedding_matrix(word_to_vec_map, word_to_index, emb_mean, emb_std):
    '''
    input:
        word_to_vec_map: a word2vec GoogleNews-vectors-negative300.bin model loaded using gensim.models
        word_to_index: word to index mapping from training set
    '''
    np.random.seed(2021)
    
    # adding 1 to fit Keras embedding (requirement)
    vocab_size = len(word_to_index) + 1
    # define dimensionality of your pre-trained word vectors (= 300)
    emb_dim = word_to_vec_map.word_vec('handsome').shape[0]
    
    # initialize the matrix with generic normal distribution values
    embed_matrix = np.random.normal(emb_mean, emb_std, (vocab_size, emb_dim))
    
    # Set each row "idx" of the embedding matrix to be 
    # the word vector representation of the idx'th word of the vocabulary
    for word, idx in word_to_index.items():
        if word in word_to_vec_map:
            embed_matrix[idx] = word_to_vec_map.get_vector(word)
            
    return embed_matrix
w_2_i = {'<UNK>': 1, 'handsome': 2, 'cool': 3, 'shit': 4 }
em_matrix = pretrained_embedding_matrix(word2vec, w_2_i, emb_mean, emb_std)
em_matrix<jupyter_output><empty_output><jupyter_text>MLP-CNN-GRU Model<jupyter_code>def define_model_2(filters = 100, kernel_size = 3, activation='relu', 
                   input_dim = None, output_dim=300, max_length = None, emb_matrix = None):
  
    # Channel 1
    input1 = Input(shape=(max_length,))
    embeddding1 = Embedding(input_dim=input_dim, 
                            output_dim=output_dim, 
                            input_length=max_length, 
                            input_shape=(max_length, ),
                            # Assign the embedding weight with word2vec embedding marix
                            weights = [emb_matrix],
                            # Set the weight to be not trainable (static)
                            trainable = False)(input1)
    conv1 = Conv1D(filters=filters, kernel_size=kernel_size, activation='relu', 
                   kernel_constraint= MaxNorm( max_value=3, axis=[0,1]))(embeddding1)
    pool1 = MaxPool1D(pool_size=2, strides=2)(conv1)
    flat1 = Flatten()(pool1)
    drop1 = Dropout(0.5)(flat1)
    dense1 = Dense(10, activation='relu')(drop1)
    drop1 = Dropout(0.5)(dense1)
    out1 = Dense(1, activation='sigmoid')(drop1)
    
    # Channel 2
    input2 = Input(shape=(max_length,))
    embeddding2 = Embedding(input_dim=input_dim, 
                            output_dim=output_dim, 
                            input_length=max_length, 
                            input_shape=(max_length, ),
                            # Assign the embedding weight with word2vec embedding marix
                            weights = [emb_matrix],
                            # Set the weight to be not trainable (static)
                            trainable = False,
                            mask_zero=True)(input2)
    gru2 = Bidirectional(GRU(64))(embeddding2)
    drop2 = Dropout(0.5)(gru2)
    out2 = Dense(1, activation='sigmoid')(drop2)
    
    # Merge
    merged = concatenate([out1, out2])
    
    # Interpretation
    outputs = Dense(1, activation='sigmoid')(merged)
    model = Model(inputs=[input1, input2], outputs=outputs)
    
    # Compile
    model.compile( loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model
model_0 = define_model_2( input_dim=1000, max_length=100, emb_matrix=np.random.rand(1000, 300))
model_0.summary()<jupyter_output>Model: "model_61"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_123 (InputLayer)          [(None, 100)]        0                                            
__________________________________________________________________________________________________
embedding_122 (Embedding)       (None, 100, 300)     300000      input_123[0][0]                  
__________________________________________________________________________________________________
conv1d_61 (Conv1D)              (None, 98, 100)      90100       embedding_122[0][0]              
__________________________________________________________________________________________________
max_pooling1d_61 (MaxPooling1D) (None, 49, 100)      0           conv1d_61[0][0]           [...]<jupyter_text>Train and Test the Model<jupyter_code>class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') >= 0.9):
            print("\nReached 90% accuracy so cancelling training!")
            self.model.stop_training=True

callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=7, verbose=2, 
                                             mode='auto', restore_best_weights=True)
# Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6, 7, 8]
emb_mean = emb_mean
emb_std = emb_std

columns = ['Activation', 'Filters', 'acc1', 'acc2', 'acc3', 'acc4', 'acc5', 'acc6', 'acc7', 'acc8', 'acc9', 'acc10', 'AVG']
record2 = pd.DataFrame(columns = columns)

# prepare cross validation with 10 splits and shuffle = True
kfold = KFold(10, True)

# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)

for activation in activations:
    for kernel_size in kernel_sizes:
        # kfold.split() will return set indices for each split
        acc_list = []
        for train, test in kfold.split(sentences):
            
            train_x, test_x = [], []
            train_y, test_y = [], []
            
            for i in train:
                train_x.append(sentences[i])
                train_y.append(labels[i])

            for i in test:
                test_x.append(sentences[i])
                test_y.append(labels[i])

            # Turn the labels into a numpy array
            train_y = np.array(train_y)
            test_y = np.array(test_y)

            # encode data using
            # Cleaning and Tokenization
            tokenizer = Tokenizer(oov_token=oov_tok)
            tokenizer.fit_on_texts(train_x)

            # Turn the text into sequence
            training_sequences = tokenizer.texts_to_sequences(train_x)
            test_sequences = tokenizer.texts_to_sequences(test_x)

            max_len = max_length(training_sequences)

            # Pad the sequence to have the same size
            Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
            Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

            word_index = tokenizer.word_index
            vocab_size = len(word_index)+1
            
            
            emb_matrix = pretrained_embedding_matrix(word2vec, word_index, emb_mean, emb_std)
            
            # Define the input shape
            model = define_model_2(filters, kernel_size, activation, input_dim=vocab_size, 
                                 max_length=max_len, emb_matrix=emb_matrix)

            # Train the model and initialize test accuracy with 0
            acc = 0
            while(acc<0.7):
                
                # Train the model
                model.fit(x=[Xtrain, Xtrain], y = train_y, batch_size=50, epochs=100, verbose=1, 
                          callbacks=[callbacks], validation_data=([Xtest, Xtest], test_y))

                # evaluate the model
                loss, acc = model.evaluate([Xtest, Xtest], test_y, verbose=0)
                print('Test Accuracy: {}'.format(acc*100))
                
                if (acc<0.7):
                    print('The model suffered from local minimum. Retrain the model!')
                    model = define_model_2(filters, kernel_size, activation, input_dim=vocab_size, 
                                           max_length=max_len, emb_matrix=emb_matrix)
                    
                else:
                    print('Done!')

            acc_list.append(acc*100)
            
        mean_acc = np.array(acc_list).mean()
        parameters = [activation, kernel_size]
        entries = parameters + acc_list + [mean_acc]

        temp = pd.DataFrame([entries], columns=columns)
        record2 = record2.append(temp, ignore_index=True)
        print()
        print(record2)
        print()<jupyter_output>C:\Users\<NAME>\Anaconda3\envs\tf-gpu\lib\site-packages\sklearn\utils\validation.py:72: FutureWarning: Pass shuffle=True as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)<jupyter_text>Summary<jupyter_code>record2.sort_values(by='AVG', ascending=False)
record2[['Activation', 'AVG']].groupby(by='Activation').max().sort_values(by='AVG', ascending=False)
report = record2.sort_values(by='AVG', ascending=False)
report = report.to_excel('HYBRID_MR_2.xlsx', sheet_name='static')<jupyter_output><empty_output><jupyter_text>Model 3: Word2Vec - Dynamic * In this part,  we will fine tune the embeddings while training (dynamic).  MLP-CNN-GRU Model<jupyter_code>def define_model_3(filters = 100, kernel_size = 3, activation='relu', 
                   input_dim = None, output_dim=300, max_length = None, emb_matrix = None):
  
    # Channel 1
    input1 = Input(shape=(max_length,))
    embeddding1 = Embedding(input_dim=input_dim, 
                            output_dim=output_dim, 
                            input_length=max_length, 
                            input_shape=(max_length, ),
                            # Assign the embedding weight with word2vec embedding marix
                            weights = [emb_matrix],
                            # Set the weight to be not trainable (static)
                            trainable = True)(input1)
    conv1 = Conv1D(filters=filters, kernel_size=kernel_size, activation='relu', 
                   kernel_constraint= MaxNorm( max_value=3, axis=[0,1]))(embeddding1)
    pool1 = MaxPool1D(pool_size=2, strides=2)(conv1)
    flat1 = Flatten()(pool1)
    drop1 = Dropout(0.5)(flat1)
    dense1 = Dense(10, activation='relu')(drop1)
    drop1 = Dropout(0.5)(dense1)
    out1 = Dense(1, activation='sigmoid')(drop1)
    
    # Channel 2
    input2 = Input(shape=(max_length,))
    embeddding2 = Embedding(input_dim=input_dim, 
                            output_dim=output_dim, 
                            input_length=max_length, 
                            input_shape=(max_length, ),
                            # Assign the embedding weight with word2vec embedding marix
                            weights = [emb_matrix],
                            # Set the weight to be not trainable (static)
                            trainable = True,
                            mask_zero=True)(input2)
    gru2 = Bidirectional(GRU(64))(embeddding2)
    drop2 = Dropout(0.5)(gru2)
    out2 = Dense(1, activation='sigmoid')(drop2)
    
    # Merge
    merged = concatenate([out1, out2])
    
    # Interpretation
    outputs = Dense(1, activation='sigmoid')(merged)
    model = Model(inputs=[input1, input2], outputs=outputs)
    
    # Compile
    model.compile( loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model
model_0 = define_model_3( input_dim=1000, max_length=100, emb_matrix=np.random.rand(1000, 300))
model_0.summary()<jupyter_output>Model: "model_126"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_253 (InputLayer)          [(None, 100)]        0                                            
__________________________________________________________________________________________________
embedding_252 (Embedding)       (None, 100, 300)     300000      input_253[0][0]                  
__________________________________________________________________________________________________
conv1d_126 (Conv1D)             (None, 98, 100)      90100       embedding_252[0][0]              
__________________________________________________________________________________________________
max_pooling1d_126 (MaxPooling1D (None, 49, 100)      0           conv1d_126[0][0]         [...]<jupyter_text>Train and Test the Model<jupyter_code>class myCallback(tf.keras.callbacks.Callback):
    # Overide the method on_epoch_end() for our benefit
    def on_epoch_end(self, epoch, logs={}):
        if (logs.get('accuracy') > 0.93):
            print("\nReached 93% accuracy so cancelling training!")
            self.model.stop_training=True

callbacks = tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', min_delta=0, 
                                             patience=7, verbose=2, 
                                             mode='auto', restore_best_weights=True)
# Parameter Initialization
trunc_type='post'
padding_type='post'
oov_tok = "<UNK>"
activations = ['relu']
filters = 100
kernel_sizes = [1, 2, 3, 4, 5, 6]
emb_mean = emb_mean
emb_std = emb_std

columns = ['Activation', 'Filters', 'acc1', 'acc2', 'acc3', 'acc4', 'acc5', 'acc6', 'acc7', 'acc8', 'acc9', 'acc10', 'AVG']
record3 = pd.DataFrame(columns = columns)

# prepare cross validation with 10 splits and shuffle = True
kfold = KFold(10, True)

# Separate the sentences and the labels
sentences, labels = list(corpus.sentence), list(corpus.label)

for activation in activations:
    for kernel_size in kernel_sizes:
        # kfold.split() will return set indices for each split
        acc_list = []
        for train, test in kfold.split(sentences):
            
            train_x, test_x = [], []
            train_y, test_y = [], []
            
            for i in train:
                train_x.append(sentences[i])
                train_y.append(labels[i])

            for i in test:
                test_x.append(sentences[i])
                test_y.append(labels[i])

            # Turn the labels into a numpy array
            train_y = np.array(train_y)
            test_y = np.array(test_y)

            # encode data using
            # Cleaning and Tokenization
            tokenizer = Tokenizer(oov_token=oov_tok)
            tokenizer.fit_on_texts(train_x)

            # Turn the text into sequence
            training_sequences = tokenizer.texts_to_sequences(train_x)
            test_sequences = tokenizer.texts_to_sequences(test_x)

            max_len = max_length(training_sequences)

            # Pad the sequence to have the same size
            Xtrain = pad_sequences(training_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)
            Xtest = pad_sequences(test_sequences, maxlen=max_len, padding=padding_type, truncating=trunc_type)

            word_index = tokenizer.word_index
            vocab_size = len(word_index)+1
            
            
            emb_matrix = pretrained_embedding_matrix(word2vec, word_index, emb_mean, emb_std)
            
            # Define the input shape
            model = define_model_3(filters, kernel_size, activation, input_dim=vocab_size, 
                                 max_length=max_len, emb_matrix=emb_matrix)

            # Train the model and initialize test accuracy with 0
            acc = 0
            while(acc<0.7):
                
                # Train the model
                model.fit(x=[Xtrain, Xtrain], y = train_y, batch_size=50, epochs=100, verbose=1, 
                          callbacks=[callbacks], validation_data=([Xtest, Xtest], test_y))

                # evaluate the model
                loss, acc = model.evaluate([Xtest, Xtest], test_y, verbose=0)
                print('Test Accuracy: {}'.format(acc*100))
                
                if (acc<0.7):
                    print('The model suffered from local minimum. Retrain the model!')
                    model = define_model_3(filters, kernel_size, activation, input_dim=vocab_size, 
                                           max_length=max_len, emb_matrix=emb_matrix)
                    
                else:
                    print('Done!')
            acc_list.append(acc*100)
            
        mean_acc = np.array(acc_list).mean()
        parameters = [activation, kernel_size]
        entries = parameters + acc_list + [mean_acc]

        temp = pd.DataFrame([entries], columns=columns)
        record3 = record3.append(temp, ignore_index=True)
        print()
        print(record3)
        print()<jupyter_output>C:\Users\<NAME>\Anaconda3\envs\tf-gpu\lib\site-packages\sklearn\utils\validation.py:72: FutureWarning: Pass shuffle=True as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)<jupyter_text>Summary<jupyter_code>record3.sort_values(by='AVG', ascending=False)
report = record3.sort_values(by='AVG', ascending=False)
report = report.to_excel('HYBRID_MR_3.xlsx', sheet_name='dynamic')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09744571763906477, 'fraction_numerical': 0.022515983568582636, 'mean_word_length': 2.875284260921604, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 73, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '39115', 'license': 'MIT', 'n_tokens_mistral': 9872, 'n_tokens_neox': 9194, 'n_words': 2482, 'path': '8_Ensemble_CNN_BiGRU/Ensemble_MR.ipynb', 'repo_name': 'jrderek/Text-Classification-', 'size': '32423'}