<jupyter_start><jupyter_text>Lab 4 - Neural Network Classifier Using Simple Word Embeddings--- In this session, we demonstrate how to solve a text classification task using simple feedforward neural network classifier. We will use IMDB Large Movie Review Dataset to train a binary classification model, able to predict whether a review is positive or negative. First, our network takes one-hot word vectors as input, averages them to make one vector and trains a fully-connected layer to predict the output. In the second part, we replace the one-hot vectors with the word embeddings and add a layer to see how much that improves the performance.We are going to use Keras Sequential API in this session. The Sequential API allows you to make models layer-by-layer. But it is not straightforward to define models where layers connect to more than just the previous and next layers.<jupyter_code>import keras
import numpy as np
from keras.layers import Lambda, GlobalAveragePooling1D, Dense, Embedding
from keras import backend as K
from keras.models import Sequential<jupyter_output>Using TensorFlow backend.<jupyter_text>Downloading data The dataset we will be using is the IMDB Large Movie Review Dataset, which consists of 50000 labeled movie reviews. These are split into 25,000 reviews for training and 25,000 reviews for testing. The  dataset contains an even number of positive and negative reviews, so randomly guessing yields 50% accuracy. The data is preprocessed. For text classification, it is ususal to limit the size of the vocabulary to stop the dataset from becoming too sparse, creating possible overfitting. We keep the top 10,000 most frequently occurring words in the training data.<jupyter_code>imdb = keras.datasets.imdb
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=10000)<jupyter_output>Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz
17465344/17464789 [==============================] - 1s 0us/step<jupyter_text>We now can start playing around with the data, let’s first see the length:<jupyter_code>print("Training entries: {}, labels: {}".format(len(X_train), len(y_train)))<jupyter_output>Training entries: 25000, labels: 25000<jupyter_text>The  reviews have been converted to integers and each integer represents a  word in a dictionary.<jupyter_code>X_train[0]<jupyter_output><empty_output><jupyter_text>We can convert integers back to words by querying a dictionary object that contains the integer to string mapping:<jupyter_code>word_index = imdb.get_word_index()<jupyter_output>Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json
1646592/1641221 [==============================] - 0s 0us/step<jupyter_text>Index 1 represents the beginning of the sentence and the index 2 is assigned to all unknown tokens. Index 0 will be used for padding.<jupyter_code>word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2  
word_index["<UNUSED>"] = 3<jupyter_output><empty_output><jupyter_text>To reverse key and values in a dictionary:<jupyter_code>reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])<jupyter_output><empty_output><jupyter_text>To view a word:<jupyter_code>reverse_word_index[25]<jupyter_output><empty_output><jupyter_text>And to recreate the whole sentence from our training data we define decode_review:<jupyter_code>def decode_review(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])
decode_review(X_train[10])<jupyter_output><empty_output><jupyter_text>Creating One-hot word vectors It is  common to use one-hot representation as input in Natural Language Processing tasks. In Keras, the Embedding layer takes an index as an input and convert it to one-hot vector with the length of the vocabulary size. Then multiplies these vectors by a normal weight matrix. But there is no way to only get a one-hot vector as the output of a layer in Keras. To solve this we use Lambda() layer and a function that creates the one-hot layer.<jupyter_code>def OneHot(input_dim=None, input_length=None):
    
    if input_dim is None or input_length is None:
        raise TypeError("input_dim or input_length is not set")

    
    def _one_hot(x, num_classes):
        return K.one_hot(K.cast(x, 'uint8'),
                          num_classes=num_classes)

    return Lambda(_one_hot,
                  arguments={'num_classes': input_dim},
                  input_shape=(input_length,))<jupyter_output><empty_output><jupyter_text>input_dim refers to the length of the one-hot vector and input_length refers to the length of the input sequence. Since the input to K.one_hot should be an integer tensor, we cast x to one (Keras passes around float tensors by default).  Each text sequence has in most cases different length of words. Here, we fill sequences with a pad token (0) to fit the size. This special tokens is then masked not to be accounted in averaging, loss calculation etc. We set the maximum length to 256.  Preparing input data<jupyter_code>VOCAB_SIZE = 10000
MAX_SEQUENCE_LENGTH = 256

X_train_enc = keras.preprocessing.sequence.pad_sequences(X_train,
                                                        value=word_index["<PAD>"],
                                                        padding='post',
                                                        maxlen=256)

X_test_enc = keras.preprocessing.sequence.pad_sequences(X_test,
                                                       value=word_index["<PAD>"],
                                                       padding='post',
                                                       maxlen=256)<jupyter_output><empty_output><jupyter_text>And to view a padded review:<jupyter_code>print(X_train_enc[1])<jupyter_output>[   1  194 1153  194 8255   78  228    5    6 1463 4369 5012  134   26
    4  715    8  118 1634   14  394   20   13  119  954  189  102    5
  207  110 3103   21   14   69  188    8   30   23    7    4  249  126
   93    4  114    9 2300 1523    5  647    4  116    9   35 8163    4
  229    9  340 1322    4  118    9    4  130 4901   19    4 1002    5
   89   29  952   46   37    4  455    9   45   43   38 1543 1905  398
    4 1649   26 6853    5  163   11 3215    2    4 1153    9  194  775
    7 8255    2  349 2637  148  605    2 8003   15  123  125   68    2
 6853   15  349  165 4362   98    5    4  228    9   43    2 1157   15
  299  120    5  120  174   11  220  175  136   50    9 4373  228 8255
    5    2  656  245 2350    5    4 9837  131  152  491   18    2   32
 7464 1212   14    9    6  371   78   22  625   64 1382    9    8  168
  145   23    4 1690   15   16    4 1355    5   28    6   52  154  462
   33   89   78  285   16  145   95    0    0    0    0    0    0    0
    0 [...]<jupyter_text>Now we want to build the neural network model. We  are going to have a hidden layer with 16 hidden units. First, we want to transform each index to an embedded vector and then average all vectors to a single one. It has been showed that unweighted average of word vectors outperforms many complicated networks that model semantic and syntactic compositionality. As an example you can take a look at this: (http://anthology.aclweb.org/P/P15/P15-1162.pdf)To average we need to ignore padded zeros:<jupyter_code>class GlobalAveragePooling1DMasked(GlobalAveragePooling1D):
    def call(self, x, mask=None):
        if mask != None:
            return K.sum(x, axis=1) / K.sum(mask, axis=1)
        else:
            return super().call(x)<jupyter_output><empty_output><jupyter_text>Neural Network model using one-hot vectors The first layer is an one-hot layer. The second layer is to compute average on all word vectors in a sentence without considering padding. The  output vector is piped through a fully-connected layer. The last layer is connected with a single output node with the sigmoid activation function. The final value is a float between 0 and 1. The vocabulary count of the movie reviews (10000) is used as the input shape. At the end we visualize the model summary.<jupyter_code>from keras.models import Sequential
from keras.optimizers import Adam
from keras.layers import Dense, Embedding

model = Sequential()
model.add(OneHot(VOCAB_SIZE, MAX_SEQUENCE_LENGTH))
model.add(GlobalAveragePooling1DMasked())
model.add(Dense(16, activation="relu"))
model.add(Dense(1, activation="sigmoid"))
model.summary()
from keras.utils import plot_model
from IPython.display import SVG
from keras.utils import vis_utils
SVG(vis_utils.model_to_dot(model, show_shapes=True, show_layer_names=True, dpi = 70).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>Training the model To compile the model we need a loss function and an optimizer. We use binary_crossentropy loss function which is just a special case of categorical cross entropy. We also use Adam optimizer that can be used instead of the classical stochastic gradient descent procedure to update network weights iterative based in training data. You can read more about it here:(https://arxiv.org/abs/1412.6980v8)<jupyter_code>model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3657: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>When training, we want to check the accuracy of the model on data it hasn't seen before. So we create a validation set:<jupyter_code>X_val = np.array(X_train_enc[:10000])
partial_X_train = np.array(X_train_enc[10000:])

y_val = np.array(y_train[:10000])
partial_y_train = np.array(y_train[10000:])<jupyter_output><empty_output><jupyter_text>Then we start to train the model for 40 epochs in mini-batches of 512 samples and monitor the model's loss and accuracy on the validation set.<jupyter_code>history = model.fit(partial_X_train,
                    partial_y_train,
                    epochs=40,
                    batch_size=512,
                    validation_data=(X_val, y_val),
                    verbose=1)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3005: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Train on 15000 samples, validate on 10000 samples
Epoch 1/40
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto inste[...]<jupyter_text>Evaluating the model To evaulate the model on test data:<jupyter_code>results = model.evaluate(X_test_enc, y_test)
print(results)
# loss, accuracay<jupyter_output>[0.5473038294219971, 0.73852]<jupyter_text>Our first model accuracy using one-hot vectors is   Plotting the accuracy graph To plot a graph of accuracy and loss over time we can use Matplotlib:<jupyter_code>import matplotlib.pyplot as plt

history_dict = history.history

acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Neural Network model using word embeddings Now instead of one-hot vectors, we want to use embedding. We change our first layer in model1 to an Embedding layer. This layer takes the integer-encoded vocabulary and looks up the embedding vector for each word-index. These vectors are learned as the model trains. The vectors add a dimension to the output array. The resulting dimensions are: (batch, sequence, embedding).<jupyter_code>VOCAB_SIZE= 10000

# put the code here
model2 = Sequential()
EMBED_SIZE = 100
model2.add(Embedding(VOCAB_SIZE, EMBED_SIZE,input_length = MAX_SEQUENCE_LENGTH ))
model2.add(GlobalAveragePooling1DMasked())
model2.add(Dense(16, activation="relu"))
model2.add(Dense(1, activation="sigmoid"))

model2.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

model2.summary()


X_val = np.array(X_train_enc[:10000])
partial_X_train = np.array(X_train_enc[10000:])

history2 = model2.fit(partial_X_train,
                    partial_y_train,
                    epochs=40,
                    batch_size=512,
                    validation_data=(X_val, y_val),
                    verbose=1)

results = model2.evaluate(X_test_enc, y_test)
print(results)
results = model2.evaluate(X_test_enc, y_test)
print (results)
from keras.utils import plot_model
from IPython.display import SVG
from keras.utils import vis_utils
SVG(vis_utils.model_to_dot(model2, show_shapes=True, show_layer_names=True, dpi = 70).create(prog='dot', format='svg'))
import matplotlib.pyplot as plt

history_dict = history2.history

acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>The accuracy of model2 is 87%. Using Embedding layer instead of one-hot layer improved the performance.  Using pre-trained word embeddings The Embedding layer can be used to load a pre-trained word embedding model. We are going to use GloVe embeddings, which you can read about it here (https://nlp.stanford.edu/projects/glove/). GloVe stands for "Global Vectors for Word Representation". It's a somewhat popular embedding technique based on factorizing a matrix of word co-occurence statistics. You can download GloVe and we can seed the Keras Embedding layer with weights from the pre-trained embedding for the words in your dataset.First, we need to read GloVe and map words to GloVe:<jupyter_code>def readGloveFile(gloveFile):
    with open(gloveFile, 'r') as f:
        wordToGlove = {}  
        wordToIndex = {}  
        indexToWord = {}  

        for line in f:
            record = line.strip().split()
            token = record[0] 
            wordToGlove[token] = np.array(record[1:], dtype=np.float64) 
            
        tokens = sorted(wordToGlove.keys())
        for idx, tok in enumerate(tokens):
            kerasIdx = idx + 1  
            wordToIndex[tok] = kerasIdx 
            indexToWord[kerasIdx] = tok 

    return wordToIndex, indexToWord, wordToGlove<jupyter_output><empty_output><jupyter_text>Now, we create our pre-trained Embedding layer:<jupyter_code>def createPretrainedEmbeddingLayer(wordToGlove, wordToIndex, isTrainable):
    vocabLen = len(wordToIndex) + 1  
    embDim = next(iter(wordToGlove.values())).shape[0]  
   
    embeddingMatrix = np.zeros((vocabLen, embDim))  
    for word, index in wordToIndex.items():
        embeddingMatrix[index, :] = wordToGlove[word] 

    embeddingLayer = Embedding(vocabLen, embDim, embeddings_initializer=keras.initializers.Constant(embeddingMatrix), trainable=isTrainable)
    return embeddingLayer<jupyter_output><empty_output><jupyter_text>We freeze the weights. To create the model:<jupyter_code># put the code here
from google.colab import drive
drive.mount('/content/drive')

wordToIndex, indexToWord, wordToGlove = readGloveFile('/content/drive/My Drive/glove.6B.300d.txt')

print(len(wordToIndex))
print(len(wordToGlove))
print(len(indexToWord))<jupyter_output>Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive
400000
400000
400000<jupyter_text>Adding another hidden layer to the network In model3, we only add another dense layer to see if that improves the performance.<jupyter_code># put your code here
model3 = Sequential()
model3.add(createPretrainedEmbeddingLayer(wordToGlove, wordToIndex, True))
model3.add(GlobalAveragePooling1DMasked())
model3.add(Dense(16, activation="relu"))
model3.add(Dense(16, activation="relu"))
model3.add(Dense(1, activation="sigmoid"))

model3.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

model3.summary()

X_val = np.array(X_train_enc[:10000])
partial_X_train = np.array(X_train_enc[10000:])

history3 = model3.fit(partial_X_train,
                    partial_y_train,
                    epochs=40,
                    batch_size=512,
                    validation_data=(X_val, y_val),
                    verbose=1)

results = model3.evaluate(X_test_enc, y_test)
print(results)
results = model3.evaluate(X_test_enc, y_test)
print(results)
from keras.utils import plot_model
from IPython.display import SVG
from keras.utils import vis_utils
SVG(vis_utils.model_to_dot(model3, show_shapes=True, show_layer_names=True, dpi = 70).create(prog='dot', format='svg'))
import matplotlib.pyplot as plt

history_dict = history3.history

acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>The accuracy of model3 with an additional layer is 85%. Adding more layers can help you to extract more features. But we can do that upto a certain extent. After some point, instead of extracting features, we tend to overfit the data. Overfitting can lead to errors in some or the other form like false positives. It is not easy to choose the number of units in a hidden layer or the number of hidden layers in a neural network. For many applications, one hidden layer is enough. As a general rule, the number of units in that hidden layer is between the number of inputs and the number of outputs. The best way to decide on the number of units and hidden layers is to try various parameters. Train several neural networks with different numbers of hidden layers and neurons, and monitor the performance of them. You will have to experiment using a series of different architectures.  ***Adding another dense layer doesn't in increasing the accuracy of the model. Adding more dense layers can actually start to overfit the model. In these cases, a dropput layer should be added to prevent overfitting. The dropout layer freezes the weights of a certain layer so that it is not update during back propagation. *** ---<jupyter_code><jupyter_output><empty_output><jupyter_text>This tutorial is substantially based on this document:https://www.tensorflow.org/tutorials/keras/basic_text_classificationTo read more about Sequential APIs you can go to: https://keras.io/getting-started/sequential-model-guide/The one-hot word vector layer is taken from:https://fdalvi.github.io/blog/2018-04-07-keras-sequential-onehot/<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08615696495800752, 'fraction_numerical': 0.04421276184959938, 'mean_word_length': 4.038667315175097, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 102, 'https://': 8, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '535388', 'license': 'MIT', 'n_tokens_mistral': 6962, 'n_tokens_neox': 6285, 'n_words': 2081, 'path': 'coursework-1/PART_B.ipynb', 'repo_name': 'ShellySrivastava/Neural-Nets-and-NLP', 'size': '20718'}