<jupyter_start><jupyter_text>Introduction to Deep NLP models This text classification tutorial trains a [recurrent neural network](https://developers.google.com/machine-learning/glossary/recurrent_neural_network) on the [IMDB large movie review dataset](http://ai.stanford.edu/~amaas/data/sentiment/) for sentiment analysis. The tutorial is heavily based on [Tensorflow text classification](https://www.tensorflow.org/text/tutorials/text_classification_rnn) tutorial available online. We will explain the notebook in depth and discuss code as we go along. We will import the required packages here. Note that we are importing `tfds` here. Tensorflow datasets package has a dataset called IMDB Reviews. We will use this dataset to train a deep learning model for text classification. Here, given a review of a movie, we will predict whether the movie review is positive or negative.<jupyter_code>import numpy as np

# We use Tensorflow Datasets package to load the data ****
import tensorflow_datasets as tfds
import tensorflow as tf

tfds.disable_progress_bar()<jupyter_output><empty_output><jupyter_text>Import `matplotlib` and create a helper function to plot graphs:<jupyter_code>import matplotlib.pyplot as plt

def plot_graphs(history, metric):
  plt.plot(history.history[metric])
  plt.plot(history.history['val_'+metric], '')
  plt.xlabel("Epochs")
  plt.ylabel(metric)
  plt.legend([metric, 'val_'+metric])<jupyter_output><empty_output><jupyter_text>Step 1: Load and Preprocess the DatasetThe IMDB large movie review dataset is a *binary classification* dataset—all the reviews have either a *positive* or *negative* sentiment.Download the dataset using [TFDS](https://www.tensorflow.org/datasets). See the [loading text tutorial](https://www.tensorflow.org/tutorials/load_data/text) for details on how to load this sort of data manually.<jupyter_code># Define the dataset
dataset, info = tfds.load('imdb_reviews', with_info=True, as_supervised=True)
# Get the train and test split from the dataset
train_dataset, test_dataset = dataset['train'], dataset['test']
# Print some details of the train_dataset
train_dataset.element_spec<jupyter_output>[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0...[0m
Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEAA35K/imdb_reviews-train.tfrecord
Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEAA35K/imdb_reviews-test.tfrecord
Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEAA35K/imdb_reviews-unsupervised.tfrecord<jupyter_text>Initially this returns a dataset of (text, label pairs):<jupyter_code>for example, label in train_dataset.take(1):
  print('text: ', example.numpy())
  print('label: ', label.numpy())<jupyter_output>text:  b"This was an absolutely terrible movie. Don't be lured in by <NAME> or <NAME>. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. <NAME> appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like <NAME>'s good name. I could barely sit through it."
label:  0<jupyter_text>Next shuffle the data for training and create batches of these `(text, label)` pairs:<jupyter_code>BUFFER_SIZE = 10000
BATCH_SIZE = 64 # hyper parameter
train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
# .shuffle(BUFFER_SIZE) --*** we want to randomize the sequence or the order in which the model sees the input data.
# .batch(BATCH_SIZE) # 64 -> amount of data that is sent at once for optimization.
# .prefetch(tf.data.AUTOTUNE) --***
test_dataset = test_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
# this will give an error because we cannot index prefetch dataset

train_dataset[0]
train_dataset.take(1)

# we see that this data has two types of variables in it
# this is the raw text dataset. now we need to preprocess the data.
# we have to iterate through the train_dataset to print the values.
examples_to_print = 6

for example, label in train_dataset.take(1):
  print('texts: ', example.numpy()[:examples_to_print])
  print()
  print('labels: ', label.numpy()[:examples_to_print])<jupyter_output>texts:  [b"Dil was a memorable movie that bring to the celluloid a great director like <NAME>. The movie followed with Beta, Ishq, Raja & Masti all of whom were superb.<br /><br />But then every successful director gives a few horrible movies alongwith some hits too. <NAME> is one such movie.<br /><br />Though the comedies are told nicely but then they fail the viewer to laugh. Comparing with the kind of comedy movies being made today this is a dumb.<br /><br />If you really want to watch a movie and laugh, please don't watch this. Because the pathetic comedy will make you cry only.<br /><br />In short, the movie is worth a miss."
 b'There are lots of extremely good-looking people in this movie. That\'s probably the best thing about it. Perhaps that even makes it worth watching.<br /><br />"Loaded" tells the story of <NAME> (<NAME>), a young man who\'s about to make his mark on the world. He\'s the son of a well-to-do family with a good reputation, and he\'s on[...]<jupyter_text>Preprocessing the dataset  Create the text encoder in Tensorflow The raw text loaded by `tfds` needs to be processed before it can be used in a model. The simplest way to process text for training is using the `TextVectorization` layer. This layer has many capabilities, but this tutorial sticks to the default behavior.From [Tensorflow TextVectorization documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization):> This layer has basic options for managing text in a Keras model. It transforms a batch of strings (one example = one string) into either a list of token indices (one example = 1D tensor of integer token indices) or a dense representation (one example = 1D tensor of float values representing data about the example's tokens).> If desired, the user can call this layer's adapt() method on a dataset. When this layer is adapted, it will analyze the dataset, determine the frequency of individual string values, and create a 'vocabulary' from them. This vocabulary can have unlimited size or be capped, depending on the configuration options for this layer; if there are more unique values in the input than the maximum vocabulary size, the most frequent terms will be used to create the vocabulary.> The processing of each example contains the following steps:1. Standardize each example (usually lowercasing + punctuation stripping)2. Split each example into substrings (usually words)3. Recombine substrings into tokens (usually ngrams)4. Index tokens (associate a unique int value with each token)5. Transform each example using this index, either into a vector of ints or a dense float vector.Now, let us create the layer, and pass the dataset's text to the layer's `.adapt` method: ```pythontf.keras.layers.TextVectorization(    max_tokens=None, standardize='lower_and_strip_punctuation',    split='whitespace', ngrams=None, output_mode='int',    output_sequence_length=None, pad_to_max_tokens=False, vocabulary=None, **kwargs)```<jupyter_code># The maximum number of words that the vocabulary can have
VOCAB_SIZE = 1000

encoder = tf.keras.layers.TextVectorization(
    max_tokens=VOCAB_SIZE, # set the max number of words to be used in the Bag of Words.
    standardize='lower_and_strip_punctuation', # standardization - lowercase the sentence and strip all puctuations.
    split='whitespace', # tokenization - split the sentence based on whitespace.
    ngrams=None, # Do not create any n-grams.
    output_mode='int') # We need our output as a list of integers.

encoder.adapt(train_dataset.map(lambda text, label: text)) # adapt the train_dataset and run the text processing pipeline on it.<jupyter_output><empty_output><jupyter_text>```python Python code to reproduce tensorflow TextVectorizationsamples_tr = train_dataset.take(1)[(example, label) for example, label in samples_tr]first_sentence = example[0].numpy()first_sentence Python code to reproduce tensorflow TextVectorizationimport stringimport nltkfrom nltk.corpus import stopwordsnltk.download('stopwords')nltk.download('punkt')from nltk.tokenize import word_tokenizereivew = str(first_sentence) lower case the corpusreivew = reivew.lower() removing punctuations !"$%&\'()*+,-./:;?@[\\]^_`{|}~reivew = reivew.translate(str.maketrans('', '', string.punctuation)) tokenize sentences to wordstokenized_reivew = word_tokenize(reivew)print("Tokenized reivew: ", tokenized_reivew)``` The `.adapt` method sets the layer's vocabulary. Short sentences are padded with a padding token in such a way that the length of the sentences are the same. Padding tokens doesn't affect the learning ability of the network as they are not considered towards the gradient updates. Any words in a new sentence that is not already in the vocabulary will be marked with an unknown token. After the padding and unknown tokens they're sorted by frequency:<jupyter_code>vocab = np.array(encoder.get_vocabulary())
vocab[:20]

# '[UNK]' -> unknown token - to specify any words that are unknown to TensorFlow.<jupyter_output><empty_output><jupyter_text>Once the vocabulary is set, the layer can encode text into indices. The tensors of indices are 0-padded to the longest sequence in the batch (unless you set a fixed `output_sequence_length`):<jupyter_code>encoded_example = encoder(example)[:3].numpy()
encoded_example
encoded_example.shape<jupyter_output><empty_output><jupyter_text>Step 2: Visualize the dataset With the default settings, the process is not completely reversible. There are three main reasons for that:1. The default value for `preprocessing.TextVectorization`'s `standardize` argument is `"lower_and_strip_punctuation"`.2. The limited vocabulary size and lack of character-based fallback results in some unknown tokens.<jupyter_code>num_words = 15
n = 0

words_in_the_sentence = str(example[n].numpy()).split(' ')[:num_words] # slicing 15 words out of the first sentence in the dataset
encodeded_id_of_the_words = encoded_example[n][:num_words] # slicing 15 words out of the encoded output of first sentence in the dataset

print("Encoding\tWord")
for word, encoded_id in zip(words_in_the_sentence, encodeded_id_of_the_words):
  print(encoded_id, "\t\t", word)
for n in range(3):
  print("Original: ", example[n].numpy())
  print("Round-trip: ", " ".join(vocab[encoded_example[n]]))
  print()<jupyter_output>Original:  b"Dil was a memorable movie that bring to the celluloid a great director like <NAME>. The movie followed with Beta, Ishq, Raja & Masti all of whom were superb.<br /><br />But then every successful director gives a few horrible movies alongwith some hits too. <NAME> is one such movie.<br /><br />Though the comedies are told nicely but then they fail the viewer to laugh. Comparing with the kind of comedy movies being made today this is a dumb.<br /><br />If you really want to watch a movie and laugh, please don't watch this. Because the pathetic comedy will make you cry only.<br /><br />In short, the movie is worth a miss."
Round-trip:  [UNK] was a memorable movie that bring to the [UNK] a great director like [UNK] [UNK] the movie [UNK] with [UNK] [UNK] [UNK] [UNK] all of whom were [UNK] br but then every [UNK] director gives a few horrible movies [UNK] some [UNK] too [UNK] [UNK] is one such moviebr br though the [UNK] are told [UNK] but then they [UNK] the viewer to[...]<jupyter_text>Step 3: Create the model ![A drawing of the information flow in the model](https://github.com/tensorflow/text/blob/master/docs/tutorials/images/bidirectional.png?raw=1) Above is a diagram of the model. 1. This model can be build as a `tf.keras.Sequential`.2. The first layer is the `encoder`, which converts the text to a sequence of token indices.3. After the encoder is an embedding layer. An embedding layer stores one vector per word. When called, it converts the sequences of word indices to sequences of vectors. These vectors are trainable. After training (on enough data), words with similar meanings often have similar vectors.  This index-lookup is much more efficient than the equivalent operation of passing a one-hot encoded vector through a `tf.keras.layers.Dense` layer.4. A recurrent neural network (RNN) processes sequence input by iterating through the elements. RNNs pass the outputs from one timestep to their input on the next timestep.  The `tf.keras.layers.Bidirectional` wrapper can also be used with an RNN layer. This propagates the input forward and backwards through the RNN layer and then concatenates the final output.   * The main advantage of a bidirectional RNN is that the signal from the beginning of the input doesn't need to be processed all the way through every timestep to affect the output.    * The main disadvantage of a bidirectional RNN is that you can't efficiently stream predictions as words are being added to the end.5. After the RNN has converted the sequence to a single vector the two `layers.Dense` do some final processing, and convert from this vector representation to a single logit as the classification output.  The code to implement this is below:<jupyter_code>model = tf.keras.Sequential([
    encoder,
    tf.keras.layers.Embedding(
        input_dim=len(encoder.get_vocabulary()),
        output_dim=64,
        # Use masking to handle the variable sequence lengths
        mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])<jupyter_output><empty_output><jupyter_text>The embedding layer [uses masking](https://www.tensorflow.org/guide/keras/masking_and_padding) to handle the varying sequence-lengths. All the layers after the `Embedding` support masking:<jupyter_code>print([layer.supports_masking for layer in model.layers])<jupyter_output>[False, True, True, True, True]<jupyter_text>To confirm that this works as expected, evaluate a sentence twice. First, alone so there's no padding to mask:<jupyter_code># predict on a sample text without padding.

sample_text = ('The movie was cool. The animation and the graphics '
               'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))
print(predictions[0])<jupyter_output>[0.00152886]<jupyter_text>Now, evaluate it again in a batch with a longer sentence. The result should be identical:<jupyter_code># predict on a sample text with padding

padding = "the " * 2000
predictions = model.predict(np.array([sample_text, padding]))
print(predictions[0])<jupyter_output>[0.00152886]<jupyter_text>Compile the Keras model to configure the training process:<jupyter_code>model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['acc'])<jupyter_output><empty_output><jupyter_text>Step 4: Train the model<jupyter_code>history = model.fit(train_dataset, epochs=5)

# changed previous line to enable validation split of th
#history = model.fit(X[train_dataset], dummy_y[train_dataset], validation_split=0.33,nb_epoch=200, batch_size=5, verbose=0)<jupyter_output>Epoch 1/5
 36/391 [=>............................] - ETA: 11:23 - loss: 0.3154 - acc: 0.8563<jupyter_text>Step 5: Evaluate the trained model<jupyter_code>test_loss, test_acc = model.evaluate(test_dataset)

print('Test Loss:', test_loss)
print('Test Accuracy:', test_acc)
history_dict = history.history
print(history_dict.keys())
plt.figure(figsize=(16, 8))
plt.subplot(1, 2, 1)
plot_graphs(history, 'accuracy')
plt.ylim(None, 1)
plt.subplot(1, 2, 2)
plot_graphs(history, 'loss')
plt.ylim(0, None)<jupyter_output><empty_output><jupyter_text>Run a prediction on a new sentence:If the prediction is >= 0.0, it is positive else it is negative.<jupyter_code>sample_text = ('The movie was cool. The animation and the graphics '
               'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))<jupyter_output><empty_output><jupyter_text>We can further improve the inference on new sentences by improving the way we present the predicted results.<jupyter_code># prediction is >= 0.0 --> positive review
# else --> negative review

inputs = [
    "This is a fantastic movie.",
    "This is a bad movie.",
    "This movie was so bad that it was good.",
    "I will never say yes to watching this movie.",
    "Skip this movie.",
    "Don't waste your time.",
]

predicted_scores = model.predict(np.array(inputs))
predicted_labels = ["Positive" if x>=0.0 else "Negative" for x in predicted_scores]

for input, label in zip(inputs, predicted_labels):
  print("Question: ", input)
  print("Predicted label: ", label)<jupyter_output><empty_output><jupyter_text>Step 3 -> Assignment: Additional Architectures - Stack two or more LSTM layersKeras recurrent layers have two available modes that are controlled by the `return_sequences` constructor argument:* If `False` it returns only the last output for each input sequence (a 2D tensor of shape (batch_size, output_features)). This is the default, used in the previous model.* If `True` the full sequences of successive outputs for each timestep is returned (a 3D tensor of shape `(batch_size, timesteps, output_features)`).Here is what the flow of information looks like with `return_sequences=True`:![layered_bidirectional](https://github.com/tensorflow/text/blob/master/docs/tutorials/images/layered_bidirectional.png?raw=1) The interesting thing about using an `RNN` with `return_sequences=True` is that the output still has 3-axes, like the input, so it can be passed to another RNN layer, like this:<jupyter_code>#step 3
model_2 = tf.keras.Sequential([
    encoder,
    tf.keras.layers.Embedding(len(encoder.get_vocabulary()), 64, mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64,  return_sequences=True)),

    # Additional layer goes there.
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(..., return_sequences=False), # fill the number of LSTM units in place of ...

    # if you want to add another layer to this network, remember to set return_sequences=True
    # in the previous layer. We want the a multi-layer LSTM network to return the sequences to the
    # following LSTM layer.
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(1)
])
model_2.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])
#step 4
history_2 = model_2.fit(train_dataset, epochs=5,
                    validation_data=test_dataset,
                    validation_steps=30)
#step 5
test_loss_2, test_acc_2 = model_2.evaluate(test_dataset)

print('Test Loss:', test_loss_2)
print('Test Accuracy:', test_acc_2)
# predict on a sample text without padding.

sample_text = ('The movie was not good. The animation and the graphics '
               'were terrible. I would not recommend this movie.')
predictions_2 = model_2.predict(np.array([sample_text]))
print(predictions_2)
plt.figure(figsize=(16, 6))
plt.subplot(1, 2, 1)
plot_graphs(history_2, 'accuracy')
plt.subplot(1, 2, 2)
plot_graphs(history_2, 'loss')
# 0 --> negative review
# 1 --> positive review
inputs = [
    "This is a fantastic movie.",
    "This is a bad movie.",
    "This movie was so bad that it was good.",
    "I will never say yes to watching this movie.",
    "Skip this movie.",
    "Don't waste your time.",
]

predicted_scores_2 = model_2.predict(inputs)
predicted_labels_2 = ["Positive" if x>=0.0 else "Negative" for x in predicted_scores_2]

for input, label in zip(inputs, predicted_labels_2):
  print("Question: ", input)
  print("Predicted label: ", label)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09178082191780822, 'fraction_numerical': 0.01066536203522505, 'mean_word_length': 5.049422906185262, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 112, 'https://': 8, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '97515', 'license': 'MIT', 'n_tokens_mistral': 6129, 'n_tokens_neox': 5844, 'n_words': 2325, 'path': 'Copy_of_Intro_to_NLP_Week_11_Day_2_Lecture_Notebook.ipynb', 'repo_name': 'DigitalYogi/my-repo', 'size': '20524'}