<jupyter_start><jupyter_text>Classifying movie reviews with tf.keras, tf.data, and eager execution.In this lab you'll learn how to classify a movie review as positive or negative on the [IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb). This tutorial was inspired by [this work](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/3.5-classifying-movie-reviews.ipynb). Here, we'll use tf.keras and tf.data.<jupyter_code>!pip install -q -U tensorflow==1.8.0
import tensorflow as tf

# Enable eager execution
tf.enable_eager_execution()

import numpy as np<jupyter_output><empty_output><jupyter_text>A first look at the dataset Step 1) Downloading the datasetThe "IMDB dataset" is a set of around 50,000 positive or negative reviews for movies from the Internet Movie Database. Let's use the TensorFlow API to download the dataset the same way we did for MNIST. The `num_words` argument is how many of the most popular words we will keep. All other "rare" words will be discarded.<jupyter_code>(train_data, train_labels), (test_data, test_labels) = tf.keras.datasets.imdb.load_data(num_words=10000)<jupyter_output><empty_output><jupyter_text>Step 2) Exploring the dataset The dataset comes preprocessed. Each example is an array of integers representing the words of the movie review. Each label is either a "0" for negative or "1" for positive.<jupyter_code>print(train_data.shape)
print(train_labels.shape)
print(train_data[0])
print(train_labels[0])<jupyter_output><empty_output><jupyter_text>Word indexingTo prove we have limited ourselves to the 10,000 most frequent words we will iterate over all the reviews and check the maximum value.<jupyter_code>print(max([max(review) for review in train_data]))<jupyter_output><empty_output><jupyter_text>Converting the integers back to wordsWe can get the dictionary from `tf.keras.datasets.imdb.get_word_index` which hashes the words to their corresponding integers. Let's try and convert a review from integers back into it's original text by first reversing this dictionary and then iterating over a review and converting the integers to strings.<jupyter_code># Dictionary that hashes words to their integer
word_to_integer = tf.keras.datasets.imdb.get_word_index()

# Print out the first ten keys in the dictionary
print(list(word_to_integer.keys())[0:10])

integer_to_word = dict([(value, key) for (key, value) in word_to_integer.items()])

# Demonstrate how to find the word from an integer
print(integer_to_word[1])
print(integer_to_word[2])

# We need to subtract 3 from the indices because 0 is "padding", 1 is "start of sequence", and 2 is "unknown"
decoded_review = ' '.join([integer_to_word.get(i - 3, 'UNK') for i in train_data[0]])
print(decoded_review)<jupyter_output><empty_output><jupyter_text>Step 3) Format the dataUnfortunately, we cannot just feed unformatted arrays into our neural network. We need to standardize the input. Here, we are going to multi-hot-encode our review which is an array of integers into a 10,000 dimensional vector. We will place 1's in the indices of word-integers that occur in the review, and 0's everywhere else.<jupyter_code>def vectorize_sequences(sequences, dimension=10000):
    # Create an all-zero matrix of shape (len(sequences), dimension)
    results = np.zeros((len(sequences), dimension), dtype=np.float32)
    for i, sequence in enumerate(sequences):
        results[i, sequence] = 1.  # set specific indices of results[i] to 1s
    return results

train_data = vectorize_sequences(train_data)
test_data = vectorize_sequences(test_data)

print(train_data.shape) # length is same as before
print(train_data[0]) # now, multi-hot encoded

# Vectorize the labels as well and reshape from (N, ) to (N, 1)
train_labels = np.reshape(np.asarray(train_labels, dtype=np.float32), (len(train_data), 1))
test_labels = np.reshape(np.asarray(test_labels, dtype=np.float32), (len(test_data), 1))<jupyter_output><empty_output><jupyter_text>Step 4) Create the modelNow that we have vectorized input, we're ready to build our neural network. We're going to use multiple hidden layers, each with 16 hidden units. The more layers and hidden units you have, the more complicated patterns the network can recognize and learn. * The downside is that a large number potentially leads to [overfitting](https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting) by finding patterns that may only exist in your training not (and not generalize to unseen testing data down the road).Instead of softmax as in the previous notebook, here our last activation will be a sigmoid. This will take the output of the first two layers and "squish" it into a number between 0 and 1 (the network's prediction for the review).<jupyter_code># Create a model
model = tf.keras.Sequential()

 # input shape here is the length of our movie review vector
model.add(tf.keras.layers.Dense(16, activation=tf.nn.relu, input_shape=(10000,)))
model.add(tf.keras.layers.Dense(16, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(1, activation=tf.nn.sigmoid))

optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001)

model.compile(loss='binary_crossentropy',
              optimizer=optimizer,
              metrics=['binary_accuracy'])

model.summary()<jupyter_output><empty_output><jupyter_text>Step 5) Create a validation setWe want to test our model on data it hasn't seen before, but before we get our final test accuracy. We'll call this dataset our validation set. In practice, it's used to tune parameters like your learning rate, or the number of layers / units in your model.<jupyter_code>VAL_SIZE = 10000

val_data = train_data[:VAL_SIZE]
partial_train_data = train_data[VAL_SIZE:]


val_labels = train_labels[:VAL_SIZE]
partial_train_labels = train_labels[VAL_SIZE:]<jupyter_output><empty_output><jupyter_text>Step 6) Create a tf.data DatasetAs before, we're working with a small in-memory dataset, so converting to tf.data isn't essential, just good practice for down the road. See the previous notebook or the `tf.data.Dataset` [API doc](https://www.tensorflow.org/api_docs/python/tf/data) for more info.<jupyter_code>BATCH_SIZE = 512
SHUFFLE_SIZE = 1000

training_set = tf.data.Dataset.from_tensor_slices((partial_train_data, partial_train_labels))
training_set = training_set.shuffle(SHUFFLE_SIZE).batch(BATCH_SIZE)<jupyter_output><empty_output><jupyter_text>Step 7) Training your modelNow we are going to train the model. As we go, we'll keep track of the training loss, training accuracy, validation loss, and validation accuracy. Please be patient as this step may take a while.<jupyter_code>EPOCHS = 10

# Store list of metric values for plotting later
tr_loss_list = []
tr_accuracy_list = []
val_loss_list = []
val_accuracy_list = []

for epoch in range(EPOCHS):
  for reviews, labels in training_set:
    # Calculate training loss and accuracy
    tr_loss, tr_accuracy = model.train_on_batch(reviews, labels)
  
  # Calculate validation loss and accuracy
  val_loss, val_accuracy = model.evaluate(val_data, val_labels)

  # Add to the lists
  tr_loss_list.append(tr_loss)
  tr_accuracy_list.append(tr_accuracy)
  val_loss_list.append(val_loss)
  val_accuracy_list.append(val_accuracy)
  
  print(('Epoch #%d\t Training Loss: %.2f\tTraining Accuracy: %.2f\t'
         'Validation Loss: %.2f\tValidation Accuracy: %.2f')
         % (epoch + 1, tr_loss, tr_accuracy,
         val_loss, val_accuracy))<jupyter_output><empty_output><jupyter_text>Step 8) Plotting loss and accuracyWe are going to use `matplotlib` to plot our training and validation metrics.<jupyter_code>import matplotlib.pyplot as plt

epochs = range(1, EPOCHS + 1)

# "bo" specifies "blue dot"
plt.plot(epochs, tr_loss_list, 'bo', label='Training loss')
# b specifies a "solid blue line"
plt.plot(epochs, val_loss_list, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()
plt.clf()   # Clear plot

plt.plot(epochs, tr_accuracy_list, 'bo', label='Training accuracy')
plt.plot(epochs, val_accuracy_list, 'b', label='Validation accuracy')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Step 9) Testing your modelNow that we have successfully trained our model and our training accuracy has jumped over 95%, we need to test it on an entirely different dataset. We are going to run our model on the test set. The test accuracy is a better evaluation metric for how our model will perform in the real world.<jupyter_code>loss, accuracy = model.evaluate(test_data, test_labels)
print('Test accuracy: %.2f' % (accuracy))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09202733485193622, 'fraction_numerical': 0.012528473804100227, 'mean_word_length': 5.642208774583963, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '86048', 'license': 'Apache-2.0', 'n_tokens_mistral': 2838, 'n_tokens_neox': 2677, 'n_words': 929, 'path': 'notebooks/3-imdb.ipynb', 'repo_name': 'ashubly25/workshops', 'size': '8780'}