<jupyter_start><jupyter_text>Using RNNs to classify sentiment on IMDB dataFor this exercise, we will train a "vanilla" RNN to predict the sentiment on IMDB reviews.  Our data consists of 25000 training sequences and 25000 test sequences.  The outcome is binary (positive/negative) and both outcomes are equally represented in both the training and the test set.Keras provides a convenient interface to load the data and immediately encode the words into integers (based on the most common words).  This will save us a lot of the drudgery that is usually involved when working with raw text.We will walk through the preparation of the data and the building of an RNN model.  Then it will be your turn to build your own models (and prepare the data how you see fit).<jupyter_code>from __future__ import print_function
import keras
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import SimpleRNN
from keras.datasets import imdb
from keras import initializers
max_features = 20000  # This is used in loading the data, picks the most common (max_features) words
maxlen = 30  # maximum length of a sequence - truncate after this
batch_size = 32
## Load in the data.  The function automatically tokenizes the text into distinct integers
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
print(len(x_train), 'train sequences')
print(len(x_test), 'test sequences')
# This pads (or truncates) the sequences so that they are of the maximum length
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)
x_train[123,:]  #Here's what an example sequence looks like<jupyter_output><empty_output><jupyter_text>Keras layers for (Vanilla) RNNsIn this exercise, we will not use pre-trained word vectors.  Rather we will learn an embedding as part of the Neural Network.  This is represented by the Embedding Layer below. Embedding Layer`keras.layers.embeddings.Embedding(input_dim, output_dim, embeddings_initializer='uniform', embeddings_regularizer=None, activity_regularizer=None, embeddings_constraint=None, mask_zero=False, input_length=None)`- This layer maps each integer into a distinct (dense) word vector of length `output_dim`.- Can think of this as learning a word vector embedding "on the fly" rather than using an existing mapping (like GloVe)- The `input_dim` should be the size of the vocabulary.- The `input_length` specifies the length of the sequences that the network expects. SimpleRNN Layer`keras.layers.recurrent.SimpleRNN(units, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)`- This is the basic RNN, where the output is also fed back as the "hidden state" to the next iteration.- The parameter `units` gives the dimensionality of the output (and therefore the hidden state).  Note that typically there will be another layer after the RNN mapping the (RNN) output to the network output.  So we should think of this value as the desired dimensionality of the hidden state and not necessarily the desired output of the network.- Recall that there are two sets of weights, one for the "recurrent" phase and the other for the "kernel" phase.  These can be configured separately in terms of their initialization, regularization, etc.<jupyter_code>## Let's build a RNN

rnn_hidden_dim = 5
word_embedding_dim = 50
model_rnn = Sequential()
model_rnn.add(Embedding(max_features, word_embedding_dim))  #This layer takes each integer in the sequence and embeds it in a 50-dimensional vector
model_rnn.add(SimpleRNN(rnn_hidden_dim,
                    kernel_initializer=initializers.RandomNormal(stddev=0.001),
                    recurrent_initializer=initializers.Identity(gain=1.0),
                    activation='relu',
                    input_shape=x_train.shape[1:]))

model_rnn.add(Dense(1, activation='sigmoid'))
## Note that most of the parameters come from the embedding layer
model_rnn.summary()
rmsprop = keras.optimizers.RMSprop(lr = .0001)

model_rnn.compile(loss='binary_crossentropy',
              optimizer=rmsprop,
              metrics=['accuracy'])
model_rnn.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=10,
          validation_data=(x_test, y_test))
score, acc = model_rnn.evaluate(x_test, y_test,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)<jupyter_output><empty_output><jupyter_text>Exercise Your TurnNow do it yourself:- Prepare the data to use sequences of length 80 rather than length 30.  Did it improve the performance?- Try different values of the "max_features".  Can you improve the performance?- Try smaller and larger sizes of the RNN hidden dimension.  How does it affect the model performance?  How does it affect the run time?<jupyter_code># Please provide your code here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06956027293404093, 'fraction_numerical': 0.009287338893100834, 'mean_word_length': 5.058553386911596, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '186650', 'license': 'MIT', 'n_tokens_mistral': 1516, 'n_tokens_neox': 1448, 'n_words': 596, 'path': 'intel-DL101-class11/Week11_RNN_IMDB_HW.ipynb', 'repo_name': 'anishmo99/intel-Deep-Learning', 'size': '5276'}