<jupyter_start><jupyter_text>07 - Deep Learning  07 - Advanced RNN ![](https://images.unsplash.com/photo-1548018628-c56f29b810b4?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1041&q=80)Picture by [<NAME>](https://unsplash.com/photos/uQMyw1VFKqI) ___ Today, we keep diving into the Recurrent Neural Networks. We will learn about more advanced techniques to handle long term dependencies, and more types of RNN!  I. Translation Machine  I.1. How to translate a sentence? The most intuitive way to translate a sentence would be to use one of the RNN types we saw yesterday: the many-to-many:![](images/many-to-many.png) So the idea is that you take a sentence in english:> `My name is Cassidy.`If we have a well trained network, this should word :The first word `My` is given as input, and outputs `Mon`.The second word `name` outputs `nom`, `is` outputs `est`, and finally `Cassidy` outputs `Cassidy`. Our translation will be > `Mon nom est Cassidy.` Alright, let's think about it a bit more: would it work every time doing that way?  I.2. Swapping words Now let's consider the following sentence, and let's do the same exercise:> `What is your favorite movie?`The translation now would be the following:> `Quel est ton préféré film ?`This is really **not good**, we want the following translation:> `Quel est ton film préféré ?`Because english and french do not always order the same adjectives and nouns. How to deal with that problem? The neural network (and even a human!) actually has to know that both words `favorite` and `movie` have to be translated!When you human, want to translate a sentence, don't you read it all first? Then let's give our RNN a chance to do the same! This would give the following architecture:![](images/many-to-many_translation.png) What does this architecture mean? Well we have two sides.The left side, we feed our RNN with the sequence of words in english. So that the RNN has a chance to "read" **all the sequence** before making any translation. Just like a human does!The right side is a bit trickier. Once all the sentence has been read, the RNN begins to predict translation. So it begins by predicting the first word, the same way we did up to now.Then, the predicted word is fed as input to predict the next word. Indeed, the **RNN uses the previous predicted word to predict the next one**!This architecture is sometimes called an **encoder-decoder sequence-to-sequence** model.  I.3. Long term dependencies Finally, let's look at a final translation example. We want to translate this sentence> `Cats, unlike my dog, don't like water.`Into this one:> `Les chats, contrairement à mon chien, n'aiment pas l'eau.` How would perform our new architecture on that sentence? Quite good, but for sure it would **miss the plural agreement** and give the following translation:> `Les chats, contrairement à mon chien, n'aime pas l'eau.`Indeed, our RNN would predict `n'aime pas` based on `mon chien` right before. While actually, it should remember that it is related to `Les chats` a long time ago...This is the problem of **long-term dependencies**. To manage them, we would like to **keep memory** of the past.  II. Memory Cells  II.1. RNN Cell structure We will change our representation now, so as a reminder, here is how we could see a RNN cell:![](images/RNN_cell.png)Where we find back what we already know:- The hidden state $h_t$ is computed using $h_{t-1}$ and $x_t$- The prediction $\hat{y}$ is computed using $h_t$Nothing new for now!  II.2. Gated Recurrent Unit structure In order to keep memory of older state, the first idea was to create an **update gate** usually noted $\Gamma_u$. This is a **Gated Recurrent Unit** (often called **GRU**) The idea of this update gate is to have a variable that says how much information from the past step we should keep.The structure would be the following:![](images/GRU_cell.png) As you can see on the diagram, the update gate is computed using the sigmoid function, using the same inputs as a regular RNN:$$\large \Gamma_u = sigmoid(W_{hu} h_{t-1} + W_{xu} X_t + b_u)$$This value $\Gamma_u$ can really be seen as a gate:* If $\Gamma_u = 1$, completely forget the previous state* If $\Gamma_u = 0$, do not update at all, keep only the previous stateThis should allow our network to remember information on longer sequences. *Optional*: For those who want to full mathematics behind, here they are:$$\hat{h_t} = g(W_{hh} \Gamma_r h_{t-1} + W_{xh} X_t + b_h)$$$$ \Gamma_u = sigmoid(W_{hu} h_{t-1} + W_{xu} X_t + b_u)$$$$ \Gamma_r = sigmoid(W_{hr} h_{t-1} + W_{xr} X_t + b_r)$$$$ \hat{h_t} = \Gamma_u \hat{h_t} + (1-\Gamma_u)h_{t-1}$$Where $\Gamma_r$ can be seen as a relevance gate. **No worries**, as usual, all of this is **already coded in TensorFlow**, all you will have to do is:```Pythonfrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import GRUmodel = Sequential()model.add(GRU(units=16))```  II.3. Long Short-Term Memory Cell  Historically, before creating GRU, Long Short-Term Memory (LSTM) cells were invented. They are even more complex than GRU.LSTM cells allow to keep information over longer sequences than GRU cells. The drawback is that they have much more parameters, and are thus harder to train: **LSTM need a larger training dataset**! A GRU cell can be seen a binary update gate: you can either keep all the past information, or rely only on current information. (Even though it is more a threshold that can keep half past and half current information.)A LSTM offers much more liberty: you can either keep all past information **and** all current information. The structure is the following:![](images/LSTM_cell.png) The structure seems now very complicated, let's have a look!We still have the update gate that keeps working the same:$$\Gamma_u = sigmoid(W_{hu} h_{t-1} + W_{xu} X_t + b_u)$$And there is now the **forget gate** $\Gamma_f$ that is computed the same way (with different weights of course!):$$\Gamma_f = sigmoid(W_{hf} h_{t-1} + W_{xf} X_t + b_f)$$And we still compute an intermediary state, called now $\hat{c}$ instead of $\hat{h}$ before, but this is pretty much the same for the moment:$$\hat{c_t} = g(W_{hc} \Gamma_r h_{t-1} + W_{xc} X_t + b_c)$$There is a new player now, the **cell state** $c_t$, that is computed thanks to the update gate, the forget gate and the current state:$$c_t = \Gamma_f c_{t-1} + \Gamma_u \hat{c}_t$$Finally, there is one more gate computed like other gates, the **output gate** $\Gamma_o$:$$\Gamma_o = sigmoid(W_{ho} h_{t-1} + W_{xo} X_t + b_o)$$That allows to compute the final hidden state $h_t$:$$h_t = \Gamma_o c_t$$ Like the GRU, the LSTM is **coded in TensorFlow**, all you will have to do is:```Pythonfrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import LSTMmodel = Sequential()model.add(LSTM(units=16))```  III. Improving the movie review Using RNN cells, we reached about 80% accuracy score on the movie review dataset. Can we do better using GRU and/or LSTM?Let's try!<jupyter_code># Load the dataset
import numpy as np
from tensorflow.keras import datasets
imdb = datasets.imdb

(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=10000)

X_train.shape, y_train.shape
# Pad the sequences
from tensorflow.keras.preprocessing import sequence

X_train = sequence.pad_sequences(X_train,
                                 value=0,
                                 padding='post',
                                 maxlen=256)

X_test = sequence.pad_sequences(X_test,
                                value=0,
                                padding='post',
                                maxlen=256)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, GRU

# Define the model
def my_RNN():

    model = Sequential()
    model.add(Embedding(input_dim=10000, output_dim=32, input_length=256))

    # We add two layers of GRU 
    model.add(GRU(units=24, return_sequences=True))
    model.add(GRU(units=24, return_sequences=False))

    model.add(Dense(units=1, activation='sigmoid'))

    return model
# Compile the model
model = my_RNN()

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
# Train the model
model.fit(x=X_train, y=y_train, validation_data=(X_test, y_test), epochs=10, batch_size=128)
from sklearn.metrics import accuracy_score

print('accuracy on train with NN:', model.evaluate(X_train, y_train)[1])
print('accuracy on test with NN:', model.evaluate(X_test, y_test)[1])<jupyter_output>25000/25000 [==============================] - 34s 1ms/sample - loss: 0.0704 - acc: 0.9823
accuracy on train with NN: 0.98232
25000/25000 [==============================] - 37s 1ms/sample - loss: 0.5598 - acc: 0.8381
accuracy on test with NN: 0.83808
{'fraction_non_alphanumeric': 0.10501816530426884, 'fraction_numerical': 0.01725703905540418, 'mean_word_length': 4.252832438878951, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '186478', 'license': 'MIT', 'n_tokens_mistral': 2762, 'n_tokens_neox': 2583, 'n_words': 1166, 'path': 'chapters/06-rnn-advanced.ipynb', 'repo_name': 'eliaswalyba/2021-deep-learning-gift-course', 'size': '8815'}