<jupyter_start><jupyter_text>| Name | Description | Date| :- |-------------: | :-:|<NAME>| Training and evaluating machine learning models - 4th PyTorch Datasets  | On 23rd of August 2019 | width="750" align="center"> Recurrent Neural Networks- Vanilla RNN- Gated Recurrent Units- Long Short Term Memory<jupyter_code>!pip3 install torch torchvision
import numpy as np
import pandas as pd
import torch, torchvision
torch.__version__
import torch.nn as nn<jupyter_output><empty_output><jupyter_text>1. Vanilla RNN![vanilla_RNN](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/LSTM3-SimpleRNN.png)- Vanilla RNN can be implemented with ```torch.nn.RNN()``` - Key Parameters  - ```input_size```:  number of expected features in the input (i.e., dimensionality of feature space)  - ```hidden_size```: number of features in hidden state (i.e., dimensionality of output space)  - ```num_layers```: number of recurrent layers (to create stacked RNN)  - ```batch_first```: If ```True```, input and output tensor shapes are ```(batch_size, seq_length, dim_feature)```. If ```False```, ```(sequence_length, batch_size, dim_feature)```  - ```bidirectional```: If ```True```, bidirectional RNN  - One thing to note is that unlike fully-connected layers or convolutional layers, RNNs take multi inputs/outputs  - In addition to (sequential) inputs, RNN has another called hidden state, which makes RNN special  - This hidden state sends information regarding current step to the next\  - Inputs to RNN: ```(x0, h0)```  - ```x0```: tensor that contains features of the input sequence    - shape      - ```(seq_len, batch_size, input_size)``` if ```batch_first == False``` (default)      - ```(batch_size, seq_len, input_size)``` if ```batch_fist == True```   - ```h0```: tensor that contains hidden state for each instance    - shape      - ```(num_layers * num_directions, batch_size, hidden_size)```<jupyter_code>rnn = nn.RNN(input_size = 10, 
             hidden_size = 5, 
             num_layers = 1)
## inputs to RNN
# input data (seq_len, batch_size, input_size)
x0 = torch.from_numpy(np.random.randn(12, 64, 10)).float()     
# hidden state (num_layers * num_directions, batch_size, hidden_size)
h0 = torch.from_numpy(np.zeros((1, 64, 5))).float()            

print(x0.shape, h0.shape)
## outputs from RNN
# output (seq_len, batch_size, num_directions * hidden_size)
# hidden state (num_layers * num_directions, batch_size, hidden_size)
out, h1 = rnn(x0, h0)

print(out.shape, h1.shape)
# when batch_first = True
rnn = nn.RNN(input_size = 10, 
             hidden_size = 5, 
             num_layers = 2,     # stacked RNN (2 layers)
             batch_first = True)
## inputs to RNN
x0 = torch.from_numpy(np.random.randn(64, 12, 10)).float()     
# note that even batch_first == True, hidden state shape order does not change
h0 = torch.from_numpy(np.zeros((2, 64, 5))).float()            

print(x0.shape, h0.shape)
## outputs from RNN
out, h1 = rnn(x0, h0)

print(out.shape, h1.shape)
# bidirectional, stacked RNN
rnn = nn.RNN(input_size = 20, 
             hidden_size = 10, 
             num_layers = 4,     
             bidirectional = True)

x0 = torch.from_numpy(np.random.randn(5, 64, 20)).float()
h0 = torch.from_numpy(np.zeros((4 * 2, 64, 10))).float()  # notice the dimensionality of hidden state
out, h1 = rnn(x0, h0)

print(out.shape, h1.shape)<jupyter_output>torch.Size([5, 64, 20]) torch.Size([8, 64, 10])<jupyter_text>2. Gated Recurrent Units (GRU)![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Gated_Recurrent_Unit%2C_base_type.svg/780px-Gated_Recurrent_Unit%2C_base_type.svg.png)- GRU has rather complicated structure compared to vanilla RNN (see below figure), but in terms of implementing it with Pytorch, largely similar to RNN, using ```torch.nn.GRU```<jupyter_code>gru = nn.GRU(input_size = 10, 
             hidden_size = 5, 
             num_layers = 1)
## inputs to GRU
# input data (seq_len, batch_size, input_size)
x0 = torch.from_numpy(np.random.randn(12, 64, 10)).float()     
# hidden state (num_layers * num_directions, batch_size, hidden_size)
h0 = torch.from_numpy(np.zeros((1, 64, 5))).float()            

print(x0.shape, h0.shape)
## outputs from GRU
# output (seq_len, batch_size, num_directions * hidden_size)
# hidden state (num_layers * num_directions, batch_size, hidden_size)
out, h1 = gru(x0, h0)

print(out.shape, h1.shape)<jupyter_output>torch.Size([12, 64, 5]) torch.Size([1, 64, 5])<jupyter_text>3. Long Short Term Memory (LSTM)![](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/The_LSTM_cell.png/1200px-The_LSTM_cell.png)- LSTM is another variant of vanilla RNN that is widely used. Though there exist some differences in structure, when implementing one just need to attend to the cell state (c_t)  - Inputs to LSTM: (x0, (h0, c0)    - ```x0```: tensor that contains features of the input sequence      - shape: ```(seq_len, batch_size, input_size)```    - ```h0```: tensor that contains initial hidden state      - shape: ```(num_layers * num_directions, batch_size, hidden_size)```    - ```c0```: tensor that contains initial cell state      - shape: ```(num_layers * num_directions, batch_size, hidden_size)``` (same as h0)  - Outputs to LSTM: (xn, (hn, cn))    - ```xn```: tensor that contains output features from the last layer      - shape: ```(seq_len, batch_size, num_directions * hidden_size)```    - ```hn```: tensor containing the hidden state      - shape: ```(num_layers * num_directions, batch_size, hidden_size)```    - ```cn```: tensor containing the cell state      - shape: ```(num_layers * num_directions, batch_size, hidden_size)```<jupyter_code>lstm = nn.LSTM(input_size = 10, 
             hidden_size = 5, 
             num_layers = 1)
## inputs to LSTM
# input data (seq_len, batch_size, input_size)
x0 = torch.from_numpy(np.random.randn(1, 64, 10)).float()     

print(x0.shape)
# outputs from LSTM
# when initial hidden & cell state are not given, they are regarded as zero
xn, (hn, cn) = lstm(x0)

print(xn.shape)               # (seq_len, batch_size, hidden_size)
print(hn.shape, cn.shape)     # (num_layers, batch_size, hidden_size)
# when initial hidden & cell states are given
x0 = torch.from_numpy(np.random.randn(1, 64, 10)).float()     
h0, c0 = torch.from_numpy(np.zeros((1, 64, 5))).float(), torch.from_numpy(np.zeros((1, 64, 5))).float()

xn, (hn, cn) = lstm(x0, (h0, c0))

print(xn.shape)               # (seq_len, batch_size, hidden_size)
print(hn.shape, cn.shape)     # (num_layers, batch_size, hidden_size)
# stacked, bidirectional LSTM
lstm = nn.LSTM(input_size = 10, 
             hidden_size = 5, 
             num_layers = 2,
             bidirectional = True)
# inputs to LSTM
x0 = torch.from_numpy(np.random.randn(5, 64, 10)).float()
h0, c0 = torch.from_numpy(np.zeros((4, 64, 5))).float(), torch.from_numpy(np.zeros((4, 64, 5))).float()

xn, (hn, cn) = lstm(x0, (h0, c0))

print(xn.shape)
print(hn.shape, cn.shape)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14549885321100917, 'fraction_numerical': 0.02981651376146789, 'mean_word_length': 3.9307420494699645, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '435288', 'license': 'MIT', 'n_tokens_mistral': 2583, 'n_tokens_neox': 2420, 'n_words': 668, 'path': 'PyTorch Exercise/PyTorch Basics/pytorch-model-basics-4 [RNN].ipynb', 'repo_name': 'rezapci/PyTorch', 'size': '6982'}