<jupyter_start><jupyter_text>Thinking in tensors in PyTorchHands-on training  by [<NAME>](https://p.migdal.pl) (2019). Version for [AI & NLP Workshop Day](https://nlpday.pl/), 31 May 2019, Warsaw, Poland: **Understanding LSTM and GRU networks in PyTorch**. NLP & AI: 4. LSTM GRU anatomy[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stared/thinking-in-tensors-writing-in-pytorch/blob/master/extra/4%20LSTM%20GRU%20anatomy.ipynb)<jupyter_code>import torch
from torch import nn<jupyter_output><empty_output><jupyter_text>LSTMMore in https://pytorch.org/docs/stable/nn.htmllstm<jupyter_code>lstm = nn.LSTM(5, 3)
# L = 8 (length)
# B = 1 (batch size)
# C = 5 (channels)
x = torch.randn(8, 1, 5)
output, (hidden, cell) = lstm(x)
output
hidden
cell
output[-1] == hidden<jupyter_output><empty_output><jupyter_text>Step by step<jupyter_code>output1, (hidden1, cell1) = lstm(x[:4])
output2, (hidden2, cell2) = lstm(x[4:], (hidden1, cell1))
output2<jupyter_output><empty_output><jupyter_text>Iteration<jupyter_code>lstm
hidden = torch.tensor([[[ 0., 0., 0.]]])
cell = torch.tensor([[[ 0., 0., 0.]]])
for i, token in enumerate(x):
    output, (hidden, cell) = lstm(x[i:i+1], (hidden, cell))
    print(output)
for e<jupyter_output><empty_output><jupyter_text>GRUMore in https://pytorch.org/docs/stable/nn.htmlgru<jupyter_code>gru = nn.GRU(5, 3)
# note that instead of (hidden, cell) there is only hidden
output, hidden = gru(x)
output
hidden<jupyter_output><empty_output><jupyter_text>Bidirectional LSTMSee also: [Understanding Bidirectional RNN in PyTorch](https://towardsdatascience.com/understanding-bidirectional-rnn-in-pytorch-5bd25a5dd66) by <NAME><jupyter_code>bilstm = nn.LSTM(5, 3, bidirectional=True)
output, (hidden, cell) = bilstm(x)
output.size()
hidden.size()
cell.size()
output
hidden<jupyter_output><empty_output><jupyter_text>Many-layered LSTM<jupyter_code>multilstm = nn.LSTM(5, 3, num_layers=2)
output, (hidden, cell) = multilstm(x)
output.size()
hidden.size()
cell.size()
output
hidden<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16189111747851004, 'fraction_numerical': 0.027698185291308502, 'mean_word_length': 6.702205882352941, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '626202', 'license': 'MIT', 'n_tokens_mistral': 881, 'n_tokens_neox': 839, 'n_words': 156, 'path': 'rnns/4 LSTM GRU anatomy.ipynb', 'repo_name': 'soumendukrg/thinking-in-tensors-writing-in-pytorch', 'size': '2105'}