<jupyter_start><jupyter_text>Basic study before Show, Attend, TellThis notebook summarize what I have studied to implement this repository. If you are newbi to image like me, I recommend to use this notebook.  1. RNNI study contents from [this blog](https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9).API list- RNNCell in pytorch [here](https://pytorch.org/docs/stable/generated/torch.nn.RNNCell.htmltorch.nn.RNNCell). RNNCell requires both of hidden and input to have exact same batch size.<jupyter_code>import torch
import torch.nn as nn

batch_size = 5
input_dim = 10
hidden_dim = 15

x = torch.rand([batch_size, input_dim])
init_h = torch.rand([batch_size, hidden_dim])
f = nn.RNNCell(input_dim, hidden_dim)
next_h = f(x, init_h)<jupyter_output><empty_output><jupyter_text>2. LSTM (from [this wikidocs in korean](https://wikidocs.net/22888))In summary, there are 3 main in/output for LSTM.- Cell State (Long-term memory)- Hidden State = Output (Short-term memory)- inputTo compute Cell State, you should compute- `forgot_gate(x, hidden_state) * cell state + input_gate(x, hidden_state)`- `forgot_gate(x, hidden_state)` = `sigmoid(W_hf@hidden_state` + `W_xf@x)`- `input_gate(x, hidden_state)` = `sigmoid(W_hi@hidden_state + W_xi@x) + tanh(W_hg@hidden_state + W_xg@x)`To compute Output (hidden_state), you should compute- Compute Cell State, `C_t`- `tanh(C_t) * output_gate(x, hidden_state)`<jupyter_code>import torch
import torch.nn as nn

batch_size = 5
input_dim = 10
hidden_dim = 15

x = torch.rand([batch_size, input_dim])
init_h = torch.rand([batch_size, hidden_dim])
init_c = torch.rand([batch_size, hidden_dim])
f = nn.LSTMCell(input_dim, hidden_dim)
next_h, next_c = f(x, (init_h, init_c))<jupyter_output><empty_output><jupyter_text>3. pytorch padded sequence<jupyter_code>import torch
import torch.nn as nn

batch = [torch.tensor([1,2,3]), torch.tensor([4,5])]
packed_batch = nn.utils.rnn.pad_sequence(batch, batch_first=True)
packed_sequence_batch = nn.utils.rnn.pack_padded_sequence(packed_batch, [3, 2], batch_first=True)
print(packed_sequence_batch)<jupyter_output>PackedSequence(data=tensor([1, 4, 2, 5, 3]), batch_sizes=tensor([2, 2, 1]), sorted_indices=None, unsorted_indices=None)
{'fraction_non_alphanumeric': 0.12672913877733155, 'fraction_numerical': 0.018741633199464525, 'mean_word_length': 6.651877133105802, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '465794', 'license': 'Apache-2.0', 'n_tokens_mistral': 857, 'n_tokens_neox': 826, 'n_words': 194, 'path': 'study.ipynb', 'repo_name': 'hankyul2/Show_Attend_Tell', 'size': '2241'}