<jupyter_start><jupyter_text>Basic Transformer Block Contents:A self attention layer, layer normalization, a feed forward layer (a single MLP applied independently to each vector), and another layer normalization. Residual connections are added around both, before the normalization.![image](http://peterbloem.nl/files/transformers/transformer-block.svg)  Input<jupyter_code>import torch
import torch.nn.functional as F
X = torch.stack((embedding_glove['the'], embedding_glove['cat'], embedding_glove['walks'], embedding_glove['on'], embedding_glove['the'], embedding_glove['street']))
print(X.shape)
X = X.reshape(1, X.shape[0], X.shape[1])
print(X.shape)<jupyter_output>torch.Size([6, 50])
torch.Size([1, 6, 50])<jupyter_text>Self-Attention<jupyter_code>from torch import nn
num_heads = 8
num_words = X.shape[1]
num_dim = X.shape[2]
queries = nn.Linear(num_dim, num_heads*num_dim, bias=False)
keys = nn.Linear(num_dim, num_heads*num_dim, bias=False)
values = nn.Linear(num_dim, num_heads*num_dim, bias=False)
unify_heads = nn.Linear(num_heads*num_dim, num_dim, bias=False)
queries = queries(X).view(1, num_words, num_heads, num_dim)
queries = queries.transpose(1, 2).contiguous().view(1*num_heads, num_words, num_dim)
keys = keys(X).view(1, num_words, num_heads, num_dim)
keys = keys.transpose(1, 2).contiguous().view(1*num_heads, num_words, num_dim)
values = values(X).view(1, num_words, num_heads, num_dim)
values = values.transpose(1, 2).contiguous().view(1*num_heads, num_words, num_dim)
queries = queries/(num_dim**(1/4))
keys = keys/(num_dim**(1/4))

raw_weights = torch.bmm(queries, keys.transpose(1, 2))

weights = F.softmax(raw_weights, dim=2)
out = torch.bmm(weights, values).view(1, num_heads, num_words, num_dim)
print(out.shape)
out = out.transpose(1, 2).contiguous().view(1, num_words, num_heads*num_dim)
Y = unify_heads(out)
print(Y.shape)
#Initializations
norm1 = nn.LayerNorm(num_dim)
layer1 = nn.Linear(num_dim, 4*num_dim)
relu = nn.ReLU()
layer2 = nn.Linear(4*num_dim, num_dim)
norm2 = nn.LayerNorm(num_dim)
out1 = norm1(Y+X)
out2 = layer1(out1)
out3 = relu(out2)
out4 = layer2(out3)
final = norm2(out1+out4)
print(final.shape)<jupyter_output>torch.Size([1, 6, 50])
{'fraction_non_alphanumeric': 0.15324794144556267, 'fraction_numerical': 0.026989935956084173, 'mean_word_length': 7.444015444015444, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '274369', 'license': 'MIT', 'n_tokens_mistral': 865, 'n_tokens_neox': 840, 'n_words': 164, 'path': 'Basic-Transformer-Block.ipynb', 'repo_name': 'chiragbajaj25/Attention-Is-All-You-ll-Get', 'size': '2186'}