<jupyter_start><jupyter_text>Self-attentionSelf-attention is a sequence-to-sequence operation: a sequence of vectors goes in, and a sequence of vectors comes out. Let’s call the input vectors 𝐱1,𝐱2,…𝐱t and the corresponding output vectors 𝐲1,𝐲2,…,𝐲t. The vectors all have dimension k.To apply self-attention, we simply assign each word t in our vocabulary an embedding vector 𝐯t (the values of which we’ll learn). This is what’s known as an embedding layer in sequence modeling. It turns the word sequence the,cat,walks,on,the,street into the vector sequence𝐯-the,𝐯-cat,𝐯-walks,𝐯-on,𝐯-the,𝐯-street.<jupyter_code>import torch
import torch.nn.functional as F
X = torch.stack((embedding_glove['the'], embedding_glove['cat'], embedding_glove['walks'], embedding_glove['on'], embedding_glove['the'], embedding_glove['street']))
print(X.shape)
X = X.reshape(1, X.shape[0], X.shape[1])
print(X.shape)<jupyter_output>torch.Size([6, 50])
torch.Size([1, 6, 50])<jupyter_text>To produce output vector 𝐲i, the self attention operation simply takes a weighted average over all the input vectors$$𝐲_{i}= \sum_{j}w_{ij}𝐱_{j}$$Where j indexes over the whole sequence and the weights sum to one over all j. The weight wij is not a parameter, as in a normal neural net, but it is derived from a function over 𝐱i and 𝐱j. The simplest option for this function is the dot product:$$w^′_{ij}=𝐱_{i}^T𝐱_{j}$$Note that 𝐱i is the input vector at the same position as the current output vector 𝐲i. For the next output vector, we get an entirely new series of dot products, and a different weighted sum.![Image](http://peterbloem.nl/files/transformers/self-attention.svg)<jupyter_code>raw_weights = torch.bmm(X, X.transpose(1, 2))
#   torch.bmm is a batched matrix multiplication. It 
#   applies matrix multiplication over batches of 
#   matrices.<jupyter_output><empty_output><jupyter_text>The dot product gives us a value anywhere between negative and positive infinity, so we apply a softmax to map the values to $[0,1]$ and to ensure that they sum to 1 over the whole sequence:$$w_{ij}= \frac{exp(w^′_{ij})}{∑_{j}exp(w^′_{ij})}$$And that’s the basic operation of self attention.To turn the raw weights $w^′_{ij}$ into positive values that sum to one, we apply a row-wise softmax:<jupyter_code>weights = F.softmax(raw_weights, dim=2)<jupyter_output><empty_output><jupyter_text>To compute the output sequence, we just multiply the weight matrix by 𝐗. This results in a batch of output matrices 𝐘 of size (b, t, k) whose rows are weighted sums over the rows of 𝐗.$$𝐲_{i}=∑_{j}w_{ij}𝐱_{j}$$<jupyter_code>Y = torch.bmm(weights, X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10862134447398406, 'fraction_numerical': 0.007595898214963919, 'mean_word_length': 5.788659793814433, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '617742', 'license': 'MIT', 'n_tokens_mistral': 940, 'n_tokens_neox': 905, 'n_words': 320, 'path': 'Basic-Self-Attention.ipynb', 'repo_name': 'chiragbajaj25/Attention-Is-All-You-ll-Get', 'size': '2633'}