<jupyter_start><jupyter_text>Transformers explained In December 2017, Vaswani et al. published their seminal paper, Attention Is All You Need. This paper toppled years of belief in RNNs and LSTMs applied to NLP tasks. They came up with a novel architecture called Transformer, that aims to solve sequence-to-sequence tasks while handling long-range dependencies with ease. In the following sections we will try to understand the Transformer architecture in detail.To begin with, The transformer model is a stack of 6 layers. There is a 6 layer encoder stack on left and a 6 layer decoder stack on the right.![](data/Attention.png)On the left, the inputs enter the encoder side of the Transformer and goes through an **Attention** sub-layer and **FeedForward Network (FFN)** sub-layer. On the right, the target outputs go into the decoder side of the Transformer through **two attention sub-layers and an FFN sub-layer**. The attention mechanism is a *"word-to-word" operation.* The attention mechanism finds out  how each word is related to all other words in a sequence, including the word being analyzed itself.  The encoder stack![](data/encoder.png) The encoder layer structure remains the same for all of the N=6 layers of the Transformer model.  Each layer contains two main sub-layers:+ A **multi-headed attention mechanism**+ A **fully connected position-wise feedforward network**.  Notice that we also have a **residual connection** surrounds each of the sublayers. These connections transport the unprocessed **input x** of a sublayer to a layer normalization function. This way, we are certain that key information such as **positional encoding** is not lost on the way.  The normalized output of each sub layer is thus:  $$LayerNormalization (x + Sublayer(x)).$$ After this the output of say, layer $l$ goes as input to layer $l+1$. Though the structure of each of the N=6 layers of the encoder is identical, the content of each layer is not strictly identical to the previous layer. Each layer learns from the previous layer and explores different ways of associating the tokens in the sequence.The designers of the Transformer introduced a very efficient constraint. The output of every sub-layer of the model has a constant dimension, including the **embedding layer** and the **residual connections**. This dimension is $d_{model}$ and can be set to another value depending on our goals. In the original Transformer architecture, $d_{model}$ =512.  Inputs**How do we represent inputs for the Transformer?**The first and foremost approch is say, We first represent each word of the input sentence using a one-hot vector. A one-hot vector is a vector in which every element is '0' except for a single element which is a '1'. The length of each one-hot vector is determined beforehand by the size of the vocabulary. If we want to represent 10,000 different words we need to use one-hot vectors of length 10,000 (so that we have a unique slot for the “one” for each word.) We don't want to feed the Transformer plain one-hot vectors because they're sparse, huge, and tell us nothing about the characteristics of the word. Therefore we learn a "word embedding" which is a smaller real-valued vector representation of the word that carries some information about the word. **Word Embeddedings:**Word embedding is a process of converting words in to vector representations in a way that similar words have similar representations. We can do this using `nn.Embedding` in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix $W$. `nn.Embedding` consists of a weight matrix $W$ that will transform a one-hot vector into a real-valued vector. The weight matrix has shape (**num_embeddings, embedding_dim**). *num_embeddings* is simply the vocabulary size  we need one embedding for each word in the vocabulary.  *embedding_dim* is the size we want our real-valued representation to be; we can choose this to be whatever we want - 3, 64, 256, 512, etc. In the Transformers paper they choose 512 (the hyperparameter $d_{model}$ = 512).People also refer nn.Embedding as a "lookup table" because you can imagine the weight matrix as merely a stack of the real-valued vector representations of the words:![](data/lookup.png)There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training as the weight matrix updates itself through training.The original Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings. Lets say after training our very own word-embedding. we obtain embeddings of size $d_{model}$ = 512 that looks soemthing like this.$$word = [1.35794589e-02,\  -2.18823571e-02, ....................., 1.34526128e-02,\  6.74355254e-02]_{1x512}$$.Now for every word in the sequence we will add information about its position in the sequence. We do this by positional encoding.**Positional Encoding**First thing to notice here, is to add any information to our word embedding which is of dimensions $d_{model}$ = 512, we need to have same dimensions. Vaswani et al. provide sine and cosine functions that we can generate different frerquencies for the positional encoding (PE) for each position and each dimension i of the $d_{model}$ = 512 of the word embedding vector:$$PE_{(pos 2i)} = \sin \bigg(\frac{pos}{10000^{\frac{2i}{d_{model}}}} \bigg)$$$$PE_{(pos 2i+1)} = \cos \bigg(\frac{pos}{10000^{\frac{2i}{d_{model}}}} \bigg)$$The Python implementation looks like this:<jupyter_code>import math
def positional_encoding(pos, d_model=512):
    pe = []
    for i in range(0, d_model):
        if i % 2 == 0:
            pe.append(math.sin(pos / (10000 ** ((2 * i)/d_model))))
        else:
            pe.append(math.cos(pos / (10000 ** ((2 * i)/d_model))))
    return pe
# For position 2 in the sequence
pe2 = positional_encoding(2)
print(f'Size of position vector: {len(pe2)}')<jupyter_output>Size of position vector: 512<jupyter_text>***Positional embedding  = word embedding vector + positional ecoding vector*** (both are of dim = 512).   There is one problem to this. If we add directly both of these vectors then we might loose some information of word embedding. So we need to increase the value of word embedding by multiplying with a scalar and here again they choose the value to be $\sqrt{d_{model}}$***Positional embedding  = ($\sqrt{d_{model}}$) x word embedding vector  +  positional ecoding vector*** These positional embeddings will be going in as inputs in our encoder stack.  Encoder![](data/encoder1.png)As discussed earlier the encoder stack consists of 6 layers. The input sentences goes to 1st layer as Positional embeddings and What comes out is a different representations of these sentences which have more context. Each of the six encoder layers contains two sub-layers:+ The first sub-layer is *'a multi-head self-attention mechanism.'*+ The second sub-layer is *'a simple, position-wise fully connected feed-forward network.'* ![](data/encoder2.png) **Sub-layer 1: Multi-head attention**The multi-head attention sub-layer contains eight heads and is followed by post-layer normalization, which will add residual connections to the output of the sublayer and normalize it. To understand multi-head attention let's first understand what the self-attention means.**Self-Attention**Consider this sentence:  *"I **swam** across the **river** to get to the other **BANK**"*  We see at the end of the sequence, there's a word called bank. Now to know the meaning of bank we need to understand the context. Without context the word has different meanings. Does the word bank here refers to a sloping raised land or a financial institution?  Without context its difficult for us to guess the true meaning of a word in a sentence. But when we attend to the neighbouring words in the sequence we can have some context and we can say that the word bank used here refers to a sloping land raise and not to a financial institution. We were able to decide this because we see the words swam and river in the sentence. So definitely swam and river has some weight in deciding the meaning of the word bank. Lets look at another example.*"I **drove** across the **road** to get to the **BANK**"*Now this bank refers to a financial institution. We can see how things change with respect to the context in whihc the word is used. So context is important to find the meaning of any word in an sentence.Attention refers to the mechanism that weighs neighbouring words to enhance the meaning of the word of interest. Like in the sentence above how much the word 'drove' or 'road' weighs to augment the meaning of the word bank. The main purpose of self attention mechanism is to add contextual information to the words in the sentence.So the way self attention works is, it takes the words across the sentence, then convert them into tokens and transforms them into word embedding vectors, then weigh each word vectors according to the context and finally produce a contextualised representations of the word vectors. ![](data/a1.png)The way weighing is done is by taking the dot product between each vectors. We know that similar words vectors are closeby in space. In such cases the value of dot product large and the words that are totally opposite or have now influence will have smaller values. We refer the dot product values as scores. In other words the higher the scores the closer their meanings, the higher the agreement between words. ![](data/a2.png)![](data/a3.png)After calcuating the scores for each words with all the other words in a sentence. We notice that the scores take values of diffrent ranges. So we normalize them via pssing them throung softmax function, so that the range is somewhere between 0-1. The values obtained are called as weights.![](data/a4.png)Now we use these weights to weight the original word vectors. $y_{1}, y_{2}, y_{3}...$ are said to be the new contextualized representations of our  intial input of word vectors. Lets understand how do we get contextualized representation of a word.![](data/a7.png)Suppose we want to get the contextualized repersentation of word vector $v_2$. First we have a dot product of  $v_2$ with all the other words in the sentence  $v_{1}$, $v_{2}$, $v_{3}$ and we get scores  $s_{21}$, $s_{22}$, $s_{23}$. Note scores are scalars they are obtained by dot product of two vectors. Then we normalize the scores to obtain weights  $w_{21}$, $w_{22}$, $w_{23}$. Then we multipy those weights with out original word vectors to get transformed contexualized word representation $y_2$ of original word vectors $v_2$.$$y_2 = w_{21}v_{1} + w_{22}v_{1} + w_{23}v_{1} $$**Note:** The original word vector and transformed word vector are of same dimensions. We can do this to all our word vectors to obtain new contextualized representations.But wait, are learning anything here.? Where are our weights? We now introduce three weight matrix $M_q, M_k, M_v$ for the input words and the word to which we were trying to find context. In data base anology we call them Queries, Keys and Values. Query is the word whose context we are looking for. Keys are all the words in the sentences and value is that we want to obtain.$Q = q * M_q$  $K = k * M_k$  $V = v * M_v$    ![](data/a8.png)Overview![](data/a9.png)![](data/scale.png)We see the architecture is very similar to what we have discussed above. Only new things that we see here are Scale and Mask.If we have lot of dimensions we can end up having large scores and when the scores goes through softmax function, the gradient signal is going to be very weak. so here they scale the dot product by dividing it with $\sqrt{d_k}$ that causes our scores to be in a good or resonable range. The Mask is used becuase the original transformer was build for predicting words so it cannot attend over future words. But this is optional in other case. This pretty much sums up all of the **Scaled Dot-Product Attention**.$$Attention(Q,K,V) = Softmax \bigg ( \frac{QK^T}{\sqrt{d_k}}\bigg )V$$**Multi head attention**Now lets look at Multi head attention.Consider again the sentence:*I swam across the river to get on the other bank.*![](data/ma1.png) If we consider the word *swam* in this sentence we can have questions like, who swam? swam where? So in different cases we have different attention. To handle this we come up with multi head attention.![](data/ma2.png) When we have multiple heads for a single word vector $v_1$ we have multiple outputs of contextualized vectors. say $y_{1}^{'} , y_{1}^{''}, y_{1}^{'''}$. Like discussed above all these attentions are computed with respect to different contexts. Now we concatenate these multiple outputs and send it across al linear layer to give a final ouptut $y_1$. Now that we have understood the concept of multi head attention lets get back to the architecture that we were discussing in the paper.Each head in the Multi-Head Attention Layer intakes the new embedding (Positional Encoding generated in the last step) which is n x 512 in the dimension where 'n' is the tokens in the sequence & produces an output of shape n x 64 each. This output from all heads is then concatenated to produce a single output of the Multi Headed Attention module of the dimension n x 512. In the paper, 8 attention heads are used.Inside each head $h_n$ of the attention mechanism, each word vector has three representations:+ A query vector ($Q$) that has a dimension of $d_q$ = 64, which is activated and trained when a word vector $x_n$  seeks all of the key-value pairs of the other word vectors, including itself in self-attention+ A key vector ($K$) that has a dimension of $d_k$ = 64, which will be trained to provide an attention value+ A value vector ($V$) that has a dimension of $d_v$ = 64, which will be trained to provide another attention valueTo obtain $Q$, $K$, and $V$, we must train the model with their respective weight matrices $Q_w$, $K_w$ and $V_w$, which have $d_k$ = 64 columns and $d_{model}$ = 512 rows. For example, $Q$ is obtained by a dot-product between $x$ and $Q_w$. $Q$ will have a dimension of $d_k$ = 64. These weight matrices can be learned during back propagation.$d_{model}$ = 512 (dimension of embedding for each token)  $d_k$ = 64 (dimension of Query & Key vector)  $d_v$ = 64 (dimension of Value vector)    weight matrices dimemsions  $Q_w = d_{model} * d_k =512 * 64$  $K_w = d_{model} * d_k = 512 * 64$  $V_w = d_{model} * d_v = 512 * 64$   Let undestand all the concepts we learned till now with the help of an example.Consider the sentence : *set yourself free*  tokenize : [1,2,3]  ; $n$ = 3  we will consider embedding size of each word is $d_{model}$ = 4 (in paper 512)<jupyter_code>import numpy as np
from scipy.special import softmax

# Lets the positonal embedding to these three words be
pe = np.array([[1, 0, 1, 0], [1, 2, 2, 0], [1, 2, 1, 1]])
pe<jupyter_output><empty_output><jupyter_text>Here we have word embeddings of lenght 4 for 3 words in sentence.  Let $d_k, d_v$ =3 (in paper 64) Lets generate weight matrices $Q_w, K_w, V_w.$  These will be of dimensions $d_{model} * (d_k or d_v)$<jupyter_code>qw = np.array([[2, 1, 1], [0, 2, 2], [0, 1, 0], [2, 2, 1]])
kw = np.array([[1, 1, 2], [2, 0, 2], [0, 2, 0], [2, 2, 1]])
vw = np.array([[2, 1, 1], [1, 2, 1], [2, 2, 0], [0, 2, 0]])
q = np.matmul(pe,qw)
q
k = np.matmul(pe,kw)
k
v = np.matmul(pe,vw)
v
atten = ((q*k.T)/np.sqrt(3))*v
atten
s1 = atten[0]
s2 = atten[1]
s3 = atten[2]
weights = softmax(atten, axis=1)
weights
w1 = weights[0]
w1<jupyter_output><empty_output><jupyter_text>For our 1 st token 'set' = [3.04591053e-06, 9.99995244e-01, 1.70992517e-06]The importance of set for+ set is 3.04591053e-06  + yourself is  9.99995244e-01  + free is 1.70992517e-06  More the weight the more importance to that token corresponding to that token (including itself)<jupyter_code>w1[0]*v[0] #A1 dims: [1x1]x[1x3]
w1[1]*v[1] #A2
w1[2]*v[2] #A3<jupyter_output><empty_output><jupyter_text>These 3 attention vectors are calculated for 1st token. Now we need to add  the vectors A1+A2+A3.<jupyter_code>w1[0]*v[0] + w1[1]*v[1] + w1[2]*v[2] # And finally, attention for 1st token is calculated.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08022004594365857, 'fraction_numerical': 0.02460403820577923, 'mean_word_length': 4.530926111668339, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '269631', 'license': 'MIT', 'n_tokens_mistral': 4743, 'n_tokens_neox': 4358, 'n_words': 2505, 'path': 'Understanding Transformer/Understanding transformer.ipynb', 'repo_name': 'luckyrabbit85/NLP', 'size': '16542'}