As mentioned in the introduction, the Transformer architecture [1]} revolutionized the analysis of sequential data, and in particular that of human-written documents. Transformers are a neural network architecture that is optimized to analyze sequential data on highly-parallel devices such as GPUs and TPUs. Differently from recurrent networks, Transformers do not have “memory” but are still able to perform the trick by a combination of position-dependent embeddings (i.e. words embeddings are supplemented by another set of vectors that depend on the position the word has in the sentence, and on the index of the embedding dimension) and attention (i.e. figuring out which parts of the inputs are relevant to calculate the output). At the core of any Transformer sits the so-called Multi-Headed Attention [1]}. The idea is to apply three different linear transformations \(W_Q\) , \(W_K\)  and \(W_V\)  to each element of the input sequence to transform each word embedding into some other internal representation states called Query (\(Q\) ), Key (\(K\) ), and Value (\(V\) ) [1]}. These states are then passed to the function that calculates the attention weights, which is simply defined as:
\(Attention(Q, K, V)= softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V\) 
