The Transformer is an attention-based encoder-decoder architecture that was first introduced in 2017 [1]}. It is designed to use the attention mechanism to handle sequential data by processing all input tokens at the same time. It replaces recurrent layers, commonly used in encoder-decoder architectures, with multi-headed self-attention where the dot-product attention is extended with a scaling factor of key-dimension \(\frac{1}{\sqrt{d_k}}\)  [1]}. Consequently, the model does not rely on sequential processing of the data and therefore creates the possibilities for parallelization and reducing the training computational times [1]}. Another commonly used attention mechanism is the additive attention, which uses a feed-forward neural network for the computation of the attention [4]}.
