Transformers [1]} are powerful neural models that are commonly adopted as the encoder and decoder in the encoder-decoder framework. These models rely on an attention mechanism to learn representations for tokens by relating them to other tokens in the sequence. Namely, a transformer-based encoder will learn representations for each token in the input sequence by “attending” to other input tokens. For the decoder, when generating a token at timestep \(t\) , it will “attend” to the representations of the output tokens generated from timestep 1 to \(t-1\)  as well as the representations of tokens from the input sequence. Transformer models can become very large with huge numbers of attention heads, encoder and decoder layers.
