There are up to three types of attention within the full <cite>Transformer</cite> model as exemplified with neural machine translation application <cite>(Vaswani et al., 2017)</cite> : 1) Encoder self-attention considers the source sentence as input, generating a sequence of encoded representations, where each encoded token has a global dependency with other tokens in the input sequence.