In the following, we itemize the corresponding designs for the variants in <cite>Transformers</cite>: (i) Encoder Self-Attention in original <cite>Transformer</cite> <cite>(Vaswani et al., 2017)</cite> : For each query x q in the encoded sequence, M (x q , S x k ) = S x k contains the keys being all the tokens in the encoded sequence.