\(\bullet\)_Absolute position embedding._ In the vanilla Transformer [22], absolute position embeddings are employed. At the bottoms of the encoder and the decoder, the absolute positional embeddings are added to the input embeddings. There are two variants of absolute position embeddings proposed in the vanilla Transformer [22], _i.e._, sinusoidal and learned position embeddings, where the latter is commonly used in existing pre-trained language models.