\(\bullet\)_Relative position embedding._ Unlike absolute position embeddings, relative positional embeddings are generated according to the offsets between keys and queries [273]. A popular variant of relative PE was introduced in Transformer-XL [274, 275]. The calculation of attention scores between keys and queries has been modified to introduce learnable embeddings corresponding to relative positions. T5 [82] further simplified relative positional embeddings, which was subsequently adopted by Gopher [64]. Specifically, it adds learnable scalars to the attention scores, where the scalars are calculated based on the distances between the positions of the query and the key. Compared with the absolute PE, Transformers with relative position embedding can generalize to sequences longer than those sequences for training, _i.e._, extrapolation [264].