Positional Embedding k(⋅, ⋅) The kernel construction on X = (F × T ) has distinct design in variants of Transformers (Vaswani et al., 2017; <cite>Dai et al., 2019</cite>; Huang et al., 2018b; Shaw et al., 2018; Child et al., 2019) .