Recent work (Shaw et al., 2018; <cite>Dai et al., 2019</cite>; Huang et al., 2018b; Child et al., 2019; Parmar et al., 2018; Tsai et al., 2019a) proposed modifications to the Transformer for the purpose of better modeling inputs positional relation (Shaw et al., 2018; Huang et al., 2018b;<cite> Dai et al., 2019)</cite> , appending additional keys in S x k<cite> (Dai et al., 2019)</cite> , modifying the mask applied to Eq. (1) (Child et al., 2019) , or applying to distinct feature types Parmar et al., 2018; Tsai et al., 2019a) . These works adopt different designs of attention as comparing to the original form (Eq. (1)). In our paper, we aim at providing an unified view via the lens of kernel.