Recent work (Shaw et al., 2018; Dai et al., 2019; Huang et al., 2018b; Child et al., 2019; Parmar et al., 2018; Tsai et al., 2019a) proposed modifications to the <cite>Transformer</cite> for the purpose of better modeling inputs positional relation (Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019) , appending additional keys in S x k (Dai et al., 2019) , modifying the mask applied to Eq. (1) (Child et al., 2019) , or applying to distinct feature types Parmar et al., 2018; Tsai et al., 2019a) .