Note that the kernel form k(x q , x k ) in the original <cite>Transformer</cite> <cite>(Vaswani et al., 2017 )</cite> is a asymmetric exponential kernel with additional mapping W q and W k (Wilson et al., 2016; Li et al., 2017) 2 .