We conduct experiments on neural machine translation (NMT) and sequence prediction (SP) tasks since these two tasks are commonly chosen for studying <cite>Transformers</cite> <cite>(Vaswani et al., 2017</cite>; Dai et al., 2019) .