We note that this operation is orderagnostic to the permutation in the input se-quence (order is encoded with extra positional embedding (Vaswani et al., 2017; Shaw et al., 2018;<cite> Dai et al., 2019)</cite> ).