We note that this operation is orderagnostic to the permutation in the input se-quence (order is encoded with extra positional embedding <cite>(Vaswani et al., 2017</cite>; Shaw et al., 2018; Dai et al., 2019) ).