Denote Π as the set of all permutations over [n] = {1, ⋯, n}. A function f unc ∶ X n → Y n is permutation equivariant iff for any permutation π ∈ Π, f unc(πx) = πf unc(x). showed that the standard attention (encoder self-attention (Vaswani et al., 2017;<cite> Dai et al., 2019)</cite> ) is permutation equivariant.