We note that this operation is orderagnostic to the permutation in the input se-quence (order is encoded with extra positional embedding (Vaswani et al., 2017; Shaw et al., 2018;<cite> Dai et al., 2019)</cite> ). The above observation inspires us to connect Transformer's attention to kernel learning (Scholkopf and Smola, 2001) : they both concurrently and order-agnostically process all inputs by calculating the similarity between the inputs.