For clarification, we are not attacking the claim made by the prior work (Vaswani et al., 2017; Shaw et al., 2018;  Huang et al., 2018b; <cite>Dai et al., 2019</cite>; Child et al., 2019 ), but we aim at providing a new look at the order-invariance problem when considering the attention mechanism with masks (masks refer to the set filtering function in our kernel formulation).