Transformer [1]} has achieved great success in NLP by serving as the basic architecture of many popular models such as BERT [2]}, RoBERTa [3]} and XLM [4]}.
In addition, Transformers also show great potentials in other fields like computer vision [5]} and speech recognition [6]} to understand data in other modalities.
Self-attention is the core of Transformer [7]}.
It aims to model the interaction between each pair of tokens to help capture their contexts, where the computational complexity is quadratic with respect to the input sequence length [1]}.
Thus, Transformer is inefficient in handling long sequences [9]}.
