The key idea of Transformer is the “self-attention” mechanism, which can capture long-term information between sequence elements. Recently, Transformer [1]} has achieved brilliant results in NLP tasks. For example, the pre-trained deep learning models (e.g., BERT [2]}, GPT [3]}) have shown effectiveness over conventional methods. Inspired by this, more and more researchers have begun to explore the application of Transformer in computer vision tasks and have achieved breakthrough results many tasks.
