Nowadays, models based on Transformer [1]} have achieved an extremely high performance in deep learning research. In the area of natural language processing, Transformer and its variants, like BERT [2]} and GPT-3 [3]}, have outperformed other models based on RNN and CNN in most tasks. Inspired by the great ability that transformers have shown in NLP tasks, researchers have started to migrate the network to other fields like computer vision and recommender system recently. Vision Transformer (ViT) [4]} is one of the representative works which apply the transformer directly to sequences of image patches.
