Transformers are initially proposed to handle the learning of long sequences in NLP tasks [1]}.
Dosovitskiy et al. [2]} and Carion et al. [3]} adapt the transformer architecture to classification and detection, respectively, and achieve competitive performance against CNN counterparts with stronger training techniques and larger-scale datasets.
DeiT [4]} further improves the training pipeline with the aid of distillation, eliminating the need for large-scale pretraining [5]}.
Inspired by the competitive performance and global receptive field of transformer models, follow-up works are proposed to refine the architecture [6]}, [7]}, explore the relationship between CONV nets and ViT [8]}, [9]}, [10]}, and adapt ViT to different computer vision tasks [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}. Other research efforts explore the essence of attention mechanism and propose insightful variants of token mixer, e.g., local attention [18]}, spatial MLP [19]}, [20]}, and pooling-mixer [21]}.
