[1]} is designed after the authors observe the feature maps reshaped from the tokens and point out that most of the token maps are meaningless in ViT. To improve the diversity of the token maps, the authors inserts a T2T-module in the beginning of the model. T2T-module is made up of few T2T-Transformers. These T2T-Transformers can be viewed as the original transformer layers or can be replaced by the Performer used in [2]}. Between the T2T-Transformers is the T2T-process, which crops the images into several overlapping patches. The overlapping strategy enables the model to share the information between the neighbors to improve the features diversity.
