The transformer architecture [1]}, initially designed for Natural Language Processing (NLP) tasks, introduces the Multi-Head Self Attention (MHSA) mechanism that allows the network to model long-term dependencies and
is easy to parallelize.
In this context, Dosovitskiy et al. [2]} adapt the attention mechanism to 2D images and propose Vision Transformer (ViT): the input image is divided into non-overlapping patches, and the inter-patch representations are learned through MHSA without inductive bias. ViTs demonstrate promising results compared to convolutional neural networks (CNNs) on computer vision tasks. Following this success, several efforts explore the potential of ViT by improving training strategies [3]}, [4]}, [5]}, introducing architecture changes [6]}, [7]}, redesigning attention mechanisms [8]}, [9]}, and elevating the performance of various vision tasks such as classification [10]}, [11]}, [12]}, segmentation [13]}, [14]}, and detection [15]}, [16]}.
