were originally introduced in the context of machine translation, replacing recurrent neural networks by an attention-based mechanism [1]}.
Transformers were later applied to image recognition, by splitting images into patches, embedding these independently, and then processing the obtained representations as a sequence [2]}.
Initially, only vision transformers pre-trained on very large collections obtained good performance, but smaller models trained on ImageNet with heavy augmentation can also yield competitive tradeoffs [3]}.
