Attention has steadily risen in adoption in vision models in recent years.
First introduced in various forms of sequence modeling [1]}, [2]}, [3]}, attention was used to attend to image features in the text generation module of image captioning models [4]}.
Attention is also closely related to non-local means [5]}, a pairwise-weighted global sum of pixels originally developed for image denoising.
[6]} applied non-local means on top of spatially downsampled convolutional features to improve video classification.
However, since these methods scale quadratically with receptive field size, they cannot be used because the spatial size is too large. In order to apply self-attention, [7]} applies local attention on images for the task of image generation. [8]} spatially downsample the features for attention and concatenate the attention outputs to convolutional features.
Instead, we directly build on top of the approach of [9]}, who compute attention on local regions in order to build a fully self-attentional vision model for classification and object detection.
Different forms of attention for pure self-attention vision models have also been proposed [10]}, [11]}, which are orthogonal and complementary to the focus on scaling in this work.
In addition to attention over the spatial extent that we focus on, components that perform attention over channels have also been used to augment convolutional models [12]}, [13]}. In recent and concurrent work, Vision Transformer [14]} show that applying transformers on projections of non-overlapping image patches can achieve accuracies comparable to SOTA when pre-trained on very large (JFT-300M [15]}) and medium sized (ImageNet-21k [16]}) classification datasets. However, their models do not adopt a multiscale architecture and our focus in this work is training on ImageNet [17]} from scratch. In Section REF , we conduct transfer experiments and compare with ViT and BiT [18]}.
