Vision and natural language processing (NLP) systems divide the landscape of computational primitives.
While self-attention is the primary workhorse in NLP, convolutions are ubiquitous in nearly all vision models.
Convolutions embody the principle of local processing, to learn local spatial features such as edges and texture that are abundant in images.
On the other hand, the Transformer [1]} showed that self-attention is an effective and computationally efficient mechanism for capturing global interactions between words in a sentence.
The success of self-attention in NLP motivates research in understanding how self-attention can improve vision.
Self-attention has several properties that make it a good fit for vision:
(a) content-based interactions as opposed to content-independent interactions of convolution;
(b) parameter-independent scaling of receptive field size as opposed to parameter-dependent scaling of convolution;
(c) empirical ability to capture long-range dependencies for use in larger images;
(d) flexibility to handle and integrate multiple types of data that appear in vision, such as pixels [2]}, [3]}, [4]}, [5]}, point clouds [6]}, sequence conditioning information [7]}, and graphs [8]}.
Self-attention may also be regarded as an adaptive nonlinearity paralleling a long history of nonlinear processing techniques in computer vision, such as bilateral filtering [9]} and non-local means [10]}.
