Although our models use self-attention instead of convolutions for capturing spatial interactions between pixels, they adopt some important architectural features of modern convolutional neural networks (CNNs).
Like CNNs, we compute multi-scale feature hierarchies [1]} which enable detecting objects at multiple sizes in tasks such as localization and instance segmentation.
For this, we develop a strided self-attention layer, a natural extension of strided convolutions (Section REF ).
To deal with the computational cost in larger resolutions where global attention is infeasible, we follow the fairly general principle of local processing, which is at the heart of convolutions and natural perceptual systems [2]}, [3]}, and use spatially restricted forms of self-attention.
However, unlike the model of [4]}, that also use local self-attention, we abstain from enforcing translation equivariance in lieu of better hardware utilization, which improves the speed-accuracy tradeoff (Section REF ).
Also note that while we use local attention, our receptive fields per pixel are quite large (up to \(18 \times 18\) ) and we show in Section REF  that larger receptive fields help with larger images.
In the remainder of this section, we will motivate self-attention for vision tasks and describe how we relax translational equivariance to efficiently map local self-attention to hardware.
<FIGURE><FIGURE>