Global self-attention, in which all locations attend to each other, is too expensive for most image scales due to the quadratic computation cost with respect to \(k\) .
Thus, multi-scale visual backbones need to use local attention to limit the size of \(k\) .
We follow the intuitive form of local attention developed in [1]}, which tries to mimic the square neighborhoods used by convolutions.
This form of local attention requires extracting local 2D grids around each pixel.
Unfortunately, while deep learning libraries automatically handle neighborhood gathering for convolutions, no such neighborhood gathering function exists for local self-attention (or any general local function).
Thus, implementing local self-attention requires explicitly gathering the local neighborhoods before the actual self-attention operation can be performed.
While the implementation of this local neighborhood gathering function might initially appear to be a relatively minor implementation detail, in practice, it must actually be carefully designed to reduce memory usage while avoiding unnecessary extra computation.
An unoptimized implementation can prevent self-attention models from scaling up due to either out-of-memory errors or excessive slowness.
The following discussion frames the design considerations of this neighborhood gathering function.
<TABLE>