Generally, the performance of computational primitives tend to improve over time due to algorithmic changes to the primitive and better software implementations.
In particular, convolution have improved over the last decade through changes in
(a) the computation of the primitive [1]}, [2]}, [3]}, [4]}, [5]}, [6]};
(b) the software implementation [7]};
(c) the structure of the primitive itself, through for example, grouped convolution [8]} and depthwise separable convolution [9]}.
Attention is in the beginning phases of this performance improvement trajectory, and given its importance in sequence modeling [10]}, it will likely see sustained effort to enhance performance.
Local attention could also receive performance improvements if it is adopted more widely to combat the general problem of processing large inputs.
Our work introduces blocked local attention to efficiently process immediate neighbors.
Other forms of non-global pixel interaction can also be implemented efficiently [11]}, [12]}, [13]}, [14]}.
