However, the downside of using attention masks is that it wastes computation that must happen regardless due to the implementation of this algorithm.
If attention masks are not used, the receptive field increases without any additional computation, as shown in Table REF  (Rows 2 and 3).
However, pixel-level translational equivariance is lost because the non-square receptive fields means that the output of a pixel is dependent on which block it falls into.
Take for example a pixel at the left edge of its block, which sees additional pixels that are to the right of its square receptive field.
If the entire image is shifted one pixel to the right, the pixel now falls into right edge of a neighboring block, and now sees additional pixels that are to the left of its square receptive field.
Thus the output of the pixel is dependent on its position in a block, which can change if the image shifts.
Another perspective is that blocked local self-attention is only translational equivariant to shifts of size \(b\) .
While pixel-level translational equivariance is considered important for achieving good performance[1]}, we find that empirically, using a non-masked block local self-attention actually improves the accuracy of the model (see Section REF ).
We suspect that the image shifting and cropping perturbations in common data augmentation strategies reduce the reliance on such inductive biases.
Thus we adopt unmasked blocked local self-attention because it improves accuracy without sacrificing performance.
