Using the implementation of local 2D self-attention with haloing detailed above, we propose a new model, HaloNet that matches state-of-the-art convolutional models on the parameter-accuracy trade-off curve.
We leverage the structure of ResNets [1]} that stack multiple residual bottleneck blocks together (see Table REF ).
HaloNet uses a few minor modifications from ResNets: (a) adding a final \(1\times 1\)  convolution before the global average pooling for larger models, following EfficientNet [2]}, (b) modifying the bottleneck block width factor, which is traditionally fixed at 4, (c) modifying the output width multiplier of the spatial operation, which is traditionally fixed at 1, (d) changing the number of blocks in the third stage from 4 to 3 for computational reasons because attention is more expensive in the higher resolution layers.
We also fix the number of heads for each of the four stages to \((4, 8, 8, 8)\)  because heads are more expensive at higher resolutions. To summarize, the scaling dimensions in HaloNet are: image size \(s\) , query block size \(b\) , halo size \(h\) , attention output width multiplier \(r_v\) , bottleneck output width multiplier \(r_b\) , number of bottleneck blocks in the third group \(l_3\) , and final \(1 \times 1\)  conv width \(d_{f}\) . Our attention neighborhoods range from \(14 \times 14\)  (\(b=8, h=3\) ) to \(18\times 18\)  (\(b=14, h=2\) ).
