We leverage these techniques to develop a new local self-attention model family, HaloNet, which achieves state-of-the-art performance across different parameter regimes.
The largest HaloNet achieves 84.9% top-1 accuracy on the ImageNet [1]} classification benchmark (Section REF ). We perform a detailed study to uncover how self-attention and convolutional models scale differently. Our self-attention layers also show promising results on harder tasks such as object detection and instance segmentation (Section REF ) using the Mask R-CNN framework on the COCO benchmark. Finally, we end with a discussion of current limitations and ideas for future work in applying self-attention to vision.
