Our results demonstrate that self-attention is competitive accuracy-wise when training on ImageNet from scratch.
Figure REF  shows that pure self-attentionBy pure attention we mean models that use self-attention in all layers except the stem, which is convolutional. based HaloNets are currently slower to train than the corresponding EfficientNets and require further optimizations for large batch training. However, our hybrids have the same speed-accuracy tradeoff as EfficientNets. On transfer from ImageNet-21k, our models outperform very strong models such as BiT [1]} and ViT [2]}, on both accuracy and speed.
Model optimizations, such as using architecture search methods to find better speed-accuracy tradeoffs or different forms of more powerful and/or efficient attention forms [3]}, [4]}, are promising directions for machine learning researchers.
Implementation optimizations, such as better memory management, can improve the practicality of these models. Also, scaling up our models to larger widths might cause our operations to transition from being memory bound to compute bound, and lead to better speed-accuracy tradeoffs. We leave this study for future work. Overall, our work shows that self-attention can be competitive in regimes traditionally dominated by computer vision.
Future work can push these boundaries further, both in terms of scale and efficiency.
