Ever since it has been demonstrated that convolutional neural networks (CNNs) generalize better than the alternatives for image classification [1]}, [2]}, further improvements have relied on simple but effective architectural changes that allow training of deeper CNNs [3]}, [4]}, [5]}, [6]}, [7]}, [8]}. On the other hand, adapting transformers that were developed for natural language processing (NLP) for vision tasks was a departure from using convolutional processing, which showed that further improvements in generalization in image classification are possible by using architectures that scale with larger labeled datasets and computational resources [9]}, [10]}. However, forsaking inductive priors suitable for images, such as 2D convolutional weight sharing, and adopting global self-attention with quadratic complexity meant that the training requirements increased to a prohibitive extent for most applications [11]}. Linear approximations to quadratic attention reduce the computational requirements only to some extent, and they do not resolve the problem of not having an appropriate inductive bias for images [12]}.
