Vision transformers and hybrid architectures, inspired by their success on NLP tasks, have pushed the image classification accuracy beyond those of the largest CNNs, albeit at the cost of several times more data and network parameters [1]}. Training such data hungry models with hundreds of millions of parameters requires access to large GPU clusters, which is impractical for resource-constrained applications. Reduction in the computational requirements of vision transformers have been made possible by architectural changes that provide image specific inductive biases creating hybrid models with elements including distillation [2]}, convolutional embeddings [3]}, [4]}, convolutional tokens [5]}, and encoding overlapping patches [6]}. The quadratic complexity with respect to the sequence length (number of pixels) for vanilla transformers has also led to the search for other linear approximations of self-attention to efficiently mix tokens [3]}.
