More recently, there have been attempts to combine the inductive prior of convolutional design with the scalability of transformers in hybrid architectures for image analysis to reduce the training requirements of neural networks (pre-training dataset size, parameters, floating point operations (FLOPs)) [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. The expensive training requirements of pure transformers also led to research into alternative token-mixing architectures that can replace self-attention [7]}, [8]}, [9]}, [10]}. These works indicate that it is worth exploring priors other than shift-invariance (via convolutions) for images to reduce the training requirements with little to no sacrifice of generalization. However, most of these architectures cannot be easily adapted for multiple image sizes and other vision tasks, such as segmentation and object detection [10]}.
