Recent work extends ViT-based models to the MetaFormer architecture [1]} consisting of MLP blocks and unspecified token mixers. Selecting a token mixer is an essential design choice when building ViT-based models. The options are many—the conventional MHSA mixer with a global receptive field, more sophisticated shifted window attention [2]}, or a non-parametric operator like pooling [1]}.
