Looking beyond ViT, we hope our practices and observations will serve as a blueprint for future work comparing pre-training methods for more advanced ViT derivatives, like Swin [1]} and MViT [2]}. To facilitate community development we will release code in Detectron2 [3]}.
