
DINO: self-distillation pre-training on ImageNet-1k dataset by [1]}.

BEiT: BERT pre-training on ImageNet-21k dataset by [2]}.

CLIP: Contrastive language-image pre-training on YFCC100M dataset by [3]}.

Sup21k: Supervised pre-training on ImageNet-21k dataset.

Sup1k: Supervised pre-training on ImageNet-1k dataset.

BEiT + Sup21k: BERT unsupervised pre-training first on ImageNet-21k dataset and then using the labels of ImageNet-21k to fine-tune the model.

<TABLE><TABLE><TABLE>