First, we start by studying the impact of the pre-training dataset size, by varying the number of ImageNet examples we use to train models.
We consider subsets of ImageNet containing 10% and 1% of the total number of examples, and use the balanced (in terms of classes) subsets from [1]}.
To decouple the effect of using smaller datasets and the effect of doing less training updates, we adapt the number of epochs to keep the number of iterations constant.
This means that we perform 3k and 30k epochs on ImageNet 10% and 1% respectively.
We report results in Table REF . Observe how pre-training with an autoencoder loss such as masked image modeling is robust to the reduction in dataset size.
In contrast, like for supervised pre-training, the performance of models pre-trained with DINO self-supervision degrades when training with smaller datasets.
