As it was already observed in previous work [1]}, [2]}, [3]}, we also see in many cases that self-supervised training outperforms supervised pre-training on ImageNet.
For example, on the iNaturalist datasets, training with the target task data alone (including a pre-training step) gives better results than pre-training on ImageNet with labels:
with a ViT-base model and the SplitMask method, we see an improvement of +2.7% in top-1 accuracy. As for the clipart, painting and sketch datasets, we see that SplitMask provides a competitive performance, outperforming an ImageNet pre-trained BEiT across all datasets for ViT-S.
However, for the aforementioned datasets, supervised pre-training achieves the best performance for both ViT-S and ViT-B.
