We compare with representative contrastive methods MoCo-v3 [1]} and DINO [2]}, as well as masked image modeling methods BeiT [3]} and MAE [4]} on ImageNet classification. We use our strongest model with five masks of ratio 80% and color augmentations. The finetuning takes 200 epochs for ViT-S and 100 epochs for ViT-B following prior works. The results are summarized in Table REF . Our approach achieves the best finetuning performance for both architectures. Our linear probing outperforms the masked image modeling methods by a large margin but underperforms contrastive counterparts. This may be due to the lack of global crops and other heavy image augmentations. The ViT-S model does not scale well with large data, potentially limited by its model size.
