Semi-supervised Learning.
Given the pretrained model, we use a small fraction of the ImageNet1k training labels (1% or 10%) for semi-supervised finetuning. We append the classification head on the first output layer of the projection head following SimCLR-v2 [1]}.
The finetuning protocol and data augmentation mainly follows BEiT [2]}.
The model is optimized using AdamW with an initial learning rate of 5e-6 for 1000 epochs and a batch size of 1024.
Comparison results are shown in Table REF .
ExtreMA outperforms the masked image modeling approaches MAE and BEiT by a large margin of 12% and 30% using 1% of the labels. Surprisingly, ExtreMA obtains better results than DINO, which is heavily tuned for ImageNet classification with higher linear probing performance than our approach.
The models pretrained with ImageNet22k perform worse, because the majority of unlabeled classes are less relevant to the target 1k classes. We thus follow the prior evaluation practice and omit the ImageNet22k entry.
