Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation
Masked image modeling (MIM) learns representations with remarkably good
fine-tuning performances, overshadowing previous prevalent pre-training
approaches such as image classification, instance contrastive learning, and
image-text alignment. In this paper, we show that the inferior fine-tuning
performance of these pre-training approaches can be significantly improved by a
simple post-processing in the form of feature distillation (FD). The feature
distillation converts the old representations to new representations that have
a few desirable properties just like those representations produced by MIM.
These properties, which we aggregately refer to as optimization friendliness,
are identified and analyzed by a set of attention- and optimization-related
diagnosis tools. With these properties, the new representations show strong
fine-tuning performance. Specifically, the contrastive self-supervised learning
methods are made as competitive in fine-tuning as the state-of-the-art masked
image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is
also significantly improved, with a CLIP ViT-L model reaching \textbf{89.0%}
top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter
SwinV2-G model, the fine-tuning accuracy on ADE20K semantic segmentation is
improved by +1.5 mIoU to \textbf{61.4 mIoU}, creating a new record. More
importantly, our work provides a way for the future research to focus more
effort on the generality and scalability of the learnt representations without
being pre-occupied with optimization friendliness since it can be enhanced
rather easily. The code will be available at
https://github.com/SwinTransformer/Feature-Distillation.