Visual Pre-training. Throughout the deep learning era, supervised classification on ImageNet [1]} has been the dominant pretraining task. It is found to deliver strong finetuning performance on numerous semantic understanding tasks [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}. Over the past several years, self-supervised pretraining has attracted more and more attention, and achieved finetuning performance on par with the supervised counterparts on several representative downstream tasks [12]}, [13]}, including two representative ones, contrastive learning [14]}, [12]}, [13]}, [17]}, [18]} and masked image modeling [19]}, [20]}, [21]}, [22]}.
In our work, we focus on understanding the different behaviors of supervised and emergent MIM pre-training.
