Pre-training of effective and general representations applicable to a wide range of tasks in a domain is the key to the success of deep learning.
In computer vision, supervised classification on ImageNet [1]} has long been the dominant pre-training task which is manifested to be effective on a wide range of vision tasks, especially on the semantic understanding tasks, such as image classification [2]}, [3]}, [4]}, [5]}, [6]}, object detection [7]}, [8]}, [9]}, [10]}, semantic segmentation [11]}, [12]}, video action recognition [13]}, [14]}, [15]}, [16]} and so on.
Over the past several years, “masked signal modeling”, which masks a portion of input signals and tries to predict these masked signals, serves as a universal and effective self-supervised pre-training task for various domains, including language, vision, and speech.
After (masked) language modeling repainted the NLP field [17]}, [18]}, recently, such task has also been shown to be a competitive challenger to the supervised pre-training in computer vision [19]}, [5]}, [21]}, [22]}, [23]}, [24]}. That is, masked image modeling (MIM) pre-trained models achieve very high fine-tuning accuracy on a wide range of vision tasks of different nature and complexity.
