Masked Image Modeling.
Masked image modeling (MIM) is a sub-task of masked signal prediction, that masks a portion of input images, and lets the deep networks predict the masked signals conditioned on the visible ones.
We use SimMIM [1]}, a simple framework for masked image modeling, as the exampled framework of pre-trained image models in our visualizations and experiments, because it is simple, effective, and generally applicable.
SimMIM consists of four major components with simple designs: 1) random masking with a moderately large masked patch size (e.g., 32); 2) the masked tokens and image tokens are fed together to the encoder; 3) the prediction head is as light as a linear layer: 4) directly predicting raw pixels of RGB values as the target with the \(\ell _1\)  loss of direct regression.
With these simple designs, SimMIM can achieve state-of-the-art performance on ImageNet-1K classification, COCO object detection, and ADE-20K semantic segmentation.
Note that, the SimMIM framework could be directly applied to different types of backbone architectures, such as Vision Transformer (ViT) [2]}, Swin Transformer [3]}, and ConvNets [4]}, [5]}. This property enables us to study the characteristics of MIM under different types of backbone architectures, as well as in multiple types of downstream tasks.
