Various tasks have been introduced for the multi-modal pre-training.
The most notable three are Masked Language Modeling (MLM), Masked Image Modeling (MIM), and Image-Text Matching (ITM) – which are the direct counterparts of the BERT objectives.
Several other variants of these three tasks have also been explored, such as predicting object tags [1]}, [2]}, masked region classification [3]}, [4]}, sequential caption generation [5]}, [6]}, image-text contrastive learning [7]}, [8]}, [9]}, etc.
Recent works [10]}, [11]} found MIM may not be essential for pre-training among those tasks.
In this work, we found that our proposed MIM is more effective than MLM.
Inspired by recent works of self-supervised learning on vision [12]}, [13]}, we propose to mask out image patches with larger proportion and follow MaskFeat [14]} to reconstruct other views of the whole image rather than recovering those masked regions only.
