The first set of pre-training tasks consists of 5 effective ones on the image-text pairs, including Masked Language Modeling (MLM), Masked Image Modeling KL-Divergence (MIM-kl), Masked Image Modeling Feature Regression (MIM-fr), Image-Text Contrastive Learning (ITC), and Image-Text Matching (ITM).
While previous works [1]}, [2]} show MIM is not helpful for multimodal pre-training and typically MLM is the most effective task [3]}, [4]}, our proposed MIM tasks are essential.
The key difference lies in (\(i\) ) we use a larger mask ratio of 50% instead of 15% used in [4]}, [6]}, [7]}; (\(ii\) ) we only recover the signal at image's [CLS] token instead of reconstructing every masked patch; (\(iii\) ) the supervision comes from the other two views (ClusterFit [8]} and GrokNet [9]}) of the intact raw image.
