Second, we validate the effectiveness of each pre-training task through a thorough ablation.
Comparing L2 and L3, our proposed MIM-kl (L3) achieves a clear gain over MLM (L2).
When applying both MIM-kl and MIM-fr, L4 further improves the performance.
This is a quite different observation compared with [1]}, [2]}, where previously MIM was not shown to be helpful.
The difference indicates the effectiveness of our proposed MIM tasks.
Interestingly, the combined MLM, MIM-kl, and MIM-fr in L5 does not quite outperform L4 (with only 0.01 gain on the Meta Average).
One possible reason might be the two MIM tasks overshadows the effect of MLM.
L6 and L7 adds ITM and ITC into the pre-training, both of which further improves the meta average.
Notably, ITC introduces a significant improvement on image-text retrieval tasks, i.e., T2I and I2T due to the same task has been well warmed up during pre-training.
