Vision-Language Pre-training (VLP) models align multi-modal data in common feature space by different pre-training tasks, which can be categorized into two categories: 1) VisualBert [1]}, UNITER [2]} and DALL-E [3]} use Language-based training strategy, including mask LM (Mask Language Modeling) such as Masked Language/Region Modeling, or autoregressive LM such as image caption and text-grounded image generation. 2) UNIMO [4]}, CLIP [5]}, ALIGN [6]} utilize cross-modal contrastive learning to align the visual and textual information into a unified semantic space.
