Vision-language pre-training focuses on how to connect visual concepts and language concepts. Early approaches [1]}, [2]}, [3]}, [4]}, [5]} was performs on some cleaned datasets with relatively small data scale. Therefore, those models usually need to be fine-tuned on some specific downstream tasks. Some recent works [6]}, [7]} have explored the benefits of large-scale noisy data obtained from web pages for vision-language pre-training. CLIP [6]}, as a representative work, employs a contrastive learning approach to distinguish the correct image-text pair in each training batch. Because many vision/language concepts are covered in large-scale data, the CLIP illustrates surprisingly strong capability on zero-shot image classification and image-text retrieval.
