Vision-language tasks includes language-based image retrieval, image captioning, visual question answering, and text-guided synthesis among others. Typically, to solve these tasks, a cross-modal vision and language representation is learned [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}.
A common approach to learn such a representation is by training a transformer [10]}.
Recently, OpenAI introduced CLIP [11]} - a powerful model that learns a joint vision-language representation. CLIP was trained on 400 million text-image pairs, collected from a variety of publicly available sources on the Internet, using a contrastive learning goal. CLIP is composed of two encoders – an image encoder, and a text encoder – which aim to map their respective inputs into a joint, multi-modal embedding space. The representations learned by CLIP have been shown to be powerful enough to guide a variety of tasks, including image synthesis [12]}, [13]} and manipulation [14]}.
