For CLIP, we used the visual CLIP-RN50 backbone (called CLIP hereafter), which was jointly trained along with a linguistic head (called CLIP-L hereafter) on a contrastive learning task on 400M image-caption pairs [1]}. Additionally we also considered visual features from TSM [2]}, another multimodal model that is trained with a contrastive objective on the HowTo100M dataset [3]}, in a task that comprises three modalities (video, text, and audio). The impact of contrastive learning objectives on the features of these models can be compared to VirTex and ICMLM, multimodal networks trained with different objectives. For Virtex, the visual backbone is trained on an image captioning task [4]}, while for ICMLM, the visual features are trained on a text-unmasking task [5]}. Both VirTex and ICMLM are trained on MS-COCO [6]}, a much smaller dataset compared to those used for CLIP or TSM.
