Our image encoder is an off-the-shelf ResNet or ViT-B/16 model, that have been weakly-supervised trained on billions of Instagram images using hashtags [1]}.
Our text encoder and multimodal fusion encoder are initialized from XLM-R [2]}, an unsupervised multi-lingual language model.
Specifically, the text encoder inherits its first \(K\)  layers and the multimodal fusion encoder inherits its remaining \(M\)  layers, whose sum equals to XLM-R's total layers (6, 12, or 24 depending on the chosen model size).
We keep \(K\)  and \(M\)  unchanged under most setting for a fixed model architecture.
