Image-Text Contrastive Learning (ITC)
Following [1]}, we add an image-text contrastive loss between the visual and textual embeddings right before feeding them into the multimodal fusion module.
It aims to align the two modalities into the same space before fusion.
Specifically, we project the textual embedding \({\mathbf {w}}_i\textsuperscript {cls}\)  and visual embedding at \({\mathbf {v}}_i\textsuperscript {cls}\)  to a normalized lower-dimensional representations via two linear transformations \(f\)  and \(g\) .
The similarity of the text \({\mathbf {w}}_i\)  and the image \({\mathbf {v}}_j\)  is then measured by the dot product of
\(\nonumber s({\mathbf {w}}_i, {\mathbf {v}}_j) = f_{\theta }({\mathbf {w}}_i\textsuperscript {cls})^T g_{\theta }({\mathbf {v}}_i\textsuperscript {cls})\) 
