Image-Text Matching (ITM)
In ITM, the inputs are a paired sentence and image and the output is a binary label \(y\in \lbrace 0, 1\rbrace \) , indicating if each input pair is a match.
We extract the hidden output of \({\mathbf {w}}\textsuperscript {cls}\)  at the last layer of multimodal fusion to represent the fused representation of both modalities, then feed it into an FC layer followed by a sigmoid function to predict a single score between 0 and 1.
We denote the output score as \(o_{\theta }(\mathbf {w}, \mathbf {v})\) .
During pre-training, we sample a positive or negative pair \((\mathbf {w}, \mathbf {v})\)  from the dataset \(D\)  at each step.
The negative pair is created by replacing the image or text in a paired sample with a randomly-selected one from other samples.
Following [1]}, when ITC is applied, we could sample the hard negative pairs from ITC's computed similarity matrix \(s({\mathbf {w}}_i, {\mathbf {v}}_j)\) .
The incorporation of those hard negatives makes ITM a harder task, which is more beneficial for the pre-training [2]}.
We apply binary cross entropy for this loss:
\(\nonumber \mathcal {L}_{\text{ITM}}(\theta ) = - \mathbb {E}_{(\mathbf {w}, \mathbf {v})\sim D} [y \log o_{\theta }(\mathbf {w}, \mathbf {v}) + (1-y) \log (1-o_{\theta }(\mathbf {w}, \mathbf {v}))] ).\) 
