Apart from intra-modal feature learning, EGE-CMP is enable to construct coherent representations of multi-modal inputs and learn their alignment.
To do this, we leverage inter-modality contrastive learning [1]}, [2]} to align the visual and textual modalities.
There are \(2N\)  data points in total for a minibatch of \(N\)  image-text samples.
We consider the related image-text pairings to be \(N\)  positive pairs and the remaining \(2(N-1)\)  mismatched pairs to be negative pairs.
Formally, given an image-text pair \((x_i,x_j)\)  and their encoded features \((\tilde{x}_i, \tilde{x}_j)\) , the cross-modal contrastive loss for this positive pair is computed as:
\(\mathcal {L}_\mathrm {VT}(x_i, x_j)=-\log \frac{\exp \left(\operatorname{sim}\left(\tilde{x}_i, \tilde{x}_j\right) / \tau \right)}{\sum _{k=1}^{2 N} \mathbb {1}_{[k \ne i]} \exp \left(\operatorname{sim}\left(\tilde{x}_i, \tilde{x}_k\right) / \tau \right)},\) 
