Loss function. Similar to the baselines, we adopt the DAMSM loss as image-text matching loss.
Moreover, we define the contrastive loss on pairs of two branches of input captions. We compute the contrastive loss to minimize the distance of textual representations related to the same image while maximizing those related to different images. We utilize the Normalized Temperature-scaled Cross Entropy Loss (NT-Xent) [1]}, [2]}, [3]} as the contrastive loss.
Given a pair, let \(sim(a, b) = a^T b / \Vert a \Vert  \Vert b\Vert \)  denote the dot product between \(l_2\)  normalized \(a\)  and \(b\) . Then the loss function for the \(i\) th sample is defined as
\(L(i)=-\log \frac{\exp (sim(u_i, u_j)/\tau )}{\sum _{k=1}^{2N}\mathbb {1}_{k\ne i}\exp (sim(u_i, u_k)/\tau )},\) 
