Then we use the margin-based triplet loss for the generated captions in two way: comparing with gold captions, and comparing between arbitrary two generated captions. Given \(N_c\)  generated candidate captions and \(N_r\)  referenced captions, we rank the generated captions in descending order based on the gold CIDEr-D [1]} scores where \(s^{\ast }_i = S^{\ast }(\widetilde{y}^i, Y)\) . The ranked captions are denoted by \(\widetilde{Y}_r = (\widetilde{y}^1_r, \widetilde{y}^2_r, ..., \widetilde{y}^{N_c}_r)\)  where \(\widetilde{y}^1_r\)  has the highest score in \(\widetilde{Y}_r\) . Then the loss is as follows:
\(\begin{aligned}\mathcal {L}_s(\theta ^s) &= \max \left(0, s(\widetilde{y}^i_r, x)-s(y^{\ast }, x) + \gamma _{1}\right)\\&+ \max \left(0, s(\widetilde{y}^j_r, x) - s(\widetilde{y}^i_r, x) + \gamma _{2}\right)\end{aligned}\) 
