The rank loss used in<cite> Kiros et al. (2014)</cite> , Socher et al. (2014) , and Karpathy & Fei-Fei (2015) is defined as follows: where α represents the margin,ĉ denotes a negative caption for the query image i, andî a negative image for the query caption c. Here, we used a shorthand notation [x] + ≡ max(x, 0).