We have shown that a new loss function that uses only violation incurred by hard negatives drastically improves performance over the typical loss that sums the violations across the negatives, typically used in previous work <cite>(Kiros et al. (2014)</cite> ; Vendrov et al. (2015) ).