We adopt the model architecture and training procedure of<cite> Kádár et al. (2018)</cite> for the task of matching images with sentences. This task is defined as learning to rank the sentences associated with an image higher than other sentences in the data set, and vice-versa (Hodosh et al., 2013) . The model is comprised of a recurrent neural network language model and a convolutional neural network image encoder. The parameters of the language encoder are randomly initialized, while the image encoder is pre-trained, frozen during training and followed by a linear layer which is tuned for the task. The model is trained to make true pairs < a, b > similar to each other, and contrastive pairs <â, b > and < a,b > dissimilar from each other in a joint embedding space by minimizing the max-violation loss function (Faghri et al., 2017) :