We largely follow<cite> Kádár et al. (2018)</cite> , however, our main interest lies in learning multimodal and bilingual representations in the scenario where the images do not come from the same data set i.e.: the data is presented is two sets of image-caption tuples rather than image-caption-caption triples. Taking a broader perspective, images have been used as pivots in multilingual multimodal language processing. On the word level this intuition is applied to visually grounded bilingual lexicon induction, which aims to learn cross-lingual word representations without aligned text using images as pivots (Bergsma and Van Durme, 2011; Kiela et al., 2015; Vulić et al., 2016; Hartmann and Søgaard, 2017; Hewitt et al., 2018) . Images have been used as pivots to learn translation models only from image-caption data sets, without parallel text (Hitschler et al., 2016; Nakayama and Nishida, 2017; Lee et al., 2017; Chen et al., 2018) .