Most work on learning a joint space for images and their descriptions is based on Canonical Correlation Analysis (CCA) or neural variants of CCA over representations of image and its descriptions (Hodosh et al., 2013; Andrew et al., 2013; Yan and Mikolajczyk, 2015; Gong et al., 2014; . Besides CCA, a few others learn a visual-semantic or multimodal embedding space of image descriptions and representations by optimizing a ranking cost function<cite> (Kiros et al., 2015</cite>; Socher et al., 2014; Vendrov et al., 2016) or by aligning image regions (objects) and segments of the description Plummer et al., 2015) in a common space.