DeVise [1]} generates visual-semantic embeddings by linearly transforming a visual embedding from a pre-trained deep neural network into the embedding space of the text representation. After DeVise, several visual semantic models have been developed by optimizing bi-directional pairwise ranking loss [2]}, [3]} and maximum mean discrepancy loss [4]}. Maximizing CCA (Canonical Correlation Analysis) [5]} is also a common way to acquire cross-modal representation. Yan et al. [6]} address the problem of matching images and text in a joint latent space learned with deep canonical correlation analysis. Dorfer et al. [7]} develop a canonical correlation analysis layer and then apply pairwise ranking loss to learn a common representation of image and text for information retrieval tasks. However, most image-text multi-modal studies focus on matching image and text. Few methods study the problem of unsupervised clustering of image-text pairs.
