A simple approach is to ignore the text and cluster the images alone. Unsupervised image clustering has received significant research attention in computer vision [1]}. However, as we will show, these single-view approaches fail to differentiate semantically different but visually similar subjects on benchmark datasets. On the other hand, using the captions alone (ignoring the image) is rarely considered viable, since captions do not fully describe the content of the image.
Current multi-modal image-text models focus on matching images and corresponding captions for information retrieval tasks [2]}, [3]} rather than unsupervised learning for image-text pairs. Jin et al. [4]} characterize correlations between image and text using Canonical Correlation Analysis (CCA). However, the textual information for the model comprised semi-structured tags rather than unstructured free-text descriptions. Free-text descriptions can capture more information to improve a model's performance, but unstructured text is often considered prohibitively noisy in practice: it can contain irrelevant or inconsistent information, and may even be associated with the wrong image. We find that these challenges have limited the uptake of machine learning in complex, human-intensive domains in science and the humanities.
