@inproceedings{
wang2024partially,
title={Partially Aligned Cross-modal Retrieval via  Optimal Transport-based Prototype Alignment Learning},
author={Junsheng Wang and Tiantian Gong and Yan Yan},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=z2aWdGeoSL}
}
Supervised cross-modal retrieval (CMR) achieves excellent performance thanks to the semantic information provided by its labels, which helps to establish semantic correlations between samples from different modalities.  However, in real-world scenarios, there often exists a large amount of unlabeled and unpaired multimodal training data, rendering existing methods unfeasible. To address this issue, we
propose a novel partially aligned cross-modal retrieval method called Optimal Transport-based Prototype Alignment Learning (OTPAL). Due to the high computational complexity involved in directly establishing matching correlations between unannotated unaligned cross-modal samples, instead, we establish matching correlations between shared prototypes and samples. To be specific, we employ the optimal transport algorithm to establish cross-modal alignment information between samples and prototypes, and then minimize the distance between samples and their corresponding prototypes through a specially designed prototype alignment loss. As an extension of this paper, we also extensively investigate the influence of incomplete multimodal data on cross-modal retrieval performance under the partially aligned setting proposed above. To further address the above more challenging scenario, we raise a scalable prototype-based neighbor feature completion method, which better captures the correlations between incomplete samples and neighbor samples through a cross-modal self-attention mechanism. Experimental results on four benchmark datasets show that our method can obtain satisfactory accuracy and scalability in various real-world scenarios.
Partially Aligned Cross-modal Retrieval via  Optimal Transport-based Prototype Alignment Learning
Junsheng Wang Tiantian Gong Yan Yan