@inproceedings{
bin2024leveraging,
title={Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning},
author={Yi Bin and Junrong Liao and Yujuan Ding and Haoxuan Li and Yang Yang and See-Kiong Ng and Heng Tao Shen},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=XkQleH84zK}
}
Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information, thereby understanding and creating content of the physical world coherently like human-beings. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. Despite of the effectiveness, labeled associated coherency information is not always available and might be costly to acquire, making the cross-modal guidance hard to leverage. To tackle this challenge, this paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency, and proposes the Weak Cross-Modal  Guided Ordering (WeGO) model. More specifically, it leverages high-confidence predicted pairwise order in one modality as reference information to guide the coherence  modeling in another. An iterative learning paradigm is further designed to jointly optimize the coherence modeling in two modalities with selected guidance from each other. The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality. Experimental results on two public datasets have demonstrated that the proposed method outperforms existing methods for cross-modal coherence modeling tasks. Major technical modules have been evaluated effective through ablation studies. \textcolor{blue}{Codes are available at: \textit{\url{https://github.com/scvready123/IterWeGO}}}.
Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning
Yi Bin Junrong Liao Yujuan Ding Haoxuan Li Yang Yang See-Kiong Ng Heng Tao Shen