@inproceedings{
zhang2024overcoming,
title={Overcoming the Pitfalls of Vision-Language Model for Image-Text Retrieval},
author={Feifei Zhang and Sijia Qu and Fan Shi and Changsheng Xu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=jNKZRGPlIr}
}
This work tackles the persistent challenge of image-text retrieval, a key problem at the intersection of computer vision and natural language processing. Despite significant advancements facilitated by large-scale Contrastive Language-Image Pretraining (CLIP) models, we found that existing methods fall short in bridging the fine-grained semantic gap between visual and textual representations, particularly in capturing the nuanced interplay of local visual details and the textual descriptions.  To address the above challenges, we propose a general framework called Local and Generative-driven Modality Gap Correction (LG-MGC), which devotes to simultaneously enhancing representation learning and alleviating the modality gap in cross-modal retrieval. Specifically, the proposed model consists of two main components: a local-driven semantic completion module, which complements specific local context information that overlooked by traditional models within global features, and a generative-driven semantic translation module, which leverages generated features as a bridge to mitigate the modality gap. This framework not only tackles the granularity of semantic correspondence and improves the performance of existing methods without requiring additional trainable parameters, but is also designed to be plug-and-play, allowing for easy integration into existing retrieval models without altering their architectures. Extensive qualitative and quantitative experiments demonstrate the effectiveness of LG-MGC by achieving consistent state-of-the-art performance over strong baselines. \emph{\color{magenta}The code is included in the supplementary material.}
Overcoming the Pitfalls of Vision-Language Model for Image-Text Retrieval
Feifei Zhang Sijia Qu Fan Shi Changsheng Xu