@inproceedings{
yang2024accurate,
title={Accurate and Lightweight Learning for Specific Domain Image-Text Retrieval},
author={Rui Yang and Shuang Wang and Jianwei Tao and Yingping Han and Qiaoling Lin and Yanhe Guo and Biao Hou and Licheng Jiao},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=1t7RW2Ixps}
}
Recent advances in vision-language pre-trained models like CLIP have greatly enhanced general domain image-text retrieval performance. This success has led scholars to develop methods for applying CLIP to Specific Domain Image-Text Retrieval (SDITR) tasks such as Remote Sensing Image-Text Retrieval (RSITR) and Text-Image Person Re-identification (TIReID). However, these methods for SDITR often neglect two critical aspects: the enhancement of modal-level distribution consistency within the retrieval space and the reduction of CLIP's computational cost during inference, resulting in suboptimal retrieval spaces and unnecessarily high inference computational loads.
To address these issues, this paper presents a novel framework, Accurate and lightweight learning for specific domain Image-text Retrieval (AIR), based on the CLIP architecture. AIR incorporates a Modal-Level distribution Consistency Enhancement regularization (MLCE) loss and a Self-Pruning Distillation Strategy (SPDS) to improve retrieval precision and computational efficiency. The MLCE loss harmonizes the sample distance distributions within image and text modalities, fostering a retrieval space closer to the ideal state. Meanwhile, SPDS employs a strategic knowledge distillation process to transfer deep multimodal insights from CLIP to a shallower level, maintaining only the essential layers for inference, thus achieving model light-weighting.
Comprehensive experiments across various datasets in RSITR and TIReID demonstrate the effectiveness of both MLCE loss and SPDS. The study also explores the limits of SPDS's performance and compares it with conventional teacher-student distillation methods. The findings reveal that MLCE loss secures optimal retrieval on several datasets, while SPDS achieves a favorable balance between accuracy and computational demand during testing.
Accurate and Lightweight Learning for Specific Domain Image-Text Retrieval
Rui Yang Shuang Wang Jianwei Tao Yingping Han Qiaoling Lin Yanhe Guo Biao Hou Licheng Jiao