@inproceedings{
chen2024querymatch,
title={QueryMatch: A Query-based Contrastive Learning Framework for Weakly Supervised Visual Grounding},
author={Shengxin Chen and Gen Luo and Yiyi Zhou and Xiaoshuai Sun and GUANNAN JIANG and Rongrong Ji},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=eLpaw4H2Nh}
}
Visual grounding is a task of locating  the object referred by a natural language description.   To reduce annotation costs, recent researchers are devoted into one-stage weakly supervised methods for visual grounding, which typically adopt the anchor-text matching paradigm. Despite the efficiency, we identify that anchor representations are often noisy and insufficient to describe object information, which inevitably hinders the vision-language alignments.  In this paper, we propose a novel query-based one-stage framework for weakly supervised visual grounding, namely QueryMatch. Different from previous work, QueryMatch represents candidate objects with a set of query features, which  inherently establish accurate one-to-one associations with visual objects.  In this case,  QueryMatch  re-formulates weakly supervised visual grounding as a query-text matching problem, which can be optimized via the query-based contrastive learning.  Based on QueryMatch, we further propose an innovative strategy for effective weakly supervised learning, namely Negative Sample Quality Estimation (NSQE). In particular, NSQE aims to augment negative training samples by actively selecting high-quality  query features. Though this strategy, NSQE can greatly  benefit the weakly supervised learning of QueryMatch. To validate our approach,  we conduct extensive experiments on three benchmark datasets of two grounding tasks, i.e., referring expression comprehension (REC) and segmentation (RES). Experimental results not only show the state-of-art performance of QueryMatch in two tasks, e.g., over +5\% IoU@0.5 on RefCOCO in REC and over +20\% mIOU on RefCOCO in RES, but also confirm the effectiveness of NSQE in weakly supervised learning. 
Source codes are available at~\url{https://anonymous.4open.science/r/QueryMatch-A82C}.
QueryMatch: A Query-based Contrastive Learning Framework for Weakly Supervised Visual Grounding
Shengxin Chen Gen Luo Yiyi Zhou Xiaoshuai Sun GUANNAN JIANG Rongrong Ji