@inproceedings{
yue2024adaptive,
title={Adaptive Selection based Referring Image Segmentation},
author={Pengfei Yue and Jianghang Lin and Shengchuan Zhang and Jie Hu and Yilin Lu and Hongwei Niu and Haixin Ding and Yan Zhang and GUANNAN JIANG and Liujuan Cao and Rongrong Ji},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=tVrwpFjsBv}
}
Referring image segmentation (RIS) aims to segment a particular region based on a specific expression. Existing one-stage methods have explored various fusion strategies, yet they encounter two significant issues. Primarily, most methods rely on manually selected visual features from the visual encoder layers, lacking the flexibility to selectively focus on language-preferred visual features. Moreover, the direct fusion of word-level features into coarse aligned features disrupts the established vision-language alignment, resulting in suboptimal performance. In this paper, we introduce an innovative framework for RIS that seeks to overcome these challenges with adaptive alignment of vision and language features, termed the Adaptive Selection with Dual Alignment (ASDA). ASDA innovates in two aspects. Firstly, we design an Adaptive Feature Selection and Fusion (AFSF) module to dynamically select visual features focusing on different regions related to various descriptions. AFSF is equipped with scale-wise feature aggregator to provide hierarchically coarse features that preserve crucial low-level details and provide robust features for successor dual alignment.  Secondly, a Word Guided Dual-Branch Aligner (WGDA) is leveraged to integrate coarse features with linguistic cues by word-guided attention, which effectively addresses the common issue of vision-language misalignment by ensuring that linguistic descriptors directly interact with masks prediction. This guides the model to focus on relevant image regions and make robust prediction. Extensive experimental results demonstrate that our ASDA framework surpasses state-of-the-art methods on RefCOCO, RefCOCO+ and G-Ref benchmark. The improvement not only underscores the superiority of ASDA in capturing fine-grained visual details but also its robustness and adaptability to diverse descriptions.
Adaptive Selection based Referring Image Segmentation
Pengfei Yue Jianghang Lin Shengchuan Zhang Jie Hu Yilin Lu Hongwei Niu Haixin Ding Yan Zhang GUANNAN JIANG Liujuan Cao Rongrong Ji