@inproceedings{
zhang2024rca,
title={{RCA}: Region Conditioned Adaptation for Visual Abductive Reasoning},
author={Hao Zhang and Ee Yeo Keat and Basura Fernando},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=0EzCOFDueS}
}
Vision foundational models (e.g., CLIP) show strong generalization on various downstream visual perception tasks. 
However, their ability to reason beyond mere perception is limited, as they are only pre-trained on image-text pairs that hold semantically equivalent meanings. 
To tackle this, we propose a simple yet effective \textit{Region Conditioned Adaptation} (RCA), a hybrid parameter-efficient fine-tuning method that equips the frozen CLIP with the ability to infer hypotheses from local visual cues. 
Specifically, the RCA contains two novel modules: regional prompt generator and Adapter$^\textbf{+}$. 
The prior encodes ''local hints'' and ''global contexts'' into visual prompts separately at fine and coarse-grained levels. 
The latter enhances the vanilla adapters with a newly designed Map Adapter, that directly steers the focus of attention map with trainable query and key projections. Finally, we train the RCA with a new Dual-Contrastive Loss to regress the visual feature simultaneously toward features of literal description (a.k.a. clue text) and plausible hypothesis (abductive inference text). The loss enables CLIP to maintain both perception and reasoning abilities. Experiments on the Sherlock visual abductive reasoning benchmark show that the RCA significantly outstands previous SOTAs, ranking the \nth{1} on the leaderboards (e.g., Human Acc: RCA 31.74 \textit{vs} CPT-CLIP 29.58, higher =better). We also validate the RCA is generalizable to local perception benchmarks like RefCOCO. We would open-source our codes for future research.
RCA: Region Conditioned Adaptation for Visual Abductive Reasoning
Hao Zhang Ee Yeo Keat Basura Fernando