Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs)  on various vision-language (VL)  tasks by guiding the model to attend more closely to these regions of interest.
For example, VLMs can be given a “visual prompt”, where visual markers such as bounding boxes delineate key image regions; this approach has become popular due to the improvement it provides in tasks requiring region-level information.
However, current VLMs that can incorporate visual guidance are either proprietary and expensive or require costly training on curated data that includes visual prompts.
We introduceContrastive Region Guidance(CRG) ,
a training-free guidance method that enables open-source VLMs to respond to visual prompts.CRGcontrasts model outputs produced with and without visual prompts,
factoring out biases revealed by the model when answeringwithoutthe information required to produce a correct answer (i.e., the model’s prior) .CRGachieves substantial improvements in a wide variety of VL tasks: When region annotations are provided,CRGincreases absolute accuracy by up toon ViP-Bench, a collection of six diverse region-based tasks such as recognition, math, and object relationship reasoning.
We also showCRG’s applicability to spatial reasoning, where we obtain up toimprovement on the hardest setting of What’sUp, as well as to compositional generalization – improving accuracy byandon two challenging splits from SugarCrepe – and to image-text alignment for generated images, where we improve by up toAUROC andF1 points on SeeTRUE.
For cases that do not have reference regions for the prompt, we also show thatCRGallows us to re-rank regions proposed by an object detection model in referring expression comprehension and phrase grounding benchmarks like RefCOCO/RefCOCO+/RefCOCOg and Flickr30K Entities, with an average improvement ofin accuracy when multiple proposals are available. In our analysis, we explore alternative masking strategies forCRG,
demonstrate howCRGimpacts the model’s probability over relevant text phrases, and evaluate the role of the region guidance strength, empirically validatingCRG’s design choices.