@inproceedings{
liu2024right,
title={Right this way: Can {VLM}s Guide Us to See More to Answer Questions?},
author={Li Liu and Diji Yang and Sijia Zhong and Kalyana Suma Sree Tholeti and Lei Ding and Yi Zhang and Leilani H. Gilpin},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=7ANmKBfP88}
}
In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating ``where to know'' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.
Right this way: Can VLMs Guide Us to See More to Answer Questions?
Li Liu Diji Yang Sijia Zhong Kalyana Suma Sree Tholeti Lei Ding Yi Zhang Leilani H. Gilpin