We propose Visual Query Detection (VQD), a new visual grounding task. In VQD, a system is guided by natural language to localize a number of objects in an image. VQD is related to visual referring expression recognition, where the task is to localize only object. We describe the first dataset for VQD and we propose baseline algorithms that demonstrate the difficulty of the task compared to referring expression recognition.
The Visual7W dataset for VQA includes a pointing' task that is closely related to RER @cite_24 . Pointing questions require choosing which box of the four boxes correctly answered a query. Systems did not generate their own boxes, and there is always one correct box.
@cite_24
2136462581
We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model’s capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.