We propose Visual Query Detection (VQD), a new visual grounding task. In VQD, a system is guided by natural language to localize a number of objects in an image. VQD is related to visual referring expression recognition, where the task is to localize only object. We describe the first dataset for VQD and we propose baseline algorithms that demonstrate the difficulty of the task compared to referring expression recognition.
Unlike VQA, RER algorithms must produce evidence to justify their outputs. A RER algorithm outputs a box around the image location matching the input string, making it easier to tell if an algorithm is behaving correctly. The RefCOCO and RefCOCO+ datasets for RER were collected from the two-player ReferIt' Game @cite_9 . The first player is asked to describe an outlined object and the second player has to correctly localize it from player one's description. The test datasets are futher split into the testA' and testB' splits. The split testA' contains object categories sampled randomly to be close to the original data distribution, while testB' contains objects sampled from the most frequent object categories, excluding categories such as sky', sand', floor', etc. Since, there is a time limit on the game, the descriptions are short, e.g., guy in a yellow t-shirt,' pink,' etc.
@cite_9
2251512949
In this paper we introduce a new game to crowd-source natural language referring expressions. By designing a two player game, we can both collect and verify referring expressions directly within the game. To date, the game has produced a dataset containing 130,525 expressions, referring to 96,654 distinct objects, in 19,894 photographs of natural scenes. This dataset is larger and more varied than previous REG datasets and allows us to study referring expressions in real-world scenes. We provide an in depth analysis of the resulting dataset. Based on our findings, we design a new optimization based model for generating referring expressions and perform experimental evaluations on 3 test sets.