In this paper , we aim to understand whether current language and vision ( LaVi ) models truly grasp the interaction between the two modalities . To this end , we propose an extension of the MS-COCO dataset , FOIL-COCO , which associates images with both correct and ' foil ' captions , that is , descriptions of the image that are highly similar to the original ones , but contain one single mistake ( ' foil word ' ) . We show that current LaVi models fall into the traps of this data and perform badly on three tasks : a ) caption classification ( correct vs. foil ) ; b ) foil word detection ; c ) foil word correction . Humans , in contrast , have near-perfect performance on those tasks . We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image . Most human language understanding is grounded in perception . There is thus growing interest in combining information from language and vision in the NLP and AI communities . So far , the primary testbeds of Language and Vision ( LaVi ) models have been ' Visual Question Answering ' ( VQA ) ( e.g. Antol et al . ( 2015 ) ; Malinowski and Fritz ( 2014 ) ; Malinowski et al . ( 2015 ) ; Gao et al . ( 2015 ) ; Ren et al . ( 2015 ) ) and ' Image Captioning ' ( IC ) ( e.g. Hodosh et al . ( 2013 ) ; Fang et al . ( 2015 ) ; Chen and Lawrence Zitnick ( 2015 ) ; Donahue et al . ( 2015 ) ; Karpathy and Fei-Fei ( 2015 ) ; Vinyals et al . ( 2015 ) ) . Whilst some models have seemed extremely successful on those tasks , it remains unclear how the reported results should be interpreted and what those Figure 1 : Is the caption correct or foil ( T1 ) ? If it is foil , where is the mistake ( T2 ) and which is the word to correct the foil one ( T3 ) ? models are actually learning . There is an emerging feeling in the community that the VQA task should be revisited , especially as many current dataset can be handled by ' blind ' models which use language input only , or by simple concatenation of language and vision features ( Agrawal et al . , 2016 ; Jabri et al . , 2016 ; Zhang et al . , 2016 ; Goyal et al . , 2016a ) . In IC too , Hodosh and Hockenmaier ( 2016 ) showed that , contrarily to what prior research had suggested , the task is far from been solved , since IC models are not able to distinguish between a correct and incorrect caption . Such results indicate that in current datasets , language provides priors that make LaVi models successful without truly understanding and integrating language and vision . But problems do not stop at biases . Johnson et al . ( 2016 ) also point out that current data ' conflate multiple sources of error , making it hard to pinpoint model weaknesses ' , thus highlighting the need for diagnostic datasets . Thirdly , existing IC evaluation metrics are sensitive to n-gram overlap and there is a need for measures that better simulate human judgments ( Hodosh et al . , 2013 ; Elliott and Keller , 2014 ; Anderson et al . , 2016 ) . Our paper tackles the identified issues by proposing an automatic method for creating a large dataset of real images with minimal language bias and some diagnostic abilities . Our dataset , FOIL ( Find One mismatch between Image and Language caption),1 consists of images associated with incorrect captions . The captions are produced by introducing one single error ( or ' foil ' ) per caption in existing , human-annotated data ( Figure 1 ) . This process results in a challenging error-detection / correction setting ( because the caption is ' nearly ' correct ) . It also provides us with a ground truth ( we know where the error is ) that can be used to objectively measure the performance of current models . We propose three tasks based on widely accepted evaluation measures : we test the ability of the system to a ) compute whether a caption is compatible with the image ( T1 ) ; b ) when it is incompatible , highlight the mismatch in the caption ( T2 ) ; c ) correct the mistake by replacing the foil word ( T3 ) . The dataset presented in this paper ( Section 3 ) is built on top of MS-COCO ( Lin et al . , 2014 ) , and contains 297,268 datapoints and 97,847 images . We will refer to it as FOIL-COCO . We evaluate two state-of-the-art VQA models : the popular one by Antol et al . ( 2015 ) , and the attention-based model by Lu et al . ( 2016 ) , and one popular IC model by ( Wang et al . , 2016 ) . We show that those models perform close to chance level , while humans can perform the tasks accurately ( Section 4 ) . Section 5 provides an analysis of our results , allowing us to diagnose three failures of LaVi models . First , their coarse representations of language and visual input do not encode suitably structured information to spot mismatches between an utterance and the corresponding scene ( tested by T1 ) . Second , their language representation is not finegrained enough to identify the part of an utterance that causes a mismatch with the image as it is ( T2 ) . Third , their visual representation is also too poor to spot and name the visual area that corresponds to a captioning error ( T3 ) . We have introduced FOIL-COCO , a large dataset of images associated with both correct and foil captions . The error production is automatically generated , but carefully thought out , making the task of spotting foils particularly challenging . By associating the dataset with a series of tasks , we allow for diagnosing various failures of current LaVi systems , from their coarse understanding of the correspondence between text and vision to their grasp of language and image structure . Our hypothesis is that systems which , like humans , deeply integrate the language and vision modalities , should spot foil captions quite easily . The SoA LaVi models we have tested fall through that test , implying that they fail to integrate the two modalities . To complete the analysis of these results , we plan to carry out a further task , namely ask the system to detect in the image the area that produces the mismatch with the foil word ( the red box around the bird in Figure 1 . ) This extra step would allow us to fully diagnose the failure of the tested systems and confirm what is implicit in our results from task 3 : that the algorithms are unable to map particular elements of the text to their visual counterparts . We note that the addition of this extra step will move this work closer to the textual / visual explanation research ( e.g. , ( Park et al . , 2016 ; Selvaraju et al . , 2016 ) ) . We will then have a pipeline able to not only test whether a mistake can be detected , but also whether the system can explain its decision : ' the wrong word is dog because the cyclists are in fact approaching a bird , there , in the image ' . LaVi models are a great success of recent research , and we are impressed by the amount of ideas , data and models produced in this stimulating area . With our work , we would like to push the community to think of ways that models can better merge language and vision modalites , instead of merely using one to supplement the other .
Despite the success of language and vision models on visual question answering tasks, what these models are learning remains unknown because of coarse-grained datasets.
They propose to automatically inject one mistake to captions in the MS-COCO dataset as a foil word and three diagnostic tasks to study models' behaviors.
Using the introduced dataset, they find that best performing models fail on the proposed tasks indicating their abilities to integrate two modalities.