Datasets and bias: Data has been a central element for machine learning progress [1]}, [2]}, [3]} in the last two to three decades. The ImageNet challenge [4]} and the development of AlexNet [5]} sparked the deep learning era. But as datasets grow, biases emerge which may go undetected for a long time. For instance, the background in ImageNet can reveal information about the object class [6]}. Also, with the increasing popularity of crowdsourcing systems like Amazon Mechanical Turk, many datasets are annotated in uncontrolled environments. Different annotators are hence injecting unknown socio-economic properties into dataset annotations [7]}. Those dataset biases can be detrimental to the considered task. For instance, meticulously collected visual question answering (VQA) data [8]} aims to provide a platform for exciting research to advance image-language understanding. However, it is non-trivial to remove biases from this type of data. Indeed, it was reported that the question solely is sufficient to detect the correct answer [9]}, [10]}, i.e., no image information is required. In an attempt to fix this bias, the dataset was re-annotated [11]}, or the train and test split were re-organized [12]}.
Similarly, [13]} reported that in Audio-visual-scene-aware dialog (AVSD) [14]}, the question cue is often stronger, making the desired video and sound reasoning implausible or unnecessary. Likewise, SNLI [15]}, aims to determine the correctness of a hypothesis given a premise. However, [16]} point out that an internal bias exists: linguistic features not related to the premise correlate with the label. Further, the Story Cloze [17]} dataset permits to develop models which estimate the correct ending of a story. [18]} show that, for this dataset, length alone is a powerful feature to determine the correct ending.
