Recently, [1]} proposed SocialIQ, a dataset intended to reason about social situations in videos, specifically, emotion detection in a social situation. In this dataset, given a video and a question, the correct social situation should be recognized, e.g., “The man is upset because he is being insulted.” We show that it is easy to pick the correct statement using only the text data. This is possible because the statement's correctness often correlates with the statement's sentiment. Again, much like for VQA and AVSD, image information doesn't seem to be necessary for reasonably accurate performance. Hence, biases exist which permit to `address the dataset' without addressing the task. For SocialIQ, we attribute these biases to the fact that social reasoning is considered difficult, even for humans. Hence an annotator's expertise is particularly important.
