Rethinking Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
Vision-and-language (V&L) models pretrained on large-scale multimodal data
have demonstrated strong performance on various tasks such as image captioning
and visual question answering (VQA). The quality of such models is commonly
assessed by measuring their performance on unseen data that typically comes
from the same distribution as the training data. However, we observe that these
models exhibit poor out-of-distribution (OOD) generalization on the task of
VQA. To better understand the underlying causes of poor generalization, we
comprehensively investigate performance of two pretrained V&L models under
different settings (i.e. classification and open-ended text generation) by
conducting cross-dataset evaluations. We find that these models tend to learn
to solve the benchmark, rather than learning the high-level skills required by
the VQA task. We also argue that in most cases generative models are less
susceptible to shifts in data distribution, while frequently performing better
on our tested benchmarks. Moreover, we find that multimodal pretraining
improves OOD performance in most settings. Finally, we revisit assumptions
underlying the use of automatic VQA evaluation metrics, and empirically show
that their stringent nature repeatedly penalizes models for correct responses.