@inproceedings{
wu2024compositional,
title={Compositional Generalization in Vision-Language Models uses the Language Modality only},
author={Chenwei Wu and Patrick haffner and Li Li and Stefano Ermon and Rong Ge},
booktitle={I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models},
year={2024},
url={https://openreview.net/forum?id=JZaTnRVuuN}
}
Compositionality is a common property in many modalities including text and images, but the compositional generalization of multi-modal models is not well-understood. In this paper, we identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We show that current attempts to improve compositional generalization rely on linguistic priors rather than on information in the image, as the strength of the language model in detecting sentences that are syntactically and semantically likely overwhelms the vision part of the model. We find in particular that a benchmark for compositionality mostly favors pure language models. Finally, we propose a new benchmark for compositionality without such linguistic priors
The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models
Chenwei Wu Patrick haffner Li Erran Li Stefano Ermon Rong Ge