@inproceedings{
alazraki2024how,
title={How (not) to ensemble {LVLM}s for {VQA}},
author={Lisa Alazraki and Lluis Castrejon and Mostafa Dehghani and Fantine Huot and Jasper Uijlings and Thomas Mensink},
booktitle={I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models},
year={2024},
url={https://openreview.net/forum?id=8Q84ensxZ1}
}
This paper studies ensembling in the era of Large Vision-Language Models (LVLMs). Ensembling is a classical method to combine different models to get increased performance. In the recent work on Encyclopedic-VQA the authors examine a wide variety of models to solve their task: from vanilla LVLMs, to models including the caption as extra context, to models augmented with Lens-based retrieval of Wikipedia pages. Intuitively these models are highly complementary, which should make them ideal for ensembling. Indeed, an oracle experiment shows potential gains from 48.8% accuracy (the best single model) all the way up to 67% (best possible ensemble). So it is a trivial exercise to create an ensemble with substantial real gains. Or is it?
How (not) to ensemble LVLMs for VQA
Lisa Alazraki Lluis Castrejon Mostafa Dehghani Fantine Huot Jasper Uijlings Thomas Mensink