@inproceedings{
zohar2023lovm,
title={{LOVM}: Language-Only Vision Model Selection},
author={Orr Zohar and Shih-Cheng Huang and Kuan-Chieh Wang and Serena Yeung},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=7Dd8uBHo90}
}
Pre-trained multi-modal vision-language models (VLMs) excel in downstream applications, especially in the few- and zero-shot settings. 
However, choosing the optimal VLM for some downstream applications is challenging due to task and dataset dependencies. 
Exhaustive evaluation of all VLMs is impractical and requires the collection of a labeled dataset for evaluation. As the number of open-source VLM variants increases, there is a need for an efficient model selection strategy that does not require access to a curated evaluation dataset. To address this, we introduce a novel task, LOVM: **L**anguage-**O**nly  **V**ision **M**odel Selection, where methods are expected to perform both model selection and performance prediction based solely on a text description of the desired downstream application. We also present an extensive LOVM benchmark consisting of ground-truth evaluations of 23 pre-trained VLMs and 35 datasets, enabling effective ranking and performance prediction of VLMs. Our code, full paper, and dataset are available at https://github.com/orrzohar/LOVM.
LOVM: Language-Only Vision Model Selection
Orr Zohar Shih-Cheng Huang Kuan-Chieh Wang Serena Yeung