@inproceedings{
schirmer2024beyond,
title={Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift},
author={Mona Schirmer and Dan Zhang and Eric Nalisnick},
booktitle={NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models},
year={2024},
url={https://openreview.net/forum?id=DwEENEkkLi}
}
Knowing if a model will generalize to data `in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback–Leibler divergence.  We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.
Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift
Mona Schirmer Dan Zhang Eric Nalisnick