@inproceedings{
dutta2023estimating,
title={Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data},
author={Shiladitya Dutta and Hongbo Wei and Lars van der Laan and Ahmed Alaa},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=f7mLYe95m4}
}
Foundation models are trained on vast amounts of data at scale using self-supervised learning, enabling adaptation to a wide range of downstream tasks. At test time, these models exhibit zero-shot capabilities through which they can classify previously unseen (user-specified) categories. In this paper, we address the problem of quantifying uncertainty in these zero-shot predictions. We propose a heuristic approach for uncertainty estimation in zero-shot settings using conformal prediction with web data. Given a set of classes at test time, we conduct zero-shot classification with CLIP-style models using a prompt template, e.g.,  ``an image of a <category>'', and use the same template as a search query to source calibration data from the open web. Given a web-based calibration set, we apply conformal prediction with a novel conformity score that accounts for potential errors in retrieved web data. We evaluate the utility of our proposed method in Biomedical foundation models; our preliminary results show that web-based conformal prediction sets achieve the target coverage with satisfactory efficiency on a variety of biomedical datasets.
Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data
Shiladitya Dutta Hongbo Wei Lars van der Laan Ahmed Alaa