PAC Prediction Sets for Meta-Learning
Uncertainty quantification is a key component of machine learning models
targeted at safety-critical systems such as in healthcare or autonomous
vehicles. We study this problem in the context of meta learning, where the goal
is to quickly adapt a predictor to new tasks. In particular, we propose a novel
algorithm to construct \emph{PAC prediction sets}, which capture uncertainty
via sets of labels, that can be adapted to new tasks with only a few training
examples. These prediction sets satisfy an extension of the typical PAC
guarantee to the meta learning setting; in particular, the PAC guarantee holds
with high probability over future tasks. We demonstrate the efficacy of our
approach on four datasets across three application domains: mini-ImageNet and
CIFAR10-C in the visual domain, FewRel in the language domain, and the CDC
Heart Dataset in the medical domain. In particular, our prediction sets satisfy
the PAC guarantee while having smaller size compared to other baselines that
also satisfy this guarantee.