Identifying how much a modelknows about the stochastic real-world processit was trained on is important to ensure it avoids producing incorrect or “hallucinated” answers or taking unsafe actions.
But this is difficult
for generative models
because
probabilistic predictions do not distinguish between per-response noise (aleatoric uncertainty)  and lack of knowledge about the process (epistemic uncertainty) , and existing epistemic uncertainty quantification techniques tend to
be overconfident
when the model underfits.
We propose a general strategy for teaching a
model to both approximateand also estimate
the remaining gaps betweenand:
train it to predictpairsof independent responses drawn from the true conditional distribution, allow it to “cheat” by observing one response while predicting the other, then measure how much it cheats.
Remarkably, we prove that being good at cheating (i.e. cheating whenever it improves your prediction)  is
equivalent to beingsecond-order calibrated, a principled extension of ordinary calibration that allows us to construct provably-correct frequentist confidence intervals forand detect incorrect responses with high probability.
We demonstrate empirically that our approach accurately estimates how much models don’t know across ambiguous image classification, (synthetic)  language modeling, and partially-observable navigation tasks, outperforming existing techniques.