Given a classifier \(f\) , what should we choose as our confidence measure \(c\) ? One natural choice is to use the maximum class probability, \(c(f(x)) = \max _i f(x)_i\) . When \(f\)  is itself calibrated, this will give rise to a calibrated confidence measure. However, we don't necessarily expect models to be well-calibrated “out of the box”, especially not in the presence of distribution shifts. While it is common to post-process predictions to improve their calibration, this approach is not always feasible as it requires additional data, is observed to have limited success in settings of distributions shifts [1]}, [2]}, and may come at the cost of sharpness [3]}.
