For evaluation, we consider both the accuracy and uncertainty on predicting \(y\)  given \(x\) . Many metrics have been proposed to evaluate the quality of uncertainty estimation. On one hand, researchers are generating calibrated probability estimates to measure model confidence [1]}, [2]}, [3]}. While expected calibration error and maximum calibration error
have been proposed to quantitatively measure calibration, such metrics do not reflect how robust the probabilities are with noise injected into the network input, and cannot capture epistemic or model uncertainty [4]}. On the other hand, the entropy of the predictive distribution as well as the mutual information, between the predictive distribution and posterior over network weights, are used as metrics to capture both epistemic and aleatoric uncertainty [5]}. However, it is often unclear
how large the
entropy or mutual information is large enough to be classified as uncertain, so such metric only provides a relative uncertainty measure.
