All machine learning models are bound to make wrong predictions on a fraction of test data. Nonetheless, one would like the confidence of predictions to be proportional to the probability of being correct. Consider a test set of \(\lbrace x_i, y_i \rbrace _{i=1:N}\)  and suppose for sample \(x_i\)  the model predicts the class \(\hat{y}_i\)  with a probability \(\hat{p}_i\) . In the ideal scenario with perfect confidence calibration, \(\text{P}(\hat{y}=y | \hat{p}=p)= p\)  [1]}.
