Calibration details.
To compute the expected calibration error (ECE) we follow the code of [1]} with their default settings using 15 equally-sized bins to compute the calibration error.
However, we change the implementation of the temperature rescaling. Since optimization of ECE over the softmax temperature is a simple one-dimensional optimization problem, it can be solved efficiently using a grid search. Moreover, we can optimize directly the metric of interest, i.e. ECE, instead of the cross-entropy loss as in [1]} who relied on a differentiable loss since they used gradient descent to optimize the temperature.
As the grid, we use the interval \(t \in [0.001, 1.0]\)  with a grid step \(0.001\)  and we test both \(t\)  and \(1/t\)  temperatures. Moreover, we make sure that for all methods the optimal \(t\)  is located not at the boundary of the grid.
We optimize the temperature only on the in-distribution samples from the test sets of CIFAR-10 and ImageNet-100 to make sure that the out-distribution samples from CIFAR-10-C and ImageNet-100-C stay unseen.
<FIGURE>