[1]} evaluate the calibration of pre-trained Transformer models in both ID and OOD settings. Their results demonstrate that Transformer-based models tend to well-calibrated ID but that the calibration error can decrease significantly OOD. [2]} empirically evaluate the relationship between scale and calibration, showing that OOD, smaller Transformer models tend to have worse calibration than larger models, even after applying TS. These works, together with earlier works [3]}, all conflate a model's confidence with the probability of the predicted label. One recent exception is [4]}, which consider confidence measures based on the margin and kurtosis of the logits. Their work is significantly different from ours as they do not directly evaluate calibration of these proposed measures and also do not consider distribution shifts.
