Binning strategy. Fix a granularity parameter \(n\)  representing the number of target bins. We distinguish between a fixed binning strategy – in which \(\)  is formed by partitioning \([0,1]\)  interval into \(n\)  equally-spaced bins; and an adaptive binning strategy – in which \(\)  is formed in a way that depend on the classifier \(f\) . Specifically, such that each bin has equal mass under \(c\circ f\) . In terms of measuring the calibration error, this is the different between the Expected Calibration Error (ECE) [1]} and the Adaptive Calibration Error (ACE) [2]}. Since our focus is on comparing different models in terms of their calibration error[2]} note several important limitations of the fixed binning strategy. Namely, that when predictions are skewed, many regions of the \([0,1]\)  interval sparsely populated, so only a few bins are “active”., we prefer to use adaptive binning.
