Can Calibration Improve Sample Prioritization?

Calibration can reduce overconfident predictions of deep neural networks, but can calibration also accelerate training? In this paper, we show that it can when used to prioritize some examples for performing subset selection. We study the effect of popular calibration techniques in selecting better subsets of samples during training (also called sample prioritization) and observe that calibration can improve the quality of subsets, reduce the number of examples per epoch (by at least 70%), and can thereby speed up the overall training process. We further study the effect of using calibrated pre-trained models coupled with calibration during training to guide sample prioritization, which again seems to improve the quality of samples selected.


Introduction
Calibration is a widely used technique in machine learning to reduce overconfidence in predictions. Modern deep neural networks are known to be overconfident classifiers or predictors, and calibrated networks provide trustworthy and reliable confidence estimates [1]. Hence, finding new calibration techniques and improving them has been an active area of research [1,6,10,11].
In this paper, we ask if calibration aids in accelerating training by using sample prioritization, i.e., we select training samples based on calibrated predictions to better steer the training performance. We explore different calibration techniques and focus on selecting a subset with the most informative samples during each epoch. We observe that calibration performed during training plays a crucial role in choosing the most informative subsets, which in turn accelerates neural network training. We then investigate the effect of an external pre-trained model which is well-calibrated (with larger capacity) on the sample selection process during training.
Our contributions are as follows, We provide an in-depth study analyzing the effect of various calibration techniques on sample prioritization during training. We also consider pre-trained calibrated target models and observe their effect on sample prioritization along with calibration during training. We benchmark our findings on widely used CIFAR-10 and CIFAR-100 datasets and observe the improved quality of the chosen subsets across different subset sizes, which ensures faster deep neural network training.

Problem Statement
We formulate the problem in the paper as follows. A calibration technique C is performed during training at each epoch, and a sample prioritization function a is then used to select the most informative samples for training each subsequent epoch. We use Expected Calibration Error (ECE) for model calibration [10], which measures the absolute difference between the model's accuracy and its confidence.
The paper discusses how a calibration technique C, when coupled with a sample prioritization function a, affects the performance (accuracy and calibration error (ECE)) of the model. In addition, we also observe if this phenomenon can aid in faster and more efficient training. We hypothesize a closer relationship between calibration and sample prioritization during training, wherein the calibrated model probabilities at each epoch are used by a sample prioritization criterion to select the most informative samples for training each subsequent epoch.

Calibration
Calibration is a technique that curbs overconfident predictions in deep neural networks, wherein the predicted (softmax) probabilities reflect true probabilities of correctness (better confidence estimates) [1]. In this paper, we consider various prominently used calibration techniques which are performed during training.

Label
Smoothing implicitly calibrates a model by discouraging overconfident prediction probabilities during training [9]. The one-hot encoded ground truth labels (y k ) are smoothened using a parameter α, that is y LS k = y k (1 − α) + α/K, where K is the number of classes. These smoothened targets y LS k and predicted outputs p k are then used to minimize the cross-entropy loss.
Mixup is a data augmentation method [14] which is shown to output well-calibrated predictive scores [13], and is again performed during training.
where x i and x j are two input data points that are randomly sampled, and y i and y j are their respective one-hot encoded labels. Here, λ ∼ Beta(α, α) with λ ∈ [0, 1].
Focal Loss is an alternative loss function to cross-entropy which yields calibrated probabilities by minimizing a regularized KL divergence between the predicted and target distributions [8].
where p is the probability assigned by the model to the ground-truth correct class, and γ is a hyperparameter. When compared with cross-entropy, Focal Loss has an added factor that encourages the samples predicted with correct classes to have lower probabilities. This enables the predicted distribution to have higher entropy, thereby helping avoid overconfident predictions.

Sample Prioritization
Sample prioritization is the process of selecting important samples during different stages of training to accelerate the training process of a deep neural network without compromising on performance. In this paper, we perform sample prioritization during training using Max Entropy, which is a de facto uncertainty sampling technique to select the most efficient samples at each epoch.
Max Entropy selects the most informative samples (top-k) that maximize the predictive entropy [12].

Pre-trained Calibrated Target models
Pre-trained models have been widely used in literature to obtain comprehensive sample representations before training a downstream task [5]. We use a pre-trained calibrated model with larger capacity which we call the target model [3] and use the Max Entropy estimates obtained from this target model at each epoch to select the samples, thereby guiding the corresponding epochs of the model during training. Further, we call the model which is being trained as the current model. In this paper, we perform sample prioritization with the target model in addition to calibrating the current model.

Experiments and Results
We perform our experiments on CIFAR-10 and CIFAR-100 [4] datasets with the setting mentioned in Section 2.1, and we use Resnet-34 [2] as the current model. For both datasets, the initial training set consisting of 50,000 samples is split into 90% training data and 10% validation data, while the test set contains 10,000 samples. In the sample prioritization setting, we start with 10 warm-up epochs in which all samples are selected during training (no subset selection), following which we select n% of total training samples in each epoch using the Max Entropy criterion. We choose different settings of subset sizes for each epoch with n -{10, 20, 30}.
We use the Stochastic Gradient Descent optimizer with initial learning rates of 0.01 and 0.1 for CIFAR-10 and CIFAR-100 respectively, trained for 200 epochs with a cosine annealing [7] scheduler, weight decay of 5e −4 and momentum of 0.9 for both datasets. The models are trained using a V100 GPU. We consider classification accuracies and ECEs across different calibration techniques with their respective parameter sweeps as follows:    As expected, we observe from Table 1 that the test accuracies reduce when the subset size becomes smaller. Moreover, Mixup consistently has higher test accuracies and low test ECEs across different subset sizes compared to other calibration techniques for both datasets. This could be attributed to Mixup being a data transformation/augmentation technique, thereby performing well even in the low-data regime. Further, Figure 2 shows that training with Mixup leads to a relatively balanced representation of classes in the chosen subsets. In contrast, Label Smoothing and Focal Loss are loss-based calibration techniques with no explicit transformation performed on the underlying training data. Here, we note that Focal Loss can be more effective when coupled with a post hoc calibration technique like temperature scaling [8]. However, this setting is not applicable in our paper since we explicitly focus on calibration techniques during training. In addition, Mixup and Focal Loss are known to outperform Label Smoothing [13,8].

Discussion on Results
Effectiveness of Target:

Conclusion
In this paper, we investigate whether existing calibration techniques improve sample prioritization during training. We empirically show that a deep neural network calibrated during training selects better subsets of samples than an uncalibrated model. We also demonstrate the effectiveness of pre-trained calibrated target models in guiding sample prioritization during training, thereby boosting calibration performance. Finally, since calibration aids in sample prioritization by improving the quality of subsets, it ensures faster neural network training.

A Common Samples between Epochs
Figures 3a and 3b show the percentage of common samples between consecutive epochs for CIFAR-10 and CIFAR-100 respectively across various calibration techniques, with Max Entropy and Random Sampling as the sample prioritization criteria with 30% subset size. In all settings, sample selection with Max Entropy leads to a significantly higher percentage of common samples between consecutive epochs throughout training, as opposed to random sampling which results in very few common samples. Although only a small percentage of samples change across epochs, sample prioritization with Max Entropy coupled with any calibration technique yields good classification performance.

B Validation Accuracies
We report the validation accuracies of CIFAR-10 and CIFAR-100 with 30% subset size for the current model across all calibration techniques with Max Entropy as the sample prioritization criterion in Figures 4a and 4b. (a) CIFAR-10 (b) CIFAR-100 Figure 4: Validation Accuracies (%) for both datasets with 30% subset size for current model.

C Reliability Diagrams
We report the reliability diagrams of CIFAR-10 and CIFAR-100 with 30% subset size for the current model across all calibration techniques with Max Entropy as the sample prioritization criterion in Figure 5 and 6.