Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in the broad comparison, we introduce the deep ensemble equivalent (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of very few independently trained networks in terms of the test log-likelihood. Deep neural networks (DNNs) have become one of the most popular families of machine learning models. The predictive performance of DNNs for classification is often measured in terms of accuracy. However, DNNs have been shown to yield inaccurate and unreliable probability estimates, or predictive uncertainty (Guo et al., 2017) . This has brought considerable attention to the problem of uncertainty estimation with deep neural networks. There are many faces to uncertainty estimation. Different desirable uncertainty estimation properties of a model require different settings and metrics to capture them. Out-of-domain uncertainty of the model is measured on data that does not follow the same distribution as the training dataset (out-of-domain data). Out-of-domain data can include images corrupted with rotations or blurring, adversarial attacks (Szegedy et al., 2013) or data points from a completely different dataset. The model is expected to be resistant to data corruptions and to be more uncertain on out-of-domain data than on in-domain data. This setting was explored in a recent study by (Ovadia et al., 2019) . On the contrary, in-domain uncertainty of the model is measured on data taken from the training data distribution, i.e. data from the same domain. In this case, a model is expected to provide correct probability estimates: it should not be overconfident in the wrong predictions, and should not be too uncertain about the correct predictions. Ensembles of deep neural networks have become a de-facto standard for uncertainty estimation and improving the quality of deep learning models (Hansen & Salamon, 1990; Krizhevsky et al., 2009; Lakshminarayanan et al., 2017) . There are two main directions in the field of training ensembles of DNNs: training stochastic computation graphs and obtaining separate snapshots of neural network weights. Methods based on the paradigm of stochastic computation graphs introduce noise over weights or activations of deep learning models. When the model is trained, each sample of the noise corresponds to a member of the ensemble. During test time, the predictions are averaged across the noise samples. These methods include (test-time) data augmentation, dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016) , variational inference (Blundell et al., 2015; Kingma et al., 2015; Louizos & Welling, 2017) , batch normalization (Ioffe & Szegedy, 2015; Teye et al., 2018; Atanov et al., 2019) , Laplace approximation (Ritter et al., 2018) and many more. Snapshot-based methods aim to obtain sets of weights for deep learning models and then to average the predictions across these weights. The weights can be trained independently (e.g., deep ensembles (Lakshminarayanan et al., 2017) ), collected on different stages of a training trajectory (e.g., snapshot ensembles (Huang et al., 2017) and fast geometric ensembles (Garipov et al., 2018) ), or obtained from a sampling process (e.g., MCMC-based methods (Welling & Teh, 2011; Zhang et al., 2019) ). These two paradigms can be combined. Some works suggest construction of ensembles of stochastic computation graphs (Tomczak et al., 2018) , while others make use of the collected snapshots to construct a stochastic computation graph (Wang et al., 2018; Maddox et al., 2019) . In this paper, we focus on assessing the quality of in-domain uncertainty estimation. We show that many common metrics in the field are either not comparable across different models or fail to provide a reliable ranking, and then address some of stated pitfalls. Following that, we perform a broad evaluation of modern DNN ensembles on CIFAR-10/100 and ImageNet datasets. To aid interpretatability, we introduce the deep ensemble equivalent score that essentially measures the number of "independent" models in an ensemble of DNNs. We draw a set of conclusions with regard to ensembling performance and metric reliability to guide future research practices. For example, we find that methods specifically designed to traverse different "optima" of the loss function (snapshot ensembles and cyclical SGLD) come close to matching the performance of deep ensembles while methods that only explore the vicinity of a single "optimum" (Dropout, FGE, K-FAC Laplace and variational inference) fall far behind. We have explored the field of in-domain uncertainty estimation and performed an extensive evaluation of modern ensembling techniques. Our main findings can be summarized as follows: • Temperature scaling is a must even for ensembles. While ensembles generally have better calibration out-of-the-box, they are not calibrated perfectly and can benefit from the procedure. Comparison of log-likelihoods of different ensembling methods without temperature scaling might not provide a fair ranking especially if some models happen to be miscalibrated. • Many common metrics for measuring in-domain uncertainty are either unreliable (ECE and analogues) or cannot be used to compare different methods (AUC-ROC, AUC-PR for misclassification detection; accuracy-confidence curves). In order to perform a fair comparison of different methods, one needs to be cautious of these pitfalls. • Many popular ensembling techniques require dozens of samples for test-time averaging, yet are essentially equivalent to a handful of independently trained models. Deep ensembles dominate other methods given a fixed test-time budget. The results indicate in particular that exploration of different modes in the loss landscape is crucial for good predictive performance. • Methods that are stuck in a single mode are unable to compete with methods that are designed to explore different modes of the loss landscape. Would more elaborate posterior approximations and better inference techniques shorten this gap? • Test-time data augmentation is a surprisingly strong baseline for in-domain uncertainty estimation and can significantly improve other methods without increasing training time or model size since data augmentation is usually already present during training. Our takeaways are aligned with the take-home messages of (Ovadia et al., 2019 ) that relate to indomain uncertainty estimation. We also observe a stable ordering of different methods in our experiments, and observe that deep ensembles with few members outperform methods based on stochastic computation graphs. A large number of unreliable metrics inhibits a fair comparison of different methods. Because of this, we urge the community to aim for more reliable benchmarks in the numerous setups of uncertainty estimation. Implied probabilistic model Conventional neural networks for classification are usually trained using the average cross-entropy loss function with weight decay regularization hidden inside an optimizer in a deep learning framework like PyTorch. The actual underlying optimization problem can be written as follows: where is the training dataset of N objects x i with corresponding labels y * i , λ is the weight decay scale andp(y * i = j | x i , w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x i . The cross-entropy loss defines a likelihood function p(y * | x, w) and weight decay regularization, or L 2 regularization, corresponds to a certain Gaussian prior distribution p(w). The whole optimization objective then corresponds to maximum a posteriori inference in the following probabilistic model: log p(y As many of the considered methods are probabilistic in nature, we use the same probabilistic model for all of them. We use the SoftMax-based likelihood for all models, and use the fully-factorized zero-mean Gaussian prior distribution with variances σ 2 = (N λ) −1 , where the number of objects N and the weight decay scale λ are dictated by the particular datasets and neural architectures, as defined in the following paragraph. In order to make the result comparable across all ensembling techniques, we use the same prababilistic model for all methods, choosing fixed weight decay parameters for each architecture. Conventional networks On CIFAR-10/100 datasets all networks were trained by SGD optimizer with batch size of 128, momentum 0.9 and model-specific parameters i.e., initial learning rate (lr init ), weight decay (wd), and number of optimization epoch (epoch). The specific hyperparameters are shown in Table 2 . The models used a unified learning rate scheduler that is shown in equation 10. All models have been trained using data augmentation that consists of horizontal flips, random crop of size 32 with padding 4. The standard data normalization has also been applied. Weight decays, initial learning rates, and the learning rate scheduler were taken from (Garipov et al., 2018) paper. Compared with hyperparameters of (Garipov et al., 2018) , the number of optimization epochs has been increased since we found that all models were underfitted. While original WideResNet28x10 includes number of dropout layers with p = 0.3 and 200 training epoch, in this setting we find that WideResNet28x10 underfits, and requires a longer training. Thus, we used p = 0, effectively it does not affect the final performance of the model in our experiments, but reduces training time. On ImageNet dataset we used ResNet50 examples with a default hyperparameters from PyTorch examples 5 . Specifically SGD optimizer with momentum 0.9, batch size of 256, initial learning rate 0.1, and with decay 1e-4. The training also includes data augmentation random crop of size 224 × 224, horizontal flips, and normalization, and learning rate scheduler lr = lr init · 0.1 epoch//30 , where // denotes integer division. We only deviated from standard parameters by increasing the number of training epochs from 90 to 130. Or models achived top-1 error of 23.81 ± 0.15 that closely matches accuracy of the ResNet50 probided by PyTorch which is 23.85 6 . Training of one model on a single NVIDIA Tesla V100 GPU takes approximately 5.5 days. Deep Ensembles Deep ensembles (Lakshminarayanan et al., 2017) average the predictions across networks trained independently starting from different initializations. To obtain Deep Ensemble we repeat the procedure of training standard networks 128 times for all architectures on CIFAR-10 and CIFAR-100 datasets (1024 networks over all) and 50 times for ImageNet dataset. Every single member of Deep Ensembles were actually trained with exactly the same hyperparameters as conventional models of the same arhitecture. Dropout The binary dropout (or MC dropout) (Srivastava et al., 2014; Gal & Ghahramani, 2016) is one of the most known ensembling techniques. It puts a multiplicative Bernoulli noise with parameter p over activations of ether fully-connected or convolutional layer, averaging predictions of the network w.r.t. the noise during test. The dropout layers have been applied to VGG, and WideResNet networks on CIFAR-10 and CIFAR-100 datasets. For VGG the dropout has been applied to fully-connected (fc) layers with p = 0.5, overall two dropout layers, one before the first fc-layer and one before the second one. While original version of VGG for CIFARs (Zagoruyko, 2015) exploits more dropout layers, we observed that any additional dropout layer deteriorates the performance on the model in ether deterministic or stochastic mode. For WideResNet network we applied dropout consistently with the original paper (Zagoruyko & Komodakis, 2016) with p = 0.3. The dropout usually increases the time to convergence, thus, VGG and WideResNet networks with dropout was trained for 400 epoch instead of 300 epoch for deterministic case. The all other hyperparameters was the same as in case of conventional models. Variational Inference The VI approximates a true posterior distribution p(w | Data) with a tractable variational approximation q θ (w), by maximizing so-called variational lower bound L (eq. 11) w.r.t. parameters of variational approximation θ. We used fully-factorized Gaussian approximation q(w), and Gaussian prior distribution p(w). In the case of such a prior p(w) the probabilistic model remains consistent with conventional training which corresponds to MAP inference in the same probabilistic model. We used variational inference for both convolutional and fully-connected layers, where variances of the weights was parameterized by log σ. For fully-connected layers we applied the LRT (Kingma et al., 2015) . While variational inference provide a theoretical grounded way to approximate a true posterior, on practice, it tends to underfit deep learning models (Kingma et al., 2015) . The following tricks are applied to deal with it: pre-training (Molchanov et al., 2017) Consistently with the practical tricks we use a pre-training, specifically, we initialize µ with a snapshot of the weights of pretrained conventional model, and initialize log σ with model-specific constant log σ init . The KL-divergence -except the term that corresponds to a weight decay -was scaled on model specific parameter β. The weigh decay term was implemented as a part of the optimizer. We used a fact that KL-divergence between two Gaussian distributions can be rewritten as two terms one of which is equal to wd regularization. On CIFAR-10 and CIFAR-100 we used β 1e-4 for VGG, ResNet100 and ResNet164 networks, and β 1e-5 for WideResNet. The initialization of log-variance log σ init was set to −5 for all models. Parameters µ were optimized with conventional SGD (with the same parameters as conventional networks, except initial learning rate lr init that was set to 1e-3). We used a separate Adam optimizer with constant learning rate 1e-3 to optimize log-variances of the weights log σ. The training was held for 100 epochs, that corresponds to 400 epochs of training (including pre-training). On ImageNet we used β = 1e-3, lr init = 0.01, log σ init = −6, and held training for a 45 epoch form a per-trained model.
We highlight the problems with common metrics of in-domain uncertainty and perform a broad study of modern ensembling techniques.
{"src": "scitldr-AIC_train", "paper_id": "BJxI5gHKDr"}