While Deep Neural Networks (DNNs) have become the go-to approach in computer vision, the vast majority of these models fail to properly capture the uncertainty inherent in their predictions. Estimating this predictive uncertainty can be crucial, for instance in automotive applications. In Bayesian deep learning, predictive uncertainty is often decomposed into the distinct types of aleatoric and epistemic uncertainty. The former can be estimated by letting a DNN output the parameters of a probability distribution. Epistemic uncertainty estimation is a more challenging problem, and while different scalable methods recently have emerged, no comprehensive comparison has been performed in a real-world setting. We therefore accept this task and propose an evaluation framework for predictive uncertainty estimation that is specifically designed to test the robustness required in real-world computer vision applications. Using the proposed framework, we perform an extensive comparison of the popular ensembling and MC-dropout methods on the tasks of depth completion and street-scene semantic segmentation. Our comparison suggests that ensembling consistently provides more reliable uncertainty estimates. Code is available at this https URL.
Lakshminarayanan @cite_10 created a parametric model @math of the conditional distribution using a DNN @math , and learned multiple point estimates @math by repeatedly minimizing the MLE objective @math with random initialization. They then averaged over the corresponding parametric models to obtain the predictive distribution, The authors considered this a non-Bayesian alternative to predictive uncertainty estimation. However, since @math always can be seen as samples from some distribution @math , we note that is virtually identical to . Ensembling can thus also be viewed as approximate Bayesian inference, where the level of approximation is determined by how well the implicit sampling distribution @math approximates the posterior @math . Ideally, we want @math to be distributed exactly according to @math . Since @math is highly in the parameter space for DNNs @cite_11 @cite_42 , so is @math . By minimizing @math multiple times, starting from initial points, we are likely to end up in different local optima. Ensembling can thus generate a compact set of samples @math that, even for small values of @math , captures this important aspect of multi-modality in @math .
@cite_42 @cite_10 @cite_11
2963238274 2101762657
Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet. We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension.