Combine contextual dropout with Deep Ensemble: Deep ensemble proposed by [1]} is a simple way to obtain uncertainty by ensembling models
trained
independently from different random initializations. In Figure REF ,
we show the performance
of combining different dropouts with deep ensemble on noisy MNIST data.
As the number of NNs increases, both accuracy and PAvPU increase for all dropouts. However, Bernoulli
contextual dropout outperforms
other dropouts by a large margin in both metrics, showing
contextual dropout is compatible with deep ensemble and their combination can lead to significant
improvements.
Out of distribution (OOD) evaluation: we evaluate different dropouts in an OOD setting, where we train our model with clean data but test it on noisy data. Contextual dropout achieves accuracy of \(78.08\) , consistently higher than MC dropout (\(75.22\) ) or concrete dropout (\(74.93\) ). Meanwhile, the proposed method is also better at uncertainty estimation with PAvPU of \(78.49\) , higher than MC (\(74.61\) ) or Concrete (\(75.49\) ).
