@inproceedings{
pitas2023something,
title={Something for (almost) nothing: improving deep ensemble calibration using unlabeled data},
author={Konstantinos Pitas and Julyan Arbel},
booktitle={Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023)},
year={2023},
url={https://openreview.net/forum?id=VN4tOgbRZU}
}
We present a method to improve the calibration of deep ensembles in the small training data regime in the presence of unlabeled data. Our approach is extremely simple to implement: given an unlabeled set, for each unlabeled data point, we simply fit a different randomly selected label with each ensemble member. We provide a theoretical analysis based on a PAC-Bayes bound which guarantees that if we fit such a labeling on unlabeled data, and the true labels on the training data, we obtain low negative log-likelihood and high ensemble diversity on testing samples. Crucially, each ensemble member can be trained independently from the rest (apart from the final validation/test step) making a parallel or distributed implementation extremely easy.
Something for (almost) nothing: improving deep ensemble calibration using unlabeled data
Konstantinos Pitas Julyan Arbel