Effective training-time stacking for ensembling of deep neural networks
Ensembling is a popular and effective method for improving machine learning
(ML) models. It proves its value not only in classical ML but also for deep
learning. Ensembles enhance the quality and trustworthiness of ML solutions,
and allow uncertainty estimation. However, they come at a price: training
ensembles of deep learning models eat a huge amount of computational resources.
  A snapshot ensembling collects models in the ensemble along a single training
path. As it runs training only one time, the computational time is similar to
the training of one model. However, the quality of models along the training
path is different: typically, later models are better if no overfitting occurs.
So, the models are of varying utility.
  Our method improves snapshot ensembling by selecting and weighting ensemble
members along the training path. It relies on training-time likelihoods without
looking at validation sample errors that standard stacking methods do.
Experimental evidence for Fashion MNIST, CIFAR-10, and CIFAR-100 datasets
demonstrates the superior quality of the proposed weighted ensembles c.t.
vanilla ensembling of deep learning models.