Ensembling combines the predictions of multiple models to improve upon the performance of a single model; see, e.g., [1]}, [2]}. Recently, ensembling over neural networks which only differ in their random seed (“deep ensembles”) has been shown to be a particularly strong baseline for a variety of benchmarks [3]}. This result, in turn, prompted further research into alternative ensemble methods, including ensembling over hyper-parameters [4]}, architectures [5]}, and joint ensemble training [6]}. In parallel, several hypotheses have been proposed to explain the performance of deep ensembles. [7]} showed empirically that deep ensembles explore diverse modes in the loss landscape. [8]} argued that the effectiveness of deep ensembles hinges on the assumption that inputs to the model have multiple correct features, which will be learned by the different ensemble members. [9]}, [10]} analyzed deep ensembling as a Bayesian averaging procedure. [11]}, [12]} focus on understanding the relationship between the generalization of neural networks and the diversity for deep ensembles, both from theoretical and empirical perspectives. [13]}, [14]} considered the interplay between model and ensemble size.
