Feature Space Particle Inference for Neural Network Ensembles
Ensembles of deep neural networks demonstrate improved performance over
single models. For enhancing the diversity of ensemble members while keeping
their performance, particle-based inference methods offer a promising approach
from a Bayesian perspective. However, the best way to apply these methods to
neural networks is still unclear: seeking samples from the weight-space
posterior suffers from inefficiency due to the over-parameterization issues,
while seeking samples directly from the function-space posterior often results
in serious underfitting. In this study, we propose optimizing particles in the
feature space where the activation of a specific intermediate layer lies to
address the above-mentioned difficulties. Our method encourages each member to
capture distinct features, which is expected to improve ensemble prediction
robustness. Extensive evaluation on real-world datasets shows that our model
significantly outperforms the gold-standard Deep Ensembles on various metrics,
including accuracy, calibration, and robustness. Code is available at
https://github.com/DensoITLab/featurePI .