Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks
Monte Carlo (MC) integration is the de facto method for approximating the
predictive distribution of Bayesian neural networks (BNNs). But, even with many
MC samples, Gaussian-based BNNs could still yield bad predictive performance
due to the posterior approximation's error. Meanwhile, alternatives to MC
integration tend to be more expensive or biased. In this work, we
experimentally show that the key to good MC-approximated predictive
distributions is the quality of the approximate posterior itself. However,
previous methods for obtaining accurate posterior approximations are expensive
and non-trivial to implement. We, therefore, propose to refine Gaussian
approximate posteriors with normalizing flows. When applied to last-layer BNNs,
it yields a simple \emph{post hoc} method for improving pre-existing parametric
approximations. We show that the resulting posterior approximation is
competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.