Out of distribution robustness with pre-trained Bayesian neural networks
We develop ShiftMatch, a new training-data-dependent likelihood for out of
distribution (OOD) robustness in Bayesian neural networks (BNNs). ShiftMatch is
inspired by the training-data-dependent "EmpCov" priors from Izmailov et al.
(2021a) and efficiently matches test-time spatial correlations to those at
training time. Critically, ShiftMatch is designed to leave neural network
training unchanged, allowing it to use publically available samples from
pretrained BNNs. Using pre-trained HMC samples, ShiftMatch gives strong
performance improvements on CIFAR-10-C, outperforms EmpCov priors, and is
perhaps the first Bayesian method capable of convincingly outperforming plain
deep ensembles. ShiftMatch can be integrated with non-Bayesian methods like
deep ensembles, where it offers smaller, but still considerable, performance
improvements. Overall, Bayesian ShiftMatch gave slightly better accuracy than
ensembles with ShiftMatch, though they both had very similar log-likelihoods.