Look beyond labels: Incorporating functional summary information in Bayesian neural networks
Bayesian deep learning offers a principled approach to train neural networks
that accounts for both aleatoric and epistemic uncertainty. In variational
inference, priors are often specified over the weight parameters, but they do
not capture the true prior knowledge in large and complex neural network
architectures. We present a simple approach to incorporate summary information
about the predicted probability (such as sigmoid or softmax score) outputs in
Bayesian neural networks (BNNs). The available summary information is
incorporated as augmented data and modeled with a Dirichlet process, and we
derive the corresponding \emph{Summary Evidence Lower BOund}. We show how the
method can inform the model about task difficulty or class imbalance. Extensive
empirical experiments show that, with negligible computational overhead, the
proposed method yields a BNN with a better calibration of uncertainty.