Excess risk analysis for epistemic uncertainty with application to variational inference
We analyze the epistemic uncertainty (EU) of supervised learning in Bayesian
inference by focusing on the excess risk. Existing analysis is limited to the
Bayesian setting, which assumes a correct model and exact Bayesian posterior
distribution. Thus we cannot apply the existing theory to modern Bayesian
algorithms, such as variational inference. To address this, we present a novel
EU analysis in the frequentist setting, where data is generated from an unknown
distribution. We show a relation between the generalization ability and the
widely used EU measurements, such as the variance and entropy of the predictive
distribution. Then we show their convergence behaviors theoretically. Finally,
we propose new variational inference that directly controls the prediction and
EU evaluation performances based on the PAC-Bayesian theory. Numerical
experiments show that our algorithm significantly improves the EU evaluation
over the existing methods.