Variational Inference with Renyi Divergence
We introduce the variational R\'enyi bound (VR) that extends traditional variational inference to R\'enyi's alpha divergences. This new family of variational lower bounds unifies a number of existing variational methods, and enables a smooth interpolation from the evidence lower bound to the log (marginal) likelihood that is controlled by the value of alpha. The reparameterization trick, Monte Carlo estimation and stochastic optimisation methods are deployed to obtain a unified implementation for the VR bound optimisation. We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework. Experiments on variational auto encoders and Bayesian neural networks demonstrate the wide applicability of the VR bound.
ArXiv