In classic papers, Zellner (1988, 2002) demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional.   Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective.   This generalized functional encompasses most modern inference procedures and suggests novel ones. Consider a data generating process φ ∼ p(φ) from which we have some N draws that constitute our training set, x P = {x 1 , x 2 , . . . , x N } ∼ p(x|φ). We can also imagine (potentially infinitely many) future draws from this same process x F = {x N +1 , . . . } ∼ p(x|φ). The predictive information I(x P ; x F ) 1 gives a unique measure of the complexity of a data generating process (Bialek et al., 2001 ). The goal of learning is to capture this complexity. To perform learning, we form a global representation of the dataset p(θ|x P ). This can be thought of as a learning algorithm, that, given a set of observations, produces a summary statistic of the dataset that we hope is useful for predicting future draws from the same process. This algorithm could be deterministic or more generally, stochastic. For example, imagine training a neural network on some data with stochastic gradient descent. Here the training data would be x P , the test data x F and the neural network parameters would be θ. Our training procedure implicitly samples from the distribution p(θ|x P ). How do we judge the utility of this learned global representation? The mutual information I(θ; x F ) quantifies the amount of information our representation captures about future draws. 2 To maximize learning we therefore aim to maximize this quantity. 1. We use I(x; y) for the mutual information between two random variables: 2. It is interesting to note that in the limit of an infinite number of future draws, I(θ; x F ) approaches I(θ; φ). Therefore, the amount of information we have about an infinite number of future draws from This is, of course, only interesting if we constrain how expressive our global representation is, for otherwise we could simply retain the full dataset. The amount of information retained about the observed data: I(θ; x P ) is a direct measure of our representation's complexity. The bits a learner extracts from data provides upper bounds on generalization (Bassily et al., 2017) . We have shown that a wide range of existing inference techniques are variational lower bounds on a single predictive information bottleneck objective. This connection highlights the drawbacks of these traditional forms of inference. In all cases considered in the previous section, we made two choices that loosened our variational bounds. First, we approximated p(x P |θ), with a factorized approximation q(x P |θ) = i q(x i |θ). Second, we approximated the future conditional marginal p(θ|x F ) = dx P p(θ|x P )p(x P |x F ) as an unconditional "prior". Neither of these approximations is necessary. For example, consider the following tighter "prior": q(θ|x F ) ∼ dx P p(θ|x P )q(x P |x F ). Here we reuse a tractable global representation p(θ|x P ) and instead create a variational approximation to the density of alternative datasets drawn from the same process: q(x P |x F ). We believe this information-theoretic, representation-first perspective on learning has the potential to motivate new and better forms of inference. 7
Rederive a wide class of inference procedures from an global information bottleneck objective.
{"src": "scitldr-AIC_train", "paper_id": "SkewFJnEtH"}