Parametric Inverse Simulation
We introduce the theory of parametric inversion, which generalizes function inversion to non invertible functions. We define parametric inverses for several common functions, and present an algorithm to symbolically transform a composition of functions (i.e. a program) into a corresponding parametric inverse. We then outline a variational approach to sampling from a conditional distribution as a form of constrained parametric inversion. Recent advances in probabilistic programming [6, 5] and deep generative modeling [4, 7] have dramatically expanded the kinds of probabilistic models we can express and learn. Unfortunately, Bayesian inference procedures have not kept pace; there is a growing chasm between our ability to simulate complex probability distribution, and invert them for probabilistic inference. In such complex probabilistic models it is often difficult to infer latent variables that are consistent with the observed values, let alone drawn from the correct posterior distribution. For example, we can view vision as inference of probable scenes (geometry, lighting, material, etc) that render to an observed image, but forward simulation of a prior distribution will in practice generate scenes which are simply inconsistent with our observations. We propose an alternative approach: to simulate a random variable in reverse, starting with the observations. Inferred latent values are then consistent by construction. To define inverse simulation when a random variable is not invertible, we present parametric inversion, which generalizes function inversion to non injective functions. A non injective function f X Y is not invertible because it lacks a unique right inverse, a function f 1 Y X which for all y satisfies: f(f 1(y) y (1) To generalize inversion, a parametric inverse represents a set of right inverses parametrically, as a mapping from a parameter space to a function in a set. Parametric inversion of random variables leads to a novel formulation of Bayesian inference by inverse simulation. In this abstract, we: (i) introduce the concept of parametric inversion, (ii) present an algorithm to construct a parametric inverse from a composition of these primitives, and (iii) outline its extension to random variables for conditional sampling. 1 Example: Bayesian Inference as Constrained Inversion To demonstrate how Bayesian inference requires function inversion, consider the following model: x exponential(l 1) y logistic(m x, s 1) We can express this model as a pair of random variables (figure 1) i.e. transformations of o [0, 1] exponential(o;l) ln(1 o)/l x(o1) exponential(o1, 1) logistic(o;m, s) m+ s ln(o/(1 o) y(o1, o2) logistic(o2, x(o1) 1) To conditionally sample a value of x which is consistent with observation y c, let (o* 1 o 2) y 1(c) then evaluate x(o* 1) Hence, conditioning requires inverting y, but what if y is not invertible? How can we draw samples from x that are not only consistent, but from the posterior P (x y c) 30th Conference on Neural Information Processing Systems (NIPS 2016) Barcelona, Spain. 2 Parametric Inversion Parametric inversion generalizes function inversion to non invertible functions. A parametric inverse of a function f X Y is a parameterized function f 1 Y xTh X which maps an element y to a single element of its preimage: f(y; th) {x f(x) y} The parameter th determines which element of the preimage is returned. In contrast to a conventional inverse, a parametric inverse always exists. For example, while f(x) |x| is not invertible, we can define a parameter space th 1, 1} and parametric inverse f 1(y; th) th y, where th determines whether the positive or negative inverse element is returned. If a function takes multiple inputs, its parametric will return a tuple. For example, inverse multiplication (we denote x 1) maps a value z to two values whose product is z: x 1(z; th) (th, z/th) where th R 0. Parametric inverses are not unique; it is equally valid to define x 1(z; th) (z/th, th) Many functions map between a value and its preimage; we reserve the term parametric inverse for those which are sound and complete in this task. Definition 1. f 1 Y xTh X is a sound and complete parametric inverse of f X Y if y: {f 1(y; th) th Th} {x f(x) y} (2) Condition 2 asserts that parametric inverses are (i) sound: for any th, f 1(y; th) is an element of the preimage of y, and (ii) complete: there exists th corresponding to every element of the preimage. Inverse addition as 1(z; th1, th2) (th1, th2) is complete but not sound; for any z, there exists a th1 and th2 which sum to z, but there are many more pairs which do not. 1(z; th) (0, z) is sound but not complete; there exists no th such that for instance 1(0; th) 2, 2) despite the fact that 2 2 0. Hence, neither of these are valid parametric inverses of addition. A parametric inverse has a dual interpretation. If y is fixed, varying the parameter th varies which element of the preimage of y is returned. If th is fixed, f 1 is a function of only y and a right inverse which satisfies equation 1. These two alternatives are useful for different applications. Figure 1: Bayesian parameter estimation. Random variable y is a function of sample space. Intersection of y with plane is subset of O1xO2 which is consistent with y 0.2. Varying th in parametric inverse y 1(0.2; th) covers this set f Th f 1 x+ y R (th, z th) x y R (z th, th) x y R 0 (z/th, th) x/y R (z th, th) x R (th, logth(z) logx(y) R 0 (th, th) min(x, y) R (z, z th) max(x, y) R (z, z th) sin(x) Z asin(x) 2pth cos(x) Z acos(x) 2pth Figure 2: Primitive parametric inverses. Column f contains forward non injective functions. Th is the parameter space used for parametric inverses shown in column f 1. 3 Parametric Inversion of Composite Functions Arbitrarily complex functions can be found by composing functions from a small set of primitive functions. Given functions f2 X Y f1 Y Z, and composition f f1 f2, to construct a parametric inverse f 1 Z xTh X we substitute f1 and f2 with their corresponding parametric inverses f 1 1 and f 1 2 and reverse the order of composition: f 1(z; th1, th2) f 1 2 (f 1 1 (z, th1) th2) (3)