Nested Compiled Inference for Hierarchical Reinforcement Learning
Probabilistic programming languages (PPLs) allow the representation of probability distributions as computer programs by means of stochastic language primitives and conditioning of variable values on observations [8, 7] Probabilistic programs are generative models as they generate samples from a joint distribution p(x,y) where x are latent variables and y are output (or observed) data. Inference in PPLs amounts to computing the posterior p(x|y) conditioned on observed data. PPLs allow decoupling model specification from inference, which can be handled by techniques including Markov chain Monte Carlo [18] variational methods [19] expectation propagation [16] and sequential Monte Carlo [20] Inference is often computationally expensive. Amortized inference [5] strategies are devised to store and reuse past inferences so that future inferences run faster.