Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods. To analyze real-world data, machine learning relies on models that can extract useful patterns. Deep Neural Networks (DNNs) are a popular choice for this purpose because they can learn flexible representations. Another popular choice are probabilistic graphical models (PGMs) which can find interpretable structures in the data. Recent work on combining these two types of models hopes to exploit their complimentary strengths and provide powerful models that are also easy to interpret BID10 BID14 BID0 BID3 .To apply such hybrid models to real-world problems, we need efficient algorithms that can extract useful structure from the data. However , the two fields of deep learning and PGMs traditionally use different types of algorithms. For deep learning, stochastic-gradient methods are the most popular choice, e.g., those based on back-propagation. These algorithms are not only widely applicable, but can also employ amortized inference to enable fast inference at test time BID17 BID12 . On the other hand , most popular algorithms for PGMs exploit the model's graphical conjugacy structure to gain computational efficiency, e.g., variational message passing (VMP) BID18 , expectation propagation BID16 , Kalman filtering BID4 BID5 , and more recently natural-gradient variational inference BID9 and stochastic variational inference BID8 . In short, the two fields of deep learning and probabilistic modelling employ fundamentally different inferential strategies and a natural question is, whether we can design algorithms that combine their respective strengths.There have been several attempts to design such methods in the recent years, e.g., BID14 ; BID3 ; BID0 ; BID10 ; BID2 . Our work in this paper is inspired by the previous work of BID10 that aims to combine message-passing, natural-gradient, and amortized inference. Our proposed method in this paper simplifies and generalizes the method of BID10 .To do so, we propose Structured Inference Networks (SIN) that incorporate the PGM structure in the standard inference networks used in variational auto-encoders (VAE) BID12 BID17 . We derive conditions under which such inference networks can enable fast amortized inference similar to VAE. By using a recent VMP method of BID11 , we The generative models are just like the decoder in VAE but they employ a structured prior, e.g., Fig. (a) has a mixture-model prior while Fig. (b) has a dynamical system prior. SINs, just like the encoder in VAE, mimic the structure of the generative model by using parameters φ. One main difference is that in SIN the arrows between y n and x n are reversed compared to the model, while rest of the arrows have the same direction.derive a variational message-passing algorithm whose messages automatically reduce to stochasticgradients for the deep components of the model, while perform natural-gradient updates for the PGM part. Overall, our algorithm enables Structured, Amortized, and Natural-gradient (SAN) updates and therefore we call our algorithm the SAN algorithm. We show that our algorithm give comparable performance to the method of BID10 while simplifying and generalizing it. The code to reproduce our results is available at https://github.com/emtiyaz/vmp-for-svae/. We propose an algorithm to simplify and generalize the algorithm of BID10 for models that contain both deep networks and graphical models. Our proposed VMP algorithm enables structured, amortized, and natural-gradient updates given that the structured inference networks satisfy two conditions. The two conditions derived in this paper generally hold for PGMs that do not force dense correlations in the latent variables x. However, it is not clear how to extend our method to models where this is the case, e.g., Gaussian process models. It is possible to use ideas from sparse Gaussian process models and we will investigate this in the future. An additional issue is that our results are limited to small scale data. We found that it is non-trivial to implement a message-passing framework that goes well with the deep learning framework. We are going to pursue this direction in the future and investigate good platforms to integrate the capabilities of these two different flavors of algorithms.
We propose a variational message-passing algorithm for models that contain both the deep model and probabilistic graphical model.
{"src": "scitldr-AIC_train", "paper_id": "HyH9lbZAW"}