Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Restricted Boltzmann Machine <s> Variational methods have been previously explored as a tractable approximation to Bayesian inference for neural networks. However the approaches proposed so far have only been applicable to a few simple network architectures. This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks. Along the way it revisits several common regularisers from a variational perspective. It also provides a simple pruning heuristic that can both drastically reduce the number of network weights and lead to improved generalisation. Experimental results are provided for a hierarchical multidimensional recurrent neural network applied to the TIMIT speech corpus. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Restricted Boltzmann Machine <s> A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies. The proposed multi-layer model employs a deep sigmoid belief network or restricted Boltzmann machine, the bottom binary layer of which selects topics for use in a Poisson factor analysis model. Under this setting, topics live on the bottom layer of the model, while the deep specification serves as a flexible prior for revealing topic structure. Scalable inference algorithms are derived by applying Bayesian conditional density filtering algorithm, in addition to extending recently proposed work on stochastic gradient thermostats. Experimental results on several corpora show that the proposed approach readily handles very large collections of text documents, infers structured topic representations, and obtains superior test perplexities when compared with related models. <s> BIB002
Similar to the deep PFA above, the restricted Boltzmann machine (RBM)  can be used in place of SBN BIB002 . If RBM is used, Equation BIB001 and (20) would be defined using the energy  : For the learning, similar algorithms as the deep PFA with SBN can be used. Specifically, the sampling process would alternate between {{φ k }, {γ k }, γ 0 } and {{W (l) }, {c (l) }}. For the former, similar conditional density as the SBN-based DPFA is used. For the latter, they use the contrastive divergence algorithm.