Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis: <s> Variational methods have been previously explored as a tractable approximation to Bayesian inference for neural networks. However the approaches proposed so far have only been applicable to a few simple network architectures. This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks. Along the way it revisits several common regularisers from a variational perspective. It also provides a simple pruning heuristic that can both drastically reduce the number of network weights and lead to improved generalisation. Experimental results are provided for a hierarchical multidimensional recurrent neural network applied to the TIMIT speech corpus. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis: <s> In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a "sampling threshold" and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis: <s> Hamiltonian Monte Carlo (HMC) sampling methods provide a mechanism for defining distant proposals with high acceptance probabilities in a Metropolis-Hastings framework, enabling more efficient exploration of the state space than standard random-walk proposals. The popularity of such methods has grown significantly in recent years. However, a limitation of HMC methods is the required gradient computation for simulation of the Hamiltonian dynamical system-such computation is infeasible in problems involving a large sample size or streaming data. Instead, we must rely on a noisy gradient estimate computed from a subset of the data. In this paper, we explore the properties of such a stochastic gradient HMC approach. Surprisingly, the natural implementation of the stochastic approximation can be arbitrarily bad. To address this problem we introduce a variant that uses second-order Langevin dynamics with a friction term that counteracts the effects of the noisy gradient, maintaining the desired target distribution as the invariant distribution. Results on simulated data validate our theory. We also provide an application of our methods to a classification task using neural networks and to online Bayesian matrix factorization. <s> BIB003 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis: <s> A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies. The proposed multi-layer model employs a deep sigmoid belief network or restricted Boltzmann machine, the bottom binary layer of which selects topics for use in a Poisson factor analysis model. Under this setting, topics live on the bottom layer of the model, while the deep specification serves as a flexible prior for revealing topic structure. Scalable inference algorithms are derived by applying Bayesian conditional density filtering algorithm, in addition to extending recently proposed work on stochastic gradient thermostats. Experimental results on several corpora show that the proposed approach readily handles very large collections of text documents, infers structured topic representations, and obtains superior test perplexities when compared with related models. <s> BIB004 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis: <s> Deep directed generative models are developed. The multi-layered model is designed by stacking sigmoid belief networks, with sparsity-encouraging priors placed on the model parameters. Learning and inference of layer-wise model parameters are implemented in a Bayesian setting. By exploring the idea of data augmentation and introducing auxiliary P olya-Gamma variables, simple and ecient Gibbs sampling and meaneld variational Bayes (VB) inference are implemented. To address large-scale datasets, an online version of VB is also developed. Experimental results are presented for three publicly available datasets: MNIST, Caltech 101 Silhouettes and OCR letters. <s> BIB005
In the deep PFA model BIB004 , the generative process can be summarized as follows: x pnk ∼ Pois(φ pk θ kn h kn ) (21) where L is the number of layers in SBN, which corresponds to Equation BIB001 and  . x pnk is the count of word p that comes from topic k in document n. In this model, the perception variables Ω p = {{H (l) }, {W l }, {b l }}, the hinge variables Ω h = {X}, and the task variables Ω t = {{φ k }, {r k }, Θ, γ 0 }. W l is the weight matrix containing columns of w in Equation  . Learning Using Bayesian Conditional Density Filtering: Efficient learning algorithms are needed for Bayesian treatments of deep PFA. BIB004 proposed to use an online version of MCMC called Bayesian conditional density filtering (BCDF) to learn both the global parameters Ψ g = ({φ k }, {r k }, γ 0 , {W l }, {b l }) and the local variables Ψ l = (Θ, {H (l) }). The key conditional densities used for the Gibbs updates are as follows: , x pnk , and ζ pnk ∝ φ pk θ kn . For the learning of h (l) kn where l > 1, the same techniques as in BIB005 can be used. Learning Using Stochastic Gradient Thermostats: An alternative way of learning deep PFA is through the use of stochastic gradient Nóse-Hoover thermostats (SGNHT), which is more accurate and scalable. SGNHT is a generalization of the stochastic gradient Langevin dynamics (SGLD) BIB002 and the stochastic gradient Hamiltonian Monte Carlo (SGHMC) BIB003 . Compared with the previous two, it introduces momentum variables into the system, helping the system to jump out of local optima. Specifically, the following stochastic differential equations (SDE) can be used: where f (Ψ g ) = −∇ Ψg U (Ψ g ) and U (Ψ g ) is the negative log-posterior of the model. t indexes time and W denotes the standard Wiener process. ξ is the thermostats variable to make sure the system has a constant temperature. D is the injected variance which is a constant. To speed up convergence, the SDE is generalized to: where I is the identity matrix, , and M is the dimensionality of the parameters.