Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Sigmoid Belief Networks <s> Abstract Connectionist learning procedures are presented for “sigmoid” and “noisy-OR” varieties of probabilistic belief networks. These networks have previously been seen primarily as a means of representing knowledge derived from experts. Here it is shown that the “Gibbs sampling” simulation procedure for such networks can support maximum-likelihood learning from empirical data through local gradient ascent. This learning procedure resembles that used for “Boltzmann machines”, and like it, allows the use of “hidden” variables to model correlations between visible variables. Due to the directed nature of the connections in a belief network, however, the “negative phase” of Boltzmann machine learning is unnecessary. Experimental results show that, as a result, learning in a sigmoid belief network can be faster than in a Boltzmann machine. These networks have other advantages over Boltzmann machines in pattern classification and decision making applications, are naturally applicable to unsupervised learning problems, and provide a link between work on connectionist learning and work on the representation of expert knowledge. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Sigmoid Belief Networks <s> A beta-negative binomial (BNB) process is proposed, leading to a beta-gamma-Poisson process, which may be viewed as a"multi-scoop"generalization of the beta-Bernoulli process. The BNB process is augmented into a beta-gamma-gamma-Poisson hierarchical structure, and applied as a nonparametric Bayesian prior for an infinite Poisson factor analysis model. A finite approximation for the beta process Levy random measure is constructed for convenient implementation. Efficient MCMC computations are performed with data augmentation and marginalization techniques. Encouraging results are shown on document count matrix factorization. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Sigmoid Belief Networks <s> A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies. The proposed multi-layer model employs a deep sigmoid belief network or restricted Boltzmann machine, the bottom binary layer of which selects topics for use in a Poisson factor analysis model. Under this setting, topics live on the bottom layer of the model, while the deep specification serves as a flexible prior for revealing topic structure. Scalable inference algorithms are derived by applying Bayesian conditional density filtering algorithm, in addition to extending recently proposed work on stochastic gradient thermostats. Experimental results on several corpora show that the proposed approach readily handles very large collections of text documents, infers structured topic representations, and obtains superior test perplexities when compared with related models. <s> BIB003 </s> Towards Bayesian Deep Learning: A Survey <s> Deep Poisson Factor Analysis with Sigmoid Belief Networks <s> The seemingly disjoint problems of count and mixture modeling are united under the negative binomial (NB) process. A gamma process is employed to model the rate measure of a Poisson process, whose normalization provides a random probability measure for mixture modeling and whose marginalization leads to an NB process for count modeling. A draw from the NB process consists of a Poisson distributed finite number of distinct atoms, each of which is associated with a logarithmic distributed number of data samples. We reveal relationships between various count- and mixture-modeling distributions and construct a Poisson-logarithmic bivariate distribution that connects the NB and Chinese restaurant table distributions. Fundamental properties of the models are developed, and we derive efficient Bayesian inference. It is shown that with augmentation and normalization, the NB process and gamma-NB process can be reduced to the Dirichlet process and hierarchical Dirichlet process, respectively. These relationships highlight theoretical, structural and computational advantages of the NB process. A variety of NB processes, including the beta-geometric, beta-NB, marked-beta-NB, marked-gamma-NB and zero-inflated-NB processes, with distinct sharing mechanisms, are also constructed. These models are applied to topic modeling, with connections made to existing algorithms under Poisson factor analysis. Example results show the importance of inferring both the NB dispersion and probability parameters. <s> BIB004
The Poisson distribution with support over nonnegative integers is known as a natural choice to model counts. It is, therefore, desirable to use it as a building block for topic models [6] . With this motivation, BIB002 proposed a model, dubbed Poisson factor analysis (PFA), for latent nonnegative matrix factorization via Poisson distributions. Poisson Factor Analysis: PFA assumes a discrete P -by-N matrix X containing word counts of N documents with a vocabulary size of P BIB003 , BIB002 . In a nutshell, PFA can be described using the following equation: where Φ (of size P -by-K where K is the number of topics) denotes the factor loading matrix in factor analysis with the k-th column φ k encoding the importance of each word in topic k. The K-by-N matrix Θ is the factor score matrix with the n-th column θ n containing topic proportions for document n. The K-by-N matrix H is a latent binary matrix with the n-th column h n defining a set of topics associated with document n. Different priors correspond to different models. For example, Dirichlet priors on φ k and θ n with an all-one matrix H would recover LDA [6] while a beta-Bernoulli prior on h n leads to the NB-FTM model in BIB004 . In BIB003 , a deep-structured prior based on sigmoid belief networks (SBN) BIB001 (an MLP variant with binary hidden units) is imposed on h n to form a deep PFA model for topic modeling.