Towards Bayesian Deep Learning: A Survey <s> Relational Stacked Denoising Autoencoders as Topic Models <s> In many applications, the data, such as web pages and research papers, contain relation (link) structure among entities in addition to textual content information. Matrix factorization (MF) methods, such as latent semantic indexing (LSI), have been successfully used to map either content information or relation information into a lower-dimensional latent space for subsequent processing. However, how to simultaneously model both the relation information and the content information effectively with an MF framework is still an open research problem. In this paper, we propose a novel MF method called relation regularized matrix factorization (RRMF) for relational data analysis. By using relation information to regularize the content MF procedure, RRMF seamlessly integrates both the relation information and the content information into a principled framework. We propose a linear-time learning algorithm with convergence guarantee to learn the parameters of RRMF. Extensive experiments on real data sets show that RRMF can achieve state-of-the-art performance. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Relational Stacked Denoising Autoencoders as Topic Models <s> Tag recommendation has become one of the most important ways of organizing and indexing online resources like articles, movies, and music. Since tagging information is usually very sparse, effective learning of the content representation for these resources is crucial to accurate tag recommendation. Recently, models proposed for tag recommendation, such as collaborative topic regression and its variants, have demonstrated promising accuracy. However, a limitation of these models is that, by using topic models like latent Dirichlet allocation as the key component, the learned representation may not be compact and effective enough. Moreover, since relational data exist as an auxiliary data source in many applications, it is desirable to incorporate such data into tag recommendation models. In this paper, we start with a deep learning model called stacked denoising autoencoder (SDAE) in an attempt to learn more effective content representation. We propose a probabilistic formulation for SDAE and then extend it to a relational SDAE (RSDAE) model. RSDAE jointly performs deep representation learning and relational learning in a principled way under a probabilistic framework. Experiments conducted on three real datasets show that both learning more effective representation and learning from relational data are beneficial steps to take to advance the state of the art. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Relational Stacked Denoising Autoencoders as Topic Models <s> A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies. The proposed multi-layer model employs a deep sigmoid belief network or restricted Boltzmann machine, the bottom binary layer of which selects topics for use in a Poisson factor analysis model. Under this setting, topics live on the bottom layer of the model, while the deep specification serves as a flexible prior for revealing topic structure. Scalable inference algorithms are derived by applying Bayesian conditional density filtering algorithm, in addition to extending recently proposed work on stochastic gradient thermostats. Experimental results on several corpora show that the proposed approach readily handles very large collections of text documents, infers structured topic representations, and obtains superior test perplexities when compared with related models. <s> BIB003
Problem Statement and Notation: Assume we have a set of items (articles or movies) X c , with X T c,j * ∈ R B denoting the content (attributes) of item j. Besides, we use I K to denote a K-dimensional identity matrix and S = [s 1 , s 2 , · · · , s J ] to denote the relational latent matrix with s j representing the relational properties of item j. From the perspective of SDAE, the J-by-B matrix X c represents the clean input to the SDAE and the noise-corrupted matrix of the same size is denoted by X 0 . Besides, we denote the output of layer l of the SDAE, a J-by-K l matrix, by X l . Row j of X l is denoted by X l,j * , W l and b l are the weight matrix and bias vector of layer l, W l, * n denotes column n of W l , and L is the number of layers. As a shorthand, we refer to the collection of weight matrices and biases in all layers as W + . Note that an L/2-layer SDAE corresponds to an L-layer network. Model Formulation: Here we will use the Bayesian SDAE introduced before as a building block for the relational stacked denoising autoencoder (RSDAE) model. As mentioned in BIB002 , RSDAE is formulated as a novel probabilistic model which can seamlessly integrate layered representation learning and the relational information available. This way, the model can simultaneously learn the feature representation from the content information and the relation between items. The graphical model of RSDAE is shown in Figure 11 and the generative process is listed as follows: 1) Draw the relational latent matrix S from a matrix variate normal distribution  : 2) For layer l of the SDAE where l = 1, 2, . . . , of the SDAE network, draw the representation vector for item j from the product of two Gaussians (PoG)  : 4) For layer l of the SDAE network where is the dimensionality of the learned representation vector for each item, S denotes the K × J relational latent matrix in which column j is the relational latent vector s j for item j. Note that N K,J (0, (13) is a matrix variate normal distribution defined as in  : where the operator ⊗ denotes the Kronecker product of two matrices  , tr(·) denotes the trace of a matrix, and L a is the Laplacian matrix incorporating the relational information. L a = D − A, where D is a diagonal matrix whose diagonal elements D ii = j A ij and A is the adjacency matrix representing the relational information with binary entries indicating the links (or relations) between items. A jj = 1 indicates that there is a link between item j and item j and , which is also a Gaussian  . According to the generative process above, maximizing the posterior probability is equivalent to maximizing the joint log-likelihood of {X l }, X c , S, {W l }, and {b l } given λ s , λ w , λ l , λ r , and λ n : Similar to the generalized SDAE, taking λ s to infinity, the joint log-likelihood becomes: where X l,j * = σ(X l−1,j * W l + b l ). Note that the first term − λ l 2 tr(SL a S T ) corresponds to log p(S) in the matrix variate distribution in Equation BIB003 . Besides, by simple manipulation, we have tr( where S k * denotes the k-th row of S. As we can see, maximizing − λ l 2 tr(S T L a S) is equivalent to making s j closer to s j if item j and item j are linked (namely A jj = 1). In RSDAE, the perception variables Ω p = {{X l }, X c , {W l }, {b l }}, the hinge variables Ω h = {S}, and the task variables Ω t = {A}. Learning Relational Representation and Topics: BIB002 provides an EM-style algorithm for MAP estimation. Here we review some of the key steps as follows. In terms of the relational latent matrix S, we first fix all rows of S except the k-th one S k * and then update S k * . Specifically, we take the gradient of L with respect to S k * , set it to 0, and get the following linear system: A naive approach is to solve the linear system by setting . Unfortunately, the complexity is O(J 3 ) for one single update. Similar to BIB001 , the steepest descent method  is used to iteratively update S k * : As discussed in BIB001 , the use of steepest descent method dramatically reduces the computation cost in each iteration from O(J 3 ) to O(J). Given S, we can learn W l and b l for each layer using the back-propagation algorithm. By alternating the update of S, W l , and b l , a local optimum for L can be found. Also, techniques such as including a momentum term may help to avoid being trapped in a local optimum.