Towards Bayesian Deep Learning: A Survey <s> Learning: <s> This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, and several variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws of large numbers to transform the original graphical model into a simplified graphical model in which inference is efficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Learning: <s> A common task of recommender systems is to improve customer experience through personalized recommendations based on prior implicit feedback. These systems passively track different sorts of user behavior, such as purchase history, watching habits and browsing activity, in order to model user preferences. Unlike the much more extensively researched explicit feedback, we do not have any direct input from the users regarding their preferences. In particular, we lack substantial evidence on which products consumer dislike. In this work we identify unique properties of implicit feedback datasets. We propose treating the data as indication of positive and negative preference associated with vastly varying confidence levels. This leads to a factor model which is especially tailored for implicit feedback recommenders. We also suggest a scalable optimization procedure, which scales linearly with the data size. The algorithm is used successfully within a recommender system for television shows. It compares favorably with well tuned implementations of other known methods. In addition, we offer a novel way to give explanations to recommendations given by this factor model. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Learning: <s> Researchers have access to large online archives of scientific articles. As a consequence, finding relevant papers has become more difficult. Newly formed online communities of researchers sharing citations provides a new way to solve this problem. In this paper, we develop an algorithm to recommend scientific articles to users of an online community. Our approach combines the merits of traditional collaborative filtering and probabilistic topic modeling. It provides an interpretable latent structure for users and items, and can form recommendations about both existing and newly published articles. We study a large subset of data from CiteULike, a bibliography sharing service, and show that our algorithm provides a more effective recommender system than traditional collaborative filtering. <s> BIB003 </s> Towards Bayesian Deep Learning: A Survey <s> Learning: <s> Denoising auto-encoders (DAEs) have been successfully used to learn new representations for a wide range of machine learning tasks. During training, DAEs make many passes over the training dataset and reconstruct it from partial corruption generated from a pre-specified corrupting distribution. This process learns robust representation, though at the expense of requiring many training epochs, in which the data is explicitly corrupted. In this paper we present the marginalized Denoising Auto-encoder (mDAE), which (approximately) marginalizes out the corruption during training. Effectively, the mDAE takes into account infinitely many corrupted copies of the training data in every epoch, and therefore is able to match or outperform the DAE with much fewer training epochs. We analyze our proposed algorithm and show that it can be understood as a classic auto-encoder with a special form of regularization. In empirical evaluations we show that it attains 1-2 order-of-magnitude speedup in training time over other competing approaches. <s> BIB004 </s> Towards Bayesian Deep Learning: A Survey <s> Learning: <s> Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address this sparsity problem, auxiliary information such as item content information may be utilized. Collaborative topic regression (CTR) is an appealing recent method taking this approach which tightly couples the two components that learn from two different sources of information. Nevertheless, the latent representation learned by CTR may not be very effective when the auxiliary information is very sparse. To address this problem, we generalize recently advances in deep learning from i.i.d. input to non-i.i.d. (CF-based) input and propose in this paper a hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix. Extensive experiments on three real-world datasets from different domains show that CDL can significantly advance the state of the art. <s> BIB005
Based on the CDL model above, all parameters could be treated as random variables so that fully Bayesian methods such as Markov chain Monte Carlo (MCMC) or variational approximation methods BIB001 may be applied. However, such treatment typically incurs high computational cost. Consequently, CDL uses an EM-style algorithm for obtaining the MAP estimates, as in BIB003 . Like in CTR BIB003 , maximizing the posterior probability is equivalent to maximizing the joint log-likelihood of U, V, {X l }, X c , {W l }, {b l }, and R given λ u , λ v , λ w , λ s , and λ n : Fig. 9 . NN representation for degenerated CDL. If λ s goes to infinity, the likelihood becomes: where the encoder function f e (·, W + ) takes the corrupted content vector X 0,j * of item j as input and computes the encoding of the item, and the function f r (·, W + ) also takes X 0,j * as input, computes the encoding and then reconstructs the content vector of item j. For example, if the number of layers L = 6, f e (X 0,j * , W + ) is the output of the third layer while f r (X 0,j * , W + ) is the output of the sixth layer. From the perspective of optimization, the third term in the objective function BIB004 above is equivalent to a multi-layer perceptron using the latent item vectors v j as the target while the fourth term is equivalent to an SDAE minimizing the reconstruction error. Seeing from the view of neural networks (NN), when λ s approaches positive infinity, training of the probabilistic graphical model of CDL in Figure 8 (left) would degenerate to simultaneously training two neural networks overlaid together with a common input layer (the corrupted input) but different output layers, as shown in Figure 9 . Note that the second network is much more complex than typical neural networks due to the involvement of the rating matrix. When the ratio λ n /λ v approaches positive infinity, it will degenerate to a two-step model in which the latent representation learned using SDAE is put directly into the CTR. Another extreme happens when λ n /λ v goes to zero where the decoder of the SDAE essentially vanishes. On the right of Figure 8 is the graphical model of the degenerated CDL when λ n /λ v goes to zero. As demonstrated in the experiments, the predictive performance will suffer greatly for both extreme cases BIB005 . For u i and v j , block coordinate descent similar to BIB002 , BIB003 is used. Given the current W + , we compute the gradients of L with respect to u i and v j and then set them to zero, leading to the following update rules: T is a column vector containing all the ratings of user i, and C ij reflects the confidence controlled by a and b as discussed in BIB002 . C j and R j are defined similarly for item j. Given U and V, we can learn the weights W l and biases b l for each layer using the back-propagation learning algorithm. The gradients of the likelihood with respect to W l and b l are as follows: By alternating the update of U, V, W l , and b l , we can find a local optimum for L . Several commonly used techniques such as using a momentum term may be applied to alleviate the local optimum problem. Prediction: Let D be the observed test data. Similar to BIB003 , CDL uses the point estimates of u i , W + and j to calculate the predicted rating: where E[·] denotes the expectation operation. In other words, we approximate the predicted rating as: Note that for any new item j with no rating in the training data, its offset * j will be 0. In the following text, we provide several extensions of CDL from different perspectives.