Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> A common task of recommender systems is to improve customer experience through personalized recommendations based on prior implicit feedback. These systems passively track different sorts of user behavior, such as purchase history, watching habits and browsing activity, in order to model user preferences. Unlike the much more extensively researched explicit feedback, we do not have any direct input from the users regarding their preferences. In particular, we lack substantial evidence on which products consumer dislike. In this work we identify unique properties of implicit feedback datasets. We propose treating the data as indication of positive and negative preference associated with vastly varying confidence levels. This leads to a factor model which is especially tailored for implicit feedback recommenders. We also suggest a scalable optimization procedure, which scales linearly with the data size. The algorithm is used successfully within a recommender system for television shows. It compares favorably with well tuned implementations of other known methods. In addition, we offer a novel way to give explanations to recommendations given by this factor model. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> Researchers have access to large online archives of scientific articles. As a consequence, finding relevant papers has become more difficult. Newly formed online communities of researchers sharing citations provides a new way to solve this problem. In this paper, we develop an algorithm to recommend scientific articles to users of an online community. Our approach combines the merits of traditional collaborative filtering and probabilistic topic modeling. It provides an interpretable latent structure for users and items, and can form recommendations about both existing and newly published articles. We study a large subset of data from CiteULike, a bibliography sharing service, and show that our algorithm provides a more effective recommender system than traditional collaborative filtering. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> Recently, tag recommendation (TR) has become a very hot research topic in data mining and related areas. However, neither co-occurrence based methods which only use the item-tag matrix nor content based methods which only use the item content information can achieve satisfactory performance in real TR applications. Hence, how to effectively combine the item-tag matrix, item content information, and other auxiliary information into the same recommendation framework is the key challenge for TR. In this paper, we first adapt the collaborative topic regression (CTR) model, which has been successfully applied for article recommendation, to combine both item-tag matrix and item content information for TR. Furthermore, by extending CTR we propose a novel hierarchical Bayesian model, called CTR with social regularization (CTR-SR), to seamlessly integrate the item-tag matrix, item content information, and social networks between items into the same principled model. Experiments on real data demonstrate the effectiveness of our proposed models. <s> BIB003 </s> Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying data-generating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty). <s> BIB004 </s> Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> Denoising auto-encoders (DAEs) have been successfully used to learn new representations for a wide range of machine learning tasks. During training, DAEs make many passes over the training dataset and reconstruct it from partial corruption generated from a pre-specified corrupting distribution. This process learns robust representation, though at the expense of requiring many training epochs, in which the data is explicitly corrupted. In this paper we present the marginalized Denoising Auto-encoder (mDAE), which (approximately) marginalizes out the corruption during training. Effectively, the mDAE takes into account infinitely many corrupted copies of the training data in every epoch, and therefore is able to match or outperform the DAE with much fewer training epochs. We analyze our proposed algorithm and show that it can be understood as a classic auto-encoder with a special form of regularization. In empirical evaluations we show that it attains 1-2 order-of-magnitude speedup in training time over other competing approaches. <s> BIB005 </s> Towards Bayesian Deep Learning: A Survey <s> Collaborative Deep Learning <s> Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address this sparsity problem, auxiliary information such as item content information may be utilized. Collaborative topic regression (CTR) is an appealing recent method taking this approach which tightly couples the two components that learn from two different sources of information. Nevertheless, the latent representation learned by CTR may not be very effective when the auxiliary information is very sparse. To address this problem, we generalize recently advances in deep learning from i.i.d. input to non-i.i.d. (CF-based) input and propose in this paper a hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix. Extensive experiments on three real-world datasets from different domains show that CDL can significantly advance the state of the art. <s> BIB006
To address the challenges above, a hierarchical Bayesian model called collaborative deep learning (CDL) as a novel tightly coupled method for RS is introduced in BIB006 . Based on a Bayesian formulation of SDAE, CDL tightly couples deep representation learning for the content information and collaborative filtering for the rating (feedback) matrix, allowing two-way interaction between the two. Experiments show that CDL significantly outperforms the state of the art. In the following text, we will start with the introduction of the notation used during our presentation of CDL. After that we will review the design and learning of CDL. Notation and Problem Formulation: Similar to the work in BIB002 , the recommendation task considered in CDL takes implicit feedback BIB001 as the training and test data. The entire collection of J items (articles or movies) is represented by a J-by-B matrix X c , where row j is the bag-of-words vector X c,j * for item j based on a vocabulary of size B. With I users, we define an I-by-J binary rating matrix R = [R ij ] I×J . For example, in the dataset citeulike-a BIB002 , BIB003 , BIB006 R ij = 1 if user i has article j in his or her personal library and R ij = 0 otherwise. Given part of the ratings in R and the content information X c , the problem is to predict the other ratings in R. Note that although CDL in its current from focuses on movie recommendation (where plots of movies are considered as content information) and article recommendation like BIB002 in this section, it is general enough to handle other recommendation tasks (e.g., tag recommendation). The matrix X c plays the role of clean input to the SDAE while the noise-corrupted matrix, also a J-by-B matrix, is denoted by X 0 . The output of layer l of the SDAE is denoted by X l which is a J-by-K l matrix. Similar to X c , row j of X l is denoted by X l,j * . W l and b l are the weight matrix and bias vector, respectively, of layer l, W l, * n denotes column n of W l , and L is the number of layers. For convenience, we use W + to denote the collection of all layers of weight matrices and biases. Note that an L/2-layer SDAE corresponds to an L-layer network. Generalized Bayesian SDAE: Following the introduction of SDAE in Section 2.2, if we assume that both the clean input X c and the corrupted input X 0 are observed, similar to BIB004 ,  , BIB005 ,  , we can define the following generative process of generalized Bayesian SDAE: 1) For each layer l of the SDAE network, a) For each column n of the weight matrix W l , draw Note that if λ s goes to infinity, the Gaussian distribution in Equation (8) will become a Dirac delta distribution  centered at σ(X l−1,j * W l + b l ), where σ(·) is the sigmoid function. The model will degenerate to be a Bayesian formulation of SDAE. That is why we call it generalized SDAE. Note that the first L/2 layers of the network act as an encoder and the last L/2 layers act as a decoder. Maximization of the posterior probability is equivalent to minimization of the reconstruction error with weight decay taken into consideration. Collaborative Deep Learning: Using the Bayesian SDAE as a component, the generative process of CDL is defined as follows: 1) For each layer l of the SDAE network, a) For each column n of the weight matrix W l , draw and then set the latent item vector: 3) Draw a latent user vector for each user i: 4) Draw a rating R ij for each user-item pair (i, j): 5. Note that while generation of the clean input Xc from X L is part of the generative process of the Bayesian SDAE, generation of the noise-corrupted input X 0 from Xc is an artificial noise injection process to help the SDAE learn a more robust feature representation. Here λ w , λ n , λ u , λ s , and λ v are hyperparameters and C ij is a confidence parameter similar to that for CTR BIB002 (C ij = a if R ij = 1 and C ij = b otherwise). Note that the middle layer X L/2 serves as a bridge between the ratings and content information. This middle layer, along with the latent offset j , is the key that enables CDL to simultaneously learn an effective feature representation and capture the similarity and (implicit) relationship between items (and users). Similar to the generalized SDAE, for computational efficiency, we can also take λ s to infinity. The graphical model of CDL when λ s approaches positive infinity is shown in Figure 8 , where, for notational simplicity, we use x 0 , x L/2 , and x L in place of X Note that according the definition in Section 4.1, here the perception variables Ω p = {{W l }, {b l }, {X l }, X c }, the hinge variables Ω h = {V}, and the task variables Ω t = {U, R}.