Towards Bayesian Deep Learning: A Survey <s> Marginalized Collaborative Deep Learning <s> Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Marginalized Collaborative Deep Learning <s> Denoising auto-encoders (DAEs) have been successfully used to learn new representations for a wide range of machine learning tasks. During training, DAEs make many passes over the training dataset and reconstruct it from partial corruption generated from a pre-specified corrupting distribution. This process learns robust representation, though at the expense of requiring many training epochs, in which the data is explicitly corrupted. In this paper we present the marginalized Denoising Auto-encoder (mDAE), which (approximately) marginalizes out the corruption during training. Effectively, the mDAE takes into account infinitely many corrupted copies of the training data in every epoch, and therefore is able to match or outperform the DAE with much fewer training epochs. We analyze our proposed algorithm and show that it can be understood as a classic auto-encoder with a special form of regularization. In empirical evaluations we show that it attains 1-2 order-of-magnitude speedup in training time over other competing approaches. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Marginalized Collaborative Deep Learning <s> Collaborative filtering (CF) has been widely employed within recommender systems to solve many real-world problems. Learning effective latent factors plays the most important role in collaborative filtering. Traditional CF methods based upon matrix factorization techniques learn the latent factors from the user-item ratings and suffer from the cold start problem as well as the sparsity problem. Some improved CF methods enrich the priors on the latent factors by incorporating side information as regularization. However, the learned latent factors may not be very effective due to the sparse nature of the ratings and the side information. To tackle this problem, we learn effective latent representations via deep learning. Deep learning models have emerged as very appealing in learning effective representations in many applications. In particular, we propose a general deep architecture for CF by integrating matrix factorization with deep feature learning. We provide a natural instantiations of our architecture by combining probabilistic matrix factorization with marginalized denoising stacked auto-encoders. The combined framework leads to a parsimonious fit over the latent features as indicated by its improved performance in comparison to prior state-of-art models over four large datasets for the tasks of movie/book recommendation and response prediction. <s> BIB003
In SDAE, corrupted input goes through encoding and decoding to recover the clean input. Usually, different epochs of training use different corrupted versions as input. Hence generally, SDAE needs to go through enough epochs of training to see sufficient corrupted versions of the input. Marginalized SDAE (mSDAE) BIB001 seeks to avoid this by marginalizing out the corrupted input and obtaining closed-form solutions directly. In this sense, mSDAE is more computationally efficient than SDAE. As mentioned in BIB003 , using mSDAE instead of the Bayesian SDAE could lead to more efficient learning algorithms. For example, in BIB003 , the objective when using a one-layer mSDAE can be written as follows: where X 0,j * is the collection of k different corrupted versions of X 0,j * (a k-by-B matrix) and X c,j * is the k-time repeated version of X c,j * (also a k-by-B matrix). P 1 is the transformation matrix for item latent factors. The solution for W 1 would be: where A solver for the expectation in the equation above is provided in BIB001 . Note that this is a linear and one-layer case which can be generalized to the nonlinear and multi-layer case using the same techniques as in BIB002 , BIB001 . As we can see, in marginalized CDL, the perception variables Ω p = {X 0 , X c , W 1 }, the hinge variables Ω h = {V}, and the task variables Ω t = {P 1 , R, U}.