Towards Bayesian Deep Learning: A Survey <s> Discussion <s> We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Discussion <s> The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline. <s> BIB002 </s> Towards Bayesian Deep Learning: A Survey <s> Discussion <s> Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address this sparsity problem, auxiliary information such as item content information may be utilized. Collaborative topic regression (CTR) is an appealing recent method taking this approach which tightly couples the two components that learn from two different sources of information. Nevertheless, the latent representation learned by CTR may not be very effective when the auxiliary information is very sparse. To address this problem, we generalize recently advances in deep learning from i.i.d. input to non-i.i.d. (CF-based) input and propose in this paper a hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix. Extensive experiments on three real-world datasets from different domains show that CDL can significantly advance the state of the art. <s> BIB003
CDL is the first hierarchical Bayesian model to bridge the gap between state-of-the-art deep learning models and RS. By performing deep learning collaboratively, CDL and its variants can simultaneously extract an effective deep feature representation from content and capture the similarity and implicit relationship between items (and users). The learned representation may also be used for tasks other than recommendation. Unlike previous deep learning models which use a simple target like classification BIB002 and reconstruction BIB001 , CDL-based models use CF as a more complex target in a probabilistic framework. As mentioned in Section 1, information exchange between two components is crucial for the performance of BDL. In the CDL-based models above, the exchange is achieved by assuming Gaussian distributions that connect the hinge variables and the variables in the perception component (drawing the hinge variable is a perception variable), which is simple but effective and efficient in computation. Among the five CDL-based models in Table 1 , three of them are HV models and the others are LV models, according to the definition of Section 4.1. Since it has been verified that the HV CDL significantly outperforms its ZV counterpart BIB003 , we can expect extra performance boost from the LV counterparts of the three HV models. Besides efficient information exchange, the designs of the models also meet the i.i.d. requirement on the distribution concerning hinge variables discussed in 4.1 and are hence easily parallelizable. In some models to be introduced later, we will see alternative designs to enable efficient and i.i.d. information exchange between the two components of BDL. In BDL-based topic models, the perception component is responsible for inferring the topic hierarchy from documents while the task-specific component is in charge of modeling the word generation, topic generation, word-topic relation, or inter-document relation. The synergy between these two components comes from the bidirectional interaction between them. On one hand, knowledge on the topic hierarchy would facilitate accurate modeling of words and topics, providing valuable information for learning inter-document relation. On the other hand, accurately modeling the words, topics, and inter-document relation could help with the discovery of topic hierarchy and learning of compact document representations. As we can see, the information exchange mechanism in some BDL-based topic models is different from that in Section 4.2. For example, in the SBN-based DPFA model, the exchange is natural since the bottom layer of SBN, H (1) , and the relationship between H  and Ω h = {X} are both inherently probabilistic, as shown in Equation  and (21), which means additional assumptions on the distribution are not necessary. The SBN-based DPFA model is equivalent to assuming that H in PFA (see Equation (18)) is generated from a Dirac delta distribution (a Gaussian distribution with zero variance) centered at the bottom layer of the SBN, H  . Hence both DPFA models in Table 1 are ZV models, according to the definition in Section 4.1. It is worth noting that RSDAE is an HV model (see Equation (14), where S is the hinge variable and the others are perception variables), and naively modifying this model to be its ZV counterpart would violate the i.i.d. requirement in Section 4.1. As mentioned in Section 1, BDL-based control models consist of two components, a perception component to see the live video and a control (task-specific) component to infer the states of the dynamical system. Inference of the system is based on the mapped states and the confidence of mapping from the perception component, and in return, the control signals sent by the control component would affect the live video received by the perception component. Only when the two components work interactively within a unified probabilistic framework can the model reach its full potential and achieve the best control performance. In terms of information exchange between the two components, the BDL-based control model discussed above uses a different mechanism from Section 4.2 and Section 4.3: it uses neural networks to separately parameterize the mean and covariance of hinge variables (e.g., in the encoding model, the hinge variable z t ∼ N (µ t , diag(σ 2 t )), where µ t and σ t are perception variables parameterized as in Equation (25) and (26)), which is more flexible (with more free parameters) than models like CDL and CDR in Section 4.2, where Gaussian distributions with fixed variance are also used. Note that this BDL-based control model is an LV model as shown in Table 1 , and since the covariance is assumed to be diagonal, the model still meets the i.i.d. requirement in Section 4.1.