Towards Bayesian Deep Learning: A Survey <s> Gated Recurrent Neural Network <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB001 </s> Towards Bayesian Deep Learning: A Survey <s> Gated Recurrent Neural Network <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB002
The problem with the vanilla RNN introduced above is that the gradients propagated over many time steps are prone to vanish or explode, which makes the optimization notoriously difficult. In addition, the signal passing through the RNN decays exponentially, making it impossible to model long-term dependencies in long sequences. Imagine we want to predict the last word in the paragraph 'I have many books ... I like reading'. In order to get the answer, we need 'long-term memory' to retrieve information (the word 'books') at the start of the text. To address this problem, the long short-term memory model (LSTM) is designed as a type of gated RNN to model and accumulate information over a relatively long duration. The intuition behind LSTM is that when processing a sequence consisting of several subsequences, it is sometimes useful for the neural network to summarize or forget the old states before moving on to process the next subsequence  . Using t = 1 . . . T j to index the words in the sequence, the formulation of LSTM is as follows (we drop the item index j for notational simplicity): where x t is the word embedding of the t-th word, W w is a K W -by-S word embedding matrix, and e t is the 1-of-S representation, stands for the element-wise product operation between two vectors, σ(·) denotes the sigmoid function, s t is the cell state of the t-th word, and b, Y, and W denote the biases, input weights, and recurrent weights respectively. The forget gate units h f t and the input gate units h i t in Equation (5) can be computed using their corresponding weights and biases  Note that in the LSTM, information of the processed sequence is contained in the cell states s t and the output states h t , both of which are column vectors of length K W . Similar to BIB001 , BIB002 , we can use the output state and cell state at the last time step (h Tj and s Tj ) of the first LSTM as the initial output state and cell state of the second LSTM. This way the two LSTMs can be concatenated to form an encoder-decoder architecture, as shown in Figure 5 . Note that there is a vast literature on deep learning and neural networks. The introduction in this section intends to serve only as the background of Bayesian deep learning. Readers are referred to  for a comprehensive survey and more details.