A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. > <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM. <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1 <s> BIB007 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB008 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> We introduce a general and simple structural design called Multiplicative Integration (MI) to improve recurrent neural networks (RNNs). MI changes the way in which information from difference sources flows and is integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models. <s> BIB009 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results. <s> BIB010 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling complex functions and capturing complex linguistic structures. However NMT systems with deep architecture in their encoder or decoder RNNs often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often make the optimization much more difficult. To address this problem we propose novel linear associative units (LAU) to reduce the gradient propagation length inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs utilizes linear associative connections between input and output of the recurrent unit, which allows unimpeded information flow through both space and time direction. The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported results in the same setting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art. <s> BIB011 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Encoder-Decoder Structure <s> In this paper, we propose an additionsubtraction twin-gated recurrent network (ATR) to simplify neural machine translation. The recurrent units of ATR are heavily simplified to have the smallest number of weight matrices among units of all existing gated RNNs. With the simple addition and subtraction operation, we introduce a twin-gated mechanism to build input and forget gates which are highly correlated. Despite this simplification, the essential non-linearities and capability of modeling long-distance dependencies are preserved. Additionally, the proposed ATR is more transparent than LSTM/GRU due to the simplification. Forward self-attention can be easily established in ATR, which makes the proposed network interpretable. Experiments on WMT14 translation tasks demonstrate that ATR-based neural machine translation can yield competitive performance on English- German and English-French language pairs in terms of both translation quality and speed. Further experiments on NIST Chinese-English translation, natural language inference and Chinese word segmentation verify the generality and applicability of ATR on different natural language processing tasks. <s> BIB012
As is known, Encoder-Decoder is the most original and classic structure of NMT; it was directly inspired by NLM and proposed by Kalchbrenner & Blunsom BIB003 and Cho et al. BIB004 . Despite all kinds of refinements in details and small tips, it was wildly accepted by almost all modern NMT models. Based on the discussion above, since RNN based NMT has held the dominant position in NMT, and to avoid being overwhelmed in describing all kinds of small distinctions between models' structures, we specifically focus our discussion just on the vanilla RNN based NMT, thus can help to trace back the development process of NMT. The original structure of Encoder-Decoder structure is conceptually simple. It contains two connected networks (the encoder and the decoder) in its architecture, each for a different part of the translation process. When the encoder network receives one source sentence, it reads the source sentence word by word and compresses the variable-length sequence into a fixed-length vector in each hidden state. This process is called encoding. Then given the final hidden state of the encoder (referred to as thought vector), the decoder does the reverse work by transforming the thought vector to the target sentence word by word. Because Encoder-Decoder structure addresses the translation task from source data directly to the target result, which means there's no visible result in the middle process, this is also called end-to-end translation. The principle of Encoder-Decoder structure of NMT can be seen as mapping the source sentence with the target sentence via an intermediate vector in semantic space. This intermediate vector actually can represent the same semantic meaning in both two languages. For specific details of this structure, besides the model selection in the network, RNN based NMT models also differ in three main terms: (a) the directionality; (b) the type of activation function; and (c) the depth of RNN layer  . In the following, we give a detailed description. Depth: For the depth of RNN, as we discussed in Section II, single layer RNN usually performs poorly comparing with multi-layer RNN. In recent years, almost all the models with competitive performance are using a deep network, which has indicated a trend of using a deeper model to get the stateof-the-art result. For example, Bahdanau et al. BIB005 used four layers RNN in their model. However, simply increasing more layers of RNN may not always be useful. In the proof proposed by Britz et al. BIB010 , they found that using 4 layers RNN in the encoder for specific dataset would produce the best performance when there is no other auxiliary method in the whole model. Besides that, stacking RNN layers may make the network become too slow and difficult to train. One major challenges is the gradient exploding and vanishing problem BIB001 , which will cause the gradient be amplified or diminished when processing back propagation in deep layers. Besides the additional gate structure in refined RNN (like LSTM and GRU), other methods have also been applied to alleviate this phenomenon. For example, in Wu et al.'s work, the residual connections are provided between layer, which can improve the value of gradient flow in the backward pass, thus can speed up the convergence process BIB008 . Another possible problem is that a deeper model often indicates larger model capacity, which may perform worse on comparatively less training data due to the over-fitting. Directionality: In respect of directionality, a simple unidirectional RNN has been chosen by some researchers. For example, Luong et al. have directly used unidirectional RNN to accept the input sentence BIB007 . In comparison, bidirectional RNN is another common choice that can empower the translation quality. This is because the model performance is affected by whether it 'knows' well about the information in context word when predicting current word. A bidirectional RNN obviously could strengthen this ability. In practice, both Bahdanau et al. and Wu et al. used bidirectional RNN on the bottom layer as an alternative to capture the context information BIB005 BIB008 . In this structure, the first layer reads the sentence "left to right", and the second layer reads the sentence in a reverse direction. Then they are concatenated and fed to the next layer. This method generally has a better performance in experiment, although the explanation is intuitive: Based on the discussion of LM in Section II, the emergence probability of a specific word is determined by all the other words in both the prior and the post positions. When applying unidirectional RNN, word dependency between the first word and the last word is hard to be captured by the thought vector, since the model has experienced too many states in all time steps. On the country, bidirectional RNN provides an additional layer of information with reverse direction of reading words, which could naturally reduce this relative length within steps. The most visible drawback of this method is that it's hard to be paralleled, considering the time-consuming in its realization, both Bahdanau et al. and Wu et al. choose to apply just one layer bidirectional RNN in the bottom layer of the encoder, and other layers are all unidirectional layers BIB008 BIB005 . This choice makes a trade-off between the feature representation ability with model efficiency, due to it can still enable the model to be distributed on multi GPUs BIB008 . The basic concept of bidirectional RNN could find in Fig. 2 . Activation Function Selection: In respect of activation function selection, there are three common choices: vanilla RNN, Long Short Term Memory (LSTM) BIB002 , and Gated Recurrent Unit (GRU) BIB004 . Comparing with the vanilla RNN, both the last two have some robustness in addessing the gradient exploding and vanishing problem BIB009 BIB001 . Another sequence processing task has also indicated better performance achieved by GRU and LSTM BIB006 . Besides, some innovative neural units have been proposed. Wang et al. proposed linear associative units, which can alleviate the gradient diffusion phenomenon in non-linear recurrent activation BIB011 . More recently, Zhang et al. have created addition-subtraction twingated recurrent network (ATR). This type of unit reduces the inefficiency in NMT training and inference by simplifying the weight matrices among units BIB012 . All in all, in NMT task, LSTM is the most common choice.