A Survey of Deep Learning Techniques for Neural Machine Translation <s> NMT with Deep Neural Network <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> NMT with Deep Neural Network <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> NMT with Deep Neural Network <s> Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> NMT with Deep Neural Network <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> NMT with Deep Neural Network <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB005
Since the traditional SMT model with NLM has got the state-of-the-art performance at that time, a pure DNN based translation approach was proposed later with an end-to-end design to model the entire MT process BIB001 BIB002 . Using DNN based NMT could capture subtle irregularities in both two languages more efficiently BIB004 , which is similar to the observation that DNNs often have a better performance than 'shallow' neural networks BIB003 . NMT with Attention Mechanism Although the initial DNN based NMT model has not outperformed the SMT completely, it still exhibited a huge potential for further research. When tracing back the major weakness, although one theoretical advantage of RNN is its ability in capturing the long dependency between words, in fact, the model performance would deteriorate with the increase of sentence length. This scenario is due to the limited feature representation ability in a fixed-length vector. Under the circumstances, since the original NMT has got a pretty good performance without any auxiliary, the idea of whether some variants in architecture could bring a breakthrough has led to the rise of Attention Mechanism. Attention Mechanism was originally proposed by Bahdanau et al. as an intermediate component BIB003 , and the objective is to provide additional word alignment information in translating the long sentence. Surprisingly, NMT model has got a considerable improvement with the help of this simple method. Later on, with tremendous popularity among both academia and industry, many refinements in Attention Mechanism have emerged, and more details will be discussed in Section IV. Fully Attention based NMT With the development of Attention Mechanism, fully Attention-based NMT has emerged as a great innovation in NMT history. In this new tendency, Attention mechanism has taken the dominate position in text feature extraction rather than a auxiliary component. And the representative model is Transformer BIB005 , which is a fully attention-based model proposed by Vaswani et al. Abandoning previous framework in neither RNN nor CNN based NMT models, Transformer is a that solely based on an intensified version of Attention Mechanism called Self-Attention with feed-forward connection, which got revolutionary progress in structure with state-of-the-art performance. Specifically, the innovative attention structure is the secret sauce to gain such significant improvement. The self-attention is a powerful feature extractor which also allows to 'read' the entire sentence and model it once a time. In the perspective of model architecture, this character can be seen as a combination of advantages from both CNN and RNN, which endows it a good feature representation ability with high inference speed. More details about self-attention will be given in Section IV. The architecture of Transformer will be discussed in Section VI.