A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> We propose multi-way, multilingual neural machine translation. The proposed approach enables a single neural translation model to translate between multiple languages, with a number of parameters that grows only linearly with the number of languages. This is made possible by having a single attention mechanism that is shared across all language pairs. We train the proposed multi-way, multilingual model on ten language pairs from WMT'15 simultaneously and observe clear performance improvements over models trained on only one language pair. In particular, we observe that the proposed model significantly improves the translation quality of low-resource language pairs. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> While neural machine translation (NMT) is making good progress in the past two years, tens of millions of bilingual sentence pairs are needed for its training. However, human labeling is very costly. To tackle this training data bottleneck, we develop a dual-learning mechanism, which can enable an NMT system to automatically learn from unlabeled data through a dual-learning game. This mechanism is inspired by the following observation: any machine translation task has a dual task, e.g., English-to-French translation (primal) versus French-to-English translation (dual); the primal and dual tasks can form a closed loop, and generate informative feedback signals to train the translation models, even if without the involvement of a human labeler. In the dual-learning mechanism, we use one agent to represent the model for the primal task and the other agent to represent the model for the dual task, then ask them to teach each other through a reinforcement learning process. Based on the feedback signals generated during this process (e.g., the language-model likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e.g., using the policy gradient methods). We call the corresponding approach to neural machine translation dual-NMT. Experiments show that dual-NMT works very well on English ↔ French translation; especially, by learning from monolingual data (with 10% bilingual data for warm start), it achieves a comparable accuracy to NMT trained from the full bilingual data for the French-to-English translation task. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Translating in real-time, a.k.a. simultaneous translation, outputs translation words before the input sentence ends, which is a challenging problem for conventional machine translation methods. We propose a neural machine translation (NMT) framework for simultaneous translation in which an agent learns to make decisions on when to translate from the interaction with a pre-trained NMT environment. To trade off quality and delay, we extensively explore various targets for delay and design a method for beam-search applicable in the simultaneous MT setting. Experiments against state-of-the-art baselines on two language pairs demonstrate the efficacy of the proposed framework both quantitatively and qualitatively. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> We explore six challenges for neural machine translation: domain mismatch, amount of training data, rare words, long sentences, word alignment, and beam search. We show both deficiencies and improvements over the quality of phrase-based statistical machine translation. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project. <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during training. We apply a slightly modified sequence-to-sequence with attention architecture that has previously been used for speech recognition and show that it can be repurposed for this more complex task, illustrating the power of attention-based models. A single model trained end-to-end obtains state-of-the-art performance on the Fisher Callhome Spanish-English speech translation task, outperforming a cascade of independently trained sequence-to-sequence speech recognition and machine translation models by 1.8 BLEU points on the Fisher test set. In addition, we find that making use of the training data in both languages by multi-task training sequence-to-sequence speech translation and recognition models with a shared encoder network can improve performance by a further 1.4 BLEU points. <s> BIB007 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> In translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we propose a cross-sentence context-aware approach and investigate the influence of historical contextual information on the performance of neural machine translation (NMT). First, this history is summarized in a hierarchical way. We then integrate the historical representation into NMT in two strategies: 1) a warm-start of encoder and decoder states, and 2) an auxiliary context source for updating decoder states. Experimental results on a large Chinese-English translation task show that our approach significantly improves upon a strong attention-based NMT system by up to +2.1 BLEU points. <s> BIB008 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures. <s> BIB009 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs. <s> BIB010 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals. <s> BIB011 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. ::: BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement). <s> BIB012 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT'16 English- German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big. <s> BIB013 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters. <s> BIB014 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail. <s> BIB015 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness. <s> BIB016 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area. <s> BIB017 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review the different neural architectures in which attention has been incorporated, and also show how attention improves interpretability of neural models. Finally, we discuss some applications in which modeling attention has a significant impact. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications. <s> BIB018 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 3) Encoder & Decoder Blocks <s> It has been shown that the performance of neural machine translation (NMT) drops starkly in low-resource conditions, underperforming phrase-based statistical machine translation (PBSMT) and requiring large amounts of auxiliary data to achieve competitive results. In this paper, we re-assess the validity of these results, arguing that they are the result of lack of system adaptation to low-resource settings. We discuss some pitfalls to be aware of when training low-resource NMT systems, and recent techniques that have shown to be especially helpful in low-resource settings, resulting in a set of best practices for low-resource NMT. In our experiments on German--English with different amounts of IWSLT14 training data, we show that, without the use of any auxiliary monolingual or multilingual data, an optimized NMT system can outperform PBSMT with far less data than previously claimed. We also apply these techniques to a low-resource Korean-English dataset, surpassing previously reported results by 4 BLEU. <s> BIB019
The encoder is built by 6 identical components, each of which contains one multi-head attention layer with a fully connected network above it. These two sub-layers are equipped with residual connection as well as layer normalization. All the sub-layers have the same dimension of 512 in output data. The decoder, however, is more complicated. There are also 6 components stacked; and in each component, three sublayers are connected, including two sub-layers of multi-head self attention and one sub-layer of fully-connected neural network. Specifically, the bottom attention layer is modified with method called masked to prevent positions from attending to subsequent positions, which is used for avoiding the model to look into the future of the target sequence when predicting the current word. Additionally, the second attention layer (the top attention layer) performs multi-head attention over the output of the encoder stack. 2) Transformer based NMT variants: Due to the tremendous performance improvement by Transformer, related refinement has got huge attention for researchers. The well accepted weakness of vanilla Transformer includes: lacking of recurrence modeling, theoretically not Turing-complete, capturing position information, as well as large model complexity. All these drawbacks have hindered its further improvement of translation performance. In response to these problems, some adjustments have been proposed for getting a better model. In respect of model architecture, some proposed modifications focused on both in depth of attention layer and network composition. Bapna et al. has proposed 2-3x deeper Transformer with a refined attention mechanism, which can be easier for the optimization of deeper models BIB009 . The refined attention mechanism extended its connection to each encoder layers, like a weighted residual connections along the encoder depth, which allows the model to flexibly adjust the gradient flow to different layers of encoder. Similarly, Wang et al. BIB013 proposed a more deeper Transformer model (25 layers of encoder), which continues the same line of Bapna et al.(2018)'s work with properly applying layer normalization and a novel output combination method. In contrast to the fixed layers of NMT model, Dehghani et al. proposed Universal Transformers, which cancelled to stack the constant number of layers by combining recurrent inductive bias of RNNs and Adaptive Computation Time halting mechanism, thus enhanced the original self-attention based representation for better learning iterative or recursive transformations. Notably, this adjustment has made the model be shown to be Turing-complete under certain assumptions [142] . As for refinement in network composition, inspired by the thinking of AutoML, So et al. applied neural architecture search (NAS) to find a comparable model with simplified architecture BIB014 . The Evolved Transformer proposed in BIB014 has an innovative combination of basic blocks achieves the same quality as the original Transformer-Big model with 37.6% less parameters. While most modification is focus on changing model structure directly, some new literature has chosen to utilize different input representation to improve model performance. One direct method is using enhanced Position Encoding for sequence order injection, where vanilla Transformer has weakness in capturing position information. Shaw et al. proposed modified self-attention mechanism with awareness of utilizing representations of relative positions, which demonstrated to have a significant improvements in two MT tasks BIB010 . Concurrently, using pre-initialized input representation with fine tune is another orientation, where some attempt have been proposed in different NLP tasks such as applying ELMo BIB011 for encoder of NMT model BIB015 . In terms of Transformer, one by-product of this innovative model is using self-attention for representing sequence, which can effectively fused word information with contextual information. In later, Two wellknown Transformer based input representation methods were been proposed named Bert (Bidirectional Encoder Representation from Transformers) BIB012 and GPT (Generative Pretrained Transformer)  , which has been indicated to bring improvement in some downstream NLP tasks. As for applying Transformer based pre-trained model in NMT task, more recently, this kind of trying has also been realized by using Bert as additional embedding layer or applying Bert as pretrained model directly BIB016 , which has been indicated to provide a bit better performance than vanilla Transformer after fine tune. Additionally, directly applying Bert as pre-trained model has been proved to have similar performance and thus can be more convenient for encoder initialization. The full structure of Transformer is illustrated in Fig. 14. VII. FUTURE TREND Although we have witnessed the fast-growing research progresses in NMT, there are still many challenges. Based on the extensive investigation BIB004 [125] BIB017 [116], we summarize the major challenges and list some potential directions in the following several aspects. (1) In terms of translation performance, NMT still doesn't perform well in translating long sentences. This is mainly because of two reasons: the practical limitation in engineering and the learning ability of the model itself. For the first reason, some academic experiments have chosen to ignore part of the long sentence that exceeds the RNN length. But we do believe that it's not the same thing when NMT has been deployed in industrial applications. For the second reason, as research progresses, the model architecture would be more complicated. For example, Transformer model has applied innovative structure in its design which brought significant improvement in translation quality and speed BIB005 . We believe that more refinements in model structure would be proposed. As we all know that RNN based NMT takes its advantages in modeling sequence order but results in computational inefficiency. More future work would consider the trade off between these two aspects. (2) Alignment mechanism is essential for both SMT and NMT models. For the vast majority of NMT models, Attention Mechanism plays the functional role in the alignment task, while it arguably does broader work than conventional alignment models. We believe this advanced alignment method would still get attraction in future research, since powerful attention method can improve the model performance directly. Later research in attention mechanism would try to relieve the weakness in NMT such as interpretation ability BIB018 . (3) Vocabulary coverage problem has always affected most of the NMT models. The research trend in handling computation load of softmax operation would pursue. And we have also found new training strategy which supports large vocabulary size. Besides, research of NMT operating in subword or character level has also aroused in recent years, which provided additional solution beyond traditional scope. More importantly, solving sophisticated translation scenario such as informal spelling is also a hot spot. Current NMT model integrated with character-level network has alleviated this phenomenon. Future work should focus on handling all kinds of OOV words in a more flexible way. (4) Low-Resource Neural Machine Translation BIB019 is another hot spot in current NMT, which tries to solve the severe performance reduction when NMT model is trained with rare bilingual corpus. Since the aforementioned scenario happened commonly in practice where some seldom-used languages don't have enough data, we do believe this filed would be extended in further research. Multilingual translation method BIB001  is the commonly proposed method which incorporated multi-language pair of data to improve NMT performance. It may need more interpretation about the different results in choosing different language pairs. Besides, unsupervised method has utilized the additional dataset and provided pre-trained model. Further research could improve its effect and provide hybrid training strategy with traditional method BIB002 [123] BIB006 . (5) Finally, research in NMT applications would also become more abundant. Currently, many applications have been developed such as speech translation [107] BIB007 and document level translation BIB008 . We believe that various applications (especially end-to-end tasks) would emerge in the future. We strongly hope that an AI based simultaneous translation system could be applied in large-scale, which can bring huge benefit to our human society BIB003 .