A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Learning an algorithm from examples is a fundamental problem that has been widely studied. Recently it has been addressed using neural networks, in particular by Neural Turing Machines (NTMs). These are fully differentiable computers that use backpropagation to learn their own programming. Despite their appeal NTMs have a weakness that is caused by their sequential nature: they are not parallel and are are hard to train due to their large depth when unfolded. ::: We present a neural network architecture to address this problem: the Neural GPU. It is based on a type of convolutional gated recurrent unit and, like the NTM, is computationally universal. Unlike the NTM, the Neural GPU is highly parallel which makes it easier to train and efficient to run. ::: An essential property of algorithms is their ability to handle inputs of arbitrary size. We show that the Neural GPU can be trained on short instances of an algorithmic task and successfully generalize to long instances. We verified it on a number of tasks including long addition and long multiplication of numbers represented in binary. We train the Neural GPU on numbers with upto 20 bits and observe no errors whatsoever while testing it, even on much longer numbers. ::: To achieve these results we introduce a technique for training deep recurrent networks: parameter sharing relaxation. We also found a small amount of dropout and gradient noise to have a large positive effect on learning and generalization. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> The recently proposed neural network joint model (NNJM) (Devlin et al., 2014) augments the n-gram target language model with a heuristically chosen source context window, achieving state-of-the-art performance in SMT. In this paper, we give a more systematic treatment by summarizing the relevant source information through a convolutional architecture guided by the target information. With different guiding signals during decoding, our specifically designed convolution+gating architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of entire source sentence to form a unified representation. This representation, together with target language words, are fed to a deep neural network (DNN) to form a stronger NNJM. Experiments on two NIST Chinese-English translation tasks show that the proposed model can achieve significant improvements over the previous NNJM by up to +1.08 BLEU points on average <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neural machine translation: a variational encoderdecoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a continuous latent variable to explicitly model underlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound. Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines. <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Neural machine translation (NMT) aims at solving machine translation (MT) problems using neural networks and has exhibited promising results in recent years. However, most of the existing NMT models are shallow and there is still a performance gap between a single NMT model and the best conventional MT system. In this work, we introduce a new type of linear connections, named fast-forward connections, based on deep Long Short-Term Memory (LSTM) networks, and an interleaved bi-directional architecture for stacking the LSTM layers. Fast-forward connections play an essential role in propagating the gradients and building a deep topology of depth 16. On the WMT'14 English-to-French task, we achieve BLEU=37.7 with a single attention model, which outperforms the corresponding single shallow model by 6.2 BLEU points. This is the first time that a single NMT model achieves state-of-the-art performance and outperforms the best conventional model by 0.7 BLEU points. We can still achieve BLEU=36.3 even without using an attention mechanism. After special handling of unknown words and model ensembling, we obtain the best score reported to date on this task with BLEU=40.4. Our models are also validated on the more difficult WMT'14 English-to-German task. <s> BIB007 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice. <s> BIB008 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT'16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and we outperform several recently published results on the WMT'15 English-German task. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline. <s> BIB009 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> We present a novel neural network for processing sequences. The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. To address the differing lengths of the source and the target, we introduce an efficient mechanism by which the decoder is dynamically unfolded over the representation of the encoder. The ByteNet uses dilation in the convolutional layers to increase its receptive field. The resulting network has two core properties: it runs in time that is linear in the length of the sequences and it sidesteps the need for excessive memorization. The ByteNet decoder attains state-of-the-art performance on character-level language modelling and outperforms the previous best results obtained with recurrent networks. The ByteNet also achieves state-of-the-art performance on character-to-character machine translation on the English-to-German WMT translation task, surpassing comparable neural translation models that are based on recurrent networks with attentional pooling and run in quadratic time. We find that the latent alignment structure contained in the representations reflects the expected alignment between the tokens. <s> BIB010 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost. <s> BIB011 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new"super-separable"convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results. <s> BIB012 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU. <s> BIB013 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Model Design of DNN based NMT <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB014
There are many variations of network design for NMT, which can be categorized into recurrent or non − recurrent models. More specifically, this category can be traced back to the early development of NMT, when RNN and CNN based models are the most common design. Many sophisticated models proposed afterwards also belong to either CNN or RNN family. This sub-section follows the development of NMT in the early years, and demonstrates some representative models by classifying them as RNN or CNN based models. 1) RNN based NMT: Although in theory, any network with enough feature extraction ability could be selected to build an NMT model, in def acto implementations, RNN based NMT models have taken the dominant position in NMT development, and they have achieved state-of-the-art performance. Based on the discussion in Section II, since many NLM literature used RNN to model the sequence data, this design has intuitively motivated the further work to build an RNN based NMT model. In the initial experiment, an RNN based NLM was applied as a feature extractor to compress the source sentence into a feature vector, which is also referred to as thought vector. Then a similar RNN was applied to do the 'inverse work' to find the target sentence that can match the previous thought vector in semantic space. The first successful RNN based NMT was proposed by Sutskever et al., who used a pure deep RNN model and got a performance that approximates the best result achieved by SMT BIB002 . Further development proposed the Attention Mechanism, which improves the translation performance significantly and exceeds the best SMT model. GNMT model was an industry-level model applied in Google Translation, and it was regarded as a milestone in RNN based NMT. Besides the above mentioned work, other researchers have also proposed different architectures with excellent performance. Zhang et al. proposed Variational NMT method, which has an innovative perspective in modeling translation task, and the corresponding experiment has indicated a better performance than the baseline of original NMT in Chinese-English and English-German tasks BIB006 . Zhou et al. have designed Fast-Forward Connections for RNN (LSTM), which can allow a deeper network in implementation and thus gets a better performance BIB007 . Shazeer et al. incorporated Mixtureof-Expert (MoE) architecture into the GNMT model, which has outperformed the original GNMT model BIB011 . Concretely, MoE is one layer in the NMT model, which contains many sparsely combined experts (which are feed-forward neural networks in this experiment) and is connected with the RNN layer by a gate function. This method requires more parameters in total for the NMT model, but still maintains the efficiency in training speed. Since more parameters often imply a better representation ability, it demonstrates huge potential in the future. 2) CNN based NMT: Related work in trying other DNN models have also been proposed. Perhaps the most noted one is the Convolutional Neural Network (CNN) based NMT. In fact, CNN based models have also undergone many variations in its concrete architecture. But for a long while, most of these models can't have competitive performance with RNN based model, especially when the Attention Mechanism has emerged. In the development of CNN based NMT models, Kalchbrenner & Blunsom once tried a CNN encoder with RNN Decoder BIB001 , and it's maybe the earliest NMT architecture applied with CNN. Cho et al. tried a gated recursive CNN encoder with RNN decoder, but it has shown worse performance than RNN encoder BIB003 . A fully CNN based NMT was proposed by Kaiser & Bengio later BIB008 , which applied Extended Neural GPU BIB004 . The best performance in the early period of CNN based NMT was achieved by Gehring et al., which was a CNN encoder NMT and got the similar translation performance with RNN based model at that time BIB009 . Concurrently, Kalchbrenner et al. also proposed ByteNet (a kind of CNN) based NMT, which achieved the state-ofthe-art performance on character-level translation but failed at word-level translation BIB010 . In addition, Meng et al. and Tu et al. proposed a CNN based model separately, which provides additional alignment information for SMT BIB005  . Compared with RNN based NMT, CNN based models have its advantage in training speed; this is due to the intrinsic structure of CNN which allows parallel computations for its different filters when handling the input data. And also, the model structure has made CNN based models easier to resolve the gradient vanishing problem. However, there are two fatal drawbacks that affect their translation quality. First, since the original CNN based model can only capture the word depen-dencies within the width of its filters, the long dependency of words can only be found in high-level convolution layers; this unnatural character often causes a worse performance than the RNN based model. Second, since the original NMT model compresses a sentence into a fixed size of the vector, a large performance reduction would happen when the sentence becomes too long. This comes from the limited representation ability in fixed size of the vector. Similar phenomenon can also be found in early proposed RNN based models, which are later alleviated by Attention Mechanism. Some advanced CNN based NMT models have also been proposed with corresponding solutions in addressing the above drawbacks. Kaiser BIB012 . BIB013 followed their previous work by proposing a CNN based NMT that is cooperated with Attention Mechanism. It even got a better result than RNN based model BIB013 , but this achievement was soon outperformed by Transformer BIB014 .