A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Statistical language modeling has been successfully used for speech recognition, part-of-speech tagging, and syntactic parsing. Recently, it has also been applied to information retrieval. According to this new paradigm, each document is viewed as a language sample, and a query as a generation process. The retrieved documents are ranked based on the probabilities of producing a query from the corresponding language models of these documents. In this paper, we will present a new language model for information retrieval, which is based on a range of data smoothing techniques, including the Good-Turning estimate, curve-fitting functions, and model combinations. Our model is conceptually simple and intuitive, and can be easily extended to incorporate probabilities of phrases such as word pairs and word triples. The experiments with the Wall Street Journal and TREC4 data sets showed that the performance of our model is comparable to that of INQUERY and better than that of another language model for information retrieval. In particular, word pairs are shown to be useful in improving the retrieval performance. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Some models of textual corpora employ text generation methods involving n-gram statistics, while others use latent topic variables inferred using the "bag-of-words" assumption, in which word order is ignored. Previously, these methods have not been combined. In this work, I explore a hierarchical generative probabilistic model that incorporates both n-gram statistics and latent topic variables by extending a unigram topic model to include properties of a hierarchical Dirichlet bigram language model. The model hyperparameters are inferred using a Gibbs EM algorithm. On two data sets, each of 150 documents, the new model exhibits better predictive accuracy than either a hierarchical Dirichlet bigram language model or a unigram topic model. Additionally, the inferred topics are less dominated by function words than are topics discovered using unigram statistics, potentially making them more meaningful. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> This paper describes the use of a neural network language model for large vocabulary continuous speech recognition. The underlying idea of this approach is to attack the data sparseness problem by performing the language model probability estimation in a continuous space. Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words. It is also shown that this approach can be incorporated into a large vocabulary continuous speech recognizer using a lattice rescoring framework at a very low additional processing time. The neural network language model was thoroughly evaluated in a state-of-the-art large vocabulary continuous speech recognizer for several international benchmark tasks, in particular the Nist evaluations on broadcast news and conversational speech recognition. The new approach is compared to four-gram back-off language models trained with modified Kneser-Ney smoothing which has often been reported to be the best known smoothing method. Usually the neural network language model is interpolated with the back-off language model. In that way, consistent word error rate reductions for all considered tasks and languages were achieved, ranging from 0.4% to almost 1% absolute. <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> While phrase-based statistical machine translation systems currently deliver state-of-the-art performance, they remain weak on word order changes. Current phrase reordering models can properly handle swaps between adjacent phrases, but they typically lack the ability to perform the kind of long-distance re-orderings possible with syntax-based systems. In this paper, we present a novel hierarchical phrase reordering model aimed at improving non-local reorderings, which seamlessly integrates with a standard phrase-based system with little loss of computational efficiency. We show that this model can successfully handle the key examples often used to motivate syntax-based systems, such as the rotation of a prepositional phrase around a noun phrase. We contrast our model with reordering models commonly used in phrase-based systems, and show that our approach provides statistically significant BLEU point gains for two language pairs: Chinese-English (+0.53 on MT05 and +0.71 on MT08) and Arabic-English (+0.55 on MT05). <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition <s> BIB007 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. <s> BIB008 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> This paper presents a new approach to perform the estimation of the translation model probabilities of a phrase-based statistical machine translation system. We use neural networks to directly learn the translation probability of phrase pairs using continuous representations. The system can be easily trained on the same data used to build standard phrase-based systems. We provide experimental evidence that the approach seems to be able to infer meaningful translation probabilities for phrase pairs not seen in the training data, or even predict a list of the most likely translations given a source phrase. The approach can be used to rescore n-best lists, but we also discuss an integration into the Moses decoder. A preliminary evaluation on the English/French IWSLT task achieved improvements in the BLEU score and a human analysis showed that the new model often chooses semantically better translations. Several extensions of this work are discussed. <s> BIB009 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We present a fast and scalable online method for tuning statistical machine translation models with large feature sets. The standard tuning algorithm—MERT—only scales to tens of features. Recent discriminative algorithms that accommodate sparse features have produced smaller than expected translation quality gains in large systems. Our method, which is based on stochastic gradient descent with an adaptive learning rate, scales to millions of features and tuning sets with tens of thousands of sentences, while still converging after only a few epochs. Large-scale experiments on Arabic-English and Chinese-English show that our method produces significant translation quality gains by exploiting sparse features. Equally important is our analysis, which suggests techniques for mitigating overfitting and domain mismatch, and applies to other recent discriminative methods for machine translation. <s> BIB010 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We explore the application of neural language models to machine translation. We develop a new model that combines the neural probabilistic language model of Bengio et al., rectified linear units, and noise-contrastive estimation, and we incorporate it into a machine translation system both by reranking k-best lists and by direct integration into the decoder. Our large-scale, large-vocabulary experiments across four language pairs show that our neural language model improves translation quality by up to 1.1 Bleu. <s> BIB011 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established using an attention mechanism: the decoder emits each symbol based on a context created with a subset of input symbols elected by the attention mechanism. We report initial results demonstrating that this new approach achieves phoneme error rates that are comparable to the state-of-the-art HMM-based decoders, on the TIMIT dataset. <s> BIB012 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Recent work has shown success in using neural network language models (NNLMs) as features in MT systems. Here, we present a novel formulation for a neural network joint model (NNJM), which augments the NNLM with a source context window. Our model is purely lexicalized and can be integrated into any MT decoder. We also present several variations of the NNJM which provide significant additive improvements. <s> BIB013 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM. <s> BIB014 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Neural network language models are often trained by optimizing likelihood, but we would prefer to optimize for a task specific metric, such as BLEU in machine translation. We show how a recurrent neural network language model can be optimized towards an expected BLEU loss instead of the usual cross-entropy criterion. Furthermore, we tackle the issue of directly integrating a recurrent network into firstpass decoding under an efficient approximation. Our best results improve a phrasebased statistical machine translation system trained on WMT 2012 French-English data by up to 2.0 BLEU, and the expected BLEU objective improves over a crossentropy trained model by up to 0.6 BLEU in a single reference setup. <s> BIB015 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1 <s> BIB016 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB017 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results. <s> BIB018 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> 1) Motivation of NMT: <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB019
The inspiration for neural machine translation comes from two aspects: the success of Deep Learning in other NLP tasks as we mentioned, and the unresolved problems in the development of MT itself. For the first reason, in many NLP tasks, traditional Machine Learning method is highly dependent on hand-crafted features that often come from linguistic intuition, which is definitely an empirical trial-and-error process BIB008  and is often far more incomplete in representing the nature of original data. For example, the context size of training language model is assigned by researchers with strong assumption in context relation BIB001 ; and in text representation method, the classic bag-of-words (BOW) method has ignored the influence of word order BIB003 . However, when applying deep neural network (DNN) in the aforementioned tasks, the DNN requires minimum domain-knowledge and avoids some pre-processing steps in human feature engineering BIB018 . DNN is a powerful neural network which has achieved excellent performance in many complex learning tasks which are traditionally considered difficult BIB019  . In NLP field, DNN has been applied in some traditional tasks, for example, speech recognition BIB012 and Named Entity Recognition (NER) BIB008 . With the exceptional performance they got, DNN-based models have found many potential applications in other NLP tasks. For the second reason, in MT field, PBSMT has got a pretty good performance in the past decades, but there are still some inherent weaknesses which require further improvement. First, since the PBSMT generates the translation by segmenting the source sentence into several phrases and doing phrase replacement, it may ignore the long dependency beyond the length of phrases and thus cause inconsistency in translation results such as incorrect gender agreements. Second, there are generally many intricate sub-components in current systems BIB006 [14] BIB010 , e.g., language model, reordering model, length/unknown penalties, etc. With the increasing number of these sub-components, it is hard to fine-tune and combine each other to get a more stable result BIB016 . All the above discussions have indicated the bottleneck in the development of SMT miniature. Specifically, this bottleneck mainly comes from the language model (LM). This is because, in MT task, language model actually can give the most important information: the emergence probability of a particular word (or phrase) that is conditioned on previous words. So building a better LM can definitely improve the translation performance. The vast majority of conventional LM is based on the Markov assumption: where x 1 , x 2 , ..., x T is a sequence of words in a sentence and T represents the length of the sentence. In this assumption, the probability of the sentence is equal to the multiplication of probability of each word. n is the total number of words that is chosen to simplify the model, which is also referred to as context window. Obviously, the dependency of words that exceed n would be ignored, which implies that the conventional LM performs poorly on modeling long dependency. Moreover, since the experimental result has indicated that a modest context size (generally 4-6 words) can be accepted, the first problem of traditional LM is the limited representation ability. Besides, the data sparsity for training has always been the problem that hinders an LM built with a larger size of context window. This is because the number of n-tuples for counting is exponential in n. In other words, when building an LM, with the increment of the number of order, the number of training samples we need would also increase remarkably, which is also referred to as "curse of dimensionality". For example, if one LM has the order of 5 with a vocabulary size of 10,000, then the possible combination of words for statistics should be about 10 25 , which requires enormous training data. And since most of these combinations have not been observed before, subsequent researches have used various trade-off and smoothing method to alleviate the sparsity problem BIB004 [128]  [130]  . While further research of the aforementioned LM with statistical method has become almost stagnant, Neural Language Model (NLM) BIB002 , on the other hand, uses a neural network to build a language model that models text data directly. In initial stage, NLM used fixed-length of a feature vector to represent each word, and then the solid number of word vectors would concatenate together as a semantic metric to represent the context BIB002 [38] BIB005 , which is very similar to the context window. This work was enhanced later by injecting additional context information from source sentence BIB013 [132] BIB009 . Comparing with the traditional LM, the original NLM alleviates the sample sparsity due to the distributed representation of the word, which enables them to share the statistical weights rather than being independent variables. And since words with similar meaning may occur in the similar context, the corresponding feature vector would have the similar value, which indicates that the semantic relation of words has been "embedded" into the feature vector. New proposals in the next stage solve the long dependency problem by using Recurrent Neural Network (RNN). RNN based NLM (RNLM) models the whole sentence by reading each word once a time-step, thus it can model the true conditional probability without limitation of context size BIB007 . Before the emergence of NMT, the RNLM, as mentioned earlier, outperformed the conventional LM in the evaluation of text perplexity and brought better performance in many practical tasks BIB007 BIB014 . The direct application of NLM in SMT has been naturally proposed BIB013 [36] BIB015 [40] BIB011 , and the preliminary experiment indicated promising results. The potential of NLM motivates further exploration for a complete DNN based translation model. Subsequently, a more "pure" model with the only neural network has emerged, with the DNN architecture that learns to do the translation task end-to-end. Section III demonstrates its basic structure (in Fig. 4 ), as well as its concrete details. 2) Formulation of NMT Task: Currently, NMT task is originally designed as an end-to-end learning task. It directly processes a source sequence to a target sequence. The learning objective is to find the correct target sequence given the source sequence, which can be seen as a high dimensional classification problem that tries to map the two sentences in the semantic space. In all mainstreams of modern NMT model, this process can be divided into two steps: encoding and decoding, and thus can functionally separate the whole model as Encoder and Decoder as illustrated in Fig. 2 . Fig. 2 . End-to-End structure in modern NMT model. The encoder is used to represent the source sentence to semantic vector, while the decoder makes prediction from this semantic vector to a target sentence. End-to-End means the model processes source data to target data directly, without explicable intermediate result. In perspective of probability, NMT generates the target sequence T (t 1 , t 2 , ..., t m ) from the max conditional probability given the source sequence S(s 1 , s 2 , ..., s n ), where n is the length of sequence S and m is the length of target sequence T . The whole task could be formulated as BIB017 : argmax P (T |S). ( More concretely, when generating each word of the target sentence, it uses the information from both the word it predicted previously and the source sentence. In that case, each generating step could be described as when generating the i-th word: Based on this formula and the discussion of NLM above, NMT task could be regarded as an NLM model with additional constraints (e.g., conditioned on a given source sequence).