A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> We present a statistical phrase-based translation model that uses hierarchical phrases---phrases that contain subphrases. The model is formally a synchronous context-free grammar but is learned from a bitext without any syntactic information. Thus it can be seen as a shift to the formal machinery of syntax-based translation systems without any linguistic commitment. In our experiments using BLEU as a metric, the hierarchical phrase-based model achieves a relative improvement of 7.5% over Pharaoh, a state-of-the-art phrase-based system. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> Many machine learning tasks can be expressed as the transformation---or \emph{transduction}---of input sequences into output sequences: speech recognition, machine translation, protein secondary structure prediction and text-to-speech to name but a few. One of the key challenges in sequence transduction is learning to represent both the input and output sequences in a way that is invariant to sequential distortions such as shrinking, stretching and translating. Recurrent neural networks (RNNs) are a powerful sequence learning architecture that has proven capable of learning such representations. However RNNs traditionally require a pre-defined alignment between the input and output sequences to perform transduction. This is a severe limitation since \emph{finding} the alignment is the most difficult aspect of many sequence transduction problems. Indeed, even determining the length of the output sequence is often challenging. This paper introduces an end-to-end, probabilistic sequence transduction system, based entirely on RNNs, that is in principle able to transform any input sequence into any finite, discrete output sequence. Experimental results for phoneme recognition are provided on the TIMIT speech corpus. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> Attention mechanism has enhanced stateof-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT. 1 <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> D. Inference method <s> Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results. <s> BIB006
After the training period, the model could be used for translation, which is called inference. The inference procedure is quite similar to the training process. Nevertheless, there is still a clear distinction between training and inference: at decoding time, we only have access to the source sentence, i.e., encoder hidden state. There is more than one way to perform decoding. Proposed decoding strategies include Sampling and Greedy search, while the latter one is generally accepted and be evolved as Beam-search. 1) General decoding work flow (greedy): The idea of greedy strategy is simple, as we illustrate in Fig. 4 . The Greedy strategy is only considering the predicted word with the highest probability. In the implementation of our illustration, the previously generated word would also be fed to the network together with the thought vector in the next time-step. The detailed steps are as follows: 1. The model still encodes the source sentence in the same way as during the training period to obtain the thought vector, and this thought vector is used to initialize the decoder. 2. The decoding (translation) process will start as soon as the decoder receives the end-of-sentence marker < EOS > of source sentence. 3. For each time-step on the decoder side, we treat the RNN's output as a set of logits. We choose the word with the highest translation probability as the emitted word, whose ID is associated with the maximum logit value. For example, in Fig. 4 , the word "moi" has the highest probability in the first decoding step. We then feed this word as an input in the next time-step. The probability is thus conditioned on the previous prediction (this is why we call it "greedy" behavior). 4. The process will continue until the ending symbol < EOS > is generated as an output symbol. 2) Beam-search: While the Greedy search method has produced a pretty good result, Beam-search is a more elaborated one with better results. Although it is not a necessary component for NMT, Beam-search has been chosen by most of NMT models to get the best performance BIB006 . The beam-search method was proposed by other sequence learning task with successful application BIB002  . It's also the conventional technique of MT task that has been used for years in finding the most appropriate translation result  [32] BIB001 . Beam-search can be simply described as retaining the top-k possible translations as candidates at each time, where k is called the beam-width. In the next time-step, each candidate word would be combined with a new word to form new possible translation. The new candidate translation would then compete with each other in log probability to get the new top-k most reasonable results. The whole process continues until the end of translation. Concretely, the beam search can be formulated in the following steps: Besides the standard Beam-search which finds the candidate translation only by sorting log probability, this evaluation function mathematically tends to find shorter sentence. This is because a negative log-probability would be added at each decoding step, which lowers the scores with the increasing length of sentences BIB003 . An efficient variant for alleviating this scenario is to add a length normalization  . A refined length normalization was also proposed by Wu et al. BIB004 . Another kind of refined method in Beam-search is adding coverage penalty, which helps to encourage the decoder to cover the words in the source sentence as much as possible when generating an output sentence BIB004 BIB005 . In addition, since this method finds k times of translation(rather than one) until getting the final result, it generally makes the decoding process more time-consuming. In practice, an intuitive solution is to limit the beam-width as a small constant, which is a trade-off between the decoding efficiency and the translation accuracy. As reported by a comparison work, an experimental beam width for best performance is 5 to 10 BIB006 .