A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> Neural probabilistic language models (NPLMs) have been shown to be competitive with and occasionally superior to the widely-used n-gram language models. The main drawback of NPLMs is their extremely long training and testing times. Morin and Bengio have proposed a hierarchical language model built around a binary tree of words, which was two orders of magnitude faster than the non-hierarchical model it was based on. However, it performed considerably worse than its non-hierarchical counterpart in spite of using a word tree created using expert knowledge. We introduce a fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data. We then show that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English!German and English!Russian by up to 1.1 and 1.3 BLEU, respectively. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1 <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model. <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Description of Vocabulary Coverage problem in NMT <s> Neural Machine translation has shown promising results in recent years. In order to control the computational complexity, NMT has to employ a small vocabulary, and massive rare words outside the vocabulary are all replaced with a single unk symbol. Besides the inability to translate rare words, this kind of simple approach leads to much increased ambiguity of the sentences since meaningless unks break the structure of sentences, and thus hurts the translation and reordering of the in-vocabulary words. To tackle this problem, we propose a novel substitution-translation-restoration method. In substitution step, the rare words in a testing sentence are replaced with similar in-vocabulary words based on a similarity model learnt from monolingual data. In translation and restoration steps, the sentence will be translated with a model trained on new bilingual data with rare words replaced, and finally the translations of the replaced words will be substituted by that of original ones. Experiments on Chinese-to-English translation demonstrate that our proposed method can achieve more than 4 BLEU points over the attention-based NMT. When compared to the recently proposed method handling rare words in NMT, our method can also obtain an improvement by nearly 3 BLEU points. <s> BIB007
Based on the scenario as mentioned above, in the practical implementation of NMT, the initial way is choosing a small set of vocabulary and converting a large number of OOV words to one uniform "UNK" symbol (or other tags) as illustrated in Fig. 10 . This intuitive solution may hurt translation performance in the following two aspects. First, the existence of "UNK" symbol in translation may hurt the semantic completeness of sentence; ambiguity may emerge when "UNK" replace some crucial words BIB006 . Second, as the NMT model hard to learn information from OOV words, the prediction quality beyond the OOV words may also be affected BIB007 . Besides the unsurprising observation that NMT performed poorly on sentence with more OOV words than with more frequent words, some other phenomena in MT task are also hard to be handled like multi-word alignment, transliteration, and spelling, .etc. BIB002 BIB003 . They are seen as a similar phenomenon which is also caused by unknown words problem or suffers from rare training data BIB004 . Fig. 10 . An example of OOV words problem presented in BIB005 . en and f r denote the source sentence in English and the corresponding target sentence in French, nn denotes the neural network's result. For most of NMT model, choosing a modest size of vocabulary list is virtually a trade-off between the computation cost with translation quality. Also, it has been found the same thing in training NLM  BIB001 . Concretely, the computation cost mainly comes from the nature of the method in getting predicting word a normalization operation, which is used repeatedly in training of the DL model. Specifically, in NMT task, since DL model needs to adjust the parameters each time, the probability of current word thus would be calculated repeatedly to get the gradient, and since NMT model calculates the probability of current word when making a prediction, it needs to normalize the all of words in the vocabulary each time. Unfortunately, the normalization process is timeconsuming due to its time complexity is linear with the vocabulary size, and this attribute has rendered the same time complexity in the training process.