A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> The device of this invention uses electrostatic recording to apply original titles, legends, or other indicia to discrete record receivers. The system includes changeable stencils which, for the purposes of this description, are constructed as counters. The desired information is set up, a voltage is applied between the stencil and an electrode behind a record receiver, ionized pigmented particles are attracted through the stencils to the record receiver, and the record receiver is moved to a fixing station. By rendering the individual items of information changeable, any desired information within the range of the apparatus can be applied to an appropriate record receiver. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural probabilistic language models (NPLMs) have been shown to be competitive with and occasionally superior to the widely-used n-gram language models. The main drawback of NPLMs is their extremely long training and testing times. Morin and Bengio have proposed a hierarchical language model built around a binary tree of words, which was two orders of magnitude faster than the non-hierarchical model it was based on. However, it performed considerably worse than its non-hierarchical counterpart in spite of using a word tree created using expert knowledge. We introduce a fast hierarchical language model along with a simple feature-based algorithm for automatic construction of word trees from the data. We then show that the resulting models can outperform non-hierarchical neural models as well as the best n-gram models. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Previous work on statistical language modeling has shown that it is possible to train a feedforward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models based on n-grams. However, training the neural network model with the maximum-likelihood criterion requires computations proportional to the number of words in the vocabulary. In this paper, we introduce adaptive importance sampling as a way to accelerate training of the model. The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network. We show that a very significant speedup can be obtained on standard problems. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. ::: ::: We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. ::: ::: We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> This paper describes challenges and solutions for building a successful voice search system as applied to Japanese and Korean at Google. We describe the techniques used to deal with an infinite vocabulary, how modeling completely in the written domain for language model and dictionary can avoid some system complexity, and how we built dictionaries, language and acoustic models in this framework. We show how to deal with the difficulty of scoring results for multiple script languages because of ambiguities. The development of voice search for these languages led to a significant simplification of the original process to build a system for any new language which in in parts became our default process for internationalization of voice search. <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> We explore the application of neural language models to machine translation. We develop a new model that combines the neural probabilistic language model of Bengio et al., rectified linear units, and noise-contrastive estimation, and we incorporate it into a machine translation system both by reranking k-best lists and by direct integration into the decoder. Our large-scale, large-vocabulary experiments across four language pairs show that our neural language model improves translation quality by up to 1.1 Bleu. <s> BIB006 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method based on importance sampling that allows us to use a very large target vocabulary without increasing training complexity. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English!German translation and almost as high performance as state-of-the-art English!French translation system. <s> BIB007 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in conventional NMT systems is their inability to correctly translate very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and implement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sentence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our experiments on the WMT’14 English to French translation task show that this method provides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT system is the first to surpass the best result achieved on a WMT’14 contest task. <s> BIB008 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1 <s> BIB009 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form‐function relationship in language, our “composed” word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish). <s> BIB010 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character ngram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English!German and English!Russian by up to 1.1 and 1.3 BLEU, respectively. <s> BIB011 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information. <s> BIB012 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> We introduce a neural machine translation model that views the input and output sentences as sequences of characters rather than words. Since word-level information provides a crucial source of bias, our input model composes representations of character sequences into representations of words (as determined by whitespace boundaries), and then these are translated using a joint attention/translation model. In the target language, the translation is modeled as a sequence of word vectors, but each word is generated one character at a time, conditional on the previous character generations in each word. As the representation and generation of words is performed at the character level, our model is capable of interpreting and generating unseen word forms. A secondary benefit of this approach is that it alleviates much of the challenges associated with preprocessing/tokenization of the source and target languages. We show that our model can achieve translation results that are on par with conventional word-based models. <s> BIB013 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> In order to capture rich language phenomena, neural machine translation models have to use a large vocabulary size, which requires high computing time and large memory usage. In this paper, we alleviate this issue by introducing a sentence-level or batch-level vocabulary, which is only a very small sub-set of the full output vocabulary. For each sentence or batch, we only predict the target words in its sentencelevel or batch-level vocabulary. Thus, we reduce both the computing time and the memory usage. Our method simply takes into account the translation options of each word or phrase in the source sentence, and picks a very small target vocabulary for each sentence based on a wordto-word translation model or a bilingual phrase library learned from a traditional machine translation model. Experimental results on the large-scale English-toFrench task show that our method achieves better translation performance by 1 BLEU point over the large vocabulary neural machine translation system of Jean et al. (2015). <s> BIB014 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model. <s> BIB015 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks. <s> BIB016 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural machine translation (NMT) often makes mistakes in translating low-frequency content words that are essential to understanding the meaning of the sentence. We propose a method to alleviate this problem by augmenting NMT systems with discrete translation lexicons that efficiently encode translations of these low-frequency words. We describe a method to calculate the lexicon probability of the next word in the translation candidate by using the attention vector of the NMT model to select which source word lexical probabilities the model should focus on. We test two methods to combine this probability with the standard NMT probability: (1) using it as a bias, and (2) linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3 BLEU and 0.13-0.44 NIST score, and faster convergence time. <s> BIB017 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB018 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional and highway layers to replace the standard lookup-based word representations. The resulting unlimited-vocabulary and affix-aware source word embeddings are tested in a state-of-the-art neural MT based on an attention-based bidirectional recurrent neural network. The proposed MT scheme provides improved results even when the source language is not morphologically rich. Improvements up to 3 BLEU points are obtained in the German-English WMT task. <s> BIB019 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Nearly all previous work on neural machine translation (NMT) has used quite restricted vocabularies, perhaps with a subsequent method to patch in unknown words. This paper presents a novel word-character solution to achieving open vocabulary NMT. We build hybrid systems that translate mostly at the word level and consult the character components for rare words. Our character-level recurrent neural networks compute source word representations and recover unknown target words when needed. The twofold advantage of such a hybrid approach is that it is much faster and easier to train than character-based ones; at the same time, it never produces unknown words as in the case of word-based models. On the WMT'15 English to Czech translation task, this hybrid approach offers an addition boost of +2.1-11.4 BLEU points over models that already handle unknown words. Our best system achieves a new state-of-the-art result with 20.7 BLEU score. We demonstrate that our character models can successfully learn to not only generate well-formed words for Czech, a highly-inflected language with a very complex vocabulary, but also build correct representations for English source words. <s> BIB020 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation. In this paper, we ask a fundamental question: can neural machine translation generate a character sequence without any explicit segmentation? To answer this question, we evaluate an attention-based encoder-decoder with a subword-level encoder and a character-level decoder on four language pairs--En-Cs, En-De, En-Ru and En-Fi-- using the parallel corpora from WMT'15. Our experiments show that the models with a character-level decoder outperform the ones with a subword-level decoder on all of the four language pairs. Furthermore, the ensembles of neural models with a character-level decoder outperform the state-of-the-art non-neural machine translation systems on En-Cs, En-De and En-Fi and perform comparably on En-Ru. <s> BIB021 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Homographs, words with different meanings but the same surface form, have long caused difficulty for machine translation systems, as it is difficult to select the correct translation based on the context. However, with the advent of neural machine translation (NMT) systems, which can theoretically take into account global sentential context, one may hypothesize that this problem has been alleviated. In this paper, we first provide empirical evidence that existing NMT systems in fact still have significant problems in properly translating ambiguous words. We then proceed to describe methods, inspired by the word sense disambiguation literature, that model the context of the input word with context-aware word embeddings that help to differentiate the word sense be- fore feeding it into the encoder. Experiments on three language pairs demonstrate that such models improve the performance of NMT systems both in terms of BLEU score and in the accuracy of translating homographs. <s> BIB022 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Different Solutions <s> Most existing machine translation systems operate at the level of words, relying on explicit segmentation to extract tokens. We introduce a neural machine translation (NMT) model that maps a source character sequence to a target character sequence without any segmentation. We employ a character-level convolutional network with max-pooling at the encoder to reduce the length of source representation, allowing the model to be trained at a speed comparable to subword-level models while capturing local regularities. Our character-to-character model outperforms a recently proposed baseline with a subword-level encoder on WMT'15 DE-EN and CS-EN, and gives comparable performance on FI-EN and RU-EN. We then demonstrate that it is possible to share a single character-level encoder across multiple languages by training a model on a many-to-one translation task. In this multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilingual character-level translation even surpasses the models specifically trained on that language pair alone, both in terms of BLEU score and human judgment. <s> BIB023
Related researches have proposed various methods in both the training and inference process. Further, these methods can be roughly divided into three categories based on their different orientations. The first one is intuitively focused on finding solutions in improving computation speedup, which could support a more extensive vocabulary. The second one focus on using context information, this kind of method can address some of the unknown words(such as Proper Noun) by copying them to translation result as well as low-frequency words which cause a poor translation quality. The last one, which is more advanced, prefers to utilize information inside the word such as characters, because of their flexibility in handling morphological variants of words, this method can support translating OOV words in a more "intelligent" way. 1) methods by computation speedup: For computation speedup method, there are lots of literature that implement their idea in NLM training. The first thought in trying computation speedup is to scale the sof tmax operation. Since an effective sof tmax calculation could obviously support a larger vocabulary, this kind of trying has got a lot of attention in NLM literature. Morin & Bengio [53] proposed hierarchical models to get an exponential speedup in the computation of normalization factor, thus help to accelerate the gradient calculation of word probabilities. In concrete details, the original model has transformed vocabulary into a binary tree structure, which was built with pre-knowledge from WordNet  . The initial experiment result shows that this hierarchical method is comparable with traditional trigram LM but fails to exceed original NLM; this is partly because of utilizing handcrafted feature from WordNet in the tree building process. As a binary tree can provide a significant improvement in cost-effective between the speed with performance, further work still focuses on this trend to find better refinement. Later on, Mnih & Hinton followed this work by removing the requirement of expert knowledge in tree building process BIB002 . A more elegant method is to retain the original model but change the method in calculating the normalization factor. Bengio & Senecal proposed importance sampling method to approximate the normalization factor BIB001 . However, this method is not stable unless with a careful control BIB003 . Mnih & Teh used noise-contrastive estimation to learn the normalization factor directly, which can be more stable in the training process of NLM BIB004 . Later, Vaswani et al. proposed a similar method with application in MT BIB006 . The above methods are difficult to be implemented parallelly by GPUs. Further consideration found solutions that are more GPU friendly. Jean et al. alleviated the computation time by utilizing a subset of vocabulary as candidate words list in the training process while used the whole vocabulary in the inference process. Based on the inspiration of using importance sampling in earlier work BIB003 , they proposed a pure data segmentation method in the training process. Specifically, they pre-processed the training data sequentially, choosing a subset of vocabulary for each training example with the number of its distinct words reached threshold t(which is still far less than the size of original vocabulary). In the inference process, they still abandon using the whole vocabulary and proposing a hybrid candidate list alternatively. They composed candidate words list from two parts. The first part is some specific candidate target words that translated from a pre-defined dictionary with the others are the K-most frequent words. In the practical performance analysis, this method remains the similar modest size of candidate words in the training process; thus, it can maintain the computational efficiency while supporting an extremely larger size of candidate words BIB007 . Similarly, Mi et al. proposed vocabulary manipulation method which provides a separate vocabulary for different sentences or batches, it contains candidate words from both word-to-word dictionary and phrase-to-phrase library BIB014 . Besides all kinds of corresponding drawbacks in the above method, the common weakness of all these methods is they still suffer from the OOV words despite a larger vocabulary size they can support. This is because the enlarged vocabulary is still size limited, and there's no solution for complementary when encountering unknown words, whereas the following category of methods can partly handle it. In addition, simply increasing the vocabulary size can merely bring little improvement due to the Zipfs Law, which means there is always a large tail of OOV words need to be addressed BIB015 . 2) methods by using context information: Besides the above variants which focus on computation speed-up, a more advanced category is using context information. Luong et al. proposed a word alignment algorithm which collaborates with Copy Mechanism to post-processing the translation result. This old but useful operation was inspired by the common word(phrase) replacement method in SMT and has achieved a pretty considerable improvement in BLEU BIB008 . Concretely, in Luong's method, for each of the OOV words, there's a "pointer" which map to the corresponding word in the source sentence. In the post-processing stage, a predefined dictionary was provided with "pointer" to find the corresponding translation, while using directly copy mechanism to handle the OOV words that not in the dictionary. The Besides that, they synthesized Copy Mechanism with general translation operation by adding a so-called switching network to decide which operation should be applied in each timestep, this could be thought to improve the generalization of the whole model. BIB015 . Gu et al. made parallel efforts in integrating different mechanisms, they proposed a kind of Attention Mechanism called CopyNet with the vanilla encoder-decoder model, which can be naturally extended to handle OOV words in NMT task BIB016 . Additionally, they found that the attention mechanism has driven more by the semantics and language model when using traditional word translation, but by location when using copying operation. Fig. 11 . An example of one kind of Copy Mechanism proposed by Luong et al. BIB009 , the subscripts of < unk > symbol (refer as d)is a relative position of corresponding source words, where the alignment relation is a target word at position j is aligned to a source word at position i = j + d. Besides the Copy Mechanism, using extra knowledge is also useful in handling some other linguistic scenarios, which is highly related to the vocabulary coverage problem. Arthur et al. incorporated lexicons knowledge to assist with translation in low-frequency words BIB017 . Feng et al. proposed a similar method with a memory-augmented NMT (M-NMT) architecture, and it used novel attention mechanism to get the extra knowledge from the dictionary that constructed by SMT  . Additionally, using context information can also be applied to improve the translation quality of ambiguous words (Homographs) BIB022 . In a nutshell, there are many context-based refinements have been proposed, most of them using Copy Mechanism to handle the OOV words with various of alignment algorithm to locate the corresponding word in the target side. However, these kinds of methods have a limited room for further improvement because the Copy Mechanism is too crude to handle the sophisticated scenarios in different languages. Practically, these methods perform poorly in languages which are rich morphology like Finnish and Turkish, which motivated method with better generalization BIB010 . 3) Methods in fine grit level: This sub-section introduce some more "intelligent" ways that focus on using additional information inside the word unit. It's obviously that such additional information could enhance the ability in covering the various of the linguistic phenomenon. In previous research, although using semantic information of word unit could provide the vast majority of learning features, other features in sub-word level are generally ignored. From the perspective of linguistic, the concept of "word" is the basic unit of language but not the minimum one in containing semantic information, and there are abundant experienced rules could be learned from the inside of word units like shape and suffix. Comparing with identity copy or dictionary translation which regards the rare words as an identical entity, the refined method using fine-grit information is more adaptable. Further, a more "radical" method was proposed, which just treats words in character level completely. It would be an innovative concern for future work. In this category, one popular choice is using the subword unit, the most remarkable achievement was proposed by Sennrich et al., which has been proved to have the best performance in some shared result BIB011 . Concretely, in this design, they treat unknown words as sequences of sub-word units, which is reasonable in terms of the composition of the vast majority of these words(e.g., named entities, loanwords, and morphologically complex words). In order to represent these OOV words completely, one intuitive solution is to build a predefined sub-word dictionary that contains enough variants of units. However, restoring the sub-words cause massive space consumption in vocabulary size, which is effectively cancels out the whole purpose of reducing the computational efficiency in both time and space. Under this circumstances, a Byte Pair Encoding (BPE) based sub-words extraction method was applied for word segmentation operation in both sides of languages, which successfully adapted this old but effective data compression method in text pre-processing. In concrete details of this adapted BPE method, it alternated merging frequent pairs of bytes with characters or sequence of characters, and word segmentation process followed the below steps: (1) Prepare a large training corpus(generally are bilingual corpus). (2) Determined the size of the sub-word vocabulary. (3) Split the words to a sequence of characters (using a special space character for marking the original spaces). (4) Merge the most frequent adjacent pair of characters (e.g., in English, this may be c and h into ch). (5) Repeat step 4 until reaching the fixed given number of times or the defined size of the vocabulary. Each of these steps would increase the vocabulary by one. The figure shows a toy example of the method. As for the practical result of word segmentation, the most frequent words will be merged as single tokens, while the rare words(which is similar to the OOV words in previous categories' work) may still contain un-merged sub-words. However, they have been found rare in processed text BIB011 . Further work of BPE method has also been proposed to obtaining a better generalization. Taku proposed subword regularization in later as an alternative to handle the spurious ambiguity phenomenon in BPE, and they further proposed a new subword segmentation algorithm based on a uni-gram language model, which shares the same idea with BPE but was more flexible in getting multiple segmentation based on their probabilities. Similarly, Wu et al. used "workpieces" concept to handle the OOV words which were once applied on the Google speech recognition system to solve Japanese/Korean segmentation problem BIB005 BIB018 . This method breaks words into word pieces to get a balance between flexibility with efficiency when using single characters and full words separately. Another notable concern is modeling sequence in characters-level. Inspired by using character-level information completely in building NLM BIB012 , Costa-jussa' & Fonollosa used CNN with highway network to model the characters directly BIB019 , they deployed this architecture in source side with a common word-level generation in target side. Similarly, Ling et al. and Ballesteros et al. have proposed model respectively that using RNN(LSTM) to build character level embedding and composes it into the word embedding BIB010 [98], this idea later has been applied in building an RNN based character-level NMT model BIB013 . More recently, BIB020 proposed a hybrid model that combines the word level RNN with character level RNN for assist BIB020 . Concretely, Luongs' method translates mostly at the word level, when encounter an OOV word, character level RNN would be used for the consult. The figure shows the detailed architecture of this model. On the other hand, the trying of designing a fully characterlevel translation model has also got attention accordingly. Chung et al. used BPE method to extract a sequence of subword in encoder side, they just varied the decoder by using pure characters, and it has indicated to provide comparable performance with models uses sub-words BIB021 . Motivated by aforementioned work, Lee et al. proposed fully character-level NMT without any segmentation, it was based on CNN pooling with highway layers, which can solve the prohibitively slow speed of training in Luong and Manning's work BIB023 . VI. ADVANCED MODELS This section gives a demonstration of some advanced models that have got the state-of-the-art performance, while all of them belong to different categories of model structure. Experimental result indicated that all these networks can achieve similar performance with different advantages in their corresponding aspects.