A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Development of Machine Translation 1) Rule-based Machine Translation: Rule-based Machine <s> A number of connectionist models capable of representing data with compositional structure have recently appeared. These new models suggest the intriguing possibility of performing holistic structure-sensitive computations with distributed representations. Two possible forms of holistic inference, transformational inference and confluent inference, are identified and compared. Transformational inference was successfully demonstrated by Chalmers; however, the pure transformational approach does not consider the eventual inference tasks during the process of learning its representations. Confluent inference is introduced as a method for achieving a tight coupling between the distributed representations of a problem and the solution for the given inference task while the net is still learning its representations. A dual-ported RAAM architecture based on Pollack's Recursive Auto-Associative Memory is implemented and demonstrated in the domain of Natural Language translation. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Development of Machine Translation 1) Rule-based Machine Translation: Rule-based Machine <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Development of Machine Translation 1) Rule-based Machine Translation: Rule-based Machine <s> Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of-the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models -- such as the reordering of verbs -- while pointing out other aspects that remain to be improved. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Development of Machine Translation 1) Rule-based Machine Translation: Rule-based Machine <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. Development of Machine Translation 1) Rule-based Machine Translation: Rule-based Machine <s> We describe an open-source toolkit for neural machine translation (NMT). The toolkit prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques. <s> BIB005
Translation is the first design in MT, which is based on the hypothesis that all different languages have its symbol in representing the same meaning. Because in usual, a word in one language could find its corresponding word in another language with the same meaning. In this method, the translation process could be treated as the word replacement in the source sentence. In terms of 'rule-based', since different languages could represent the same meaning of sentence in different word order, the word replacement method should base on the syntax rules of both two languages. Thus every word in the source sentence should take its corresponding position in the target language. The rule-based method has a beautiful theory but hardly achieves satisfactory performance in implementation. This is because of the computational inefficiency in determining the adaptive rule of one sentence. Besides, grammar rules are also hard to be organized, since linguists summarize the grammar rules, and there are too many syntax rules in one language (especially language with more relaxed grammar rules). It is even possible that two syntax rules conflict with each other. The most severe drawback of rule-based method is that it has ignored the need of context information in the translation process, which destroys the robustness of rule-based machine translation. One famous example was given by Marvin Minsky in 1966, where he used two sentences given below: "T he pen is in the box" "T he box is in the pen" Both sentences have the same syntax structure. The first sentence is easy to understand; but the second one is more confusing, since the word "pen" is a polysemant, which also means "fence" in English. But it is difficult for the computer to translate the "pen" to that meaning; the word replacement is thus an unsuccessful method. 2) Statistical Machine Translation: Statistical Machine Translation (SMT) has been the mainstream technology for the past 20 years. It has been successfully applied in the industry, including Google translation, Baidu translation, etc. Different from Rule-based machine translation, SMT tackles the translation task from a statistical perspective. Concretely, the SMT model finds the words (or phrases) which have the same meaning through bilingual corpus by statistics. Given one sentence, SMT divides it into several sub-sentences, then every part could be replaced by target word or phrase. The most prevalent version of SMT is Phrase-based SMT (PBSMT), which in general includes pre-processing, sentence alignment, word alignment, phrase extraction, phrase feature preparation, and language model training. The key component of a PBSMT model is a phrase-based lexicon, which pairs phrases in the source language with phrases in the target language. The lexicon is built from the training data set which is a bilingual corpus. By using phrases in this translation, the translation model could utilize the context information within phrases. Thus PBSMT could outperform the simple word-toword translation methods. 3) Neural Machine Translation: It has been a long time since the first try on MT task by neural network  BIB001 . Because of the poor performance in the early period and the computing hardware limitation, related research in translation by neural network has been ignored for many years. Due to the proliferation of Deep Learning in 2010, more and more NLP tasks have achieved great improvement. Using deep neural networks for MT task has received great attention as well. A successful DNN based Machine Translation (NMT) model was first proposed by Kalchbrenner and Blunsom BIB002 , which is a totally new concept for MT by that time. Comparing with other models, the NMT model needs less linguistic knowledge but can produce a competitive performance. Since then, many researchers have reported that NMT can perform much better than the traditional SMT model BIB005 [112]  [114] BIB003 , and it has also been massively applied to the industrial field BIB004 .