A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. The Recent Development in NMT <s> Abstract A longstanding difficulty for connectionist modeling has been how to represent variable-sized recursive data structures, such as trees and lists, in fixed-width patterns. This paper presents a connectionist architecture which automatically develops compact distributed representations for such compositional structures, as well as efficient accessing mechanisms for them. Patterns which stand for the internal nodes of fixed-valence trees are devised through the recursive use of backpropagation on three-layer auto-associative encoder networks. The resulting representations are novel, in that they combine apparently immiscible aspects of features, pointers, and symbol structures. They form a bridge between the data structures necessary for high-level cognitive tasks and the associative, pattern recognition machinery provided by neural networks. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. The Recent Development in NMT <s> A number of connectionist models capable of representing data with compositional structure have recently appeared. These new models suggest the intriguing possibility of performing holistic structure-sensitive computations with distributed representations. Two possible forms of holistic inference, transformational inference and confluent inference, are identified and compared. Transformational inference was successfully demonstrated by Chalmers; however, the pure transformational approach does not consider the eventual inference tasks during the process of learning its representations. Confluent inference is introduced as a method for achieving a tight coupling between the distributed representations of a problem and the solution for the given inference task while the net is still learning its representations. A dual-ported RAAM architecture based on Pollack's Recursive Auto-Associative Memory is implemented and demonstrated in the domain of Natural Language translation. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. The Recent Development in NMT <s> In this paper, we propose a novel neural network model called RNN Encoder‐ Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixedlength vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder‐Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases. <s> BIB003
We devide the recent developement of NMT in five main stages: (a) the original NMT with a shallow layer, (b) SMT assisted by NLM, (c) the DNN based NMT, (d) NMT with attention mechanism, (e) the attention-based NMT. NMT with Shallow Layer Even before the Deep Learning, Allen has used binary encoding to train an NMT model in 1987  . Later in 1991, Chrisman used Dual-ported RAAM architecture BIB001 to build an original NMT model BIB002 . Although both of them have a pretty primitive design with the limited result when looking back, their work has indicated the original idea of this field. The further related work has almost stagnated in the following decades, due to the huge progress that SMT method acquired at that period, as well as the limited computing power and data samples. SMT assisted by NLM Based on the above discussion, NLM has revolutionized the traditional LM even before the rise of deep learning. Later on, deep RNN based NLM has been applied in the SMT system. Cho et al. proposed an SMT model along with an NLM model BIB003 . Although the main body is still SMT, this hybrid method provides a new direction for the emergence of a pure deep learning-based NMT.