A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Structure of Attention Mechanism <s> Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Structure of Attention Mechanism <s> An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems that already incorporate known techniques such as dropout. Our ensemble model using different attention architectures yields a new state-of-the-art result in the WMT’15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker. 1 <s> BIB002
There are many variants in the implementation of Attention Mechanism. Here we just give the detailed description of Attention Mechanism which has been widely accepted as bringing significant contribution in the development of NMT. 1) basic structure: The structure of attention mechanism was originally proposed by Bahdanau et al. In later, Luong et al. proposed similar structure with small distinctions and extends this work BIB002 BIB001 . To simplify the discussion, here we take Luong et al.'s method as an example. Concretely, in encoding period, this mechanism receiving the input words like the basic NMT model, but instead of compressing all the information in one vector, every unit in the top layer of encoder will generate one vector that represents one time-step in the source sentence. In the decoding period, the decoder wont predict the word just use its own information. However, it collaborates with the attention layer to get the translation. The input of attention mechanism is the hidden states in the top layer of the encoder and the current decoder. It gets the relativity order by calculating the following steps: 1. The current decoding hidden state h t will be used to compare with all source states h s to derive the attention weights score s t . 2. The attention weights a t is driven by normalization operation for all attention weight score. 3. Based on the attention weights, we then compute the weighted average of the source states as a context vector c t . 4. Concatenate the context vector with the current decoding hidden state to yield the final attention vector(the exact combination method can be different). 5. The attention vector is fed as an input to decoder in the next time-step (applicable for input feeding). The first three steps can be summarized by the equations below: Among the above function, The score function could be defined in different ways. Here, we two classic definitions: Back to the decoding period, it receives the information from both two sides, the decoder hidden state and the attention vector, given the current two vectors, it then predicts the words by alignment them to a new vector, then it usually has another layer to predict the current target word.