A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Transformer and Transformer based models <s> While phrase-based statistical machine translation systems currently deliver state-of-the-art performance, they remain weak on word order changes. Current phrase reordering models can properly handle swaps between adjacent phrases, but they typically lack the ability to perform the kind of long-distance re-orderings possible with syntax-based systems. In this paper, we present a novel hierarchical phrase reordering model aimed at improving non-local reorderings, which seamlessly integrates with a standard phrase-based system with little loss of computational efficiency. We show that this model can successfully handle the key examples often used to motivate syntax-based systems, such as the rotation of a prepositional phrase around a noun phrase. We contrast our model with reordering models commonly used in phrase-based systems, and show that our approach provides statistically significant BLEU point gains for two language pairs: Chinese-English (+0.53 on MT05 and +0.71 on MT08) and Arabic-English (+0.55 on MT05). <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Transformer and Transformer based models <s> We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Transformer and Transformer based models <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> C. Transformer and Transformer based models <s> The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU. <s> BIB004
Transformer is a new NMT structure proposed by Vaswani et al. BIB003 . Different from existing NMT models, it has abandoned the standard RNN/CNN structures and designed an innovative multi-layer self-attention blocks that are incorporated with a positional encoding method. This new trend of structure design takes the advantages from both RNN and CNN based model, which has been further used for initializing the input representation for other NLP tasks. Notably, Transformer is a complete Attention based NMT model. 1) Structure of the model: Transformer has its distinct structure in its model, where the major differences are the input representation and multi-head attention. (1) Input representation Transformer has its unique representation in handling input data that quite different with recurrent or convolution model. For computing Self-Attention we mentioned, transformer handles the input as three kind of vectors for different purpose. They are Key,Value and Query vectors. And all these vectors are driven by multiplying the input embedding with three matrices that we trained during the training process. Also, there's Positional Encoding method was applied for enhancing the modeling ability of sequence order, since Transformer has abandoned recurrence structure, this kind of method made a compensation by inject word order information in to feature vector, which can avoid the model to become invariant to sequence ordering BIB002 . Specifically, Transformer add Positional Encoding to the input embedding at the bottoms of the encoder and decoder stacks, the Positional Encoding has been designed to have the same dimension with model embedding and thus could be summed. Positional Encoding can be calculated by applying positional functions directly or be learned BIB004 , with be proven to have similar performance in final evaluation. In Transformer, adding Positional Encoding by using since and cosine functions is finally chosen, and each position can be encoded in the following way: P E (pos,2i) = sin pos/10000 2i/dmodel ) P E (pos, 2i+1) = cos pos/10000 2i/dmodel BIB001 where pos indicates the position and i indicates the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid function, and the wavelengths form a geometric progression from 2π to 20000π. The reason of choosing these functions is that they have been assumed theoretically in helping the model to learn to attend by relative positions easily, due to the characters that for any fixed offset k, P E pos + k can be represented as a linear function of P E pos . (2) Multi-head Self-Attention Self-Attention is the major innovation in Transformer. But in implementation, rather than just computing the Self-Attention once, the multi-head mechanism runs through the scaled dotproduct attention multiple times in parallel, and the outputs of these independent attention are then concatenated and linearly transformed into the expected dimensions. This multiple times Self-Attention computation is called Multi-head Self-Attention, which is applied for allowing the model to jointly attend to information from different representation sub-spaces at different positions. (