A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB003 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system. <s> BIB004 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies. <s> BIB005 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> A. ConvS2S <s> The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU. <s> BIB006
ConvS2S is short for Convolutional Sequence to Sequence, which is an end-to-end NMT model proposed by Gehring et al. BIB006 . Different with most of RNN based NMT models, ConvS2S is entirely CNN based model both in encoder and decoder. In the network structure, ConvS2S stacked 15 layers of CNN in its encoder and decoder with fixed kernel width of 3. This deep structure helps to mitigate the weakness in capturing context information. In respect of network details, ConvS2S applied Gated Linear Units (GLU) BIB001 in building network, which provide a gated function for output of convolution layer. Specifically, the output of convolution layer Y ∈ R 2d which is a vector with double times dimensions (2d numbers of dimensions) of each input element's embedding (d numbers of dimensions), the gated function processes the output Y = [AB] ∈ R 2d by implementing the equation 8, where both A and B are d dimensions vector, and the function σ(B) is a gated function used to control which inputs A of the current context are relevant. This non-linearity operation has been proved to be more effective in applying training language model BIB001 , surpassing those only applying tanh function on A BIB002 . In addition, ConvS2S also used residual connection BIB003 between different convolution layers.  Besides the innovation of CNN based encoder-decoder structure, ConvS2S also applied similar Attention Mechanism that has been wildly accepted by RNN model, called Multistep Attention. Concretely, Multi-step Attention is a separate attention structure applied in each decoder layer. In the process of calculating attention, the current hidden state d l i (i.e., the output of the lth layer) has combined with previous output embedding g i as a vector of decoder state summary d l i : Then the attention vector a l ij (i.e., the attention of state i with source element j in decoder layer l) would be driven by the dot-product of the summary vector with the output of the final encoder layer z u j . Lastly, the context vector is calculated as the weighted average of the attention vector a l ij with the encoder output z u j as well as the encoder input e j . . This model has directly inherited the structure of GNMT model that was proposed by Wu et al. BIB004 . Specifically, RNMT+ can be seen as an enhanced GNMT model, which demonstrated the best performance of RNN based NMT model. In model structure, RNMT+ mainly differs from the GNMT model in the following several perspectives: First, RNMT+ used six bi-directional RNN (LSTM) in its decoder, whereas GNMT used one layer of bi-directional RNN with seven layers of unidirectional RNN. This structure has sacrificed the computation efficiency in return for the extreme performance. Second, RNMT+ applied Multi-head additive attention instead of the single-head attention in conventional NMT model, which can be seen as taking advantage of Transformer model. Third, synchronous training strategy was provided in the training process, which improved the convergence speed with model performance based on empirical results BIB005 . In addition, inspired by Transformer model, per-gate layer normalization [101] was applied, which has indicated to be helpful in stabilizing model training. Fig. 13 . The structure of RNMT+ model,which has the similar structure of GNMT with adaptive innovation in Attention Mechanism