A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> We propose a new phrase-based translation model and decoding algorithm that enables us to evaluate and compare several, previously proposed phrase-based translation models. Within our framework, we carry out a large number of experiments to understand better and explain why phrase-based models out-perform word-based models. Our empirical results, which hold for all examined language pairs, suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translations. Surprisingly, learning phrases longer than three words and learning phrases from high-accuracy word-level alignment models does not have a strong impact on performance. Learning only syntactically motivated phrases degrades the performance of our systems. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> We introduce a class of probabilistic continuous translation models called Recurrent Continuous Translation Models that are purely based on continuous representations for words, phrases and sentences and do not rely on alignments or phrasal translation units. The models have a generation and a conditioning aspect. The generation of the translation is modelled with a target Recurrent Language Model, whereas the conditioning on the source sentence is modelled with a Convolutional Sentence Model. Through various experiments, we show first that our models obtain a perplexity with respect to gold translations that is > 43% lower than that of stateof-the-art alignment-based translation models. Secondly, we show that they are remarkably sensitive to the word order, syntax, and meaning of the source sentence despite lacking alignments. Finally we show that they match a state-of-the-art system when rescoring n-best lists of translations. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic information. In this paper we show that the strong learning capability of neural MT models does not make linguistic features redundant; they can be easily incorporated to provide further improvements in performance. We generalize the embedding layer of the encoder in the attentional encoder--decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature. We add morphological features, part-of-speech tags, and syntactic dependency labels as input features to English German, and English->Romanian neural machine translation systems. In experiments on WMT16 training and test sets, we find that linguistic input features improve model quality according to three metrics: perplexity, BLEU and CHRF3. An open-source implementation of our neural MT system is available, as are sample files and configurations. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> We present Nematus, a toolkit for Neural Machine Translation. The toolkit prioritizes high translation accuracy, usability, and extensibility. Nematus has been used to build top-performing submissions to shared translation tasks at WMT and IWSLT, and has been used to train systems for production environments. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et. al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task. <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Machine Translation <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB009
Machine translation (MT) is the quintessential application of natural language processing. It involves the use of mathematical and algorithmic techniques to translate documents in one language to another. Performing effective translation is intrinsically onerous even for humans, requiring expert proficiency and prowess in areas such as morphology, syntax, and semantics, as well as an adept understanding and discernment of cultural sensitivities, for both of the languages (and associated societies) under consideration  . Historically, this work included statistical approaches to phrase BIB001 ], syntax  , and word-based translation  , each having its own limitations. Recent attempts at this task use neural machine translation (NMT), which uses end-to-end deep neural networks. The first use of such a model BIB004 stemmed from the success of continuous recurrent representations in capturing syntax, semantics, and morphology BIB002 in addition to the ability of recurrent neural networks to build robust language models BIB003 . This original NMT model used a combination of generative convolutional and recurrent layers to encode and optimize a source language model and cast this into a target language. Numerous novel and effective advances to this model have since been made , as derived models are continually improving, finding answers to the shortcomings of their predecessors and overcoming any need for hand engineering BIB006 . Recent progress includes effective initialization of decoder hidden states, use of conditional gated attentional cells, removal of bias in embedding layers, use of alternative decoding phases, factorization of embeddings, large-scale stacking of recurrent layers, and test time use of the beam search algorithm BIB007 . The standard initialization for the decoder state is that proposed by Bahsanau et al. [2014] , using the last backward encoder state. However, as noted by BIB006 , using the average of the embedding or annotation layer seems to lead to the best translations. Gated recurrent cells have been the gold standard for sequence-to-sequence tasks, a variation of which is a conditional GRU (cGRU), proposed by BIB007 , most effectively utilized with an attention mechanism. A cGRU cell consists of three key components: two GRU transition blocks and an attention mechanism between them. These three blocks combine the previous hidden state, along with the attention context window to generate the next hidden state. Altering the decoding process from Look, Generate, Update (Look at input, generate output token, update hidden representation) to a process of Look, Update, Generate can simplify the final decoding implementation. Adding further source attributes such as morphological segmentation labels, part-of-speech tags, and syntactic dependency labels has been shown to improve models, and concatenating or factorizing these with embeddings has been shown to increase robustness further BIB007 BIB005 . For remembering long-term dependencies, vertically stacked recurrent units have been the standard, with the optimum number of layers having been determined to be roughly between two and sixteen BIB006 , depending on the desired input length as well as the presence and density of residual connections. At test time, a beam search algorithm can be used beside the final softmax layer for considering multiple target predictions in a greedy fashion, allowing the best predictions to be found without looking through the entire hypothesis space . In a direction diverging from previous work, some researchers have proposed discarding the large number of recurrent and convolutional layers and instead focusing exclusively on attention mechanisms to encode a language globally from input to output BIB008 BIB009 . Preferring such "self-attention" mechanisms over traditional layers is motivated by the following three principles: reducing the complexity of computations required per layer, minimizing sequential training steps, and lastly, abating the path length from input to output and its handicap on the learning of the long-range dependencies which are necessary in many sequencing tasks  . Apart from increased accuracy across translation tasks, self-attention models allow more parallelization throughout architectures, decreasing the possible training times and minimizing necessary sequential steps.