A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Related Work and Our Contribution <s> Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results. <s> BIB001 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Related Work and Our Contribution <s> Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review the different neural architectures in which attention has been incorporated, and also show how attention improves interpretability of neural models. Finally, we discuss some applications in which modeling attention has a significant impact. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications. <s> BIB002 </s> A Survey of Deep Learning Techniques for Neural Machine Translation <s> B. Related Work and Our Contribution <s> Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area. <s> BIB003
Although there is little work in the literature survey of NMT, some other works are highly related. Lipton et al. have summarized the conventional methods in sequence learning  , which provided essential information for the origin of NMT as well as the related base knowledge. Britz et al. and Tobias Domhan have done some model comparison work in NMT with experiment and evaluation in the practical performance of some wildly accepted technologies, but they have rare theoretical analysis, especially in presenting the relationship between different proposed models BIB001  . On the other hand, some researchers limited their survey work on a special part related to NMT the Attention Mechanism, but both of them have a general scope that oriented to all kinds of AI tasks with Attention BIB002 BIB003 . Maybe the most related work was an earlier doctoral thesis written by Minh-Thang Luong in 2016  , which included a comprehensive description about the original structure of NMT as well as some wildly applied tips. This paper, however, focuses on a direct and up-to-date literature survey about NMT. We have investigated a lot about the relevant literature in this new trend and provided comprehensive interpretation for current mainstream technology in NMT. As for concrete components, this literature survey investigates the origin and recent progresses in NMT, categorizes these models by their different orientation in the model structure. Then we demonstrate the insight of these NMT types, summarize the strengths and weaknesses by reviewing their design principal and corresponding performance analysis in translation quality and speed. We also give a comprehensive overview of two components in NMT development, namely attention mechanism and vocabulary coverage mechanism, both of which are indispensable for current achievement. At last, we give concentration on some literature which proposed advanced models with comparison work; we introduce these considerable models as well as the potential direction in future work. Regarding the survey scope, some subareas of NMT with less attention were deliberately left out of the scope except with brief description in future trend. These include but are not limited to the literature of robustness of NMT, domain adaptation in NMT and other applications that embed NMT method (such as speech translation, document translation). Although the research scope has been specifically designed, due to the numerous of researches and the inevitable expert selection bias, we believe that our work is merely a snapshot of part of current research rather than all of them. We are hoping that our work could provide convenience for further research. The remaining of the paper is organized as follows. Section II provides an introduction of machine translation and presents its history of development. Section III introduces the structure of NMT and the procedure of training and testing. Section IV discusses attention mechanism, an essential innovation in the development of NMT. Section V surveys a variety of methods in handling word coverage problem and some fluent divisions. Section VI describes three advanced models in NMT. Finally, Section VII discusses the future trend in this field.