A Survey of the Usages of Deep Learning in Natural Language Processing <s> CONCLUSIONS <s> Abstract : The paper describes a system for the computer understanding of English. The system answers questions, executes commands, and accepts information in normal English dialog. It uses semantic information and context to understand discourse and to disambiguate sentences. It combines a complete syntactic analysis of each sentence with a 'heuristic understander' which uses different kinds of information about a sentence, other parts of the discourse, and general information about the world in deciding what the sentence means. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> CONCLUSIONS <s> This best-selling title, considered for over a decade to be essential reading for every serious student and practitioner of computer design, has been updated throughout to address the most important trends facing computer designers today. In this edition, the authors bring their trademark method of quantitative analysis not only to high-performance desktop machine design, but also to the design of embedded and server systems. They have illustrated their principles with designs from all three of these domains, including examples from consumer electronics, multimedia and Web technologies, and high-performance computing. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> CONCLUSIONS <s> We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> CONCLUSIONS <s> A technology inspired by biological principles but 'steamrolled for decades' prepares to take off as Moore's Law approaches its long-anticipated end. <s> BIB004
Early applications of natural language processing included a well-acclaimed but simpleminded algebra word problem solver program called STUDENT  , as well as interesting but severely constrained conversational systems such as Eliza, which acted as a "psycho-therapist"  , and another that conversed about manipulating blocks in a microworld BIB001 ]. Nowadays, highly advanced applications of state-of-the-art NLP programs are ubiquitous. These include Google's and Microsoft's machine translators, which translate more or less competently from a language to scores of other languages, as well as a number of devices which process voice commands and respond in like. The emergence of these sophisticated applications, particularly in deployed settings, acts as a testament to the impressive accomplishments that have been made in this domain over the last sixty or so years. Without a doubt, incredible progress has taken place, particularly in the last several years. As has been shown, this recent progress has a clear causal relationship with the remarkable advances in Artificial Neural Networks. Considered an "old" technology just a decade ago, these machine learning constructs have ushered in progress at an unprecedented rate, breaking performance records in myriad tasks in miscellaneous fields. In particular, deep neural architectures, have instilled models with greater "understanding" of natural languages. Both convolutional and recurrent specimen contribute to the state of the art in the field, however, it is not clear which produces superior results across the rich and varying terrain of the NLP field. Consolidating the analysis of all the models surveyed, a few general trends can be surmised. First, recurrent networks (particularly those with explicit memory, such as LSTMs and GRUs) with attention mechanisms are the strongest decoders. Second, the best decoders tend to be those that implement CNNs capped by RNNs, with the convolutional aspect seeming slightly more important. Third, while highly engineering networks usually optimizes results, there is no substitute for cultivating networks with large quantities of high quality data. Following from this final observation, it may be useful to direct more research effort toward training methodologies, rather than developing expensive, highly-specialized components to squeeze the last drops of performance from complex models. While the numerous stellar architectures being proposed each month are highly competitive, muddling the process of identifying a winning architecture, the methods of evaluation used add just as much complexity to the problem. Many datasets used to evaluate new models are generated specifically for that model and are then used only several more times, if at all. As the features and sizes of these datasets are highly variable, this makes comparison difficult, particularly over time. Most subfields of NLP, as well as the field as a whole, would be behooved by extensive, large-scale discussions regarding the necessary contents of such datasets, followed by the compilation of such sets. In addition to high variability in evaluation data, there are numerous metrics used to evaluate performance on each task. Oftentimes, comparing similar models is difficult due to the fact that different metrics are reported for each. Agreement on particular sets of metrics would go a long way toward ensuring clear comparisons in the field. Furthermore, metrics are usually only reported for the best case, with few mentions of average cases and variability, or of worst cases. While it is important to understand the possible performance of new models, it is just as important to understand the standard performance. If models produce highly variable results, they may take many attempts to train to the cutting-edge levels reported in research. In most cases, this is undesirable, and models that can be consistently trained to relatively high levels of performance are preferable. While increasingly large numbers of randomized parameters do reduce variation in performance, some variance will always exist, necessitating the reporting of more than just best-case metrics. One final recommendation for future work is that it be directed toward a wider variety of languages than it is at present. Currently, the vast majority of research in NLP is conducted on the English language, with another sizeable portion using Mandarin Chinese. In translation tasks, English is almost always either the input language or the output language, with the other end usually being one of a dozen major European or Eastern Asian languages. This neglects entire families of languages, as well as the people who speak them. Many linguistic intricacies may not be expressed in any of the languages used, and therefore are not captured in current NLP software. Furthermore, there are thousands of languages spoken throughout the world, with at least eighty spoken by more than 10 million people, meaning that current research excludes an immense segment of humankind. Collection and validation of data in underanalyzed languages, as well as testing NLP models using such data, will be a tremendous contribution to not only the field of natural language processing, but to human society as a whole. Due to the small amounts of data available in many languages, the authors do not foresee the complete usurpation of traditional NLP models by deep learning any time in the near future. Deep learning models (and even shallow ANNs) are extremely data-hungry. Contrastingly, many traditional models require only relatively small amounts of training data. However, looking further forward, it can be anticipated that deep learning models will become the norm in computational linguistics. BIB003 sparked the deep learning revolution in NLP, although one of the key contributions of their work-that of a single unified model-was not realized widely. Instead, neural networks were introduced into traditional NLP tasks, and are only now reconnecting. In the field of parsing, for example, most models continue to implement non-neural structures, simply using ANNs on the side to make the decisions that were previously done using rules and probability models. While more complete NLP architectures are obviously becoming more and more of a reality, understanding the abstract concepts handled by such networks is important to understanding how to build and train better networks. Furthermore, as abstraction is a hallmark of human intelligence, understanding of the abstractions that take place inside an ANN may aid in the understanding of human intelligence and the processes that underlie it. Just as human linguistic ability is only a piece of our sentience, so is linguistic processing just a small piece of artificial intelligence. Understanding how such components are interrelated is important in constructing more complete AI systems, and creating a unified NLP architecture is another step toward making such a system a reality. This goal will also be aided by further advances in computational equipment. While GPUs have significantly improved the ability to train deep networks, they are only a step in the right direction  . The next step is the wider availability of chips designed specifically for this purpose, such as Google's Tensor Processing Unit (TPU), Microsoft's Catapult, and Intel's Lake Crest BIB002 . Ultimately, artificial neural networks implemented in traditional von Neumann style computers may not be able to reach their full potential. Luckily, another old line of work in computer science and engineering has seen a resurgance in recent years: neuromorphic computing. With neuromorphic chips, which implement neural structures at the hardware level, expected much more widely in coming years BIB004 ], the continuation of deep learning and the longevity of its success can be highly anticipated, ensuring the opportunity for sustained progress in natural language processing.