A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> Abstract Because of the “all-or-none” character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms, with the addition of more complicated logical means for nets containing circles; and that for any logical expression satisfying certain conditions, one can find a net behaving in the fashion it describes. It is shown that many particular choices among possible neurophysiological assumptions are equivalent, in the sense that for every net behaving under one assumption, there exists another net which behaves under the other and gives the same results, although perhaps not in the same time. Various applications of the calculus are discussed. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> The promise of unsupervised learning methods lies in their potential to use vast amounts of unlabeled data to learn complex, highly nonlinear models with millions of free parameters. We consider two well-known unsupervised learning models, deep belief networks (DBNs) and sparse coding, that have recently been applied to a flurry of machine learning applications (Hinton & Salakhutdinov, 2006; Raina et al., 2007). Unfortunately, current learning algorithms for both models are too slow for large-scale applications, forcing researchers to focus on smaller-scale models, or to use fewer training examples. In this paper, we suggest massively parallel methods to help resolve these problems. We argue that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods. We develop general principles for massively parallelizing unsupervised learning tasks using graphics processors. We show that these principles can be applied to successfully scaling up learning algorithms for both DBNs and sparse coding. Our implementation of DBN learning is up to 70 times faster than a dual-core CPU implementation for large models. For example, we are able to reduce the time required to learn a four-layer DBN with 100 million free parameters from several weeks to around a single day. For sparse coding, we develop a simple, inherently parallel algorithm, that leads to a 5 to 15-fold speedup over previous methods. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> Scaling up deep learning algorithms has been shown to lead to increased performance in benchmark tasks and to enable discovery of complex high-level features. Recent efforts to train extremely large networks (with over 1 billion parameters) have relied on cloudlike computing infrastructure and thousands of CPU cores. In this paper, we present technical details and results from our own system based on Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology: a cluster of GPU servers with Infiniband interconnects and MPI. Our system is able to train 1 billion parameter networks on just 3 machines in a couple of days, and we show that it can scale to networks with over 11 billion parameters using just 16 machines. As this infrastructure is much more easily marshaled by others, the approach enables much wider-spread research with extremely large neural networks. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> INTRODUCTION <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB007
The field of natural language processing (NLP) encompasses a wide variety of topics which involve the computational processing and understanding of human languages. Beginning in the 1930s  ], the field historically has focused on the use of human-crafted rules, ad-hoc processing, and mathematical logic. Since the 1980s, it has increasingly relied on data-driven computation involving statistics, probability, and machine learning   . The first artificial neural networks (ANNs) were conceived in the 1940s BIB001 as computational devices to capture human intelligence. Recent increases in computational power and parallelization, harnessed by Graphical Processing Units (GPUs) BIB005 BIB002 ], now allow for "deep learning", which utilizes ANNs, sometimes with billions of trainable parameters  . Additionally, the contemporary availability of large datasets, facilitated by sophisticated data collection processes, enables the training of such deep architectures via their associated learning algorithms BIB003 BIB007 BIB006 . In recent years, researchers and practitioners in natural language processing have leveraged the power of modern artificial neural networks with many propitious results, beginning in large part with the pioneering work of BIB004 . Over the last half decade, the use of artificial neural networks and deep learning has upsurged considerably in the field   . This has led to significant advances both in core areas of NLP and in areas in which it is directly applied to achieve practical and useful objectives. This survey provides a brief introduction to both natural language processing and deep neural networks, and then presents an extensive discussion on how deep learning is being used to solve current problems in NLP. This discussion will be found useful by readers who want to familiarize themselves quickly with the current state of the art before embarking upon further advanced research and practice themselves. The topics of natural language processing and artificial neural networks, including deep learning, are introduced in Section 2. The ways in which deep learning has been used to solve problems in core areas of NLP are presented in Section 3. The section is broken down into several subsections, namely natural language modeling (3.1), morphology (3.2), parsing (3.3), and semantics (3.4). Applications of deep learning to more practical areas in natural language processing are then discussed in Section 4. Specifically discussed are information extraction (4.1), text classification (4.2), summarization (4.3), question answering (4.4), machine translation (4.5), and image and video captioning (4.6). Conclusions are then drawn in Section 5 with a brief summary of the state of the art as well as predictions, suggestions, and other thoughts on the future of this dynamically evolving area.