A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> Abstract : At this early stage in the development of cognitive science, methodological issues are both open and central. There may have been times when developments in neuroscience, artificial intelligence, or cognitive psychology seduced researchers into believing that their discipline was on the verge of discovering the secret of intelligence. But a humbling history of hopes disappointed has produced the realization that understanding the mind will challenge the power of all these methodologies combined. The work reported in this chapter rests on the conviction that a methodology that has a crucial role to play in the development of cognitive science is mathematical analysis. The success of cognitive science, like that of many other sciences, will, I believe, depend upon the construction of a solid body of theoretical results: results that express in a mathematical language the conceptual insights of the field; results that squeeze all possible implications out of those insights by exploiting powerful mathematical techniques. This body of results, which I will call the theory of information processing, exists because information is a concept that lends itself to mathematical formalization. One part of the theory of information processing is already well-developed. The classical theory of computation provides powerful and elegant results about the notion of effective procedure, including languages for precisely expressing them and theoretical machines for realizing them. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> Preface Table of Notation Part 1: Unconstrained Optimization Introduction Structure of Methods Newton-like Methods Conjugate Direction Methods Restricted Step Methods Sums of Squares and Nonlinear Equations Part 2: Constrained Optimization Introduction Linear Programming The Theory of Constrained Optimization Quadratic Programming General Linearly Constrained Optimization Nonlinear Programming Other Optimization Problems Non-Smooth Optimization References Subject Index. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> Classification is an important problem in the emerging field of data mining. Although classification has been studied extensively in the past, most of the classification algorithms are designed only for memory-resident data, thus limiting their suitability for data mining large data sets. This paper discusses issues in building a scalable classifier and presents the design of SLIQ, a new classifier. SLIQ is a decision tree classifier that can handle both numeric and categorical attributes. It uses a novel pre-sorting technique in the tree-growth phase. This sorting procedure is integrated with a breadth-first tree growing strategy to enable classification of disk-resident datasets. SLIQ also uses a new tree-pruning algorithm that is inexpensive, and results in compact and accurate trees. The combination of these techniques enables SLIQ to scale for large data sets and classify data sets irrespective of the number of classes, attributes, and examples (records), thus making it an attractive tool for data mining. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> This paper explores the use of Support Vector Machines (SVMs) for learning text classifiers from examples. It analyzes the particular properties of learning with text data and identifies why SVMs are appropriate for this task. Empirical results support the theoretical findings. SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of different learning tasks. Furthermore they are fully automatic, eliminating the need for manual parameter tuning. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Text Classification <s> The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vision. We present a new architecture (VDCNN) for text processing which operates directly at the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with depth: using up to 29 convolutional layers, we report improvements over the state-of-the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to text processing. <s> BIB007
Another classic application for natural language processing is text classification, or the assignment of free-text documents to predefined classes. Document classification has numerous applications. Many techniques have produced valuable results on this task-the Rocchio algorithm used in information retrieval  ], support vector machines BIB004 ], decision trees BIB003 , and the focus of this section, deep neural networks, among them.  was the first to use pretrained word vectors in a CNN for sentence-level classification tasks. Kim's work with classification was motivating, and showed that simple CNNs, with one convolutional layer followed by a softmax layer with dropout, could achieve excellent results on multiple benchmarks using little hyperparameter tuning. The CNN models proposed by Kim were able to improve upon the state of the art on 4 out of 7 different tasks cast as sentence classification, including sentiment analysis and question classification. BIB007 later showed that networks that employ a large number of convolutional neural networks work well for document classification. Instead of using various sized convolutions, their architecture used only convolutions with a window size of three. A novel approach proposed by used a hybrid architecture combining a deep belief network BIB006 and softmax regression BIB005 ]. A deep belief network is a feedforward network where pairs of hidden layers are designed to resemble the restricted Boltzmann machines (RBMs) proposed by BIB001 . Restricted Bolzmann Machines are trained using unsupervised learning and are designed to increase or decrease dimensionality of data. This is achieved by making passes over the data using forward and backward propagation many times until a minimum error is found over an energy-loss-based function. This process was independent of the labeled or classification portion of the task, and was therefore initially trained without the softmax regression output layer. Once both sections of the architecture were pretrained, they were then combined and trained like a regular deep neural net with backpropagation and quasi-Newton methods BIB002 ].