A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> Abstract Suggested by the structure of the visual nervous system, a new algorithm is proposed for pattern recognition. This algorithm can be realized with a multilayered network consisting of neuron-like cells. The network, “neocognitron”, is self-organized by unsupervised learning, and acquires the ability to recognize stimulus patterns according to the differences in their shapes: Any patterns which we human beings judge to be alike are also judged to be of the same category by the neocognitron. The neocognitron recognizes stimulus patterns correctly without being affected by shifts in position or even by considerable distortions in shape of the stimulus patterns. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> Sentiment analysis of short texts such as single sentences and Twitter messages is challenging because of the limited contextual information that they normally contain. Effectively solving this task requires strategies that combine the small text content with prior knowledge and use more than just bag-of-words. In this work we propose a new deep convolutional neural network that exploits from characterto sentence-level information to perform sentiment analysis of short texts. We apply our approach for two corpora of two different domains: the Stanford Sentiment Treebank (SSTb), which contains sentences from movie reviews; and the Stanford Twitter Sentiment corpus (STS), which contains Twitter messages. For the SSTb corpus, our approach achieves state-of-the-art results for single sentence sentiment prediction in both binary positive/negative classification, with 85.7% accuracy, and fine-grained classification, with 48.3% accuracy. For the STS corpus, our approach achieves a sentiment prediction accuracy of 86.4%. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural <s> The state-of-the-art methods used for relation classification are primarily based on statistical machine learning, and their performance strongly depends on the quality of the extracted features. The extracted features are often derived from the output of pre-existing natural language processing (NLP) systems, which leads to the propagation of the errors in the existing tools and hinders the performance of these systems. In this paper, we exploit a convolutional deep neural network (DNN) to extract lexical and sentence level features. Our method takes all of the word tokens as input without complicated pre-processing. First, the word tokens are transformed to vectors by looking up word embeddings 1 . Then, lexical level features are extracted according to the given nouns. Meanwhile, sentence level features are learned using a convolutional approach. These two level features are concatenated to form the final extracted feature vector. Finally, the features are fed into a softmax classifier to predict the relationship between two marked nouns. The experimental results demonstrate that our approach significantly outperforms the state-of-the-art methods. <s> BIB007
Networks. Some networks contain layers in which the nodes receive input from only some of the nodes in previous layers, and therefore are not fully connected. One such type of network is the convolutional neural network (CNN) BIB002 BIB003 ], built upon Fukashima's [1980; BIB001 neocognitron. Deriving their name from the convolution operation in mathematics and signal processing, convolutional neural networks use an assortment of different fully-enumerated functions, known as filters, to augment the data in varying ways, allowing for simultaneous analysis of different features in the data BIB004  . The structure is inspired by the biological neural networks (BNNs) found in the visual cortex  . A single neuron receives signals from a number of co-located rods and cones (visual receptors in the eye) in an area known as that neuron's receptive field. The neuron, in essence, performs a function on the received signals to identify if a particular low-level feature is present in its receptive field. Likewise, a number of other neurons with identical or similar receptive fields perform other operations on the stimuli to identify other features. Further neurons receive signals from these neurons as well as neurons with different receptive fields, and use this information to identify higher level features. Consequently, CNNs are used extensively in image and video processing. A simple CNN architecture is show in Fig. 2a . Due to some of the advantages they provide, they have also found use in a number of other fields, including speech and natural language processing BIB005 BIB006  BIB007 . Two of the largest advantages of convolutional networks are their ability to share weights and their ability to account for poorly processed data  . Since many nodes in a CNN are looking for the same features as other neurons in their layer, just in different receptive fields, it makes sense that they would use the same functions (i.e., the same weights for their inputs). Therefore, for example, instead of having to train one million neurons to identify one hundred different features in ten thousand different regions, one only has to train one hundred different neurons, which can then be used ten thousand times each. This reduces the scope of the problem immensely (in this example by a factor of ten thousand), and with it the training time. Another appealing feature of CNNs is their ability to account for data that may not be uniformly or systematically formatted. Since CNNs are capable of learning features that may be present in different regions of the input data, rather than relying on certain features being present in particular locations, less preprocessing is required. This allows CNNs to identify objects in images no matter where in the image the object occurs or what proportion of the frame is occupied by the object. Likewise, it allows linguistic features to be identified no matter where in a sample text they occur. This allows for processing of diverse forms of word morphology, sentence syntax, and other characteristics that may not always follow the same structure BIB006 . Often, it is not important precisely where certain features occur, but rather whether or not they appear in particular localities. Therefore, pooling operations, such as that show in Figure 2b , can be used to minimize the size of feature maps (the outputs of the convolutional filters). The most common pooling operation is k-max pooling, in which the k largest values within the pool are retained and all others discarded. The sizes of such pools are generally small in order to prevent the loss of too much precision. As pooling can significantly reduce the number of locations to be examined by the following layers, its use can dramatically decrease the number of parameters in a network, leading to substantial improvements in training times. Convolutional neural networks usually consist of multiple convolutional and pooling layers followed by several fully-connected layers.