A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> This chapter contains sections titled: The Problem, The Generalized Delta Rule, Simulation Results, Some Further Generalizations, Conclusion <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Time underlies many interesting human behaviors. Thus, the question of how to represent time in connectionist models is very important. One approach is to represent time implicitly by its effects on processing rather than explicitly (as in a spatial representation). The current report develops a proposal along these lines first described by Jordan (1986) which involves the use of recurrent links in order to provide networks with a dynamic memory. In this approach, hidden unit patterns are fed back to themselves; the internal representations which develop thus reflect task demands in the context of prior internal states. A set of simulations is reported which range from relatively simple problems (temporal version of XOR) to discovering syntactic/semantic features for words. The networks are able to learn interesting internal representations which incorporate task demands with memory demands; indeed, in this approach the notion of memory is inextricably bound up with task processing. These representations reveal a rich structure, which allows them to be highly context-dependent while also expressing generalizations across classes of items. These representations suggest a method for representing lexical categories and the type/token distinction. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to “divide and conquer” by recursively decomposing sequences. I describe two architectures. The first functions as a self-organizing multilevel hierarchy of recurrent networks. The second, involving only two recurrent networks, tries to collapse a multilevel predictor hierarchy into a single recurrent net. Experiments show that the system can require less computation per time step and many fewer training sequences than conventional training algorithms for recurrent nets. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. > <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> We have already shown that extracting long-term dependencies from sequential data is difficult, both for determimstic dynamical systems such as recurrent networks, and probabilistic models such as hidden Markov models (HMMs) or input/output hidden Markov models (IOHMMs). In practice, to avoid this problem, researchers have used domain specific a-priori knowledge to give meaning to the hidden or state variables representing past context. In this paper, we propose to use a more general type of a-priori knowledge, namely that the temporal dependencies are structured hierarchically. This implies that long-term dependencies are represented by variables with a long time scale. This principle is applied to a recurrent network which includes delays and multiple time scales. Experiments confirm the advantages of such structures. A similar approach is proposed for HMMs and IOHMMs. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Keyword-based search engines are in widespread use today as a popular means for Web-based information retrieval. Although such systems seem deceptively simple, a considerable amount of skill is required in order to satisfy non-trivial information needs. This paper presents a new conceptual paradigm for performing search in context, that largely automates the search process, providing even non-professional users with highly relevant results. This paradigm is implemented in practice in the IntelliZap system, where search is initiated from a text query marked by the user in a document she views, and is guided by the text surrounding the marked query in that document (“the context”). The context-driven information retrieval process involves semantic keyword extraction and clustering to automatically generate new, augmented queries. The latter are submitted to a host of general and domain-specific search engines. Search results are then semantically reranked, using context. Experimental results testify that using context to guide search, effectively offers even inexperienced users an advanced search tool on the Web. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Restricted Boltzmann machines were developed using binary stochastic hidden units. These can be generalized by replacing each binary unit by an infinite number of copies that all have the same weights but have progressively more negative biases. The learning and inference rules for these "Stepped Sigmoid Units" are unchanged. They can be approximated efficiently by noisy, rectified linear units. Compared with binary units, these units learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset. Unlike binary units, rectified linear units preserve information about relative intensities as information travels through multiple layers of feature detectors. <s> BIB009 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> We describe how to effectively train neural network based language models on large data sets. Fast convergence during training and better overall performance is observed when the training data are sorted by their relevance. We introduce hash-based implementation of a maximum entropy model, that can be trained as a part of the neural network model. This leads to significant reduction of computational complexity. We achieved around 10% relative reduction of word error rate on English Broadcast News speech recognition task, against large 4-gram model trained on 400M tokens. <s> BIB010 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> We present several modifications of the original recurrent neural network language model (RNN LM).While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one. <s> BIB011 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM. <s> BIB012 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks. <s> BIB013 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Several variants of the long short-term memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful functional ANalysis Of VAriance framework. In total, we summarize the results of 5400 experimental runs ( $\approx 15$ years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment. <s> BIB014 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Summarization based on text extraction is inherently limited, but generation-style abstractive methods have proven challenging to build. In this work, we propose a fully data-driven approach to abstractive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary conditioned on the input sentence. While the model is structurally simple, it can easily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines. <s> BIB015 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. <s> BIB016 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Recurrent Neural Networks. <s> Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at this https URL . <s> BIB017
A simple type of recursive neural network that is used heavily in NLP is the recurrent neural network (RNN) BIB002 BIB007 ]. An RNN, in its simplest form, has an edge on a node, usually in a hidden layer, that feeds back to itself, as seen in Figure 3a . To demonstrate easily the creation of a chain like structure, which allows the network to remember the previous input in ordered sequences of input, an RNN can be visualized in an unrolled manner If an entire data sequence is known prior to computation beginning, RNNs can also operate over the reverse sequence of the data. Bidirectional networks such as (e) utilize two RNNs working in opposite directions and then combine the outputs. [ Medsker and Jain 2001] , such as shown in Figure 3c . Since much of NLP is dependent on the order of words or other elements such as phonemes or sentences, it is extremely useful to have memory of the previous elements when processing new ones BIB010 BIB008 BIB011 . Sometimes, backwards dependencies exist, i.e., correct processing of some words may depend on words that follow. These dependencies may be better captured if sentences are analyzed in the backwards direction. Thus, it may be beneficial to look at sentences in both directions, forwards and backwards, using two RNN cells, and combine their outputs. This arrangement of RNNs is called a bidirectional RNN and is shown in Figure 3e . Since an RNN's memory of data early in the chain gradually decays, later outputs may be erroneous if they rely heavily on early information. Examining chains in both directions counteracts this, as the RNNs focus separately on early data and late data. Note that all information must be present prior to computation in order to view it in reverse, preventing the use of bidirectional RNNs in some live applications such as speech processing programs. It has also been noticed that sometimes it leads to a better final representation if instead of having one RNN cell, there is a sequence of RNN cells, one after the other. This may allow the effect of an input to linger longer than a single RNN cell, allowing for longer-term effects. This setup of sequential RNN cells is called an RNN stack BIB005 BIB003 . Of course, there can be more than two stacked RNN cells as well, as shown in Figures 3b and 3d. 2.2.4 Long Short-Term Memory Networks. The internal workings of nodes in RvNNs (RNNs included) may vary. Single neurons may be used or complex networks may be used. One highly engineered RNN is the long short-term memory (LSTM) network BIB014 BIB006 . In LSTMs, the recursive nodes are composed of several individual neurons (or in some variants small ANNs) connected in a manner designed to retain specific information. Whereas generic RNNs with single neurons feeding back to themselves technically have some memory of long passed results, these results are diluted with each successive iteration. Furthermore, each of the elements of the results are remembered equally. Oftentimes, it is important to remember fully information from the distant past, while at the same time, other very recent information may be of zero importance. By using LSTM blocks in RNNs to form LSTM networks, this important information can be retained almost indefinitely while irrelevant information can be forgotten. However, if information is disregarded, it cannot be recovered, even if its presence is desired later. Current research looks to correct this problem, possibly by retaining larger amounts of data and Fig. 4 . Long Short-Term Memory Network. In this network, the input is concatenated with the previous output and is then distributed to an activation node and to three sigmoid nodes followed by gate nodes. Depending on the outputs of the sigmoid nodes, the gate nodes choose whether or not to allow information to pass through. These gates control i) the input, ii) the recurrent internal state, and iii) the output. simply altering which parts of it particular examples are attentive to . In another line of work, a slightly simpler variant of the LSTM, called the Gated Recurrent Unit (GRU), has been shown to perform as well as or better than standard LSTMs in many NLP tasks BIB012 . 2.2.5 Attention Mechanisms, Residual Connections, and Dropout. A number of other forms of artificial neural networks exist BIB013 ], however, those listed above are the most commonly used in current natural language processing research. Furthermore, many neural networks, including some mentioned in the discussions that follow, contain features from multiple types of networks. In order to understand some of these networks, one must understand some additional components and methods used in neural networks. One such component is the attention mechanism. For many tasks such as machine translation, text summarization, or image or video captioning, the output is in textual form. (This differs from many tasks in which text is input and a classification decision must be made.) Typically, this is done through the use of encoder-decoder pairs. An encoding ANN is used to produce a vector of a particular length and a decoding ANN is used to return text based on this vector. Since the vector must be of this specified length in order to be fed into the decoder, the encoder must also generate a vector of this length. While RNNs produce an output at each time step, the number of time steps is variable. However, ideally speaking, the final output will account for all inputs. This output will be a fixed size, and therefore can be passed into the decoder. The problem with this scheme, which is shown in Figure 5a , is that the RNN is forced to encode an entire sequence to a finite length vector, without regards to whether or not any of the inputs are more important than others. A robust solution to this is that of attention. The first noted use of an attention mechanism was that of . This approach used a dense layer for annotated weighting of an RNN's hidden state, allowing the network to learn what to pay attention to in accordance with the current hidden state and annotation. Such a mechanism is present in Figure 5b . Many variants of the mechanism have been introduced, popular ones including convolutional BIB015 , intra-temporal [Paulus et al. 2017] , gated  , and self-attention BIB016 . Deep neural networks are usually trained via backpropagation BIB001 . This method computes the error at the output nodes, and makes updates to weights on all edges connecting to them from lower level nodes. The backward propagation of error and correction of edge weights happens layer by layer in a similar manner, all the way back to the input layer. In increasingly deep networks, the gradients used to correct for error often vanish or explode by the time early layers are reached. BIB004 ]. This can be somewhat mitigated by choosing activation functions, such as the Rectified Linear Unit (ReLU) BIB009 , which do not exhibit regions that are arêtically steep or have bosonically small gradients. Also in response to this issue, as well as others , residual connections are often used. Such connections are simply those that skip layers (usually one). If used in every alternating layer, this cuts in half the number of layers through which the gradient must backpropagate. Such a network is known as a residual network (ResNet). A number of variants exist, including Highway Networks [Srivastava et al. 2015] and DenseNets BIB017 . Another important method used in training ANNs is dropout. In dropout, some nodes are deactivated, usually randomly, for each training batch (small set of examples), varying which nodes are deactivated each batch. This forces the network to distribute its memory across multiple paths, helping with generalization and lessening the likelihood of overfitting to the training data. For simplicity in training, this is usually implemented via dropout layers, which are simply single feedforward layers (generally fully connected) in which all the dropped nodes are contained. Dropout rates between 0 and 1 designate the proportion of nodes dropped during each training batch.