A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> We propose a novel deep network structure called "Network In Network" (NIN) to enhance model discriminability for local patches within the receptive field. The conventional convolutional layer uses linear filters followed by a nonlinear activation function to scan the input. Instead, we build micro neural networks with more complex structures to abstract the data within the receptive field. We instantiate the micro neural network with a multilayer perceptron, which is a potent function approximator. The feature maps are obtained by sliding the micro networks over the input in a similar manner as CNN; they are then fed into the next layer. Deep NIN can be implemented by stacking mutiple of the above described structure. With enhanced local modeling via the micro network, we are able to utilize global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers. We demonstrated the state-of-the-art classification performances with NIN on CIFAR-10 and CIFAR-100, and reasonable performances on SVHN and MNIST datasets. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned KneserNey 5-gram model achieves perplexity 67.6. A combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, computational costs and memory. Our analysis shows that despite being more costly to train, RNNLMs obtain much lower perplexities on standard benchmarks than n-gram models. We train the largest known RNNs and present relative word error rates gains of 18% on an ASR task. We also present the new lowest perplexities on the recently released billion word language modelling benchmark, 1 BLEU point gain on machine translation and a 17% relative hit rate gain in word prediction. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Convolutional Neural Networks in Language Modeling. <s> In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon. <s> BIB008
A convolutional neural network that was used recently in language modeling was one in which the pooling layers were replaced with fully-connected layers  . These layers allowed the feature maps to be reduced to lower dimensional spaces just as the pooling layers usually do. However, whereas any references to location of such features are lost in pooling layers, fully-connected layers somewhat retain this information. Several variations of CNNs were tested, all incorporating this characteristic. In total, three different architectures were implemented: a multilayer perceptron CNN (MLPConv) in which the filters were not simply linear, but instead small MLPs BIB001 ]; a multilayer CNN (ML-CNN) in which multiple convolutional layers are stacked on top of each other; and a combination of these networks called COM, in which kernel sizes for filters varied (in this case they were three and five). All of the networks were tested on three different datasets: the Penn Treebank, Europarl-NC [Bojar et al. [n. d.] ], and ukWaC  ]. The results of this study showed that stacking convolutional layers was actually detrimental in language modeling, but that both MLPConv and COM were effective at reducing the perplexity on each of the datasets tested. Furthermore, combining MLPConv with the varying kernel sizes of COM provided even better results, achieving a perplexity of ninety-one on the Penn Treebank. Analysis was performed by the researchers, showing that the networks learned specific patterns of words, such as, "as . . . as". Lastly, this study showed that CNNs can be used to capture long term dependencies in sentences. Closer words were found to be of greatest importance, but words located farther away were found to be of some significance as well. Words as far as sixteen positions away were considered. 3.1.6 Character Aware Neural Language Models. While most CNNs used in NLP receive words (or word embeddings) as input, some recent networks have analyzed character level input instead. One such network is that of BIB005 , which unlike previous networks that accepted character level input BIB003 , accepted only such input, rather than combining it with word embeddings. A CNN was used to process the character level input in order to provide representations of the words. In a similar manner as word embeddings usually are, these representations were then fed into an encoder-decoder pair composed of a highway network (a gated network resembling an LSTM) [Srivastava et al. 2015] and an LSTM. They trained the network on the English Penn Treebank, as well as on datasets for Czech, German, Spanish, French, Russian, and Arabic. The Arabic data was from the News-Commentary corpus and all of the other data was from the 2013 ACL Workshop on Machine Translation (both the small and large datasets). For every non-English language except Russian, the network outperformed previously published results BIB003 in both the large and small datasets. For Russian, better results were achieved on the small dataset, but not the large. On the Penn Treebank, results were produced on par with the existing state of the art BIB004 , achieving a perplexity of 78.9 (compared to 78.4). However, the network had only 19 million trainable parameters, which is considerably lower than others, and was handicapped by the number of words marked as unknown in the Penn Treebank. Since the network focused on morphological similarities produced by character level analysis, it was more capable than previous models of handling words that were rarely seen in the corpus. Analysis showed that without the use of highway layers, many words had nearest neighbors that were orthographically similar (spelled similarly), but not necessarily semantically similar. The addition of highway layers (which were used in the highest performing network) seemed to solve this problem. Additionally, the network was capable of recognizing misspelled words or words not spelled in the standard way (e.g. looooook instead of look) and of recognizing out of vocabulary words. The analysis also showed that the network was capable of identifying prefixes, roots, and suffixes, as well as understanding hyphenated words, making it an extremely robust model. BIB008 tested a number of architectures including some mentioned above, and a few others producing character level outputs BIB002 BIB006  BIB007 . Whereas many of these models had only been tested on small scale language modeling, this study tested them on large scale language modeling, testing them with the Billion Word Benchmark. The most effective model, achieving a state-of-the-art (for single models) perplexity of 30.0 with 1.04 billion trainable parameters (compared to a previous best by a single model of 51.3 with 20 billion parameters BIB002 ), was a large LSTM using a character level CNN as an input network. The best performance, however, was achieved using an ensemble (group of parallel networks whose outputs are averaged) of ten LSTMs. The perplexity of this ensemble, 23.7, far surpassed the previous state-of-the-art ensemble  , which had a perplexity of 41.0.