A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> Abstract : As a result of this grant, the researchers have now published oil CDROM a corpus of over 4 million words of running text annotated with part-of- speech (POS) tags, with over 3 million words of that material assigned skeletal grammatical structure. This material now includes a fully hand-parsed version of the classic Brown corpus. About one half of the papers at the ACL Workshop on Using Large Text Corpora this past summer were based on the materials generated by this grant. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> Statistical n-gram language models are traditionally developed using perplexity as a measure of goodness. However, perplexity often demonstrates a poor correlation with recognition improvements, mainly because it fails to account for the acoustic confusability between words and for search errors in a recognizer. In this paper, we study alternatives to perplexity for predicting language model performance, including other global features as well as a new approach that predicts, with a high correlation (0.96), performance differences associated with localized changes in language models, given a recognition system. Experiments focus on the problem of augmenting in-domain Switchboard text with out-of-domain text from the Wall Street Journal and broadcast news that differ in both style and content from the in-domain data. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> The most widely-used evaluation metric for language models for speech recognition is the perplexity of test data. While perplexities can be calculated efficiently and without access to a speech recognizer, they often do not correlate well with speech recognition word-error rates. In this research, we attempt to find a measure that like perplexity is easily calculated but which better predicts speech recognition performance. We investigate two approaches; first, we attempt to extend perplexity by using similar measures that utilize information about language models that perplexity ignores. Second, we attempt to imitate the word-error calculation without using a speech recognizer by artificially generating speech recognition lattices. To test our new metrics, we have built over thirty varied language models. We find that perplexity correlates with word-error rate remarkably well when only considering -gram models trained on in-domain data. When considering other types of models, our novel metrics are superior to perplexity for predicting speech recognition performance. However, we conclude that none of these measures predict word-error rate sufficiently accurately to be effective tools for language model evaluation in speech recognition. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> This paper explores the interaction between a language model?s perplexity and its effect on the word error rate of a speech recognition system. Much recent research has indicated that these two measures are not as well correlated as was once thought, and many examples exist of models which have a much lower perplexity than the equivalent N -gram model, yet lead to no improvement in recognition accuracy. This paper investigates the reasons for this apparent discrepancy. Perplexity?s calculation is based solely on the probabilities of words contained within the test text; it disregards the probabilities of alternative words which will be competing with the correct word within the decoder. It is shown that by considering the probabilities of the alternative words it is possible to derive measures of language model quality which are better correlated with word error rate than perplexity is. Furthermore, optimizing language model parameters with respect to these new measures leads to a significant reduction in the word error rate. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Neural Language Modeling. <s> We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned KneserNey 5-gram model achieves perplexity 67.6. A combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models. <s> BIB006
A major problem with statistical language models was their inability to deal well with synonyms or out-of-vocabulary (OOV) words. While they could predict the re-occurrence of word sets that were present in the training corpus, they had difficulty predicting new sets of words; for example, a set of words from the training corpus in which one word was replaced with a synonym or a similar word with a different meaning (e.g. car and truck or red and blue). Progress was made in solving these problems with the introduction of the neural language model BIB005 ]. While much of NLP took another decade to begin to utilize ANNs heavily, the LM community immediately took advantage of them, and continued to develop sophisticated models, many of which were summarized by DeMulder et al. [2015] . Not only do they allow for the prediction of synonymous words, they also allow for modeling the relationships between words [Mikolov et al. 2013a,b] . The word vectors with numeric components, obtained by language modeling techniques are called embeddings. Typically, word embeddings have between 50 and 300 dimensions. An overused example is that of the distributed representations of the words king, queen, man, and woman. If one takes the embedding vectors for each of these words, mathematics can be performed to obtain highly sensible results. If the vectors representing these words are respectively represented as ì k, ì q, ì m, and ì w, it can be observed that ì k − ì q ≈ ì m − ì w, which is extremely intuitive to human reasoning. 3.1.3 Evaluation of Language Models. While neural networks have certainly made breakthroughs in the LM field, it is often rather hard to quantify improvements. In some applications such as speech recognition or machine translation, metrics such as word error rate (WER) can be used. (Word error rate is the fraction of actual output words not correctly matching the desired output words.) However, typically it is desirable to evaluate language models independently of the applications in which they appear. A number of metrics have been proposed, but no perfect solution has yet been found BIB003 BIB004 BIB002 . The most commonly used metric is perplexity, which is the inverse probability of a test set normalized by the number of words. Perplexity is a reasonable measurement for LMs trained on the same datasets, but when they are trained on different vocabularies, the metric becomes less meaningful. Luckily, there are several benchmark datasets that are used widely in the field, allowing for reasonable comparison. Two such datasets are the Penn Treebank (PTB) BIB001 , and more recently, in order to allow for comparison of large models that require extensive amounts of training data, the Billion Word Benchmark BIB006 ].