A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> This paper describes ongoing work on a new approach for language modeling for large vocabulary continuous speech recognition. Almost all state.. o. f-the-art systems use statistical n-gram language models estimated on text corpora. One principle problem with such language models is the fact that many of the n-grams are never observed even in very large training corpora, and therefore it is common to back-off to a lower-order model. In this paper we propose to address this problem by carrying out the estimation task in a continuous space, enabling a smooth interpolation of the probabilities. A neural network is used to learn the projection of the words onto a continuous space and to estimate the n-gram probabilities. The connectionist language model is being evaluated on the DARPA HUB5 conversational telephone speech recognition task and preliminary results show consistent improvements in both perplexity and word error rate. <s> BIB001 </s> A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB002 </s> A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> During the last years there has been growing interest in using neural networks for language modeling. In contrast to the well known back-off n-gram language models, the neural network approach attempts to overcome the data sparseness problem by performing the estimation in a continuous space. This type of language model was mostly used for tasks for which only a very limited amount of in-domain training data is available.In this paper we present new algorithms to train a neural network language model on very large text corpora. This makes possible the use of the approach in domains where several hundreds of millions words of texts are available. The neural network language model is evaluated in a state-of-the-art real-time continuous speech recognizer for French Broadcast News. Word error reductions of 0.5% absolute are reported using only a very limited amount of additional processing time. <s> BIB003 </s> A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> This paper introduces a new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM. This model is able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which uses the continuous word representation induced by a NNLM. The GALE Mandarin data was used to carry out the speech-to-text experiments and evaluate the NNLMs. On this data the well tuned baseline system has a character error rate under 10%. Our model achieves consistent improvements over the combination of an n-gram model and classical short-list NNLMs both in terms of perplexity and recognition accuracy. <s> BIB004 </s> A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> In this paper, we propose a general framework to incorporate semantic knowledge into the popular data-driven learning process of word embeddings to improve the quality of them. Under this framework, we represent semantic knowledge as many ordinal ranking inequalities and formulate the learning of semantic word embeddings (SWE) as a constrained optimization problem, where the data-derived objective function is optimized subject to all ordinal knowledge inequality constraints extracted from available knowledge resources such as Thesaurus and WordNet. We have demonstrated that this constrained optimization problem can be efficiently solved by the stochastic gradient descent (SGD) algorithm, even for a large number of inequality constraints. Experimental results on four standard NLP tasks, including word similarity measure, sentence completion, name entity recognition, and the TOEFL synonym selection, have all demonstrated that the quality of learned word vectors can be significantly improved after semantic knowledge is incorporated as inequality constraints during the learning process of word embeddings. <s> BIB005 </s> A Survey on Context Learning <s> Explicit Analysis + Neural Network <s> Neural network methods have achieved promising results for sentiment classification of text. However, these models only use semantics of texts, while ignoring users who express the sentiment and products which are evaluated, both of which have great influences on interpreting the sentiment of text. In this paper, we address this issue by incorporating userand productlevel information into a neural network approach for document level sentiment classification. Users and products are modeled using vector space models, the representations of which capture important global clues such as individual preferences of users or overall qualities of products. Such global evidence in turn facilitates embedding learning procedure at document level, yielding better text representations. By combining evidence at user-, productand documentlevel in a unified neural framework, the proposed model achieves state-of-the-art performances on IMDB and Yelp datasets1. <s> BIB006
When dealing with natural language processing tasks, such as part-of-speech tagging and language modeling, the neural network based models outperform the stateof-the-art n-gram models and are able to overcome many drawbacks inherent in n-gram models (the curse of dimensionality, out-of-vocabulary words, etc.). However, on the other hand, as the most widely used language model, n-gram models are very effective and efficient despite those drawbacks. In order to handle this tradeoff between computational cost and model performance, an intuitive idea is to combine the neural probabilistic language model and the state-of-the-art n-gram model to achieve both model quality and efficiency. Ref. BIB001 proposes a combined model of neural network and n-gram to reduce the computation of NPLM (Section 4.2). The main idea is to use NPLM to predict only the "interesting words", which are referred to as a shortlist in this paper, and use a standard three-gram model for the rest of the words. The authors introduce two ways to define a shortlist: a static shortlist, which contains l ( jV j most frequent words in the vocabulary, and a dynamic shortlist, which contains l ( jV j words depending on the input context. With the utilization of the shortlist, this model is able to reduce the perplexity by up to 8.9 percent compared to the standard threegram model with a much faster speed than the NPLM. The size of the shortlist affects the trade-off between the computational cost and the model perplexity: the more words in the shortlist, the more reduction in the perplexity can be achieved, and the longer training time the model will need. BIB003 extends this combined model with several new techniques including lattice rescoring, regrouping, block model, CPU optimization and training data sub-sampling. Ref. BIB004 proposes a unified composite model of neural network and n-gram models based language model called Structured OUtput Layer (SOUL). The proposed model structures the output vocabulary based on word clustering and also adopts the shortlist to reduce the computation. Compared to a four-gram model, SOUL can achieve an improvement of up to 23 percent in terms of perplexity. Ref. BIB002 shows that combining the predictions of neural network and n-gram model may bring in a 24 percent improvement in terms of perplexity compared with individual models since these two models appear to make "errors" in very different ways. This paper introduces three strategies to combine the prediction probability of the NPLM with that of the tri-gram model: a fixed weight of 0.5, a learned weight based on the likelihood on the validation set and weights conditioned on the input context. In their experiment, the performance of the composite model doesn't change as the size of the hidden layer increases when the weight is fixed, and this implies these two models make errors in different ways. Besides improving efficiency, explicit analysis can also incorporate prior context knowledge into neural network based models. Ref. BIB005 proposes a general framework to incorporate ordinal ranking inequalities into neural network based language models. The ordinal ranking inequalities represent semantic prior knowledge extracted from Thesaurus and WordNet, for example, similarity(football, basketball) > similarity(football, art) is a prior inequality constraint on the corresponding word vectors. Ref. BIB006 shows how to utilize user and product context information when dealing with sentiment classification task of text. Not only the semantics of the text, but also who expressed the sentiment and which product is being evaluated have great influences on interpreting the sentiment of text. As neural network based models have achieved promising results for sentiment classification task, they adopt a convolutional neural network architecture as their foundation. By incorporating user and product level information into the neural network for document level sentiment classification, the proposed model outperforms the baselines by averagely 19 percent in terms of RMSE on the IMDB dataset.