A Survey on Context Learning <s> Speed-Up with a Scoring Mechanism <s> We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance. <s> BIB001 </s> A Survey on Context Learning <s> Speed-Up with a Scoring Mechanism <s> This paper describes how to automatically cross-reference documents with Wikipedia: the largest knowledge base ever known. It explains how machine learning can be used to identify significant terms within unstructured text, and enrich it with links to the appropriate Wikipedia articles. The resulting link detector and disambiguator performs very well, with recall and precision of almost 75%. This performance is constant whether the system is evaluated on Wikipedia articles or "real world" documents. This work has implications far beyond enriching documents with explanatory links. It can provide structured knowledge about any unstructured fragment of text. Any task that is currently addressed with bags of words - indexing, clustering, retrieval, and summarization to name a few - could use the techniques described here to draw on a vast network of concepts and semantics. <s> BIB002 </s> A Survey on Context Learning <s> Speed-Up with a Scoring Mechanism <s> We consider the task of estimating, from observed data, a probabilistic model that is parameterized by a finite number of parameters. In particular, we are considering the situation where the model probability density function is unnormalized. That is, the model is only specified up to the partition function. The partition function normalizes a model so that it integrates to one for any choice of the parameters. However, it is often impossible to obtain it in closed form. Gibbs distributions, Markov and multi-layer networks are examples of models where analytical normalization is often impossible. Maximum likelihood estimation can then not be used without resorting to numerical approximations which are often computationally expensive. We propose here a new objective function for the estimation of both normalized and unnormalized models. The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise. With this approach, the normalizing partition function can be estimated like any other parameter. We prove that the new estimation method leads to a consistent (convergent) estimator of the parameters. For large noise sample sizes, the new estimator is furthermore shown to behave like the maximum likelihood estimator. In the estimation of unnormalized models, there is a trade-off between statistical and computational performance. We show that the new method strikes a competitive trade-off in comparison to other estimation methods for unnormalized models. As an application to real data, we estimate novel two-layer models of natural image statistics with spline nonlinearities. <s> BIB003 </s> A Survey on Context Learning <s> Speed-Up with a Scoring Mechanism <s> In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. ::: ::: We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. ::: ::: We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset. <s> BIB004 </s> A Survey on Context Learning <s> Speed-Up with a Scoring Mechanism <s> The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. <s> BIB005
As we mentioned in the previous section, the jV j-sized output layer is the most time consuming part. Therefore, reducing the size of the output layer would be a good strategy to speed up the model. Ref. BIB001 proposed a deep neural network structure with an output layer of only size one. Rather than training a language model, this model focuses on learning a good representation of words and applies this representation to several natural language processing tasks, such as part-of-speech tagging, chunking, name entity recognition, semantic role labeling. With a special goal of only wanting the representation of words, the way to train this model is different. The model is trying to give the n words in each context window a score, scoreðw tÀnþ1 ; . . . ; w tÀ1 ; w t Þ, instead of calculating the conditional probability pðw t jw 1 ; w 2 ; . . . ; w tÀ1 Þ, where scoreðw tÀnþ1 ; . . . ; w tÀ1 ; w t Þ is the scoring function implemented by the neural network architecture. Thus this score is the only unit of the output layer. A higher score means the context makes more sense, while a context composed of random words should have a negative score. The goal is to discriminate the correct word from a random word. The model wants the score of the correct word to be larger by a margin of 1 than the score of any other random word in the vocabulary. Denoting scoreðw tÀnþ1 ; . . . ; w tÀ1 ; w t Þ by fðsÞ, therefore the training objective to minimize is X s2S X w2D maxð0; 1 À fðsÞ þ fðs w ÞÞ; BIB002 where S is the set of context windows, D is the vocabulary and s w is a context window where the current word is replaced by a random word w. Mostly, replacing an existing word in a context window with another random word will result in the new sentence not making any sense, i.e., fðs w Þ much lower than fðsÞ. Negative sampling BIB005 is a similar strategy that ranks the data above noise. Negative sampling is a simplified version of Noise Contrastive Estimation BIB003 , BIB004 . Rather than approximately maximizing the log probability of the softmax, negative sampling is only concerned with learning high-quality word vectors. Without loss in word vector qualities, this strategy is able to reduce the training time on large corpus from days, even weeks to one day.