A Survey on Context Learning <s> Utilizing Global Statistics <s> Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are built with only local context and one representation per word. This is problematic because words are often polysemous and global context can also provide useful information for learning word meanings. We present a new neural network architecture which 1) learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and 2) accounts for homonymy and polysemy by learning multiple embeddings per word. We introduce a new dataset with human judgments on pairs of words in sentential context, and evaluate our model on it, showing that our model outperforms competitive baselines and other neural language models. <s> BIB001 </s> A Survey on Context Learning <s> Utilizing Global Statistics <s> Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition. <s> BIB002 </s> A Survey on Context Learning <s> Utilizing Global Statistics <s> Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperforms bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks. <s> BIB003
Ref. BIB002 provides another way to utilize the global statistical information of the corpus by building their log-bilinear regression model on global word-word co-occurrence counts. The proposed model is called Global Vectors (GloVe), because compared with other neural network based language models, GloVe is directly based on the global corpus statistics. Let X denote the matrix of wordword co-occurrence counts and X i ¼ P k X ik be the total number of words appear in the context of word i. And P ðjjiÞ ¼ X ij =X i denotes the probability of word j appearing in the context of word i. The paper proposes that the relationship between two words can be examined by studying the ratio of their co-occurrence probabilities with other probe words, k. For example, if word k is related to word i but not j, the ratio P ðkjiÞ=P ðkjjÞ should be large. In contrast, if word k is related to word j but not i, then the ratio P ðkjiÞ=P ðkjjÞ should be small. Based on this ratio of cooccurrence probabilities rather than the probabilities themselves, the most general model takes the form where w i , w j are word vectors andw is from another set of word vectors called context word vectors. In order to capture the linear structure of the word vector space, vector difference and dot product are applied to the input word vectors, and F is appointed the exponential function. Thus after adding bias, a drastically simplified model is obtained Considering the co-occurrence matrix X is very sparse, the final objective function is a weighted least squares regression where fðX ij Þ is a non-decreasing weighting function that fð0Þ ¼ 0. The model generates two sets of word vectors, regular vectors W and context vectorsW . They use W þW as the final word vectors so as to reduce overfitting and noise. By using the global statistics, GloVe is able to outperform the previous neural network based language models by 6.4 on the word analogy task. Other than encoding the global statistics as a word-word co-occurrence matrix, BIB003 and BIB001 utilize the global context information by incorporating a global feature vector in the models. 