Computational Approaches to Measuring the Similarity of Short Contexts : A Review of Applications and Methods <s> Context as an Average of Word Vectors <s> A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents (“semantic structure”) in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. initial tests find this completely automatic method for retrieval to be promising. <s> BIB001 </s> Computational Approaches to Measuring the Similarity of Short Contexts : A Review of Applications and Methods <s> Context as an Average of Word Vectors <s> How do people know as much as they do with as little information as they get? The problem takes many forms; learning vocabulary from text is an especially dramatic and convenient case for research. A new general theory of acquired similarity and knowledge representation, latent semantic analysis (LSA), is presented and used to successfully simulate such learning and several other psycholinguistic phenomena. By inducing global knowledge indirectly from local co-occurrence data in a large body of representative text, LSA acquired knowledge about the full vocabulary of English at a comparable rate to schoolchildren. LSA uses no prior linguistic or perceptual similarity knowledge; it is based solely on a general mathematical learning method that achieves powerful inductive effects by extracting the right number of dimensions (e.g., 300) to represent objects and contexts. Relations to other theories, phenomena, and problems are sketched. <s> BIB002 </s> Computational Approaches to Measuring the Similarity of Short Contexts : A Review of Applications and Methods <s> Context as an Average of Word Vectors <s> This paper systematically compares unsupervised word sense discrimination techniques that cluster instances of a target word that occur in raw text using both vector and similarity spaces. The context of each instance is represented as a vector in a high dimensional feature space. Discrimination is achieved by clustering these context vectors directly in vector space and also by finding pairwise similarities among the vectors and then clustering in similarity space. We employ two different representations of the context in which a target word occurs. First order context vectors represent the context of each instance of a target word as a vector of features that occur in that context. Second order context vectors are an indirect representation of the context based on the average of vectors that represent the words that occur in the context. We evaluate the discriminated clusters by carrying out experiments using sense–tagged instances of 24 SENSEVAL2 words and the well known Line, Hard and Serve sense–tagged corpora. <s> BIB003 </s> Computational Approaches to Measuring the Similarity of Short Contexts : A Review of Applications and Methods <s> Context as an Average of Word Vectors <s> In this paper, we introduce a WordNetbased measure of semantic relatedness by combining the structure and content of WordNet with co–occurrence information derived from raw text. We use the co–occurrence information along with the WordNet definitions to build gloss vectors corresponding to each concept in WordNet. Numeric scores of relatedness are assigned to a pair of concepts by measuring the cosine of the angle between their respective gloss vectors. We show that this measure compares favorably to other measures with respect to human judgments of semantic relatedness, and that it performs well when used in a word sense disambiguation algorithm that relies on semantic relatedness. This measure is flexible in that it can make comparisons between any two concepts without regard to their part of speech. In addition, it can be adapted to different domains, since any plain text corpus can be used to derive the co–occurrence information. <s> BIB004
Short contexts normally consist of more than one word, so even after a word vector has been created based on the micro or macro view of context, there is still the question how to represent an entire short context, not just individual words. Since each of the words in a context is replaced with a vector, a natural solution is to represent the context with the centroid of the word vectors. Then, contexts can be measured for similarity using the cosine or any other vector based measure of similarity. The centroid is a vector that has the same dimensions as the word vectors, and is equally distant from all of them, and is therefore located in the middle of the word vectors. Mathematically this is accomplished by finding the average of the vectors, so the centroid is simply an averaged word vector for that context. Averaging word vectors to represent a context has been used with the macro view of context in Latent Semantic Indexing BIB001 ) and Latent Semantic Analysis BIB002 . Averaging word vectors based on the micro view of context has been pursued by  and BIB003 for carrying out word sense discrimination, and BIB004 for measuring semantic relatedness between concepts. Prior to creating the averaged vectors to represent the contexts, the micro view of context results in the construction of a word by word matrix, where the rows of this matrix represent the words in the contexts, and the columns represent the co-occurring words that may be obtained locally (from the short contexts themselves) or globally (from other resources like corpora or the Web). The values in the cells of this matrix may contain binary values that indicate if the words have co-occurred, frequency counts showing how often they have occurred together, or measures of association that show how dependent they are on each other. The macro view results in a word by context matrix, where the rows represent the words that occur in the context. The columns represent the contexts in which the words have occurred, and these may be obtained locally (from the contexts themselves) or globally (from some other resource outside of the contexts). The values in the matrix cells can be binary to indicate that a word occurred in a context, or they can include a frequency count that shows how many times a word occurred in a context. In both cases the resulting matrix will be very sparse, and it may be beneficial to perform dimensionality reduction on the either the word by word matrix (micro view) or the word by context matrix (macro view) to reduce the sparsity and dimensionality of the matrix, and smooth the values in the cells so that there are fewer zero valued cells. This will make it possible to draw finer grained distinctions between contexts when measuring their cosine. Dimensionality reduction is typically performed via Singular Value Decomposition (SVD), which has the effect (simply put) of trying to group together the columns in these matrices that are similar to each other, and reduce the dimensionality of the matrix by compressing similar columns together. SVD is an optional step however, and is not required to create the context representations from the averaged word vectors. The resulting word matrices are used to create the context representations, and this proceeds very simply. Each word in a context is replaced with a word vector (that has been optionally reduced via SVD). If there is no word vector available, then the word is dropped from the context. Once all the word are replaced, then the word vectors are averaged to find the centroid, and that centroid represents the context. This is done for every context of interest, so there is one vector per context. These can be clustered or measured for pair-wise similarity. Clustering and measuring the similarity of vectors are well studied areas, and many alternative methods exist. For short context problems the important point is not how the clustering or pair-wise measurement will be done, but rather how the short context representations can be created to allow for similarity judgments to be made even when there are few shared words between the contexts.