Dense word vectors have proven their values in many downstream NLP tasks over the past few years. However, the dimensions of such embeddings are not easily interpretable. Out of the d-dimensions in a word vector, we would not be able to understand what high or low values mean. Previous approaches addressing this issue have mainly focused on either training sparse/non-negative constrained word embeddings, or post-processing standard pre-trained word embeddings. On the other hand, we analyze conventional word embeddings trained with Singular Value Decomposition, and reveal similar interpretability. We use a novel eigenvector analysis method inspired from Random Matrix Theory and show that semantically coherent groups not only form in the row space, but also the column space. This allows us to view individual word vector dimensions as human-interpretable semantic features. Understanding words has a fundamental impact on many natural language processing tasks, and has been modeled with the Distributional Hypothesis BID0 . Dense d-dimensional vector representations of words created from this model are often referred to as word embeddings, and have successfully captured similarities between words, such as word2vec and GloVe BID1 BID2 . They have also been applied to downstream NLP tasks as word representation features, ranging from sentiment analysis to machine translation BID3 BID4 .Despite their widespread popularity in usage, the dimensions of these word vectors are difficult to interpret BID5 . Consider w president = [0.1, 2.4, 0.3] as the 3-dimensional vector of "president" from word2vec. In this 3-dimensional space (or the row space), semantically similar words like "minister" and "president" are closely located. However , it is unclear what the dimensions represent, as we do not know the meaning of the 2.4 in w president . It is difficult to answer questions like 'what is the meaning of high and low values in the columns of W' and 'how can we interpret the dimensions of word vectors'. To address this problem, previous literature focused on the column space by either training word embeddings with sparse and non-negative constraints BID6 BID7 BID8 , or post-processing pre-trained word embeddings BID5 BID9 BID10 . We instead investigate this problem from a random matrix perspective.In our work, we analyze the eigenvectors of word embeddings obtained with truncated Singular Value Decomposition (SVD) BID11 BID12 of the Positive Pointwise Mutual Information (PPMI) matrix BID13 . Moreover, we compare this analysis with the row and column space analysis of Skip Gram Negative Sampling (SGNS), a model used to train word2vec BID14 . From the works of BID15 proving that both SVD and SGNS factorizes and approximates the same matrix, we hypothesize that a study of the principal eigenvectors of the PPMI matrix reflects the information contained in SGNS.Contributions: Without requiring any constraints or post-processing, we show that the dimensions of word vectors can be interpreted as semantic features. In doing so, we also introduce novel word embedding analysis methods inspired by the literature of eigenvector analysis techniques from Random Matrix Theory. In this work, we analyzed the eigenvectors, or the column space, of the word embeddings obtained from the Singular Value Decomposition of PPMI matrix. We revealed that the significant participants of the eigenvectors form semantically coherent groups, allowing us to view each word vector as an interpretable feature vector composed of semantic groups. These results can be very useful in error analysis in downstream NLP tasks, or cherry-picking useful feature dimensions to easily create compressed and efficient task-specific embeddings. Future work will proceed in this direction on applying interpretability to practical usage.
Without requiring any constraints or post-processing, we show that the salient dimensions of word vectors can be interpreted as semantic features. 
{"src": "scitldr-AIC_train", "paper_id": "rJfJiR5ooX"}