Although word analogy problems have become a standard tool for evaluating word vectors, little is known about why word vectors are so good at solving these problems. In this paper, I attempt to further our understanding of the subject, by developing a simple, but highly accurate generative approach to solve the word analogy problem for the case when all terms involved in the problem are nouns. My results demonstrate the ambiguities associated with learning the relationship between a word pair, and the role of the training dataset in determining the relationship which gets most highlighted. Furthermore, my results show that the ability of a model to accurately solve the word analogy problem may not be indicative of a model’s ability to learn the relationship between a word pair the way a human does.
 Word vectors constructed using Word2vec BID6 , BID8 ) and Glove BID9 ) are central to the success of several state of the art models in natural language processing BID1 , BID2 , BID7 , BID11 ). These vectors are low dimensional vector representations of words that accurately capture the semantic and syntactic information about the word in a document.The ability of these vectors to encode language is best illustrated by their efficiency at solving word analogy problems. The problem involves predicting a word, D, which completes analogies of the form 'A:B :: C:D'. For example, if the phrase is ''King:Queen :: Man:D', then the appropriate value of D is Woman. Word2vec solves these problems by observing that the word vectors for A, B, C and D satisfy the equation V ec(D) ≈ V ec(C) + V ec(B) − V ec(A) in several cases.Although this equation accurately resolves the word analogy for a wide variety of semantic and syntactic problems, the precise dynamics underlying this equation are largely unknown. Part of the difficulty in understanding the dynamics is that word vectors are essentially 'black boxes' which lack interpretability. This difficulty has been overcome in large part due to the systematic analyses of Levy, Goldberg and colleagues, who have derived connections between word vectors and the more human-interpretable count based approach of representing words. They show that 1) there are mathematical equivalences between Word2vec and the count based approach BID4 ), 2) that the count based approach can produce results comparable to Word2vec on word analogy problems BID3 ) and more generally, 3) that the count based approach can perform as well as Word2vec on most NLP tasks when the hyper-parameters in the model are properly tuned BID5 . Their results (see section 9 in BID3 ) demonstrate that V ec(B) − V ec(A) is likely capturing the 'common information' between A and B, and this information is somehow being 'transferred' on to C to compute D.Still the question remains, how is this transference process taking place? The answer would provide insight into the topology of word vectors and would help us to identify gaps in our understanding of word vectors. In this paper, I attempt to gain insights into the transference process by building a simple generative algorithm for solving semantic word analogy problems in the case where A, B, C and D are nouns. My algorithm works in two steps: In the first step, I compute a list of nouns that likely represent the information that is common to both A and B. In the second step, I impose the information about the nouns obtained in the first step on to C to compute D. Both steps of the algorithm work only on word counts; therefore, it is possible to precisely understand how and why D is generated in every word analogy question.Despite the simplicity of my approach, the algorithm is able to produce results comparable to Word2vec on the semantic word analogy questions, even using a very small dataset. My study reveals insights into why word vectors solve certain classes of word analogy problems much better than others. I show that there is no universal interpretation of the information contained in V ec(B) − V ec(A) because the 'common information' between A and B is strongly dependent on the training dataset. My results reveal that a machine may not be 'learning' the relationship between a pair of words the way a human does, even when it accurately solves an analogy problem.
Simple generative approach to solve the word analogy problem which yields insights into word relationships, and the problems with estimating them
{"src": "scitldr-AIC_train", "paper_id": "ryA-jdlA-"}