Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, relation extraction, and question answering. Supervised learning from labeled hypernym sources, such as WordNet, limit the coverage of these models, which can be addressed by learning hypernyms from unlabeled text.   Existing unsupervised methods either do not scale to large vocabularies or yield unacceptably poor accuracy.   This paper introduces {\it distributional inclusion vector embedding (DIVE)}, a simple-to-implement unsupervised method of hypernym discovery via per-word non-negative vector embeddings which preserve the inclusion property of word contexts. In experimental evaluations more comprehensive than any previous literature of which we are aware---evaluating on 11 datasets using multiple existing as well as newly proposed scoring functions---we find that our method provides up to double the precision of previous unsupervised methods, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art results. In addition, the meaning of each dimension in DIVE is interpretable, which leads to a novel approach on word sense disambiguation as another promising application of DIVE. Numerous applications benefit from compactly representing context distributions, which assign meaning to objects under the rubric of distributional semantics. In natural language processing, distributional semantics has long been used to assign meanings to words (that is, to lexemes in the dictionary, not individual instances of word tokens). The meaning of a word in the distributional sense is often taken to be the set of textual contexts (nearby tokens) in which that word appears, represented as a large sparse bag of words (SBOW). Without any supervision, word2vec BID22 , among other approaches based on matrix factorization BID20 , successfully compress the SBOW into a much lower dimensional embedding space, increasing the scalability and applicability of the embeddings while preserving (or even improving) the correlation of geometric embedding similarities with human word similarity judgments.While embedding models have achieved impressive results, context distributions capture more semantic features than just word similarity. The distributional inclusion hypothesis (DIH) BID49 BID11 BID6 posits that the context set of a word tends to be a subset of the contexts of its hypernyms. For a concrete example, most adjectives that can be applied to poodle can also be applied to dog, because dog is a hypernym of poodle. For instance, both can be obedient. However, the converse is not necessarily true -a dog can be straight-haired but a poodle cannot. Therefore, dog tends to have a broader context set than poodle. Many asymmetric scoring functions comparing SBOW based on DIH have been developed for automatic hypernymy detection BID49 BID11 BID38 .Hypernymy detection plays a key role in many challenging NLP tasks, such as textual entailment BID34 , coreference BID32 , relation extraction BID8 and question answering BID13 . Leveraging the variety of contexts and inclusion properties in context distributions can greatly increase the ability to discover taxonomic structure among words BID38 . The inability to preserve these features limits the semantic representation power and downstream applicability of some popular existing unsupervised learning approaches such as word2vec.Several recently proposed methods aim to encode hypernym relations between words in dense embeddings, such as Gaussian embedding BID45 BID0 , order embedding BID44 , H-feature detector BID33 , HyperScore (Nguyen et al., 2017) , dual tensor BID12 , Poincaré embedding BID28 , and LEAR BID46 . However, the methods focus on supervised or semi-supervised setting BID44 BID33 BID27 BID12 BID46 , do not learn from raw text BID28 or lack comprehensive experiments on the hypernym detection task BID45 BID0 .Recent studies BID21 BID38 have underscored the difficulty of generalizing supervised hypernymy annotations to unseen pairs -classifiers often effectively memorize prototypical hypernyms ('general' words) and ignore relations between words. These findings motivate us to develop more accurate and scalable unsupervised embeddings to detect hypernymy and propose several scoring functions to analyze the embeddings from different perspectives. We show the micro average AP@all on 10 datasets using different hypernymy scoring functions in TAB2 . We can see the combination functions such as C·∆S and W·∆S perform the best overall. Among the unnormalized inclusion based scoring functions, CDE works the best. AL 1 performs well compared with other functions which remove the frequency signal such as Word2vec, Cosine, and SLQS Row. The summation is the most robust generality measurement. In the In TAB4 , DIVE with two of the best scoring functions (C·∆S and W·∆S) is compared with the previous unsupervised state-of-the-art approaches based on SBOW on different datasets.There are several reasons which might cause the large performance gaps in some datasets. In addition to the effectiveness of DIVE, some improvements come from our proposed scoring functions. The fact that every paper uses a different training corpus also affects the performances. Furthermore, BID38 select the scoring functions and feature space for the first 4 datasets based on AP@100, which we believe is too sensitive to the hyper-parameter settings of different methods. To isolate the impact of each factor, we perform a more comprehensive comparison next. In TAB6 , we first confirm the finding of the previous review study of BID38 : there is no single hypernymy scoring function which always outperforms others. One of the main reasons is that different datasets collect negative samples differently. This is also why we evaluate our method on many datasets to make sure our conclusions hold in general. For example, if negative samples come from random word pairs (e.g. WordNet dataset), a symmetric similarity measure is already a pretty good scoring function. On the other hand, negative samples come from related or similar words in HyperNet, EVALution, Lenci/Benotto, and Weeds, so only computing generality difference leads to the best (or close to the best) performance. The negative samples in many datasets are composed of both random samples and similar words (such as BLESS), so the combination of similarity and generality difference yields the most stable results.DIVE performs similar or better on all the scoring functions compared with SBOW consistently across all datasets in TAB6 , while using many fewer dimensions (see TAB8 ). Its results on combination scoring functions outperform SBOW Freq. Meanwhile, its results on AL 1 outperform SBOW PPMI. The fact that combination scoring functions (i.e., W·∆S or C·∆S) usually outperform generality functions suggests that only memorizing general words is not sufficient. The best average performance on 4 and 10 datasets are both produced by W·∆S on DIVE.SBOW PPMI improves the combination functions from SBOW Freq but sacrifices AP on the inclusion functions. It generally hurts performance to change the frequency sampling of PPMI (PPMI w/ FW) or compute SBOW PPMI on the whole WaCkypedia (all wiki) instead of the first 51.2 million tokens. The similar trend can also be seen in TAB7 . Note that AL 1 completely fails in HyperLex dataset using SBOW PPMI, which suggests that PPMI might not necessarily preserve the distributional inclusion property, even though it can have good performance on combination functions.Removing the PMI filter from DIVE slightly drops the overall precision while removing frequency weights on shifted PMI (w/o FW) leads to poor performances. K-means (Freq NMF) produces similar AP compared with SBOW Freq, but has worse AL 1 scores. Its best AP scores on different datasets are also significantly worse than the best AP of DIVE. This means that only making word2vec (skip-grams with negative sampling) non-negative or naively accumulating topic distribution in contexts cannot lead to satisfactory embeddings. In addition to hypernymy detection, BID0 show that the mixture of Gaussian distributions can also be used to discover multiple senses of each word. In our qualitative experiment, we show that DIVE can achieve the similar goal without fixing the number of senses before training the embedding.Recall that each dimension roughly corresponds to one topic. Given a query word, the higher embedding value on a dimension implies higher likelihood to observe the word in the context of the topic. The embedding of a polysemy would have high values on different groups of topics/dimensions. This allows us to discover the senses by clustering the topics/dimensions of the polysemy. We use the embedding values as the feature each dimension, compute the pairwise similarity between dimensions, and apply spectral clustering BID41 to group topics as shown in the TAB9 . See more implementation details in the supplementary materials.In the word sense disambiguation tasks, it is usually challenging to determine how many senses/clusters each word should have. Many existing approaches fix the number of senses before training the embedding BID42 BID0 . BID26 make the number of clusters approximately proportional to the diversity of the context, but the assumption does not always hold. Furthermore, the training process cannot capture different granularity of senses. For instance, race in the car context could share the same sense with the race in the game topic because they all mean contest, but the race in the car context actually refers to the specific contest of speed. Therefore, they can also be viewed as separate senses (like the results in TAB9 ). This means the correct number of clusters is not unique, and the methods, which fixes the cluster numbers, need to re-train the embedding many times to capture such granularity.In our approach, clustering dimensions is done after the training process of DIVE is completed, so it is fairly efficient to change the cluster numbers and hierarchical clustering is also an option. Similar to our method, BID31 also discover word senses by graph-based clustering. The main difference is that they cluster the top n words which are most related to the query word instead of topics. However, choosing the hyper-parameter n is difficult. Large n would make graph clustering algorithm inefficient, while small n would make less frequent senses difficult to discover. Compressing unsupervised SBOW models into a compact representation is challenging while preserving the inclusion, generality, and similarity signals which are important for hypernym detection. Our experiments suggest that simple baselines such as accumulating K-mean clusters and non-negative skip-grams do not lead to satisfactory performances in this task.To achieve this goal, we proposed an interpretable and scalable embedding method called distributional inclusion vector embedding (DIVE) by performing non-negative matrix factorization (NMF) on a weighted PMI matrix. We demonstrate that scoring functions which measure inclusion and generality properties in SBOW can also be applied to DIVE to detect hypernymy, and DIVE performs the best on average, slightly better than SBOW while using many fewer dimensions.Our experiments also indicate that unsupervised scoring functions, which combine similarity and generality measurements, work the best in general, but no one scoring function dominates across all datasets. A combination of unsupervised DIVE with the proposed scoring functions produces new state-of-the-art performances on many datasets under the unsupervised setup.Finally, a qualitative experiment shows that clusters of the topics discovered by DIVE often correspond to the word senses, which allow us to do word sense disambiguation without the need to know the number of senses before training the embeddings. In addition to the unsupervised approach, we also compare DIVE with semi-supervised approaches. When there are sufficient training data, there is no doubt that the semi-supervised embedding approaches such as HyperNet BID40 , H-feature detector BID33 , and HyperScore (Nguyen et al., 2017) can achieve better performance than all unsupervised methods. However, in many domains such as scientific literature, there are often not many annotated hypernymy pairs (e.g. Medical dataset ).Since we are comparing an unsupervised method with semi-supervised methods, it is hard to fairly control the experimental setups and tune the hyper-parameters. In TAB10 , we only show several performances which are copied from the original paper when training data are limited 3 . As we can see, the performance from DIVE is roughly comparable to the previous semi-supervised approaches trained on small amount of hypernym pairs. This demonstrates the robustness of our approach and the difficulty of generalizing hypernymy annotations with semi-supervised approaches.
We propose a novel unsupervised word embedding which preserves the inclusion property in the context distribution and achieve state-of-the-art results on unsupervised hypernymy detection
{"src": "scitldr-AIC_train", "paper_id": "SywMS6ZfM"}