In many applications of natural language processing (NLP) it is necessary to determine the likelihood of a given word combination. For example, a speech recognizer may need to determine which of the two word combinations eat a peach'' and eat a beach'' is more likely. Statistical NLP methods determine the likelihood of a word combination from its frequency in a training corpus. However, the nature of language is such that many word combinations are infrequent and do not occur in any given corpus. In this work we propose a method for estimating the probability of such previously unseen word combinations using available information on most similar'' words. We describe probabilistic word association models based on distributional word similarity, and apply them to two tasks, language modeling and pseudo-word disambiguation. In the language modeling task, a similarity-based model is used to improve probability estimates for unseen bigrams in a back-off language model. The similarity-based method yields a 20 perplexity improvement in the prediction of unseen bigrams and statistically significant reductions in speech-recognition error. We also compare four similarity-based estimation methods against back-off and maximum-likelihood estimation methods on a pseudo-word sense disambiguation task in which we controlled for both unigram and bigram frequency to avoid giving too much weight to easy-to-disambiguate high-frequency configurations. The similarity-based methods perform up to 40 better on this particular task.
An early attempt to automatically classify words into semantic classes was carried out in the Linguistic String Project @cite_28 . Semantic classes were derived from similar cooccurrence patterns of words within syntactic relations. Cooccurrence statistics were then considered at the class level and used to alleviate data sparseness in syntactic disambiguation.
@cite_28
53773398 69597389 2144108169 2145328028
Abstract Automatically extracting semantic information about word mean-ing and document topic from text typically involves an extensivenumber of classes. Such classes may represent prede ned wordsenses, topics or document categories and are often organized in ataxonomy. Thelatterencodesimportantinformation,whichshouldbe exploited in learning classi ers from labeled training data. Tothat extent, this paper presents an extension of multiclass SupportVector Machine learning which can incorporate prior knowledgeabout class relationships. The latter can be encoded in the form ofclass attributes, similarities between classes or even a kernel func-tion de ned over the set of classes. The paper also discusses howto specify and optimize meaningful loss functions based on the rel-ative position of classes in the taxonomy. We include experimentalresults for text categorization and for word sense classi cation. 1 Introduction Manyreal-worldclassi cationtasksaremulticlassproblemsinvolvinglargenumbersof classes. This is in particular true for application domains like information re-trieval and natural language processing, where classes may correspond to documentcategories or word senses: several thousand or even tens of thousands of classes arenot uncommon. For instance, the International Patent Classi cation (IPC) scheme[8] consists of approximately 69,000 classes (called groups) that are used to catego-rize patent documents and WordNet 2.0 [3] consists of almost 80,000 word senses(called synsets) de ned by lexicographers to classify the meaning of English nouns.Multiclass problems of this scale pose a severe challenge for learning algorithms andclassi cation accuracies obtained by even the best classi cation methods are oftendisappointingly poor. Systems that automatically discover semantic classes have emerged in part to address the limitations of broad-coverage lexical resources such as WordNet and Cyc. The current state of the art discovers many semantic classes but fails to label their concepts. We propose an algorithm labeling semantic classes and for leveraging them to extract is-a relationships using a top-down approach. We propose a novel algorithm for inducing semantic taxonomies. Previous algorithms for taxonomy induction have typically focused on independent classifiers for discovering new single relationships based on hand-constructed or automatically discovered textual patterns. By contrast, our algorithm flexibly incorporates evidence from multiple classifiers over heterogenous relationships to optimize the entire structure of the taxonomy, using knowledge of a word's coordinate terms to help in determining its hypernyms, and vice versa. We apply our algorithm on the problem of sense-disambiguated noun hyponym acquisition, where we combine the predictions of hypernym and coordinate term classifiers with the knowledge in a preexisting semantic taxonomy (WordNet 2.1). We add 10,000 novel synsets to WordNet 2.1 at 84 precision, a relative error reduction of 70 over a non-joint algorithm using the same component classifiers. Finally, we show that a taxonomy built using our algorithm shows a 23 relative F-score improvement over WordNet 2.1 on an independent testset of hypernym pairs. We present a novel approach to weakly supervised semantic class learning from the web, using a single powerful hyponym pattern combined with graph structures, which capture two properties associated with pattern-based extractions: popularity and productivity. Intuitively, a candidate is popular if it was discovered many times by other instances in the hyponym pattern. A candidate is productive if it frequently leads to the discovery of other instances. Together, these two measures capture not only frequency of occurrence, but also cross-checking that the candidate occurs both near the class name and near other class members. We developed two algorithms that begin with just a class name and one seed instance and then automatically generate a ranked list of new class instances. We conducted experiments on four semantic classes and consistently achieved high accuracies.