This paper studies how word embeddings trained on the British National Corpus interact with part of speech boundaries. Our work targets the Universal PoS tag set, which is currently actively being used for annotation of a range of languages. We experiment with training classifiers for predicting PoS tags for words based on their embeddings. The results show that the information about PoS affiliation contained in the distributional vectors allows us to discover groups of words with distributional patterns that differ from other words of the same part of speech. This data often reveals hidden inconsistencies of the annotation process or guidelines. At the same time, it supports the notion of soft' or graded' part of speech affiliations. Finally, we show that information about PoS is distributed among dozens of vector components, not limited to only one or two features.
Various types of distributional information has also played an important role in previous work done on the related problem of unsupervised PoS acquisition. As discussed in , we can separate at least three main directions within this line of work: approaches @cite_17 @cite_3 @cite_13 that start out from a dictionary providing possible tags for different words; approaches @cite_14 @cite_8 based on a small number of prototypical examples for each PoS; approaches that are completely unsupervised and make no use of prior knowledge. This is also the main focus of the comparative survey provided by @cite_8 .
@cite_14 @cite_8 @cite_3 @cite_13 @cite_17
1574126082 2109757083 1801581093 113586804
Despite significant recent work, purely unsupervised techniques for part-of-speech (POS) tagging have not achieved useful accuracies required by many language processing tasks. Use of parallel text between resource-rich and resource-poor languages is one source of weak supervision that significantly improves accuracy. However, parallel text is not always available and techniques for using it require multiple complex algorithmic steps. In this paper we show that we can build POS-taggers exceeding state-of-the-art bilingual methods by using simple hidden Markov models and a freely available and naturally growing resource, the Wiktionary. Across eight languages for which we have labeled data to evaluate results, we achieve accuracy that significantly exceeds best unsupervised and parallel text methods. We achieve highest accuracy reported for several languages and show that our approach yields better out-of-domain taggers than those trained using fully supervised Penn Treebank. In this paper, a method based on part-of-speech tagging (PoS) is used for bibliographic reference structure. This method operates on a roughly structured ASCII file, produced by OCR. Because of the heterogeneity of the reference structure, the method acts in a bottom-up way, without an a priori model, gathering structural elements from basic tags to sub-fields and fields. Significant tags are first grouped in homogeneous classes according to their grammar categories and then reduced in canonical forms corresponding to record fields: "authors", "title", "conference name", "date", etc. Non labelled tokens are integrated in one or another field by either applying PoS correction rules or using a structure model generated from well-detected records. The designed prototype operates with a great satisfaction on different record layouts and character recognition qualities. Without manual intervention, 96.6 words are correctly attributed, and about 75.9 references are completely segmented from 2500 references. Unsupervised word embeddings have been shown to be valuable as features in supervised learning problems; however, their role in unsupervised problems has been less thoroughly explored. In this paper, we show that embeddings can likewise add value to the problem of unsupervised POS induction. In two representative models of POS induction, we replace multinomial distributions over the vocabulary with multivariate Gaussian distributions over word embeddings and observe consistent improvements in eight languages. We also analyze the effect of various choices while inducing word embeddings on "downstream" POS induction results. Distributional approaches to meaning acquisition utilize distributional properties of linguistic entities as the building blocks of semantics. In doing so, they rely fundamentally on a set of assumptions about the nature of language and meaning referred to as the distributional hypothesis. This hypothesis is often stated in terms like “words which are similar in meaning occur in similar contexts” (Rubenstein & Goodenough, 1965); “words with similar meanings will occur with similar neighbors if enough text material is available” (Schutze & Pedersen, 1995); “a representation that captures much of how words are used in natural context will capture much of what we mean by meaning” (Landauer & Dumais, 1997); and “words that occur in the same contexts tend to have similar meanings” (Pantel, 2005), just to quote a few representative examples. The general idea behind the distributional hypothesis seems clear enough: there is a correlation between distributional similarity and meaning similarity, which allows us to utilize the former in order to estimate the latter. However, one can pose two very basic questions concerning the distributional hypothesis. The first is what kind of distributional properties we should look for, and what — if any — the differences are between different kinds of distributional properties. Looking at algorithms for distributional meaning acquisition we can discern two distinct approaches. The first is to build distributional profiles for words based on which other words surround them, as exemplified by Schutze (1992) and the Hyperspace Analogue to Language (HAL) model (Lund, Burgess, & Atchley, 1995). The second is to build distributional profiles based on in which text regions words occur, as exemplified by the Latent Semantic Analysis (LSA) model (Landauer & Dumais, 1997). These approaches are often treated as functionally equivalent when it comes to representing meaning similarities, despite the fact that they are based on different types of distributional raw materials. The second question is in what sense it is meaning that is conveyed by distributional patterns. Proponents of distributional methods often seem comfortable to ascribe meaning to distributional representations without explaining