We propose a new model for unsupervised POS tagging based on linguistic distinctions between open and closed-class items . Exploiting notions from current linguistic theory , the system uses far less information than previous systems , far simpler computational methods , and far sparser descriptions in learning contexts . By applying simple language acquisition techniques based on counting , the system is given the closed-class lexicon , acquires a large open-class lexicon and then acquires disambiguation rules for both . This system achieves a 20 % error reduction for POS tagging over state-of-the-art unsupervised systems tested under the same conditions , and achieves comparable accuracy when trained with much less prior information . All recent research on unsupervised tagging , as well as the majority of work on supervised taggers , views POS tagging as a sequential labeling problem and treats all POS tags , both closed-and open-class , as roughly equivalent . In this work we explore a different understanding of the tagging problem , viewing it as a process of first identifying functional syntactic contexts , which are flagged by closed-class items , and then using these functional contexts to determine the POS labels . This disambiguation model differs from most previous work in three ways : 1 ) it uses different encodings over two distinct domains ( roughly open-and closed-class words ) with complementary distribution ( and so decodes separately ) ; 2 ) it is deterministic and 3 ) it is non-lexicalized . By learning disambiguation models for open-and closed-classes separately , we found that the deterministic , rulebased model can be learned from unannotated data by a simple strategy of selecting a rule in each appropriate context with the highest count . In contrast to this , most previous work on unsupervised tagging ( especially for English ) concentrates on improving the parameter estimation techniques for training statistical disambiguation models from unannotated data . For example , ( Smith&Eisner , 2005 ) proposes contrastive estimation ( CE ) for log-linear models ( CRF ) , achieving the current state-of-the-art performance of 90.4 % ; ( Goldwater&Griffiths , 2007 ) applies a Bayesian approach to improve maximumlikelihood estimation ( MLE ) for training generative models ( HMM ) . In the main experiments of both of these papers , the disambiguation model is learned , but the algorithms assume a complete knowledge of the lexicon with all possible tags for each word . In this work , we propose making such a large lexicon unnecessary by learning the bulk of the lexicon along with learning a disambiguation model . Little previous work has been done on this natural and simple idea because the clusters found by previous induction schemes are not in line with the lexical categories that we care about . ( Chan , 2008 ) is perhaps the first with the intention of generating " a discrete set of clusters . " By applying similar techniques to ( Chan , 2008 ) , which we discuss later , we can generate clusters that closely approximate the central open-class lexical categories , a major advance , but we still require a closed-class lexicon specifying possible tags for these words . This asymmetry in our lexicon acquisition model conforms with our understanding of natural language as structured data over two distinct domains with complementary distribution : open-class ( lexical ) and closed-class ( functional ) . Provided with only a closed-class lexicon of 288 words , about 0.6 % of the full lexicon , the system acquires a large open-class lexicon and then acquires disambiguation rules for both closed-and open-class words , achieving a tagging accuracy of 90.6 % for a 24k dataset , as high as the current state-of-the-art ( 90.4 % ) achieved with a complete dictionary . In the test condition where both algorithms are provided with a full lexicon , and are trained and evaluated over the same 96k dataset , we reduce the tagging error by up to 20 % . In Section 2 we explain our understanding of the POS tagging problem in detail and define the notions of functional context and open-and closedclass elements . Then we will introduce our methods for acquiring the lexicon ( Section 3 ) and learning disambiguation models ( Section 4 , 5 and 6 ) step by step . Results are reported in Section 7 followed by Section 8 which discusses the linguistic motivation behind this work and the simplicity and efficiency of our model . In this work on unsupervised tagging , we combine lexicon acquisition with the learning of a POS disambiguation model . Moreover , the disambiguation model we used is deterministic , nonlexicalized and defined over two distinct domains with complementary distribution ( open-and closed-class ) . Building a lexicon based on induced clusters requires our morphological knowledge of three special endings in English : -ing , -ed and -s ; on the other hand , to reduce the feature space used for category induction , we utilize vectors of functional features only , exploiting our knowledge of the role of determiners and modal verbs . However , the above information is restricted to the lexicon acquisition model . Taking a lexicon as input , which either consists of a known closed-class lexicon together with an acquired open-class lexicon or is composed by automatic extraction from the Penn Treebank , we need NO language-specific knowledge for learning the disambiguation model . We would like to point the reader to ( Chan , 2008 ) for more discussion on Category induction14 ; and discussions below will concentrate on the proposed disambiguation model . Current Chomskian theory , developed in the Minimalist Program ( MP ) ( Chomsky , 2006 ) , argues ( very roughly speaking ) that the syntactic structure of a sentence is built around a scaffolding provided by a set of functional elements15 . Each of these provides a large tree fragment ( roughly corresponding to what Chomsky calls a phase ) that provide the piece parts for full utterances . Chomsky observes that when these fragments combine , only the very edge of the fragments can change and that the internal structure of these fragments is rigid ( he labels this observation the Phase Impenetrability Condition , PIC ) . With the belief in PIC , we propose the concept of functional context , in which category property can be determined ; also we notice the distinct distribution of the elements ( functional ) on the edge of phase and those ( lexical ) assembled within the phase . Instead of chasing the highest possible performance by using the strongest method possible , we wanted to explore how well a deterministic , nonlexicalized model , following certain linguistic intuitions , can approach the NLP problem . For the unsupervised tagging task , this simple model , with less than two hundred rules learned , even outperforms non-deterministic generative models with ten of thousands of parameters . Another motivation for our pursuit of this deterministic , non-lexicalized model is computational efficiency 16 . It takes less than 3 minutes total for our model to acquire the lexicon , learn the disambiguation model , tag raw data and evaluate the output for a 96k dataset on a small laptop17 . And a model using only counting and selecting is common in the research field of language acquisition and perhaps more compatible to the way humans process language . We are certainly aware that our work does not yet address two problems : 1 ) . How the system can be adapted to work for other languages and 2 ) How to automatically obtain the knowledge of functional elements . We believe that , given the proper understanding of functional elements , our system will be easily adapted to other languages , but we clearly need to test this hypothesis . Also , we are highly interested in completing our system by incorporating the acquisition of functional elements . ( Chan , 2008 ) presents an extensive discussion of his work on morphological induction and ( Mintz et al . , 2002 ) presents interesting psychological experiments we can build on to acquire closed-class words .
Current approaches tackle unsupervised POS tagging as a sequential labelling problem and require a complete knowledge of the lexicon.
They propose to first identify functional syntactic contexts and then use them to make predictions for POS tagging.
The proposed method achieves equivalent performance by using 0.6% of the lexical knowledge used in baseline models.