Part-of-speech ( POS ) tag distributions are known to exhibit sparsity -a word is likely to take a single predominant tag in a corpus . Recent research has demonstrated that incorporating this sparsity constraint improves tagging accuracy . However , in existing systems , this expansion come with a steep increase in model complexity . This paper proposes a simple and effective tagging method that directly models tag sparsity and other distributional properties of valid POS tag assignments . In addition , this formulation results in a dramatic reduction in the number of model parameters thereby , enabling unusually rapid training . Our experiments consistently demonstrate that this model architecture yields substantial performance gains over more complex tagging counterparts . On several languages , we report performance exceeding that of more complex state-of-the art systems . 1 Since the early days of statistical NLP , researchers have observed that a part-of-speech tag distribution exhibits " one tag per discourse " sparsity -words are likely to select a single predominant tag in a corpus , even when several tags are possible . Simply assigning to each word its most frequent associated tag in a corpus achieves 94.6 % accuracy on the WSJ portion of the Penn Treebank . This distributional sparsity of syntactic tags is not unique to English 1 The source code for the work presented in this paper is available at http://groups.csail.mit.edu / rbg / code / typetagging/. -similar results have been observed across multiple languages . Clearly , explicitly modeling such a powerful constraint on tagging assignment has a potential to significantly improve the accuracy of an unsupervised part-of-speech tagger learned without a tagging dictionary . In practice , this sparsity constraint is difficult to incorporate in a traditional POS induction system ( Mérialdo , 1994 ; Johnson , 2007 ; Gao and Johnson , 2008 ; Grac ¸a et al . , 2009 ; Berg-Kirkpatrick et al . , 2010 ) . These sequence models-based approaches commonly treat token-level tag assignment as the primary latent variable . By design , they readily capture regularities at the token-level . However , these approaches are ill-equipped to directly represent type-based constraints such as sparsity . Previous work has attempted to incorporate such constraints into token-level models via heavy-handed modifications to inference procedure and objective function ( e.g. , posterior regularization and ILP decoding ) ( Grac ¸a et al . , 2009 ; Ravi and Knight , 2009 ) . In most cases , however , these expansions come with a steep increase in model complexity , with respect to training procedure and inference time . In this work , we take a more direct approach and treat a word type and its allowed POS tags as a primary element of the model . The model starts by generating a tag assignment for each word type in a vocabulary , assuming one tag per word . Then , tokenlevel HMM emission parameters are drawn conditioned on these assignments such that each word is only allowed probability mass on a single assigned tag . In this way we restrict the parameterization of a token-level HMM to reflect lexicon sparsity . This model admits a simple Gibbs sampling algorithm where the number of latent variables is proportional to the number of word types , rather than the size of a corpus as for a standard HMM sampler ( Johnson , 2007 ) . There are two key benefits of this model architecture . First , it directly encodes linguistic intuitions about POS tag assignments : the model structure reflects the one-tag-per-word property , and a typelevel tag prior captures the skew on tag assignments ( e.g. , there are fewer unique determiners than unique nouns ) . Second , the reduced number of hidden variables and parameters dramatically speeds up learning and inference . We evaluate our model on seven languages exhibiting substantial syntactic variation . On several languages , we report performance exceeding that of state-of-the art systems . Our analysis identifies three key factors driving our performance gain : 1 ) selecting a model structure which directly encodes tag sparsity , 2 ) a type-level prior on tag assignments , and 3 ) a straightforward naïve-Bayes approach to incorporate features . The observed performance gains , coupled with the simplicity of model implementation , makes it a compelling alternative to existing more complex counterparts . We have presented a method for unsupervised partof-speech tagging that considers a word type and its allowed POS tags as a primary element of the model . This departure from the traditional token-based tagging approach allows us to explicitly capture typelevel distributional properties of valid POS tag as-signments as part of the model . The resulting model is compact , efficiently learnable and linguistically expressive . Our empirical results demonstrate that the type-based tagger rivals state-of-the-art tag-level taggers which employ more sophisticated learning mechanisms to exploit similar constraints . In this paper , we make a simplifying assumption of one-tag-per-word . This assumption , however , is not inherent to type-based tagging models . A promising direction for future work is to explicitly model a distribution over tags for each word type . We hypothesize that modeling morphological information will greatly constrain the set of possible tags , thereby further refining the representation of the tag lexicon .
Assuming there is only one tag for a word is a powerful heuristic for Part-of-speech tagging but incorporating this into a model leads to complexity.
They propose an unsupervised method that directly incorporates a one-tag-per-word assumption into a HMM-based model.
Their proposed method reduces the number of model parameters which results in faster training speed and also outperforms more complex systems.