Part of Speech Taggers for Morphologically Rich Indian Languages: A Survey <s> Experiments & Results <s> Five language and tagset independent stochastic taggers, handling morphological and contextual information, are presented and tested in corpora of seven European languages (Dutch, English, French, German, Greek, Italian and Spanish), using two sets of grammatical tags; a small set containing the eleven main grammatical classes and a large set of grammatical categories common to all languages. The unknown words are tagged using an experimentally proven stochastic hypothesis that links the stochastic behavior of the unknown words with that of the less probable known words. A fully automatic training and tagging program has been implemented on an IBM PC-compatible 80386-based computer. Measurements of error rate, time response, and memory requirements have shown that the taggers' performance is satisfactory, even though a small training text is available. The error rate is improved when new texts are used to update the stochastic model parameters. <s> BIB001 </s> Part of Speech Taggers for Morphologically Rich Indian Languages: A Survey <s> Experiments & Results <s> This paper describes our work on building Part-of-Speech (POS) tagger for Bengali. We have use Hidden Markov Model (HMM) and Maximum Entropy (ME) based stochastic taggers. Bengali is a morphologically rich language and our taggers make use of morphological and contextual information of the words. Since only a small labeled training set is available (45,000 words), simple stochastic approach does not yield very good results. In this work, we have studied the effect of using a morphological analyzer to improve the performance of the tagger. We find that the use of morphology helps improve the accuracy of the tagger especially when less amount of tagged corpora are available. <s> BIB002
A total of 12 models were considered under different stochastic tagging schemes. To estimate the parameters for all the models the same training text has been used. The model parameters for supervised HMM and ME models are estimated from the annotated text corpus. For semi-supervised learning, the HMM learned through supervised training is considered as the initial model. Further, a larger unlabelled training data has been used to re-estimate the model parameters of the semi-supervised HMM. The experiments were conducted with three different sizes (10K, 20K and 40K words) of the training data to understand the relative performance of the models as we keep on increasing the size of the annotated data. The training data consists of manually annotated 3625 sentences (approximately 40,000 words) for both supervised HMM and ME model. A fixed set of 11,000 unlabeled sentences (approximately 100,000 words) taken from CIIL (Central Institute of Indian Languages) corpus are used to re-estimate the model parameter during semi-supervised learning BIB002 . The corpus ambiguity (mean number of possible tags for each word) in the training text is 1.77 which is much larger compared to the European languages BIB001 A set of randomly drawn 400 sentences (5000 words) have been used for testing all models. Out of these 14% words in the open testing text are unknown with respect to the training set, which is also a little higher compared to the European languages BIB001 The results are obtained on the basis of final accuracies achieved by different models with the varying size of training data 