Short vowels and other diacritics are not part of written Arabic scripts . Exceptions are made for important political and religious texts and in scripts for beginning students of Arabic . Script without diacritics have considerable ambiguity because many words with different diacritic patterns appear identical in a diacritic-less setting . We propose in this paper a maximum entropy approach for restoring diacritics in a document . The approach can easily integrate and make effective use of diverse types of information ; the model we propose integrates a wide array of lexical , segmentbased and part-of-speech tag features . The combination of these feature types leads to a state-of-the-art diacritization model . Using a publicly available corpus ( LDC 's Arabic Treebank Part 3 ) , we achieve a diacritic error rate of 5.1 % , a segment error rate 8.5 % , and a word error rate of 17.3 % . In case-ending-less setting , we obtain a diacritic error rate of 2.2 % , a segment error rate 4.0 % , and a word error rate of 7.2 % . Modern Arabic written texts are composed of scripts without short vowels and other diacritic marks . This often leads to considerable ambiguity since several words that have different diacritic patterns may appear identical in a diacritic-less setting . Educated modern Arabic speakers are able to accurately restore diacritics in a document . This is based on the context and their knowledge of the grammar and the lexicon of Arabic . However , a text without diacritics becomes a source of confusion for beginning readers and people with learning disabilities . A text without diacritics is also problematic for applications such as text-to-speech or speech-to-text , where the lack of diacritics adds another layer of ambiguity when processing the data . As an example , full vocalization of text is required for text-to-speech applications , where the mapping from graphemes to phonemes is simple compared to languages such as English and French ; where there is , in most cases , one-to-one relationship . Also , using data with diacritics shows an improvement in the accuracy of speech-recognition applications ( Afify et al . , 2004 ) . Currently , text-tospeech , speech-to-text , and other applications use data where diacritics are placed manually , which is a tedious and time consuming excercise . A diacritization system that restores the diacritics of scripts , i.e. supply the full diacritical markings , would be of interest to these applications . It also would greatly benefit nonnative speakers , sufferers of dyslexia and could assist in restoring diacritics of children 's and poetry books , a task that is currently done manually . We propose in this paper a statistical approach that restores diacritics in a text document . The proposed approach is based on the maximum entropy framework where several diverse sources of information are employed . The model implicitly learns the correlation between these types of information and the output diacritics . In the next section , we present the set of diacritics to be restored and the ambiguity we face when processing a non-diacritized text . Section 3 gives a brief summary of previous related works . Section 4 presents our diacritization model ; we explain the training and decoding process as well as the different feature categories employed to restore the diacritics . Section 5 describes a clearly defined and replicable split of the LDC 's Arabic Treebank Part 3 corpus , used to built and evaluate the system , so that the reproduction of the results and future comparison can accurately be established . Section 6 presents the experimental results . Section 7 reports a comparison of our approach to the finite state machine modeling technique that showed promissing results in ( Nelken and Shieber , 2005 ) . Finally , section 8 concludes the paper and discusses future directions . We presented in this paper a statistical model for Arabic diacritic restoration . The approach we propose is based on the Maximum entropy framework , which gives the system the ability to integrate different sources of knowledge . Our model has the advantage of successfully combining diverse sources of information ranging from lexical , segment-based and POS features . Both POS and segment-based features are generated by separate statistical systems -not extracted manually -in order to simulate real world applications . The segment-based features are extracted from a statistical morphological analysis system using WFST approach and the POS features are generated by a parsing model Table 5 : Error Rate in % for n-gram diacritic restoration and segmentation using FST and Kneser-Ney LM . Columns marked with " True shadda " represent results on documents containing the original consonant doubling " shadda " while columns marked with " Predicted shadda " represent results where the system restored all diacritics including shadda . that also uses Maximum entropy framework . Evaluation results show that combining these sources of information lead to state-of-the-art performance . As future work , we plan to incorporate Buckwalter morphological analyzer information to extract new features that reduce the search space . One idea will be to reduce the search to the number of hypotheses , if any , proposed by the morphological analyzer . We also plan to investigate additional conjunction features to improve the accuracy of the model .
Short vowels and other diacritics are not expressed in written Arabic, making it difficult to read for beginner readers or system developments.
They propose an approach that uses maximum entropy to restore diacritics in Arabic documents by learning relations between a wide varieties of features.
They show that by taking various kinds of features their system outperforms the existing state-of-the-art decritization model.