Chinese input methods are used to convert pinyin sequence or other Latin encoding systems into Chinese character sentences. For more effective pinyin-to-character conversion, typical Input Method Engines (IMEs) rely on a predefined vocabulary that demands manually maintenance on schedule. For the purpose of removing the inconvenient vocabulary setting, this work focuses on automatic wordhood acquisition by fully considering that Chinese inputting is a free human-computer interaction procedure. Instead of strictly defining words, a loose word likelihood is introduced for measuring how likely a character sequence can be a user-recognized word with respect to using IME. Then an online algorithm is proposed to adjust the word likelihood or generate new words by comparing user true choice for inputting and the algorithm prediction. The experimental results show that the proposed solution can agilely adapt to diverse typings and demonstrate performance approaching highly-optimized IME with fixed vocabulary.
Early work on IMEs include trigram statistical language model @cite_0 and joint source-channel model @cite_27 that is applicable to Chinese, Japanese and Korean languages. Recent studies pay more attention on IME input typo correction @cite_1 @cite_11 along with various approaches on Chinese spell checking @cite_2 @cite_24 @cite_21 . There are also stuides on Janpanese IME which focus on kana-kanji conversion, including probabilistic language model based method @cite_16 , discriminative method @cite_10 @cite_20 @cite_9 , improved @math -pos model @cite_12 , and ensemble method @cite_23 . However, all the above are based on predefined vocabulary and did not exploit the interactive property of IME inputting. @cite_14 and @cite_26 tried to address unknown word problem, but they still require an initial vocabulary or a segmented training corpus.
@cite_14 @cite_26 @cite_9 @cite_21 @cite_1 @cite_0 @cite_24 @cite_27 @cite_23 @cite_2 @cite_16 @cite_10 @cite_12 @cite_20 @cite_11
2251238909 2252213328 2251256737 2406728828
Since Japanese and Chinese languages have too many characters to be input directly using a standard keyboard, input methods for these languages that enable users to input the characters are required. Recently, input methods based on statistical models have become popular because of their accuracy and ease of maintenance. Most of them adopt word-based models because they utilize word-segmented corpora to train the models. However, such word-based models suffer from unknown words because they cannot convert words correctly which are not in corpora. To handle this problem, we propose a character-based model that enables input methods to convert unknown words by exploiting character-aligned corpora automatically generated by a monotonic alignment tool. In addition to the character-based model, we propose an ensemble model of both character-based and word-based models to achieve higher accuracy. The ensemble model combines these two models by linear interpolation. All of these models are based on joint source channel model to utilize rich context through higher order joint n-gram. Experiments on Japanese and Chinese datasets showed that the character-based model performs reasonably and the ensemble model outperforms the word-based baseline model. As a future work, the effectiveness of incorporating large raw data should be investigated. Kana-Kanji conversion is known as one of the representative applications of Natural Language Processing (NLP) for the Japanese language. The N-pos model, presenting the probability of a Kanji candidate sequence by the product of bi-gram Part-of-Speech (POS) probabilities and POS-to-word emission probabilities, has been successfully applied in a number of well-known Japanese Input Method Editor (IME) systems. However, since N-pos model is an approximation of n-gram word-based language model, important word-to-word collocation information are lost during this compression and lead to a drop of the conversion accuracies. In order to overcome this problem, we propose ways to improve current N-pos model. One way is to append the highfrequency collocations and the other way is to sub-categorize the huge POS sets to make them more representative. Experiments on large-scale data verified our proposals. It is very import for Chinese language processing with the aid of an efficient input method engine (IME), of which pinyinto-Chinese (PTC) conversion is the core part. Meanwhile, though typos are inevitable during user pinyin inputting, existing IMEs paid little attention to such big inconvenience. In this paper, motivated by a key equivalence of two decoding algorithms, we propose a joint graph model to globally optimize PTC and typo correction for IME. The evaluation results show that the proposed method outperforms both existing academic and commercial IMEs. This paper describes our system designed for the NLPCC 2015 shared task on Chinese word segmentation WS and POS tagging for Weibo Text. We treat WS and POS tagging as two separate tasks and use a cascaded approach. Our major focus is how to effectively exploit multiple heterogeneous data to boost performance of statistical models. This work considers three sets of heterogeneous data, i.e., Weibo @math , 10K sentences, Penn Chinese Treebank 7.0 @math , 50K, and People's Daily @math , 280K. For WS, we adopt the recently proposed coupled sequence labeling to combine @math , @math , and @math , boosting F1 score from @math baseline model trained on only @math to @math @math . For POS tagging, we adopt an ensemble approach combining coupled sequence labeling and the guide-feature based method, since the three datasets have three different annotation standards. First, we convert @math into the annotation style of @math based on coupled sequence labeling, denoted by @math . Then, we merge CTB7 and @math to train a POS tagger, denoted by @math , which is further used to produce guide features on @math . Finally, the tagging F1 score is improved from 87.93 to 88.99 +1.06 .