This specification relates to name detection, specifically name detection for Chinese, Japanese, and Korean (“CJK”) languages.
Name detection is typically used in natural language processing, for example, automatic speech recognition (ASR), machine translation (MT), optical character recognition (OCR), sentence parsing, non-Roman character input method editor (IME), and web search applications.
Naïve Bayesian classification methods can be used to detect if a sequence of characters “X” identifies a name, depending on the ratio of the probability of “X” identifying a name given its context (e.g., characters occurring before or after “X”) and the probability of “X” not identifying a name given its context. Language models are used to compute these conditional probabilities. A typical statistical language model is a probability measurement of a word or a sequence of characters given its history (e.g., the occurrence of previous word or character sequences in a collection of data). In particular, a conventional n-gram language model based on a Markov assumption, is used to predict a word or a sequence of characters.
A n-gram is a sequence of n consecutive tokens, e.g. words or characters. A n-gram has an order, which is the number of tokens in the n-gram. For example, a 1-gram (or unigram) includes one token; a 2-gram (or bi-gram) includes two tokens.
A given n-gram can be described according to different portions of the n-gram. A n-gram can be described as a context and a future token (context, c), where the context has a length n−1 and c represents the future token. For example, the 3-gram “x y z” can be described in terms of a n-gram context and a future token. The n-gram context includes all tokens of the n-gram preceding the last token of the n-gram. In the given example, “x y” is the context. The left most token in the context is referred to as the left token. The future token is the last token of the n-gram, which in the example is “z”. The n-gram can also be described with respect to a right context and a backed off context. The right context includes all tokens of the n-gram following the first token of the n-gram, represented as a (n−1)-gram. In the example above, “y z” is the right context. Additionally, the backed off context is the context of the n-gram less the left most token in the context. In the example above, “y” is the backed off context.
Each n-gram has an associated probability estimate that is calculated as a function of n-gram relative frequency in training data. For example, a string of L tokens is represented as C1L=(c1, c2, . . . , cL). A probability can be assigned to the string C1L as:
            P      ⁡              (                  c          1          L                )              =                            ∏                      i            =            1                    L                ⁢                  P          ⁡                      (                                          c                i                            ❘                              c                1                                  i                  -                  1                                                      )                              ≈                        ∏                      i            =            1                    L                ⁢                              P            ^                    ⁡                      (                                          c                i                            ❘                              c                                  i                  -                  n                  +                  1                                                  i                  -                  1                                                      )                                ,where the approximation is based on a Markov assumption that only the most recent (n−1) tokens are relevant when predicting a next token in the string, and the “^” notation for P indicates that it is an approximation of the probability function.
In CJK languages, sentences do not have word boundaries. As a result, sentences need to be segmented automatically before the detection of people's names. Therefore, segmentation errors will be propagated to name detection.
CJK names have morphologic laws that can be obtained from large statistics. For example, 300 common Chinese family names cover 99% or more of the population. Female names often contain characters such as  (na, hong, bing, li). Usually, common given names are independent of family names. For example, if statistics are available for a combination of the family name  and a given name  a combination of another family name  and the given name  identifying a name can be predicted using the statistics of  identifying a family name and the statistics of  identifying a given name. Furthermore, some words in Chinese can either be a person's name or a regular word, e.g.,  can be either the name of a famous singer in China, or a common word meaning daybreak. The detection of such a name largely depends on the context.
In addition, CJK names are generally identified using 2-grams (bigrams) or 3-grams (trigrams). Assuming a horizontal convention of reading CJK text from left to the right, the left most character in the context is a family name. The right context is a given name. For example, if “x y z” is a CJK name, then “x” is a family name and “y z” is a given name. As a further example, if “x y” is a CJK name, then “x” is a family name and “y” is a given name.