Computer Science
ComputerScience,1970-2008-Computer Science-2004.jsonl.gz.gz
Computer Science
0bba1958ecd4a5cc5f716e91ede4cbe0eec77a4e
Mining New Word Translations from Comparable Corpora

New words such as names, technical terms, etc appear frequently. As such, the bilingual lexicon of a machine translation system has to be constantly updated with these new word translations. Comparable corpora such as news documents of the same period from different news agencies are readily available. In this paper, we present a new approach to mining new word translations from comparable corpora, by using context information to complement transliteration information. We evaluated our approach on six months of Chinese and English Gigaword corpora, with encouraging results.


Introduction
New words such as person names, organization names, technical terms, etc. appear frequently. In order for a machine translation system to translate these new words correctly, its bilingual lexicon needs to be constantly updated with new word translations.
Much research has been done on using parallel corpora to learn bilingual lexicons (Melamed, 1997;Moore, 2003). But parallel corpora are scarce resources, especially for uncommon language pairs. Comparable corpora refer to texts that are not direct translation but are about the same topic. For example, various news agencies report major world events in different languages, and such news documents form a readily available source of comparable corpora. Being more readily available, comparable corpora are thus more suitable than parallel corpora for the task of acquiring new word translations, although relatively less research has been done in the past on comparable corpora. Previous research efforts on acquiring translations from comparable corpora include (Fung and Yee, 1998;Rapp, 1995;Rapp, 1999).
When translating a word w, two sources of information can be used to determine its translation: the word w itself and the surrounding words in the neighborhood (i.e., the context) of w. Most previous research only considers one of the two sources of information, but not both. For example, the work of (Al-Onaizan and Knight, 2002a;Al-Onaizan and Knight, 2002b;Knight and Graehl, 1998) used the pronunciation of w in translation. On the other hand, the work of (Cao and Li, 2002;Fung and Yee, 1998;Koehn and Knight, 2002;Rapp, 1995;Rapp, 1999) used the context of w to locate its translation in a second language.
In this paper, we propose a new approach for the task of mining new word translations from comparable corpora, by combining both context and transliteration information. Since both sources of information are complementary, the accuracy of our combined approach is better than the accuracy of using just context or transliteration information alone. We fully implemented our method and tested it on Chinese-English comparable corpora. We translated Chinese words into English. That is, Chinese is the source language and English is the target language. We achieved encouraging results.
While we have only tested our method on Chinese-English comparable corpora, our method is general and applicable to other language pairs.

Our approach
The work of (Fung and Yee, 1998;Rapp, 1995;Rapp, 1999) noted that if an English word e is the translation of a Chinese word c , then the contexts of the two words are similar. We could view this as a document retrieval problem. The context (i.e., the surrounding words) of c is viewed as a query. The context of each candidate translation ' e is viewed as a document. Since the context of the correct translation e is similar to the context of c , we are likely to retrieve the context of e when we use the context of c as the query and try to retrieve the most similar document. We employ the language modeling approach (Ng, 2000;Ponte and Croft, 1998) for this retrieval problem. More details are given in Section 3.
On the other hand, when we only look at the word w itself, we can rely on the pronunciation of w to locate its translation. We use a variant of the machine transliteration method proposed by (Knight and Graehl, 1998). More details are given in Section 4.
Each of the two individual methods provides a ranked list of candidate words, associating with each candidate a score estimated by the particular method. If a word e in English is indeed the translation of a word c in Chinese, then we would expect e to be ranked very high in both The candidate e i that is ranked the highest according to the average rank is taken to be the correct translation and is output. If no words appear within the top M positions in both lists, then no translation is output.
Since we are using comparable corpora, it is possible that the translation of a new word does not exist in the target corpus. In particular, our experiment was conducted on comparable corpora that are not very closely related and as such, most of the Chinese words have no translations in the English target corpus.

Translation by context
In a typical information retrieval (IR) problem, a query is given and a ranked list of documents most relevant to the query is returned from a document collection.
For our task, the query is Within IR, there is a new approach to document retrieval called the language modeling approach (Ponte & Croft,98). In this approach, a language model is derived from each document D . Then the probability of generating the query Q according to that language model, ) is the one that best matches the query. The language modeling approach to IR has been shown to give superior retrieval performance (Ponte & Croft, 98;Ng, 2000), compared with traditional vector space model, and we adopt this approach in our current work.
To estimate ) | ( D Q P , we use the approach of (Ng, 2000). We view the document D as a multinomial distribution of terms and assume that query Q is generated by this model: where t is a term in the corpus, t c is the number of times term t occurs in the query Q , is the total number of terms in query Q .
For ranking purpose, the first fraction can be omitted as this part depends on the query only and thus is the same for all the documents. In our translation problem, ) (c C is viewed as the query and ) (e C is viewed as a document. So our task is to compute English word e and find the e that gives the is the bag of Chinese words obtained by translating the English words in ) (e C , as determined by a bilingual dictionary. If an English word is ambiguous and has K translated Chinese words listed in the bilingual dictionary, then each of the K translated Chinese words is counted as occurring 1/K times in )) ( ( e C T c for the purpose of probability estimation.
We use backoff and linear interpolation for probability estimation: , and ) ( c ml t P is estimated similarly by counting the occurrences of c t in the Chinese translation of the whole English corpus. α is set to 0.6 in our experiments.

Translation by transliteration
For the transliteration model, we use a modified model of (Knight and Graehl, 1998) and (Al-Onaizan and Knight, 2002b). Knight and Graehl (1998) proposed a probabilistic model for machine transliteration. In this model, a word in the target language (i.e., English in our task) is written and pronounced. This pronunciation is converted to source language pronunciation and then to source language word (i.e., Chinese in our task). Al-Onaizan and Knight (2002b) suggested that pronunciation can be skipped and the target language letters can be mapped directly to source language letters.
Pinyin is the standard Romanization system of Chinese characters. It is phonetic-based. For transliteration, we estimate ) | ( c e P as follows: First, each Chinese character in a Chinese word c is converted to pinyin form. Then we sum over all the alignments that this pinyin form of c can map to an English word e. For each possible alignment, we calculate the probability by taking the product of each mapping. i p is the ith syllable of pinyin, a i l is the English letter sequence that the ith pinyin syllable maps to in the particular alignment a. Since most Chinese characters have only one pronunciation and hence one pinyin form, we assume that Chinese character-to-pinyin mapping is one-to-one to simplify the problem. We use the expectation maximization (EM) algorithm to generate mapping probabilities from pinyin syllables to English letter sequences. To reduce the search space, we limit the number of English letters that each pinyin syllable can map to as 0, 1, or 2. Also we do not allow cross mappings. That is, if an English letter sequence 1 e precedes another English letter sequence 2 e in an English word, then the pinyin syllable mapped to 1 e must precede the pinyin syllable mapped to 2 e .
Our method differs from (Knight and Graehl, 1998) and (Al-Onaizan and Knight, 2002b)  We used a Chinese-English dictionary which contained about 10,000 entries for translating the words in the context. For the training of transliteration probability, we required a Chinese-English name list. We used a list of 1,580 Chinese-English name pairs as training data for the EM algorithm.

Preprocessing
Unlike English, Chinese text is composed of Chinese characters with no demarcation for words. So we first segmented Chinese text with a Chinese word segmenter that was based on maximum entropy modeling (Ng and Low, 2004). We then divided the Chinese corpus from Jul to Dec 1995 into 12 periods, each containing text from a half-month period. Then we determined the new Chinese words in each half-month period p. By new Chinese words, we refer to those words that appeared in this period p but not from Jan to Jun 1995 or any other periods that preceded p. Among all these new words, we selected those occurring at least 5 times. These words made up our test set. We call these words Chinese source words. They were the words that we were supposed to find translations from the English corpus.
For the English corpus, we performed sentence segmentation and converted each word to its morphological root form and to lower case.
We also divided the English corpus into 12 periods, each containing text from a half-month period. For each period, we selected those English words occurring at least 10 times and were not present in the 10,000-word Chinese-English dictionary we used and were not stop words. We considered these English words as potential translations of the Chinese source words. We call them English translation candidate words. For a Chinese source word occurring within a halfmonth period p, we looked for its English translation candidate words occurring in news documents in the same period p.

Translation candidates
The context ) (c C of a Chinese word c was collected as follows: For each occurrence of c, we set a window of size 50 characters centered at c. We discarded all the Chinese words in the context that were not in the dictionary we used. The contexts of all occurrences of a word c were then concatenated together to form ) (c C . The context of an English translation candidate word e, ) (e C , was similarly collected. The window size of English context was 100 words.
After all the counts were collected, we estimated )) ( | ) ( ( e C c C P as described in Section 3, for each pair of Chinese source word and English translation candidate word. For each Chinese source word, we ranked all its English translation candidate words according to the estimated For each Chinese source word c and an English translation candidate word e , we also calculated the probability ) | ( c e P (as described in Section 4), which was used to rank the English candidate words based on transliteration.
Finally, the English candidate word with the smallest average rank position and that appears within the top M positions of both ranked lists is the chosen English translation (as described in Section 2). If no words appear within the top M positions in both ranked lists, then no translation is output.
Note that for many Chinese words, only one In Table 1, period 1 is Jul 01 -Jul 15, period 2 is Jul 16 -Jul 31, …, period 12 is Dec 16 -Dec 31. #c is the total number of new Chinese source words in the period. #e is the total number of English translation candidates in the period. #o is the total number of output English translations. #Cor is the number of correct English translations output. Prec. is the precision. The correctness of the English translations was manually checked.
Recall is somewhat difficult to estimate because we do not know whether the English translation of a Chinese word appears in the English part of the corpus. We attempted to estimate recall by manually finding the English translations for all the Chinese source words for the two periods Dec 01 -Dec 15 and Dec 16 -Dec 31 in the English part of the corpus. During the whole December period, we only managed to find English translations which were present in the English side of the comparable corpora for 43 Chinese words. = .
We also investigated the effect of varying M .
The results are shown in Table 2.

M
The past research of (Fung and Yee, 1998;Rapp, 1995;Rapp, 1999) utilized context information alone and was evaluated on different corpora from ours, so it is difficult to directly compare our current results with theirs. Similarly, Al-Onaizan and Knight (2002a;2002b) only made use of transliteration information alone and so was not directly comparable.
To investigate the effect of the two individual sources of information (context and transliteration), we checked how many translations could be found using only one source of information (i.e., context alone or transliteration alone), on those Chinese words that have translations in the English part of the comparable corpus. As mentioned earlier, for the month of Dec 1995, there are altogether 43 Chinese words that have their translations in the English part of the corpus. This list of 43 words is shown in Table 3. 8 of the 43 words are translated to English multi-word phrases (denoted as "phrase" in Table 3). Since our method currently only considers unigram English words, we are not able to find translations for these words. But it is not difficult to extend our method to handle this problem. We can first use a named entity recognizer and noun phrase chunker to extract English names and noun phrases.
The translations of 6 of the 43 words are words in the dictionary (denoted as "comm." in Table 3) and 4 of the 43 words appear less than 10 times in the English part of the corpus (denoted as "insuff"). Our method is not able to find these translations. But this is due to search space pruning. If we are willing to spend more time on searching, then in principle we can find these translations. 'NA' means the word cannot be transliterated. 'insuff' means the correct translation appears less than 10 times in the English part of the comparable corpus. 'comm' means the correct translation is a word appearing in the dictionary we used or is a stop word. 'phrase' means the correct translation contains multiple English words.
As shown in Table 3, using just context information alone, 10 Chinese words (the first 10) have their correct English translations at rank one position. And using just transliteration information alone, 9 Chinese words have their correct English translations at rank one position.
On the other hand, using our method of combining both sources of information and setting M = ∞, 19 Chinese words (i.e., the first 22 Chinese words in Table 3 except 巴佐亚,坩埚,普利法) have their correct English translations at rank one position. If M = 10, 15 Chinese words (i.e., the first 19 Chinese words in Table 3 except 叶玛斯,巴佐亚,坩埚,普利法) have their correct English translations at rank one position. Hence, our method of using both sources of information outperforms using either information source alone.

Related work
As pointed out earlier, most previous research only considers either transliteration or context information in determining the translation of a source language word w, but not both sources of information. For example, the work of (Al-Onaizan and Knight, 2002a;Al-Onaizan and Knight, 2002b;Knight and Graehl, 1998) used only the pronunciation or spelling of w in translation. On the other hand, the work of (Cao and Li, 2002;Fung and Yee, 1998;Rapp, 1995;Rapp, 1999) used only the context of w to locate its translation in a second language. In contrast, our current work attempts to combine both complementary sources of information, yielding higher accuracy than using either source of information alone. Koehn and Knight (2002) attempted to combine multiple clues, including similar context and spelling. But their similar spelling clue uses the longest common subsequence ratio and works only for cognates (words with a very similar spelling).
The work that is most similar to ours is the recent research of (Huang et al., 2004). They attempted to improve named entity translation by combining phonetic and semantic information. Their contextual semantic similarity model is different from our language modeling approach to measuring context similarity. It also made use of part-of-speech tag information, whereas our method is simpler and does not require part-ofspeech tagging. They combined the two sources of information by weighting the two individual scores, whereas we made use of the average rank for combination.

Conclusion
In this paper, we proposed a new method to mine new word translations from comparable corpora, by combining context and transliteration information, which are complementary sources of information. We evaluated our approach on six months of Chinese and English Gigaword corpora, with encouraging results.