DCU-Lingo24 Participation in WMT 2014 Hindi-English Translation task

This paper describes the DCU-Lingo24 submission to WMT 2014 for the HindiEnglish translation task. We exploit miscellaneous methods in our system, including: Context-Informed PB-SMT, OOV Word Conversion (OWC), MultiAlignment Combination (MAC), Operation Sequence Model (OSM), Stemming Align and Normal Phrase Extraction (SANPE), and Language Model Interpolation (LMI). We also describe various preprocessing steps we tried for Hindi in this task.


Introduction
This paper describes the DCU-Lingo24 submission to WMT 2014 for the Hindi-English translation task.
In the next section, the preprocessing steps are explained. In Section 3 a detailed explanation of the technique we exploit is provided. Then in Section 4, we provide our experimental results and resultant discussion.

Pre-processing Steps
We use all the training data provided for Hindi-English translation. Following Bojar et al. (2010), we apply a number of normalisation methods on the Hindi corpus. The HindEnCorp parallel corpus compiles several sources of parallel data. We observe that the source-side (Hindi) of the TIDES data source contains font-related noise, i.e. many Hindi sentences are a mixture of two different encodings: UTF-8 1 and WX 2 notations. We prepared a WX-to-UTF-8 font conversion script for Hindi which converts all WX encoded characters into UTF-8, thus removing all WX encoding appearing in the TIDES data.
We also observe that a portion of the English training corpus contained the following bracketlike sequences of characters: -LRB-, -LSB-, -LCB-, -RRB-, -RSB-, and -RCB-. 3 For consistency, those character sequences in the training data were replaced by the corresponding brackets.
For English -both monolingual and the target side of the bilingual data -we perform tokenization, normalization of punctuation, and truecasing. For parallel training data, we filter sentences pairs containing more than 80 tokens on either side and Clearly, Hindi and English have quite different word orders, so we adopt three lexical reordering models to address this problem. They are wordbased LRM and phrase-based LRM, which mainly focus on local reordering phenomena, and hierarchical phrase-based LRM, which mainly focuses on longer distance reordering (Galley and Manning, 2008).

Operation Sequence Model
The  (1): where n indicates the number of previous operations used. We employ a 9-order OSM in our framework.

Language Model Interpolation (LMI)
We build a large language model by including data from the English Gigaword fifth edition, the English side of the UN corpus, the English side of the 10 9 French-English corpus and the English side of the Hindi-English parallel data provided by the organisers. We interpolate language models trained using each dataset, with the monolingual data provided split into three parts (news 2007-2013, Europarl (?) and news commentary) and the weights tuned to minimize perplexity on the target side of the devset. The language models in our systems are trained with SRILM (Stolcke, 2002). We train a 5-gram model with Kneser-Ney discounting (Chen and Goodman, 1996).

Context-informed PB-SMT
Haque et al. (2011) express a context-dependent phrase translation as a multi-class classification problem, where a source phrase with given additional context information is classified into a distribution over possible target phrases. The size of this distribution needs to be limited, and would ideally omit irrelevant target phrase translations that the standard PB-SMT (Koehn et al., 2003) approach would normally include. Following Haque et al. (2011), we derive a context-informed featurê h mbl that is expressed as the conditional probability of the target phraseê k given the source phrasê f k and its context information (CI), as in (2): Here, CI may include any feature that can provide useful information to disambiguate the given source phrase. In our experiment, we use CCG supertag (Steedman, 2000) as a contextual features. CCG supertag expresses the specific syntactic behaviour of a word in terms of the arguments it takes, and more generally the syntactic environment in which it appears. We consider the CCG supertags of the context words, as well as of the focus phrase itself. In our model, the supertag of a multi-word focus phrase is the concatenation of the supertags of the words composing that phrase. We generate a window of size 2l + 1 features (we set l:=2), including the concatenated complex supertag of the focus phrase. Accordingly, the supertag-based contextual information (CI st ) is described as in (3): For the Hindi-to-English translation task, we use part-of-speech (PoS) tags 4 of the source phrase and the neighbouring words as the contextual feature, owing to the fact that supertaggers are readily available only for English. We use a memory-based machine learning (MBL) classifier (TRIBL: (Daelemans, 2005)) 5 that is able to estimate P(ê k |f k , CI(f k )) by similarity-based reasoning over memorized nearest-neighbour examples of source-target phrase translations. Thus, we derive the featurê h mbl defined in Equation (2). In addition toĥ mbl , we derive a simple two-valued featureĥ best , defined in Equation (4): (4) whereĥ best is set to 1 whenê k is one of the target phrases with highest probability according to P(ê k |f k , CI(f k )) for each source phrasef k ; otherwiseĥ best is set to 0.000001. We performed experiments by integrating these two featuresĥ mbl andĥ best directly into the log-linear model of Moses. Their weights are optimized using minimum error-rate training (MERT) (Och, 2003) on a held-out development set for each of the experiments.

Morphological Segmentation
Haque et al. (2012) applied a morphological suffix separation process in a Bengali-to-English translation task and showed that suffix separation significantly reduces data sparseness in the Bengali corpus. They also showed an SMT model trained on the suffix-stripped training data significantly outperforms the state-of-the-art PB-SMT baseline. Like Bengali, Hindi is a morphologically very rich and highly inflected Indian language. As done previously for Bengali-to-English (Haque et al., 2012), we employ a suffix-stripping method for lemmatizing inflected Hindi words in the WMT Hindi-to-English translation task. Following Dasgupta and Ng (2006), we developed an unsupervised morphological segmentation method for Hindi. We also used a Hindi lightweight stemmer (Ramanathan and Rao, 2003) in order to prepare a training corpus with only Hindi stems. We prepared Hindi-to-English SMT systems on the both types of training data (i.e. suffix-stripped and stemmed). 6

Multi-Alignment Combination (MAC)
Word alignment is a critical component of MT systems. Various methods for word alignment have been proposed, and different models can produce signicantly different outputs. For example, Tu et al. (2012) demonstrates that the alignment agreement between the two best-known alignment tools, namely Giza++ (Och and Ney, 2003) and the Berkeley aligner 7 (Liang et al., 2006), is below 70%. Taking into consideration the small size of the the corpus, in order to extract more effective phrase tables, we concatenate three alignments: Giza++ with grow-diag-final-and, Giza++ with intersection, and that derived from the Berkeley aligner.

Stemming Alignment and Normal Phrase Extraction (SANPE)
The rich morphology of Hindi will cause word alignment sparsity, which results in poor alignment quality. Furthermore, word stemming on the Hindi side usually results in too many English words being aligned to one stemmed Hindi word, i.e. we encounter the problem of phrase overextraction. Therefore, we conduct word alignment with the stemmed version of Hindi, and then at the phrase extraction step, we replace the stemmed form with the original Hindi form.

OOV Word Conversion Method
Our algorithm for OOV word conversion uses the recently developed zero-shot learning (Palatucci et al., 2009) using neural network language modelling (Bengio et al., 2000;Mikolov et al., 2013). The same technique is used in (Okita et al., 2014). This method requires neither parallel nor comparable corpora, but rather two monolingual corpora. In our context, we prepare two monolingual corpora on both sides, which are neither parallel nor comparable, and a small amount of already known correspondences between words on the source and target sides (henceforth, we refer to this as the 'dictionary'). Then, we train both sides with the neural network language model, and use a continuous space representation to project words to each other on the basis of a small amount of correspondences in the dictionary. The following algorithm shows the steps involved: 1. Prepare the monolingual source and target sentences.
2. Prepare the dictionary which consists of U entries of source and target sentences comprising non-stop-words.
3. Train the neural network language model on the source side and obtain the real vectors of X dimensions for each word.
4. Train the neural network language model on the target side and obtain the real vectors of X dimensions for each word.
5. Using the real vectors obtained in the above steps, obtain the linear mapping between the dictionary items in two continuous spaces using canonical component analysis (CCA).
In our experiments we use U the same as the entries of Wiki corpus, which is provided among WMT14 corpora, and X as 50. The resulted projection by this algorithm can be used as the OOV word conversion which projects from the source language which among OOV words into the target language. The overall algorithm which uses the projection which we build in the above step is shown in the following.
1. Collect unknown words in the translation outputs.
3. If they are noun phrases, do the projection from each unknown word in the source side into the target words (We use the projection prepared in the above steps). If they are not noun phrases, run the transliteration to convert each of them.
We perform Hindi NER by training CRF++ (Kudo et al., 2004) using the Hindi named entity corpus, and use the Hindi shallow parser (Begum et al., 2008) for preprocessing of the inputs.

Data
We conduct our experiments on the standard datasets released in the WMT14 shared translation task. We use HindEnCorp 8 (Bojar et al., 2014) parallel corpus for MT system building. We also used the CommonCrawl Hindi monolingual corpus (Bojar et al., 2014) in order to build an additional language model for Hindi. For the Hindi-to-English direction, we also employed monolingual English data used in the other translation tasks for building the English language model. 8 http://ufallab.ms.mff.cuni.cz/ bojar/hindencorp/

Moses Baseline
We employ a standard Moses PB-SMT model as our baseline. The Hindi side is preprocessed but unstemmed. We use Giza++ to perform word alignment, the phrase table is extracted via the grow-diag-final-and heuristic and the max-phraselength is set to 7.  We prepared a number of MT systems for both English-to-Hindi and Hindi-to-English, and submitted their runs in the WMT 2014 Evaluation Matrix. The BLEU scores of the different Englishto-Hindi MT systems (Moses Baseline, Context-Based (CCG) MT system, and Context-Based (CCG) MT system with an additional LM built on the CommonCrawl Hindi monolingual corpus (Bojar et al., 2014)) on the WMT 2014 Englishto-Hindi test set are reported in Table 1. As can be seen from Table 1, Context-Based (CCG) MT system produces 0.7 BLEU points improvement (8.04% relative) over the Moses Baseline. When we add an additional large LM built on the Com-monCrawl data to the Context-Based (CCG) MT system, we achieved a 2 BLEU-point improvement (21.3% relative) (cf. last row in Table 1) over the Context-Based (CCG) MT system. 9 The BLEU scores of the different Hindi-to-English MT systems on the WMT 2014 Hindito-English test set are reported in Table 2. The first row of Table 2 shows the BLEU score for the Baseline MT system. We note that the performance of the Context-Based (PoS) MT system obtains identical performance to the Moses baseline (10.1 BLEU points) on the WMT 2014 Hindito-English test set.

Automatic Evaluation
We employed a source language (Hindi) normalisation technique, namely suffix separation, but unfortunately this did not bring about any improvement for the Hindi-to-English translation task. The improvement gained by individually employing OSM, three lexical reordering models, Multi-alignment Combination, Stem-align and normal Phrase Extraction and Language Model Interpolation can be seen in Table 2. Our best system is achieved by combining OSM, Three LMR, MAC, SANPE and LMI, which results in a 1.6 BLEU point improvement over the Baseline.