Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> We present an approach to using a morphological analyzer for tokenizing and morphologically tagging (including part-of-speech tagging) Arabic words in one process. We learn classifiers for individual morphological features, as well as ways of using these classifiers to choose among entries from the output of the analyzer. We obtain accuracy rates on all tasks in the high nineties. <s> BIB001 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> This paper introduces two recent open source software packages developed for unsupervised natural language modeling. The Morfessor program segments words automatically into morpheme-like units without any rule-based morphological analyzers. The VariKN toolkit trains language models producing a compact set of high-order n-grams utilizing state-of-art KneserNey smoothing. As an example, this paper shows how to construct a language model for speech recognition in multiple languages utilizing only a minimal amount of linguistic resources. Morfessor and VariKN also have other applications in text understanding, information retrieval and machine translation. Unsupervised machine learning techniques are particularly well suited for the development of systems for less-resourced languages, because they do not depend on manually designed morphological or syntactical analyzers or annotated data. <s> BIB002 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> This paper is about improving the quality of Arabic-English statistical machine translation (SMT) on dialectal Arabic text using morphological knowledge. We present a light-weight rule-based approach to producing Modern Standard Arabic (MSA) paraphrases of dialectal Arabic out-of-vocabulary (OOV) words and low frequency words. Our approach extends an existing MSA analyzer with a small number of morphological clitics, and uses transfer rules to generate paraphrase lattices that are input to a state-of-the-art phrase-based SMT system. This approach improves BLEU scores on a blind test set by 0.56 absolute BLEU (or 1.5% relative). A manual error analysis of translated dialectal words shows that our system produces correct translations in 74% of the time for OOVs and 60% of the time for low frequency words. <s> BIB003 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> Modern Standard Arabic (MSA) has a wealth of natural language processing (NLP) tools and resources. In comparison, resources for dialectal Arabic (DA), the unstandardized spoken varieties of Arabic, are still lacking. We present ELISSA, a machine translation (MT) system for DA to MSA. ELISSA employs a rule-based approach that relies on morphological analysis, transfer rules and dictionaries in addition to language models to produce MSA paraphrases of DA sentences. ELISSA can be employed as a general preprocessor for DA when using MSA NLP tools. A manual error analysis of ELISSA’s output shows that it produces correct MSA translations over 93% of the time. Using ELISSA to produce MSA versions of DA sentences as part of an MSA-pivoting DA-to-English MT solution, improves BLEU scores on multiple blind test sets between 0.6% and 1.4%. <s> BIB004 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> We present a dialectal Egyptian Arabic to English statistical machine translation system that leverages dialectal to Modern Standard Arabic (MSA) adaptation. In contrast to previous work, we first narrow down the gap between Egyptian and MSA by applying an automatic characterlevel transformational model that changes Egyptian to EG 0 , which looks similar to MSA. The transformations include morphological, phonological and spelling changes. The transformation reduces the out-of-vocabulary (OOV) words from 5.2% to 2.6% and gives a gain of 1.87 BLEU points. Further, adapting large MSA/English parallel data increases the lexical coverage, reduces OOVs to 0.7% and leads to an absolute BLEU improvement of 2.73 points. <s> BIB005 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> In this paper we study the use of sentencelevel dialect identification in optimizing machine translation system selection when translating mixed dialect input. We test our approach on Arabic, a prototypical diglossic language; and we optimize the combination of four different machine translation systems. Our best result improves over the best single MT system baseline by 1.0% BLEU and over a strong system selection baseline by 0.6% BLEU on a blind test set. <s> BIB006 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> In this paper, we present a statistical machine translation system for English to Dialectal Arabic (DA), using Modern Standard Arabic (MSA) as a pivot. We create a core system to translate from English to MSA using a large bilingual parallel corpus. Then, we design two separate pathways for translation from MSA into DA: a two-step domain and dialect adaptation system and a one-step simultaneous domain and dialect adaptation system. Both variants of the adaptation systems are trained on a 100k sentence tri-parallel corpus of English, MSA, and Egyptian Arabic generated by a rule-based transformation. We test our systems on a held-out Egyptian Arabic test set from the 100k sentence corpus and we achieve our best performance using the two-step domain and dialect adaptation system with a BLEU score of 42.9. <s> BIB007 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> One of the aims of DARPA BOLT project is to translate the Egyptian blog data into English. While the parallel data for MSA-English is abundantly available, sparsely exists for Egyptian-English and Egyptian-MSA. A notable drop in the translation quality is observed when translating Egyptian to English in comparison with translating from MSA to English. One of the reasons for this drop is the high OOV rate, where as another is the dialectal differences between training and test data. This work is focused on improving Egyptian-to-English translation by bridging the gap between Egyptian and MSA. First we try to reduce the OOV rate by proposing MSA candidates for the unknown Egyptian words through different methods such as spelling correction, suggesting synonyms based on context etc. Secondly we apply convolution model using English as a pivot to map Egyptian words into MSA. We then evaluate our edits by running decoder built on MSA-to-English data. Our spelling-based correction shows an improvement of i¾?1.7 BLEU points over the baseline system, that translates unedited Egyptian into English. <s> BIB008 </s> Machine translation for Arabic dialects (survey) <s> Translating between Arabic dialects and foreign languages <s> Dialects and standard forms of a language typically share a set of cognates that could bear the same meaning in both varieties or only be shared homographs but serve as faux amis. Moreover, there are words that are used exclusively in the dialect or the standard variety. Both phenomena, faux amis and exclusive vocabulary, are considered out of vocabulary (OOV) phenomena. In this paper, we present this problem of OOV in the context of machine translation. We present a new approach for dialect to English Statistical Machine Translation (SMT) enhancement based on normalizing dialectal language into standard form to provide equivalents to address both aspects of the OOV problem posited by dialectal language use. We specifically focus on Arabic to English SMT. We use two publicly available dialect identification tools: AIDA and MADAMIRA, to identify and replace dialectal Arabic OOV words with their modern standard Arabic (MSA) equivalents. The results of evaluation on two blind test sets show that using AIDA to identify and replace MSA equivalents enhances translation results by 0.4% absolute BLEU (1.6% relative BLEU) and using MADAMIRA achieves 0.3% absolute BLEU (1.2% relative BLEU) enhancement over the baseline. We show our replacement scheme reaches a noticeable enhancement in SMT performance for faux amis words. <s> BIB009
Sawaf (2010) built a hybrid MT system combining statistical and rulebased approaches. This system translates from Arabic dialects (spontaneous and noisy text from broadcast transmissions and web content) to English using MSA as pivot language. Dialect texts were normalized into MSA using character-based rules which utilizes simple rules to convert words into the most similar MSA words, then the text is analyzed by a dialect-specific and a MSA morphological analyzers. The results are entered into dialect normalization decoder that relies on language models and a lexicon. The work deals with a set of Arabic dialects: Levantine (Lebanese, North Syria, Damascus, Palestine and Jordan), Gulf Arabic (Northern Iraq, Baghdad, Southern Iraq, Gulf, Saudi-Arabia, and Southern Arabic Peninsula), Nile Region (Egypt and Sudan) and Maghreb Arabic (Libya, Morocco and Tunisia). Achieved results showed that hybrid MT performs better than statistical MT and rule-based MT and that normalizing and processing the text (both training and test corpora) improve translation quality in terms of BLEU by 2% for Web text and about 1% for broadcast news/conversations. In BIB003 , the authors improved an Arabic-English SMT system by producing MSA paraphrases for OOV dialectal words and low-frequency words through a light-weight rule-based approach. They created ADAM (Arabic Dialect Morphological Analyzer) by extending the wellknown BAMA  with Levantine/Egyptian dialectal affixes and clitics. In addition to ADAM, they used a set of hand-write morphosyntactic transfer rules. This allows to generating paraphrases that are input as a lattice to a state-of-the-art phrase-based SMT system. This last point is the main difference between this work and the one presented above  . The latter produces unique MSA version for a dialect word where the former produces multiple MSA paraphrases (or alternative normalizations). Two SMT systems were built within this work, they were trained on two different data conditions, a MSA(only)-English parallel corpus (of 12M words on the Arabic side) and a large (MSA&Dialect)-English parallel corpus (of 64M words on the Arabic side). When evaluated on a blind test set, the SMT system trained on the large corpus using ADAM and transfer rules outperformed the baseline system (SMT system trained on the same data) by 0.56 absolute BLEU. The same authors in BIB004 , presented a manual evaluation of Elissa (cited above). It was shown that 93% of MSA sentences produced by Elissa were correct. In addition, Elissa was used for pivoting through MSA in a dialect-English SMT system whose BLEU score was improved between 0.6% and 1.4%. BIB005 provided a dialectal Egyptian Arabic to English statistical machine translation system. They converted Egyptian to MSA by applying a character level transformational model (including morphological, phonological and spelling changes) learned from Egyptian-MSA words pairs. The MT system built on the adapted parallel data showed improvement in the quality of machine translation. Transformation task reduces the OOV words rate from 5.2% to 2.6% and improves BLEU score by 1.87 points. Whereas adapting large MSA/English parallel data gives significant reduction of OOV rate to 0.7% and leads to an absolute BLEU increase of 2.73 points. BIB006 explored the impact of sentence-level dialect identification used with various linguistic features on machine translation performance. They attempted to optimize the selection of outputs produced by different MT systems given an input text including a mixture of dialects and MSA. The study concerns machine translation from Arabic dialect, namely Egyptian and Levantine to English. Four MT systems were used for this purpose, the first three ones are SMT systems trained on different corpora 4 : dialect-English (5M tokenized words of Egyptian and Levantine), MSA-English (57M tokenized words) and dialect+MSA-English (62M tokenized words). The fourth one is a MSA-pivoting system that combines dialect-to-MSA MT system BIB004 and an Arabic-English SMT system. This last system is trained on dialect+MSA-English corpus augmented with dialect-English corpus where the dialectal side has been preprocessed with the dialect-MSA MT previously cited BIB004 . The size of this training corpus is 67M. We note that the MSA-pivoting system (the fourth one) produces the best BLEU score among all systems, it is the first baseline system. In this wrok, the same MT algorithms are used for training, tuning and testing each MT system, but as regards data each system is trained on a different dataset (as we saw above) in terms of the degree of source language dialectness. An interesting approach was adopted in this reaserach, instead of finding the most performant MT, the authors tried to identify automatically the most suitable MT system for a given sentence. They assume that these systems complement each other and combining their selections could lead to better overall performance. A baseline MT system selection based on a binary classification was built by using a sentence-level dialect identifier  . This baseline selection system decides what MT system to use among the four systems described above. According to the authors, the best configuration defined is to select the MSA-English system for sentences tagged as MSA sentences and MSA-pivoting for sentences tagged as dialectal ones. The main contribution of this work is a MT selection system created using machine learning techniques trained on only source language features to select the best MT system that should translate each sentence in the test set. This selection system is a Naive Bayes Classifier (NBC) with four classes corresponding to the four MT systems. The training data of the classifier is a set of 5562 sentences labeled with the class label of the MT system that has produced the highest BLEU score (at sentence-level). The NBC uses a set of basic features such as: token-Level features which use language models, MSA & dialectal morphological analyzers and a dialectal lexicon (to decide whether each word is MSA, dialectal, both, or OOV), perplexity features that include two features related to the perplexity of a sentence computed on the two languages models (MSA and dialect). In addition, the classifier uses some extended features extracted from the cited dialect-MSA MT system like sentence length (in words), number of punctuation marks, and number of words that are written in Latin script. Another set of extended features are used like the sentence perplexity computed on each source-side of the training data of each of the four MT systems. Using the NBC to predict the best MT system to use for translating a sentence had improved the BLEU score by 1% over the best score recorded for a single MT (which corresponds to the MSA-pivoting system). It also outperforms the baseline selection system by 0.6% BLEU. BIB007 presented a SMT system that translates (in contrast to all other research efforts) from English to Arabic dialect by pivoting through MSA. The translator is based on a core SMT system trained on a parallel English-MSA corpus of (5M pairs of sentences), the output of this system is translated to Egyptian dialect by using both dialect and domain adaptation system. It should be noted that for adaptation systems the authors created a tri-side parallel corpus (English, MSA and Egyptian dialect) of 100k sentences by using a rule-based method. For convenience of reading we refer to each side of this corpus as Eng-100k, MSA-100k and Egy-100k. Two variants of adaptation system were presented. The first variant translated with the core SMT system the English side (Eng-100k) of the tri-parallel corpus to MSA (we call the result MSA-100k-trans). This dataset is used with the Egyptian side (Egy-100k) of the corpus as training data to translate from MSA to Egyptian. An English test set is translated to MSA (by using the core SMT English-MSA), the result is then translated to Egyptian dialect by using the SMT trained on the parallel corpus (MSA-100k-trans, Egy100k). The second variant includes two adaptation steps. The first one is used to adapt the MSA output of the core system to the domain of the MSA side in the tri-parallel corpus and a second one to translate the MSA output of the domain adaptation system into Egyptian Arabic. An English test set is translated to MSA with the SMT core system, the result is then translated by the first adaptation system trained on (MSA-100k-trans, MSA-100k). The output of this step is then translated into Egyptian by using the second adaptation system trained on (MSA100k, Egy100k). The main result of this work showed that it is possible to increase the MT quality by using domain adaptation between MSA and Egyptian dialect as adapting between different domains of the same language. Furthermore, using MSA as a pivot then adapting to dialect could improve MT performance. Al-Mannai et al. (2014) proposed an unsupervised morphological segmentation for Arabic dialects to improve machine translation quality. The study concerned a Qatari Arabic to English SMT. It was shown that segmentation with Morfessor BIB002 (unsupervised morphological segmenter) improves the translation quality compared to a system without segmentation at all or to a system using Arabic Treebank (ATB) segmentation. In addition, a multi-dialectal word segmentation model was trained on the Arabic part of a parallel corpus including Qatari Arabic, Egyptian, Levantine, MSA and English. This segmented corpus was used to train the Qatari Arabic to English SMT, the BLEU score increased by 1.5 points when compared to a baseline system which does not use segmentation. In the other direction, a preliminary SMT system was trained to translate English to Qatari Arabic using the same parallel corpus without segmentation and by training the language model with other dialect corpora. The best system shows an absolute improvement of 0.22 in terms of BLEU compared to the baseline system that only uses the Arabic side of the Qatari Arabic corpus for language model (LM) training. BIB008 improved Egyptian-to-English translation quality by handling OOV words. They first proceed to convert Egyptian to MSA by using a large monolingual language model to score the MSA-candidates for Egyptian OOV words (via a stack-based search with a beam-search algorithm). These candidates are got mainly through spelling correction and suggesting synonyms on context, MSA results are then translated to English via a SMT system. They showed that the spelling-based correction could improve the BLEU score by 1.7 points over the baseline system that translates unedited Egyptian into English. This work introduced an interesting idea to map Egyptian words into MSA by applying a convolution model using English as a pivot, the model relies on two corpora of 8.5K parallel sentences of Egyptian-English and 300K sentences of MSA-English. Bolt Project 5 . DARPA launched the Broad Operational Language Translation (BOLT) program (2011) (2012) (2013) (2014) to attempt to create new techniques for automated translation and linguistic analysis that can be applied to the informal genres of text and speech common in online and in-person communication in English, Chinese and Egyptian Arabic. BOLT has three technical areas: developing algorithms and integrated systems to support the translation, data collection and an evaluation step. Under this program, in  , two parallel corpora Levantine-English (1.1M words) and Egyptian-English (380K words) were built by translating parts extracted from a large corpus of Arabic web texts to English. Classification by dialect and translation were done by using Amazon's Mechanical Turk. Authors performed several experiments on a SMT system using these corpora in addition to a MSA-English parallel corpus (150M tokens for Arabic side). It was shown that morphological segmentation (using MADA BIB001 morphological analyzer) uniformly improves translation quality. The work studied also the impact of dialectal training data size on MT performance. They show that a system trained on the combined dialectal-MSA data is likely to give the best performance, since informal Arabic data is usually a mixture of dialectal Arabic and MSA. Another interesting result was presented regards to pivoting through MSA or translating directly from dialect into English (the experiment was performed for Levantine only). The performance of the system improves by 2.3 BLEU points when pivoting through MSA for first experiment, but when adding more dialectal data to training set (400k words) direct translation becomes better than mapping to MSA despite the significantly low OOV rate with MSA-mapping. BIB009 dealt with OOV words in the context of Arabic to English SMT system. They adopted an approach that normalizes dialectal words to MSA words by using AIDA 6 (Elfardy et al., 2014) and MADAMIRA 7  , to identify and replace dialectal Arabic OOV words with their MSA equivalents. When tested on a blind dataset test, this approach improved SMT quality by 0.4% and 0.3% absolute BLEU for AIDA and MADAMIRA, respectively. Within the same program, in  a focus was made on Arabic dialect to English translation especially for Egyptian dialect. Several techniques have been implemented such as adapting SMT systems to the Egyptian dialect since the available training corpora, in the context of Bolt project, contain MSA and several dialects (Egyptian, Levantine and Iraqi). 6 A dialect identification tool that identifies and classifies dialectal words on the token and sentence levels. 7 A morphological analysis and disambiguation system for MSA and Egyptian dialect.