Machine translation for Arabic dialects (survey) <s> Discussion <s> Recently the rate of written colloquial text has increased dramatically. It is being used as a medium of expressing ideas especially across the WWW, usually in the form of blogs and partially colloquial articles. Most of these written colloquial has been in the Egyptian colloquial dialect, which is considered the most widely dialect understood and used throughout the Arab world. Modern Standard Arabic is the official Arabic language taught and understood all over the Arab world. Diacritics play a key role in disambiguating Arabic text. The reader is expected to infer or predict vowels from the context of the sentence. Inferring the full form of the Arabic word is also useful when developing Arabic natural language processing tools and applications. In this paper, we introduce a generic method for converting a written Egyptian colloquial sentence into its corresponding diacritized Modern Standard Arabic sentence which could easily be extended to be applied to other dialects of Arabic. In spite of the non-availability of linguistic Arabic resources for this task, we have developed techniques for lexical acquisition of colloquial words which are used for transferring written Egyptian Arabic into Modern Standard Arabic. We successfully used Support Vector Machine approach for the diacritization (aka vocalization or vowelling) of Arabic text. <s> BIB001 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> We present a method for generating Colloquial Egyptian Arabic (CEA) from morphologically disambiguated Modern Standard Arabic (MSA). When used in POS tagging, this process improves the accuracy from 73.24% to 86.84% on unseen CEA text, and reduces the percentage of out-of-vocabulary words from 28.98% to 16.66%. The process holds promise for any NLP task targeting the dialectal varieties of Arabic; e.g., this approach may provide a cheap way to leverage MSA data and morphological resources to create resources for colloquial Arabic to English machine translation. It can also considerably speed up the annotation of Arabic dialects. <s> BIB002 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> The development of natural language processing tools for dialects faces the severe problem of lack of resources. In cases of diglossia, as in Arabic, one variant, Modern Standard Arabic (MSA), has many resources that can be used to build natural language processing tools. Whereas other variants, Arabic dialects, are resource poor. Taking advantage of the closeness of MSA and its dialects, one way to solve the problem of limited resources, consists in performing a translation of the dialect into MSA in order to use the tools developed for MSA. We describe in this paper an architecture for such a translation and we evaluate it on Tunisian Arabic verbs. Our approach relies on modeling the translation process over the deep morphological representations of roots and patterns, commonly used to model Semitic morphology. We compare different techniques for how to perform the cross-lingual mapping. Our evaluation demonstrates that the use of a decent coverage root+pattern lexicon of Tunisian and MSA with a backoff that assumes independence of mapping roots and patterns is optimal in reducing overall ambiguity and increasing recall. <s> BIB003 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> We present a dialectal Egyptian Arabic to English statistical machine translation system that leverages dialectal to Modern Standard Arabic (MSA) adaptation. In contrast to previous work, we first narrow down the gap between Egyptian and MSA by applying an automatic characterlevel transformational model that changes Egyptian to EG 0 , which looks similar to MSA. The transformations include morphological, phonological and spelling changes. The transformation reduces the out-of-vocabulary (OOV) words from 5.2% to 2.6% and gives a gain of 1.87 BLEU points. Further, adapting large MSA/English parallel data increases the lexical coverage, reduces OOVs to 0.7% and leads to an absolute BLEU improvement of 2.73 points. <s> BIB004 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> Modern Standard Arabic (MSA) is the formal language in most Arabic countries. Arabic Dialects (AD) or daily language differs from MSA especially in social media communication. However, most Arabic social media texts have mixed forms and many variations especially between MSA and AD. This paper aims to bridge the gap between MSA and AD by providing a framework for the translation of texts of social media. More precisely, this paper focuses on the Tunisian Dialect of Arabic (TAD) with an application on automatic machine translation for a social media text into MSA and any other target language. Linguistic tools such as a bilingual TAD-MSA lexicon and a set of grammatical mapping rules are collaboratively constructed and exploited in addition to a language model to produce MSA sentences of Tunisian dialectal sentences. This work is a first-step towards collaboratively constructed semantic and lexical resources for Arabic Social Media within the ASMAT (Arabic Social Media Analysis Tools) project. <s> BIB005 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> Today, several tools exist for automatic processing of standard Arabic. In comparison, resources and tools for Arabic dialects especially Moroccan dialect are still lacking. Given the proximity of the Moroccan dialect with standard Arabic, a way is to provide a translation of the first to the second by combining a rule-based approach and a statistical approach, using tools designed for Arabic standard and adapting these tools to Moroccan dialect. We describe in this paper an architecture for such a translation. <s> BIB006 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> In this paper, we present a statistical machine translation system for English to Dialectal Arabic (DA), using Modern Standard Arabic (MSA) as a pivot. We create a core system to translate from English to MSA using a large bilingual parallel corpus. Then, we design two separate pathways for translation from MSA into DA: a two-step domain and dialect adaptation system and a one-step simultaneous domain and dialect adaptation system. Both variants of the adaptation systems are trained on a 100k sentence tri-parallel corpus of English, MSA, and Egyptian Arabic generated by a rule-based transformation. We test our systems on a held-out Egyptian Arabic test set from the 100k sentence corpus and we achieve our best performance using the two-step domain and dialect adaptation system with a BLEU score of 42.9. <s> BIB007 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> We demonstrate the feasibility of using unsupervised morphological segmentation for dialects of Arabic, which are poor in linguistics resources. Our experiments using a Qatari Arabic to English machine translation system show that unsupervised segmentation helps to improve the translation quality as compared to using no segmentation or to using ATB segmentation, which was especially designed for Modern Standard Arabic (MSA). We use MSA and other dialects to improve Qatari Arabic to English machine translation, and we show that a uniform segmentation scheme across them yields an improvement of 1.5 BLEU points over using no segmentation. <s> BIB008 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> Dialects and standard forms of a language typically share a set of cognates that could bear the same meaning in both varieties or only be shared homographs but serve as faux amis. Moreover, there are words that are used exclusively in the dialect or the standard variety. Both phenomena, faux amis and exclusive vocabulary, are considered out of vocabulary (OOV) phenomena. In this paper, we present this problem of OOV in the context of machine translation. We present a new approach for dialect to English Statistical Machine Translation (SMT) enhancement based on normalizing dialectal language into standard form to provide equivalents to address both aspects of the OOV problem posited by dialectal language use. We specifically focus on Arabic to English SMT. We use two publicly available dialect identification tools: AIDA and MADAMIRA, to identify and replace dialectal Arabic OOV words with their modern standard Arabic (MSA) equivalents. The results of evaluation on two blind test sets show that using AIDA to identify and replace MSA equivalents enhances translation results by 0.4% absolute BLEU (1.6% relative BLEU) and using MADAMIRA achieves 0.3% absolute BLEU (1.2% relative BLEU) enhancement over the baseline. We show our replacement scheme reaches a noticeable enhancement in SMT performance for faux amis words. <s> BIB009 </s> Machine translation for Arabic dialects (survey) <s> Discussion <s> In this paper we study the use of sentencelevel dialect identification in optimizing machine translation system selection when translating mixed dialect input. We test our approach on Arabic, a prototypical diglossic language; and we optimize the combination of four different machine translation systems. Our best result improves over the best single MT system baseline by 1.0% BLEU and over a strong system selection baseline by 0.6% BLEU on a blind test set. <s> BIB010
We presented above a set of recent machine translation studies dedicated to Arabic dialects. This research work has been described in terms of used approach, data configuration and relevant results. In the following, we sum up the most significant findings of these different contributions: • The limited number of covered languages shows that MT for Arabic dialects is just beginning. Indeed, all contributions are dedicated to translate between dialects, MSA and English. We note that there is only one work which translates to French but unfortunately, no results are available for it. In terms of translation direction, most of the contributions translate from dialects to MSA or English, whereas there is very little work that uses the dialect as target language. This may be explained by the fact that using dialect as target language for a SMT system for example requires important amount of cleaned data in order to build reliable language models. Even for rule-based MT systems, it requires adapted tools (morphological, syntactic and semantic generators). Such requirements are still unavailable for most Arabic dialects. • Regards to the used dialects (see Table 3 ), it is clear that middleeast dialects are the most used ones especially Egyptian (spoken in the most populous Arab country 20 ), followed by Levantine, whilst Maghrebi dialects are less present when for the other dialects like Koweitian, Bahraini, Omani and Mauritanian no work in this field was found. available, the rule-based approach is adopted despite its drawbacks. BIB001 Statistical 1k sentences Accuracy: 88% tokenization & tagging + Rule-based transformation  Rule-based approach + LM 300 sentences Accuracy: 93.15% BIB002 Rule-based approach 100 user comments POS tagging evaluation Accuracy: 73.24% (Al-Gaphari and Al-Yadoumi, 2012) Rule-based approach 9386 words Accuracy: 77.32% BIB003 Rule based approach Parallel Tunisian/MSA Accuracy: (deep morphological corpus of 1500 sentence pairs Tunisian-to-MSA 84% representation of data) Dev/test set 750 sentence pairs, MSA-to-Tunisian 80% BIB005 Rule-based approach 50 sentences BLEU score: 14.32 +Bilingual lexicon+LM BIB006 Rule-based approach -+Bilingual lexicon+LM  Statistical approach 6 sides parallel corpus A set of BLEU scores of 6400 sentences Dev/test set 500 sentence for each corpus • For machine translation between Arabic dialects and English, the dominant methodology is hybridizing rule-based and statistical approaches, especially for the first research work (see table 5 ). The SMT systems are trained on large MSA/English corpora in addition to relatively smaller dialectal corpora. The rule-based methods rely on morphological analysis and transfer rules to normalize dialectal words into MSA words  Habash, 2011, 2013; BIB004 . Other work uses domain adaptation techniques by considering dialect adaptation as a domain adaption problem BIB004 BIB007 BIB008  . The availability of some parallel corpora makes this research direction possible. Furthermore, availability of new tools related to dialect identification (at word and sentences levels) has a positive impact on machine trans-lation performance as it was shown in BIB009 BIB010 . Indeed, in this last work, identifying either the sentence is dialectal or MSA guides the selection of the MT system to use. Also, It must be stressed that the impact of segmentation have been showed in most work, it improves MT scores significantly. • An important point related to Arabic dialects MT is using MSA as a pivot language when translating to or from English. As mentionned above, exploiting the proximity between close languages has been used in NLP research dedicated to under-resourced languages. The idea is to adapt existing resources of a rich-resourced language to process an under-resourced language, particularly, in the context of standard langauges and their dialects. This research direction has been adopted in the area of Arabic dialects NLP and especially for machine translation. We observe that the first efforts were (naturally) dedicated to translating between dialects and MSA, probably with a view to reaching other standard languages. Pivoting through MSA has been used in a majority of contributions, they state that it improves MT quality, except one work  which shows that increasing the dialect training data increases MT performance better than pivoting through MSA, but it noticed that the OOV rate is lower with MSA-mapping. The authors concluded that differences in genre between MSA and dialects make vocabulary coverage insufficient and considering the domain is an important research direction. We note that an intersting idea has been introduced in the work of BIB010 where authors have combined four MT systems among them a system which pivots through MSA and a system that translates directly from dialect to English. By using learning machine techniques and according to dialect level of the sentence, they select the adequate MT system (among the four ones) to translate (the considered sentence). Consequently, they continue to take profit of MSA-mapping whenever it is possible. In this way, the MT systems form a whole and complement each other. • Another big challenge of Arabic dialects MT is Arabizi (aka Arabish or Romanized Arabic). Indeed, important amount of user-generated data from social networks are a mixture of dialect written in both Arabic and Roman script. Given their size, these data could be an important source of dialect corpora if they are processed. It is in this 20 context that most recent MT work for Arabic dialects attempt to deal with translation from Arabizi to English. But, despite its large use, Arabizi is still a new research direction, few work are dedicated to it and they concern only Egyptian dialect. Other Arabic dialects are at a preliminary stage. The contributions presented in this survey related to Arabizi are based on a SMT system built on the top of a deromanization module that converts Arabizi texts to Arabic script. The importance of deromanization is evident, it was shown that its accuracy rates correlate with MT scores (in both of the two papers presented above). We expect that future work attempt to bypass the step of deromanization when more parallel corpora including Arabizi will be available. Thus, Arabicscript pivoting and direct translation will be certainly experimented. In this respect, direct translation from Arabizi to English or French will probably reduces the complexity of two serious problems related to Arabic dialects MT; proper nouns translation and code-switching. Since Arabizi uses Roman script, there is no need to translate proper nouns even more English or French words 21 (included in dialect text in the case of code-switching) • Regards to the data, we notice a significant lack of textual resources dedicated for dialects. All research efforts deal with this issue. We can see that MSA-dialect parallel corpora are fewer than English-dialect ones (see data description in Tables 4 and 5 ). This is due to the fact that MT projects between Arabic dialects and English are more funded mainly in the case of BOLT project. Yet, even with this funding, the corpora including dialects are smaller than those of standard languages (MSA/English for example). Also, in terms of coverage, Egyptian and Levantine remain the most resourced dialects in contrast to all others. It is worth noting that an important portion of several MT efforts is dedicated to produce dialect resources. 22