5
ABSTRACTIVE
Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks.
EXTRACTIVE
We experimented with English INLINEFORM0 Hindi translation as the parent task. English is the assisting source language. Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks. Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages. All these languages have a canonical SOV word order.
Languages We experimented with English INLINEFORM0 Hindi translation as the parent task. English is the assisting source language. Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks. Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages. All these languages have a canonical SOV word order.
1811.00383
Introduction </s> Deep ĠLearning Ġapproaches Ġhave Ġachieved Ġimpressive Ġresults Ġon Ġvarious ĠN LP Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġhave Ġbecome Ġthe Ġde Ġfacto Ġapproach Ġfor Ġany ĠN LP Ġtask . ĠHowever , Ġthese Ġdeep Ġlearning Ġtechniques Ġhave Ġfound Ġto Ġbe Ġless Ġeffective Ġfor Ġlow - resource Ġlanguages Ġwhen Ġthe Ġavailable Ġtraining Ġdata Ġis Ġvery Ġless ĠB IB REF 3 Ġ. ĠRecently , Ġseveral Ġapproaches Ġlike ĠMulti - task Ġlearning ĠB IB REF 4 Ġ, Ġmult ilingual Ġlearning ĠB IB REF 5 Ġ, Ġsemi - super vised Ġlearning ĠB IB REF 2 Ġ, ĠB IB REF 6 Ġand Ġtransfer Ġlearning ĠB IB REF 7 Ġ, ĠB IB REF 3 Ġhave Ġbeen Ġexplored Ġby Ġthe Ġdeep Ġlearning Ġcommunity Ġto Ġovercome Ġdata Ġsp arsity Ġin Ġlow - resource Ġlanguages . ĠTransfer Ġlearning Ġtrains Ġa Ġmodel Ġfor Ġa Ġparent Ġtask Ġand Ġfine - t unes Ġthe Ġlearned Ġparent Ġmodel Ġweights Ġ( features ) Ġfor Ġa Ġrelated Ġchild Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠThis Ġeffectively Ġreduces Ġthe Ġrequirement Ġon Ġtraining Ġdata Ġfor Ġthe Ġchild Ġtask Ġas Ġthe Ġmodel Ġwould Ġhave Ġlearned Ġrelevant Ġfeatures Ġfrom Ġthe Ġparent Ġtask Ġdata Ġthereby , Ġimproving Ġthe Ġperformance Ġon Ġthe Ġchild Ġtask . </s> Transfer Ġlearning Ġhas Ġalso Ġbeen Ġexplored Ġin Ġthe Ġmult ilingual ĠNeural ĠMachine ĠTranslation ĠB IB REF 3 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠThe Ġgoal Ġis Ġto Ġimprove Ġthe ĠN MT Ġperformance Ġon Ġthe Ġsource Ġto Ġtarget Ġlanguage Ġpair Ġ( child Ġtask ) Ġusing Ġan Ġassisting Ġsource Ġlanguage Ġ( ass isting Ġto Ġtarget Ġtranslation Ġis Ġthe Ġparent Ġtask ). ĠHere , Ġthe Ġparent Ġmodel Ġis Ġtrained Ġon Ġthe Ġassisting Ġand Ġtarget Ġlanguage Ġparallel Ġcorpus Ġand Ġthe Ġtrained Ġweights Ġare Ġused Ġto Ġinitialize Ġthe Ġchild Ġmodel . ĠThe Ġchild Ġmodel Ġcan Ġnow Ġbe Ġfine - tun ed Ġon Ġthe Ġsource - target Ġlanguage Ġpairs , Ġif Ġparallel Ġcorpus Ġis Ġavailable . ĠThe Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġthe Ġassisting Ġlanguage Ġcan Ġadversely Ġimpact Ġthe Ġbenefits Ġobtained Ġfrom Ġtransfer Ġlearning . ĠMultiple Ġstudies Ġhave Ġshown Ġthat Ġtransfer Ġlearning Ġworks Ġbest Ġwhen Ġthe Ġlanguages Ġare Ġrelated ĠB IB REF 3 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 9 Ġ. ĠSeveral Ġstudies Ġhave Ġtried Ġto Ġaddress Ġlex ical Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġthe Ġtarget Ġlanguages ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠHowever , Ġthe Ġeffect Ġof Ġword Ġorder Ġdivergence Ġand Ġits Ġmitigation Ġhas Ġnot Ġbeen Ġexplored . ĠIn Ġa Ġpractical Ġsetting , Ġit Ġis Ġnot Ġuncommon Ġto Ġhave Ġsource Ġand Ġassisting Ġlanguages Ġwith Ġdifferent Ġword Ġorder . ĠFor Ġinstance , Ġit Ġis Ġpossible Ġto Ġfind Ġparallel Ġcorpor a Ġbetween ĠEnglish Ġand Ġsome ĠIndian Ġlanguages , Ġbut Ġvery Ġlittle Ġparallel Ġcorpor a Ġbetween ĠIndian Ġlanguages . ĠHence , Ġit Ġis Ġnatural Ġto Ġuse ĠEnglish Ġas Ġan Ġassisting Ġlanguage Ġfor Ġinter - Indian Ġlanguage Ġtranslation . </s> To Ġsee Ġhow Ġword Ġorder Ġdivergence Ġcan Ġbe Ġdetrimental , Ġlet Ġus Ġconsider Ġthe Ġcase Ġof Ġthe Ġstandard ĠR NN Ġ( Bi - L ST M ) Ġenc oder - att ention - dec oder Ġarchitecture ĠB IB REF 13 Ġ. ĠThe Ġenc oder Ġgenerates Ġcontextual Ġrepresentations Ġ( ann otation Ġvectors ) Ġfor Ġeach Ġsource Ġword , Ġwhich Ġare Ġused Ġby Ġthe Ġattention Ġnetwork Ġto Ġmatch Ġthe Ġsource Ġwords Ġto Ġthe Ġcurrent Ġdec oder Ġstate . ĠThe Ġcontextual Ġrepresentation Ġis Ġword - order Ġdependent . ĠHence , Ġif Ġthe Ġassisting Ġand Ġthe Ġsource Ġlanguages Ġdo Ġnot Ġhave Ġsimilar Ġword Ġorder Ġthe Ġgenerated Ġcontextual Ġrepresentations Ġwill Ġnot Ġbe Ġconsistent . ĠThe Ġattention Ġnetwork Ġ( and Ġhence Ġthe Ġdec oder ) Ġsees Ġdifferent Ġcontextual Ġrepresentations Ġfor Ġsimilar Ġwords Ġin Ġparallel Ġsentences Ġacross Ġdifferent Ġlanguages . ĠThis Ġmakes Ġit Ġdifficult Ġto Ġtransfer Ġknowledge Ġlearned Ġfrom Ġthe Ġassisting Ġlanguage Ġto Ġthe Ġsource Ġlanguage . </s> We Ġillustrate Ġthis Ġby Ġvisual izing Ġthe Ġcontextual Ġrepresentations Ġgenerated Ġby Ġthe Ġenc oder Ġof Ġan ĠEnglish Ġto ĠHindi ĠN MT Ġsystem Ġfor Ġtwo Ġversions Ġof Ġthe ĠEnglish Ġinput : Ġ( a ) Ġoriginal Ġword Ġorder Ġ( S VO ) Ġ( b ) Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage Ġ( S OV , Ġfor ĠBeng ali ). ĠFigure ĠFIG REF 1 Ġshows Ġthat Ġthe Ġenc oder Ġrepresentations Ġobtained Ġare Ġvery Ġdifferent . ĠThe Ġattention Ġnetwork Ġand Ġthe Ġdec oder Ġnow Ġhave Ġto Ġwork Ġwith Ġvery Ġdifferent Ġrepresentations . ĠNote Ġthat Ġthe Ġplot Ġbelow Ġdoes Ġnot Ġtake Ġinto Ġaccount Ġfurther Ġlex ical Ġand Ġother Ġdiver g ences Ġbetween Ġsource Ġand Ġassisting Ġlanguages , Ġsince Ġwe Ġdemonstrated Ġword Ġorder Ġdivergence Ġwith Ġthe Ġsame Ġlanguage Ġon Ġthe Ġsource Ġside . </s> To Ġaddress Ġthis Ġword Ġorder Ġdivergence , Ġwe Ġpropose Ġto Ġpre - order Ġthe Ġassisting Ġlanguage Ġsentences Ġto Ġmatch Ġthe Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage . ĠWe Ġconsider Ġan Ġextremely Ġresource Ġconstrained Ġscenario , Ġwhere Ġwe Ġdo Ġnot Ġhave Ġany Ġparallel Ġcorpus Ġfor Ġthe Ġchild Ġtask . ĠWe Ġare Ġlimited Ġto Ġa Ġbilingual Ġdictionary Ġfor Ġtransfer Ġinformation Ġfrom Ġthe Ġassisting Ġto Ġthe Ġsource Ġlanguage . ĠFrom Ġour Ġexperiments , Ġwe Ġshow Ġthat Ġthere Ġis Ġa Ġsignificant Ġincrease Ġin Ġthe Ġtranslation Ġaccuracy Ġfor Ġthe Ġunseen Ġsource - target Ġlanguage Ġpair . </s> Add ressing ĠLex ical ĠDiver gence </s> BI B REF 3 Ġexplored Ġtransfer Ġlearning Ġfor ĠN MT Ġon Ġlow - resource Ġlanguages . ĠThey Ġstudied Ġthe Ġinfluence Ġof Ġlanguage Ġdivergence Ġbetween Ġlanguages Ġchosen Ġfor Ġtraining Ġthe Ġparent Ġand Ġchild Ġmodel , Ġand Ġshowed Ġthat Ġchoosing Ġsimilar Ġlanguages Ġfor Ġtraining Ġthe Ġparent Ġand Ġchild Ġmodel Ġleads Ġto Ġbetter Ġimprovements Ġfrom Ġtransfer Ġlearning . ĠA Ġlimitation Ġof ĠB IB REF 3 Ġapproach Ġis Ġthat Ġthey Ġignore Ġthe Ġlex ical Ġsimilarity Ġbetween Ġlanguages Ġand Ġalso Ġthe Ġsource Ġlanguage Ġembed d ings Ġare Ġrandomly Ġinitialized . ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġtake Ġadvantage Ġof Ġlex ical Ġsimilarity Ġbetween Ġlanguages Ġin Ġtheir Ġwork . ĠB IB REF 10 Ġproposed Ġto Ġuse ĠByte - P air ĠEnc oding Ġ( B PE ) Ġto Ġrepresent Ġthe Ġsentences Ġin Ġboth Ġthe Ġparent Ġand Ġthe Ġchild Ġlanguage Ġto Ġovercome Ġthe Ġabove Ġlimitation . ĠThey Ġshow Ġusing ĠB PE Ġbenefits Ġtransfer Ġlearning Ġespecially Ġwhen Ġthe Ġinvolved Ġlanguages Ġare Ġclosely - related Ġagg l ut in ative Ġlanguages . ĠSimilarly , ĠB IB REF 11 Ġutilize Ġlex ical Ġsimilarity Ġbetween Ġthe Ġsource Ġand Ġassisting Ġlanguages Ġby Ġtraining Ġa Ġcharacter - level ĠN MT Ġsystem . ĠB IB REF 12 Ġaddress Ġlex ical Ġdivergence Ġby Ġusing Ġbilingual Ġembed d ings Ġand Ġmixture Ġof Ġuniversal Ġtoken Ġembed d ings . ĠOne Ġof Ġthe Ġlanguages ' Ġvocabulary , Ġusually ĠEnglish Ġvocabulary Ġis Ġconsidered Ġas Ġuniversal Ġtokens Ġand Ġevery Ġword Ġin Ġthe Ġother Ġlanguages Ġis Ġrepresented Ġas Ġa Ġmixture Ġof Ġuniversal Ġtokens . ĠThey Ġshow Ġresults Ġon Ġextremely Ġlow - resource Ġlanguages . </s> Add ressing ĠWord ĠOrder ĠDiver gence </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġwork Ġhas Ġaddressed Ġword Ġorder Ġdivergence Ġin Ġtransfer Ġlearning Ġfor Ġmult ilingual ĠN MT . ĠHowever , Ġsome Ġwork Ġexists Ġfor Ġother ĠN LP Ġtasks Ġthat Ġcould Ġpotentially Ġaddress Ġword Ġorder . ĠFor ĠNamed ĠEntity ĠRecogn ition Ġ( NER ), ĠB IB REF 14 Ġuse Ġa Ġself - att ention Ġlayer Ġafter Ġthe ĠBi - L ST M Ġlayer Ġto Ġaddress Ġword - order Ġdivergence Ġfor ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġtask . ĠThe Ġapproach Ġdoes Ġnot Ġshow Ġany Ġsignificant Ġimprovements Ġover Ġmultiple Ġlanguages . ĠA Ġpossible Ġreason Ġis Ġthat Ġthe Ġdivergence Ġhas Ġto Ġbe Ġaddressed Ġbefore / during Ġconstruction Ġof Ġthe Ġcontextual Ġembed d ings Ġin Ġthe ĠBi - L ST M Ġlayer , Ġand Ġthe Ġsubsequent Ġself - att ention Ġlayer Ġdoes Ġnot Ġaddress Ġword - order Ġdivergence . ĠB IB REF 15 Ġuse Ġadvers arial Ġtraining Ġfor Ġcross - ling ual Ġquestion - question Ġsimilarity Ġranking Ġin Ġcommunity Ġquestion Ġanswering . ĠThe Ġadvers arial Ġtraining Ġtries Ġto Ġforce Ġthe Ġenc oder Ġrepresentations Ġof Ġsimilar Ġsentences Ġfrom Ġdifferent Ġinput Ġlanguages Ġto Ġhave Ġsimilar Ġrepresentations . </s> Use Ġof ĠPre - ordering </s> Pre - ordering Ġthe Ġsource Ġlanguage Ġsentences Ġto Ġmatch Ġthe Ġtarget Ġlanguage Ġword Ġorder Ġhas Ġbeen Ġuseful Ġin Ġaddressing Ġword - order Ġdivergence Ġfor ĠPh rase - Based ĠSM T ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠRecently , ĠB IB REF 20 Ġproposed Ġa Ġway Ġto Ġmeasure Ġand Ġreduce Ġthe Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġtarget Ġlanguages Ġbased Ġon Ġmorph ological Ġand Ġsynt actic Ġproperties , Ġalso Ġtermed Ġas Ġan is omorph ism . ĠThey Ġdemonstrated Ġthat Ġby Ġreducing Ġthe Ġan is omorph ism Ġbetween Ġthe Ġsource Ġand Ġtarget Ġlanguages , Ġconsistent Ġimprovements Ġin ĠN MT Ġperformance Ġwere Ġobtained . ĠThe ĠN MT Ġsystem Ġused Ġadditional Ġfeatures Ġlike Ġword Ġforms , ĠPOS Ġtags Ġand Ġdependency Ġrelations Ġin Ġaddition Ġto Ġparallel Ġcorpor a . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 21 Ġobserved Ġa Ġdrop Ġin Ġperformance Ġdue Ġto Ġpre - ordering Ġfor ĠN MT . ĠUnlike ĠB IB REF 20 Ġ, Ġthe ĠN MT Ġsystem Ġwas Ġtrained Ġon Ġpre - ordered Ġsentences Ġand Ġno Ġadditional Ġfeatures Ġwere Ġprovided Ġto Ġthe Ġsystem . ĠNote Ġthat Ġall Ġthese Ġworks Ġaddress Ġsource - target Ġdivergence , Ġnot Ġdivergence Ġbetween Ġsource Ġlanguages Ġin Ġmult ilingual ĠN MT . </s> Prop osed ĠSolution </s> Consider Ġthe Ġtask Ġof Ġtranslating Ġfrom Ġan Ġextremely Ġlow - resource Ġlanguage Ġ( source ) Ġto Ġa Ġtarget Ġlanguage . ĠThe Ġparallel Ġcorpus Ġbetween Ġthe Ġtwo Ġlanguages Ġif Ġavailable Ġmay Ġbe Ġtoo Ġsmall Ġto Ġtrain Ġa ĠN MT Ġmodel . ĠSimilar Ġto Ġexisting Ġworks ĠB IB REF 3 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 12 Ġ, Ġwe Ġuse Ġtransfer Ġlearning Ġto Ġovercome Ġdata Ġsp arsity Ġand Ġtrain Ġa ĠN MT Ġmodel Ġbetween Ġthe Ġsource Ġand Ġthe Ġtarget Ġlanguages . ĠSpecifically , Ġthe ĠN MT Ġmodel Ġ( parent Ġmodel ) Ġis Ġtrained Ġon Ġthe Ġassisting Ġlanguage Ġand Ġtarget Ġlanguage Ġpairs . ĠWe Ġchoose ĠEnglish Ġas Ġthe Ġassisting Ġlanguage Ġin Ġall Ġour Ġexperiments . ĠIn Ġour Ġresource - scar ce Ġscenario , Ġwe Ġhave Ġno Ġparallel Ġcorpus Ġfor Ġthe Ġchild Ġtask . ĠHence , Ġat Ġtest Ġtime , Ġthe Ġsource Ġlanguage Ġsentence Ġis Ġtranslated Ġusing Ġthe Ġparent Ġmodel Ġafter Ġperforming Ġa Ġword - by - word Ġtranslation Ġinto Ġthe Ġassisting Ġlanguage . </s> Since Ġthe Ġsource Ġlanguage Ġand Ġthe Ġassisting Ġlanguage Ġ( English ) Ġhave Ġdifferent Ġword Ġorder , Ġwe Ġhypothes ize Ġthat Ġit Ġleads Ġto Ġinconsistencies Ġin Ġthe Ġcontextual Ġrepresentations Ġgenerated Ġby Ġthe Ġenc oder Ġfor Ġthe Ġtwo Ġlanguages . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġpre - order ĠEnglish Ġsentences Ġ( ass isting Ġlanguage Ġsentences ) Ġto Ġmatch Ġthe Ġword - order Ġof Ġthe Ġsource Ġlanguage Ġand Ġtrain Ġthe Ġparent Ġmodel Ġon Ġthis Ġpre - ordered Ġcorpus . ĠIn Ġour Ġexperiments , Ġwe Ġlook Ġat Ġscenarios Ġwhere Ġthe Ġassisting Ġlanguage Ġhas ĠSV O Ġword Ġorder Ġand Ġthe Ġsource Ġlanguage Ġhas ĠS OV Ġword Ġorder . </s> For Ġinstance , Ġconsider Ġthe ĠEnglish Ġsentence ĠAn ur ag Ġwill Ġmeet ĠTh ak ur . ĠOne Ġof Ġthe Ġpre - ordering Ġrule Ġswaps Ġthe Ġposition Ġof Ġthe Ġnoun Ġphrase Ġfollowed Ġby Ġa Ġtrans itive Ġverb Ġwith Ġthe Ġtrans itive Ġverb . ĠThe Ġoriginal Ġand Ġthe Ġresulting Ġre - ordered Ġparse Ġtree Ġwill Ġbe Ġas Ġshown Ġin Ġthe ĠTable ĠT AB REF 5 Ġ. ĠApp lying Ġthis Ġre ordering Ġrule Ġto Ġthe Ġabove Ġsentence ĠAn ur ag Ġwill Ġmeet ĠTh ak ur Ġwill Ġyield Ġthe Ġre ordered Ġsentence ĠAn ur ag ĠTh ak ur Ġwill Ġmeet . ĠAdditionally , Ġthe ĠTable ĠT AB REF 5 Ġshows Ġthe Ġparse Ġtrees Ġfor Ġthe Ġabove Ġsentence Ġwith Ġand Ġwithout Ġpre - ordering . </s> Pre - ordering Ġshould Ġalso Ġbe Ġbeneficial Ġfor Ġother Ġword Ġorder Ġdivergence Ġscenarios Ġ( e . g ., ĠS OV Ġto ĠSV O ), Ġbut Ġwe Ġleave Ġverification Ġof Ġthese Ġadditional Ġscenarios Ġfor Ġfuture Ġwork . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġlanguages Ġexperimented Ġwith , Ġdatasets Ġused , Ġthe Ġnetwork Ġhyper - param eters Ġused Ġin Ġour Ġexperiments . </s> L anguages </s> We Ġexperimented Ġwith ĠEnglish ĠIN LINE FORM 0 ĠHindi Ġtranslation Ġas Ġthe Ġparent Ġtask . ĠEnglish Ġis Ġthe Ġassisting Ġsource Ġlanguage . ĠBeng ali , ĠGujar ati , ĠMar athi , ĠMal ay alam Ġand ĠTamil Ġare Ġthe Ġprimary Ġsource Ġlanguages , Ġand Ġtranslation Ġfrom Ġthese Ġto ĠHindi Ġconstitute Ġthe Ġchild Ġtasks . ĠHindi , ĠBeng ali , ĠGujar ati Ġand ĠMar athi Ġare ĠIndo - A ryan Ġlanguages , Ġwhile ĠMal ay alam Ġand ĠTamil Ġare ĠD rav idian Ġlanguages . ĠAll Ġthese Ġlanguages Ġhave Ġa Ġcanonical ĠS OV Ġword Ġorder . </s> Dat as ets </s> For Ġtraining ĠEnglish - H ind i ĠN MT Ġsystems , Ġwe Ġuse Ġthe ĠI IT B ĠEnglish - H ind i Ġparallel Ġcorpus ĠB IB REF 22 Ġ( ĠIN LINE FORM 0 Ġsentences Ġfrom Ġthe Ġtraining Ġset ) Ġand Ġthe ĠI LC I ĠEnglish - H ind i Ġparallel Ġcorpus Ġ( ĠIN LINE FORM 1 Ġsentences ). ĠThe ĠI LC I Ġ( Indian ĠLanguage ĠCorpor a ĠInitiative ) Ġmult ilingual Ġparallel Ġcorpus ĠB IB REF 23 Ġspans Ġmultiple ĠIndian Ġlanguages Ġfrom Ġthe Ġhealth Ġand Ġtourism Ġdomains . ĠWe Ġuse Ġthe Ġ520 - sent ence Ġdev - set Ġof Ġthe ĠI IT B Ġparallel Ġcorpus Ġfor Ġvalidation . ĠFor Ġeach Ġchild Ġtask , Ġwe Ġuse ĠIN LINE FORM 2 Ġsentences Ġfrom ĠI LC I Ġcorpus Ġas Ġthe Ġtest Ġset . </s> Network </s> We Ġuse ĠOpen N MT - Tor ch ĠB IB REF 24 Ġto Ġtrain Ġthe ĠN MT Ġsystem . ĠWe Ġuse Ġthe Ġstandard Ġsequence - to - sequence Ġarchitecture Ġwith Ġattention ĠB IB REF 13 Ġ. ĠWe Ġuse Ġan Ġenc oder Ġwhich Ġcontains Ġtwo Ġlayers Ġof Ġbid irection al ĠL ST Ms Ġwith Ġ500 Ġneurons Ġeach . ĠThe Ġdec oder Ġcontains Ġtwo ĠL ST M Ġlayers Ġwith Ġ500 Ġneurons Ġeach . ĠInput Ġfeeding Ġapproach ĠB IB REF 1 Ġis Ġused Ġwhere Ġthe Ġprevious Ġattention Ġhidden Ġstate Ġis Ġfed Ġas Ġinput Ġto Ġthe Ġdec oder ĠL ST M . ĠWe Ġuse Ġa Ġmini - batch Ġof Ġsize Ġ50 Ġand Ġuse Ġa Ġdrop out Ġlayer . ĠWe Ġbegin Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġand Ġdecay Ġthe Ġlearning Ġrate Ġby Ġa Ġfactor Ġof ĠIN LINE FORM 1 Ġwhen Ġthe Ġperplex ity Ġon Ġvalidation Ġset Ġincreases . ĠThe Ġtraining Ġis Ġstopped Ġwhen Ġthe Ġlearning Ġrate Ġfalls Ġbelow ĠIN LINE FORM 2 Ġor Ġnumber Ġof Ġepoch s = 22 . ĠThe ĠEnglish Ġinput Ġis Ġinitialized Ġwith Ġpre - trained Ġembed d ings Ġtrained Ġusing Ġfast Text ĠB IB REF 25 Ġ. </s> English Ġvocabulary Ġconsists Ġof ĠIN LINE FORM 0 Ġtokens Ġappearing Ġat Ġleast Ġ2 Ġtimes Ġin Ġthe ĠEnglish Ġtraining Ġcorpus . ĠFor Ġconstructing Ġthe ĠHindi Ġvocabulary Ġwe Ġconsidered Ġonly Ġthose Ġtokens Ġappearing Ġat Ġleast Ġ5 Ġtimes Ġin Ġthe Ġtraining Ġsplit Ġresulting Ġin Ġa Ġvocabulary Ġsize Ġof ĠIN LINE FORM 1 Ġtokens . ĠFor Ġrepresenting ĠEnglish Ġand Ġother Ġsource Ġlanguages Ġinto Ġa Ġcommon Ġspace , Ġwe Ġtranslate Ġeach Ġword Ġin Ġthe Ġsource Ġlanguage Ġinto ĠEnglish Ġusing Ġa Ġbilingual Ġdictionary Ġ( Google ĠTrans late Ġword Ġtranslation Ġin Ġour Ġcase ). ĠIn Ġan Ġend - to - end Ġsolution , Ġit Ġwould Ġhave Ġbeen Ġideal Ġto Ġuse Ġbilingual Ġembed d ings Ġor Ġobtain Ġword - by - word Ġtranslations Ġvia Ġbilingual Ġembed d ings ĠB IB REF 14 Ġ. ĠBut , Ġthe Ġquality Ġof Ġpublicly Ġavailable Ġbilingual Ġembed d ings Ġfor ĠEnglish - Indian Ġlanguages Ġis Ġvery Ġlow Ġfor Ġobtaining Ġgood - quality , Ġbilingual Ġrepresentations ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠWe Ġalso Ġfound Ġthat Ġthese Ġembed d ings Ġwere Ġnot Ġuseful Ġfor Ġtransfer Ġlearning . </s> We Ġuse Ġthe ĠCF IL T - pre order Ġsystem Ġfor Ġre ordering ĠEnglish Ġsentences Ġto Ġmatch Ġthe ĠIndian Ġlanguage Ġword Ġorder . ĠIt Ġcontains Ġtwo Ġre - ordering Ġsystems : Ġ( 1 ) Ġgeneric Ġrules Ġthat Ġapply Ġto Ġall ĠIndian Ġlanguages ĠB IB REF 17 Ġ, Ġand Ġ( 2 ) Ġhind i - tun ed Ġrules Ġwhich Ġimprove Ġthe Ġgeneric Ġrules Ġby Ġincorporating Ġimprovements Ġfound Ġthrough Ġan Ġerror Ġanalysis Ġof ĠEnglish - H ind i Ġre ordering ĠB IB REF 28 Ġ. ĠThese ĠHindi - tun ed Ġrules Ġhave Ġbeen Ġfound Ġto Ġimprove Ġre ordering Ġfor Ġmany ĠEnglish Ġto ĠIndian Ġlanguage Ġpairs ĠB IB REF 29 Ġ. </s> Results </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġresults Ġfrom Ġour Ġexperiments Ġon ĠN MT Ġtask . ĠWe Ġreport Ġthe Ġresults Ġon ĠX - H ind i Ġpair , Ġwhere ĠX Ġis Ġone Ġof ĠBeng ali , ĠGujar ati , ĠMar athi , ĠTamil , Ġand ĠMal ay alam . ĠThe Ġresults Ġare Ġpresented Ġin Ġthe ĠTable ĠT AB REF 6 Ġ. ĠWe Ġreport ĠB LE U Ġscores Ġand ĠLe BLE U Ġscores ĠB IB REF 30 Ġ. ĠWe Ġobserve Ġthat Ġboth Ġthe Ġpre - ordering Ġconfigurations Ġsignificantly Ġimprove Ġthe ĠB LE U Ġscores Ġover Ġthe Ġbaseline Ġscores . ĠWe Ġobserve Ġlarger Ġgains Ġwhen Ġgeneric Ġpre - ordering Ġrules Ġare Ġused Ġcompared Ġto Ġthe ĠHindi - tun ed Ġpre - ordering Ġrules . </s> These Ġresults Ġsupport Ġour Ġhypothesis Ġthat Ġword - order Ġdivergence Ġcan Ġlimit Ġthe Ġbenefits Ġof Ġmult ilingual Ġtranslation . ĠRed ucing Ġthe Ġword Ġorder Ġdivergence Ġcan Ġimprove Ġtranslation Ġin Ġextremely Ġlow - resource Ġscenarios . </s> An Ġanalysis Ġof Ġthe Ġoutputs Ġrevealed Ġthat Ġpre - ordering Ġsignificantly Ġreducing Ġthe Ġnumber Ġof ĠUN K Ġtokens Ġ( place holder Ġfor Ġunknown Ġwords ) Ġin Ġthe Ġtest Ġoutput Ġ( Table ĠT AB REF 14 Ġ). ĠWe Ġhypothes ize Ġthat Ġdue Ġto Ġword Ġorder Ġdivergence Ġbetween ĠEnglish Ġand ĠIndian Ġlanguages , Ġthe Ġenc oder Ġrepresentation Ġgenerated Ġis Ġnot Ġconsistent Ġleading Ġto Ġdec oder Ġgenerating Ġunknown Ġwords . ĠHowever , Ġthe Ġpre - ordered Ġmodels Ġgenerate Ġbetter Ġcontextual Ġrepresentations Ġleading Ġto Ġless Ġnumber Ġof Ġunknown Ġtokens Ġand Ġbetter Ġtranslation Ġwhich Ġis Ġalso Ġreflected Ġin Ġthe ĠB LE U Ġscores . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshow Ġthat Ġhandling Ġword - order Ġdivergence Ġbetween Ġsource Ġand Ġassisting Ġlanguages Ġis Ġcrucial Ġfor Ġthe Ġsuccess Ġof Ġmult ilingual ĠN MT Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWe Ġshow Ġthat Ġpre - ordering Ġthe Ġassisting Ġlanguage Ġto Ġmatch Ġthe Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage Ġsignificantly Ġimproves Ġtranslation Ġquality Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWhile Ġthe Ġcurrent Ġwork Ġfocused Ġon ĠIndian Ġlanguages , Ġwe Ġwould Ġlike Ġto Ġvalidate Ġthe Ġhypothesis Ġon Ġa Ġmore Ġdiverse Ġset Ġof Ġlanguages .
Introduction Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task. However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3 . Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages. Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8 . This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task. Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10 . The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task). Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model. The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available. The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning. Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9 . Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12 . However, the effect of word order divergence and its mitigation has not been explored. In a practical setting, it is not uncommon to have source and assisting languages with different word order. For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages. Hence, it is natural to use English as an assisting language for inter-Indian language translation. To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13 . The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state. The contextual representation is word-order dependent. Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent. The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages. This makes it difficult to transfer knowledge learned from the assisting language to the source language. We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali). Figure FIGREF1 shows that the encoder representations obtained are very different. The attention network and the decoder now have to work with very different representations. Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side. To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language. We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task. We are limited to a bilingual dictionary for transfer information from the assisting to the source language. From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair. Addressing Lexical Divergence BIBREF3 explored transfer learning for NMT on low-resource languages. They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning. A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized. BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work. BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation. They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages. Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system. BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings. One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens. They show results on extremely low-resource languages. Addressing Word Order Divergence To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT. However, some work exists for other NLP tasks that could potentially address word order. For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task. The approach does not show any significant improvements over multiple languages. A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence. BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering. The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations. Use of Pre-ordering Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 . Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism. They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained. The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora. On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT. Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system. Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT. Proposed Solution Consider the task of translating from an extremely low-resource language (source) to a target language. The parallel corpus between the two languages if available may be too small to train a NMT model. Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages. Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs. We choose English as the assisting language in all our experiments. In our resource-scarce scenario, we have no parallel corpus for the child task. Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language. Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages. In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus. In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order. For instance, consider the English sentence Anurag will meet Thakur. One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb. The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5 . Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet. Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering. Pre-ordering should also be beneficial for other word order divergence scenarios (e.g., SOV to SVO), but we leave verification of these additional scenarios for future work. Experimental Setup In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments. Languages We experimented with English INLINEFORM0 Hindi translation as the parent task. English is the assisting source language. Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks. Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages. All these languages have a canonical SOV word order. Datasets For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences). The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains. We use the 520-sentence dev-set of the IITB parallel corpus for validation. For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set. Network We use OpenNMT-Torch BIBREF24 to train the NMT system. We use the standard sequence-to-sequence architecture with attention BIBREF13 . We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each. The decoder contains two LSTM layers with 500 neurons each. Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM. We use a mini-batch of size 50 and use a dropout layer. We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases. The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22. The English input is initialized with pre-trained embeddings trained using fastText BIBREF25 . English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus. For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens. For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case). In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14 . But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27 . We also found that these embeddings were not useful for transfer learning. We use the CFILT-preorder system for reordering English sentences to match the Indian language word order. It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28 . These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29 . Results In this section, we describe the results from our experiments on NMT task. We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam. The results are presented in the Table TABREF6 . We report BLEU scores and LeBLEU scores BIBREF30 . We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores. We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules. These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation. Reducing the word order divergence can improve translation in extremely low-resource scenarios. An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 ). We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words. However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores. Conclusion In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting. We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting. While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages.
On how many language pairs do they show that preordering assisting language sentences helps translation quality?
37861be6aecd9242c4fdccdfcd06e48f3f1f8f81
On Ġhow Ġmany Ġlanguage Ġpairs Ġdo Ġthey Ġshow Ġthat Ġpre ordering Ġassisting Ġlanguage Ġsentences Ġhelps Ġtranslation Ġquality ?
5
<s> On Ġhow Ġmany Ġlanguage Ġpairs Ġdo Ġthey Ġshow Ġthat Ġpre ordering Ġassisting Ġlanguage Ġsentences Ġhelps Ġtranslation Ġquality ? </s> Introduction </s> Deep ĠLearning Ġapproaches Ġhave Ġachieved Ġimpressive Ġresults Ġon Ġvarious ĠN LP Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġand Ġhave Ġbecome Ġthe Ġde Ġfacto Ġapproach Ġfor Ġany ĠN LP Ġtask . ĠHowever , Ġthese Ġdeep Ġlearning Ġtechniques Ġhave Ġfound Ġto Ġbe Ġless Ġeffective Ġfor Ġlow - resource Ġlanguages Ġwhen Ġthe Ġavailable Ġtraining Ġdata Ġis Ġvery Ġless ĠB IB REF 3 Ġ. ĠRecently , Ġseveral Ġapproaches Ġlike ĠMulti - task Ġlearning ĠB IB REF 4 Ġ, Ġmult ilingual Ġlearning ĠB IB REF 5 Ġ, Ġsemi - super vised Ġlearning ĠB IB REF 2 Ġ, ĠB IB REF 6 Ġand Ġtransfer Ġlearning ĠB IB REF 7 Ġ, ĠB IB REF 3 Ġhave Ġbeen Ġexplored Ġby Ġthe Ġdeep Ġlearning Ġcommunity Ġto Ġovercome Ġdata Ġsp arsity Ġin Ġlow - resource Ġlanguages . ĠTransfer Ġlearning Ġtrains Ġa Ġmodel Ġfor Ġa Ġparent Ġtask Ġand Ġfine - t unes Ġthe Ġlearned Ġparent Ġmodel Ġweights Ġ( features ) Ġfor Ġa Ġrelated Ġchild Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠThis Ġeffectively Ġreduces Ġthe Ġrequirement Ġon Ġtraining Ġdata Ġfor Ġthe Ġchild Ġtask Ġas Ġthe Ġmodel Ġwould Ġhave Ġlearned Ġrelevant Ġfeatures Ġfrom Ġthe Ġparent Ġtask Ġdata Ġthereby , Ġimproving Ġthe Ġperformance Ġon Ġthe Ġchild Ġtask . </s> Transfer Ġlearning Ġhas Ġalso Ġbeen Ġexplored Ġin Ġthe Ġmult ilingual ĠNeural ĠMachine ĠTranslation ĠB IB REF 3 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠThe Ġgoal Ġis Ġto Ġimprove Ġthe ĠN MT Ġperformance Ġon Ġthe Ġsource Ġto Ġtarget Ġlanguage Ġpair Ġ( child Ġtask ) Ġusing Ġan Ġassisting Ġsource Ġlanguage Ġ( ass isting Ġto Ġtarget Ġtranslation Ġis Ġthe Ġparent Ġtask ). ĠHere , Ġthe Ġparent Ġmodel Ġis Ġtrained Ġon Ġthe Ġassisting Ġand Ġtarget Ġlanguage Ġparallel Ġcorpus Ġand Ġthe Ġtrained Ġweights Ġare Ġused Ġto Ġinitialize Ġthe Ġchild Ġmodel . ĠThe Ġchild Ġmodel Ġcan Ġnow Ġbe Ġfine - tun ed Ġon Ġthe Ġsource - target Ġlanguage Ġpairs , Ġif Ġparallel Ġcorpus Ġis Ġavailable . ĠThe Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġthe Ġassisting Ġlanguage Ġcan Ġadversely Ġimpact Ġthe Ġbenefits Ġobtained Ġfrom Ġtransfer Ġlearning . ĠMultiple Ġstudies Ġhave Ġshown Ġthat Ġtransfer Ġlearning Ġworks Ġbest Ġwhen Ġthe Ġlanguages Ġare Ġrelated ĠB IB REF 3 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 9 Ġ. ĠSeveral Ġstudies Ġhave Ġtried Ġto Ġaddress Ġlex ical Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġthe Ġtarget Ġlanguages ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠHowever , Ġthe Ġeffect Ġof Ġword Ġorder Ġdivergence Ġand Ġits Ġmitigation Ġhas Ġnot Ġbeen Ġexplored . ĠIn Ġa Ġpractical Ġsetting , Ġit Ġis Ġnot Ġuncommon Ġto Ġhave Ġsource Ġand Ġassisting Ġlanguages Ġwith Ġdifferent Ġword Ġorder . ĠFor Ġinstance , Ġit Ġis Ġpossible Ġto Ġfind Ġparallel Ġcorpor a Ġbetween ĠEnglish Ġand Ġsome ĠIndian Ġlanguages , Ġbut Ġvery Ġlittle Ġparallel Ġcorpor a Ġbetween ĠIndian Ġlanguages . ĠHence , Ġit Ġis Ġnatural Ġto Ġuse ĠEnglish Ġas Ġan Ġassisting Ġlanguage Ġfor Ġinter - Indian Ġlanguage Ġtranslation . </s> To Ġsee Ġhow Ġword Ġorder Ġdivergence Ġcan Ġbe Ġdetrimental , Ġlet Ġus Ġconsider Ġthe Ġcase Ġof Ġthe Ġstandard ĠR NN Ġ( Bi - L ST M ) Ġenc oder - att ention - dec oder Ġarchitecture ĠB IB REF 13 Ġ. ĠThe Ġenc oder Ġgenerates Ġcontextual Ġrepresentations Ġ( ann otation Ġvectors ) Ġfor Ġeach Ġsource Ġword , Ġwhich Ġare Ġused Ġby Ġthe Ġattention Ġnetwork Ġto Ġmatch Ġthe Ġsource Ġwords Ġto Ġthe Ġcurrent Ġdec oder Ġstate . ĠThe Ġcontextual Ġrepresentation Ġis Ġword - order Ġdependent . ĠHence , Ġif Ġthe Ġassisting Ġand Ġthe Ġsource Ġlanguages Ġdo Ġnot Ġhave Ġsimilar Ġword Ġorder Ġthe Ġgenerated Ġcontextual Ġrepresentations Ġwill Ġnot Ġbe Ġconsistent . ĠThe Ġattention Ġnetwork Ġ( and Ġhence Ġthe Ġdec oder ) Ġsees Ġdifferent Ġcontextual Ġrepresentations Ġfor Ġsimilar Ġwords Ġin Ġparallel Ġsentences Ġacross Ġdifferent Ġlanguages . ĠThis Ġmakes Ġit Ġdifficult Ġto Ġtransfer Ġknowledge Ġlearned Ġfrom Ġthe Ġassisting Ġlanguage Ġto Ġthe Ġsource Ġlanguage . </s> We Ġillustrate Ġthis Ġby Ġvisual izing Ġthe Ġcontextual Ġrepresentations Ġgenerated Ġby Ġthe Ġenc oder Ġof Ġan ĠEnglish Ġto ĠHindi ĠN MT Ġsystem Ġfor Ġtwo Ġversions Ġof Ġthe ĠEnglish Ġinput : Ġ( a ) Ġoriginal Ġword Ġorder Ġ( S VO ) Ġ( b ) Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage Ġ( S OV , Ġfor ĠBeng ali ). ĠFigure ĠFIG REF 1 Ġshows Ġthat Ġthe Ġenc oder Ġrepresentations Ġobtained Ġare Ġvery Ġdifferent . ĠThe Ġattention Ġnetwork Ġand Ġthe Ġdec oder Ġnow Ġhave Ġto Ġwork Ġwith Ġvery Ġdifferent Ġrepresentations . ĠNote Ġthat Ġthe Ġplot Ġbelow Ġdoes Ġnot Ġtake Ġinto Ġaccount Ġfurther Ġlex ical Ġand Ġother Ġdiver g ences Ġbetween Ġsource Ġand Ġassisting Ġlanguages , Ġsince Ġwe Ġdemonstrated Ġword Ġorder Ġdivergence Ġwith Ġthe Ġsame Ġlanguage Ġon Ġthe Ġsource Ġside . </s> To Ġaddress Ġthis Ġword Ġorder Ġdivergence , Ġwe Ġpropose Ġto Ġpre - order Ġthe Ġassisting Ġlanguage Ġsentences Ġto Ġmatch Ġthe Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage . ĠWe Ġconsider Ġan Ġextremely Ġresource Ġconstrained Ġscenario , Ġwhere Ġwe Ġdo Ġnot Ġhave Ġany Ġparallel Ġcorpus Ġfor Ġthe Ġchild Ġtask . ĠWe Ġare Ġlimited Ġto Ġa Ġbilingual Ġdictionary Ġfor Ġtransfer Ġinformation Ġfrom Ġthe Ġassisting Ġto Ġthe Ġsource Ġlanguage . ĠFrom Ġour Ġexperiments , Ġwe Ġshow Ġthat Ġthere Ġis Ġa Ġsignificant Ġincrease Ġin Ġthe Ġtranslation Ġaccuracy Ġfor Ġthe Ġunseen Ġsource - target Ġlanguage Ġpair . </s> Add ressing ĠLex ical ĠDiver gence </s> BI B REF 3 Ġexplored Ġtransfer Ġlearning Ġfor ĠN MT Ġon Ġlow - resource Ġlanguages . ĠThey Ġstudied Ġthe Ġinfluence Ġof Ġlanguage Ġdivergence Ġbetween Ġlanguages Ġchosen Ġfor Ġtraining Ġthe Ġparent Ġand Ġchild Ġmodel , Ġand Ġshowed Ġthat Ġchoosing Ġsimilar Ġlanguages Ġfor Ġtraining Ġthe Ġparent Ġand Ġchild Ġmodel Ġleads Ġto Ġbetter Ġimprovements Ġfrom Ġtransfer Ġlearning . ĠA Ġlimitation Ġof ĠB IB REF 3 Ġapproach Ġis Ġthat Ġthey Ġignore Ġthe Ġlex ical Ġsimilarity Ġbetween Ġlanguages Ġand Ġalso Ġthe Ġsource Ġlanguage Ġembed d ings Ġare Ġrandomly Ġinitialized . ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġtake Ġadvantage Ġof Ġlex ical Ġsimilarity Ġbetween Ġlanguages Ġin Ġtheir Ġwork . ĠB IB REF 10 Ġproposed Ġto Ġuse ĠByte - P air ĠEnc oding Ġ( B PE ) Ġto Ġrepresent Ġthe Ġsentences Ġin Ġboth Ġthe Ġparent Ġand Ġthe Ġchild Ġlanguage Ġto Ġovercome Ġthe Ġabove Ġlimitation . ĠThey Ġshow Ġusing ĠB PE Ġbenefits Ġtransfer Ġlearning Ġespecially Ġwhen Ġthe Ġinvolved Ġlanguages Ġare Ġclosely - related Ġagg l ut in ative Ġlanguages . ĠSimilarly , ĠB IB REF 11 Ġutilize Ġlex ical Ġsimilarity Ġbetween Ġthe Ġsource Ġand Ġassisting Ġlanguages Ġby Ġtraining Ġa Ġcharacter - level ĠN MT Ġsystem . ĠB IB REF 12 Ġaddress Ġlex ical Ġdivergence Ġby Ġusing Ġbilingual Ġembed d ings Ġand Ġmixture Ġof Ġuniversal Ġtoken Ġembed d ings . ĠOne Ġof Ġthe Ġlanguages ' Ġvocabulary , Ġusually ĠEnglish Ġvocabulary Ġis Ġconsidered Ġas Ġuniversal Ġtokens Ġand Ġevery Ġword Ġin Ġthe Ġother Ġlanguages Ġis Ġrepresented Ġas Ġa Ġmixture Ġof Ġuniversal Ġtokens . ĠThey Ġshow Ġresults Ġon Ġextremely Ġlow - resource Ġlanguages . </s> Add ressing ĠWord ĠOrder ĠDiver gence </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġwork Ġhas Ġaddressed Ġword Ġorder Ġdivergence Ġin Ġtransfer Ġlearning Ġfor Ġmult ilingual ĠN MT . ĠHowever , Ġsome Ġwork Ġexists Ġfor Ġother ĠN LP Ġtasks Ġthat Ġcould Ġpotentially Ġaddress Ġword Ġorder . ĠFor ĠNamed ĠEntity ĠRecogn ition Ġ( NER ), ĠB IB REF 14 Ġuse Ġa Ġself - att ention Ġlayer Ġafter Ġthe ĠBi - L ST M Ġlayer Ġto Ġaddress Ġword - order Ġdivergence Ġfor ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġtask . ĠThe Ġapproach Ġdoes Ġnot Ġshow Ġany Ġsignificant Ġimprovements Ġover Ġmultiple Ġlanguages . ĠA Ġpossible Ġreason Ġis Ġthat Ġthe Ġdivergence Ġhas Ġto Ġbe Ġaddressed Ġbefore / during Ġconstruction Ġof Ġthe Ġcontextual Ġembed d ings Ġin Ġthe ĠBi - L ST M Ġlayer , Ġand Ġthe Ġsubsequent Ġself - att ention Ġlayer Ġdoes Ġnot Ġaddress Ġword - order Ġdivergence . ĠB IB REF 15 Ġuse Ġadvers arial Ġtraining Ġfor Ġcross - ling ual Ġquestion - question Ġsimilarity Ġranking Ġin Ġcommunity Ġquestion Ġanswering . ĠThe Ġadvers arial Ġtraining Ġtries Ġto Ġforce Ġthe Ġenc oder Ġrepresentations Ġof Ġsimilar Ġsentences Ġfrom Ġdifferent Ġinput Ġlanguages Ġto Ġhave Ġsimilar Ġrepresentations . </s> Use Ġof ĠPre - ordering </s> Pre - ordering Ġthe Ġsource Ġlanguage Ġsentences Ġto Ġmatch Ġthe Ġtarget Ġlanguage Ġword Ġorder Ġhas Ġbeen Ġuseful Ġin Ġaddressing Ġword - order Ġdivergence Ġfor ĠPh rase - Based ĠSM T ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠRecently , ĠB IB REF 20 Ġproposed Ġa Ġway Ġto Ġmeasure Ġand Ġreduce Ġthe Ġdivergence Ġbetween Ġthe Ġsource Ġand Ġtarget Ġlanguages Ġbased Ġon Ġmorph ological Ġand Ġsynt actic Ġproperties , Ġalso Ġtermed Ġas Ġan is omorph ism . ĠThey Ġdemonstrated Ġthat Ġby Ġreducing Ġthe Ġan is omorph ism Ġbetween Ġthe Ġsource Ġand Ġtarget Ġlanguages , Ġconsistent Ġimprovements Ġin ĠN MT Ġperformance Ġwere Ġobtained . ĠThe ĠN MT Ġsystem Ġused Ġadditional Ġfeatures Ġlike Ġword Ġforms , ĠPOS Ġtags Ġand Ġdependency Ġrelations Ġin Ġaddition Ġto Ġparallel Ġcorpor a . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 21 Ġobserved Ġa Ġdrop Ġin Ġperformance Ġdue Ġto Ġpre - ordering Ġfor ĠN MT . ĠUnlike ĠB IB REF 20 Ġ, Ġthe ĠN MT Ġsystem Ġwas Ġtrained Ġon Ġpre - ordered Ġsentences Ġand Ġno Ġadditional Ġfeatures Ġwere Ġprovided Ġto Ġthe Ġsystem . ĠNote Ġthat Ġall Ġthese Ġworks Ġaddress Ġsource - target Ġdivergence , Ġnot Ġdivergence Ġbetween Ġsource Ġlanguages Ġin Ġmult ilingual ĠN MT . </s> Prop osed ĠSolution </s> Consider Ġthe Ġtask Ġof Ġtranslating Ġfrom Ġan Ġextremely Ġlow - resource Ġlanguage Ġ( source ) Ġto Ġa Ġtarget Ġlanguage . ĠThe Ġparallel Ġcorpus Ġbetween Ġthe Ġtwo Ġlanguages Ġif Ġavailable Ġmay Ġbe Ġtoo Ġsmall Ġto Ġtrain Ġa ĠN MT Ġmodel . ĠSimilar Ġto Ġexisting Ġworks ĠB IB REF 3 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 12 Ġ, Ġwe Ġuse Ġtransfer Ġlearning Ġto Ġovercome Ġdata Ġsp arsity Ġand Ġtrain Ġa ĠN MT Ġmodel Ġbetween Ġthe Ġsource Ġand Ġthe Ġtarget Ġlanguages . ĠSpecifically , Ġthe ĠN MT Ġmodel Ġ( parent Ġmodel ) Ġis Ġtrained Ġon Ġthe Ġassisting Ġlanguage Ġand Ġtarget Ġlanguage Ġpairs . ĠWe Ġchoose ĠEnglish Ġas Ġthe Ġassisting Ġlanguage Ġin Ġall Ġour Ġexperiments . ĠIn Ġour Ġresource - scar ce Ġscenario , Ġwe Ġhave Ġno Ġparallel Ġcorpus Ġfor Ġthe Ġchild Ġtask . ĠHence , Ġat Ġtest Ġtime , Ġthe Ġsource Ġlanguage Ġsentence Ġis Ġtranslated Ġusing Ġthe Ġparent Ġmodel Ġafter Ġperforming Ġa Ġword - by - word Ġtranslation Ġinto Ġthe Ġassisting Ġlanguage . </s> Since Ġthe Ġsource Ġlanguage Ġand Ġthe Ġassisting Ġlanguage Ġ( English ) Ġhave Ġdifferent Ġword Ġorder , Ġwe Ġhypothes ize Ġthat Ġit Ġleads Ġto Ġinconsistencies Ġin Ġthe Ġcontextual Ġrepresentations Ġgenerated Ġby Ġthe Ġenc oder Ġfor Ġthe Ġtwo Ġlanguages . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġpre - order ĠEnglish Ġsentences Ġ( ass isting Ġlanguage Ġsentences ) Ġto Ġmatch Ġthe Ġword - order Ġof Ġthe Ġsource Ġlanguage Ġand Ġtrain Ġthe Ġparent Ġmodel Ġon Ġthis Ġpre - ordered Ġcorpus . ĠIn Ġour Ġexperiments , Ġwe Ġlook Ġat Ġscenarios Ġwhere Ġthe Ġassisting Ġlanguage Ġhas ĠSV O Ġword Ġorder Ġand Ġthe Ġsource Ġlanguage Ġhas ĠS OV Ġword Ġorder . </s> For Ġinstance , Ġconsider Ġthe ĠEnglish Ġsentence ĠAn ur ag Ġwill Ġmeet ĠTh ak ur . ĠOne Ġof Ġthe Ġpre - ordering Ġrule Ġswaps Ġthe Ġposition Ġof Ġthe Ġnoun Ġphrase Ġfollowed Ġby Ġa Ġtrans itive Ġverb Ġwith Ġthe Ġtrans itive Ġverb . ĠThe Ġoriginal Ġand Ġthe Ġresulting Ġre - ordered Ġparse Ġtree Ġwill Ġbe Ġas Ġshown Ġin Ġthe ĠTable ĠT AB REF 5 Ġ. ĠApp lying Ġthis Ġre ordering Ġrule Ġto Ġthe Ġabove Ġsentence ĠAn ur ag Ġwill Ġmeet ĠTh ak ur Ġwill Ġyield Ġthe Ġre ordered Ġsentence ĠAn ur ag ĠTh ak ur Ġwill Ġmeet . ĠAdditionally , Ġthe ĠTable ĠT AB REF 5 Ġshows Ġthe Ġparse Ġtrees Ġfor Ġthe Ġabove Ġsentence Ġwith Ġand Ġwithout Ġpre - ordering . </s> Pre - ordering Ġshould Ġalso Ġbe Ġbeneficial Ġfor Ġother Ġword Ġorder Ġdivergence Ġscenarios Ġ( e . g ., ĠS OV Ġto ĠSV O ), Ġbut Ġwe Ġleave Ġverification Ġof Ġthese Ġadditional Ġscenarios Ġfor Ġfuture Ġwork . </s> Exper imental ĠSetup </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġlanguages Ġexperimented Ġwith , Ġdatasets Ġused , Ġthe Ġnetwork Ġhyper - param eters Ġused Ġin Ġour Ġexperiments . </s> L anguages </s> We Ġexperimented Ġwith ĠEnglish ĠIN LINE FORM 0 ĠHindi Ġtranslation Ġas Ġthe Ġparent Ġtask . ĠEnglish Ġis Ġthe Ġassisting Ġsource Ġlanguage . ĠBeng ali , ĠGujar ati , ĠMar athi , ĠMal ay alam Ġand ĠTamil Ġare Ġthe Ġprimary Ġsource Ġlanguages , Ġand Ġtranslation Ġfrom Ġthese Ġto ĠHindi Ġconstitute Ġthe Ġchild Ġtasks . ĠHindi , ĠBeng ali , ĠGujar ati Ġand ĠMar athi Ġare ĠIndo - A ryan Ġlanguages , Ġwhile ĠMal ay alam Ġand ĠTamil Ġare ĠD rav idian Ġlanguages . ĠAll Ġthese Ġlanguages Ġhave Ġa Ġcanonical ĠS OV Ġword Ġorder . </s> Dat as ets </s> For Ġtraining ĠEnglish - H ind i ĠN MT Ġsystems , Ġwe Ġuse Ġthe ĠI IT B ĠEnglish - H ind i Ġparallel Ġcorpus ĠB IB REF 22 Ġ( ĠIN LINE FORM 0 Ġsentences Ġfrom Ġthe Ġtraining Ġset ) Ġand Ġthe ĠI LC I ĠEnglish - H ind i Ġparallel Ġcorpus Ġ( ĠIN LINE FORM 1 Ġsentences ). ĠThe ĠI LC I Ġ( Indian ĠLanguage ĠCorpor a ĠInitiative ) Ġmult ilingual Ġparallel Ġcorpus ĠB IB REF 23 Ġspans Ġmultiple ĠIndian Ġlanguages Ġfrom Ġthe Ġhealth Ġand Ġtourism Ġdomains . ĠWe Ġuse Ġthe Ġ520 - sent ence Ġdev - set Ġof Ġthe ĠI IT B Ġparallel Ġcorpus Ġfor Ġvalidation . ĠFor Ġeach Ġchild Ġtask , Ġwe Ġuse ĠIN LINE FORM 2 Ġsentences Ġfrom ĠI LC I Ġcorpus Ġas Ġthe Ġtest Ġset . </s> Network </s> We Ġuse ĠOpen N MT - Tor ch ĠB IB REF 24 Ġto Ġtrain Ġthe ĠN MT Ġsystem . ĠWe Ġuse Ġthe Ġstandard Ġsequence - to - sequence Ġarchitecture Ġwith Ġattention ĠB IB REF 13 Ġ. ĠWe Ġuse Ġan Ġenc oder Ġwhich Ġcontains Ġtwo Ġlayers Ġof Ġbid irection al ĠL ST Ms Ġwith Ġ500 Ġneurons Ġeach . ĠThe Ġdec oder Ġcontains Ġtwo ĠL ST M Ġlayers Ġwith Ġ500 Ġneurons Ġeach . ĠInput Ġfeeding Ġapproach ĠB IB REF 1 Ġis Ġused Ġwhere Ġthe Ġprevious Ġattention Ġhidden Ġstate Ġis Ġfed Ġas Ġinput Ġto Ġthe Ġdec oder ĠL ST M . ĠWe Ġuse Ġa Ġmini - batch Ġof Ġsize Ġ50 Ġand Ġuse Ġa Ġdrop out Ġlayer . ĠWe Ġbegin Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġand Ġdecay Ġthe Ġlearning Ġrate Ġby Ġa Ġfactor Ġof ĠIN LINE FORM 1 Ġwhen Ġthe Ġperplex ity Ġon Ġvalidation Ġset Ġincreases . ĠThe Ġtraining Ġis Ġstopped Ġwhen Ġthe Ġlearning Ġrate Ġfalls Ġbelow ĠIN LINE FORM 2 Ġor Ġnumber Ġof Ġepoch s = 22 . ĠThe ĠEnglish Ġinput Ġis Ġinitialized Ġwith Ġpre - trained Ġembed d ings Ġtrained Ġusing Ġfast Text ĠB IB REF 25 Ġ. </s> English Ġvocabulary Ġconsists Ġof ĠIN LINE FORM 0 Ġtokens Ġappearing Ġat Ġleast Ġ2 Ġtimes Ġin Ġthe ĠEnglish Ġtraining Ġcorpus . ĠFor Ġconstructing Ġthe ĠHindi Ġvocabulary Ġwe Ġconsidered Ġonly Ġthose Ġtokens Ġappearing Ġat Ġleast Ġ5 Ġtimes Ġin Ġthe Ġtraining Ġsplit Ġresulting Ġin Ġa Ġvocabulary Ġsize Ġof ĠIN LINE FORM 1 Ġtokens . ĠFor Ġrepresenting ĠEnglish Ġand Ġother Ġsource Ġlanguages Ġinto Ġa Ġcommon Ġspace , Ġwe Ġtranslate Ġeach Ġword Ġin Ġthe Ġsource Ġlanguage Ġinto ĠEnglish Ġusing Ġa Ġbilingual Ġdictionary Ġ( Google ĠTrans late Ġword Ġtranslation Ġin Ġour Ġcase ). ĠIn Ġan Ġend - to - end Ġsolution , Ġit Ġwould Ġhave Ġbeen Ġideal Ġto Ġuse Ġbilingual Ġembed d ings Ġor Ġobtain Ġword - by - word Ġtranslations Ġvia Ġbilingual Ġembed d ings ĠB IB REF 14 Ġ. ĠBut , Ġthe Ġquality Ġof Ġpublicly Ġavailable Ġbilingual Ġembed d ings Ġfor ĠEnglish - Indian Ġlanguages Ġis Ġvery Ġlow Ġfor Ġobtaining Ġgood - quality , Ġbilingual Ġrepresentations ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠWe Ġalso Ġfound Ġthat Ġthese Ġembed d ings Ġwere Ġnot Ġuseful Ġfor Ġtransfer Ġlearning . </s> We Ġuse Ġthe ĠCF IL T - pre order Ġsystem Ġfor Ġre ordering ĠEnglish Ġsentences Ġto Ġmatch Ġthe ĠIndian Ġlanguage Ġword Ġorder . ĠIt Ġcontains Ġtwo Ġre - ordering Ġsystems : Ġ( 1 ) Ġgeneric Ġrules Ġthat Ġapply Ġto Ġall ĠIndian Ġlanguages ĠB IB REF 17 Ġ, Ġand Ġ( 2 ) Ġhind i - tun ed Ġrules Ġwhich Ġimprove Ġthe Ġgeneric Ġrules Ġby Ġincorporating Ġimprovements Ġfound Ġthrough Ġan Ġerror Ġanalysis Ġof ĠEnglish - H ind i Ġre ordering ĠB IB REF 28 Ġ. ĠThese ĠHindi - tun ed Ġrules Ġhave Ġbeen Ġfound Ġto Ġimprove Ġre ordering Ġfor Ġmany ĠEnglish Ġto ĠIndian Ġlanguage Ġpairs ĠB IB REF 29 Ġ. </s> Results </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġresults Ġfrom Ġour Ġexperiments Ġon ĠN MT Ġtask . ĠWe Ġreport Ġthe Ġresults Ġon ĠX - H ind i Ġpair , Ġwhere ĠX Ġis Ġone Ġof ĠBeng ali , ĠGujar ati , ĠMar athi , ĠTamil , Ġand ĠMal ay alam . ĠThe Ġresults Ġare Ġpresented Ġin Ġthe ĠTable ĠT AB REF 6 Ġ. ĠWe Ġreport ĠB LE U Ġscores Ġand ĠLe BLE U Ġscores ĠB IB REF 30 Ġ. ĠWe Ġobserve Ġthat Ġboth Ġthe Ġpre - ordering Ġconfigurations Ġsignificantly Ġimprove Ġthe ĠB LE U Ġscores Ġover Ġthe Ġbaseline Ġscores . ĠWe Ġobserve Ġlarger Ġgains Ġwhen Ġgeneric Ġpre - ordering Ġrules Ġare Ġused Ġcompared Ġto Ġthe ĠHindi - tun ed Ġpre - ordering Ġrules . </s> These Ġresults Ġsupport Ġour Ġhypothesis Ġthat Ġword - order Ġdivergence Ġcan Ġlimit Ġthe Ġbenefits Ġof Ġmult ilingual Ġtranslation . ĠRed ucing Ġthe Ġword Ġorder Ġdivergence Ġcan Ġimprove Ġtranslation Ġin Ġextremely Ġlow - resource Ġscenarios . </s> An Ġanalysis Ġof Ġthe Ġoutputs Ġrevealed Ġthat Ġpre - ordering Ġsignificantly Ġreducing Ġthe Ġnumber Ġof ĠUN K Ġtokens Ġ( place holder Ġfor Ġunknown Ġwords ) Ġin Ġthe Ġtest Ġoutput Ġ( Table ĠT AB REF 14 Ġ). ĠWe Ġhypothes ize Ġthat Ġdue Ġto Ġword Ġorder Ġdivergence Ġbetween ĠEnglish Ġand ĠIndian Ġlanguages , Ġthe Ġenc oder Ġrepresentation Ġgenerated Ġis Ġnot Ġconsistent Ġleading Ġto Ġdec oder Ġgenerating Ġunknown Ġwords . ĠHowever , Ġthe Ġpre - ordered Ġmodels Ġgenerate Ġbetter Ġcontextual Ġrepresentations Ġleading Ġto Ġless Ġnumber Ġof Ġunknown Ġtokens Ġand Ġbetter Ġtranslation Ġwhich Ġis Ġalso Ġreflected Ġin Ġthe ĠB LE U Ġscores . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġshow Ġthat Ġhandling Ġword - order Ġdivergence Ġbetween Ġsource Ġand Ġassisting Ġlanguages Ġis Ġcrucial Ġfor Ġthe Ġsuccess Ġof Ġmult ilingual ĠN MT Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWe Ġshow Ġthat Ġpre - ordering Ġthe Ġassisting Ġlanguage Ġto Ġmatch Ġthe Ġword Ġorder Ġof Ġthe Ġsource Ġlanguage Ġsignificantly Ġimproves Ġtranslation Ġquality Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWhile Ġthe Ġcurrent Ġwork Ġfocused Ġon ĠIndian Ġlanguages , Ġwe Ġwould Ġlike Ġto Ġvalidate Ġthe Ġhypothesis Ġon Ġa Ġmore Ġdiverse Ġset Ġof Ġlanguages .