BIBREF19, BIBREF20
EXTRACTIVE
multilingual NMT (MNMT) BIBREF19
EXTRACTIVE
Table TABREF19 and TABREF26 report zero-shot results on Europarl and Multi-UN evaluation sets, respectively. We compare our approaches with related approaches of pivoting, multilingual NMT (MNMT) BIBREF19, and cross-lingual transfer without pretraining BIBREF16. The results show that our approaches consistently outperform other approaches across languages and datasets, especially surpass pivoting, which is a strong baseline in the zero-shot scenario that multilingual NMT systems often fail to beat BIBREF19, BIBREF20, BIBREF23. Pivoting translates source to pivot then to target in two steps, causing inefficient translation process. Our approaches use one encoder-decoder model to translate between any zero-shot directions, which is more efficient than pivoting. Regarding the comparison between transfer approaches, our cross-lingual pretraining based transfer outperforms transfer method that does not use pretraining by a large margin.
Table TABREF19 and TABREF26 report zero-shot results on Europarl and Multi-UN evaluation sets, respectively. We compare our approaches with related approaches of pivoting, multilingual NMT (MNMT) BIBREF19, and cross-lingual transfer without pretraining BIBREF16. The results show that our approaches consistently outperform other approaches across languages and datasets, especially surpass pivoting, which is a strong baseline in the zero-shot scenario that multilingual NMT systems often fail to beat BIBREF19, BIBREF20, BIBREF23. Pivoting translates source to pivot then to target in two steps, causing inefficient translation process. Our approaches use one encoder-decoder model to translate between any zero-shot directions, which is more efficient than pivoting. Regarding the comparison between transfer approaches, our cross-lingual pretraining based transfer outperforms transfer method that does not use pretraining by a large margin.
1912.01214
Introduction </s> Although ĠNeural ĠMachine ĠTranslation Ġ( N MT ) Ġhas Ġdominated Ġrecent Ġresearch Ġon Ġtranslation Ġtasks ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠN MT Ġheavily Ġrelies Ġon Ġlarge - scale Ġparallel Ġdata , Ġresulting Ġin Ġpoor Ġperformance Ġon Ġlow - resource Ġor Ġzero - resource Ġlanguage Ġpairs ĠB IB REF 3 . ĠTranslation Ġbetween Ġthese Ġlow - resource Ġlanguages Ġ( e . g ., ĠArabic $ \ right arrow Ġ$ Spanish ) Ġis Ġusually Ġaccomplished Ġwith Ġpiv oting Ġthrough Ġa Ġrich - resource Ġlanguage Ġ( such Ġas ĠEnglish ), Ġi . e ., ĠArabic Ġ( source ) Ġsentence Ġis Ġtranslated Ġto ĠEnglish Ġ( p ivot ) Ġfirst Ġwhich Ġis Ġlater Ġtranslated Ġto ĠSpanish Ġ( target ) ĠB IB REF 4 , ĠB IB REF 5 . ĠHowever , Ġthe Ġpivot - based Ġmethod Ġrequires Ġdoubled Ġdecoding Ġtime Ġand Ġsuffers Ġfrom Ġthe Ġpropagation Ġof Ġtranslation Ġerrors . </s> One Ġcommon Ġalternative Ġto Ġavoid Ġpiv oting Ġin ĠN MT Ġis Ġtransfer Ġlearning ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 Ġwhich Ġlever ages Ġa Ġhigh - resource Ġpivot $ \ right arrow Ġ$ target Ġmodel Ġ( parent ) Ġto Ġinitialize Ġa Ġlow - resource Ġsource $ \ right arrow Ġ$ target Ġmodel Ġ( child ) Ġthat Ġis Ġfurther Ġoptimized Ġwith Ġa Ġsmall Ġamount Ġof Ġavailable Ġparallel Ġdata . ĠAlthough Ġthis Ġapproach Ġhas Ġachieved Ġsuccess Ġin Ġsome Ġlow - resource Ġlanguage Ġpairs , Ġit Ġstill Ġperforms Ġvery Ġpoorly Ġin Ġextremely Ġlow - resource Ġor Ġzero - resource Ġtranslation Ġscenario . ĠSpecifically , ĠB IB REF 8 Ġreports Ġthat Ġwithout Ġany Ġchild Ġmodel Ġtraining Ġdata , Ġthe Ġperformance Ġof Ġthe Ġparent Ġmodel Ġon Ġthe Ġchild Ġtest Ġset Ġis Ġmiserable . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġthe Ġlanguage Ġspace Ġmismatch Ġproblem , Ġalso Ġnamed Ġdomain Ġshift Ġproblem ĠB IB REF 10 , Ġbrings Ġabout Ġthe Ġzero - shot Ġtranslation Ġfailure Ġin Ġtransfer Ġlearning . ĠIt Ġis Ġbecause Ġtransfer Ġlearning Ġhas Ġno Ġexplicit Ġtraining Ġprocess Ġto Ġguarantee Ġthat Ġthe Ġsource Ġand Ġpivot Ġlanguages Ġshare Ġthe Ġsame Ġfeature Ġdistributions , Ġcausing Ġthat Ġthe Ġchild Ġmodel Ġinherited Ġfrom Ġthe Ġparent Ġmodel Ġfails Ġin Ġsuch Ġa Ġsituation . ĠFor Ġinstance , Ġas Ġillustrated Ġin Ġthe Ġleft Ġof ĠFigure ĠFIG REF 1 , Ġthe Ġpoints Ġof Ġthe Ġsentence Ġpair Ġwith Ġthe Ġsame Ġsemantics Ġare Ġnot Ġoverlapping Ġin Ġsource Ġspace , Ġresulting Ġin Ġthat Ġthe Ġshared Ġdec oder Ġwill Ġgenerate Ġdifferent Ġtranslations Ġden oted Ġby Ġdifferent Ġpoints Ġin Ġtarget Ġspace . ĠActually , Ġtransfer Ġlearning Ġfor ĠN MT Ġcan Ġbe Ġviewed Ġas Ġa Ġmulti - domain Ġproblem Ġwhere Ġeach Ġsource Ġlanguage Ġforms Ġa Ġnew Ġdomain . ĠMin im izing Ġthe Ġdiscrepancy Ġbetween Ġthe Ġfeature Ġdistributions Ġof Ġdifferent Ġsource Ġlanguages , Ġi . e ., Ġdifferent Ġdomains , Ġwill Ġensure Ġthe Ġsmooth Ġtransition Ġbetween Ġthe Ġparent Ġand Ġchild Ġmodels , Ġas Ġshown Ġin Ġthe Ġright Ġof ĠFigure ĠFIG REF 1 . ĠOne Ġway Ġto Ġachieve Ġthis Ġgoal Ġis Ġthe Ġfine - tun ing Ġtechnique , Ġwhich Ġforces Ġthe Ġmodel Ġto Ġforget Ġthe Ġspecific Ġknowledge Ġfrom Ġparent Ġdata Ġand Ġlearn Ġnew Ġfeatures Ġfrom Ġchild Ġdata . ĠHowever , Ġthe Ġdomain Ġshift Ġproblem Ġstill Ġexists , Ġand Ġthe Ġdemand Ġof Ġparallel Ġchild Ġdata Ġfor Ġfine - tun ing Ġheavily Ġhind ers Ġtransfer Ġlearning Ġfor ĠN MT Ġtowards Ġthe Ġzero - resource Ġsetting . </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe Ġtransfer Ġlearning Ġin Ġa Ġcommon Ġzero - shot Ġscenario Ġwhere Ġthere Ġare Ġa Ġlot Ġof Ġsource $ \ left right arrow Ġ$ p ivot Ġand Ġpivot $ \ left right arrow Ġ$ target Ġparallel Ġdata Ġbut Ġno Ġsource $ \ left right arrow Ġ$ target Ġparallel Ġdata . ĠIn Ġthis Ġscenario , Ġwe Ġpropose Ġa Ġsimple Ġbut Ġeffective Ġtransfer Ġapproach , Ġthe Ġkey Ġidea Ġof Ġwhich Ġis Ġto Ġrelieve Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem Ġby Ġmeans Ġof Ġcross - ling ual Ġpre - training . ĠTo Ġthis Ġend , Ġwe Ġfirst ly Ġinvestigate Ġthe Ġperformance Ġof Ġtwo Ġexisting Ġcross - ling ual Ġpre - training Ġmethods Ġproposed Ġby ĠB IB REF 11 Ġin Ġzero - shot Ġtranslation Ġscenario . ĠBesides , Ġa Ġnovel Ġpre - training Ġmethod Ġcalled ĠBR idge ĠLanguage ĠModel ing Ġ( B RL M ) Ġis Ġdesigned Ġto Ġmake Ġfull Ġuse Ġof Ġthe Ġsource $ \ left right arrow Ġ$ p ivot Ġbilingual Ġdata Ġto Ġobtain Ġa Ġuniversal Ġenc oder Ġfor Ġdifferent Ġlanguages . ĠOnce Ġthe Ġuniversal Ġenc oder Ġis Ġconstructed , Ġwe Ġonly Ġneed Ġto Ġtrain Ġthe Ġpivot $ \ right arrow Ġ$ target Ġmodel Ġand Ġthen Ġtest Ġthis Ġmodel Ġin Ġsource $ \ right arrow Ġ$ target Ġdirection Ġdirectly . ĠThe Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> We Ġpropose Ġa Ġnew Ġtransfer Ġlearning Ġapproach Ġfor ĠN MT Ġwhich Ġuses Ġthe Ġcross - ling ual Ġlanguage Ġmodel Ġpre - training Ġto Ġenable Ġa Ġhigh Ġperformance Ġon Ġzero - shot Ġtranslation . </s> We Ġpropose Ġa Ġnovel Ġpre - training Ġmethod Ġcalled ĠB RL M , Ġwhich Ġcan Ġeffectively Ġallev iates Ġthe Ġdistance Ġbetween Ġdifferent Ġsource Ġlanguage Ġspaces . </s> Our Ġproposed Ġapproach Ġsignificantly Ġimproves Ġzero - shot Ġtranslation Ġperformance , Ġconsistently Ġsurpass ing Ġpiv oting Ġand Ġmult ilingual Ġapproaches . ĠMeanwhile , Ġthe Ġperformance Ġon Ġsupervised Ġtranslation Ġdirection Ġremains Ġthe Ġsame Ġlevel Ġor Ġeven Ġbetter Ġwhen Ġusing Ġour Ġmethod . </s> Related ĠWork </s> In Ġrecent Ġyears , Ġzero - shot Ġtranslation Ġin ĠN MT Ġhas Ġattracted Ġwidespread Ġattention Ġin Ġacademic Ġresearch . ĠEx isting Ġmethods Ġare Ġmainly Ġdivided Ġinto Ġfour Ġcategories : Ġpivot - based Ġmethod , Ġtransfer Ġlearning , Ġmult ilingual ĠN MT , Ġand Ġun super vised ĠN MT . </s> P ivot - based ĠMethod Ġis Ġa Ġcommon Ġstrategy Ġto Ġobtain Ġa Ġsource $ \ right arrow Ġ$ target Ġmodel Ġby Ġintroducing Ġa Ġpivot Ġlanguage . ĠThis Ġapproach Ġis Ġfurther Ġdivided Ġinto Ġpiv oting Ġand Ġpivot - sy nt hetic . ĠWhile Ġthe Ġformer Ġfirst ly Ġtranslates Ġa Ġsource Ġlanguage Ġinto Ġthe Ġpivot Ġlanguage Ġwhich Ġis Ġlater Ġtranslated Ġto Ġthe Ġtarget Ġlanguage ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 12 , Ġthe Ġlatter Ġtrains Ġa Ġsource $ \ right arrow Ġ$ target Ġmodel Ġwith Ġpseudo Ġdata Ġgenerated Ġfrom Ġsource - p ivot Ġor Ġpivot - target Ġparallel Ġdata ĠB IB REF 13 , ĠB IB REF 14 . ĠAlthough Ġthe Ġpivot - based Ġmethods Ġcan Ġachieve Ġnot Ġbad Ġperformance , Ġit Ġalways Ġfalls Ġinto Ġa Ġcomputation - expensive Ġand Ġparameter - v ast Ġdilemma Ġof Ġquad r atic Ġgrowth Ġin Ġthe Ġnumber Ġof Ġsource Ġlanguages , Ġand Ġsuffers Ġfrom Ġthe Ġerror Ġpropagation Ġproblem ĠB IB REF 15 . </s> Transfer ĠLearning Ġis Ġfirst ly Ġintroduced Ġfor ĠN MT Ġby ĠB IB REF 6 , Ġwhich Ġlever ages Ġa Ġhigh - resource Ġparent Ġmodel Ġto Ġinitialize Ġthe Ġlow - resource Ġchild Ġmodel . ĠOn Ġthis Ġbasis , ĠB IB REF 7 Ġand ĠB IB REF 8 Ġuse Ġshared Ġvoc ab ular ies Ġfor Ġsource / target Ġlanguage Ġto Ġimprove Ġtransfer Ġlearning , Ġwhile ĠB IB REF 16 Ġrelieve Ġthe Ġvocabulary Ġmismatch Ġby Ġmainly Ġusing Ġcross - ling ual Ġword Ġembed ding . ĠAlthough Ġthese Ġmethods Ġare Ġsuccessful Ġin Ġthe Ġlow - resource Ġscene , Ġthey Ġhave Ġlimited Ġeffects Ġin Ġzero - shot Ġtranslation . </s> Mult ilingual ĠN MT Ġ( MN MT ) Ġenables Ġtraining Ġa Ġsingle Ġmodel Ġthat Ġsupports Ġtranslation Ġfrom Ġmultiple Ġsource Ġlanguages Ġinto Ġmultiple Ġtarget Ġlanguages , Ġeven Ġthose Ġunseen Ġlanguage Ġpairs ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠAside Ġfrom Ġsimpler Ġdeployment , ĠMN MT Ġbenefits Ġfrom Ġtransfer Ġlearning Ġwhere Ġlow - resource Ġlanguage Ġpairs Ġare Ġtrained Ġtogether Ġwith Ġhigh - resource Ġones . ĠHowever , ĠB IB REF 22 Ġpoint Ġout Ġthat ĠMN MT Ġfor Ġzero - shot Ġtranslation Ġeasily Ġfails , Ġand Ġis Ġsensitive Ġto Ġthe Ġhyper - param eter Ġsetting . ĠAlso , ĠMN MT Ġusually Ġperforms Ġworse Ġthan Ġthe Ġpivot - based Ġmethod Ġin Ġzero - shot Ġtranslation Ġsetting ĠB IB REF 23 . </s> Un super vised ĠN MT Ġ( UN MT ) Ġconsiders Ġa Ġharder Ġsetting , Ġin Ġwhich Ġonly Ġlarge - scale Ġmon oling ual Ġcorpor a Ġare Ġavailable Ġfor Ġtraining . ĠRecently , Ġmany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof ĠUN MT , Ġincluding Ġusing Ġden o ising Ġauto - enc oder , Ġstatistic Ġmachine Ġtranslation Ġ( SM T ) Ġand Ġun super vised Ġpre - training ĠB IB REF 24 , ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 11 . ĠSince ĠUN MT Ġperforms Ġwell Ġbetween Ġsimilar Ġlanguages Ġ( e . g ., ĠEnglish - German Ġtranslation ), Ġits Ġperformance Ġbetween Ġdistant Ġlanguages Ġis Ġstill Ġfar Ġfrom Ġexpectation . </s> Our Ġproposed Ġmethod Ġbelongs Ġto Ġthe Ġtransfer Ġlearning , Ġbut Ġit Ġis Ġdifferent Ġfrom Ġtraditional Ġtransfer Ġmethods Ġwhich Ġtrain Ġa Ġparent Ġmodel Ġas Ġstarting Ġpoint . ĠBefore Ġtraining Ġa Ġparent Ġmodel , Ġour Ġapproach Ġfully Ġlever ages Ġcross - ling ual Ġpre - training Ġmethods Ġto Ġmake Ġall Ġsource Ġlanguages Ġshare Ġthe Ġsame Ġfeature Ġspace Ġand Ġthus Ġenables Ġa Ġsmooth Ġtransition Ġfor Ġzero - shot Ġtranslation . </s> App roach </s> In Ġthis Ġsection , Ġwe Ġwill Ġpresent Ġa Ġcross - ling ual Ġpre - training Ġbased Ġtransfer Ġapproach . ĠThis Ġmethod Ġis Ġdesigned Ġfor Ġa Ġcommon Ġzero - shot Ġscenario Ġwhere Ġthere Ġare Ġa Ġlot Ġof Ġsource $ \ left right arrow Ġ$ p ivot Ġand Ġpivot $ \ left right arrow Ġ$ target Ġbilingual Ġdata Ġbut Ġno Ġsource $ \ left right arrow Ġ$ target Ġparallel Ġdata , Ġand Ġthe Ġwhole Ġtraining Ġprocess Ġcan Ġbe Ġsummarized Ġas Ġfollows Ġstep Ġby Ġstep : </s> Pre - train Ġa Ġuniversal Ġenc oder Ġwith Ġsource / p ivot Ġmon oling ual Ġor Ġsource $ \ left right arrow Ġ$ p ivot Ġbilingual Ġdata . </s> Train Ġa Ġpivot $ \ right arrow Ġ$ target Ġparent Ġmodel Ġbuilt Ġon Ġthe Ġpre - trained Ġuniversal Ġenc oder Ġwith Ġthe Ġavailable Ġparallel Ġdata . ĠDuring Ġthe Ġtraining Ġprocess , Ġwe Ġfreeze Ġseveral Ġlayers Ġof Ġthe Ġpre - trained Ġuniversal Ġenc oder Ġto Ġavoid Ġthe Ġdegener acy Ġissue ĠB IB REF 27 . </s> Direct ly Ġtranslate Ġsource Ġsentences Ġinto Ġtarget Ġsentences Ġwith Ġthe Ġparent Ġmodel , Ġwhich Ġbenefits Ġfrom Ġthe Ġavailability Ġof Ġthe Ġuniversal Ġenc oder . </s> The Ġkey Ġdifficulty Ġof Ġthis Ġmethod Ġis Ġto Ġensure Ġthe Ġintermediate Ġrepresentations Ġof Ġthe Ġuniversal Ġenc oder Ġare Ġlanguage Ġinvari ant . ĠIn Ġthe Ġrest Ġof Ġthis Ġsection , Ġwe Ġfirst Ġpresent Ġtwo Ġexisting Ġmethods Ġyet Ġto Ġbe Ġexplored Ġin Ġzero - shot Ġtranslation , Ġand Ġthen Ġpropose Ġa Ġstraightforward Ġbut Ġeffective Ġcross - ling ual Ġpre - training Ġmethod . ĠIn Ġthe Ġend , Ġwe Ġpresent Ġthe Ġwhole Ġtraining Ġand Ġinference Ġprotocol Ġfor Ġtransfer . </s> App roach Ġ: :: ĠMask ed Ġand ĠTranslation ĠLanguage ĠModel ĠPret raining </s> Two Ġexisting Ġcross - ling ual Ġpre - training Ġmethods , ĠMask ed ĠLanguage ĠModel ing Ġ( ML M ) Ġand ĠTranslation ĠLanguage ĠModel ing Ġ( TL M ), Ġhave Ġshown Ġtheir Ġeffectiveness Ġon ĠX N LI Ġcross - ling ual Ġclassification Ġtask ĠB IB REF 11 , ĠB IB REF 28 , Ġbut Ġthese Ġmethods Ġhave Ġnot Ġbeen Ġwell Ġstudied Ġon Ġcross - ling ual Ġgeneration Ġtasks Ġin Ġzero - shot Ġcondition . ĠWe Ġattempt Ġto Ġtake Ġadvantage Ġof Ġthe Ġcross - ling ual Ġability Ġof Ġthe Ġtwo Ġmethods Ġfor Ġzero - shot Ġtranslation . </s> Specifically , ĠML M Ġadop ts Ġthe ĠClo ze Ġobjective Ġof ĠB ERT ĠB IB REF 29 Ġand Ġpredicts Ġthe Ġmasked Ġwords Ġthat Ġare Ġrandomly Ġselected Ġand Ġreplaced Ġwith Ġ[ MAS K ] Ġtoken Ġon Ġmon oling ual Ġcorpus . ĠIn Ġpractice , ĠML M Ġtakes Ġdifferent Ġlanguage Ġmon oling ual Ġcorpor a Ġas Ġinput Ġto Ġfind Ġfeatures Ġshared Ġacross Ġdifferent Ġlanguages . ĠWith Ġthis Ġmethod , Ġword Ġpieces Ġshared Ġin Ġall Ġlanguages Ġhave Ġbeen Ġmapped Ġinto Ġa Ġshared Ġspace , Ġwhich Ġmakes Ġthe Ġsentence Ġrepresentations Ġacross Ġdifferent Ġlanguages Ġclose ĠB IB REF 30 . </s> Since ĠML M Ġobjective Ġis Ġun super vised Ġand Ġonly Ġrequires Ġmon oling ual Ġdata , ĠTL M Ġis Ġdesigned Ġto Ġleverage Ġparallel Ġdata Ġwhen Ġit Ġis Ġavailable . ĠActually , ĠTL M Ġis Ġa Ġsimple Ġextension Ġof ĠML M , Ġwith Ġthe Ġdifference Ġthat ĠTL M Ġconc aten ates Ġsentence Ġpair Ġinto Ġa Ġwhole Ġsentence , Ġand Ġthen Ġrandomly Ġmasks Ġwords Ġin Ġboth Ġthe Ġsource Ġand Ġtarget Ġsentences . ĠIn Ġthis Ġway , Ġthe Ġmodel Ġcan Ġeither Ġattend Ġto Ġsurrounding Ġwords Ġor Ġto Ġthe Ġtranslation Ġsentence , Ġimplicitly Ġencouraging Ġthe Ġmodel Ġto Ġalign Ġthe Ġsource Ġand Ġtarget Ġlanguage Ġrepresentations . ĠNote Ġthat Ġalthough Ġeach Ġsentence Ġpair Ġis Ġformed Ġinto Ġone Ġsentence , Ġthe Ġpositions Ġof Ġthe Ġtarget Ġsentence Ġare Ġreset Ġto Ġcount Ġform Ġzero . </s> App roach Ġ: :: ĠBridge ĠLanguage ĠModel ĠPret raining </s> Aside Ġfrom ĠML M Ġand ĠTL M , Ġwe Ġpropose ĠBR idge ĠLanguage ĠModel ing Ġ( B RL M ) Ġto Ġfurther Ġobtain Ġword - level Ġrepresentation Ġalignment Ġbetween Ġdifferent Ġlanguages . ĠThis Ġmethod Ġis Ġinspired Ġby Ġthe Ġassumption Ġthat Ġif Ġthe Ġfeature Ġspaces Ġof Ġdifferent Ġlanguages Ġare Ġaligned Ġvery Ġwell , Ġthe Ġmasked Ġwords Ġin Ġthe Ġcorrupted Ġsentence Ġcan Ġalso Ġbe Ġguessed Ġby Ġthe Ġcontext Ġof Ġthe Ġcorrespond ingly Ġaligned Ġwords Ġon Ġthe Ġother Ġside . ĠTo Ġachieve Ġthis Ġgoal , ĠB RL M Ġis Ġdesigned Ġto Ġstrengthen Ġthe Ġability Ġto Ġinfer Ġwords Ġacross Ġlanguages Ġbased Ġon Ġalignment Ġinformation , Ġinstead Ġof Ġinfer ring Ġwords Ġwithin Ġmon oling ual Ġsentence Ġas Ġin ĠML M Ġor Ġwithin Ġthe Ġpseudo Ġsentence Ġformed Ġby Ġconc aten ating Ġsentence Ġpair Ġas Ġin ĠTL M . </s> As Ġillustrated Ġin ĠFigure ĠFIG REF 9 , ĠB RL M Ġstacks Ġshared Ġenc oder Ġover Ġboth Ġside Ġsentences Ġseparately . ĠIn Ġparticular , Ġwe Ġdesign Ġtwo Ġnetwork Ġstructures Ġfor ĠB RL M , Ġwhich Ġare Ġdivided Ġinto ĠHard ĠAl ignment Ġ( B RL M - HA ) Ġand ĠSoft ĠAl ignment Ġ( B RL M - SA ) Ġaccording Ġto Ġthe Ġway Ġof Ġgenerating Ġthe Ġalignment Ġinformation . ĠThese Ġtwo Ġstructures Ġactually Ġextend ĠML M Ġinto Ġa Ġbilingual Ġscenario , Ġwith Ġthe Ġdifference Ġthat ĠB RL M Ġlever ages Ġexternal Ġalign er Ġtool Ġor Ġadditional Ġattention Ġlayer Ġto Ġexplicitly Ġintroduce Ġalignment Ġinformation Ġduring Ġmodel Ġtraining . </s> Hard ĠAl ignment Ġ( B RL M - HA ). ĠWe Ġfirst Ġuse Ġexternal Ġalign er Ġtool Ġon Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata Ġto Ġextract Ġthe Ġalignment Ġinformation Ġof Ġsentence Ġpair . ĠDuring Ġmodel Ġtraining , Ġgiven Ġsource $ \ left right arrow Ġ$ p ivot Ġsentence Ġpair , ĠB RL M - HA Ġrandomly Ġmasks Ġsome Ġwords Ġin Ġsource Ġsentence Ġand Ġlever ages Ġalignment Ġinformation Ġto Ġobtain Ġthe Ġaligned Ġwords Ġin Ġpivot Ġsentence Ġfor Ġmasked Ġwords . ĠBased Ġon Ġthe Ġprocessed Ġinput , ĠB RL M - HA Ġadop ts Ġthe ĠTrans former ĠB IB REF 1 Ġenc oder Ġto Ġgain Ġthe Ġhidden Ġstates Ġfor Ġsource Ġand Ġpivot Ġsentences Ġrespectively . ĠThen Ġthe Ġtraining Ġobjective Ġof ĠB RL M - HA Ġis Ġto Ġpredict Ġthe Ġmasked Ġwords Ġby Ġnot Ġonly Ġthe Ġsurrounding Ġwords Ġin Ġsource Ġsentence Ġbut Ġalso Ġthe Ġenc oder Ġoutputs Ġof Ġthe Ġaligned Ġwords . ĠNote Ġthat Ġthis Ġtraining Ġprocess Ġis Ġalso Ġcarried Ġout Ġin Ġa Ġsymm etric Ġsituation , Ġin Ġwhich Ġwe Ġmask Ġsome Ġwords Ġin Ġpivot Ġsentence Ġand Ġobtain Ġthe Ġaligned Ġwords Ġin Ġthe Ġsource Ġsentence . </s> Soft ĠAl ignment Ġ( B RL M - SA ). ĠInstead Ġof Ġusing Ġexternal Ġalign er Ġtool , ĠB RL M - SA Ġintroduces Ġan Ġadditional Ġattention Ġlayer Ġto Ġlearn Ġthe Ġalignment Ġinformation Ġtogether Ġwith Ġmodel Ġtraining . ĠIn Ġthis Ġway , ĠB RL M - SA Ġavoids Ġthe Ġeffect Ġcaused Ġby Ġexternal Ġwrong Ġalignment Ġinformation Ġand Ġenables Ġmany - to - one Ġsoft Ġalignment Ġduring Ġmodel Ġtraining . ĠSimilar Ġwith ĠB RL M - HA , Ġthe Ġtraining Ġobjective Ġof ĠB RL M - SA Ġis Ġto Ġpredict Ġthe Ġmasked Ġwords Ġby Ġnot Ġonly Ġthe Ġsurrounding Ġwords Ġin Ġsource Ġsentence Ġbut Ġalso Ġthe Ġoutputs Ġof Ġattention Ġlayer . ĠIn Ġour Ġimplementation , Ġthe Ġattention Ġlayer Ġis Ġa Ġmulti - head Ġattention Ġlayer Ġadopted Ġin ĠTrans former , Ġwhere Ġthe Ġqueries Ġcome Ġfrom Ġthe Ġmasked Ġsource Ġsentence , Ġthe Ġkeys Ġand Ġvalues Ġcome Ġfrom Ġthe Ġpivot Ġsentence . </s> In Ġprinciple , ĠML M Ġand ĠTL M Ġcan Ġlearn Ġsome Ġimplicit Ġalignment Ġinformation Ġduring Ġmodel Ġtraining . ĠHowever , Ġthe Ġalignment Ġprocess Ġin ĠML M Ġis Ġinefficient Ġsince Ġthe Ġshared Ġword Ġpieces Ġonly Ġaccount Ġfor Ġa Ġsmall Ġproportion Ġof Ġthe Ġwhole Ġcorpus , Ġresulting Ġin Ġthe Ġdifficulty Ġof Ġexpanding Ġthe Ġshared Ġinformation Ġto Ġalign Ġthe Ġwhole Ġcorpus . ĠTL M Ġalso Ġlacks Ġeffort Ġin Ġalignment Ġbetween Ġthe Ġsource Ġand Ġtarget Ġsentences Ġsince ĠTL M Ġconc aten ates Ġthe Ġsentence Ġpair Ġinto Ġone Ġsequence , Ġmaking Ġthe Ġexplicit Ġalignment Ġbetween Ġthe Ġsource Ġand Ġtarget Ġinf eas ible . ĠB RL M Ġfully Ġutilizes Ġthe Ġalignment Ġinformation Ġto Ġobtain Ġbetter Ġword - level Ġrepresentation Ġalignment Ġbetween Ġdifferent Ġlanguages , Ġwhich Ġbetter Ġrelie ves Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem . </s> App roach Ġ: :: ĠTransfer ĠProtocol </s> We Ġconsider Ġthe Ġtypical Ġzero - shot Ġtranslation Ġscenario Ġin Ġwhich Ġa Ġhigh Ġresource Ġpivot Ġlanguage Ġhas Ġparallel Ġdata Ġwith Ġboth Ġsource Ġand Ġtarget Ġlanguages , Ġwhile Ġsource Ġand Ġtarget Ġlanguages Ġhas Ġno Ġparallel Ġdata Ġbetween Ġthemselves . ĠOur Ġproposed Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproach Ġfor Ġsource $ \ right arrow Ġ$ target Ġzero - shot Ġtranslation Ġis Ġmainly Ġdivided Ġinto Ġtwo Ġphrases : Ġthe Ġpret raining Ġphase Ġand Ġthe Ġtransfer Ġphase . </s> In Ġthe Ġpret raining Ġphase , Ġwe Ġfirst Ġpret rain ĠML M Ġon Ġmon oling ual Ġcorpor a Ġof Ġboth Ġsource Ġand Ġpivot Ġlanguages , Ġand Ġcontinue Ġto Ġpret rain ĠTL M Ġor Ġthe Ġproposed ĠB RL M Ġon Ġthe Ġavailable Ġparallel Ġdata Ġbetween Ġsource Ġand Ġpivot Ġlanguages , Ġin Ġorder Ġto Ġbuild Ġa Ġcross - ling ual Ġenc oder Ġshared Ġby Ġthe Ġsource Ġand Ġpivot Ġlanguages . </s> In Ġthe Ġtransfer Ġphase , Ġwe Ġtrain Ġpivot $ \ right arrow Ġ$ target ĠN MT Ġmodel Ġinitialized Ġby Ġthe Ġcross - ling ually Ġpre - trained Ġenc oder , Ġand Ġfinally Ġtransfer Ġthe Ġtrained ĠN MT Ġmodel Ġto Ġsource $ \ right arrow Ġ$ target Ġtranslation Ġthanks Ġto Ġthe Ġshared Ġenc oder . ĠNote Ġthat Ġduring Ġtraining Ġpivot $ \ right arrow Ġ$ target ĠN MT Ġmodel , Ġwe Ġfreeze Ġseveral Ġlayers Ġof Ġthe Ġcross - ling ually Ġpre - trained Ġenc oder Ġto Ġavoid Ġthe Ġdegener acy Ġissue . </s> For Ġthe Ġmore Ġcomplicated Ġscenario Ġthat Ġeither Ġthe Ġsource Ġside Ġor Ġthe Ġtarget Ġside Ġhas Ġmultiple Ġlanguages , Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġare Ġalso Ġshared Ġacross Ġeach Ġside Ġlanguages Ġfor Ġefficient Ġdeployment Ġof Ġtranslation Ġbetween Ġmultiple Ġlanguages . </s> Exper iments Ġ: :: ĠSetup </s> We Ġevaluate Ġour Ġcross - ling ual Ġpre - training Ġbased Ġtransfer Ġapproach Ġagainst Ġseveral Ġstrong Ġbas elines Ġon Ġtwo Ġpublic Ġdat ats ets , ĠEuro par l ĠB IB REF 31 Ġand ĠMulti UN ĠB IB REF 32 , Ġwhich Ġcontain Ġmulti - par allel Ġevaluation Ġdata Ġto Ġassess Ġthe Ġzero - shot Ġperformance . ĠIn Ġall Ġexperiments , Ġwe Ġuse ĠB LE U Ġas Ġthe Ġautomatic Ġmetric Ġfor Ġtranslation Ġevaluation . </s> Exper iments Ġ: :: ĠSetup Ġ: :: ĠDat as ets . </s> The Ġstatistics Ġof ĠEuro par l Ġand ĠMulti UN Ġcorpor a Ġare Ġsummarized Ġin ĠTable ĠT AB REF 18 . ĠFor ĠEuro par l Ġcorpus , Ġwe Ġevaluate Ġon ĠFrench - English - Spanish Ġ( Fr - En - Es ), ĠGerman - English - French Ġ( De - En - Fr ) Ġand ĠRomanian - English - German Ġ( Ro - En - De ), Ġwhere ĠEnglish Ġacts Ġas Ġthe Ġpivot Ġlanguage , Ġits Ġleft Ġside Ġis Ġthe Ġsource Ġlanguage , Ġand Ġits Ġright Ġside Ġis Ġthe Ġtarget Ġlanguage . ĠWe Ġremove Ġthe Ġmulti - par allel Ġsentences Ġbetween Ġdifferent Ġtraining Ġcorpor a Ġto Ġensure Ġzero - shot Ġsettings . ĠWe Ġuse Ġthe Ġdev test 2006 Ġas Ġthe Ġvalidation Ġset Ġand Ġthe Ġtest 2006 Ġas Ġthe Ġtest Ġset Ġfor ĠFr $ \ right arrow Ġ$ Es Ġand ĠDe $ \ right arrow Ġ$ Fr . ĠFor Ġdistant Ġlanguage Ġpair ĠRo $ \ right arrow Ġ$ De , Ġwe Ġextract Ġ1 , 000 Ġoverlapping Ġsentences Ġfrom Ġnew st est 2016 Ġas Ġthe Ġtest Ġset Ġand Ġthe Ġ2 , 000 Ġoverlapping Ġsentences Ġsplit Ġfrom Ġthe Ġtraining Ġset Ġas Ġthe Ġvalidation Ġset Ġsince Ġthere Ġis Ġno Ġofficial Ġvalidation Ġand Ġtest Ġsets . ĠFor Ġvocabulary , Ġwe Ġuse Ġ60 K Ġsub - word Ġtokens Ġbased Ġon ĠByte ĠPair ĠEnc oding Ġ( B PE ) ĠB IB REF 33 . </s> For ĠMulti UN Ġcorpus , Ġwe Ġuse Ġfour Ġlanguages : ĠEnglish Ġ( En ) Ġis Ġset Ġas Ġthe Ġpivot Ġlanguage , Ġwhich Ġhas Ġparallel Ġdata Ġwith Ġother Ġthree Ġlanguages Ġwhich Ġdo Ġnot Ġhave Ġparallel Ġdata Ġbetween Ġeach Ġother . ĠThe Ġthree Ġlanguages Ġare ĠArabic Ġ( Ar ), ĠSpanish Ġ( Es ), Ġand ĠRussian Ġ( Ru ), Ġand Ġmutual Ġtranslation Ġbetween Ġthemselves Ġconstitutes Ġsix Ġzero - shot Ġtranslation Ġdirection Ġfor Ġevaluation . ĠWe Ġuse Ġ80 K ĠB PE Ġsplits Ġas Ġthe Ġvocabulary . ĠNote Ġthat Ġall Ġsentences Ġare Ġtoken ized Ġby Ġthe Ġtoken ize . per l Ġscript , Ġand Ġwe Ġlower case Ġall Ġdata Ġto Ġavoid Ġa Ġlarge Ġvocabulary Ġfor Ġthe ĠMulti UN Ġcorpus . </s> Exper iments Ġ: :: ĠSetup Ġ: :: ĠExperimental ĠDetails . </s> We Ġuse Ġtraditional Ġtransfer Ġlearning , Ġpivot - based Ġmethod Ġand Ġmult ilingual ĠN MT Ġas Ġour Ġbas elines . ĠFor Ġthe Ġfair Ġcomparison , Ġthe ĠTrans former - big Ġmodel Ġwith Ġ1024 Ġembed ding / hidden Ġunits , Ġ4096 Ġfeed - forward Ġfilter Ġsize , Ġ6 Ġlayers Ġand Ġ8 Ġheads Ġper Ġlayer Ġis Ġadopted Ġfor Ġall Ġtranslation Ġmodels Ġin Ġour Ġexperiments . ĠWe Ġset Ġthe Ġbatch Ġsize Ġto Ġ2400 Ġper Ġbatch Ġand Ġlimit Ġsentence Ġlength Ġto Ġ100 ĠB PE Ġtokens . ĠWe Ġset Ġthe Ġ$\ text { att n }\ _ \ text { drop } = 0 $ Ġ( a Ġdrop out Ġrate Ġon Ġeach Ġattention Ġhead ), Ġwhich Ġis Ġfavorable Ġto Ġthe Ġzero - shot Ġtranslation Ġand Ġhas Ġno Ġeffect Ġon Ġsupervised Ġtranslation Ġdirections ĠB IB REF 22 . ĠFor Ġthe Ġmodel Ġinitialization , Ġwe Ġuse ĠFacebook 's Ġcross - ling ual Ġpret rained Ġmodels Ġreleased Ġby ĠXL M Ġto Ġinitialize Ġthe Ġenc oder Ġpart , Ġand Ġthe Ġrest Ġparameters Ġare Ġinitialized Ġwith Ġx avier Ġuniform . ĠWe Ġemploy Ġthe ĠAdam Ġoptim izer Ġwith Ġ$\ text { lr } = 0 . 0001 $, Ġ$ t _{ \ text { warm }\ _ \ text { up }} = 4000 $ Ġand Ġ$\ text { drop out } = 0 . 1 $. ĠAt Ġdecoding Ġtime , Ġwe Ġgenerate Ġgreed ily Ġwith Ġlength Ġpenalty Ġ$\ alpha Ġ= 1 . 0 $. </s> Regarding ĠML M , ĠTL M Ġand ĠB RL M , Ġas Ġmentioned Ġin Ġthe Ġpre - training Ġphase Ġof Ġtransfer Ġprotocol , Ġwe Ġfirst Ġpre - train ĠML M Ġon Ġmon oling ual Ġdata Ġof Ġboth Ġsource Ġand Ġpivot Ġlanguages , Ġthen Ġleverage Ġthe Ġparameters Ġof ĠML M Ġto Ġinitialize ĠTL M Ġand Ġthe Ġproposed ĠB RL M , Ġwhich Ġare Ġcontinued Ġto Ġbe Ġoptimized Ġwith Ġsource - p ivot Ġbilingual Ġdata . ĠIn Ġour Ġexperiments , Ġwe Ġuse ĠML M + TL M , ĠML M + B RL M Ġto Ġrepresent Ġthis Ġtraining Ġprocess . ĠFor Ġthe Ġmask ing Ġstrategy Ġduring Ġtraining , Ġfollowing ĠB IB REF 29 , Ġ$ 15 \ % $ Ġof ĠB PE Ġtokens Ġare Ġselected Ġto Ġbe Ġmasked . ĠAmong Ġthe Ġselected Ġtokens , Ġ$ 80 \ % $ Ġof Ġthem Ġare Ġreplaced Ġwith Ġ[ MAS K ] Ġtoken , Ġ$ 10 \ % $ Ġare Ġreplaced Ġwith Ġa Ġrandom ĠB PE Ġtoken , Ġand Ġ$ 10 \ % $ Ġunchanged . ĠThe Ġprediction Ġaccuracy Ġof Ġmasked Ġwords Ġis Ġused Ġas Ġa Ġstopping Ġcriterion Ġin Ġthe Ġpre - training Ġstage . ĠBesides , Ġwe Ġuse Ġfast align Ġtool ĠB IB REF 34 Ġto Ġextract Ġword Ġalign ments Ġfor ĠB RL M - HA . </s> Exper iments Ġ: :: ĠMain ĠResults </s> Table ĠT AB REF 19 Ġand ĠT AB REF 26 Ġreport Ġzero - shot Ġresults Ġon ĠEuro par l Ġand ĠMulti - UN Ġevaluation Ġsets , Ġrespectively . ĠWe Ġcompare Ġour Ġapproaches Ġwith Ġrelated Ġapproaches Ġof Ġpiv oting , Ġmult ilingual ĠN MT Ġ( MN MT ) ĠB IB REF 19 , Ġand Ġcross - ling ual Ġtransfer Ġwithout Ġpret raining ĠB IB REF 16 . ĠThe Ġresults Ġshow Ġthat Ġour Ġapproaches Ġconsistently Ġoutper form Ġother Ġapproaches Ġacross Ġlanguages Ġand Ġdatasets , Ġespecially Ġsurpass Ġpiv oting , Ġwhich Ġis Ġa Ġstrong Ġbaseline Ġin Ġthe Ġzero - shot Ġscenario Ġthat Ġmult ilingual ĠN MT Ġsystems Ġoften Ġfail Ġto Ġbeat ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 23 . ĠP iv oting Ġtranslates Ġsource Ġto Ġpivot Ġthen Ġto Ġtarget Ġin Ġtwo Ġsteps , Ġcausing Ġinefficient Ġtranslation Ġprocess . ĠOur Ġapproaches Ġuse Ġone Ġenc oder - dec oder Ġmodel Ġto Ġtranslate Ġbetween Ġany Ġzero - shot Ġdirections , Ġwhich Ġis Ġmore Ġefficient Ġthan Ġpiv oting . ĠRegarding Ġthe Ġcomparison Ġbetween Ġtransfer Ġapproaches , Ġour Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġoutper forms Ġtransfer Ġmethod Ġthat Ġdoes Ġnot Ġuse Ġpret raining Ġby Ġa Ġlarge Ġmargin . </s> Exper iments Ġ: :: ĠMain ĠResults Ġ: :: ĠResults Ġon ĠEuro par l ĠDat as et . </s> Regarding Ġcomparison Ġbetween Ġthe Ġbas elines Ġin Ġtable ĠT AB REF 19 , Ġwe Ġfind Ġthat Ġpiv oting Ġis Ġthe Ġstrongest Ġbaseline Ġthat Ġhas Ġsignificant Ġadvantage Ġover Ġother Ġtwo Ġbas elines . ĠCross - ling ual Ġtransfer Ġfor Ġlanguages Ġwithout Ġshared Ġvoc ab ular ies ĠB IB REF 16 Ġmanifests Ġthe Ġworst Ġperformance Ġbecause Ġof Ġnot Ġusing Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata , Ġwhich Ġis Ġutilized Ġas Ġbeneficial Ġsupervised Ġsignal Ġfor Ġthe Ġother Ġtwo Ġbas elines . </s> Our Ġbest Ġapproach Ġof ĠML M + B RL M - SA Ġachieves Ġthe Ġsignificant Ġsuperior Ġperformance Ġto Ġall Ġbas elines Ġin Ġthe Ġzero - shot Ġdirections , Ġimproving Ġby Ġ0 . 9 - 4 . 8 ĠB LE U Ġpoints Ġover Ġthe Ġstrong Ġpiv oting . ĠMeanwhile , Ġin Ġthe Ġsupervised Ġdirection Ġof Ġpivot $ \ right arrow Ġ$ target , Ġour Ġapproaches Ġperforms Ġeven Ġbetter Ġthan Ġthe Ġoriginal Ġsupervised ĠTrans former Ġthanks Ġto Ġthe Ġshared Ġenc oder Ġtrained Ġon Ġboth Ġlarge - scale Ġmon oling ual Ġdata Ġand Ġparallel Ġdata Ġbetween Ġmultiple Ġlanguages . </s> ML M Ġalone Ġthat Ġdoes Ġnot Ġuse Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata Ġperforms Ġmuch Ġbetter Ġthan Ġthe Ġcross - ling ual Ġtransfer , Ġand Ġachieves Ġcomparable Ġresults Ġto Ġpiv oting . ĠWhen ĠML M Ġis Ġcombined Ġwith ĠTL M Ġor Ġthe Ġproposed ĠB RL M , Ġthe Ġperformance Ġis Ġfurther Ġimproved . ĠML M + B RL M - SA Ġperforms Ġthe Ġbest , Ġand Ġis Ġbetter Ġthan ĠML M + B RL M - HA Ġindicating Ġthat Ġsoft Ġalignment Ġis Ġhelpful Ġthan Ġhard Ġalignment Ġfor Ġthe Ġcross - ling ual Ġpret raining . </s> Exper iments Ġ: :: ĠMain ĠResults Ġ: :: ĠResults Ġon ĠMulti UN ĠDat as et . </s> Like Ġexperimental Ġresults Ġon ĠEuro par l , ĠML M + B RL M - SA Ġperforms Ġthe Ġbest Ġamong Ġall Ġproposed Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproaches Ġas Ġshown Ġin ĠTable ĠT AB REF 26 . ĠWhen Ġcomparing Ġsystems Ġconsisting Ġof Ġone Ġenc oder - dec oder Ġmodel Ġfor Ġall Ġzero - shot Ġtranslation , Ġour Ġapproaches Ġperforms Ġsignificantly Ġbetter Ġthan ĠMN MT ĠB IB REF 19 . </s> Although Ġit Ġis Ġchallenging Ġfor Ġone Ġmodel Ġto Ġtranslate Ġall Ġzero - shot Ġdirections Ġbetween Ġmultiple Ġdistant Ġlanguage Ġpairs Ġof ĠMulti UN , ĠML M + B RL M - SA Ġstill Ġachieves Ġbetter Ġperformances Ġon ĠEs Ġ$\ right arrow Ġ$ ĠAr Ġand ĠEs Ġ$\ right arrow Ġ$ ĠRu Ġthan Ġstrong Ġpiv oting $ _{ \ rm Ġm } $, Ġwhich Ġuses ĠMN MT Ġto Ġtranslate Ġsource Ġto Ġpivot Ġthen Ġto Ġtarget Ġin Ġtwo Ġseparate Ġsteps Ġwith Ġeach Ġstep Ġreceiving Ġsupervised Ġsignal Ġof Ġparallel Ġcorpor a . ĠOur Ġapproaches Ġsurpass Ġpiv oting $ _{ \ rm Ġm } $ Ġin Ġall Ġzero - shot Ġdirections Ġby Ġadding Ġback Ġtranslation ĠB IB REF 33 Ġto Ġgenerate Ġpseudo Ġparallel Ġsentences Ġfor Ġall Ġzero - shot Ġdirections Ġbased Ġon Ġour Ġpret rained Ġmodels Ġsuch Ġas ĠML M + B RL M - SA , Ġand Ġfurther Ġtraining Ġour Ġuniversal Ġenc oder - dec oder Ġmodel Ġwith Ġthese Ġpseudo Ġdata . ĠB IB REF 22 Ġgu 2019 impro ved Ġintroduces Ġback Ġtranslation Ġinto ĠMN MT , Ġwhile Ġwe Ġadopt Ġit Ġin Ġour Ġtransfer Ġapproaches . ĠFinally , Ġour Ġbest ĠML M + B RL M - SA Ġwith Ġback Ġtranslation Ġoutper forms Ġpiv oting $ _{ \ rm Ġm } $ Ġby Ġ2 . 4 ĠB LE U Ġpoints Ġaverage ly , Ġand Ġoutper forms ĠMN MT ĠB IB REF 22 Ġby Ġ4 . 6 ĠB LE U Ġpoints Ġaverage ly . ĠAgain , Ġin Ġsupervised Ġtranslation Ġdirections , ĠML M + B RL M - SA Ġwith Ġback Ġtranslation Ġalso Ġachieves Ġbetter Ġperformance Ġthan Ġthe Ġoriginal Ġsupervised ĠTrans former . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠSent ence ĠRepresent ation . </s> We Ġfirst Ġevaluate Ġthe Ġrepresent ational Ġinvari ance Ġacross Ġlanguages Ġfor Ġall Ġcross - ling ual Ġpre - training Ġmethods . ĠFollowing ĠB IB REF 23 , Ġwe Ġadopt Ġmax - pool ing Ġoperation Ġto Ġcollect Ġthe Ġsentence Ġrepresentation Ġof Ġeach Ġenc oder Ġlayer Ġfor Ġall Ġsource - p ivot Ġsentence Ġpairs Ġin Ġthe ĠEuro par l Ġvalidation Ġsets . ĠThen Ġwe Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġfor Ġeach Ġsentence Ġpair Ġand Ġaverage Ġall Ġcos ine Ġscores . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 27 , Ġwe Ġcan Ġobserve Ġthat , ĠML M + B RL M - SA Ġhas Ġthe Ġmost Ġstable Ġand Ġsimilar Ġcross - ling ual Ġrepresentations Ġof Ġsentence Ġpairs Ġon Ġall Ġlayers , Ġwhile Ġit Ġachieves Ġthe Ġbest Ġperformance Ġin Ġzero - shot Ġtranslation . ĠThis Ġdemonstrates Ġthat Ġbetter Ġcross - ling ual Ġrepresentations Ġcan Ġbenefit Ġfor Ġthe Ġprocess Ġof Ġtransfer Ġlearning . ĠBesides , ĠML M + B RL M - HA Ġis Ġnot Ġas Ġsuperior Ġas ĠML M + B RL M - SA Ġand Ġeven Ġworse Ġthan ĠML M + TL M Ġon ĠFr - En , Ġsince ĠML M + B RL M - HA Ġmay Ġsuffer Ġfrom Ġthe Ġwrong Ġalignment Ġknowledge Ġfrom Ġan Ġexternal Ġalign er Ġtool . ĠWe Ġalso Ġfind Ġan Ġinteresting Ġphenomenon Ġthat Ġas Ġthe Ġnumber Ġof Ġlayers Ġincreases , Ġthe Ġcos ine Ġsimilarity Ġdecreases . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠContext ual ized ĠWord ĠRepresent ation . </s> We Ġfurther Ġsample Ġan ĠEnglish - Russian Ġsentence Ġpair Ġfrom Ġthe ĠMulti UN Ġvalidation Ġsets Ġand Ġvisualize Ġthe Ġcos ine Ġsimilarity Ġbetween Ġhidden Ġstates Ġof Ġthe Ġtop Ġenc oder Ġlayer Ġto Ġfurther Ġinvestigate Ġthe Ġdifference Ġof Ġall Ġcross - ling ual Ġpre - training Ġmethods . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 38 , Ġthe Ġhidden Ġstates Ġgenerated Ġby ĠML M + B RL M - SA Ġhave Ġhigher Ġsimilarity Ġfor Ġtwo Ġaligned Ġwords . ĠIt Ġindicates Ġthat ĠML M + B RL M - SA Ġcan Ġgain Ġbetter Ġword - level Ġrepresentation Ġalignment Ġbetween Ġsource Ġand Ġpivot Ġlanguages , Ġwhich Ġbetter Ġrelie ves Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠThe ĠEffect Ġof ĠFree zing ĠParameters . </s> To Ġfreeze Ġparameters Ġis Ġa Ġcommon Ġstrategy Ġto Ġavoid Ġcatastrophic Ġforgetting Ġin Ġtransfer Ġlearning ĠB IB REF 27 . ĠTable ĠT AB REF 43 Ġshows Ġthe Ġperformance Ġof Ġtransfer Ġlearning Ġwith Ġfreezing Ġdifferent Ġlayers Ġon ĠMulti UN Ġtest Ġset , Ġin Ġwhich ĠEn $ \ right arrow Ġ$ Ru Ġdenotes Ġthe Ġparent Ġmodel , ĠAr $ \ right arrow Ġ$ Ru Ġand ĠEs $ \ right arrow Ġ$ Ru Ġare Ġtwo Ġchild Ġmodels , Ġand Ġall Ġmodels Ġare Ġbased Ġon ĠML M + B RL M - SA . ĠWe Ġcan Ġfind Ġthat Ġupdating Ġall Ġparameters Ġduring Ġtraining Ġwill Ġcause Ġa Ġnotable Ġdrop Ġon Ġthe Ġzero - shot Ġdirection Ġdue Ġto Ġthe Ġcatastrophic Ġforgetting . ĠOn Ġthe Ġcontrary , Ġfreezing Ġall Ġthe Ġparameters Ġleads Ġto Ġthe Ġdecline Ġon Ġsupervised Ġdirection Ġbecause Ġthe Ġlanguage Ġfeatures Ġextracted Ġduring Ġpre - training Ġis Ġnot Ġsufficient Ġfor ĠMT Ġtask . ĠFree zing Ġthe Ġfirst Ġfour Ġlayers Ġof Ġthe Ġtransformer Ġshows Ġthe Ġbest Ġperformance Ġand Ġkeeps Ġthe Ġbalance Ġbetween Ġpre - training Ġand Ġfine - tun ing . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproach Ġfor Ġthe Ġchallenging Ġzero - shot Ġtranslation Ġtask , Ġin Ġwhich Ġsource Ġand Ġtarget Ġlanguages Ġhave Ġno Ġparallel Ġdata , Ġwhile Ġthey Ġboth Ġhave Ġparallel Ġdata Ġwith Ġa Ġhigh Ġresource Ġpivot Ġlanguage . ĠWith Ġthe Ġaim Ġof Ġbuilding Ġthe Ġlanguage Ġinvari ant Ġrepresentation Ġbetween Ġsource Ġand Ġpivot Ġlanguages Ġfor Ġsmooth Ġtransfer Ġof Ġthe Ġparent Ġmodel Ġof Ġpivot $ \ right arrow Ġ$ target Ġdirection Ġto Ġthe Ġchild Ġmodel Ġof Ġsource $ \ right arrow Ġ$ target Ġdirection , Ġwe Ġintroduce Ġone Ġmon oling ual Ġpret raining Ġmethod Ġand Ġtwo Ġbilingual Ġpret raining Ġmethods Ġto Ġconstruct Ġan Ġuniversal Ġenc oder Ġfor Ġthe Ġsource Ġand Ġpivot Ġlanguages . ĠExper iments Ġon Ġpublic Ġdatasets Ġshow Ġthat Ġour Ġapproaches Ġsignificantly Ġoutper forms Ġseveral Ġstrong Ġbaseline Ġsystems , Ġand Ġmanifest Ġthe Ġlanguage Ġinvari ance Ġcharacteristics Ġin Ġboth Ġsentence Ġlevel Ġand Ġword Ġlevel Ġneural Ġrepresentations . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġthe Ġhelpful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby ĠNational ĠKey ĠR & D ĠProgram Ġof ĠChina Ġ( Grant ĠNo . Ġ2016 Y FE 013 21 00 ), ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 15 25 205 , Ġ6 16 73 289 ). ĠThis Ġwork Ġwas Ġalso Ġpartially Ġsupported Ġby ĠAlibaba ĠGroup Ġthrough ĠAlibaba ĠInnov ative ĠResearch ĠProgram Ġand Ġthe ĠPriority ĠAcademic ĠProgram ĠDevelopment Ġ( P APD ) Ġof ĠJiang su ĠHigher ĠEducation ĠInstit utions .
Introduction Although Neural Machine Translation (NMT) has dominated recent research on translation tasks BIBREF0, BIBREF1, BIBREF2, NMT heavily relies on large-scale parallel data, resulting in poor performance on low-resource or zero-resource language pairs BIBREF3. Translation between these low-resource languages (e.g., Arabic$\rightarrow $Spanish) is usually accomplished with pivoting through a rich-resource language (such as English), i.e., Arabic (source) sentence is translated to English (pivot) first which is later translated to Spanish (target) BIBREF4, BIBREF5. However, the pivot-based method requires doubled decoding time and suffers from the propagation of translation errors. One common alternative to avoid pivoting in NMT is transfer learning BIBREF6, BIBREF7, BIBREF8, BIBREF9 which leverages a high-resource pivot$\rightarrow $target model (parent) to initialize a low-resource source$\rightarrow $target model (child) that is further optimized with a small amount of available parallel data. Although this approach has achieved success in some low-resource language pairs, it still performs very poorly in extremely low-resource or zero-resource translation scenario. Specifically, BIBREF8 reports that without any child model training data, the performance of the parent model on the child test set is miserable. In this work, we argue that the language space mismatch problem, also named domain shift problem BIBREF10, brings about the zero-shot translation failure in transfer learning. It is because transfer learning has no explicit training process to guarantee that the source and pivot languages share the same feature distributions, causing that the child model inherited from the parent model fails in such a situation. For instance, as illustrated in the left of Figure FIGREF1, the points of the sentence pair with the same semantics are not overlapping in source space, resulting in that the shared decoder will generate different translations denoted by different points in target space. Actually, transfer learning for NMT can be viewed as a multi-domain problem where each source language forms a new domain. Minimizing the discrepancy between the feature distributions of different source languages, i.e., different domains, will ensure the smooth transition between the parent and child models, as shown in the right of Figure FIGREF1. One way to achieve this goal is the fine-tuning technique, which forces the model to forget the specific knowledge from parent data and learn new features from child data. However, the domain shift problem still exists, and the demand of parallel child data for fine-tuning heavily hinders transfer learning for NMT towards the zero-resource setting. In this paper, we explore the transfer learning in a common zero-shot scenario where there are a lot of source$\leftrightarrow $pivot and pivot$\leftrightarrow $target parallel data but no source$\leftrightarrow $target parallel data. In this scenario, we propose a simple but effective transfer approach, the key idea of which is to relieve the burden of the domain shift problem by means of cross-lingual pre-training. To this end, we firstly investigate the performance of two existing cross-lingual pre-training methods proposed by BIBREF11 in zero-shot translation scenario. Besides, a novel pre-training method called BRidge Language Modeling (BRLM) is designed to make full use of the source$\leftrightarrow $pivot bilingual data to obtain a universal encoder for different languages. Once the universal encoder is constructed, we only need to train the pivot$\rightarrow $target model and then test this model in source$\rightarrow $target direction directly. The main contributions of this paper are as follows: We propose a new transfer learning approach for NMT which uses the cross-lingual language model pre-training to enable a high performance on zero-shot translation. We propose a novel pre-training method called BRLM, which can effectively alleviates the distance between different source language spaces. Our proposed approach significantly improves zero-shot translation performance, consistently surpassing pivoting and multilingual approaches. Meanwhile, the performance on supervised translation direction remains the same level or even better when using our method. Related Work In recent years, zero-shot translation in NMT has attracted widespread attention in academic research. Existing methods are mainly divided into four categories: pivot-based method, transfer learning, multilingual NMT, and unsupervised NMT. Pivot-based Method is a common strategy to obtain a source$\rightarrow $target model by introducing a pivot language. This approach is further divided into pivoting and pivot-synthetic. While the former firstly translates a source language into the pivot language which is later translated to the target language BIBREF4, BIBREF5, BIBREF12, the latter trains a source$\rightarrow $target model with pseudo data generated from source-pivot or pivot-target parallel data BIBREF13, BIBREF14. Although the pivot-based methods can achieve not bad performance, it always falls into a computation-expensive and parameter-vast dilemma of quadratic growth in the number of source languages, and suffers from the error propagation problem BIBREF15. Transfer Learning is firstly introduced for NMT by BIBREF6, which leverages a high-resource parent model to initialize the low-resource child model. On this basis, BIBREF7 and BIBREF8 use shared vocabularies for source/target language to improve transfer learning, while BIBREF16 relieve the vocabulary mismatch by mainly using cross-lingual word embedding. Although these methods are successful in the low-resource scene, they have limited effects in zero-shot translation. Multilingual NMT (MNMT) enables training a single model that supports translation from multiple source languages into multiple target languages, even those unseen language pairs BIBREF17, BIBREF18, BIBREF19, BIBREF20, BIBREF21. Aside from simpler deployment, MNMT benefits from transfer learning where low-resource language pairs are trained together with high-resource ones. However, BIBREF22 point out that MNMT for zero-shot translation easily fails, and is sensitive to the hyper-parameter setting. Also, MNMT usually performs worse than the pivot-based method in zero-shot translation setting BIBREF23. Unsupervised NMT (UNMT) considers a harder setting, in which only large-scale monolingual corpora are available for training. Recently, many methods have been proposed to improve the performance of UNMT, including using denoising auto-encoder, statistic machine translation (SMT) and unsupervised pre-training BIBREF24, BIBREF25, BIBREF26, BIBREF11. Since UNMT performs well between similar languages (e.g., English-German translation), its performance between distant languages is still far from expectation. Our proposed method belongs to the transfer learning, but it is different from traditional transfer methods which train a parent model as starting point. Before training a parent model, our approach fully leverages cross-lingual pre-training methods to make all source languages share the same feature space and thus enables a smooth transition for zero-shot translation. Approach In this section, we will present a cross-lingual pre-training based transfer approach. This method is designed for a common zero-shot scenario where there are a lot of source$\leftrightarrow $pivot and pivot$\leftrightarrow $target bilingual data but no source$\leftrightarrow $target parallel data, and the whole training process can be summarized as follows step by step: Pre-train a universal encoder with source/pivot monolingual or source$\leftrightarrow $pivot bilingual data. Train a pivot$\rightarrow $target parent model built on the pre-trained universal encoder with the available parallel data. During the training process, we freeze several layers of the pre-trained universal encoder to avoid the degeneracy issue BIBREF27. Directly translate source sentences into target sentences with the parent model, which benefits from the availability of the universal encoder. The key difficulty of this method is to ensure the intermediate representations of the universal encoder are language invariant. In the rest of this section, we first present two existing methods yet to be explored in zero-shot translation, and then propose a straightforward but effective cross-lingual pre-training method. In the end, we present the whole training and inference protocol for transfer. Approach ::: Masked and Translation Language Model Pretraining Two existing cross-lingual pre-training methods, Masked Language Modeling (MLM) and Translation Language Modeling (TLM), have shown their effectiveness on XNLI cross-lingual classification task BIBREF11, BIBREF28, but these methods have not been well studied on cross-lingual generation tasks in zero-shot condition. We attempt to take advantage of the cross-lingual ability of the two methods for zero-shot translation. Specifically, MLM adopts the Cloze objective of BERT BIBREF29 and predicts the masked words that are randomly selected and replaced with [MASK] token on monolingual corpus. In practice, MLM takes different language monolingual corpora as input to find features shared across different languages. With this method, word pieces shared in all languages have been mapped into a shared space, which makes the sentence representations across different languages close BIBREF30. Since MLM objective is unsupervised and only requires monolingual data, TLM is designed to leverage parallel data when it is available. Actually, TLM is a simple extension of MLM, with the difference that TLM concatenates sentence pair into a whole sentence, and then randomly masks words in both the source and target sentences. In this way, the model can either attend to surrounding words or to the translation sentence, implicitly encouraging the model to align the source and target language representations. Note that although each sentence pair is formed into one sentence, the positions of the target sentence are reset to count form zero. Approach ::: Bridge Language Model Pretraining Aside from MLM and TLM, we propose BRidge Language Modeling (BRLM) to further obtain word-level representation alignment between different languages. This method is inspired by the assumption that if the feature spaces of different languages are aligned very well, the masked words in the corrupted sentence can also be guessed by the context of the correspondingly aligned words on the other side. To achieve this goal, BRLM is designed to strengthen the ability to infer words across languages based on alignment information, instead of inferring words within monolingual sentence as in MLM or within the pseudo sentence formed by concatenating sentence pair as in TLM. As illustrated in Figure FIGREF9, BRLM stacks shared encoder over both side sentences separately. In particular, we design two network structures for BRLM, which are divided into Hard Alignment (BRLM-HA) and Soft Alignment (BRLM-SA) according to the way of generating the alignment information. These two structures actually extend MLM into a bilingual scenario, with the difference that BRLM leverages external aligner tool or additional attention layer to explicitly introduce alignment information during model training. Hard Alignment (BRLM-HA). We first use external aligner tool on source$\leftrightarrow $pivot parallel data to extract the alignment information of sentence pair. During model training, given source$\leftrightarrow $pivot sentence pair, BRLM-HA randomly masks some words in source sentence and leverages alignment information to obtain the aligned words in pivot sentence for masked words. Based on the processed input, BRLM-HA adopts the Transformer BIBREF1 encoder to gain the hidden states for source and pivot sentences respectively. Then the training objective of BRLM-HA is to predict the masked words by not only the surrounding words in source sentence but also the encoder outputs of the aligned words. Note that this training process is also carried out in a symmetric situation, in which we mask some words in pivot sentence and obtain the aligned words in the source sentence. Soft Alignment (BRLM-SA). Instead of using external aligner tool, BRLM-SA introduces an additional attention layer to learn the alignment information together with model training. In this way, BRLM-SA avoids the effect caused by external wrong alignment information and enables many-to-one soft alignment during model training. Similar with BRLM-HA, the training objective of BRLM-SA is to predict the masked words by not only the surrounding words in source sentence but also the outputs of attention layer. In our implementation, the attention layer is a multi-head attention layer adopted in Transformer, where the queries come from the masked source sentence, the keys and values come from the pivot sentence. In principle, MLM and TLM can learn some implicit alignment information during model training. However, the alignment process in MLM is inefficient since the shared word pieces only account for a small proportion of the whole corpus, resulting in the difficulty of expanding the shared information to align the whole corpus. TLM also lacks effort in alignment between the source and target sentences since TLM concatenates the sentence pair into one sequence, making the explicit alignment between the source and target infeasible. BRLM fully utilizes the alignment information to obtain better word-level representation alignment between different languages, which better relieves the burden of the domain shift problem. Approach ::: Transfer Protocol We consider the typical zero-shot translation scenario in which a high resource pivot language has parallel data with both source and target languages, while source and target languages has no parallel data between themselves. Our proposed cross-lingual pretraining based transfer approach for source$\rightarrow $target zero-shot translation is mainly divided into two phrases: the pretraining phase and the transfer phase. In the pretraining phase, we first pretrain MLM on monolingual corpora of both source and pivot languages, and continue to pretrain TLM or the proposed BRLM on the available parallel data between source and pivot languages, in order to build a cross-lingual encoder shared by the source and pivot languages. In the transfer phase, we train pivot$\rightarrow $target NMT model initialized by the cross-lingually pre-trained encoder, and finally transfer the trained NMT model to source$\rightarrow $target translation thanks to the shared encoder. Note that during training pivot$\rightarrow $target NMT model, we freeze several layers of the cross-lingually pre-trained encoder to avoid the degeneracy issue. For the more complicated scenario that either the source side or the target side has multiple languages, the encoder and the decoder are also shared across each side languages for efficient deployment of translation between multiple languages. Experiments ::: Setup We evaluate our cross-lingual pre-training based transfer approach against several strong baselines on two public datatsets, Europarl BIBREF31 and MultiUN BIBREF32, which contain multi-parallel evaluation data to assess the zero-shot performance. In all experiments, we use BLEU as the automatic metric for translation evaluation. Experiments ::: Setup ::: Datasets. The statistics of Europarl and MultiUN corpora are summarized in Table TABREF18. For Europarl corpus, we evaluate on French-English-Spanish (Fr-En-Es), German-English-French (De-En-Fr) and Romanian-English-German (Ro-En-De), where English acts as the pivot language, its left side is the source language, and its right side is the target language. We remove the multi-parallel sentences between different training corpora to ensure zero-shot settings. We use the devtest2006 as the validation set and the test2006 as the test set for Fr$\rightarrow $Es and De$\rightarrow $Fr. For distant language pair Ro$\rightarrow $De, we extract 1,000 overlapping sentences from newstest2016 as the test set and the 2,000 overlapping sentences split from the training set as the validation set since there is no official validation and test sets. For vocabulary, we use 60K sub-word tokens based on Byte Pair Encoding (BPE) BIBREF33. For MultiUN corpus, we use four languages: English (En) is set as the pivot language, which has parallel data with other three languages which do not have parallel data between each other. The three languages are Arabic (Ar), Spanish (Es), and Russian (Ru), and mutual translation between themselves constitutes six zero-shot translation direction for evaluation. We use 80K BPE splits as the vocabulary. Note that all sentences are tokenized by the tokenize.perl script, and we lowercase all data to avoid a large vocabulary for the MultiUN corpus. Experiments ::: Setup ::: Experimental Details. We use traditional transfer learning, pivot-based method and multilingual NMT as our baselines. For the fair comparison, the Transformer-big model with 1024 embedding/hidden units, 4096 feed-forward filter size, 6 layers and 8 heads per layer is adopted for all translation models in our experiments. We set the batch size to 2400 per batch and limit sentence length to 100 BPE tokens. We set the $\text{attn}\_\text{drop}=0$ (a dropout rate on each attention head), which is favorable to the zero-shot translation and has no effect on supervised translation directions BIBREF22. For the model initialization, we use Facebook's cross-lingual pretrained models released by XLM to initialize the encoder part, and the rest parameters are initialized with xavier uniform. We employ the Adam optimizer with $\text{lr}=0.0001$, $t_{\text{warm}\_\text{up}}=4000$ and $\text{dropout}=0.1$. At decoding time, we generate greedily with length penalty $\alpha =1.0$. Regarding MLM, TLM and BRLM, as mentioned in the pre-training phase of transfer protocol, we first pre-train MLM on monolingual data of both source and pivot languages, then leverage the parameters of MLM to initialize TLM and the proposed BRLM, which are continued to be optimized with source-pivot bilingual data. In our experiments, we use MLM+TLM, MLM+BRLM to represent this training process. For the masking strategy during training, following BIBREF29, $15\%$ of BPE tokens are selected to be masked. Among the selected tokens, $80\%$ of them are replaced with [MASK] token, $10\%$ are replaced with a random BPE token, and $10\%$ unchanged. The prediction accuracy of masked words is used as a stopping criterion in the pre-training stage. Besides, we use fastalign tool BIBREF34 to extract word alignments for BRLM-HA. Experiments ::: Main Results Table TABREF19 and TABREF26 report zero-shot results on Europarl and Multi-UN evaluation sets, respectively. We compare our approaches with related approaches of pivoting, multilingual NMT (MNMT) BIBREF19, and cross-lingual transfer without pretraining BIBREF16. The results show that our approaches consistently outperform other approaches across languages and datasets, especially surpass pivoting, which is a strong baseline in the zero-shot scenario that multilingual NMT systems often fail to beat BIBREF19, BIBREF20, BIBREF23. Pivoting translates source to pivot then to target in two steps, causing inefficient translation process. Our approaches use one encoder-decoder model to translate between any zero-shot directions, which is more efficient than pivoting. Regarding the comparison between transfer approaches, our cross-lingual pretraining based transfer outperforms transfer method that does not use pretraining by a large margin. Experiments ::: Main Results ::: Results on Europarl Dataset. Regarding comparison between the baselines in table TABREF19, we find that pivoting is the strongest baseline that has significant advantage over other two baselines. Cross-lingual transfer for languages without shared vocabularies BIBREF16 manifests the worst performance because of not using source$\leftrightarrow $pivot parallel data, which is utilized as beneficial supervised signal for the other two baselines. Our best approach of MLM+BRLM-SA achieves the significant superior performance to all baselines in the zero-shot directions, improving by 0.9-4.8 BLEU points over the strong pivoting. Meanwhile, in the supervised direction of pivot$\rightarrow $target, our approaches performs even better than the original supervised Transformer thanks to the shared encoder trained on both large-scale monolingual data and parallel data between multiple languages. MLM alone that does not use source$\leftrightarrow $pivot parallel data performs much better than the cross-lingual transfer, and achieves comparable results to pivoting. When MLM is combined with TLM or the proposed BRLM, the performance is further improved. MLM+BRLM-SA performs the best, and is better than MLM+BRLM-HA indicating that soft alignment is helpful than hard alignment for the cross-lingual pretraining. Experiments ::: Main Results ::: Results on MultiUN Dataset. Like experimental results on Europarl, MLM+BRLM-SA performs the best among all proposed cross-lingual pretraining based transfer approaches as shown in Table TABREF26. When comparing systems consisting of one encoder-decoder model for all zero-shot translation, our approaches performs significantly better than MNMT BIBREF19. Although it is challenging for one model to translate all zero-shot directions between multiple distant language pairs of MultiUN, MLM+BRLM-SA still achieves better performances on Es $\rightarrow $ Ar and Es $\rightarrow $ Ru than strong pivoting$_{\rm m}$, which uses MNMT to translate source to pivot then to target in two separate steps with each step receiving supervised signal of parallel corpora. Our approaches surpass pivoting$_{\rm m}$ in all zero-shot directions by adding back translation BIBREF33 to generate pseudo parallel sentences for all zero-shot directions based on our pretrained models such as MLM+BRLM-SA, and further training our universal encoder-decoder model with these pseudo data. BIBREF22 gu2019improved introduces back translation into MNMT, while we adopt it in our transfer approaches. Finally, our best MLM+BRLM-SA with back translation outperforms pivoting$_{\rm m}$ by 2.4 BLEU points averagely, and outperforms MNMT BIBREF22 by 4.6 BLEU points averagely. Again, in supervised translation directions, MLM+BRLM-SA with back translation also achieves better performance than the original supervised Transformer. Experiments ::: Analysis ::: Sentence Representation. We first evaluate the representational invariance across languages for all cross-lingual pre-training methods. Following BIBREF23, we adopt max-pooling operation to collect the sentence representation of each encoder layer for all source-pivot sentence pairs in the Europarl validation sets. Then we calculate the cosine similarity for each sentence pair and average all cosine scores. As shown in Figure FIGREF27, we can observe that, MLM+BRLM-SA has the most stable and similar cross-lingual representations of sentence pairs on all layers, while it achieves the best performance in zero-shot translation. This demonstrates that better cross-lingual representations can benefit for the process of transfer learning. Besides, MLM+BRLM-HA is not as superior as MLM+BRLM-SA and even worse than MLM+TLM on Fr-En, since MLM+BRLM-HA may suffer from the wrong alignment knowledge from an external aligner tool. We also find an interesting phenomenon that as the number of layers increases, the cosine similarity decreases. Experiments ::: Analysis ::: Contextualized Word Representation. We further sample an English-Russian sentence pair from the MultiUN validation sets and visualize the cosine similarity between hidden states of the top encoder layer to further investigate the difference of all cross-lingual pre-training methods. As shown in Figure FIGREF38, the hidden states generated by MLM+BRLM-SA have higher similarity for two aligned words. It indicates that MLM+BRLM-SA can gain better word-level representation alignment between source and pivot languages, which better relieves the burden of the domain shift problem. Experiments ::: Analysis ::: The Effect of Freezing Parameters. To freeze parameters is a common strategy to avoid catastrophic forgetting in transfer learning BIBREF27. Table TABREF43 shows the performance of transfer learning with freezing different layers on MultiUN test set, in which En$\rightarrow $Ru denotes the parent model, Ar$\rightarrow $Ru and Es$\rightarrow $Ru are two child models, and all models are based on MLM+BRLM-SA. We can find that updating all parameters during training will cause a notable drop on the zero-shot direction due to the catastrophic forgetting. On the contrary, freezing all the parameters leads to the decline on supervised direction because the language features extracted during pre-training is not sufficient for MT task. Freezing the first four layers of the transformer shows the best performance and keeps the balance between pre-training and fine-tuning. Conclusion In this paper, we propose a cross-lingual pretraining based transfer approach for the challenging zero-shot translation task, in which source and target languages have no parallel data, while they both have parallel data with a high resource pivot language. With the aim of building the language invariant representation between source and pivot languages for smooth transfer of the parent model of pivot$\rightarrow $target direction to the child model of source$\rightarrow $target direction, we introduce one monolingual pretraining method and two bilingual pretraining methods to construct an universal encoder for the source and pivot languages. Experiments on public datasets show that our approaches significantly outperforms several strong baseline systems, and manifest the language invariance characteristics in both sentence level and word level neural representations. Acknowledgments We would like to thank the anonymous reviewers for the helpful comments. This work was supported by National Key R&D Program of China (Grant No. 2016YFE0132100), National Natural Science Foundation of China (Grant No. 61525205, 61673289). This work was also partially supported by Alibaba Group through Alibaba Innovative Research Program and the Priority Academic Program Development (PAPD) of Jiangsu Higher Education Institutions.
which multilingual approaches do they compare with?
b6f15fb6279b82e34a5bf4828b7b5ddabfdf1d54
which Ġmult ilingual Ġapproaches Ġdo Ġthey Ġcompare Ġwith ?
BI B REF 19 , ĠB IB REF 20
<s> which Ġmult ilingual Ġapproaches Ġdo Ġthey Ġcompare Ġwith ? </s> Introduction </s> Although ĠNeural ĠMachine ĠTranslation Ġ( N MT ) Ġhas Ġdominated Ġrecent Ġresearch Ġon Ġtranslation Ġtasks ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠN MT Ġheavily Ġrelies Ġon Ġlarge - scale Ġparallel Ġdata , Ġresulting Ġin Ġpoor Ġperformance Ġon Ġlow - resource Ġor Ġzero - resource Ġlanguage Ġpairs ĠB IB REF 3 . ĠTranslation Ġbetween Ġthese Ġlow - resource Ġlanguages Ġ( e . g ., ĠArabic $ \ right arrow Ġ$ Spanish ) Ġis Ġusually Ġaccomplished Ġwith Ġpiv oting Ġthrough Ġa Ġrich - resource Ġlanguage Ġ( such Ġas ĠEnglish ), Ġi . e ., ĠArabic Ġ( source ) Ġsentence Ġis Ġtranslated Ġto ĠEnglish Ġ( p ivot ) Ġfirst Ġwhich Ġis Ġlater Ġtranslated Ġto ĠSpanish Ġ( target ) ĠB IB REF 4 , ĠB IB REF 5 . ĠHowever , Ġthe Ġpivot - based Ġmethod Ġrequires Ġdoubled Ġdecoding Ġtime Ġand Ġsuffers Ġfrom Ġthe Ġpropagation Ġof Ġtranslation Ġerrors . </s> One Ġcommon Ġalternative Ġto Ġavoid Ġpiv oting Ġin ĠN MT Ġis Ġtransfer Ġlearning ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 Ġwhich Ġlever ages Ġa Ġhigh - resource Ġpivot $ \ right arrow Ġ$ target Ġmodel Ġ( parent ) Ġto Ġinitialize Ġa Ġlow - resource Ġsource $ \ right arrow Ġ$ target Ġmodel Ġ( child ) Ġthat Ġis Ġfurther Ġoptimized Ġwith Ġa Ġsmall Ġamount Ġof Ġavailable Ġparallel Ġdata . ĠAlthough Ġthis Ġapproach Ġhas Ġachieved Ġsuccess Ġin Ġsome Ġlow - resource Ġlanguage Ġpairs , Ġit Ġstill Ġperforms Ġvery Ġpoorly Ġin Ġextremely Ġlow - resource Ġor Ġzero - resource Ġtranslation Ġscenario . ĠSpecifically , ĠB IB REF 8 Ġreports Ġthat Ġwithout Ġany Ġchild Ġmodel Ġtraining Ġdata , Ġthe Ġperformance Ġof Ġthe Ġparent Ġmodel Ġon Ġthe Ġchild Ġtest Ġset Ġis Ġmiserable . </s> In Ġthis Ġwork , Ġwe Ġargue Ġthat Ġthe Ġlanguage Ġspace Ġmismatch Ġproblem , Ġalso Ġnamed Ġdomain Ġshift Ġproblem ĠB IB REF 10 , Ġbrings Ġabout Ġthe Ġzero - shot Ġtranslation Ġfailure Ġin Ġtransfer Ġlearning . ĠIt Ġis Ġbecause Ġtransfer Ġlearning Ġhas Ġno Ġexplicit Ġtraining Ġprocess Ġto Ġguarantee Ġthat Ġthe Ġsource Ġand Ġpivot Ġlanguages Ġshare Ġthe Ġsame Ġfeature Ġdistributions , Ġcausing Ġthat Ġthe Ġchild Ġmodel Ġinherited Ġfrom Ġthe Ġparent Ġmodel Ġfails Ġin Ġsuch Ġa Ġsituation . ĠFor Ġinstance , Ġas Ġillustrated Ġin Ġthe Ġleft Ġof ĠFigure ĠFIG REF 1 , Ġthe Ġpoints Ġof Ġthe Ġsentence Ġpair Ġwith Ġthe Ġsame Ġsemantics Ġare Ġnot Ġoverlapping Ġin Ġsource Ġspace , Ġresulting Ġin Ġthat Ġthe Ġshared Ġdec oder Ġwill Ġgenerate Ġdifferent Ġtranslations Ġden oted Ġby Ġdifferent Ġpoints Ġin Ġtarget Ġspace . ĠActually , Ġtransfer Ġlearning Ġfor ĠN MT Ġcan Ġbe Ġviewed Ġas Ġa Ġmulti - domain Ġproblem Ġwhere Ġeach Ġsource Ġlanguage Ġforms Ġa Ġnew Ġdomain . ĠMin im izing Ġthe Ġdiscrepancy Ġbetween Ġthe Ġfeature Ġdistributions Ġof Ġdifferent Ġsource Ġlanguages , Ġi . e ., Ġdifferent Ġdomains , Ġwill Ġensure Ġthe Ġsmooth Ġtransition Ġbetween Ġthe Ġparent Ġand Ġchild Ġmodels , Ġas Ġshown Ġin Ġthe Ġright Ġof ĠFigure ĠFIG REF 1 . ĠOne Ġway Ġto Ġachieve Ġthis Ġgoal Ġis Ġthe Ġfine - tun ing Ġtechnique , Ġwhich Ġforces Ġthe Ġmodel Ġto Ġforget Ġthe Ġspecific Ġknowledge Ġfrom Ġparent Ġdata Ġand Ġlearn Ġnew Ġfeatures Ġfrom Ġchild Ġdata . ĠHowever , Ġthe Ġdomain Ġshift Ġproblem Ġstill Ġexists , Ġand Ġthe Ġdemand Ġof Ġparallel Ġchild Ġdata Ġfor Ġfine - tun ing Ġheavily Ġhind ers Ġtransfer Ġlearning Ġfor ĠN MT Ġtowards Ġthe Ġzero - resource Ġsetting . </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe Ġtransfer Ġlearning Ġin Ġa Ġcommon Ġzero - shot Ġscenario Ġwhere Ġthere Ġare Ġa Ġlot Ġof Ġsource $ \ left right arrow Ġ$ p ivot Ġand Ġpivot $ \ left right arrow Ġ$ target Ġparallel Ġdata Ġbut Ġno Ġsource $ \ left right arrow Ġ$ target Ġparallel Ġdata . ĠIn Ġthis Ġscenario , Ġwe Ġpropose Ġa Ġsimple Ġbut Ġeffective Ġtransfer Ġapproach , Ġthe Ġkey Ġidea Ġof Ġwhich Ġis Ġto Ġrelieve Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem Ġby Ġmeans Ġof Ġcross - ling ual Ġpre - training . ĠTo Ġthis Ġend , Ġwe Ġfirst ly Ġinvestigate Ġthe Ġperformance Ġof Ġtwo Ġexisting Ġcross - ling ual Ġpre - training Ġmethods Ġproposed Ġby ĠB IB REF 11 Ġin Ġzero - shot Ġtranslation Ġscenario . ĠBesides , Ġa Ġnovel Ġpre - training Ġmethod Ġcalled ĠBR idge ĠLanguage ĠModel ing Ġ( B RL M ) Ġis Ġdesigned Ġto Ġmake Ġfull Ġuse Ġof Ġthe Ġsource $ \ left right arrow Ġ$ p ivot Ġbilingual Ġdata Ġto Ġobtain Ġa Ġuniversal Ġenc oder Ġfor Ġdifferent Ġlanguages . ĠOnce Ġthe Ġuniversal Ġenc oder Ġis Ġconstructed , Ġwe Ġonly Ġneed Ġto Ġtrain Ġthe Ġpivot $ \ right arrow Ġ$ target Ġmodel Ġand Ġthen Ġtest Ġthis Ġmodel Ġin Ġsource $ \ right arrow Ġ$ target Ġdirection Ġdirectly . ĠThe Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> We Ġpropose Ġa Ġnew Ġtransfer Ġlearning Ġapproach Ġfor ĠN MT Ġwhich Ġuses Ġthe Ġcross - ling ual Ġlanguage Ġmodel Ġpre - training Ġto Ġenable Ġa Ġhigh Ġperformance Ġon Ġzero - shot Ġtranslation . </s> We Ġpropose Ġa Ġnovel Ġpre - training Ġmethod Ġcalled ĠB RL M , Ġwhich Ġcan Ġeffectively Ġallev iates Ġthe Ġdistance Ġbetween Ġdifferent Ġsource Ġlanguage Ġspaces . </s> Our Ġproposed Ġapproach Ġsignificantly Ġimproves Ġzero - shot Ġtranslation Ġperformance , Ġconsistently Ġsurpass ing Ġpiv oting Ġand Ġmult ilingual Ġapproaches . ĠMeanwhile , Ġthe Ġperformance Ġon Ġsupervised Ġtranslation Ġdirection Ġremains Ġthe Ġsame Ġlevel Ġor Ġeven Ġbetter Ġwhen Ġusing Ġour Ġmethod . </s> Related ĠWork </s> In Ġrecent Ġyears , Ġzero - shot Ġtranslation Ġin ĠN MT Ġhas Ġattracted Ġwidespread Ġattention Ġin Ġacademic Ġresearch . ĠEx isting Ġmethods Ġare Ġmainly Ġdivided Ġinto Ġfour Ġcategories : Ġpivot - based Ġmethod , Ġtransfer Ġlearning , Ġmult ilingual ĠN MT , Ġand Ġun super vised ĠN MT . </s> P ivot - based ĠMethod Ġis Ġa Ġcommon Ġstrategy Ġto Ġobtain Ġa Ġsource $ \ right arrow Ġ$ target Ġmodel Ġby Ġintroducing Ġa Ġpivot Ġlanguage . ĠThis Ġapproach Ġis Ġfurther Ġdivided Ġinto Ġpiv oting Ġand Ġpivot - sy nt hetic . ĠWhile Ġthe Ġformer Ġfirst ly Ġtranslates Ġa Ġsource Ġlanguage Ġinto Ġthe Ġpivot Ġlanguage Ġwhich Ġis Ġlater Ġtranslated Ġto Ġthe Ġtarget Ġlanguage ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 12 , Ġthe Ġlatter Ġtrains Ġa Ġsource $ \ right arrow Ġ$ target Ġmodel Ġwith Ġpseudo Ġdata Ġgenerated Ġfrom Ġsource - p ivot Ġor Ġpivot - target Ġparallel Ġdata ĠB IB REF 13 , ĠB IB REF 14 . ĠAlthough Ġthe Ġpivot - based Ġmethods Ġcan Ġachieve Ġnot Ġbad Ġperformance , Ġit Ġalways Ġfalls Ġinto Ġa Ġcomputation - expensive Ġand Ġparameter - v ast Ġdilemma Ġof Ġquad r atic Ġgrowth Ġin Ġthe Ġnumber Ġof Ġsource Ġlanguages , Ġand Ġsuffers Ġfrom Ġthe Ġerror Ġpropagation Ġproblem ĠB IB REF 15 . </s> Transfer ĠLearning Ġis Ġfirst ly Ġintroduced Ġfor ĠN MT Ġby ĠB IB REF 6 , Ġwhich Ġlever ages Ġa Ġhigh - resource Ġparent Ġmodel Ġto Ġinitialize Ġthe Ġlow - resource Ġchild Ġmodel . ĠOn Ġthis Ġbasis , ĠB IB REF 7 Ġand ĠB IB REF 8 Ġuse Ġshared Ġvoc ab ular ies Ġfor Ġsource / target Ġlanguage Ġto Ġimprove Ġtransfer Ġlearning , Ġwhile ĠB IB REF 16 Ġrelieve Ġthe Ġvocabulary Ġmismatch Ġby Ġmainly Ġusing Ġcross - ling ual Ġword Ġembed ding . ĠAlthough Ġthese Ġmethods Ġare Ġsuccessful Ġin Ġthe Ġlow - resource Ġscene , Ġthey Ġhave Ġlimited Ġeffects Ġin Ġzero - shot Ġtranslation . </s> Mult ilingual ĠN MT Ġ( MN MT ) Ġenables Ġtraining Ġa Ġsingle Ġmodel Ġthat Ġsupports Ġtranslation Ġfrom Ġmultiple Ġsource Ġlanguages Ġinto Ġmultiple Ġtarget Ġlanguages , Ġeven Ġthose Ġunseen Ġlanguage Ġpairs ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠAside Ġfrom Ġsimpler Ġdeployment , ĠMN MT Ġbenefits Ġfrom Ġtransfer Ġlearning Ġwhere Ġlow - resource Ġlanguage Ġpairs Ġare Ġtrained Ġtogether Ġwith Ġhigh - resource Ġones . ĠHowever , ĠB IB REF 22 Ġpoint Ġout Ġthat ĠMN MT Ġfor Ġzero - shot Ġtranslation Ġeasily Ġfails , Ġand Ġis Ġsensitive Ġto Ġthe Ġhyper - param eter Ġsetting . ĠAlso , ĠMN MT Ġusually Ġperforms Ġworse Ġthan Ġthe Ġpivot - based Ġmethod Ġin Ġzero - shot Ġtranslation Ġsetting ĠB IB REF 23 . </s> Un super vised ĠN MT Ġ( UN MT ) Ġconsiders Ġa Ġharder Ġsetting , Ġin Ġwhich Ġonly Ġlarge - scale Ġmon oling ual Ġcorpor a Ġare Ġavailable Ġfor Ġtraining . ĠRecently , Ġmany Ġmethods Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof ĠUN MT , Ġincluding Ġusing Ġden o ising Ġauto - enc oder , Ġstatistic Ġmachine Ġtranslation Ġ( SM T ) Ġand Ġun super vised Ġpre - training ĠB IB REF 24 , ĠB IB REF 25 , ĠB IB REF 26 , ĠB IB REF 11 . ĠSince ĠUN MT Ġperforms Ġwell Ġbetween Ġsimilar Ġlanguages Ġ( e . g ., ĠEnglish - German Ġtranslation ), Ġits Ġperformance Ġbetween Ġdistant Ġlanguages Ġis Ġstill Ġfar Ġfrom Ġexpectation . </s> Our Ġproposed Ġmethod Ġbelongs Ġto Ġthe Ġtransfer Ġlearning , Ġbut Ġit Ġis Ġdifferent Ġfrom Ġtraditional Ġtransfer Ġmethods Ġwhich Ġtrain Ġa Ġparent Ġmodel Ġas Ġstarting Ġpoint . ĠBefore Ġtraining Ġa Ġparent Ġmodel , Ġour Ġapproach Ġfully Ġlever ages Ġcross - ling ual Ġpre - training Ġmethods Ġto Ġmake Ġall Ġsource Ġlanguages Ġshare Ġthe Ġsame Ġfeature Ġspace Ġand Ġthus Ġenables Ġa Ġsmooth Ġtransition Ġfor Ġzero - shot Ġtranslation . </s> App roach </s> In Ġthis Ġsection , Ġwe Ġwill Ġpresent Ġa Ġcross - ling ual Ġpre - training Ġbased Ġtransfer Ġapproach . ĠThis Ġmethod Ġis Ġdesigned Ġfor Ġa Ġcommon Ġzero - shot Ġscenario Ġwhere Ġthere Ġare Ġa Ġlot Ġof Ġsource $ \ left right arrow Ġ$ p ivot Ġand Ġpivot $ \ left right arrow Ġ$ target Ġbilingual Ġdata Ġbut Ġno Ġsource $ \ left right arrow Ġ$ target Ġparallel Ġdata , Ġand Ġthe Ġwhole Ġtraining Ġprocess Ġcan Ġbe Ġsummarized Ġas Ġfollows Ġstep Ġby Ġstep : </s> Pre - train Ġa Ġuniversal Ġenc oder Ġwith Ġsource / p ivot Ġmon oling ual Ġor Ġsource $ \ left right arrow Ġ$ p ivot Ġbilingual Ġdata . </s> Train Ġa Ġpivot $ \ right arrow Ġ$ target Ġparent Ġmodel Ġbuilt Ġon Ġthe Ġpre - trained Ġuniversal Ġenc oder Ġwith Ġthe Ġavailable Ġparallel Ġdata . ĠDuring Ġthe Ġtraining Ġprocess , Ġwe Ġfreeze Ġseveral Ġlayers Ġof Ġthe Ġpre - trained Ġuniversal Ġenc oder Ġto Ġavoid Ġthe Ġdegener acy Ġissue ĠB IB REF 27 . </s> Direct ly Ġtranslate Ġsource Ġsentences Ġinto Ġtarget Ġsentences Ġwith Ġthe Ġparent Ġmodel , Ġwhich Ġbenefits Ġfrom Ġthe Ġavailability Ġof Ġthe Ġuniversal Ġenc oder . </s> The Ġkey Ġdifficulty Ġof Ġthis Ġmethod Ġis Ġto Ġensure Ġthe Ġintermediate Ġrepresentations Ġof Ġthe Ġuniversal Ġenc oder Ġare Ġlanguage Ġinvari ant . ĠIn Ġthe Ġrest Ġof Ġthis Ġsection , Ġwe Ġfirst Ġpresent Ġtwo Ġexisting Ġmethods Ġyet Ġto Ġbe Ġexplored Ġin Ġzero - shot Ġtranslation , Ġand Ġthen Ġpropose Ġa Ġstraightforward Ġbut Ġeffective Ġcross - ling ual Ġpre - training Ġmethod . ĠIn Ġthe Ġend , Ġwe Ġpresent Ġthe Ġwhole Ġtraining Ġand Ġinference Ġprotocol Ġfor Ġtransfer . </s> App roach Ġ: :: ĠMask ed Ġand ĠTranslation ĠLanguage ĠModel ĠPret raining </s> Two Ġexisting Ġcross - ling ual Ġpre - training Ġmethods , ĠMask ed ĠLanguage ĠModel ing Ġ( ML M ) Ġand ĠTranslation ĠLanguage ĠModel ing Ġ( TL M ), Ġhave Ġshown Ġtheir Ġeffectiveness Ġon ĠX N LI Ġcross - ling ual Ġclassification Ġtask ĠB IB REF 11 , ĠB IB REF 28 , Ġbut Ġthese Ġmethods Ġhave Ġnot Ġbeen Ġwell Ġstudied Ġon Ġcross - ling ual Ġgeneration Ġtasks Ġin Ġzero - shot Ġcondition . ĠWe Ġattempt Ġto Ġtake Ġadvantage Ġof Ġthe Ġcross - ling ual Ġability Ġof Ġthe Ġtwo Ġmethods Ġfor Ġzero - shot Ġtranslation . </s> Specifically , ĠML M Ġadop ts Ġthe ĠClo ze Ġobjective Ġof ĠB ERT ĠB IB REF 29 Ġand Ġpredicts Ġthe Ġmasked Ġwords Ġthat Ġare Ġrandomly Ġselected Ġand Ġreplaced Ġwith Ġ[ MAS K ] Ġtoken Ġon Ġmon oling ual Ġcorpus . ĠIn Ġpractice , ĠML M Ġtakes Ġdifferent Ġlanguage Ġmon oling ual Ġcorpor a Ġas Ġinput Ġto Ġfind Ġfeatures Ġshared Ġacross Ġdifferent Ġlanguages . ĠWith Ġthis Ġmethod , Ġword Ġpieces Ġshared Ġin Ġall Ġlanguages Ġhave Ġbeen Ġmapped Ġinto Ġa Ġshared Ġspace , Ġwhich Ġmakes Ġthe Ġsentence Ġrepresentations Ġacross Ġdifferent Ġlanguages Ġclose ĠB IB REF 30 . </s> Since ĠML M Ġobjective Ġis Ġun super vised Ġand Ġonly Ġrequires Ġmon oling ual Ġdata , ĠTL M Ġis Ġdesigned Ġto Ġleverage Ġparallel Ġdata Ġwhen Ġit Ġis Ġavailable . ĠActually , ĠTL M Ġis Ġa Ġsimple Ġextension Ġof ĠML M , Ġwith Ġthe Ġdifference Ġthat ĠTL M Ġconc aten ates Ġsentence Ġpair Ġinto Ġa Ġwhole Ġsentence , Ġand Ġthen Ġrandomly Ġmasks Ġwords Ġin Ġboth Ġthe Ġsource Ġand Ġtarget Ġsentences . ĠIn Ġthis Ġway , Ġthe Ġmodel Ġcan Ġeither Ġattend Ġto Ġsurrounding Ġwords Ġor Ġto Ġthe Ġtranslation Ġsentence , Ġimplicitly Ġencouraging Ġthe Ġmodel Ġto Ġalign Ġthe Ġsource Ġand Ġtarget Ġlanguage Ġrepresentations . ĠNote Ġthat Ġalthough Ġeach Ġsentence Ġpair Ġis Ġformed Ġinto Ġone Ġsentence , Ġthe Ġpositions Ġof Ġthe Ġtarget Ġsentence Ġare Ġreset Ġto Ġcount Ġform Ġzero . </s> App roach Ġ: :: ĠBridge ĠLanguage ĠModel ĠPret raining </s> Aside Ġfrom ĠML M Ġand ĠTL M , Ġwe Ġpropose ĠBR idge ĠLanguage ĠModel ing Ġ( B RL M ) Ġto Ġfurther Ġobtain Ġword - level Ġrepresentation Ġalignment Ġbetween Ġdifferent Ġlanguages . ĠThis Ġmethod Ġis Ġinspired Ġby Ġthe Ġassumption Ġthat Ġif Ġthe Ġfeature Ġspaces Ġof Ġdifferent Ġlanguages Ġare Ġaligned Ġvery Ġwell , Ġthe Ġmasked Ġwords Ġin Ġthe Ġcorrupted Ġsentence Ġcan Ġalso Ġbe Ġguessed Ġby Ġthe Ġcontext Ġof Ġthe Ġcorrespond ingly Ġaligned Ġwords Ġon Ġthe Ġother Ġside . ĠTo Ġachieve Ġthis Ġgoal , ĠB RL M Ġis Ġdesigned Ġto Ġstrengthen Ġthe Ġability Ġto Ġinfer Ġwords Ġacross Ġlanguages Ġbased Ġon Ġalignment Ġinformation , Ġinstead Ġof Ġinfer ring Ġwords Ġwithin Ġmon oling ual Ġsentence Ġas Ġin ĠML M Ġor Ġwithin Ġthe Ġpseudo Ġsentence Ġformed Ġby Ġconc aten ating Ġsentence Ġpair Ġas Ġin ĠTL M . </s> As Ġillustrated Ġin ĠFigure ĠFIG REF 9 , ĠB RL M Ġstacks Ġshared Ġenc oder Ġover Ġboth Ġside Ġsentences Ġseparately . ĠIn Ġparticular , Ġwe Ġdesign Ġtwo Ġnetwork Ġstructures Ġfor ĠB RL M , Ġwhich Ġare Ġdivided Ġinto ĠHard ĠAl ignment Ġ( B RL M - HA ) Ġand ĠSoft ĠAl ignment Ġ( B RL M - SA ) Ġaccording Ġto Ġthe Ġway Ġof Ġgenerating Ġthe Ġalignment Ġinformation . ĠThese Ġtwo Ġstructures Ġactually Ġextend ĠML M Ġinto Ġa Ġbilingual Ġscenario , Ġwith Ġthe Ġdifference Ġthat ĠB RL M Ġlever ages Ġexternal Ġalign er Ġtool Ġor Ġadditional Ġattention Ġlayer Ġto Ġexplicitly Ġintroduce Ġalignment Ġinformation Ġduring Ġmodel Ġtraining . </s> Hard ĠAl ignment Ġ( B RL M - HA ). ĠWe Ġfirst Ġuse Ġexternal Ġalign er Ġtool Ġon Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata Ġto Ġextract Ġthe Ġalignment Ġinformation Ġof Ġsentence Ġpair . ĠDuring Ġmodel Ġtraining , Ġgiven Ġsource $ \ left right arrow Ġ$ p ivot Ġsentence Ġpair , ĠB RL M - HA Ġrandomly Ġmasks Ġsome Ġwords Ġin Ġsource Ġsentence Ġand Ġlever ages Ġalignment Ġinformation Ġto Ġobtain Ġthe Ġaligned Ġwords Ġin Ġpivot Ġsentence Ġfor Ġmasked Ġwords . ĠBased Ġon Ġthe Ġprocessed Ġinput , ĠB RL M - HA Ġadop ts Ġthe ĠTrans former ĠB IB REF 1 Ġenc oder Ġto Ġgain Ġthe Ġhidden Ġstates Ġfor Ġsource Ġand Ġpivot Ġsentences Ġrespectively . ĠThen Ġthe Ġtraining Ġobjective Ġof ĠB RL M - HA Ġis Ġto Ġpredict Ġthe Ġmasked Ġwords Ġby Ġnot Ġonly Ġthe Ġsurrounding Ġwords Ġin Ġsource Ġsentence Ġbut Ġalso Ġthe Ġenc oder Ġoutputs Ġof Ġthe Ġaligned Ġwords . ĠNote Ġthat Ġthis Ġtraining Ġprocess Ġis Ġalso Ġcarried Ġout Ġin Ġa Ġsymm etric Ġsituation , Ġin Ġwhich Ġwe Ġmask Ġsome Ġwords Ġin Ġpivot Ġsentence Ġand Ġobtain Ġthe Ġaligned Ġwords Ġin Ġthe Ġsource Ġsentence . </s> Soft ĠAl ignment Ġ( B RL M - SA ). ĠInstead Ġof Ġusing Ġexternal Ġalign er Ġtool , ĠB RL M - SA Ġintroduces Ġan Ġadditional Ġattention Ġlayer Ġto Ġlearn Ġthe Ġalignment Ġinformation Ġtogether Ġwith Ġmodel Ġtraining . ĠIn Ġthis Ġway , ĠB RL M - SA Ġavoids Ġthe Ġeffect Ġcaused Ġby Ġexternal Ġwrong Ġalignment Ġinformation Ġand Ġenables Ġmany - to - one Ġsoft Ġalignment Ġduring Ġmodel Ġtraining . ĠSimilar Ġwith ĠB RL M - HA , Ġthe Ġtraining Ġobjective Ġof ĠB RL M - SA Ġis Ġto Ġpredict Ġthe Ġmasked Ġwords Ġby Ġnot Ġonly Ġthe Ġsurrounding Ġwords Ġin Ġsource Ġsentence Ġbut Ġalso Ġthe Ġoutputs Ġof Ġattention Ġlayer . ĠIn Ġour Ġimplementation , Ġthe Ġattention Ġlayer Ġis Ġa Ġmulti - head Ġattention Ġlayer Ġadopted Ġin ĠTrans former , Ġwhere Ġthe Ġqueries Ġcome Ġfrom Ġthe Ġmasked Ġsource Ġsentence , Ġthe Ġkeys Ġand Ġvalues Ġcome Ġfrom Ġthe Ġpivot Ġsentence . </s> In Ġprinciple , ĠML M Ġand ĠTL M Ġcan Ġlearn Ġsome Ġimplicit Ġalignment Ġinformation Ġduring Ġmodel Ġtraining . ĠHowever , Ġthe Ġalignment Ġprocess Ġin ĠML M Ġis Ġinefficient Ġsince Ġthe Ġshared Ġword Ġpieces Ġonly Ġaccount Ġfor Ġa Ġsmall Ġproportion Ġof Ġthe Ġwhole Ġcorpus , Ġresulting Ġin Ġthe Ġdifficulty Ġof Ġexpanding Ġthe Ġshared Ġinformation Ġto Ġalign Ġthe Ġwhole Ġcorpus . ĠTL M Ġalso Ġlacks Ġeffort Ġin Ġalignment Ġbetween Ġthe Ġsource Ġand Ġtarget Ġsentences Ġsince ĠTL M Ġconc aten ates Ġthe Ġsentence Ġpair Ġinto Ġone Ġsequence , Ġmaking Ġthe Ġexplicit Ġalignment Ġbetween Ġthe Ġsource Ġand Ġtarget Ġinf eas ible . ĠB RL M Ġfully Ġutilizes Ġthe Ġalignment Ġinformation Ġto Ġobtain Ġbetter Ġword - level Ġrepresentation Ġalignment Ġbetween Ġdifferent Ġlanguages , Ġwhich Ġbetter Ġrelie ves Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem . </s> App roach Ġ: :: ĠTransfer ĠProtocol </s> We Ġconsider Ġthe Ġtypical Ġzero - shot Ġtranslation Ġscenario Ġin Ġwhich Ġa Ġhigh Ġresource Ġpivot Ġlanguage Ġhas Ġparallel Ġdata Ġwith Ġboth Ġsource Ġand Ġtarget Ġlanguages , Ġwhile Ġsource Ġand Ġtarget Ġlanguages Ġhas Ġno Ġparallel Ġdata Ġbetween Ġthemselves . ĠOur Ġproposed Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproach Ġfor Ġsource $ \ right arrow Ġ$ target Ġzero - shot Ġtranslation Ġis Ġmainly Ġdivided Ġinto Ġtwo Ġphrases : Ġthe Ġpret raining Ġphase Ġand Ġthe Ġtransfer Ġphase . </s> In Ġthe Ġpret raining Ġphase , Ġwe Ġfirst Ġpret rain ĠML M Ġon Ġmon oling ual Ġcorpor a Ġof Ġboth Ġsource Ġand Ġpivot Ġlanguages , Ġand Ġcontinue Ġto Ġpret rain ĠTL M Ġor Ġthe Ġproposed ĠB RL M Ġon Ġthe Ġavailable Ġparallel Ġdata Ġbetween Ġsource Ġand Ġpivot Ġlanguages , Ġin Ġorder Ġto Ġbuild Ġa Ġcross - ling ual Ġenc oder Ġshared Ġby Ġthe Ġsource Ġand Ġpivot Ġlanguages . </s> In Ġthe Ġtransfer Ġphase , Ġwe Ġtrain Ġpivot $ \ right arrow Ġ$ target ĠN MT Ġmodel Ġinitialized Ġby Ġthe Ġcross - ling ually Ġpre - trained Ġenc oder , Ġand Ġfinally Ġtransfer Ġthe Ġtrained ĠN MT Ġmodel Ġto Ġsource $ \ right arrow Ġ$ target Ġtranslation Ġthanks Ġto Ġthe Ġshared Ġenc oder . ĠNote Ġthat Ġduring Ġtraining Ġpivot $ \ right arrow Ġ$ target ĠN MT Ġmodel , Ġwe Ġfreeze Ġseveral Ġlayers Ġof Ġthe Ġcross - ling ually Ġpre - trained Ġenc oder Ġto Ġavoid Ġthe Ġdegener acy Ġissue . </s> For Ġthe Ġmore Ġcomplicated Ġscenario Ġthat Ġeither Ġthe Ġsource Ġside Ġor Ġthe Ġtarget Ġside Ġhas Ġmultiple Ġlanguages , Ġthe Ġenc oder Ġand Ġthe Ġdec oder Ġare Ġalso Ġshared Ġacross Ġeach Ġside Ġlanguages Ġfor Ġefficient Ġdeployment Ġof Ġtranslation Ġbetween Ġmultiple Ġlanguages . </s> Exper iments Ġ: :: ĠSetup </s> We Ġevaluate Ġour Ġcross - ling ual Ġpre - training Ġbased Ġtransfer Ġapproach Ġagainst Ġseveral Ġstrong Ġbas elines Ġon Ġtwo Ġpublic Ġdat ats ets , ĠEuro par l ĠB IB REF 31 Ġand ĠMulti UN ĠB IB REF 32 , Ġwhich Ġcontain Ġmulti - par allel Ġevaluation Ġdata Ġto Ġassess Ġthe Ġzero - shot Ġperformance . ĠIn Ġall Ġexperiments , Ġwe Ġuse ĠB LE U Ġas Ġthe Ġautomatic Ġmetric Ġfor Ġtranslation Ġevaluation . </s> Exper iments Ġ: :: ĠSetup Ġ: :: ĠDat as ets . </s> The Ġstatistics Ġof ĠEuro par l Ġand ĠMulti UN Ġcorpor a Ġare Ġsummarized Ġin ĠTable ĠT AB REF 18 . ĠFor ĠEuro par l Ġcorpus , Ġwe Ġevaluate Ġon ĠFrench - English - Spanish Ġ( Fr - En - Es ), ĠGerman - English - French Ġ( De - En - Fr ) Ġand ĠRomanian - English - German Ġ( Ro - En - De ), Ġwhere ĠEnglish Ġacts Ġas Ġthe Ġpivot Ġlanguage , Ġits Ġleft Ġside Ġis Ġthe Ġsource Ġlanguage , Ġand Ġits Ġright Ġside Ġis Ġthe Ġtarget Ġlanguage . ĠWe Ġremove Ġthe Ġmulti - par allel Ġsentences Ġbetween Ġdifferent Ġtraining Ġcorpor a Ġto Ġensure Ġzero - shot Ġsettings . ĠWe Ġuse Ġthe Ġdev test 2006 Ġas Ġthe Ġvalidation Ġset Ġand Ġthe Ġtest 2006 Ġas Ġthe Ġtest Ġset Ġfor ĠFr $ \ right arrow Ġ$ Es Ġand ĠDe $ \ right arrow Ġ$ Fr . ĠFor Ġdistant Ġlanguage Ġpair ĠRo $ \ right arrow Ġ$ De , Ġwe Ġextract Ġ1 , 000 Ġoverlapping Ġsentences Ġfrom Ġnew st est 2016 Ġas Ġthe Ġtest Ġset Ġand Ġthe Ġ2 , 000 Ġoverlapping Ġsentences Ġsplit Ġfrom Ġthe Ġtraining Ġset Ġas Ġthe Ġvalidation Ġset Ġsince Ġthere Ġis Ġno Ġofficial Ġvalidation Ġand Ġtest Ġsets . ĠFor Ġvocabulary , Ġwe Ġuse Ġ60 K Ġsub - word Ġtokens Ġbased Ġon ĠByte ĠPair ĠEnc oding Ġ( B PE ) ĠB IB REF 33 . </s> For ĠMulti UN Ġcorpus , Ġwe Ġuse Ġfour Ġlanguages : ĠEnglish Ġ( En ) Ġis Ġset Ġas Ġthe Ġpivot Ġlanguage , Ġwhich Ġhas Ġparallel Ġdata Ġwith Ġother Ġthree Ġlanguages Ġwhich Ġdo Ġnot Ġhave Ġparallel Ġdata Ġbetween Ġeach Ġother . ĠThe Ġthree Ġlanguages Ġare ĠArabic Ġ( Ar ), ĠSpanish Ġ( Es ), Ġand ĠRussian Ġ( Ru ), Ġand Ġmutual Ġtranslation Ġbetween Ġthemselves Ġconstitutes Ġsix Ġzero - shot Ġtranslation Ġdirection Ġfor Ġevaluation . ĠWe Ġuse Ġ80 K ĠB PE Ġsplits Ġas Ġthe Ġvocabulary . ĠNote Ġthat Ġall Ġsentences Ġare Ġtoken ized Ġby Ġthe Ġtoken ize . per l Ġscript , Ġand Ġwe Ġlower case Ġall Ġdata Ġto Ġavoid Ġa Ġlarge Ġvocabulary Ġfor Ġthe ĠMulti UN Ġcorpus . </s> Exper iments Ġ: :: ĠSetup Ġ: :: ĠExperimental ĠDetails . </s> We Ġuse Ġtraditional Ġtransfer Ġlearning , Ġpivot - based Ġmethod Ġand Ġmult ilingual ĠN MT Ġas Ġour Ġbas elines . ĠFor Ġthe Ġfair Ġcomparison , Ġthe ĠTrans former - big Ġmodel Ġwith Ġ1024 Ġembed ding / hidden Ġunits , Ġ4096 Ġfeed - forward Ġfilter Ġsize , Ġ6 Ġlayers Ġand Ġ8 Ġheads Ġper Ġlayer Ġis Ġadopted Ġfor Ġall Ġtranslation Ġmodels Ġin Ġour Ġexperiments . ĠWe Ġset Ġthe Ġbatch Ġsize Ġto Ġ2400 Ġper Ġbatch Ġand Ġlimit Ġsentence Ġlength Ġto Ġ100 ĠB PE Ġtokens . ĠWe Ġset Ġthe Ġ$\ text { att n }\ _ \ text { drop } = 0 $ Ġ( a Ġdrop out Ġrate Ġon Ġeach Ġattention Ġhead ), Ġwhich Ġis Ġfavorable Ġto Ġthe Ġzero - shot Ġtranslation Ġand Ġhas Ġno Ġeffect Ġon Ġsupervised Ġtranslation Ġdirections ĠB IB REF 22 . ĠFor Ġthe Ġmodel Ġinitialization , Ġwe Ġuse ĠFacebook 's Ġcross - ling ual Ġpret rained Ġmodels Ġreleased Ġby ĠXL M Ġto Ġinitialize Ġthe Ġenc oder Ġpart , Ġand Ġthe Ġrest Ġparameters Ġare Ġinitialized Ġwith Ġx avier Ġuniform . ĠWe Ġemploy Ġthe ĠAdam Ġoptim izer Ġwith Ġ$\ text { lr } = 0 . 0001 $, Ġ$ t _{ \ text { warm }\ _ \ text { up }} = 4000 $ Ġand Ġ$\ text { drop out } = 0 . 1 $. ĠAt Ġdecoding Ġtime , Ġwe Ġgenerate Ġgreed ily Ġwith Ġlength Ġpenalty Ġ$\ alpha Ġ= 1 . 0 $. </s> Regarding ĠML M , ĠTL M Ġand ĠB RL M , Ġas Ġmentioned Ġin Ġthe Ġpre - training Ġphase Ġof Ġtransfer Ġprotocol , Ġwe Ġfirst Ġpre - train ĠML M Ġon Ġmon oling ual Ġdata Ġof Ġboth Ġsource Ġand Ġpivot Ġlanguages , Ġthen Ġleverage Ġthe Ġparameters Ġof ĠML M Ġto Ġinitialize ĠTL M Ġand Ġthe Ġproposed ĠB RL M , Ġwhich Ġare Ġcontinued Ġto Ġbe Ġoptimized Ġwith Ġsource - p ivot Ġbilingual Ġdata . ĠIn Ġour Ġexperiments , Ġwe Ġuse ĠML M + TL M , ĠML M + B RL M Ġto Ġrepresent Ġthis Ġtraining Ġprocess . ĠFor Ġthe Ġmask ing Ġstrategy Ġduring Ġtraining , Ġfollowing ĠB IB REF 29 , Ġ$ 15 \ % $ Ġof ĠB PE Ġtokens Ġare Ġselected Ġto Ġbe Ġmasked . ĠAmong Ġthe Ġselected Ġtokens , Ġ$ 80 \ % $ Ġof Ġthem Ġare Ġreplaced Ġwith Ġ[ MAS K ] Ġtoken , Ġ$ 10 \ % $ Ġare Ġreplaced Ġwith Ġa Ġrandom ĠB PE Ġtoken , Ġand Ġ$ 10 \ % $ Ġunchanged . ĠThe Ġprediction Ġaccuracy Ġof Ġmasked Ġwords Ġis Ġused Ġas Ġa Ġstopping Ġcriterion Ġin Ġthe Ġpre - training Ġstage . ĠBesides , Ġwe Ġuse Ġfast align Ġtool ĠB IB REF 34 Ġto Ġextract Ġword Ġalign ments Ġfor ĠB RL M - HA . </s> Exper iments Ġ: :: ĠMain ĠResults </s> Table ĠT AB REF 19 Ġand ĠT AB REF 26 Ġreport Ġzero - shot Ġresults Ġon ĠEuro par l Ġand ĠMulti - UN Ġevaluation Ġsets , Ġrespectively . ĠWe Ġcompare Ġour Ġapproaches Ġwith Ġrelated Ġapproaches Ġof Ġpiv oting , Ġmult ilingual ĠN MT Ġ( MN MT ) ĠB IB REF 19 , Ġand Ġcross - ling ual Ġtransfer Ġwithout Ġpret raining ĠB IB REF 16 . ĠThe Ġresults Ġshow Ġthat Ġour Ġapproaches Ġconsistently Ġoutper form Ġother Ġapproaches Ġacross Ġlanguages Ġand Ġdatasets , Ġespecially Ġsurpass Ġpiv oting , Ġwhich Ġis Ġa Ġstrong Ġbaseline Ġin Ġthe Ġzero - shot Ġscenario Ġthat Ġmult ilingual ĠN MT Ġsystems Ġoften Ġfail Ġto Ġbeat ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 23 . ĠP iv oting Ġtranslates Ġsource Ġto Ġpivot Ġthen Ġto Ġtarget Ġin Ġtwo Ġsteps , Ġcausing Ġinefficient Ġtranslation Ġprocess . ĠOur Ġapproaches Ġuse Ġone Ġenc oder - dec oder Ġmodel Ġto Ġtranslate Ġbetween Ġany Ġzero - shot Ġdirections , Ġwhich Ġis Ġmore Ġefficient Ġthan Ġpiv oting . ĠRegarding Ġthe Ġcomparison Ġbetween Ġtransfer Ġapproaches , Ġour Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġoutper forms Ġtransfer Ġmethod Ġthat Ġdoes Ġnot Ġuse Ġpret raining Ġby Ġa Ġlarge Ġmargin . </s> Exper iments Ġ: :: ĠMain ĠResults Ġ: :: ĠResults Ġon ĠEuro par l ĠDat as et . </s> Regarding Ġcomparison Ġbetween Ġthe Ġbas elines Ġin Ġtable ĠT AB REF 19 , Ġwe Ġfind Ġthat Ġpiv oting Ġis Ġthe Ġstrongest Ġbaseline Ġthat Ġhas Ġsignificant Ġadvantage Ġover Ġother Ġtwo Ġbas elines . ĠCross - ling ual Ġtransfer Ġfor Ġlanguages Ġwithout Ġshared Ġvoc ab ular ies ĠB IB REF 16 Ġmanifests Ġthe Ġworst Ġperformance Ġbecause Ġof Ġnot Ġusing Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata , Ġwhich Ġis Ġutilized Ġas Ġbeneficial Ġsupervised Ġsignal Ġfor Ġthe Ġother Ġtwo Ġbas elines . </s> Our Ġbest Ġapproach Ġof ĠML M + B RL M - SA Ġachieves Ġthe Ġsignificant Ġsuperior Ġperformance Ġto Ġall Ġbas elines Ġin Ġthe Ġzero - shot Ġdirections , Ġimproving Ġby Ġ0 . 9 - 4 . 8 ĠB LE U Ġpoints Ġover Ġthe Ġstrong Ġpiv oting . ĠMeanwhile , Ġin Ġthe Ġsupervised Ġdirection Ġof Ġpivot $ \ right arrow Ġ$ target , Ġour Ġapproaches Ġperforms Ġeven Ġbetter Ġthan Ġthe Ġoriginal Ġsupervised ĠTrans former Ġthanks Ġto Ġthe Ġshared Ġenc oder Ġtrained Ġon Ġboth Ġlarge - scale Ġmon oling ual Ġdata Ġand Ġparallel Ġdata Ġbetween Ġmultiple Ġlanguages . </s> ML M Ġalone Ġthat Ġdoes Ġnot Ġuse Ġsource $ \ left right arrow Ġ$ p ivot Ġparallel Ġdata Ġperforms Ġmuch Ġbetter Ġthan Ġthe Ġcross - ling ual Ġtransfer , Ġand Ġachieves Ġcomparable Ġresults Ġto Ġpiv oting . ĠWhen ĠML M Ġis Ġcombined Ġwith ĠTL M Ġor Ġthe Ġproposed ĠB RL M , Ġthe Ġperformance Ġis Ġfurther Ġimproved . ĠML M + B RL M - SA Ġperforms Ġthe Ġbest , Ġand Ġis Ġbetter Ġthan ĠML M + B RL M - HA Ġindicating Ġthat Ġsoft Ġalignment Ġis Ġhelpful Ġthan Ġhard Ġalignment Ġfor Ġthe Ġcross - ling ual Ġpret raining . </s> Exper iments Ġ: :: ĠMain ĠResults Ġ: :: ĠResults Ġon ĠMulti UN ĠDat as et . </s> Like Ġexperimental Ġresults Ġon ĠEuro par l , ĠML M + B RL M - SA Ġperforms Ġthe Ġbest Ġamong Ġall Ġproposed Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproaches Ġas Ġshown Ġin ĠTable ĠT AB REF 26 . ĠWhen Ġcomparing Ġsystems Ġconsisting Ġof Ġone Ġenc oder - dec oder Ġmodel Ġfor Ġall Ġzero - shot Ġtranslation , Ġour Ġapproaches Ġperforms Ġsignificantly Ġbetter Ġthan ĠMN MT ĠB IB REF 19 . </s> Although Ġit Ġis Ġchallenging Ġfor Ġone Ġmodel Ġto Ġtranslate Ġall Ġzero - shot Ġdirections Ġbetween Ġmultiple Ġdistant Ġlanguage Ġpairs Ġof ĠMulti UN , ĠML M + B RL M - SA Ġstill Ġachieves Ġbetter Ġperformances Ġon ĠEs Ġ$\ right arrow Ġ$ ĠAr Ġand ĠEs Ġ$\ right arrow Ġ$ ĠRu Ġthan Ġstrong Ġpiv oting $ _{ \ rm Ġm } $, Ġwhich Ġuses ĠMN MT Ġto Ġtranslate Ġsource Ġto Ġpivot Ġthen Ġto Ġtarget Ġin Ġtwo Ġseparate Ġsteps Ġwith Ġeach Ġstep Ġreceiving Ġsupervised Ġsignal Ġof Ġparallel Ġcorpor a . ĠOur Ġapproaches Ġsurpass Ġpiv oting $ _{ \ rm Ġm } $ Ġin Ġall Ġzero - shot Ġdirections Ġby Ġadding Ġback Ġtranslation ĠB IB REF 33 Ġto Ġgenerate Ġpseudo Ġparallel Ġsentences Ġfor Ġall Ġzero - shot Ġdirections Ġbased Ġon Ġour Ġpret rained Ġmodels Ġsuch Ġas ĠML M + B RL M - SA , Ġand Ġfurther Ġtraining Ġour Ġuniversal Ġenc oder - dec oder Ġmodel Ġwith Ġthese Ġpseudo Ġdata . ĠB IB REF 22 Ġgu 2019 impro ved Ġintroduces Ġback Ġtranslation Ġinto ĠMN MT , Ġwhile Ġwe Ġadopt Ġit Ġin Ġour Ġtransfer Ġapproaches . ĠFinally , Ġour Ġbest ĠML M + B RL M - SA Ġwith Ġback Ġtranslation Ġoutper forms Ġpiv oting $ _{ \ rm Ġm } $ Ġby Ġ2 . 4 ĠB LE U Ġpoints Ġaverage ly , Ġand Ġoutper forms ĠMN MT ĠB IB REF 22 Ġby Ġ4 . 6 ĠB LE U Ġpoints Ġaverage ly . ĠAgain , Ġin Ġsupervised Ġtranslation Ġdirections , ĠML M + B RL M - SA Ġwith Ġback Ġtranslation Ġalso Ġachieves Ġbetter Ġperformance Ġthan Ġthe Ġoriginal Ġsupervised ĠTrans former . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠSent ence ĠRepresent ation . </s> We Ġfirst Ġevaluate Ġthe Ġrepresent ational Ġinvari ance Ġacross Ġlanguages Ġfor Ġall Ġcross - ling ual Ġpre - training Ġmethods . ĠFollowing ĠB IB REF 23 , Ġwe Ġadopt Ġmax - pool ing Ġoperation Ġto Ġcollect Ġthe Ġsentence Ġrepresentation Ġof Ġeach Ġenc oder Ġlayer Ġfor Ġall Ġsource - p ivot Ġsentence Ġpairs Ġin Ġthe ĠEuro par l Ġvalidation Ġsets . ĠThen Ġwe Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġfor Ġeach Ġsentence Ġpair Ġand Ġaverage Ġall Ġcos ine Ġscores . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 27 , Ġwe Ġcan Ġobserve Ġthat , ĠML M + B RL M - SA Ġhas Ġthe Ġmost Ġstable Ġand Ġsimilar Ġcross - ling ual Ġrepresentations Ġof Ġsentence Ġpairs Ġon Ġall Ġlayers , Ġwhile Ġit Ġachieves Ġthe Ġbest Ġperformance Ġin Ġzero - shot Ġtranslation . ĠThis Ġdemonstrates Ġthat Ġbetter Ġcross - ling ual Ġrepresentations Ġcan Ġbenefit Ġfor Ġthe Ġprocess Ġof Ġtransfer Ġlearning . ĠBesides , ĠML M + B RL M - HA Ġis Ġnot Ġas Ġsuperior Ġas ĠML M + B RL M - SA Ġand Ġeven Ġworse Ġthan ĠML M + TL M Ġon ĠFr - En , Ġsince ĠML M + B RL M - HA Ġmay Ġsuffer Ġfrom Ġthe Ġwrong Ġalignment Ġknowledge Ġfrom Ġan Ġexternal Ġalign er Ġtool . ĠWe Ġalso Ġfind Ġan Ġinteresting Ġphenomenon Ġthat Ġas Ġthe Ġnumber Ġof Ġlayers Ġincreases , Ġthe Ġcos ine Ġsimilarity Ġdecreases . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠContext ual ized ĠWord ĠRepresent ation . </s> We Ġfurther Ġsample Ġan ĠEnglish - Russian Ġsentence Ġpair Ġfrom Ġthe ĠMulti UN Ġvalidation Ġsets Ġand Ġvisualize Ġthe Ġcos ine Ġsimilarity Ġbetween Ġhidden Ġstates Ġof Ġthe Ġtop Ġenc oder Ġlayer Ġto Ġfurther Ġinvestigate Ġthe Ġdifference Ġof Ġall Ġcross - ling ual Ġpre - training Ġmethods . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 38 , Ġthe Ġhidden Ġstates Ġgenerated Ġby ĠML M + B RL M - SA Ġhave Ġhigher Ġsimilarity Ġfor Ġtwo Ġaligned Ġwords . ĠIt Ġindicates Ġthat ĠML M + B RL M - SA Ġcan Ġgain Ġbetter Ġword - level Ġrepresentation Ġalignment Ġbetween Ġsource Ġand Ġpivot Ġlanguages , Ġwhich Ġbetter Ġrelie ves Ġthe Ġburden Ġof Ġthe Ġdomain Ġshift Ġproblem . </s> Exper iments Ġ: :: ĠAnalysis Ġ: :: ĠThe ĠEffect Ġof ĠFree zing ĠParameters . </s> To Ġfreeze Ġparameters Ġis Ġa Ġcommon Ġstrategy Ġto Ġavoid Ġcatastrophic Ġforgetting Ġin Ġtransfer Ġlearning ĠB IB REF 27 . ĠTable ĠT AB REF 43 Ġshows Ġthe Ġperformance Ġof Ġtransfer Ġlearning Ġwith Ġfreezing Ġdifferent Ġlayers Ġon ĠMulti UN Ġtest Ġset , Ġin Ġwhich ĠEn $ \ right arrow Ġ$ Ru Ġdenotes Ġthe Ġparent Ġmodel , ĠAr $ \ right arrow Ġ$ Ru Ġand ĠEs $ \ right arrow Ġ$ Ru Ġare Ġtwo Ġchild Ġmodels , Ġand Ġall Ġmodels Ġare Ġbased Ġon ĠML M + B RL M - SA . ĠWe Ġcan Ġfind Ġthat Ġupdating Ġall Ġparameters Ġduring Ġtraining Ġwill Ġcause Ġa Ġnotable Ġdrop Ġon Ġthe Ġzero - shot Ġdirection Ġdue Ġto Ġthe Ġcatastrophic Ġforgetting . ĠOn Ġthe Ġcontrary , Ġfreezing Ġall Ġthe Ġparameters Ġleads Ġto Ġthe Ġdecline Ġon Ġsupervised Ġdirection Ġbecause Ġthe Ġlanguage Ġfeatures Ġextracted Ġduring Ġpre - training Ġis Ġnot Ġsufficient Ġfor ĠMT Ġtask . ĠFree zing Ġthe Ġfirst Ġfour Ġlayers Ġof Ġthe Ġtransformer Ġshows Ġthe Ġbest Ġperformance Ġand Ġkeeps Ġthe Ġbalance Ġbetween Ġpre - training Ġand Ġfine - tun ing . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġcross - ling ual Ġpret raining Ġbased Ġtransfer Ġapproach Ġfor Ġthe Ġchallenging Ġzero - shot Ġtranslation Ġtask , Ġin Ġwhich Ġsource Ġand Ġtarget Ġlanguages Ġhave Ġno Ġparallel Ġdata , Ġwhile Ġthey Ġboth Ġhave Ġparallel Ġdata Ġwith Ġa Ġhigh Ġresource Ġpivot Ġlanguage . ĠWith Ġthe Ġaim Ġof Ġbuilding Ġthe Ġlanguage Ġinvari ant Ġrepresentation Ġbetween Ġsource Ġand Ġpivot Ġlanguages Ġfor Ġsmooth Ġtransfer Ġof Ġthe Ġparent Ġmodel Ġof Ġpivot $ \ right arrow Ġ$ target Ġdirection Ġto Ġthe Ġchild Ġmodel Ġof Ġsource $ \ right arrow Ġ$ target Ġdirection , Ġwe Ġintroduce Ġone Ġmon oling ual Ġpret raining Ġmethod Ġand Ġtwo Ġbilingual Ġpret raining Ġmethods Ġto Ġconstruct Ġan Ġuniversal Ġenc oder Ġfor Ġthe Ġsource Ġand Ġpivot Ġlanguages . ĠExper iments Ġon Ġpublic Ġdatasets Ġshow Ġthat Ġour Ġapproaches Ġsignificantly Ġoutper forms Ġseveral Ġstrong Ġbaseline Ġsystems , Ġand Ġmanifest Ġthe Ġlanguage Ġinvari ance Ġcharacteristics Ġin Ġboth Ġsentence Ġlevel Ġand Ġword Ġlevel Ġneural Ġrepresentations . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġthe Ġhelpful Ġcomments . ĠThis Ġwork Ġwas Ġsupported Ġby ĠNational ĠKey ĠR & D ĠProgram Ġof ĠChina Ġ( Grant ĠNo . Ġ2016 Y FE 013 21 00 ), ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( Grant ĠNo . Ġ6 15 25 205 , Ġ6 16 73 289 ). ĠThis Ġwork Ġwas Ġalso Ġpartially Ġsupported Ġby ĠAlibaba ĠGroup Ġthrough ĠAlibaba ĠInnov ative ĠResearch ĠProgram Ġand Ġthe ĠPriority ĠAcademic ĠProgram ĠDevelopment Ġ( P APD ) Ġof ĠJiang su ĠHigher ĠEducation ĠInstit utions .