translation probabilities, Labeled Attachment Scores (LAS)
EXTRACTIVE
accuracy, Labeled Attachment Scores (LAS)
EXTRACTIVE
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities. Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13. We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.
Table TABREF32 shows the XNLI test accuracy. For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6. Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only. We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.
2002.07306
Introduction </s> Pre - trained Ġmodels ĠB IB REF 0 , ĠB IB REF 1 Ġhave Ġreceived Ġmuch Ġof Ġattention Ġrecently Ġthanks Ġto Ġtheir Ġimpressive Ġresults Ġin Ġmany Ġdown Ġstream ĠN LP Ġtasks . ĠAdditionally , Ġmult ilingual Ġpre - trained Ġmodels Ġenable Ġmany ĠN LP Ġapplications Ġfor Ġother Ġlanguages Ġvia Ġzero - short Ġcross - ling ual Ġtransfer . ĠZero - shot Ġcross - ling ual Ġtransfer Ġhas Ġshown Ġpromising Ġresults Ġfor Ġrapidly Ġbuilding Ġapplications Ġfor Ġlow Ġresource Ġlanguages . ĠB IB REF 2 Ġshow Ġthe Ġpotential Ġof Ġmult ilingual - BER T ĠB IB REF 0 Ġin Ġzero - shot Ġtransfer Ġfor Ġa Ġlarge Ġnumber Ġof Ġlanguages Ġfrom Ġdifferent Ġlanguage Ġfamilies Ġon Ġfive ĠN LP Ġtasks , Ġnamely , Ġnatural Ġlanguage Ġinference , Ġdocument Ġclassification , Ġnamed Ġentity Ġrecognition , Ġpart - of - speech Ġtagging , Ġand Ġdependency Ġparsing . </s> Although Ġmult ilingual Ġmodels Ġare Ġan Ġimportant Ġingredient Ġfor Ġenhancing Ġlanguage Ġtechnology Ġin Ġmany Ġlanguages , Ġrecent Ġresearch Ġon Ġimproving Ġpre - trained Ġmodels Ġputs Ġmuch Ġemphasis Ġon ĠEnglish ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 . ĠThe Ġcurrent Ġstate Ġof Ġaffairs Ġmakes Ġit Ġdifficult Ġto Ġtranslate Ġadvancements Ġin Ġpre - training Ġfrom ĠEnglish Ġto Ġnon - English Ġlanguages . ĠTo Ġour Ġbest Ġknowledge , Ġthere Ġare Ġonly Ġthree Ġavailable Ġmult ilingual Ġpre - trained Ġmodels Ġto Ġdate : Ġ( 1 ) Ġthe Ġmult ilingual - BER T Ġ( m BER T ) Ġthat Ġsupports Ġ104 Ġlanguages , Ġ( 2 ) Ġcross - ling ual Ġlanguage Ġmodel ĠB IB REF 6 Ġthat Ġsupports Ġ100 Ġlanguages , Ġand Ġ( 3 ) ĠLanguage ĠAg nostic ĠSE nt ence ĠRepresent ations ĠB IB REF 7 Ġthat Ġsupports Ġ93 Ġlanguages . ĠAmong Ġthe Ġthree Ġmodels , ĠL AS ER Ġis Ġbased Ġon Ġneural Ġmachine Ġtranslation Ġapproach Ġand Ġstrictly Ġrequires Ġparallel Ġdata Ġto Ġtrain . </s> Do Ġmult ilingual Ġmodels Ġalways Ġneed Ġto Ġbe Ġtrained Ġfrom Ġscratch ? ĠCan Ġwe Ġtransfer Ġlinguistic Ġknowledge Ġlearned Ġby ĠEnglish Ġpre - trained Ġmodels Ġto Ġother Ġlanguages ? ĠIn Ġthis Ġwork , Ġwe Ġdevelop Ġa Ġtechnique Ġto Ġrapidly Ġtransfer Ġan Ġexisting Ġpre - trained Ġmodel Ġfrom ĠEnglish Ġto Ġother Ġlanguages Ġin Ġan Ġenergy Ġefficient Ġway ĠB IB REF 8 . ĠAs Ġthe Ġfirst Ġstep , Ġwe Ġfocus Ġon Ġbuilding Ġa Ġbilingual Ġlanguage Ġmodel Ġ( LM ) Ġof ĠEnglish Ġand Ġa Ġtarget Ġlanguage . ĠStarting Ġfrom Ġa Ġpre - trained ĠEnglish ĠLM , Ġwe Ġlearn Ġthe Ġtarget Ġlanguage Ġspecific Ġparameters Ġ( i . e ., Ġword Ġembed d ings ), Ġwhile Ġkeeping Ġthe Ġenc oder Ġlayers Ġof Ġthe Ġpre - trained ĠEnglish ĠLM Ġfixed . ĠWe Ġthen Ġfine - t une Ġboth ĠEnglish Ġand Ġtarget Ġmodel Ġto Ġobtain Ġthe Ġbilingual ĠLM . ĠWe Ġapply Ġour Ġapproach Ġto Ġaut oen c oding Ġlanguage Ġmodels Ġwith Ġmasked Ġlanguage Ġmodel Ġobjective Ġand Ġshow Ġthe Ġadvantage Ġof Ġthe Ġproposed Ġapproach Ġin Ġzero - shot Ġtransfer . ĠOur Ġmain Ġcontributions Ġin Ġthis Ġwork Ġare : </s> We Ġpropose Ġa Ġfast Ġadaptation Ġmethod Ġfor Ġobtaining Ġa Ġbilingual ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġof ĠEnglish Ġand Ġa Ġtarget Ġlanguage Ġwithin Ġa Ġday Ġusing Ġone ĠTesla ĠV 100 Ġ16 GB ĠGPU . </s> We Ġevaluate Ġour Ġbilingual ĠL Ms Ġfor Ġsix Ġlanguages Ġon Ġtwo Ġzero - shot Ġcross - ling ual Ġtransfer Ġtasks , Ġnamely Ġnatural Ġlanguage Ġinference ĠB IB REF 9 Ġand Ġuniversal Ġdependency Ġparsing . ĠWe Ġshow Ġthat Ġour Ġmodels Ġoffer Ġcompetitive Ġperformance Ġor Ġeven Ġbetter Ġthat Ġm BER T . </s> We Ġillustrate Ġthat Ġour Ġbilingual ĠL Ms Ġcan Ġserve Ġas Ġan Ġexcellent Ġfeature Ġextract or Ġin Ġsupervised Ġdependency Ġparsing Ġtask . </s> B ilingual ĠPre - trained ĠL Ms </s> We Ġfirst Ġprovide Ġsome Ġbackground Ġof Ġpre - trained Ġlanguage Ġmodels . ĠLet Ġ$_ e $ Ġbe ĠEnglish Ġword - embed d ings Ġand Ġ$\ Ps i Ġ() $ Ġbe Ġthe ĠTrans former ĠB IB REF 10 Ġenc oder Ġwith Ġparameters Ġ$$ . ĠLet Ġ$ _{ w _ i } $ Ġdenote Ġthe Ġembed ding Ġof Ġword Ġ$ w _ i $ Ġ( i . e ., Ġ$ _{ w _ i } Ġ= Ġ_ e [ w _ 1 ] $ ). ĠWe Ġomit Ġpositional Ġembed d ings Ġand Ġbias Ġfor Ġclarity . ĠA Ġpre - trained ĠLM Ġtypically Ġperforms Ġthe Ġfollowing Ġcomput ations : Ġ( i ) Ġtransform Ġa Ġsequence Ġof Ġinput Ġtokens Ġto Ġcontextual ized Ġrepresentations Ġ$ [ _{ w _ 1 }, \ d ots Ġ, _{ w _ n } ] Ġ= Ġ\ Ps i Ġ( _{ w _ 1 }, Ġ\ d ots Ġ, Ġ_ { w _ n }; Ġ) $, Ġand Ġ( ii ) Ġpredict Ġan Ġoutput Ġword Ġ$ y _ i $ Ġat Ġ$ i ^ {\ text { th }} $ Ġposition Ġ$ p ( y _ i Ġ| Ġ_ { w _ i }) Ġ\ pro pt o Ġ\ exp Ġ( _{ w _ i } ^ \ top Ġ_ { y _ i }) $. </s> Aut oen c oding ĠLM ĠB IB REF 0 Ġcorrupt s Ġsome Ġinput Ġtokens Ġ$ w _ i $ Ġby Ġreplacing Ġthem Ġwith Ġa Ġspecial Ġtoken Ġ[ MAS K ]. ĠIt Ġthen Ġpredicts Ġthe Ġoriginal Ġtokens Ġ$ y _ i Ġ= Ġw _ i $ Ġfrom Ġthe Ġcorrupted Ġtokens . ĠAut ore gressive ĠLM ĠB IB REF 3 Ġpredicts Ġthe Ġnext Ġtoken Ġ($ y _ i Ġ= Ġw _{ i + 1 } $ ) Ġgiven Ġall Ġthe Ġprevious Ġtokens . ĠThe Ġrecently Ġproposed ĠXL Net Ġmodel ĠB IB REF 5 Ġis Ġan Ġaut ore gressive ĠLM Ġthat Ġfactor izes Ġoutput Ġwith Ġall Ġpossible Ġperm utations , Ġwhich Ġshows Ġempirical Ġperformance Ġimprovement Ġover ĠG PT - 2 Ġdue Ġto Ġthe Ġability Ġto Ġcapture Ġbid irection al Ġcontext . ĠHere Ġwe Ġassume Ġthat Ġthe Ġenc oder Ġperforms Ġnecessary Ġmask ing Ġwith Ġrespect Ġto Ġeach Ġtraining Ġobjective . </s> Given Ġan ĠEnglish Ġpre - trained ĠLM , Ġwe Ġwish Ġto Ġlearn Ġa Ġbilingual ĠLM Ġfor ĠEnglish Ġand Ġa Ġgiven Ġtarget Ġlanguage Ġ$ f $ Ġunder Ġa Ġlimited Ġcomputational Ġresource Ġbudget . ĠTo Ġquickly Ġbuild Ġa Ġbilingual ĠLM , Ġwe Ġdirectly Ġadapt Ġthe ĠEnglish Ġpre - train d Ġmodel Ġto Ġthe Ġtarget Ġmodel . ĠOur Ġapproach Ġconsists Ġof Ġthree Ġsteps . ĠFirst , Ġwe Ġinitialize Ġtarget Ġlanguage Ġword - embed d ings Ġ$_ f $ Ġin Ġthe ĠEnglish Ġembed ding Ġspace Ġsuch Ġthat Ġembed d ings Ġof Ġa Ġtarget Ġword Ġand Ġits ĠEnglish Ġequivalents Ġare Ġclose Ġtogether Ġ( Â§ SEC REF 8 ). ĠNext , Ġwe Ġcreate Ġa Ġtarget ĠLM Ġfrom Ġthe Ġtarget Ġembed d ings Ġand Ġthe ĠEnglish Ġenc oder Ġ$\ Ps i Ġ() $. ĠWe Ġthen Ġfine - t une Ġtarget Ġembed d ings Ġwhile Ġkeeping Ġ$\ Ps i Ġ() $ Ġfixed Ġ( Â§ SEC REF 14 ). ĠFinally , Ġwe Ġconstruct Ġa Ġbilingual ĠLM Ġof Ġ$_ e $, Ġ$_ f $, Ġand Ġ$\ Ps i Ġ() $ Ġand Ġfine - t une Ġall Ġthe Ġparameters Ġ( Â§ SEC REF 15 ). ĠFigure ĠFIG REF 7 Ġillustrates Ġthe Ġlast Ġtwo Ġsteps Ġin Ġour Ġapproach . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings </s> Our Ġapproach Ġto Ġlearn Ġthe Ġinitial Ġforeign Ġword Ġembed d ings Ġ$_ f Ġ\ in Ġ^ { | V _ f | Ġ\ times Ġd } $ Ġis Ġbased Ġon Ġthe Ġidea Ġof Ġmapping Ġthe Ġtrained ĠEnglish Ġword Ġembed d ings Ġ$_ e Ġ\ in Ġ^ { | V _ e | Ġ\ times Ġd } $ Ġto Ġ$_ f $ Ġsuch Ġthat Ġif Ġa Ġforeign Ġword Ġand Ġan ĠEnglish Ġword Ġare Ġsimilar Ġin Ġmeaning Ġthen Ġtheir Ġembed d ings Ġare Ġsimilar . ĠB orrow ing Ġthe Ġidea Ġof Ġuniversal Ġlex ical Ġsharing Ġfrom ĠB IB REF 11 , Ġwe Ġrepresent Ġeach Ġforeign Ġword Ġembed ding Ġ$_ f [ i ] Ġ\ in Ġ^ d $ Ġas Ġa Ġlinear Ġcombination Ġof ĠEnglish Ġword Ġembed d ings Ġ$_ e [ j ] Ġ\ in Ġ^ d $ </s> where Ġ$_ i \ in Ġ^ { | V _ e | } $ Ġis Ġa Ġsparse Ġvector Ġand Ġ$\ sum Ġ_ j ^{ | V _ e | } Ġ\ alpha Ġ_ { ij } Ġ= Ġ1 $. </s> In Ġthis Ġstep Ġof Ġinitial izing Ġforeign Ġembed d ings , Ġhaving Ġa Ġgood Ġestimation Ġof Ġ$$ Ġcould Ġspeed Ġof Ġthe Ġconvergence Ġwhen Ġtuning Ġthe Ġforeign Ġmodel Ġand Ġenable Ġzero - shot Ġtransfer Ġ( Â§ SEC REF 5 ). ĠIn Ġthe Ġfollowing , Ġwe Ġdiscuss Ġhow Ġto Ġestimate Ġ$_ i \ ; \ for all Ġi \ in Ġ\ l brace Ġ1 , 2 , Ġ\ d ots Ġ, Ġ| V _ f | \ r brace Ġ$ Ġunder Ġtwo Ġscenarios : Ġ( i ) Ġwe Ġhave Ġparallel Ġdata Ġof ĠEnglish - foreign , Ġand Ġ( ii ) Ġwe Ġonly Ġrely Ġon ĠEnglish Ġand Ġforeign Ġmon oling ual Ġdata . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings Ġ: :: ĠLearning Ġfrom ĠParallel ĠCorpus </s> Given Ġan ĠEnglish - foreign Ġparallel Ġcorpus , Ġwe Ġcan Ġestimate Ġword Ġtranslation Ġprobability Ġ$ p ( e \ , | \ , f ) $ Ġfor Ġany Ġ( English - foreign ) Ġpair Ġ$( e , Ġf ) $ Ġusing Ġpopular Ġword - al ignment ĠB IB REF 12 Ġtool k its Ġsuch Ġas Ġfast - align ĠB IB REF 13 . ĠWe Ġthen Ġassign : </s> Since Ġ$_ i $ Ġis Ġestimated Ġfrom Ġword Ġalignment , Ġit Ġis Ġa Ġsparse Ġvector . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings Ġ: :: ĠLearning Ġfrom ĠMon oling ual ĠCorpus </s> For Ġlow Ġresource Ġlanguages , Ġparallel Ġdata Ġmay Ġnot Ġbe Ġavailable . ĠIn Ġthis Ġcase , Ġwe Ġrely Ġonly Ġon Ġmon oling ual Ġdata Ġ( e . g ., ĠWik iped ias ). ĠWe Ġestimate Ġword Ġtranslation Ġprobabilities Ġfrom Ġword Ġembed d ings Ġof Ġthe Ġtwo Ġlanguages . ĠWord Ġvectors Ġof Ġthese Ġlanguages Ġcan Ġbe Ġlearned Ġusing Ġfast Text ĠB IB REF 14 Ġand Ġthen Ġare Ġaligned Ġinto Ġa Ġshared Ġspace Ġwith ĠEnglish ĠB IB REF 15 , ĠB IB REF 16 . ĠUnlike Ġlearning Ġcontextual ized Ġrepresentations , Ġlearning Ġword Ġvectors Ġis Ġfast Ġand Ġcomput ationally Ġcheap . ĠGiven Ġthe Ġaligned Ġvectors Ġ$\ bar { } _ f $ Ġof Ġforeign Ġand Ġ$\ bar { } _ e $ Ġof ĠEnglish , Ġwe Ġcalculate Ġthe Ġword Ġtranslation Ġmatrix Ġ$\ in Ġ^ { | V _ f | \ times Ġ| V _ e | } $ Ġas </s> Here , Ġwe Ġuse Ġ$\ math rm Ġ{ s parse max } $ ĠB IB REF 17 Ġinstead Ġof Ġsoft max . ĠSp arse max Ġis Ġa Ġsparse Ġversion Ġof Ġsoft max Ġand Ġit Ġputs Ġzero Ġprobabilities Ġon Ġmost Ġof Ġthe Ġword Ġin Ġthe ĠEnglish Ġvocabulary Ġexcept Ġfew ĠEnglish Ġwords Ġthat Ġare Ġsimilar Ġto Ġa Ġgiven Ġforeign Ġword . ĠThis Ġproperty Ġis Ġdesirable Ġin Ġour Ġapproach Ġsince Ġit Ġleads Ġto Ġa Ġbetter Ġinitialization Ġof Ġthe Ġforeign Ġembed d ings . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠFine - tun ing ĠTarget ĠEmb edd ings </s> After Ġinitial izing Ġforeign Ġword - embed d ings , Ġwe Ġreplace ĠEnglish Ġword - embed d ings Ġin Ġthe ĠEnglish Ġpre - trained ĠLM Ġwith Ġforeign Ġword - embed d ings Ġto Ġobtain Ġthe Ġforeign ĠLM . ĠWe Ġthen Ġfine - t une Ġonly Ġforeign Ġword - embed d ings Ġon Ġmon oling ual Ġdata . ĠThe Ġtraining Ġobjective Ġis Ġthe Ġsame Ġas Ġthe Ġtraining Ġobjective Ġof Ġthe ĠEnglish Ġpre - trained ĠLM Ġ( i . e ., Ġmasked ĠLM Ġfor ĠB ERT ). ĠSince Ġthe Ġtrained Ġenc oder Ġ$\ Ps i Ġ() $ Ġis Ġgood Ġat Ġcapturing Ġassociation , Ġthe Ġpurpose Ġof Ġthis Ġstep Ġis Ġto Ġfurther Ġoptimize Ġtarget Ġembed d ings Ġsuch Ġthat Ġthe Ġtarget ĠLM Ġcan Ġutilized Ġthe Ġtrained Ġenc oder Ġfor Ġassociation Ġtask . ĠFor Ġexample , Ġif Ġthe Ġwords ĠAlbert ĠCam us Ġpresented Ġin Ġa ĠFrench Ġinput Ġsequence , Ġthe Ġself - att ention Ġin Ġthe Ġenc oder Ġmore Ġlikely Ġattends Ġto Ġwords Ġabsurd e Ġand Ġexistential ism e Ġonce Ġtheir Ġembed d ings Ġare Ġtuned . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠFine - tun ing ĠB ilingual ĠLM </s> We Ġcreate Ġa Ġbilingual ĠLM Ġby Ġplug ging Ġforeign Ġlanguage Ġspecific Ġparameters Ġto Ġthe Ġpre - trained ĠEnglish ĠLM Ġ( Figure ĠFIG REF 7 ). ĠThe Ġnew Ġmodel Ġhas Ġtwo Ġseparate Ġembed ding Ġlayers Ġand Ġoutput Ġlayers , Ġone Ġfor ĠEnglish Ġand Ġone Ġfor Ġforeign Ġlanguage . ĠThe Ġenc oder Ġlayer Ġin Ġbetween Ġis Ġshared . ĠWe Ġthen Ġfine - t une Ġthis Ġmodel Ġusing ĠEnglish Ġand Ġforeign Ġmon oling ual Ġdata . ĠHere , Ġwe Ġkeep Ġtuning Ġthe Ġmodel Ġon ĠEnglish Ġto Ġensure Ġthat Ġit Ġdoes Ġnot Ġforget Ġwhat Ġit Ġhas Ġlearned Ġin ĠEnglish Ġand Ġthat Ġwe Ġcan Ġuse Ġthe Ġresulting Ġmodel Ġfor Ġzero - shot Ġtransfer Ġ( Â§ SEC REF 3 ). ĠIn Ġthis Ġstep , Ġthe Ġenc oder Ġparameters Ġare Ġalso Ġupdated Ġso Ġthat Ġin Ġcan Ġlearn Ġsynt actic Ġaspects Ġ( i . e ., Ġword Ġorder , Ġmorph ological Ġagreement ) Ġof Ġthe Ġtarget Ġlanguages . </s> Zero - shot ĠExper iments </s> We Ġbuild Ġour Ġbilingual ĠL Ms , Ġnamed ĠRAM EN , Ġstarting Ġfrom ĠB ERT $ _{ \ text sc Ġ{ base }} $, ĠB ERT $ _{ \ text sc Ġ{ large }} $, ĠRo BER Ta $ _{ \ text sc Ġ{ base }} $, Ġand ĠRo BER Ta $ _{ \ text sc Ġ{ large }} $ Ġpre - trained Ġmodels . ĠUsing ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġallows Ġus Ġto Ġcompare Ġthe Ġresults Ġwith Ġm BER T Ġmodel . ĠUsing ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġand ĠRo BER Ta Ġallows Ġus Ġto Ġinvestigate Ġwhether Ġthe Ġperformance Ġof Ġthe Ġtarget ĠLM Ġcorrelates Ġwith Ġthe Ġperformance Ġof Ġthe Ġsource ĠLM . ĠWe Ġevaluate Ġour Ġmodels Ġon Ġtwo Ġcross - ling ual Ġzero - shot Ġtasks : Ġ( 1 ) ĠCross - ling ual ĠNatural ĠLanguage ĠIn ference Ġ( X N LI ) Ġand Ġ( 2 ) Ġdependency Ġparsing . </s> Zero - shot ĠExper iments Ġ: :: ĠData </s> We Ġevaluate Ġour Ġapproach Ġfor Ġsix Ġtarget Ġlanguages : ĠFrench Ġ( fr ), ĠRussian Ġ( ru ), ĠArabic Ġ( ar ), ĠChinese Ġ( zh ), ĠHindi Ġ( hi ), Ġand ĠVietnamese Ġ( vi ). ĠThese Ġlanguages Ġbelong Ġto Ġfour Ġdifferent Ġlanguage Ġfamilies . ĠFrench , ĠRussian , Ġand ĠHindi Ġare ĠIndo - European Ġlanguages , Ġsimilar Ġto ĠEnglish . ĠArabic , ĠChinese , Ġand ĠVietnamese Ġbelong Ġto ĠAf ro - As i atic , ĠS ino - T ib et an , Ġand ĠAust ro - As i atic Ġfamily Ġrespectively . ĠThe Ġchoice Ġof Ġthe Ġsix Ġlanguages Ġalso Ġreflects Ġdifferent Ġtraining Ġconditions Ġdepending Ġon Ġthe Ġamount Ġof Ġmon oling ual Ġdata . ĠFrench Ġand ĠRussian , Ġand ĠArabic Ġcan Ġbe Ġregarded Ġas Ġhigh Ġresource Ġlanguages Ġwhereas ĠHindi Ġhas Ġfar Ġless Ġdata Ġand Ġcan Ġbe Ġconsidered Ġas Ġlow Ġresource . </s> For Ġexperiments Ġthat Ġuse Ġparallel Ġdata Ġto Ġinitialize Ġforeign Ġspecific Ġparameters , Ġwe Ġuse Ġthe Ġsame Ġdatasets Ġin Ġthe Ġwork Ġof ĠB IB REF 6 . ĠSpecifically , Ġwe Ġuse ĠUnited ĠNations ĠParallel ĠCorpus ĠB IB REF 18 Ġfor Ġen - ru , Ġen - ar , Ġen - zh , Ġand Ġen - fr . ĠWe Ġcollect Ġen - hi Ġparallel Ġdata Ġfrom ĠI IT ĠBombay Ġcorpus ĠB IB REF 19 Ġand Ġen - vi Ġdata Ġfrom ĠOpen Sub t itles Ġ2018 . ĠFor Ġexperiments Ġthat Ġuse Ġonly Ġmon oling ual Ġdata Ġto Ġinitialize Ġforeign Ġparameters , Ġinstead Ġof Ġtraining Ġword - ve ctors Ġfrom Ġthe Ġscratch , Ġwe Ġuse Ġthe Ġpre - trained Ġword Ġvectors Ġfrom Ġfast Text ĠB IB REF 14 Ġto Ġestimate Ġword Ġtranslation Ġprobabilities Ġ( E q . ĠDIS PLAY _ FORM 13 ). ĠWe Ġalign Ġthese Ġvectors Ġinto Ġa Ġcommon Ġspace Ġusing Ġorth og onal ĠPro cr ust es ĠB IB REF 20 , ĠB IB REF 15 , ĠB IB REF 16 . ĠWe Ġonly Ġuse Ġidentical Ġwords Ġbetween Ġthe Ġtwo Ġlanguages Ġas Ġthe Ġsupervised Ġsignal . ĠWe Ġuse ĠWiki Ext ractor Ġto Ġextract Ġextract Ġraw Ġsentences Ġfrom ĠWik iped ias Ġas Ġmon oling ual Ġdata Ġfor Ġfine - tun ing Ġtarget Ġembed d ings Ġand Ġbilingual ĠL Ms Ġ( Â§ SEC REF 15 ). ĠWe Ġdo Ġnot Ġlower case Ġor Ġremove Ġaccents Ġin Ġour Ġdata Ġpre processing Ġpipeline . </s> We Ġtoken ize ĠEnglish Ġusing Ġthe Ġprovided Ġtoken izer Ġfrom Ġpre - trained Ġmodels . ĠFor Ġtarget Ġlanguages , Ġwe Ġuse Ġfast B PE Ġto Ġlearn Ġ30 , 000 ĠB PE Ġcodes Ġand Ġ50 , 000 Ġcodes Ġwhen Ġtransferring Ġfrom ĠB ERT Ġand ĠRo BER Ta Ġrespectively . ĠWe Ġtrunc ate Ġthe ĠB PE Ġvocabulary Ġof Ġforeign Ġlanguages Ġto Ġmatch Ġthe Ġsize Ġof Ġthe ĠEnglish Ġvocabulary Ġin Ġthe Ġsource Ġmodels . ĠPrec ise ly , Ġthe Ġsize Ġof Ġforeign Ġvocabulary Ġis Ġset Ġto Ġ32 , 000 Ġwhen Ġtransferring Ġfrom ĠB ERT Ġand Ġ50 , 000 Ġwhen Ġtransferring Ġfrom ĠRo BER Ta . </s> We Ġuse ĠX N LI Ġdataset ĠB IB REF 9 Ġfor Ġclassification Ġtask Ġand ĠUniversal ĠDepend encies Ġv 2 . 4 ĠB IB REF 21 Ġfor Ġparsing Ġtask . ĠSince Ġa Ġlanguage Ġmight Ġhave Ġmore Ġthan Ġone Ġtree bank Ġin ĠUniversal ĠDepend encies , Ġwe Ġuse Ġthe Ġfollowing Ġtree banks : Ġen _ ew t Ġ( English ), Ġfr _ gs d Ġ( French ), Ġru _ sy nt ag rus Ġ( Russian ) Ġar _ pad t Ġ( Arab ic ), Ġvi _ vt b Ġ( V iet namese ), Ġhi _ h dt b Ġ( H ind i ), Ġand Ġz h _ gs d Ġ( Chinese ). </s> Zero - shot ĠExper iments Ġ: :: ĠData Ġ: :: ĠRem ark Ġon ĠB PE </s> BI B REF 22 Ġshow Ġthat Ġsharing Ġsub words Ġbetween Ġlanguages Ġimproves Ġalign ments Ġbetween Ġembed ding Ġspaces . ĠB IB REF 2 Ġobserve Ġa Ġstrong Ġcorrelation Ġbetween Ġthe Ġpercentage Ġof Ġoverlapping Ġsub words Ġand Ġm BER T 's Ġperformances Ġfor Ġcross - ling ual Ġzero - shot Ġtransfer . ĠHowever , Ġin Ġour Ġcurrent Ġapproach , Ġsub words Ġbetween Ġsource Ġand Ġtarget Ġare Ġnot Ġshared . ĠA Ġsub word Ġthat Ġis Ġin Ġboth ĠEnglish Ġand Ġforeign Ġvocabulary Ġhas Ġtwo Ġdifferent Ġembed d ings . </s> Zero - shot ĠExper iments Ġ: :: ĠEst imating Ġtranslation Ġprobabilities </s> Since Ġpre - trained Ġmodels Ġoperate Ġon Ġsub word Ġlevel , Ġwe Ġneed Ġto Ġestimate Ġsub word Ġtranslation Ġprobabilities . ĠTherefore , Ġwe Ġsubs ample Ġ2 M Ġsentence Ġpairs Ġfrom Ġeach Ġparallel Ġcorpus Ġand Ġtoken ize Ġthe Ġdata Ġinto Ġsub words Ġbefore Ġrunning Ġfast - align ĠB IB REF 13 . </s> Est imating Ġsub word Ġtranslation Ġprobabilities Ġfrom Ġaligned Ġword Ġvectors Ġrequires Ġan Ġadditional Ġprocessing Ġstep Ġsince Ġthe Ġprovided Ġvectors Ġfrom Ġfast Text Ġare Ġnot Ġat Ġsub word Ġlevel . ĠWe Ġuse Ġthe Ġfollowing Ġapproximation Ġto Ġobtain Ġsub word Ġvectors : Ġthe Ġvector Ġ$_ s $ Ġof Ġsub word Ġ$ s $ Ġis Ġthe Ġweighted Ġaverage Ġof Ġall Ġthe Ġaligned Ġword Ġvectors Ġ$ _{ w _ i } $ Ġthat Ġhave Ġ$ s $ Ġas Ġan Ġsub word </s> where Ġ$ p ( w _ j ) $ Ġis Ġthe Ġun ig ram Ġprobability Ġof Ġword Ġ$ w _ j $ Ġand Ġ$ n _ s Ġ= Ġ\ sum Ġ_ { w _ j :\ , Ġs \ in Ġw _ j } Ġp ( w _ j ) $. ĠWe Ġtake Ġthe Ġtop Ġ50 , 000 Ġwords Ġin Ġeach Ġaligned Ġword - ve ctors Ġto Ġcompute Ġsub word Ġvectors . </s> In Ġboth Ġcases , Ġnot Ġall Ġthe Ġwords Ġin Ġthe Ġforeign Ġvocabulary Ġcan Ġbe Ġinitialized Ġfrom Ġthe ĠEnglish Ġword - embed d ings . ĠThose Ġwords Ġare Ġinitialized Ġrandomly Ġfrom Ġa ĠGa ussian Ġ$\ math cal Ġ{ N } ( 0 , Ġ{ 1 }{ d ^ 2 }) $. </s> Zero - shot ĠExper iments Ġ: :: ĠHyper - param eters </s> In Ġall Ġthe Ġexperiments , Ġwe Ġtune ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġfor Ġ175 , 000 Ġupdates Ġand ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġfor Ġ275 , 000 Ġupdates Ġwhere Ġthe Ġfirst Ġ25 , 000 Ġupdates Ġare Ġfor Ġlanguage Ġspecific Ġparameters . ĠThe Ġsequence Ġlength Ġis Ġset Ġto Ġ256 . ĠThe Ġmini - batch Ġsize Ġare Ġ64 Ġand Ġ24 Ġwhen Ġtuning Ġlanguage Ġspecific Ġparameters Ġusing ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġand ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġrespectively . ĠFor Ġtuning Ġbilingual ĠL Ms , Ġwe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ64 Ġfor ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġand Ġ24 Ġfor ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġwhere Ġhalf Ġof Ġthe Ġbatch Ġare ĠEnglish Ġsequences Ġand Ġthe Ġother Ġhalf Ġare Ġforeign Ġsequences . ĠThis Ġstrategy Ġof Ġbalancing Ġmini - batch Ġhas Ġbeen Ġused Ġin Ġmult ilingual Ġneural Ġmachine Ġtranslation ĠB IB REF 23 , ĠB IB REF 24 . </s> We Ġoptimize ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġusing ĠLook ahead Ġoptim izer ĠB IB REF 25 Ġwrapped Ġaround ĠAdam Ġwith Ġthe Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 4 } $, Ġthe Ġnumber Ġof Ġfast Ġweight Ġupdates Ġ$ k = 5 $, Ġand Ġinterpol ation Ġparameter Ġ$\ alpha Ġ= 0 . 5 $. ĠWe Ġchoose ĠLook ahead Ġoptim izer Ġbecause Ġit Ġhas Ġbeen Ġshown Ġto Ġbe Ġrobust Ġto Ġthe Ġinitial Ġparameters Ġof Ġthe Ġbased Ġoptim izer Ġ( Adam ). ĠFor ĠAdam Ġoptim izer , Ġwe Ġlin early Ġincrease Ġthe Ġlearning Ġrate Ġfrom Ġ$ 10 ^{ - 7 } $ Ġto Ġ$ 10 ^{ - 4 } $ Ġin Ġthe Ġfirst Ġ4000 Ġupdates Ġand Ġthen Ġfollow Ġan Ġinverse Ġsquare Ġroot Ġdecay . ĠAll ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġmodels Ġare Ġoptimized Ġwith ĠAdam Ġdue Ġto Ġmemory Ġlimit . </s> When Ġfine - tun ing ĠRAM EN Ġon ĠX N LI Ġand ĠUD , Ġwe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ32 , ĠAdam 's Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 5 } $. ĠThe Ġnumber Ġof Ġepoch s Ġare Ġset Ġto Ġ4 Ġand Ġ50 Ġfor ĠX N LI Ġand ĠUD Ġtasks Ġrespectively . ĠAll Ġexperiments Ġare Ġcarried Ġout Ġon Ġa Ġsingle ĠTesla ĠV 100 Ġ16 GB ĠGPU . ĠEach ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġmodel Ġis Ġtrained Ġwithin Ġa Ġday Ġand Ġeach ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġis Ġtrained Ġwithin Ġtwo Ġdays . </s> Results </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġresults Ġof Ġout Ġmodels Ġfor Ġtwo Ġzero - shot Ġcross Ġling ual Ġtransfer Ġtasks : ĠX N LI Ġand Ġdependency Ġparsing . </s> Results Ġ: :: ĠCross - ling ual ĠNatural ĠLanguage ĠIn ference </s> Table ĠT AB REF 32 Ġshows Ġthe ĠX N LI Ġtest Ġaccuracy . ĠFor Ġreference , Ġwe Ġalso Ġinclude Ġthe Ġscores Ġfrom Ġthe Ġprevious Ġwork , Ġnotably Ġthe Ġstate - of - the - art Ġsystem ĠXL M ĠB IB REF 6 . ĠBefore Ġdiscussing Ġthe Ġresults , Ġwe Ġspell Ġout Ġthat Ġthe Ġfaire st Ġcomparison Ġin Ġthis Ġexperiment Ġis Ġthe Ġcomparison Ġbetween Ġm BER T Ġand ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġtrained Ġwith Ġmon oling ual Ġonly . </s> We Ġfirst Ġdiscuss Ġthe Ġtransfer Ġresults Ġfrom ĠB ERT . ĠIn itialized Ġfrom Ġfast Text Ġvectors , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġslightly Ġoutper forms Ġm BER T Ġby Ġ1 . 9 Ġpoints Ġon Ġaverage Ġand Ġwiden Ġthe Ġgap Ġof Ġ3 . 3 Ġpoints Ġon ĠArabic . ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġgains Ġextra Ġ0 . 8 Ġpoints Ġon Ġaverage Ġwhen Ġinitialized Ġfrom Ġparallel Ġdata . ĠWith Ġtriple Ġnumber Ġof Ġparameters , ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġoffers Ġan Ġadditional Ġboost Ġin Ġterm Ġof Ġaccuracy Ġand Ġinitialization Ġwith Ġparallel Ġdata Ġconsistently Ġimproves Ġthe Ġperformance . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġsignificantly Ġoutper forms ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġon Ġ11 ĠEnglish ĠN LP Ġtasks ĠB IB REF 0 , Ġthe Ġstrength Ġof ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġalso Ġshows Ġup Ġwhen Ġadapted Ġto Ġforeign Ġlanguages . </s> Transfer ring Ġfrom ĠRo BER Ta Ġleads Ġto Ġbetter Ġzero - shot Ġaccur acies . ĠWith Ġthe Ġsame Ġinitial izing Ġcondition , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + Ro BER Ta Ġoutper forms ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġon Ġaverage Ġby Ġ2 . 9 Ġand Ġ2 . 3 Ġpoints Ġwhen Ġinitial izing Ġfrom Ġmon oling ual Ġand Ġparallel Ġdata Ġrespectively . ĠThis Ġresult Ġshow Ġthat Ġwith Ġsimilar Ġnumber Ġof Ġparameters , Ġour Ġapproach Ġbenefits Ġfrom Ġa Ġbetter ĠEnglish Ġpre - trained Ġmodel . ĠWhen Ġtransferring Ġfrom ĠRo BER Ta $ _{ \ text sc Ġ{ large }} $, Ġwe Ġobtain Ġstate - of - the - art Ġresults Ġfor Ġfive Ġlanguages . </s> Currently , ĠRAM EN Ġonly Ġuses Ġparallel Ġdata Ġto Ġinitialize Ġforeign Ġembed d ings . ĠRAM EN Ġcan Ġalso Ġexploit Ġparallel Ġdata Ġthrough Ġtranslation Ġobjective Ġproposed Ġin ĠXL M . ĠWe Ġbelieve Ġthat Ġby Ġutilizing Ġparallel Ġdata Ġduring Ġthe Ġfine - tun ing Ġof ĠRAM EN Ġwould Ġbring Ġadditional Ġbenefits Ġfor Ġzero - shot Ġtasks . ĠWe Ġleave Ġthis Ġexploration Ġto Ġfuture Ġwork . ĠIn Ġsummary , Ġstarting Ġfrom ĠB ERT $ _{ \ text sc Ġ{ base }} $, Ġour Ġapproach Ġob tains Ġcompetitive Ġbilingual ĠL Ms Ġwith Ġm BER T Ġfor Ġzero - shot ĠX N LI . ĠOur Ġapproach Ġshows Ġthe Ġaccuracy Ġgains Ġwhen Ġadapting Ġfrom Ġa Ġbetter Ġpre - trained Ġmodel . </s> Results Ġ: :: ĠUniversal ĠDepend ency ĠPars ing </s> We Ġbuild Ġon Ġtop Ġof ĠRAM EN Ġa Ġgraph - based Ġdependency Ġparser ĠB IB REF 27 . ĠFor Ġthe Ġpurpose Ġof Ġevaluating Ġthe Ġcontextual Ġrepresentations Ġlearned Ġby Ġour Ġmodel , Ġwe Ġdo Ġnot Ġuse Ġpart - of - speech Ġtags . ĠContext ual ized Ġrepresentations Ġare Ġdirectly Ġfed Ġinto ĠDeep - B ia ff ine Ġlayers Ġto Ġpredict Ġarc Ġand Ġlabel Ġscores . ĠTable ĠT AB REF 34 Ġpresents Ġthe ĠLab eled ĠAtt achment ĠScores Ġ( L AS ) Ġfor Ġzero - shot Ġdependency Ġparsing . </s> We Ġfirst Ġlook Ġat Ġthe Ġfaire st Ġcomparison Ġbetween Ġm BER T Ġand Ġmon oling ually Ġinitialized ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T . ĠThe Ġlatter Ġoutper forms Ġthe Ġformer Ġon Ġfive Ġlanguages Ġexcept ĠArabic . ĠWe Ġobserve Ġthe Ġlargest Ġgain Ġof Ġ+ 5 . 2 ĠL AS Ġfor ĠFrench . ĠChinese Ġenjoys Ġ+ 3 . 1 ĠL AS Ġfrom Ġour Ġapproach . ĠWith Ġsimilar Ġarchitecture Ġ( 12 Ġor Ġ24 Ġlayers ) Ġand Ġinitialization Ġ( using Ġmon oling ual Ġor Ġparallel Ġdata ), ĠRAM EN + Ro BER Ta Ġperforms Ġbetter Ġthan ĠRAM EN + BER T Ġfor Ġmost Ġof Ġthe Ġlanguages . ĠArabic Ġand ĠHindi Ġbenefit Ġthe Ġmost Ġfrom Ġbigger Ġmodels . ĠFor Ġthe Ġother Ġfour Ġlanguages , ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġrenders Ġa Ġmodest Ġimprovement Ġover ĠRAM EN $ _{ \ text sc Ġ{ base }} $. </s> Analysis Ġ: :: ĠImpact Ġof Ġinitialization </s> Initial izing Ġforeign Ġembed d ings Ġis Ġthe Ġbackbone Ġof Ġour Ġapproach . ĠA Ġgood Ġinitialization Ġleads Ġto Ġbetter Ġzero - shot Ġtransfer Ġresults Ġand Ġenables Ġfast Ġadaptation . ĠTo Ġverify Ġthe Ġimportance Ġof Ġa Ġgood Ġinitialization , Ġwe Ġtrain Ġa ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + Ro BER Ta Ġwith Ġforeign Ġword - embed d ings Ġare Ġinitialized Ġrandomly Ġfrom Ġ$\ math cal Ġ{ N } ( 0 , Ġ{ 1 }{ d ^ 2 }) $. ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġsame Ġhyper - param eters Ġin ĠÂ§ SEC REF 27 . ĠTable ĠT AB REF 36 Ġshows Ġthe Ġresults Ġof ĠX N LI Ġand ĠUD Ġparsing Ġof Ġrandom Ġinitialization . ĠIn Ġcomparison Ġto Ġthe Ġinitialization Ġusing Ġaligned Ġfast Text Ġvectors , Ġrandom Ġinitialization Ġdecreases Ġthe Ġzero - shot Ġperformance Ġof ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġby Ġ15 . 9 % Ġfor ĠX N LI Ġand Ġ27 . 8 Ġpoints Ġfor ĠUD Ġparsing Ġon Ġaverage . ĠWe Ġalso Ġsee Ġthat Ġzero - shot Ġparsing Ġof ĠS OV Ġlanguages Ġ( Arab ic Ġand ĠHindi ) Ġsuffers Ġrandom Ġinitialization . </s> Analysis Ġ: :: ĠAre Ġcontextual Ġrepresentations Ġfrom ĠRAM EN Ġalso Ġgood Ġfor Ġsupervised Ġparsing ? </s> All Ġthe ĠRAM EN Ġmodels Ġare Ġbuilt Ġfrom ĠEnglish Ġand Ġtuned Ġon ĠEnglish Ġfor Ġzero - shot Ġcross - ling ual Ġtasks . ĠIt Ġis Ġreasonable Ġto Ġexpect ĠRAM EN s Ġdo Ġwell Ġin Ġthose Ġtasks Ġas Ġwe Ġhave Ġshown Ġin Ġour Ġexperiments . ĠBut Ġare Ġthey Ġalso Ġa Ġgood Ġfeature Ġextract or Ġfor Ġsupervised Ġtasks ? ĠWe Ġoffer Ġa Ġpartial Ġanswer Ġto Ġthis Ġquestion Ġby Ġevaluating Ġour Ġmodel Ġfor Ġsupervised Ġdependency Ġparsing Ġon ĠUD Ġdatasets . </s> We Ġused Ġtrain / dev / test Ġsplits Ġprovided Ġin ĠUD Ġto Ġtrain Ġand Ġevaluate Ġour ĠRAM EN - based Ġparser . ĠTable ĠT AB REF 38 Ġsummarizes Ġthe Ġresults Ġ( L AS ) Ġof Ġour Ġsupervised Ġparser . ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġchoose Ġm BER T Ġas Ġthe Ġbaseline Ġand Ġall Ġthe ĠRAM EN Ġmodels Ġare Ġinitialized Ġfrom Ġaligned Ġfast Text Ġvectors . ĠWith Ġthe Ġsame Ġarchitecture Ġof Ġ12 ĠTrans former Ġlayers , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġperforms Ġcompetitive Ġto Ġm BER T Ġand Ġout sh ines Ġm BER T Ġby Ġ+ 1 . 2 Ġpoints Ġfor ĠVietnamese . ĠThe Ġbest ĠL AS Ġresults Ġare Ġobtained Ġby ĠRAM EN $ _{ \ text sc Ġ{ large }} $ + Ro BER Ta Ġwith Ġ24 ĠTrans former Ġlayers . ĠOverall , Ġour Ġresults Ġindicate Ġthe Ġpotential Ġof Ġusing Ġcontextual Ġrepresentations Ġfrom ĠRAM EN Ġfor Ġsupervised Ġtasks . </s> Analysis Ġ: :: ĠHow Ġdoes Ġlinguistic Ġknowledge Ġtransfer Ġhappen Ġthrough Ġeach Ġtraining Ġstages ? </s> We Ġevaluate Ġthe Ġperformance Ġof ĠRAM EN + Ro BER Ta $ _{ \ text sc Ġ{ base }} $ Ġ( initialized Ġfrom Ġmon oling ual Ġdata ) Ġat Ġeach Ġtraining Ġsteps : Ġinitialization Ġof Ġword Ġembed d ings Ġ( 0 K Ġupdate ), Ġfine - tun ing Ġtarget Ġembed d ings Ġ( 25 K ), Ġand Ġfine - tun ing Ġthe Ġmodel Ġon Ġboth ĠEnglish Ġand Ġtarget Ġlanguage Ġ( at Ġeach Ġ25 K Ġupdates ). ĠThe Ġresults Ġare Ġpresented Ġin ĠFigure ĠFIG REF 40 . </s> Without Ġfine - tun ing , Ġthe Ġaverage Ġaccuracy Ġof ĠXL NI Ġis Ġ39 . 7 % Ġfor Ġa Ġthree - ways Ġclassification Ġtask , Ġand Ġthe Ġaverage ĠL AS Ġscore Ġis Ġ3 . 6 Ġfor Ġdependency Ġparsing . ĠWe Ġsee Ġthe Ġbiggest Ġleap Ġin Ġthe Ġperformance Ġafter Ġ50 K Ġupdates . ĠWhile Ġsemantic Ġsimilarity Ġtask Ġprofits Ġsignificantly Ġat Ġ25 K Ġupdates Ġof Ġthe Ġtarget Ġembed d ings , Ġsynt actic Ġtask Ġbenefits Ġwith Ġfurther Ġfine - tun ing Ġthe Ġenc oder . ĠThis Ġis Ġexpected Ġsince Ġthe Ġtarget Ġlanguages Ġmight Ġexhibit Ġdifferent Ġsynt actic Ġstructures Ġthan ĠEnglish Ġand Ġfine - tun ing Ġenc oder Ġhelps Ġto Ġcapture Ġlanguage Ġspecific Ġstructures . ĠWe Ġobserve Ġa Ġsubstantial Ġgain Ġof Ġ19 - 30 ĠL AS Ġfor Ġall Ġlanguages Ġexcept ĠFrench Ġafter Ġ50 K Ġupdates . </s> Language Ġsimilarities Ġhave Ġmore Ġimpact Ġon Ġtransferring Ġsyntax Ġthan Ġsemantics . ĠWithout Ġtuning Ġthe ĠEnglish Ġenc oder , ĠFrench Ġenjoys Ġ50 . 3 ĠL AS Ġfor Ġbeing Ġclosely Ġrelated Ġto ĠEnglish , Ġwhereas ĠArabic Ġand ĠHindi , ĠS OV Ġlanguages , Ġmodest ly Ġreach Ġ4 . 2 Ġand Ġ6 . 4 Ġpoints Ġusing Ġthe ĠSV O Ġenc oder . ĠAlthough ĠChinese Ġhas ĠSV O Ġorder , Ġit Ġis Ġoften Ġseen Ġas Ġhead - final Ġwhile ĠEnglish Ġis Ġstrong Ġhead - initial . ĠPerhaps , Ġthis Ġexplains Ġthe Ġpoor Ġperformance Ġfor ĠChinese . </s> Lim itations </s> While Ġwe Ġhave Ġsuccessfully Ġadapted Ġaut oen c oding Ġpre - trained ĠL Ms Ġfrom ĠEnglish Ġto Ġother Ġlanguages , Ġthe Ġquestion Ġwhether Ġour Ġapproach Ġcan Ġalso Ġbe Ġapplied Ġfor Ġaut ore gressive ĠLM Ġsuch Ġas ĠXL Net Ġstill Ġremains . ĠWe Ġleave Ġthe Ġinvestigation Ġto Ġfuture Ġwork . </s> Con clusions </s> In Ġthis Ġwork , Ġwe Ġhave Ġpresented Ġa Ġsimple Ġand Ġeffective Ġapproach Ġfor Ġrapidly Ġbuilding Ġa Ġbilingual ĠLM Ġunder Ġa Ġlimited Ġcomputational Ġbudget . ĠUsing ĠB ERT Ġas Ġthe Ġstarting Ġpoint , Ġwe Ġdemonstrate Ġthat Ġour Ġapproach Ġproduces Ġbetter Ġthan Ġm BER T Ġon Ġtwo Ġcross - ling ual Ġzero - shot Ġsentence Ġclassification Ġand Ġdependency Ġparsing . ĠWe Ġfind Ġthat Ġthe Ġperformance Ġof Ġour Ġbilingual ĠLM , ĠRAM EN , Ġcorrelates Ġwith Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġpre - trained ĠEnglish Ġmodels . ĠWe Ġalso Ġfind Ġthat ĠRAM EN Ġis Ġalso Ġa Ġpowerful Ġfeature Ġextract or Ġin Ġsupervised Ġdependency Ġparsing . ĠFinally , Ġwe Ġhope Ġthat Ġour Ġwork Ġsparks Ġof Ġinterest Ġin Ġdeveloping Ġfast Ġand Ġeffective Ġmethods Ġfor Ġtransferring Ġpre - trained ĠEnglish Ġmodels Ġto Ġother Ġlanguages .
Introduction Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks. Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer. Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages. BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing. Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5. The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages. To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages. Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train. Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM. We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer. Our main contributions in this work are: We propose a fast adaptation method for obtaining a bilingual BERT$_{\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU. We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing. We show that our models offer competitive performance or even better that mBERT. We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task. Bilingual Pre-trained LMs We first provide some background of pre-trained language models. Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$. Let $_{w_i}$ denote the embedding of word $w_i$ (i.e., $_{w_i} = _e[w_1]$). We omit positional embeddings and bias for clarity. A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$. Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]. It then predicts the original tokens $y_i = w_i$ from the corrupted tokens. Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens. The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context. Here we assume that the encoder performs necessary masking with respect to each training objective. Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget. To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model. Our approach consists of three steps. First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8). Next, we create a target LM from the target embeddings and the English encoder $\Psi ()$. We then fine-tune target embeddings while keeping $\Psi ()$ fixed (§SECREF14). Finally, we construct a bilingual LM of $_e$, $_f$, and $\Psi ()$ and fine-tune all the parameters (§SECREF15). Figure FIGREF7 illustrates the last two steps in our approach. Bilingual Pre-trained LMs ::: Initializing Target Embeddings Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar. Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$ where $_i\in ^{|V_e|}$ is a sparse vector and $\sum _j^{|V_e|} \alpha _{ij} = 1$. In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5). In the following, we discuss how to estimate $_i\;\forall i\in \lbrace 1,2, \dots , |V_f|\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data. Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13. We then assign: Since $_i$ is estimated from word alignment, it is a sparse vector. Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus For low resource languages, parallel data may not be available. In this case, we rely only on monolingual data (e.g., Wikipedias). We estimate word translation probabilities from word embeddings of the two languages. Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16. Unlike learning contextualized representations, learning word vectors is fast and computationally cheap. Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as Here, we use $\mathrm {sparsemax}$ BIBREF17 instead of softmax. Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word. This property is desirable in our approach since it leads to a better initialization of the foreign embeddings. Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM. We then fine-tune only foreign word-embeddings on monolingual data. The training objective is the same as the training objective of the English pre-trained LM (i.e., masked LM for BERT). Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task. For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned. Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7). The new model has two separate embedding layers and output layers, one for English and one for foreign language. The encoder layer in between is shared. We then fine-tune this model using English and foreign monolingual data. Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3). In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i.e., word order, morphological agreement) of the target languages. Zero-shot Experiments We build our bilingual LMs, named RAMEN, starting from BERT$_{\textsc {base}}$, BERT$_{\textsc {large}}$, RoBERTa$_{\textsc {base}}$, and RoBERTa$_{\textsc {large}}$ pre-trained models. Using BERT$_{\textsc {base}}$ allows us to compare the results with mBERT model. Using BERT$_{\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM. We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing. Zero-shot Experiments ::: Data We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource. For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6. Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr. We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018. For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq. DISPLAY_FORM13). We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16. We only use identical words between the two languages as the supervised signal. We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15). We do not lowercase or remove accents in our data preprocessing pipeline. We tokenize English using the provided tokenizer from pre-trained models. For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively. We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models. Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa. We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2.4 BIBREF21 for parsing task. Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese). Zero-shot Experiments ::: Data ::: Remark on BPE BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces. BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer. However, in our current approach, subwords between source and target are not shared. A subword that is in both English and foreign vocabulary has two different embeddings. Zero-shot Experiments ::: Estimating translation probabilities Since pre-trained models operate on subword level, we need to estimate subword translation probabilities. Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13. Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level. We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$. We take the top 50,000 words in each aligned word-vectors to compute subword vectors. In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings. Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$. Zero-shot Experiments ::: Hyper-parameters In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters. The sequence length is set to 256. The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively. For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences. This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24. We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0.5$. We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam). For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay. All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit. When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$. The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively. All experiments are carried out on a single Tesla V100 16GB GPU. Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days. Results In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing. Results ::: Cross-lingual Natural Language Inference Table TABREF32 shows the XNLI test accuracy. For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6. Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only. We first discuss the transfer results from BERT. Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1.9 points on average and widen the gap of 3.3 points on Arabic. RAMEN$_{\textsc {base}}$ gains extra 0.8 points on average when initialized from parallel data. With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance. It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages. Transferring from RoBERTa leads to better zero-shot accuracies. With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2.9 and 2.3 points when initializing from monolingual and parallel data respectively. This result show that with similar number of parameters, our approach benefits from a better English pre-trained model. When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages. Currently, RAMEN only uses parallel data to initialize foreign embeddings. RAMEN can also exploit parallel data through translation objective proposed in XLM. We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks. We leave this exploration to future work. In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI. Our approach shows the accuracy gains when adapting from a better pre-trained model. Results ::: Universal Dependency Parsing We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing. We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT. The latter outperforms the former on five languages except Arabic. We observe the largest gain of +5.2 LAS for French. Chinese enjoys +3.1 LAS from our approach. With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages. Arabic and Hindi benefit the most from bigger models. For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$. Analysis ::: Impact of initialization Initializing foreign embeddings is the backbone of our approach. A good initialization leads to better zero-shot transfer results and enables fast adaptation. To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$. For a fair comparison, we use the same hyper-parameters in §SECREF27. Table TABREF36 shows the results of XNLI and UD parsing of random initialization. In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15.9% for XNLI and 27.8 points for UD parsing on average. We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization. Analysis ::: Are contextual representations from RAMEN also good for supervised parsing? All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks. It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments. But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets. We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser. Table TABREF38 summarizes the results (LAS) of our supervised parser. For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors. With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1.2 points for Vietnamese. The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers. Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks. Analysis ::: How does linguistic knowledge transfer happen through each training stages? We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates). The results are presented in Figure FIGREF40. Without fine-tuning, the average accuracy of XLNI is 39.7% for a three-ways classification task, and the average LAS score is 3.6 for dependency parsing. We see the biggest leap in the performance after 50K updates. While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder. This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures. We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates. Language similarities have more impact on transferring syntax than semantics. Without tuning the English encoder, French enjoys 50.3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4.2 and 6.4 points using the SVO encoder. Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial. Perhaps, this explains the poor performance for Chinese. Limitations While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains. We leave the investigation to future work. Conclusions In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget. Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing. We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models. We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing. Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages.
What metrics are used for evaluation?
37f8c034a14c7b4d0ab2e0ed1b827cc0eaa71ac6
What Ġmetrics Ġare Ġused Ġfor Ġevaluation ?
translation Ġprobabilities , ĠLab eled ĠAtt achment ĠScores Ġ( L AS )
<s> What Ġmetrics Ġare Ġused Ġfor Ġevaluation ? </s> Introduction </s> Pre - trained Ġmodels ĠB IB REF 0 , ĠB IB REF 1 Ġhave Ġreceived Ġmuch Ġof Ġattention Ġrecently Ġthanks Ġto Ġtheir Ġimpressive Ġresults Ġin Ġmany Ġdown Ġstream ĠN LP Ġtasks . ĠAdditionally , Ġmult ilingual Ġpre - trained Ġmodels Ġenable Ġmany ĠN LP Ġapplications Ġfor Ġother Ġlanguages Ġvia Ġzero - short Ġcross - ling ual Ġtransfer . ĠZero - shot Ġcross - ling ual Ġtransfer Ġhas Ġshown Ġpromising Ġresults Ġfor Ġrapidly Ġbuilding Ġapplications Ġfor Ġlow Ġresource Ġlanguages . ĠB IB REF 2 Ġshow Ġthe Ġpotential Ġof Ġmult ilingual - BER T ĠB IB REF 0 Ġin Ġzero - shot Ġtransfer Ġfor Ġa Ġlarge Ġnumber Ġof Ġlanguages Ġfrom Ġdifferent Ġlanguage Ġfamilies Ġon Ġfive ĠN LP Ġtasks , Ġnamely , Ġnatural Ġlanguage Ġinference , Ġdocument Ġclassification , Ġnamed Ġentity Ġrecognition , Ġpart - of - speech Ġtagging , Ġand Ġdependency Ġparsing . </s> Although Ġmult ilingual Ġmodels Ġare Ġan Ġimportant Ġingredient Ġfor Ġenhancing Ġlanguage Ġtechnology Ġin Ġmany Ġlanguages , Ġrecent Ġresearch Ġon Ġimproving Ġpre - trained Ġmodels Ġputs Ġmuch Ġemphasis Ġon ĠEnglish ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 . ĠThe Ġcurrent Ġstate Ġof Ġaffairs Ġmakes Ġit Ġdifficult Ġto Ġtranslate Ġadvancements Ġin Ġpre - training Ġfrom ĠEnglish Ġto Ġnon - English Ġlanguages . ĠTo Ġour Ġbest Ġknowledge , Ġthere Ġare Ġonly Ġthree Ġavailable Ġmult ilingual Ġpre - trained Ġmodels Ġto Ġdate : Ġ( 1 ) Ġthe Ġmult ilingual - BER T Ġ( m BER T ) Ġthat Ġsupports Ġ104 Ġlanguages , Ġ( 2 ) Ġcross - ling ual Ġlanguage Ġmodel ĠB IB REF 6 Ġthat Ġsupports Ġ100 Ġlanguages , Ġand Ġ( 3 ) ĠLanguage ĠAg nostic ĠSE nt ence ĠRepresent ations ĠB IB REF 7 Ġthat Ġsupports Ġ93 Ġlanguages . ĠAmong Ġthe Ġthree Ġmodels , ĠL AS ER Ġis Ġbased Ġon Ġneural Ġmachine Ġtranslation Ġapproach Ġand Ġstrictly Ġrequires Ġparallel Ġdata Ġto Ġtrain . </s> Do Ġmult ilingual Ġmodels Ġalways Ġneed Ġto Ġbe Ġtrained Ġfrom Ġscratch ? ĠCan Ġwe Ġtransfer Ġlinguistic Ġknowledge Ġlearned Ġby ĠEnglish Ġpre - trained Ġmodels Ġto Ġother Ġlanguages ? ĠIn Ġthis Ġwork , Ġwe Ġdevelop Ġa Ġtechnique Ġto Ġrapidly Ġtransfer Ġan Ġexisting Ġpre - trained Ġmodel Ġfrom ĠEnglish Ġto Ġother Ġlanguages Ġin Ġan Ġenergy Ġefficient Ġway ĠB IB REF 8 . ĠAs Ġthe Ġfirst Ġstep , Ġwe Ġfocus Ġon Ġbuilding Ġa Ġbilingual Ġlanguage Ġmodel Ġ( LM ) Ġof ĠEnglish Ġand Ġa Ġtarget Ġlanguage . ĠStarting Ġfrom Ġa Ġpre - trained ĠEnglish ĠLM , Ġwe Ġlearn Ġthe Ġtarget Ġlanguage Ġspecific Ġparameters Ġ( i . e ., Ġword Ġembed d ings ), Ġwhile Ġkeeping Ġthe Ġenc oder Ġlayers Ġof Ġthe Ġpre - trained ĠEnglish ĠLM Ġfixed . ĠWe Ġthen Ġfine - t une Ġboth ĠEnglish Ġand Ġtarget Ġmodel Ġto Ġobtain Ġthe Ġbilingual ĠLM . ĠWe Ġapply Ġour Ġapproach Ġto Ġaut oen c oding Ġlanguage Ġmodels Ġwith Ġmasked Ġlanguage Ġmodel Ġobjective Ġand Ġshow Ġthe Ġadvantage Ġof Ġthe Ġproposed Ġapproach Ġin Ġzero - shot Ġtransfer . ĠOur Ġmain Ġcontributions Ġin Ġthis Ġwork Ġare : </s> We Ġpropose Ġa Ġfast Ġadaptation Ġmethod Ġfor Ġobtaining Ġa Ġbilingual ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġof ĠEnglish Ġand Ġa Ġtarget Ġlanguage Ġwithin Ġa Ġday Ġusing Ġone ĠTesla ĠV 100 Ġ16 GB ĠGPU . </s> We Ġevaluate Ġour Ġbilingual ĠL Ms Ġfor Ġsix Ġlanguages Ġon Ġtwo Ġzero - shot Ġcross - ling ual Ġtransfer Ġtasks , Ġnamely Ġnatural Ġlanguage Ġinference ĠB IB REF 9 Ġand Ġuniversal Ġdependency Ġparsing . ĠWe Ġshow Ġthat Ġour Ġmodels Ġoffer Ġcompetitive Ġperformance Ġor Ġeven Ġbetter Ġthat Ġm BER T . </s> We Ġillustrate Ġthat Ġour Ġbilingual ĠL Ms Ġcan Ġserve Ġas Ġan Ġexcellent Ġfeature Ġextract or Ġin Ġsupervised Ġdependency Ġparsing Ġtask . </s> B ilingual ĠPre - trained ĠL Ms </s> We Ġfirst Ġprovide Ġsome Ġbackground Ġof Ġpre - trained Ġlanguage Ġmodels . ĠLet Ġ$_ e $ Ġbe ĠEnglish Ġword - embed d ings Ġand Ġ$\ Ps i Ġ() $ Ġbe Ġthe ĠTrans former ĠB IB REF 10 Ġenc oder Ġwith Ġparameters Ġ$$ . ĠLet Ġ$ _{ w _ i } $ Ġdenote Ġthe Ġembed ding Ġof Ġword Ġ$ w _ i $ Ġ( i . e ., Ġ$ _{ w _ i } Ġ= Ġ_ e [ w _ 1 ] $ ). ĠWe Ġomit Ġpositional Ġembed d ings Ġand Ġbias Ġfor Ġclarity . ĠA Ġpre - trained ĠLM Ġtypically Ġperforms Ġthe Ġfollowing Ġcomput ations : Ġ( i ) Ġtransform Ġa Ġsequence Ġof Ġinput Ġtokens Ġto Ġcontextual ized Ġrepresentations Ġ$ [ _{ w _ 1 }, \ d ots Ġ, _{ w _ n } ] Ġ= Ġ\ Ps i Ġ( _{ w _ 1 }, Ġ\ d ots Ġ, Ġ_ { w _ n }; Ġ) $, Ġand Ġ( ii ) Ġpredict Ġan Ġoutput Ġword Ġ$ y _ i $ Ġat Ġ$ i ^ {\ text { th }} $ Ġposition Ġ$ p ( y _ i Ġ| Ġ_ { w _ i }) Ġ\ pro pt o Ġ\ exp Ġ( _{ w _ i } ^ \ top Ġ_ { y _ i }) $. </s> Aut oen c oding ĠLM ĠB IB REF 0 Ġcorrupt s Ġsome Ġinput Ġtokens Ġ$ w _ i $ Ġby Ġreplacing Ġthem Ġwith Ġa Ġspecial Ġtoken Ġ[ MAS K ]. ĠIt Ġthen Ġpredicts Ġthe Ġoriginal Ġtokens Ġ$ y _ i Ġ= Ġw _ i $ Ġfrom Ġthe Ġcorrupted Ġtokens . ĠAut ore gressive ĠLM ĠB IB REF 3 Ġpredicts Ġthe Ġnext Ġtoken Ġ($ y _ i Ġ= Ġw _{ i + 1 } $ ) Ġgiven Ġall Ġthe Ġprevious Ġtokens . ĠThe Ġrecently Ġproposed ĠXL Net Ġmodel ĠB IB REF 5 Ġis Ġan Ġaut ore gressive ĠLM Ġthat Ġfactor izes Ġoutput Ġwith Ġall Ġpossible Ġperm utations , Ġwhich Ġshows Ġempirical Ġperformance Ġimprovement Ġover ĠG PT - 2 Ġdue Ġto Ġthe Ġability Ġto Ġcapture Ġbid irection al Ġcontext . ĠHere Ġwe Ġassume Ġthat Ġthe Ġenc oder Ġperforms Ġnecessary Ġmask ing Ġwith Ġrespect Ġto Ġeach Ġtraining Ġobjective . </s> Given Ġan ĠEnglish Ġpre - trained ĠLM , Ġwe Ġwish Ġto Ġlearn Ġa Ġbilingual ĠLM Ġfor ĠEnglish Ġand Ġa Ġgiven Ġtarget Ġlanguage Ġ$ f $ Ġunder Ġa Ġlimited Ġcomputational Ġresource Ġbudget . ĠTo Ġquickly Ġbuild Ġa Ġbilingual ĠLM , Ġwe Ġdirectly Ġadapt Ġthe ĠEnglish Ġpre - train d Ġmodel Ġto Ġthe Ġtarget Ġmodel . ĠOur Ġapproach Ġconsists Ġof Ġthree Ġsteps . ĠFirst , Ġwe Ġinitialize Ġtarget Ġlanguage Ġword - embed d ings Ġ$_ f $ Ġin Ġthe ĠEnglish Ġembed ding Ġspace Ġsuch Ġthat Ġembed d ings Ġof Ġa Ġtarget Ġword Ġand Ġits ĠEnglish Ġequivalents Ġare Ġclose Ġtogether Ġ( Â§ SEC REF 8 ). ĠNext , Ġwe Ġcreate Ġa Ġtarget ĠLM Ġfrom Ġthe Ġtarget Ġembed d ings Ġand Ġthe ĠEnglish Ġenc oder Ġ$\ Ps i Ġ() $. ĠWe Ġthen Ġfine - t une Ġtarget Ġembed d ings Ġwhile Ġkeeping Ġ$\ Ps i Ġ() $ Ġfixed Ġ( Â§ SEC REF 14 ). ĠFinally , Ġwe Ġconstruct Ġa Ġbilingual ĠLM Ġof Ġ$_ e $, Ġ$_ f $, Ġand Ġ$\ Ps i Ġ() $ Ġand Ġfine - t une Ġall Ġthe Ġparameters Ġ( Â§ SEC REF 15 ). ĠFigure ĠFIG REF 7 Ġillustrates Ġthe Ġlast Ġtwo Ġsteps Ġin Ġour Ġapproach . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings </s> Our Ġapproach Ġto Ġlearn Ġthe Ġinitial Ġforeign Ġword Ġembed d ings Ġ$_ f Ġ\ in Ġ^ { | V _ f | Ġ\ times Ġd } $ Ġis Ġbased Ġon Ġthe Ġidea Ġof Ġmapping Ġthe Ġtrained ĠEnglish Ġword Ġembed d ings Ġ$_ e Ġ\ in Ġ^ { | V _ e | Ġ\ times Ġd } $ Ġto Ġ$_ f $ Ġsuch Ġthat Ġif Ġa Ġforeign Ġword Ġand Ġan ĠEnglish Ġword Ġare Ġsimilar Ġin Ġmeaning Ġthen Ġtheir Ġembed d ings Ġare Ġsimilar . ĠB orrow ing Ġthe Ġidea Ġof Ġuniversal Ġlex ical Ġsharing Ġfrom ĠB IB REF 11 , Ġwe Ġrepresent Ġeach Ġforeign Ġword Ġembed ding Ġ$_ f [ i ] Ġ\ in Ġ^ d $ Ġas Ġa Ġlinear Ġcombination Ġof ĠEnglish Ġword Ġembed d ings Ġ$_ e [ j ] Ġ\ in Ġ^ d $ </s> where Ġ$_ i \ in Ġ^ { | V _ e | } $ Ġis Ġa Ġsparse Ġvector Ġand Ġ$\ sum Ġ_ j ^{ | V _ e | } Ġ\ alpha Ġ_ { ij } Ġ= Ġ1 $. </s> In Ġthis Ġstep Ġof Ġinitial izing Ġforeign Ġembed d ings , Ġhaving Ġa Ġgood Ġestimation Ġof Ġ$$ Ġcould Ġspeed Ġof Ġthe Ġconvergence Ġwhen Ġtuning Ġthe Ġforeign Ġmodel Ġand Ġenable Ġzero - shot Ġtransfer Ġ( Â§ SEC REF 5 ). ĠIn Ġthe Ġfollowing , Ġwe Ġdiscuss Ġhow Ġto Ġestimate Ġ$_ i \ ; \ for all Ġi \ in Ġ\ l brace Ġ1 , 2 , Ġ\ d ots Ġ, Ġ| V _ f | \ r brace Ġ$ Ġunder Ġtwo Ġscenarios : Ġ( i ) Ġwe Ġhave Ġparallel Ġdata Ġof ĠEnglish - foreign , Ġand Ġ( ii ) Ġwe Ġonly Ġrely Ġon ĠEnglish Ġand Ġforeign Ġmon oling ual Ġdata . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings Ġ: :: ĠLearning Ġfrom ĠParallel ĠCorpus </s> Given Ġan ĠEnglish - foreign Ġparallel Ġcorpus , Ġwe Ġcan Ġestimate Ġword Ġtranslation Ġprobability Ġ$ p ( e \ , | \ , f ) $ Ġfor Ġany Ġ( English - foreign ) Ġpair Ġ$( e , Ġf ) $ Ġusing Ġpopular Ġword - al ignment ĠB IB REF 12 Ġtool k its Ġsuch Ġas Ġfast - align ĠB IB REF 13 . ĠWe Ġthen Ġassign : </s> Since Ġ$_ i $ Ġis Ġestimated Ġfrom Ġword Ġalignment , Ġit Ġis Ġa Ġsparse Ġvector . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠInitial izing ĠTarget ĠEmb edd ings Ġ: :: ĠLearning Ġfrom ĠMon oling ual ĠCorpus </s> For Ġlow Ġresource Ġlanguages , Ġparallel Ġdata Ġmay Ġnot Ġbe Ġavailable . ĠIn Ġthis Ġcase , Ġwe Ġrely Ġonly Ġon Ġmon oling ual Ġdata Ġ( e . g ., ĠWik iped ias ). ĠWe Ġestimate Ġword Ġtranslation Ġprobabilities Ġfrom Ġword Ġembed d ings Ġof Ġthe Ġtwo Ġlanguages . ĠWord Ġvectors Ġof Ġthese Ġlanguages Ġcan Ġbe Ġlearned Ġusing Ġfast Text ĠB IB REF 14 Ġand Ġthen Ġare Ġaligned Ġinto Ġa Ġshared Ġspace Ġwith ĠEnglish ĠB IB REF 15 , ĠB IB REF 16 . ĠUnlike Ġlearning Ġcontextual ized Ġrepresentations , Ġlearning Ġword Ġvectors Ġis Ġfast Ġand Ġcomput ationally Ġcheap . ĠGiven Ġthe Ġaligned Ġvectors Ġ$\ bar { } _ f $ Ġof Ġforeign Ġand Ġ$\ bar { } _ e $ Ġof ĠEnglish , Ġwe Ġcalculate Ġthe Ġword Ġtranslation Ġmatrix Ġ$\ in Ġ^ { | V _ f | \ times Ġ| V _ e | } $ Ġas </s> Here , Ġwe Ġuse Ġ$\ math rm Ġ{ s parse max } $ ĠB IB REF 17 Ġinstead Ġof Ġsoft max . ĠSp arse max Ġis Ġa Ġsparse Ġversion Ġof Ġsoft max Ġand Ġit Ġputs Ġzero Ġprobabilities Ġon Ġmost Ġof Ġthe Ġword Ġin Ġthe ĠEnglish Ġvocabulary Ġexcept Ġfew ĠEnglish Ġwords Ġthat Ġare Ġsimilar Ġto Ġa Ġgiven Ġforeign Ġword . ĠThis Ġproperty Ġis Ġdesirable Ġin Ġour Ġapproach Ġsince Ġit Ġleads Ġto Ġa Ġbetter Ġinitialization Ġof Ġthe Ġforeign Ġembed d ings . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠFine - tun ing ĠTarget ĠEmb edd ings </s> After Ġinitial izing Ġforeign Ġword - embed d ings , Ġwe Ġreplace ĠEnglish Ġword - embed d ings Ġin Ġthe ĠEnglish Ġpre - trained ĠLM Ġwith Ġforeign Ġword - embed d ings Ġto Ġobtain Ġthe Ġforeign ĠLM . ĠWe Ġthen Ġfine - t une Ġonly Ġforeign Ġword - embed d ings Ġon Ġmon oling ual Ġdata . ĠThe Ġtraining Ġobjective Ġis Ġthe Ġsame Ġas Ġthe Ġtraining Ġobjective Ġof Ġthe ĠEnglish Ġpre - trained ĠLM Ġ( i . e ., Ġmasked ĠLM Ġfor ĠB ERT ). ĠSince Ġthe Ġtrained Ġenc oder Ġ$\ Ps i Ġ() $ Ġis Ġgood Ġat Ġcapturing Ġassociation , Ġthe Ġpurpose Ġof Ġthis Ġstep Ġis Ġto Ġfurther Ġoptimize Ġtarget Ġembed d ings Ġsuch Ġthat Ġthe Ġtarget ĠLM Ġcan Ġutilized Ġthe Ġtrained Ġenc oder Ġfor Ġassociation Ġtask . ĠFor Ġexample , Ġif Ġthe Ġwords ĠAlbert ĠCam us Ġpresented Ġin Ġa ĠFrench Ġinput Ġsequence , Ġthe Ġself - att ention Ġin Ġthe Ġenc oder Ġmore Ġlikely Ġattends Ġto Ġwords Ġabsurd e Ġand Ġexistential ism e Ġonce Ġtheir Ġembed d ings Ġare Ġtuned . </s> B ilingual ĠPre - trained ĠL Ms Ġ: :: ĠFine - tun ing ĠB ilingual ĠLM </s> We Ġcreate Ġa Ġbilingual ĠLM Ġby Ġplug ging Ġforeign Ġlanguage Ġspecific Ġparameters Ġto Ġthe Ġpre - trained ĠEnglish ĠLM Ġ( Figure ĠFIG REF 7 ). ĠThe Ġnew Ġmodel Ġhas Ġtwo Ġseparate Ġembed ding Ġlayers Ġand Ġoutput Ġlayers , Ġone Ġfor ĠEnglish Ġand Ġone Ġfor Ġforeign Ġlanguage . ĠThe Ġenc oder Ġlayer Ġin Ġbetween Ġis Ġshared . ĠWe Ġthen Ġfine - t une Ġthis Ġmodel Ġusing ĠEnglish Ġand Ġforeign Ġmon oling ual Ġdata . ĠHere , Ġwe Ġkeep Ġtuning Ġthe Ġmodel Ġon ĠEnglish Ġto Ġensure Ġthat Ġit Ġdoes Ġnot Ġforget Ġwhat Ġit Ġhas Ġlearned Ġin ĠEnglish Ġand Ġthat Ġwe Ġcan Ġuse Ġthe Ġresulting Ġmodel Ġfor Ġzero - shot Ġtransfer Ġ( Â§ SEC REF 3 ). ĠIn Ġthis Ġstep , Ġthe Ġenc oder Ġparameters Ġare Ġalso Ġupdated Ġso Ġthat Ġin Ġcan Ġlearn Ġsynt actic Ġaspects Ġ( i . e ., Ġword Ġorder , Ġmorph ological Ġagreement ) Ġof Ġthe Ġtarget Ġlanguages . </s> Zero - shot ĠExper iments </s> We Ġbuild Ġour Ġbilingual ĠL Ms , Ġnamed ĠRAM EN , Ġstarting Ġfrom ĠB ERT $ _{ \ text sc Ġ{ base }} $, ĠB ERT $ _{ \ text sc Ġ{ large }} $, ĠRo BER Ta $ _{ \ text sc Ġ{ base }} $, Ġand ĠRo BER Ta $ _{ \ text sc Ġ{ large }} $ Ġpre - trained Ġmodels . ĠUsing ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġallows Ġus Ġto Ġcompare Ġthe Ġresults Ġwith Ġm BER T Ġmodel . ĠUsing ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġand ĠRo BER Ta Ġallows Ġus Ġto Ġinvestigate Ġwhether Ġthe Ġperformance Ġof Ġthe Ġtarget ĠLM Ġcorrelates Ġwith Ġthe Ġperformance Ġof Ġthe Ġsource ĠLM . ĠWe Ġevaluate Ġour Ġmodels Ġon Ġtwo Ġcross - ling ual Ġzero - shot Ġtasks : Ġ( 1 ) ĠCross - ling ual ĠNatural ĠLanguage ĠIn ference Ġ( X N LI ) Ġand Ġ( 2 ) Ġdependency Ġparsing . </s> Zero - shot ĠExper iments Ġ: :: ĠData </s> We Ġevaluate Ġour Ġapproach Ġfor Ġsix Ġtarget Ġlanguages : ĠFrench Ġ( fr ), ĠRussian Ġ( ru ), ĠArabic Ġ( ar ), ĠChinese Ġ( zh ), ĠHindi Ġ( hi ), Ġand ĠVietnamese Ġ( vi ). ĠThese Ġlanguages Ġbelong Ġto Ġfour Ġdifferent Ġlanguage Ġfamilies . ĠFrench , ĠRussian , Ġand ĠHindi Ġare ĠIndo - European Ġlanguages , Ġsimilar Ġto ĠEnglish . ĠArabic , ĠChinese , Ġand ĠVietnamese Ġbelong Ġto ĠAf ro - As i atic , ĠS ino - T ib et an , Ġand ĠAust ro - As i atic Ġfamily Ġrespectively . ĠThe Ġchoice Ġof Ġthe Ġsix Ġlanguages Ġalso Ġreflects Ġdifferent Ġtraining Ġconditions Ġdepending Ġon Ġthe Ġamount Ġof Ġmon oling ual Ġdata . ĠFrench Ġand ĠRussian , Ġand ĠArabic Ġcan Ġbe Ġregarded Ġas Ġhigh Ġresource Ġlanguages Ġwhereas ĠHindi Ġhas Ġfar Ġless Ġdata Ġand Ġcan Ġbe Ġconsidered Ġas Ġlow Ġresource . </s> For Ġexperiments Ġthat Ġuse Ġparallel Ġdata Ġto Ġinitialize Ġforeign Ġspecific Ġparameters , Ġwe Ġuse Ġthe Ġsame Ġdatasets Ġin Ġthe Ġwork Ġof ĠB IB REF 6 . ĠSpecifically , Ġwe Ġuse ĠUnited ĠNations ĠParallel ĠCorpus ĠB IB REF 18 Ġfor Ġen - ru , Ġen - ar , Ġen - zh , Ġand Ġen - fr . ĠWe Ġcollect Ġen - hi Ġparallel Ġdata Ġfrom ĠI IT ĠBombay Ġcorpus ĠB IB REF 19 Ġand Ġen - vi Ġdata Ġfrom ĠOpen Sub t itles Ġ2018 . ĠFor Ġexperiments Ġthat Ġuse Ġonly Ġmon oling ual Ġdata Ġto Ġinitialize Ġforeign Ġparameters , Ġinstead Ġof Ġtraining Ġword - ve ctors Ġfrom Ġthe Ġscratch , Ġwe Ġuse Ġthe Ġpre - trained Ġword Ġvectors Ġfrom Ġfast Text ĠB IB REF 14 Ġto Ġestimate Ġword Ġtranslation Ġprobabilities Ġ( E q . ĠDIS PLAY _ FORM 13 ). ĠWe Ġalign Ġthese Ġvectors Ġinto Ġa Ġcommon Ġspace Ġusing Ġorth og onal ĠPro cr ust es ĠB IB REF 20 , ĠB IB REF 15 , ĠB IB REF 16 . ĠWe Ġonly Ġuse Ġidentical Ġwords Ġbetween Ġthe Ġtwo Ġlanguages Ġas Ġthe Ġsupervised Ġsignal . ĠWe Ġuse ĠWiki Ext ractor Ġto Ġextract Ġextract Ġraw Ġsentences Ġfrom ĠWik iped ias Ġas Ġmon oling ual Ġdata Ġfor Ġfine - tun ing Ġtarget Ġembed d ings Ġand Ġbilingual ĠL Ms Ġ( Â§ SEC REF 15 ). ĠWe Ġdo Ġnot Ġlower case Ġor Ġremove Ġaccents Ġin Ġour Ġdata Ġpre processing Ġpipeline . </s> We Ġtoken ize ĠEnglish Ġusing Ġthe Ġprovided Ġtoken izer Ġfrom Ġpre - trained Ġmodels . ĠFor Ġtarget Ġlanguages , Ġwe Ġuse Ġfast B PE Ġto Ġlearn Ġ30 , 000 ĠB PE Ġcodes Ġand Ġ50 , 000 Ġcodes Ġwhen Ġtransferring Ġfrom ĠB ERT Ġand ĠRo BER Ta Ġrespectively . ĠWe Ġtrunc ate Ġthe ĠB PE Ġvocabulary Ġof Ġforeign Ġlanguages Ġto Ġmatch Ġthe Ġsize Ġof Ġthe ĠEnglish Ġvocabulary Ġin Ġthe Ġsource Ġmodels . ĠPrec ise ly , Ġthe Ġsize Ġof Ġforeign Ġvocabulary Ġis Ġset Ġto Ġ32 , 000 Ġwhen Ġtransferring Ġfrom ĠB ERT Ġand Ġ50 , 000 Ġwhen Ġtransferring Ġfrom ĠRo BER Ta . </s> We Ġuse ĠX N LI Ġdataset ĠB IB REF 9 Ġfor Ġclassification Ġtask Ġand ĠUniversal ĠDepend encies Ġv 2 . 4 ĠB IB REF 21 Ġfor Ġparsing Ġtask . ĠSince Ġa Ġlanguage Ġmight Ġhave Ġmore Ġthan Ġone Ġtree bank Ġin ĠUniversal ĠDepend encies , Ġwe Ġuse Ġthe Ġfollowing Ġtree banks : Ġen _ ew t Ġ( English ), Ġfr _ gs d Ġ( French ), Ġru _ sy nt ag rus Ġ( Russian ) Ġar _ pad t Ġ( Arab ic ), Ġvi _ vt b Ġ( V iet namese ), Ġhi _ h dt b Ġ( H ind i ), Ġand Ġz h _ gs d Ġ( Chinese ). </s> Zero - shot ĠExper iments Ġ: :: ĠData Ġ: :: ĠRem ark Ġon ĠB PE </s> BI B REF 22 Ġshow Ġthat Ġsharing Ġsub words Ġbetween Ġlanguages Ġimproves Ġalign ments Ġbetween Ġembed ding Ġspaces . ĠB IB REF 2 Ġobserve Ġa Ġstrong Ġcorrelation Ġbetween Ġthe Ġpercentage Ġof Ġoverlapping Ġsub words Ġand Ġm BER T 's Ġperformances Ġfor Ġcross - ling ual Ġzero - shot Ġtransfer . ĠHowever , Ġin Ġour Ġcurrent Ġapproach , Ġsub words Ġbetween Ġsource Ġand Ġtarget Ġare Ġnot Ġshared . ĠA Ġsub word Ġthat Ġis Ġin Ġboth ĠEnglish Ġand Ġforeign Ġvocabulary Ġhas Ġtwo Ġdifferent Ġembed d ings . </s> Zero - shot ĠExper iments Ġ: :: ĠEst imating Ġtranslation Ġprobabilities </s> Since Ġpre - trained Ġmodels Ġoperate Ġon Ġsub word Ġlevel , Ġwe Ġneed Ġto Ġestimate Ġsub word Ġtranslation Ġprobabilities . ĠTherefore , Ġwe Ġsubs ample Ġ2 M Ġsentence Ġpairs Ġfrom Ġeach Ġparallel Ġcorpus Ġand Ġtoken ize Ġthe Ġdata Ġinto Ġsub words Ġbefore Ġrunning Ġfast - align ĠB IB REF 13 . </s> Est imating Ġsub word Ġtranslation Ġprobabilities Ġfrom Ġaligned Ġword Ġvectors Ġrequires Ġan Ġadditional Ġprocessing Ġstep Ġsince Ġthe Ġprovided Ġvectors Ġfrom Ġfast Text Ġare Ġnot Ġat Ġsub word Ġlevel . ĠWe Ġuse Ġthe Ġfollowing Ġapproximation Ġto Ġobtain Ġsub word Ġvectors : Ġthe Ġvector Ġ$_ s $ Ġof Ġsub word Ġ$ s $ Ġis Ġthe Ġweighted Ġaverage Ġof Ġall Ġthe Ġaligned Ġword Ġvectors Ġ$ _{ w _ i } $ Ġthat Ġhave Ġ$ s $ Ġas Ġan Ġsub word </s> where Ġ$ p ( w _ j ) $ Ġis Ġthe Ġun ig ram Ġprobability Ġof Ġword Ġ$ w _ j $ Ġand Ġ$ n _ s Ġ= Ġ\ sum Ġ_ { w _ j :\ , Ġs \ in Ġw _ j } Ġp ( w _ j ) $. ĠWe Ġtake Ġthe Ġtop Ġ50 , 000 Ġwords Ġin Ġeach Ġaligned Ġword - ve ctors Ġto Ġcompute Ġsub word Ġvectors . </s> In Ġboth Ġcases , Ġnot Ġall Ġthe Ġwords Ġin Ġthe Ġforeign Ġvocabulary Ġcan Ġbe Ġinitialized Ġfrom Ġthe ĠEnglish Ġword - embed d ings . ĠThose Ġwords Ġare Ġinitialized Ġrandomly Ġfrom Ġa ĠGa ussian Ġ$\ math cal Ġ{ N } ( 0 , Ġ{ 1 }{ d ^ 2 }) $. </s> Zero - shot ĠExper iments Ġ: :: ĠHyper - param eters </s> In Ġall Ġthe Ġexperiments , Ġwe Ġtune ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġfor Ġ175 , 000 Ġupdates Ġand ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġfor Ġ275 , 000 Ġupdates Ġwhere Ġthe Ġfirst Ġ25 , 000 Ġupdates Ġare Ġfor Ġlanguage Ġspecific Ġparameters . ĠThe Ġsequence Ġlength Ġis Ġset Ġto Ġ256 . ĠThe Ġmini - batch Ġsize Ġare Ġ64 Ġand Ġ24 Ġwhen Ġtuning Ġlanguage Ġspecific Ġparameters Ġusing ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġand ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġrespectively . ĠFor Ġtuning Ġbilingual ĠL Ms , Ġwe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ64 Ġfor ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġand Ġ24 Ġfor ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġwhere Ġhalf Ġof Ġthe Ġbatch Ġare ĠEnglish Ġsequences Ġand Ġthe Ġother Ġhalf Ġare Ġforeign Ġsequences . ĠThis Ġstrategy Ġof Ġbalancing Ġmini - batch Ġhas Ġbeen Ġused Ġin Ġmult ilingual Ġneural Ġmachine Ġtranslation ĠB IB REF 23 , ĠB IB REF 24 . </s> We Ġoptimize ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġusing ĠLook ahead Ġoptim izer ĠB IB REF 25 Ġwrapped Ġaround ĠAdam Ġwith Ġthe Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 4 } $, Ġthe Ġnumber Ġof Ġfast Ġweight Ġupdates Ġ$ k = 5 $, Ġand Ġinterpol ation Ġparameter Ġ$\ alpha Ġ= 0 . 5 $. ĠWe Ġchoose ĠLook ahead Ġoptim izer Ġbecause Ġit Ġhas Ġbeen Ġshown Ġto Ġbe Ġrobust Ġto Ġthe Ġinitial Ġparameters Ġof Ġthe Ġbased Ġoptim izer Ġ( Adam ). ĠFor ĠAdam Ġoptim izer , Ġwe Ġlin early Ġincrease Ġthe Ġlearning Ġrate Ġfrom Ġ$ 10 ^{ - 7 } $ Ġto Ġ$ 10 ^{ - 4 } $ Ġin Ġthe Ġfirst Ġ4000 Ġupdates Ġand Ġthen Ġfollow Ġan Ġinverse Ġsquare Ġroot Ġdecay . ĠAll ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġmodels Ġare Ġoptimized Ġwith ĠAdam Ġdue Ġto Ġmemory Ġlimit . </s> When Ġfine - tun ing ĠRAM EN Ġon ĠX N LI Ġand ĠUD , Ġwe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ32 , ĠAdam 's Ġlearning Ġrate Ġof Ġ$ 10 ^{ - 5 } $. ĠThe Ġnumber Ġof Ġepoch s Ġare Ġset Ġto Ġ4 Ġand Ġ50 Ġfor ĠX N LI Ġand ĠUD Ġtasks Ġrespectively . ĠAll Ġexperiments Ġare Ġcarried Ġout Ġon Ġa Ġsingle ĠTesla ĠV 100 Ġ16 GB ĠGPU . ĠEach ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġmodel Ġis Ġtrained Ġwithin Ġa Ġday Ġand Ġeach ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġis Ġtrained Ġwithin Ġtwo Ġdays . </s> Results </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġresults Ġof Ġout Ġmodels Ġfor Ġtwo Ġzero - shot Ġcross Ġling ual Ġtransfer Ġtasks : ĠX N LI Ġand Ġdependency Ġparsing . </s> Results Ġ: :: ĠCross - ling ual ĠNatural ĠLanguage ĠIn ference </s> Table ĠT AB REF 32 Ġshows Ġthe ĠX N LI Ġtest Ġaccuracy . ĠFor Ġreference , Ġwe Ġalso Ġinclude Ġthe Ġscores Ġfrom Ġthe Ġprevious Ġwork , Ġnotably Ġthe Ġstate - of - the - art Ġsystem ĠXL M ĠB IB REF 6 . ĠBefore Ġdiscussing Ġthe Ġresults , Ġwe Ġspell Ġout Ġthat Ġthe Ġfaire st Ġcomparison Ġin Ġthis Ġexperiment Ġis Ġthe Ġcomparison Ġbetween Ġm BER T Ġand ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġtrained Ġwith Ġmon oling ual Ġonly . </s> We Ġfirst Ġdiscuss Ġthe Ġtransfer Ġresults Ġfrom ĠB ERT . ĠIn itialized Ġfrom Ġfast Text Ġvectors , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġslightly Ġoutper forms Ġm BER T Ġby Ġ1 . 9 Ġpoints Ġon Ġaverage Ġand Ġwiden Ġthe Ġgap Ġof Ġ3 . 3 Ġpoints Ġon ĠArabic . ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġgains Ġextra Ġ0 . 8 Ġpoints Ġon Ġaverage Ġwhen Ġinitialized Ġfrom Ġparallel Ġdata . ĠWith Ġtriple Ġnumber Ġof Ġparameters , ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġoffers Ġan Ġadditional Ġboost Ġin Ġterm Ġof Ġaccuracy Ġand Ġinitialization Ġwith Ġparallel Ġdata Ġconsistently Ġimproves Ġthe Ġperformance . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġsignificantly Ġoutper forms ĠB ERT $ _{ \ text sc Ġ{ base }} $ Ġon Ġ11 ĠEnglish ĠN LP Ġtasks ĠB IB REF 0 , Ġthe Ġstrength Ġof ĠB ERT $ _{ \ text sc Ġ{ large }} $ Ġalso Ġshows Ġup Ġwhen Ġadapted Ġto Ġforeign Ġlanguages . </s> Transfer ring Ġfrom ĠRo BER Ta Ġleads Ġto Ġbetter Ġzero - shot Ġaccur acies . ĠWith Ġthe Ġsame Ġinitial izing Ġcondition , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + Ro BER Ta Ġoutper forms ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġon Ġaverage Ġby Ġ2 . 9 Ġand Ġ2 . 3 Ġpoints Ġwhen Ġinitial izing Ġfrom Ġmon oling ual Ġand Ġparallel Ġdata Ġrespectively . ĠThis Ġresult Ġshow Ġthat Ġwith Ġsimilar Ġnumber Ġof Ġparameters , Ġour Ġapproach Ġbenefits Ġfrom Ġa Ġbetter ĠEnglish Ġpre - trained Ġmodel . ĠWhen Ġtransferring Ġfrom ĠRo BER Ta $ _{ \ text sc Ġ{ large }} $, Ġwe Ġobtain Ġstate - of - the - art Ġresults Ġfor Ġfive Ġlanguages . </s> Currently , ĠRAM EN Ġonly Ġuses Ġparallel Ġdata Ġto Ġinitialize Ġforeign Ġembed d ings . ĠRAM EN Ġcan Ġalso Ġexploit Ġparallel Ġdata Ġthrough Ġtranslation Ġobjective Ġproposed Ġin ĠXL M . ĠWe Ġbelieve Ġthat Ġby Ġutilizing Ġparallel Ġdata Ġduring Ġthe Ġfine - tun ing Ġof ĠRAM EN Ġwould Ġbring Ġadditional Ġbenefits Ġfor Ġzero - shot Ġtasks . ĠWe Ġleave Ġthis Ġexploration Ġto Ġfuture Ġwork . ĠIn Ġsummary , Ġstarting Ġfrom ĠB ERT $ _{ \ text sc Ġ{ base }} $, Ġour Ġapproach Ġob tains Ġcompetitive Ġbilingual ĠL Ms Ġwith Ġm BER T Ġfor Ġzero - shot ĠX N LI . ĠOur Ġapproach Ġshows Ġthe Ġaccuracy Ġgains Ġwhen Ġadapting Ġfrom Ġa Ġbetter Ġpre - trained Ġmodel . </s> Results Ġ: :: ĠUniversal ĠDepend ency ĠPars ing </s> We Ġbuild Ġon Ġtop Ġof ĠRAM EN Ġa Ġgraph - based Ġdependency Ġparser ĠB IB REF 27 . ĠFor Ġthe Ġpurpose Ġof Ġevaluating Ġthe Ġcontextual Ġrepresentations Ġlearned Ġby Ġour Ġmodel , Ġwe Ġdo Ġnot Ġuse Ġpart - of - speech Ġtags . ĠContext ual ized Ġrepresentations Ġare Ġdirectly Ġfed Ġinto ĠDeep - B ia ff ine Ġlayers Ġto Ġpredict Ġarc Ġand Ġlabel Ġscores . ĠTable ĠT AB REF 34 Ġpresents Ġthe ĠLab eled ĠAtt achment ĠScores Ġ( L AS ) Ġfor Ġzero - shot Ġdependency Ġparsing . </s> We Ġfirst Ġlook Ġat Ġthe Ġfaire st Ġcomparison Ġbetween Ġm BER T Ġand Ġmon oling ually Ġinitialized ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T . ĠThe Ġlatter Ġoutper forms Ġthe Ġformer Ġon Ġfive Ġlanguages Ġexcept ĠArabic . ĠWe Ġobserve Ġthe Ġlargest Ġgain Ġof Ġ+ 5 . 2 ĠL AS Ġfor ĠFrench . ĠChinese Ġenjoys Ġ+ 3 . 1 ĠL AS Ġfrom Ġour Ġapproach . ĠWith Ġsimilar Ġarchitecture Ġ( 12 Ġor Ġ24 Ġlayers ) Ġand Ġinitialization Ġ( using Ġmon oling ual Ġor Ġparallel Ġdata ), ĠRAM EN + Ro BER Ta Ġperforms Ġbetter Ġthan ĠRAM EN + BER T Ġfor Ġmost Ġof Ġthe Ġlanguages . ĠArabic Ġand ĠHindi Ġbenefit Ġthe Ġmost Ġfrom Ġbigger Ġmodels . ĠFor Ġthe Ġother Ġfour Ġlanguages , ĠRAM EN $ _{ \ text sc Ġ{ large }} $ Ġrenders Ġa Ġmodest Ġimprovement Ġover ĠRAM EN $ _{ \ text sc Ġ{ base }} $. </s> Analysis Ġ: :: ĠImpact Ġof Ġinitialization </s> Initial izing Ġforeign Ġembed d ings Ġis Ġthe Ġbackbone Ġof Ġour Ġapproach . ĠA Ġgood Ġinitialization Ġleads Ġto Ġbetter Ġzero - shot Ġtransfer Ġresults Ġand Ġenables Ġfast Ġadaptation . ĠTo Ġverify Ġthe Ġimportance Ġof Ġa Ġgood Ġinitialization , Ġwe Ġtrain Ġa ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + Ro BER Ta Ġwith Ġforeign Ġword - embed d ings Ġare Ġinitialized Ġrandomly Ġfrom Ġ$\ math cal Ġ{ N } ( 0 , Ġ{ 1 }{ d ^ 2 }) $. ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġsame Ġhyper - param eters Ġin ĠÂ§ SEC REF 27 . ĠTable ĠT AB REF 36 Ġshows Ġthe Ġresults Ġof ĠX N LI Ġand ĠUD Ġparsing Ġof Ġrandom Ġinitialization . ĠIn Ġcomparison Ġto Ġthe Ġinitialization Ġusing Ġaligned Ġfast Text Ġvectors , Ġrandom Ġinitialization Ġdecreases Ġthe Ġzero - shot Ġperformance Ġof ĠRAM EN $ _{ \ text sc Ġ{ base }} $ Ġby Ġ15 . 9 % Ġfor ĠX N LI Ġand Ġ27 . 8 Ġpoints Ġfor ĠUD Ġparsing Ġon Ġaverage . ĠWe Ġalso Ġsee Ġthat Ġzero - shot Ġparsing Ġof ĠS OV Ġlanguages Ġ( Arab ic Ġand ĠHindi ) Ġsuffers Ġrandom Ġinitialization . </s> Analysis Ġ: :: ĠAre Ġcontextual Ġrepresentations Ġfrom ĠRAM EN Ġalso Ġgood Ġfor Ġsupervised Ġparsing ? </s> All Ġthe ĠRAM EN Ġmodels Ġare Ġbuilt Ġfrom ĠEnglish Ġand Ġtuned Ġon ĠEnglish Ġfor Ġzero - shot Ġcross - ling ual Ġtasks . ĠIt Ġis Ġreasonable Ġto Ġexpect ĠRAM EN s Ġdo Ġwell Ġin Ġthose Ġtasks Ġas Ġwe Ġhave Ġshown Ġin Ġour Ġexperiments . ĠBut Ġare Ġthey Ġalso Ġa Ġgood Ġfeature Ġextract or Ġfor Ġsupervised Ġtasks ? ĠWe Ġoffer Ġa Ġpartial Ġanswer Ġto Ġthis Ġquestion Ġby Ġevaluating Ġour Ġmodel Ġfor Ġsupervised Ġdependency Ġparsing Ġon ĠUD Ġdatasets . </s> We Ġused Ġtrain / dev / test Ġsplits Ġprovided Ġin ĠUD Ġto Ġtrain Ġand Ġevaluate Ġour ĠRAM EN - based Ġparser . ĠTable ĠT AB REF 38 Ġsummarizes Ġthe Ġresults Ġ( L AS ) Ġof Ġour Ġsupervised Ġparser . ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġchoose Ġm BER T Ġas Ġthe Ġbaseline Ġand Ġall Ġthe ĠRAM EN Ġmodels Ġare Ġinitialized Ġfrom Ġaligned Ġfast Text Ġvectors . ĠWith Ġthe Ġsame Ġarchitecture Ġof Ġ12 ĠTrans former Ġlayers , ĠRAM EN $ _{ \ text sc Ġ{ base }} $ + BER T Ġperforms Ġcompetitive Ġto Ġm BER T Ġand Ġout sh ines Ġm BER T Ġby Ġ+ 1 . 2 Ġpoints Ġfor ĠVietnamese . ĠThe Ġbest ĠL AS Ġresults Ġare Ġobtained Ġby ĠRAM EN $ _{ \ text sc Ġ{ large }} $ + Ro BER Ta Ġwith Ġ24 ĠTrans former Ġlayers . ĠOverall , Ġour Ġresults Ġindicate Ġthe Ġpotential Ġof Ġusing Ġcontextual Ġrepresentations Ġfrom ĠRAM EN Ġfor Ġsupervised Ġtasks . </s> Analysis Ġ: :: ĠHow Ġdoes Ġlinguistic Ġknowledge Ġtransfer Ġhappen Ġthrough Ġeach Ġtraining Ġstages ? </s> We Ġevaluate Ġthe Ġperformance Ġof ĠRAM EN + Ro BER Ta $ _{ \ text sc Ġ{ base }} $ Ġ( initialized Ġfrom Ġmon oling ual Ġdata ) Ġat Ġeach Ġtraining Ġsteps : Ġinitialization Ġof Ġword Ġembed d ings Ġ( 0 K Ġupdate ), Ġfine - tun ing Ġtarget Ġembed d ings Ġ( 25 K ), Ġand Ġfine - tun ing Ġthe Ġmodel Ġon Ġboth ĠEnglish Ġand Ġtarget Ġlanguage Ġ( at Ġeach Ġ25 K Ġupdates ). ĠThe Ġresults Ġare Ġpresented Ġin ĠFigure ĠFIG REF 40 . </s> Without Ġfine - tun ing , Ġthe Ġaverage Ġaccuracy Ġof ĠXL NI Ġis Ġ39 . 7 % Ġfor Ġa Ġthree - ways Ġclassification Ġtask , Ġand Ġthe Ġaverage ĠL AS Ġscore Ġis Ġ3 . 6 Ġfor Ġdependency Ġparsing . ĠWe Ġsee Ġthe Ġbiggest Ġleap Ġin Ġthe Ġperformance Ġafter Ġ50 K Ġupdates . ĠWhile Ġsemantic Ġsimilarity Ġtask Ġprofits Ġsignificantly Ġat Ġ25 K Ġupdates Ġof Ġthe Ġtarget Ġembed d ings , Ġsynt actic Ġtask Ġbenefits Ġwith Ġfurther Ġfine - tun ing Ġthe Ġenc oder . ĠThis Ġis Ġexpected Ġsince Ġthe Ġtarget Ġlanguages Ġmight Ġexhibit Ġdifferent Ġsynt actic Ġstructures Ġthan ĠEnglish Ġand Ġfine - tun ing Ġenc oder Ġhelps Ġto Ġcapture Ġlanguage Ġspecific Ġstructures . ĠWe Ġobserve Ġa Ġsubstantial Ġgain Ġof Ġ19 - 30 ĠL AS Ġfor Ġall Ġlanguages Ġexcept ĠFrench Ġafter Ġ50 K Ġupdates . </s> Language Ġsimilarities Ġhave Ġmore Ġimpact Ġon Ġtransferring Ġsyntax Ġthan Ġsemantics . ĠWithout Ġtuning Ġthe ĠEnglish Ġenc oder , ĠFrench Ġenjoys Ġ50 . 3 ĠL AS Ġfor Ġbeing Ġclosely Ġrelated Ġto ĠEnglish , Ġwhereas ĠArabic Ġand ĠHindi , ĠS OV Ġlanguages , Ġmodest ly Ġreach Ġ4 . 2 Ġand Ġ6 . 4 Ġpoints Ġusing Ġthe ĠSV O Ġenc oder . ĠAlthough ĠChinese Ġhas ĠSV O Ġorder , Ġit Ġis Ġoften Ġseen Ġas Ġhead - final Ġwhile ĠEnglish Ġis Ġstrong Ġhead - initial . ĠPerhaps , Ġthis Ġexplains Ġthe Ġpoor Ġperformance Ġfor ĠChinese . </s> Lim itations </s> While Ġwe Ġhave Ġsuccessfully Ġadapted Ġaut oen c oding Ġpre - trained ĠL Ms Ġfrom ĠEnglish Ġto Ġother Ġlanguages , Ġthe Ġquestion Ġwhether Ġour Ġapproach Ġcan Ġalso Ġbe Ġapplied Ġfor Ġaut ore gressive ĠLM Ġsuch Ġas ĠXL Net Ġstill Ġremains . ĠWe Ġleave Ġthe Ġinvestigation Ġto Ġfuture Ġwork . </s> Con clusions </s> In Ġthis Ġwork , Ġwe Ġhave Ġpresented Ġa Ġsimple Ġand Ġeffective Ġapproach Ġfor Ġrapidly Ġbuilding Ġa Ġbilingual ĠLM Ġunder Ġa Ġlimited Ġcomputational Ġbudget . ĠUsing ĠB ERT Ġas Ġthe Ġstarting Ġpoint , Ġwe Ġdemonstrate Ġthat Ġour Ġapproach Ġproduces Ġbetter Ġthan Ġm BER T Ġon Ġtwo Ġcross - ling ual Ġzero - shot Ġsentence Ġclassification Ġand Ġdependency Ġparsing . ĠWe Ġfind Ġthat Ġthe Ġperformance Ġof Ġour Ġbilingual ĠLM , ĠRAM EN , Ġcorrelates Ġwith Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġpre - trained ĠEnglish Ġmodels . ĠWe Ġalso Ġfind Ġthat ĠRAM EN Ġis Ġalso Ġa Ġpowerful Ġfeature Ġextract or Ġin Ġsupervised Ġdependency Ġparsing . ĠFinally , Ġwe Ġhope Ġthat Ġour Ġwork Ġsparks Ġof Ġinterest Ġin Ġdeveloping Ġfast Ġand Ġeffective Ġmethods Ġfor Ġtransferring Ġpre - trained ĠEnglish Ġmodels Ġto Ġother Ġlanguages .