we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged
EXTRACTIVE
We have training and testing sets in three different languages: English, Chinese and Korean. The English dataset is SQuAD BIBREF2. The Chinese dataset is DRCD BIBREF14, a Chinese RC dataset with 30,000+ examples in the training set and 10,000+ examples in the development set. The Korean dataset is KorQuAD BIBREF15, a Korean RC dataset with 60,000+ examples in the training set and 10,000+ examples in the development set, created in exactly the same procedure as SQuAD. We always use the development sets of SQuAD, DRCD and KorQuAD for testing since the testing sets of the corpora have not been released yet. The pre-trained multi-BERT is the official released one. This multi-lingual version of BERT were pre-trained on corpus in 104 languages. Data in different languages were simply mixed in batches while pre-training, without additional effort to align between languages. When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged.
1909.09587
Introduction </s> Reading ĠComp rehens ion Ġ( RC ) Ġhas Ġbecome Ġa Ġcentral Ġtask Ġin Ġnatural Ġlanguage Ġprocessing , Ġwith Ġgreat Ġpractical Ġvalue Ġin Ġvarious Ġindustries . ĠIn Ġrecent Ġyears , Ġmany Ġlarge - scale ĠRC Ġdatasets Ġin ĠEnglish ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġhave Ġnour ished Ġthe Ġdevelopment Ġof Ġnumerous Ġpowerful Ġand Ġdiverse ĠRC Ġmodels ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 . ĠThe Ġstate - of - the - art Ġmodel ĠB IB REF 12 Ġon ĠS Qu AD , Ġone Ġof Ġthe Ġmost Ġwidely Ġused ĠRC Ġbenchmarks , Ġeven Ġsurpass es Ġhuman - level Ġperformance . ĠNonetheless , ĠRC Ġon Ġlanguages Ġother Ġthan ĠEnglish Ġhas Ġbeen Ġlimited Ġdue Ġto Ġthe Ġabsence Ġof Ġsufficient Ġtraining Ġdata . ĠAlthough Ġsome Ġefforts Ġhave Ġbeen Ġmade Ġto Ġcreate ĠRC Ġdatasets Ġfor ĠChinese ĠB IB REF 13 , ĠB IB REF 14 Ġand ĠKorean ĠB IB REF 15 , Ġit Ġis Ġnot Ġfeasible Ġto Ġcollect ĠRC Ġdatasets Ġfor Ġevery Ġlanguage Ġsince Ġannotation Ġefforts Ġto Ġcollect Ġa Ġnew ĠRC Ġdataset Ġare Ġoften Ġfar Ġfrom Ġtrivial . ĠTherefore , Ġthe Ġsetup Ġof Ġtransfer Ġlearning , Ġespecially Ġzero - shot Ġlearning , Ġis Ġof Ġextraordinary Ġimportance . </s> Ex isting Ġmethods ĠB IB REF 16 Ġof Ġcross - ling ual Ġtransfer Ġlearning Ġon ĠRC Ġdatasets Ġoften Ġcount Ġon Ġmachine Ġtranslation Ġ( MT ) Ġto Ġtranslate Ġdata Ġfrom Ġsource Ġlanguage Ġinto Ġtarget Ġlanguage , Ġor Ġvice Ġversa . ĠThese Ġmethods Ġmay Ġnot Ġrequire Ġa Ġwell - annot ated ĠRC Ġdataset Ġfor Ġthe Ġtarget Ġlanguage , Ġwhereas Ġa Ġhigh - quality ĠMT Ġmodel Ġis Ġneeded Ġas Ġa Ġtrade - off , Ġwhich Ġmight Ġnot Ġbe Ġavailable Ġwhen Ġit Ġcomes Ġto Ġlow - resource Ġlanguages . </s> In Ġthis Ġpaper , Ġwe Ġleverage Ġpre - trained Ġmult ilingual Ġlanguage Ġrepresentation , Ġfor Ġexample , ĠB ERT Ġlearned Ġfrom Ġmult ilingual Ġun - annot ated Ġsentences Ġ( multi - BER T ), Ġin Ġcross - ling ual Ġzero - shot ĠRC . ĠWe Ġfine - t une Ġmulti - BER T Ġon Ġthe Ġtraining Ġset Ġin Ġsource Ġlanguage , Ġthen Ġtest Ġthe Ġmodel Ġin Ġtarget Ġlanguage , Ġwith Ġa Ġnumber Ġof Ġcombinations Ġof Ġsource - target Ġlanguage Ġpair Ġto Ġexplore Ġthe Ġcross - ling ual Ġability Ġof Ġmulti - BER T . ĠSurprisingly , Ġwe Ġfind Ġthat Ġthe Ġmodels Ġhave Ġthe Ġability Ġto Ġtransfer Ġbetween Ġlow Ġlex ical Ġsimilarity Ġlanguage Ġpair , Ġsuch Ġas ĠEnglish Ġand ĠChinese . ĠRecent Ġstudies ĠB IB REF 17 , ĠB IB REF 12 , ĠB IB REF 18 Ġshow Ġthat Ġcross - ling ual Ġlanguage Ġmodels Ġhave Ġthe Ġability Ġto Ġenable Ġpreliminary Ġzero - shot Ġtransfer Ġon Ġsimple Ġnatural Ġlanguage Ġunderstanding Ġtasks , Ġbut Ġzero - shot Ġtransfer Ġof ĠRC Ġhas Ġnot Ġbeen Ġstudied . ĠTo Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġsystematically Ġexploring Ġthe Ġcross - ling ual Ġtransferring Ġability Ġof Ġmulti - BER T Ġon ĠRC Ġtasks . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T </s> Multi - BER T Ġhas Ġshowcased Ġits Ġability Ġto Ġenable Ġcross - ling ual Ġzero - shot Ġlearning Ġon Ġthe Ġnatural Ġlanguage Ġunderstanding Ġtasks Ġincluding ĠX N LI ĠB IB REF 19 , ĠN ER , ĠPOS , ĠDepend ency ĠPars ing , Ġand Ġso Ġon . ĠWe Ġnow Ġseek Ġto Ġknow Ġif Ġa Ġpre - trained Ġmulti - BER T Ġhas Ġability Ġto Ġsolve ĠRC Ġtasks Ġin Ġthe Ġzero - shot Ġsetting . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠExperimental ĠSetup Ġand ĠData </s> We Ġhave Ġtraining Ġand Ġtesting Ġsets Ġin Ġthree Ġdifferent Ġlanguages : ĠEnglish , ĠChinese Ġand ĠKorean . ĠThe ĠEnglish Ġdataset Ġis ĠS Qu AD ĠB IB REF 2 . ĠThe ĠChinese Ġdataset Ġis ĠD RC D ĠB IB REF 14 , Ġa ĠChinese ĠRC Ġdataset Ġwith Ġ30 , 000 + Ġexamples Ġin Ġthe Ġtraining Ġset Ġand Ġ10 , 000 + Ġexamples Ġin Ġthe Ġdevelopment Ġset . ĠThe ĠKorean Ġdataset Ġis ĠKor Qu AD ĠB IB REF 15 , Ġa ĠKorean ĠRC Ġdataset Ġwith Ġ60 , 000 + Ġexamples Ġin Ġthe Ġtraining Ġset Ġand Ġ10 , 000 + Ġexamples Ġin Ġthe Ġdevelopment Ġset , Ġcreated Ġin Ġexactly Ġthe Ġsame Ġprocedure Ġas ĠS Qu AD . ĠWe Ġalways Ġuse Ġthe Ġdevelopment Ġsets Ġof ĠS Qu AD , ĠD RC D Ġand ĠKor Qu AD Ġfor Ġtesting Ġsince Ġthe Ġtesting Ġsets Ġof Ġthe Ġcorpor a Ġhave Ġnot Ġbeen Ġreleased Ġyet . </s> Next , Ġto Ġconstruct Ġa Ġdiverse Ġcross - ling ual ĠRC Ġdataset Ġwith Ġcompromised Ġquality , Ġwe Ġtranslated Ġthe ĠEnglish Ġand ĠChinese Ġdatasets Ġinto Ġmore Ġlanguages , Ġwith ĠGoogle ĠTrans late . ĠAn Ġobvious Ġissue Ġwith Ġthis Ġmethod Ġis Ġthat Ġsome Ġexamples Ġmight Ġno Ġlonger Ġhave Ġa Ġrecover able Ġspan . ĠTo Ġsolve Ġthe Ġproblem , Ġwe Ġuse Ġfuzzy Ġmatching Ġto Ġfind Ġthe Ġmost Ġpossible Ġanswer , Ġwhich Ġcalculates Ġminimal Ġedit Ġdistance Ġbetween Ġtranslated Ġanswer Ġand Ġall Ġpossible Ġspans . ĠIf Ġthe Ġminimal Ġedit Ġdistance Ġis Ġlarger Ġthan Ġmin ( 10 , Ġlengths Ġof Ġtranslated Ġanswer Ġ- Ġ1 ), Ġwe Ġdrop Ġthe Ġexamples Ġduring Ġtraining , Ġand Ġtreat Ġthem Ġas Ġnoise Ġwhen Ġtesting . ĠIn Ġthis Ġway , Ġwe Ġcan Ġrecover Ġmore Ġthan Ġ95 % Ġof Ġexamples . ĠThe Ġfollowing Ġgenerated Ġdatasets Ġare Ġrecovered Ġwith Ġsame Ġsetting . </s> The Ġpre - trained Ġmulti - BER T Ġis Ġthe Ġofficial Ġreleased Ġone . ĠThis Ġmulti - ling ual Ġversion Ġof ĠB ERT Ġwere Ġpre - trained Ġon Ġcorpus Ġin Ġ104 Ġlanguages . ĠData Ġin Ġdifferent Ġlanguages Ġwere Ġsimply Ġmixed Ġin Ġbatches Ġwhile Ġpre - training , Ġwithout Ġadditional Ġeffort Ġto Ġalign Ġbetween Ġlanguages . ĠWhen Ġfine - tun ing , Ġwe Ġsimply Ġadopted Ġthe Ġofficial Ġtraining Ġscript Ġof ĠB ERT , Ġwith Ġdefault Ġhyper param eters , Ġto Ġfine - t une Ġeach Ġmodel Ġuntil Ġtraining Ġloss Ġconver ged . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠExperimental ĠResults </s> Table ĠT AB REF 6 Ġshows Ġthe Ġresult Ġof Ġdifferent Ġmodels Ġtrained Ġon Ġeither ĠChinese Ġor ĠEnglish Ġand Ġtested Ġon ĠChinese . ĠIn Ġrow Ġ( f ), Ġmulti - BER T Ġis Ġfine - tun ed Ġon ĠEnglish Ġbut Ġtested Ġon ĠChinese , Ġwhich Ġachieves Ġcompetitive Ġperformance Ġcompared Ġwith ĠQ AN et Ġtrained Ġon ĠChinese . ĠWe Ġalso Ġfind Ġthat Ġmulti - BER T Ġtrained Ġon ĠEnglish Ġhas Ġrelatively Ġlower ĠEM Ġcompared Ġwith Ġthe Ġmodel Ġwith Ġcomparable ĠF 1 Ġscores . ĠThis Ġshows Ġthat Ġthe Ġmodel Ġlearned Ġwith Ġzero - shot Ġcan Ġroughly Ġidentify Ġthe Ġanswer Ġspans Ġin Ġcontext Ġbut Ġless Ġaccurate . ĠIn Ġrow Ġ( c ), Ġwe Ġfine - tun ed Ġa ĠB ERT Ġmodel Ġpre - trained Ġon ĠEnglish Ġmon oling ual Ġcorpus Ġ( English ĠB ERT ) Ġon ĠChinese ĠRC Ġtraining Ġdata Ġdirectly Ġby Ġapp ending Ġfast Text - initialized ĠChinese Ġword Ġembed d ings Ġto Ġthe Ġoriginal Ġword Ġembed d ings Ġof ĠEnglish - BER T . ĠIts ĠF 1 Ġscore Ġis Ġeven Ġlower Ġthan Ġthat Ġof Ġzero - shot Ġtransferring Ġmulti - BER T Ġ( rows Ġ( c ) Ġv . s . Ġ( e )). ĠThe Ġresult Ġimplies Ġmulti - BER T Ġdoes Ġacquire Ġbetter Ġcross - ling ual Ġcapability Ġthrough Ġpre - training Ġon Ġmult ilingual Ġcorpus . ĠTable ĠT AB REF 8 Ġshows Ġthe Ġresults Ġof Ġmulti - BER T Ġfine - tun ed Ġon Ġdifferent Ġlanguages Ġand Ġthen Ġtested Ġon ĠEnglish Ġ, ĠChinese Ġand ĠKorean . ĠThe Ġtop Ġhalf Ġof Ġthe Ġtable Ġshows Ġthe Ġresults Ġof Ġtraining Ġdata Ġwithout Ġtranslation . ĠIt Ġis Ġnot Ġsurprising Ġthat Ġwhen Ġthe Ġtraining Ġand Ġtesting Ġsets Ġare Ġin Ġthe Ġsame Ġlanguage , Ġthe Ġbest Ġresults Ġare Ġachieved , Ġand Ġmulti - BER T Ġshows Ġtransfer Ġcapability Ġwhen Ġtraining Ġand Ġtesting Ġsets Ġare Ġin Ġdifferent Ġlanguages , Ġespecially Ġbetween ĠChinese Ġand ĠKorean . </s> In Ġthe Ġlower Ġhalf Ġof ĠTable ĠT AB REF 8 , Ġthe Ġresults Ġare Ġobtained Ġby Ġthe Ġtranslated Ġtraining Ġdata . ĠFirst , Ġwe Ġfound Ġthat Ġwhen Ġtesting Ġon ĠEnglish Ġand ĠChinese , Ġtranslation Ġalways Ġde grades Ġthe Ġperformance Ġ( En Ġv . s . ĠEn - XX , ĠZh Ġv . s . ĠZh - XX ). ĠEven Ġthough Ġwe Ġtranslate Ġthe Ġtraining Ġdata Ġinto Ġthe Ġsame Ġlanguage Ġas Ġtesting Ġdata , Ġusing Ġthe Ġunt rans lated Ġdata Ġstill Ġyield Ġbetter Ġresults . ĠFor Ġexample , Ġwhen Ġtesting Ġon ĠEnglish , Ġthe ĠF 1 Ġscore Ġof Ġthe Ġmodel Ġtraining Ġon ĠChinese Ġ( Z h ) Ġis Ġ53 . 8 , Ġwhile Ġthe ĠF 1 Ġscore Ġis Ġonly Ġ44 . 1 Ġfor Ġthe Ġmodel Ġtraining Ġon ĠZh - En . ĠThis Ġshows Ġthat Ġtranslation Ġde grades Ġthe Ġquality Ġof Ġdata . ĠThere Ġare Ġsome Ġexceptions Ġwhen Ġtesting Ġon ĠKorean . ĠTrans l ating Ġthe ĠEnglish Ġtraining Ġdata Ġinto ĠChinese , ĠJapanese Ġand ĠKorean Ġstill Ġimprove Ġthe Ġperformance Ġon ĠKorean . ĠWe Ġalso Ġfound Ġthat Ġwhen Ġtranslated Ġinto Ġthe Ġsame Ġlanguage , Ġthe ĠEnglish Ġtraining Ġdata Ġis Ġalways Ġbetter Ġthan Ġthe ĠChinese Ġdata Ġ( En - XX Ġv . s . ĠZh - XX ), Ġwith Ġonly Ġone Ġexception Ġ( En - Fr Ġv . s . ĠZh - Fr Ġwhen Ġtesting Ġon ĠKor Qu AD ). ĠThis Ġmay Ġbe Ġbecause Ġwe Ġhave Ġless ĠChinese Ġtraining Ġdata Ġthan ĠEnglish . ĠThese Ġresults Ġshow Ġthat Ġthe Ġquality Ġand Ġthe Ġsize Ġof Ġdataset Ġare Ġmuch Ġmore Ġimportant Ġthan Ġwhether Ġthe Ġtraining Ġand Ġtesting Ġare Ġin Ġthe Ġsame Ġlanguage Ġor Ġnot . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠDiscussion Ġ: :: ĠThe ĠEffect Ġof ĠMachine ĠTranslation </s> Table ĠT AB REF 8 Ġshows Ġthat Ġfine - tun ing Ġon Ġun - trans lated Ġtarget Ġlanguage Ġdata Ġachieves Ġmuch Ġbetter Ġperformance Ġthan Ġdata Ġtranslated Ġinto Ġthe Ġtarget Ġlanguage . ĠBecause Ġthe Ġabove Ġstatement Ġis Ġtrue Ġacross Ġall Ġthe Ġlanguages , Ġit Ġis Ġa Ġstrong Ġevidence Ġthat Ġtranslation Ġde grades Ġthe Ġperformance . We Ġnotice Ġthat Ġthe Ġtranslated Ġcorpus Ġand Ġunt rans lated Ġcorpus Ġare Ġnot Ġthe Ġsame . ĠThis Ġmay Ġbe Ġanother Ġfactor Ġthat Ġinfluences Ġthe Ġresults . ĠConduct ing Ġan Ġexperiment Ġbetween Ġun - trans lated Ġand Ġback - trans lated Ġdata Ġmay Ġdeal Ġwith Ġthis Ġproblem . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠDiscussion Ġ: :: ĠThe ĠEffect Ġof ĠOther ĠFactors </s> Here Ġwe Ġdiscuss Ġthe Ġcase Ġthat Ġthe Ġtraining Ġdata Ġare Ġtranslated . ĠWe Ġconsider Ġeach Ġresult Ġis Ġaffected Ġby Ġat Ġleast Ġthree Ġfactors : Ġ( 1 ) Ġtraining Ġcorpus , Ġ( 2 ) Ġdata Ġsize , Ġ( 3 ) Ġwhether Ġthe Ġsource Ġcorpus Ġis Ġtranslated Ġinto Ġthe Ġtarget Ġlanguage . ĠTo Ġstudy Ġthe Ġeffect Ġof Ġdata - size , Ġwe Ġconducted Ġan Ġextra Ġexperiment Ġwhere Ġwe Ġdown - sam pled Ġthe Ġsize Ġof ĠEnglish Ġdata Ġto Ġbe Ġthe Ġsame Ġas ĠChinese Ġcorpus , Ġand Ġused Ġthe Ġdown - sam pled Ġcorpus Ġto Ġtrain . ĠThen ĠWe Ġcarried Ġout Ġone - way ĠAN OVA Ġtest Ġand Ġfound Ġout Ġthe Ġsignificance Ġof Ġthe Ġthree Ġfactors Ġare Ġranked Ġas Ġbelow : Ġ( 1 ) Ġ> Ġ( 2 ) Ġ>> Ġ( 3 ). ĠThe Ġanalysis Ġsupports Ġthat Ġthe Ġcharacteristics Ġof Ġtraining Ġdata Ġis Ġmore Ġimportant Ġthan Ġtranslated Ġinto Ġtarget Ġlanguage Ġor Ġnot . ĠTherefore , Ġalthough Ġtranslation Ġde grades Ġthe Ġperformance , Ġwhether Ġtranslating Ġthe Ġcorpus Ġinto Ġthe Ġtarget Ġlanguage Ġis Ġnot Ġcritical . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠUn seen ĠLanguage ĠDat as et </s> It Ġhas Ġbeen Ġshown Ġthat Ġextract ive ĠQ A Ġtasks Ġlike ĠS Qu AD Ġmay Ġbe Ġtackled Ġby Ġsome Ġlanguage Ġindependent Ġstrategies , Ġfor Ġexample , Ġmatching Ġwords Ġin Ġquestions Ġand Ġcontext ĠB IB REF 20 . ĠIs Ġzero - shot Ġlearning Ġfeasible Ġbecause Ġthe Ġmodel Ġsimply Ġlearns Ġthis Ġkind Ġof Ġlanguage Ġindependent Ġstrategies Ġon Ġone Ġlanguage Ġand Ġapply Ġto Ġthe Ġother ? </s> To Ġverify Ġwhether Ġmulti - BER T Ġlargely Ġcounts Ġon Ġa Ġlanguage Ġindependent Ġstrategy , Ġwe Ġtest Ġthe Ġmodel Ġon Ġthe Ġlanguages Ġunseen Ġduring Ġpre - training . ĠTo Ġmake Ġsure Ġthe Ġlanguages Ġhave Ġnever Ġbeen Ġseen Ġbefore , Ġwe Ġartificially Ġmake Ġunseen Ġlanguages Ġby Ġperm uting Ġthe Ġwhole Ġvocabulary Ġof Ġexisting Ġlanguages . ĠThat Ġis , Ġall Ġthe Ġwords Ġin Ġthe Ġsentences Ġof Ġa Ġspecific Ġlanguage Ġare Ġreplaced Ġby Ġother Ġwords Ġin Ġthe Ġsame Ġlanguage Ġto Ġform Ġthe Ġsentences Ġin Ġthe Ġcreated Ġunseen Ġlanguage . ĠIt Ġis Ġassumed Ġthat Ġif Ġmulti - BER T Ġused Ġto Ġfind Ġanswers Ġby Ġlanguage Ġindependent Ġstrategy , Ġthen Ġmulti - BER T Ġshould Ġalso Ġdo Ġwell Ġon Ġunseen Ġlanguages . ĠTable ĠT AB REF 14 Ġshows Ġthat Ġthe Ġperformance Ġof Ġmulti - BER T Ġdrops Ġdrastically Ġon Ġthe Ġdataset . ĠIt Ġimplies Ġthat Ġmulti - BER T Ġmight Ġnot Ġtotally Ġrely Ġon Ġpattern Ġmatching Ġwhen Ġfinding Ġanswers . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠEmb edd ing Ġin ĠMulti - BER T </s> PC A Ġprojection Ġof Ġhidden Ġrepresentations Ġof Ġthe Ġlast Ġlayer Ġof Ġmulti - BER T Ġbefore Ġand Ġafter Ġfine - tun ing Ġare Ġshown Ġin ĠFig . ĠFIG REF 15 . ĠThe Ġred Ġpoints Ġrepresent ĠChinese Ġtokens , Ġand Ġthe Ġblue Ġpoints Ġare Ġfor ĠEnglish . ĠThe Ġresults Ġshow Ġthat Ġtokens Ġfrom Ġdifferent Ġlanguages Ġmight Ġbe Ġembedded Ġinto Ġthe Ġsame Ġspace Ġwith Ġclose Ġspatial Ġdistribution . ĠEven Ġthough Ġduring Ġthe Ġfine - tun ing Ġonly Ġthe ĠEnglish Ġdata Ġis Ġused , Ġthe Ġembed ding Ġof Ġthe ĠChinese Ġtoken Ġchanged Ġaccordingly . ĠWe Ġalso Ġquant itatively Ġevaluate Ġthe Ġsimilarities Ġbetween Ġthe Ġembed ding Ġof Ġthe Ġlanguages . ĠThe Ġresults Ġcan Ġbe Ġfound Ġin Ġthe ĠAppendix . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠCode - sw itching ĠDat as et </s> We Ġobserve Ġlinguistic - agn ostic Ġrepresentations Ġin Ġthe Ġlast Ġsubsection . ĠIf Ġtokens Ġare Ġrepresented Ġin Ġa Ġlanguage - agn ostic Ġway , Ġthe Ġmodel Ġmay Ġbe Ġable Ġto Ġhandle Ġcode - sw itching Ġdata . ĠBecause Ġthere Ġis Ġno Ġcode - sw itching Ġdata Ġfor ĠRC , Ġwe Ġcreate Ġartificial Ġcode - sw itching Ġdatasets Ġby Ġreplacing Ġsome Ġof Ġthe Ġwords Ġin Ġcontexts Ġor Ġquestions Ġwith Ġtheir Ġsyn onyms Ġin Ġanother Ġlanguage . ĠThe Ġsyn onyms Ġare Ġfound Ġby Ġword - by - word Ġtranslation Ġwith Ġgiven Ġdiction aries . ĠWe Ġuse Ġthe Ġbilingual Ġdiction aries Ġcollected Ġand Ġreleased Ġin Ġfacebook research / M USE ĠGitHub Ġrepository . ĠWe Ġsubstitute Ġthe Ġwords Ġif Ġand Ġonly Ġif Ġthe Ġwords Ġare Ġin Ġthe Ġbilingual Ġdiction aries . </s> Table ĠT AB REF 14 Ġshows Ġthat Ġon Ġall Ġthe Ġcode - sw itching Ġdatasets , Ġthe ĠEM / F 1 Ġscore Ġdrops , Ġindicating Ġthat Ġthe Ġsemantics Ġof Ġrepresentations Ġare Ġnot Ġtotally Ġdis ent angled Ġfrom Ġlanguage . ĠHowever , Ġthe Ġexamples Ġof Ġthe Ġanswers Ġof Ġthe Ġmodel Ġ( Table ĠT AB REF 21 ) Ġshow Ġthat Ġmulti - BER T Ġcould Ġfind Ġthe Ġcorrect Ġanswer Ġspans Ġalthough Ġsome Ġkeywords Ġin Ġthe Ġspans Ġhave Ġbeen Ġtranslated Ġinto Ġanother Ġlanguage . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠTyp ology - man ip ulated ĠDat as et </s> There Ġare Ġvarious Ġtypes Ġof Ġtyp ology Ġin Ġlanguages . ĠFor Ġexample , Ġin ĠEnglish Ġthe Ġtyp ology Ġorder Ġis Ġsubject - verb - object Ġ( S VO ) Ġorder , Ġbut Ġin ĠJapanese Ġand ĠKorean Ġthe Ġorder Ġis Ġsubject - object - verb Ġ( S OV ). ĠWe Ġconstruct Ġa Ġtyp ology - man ip ulated Ġdataset Ġto Ġexamine Ġif Ġthe Ġtyp ology Ġorder Ġof Ġthe Ġtraining Ġdata Ġinfluences Ġthe Ġtransfer Ġlearning Ġresults . ĠIf Ġthe Ġmodel Ġonly Ġlearns Ġthe Ġsemantic Ġmapping Ġbetween Ġdifferent Ġlanguages , Ġchanging ĠEnglish Ġtyp ology Ġorder Ġfrom ĠSV O Ġto ĠS OV Ġshould Ġimprove Ġthe Ġtransfer Ġability Ġfrom ĠEnglish Ġto ĠJapanese . ĠThe Ġmethod Ġused Ġto Ġgenerate Ġdatasets Ġis Ġthe Ġsame Ġas ĠB IB REF 21 . </s> The Ġsource Ġcode Ġis Ġfrom Ġa ĠGitHub Ġrepository Ġnamed ĠSh aul 13 21 / r nn _ typ ology , Ġwhich Ġlabels Ġgiven Ġsentences Ġto ĠCo N LL Ġformat Ġwith ĠStanford Core N LP Ġand Ġthen Ġre - arr anges Ġthem Ġgreed ily . </s> Table ĠT AB REF 23 Ġshows Ġthat Ġwhen Ġwe Ġchange Ġthe ĠEnglish Ġtyp ology Ġorder Ġto ĠS OV Ġor ĠOS V Ġorder , Ġthe Ġperformance Ġon ĠKorean Ġis Ġimproved Ġand Ġworsen Ġon ĠEnglish Ġand ĠChinese , Ġbut Ġvery Ġslightly . ĠThe Ġresults Ġshow Ġthat Ġthe Ġtyp ology Ġmanipulation Ġon Ġthe Ġtraining Ġset Ġhas Ġlittle Ġinfluence . ĠIt Ġis Ġpossible Ġthat Ġmulti - BER T Ġnormal izes Ġthe Ġtyp ology Ġorder Ġof Ġdifferent Ġlanguages Ġto Ġsome Ġextent . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġsystematically Ġexplore Ġzero - shot Ġcross - ling ual Ġtransfer Ġlearning Ġon ĠRC Ġwith Ġmulti - BER T . ĠThe Ġexperimental Ġresults Ġon ĠEnglish , ĠChinese Ġand ĠKorean Ġcorpor a Ġshow Ġthat Ġeven Ġwhen Ġthe Ġlanguages Ġfor Ġtraining Ġand Ġtesting Ġare Ġdifferent , Ġreasonable Ġperformance Ġcan Ġbe Ġobtained . ĠFurthermore , Ġwe Ġcreated Ġseveral Ġartificial Ġdata Ġto Ġstudy Ġthe Ġcross - ling ual Ġability Ġof Ġmulti - BER T Ġin Ġthe Ġpresence Ġof Ġtyp ology Ġvariation Ġand Ġcode - sw itching . ĠWe Ġshowed Ġthat Ġonly Ġtoken - level Ġpattern Ġmatching Ġis Ġnot Ġsufficient Ġfor Ġmulti - BER T Ġto Ġanswer Ġquestions Ġand Ġtyp ology Ġvariation Ġand Ġcode - sw itching Ġonly Ġcaused Ġminor Ġeffects Ġon Ġtesting Ġperformance . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T </s> The Ġarchitecture Ġof Ġmulti - BER T Ġis Ġa ĠTrans former Ġenc oder ĠB IB REF 25 . ĠWhile Ġfine - tun ing Ġon ĠS Qu AD - like Ġdataset , Ġthe Ġbottom Ġlayers Ġof Ġmulti - BER T Ġare Ġinitialized Ġfrom ĠGoogle - pret rained Ġparameters , Ġwith Ġan Ġadded Ġoutput Ġlayer Ġinitialized Ġfrom Ġrandom Ġparameters . ĠTokens Ġrepresentations Ġfrom Ġthe Ġlast Ġlayer Ġof Ġbottom - part Ġof Ġmulti - BER T Ġare Ġinputs Ġto Ġthe Ġoutput Ġlayer Ġand Ġthen Ġthe Ġoutput Ġlayer Ġoutputs Ġa Ġdistribution Ġover Ġall Ġtokens Ġthat Ġindicates Ġthe Ġprobability Ġof Ġa Ġtoken Ġbeing Ġthe ĠSTART / END Ġof Ġan Ġanswer Ġspan . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T Ġ: :: ĠCos ine ĠSimilar ity </s> As Ġall Ġtranslated Ġversions Ġof ĠS Qu AD / D RC D Ġare Ġparallel Ġto Ġeach Ġother . ĠGiven Ġa Ġsource - target Ġlanguage Ġpair , Ġwe Ġcalculate Ġcos ine Ġsimilarity Ġof Ġthe Ġmean Ġpool ing Ġof Ġtokens Ġrepresentation Ġwithin Ġcorresponding Ġanswer - span Ġas Ġa Ġmeasure Ġof Ġhow Ġmuch Ġthey Ġlook Ġlike Ġin Ġterms Ġof Ġthe Ġinternal Ġrepresentation Ġof Ġmulti - BER T . ĠThe Ġresults Ġare Ġshown Ġin ĠFig . ĠFIG REF 26 . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T Ġ: :: ĠSV CC A </s> Sing ular ĠVector ĠCanon ical ĠCor relation ĠAnalysis Ġ( S V CC A ) Ġis Ġa Ġgeneral Ġmethod Ġto Ġcompare Ġthe Ġcorrelation Ġof Ġtwo Ġsets Ġof Ġvector Ġrepresentations . ĠSV CC A Ġhas Ġbeen Ġproposed Ġto Ġcompare Ġlearned Ġrepresentations Ġacross Ġlanguage Ġmodels ĠB IB REF 24 . ĠHere Ġwe Ġadopt ĠSV CC A Ġto Ġmeasure Ġthe Ġlinear Ġsimilarity Ġof Ġtwo Ġsets Ġof Ġrepresentations Ġin Ġthe Ġsame Ġmulti - BER T Ġfrom Ġdifferent Ġtranslated Ġdatasets , Ġwhich Ġare Ġparallel Ġto Ġeach Ġother . ĠThe Ġresults Ġare Ġshown Ġin ĠFig ĠFIG REF 28 . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering </s> In Ġthe Ġpaper , Ġwe Ġshow Ġthat Ġinternal Ġrepresentations Ġof Ġmulti - BER T Ġare Ġlinear - m app able Ġto Ġsome Ġextent Ġbetween Ġdifferent Ġlanguages . ĠThis Ġimplies Ġthat Ġmulti - BER T Ġmodel Ġmight Ġencode Ġsemantic Ġand Ġsynt actic Ġinformation Ġin Ġlanguage - agn ostic Ġways Ġand Ġexplains Ġhow Ġzero - shot Ġtransfer Ġlearning Ġcould Ġbe Ġdone . </s> To Ġtake Ġa Ġstep Ġfurther , Ġwhile Ġtransf ering Ġmodel Ġfrom Ġsource Ġdataset Ġto Ġtarget Ġdataset , Ġwe Ġalign Ġrepresentations Ġin Ġtwo Ġproposed Ġway , Ġto Ġimprove Ġperformance Ġon Ġtarget Ġdataset . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠLinear ĠM apping ĠMethod </s> Al gorith ms Ġproposed Ġin ĠB IB REF 23 , ĠB IB REF 22 , ĠB IB REF 26 Ġto Ġun super vised ly Ġlearn Ġlinear Ġmapping Ġbetween Ġtwo Ġsets Ġof Ġembed d ings Ġare Ġused Ġhere Ġto Ġalign Ġrepresentations Ġof Ġsource Ġ( training Ġdata ) Ġto Ġthose Ġof Ġtarget . ĠWe Ġobtain Ġthe Ġmapping Ġgenerated Ġby Ġembed d ings Ġfrom Ġone Ġspecific Ġlayer Ġof Ġpre - trained Ġmulti - BER T Ġthen Ġwe Ġapply Ġthis Ġmapping Ġto Ġtransform Ġthe Ġinternal Ġrepresentations Ġof Ġmulti - BER T Ġwhile Ġfine - tun ing Ġon Ġtraining Ġdata . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠAd vers arial ĠMethod </s> In ĠAd vers arial ĠMethod , Ġwe Ġadd Ġan Ġadditional Ġtransform Ġlayer Ġto Ġtransform Ġrepresentations Ġand Ġa Ġdiscrimination Ġlayer Ġto Ġdiscriminate Ġbetween Ġtransformed Ġrepresentations Ġfrom Ġsource Ġlanguage Ġ( training Ġset ) Ġand Ġtarget Ġlanguage Ġ( development Ġset ). ĠAnd Ġthe ĠG AN Ġloss Ġis Ġapplied Ġin Ġthe Ġtotal Ġloss Ġof Ġfine - tun ing . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠDiscussion </s> As Ġtable ĠT AB REF 33 Ġshows , Ġthere Ġare Ġno Ġimprovements Ġamong Ġabove Ġmethods . ĠSome Ġlinear Ġmapping Ġmethods Ġeven Ġcauses Ġdevastating Ġeffect Ġon ĠEM / F 1 Ġscores .
Introduction Reading Comprehension (RC) has become a central task in natural language processing, with great practical value in various industries. In recent years, many large-scale RC datasets in English BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 have nourished the development of numerous powerful and diverse RC models BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11. The state-of-the-art model BIBREF12 on SQuAD, one of the most widely used RC benchmarks, even surpasses human-level performance. Nonetheless, RC on languages other than English has been limited due to the absence of sufficient training data. Although some efforts have been made to create RC datasets for Chinese BIBREF13, BIBREF14 and Korean BIBREF15, it is not feasible to collect RC datasets for every language since annotation efforts to collect a new RC dataset are often far from trivial. Therefore, the setup of transfer learning, especially zero-shot learning, is of extraordinary importance. Existing methods BIBREF16 of cross-lingual transfer learning on RC datasets often count on machine translation (MT) to translate data from source language into target language, or vice versa. These methods may not require a well-annotated RC dataset for the target language, whereas a high-quality MT model is needed as a trade-off, which might not be available when it comes to low-resource languages. In this paper, we leverage pre-trained multilingual language representation, for example, BERT learned from multilingual un-annotated sentences (multi-BERT), in cross-lingual zero-shot RC. We fine-tune multi-BERT on the training set in source language, then test the model in target language, with a number of combinations of source-target language pair to explore the cross-lingual ability of multi-BERT. Surprisingly, we find that the models have the ability to transfer between low lexical similarity language pair, such as English and Chinese. Recent studies BIBREF17, BIBREF12, BIBREF18 show that cross-lingual language models have the ability to enable preliminary zero-shot transfer on simple natural language understanding tasks, but zero-shot transfer of RC has not been studied. To our knowledge, this is the first work systematically exploring the cross-lingual transferring ability of multi-BERT on RC tasks. Zero-shot Transfer with Multi-BERT Multi-BERT has showcased its ability to enable cross-lingual zero-shot learning on the natural language understanding tasks including XNLI BIBREF19, NER, POS, Dependency Parsing, and so on. We now seek to know if a pre-trained multi-BERT has ability to solve RC tasks in the zero-shot setting. Zero-shot Transfer with Multi-BERT ::: Experimental Setup and Data We have training and testing sets in three different languages: English, Chinese and Korean. The English dataset is SQuAD BIBREF2. The Chinese dataset is DRCD BIBREF14, a Chinese RC dataset with 30,000+ examples in the training set and 10,000+ examples in the development set. The Korean dataset is KorQuAD BIBREF15, a Korean RC dataset with 60,000+ examples in the training set and 10,000+ examples in the development set, created in exactly the same procedure as SQuAD. We always use the development sets of SQuAD, DRCD and KorQuAD for testing since the testing sets of the corpora have not been released yet. Next, to construct a diverse cross-lingual RC dataset with compromised quality, we translated the English and Chinese datasets into more languages, with Google Translate. An obvious issue with this method is that some examples might no longer have a recoverable span. To solve the problem, we use fuzzy matching to find the most possible answer, which calculates minimal edit distance between translated answer and all possible spans. If the minimal edit distance is larger than min(10, lengths of translated answer - 1), we drop the examples during training, and treat them as noise when testing. In this way, we can recover more than 95% of examples. The following generated datasets are recovered with same setting. The pre-trained multi-BERT is the official released one. This multi-lingual version of BERT were pre-trained on corpus in 104 languages. Data in different languages were simply mixed in batches while pre-training, without additional effort to align between languages. When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged. Zero-shot Transfer with Multi-BERT ::: Experimental Results Table TABREF6 shows the result of different models trained on either Chinese or English and tested on Chinese. In row (f), multi-BERT is fine-tuned on English but tested on Chinese, which achieves competitive performance compared with QANet trained on Chinese. We also find that multi-BERT trained on English has relatively lower EM compared with the model with comparable F1 scores. This shows that the model learned with zero-shot can roughly identify the answer spans in context but less accurate. In row (c), we fine-tuned a BERT model pre-trained on English monolingual corpus (English BERT) on Chinese RC training data directly by appending fastText-initialized Chinese word embeddings to the original word embeddings of English-BERT. Its F1 score is even lower than that of zero-shot transferring multi-BERT (rows (c) v.s. (e)). The result implies multi-BERT does acquire better cross-lingual capability through pre-training on multilingual corpus. Table TABREF8 shows the results of multi-BERT fine-tuned on different languages and then tested on English , Chinese and Korean. The top half of the table shows the results of training data without translation. It is not surprising that when the training and testing sets are in the same language, the best results are achieved, and multi-BERT shows transfer capability when training and testing sets are in different languages, especially between Chinese and Korean. In the lower half of Table TABREF8, the results are obtained by the translated training data. First, we found that when testing on English and Chinese, translation always degrades the performance (En v.s. En-XX, Zh v.s. Zh-XX). Even though we translate the training data into the same language as testing data, using the untranslated data still yield better results. For example, when testing on English, the F1 score of the model training on Chinese (Zh) is 53.8, while the F1 score is only 44.1 for the model training on Zh-En. This shows that translation degrades the quality of data. There are some exceptions when testing on Korean. Translating the English training data into Chinese, Japanese and Korean still improve the performance on Korean. We also found that when translated into the same language, the English training data is always better than the Chinese data (En-XX v.s. Zh-XX), with only one exception (En-Fr v.s. Zh-Fr when testing on KorQuAD). This may be because we have less Chinese training data than English. These results show that the quality and the size of dataset are much more important than whether the training and testing are in the same language or not. Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Machine Translation Table TABREF8 shows that fine-tuning on un-translated target language data achieves much better performance than data translated into the target language. Because the above statement is true across all the languages, it is a strong evidence that translation degrades the performance.We notice that the translated corpus and untranslated corpus are not the same. This may be another factor that influences the results. Conducting an experiment between un-translated and back-translated data may deal with this problem. Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Other Factors Here we discuss the case that the training data are translated. We consider each result is affected by at least three factors: (1) training corpus, (2) data size, (3) whether the source corpus is translated into the target language. To study the effect of data-size, we conducted an extra experiment where we down-sampled the size of English data to be the same as Chinese corpus, and used the down-sampled corpus to train. Then We carried out one-way ANOVA test and found out the significance of the three factors are ranked as below: (1) > (2) >> (3). The analysis supports that the characteristics of training data is more important than translated into target language or not. Therefore, although translation degrades the performance, whether translating the corpus into the target language is not critical. What Does Zero-shot Transfer Model Learn? ::: Unseen Language Dataset It has been shown that extractive QA tasks like SQuAD may be tackled by some language independent strategies, for example, matching words in questions and context BIBREF20. Is zero-shot learning feasible because the model simply learns this kind of language independent strategies on one language and apply to the other? To verify whether multi-BERT largely counts on a language independent strategy, we test the model on the languages unseen during pre-training. To make sure the languages have never been seen before, we artificially make unseen languages by permuting the whole vocabulary of existing languages. That is, all the words in the sentences of a specific language are replaced by other words in the same language to form the sentences in the created unseen language. It is assumed that if multi-BERT used to find answers by language independent strategy, then multi-BERT should also do well on unseen languages. Table TABREF14 shows that the performance of multi-BERT drops drastically on the dataset. It implies that multi-BERT might not totally rely on pattern matching when finding answers. What Does Zero-shot Transfer Model Learn? ::: Embedding in Multi-BERT PCA projection of hidden representations of the last layer of multi-BERT before and after fine-tuning are shown in Fig. FIGREF15. The red points represent Chinese tokens, and the blue points are for English. The results show that tokens from different languages might be embedded into the same space with close spatial distribution. Even though during the fine-tuning only the English data is used, the embedding of the Chinese token changed accordingly. We also quantitatively evaluate the similarities between the embedding of the languages. The results can be found in the Appendix. What Does Zero-shot Transfer Model Learn? ::: Code-switching Dataset We observe linguistic-agnostic representations in the last subsection. If tokens are represented in a language-agnostic way, the model may be able to handle code-switching data. Because there is no code-switching data for RC, we create artificial code-switching datasets by replacing some of the words in contexts or questions with their synonyms in another language. The synonyms are found by word-by-word translation with given dictionaries. We use the bilingual dictionaries collected and released in facebookresearch/MUSE GitHub repository. We substitute the words if and only if the words are in the bilingual dictionaries. Table TABREF14 shows that on all the code-switching datasets, the EM/F1 score drops, indicating that the semantics of representations are not totally disentangled from language. However, the examples of the answers of the model (Table TABREF21) show that multi-BERT could find the correct answer spans although some keywords in the spans have been translated into another language. What Does Zero-shot Transfer Model Learn? ::: Typology-manipulated Dataset There are various types of typology in languages. For example, in English the typology order is subject-verb-object (SVO) order, but in Japanese and Korean the order is subject-object-verb (SOV). We construct a typology-manipulated dataset to examine if the typology order of the training data influences the transfer learning results. If the model only learns the semantic mapping between different languages, changing English typology order from SVO to SOV should improve the transfer ability from English to Japanese. The method used to generate datasets is the same as BIBREF21. The source code is from a GitHub repository named Shaul1321/rnn_typology, which labels given sentences to CoNLL format with StanfordCoreNLP and then re-arranges them greedily. Table TABREF23 shows that when we change the English typology order to SOV or OSV order, the performance on Korean is improved and worsen on English and Chinese, but very slightly. The results show that the typology manipulation on the training set has little influence. It is possible that multi-BERT normalizes the typology order of different languages to some extent. Conclusion In this paper, we systematically explore zero-shot cross-lingual transfer learning on RC with multi-BERT. The experimental results on English, Chinese and Korean corpora show that even when the languages for training and testing are different, reasonable performance can be obtained. Furthermore, we created several artificial data to study the cross-lingual ability of multi-BERT in the presence of typology variation and code-switching. We showed that only token-level pattern matching is not sufficient for multi-BERT to answer questions and typology variation and code-switching only caused minor effects on testing performance. Supplemental Material ::: Internal Representation of multi-BERT The architecture of multi-BERT is a Transformer encoder BIBREF25. While fine-tuning on SQuAD-like dataset, the bottom layers of multi-BERT are initialized from Google-pretrained parameters, with an added output layer initialized from random parameters. Tokens representations from the last layer of bottom-part of multi-BERT are inputs to the output layer and then the output layer outputs a distribution over all tokens that indicates the probability of a token being the START/END of an answer span. Supplemental Material ::: Internal Representation of multi-BERT ::: Cosine Similarity As all translated versions of SQuAD/DRCD are parallel to each other. Given a source-target language pair, we calculate cosine similarity of the mean pooling of tokens representation within corresponding answer-span as a measure of how much they look like in terms of the internal representation of multi-BERT. The results are shown in Fig. FIGREF26. Supplemental Material ::: Internal Representation of multi-BERT ::: SVCCA Singular Vector Canonical Correlation Analysis (SVCCA) is a general method to compare the correlation of two sets of vector representations. SVCCA has been proposed to compare learned representations across language models BIBREF24. Here we adopt SVCCA to measure the linear similarity of two sets of representations in the same multi-BERT from different translated datasets, which are parallel to each other. The results are shown in Fig FIGREF28. Supplemental Material ::: Improve Transfering In the paper, we show that internal representations of multi-BERT are linear-mappable to some extent between different languages. This implies that multi-BERT model might encode semantic and syntactic information in language-agnostic ways and explains how zero-shot transfer learning could be done. To take a step further, while transfering model from source dataset to target dataset, we align representations in two proposed way, to improve performance on target dataset. Supplemental Material ::: Improve Transfering ::: Linear Mapping Method Algorithms proposed in BIBREF23, BIBREF22, BIBREF26 to unsupervisedly learn linear mapping between two sets of embeddings are used here to align representations of source (training data) to those of target. We obtain the mapping generated by embeddings from one specific layer of pre-trained multi-BERT then we apply this mapping to transform the internal representations of multi-BERT while fine-tuning on training data. Supplemental Material ::: Improve Transfering ::: Adversarial Method In Adversarial Method, we add an additional transform layer to transform representations and a discrimination layer to discriminate between transformed representations from source language (training set) and target language (development set). And the GAN loss is applied in the total loss of fine-tuning. Supplemental Material ::: Improve Transfering ::: Discussion As table TABREF33 shows, there are no improvements among above methods. Some linear mapping methods even causes devastating effect on EM/F1 scores.
what does the model learn in zero-shot setting?
55569d0a4586d20c01268a80a7e31a17a18198e2
what Ġdoes Ġthe Ġmodel Ġlearn Ġin Ġzero - shot Ġsetting ?
we Ġsimply Ġadopted Ġthe Ġofficial Ġtraining Ġscript Ġof ĠB ERT , Ġwith Ġdefault Ġhyper param eters , Ġto Ġfine - t une Ġeach Ġmodel Ġuntil Ġtraining Ġloss Ġconver ged
<s> what Ġdoes Ġthe Ġmodel Ġlearn Ġin Ġzero - shot Ġsetting ? </s> Introduction </s> Reading ĠComp rehens ion Ġ( RC ) Ġhas Ġbecome Ġa Ġcentral Ġtask Ġin Ġnatural Ġlanguage Ġprocessing , Ġwith Ġgreat Ġpractical Ġvalue Ġin Ġvarious Ġindustries . ĠIn Ġrecent Ġyears , Ġmany Ġlarge - scale ĠRC Ġdatasets Ġin ĠEnglish ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġhave Ġnour ished Ġthe Ġdevelopment Ġof Ġnumerous Ġpowerful Ġand Ġdiverse ĠRC Ġmodels ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 . ĠThe Ġstate - of - the - art Ġmodel ĠB IB REF 12 Ġon ĠS Qu AD , Ġone Ġof Ġthe Ġmost Ġwidely Ġused ĠRC Ġbenchmarks , Ġeven Ġsurpass es Ġhuman - level Ġperformance . ĠNonetheless , ĠRC Ġon Ġlanguages Ġother Ġthan ĠEnglish Ġhas Ġbeen Ġlimited Ġdue Ġto Ġthe Ġabsence Ġof Ġsufficient Ġtraining Ġdata . ĠAlthough Ġsome Ġefforts Ġhave Ġbeen Ġmade Ġto Ġcreate ĠRC Ġdatasets Ġfor ĠChinese ĠB IB REF 13 , ĠB IB REF 14 Ġand ĠKorean ĠB IB REF 15 , Ġit Ġis Ġnot Ġfeasible Ġto Ġcollect ĠRC Ġdatasets Ġfor Ġevery Ġlanguage Ġsince Ġannotation Ġefforts Ġto Ġcollect Ġa Ġnew ĠRC Ġdataset Ġare Ġoften Ġfar Ġfrom Ġtrivial . ĠTherefore , Ġthe Ġsetup Ġof Ġtransfer Ġlearning , Ġespecially Ġzero - shot Ġlearning , Ġis Ġof Ġextraordinary Ġimportance . </s> Ex isting Ġmethods ĠB IB REF 16 Ġof Ġcross - ling ual Ġtransfer Ġlearning Ġon ĠRC Ġdatasets Ġoften Ġcount Ġon Ġmachine Ġtranslation Ġ( MT ) Ġto Ġtranslate Ġdata Ġfrom Ġsource Ġlanguage Ġinto Ġtarget Ġlanguage , Ġor Ġvice Ġversa . ĠThese Ġmethods Ġmay Ġnot Ġrequire Ġa Ġwell - annot ated ĠRC Ġdataset Ġfor Ġthe Ġtarget Ġlanguage , Ġwhereas Ġa Ġhigh - quality ĠMT Ġmodel Ġis Ġneeded Ġas Ġa Ġtrade - off , Ġwhich Ġmight Ġnot Ġbe Ġavailable Ġwhen Ġit Ġcomes Ġto Ġlow - resource Ġlanguages . </s> In Ġthis Ġpaper , Ġwe Ġleverage Ġpre - trained Ġmult ilingual Ġlanguage Ġrepresentation , Ġfor Ġexample , ĠB ERT Ġlearned Ġfrom Ġmult ilingual Ġun - annot ated Ġsentences Ġ( multi - BER T ), Ġin Ġcross - ling ual Ġzero - shot ĠRC . ĠWe Ġfine - t une Ġmulti - BER T Ġon Ġthe Ġtraining Ġset Ġin Ġsource Ġlanguage , Ġthen Ġtest Ġthe Ġmodel Ġin Ġtarget Ġlanguage , Ġwith Ġa Ġnumber Ġof Ġcombinations Ġof Ġsource - target Ġlanguage Ġpair Ġto Ġexplore Ġthe Ġcross - ling ual Ġability Ġof Ġmulti - BER T . ĠSurprisingly , Ġwe Ġfind Ġthat Ġthe Ġmodels Ġhave Ġthe Ġability Ġto Ġtransfer Ġbetween Ġlow Ġlex ical Ġsimilarity Ġlanguage Ġpair , Ġsuch Ġas ĠEnglish Ġand ĠChinese . ĠRecent Ġstudies ĠB IB REF 17 , ĠB IB REF 12 , ĠB IB REF 18 Ġshow Ġthat Ġcross - ling ual Ġlanguage Ġmodels Ġhave Ġthe Ġability Ġto Ġenable Ġpreliminary Ġzero - shot Ġtransfer Ġon Ġsimple Ġnatural Ġlanguage Ġunderstanding Ġtasks , Ġbut Ġzero - shot Ġtransfer Ġof ĠRC Ġhas Ġnot Ġbeen Ġstudied . ĠTo Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġwork Ġsystematically Ġexploring Ġthe Ġcross - ling ual Ġtransferring Ġability Ġof Ġmulti - BER T Ġon ĠRC Ġtasks . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T </s> Multi - BER T Ġhas Ġshowcased Ġits Ġability Ġto Ġenable Ġcross - ling ual Ġzero - shot Ġlearning Ġon Ġthe Ġnatural Ġlanguage Ġunderstanding Ġtasks Ġincluding ĠX N LI ĠB IB REF 19 , ĠN ER , ĠPOS , ĠDepend ency ĠPars ing , Ġand Ġso Ġon . ĠWe Ġnow Ġseek Ġto Ġknow Ġif Ġa Ġpre - trained Ġmulti - BER T Ġhas Ġability Ġto Ġsolve ĠRC Ġtasks Ġin Ġthe Ġzero - shot Ġsetting . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠExperimental ĠSetup Ġand ĠData </s> We Ġhave Ġtraining Ġand Ġtesting Ġsets Ġin Ġthree Ġdifferent Ġlanguages : ĠEnglish , ĠChinese Ġand ĠKorean . ĠThe ĠEnglish Ġdataset Ġis ĠS Qu AD ĠB IB REF 2 . ĠThe ĠChinese Ġdataset Ġis ĠD RC D ĠB IB REF 14 , Ġa ĠChinese ĠRC Ġdataset Ġwith Ġ30 , 000 + Ġexamples Ġin Ġthe Ġtraining Ġset Ġand Ġ10 , 000 + Ġexamples Ġin Ġthe Ġdevelopment Ġset . ĠThe ĠKorean Ġdataset Ġis ĠKor Qu AD ĠB IB REF 15 , Ġa ĠKorean ĠRC Ġdataset Ġwith Ġ60 , 000 + Ġexamples Ġin Ġthe Ġtraining Ġset Ġand Ġ10 , 000 + Ġexamples Ġin Ġthe Ġdevelopment Ġset , Ġcreated Ġin Ġexactly Ġthe Ġsame Ġprocedure Ġas ĠS Qu AD . ĠWe Ġalways Ġuse Ġthe Ġdevelopment Ġsets Ġof ĠS Qu AD , ĠD RC D Ġand ĠKor Qu AD Ġfor Ġtesting Ġsince Ġthe Ġtesting Ġsets Ġof Ġthe Ġcorpor a Ġhave Ġnot Ġbeen Ġreleased Ġyet . </s> Next , Ġto Ġconstruct Ġa Ġdiverse Ġcross - ling ual ĠRC Ġdataset Ġwith Ġcompromised Ġquality , Ġwe Ġtranslated Ġthe ĠEnglish Ġand ĠChinese Ġdatasets Ġinto Ġmore Ġlanguages , Ġwith ĠGoogle ĠTrans late . ĠAn Ġobvious Ġissue Ġwith Ġthis Ġmethod Ġis Ġthat Ġsome Ġexamples Ġmight Ġno Ġlonger Ġhave Ġa Ġrecover able Ġspan . ĠTo Ġsolve Ġthe Ġproblem , Ġwe Ġuse Ġfuzzy Ġmatching Ġto Ġfind Ġthe Ġmost Ġpossible Ġanswer , Ġwhich Ġcalculates Ġminimal Ġedit Ġdistance Ġbetween Ġtranslated Ġanswer Ġand Ġall Ġpossible Ġspans . ĠIf Ġthe Ġminimal Ġedit Ġdistance Ġis Ġlarger Ġthan Ġmin ( 10 , Ġlengths Ġof Ġtranslated Ġanswer Ġ- Ġ1 ), Ġwe Ġdrop Ġthe Ġexamples Ġduring Ġtraining , Ġand Ġtreat Ġthem Ġas Ġnoise Ġwhen Ġtesting . ĠIn Ġthis Ġway , Ġwe Ġcan Ġrecover Ġmore Ġthan Ġ95 % Ġof Ġexamples . ĠThe Ġfollowing Ġgenerated Ġdatasets Ġare Ġrecovered Ġwith Ġsame Ġsetting . </s> The Ġpre - trained Ġmulti - BER T Ġis Ġthe Ġofficial Ġreleased Ġone . ĠThis Ġmulti - ling ual Ġversion Ġof ĠB ERT Ġwere Ġpre - trained Ġon Ġcorpus Ġin Ġ104 Ġlanguages . ĠData Ġin Ġdifferent Ġlanguages Ġwere Ġsimply Ġmixed Ġin Ġbatches Ġwhile Ġpre - training , Ġwithout Ġadditional Ġeffort Ġto Ġalign Ġbetween Ġlanguages . ĠWhen Ġfine - tun ing , Ġwe Ġsimply Ġadopted Ġthe Ġofficial Ġtraining Ġscript Ġof ĠB ERT , Ġwith Ġdefault Ġhyper param eters , Ġto Ġfine - t une Ġeach Ġmodel Ġuntil Ġtraining Ġloss Ġconver ged . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠExperimental ĠResults </s> Table ĠT AB REF 6 Ġshows Ġthe Ġresult Ġof Ġdifferent Ġmodels Ġtrained Ġon Ġeither ĠChinese Ġor ĠEnglish Ġand Ġtested Ġon ĠChinese . ĠIn Ġrow Ġ( f ), Ġmulti - BER T Ġis Ġfine - tun ed Ġon ĠEnglish Ġbut Ġtested Ġon ĠChinese , Ġwhich Ġachieves Ġcompetitive Ġperformance Ġcompared Ġwith ĠQ AN et Ġtrained Ġon ĠChinese . ĠWe Ġalso Ġfind Ġthat Ġmulti - BER T Ġtrained Ġon ĠEnglish Ġhas Ġrelatively Ġlower ĠEM Ġcompared Ġwith Ġthe Ġmodel Ġwith Ġcomparable ĠF 1 Ġscores . ĠThis Ġshows Ġthat Ġthe Ġmodel Ġlearned Ġwith Ġzero - shot Ġcan Ġroughly Ġidentify Ġthe Ġanswer Ġspans Ġin Ġcontext Ġbut Ġless Ġaccurate . ĠIn Ġrow Ġ( c ), Ġwe Ġfine - tun ed Ġa ĠB ERT Ġmodel Ġpre - trained Ġon ĠEnglish Ġmon oling ual Ġcorpus Ġ( English ĠB ERT ) Ġon ĠChinese ĠRC Ġtraining Ġdata Ġdirectly Ġby Ġapp ending Ġfast Text - initialized ĠChinese Ġword Ġembed d ings Ġto Ġthe Ġoriginal Ġword Ġembed d ings Ġof ĠEnglish - BER T . ĠIts ĠF 1 Ġscore Ġis Ġeven Ġlower Ġthan Ġthat Ġof Ġzero - shot Ġtransferring Ġmulti - BER T Ġ( rows Ġ( c ) Ġv . s . Ġ( e )). ĠThe Ġresult Ġimplies Ġmulti - BER T Ġdoes Ġacquire Ġbetter Ġcross - ling ual Ġcapability Ġthrough Ġpre - training Ġon Ġmult ilingual Ġcorpus . ĠTable ĠT AB REF 8 Ġshows Ġthe Ġresults Ġof Ġmulti - BER T Ġfine - tun ed Ġon Ġdifferent Ġlanguages Ġand Ġthen Ġtested Ġon ĠEnglish Ġ, ĠChinese Ġand ĠKorean . ĠThe Ġtop Ġhalf Ġof Ġthe Ġtable Ġshows Ġthe Ġresults Ġof Ġtraining Ġdata Ġwithout Ġtranslation . ĠIt Ġis Ġnot Ġsurprising Ġthat Ġwhen Ġthe Ġtraining Ġand Ġtesting Ġsets Ġare Ġin Ġthe Ġsame Ġlanguage , Ġthe Ġbest Ġresults Ġare Ġachieved , Ġand Ġmulti - BER T Ġshows Ġtransfer Ġcapability Ġwhen Ġtraining Ġand Ġtesting Ġsets Ġare Ġin Ġdifferent Ġlanguages , Ġespecially Ġbetween ĠChinese Ġand ĠKorean . </s> In Ġthe Ġlower Ġhalf Ġof ĠTable ĠT AB REF 8 , Ġthe Ġresults Ġare Ġobtained Ġby Ġthe Ġtranslated Ġtraining Ġdata . ĠFirst , Ġwe Ġfound Ġthat Ġwhen Ġtesting Ġon ĠEnglish Ġand ĠChinese , Ġtranslation Ġalways Ġde grades Ġthe Ġperformance Ġ( En Ġv . s . ĠEn - XX , ĠZh Ġv . s . ĠZh - XX ). ĠEven Ġthough Ġwe Ġtranslate Ġthe Ġtraining Ġdata Ġinto Ġthe Ġsame Ġlanguage Ġas Ġtesting Ġdata , Ġusing Ġthe Ġunt rans lated Ġdata Ġstill Ġyield Ġbetter Ġresults . ĠFor Ġexample , Ġwhen Ġtesting Ġon ĠEnglish , Ġthe ĠF 1 Ġscore Ġof Ġthe Ġmodel Ġtraining Ġon ĠChinese Ġ( Z h ) Ġis Ġ53 . 8 , Ġwhile Ġthe ĠF 1 Ġscore Ġis Ġonly Ġ44 . 1 Ġfor Ġthe Ġmodel Ġtraining Ġon ĠZh - En . ĠThis Ġshows Ġthat Ġtranslation Ġde grades Ġthe Ġquality Ġof Ġdata . ĠThere Ġare Ġsome Ġexceptions Ġwhen Ġtesting Ġon ĠKorean . ĠTrans l ating Ġthe ĠEnglish Ġtraining Ġdata Ġinto ĠChinese , ĠJapanese Ġand ĠKorean Ġstill Ġimprove Ġthe Ġperformance Ġon ĠKorean . ĠWe Ġalso Ġfound Ġthat Ġwhen Ġtranslated Ġinto Ġthe Ġsame Ġlanguage , Ġthe ĠEnglish Ġtraining Ġdata Ġis Ġalways Ġbetter Ġthan Ġthe ĠChinese Ġdata Ġ( En - XX Ġv . s . ĠZh - XX ), Ġwith Ġonly Ġone Ġexception Ġ( En - Fr Ġv . s . ĠZh - Fr Ġwhen Ġtesting Ġon ĠKor Qu AD ). ĠThis Ġmay Ġbe Ġbecause Ġwe Ġhave Ġless ĠChinese Ġtraining Ġdata Ġthan ĠEnglish . ĠThese Ġresults Ġshow Ġthat Ġthe Ġquality Ġand Ġthe Ġsize Ġof Ġdataset Ġare Ġmuch Ġmore Ġimportant Ġthan Ġwhether Ġthe Ġtraining Ġand Ġtesting Ġare Ġin Ġthe Ġsame Ġlanguage Ġor Ġnot . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠDiscussion Ġ: :: ĠThe ĠEffect Ġof ĠMachine ĠTranslation </s> Table ĠT AB REF 8 Ġshows Ġthat Ġfine - tun ing Ġon Ġun - trans lated Ġtarget Ġlanguage Ġdata Ġachieves Ġmuch Ġbetter Ġperformance Ġthan Ġdata Ġtranslated Ġinto Ġthe Ġtarget Ġlanguage . ĠBecause Ġthe Ġabove Ġstatement Ġis Ġtrue Ġacross Ġall Ġthe Ġlanguages , Ġit Ġis Ġa Ġstrong Ġevidence Ġthat Ġtranslation Ġde grades Ġthe Ġperformance . We Ġnotice Ġthat Ġthe Ġtranslated Ġcorpus Ġand Ġunt rans lated Ġcorpus Ġare Ġnot Ġthe Ġsame . ĠThis Ġmay Ġbe Ġanother Ġfactor Ġthat Ġinfluences Ġthe Ġresults . ĠConduct ing Ġan Ġexperiment Ġbetween Ġun - trans lated Ġand Ġback - trans lated Ġdata Ġmay Ġdeal Ġwith Ġthis Ġproblem . </s> Zero - shot ĠTransfer Ġwith ĠMulti - BER T Ġ: :: ĠDiscussion Ġ: :: ĠThe ĠEffect Ġof ĠOther ĠFactors </s> Here Ġwe Ġdiscuss Ġthe Ġcase Ġthat Ġthe Ġtraining Ġdata Ġare Ġtranslated . ĠWe Ġconsider Ġeach Ġresult Ġis Ġaffected Ġby Ġat Ġleast Ġthree Ġfactors : Ġ( 1 ) Ġtraining Ġcorpus , Ġ( 2 ) Ġdata Ġsize , Ġ( 3 ) Ġwhether Ġthe Ġsource Ġcorpus Ġis Ġtranslated Ġinto Ġthe Ġtarget Ġlanguage . ĠTo Ġstudy Ġthe Ġeffect Ġof Ġdata - size , Ġwe Ġconducted Ġan Ġextra Ġexperiment Ġwhere Ġwe Ġdown - sam pled Ġthe Ġsize Ġof ĠEnglish Ġdata Ġto Ġbe Ġthe Ġsame Ġas ĠChinese Ġcorpus , Ġand Ġused Ġthe Ġdown - sam pled Ġcorpus Ġto Ġtrain . ĠThen ĠWe Ġcarried Ġout Ġone - way ĠAN OVA Ġtest Ġand Ġfound Ġout Ġthe Ġsignificance Ġof Ġthe Ġthree Ġfactors Ġare Ġranked Ġas Ġbelow : Ġ( 1 ) Ġ> Ġ( 2 ) Ġ>> Ġ( 3 ). ĠThe Ġanalysis Ġsupports Ġthat Ġthe Ġcharacteristics Ġof Ġtraining Ġdata Ġis Ġmore Ġimportant Ġthan Ġtranslated Ġinto Ġtarget Ġlanguage Ġor Ġnot . ĠTherefore , Ġalthough Ġtranslation Ġde grades Ġthe Ġperformance , Ġwhether Ġtranslating Ġthe Ġcorpus Ġinto Ġthe Ġtarget Ġlanguage Ġis Ġnot Ġcritical . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠUn seen ĠLanguage ĠDat as et </s> It Ġhas Ġbeen Ġshown Ġthat Ġextract ive ĠQ A Ġtasks Ġlike ĠS Qu AD Ġmay Ġbe Ġtackled Ġby Ġsome Ġlanguage Ġindependent Ġstrategies , Ġfor Ġexample , Ġmatching Ġwords Ġin Ġquestions Ġand Ġcontext ĠB IB REF 20 . ĠIs Ġzero - shot Ġlearning Ġfeasible Ġbecause Ġthe Ġmodel Ġsimply Ġlearns Ġthis Ġkind Ġof Ġlanguage Ġindependent Ġstrategies Ġon Ġone Ġlanguage Ġand Ġapply Ġto Ġthe Ġother ? </s> To Ġverify Ġwhether Ġmulti - BER T Ġlargely Ġcounts Ġon Ġa Ġlanguage Ġindependent Ġstrategy , Ġwe Ġtest Ġthe Ġmodel Ġon Ġthe Ġlanguages Ġunseen Ġduring Ġpre - training . ĠTo Ġmake Ġsure Ġthe Ġlanguages Ġhave Ġnever Ġbeen Ġseen Ġbefore , Ġwe Ġartificially Ġmake Ġunseen Ġlanguages Ġby Ġperm uting Ġthe Ġwhole Ġvocabulary Ġof Ġexisting Ġlanguages . ĠThat Ġis , Ġall Ġthe Ġwords Ġin Ġthe Ġsentences Ġof Ġa Ġspecific Ġlanguage Ġare Ġreplaced Ġby Ġother Ġwords Ġin Ġthe Ġsame Ġlanguage Ġto Ġform Ġthe Ġsentences Ġin Ġthe Ġcreated Ġunseen Ġlanguage . ĠIt Ġis Ġassumed Ġthat Ġif Ġmulti - BER T Ġused Ġto Ġfind Ġanswers Ġby Ġlanguage Ġindependent Ġstrategy , Ġthen Ġmulti - BER T Ġshould Ġalso Ġdo Ġwell Ġon Ġunseen Ġlanguages . ĠTable ĠT AB REF 14 Ġshows Ġthat Ġthe Ġperformance Ġof Ġmulti - BER T Ġdrops Ġdrastically Ġon Ġthe Ġdataset . ĠIt Ġimplies Ġthat Ġmulti - BER T Ġmight Ġnot Ġtotally Ġrely Ġon Ġpattern Ġmatching Ġwhen Ġfinding Ġanswers . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠEmb edd ing Ġin ĠMulti - BER T </s> PC A Ġprojection Ġof Ġhidden Ġrepresentations Ġof Ġthe Ġlast Ġlayer Ġof Ġmulti - BER T Ġbefore Ġand Ġafter Ġfine - tun ing Ġare Ġshown Ġin ĠFig . ĠFIG REF 15 . ĠThe Ġred Ġpoints Ġrepresent ĠChinese Ġtokens , Ġand Ġthe Ġblue Ġpoints Ġare Ġfor ĠEnglish . ĠThe Ġresults Ġshow Ġthat Ġtokens Ġfrom Ġdifferent Ġlanguages Ġmight Ġbe Ġembedded Ġinto Ġthe Ġsame Ġspace Ġwith Ġclose Ġspatial Ġdistribution . ĠEven Ġthough Ġduring Ġthe Ġfine - tun ing Ġonly Ġthe ĠEnglish Ġdata Ġis Ġused , Ġthe Ġembed ding Ġof Ġthe ĠChinese Ġtoken Ġchanged Ġaccordingly . ĠWe Ġalso Ġquant itatively Ġevaluate Ġthe Ġsimilarities Ġbetween Ġthe Ġembed ding Ġof Ġthe Ġlanguages . ĠThe Ġresults Ġcan Ġbe Ġfound Ġin Ġthe ĠAppendix . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠCode - sw itching ĠDat as et </s> We Ġobserve Ġlinguistic - agn ostic Ġrepresentations Ġin Ġthe Ġlast Ġsubsection . ĠIf Ġtokens Ġare Ġrepresented Ġin Ġa Ġlanguage - agn ostic Ġway , Ġthe Ġmodel Ġmay Ġbe Ġable Ġto Ġhandle Ġcode - sw itching Ġdata . ĠBecause Ġthere Ġis Ġno Ġcode - sw itching Ġdata Ġfor ĠRC , Ġwe Ġcreate Ġartificial Ġcode - sw itching Ġdatasets Ġby Ġreplacing Ġsome Ġof Ġthe Ġwords Ġin Ġcontexts Ġor Ġquestions Ġwith Ġtheir Ġsyn onyms Ġin Ġanother Ġlanguage . ĠThe Ġsyn onyms Ġare Ġfound Ġby Ġword - by - word Ġtranslation Ġwith Ġgiven Ġdiction aries . ĠWe Ġuse Ġthe Ġbilingual Ġdiction aries Ġcollected Ġand Ġreleased Ġin Ġfacebook research / M USE ĠGitHub Ġrepository . ĠWe Ġsubstitute Ġthe Ġwords Ġif Ġand Ġonly Ġif Ġthe Ġwords Ġare Ġin Ġthe Ġbilingual Ġdiction aries . </s> Table ĠT AB REF 14 Ġshows Ġthat Ġon Ġall Ġthe Ġcode - sw itching Ġdatasets , Ġthe ĠEM / F 1 Ġscore Ġdrops , Ġindicating Ġthat Ġthe Ġsemantics Ġof Ġrepresentations Ġare Ġnot Ġtotally Ġdis ent angled Ġfrom Ġlanguage . ĠHowever , Ġthe Ġexamples Ġof Ġthe Ġanswers Ġof Ġthe Ġmodel Ġ( Table ĠT AB REF 21 ) Ġshow Ġthat Ġmulti - BER T Ġcould Ġfind Ġthe Ġcorrect Ġanswer Ġspans Ġalthough Ġsome Ġkeywords Ġin Ġthe Ġspans Ġhave Ġbeen Ġtranslated Ġinto Ġanother Ġlanguage . </s> What ĠDoes ĠZero - shot ĠTransfer ĠModel ĠLearn ? Ġ: :: ĠTyp ology - man ip ulated ĠDat as et </s> There Ġare Ġvarious Ġtypes Ġof Ġtyp ology Ġin Ġlanguages . ĠFor Ġexample , Ġin ĠEnglish Ġthe Ġtyp ology Ġorder Ġis Ġsubject - verb - object Ġ( S VO ) Ġorder , Ġbut Ġin ĠJapanese Ġand ĠKorean Ġthe Ġorder Ġis Ġsubject - object - verb Ġ( S OV ). ĠWe Ġconstruct Ġa Ġtyp ology - man ip ulated Ġdataset Ġto Ġexamine Ġif Ġthe Ġtyp ology Ġorder Ġof Ġthe Ġtraining Ġdata Ġinfluences Ġthe Ġtransfer Ġlearning Ġresults . ĠIf Ġthe Ġmodel Ġonly Ġlearns Ġthe Ġsemantic Ġmapping Ġbetween Ġdifferent Ġlanguages , Ġchanging ĠEnglish Ġtyp ology Ġorder Ġfrom ĠSV O Ġto ĠS OV Ġshould Ġimprove Ġthe Ġtransfer Ġability Ġfrom ĠEnglish Ġto ĠJapanese . ĠThe Ġmethod Ġused Ġto Ġgenerate Ġdatasets Ġis Ġthe Ġsame Ġas ĠB IB REF 21 . </s> The Ġsource Ġcode Ġis Ġfrom Ġa ĠGitHub Ġrepository Ġnamed ĠSh aul 13 21 / r nn _ typ ology , Ġwhich Ġlabels Ġgiven Ġsentences Ġto ĠCo N LL Ġformat Ġwith ĠStanford Core N LP Ġand Ġthen Ġre - arr anges Ġthem Ġgreed ily . </s> Table ĠT AB REF 23 Ġshows Ġthat Ġwhen Ġwe Ġchange Ġthe ĠEnglish Ġtyp ology Ġorder Ġto ĠS OV Ġor ĠOS V Ġorder , Ġthe Ġperformance Ġon ĠKorean Ġis Ġimproved Ġand Ġworsen Ġon ĠEnglish Ġand ĠChinese , Ġbut Ġvery Ġslightly . ĠThe Ġresults Ġshow Ġthat Ġthe Ġtyp ology Ġmanipulation Ġon Ġthe Ġtraining Ġset Ġhas Ġlittle Ġinfluence . ĠIt Ġis Ġpossible Ġthat Ġmulti - BER T Ġnormal izes Ġthe Ġtyp ology Ġorder Ġof Ġdifferent Ġlanguages Ġto Ġsome Ġextent . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġsystematically Ġexplore Ġzero - shot Ġcross - ling ual Ġtransfer Ġlearning Ġon ĠRC Ġwith Ġmulti - BER T . ĠThe Ġexperimental Ġresults Ġon ĠEnglish , ĠChinese Ġand ĠKorean Ġcorpor a Ġshow Ġthat Ġeven Ġwhen Ġthe Ġlanguages Ġfor Ġtraining Ġand Ġtesting Ġare Ġdifferent , Ġreasonable Ġperformance Ġcan Ġbe Ġobtained . ĠFurthermore , Ġwe Ġcreated Ġseveral Ġartificial Ġdata Ġto Ġstudy Ġthe Ġcross - ling ual Ġability Ġof Ġmulti - BER T Ġin Ġthe Ġpresence Ġof Ġtyp ology Ġvariation Ġand Ġcode - sw itching . ĠWe Ġshowed Ġthat Ġonly Ġtoken - level Ġpattern Ġmatching Ġis Ġnot Ġsufficient Ġfor Ġmulti - BER T Ġto Ġanswer Ġquestions Ġand Ġtyp ology Ġvariation Ġand Ġcode - sw itching Ġonly Ġcaused Ġminor Ġeffects Ġon Ġtesting Ġperformance . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T </s> The Ġarchitecture Ġof Ġmulti - BER T Ġis Ġa ĠTrans former Ġenc oder ĠB IB REF 25 . ĠWhile Ġfine - tun ing Ġon ĠS Qu AD - like Ġdataset , Ġthe Ġbottom Ġlayers Ġof Ġmulti - BER T Ġare Ġinitialized Ġfrom ĠGoogle - pret rained Ġparameters , Ġwith Ġan Ġadded Ġoutput Ġlayer Ġinitialized Ġfrom Ġrandom Ġparameters . ĠTokens Ġrepresentations Ġfrom Ġthe Ġlast Ġlayer Ġof Ġbottom - part Ġof Ġmulti - BER T Ġare Ġinputs Ġto Ġthe Ġoutput Ġlayer Ġand Ġthen Ġthe Ġoutput Ġlayer Ġoutputs Ġa Ġdistribution Ġover Ġall Ġtokens Ġthat Ġindicates Ġthe Ġprobability Ġof Ġa Ġtoken Ġbeing Ġthe ĠSTART / END Ġof Ġan Ġanswer Ġspan . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T Ġ: :: ĠCos ine ĠSimilar ity </s> As Ġall Ġtranslated Ġversions Ġof ĠS Qu AD / D RC D Ġare Ġparallel Ġto Ġeach Ġother . ĠGiven Ġa Ġsource - target Ġlanguage Ġpair , Ġwe Ġcalculate Ġcos ine Ġsimilarity Ġof Ġthe Ġmean Ġpool ing Ġof Ġtokens Ġrepresentation Ġwithin Ġcorresponding Ġanswer - span Ġas Ġa Ġmeasure Ġof Ġhow Ġmuch Ġthey Ġlook Ġlike Ġin Ġterms Ġof Ġthe Ġinternal Ġrepresentation Ġof Ġmulti - BER T . ĠThe Ġresults Ġare Ġshown Ġin ĠFig . ĠFIG REF 26 . </s> Supplement al ĠMaterial Ġ: :: ĠInternal ĠRepresent ation Ġof Ġmulti - BER T Ġ: :: ĠSV CC A </s> Sing ular ĠVector ĠCanon ical ĠCor relation ĠAnalysis Ġ( S V CC A ) Ġis Ġa Ġgeneral Ġmethod Ġto Ġcompare Ġthe Ġcorrelation Ġof Ġtwo Ġsets Ġof Ġvector Ġrepresentations . ĠSV CC A Ġhas Ġbeen Ġproposed Ġto Ġcompare Ġlearned Ġrepresentations Ġacross Ġlanguage Ġmodels ĠB IB REF 24 . ĠHere Ġwe Ġadopt ĠSV CC A Ġto Ġmeasure Ġthe Ġlinear Ġsimilarity Ġof Ġtwo Ġsets Ġof Ġrepresentations Ġin Ġthe Ġsame Ġmulti - BER T Ġfrom Ġdifferent Ġtranslated Ġdatasets , Ġwhich Ġare Ġparallel Ġto Ġeach Ġother . ĠThe Ġresults Ġare Ġshown Ġin ĠFig ĠFIG REF 28 . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering </s> In Ġthe Ġpaper , Ġwe Ġshow Ġthat Ġinternal Ġrepresentations Ġof Ġmulti - BER T Ġare Ġlinear - m app able Ġto Ġsome Ġextent Ġbetween Ġdifferent Ġlanguages . ĠThis Ġimplies Ġthat Ġmulti - BER T Ġmodel Ġmight Ġencode Ġsemantic Ġand Ġsynt actic Ġinformation Ġin Ġlanguage - agn ostic Ġways Ġand Ġexplains Ġhow Ġzero - shot Ġtransfer Ġlearning Ġcould Ġbe Ġdone . </s> To Ġtake Ġa Ġstep Ġfurther , Ġwhile Ġtransf ering Ġmodel Ġfrom Ġsource Ġdataset Ġto Ġtarget Ġdataset , Ġwe Ġalign Ġrepresentations Ġin Ġtwo Ġproposed Ġway , Ġto Ġimprove Ġperformance Ġon Ġtarget Ġdataset . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠLinear ĠM apping ĠMethod </s> Al gorith ms Ġproposed Ġin ĠB IB REF 23 , ĠB IB REF 22 , ĠB IB REF 26 Ġto Ġun super vised ly Ġlearn Ġlinear Ġmapping Ġbetween Ġtwo Ġsets Ġof Ġembed d ings Ġare Ġused Ġhere Ġto Ġalign Ġrepresentations Ġof Ġsource Ġ( training Ġdata ) Ġto Ġthose Ġof Ġtarget . ĠWe Ġobtain Ġthe Ġmapping Ġgenerated Ġby Ġembed d ings Ġfrom Ġone Ġspecific Ġlayer Ġof Ġpre - trained Ġmulti - BER T Ġthen Ġwe Ġapply Ġthis Ġmapping Ġto Ġtransform Ġthe Ġinternal Ġrepresentations Ġof Ġmulti - BER T Ġwhile Ġfine - tun ing Ġon Ġtraining Ġdata . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠAd vers arial ĠMethod </s> In ĠAd vers arial ĠMethod , Ġwe Ġadd Ġan Ġadditional Ġtransform Ġlayer Ġto Ġtransform Ġrepresentations Ġand Ġa Ġdiscrimination Ġlayer Ġto Ġdiscriminate Ġbetween Ġtransformed Ġrepresentations Ġfrom Ġsource Ġlanguage Ġ( training Ġset ) Ġand Ġtarget Ġlanguage Ġ( development Ġset ). ĠAnd Ġthe ĠG AN Ġloss Ġis Ġapplied Ġin Ġthe Ġtotal Ġloss Ġof Ġfine - tun ing . </s> Supplement al ĠMaterial Ġ: :: ĠImprove ĠTrans f ering Ġ: :: ĠDiscussion </s> As Ġtable ĠT AB REF 33 Ġshows , Ġthere Ġare Ġno Ġimprovements Ġamong Ġabove Ġmethods . ĠSome Ġlinear Ġmapping Ġmethods Ġeven Ġcauses Ġdevastating Ġeffect Ġon ĠEM / F 1 Ġscores .