English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish
ABSTRACTIVE
English, Spanish, Finnish
EXTRACTIVE
FLOAT SELECTED: Table 1: XNLI dev results (acc). BT-XX and MT-XX consistently outperform ORIG in all cases. We start by analyzing XNLI development results for Translate-Test. Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data. Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation. As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4.6 points in the best case. Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0.8 points below their BT-ES and BT-FI counterparts. Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English. This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another. FLOAT SELECTED: Table 5: XNLI dev results with class distribution unbiasing (average acc across all languages). Adjusting the bias term of the classifier to match the true class distribution brings large improvements for ORIG, but is less effective for BT-FI and MT-FI.
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI). For sentences occurring multiple times in the training set (e.g. premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method.
2004.04721
Introduction </s> While Ġmost ĠN LP Ġresources Ġare ĠEnglish - specific , Ġthere Ġhave Ġbeen Ġseveral Ġrecent Ġefforts Ġto Ġbuild Ġmult ilingual Ġbenchmarks . ĠOne Ġpossibility Ġis Ġto Ġcollect Ġand Ġannot ate Ġdata Ġin Ġmultiple Ġlanguages Ġseparately ĠB IB REF 0 , Ġbut Ġmost Ġexisting Ġdatasets Ġhave Ġbeen Ġcreated Ġthrough Ġtranslation ĠB IB REF 1 , ĠB IB REF 2 . ĠThis Ġapproach Ġhas Ġtwo Ġdesirable Ġproperties : Ġit Ġrelies Ġon Ġexisting Ġprofessional Ġtranslation Ġservices Ġrather Ġthan Ġrequiring Ġexpertise Ġin Ġmultiple Ġlanguages , Ġand Ġit Ġresults Ġin Ġparallel Ġevaluation Ġsets Ġthat Ġoffer Ġa Ġmeaningful Ġmeasure Ġof Ġthe Ġcross - ling ual Ġtransfer Ġgap Ġof Ġdifferent Ġmodels . ĠThe Ġresulting Ġmult ilingual Ġdatasets Ġare Ġgenerally Ġused Ġfor Ġevaluation Ġonly , Ġrelying Ġon Ġexisting ĠEnglish Ġdatasets Ġfor Ġtraining . </s> Close ly Ġrelated Ġto Ġthat , Ġcross - ling ual Ġtransfer Ġlearning Ġaims Ġto Ġleverage Ġlarge Ġdatasets Ġavailable Ġin Ġone Ġlanguage âĢĶ typically ĠEnglish âĢĶ to Ġbuild Ġmult ilingual Ġmodels Ġthat Ġcan Ġgeneral ize Ġto Ġother Ġlanguages . ĠPrevious Ġwork Ġhas Ġexplored Ġ3 Ġmain Ġapproaches Ġto Ġthat Ġend : Ġmachine Ġtranslating Ġthe Ġtest Ġset Ġinto ĠEnglish Ġand Ġusing Ġa Ġmon oling ual ĠEnglish Ġmodel Ġ( Trans late - Test ), Ġmachine Ġtranslating Ġthe Ġtraining Ġset Ġinto Ġeach Ġtarget Ġlanguage Ġand Ġtraining Ġthe Ġmodels Ġon Ġtheir Ġrespective Ġlanguages Ġ( Trans late - Train ), Ġor Ġusing ĠEnglish Ġdata Ġto Ġfine - t une Ġa Ġmult ilingual Ġmodel Ġthat Ġis Ġthen Ġtransferred Ġto Ġthe Ġrest Ġof Ġlanguages Ġ( Zero - Shot ). </s> The Ġdataset Ġcreation Ġand Ġtransfer Ġprocedures Ġdescribed Ġabove Ġresult Ġin Ġa Ġmixture Ġof Ġoriginal , Ġhuman Ġtranslated Ġand Ġmachine Ġtranslated Ġdata Ġwhen Ġdealing Ġwith Ġcross - ling ual Ġmodels . ĠIn Ġfact , Ġthe Ġtype Ġof Ġtext Ġa Ġsystem Ġis Ġtrained Ġon Ġdoes Ġnot Ġtypically Ġmatch Ġthe Ġtype Ġof Ġtext Ġit Ġis Ġexposed Ġto Ġat Ġtest Ġtime : ĠTrans late - Test Ġsystems Ġare Ġtrained Ġon Ġoriginal Ġdata Ġand Ġevaluated Ġon Ġmachine Ġtranslated Ġtest Ġsets , ĠZero - Shot Ġsystems Ġare Ġtrained Ġon Ġoriginal Ġdata Ġand Ġevaluated Ġon Ġhuman Ġtranslated Ġtest Ġsets , Ġand ĠTrans late - Train Ġsystems Ġare Ġtrained Ġon Ġmachine Ġtranslated Ġdata Ġand Ġevaluated Ġon Ġhuman Ġtranslated Ġtest Ġsets . </s> Despite Ġoverlooked Ġto Ġdate , Ġwe Ġshow Ġthat Ġsuch Ġmismatch Ġhas Ġa Ġnotable Ġimpact Ġin Ġthe Ġperformance Ġof Ġexisting Ġcross - ling ual Ġmodels . ĠBy Ġusing Ġback - translation ĠB IB REF 3 Ġto Ġparaph rase Ġeach Ġtraining Ġinstance , Ġwe Ġobtain Ġanother ĠEnglish Ġversion Ġof Ġthe Ġtraining Ġset Ġthat Ġbetter Ġresembles Ġthe Ġtest Ġset , Ġobtaining Ġsubstantial Ġimprovements Ġfor Ġthe ĠTrans late - Test Ġand ĠZero - Shot Ġapproaches Ġin Ġcross - ling ual ĠNatural ĠLanguage ĠIn ference Ġ( N LI ). ĠWhile Ġimprovements Ġbrought Ġby Ġmachine Ġtranslation Ġhave Ġpreviously Ġbeen Ġattributed Ġto Ġdata Ġaug mentation ĠB IB REF 4 , Ġwe Ġreject Ġthis Ġhypothesis Ġand Ġshow Ġthat Ġthe Ġphenomenon Ġis Ġonly Ġpresent Ġin Ġtranslated Ġtest Ġsets , Ġbut Ġnot Ġin Ġoriginal Ġones . ĠInstead , Ġour Ġanalysis Ġreveals Ġthat Ġthis Ġbehavior Ġis Ġcaused Ġby Ġsubtle Ġartifacts Ġarising Ġfrom Ġthe Ġtranslation Ġprocess Ġitself . ĠIn Ġparticular , Ġwe Ġshow Ġthat Ġtranslating Ġdifferent Ġparts Ġof Ġeach Ġinstance Ġseparately Ġ( e . g . Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġin ĠNL I ) Ġcan Ġalter Ġsuperficial Ġpatterns Ġin Ġthe Ġdata Ġ( e . g . Ġthe Ġdegree Ġof Ġlex ical Ġoverlap Ġbetween Ġthem ), Ġwhich Ġseverely Ġaffects Ġthe Ġgeneral ization Ġability Ġof Ġcurrent Ġmodels . ĠBased Ġon Ġthe Ġgained Ġinsights , Ġwe Ġimprove Ġthe Ġstate - of - the - art Ġin ĠX N LI , Ġand Ġshow Ġthat Ġsome Ġprevious Ġfindings Ġneed Ġto Ġbe Ġrecons idered Ġin Ġthe Ġlight Ġof Ġthis Ġphenomenon . </s> Related Ġwork Ġ: :: ĠCross - ling ual Ġtransfer Ġlearning . </s> Current Ġcross - ling ual Ġmodels Ġwork Ġby Ġpre - training Ġmult ilingual Ġrepresentations Ġusing Ġsome Ġform Ġof Ġlanguage Ġmodeling , Ġwhich Ġare Ġthen Ġfine - tun ed Ġon Ġthe Ġrelevant Ġtask Ġand Ġtransferred Ġto Ġdifferent Ġlanguages . ĠSome Ġauthors Ġleverage Ġparallel Ġdata Ġto Ġthat Ġend ĠB IB REF 5 , ĠB IB REF 6 , Ġbut Ġtraining Ġa Ġmodel Ġakin Ġto ĠB ERT ĠB IB REF 7 Ġon Ġthe Ġcombination Ġof Ġmon oling ual Ġcorpor a Ġin Ġmultiple Ġlanguages Ġis Ġalso Ġeffective ĠB IB REF 8 . ĠClose ly Ġrelated Ġto Ġour Ġwork , ĠB IB REF 4 Ġshowed Ġthat Ġreplacing Ġsegments Ġof Ġthe Ġtraining Ġdata Ġwith Ġtheir Ġtranslation Ġduring Ġfine - tun ing Ġis Ġhelpful . ĠHowever , Ġthey Ġattribute Ġthis Ġbehavior Ġto Ġa Ġdata Ġaug mentation Ġeffect , Ġwhich Ġwe Ġbelieve Ġshould Ġbe Ġrecons idered Ġgiven Ġthe Ġnew Ġevidence Ġwe Ġprovide . </s> Related Ġwork Ġ: :: ĠMult ilingual Ġbenchmarks . </s> Most Ġbenchmarks Ġcovering Ġa Ġwide Ġset Ġof Ġlanguages Ġhave Ġbeen Ġcreated Ġthrough Ġtranslation , Ġas Ġit Ġis Ġthe Ġcase Ġof ĠX N LI ĠB IB REF 1 Ġfor ĠNL I , ĠPA WS - X ĠB IB REF 9 Ġfor Ġadvers arial Ġparaph rase Ġidentification , Ġand ĠX Qu AD ĠB IB REF 2 Ġand ĠML Q A ĠB IB REF 10 Ġfor ĠQuestion ĠAn sw ering Ġ( Q A ). ĠA Ġnotable Ġexception Ġis ĠTy Di ĠQ A ĠB IB REF 0 , Ġa Ġcontempor aneous ĠQ A Ġdataset Ġthat Ġwas Ġseparately Ġannot ated Ġin Ġ11 Ġlanguages . ĠOther Ġcross - ling ual Ġdatasets Ġleverage Ġexisting Ġmult ilingual Ġresources , Ġas Ġit Ġis Ġthe Ġcase Ġof ĠML Doc ĠB IB REF 11 Ġfor Ġdocument Ġclassification Ġand ĠWik iann ĠB IB REF 12 Ġfor Ġnamed Ġentity Ġrecognition . ĠConc urrent Ġto Ġour Ġwork , ĠB IB REF 13 Ġcombine Ġsome Ġof Ġthese Ġdatasets Ġinto Ġa Ġsingle Ġmult ilingual Ġbenchmark , Ġand Ġevaluate Ġsome Ġwell - known Ġmethods Ġon Ġit . </s> Related Ġwork Ġ: :: ĠAn notation Ġartifacts . </s> Several Ġstudies Ġhave Ġshown Ġthat ĠNL I Ġdatasets Ġlike ĠSN LI ĠB IB REF 14 Ġand ĠMulti N LI ĠB IB REF 15 Ġcontain Ġspurious Ġpatterns Ġthat Ġcan Ġbe Ġexploited Ġto Ġobtain Ġstrong Ġresults Ġwithout Ġmaking Ġreal Ġinf erential Ġdecisions . ĠFor Ġinstance , ĠB IB REF 16 Ġand ĠB IB REF 17 Ġshowed Ġthat Ġa Ġhypothesis - only Ġbaseline Ġperforms Ġbetter Ġthan Ġchance Ġdue Ġto Ġcues Ġon Ġtheir Ġlex ical Ġchoice Ġand Ġsentence Ġlength . ĠSimilarly , ĠB IB REF 18 Ġshowed Ġthat ĠNL I Ġmodels Ġtend Ġto Ġpredict Ġentail ment Ġfor Ġsentence Ġpairs Ġwith Ġa Ġhigh Ġlex ical Ġoverlap . ĠSeveral Ġauthors Ġhave Ġworked Ġon Ġadvers arial Ġdatasets Ġto Ġdiagnose Ġthese Ġissues Ġand Ġprovide Ġa Ġmore Ġchallenging Ġbenchmark ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠBesides ĠNL I , Ġother Ġtasks Ġlike ĠQ A Ġhave Ġalso Ġbeen Ġfound Ġto Ġbe Ġsusceptible Ġto Ġannotation Ġartifacts ĠB IB REF 22 , ĠB IB REF 23 . ĠWhile Ġprevious Ġwork Ġhas Ġfocused Ġon Ġthe Ġmon oling ual Ġscenario , Ġwe Ġshow Ġthat Ġtranslation Ġcan Ġinterfere Ġwith Ġthese Ġartifacts Ġin Ġmult ilingual Ġsettings . </s> Related Ġwork Ġ: :: ĠTranslation ese . </s> Trans lated Ġtexts Ġare Ġknown Ġto Ġhave Ġunique Ġfeatures Ġlike Ġsimpl ification , Ġexplicit ation , Ġnormal ization Ġand Ġinterference , Ġwhich Ġare Ġrefer Ġto Ġas Ġtranslation ese ĠB IB REF 24 . ĠThis Ġphenomenon Ġhas Ġbeen Ġreported Ġto Ġhave Ġa Ġnotable Ġimpact Ġin Ġmachine Ġtranslation Ġevaluation ĠB IB REF 25 , ĠB IB REF 26 . ĠFor Ġinstance , Ġback - translation Ġbrings Ġlarge ĠB LE U Ġgains Ġfor Ġreversed Ġtest Ġsets Ġ( i . e . Ġwhen Ġtranslation ese Ġis Ġon Ġthe Ġsource Ġside Ġand Ġoriginal Ġtext Ġis Ġused Ġas Ġreference ), Ġbut Ġits Ġeffect Ġdimin ishes Ġin Ġthe Ġnatural Ġdirection ĠB IB REF 27 . ĠWhile Ġconnected , Ġthe Ġphenomenon Ġwe Ġanalyze Ġis Ġdifferent Ġin Ġthat Ġit Ġarises Ġfrom Ġtranslation Ġinconsistencies Ġdue Ġto Ġthe Ġlack Ġof Ġcontext , Ġand Ġaffects Ġcross - ling ual Ġtransfer Ġlearning Ġrather Ġthan Ġmachine Ġtranslation . </s> Exper imental Ġdesign </s> Our Ġgoal Ġis Ġto Ġanalyze Ġthe Ġeffect Ġof Ġboth Ġhuman Ġand Ġmachine Ġtranslation Ġin Ġcross - ling ual Ġmodels . ĠFor Ġthat Ġpurpose , Ġthe Ġcore Ġidea Ġof Ġour Ġwork Ġis Ġto Ġ( i ) Ġuse Ġmachine Ġtranslation Ġto Ġeither Ġtranslate Ġthe Ġtraining Ġset Ġinto Ġother Ġlanguages , Ġor Ġgenerate ĠEnglish Ġparaph r ases Ġof Ġit Ġthrough Ġback - translation , Ġand Ġ( ii ) Ġevaluate Ġthe Ġresulting Ġsystems Ġon Ġoriginal , Ġhuman Ġtranslated Ġand Ġmachine Ġtranslated Ġtest Ġsets Ġin Ġcomparison Ġwith Ġsystems Ġtrained Ġon Ġoriginal Ġdata . ĠWe Ġnext Ġdescribe Ġthe Ġmodels Ġused Ġin Ġour Ġexperiments Ġ( Â§ SEC REF 6 ), Ġthe Ġspecific Ġtraining Ġvariants Ġexplored Ġ( Â§ SEC REF 8 ), Ġand Ġthe Ġevaluation Ġprocedure Ġfollowed Ġ( Â§ SEC REF 10 ). </s> Exper imental Ġdesign Ġ: :: ĠModels Ġand Ġtransfer Ġmethods </s> We Ġexperiment Ġwith Ġtwo Ġmodels Ġthat Ġare Ġrepresentative Ġof Ġthe Ġstate - of - the - art Ġin Ġmon oling ual Ġand Ġcross - ling ual Ġpre - training : Ġ( i ) ĠRobert a ĠB IB REF 28 , Ġwhich Ġis Ġan Ġimproved Ġversion Ġof ĠB ERT Ġthat Ġuses Ġmasked Ġlanguage Ġmodeling Ġto Ġpre - train Ġan ĠEnglish ĠTrans former Ġmodel , Ġand Ġ( ii ) ĠXL M - R ĠB IB REF 8 , Ġwhich Ġis Ġa Ġmult ilingual Ġextension Ġof Ġthe Ġformer Ġpre - trained Ġon Ġ100 Ġlanguages . ĠIn Ġboth Ġcases , Ġwe Ġuse Ġthe Ġlarge Ġmodels Ġreleased Ġby Ġthe Ġauthors Ġunder Ġthe Ġfair seq Ġrepository . ĠAs Ġdiscussed Ġnext , Ġwe Ġexplore Ġdifferent Ġvariants Ġof Ġthe Ġtraining Ġset Ġto Ġfine - t une Ġeach Ġmodel Ġon Ġdifferent Ġtasks . ĠAt Ġtest Ġtime , Ġwe Ġtry Ġboth Ġmachine Ġtranslating Ġthe Ġtest Ġset Ġinto ĠEnglish Ġ( Trans late - Test ) Ġand , Ġin Ġthe Ġcase Ġof ĠXL M - R , Ġusing Ġthe Ġactual Ġtest Ġset Ġin Ġthe Ġtarget Ġlanguage Ġ( Zero - Shot ). </s> Exper imental Ġdesign Ġ: :: ĠTraining Ġvariants </s> We Ġtry Ġ3 Ġvariants Ġof Ġeach Ġtraining Ġset Ġto Ġfine - t une Ġour Ġmodels : Ġ( i ) Ġthe Ġoriginal Ġone Ġin ĠEnglish Ġ( Orig ), Ġ( ii ) Ġan ĠEnglish Ġparaph rase Ġof Ġit Ġgenerated Ġthrough Ġback - translation Ġusing ĠSpanish Ġor ĠFinnish Ġas Ġpivot Ġ( BT - ES Ġand ĠBT - FI ), Ġand Ġ( iii ) Ġa Ġmachine Ġtranslated Ġversion Ġin ĠSpanish Ġor ĠFinnish Ġ( MT - ES Ġand ĠMT - FI ). ĠFor Ġsentences Ġoccurring Ġmultiple Ġtimes Ġin Ġthe Ġtraining Ġset Ġ( e . g . Ġpremises Ġrepeated Ġfor Ġmultiple Ġhypotheses ), Ġwe Ġuse Ġthe Ġexact Ġsame Ġtranslation Ġfor Ġall Ġoccurrences , Ġas Ġour Ġgoal Ġis Ġto Ġunderstand Ġthe Ġinherent Ġeffect Ġof Ġtranslation Ġrather Ġthan Ġits Ġpotential Ġapplication Ġas Ġa Ġdata Ġaug mentation Ġmethod . </s> In Ġorder Ġto Ġtrain Ġthe Ġmachine Ġtranslation Ġsystems Ġfor ĠMT - XX Ġand ĠBT - XX , Ġwe Ġuse Ġthe Ġbig ĠTrans former Ġmodel ĠB IB REF 29 Ġwith Ġthe Ġsame Ġsettings Ġas ĠB IB REF 30 Ġand ĠSent ence P iece Ġtoken ization ĠB IB REF 31 Ġwith Ġa Ġjoint Ġvocabulary Ġof Ġ32 k Ġsub words . ĠFor ĠEnglish - Spanish , Ġwe Ġtrain Ġfor Ġ10 Ġepoch s Ġon Ġall Ġparallel Ġdata Ġfrom ĠW MT Ġ2013 ĠB IB REF 32 Ġand ĠPar a C rawl Ġv 5 . 0 ĠB IB REF 33 . ĠFor ĠEnglish - F inn ish , Ġwe Ġtrain Ġfor Ġ40 Ġepoch s Ġon ĠEuro par l Ġand ĠWiki ĠTit les Ġfrom ĠW MT Ġ2019 ĠB IB REF 34 , ĠPar a C rawl Ġv 5 . 0 , Ġand ĠD GT , ĠEU book shop Ġand ĠT ilde MOD EL Ġfrom ĠOP US ĠB IB REF 35 . ĠIn Ġboth Ġcases , Ġwe Ġremove Ġsentences Ġlonger Ġthan Ġ250 Ġtokens , Ġwith Ġa Ġsource / target Ġratio Ġexceeding Ġ1 . 5 , Ġor Ġfor Ġwhich Ġlang id . py ĠB IB REF 36 Ġpredicts Ġa Ġdifferent Ġlanguage , Ġresulting Ġin Ġa Ġfinal Ġcorpus Ġsize Ġof Ġ48 M Ġand Ġ7 M Ġsentence Ġpairs , Ġrespectively . ĠWe Ġuse Ġsampling Ġdecoding Ġwith Ġa Ġtemperature Ġof Ġ0 . 5 Ġfor Ġinference , Ġwhich Ġproduces Ġmore Ġdiverse Ġtranslations Ġthan Ġbeam Ġsearch ĠB IB REF 37 Ġand Ġperformed Ġbetter Ġin Ġour Ġpreliminary Ġexperiments . </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure </s> We Ġuse Ġthe Ġfollowing Ġtasks Ġfor Ġour Ġexperiments : </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure Ġ: :: ĠNatural ĠLanguage ĠIn ference Ġ( N LI ). </s> Given Ġa Ġpremise Ġand Ġa Ġhypothesis , Ġthe Ġtask Ġis Ġto Ġdetermine Ġwhether Ġthere Ġis Ġan Ġentail ment , Ġneutral Ġor Ġcontradiction Ġrelation Ġbetween Ġthem . ĠWe Ġfine - t une Ġour Ġmodels Ġon ĠMulti N LI ĠB IB REF 15 Ġfor Ġ10 Ġepoch s Ġusing Ġthe Ġsame Ġsettings Ġas ĠB IB REF 28 . ĠIn Ġmost Ġof Ġour Ġexperiments , Ġwe Ġevaluate Ġon ĠX N LI ĠB IB REF 1 , Ġwhich Ġcomprises Ġ24 90 Ġdevelopment Ġand Ġ5 010 Ġtest Ġinstances Ġin Ġ15 Ġlanguages . ĠThese Ġwere Ġoriginally Ġannot ated Ġin ĠEnglish , Ġand Ġthe Ġresulting Ġpremises Ġand Ġhypotheses Ġwere Ġindependently Ġtranslated Ġinto Ġthe Ġrest Ġof Ġthe Ġlanguages Ġby Ġprofessional Ġtransl ators . ĠFor Ġthe ĠTrans late - Test Ġapproach , Ġwe Ġuse Ġthe Ġmachine Ġtranslated Ġversions Ġfrom Ġthe Ġauthors . ĠFollowing ĠB IB REF 8 , Ġwe Ġselect Ġthe Ġbest Ġepoch Ġcheckpoint Ġaccording Ġto Ġthe Ġaverage Ġaccuracy Ġin Ġthe Ġdevelopment Ġset . </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure Ġ: :: ĠQuestion ĠAn sw ering Ġ( Q A ). </s> Given Ġa Ġcontext Ġparagraph Ġand Ġa Ġquestion , Ġthe Ġtask Ġis Ġto Ġidentify Ġthe Ġspan Ġanswering Ġthe Ġquestion Ġin Ġthe Ġcontext . ĠWe Ġfine - t une Ġour Ġmodels Ġon ĠS Qu AD Ġv 1 . 1 ĠB IB REF 38 Ġfor Ġ2 Ġepoch s Ġusing Ġthe Ġsame Ġsettings Ġas ĠB IB REF 28 , Ġand Ġreport Ġtest Ġresults Ġfor Ġthe Ġlast Ġepoch . ĠWe Ġuse Ġtwo Ġdatasets Ġfor Ġevaluation : ĠX Qu AD ĠB IB REF 2 , Ġa Ġsubset Ġof Ġthe ĠS Qu AD Ġdevelopment Ġset Ġtranslated Ġinto Ġ10 Ġother Ġlanguages , Ġand ĠML Q A ĠB IB REF 10 Ġa Ġdataset Ġconsisting Ġof Ġparallel Ġcontext Ġparagraphs Ġplus Ġthe Ġcorresponding Ġquestions Ġannot ated Ġin ĠEnglish Ġand Ġtranslated Ġinto Ġ6 Ġother Ġlanguages . ĠIn Ġboth Ġcases , Ġthe Ġtranslation Ġwas Ġdone Ġby Ġprofessional Ġtransl ators Ġat Ġthe Ġdocument Ġlevel Ġ( i . e . Ġwhen Ġtranslating Ġa Ġquestion , Ġthe Ġtext Ġanswering Ġit Ġwas Ġalso Ġshown ). ĠFor Ġour ĠBT - XX Ġand ĠMT - XX Ġvariants , Ġwe Ġtranslate Ġthe Ġcontext Ġparagraph Ġand Ġthe Ġquestions Ġindependently , Ġand Ġmap Ġthe Ġanswer Ġspans Ġusing Ġthe Ġsame Ġprocedure Ġas ĠB IB REF 39 . ĠFor Ġthe ĠTrans late - Test Ġapproach , Ġwe Ġuse Ġthe Ġofficial Ġmachine Ġtranslated Ġversions Ġof ĠML Q A , Ġrun Ġinference Ġover Ġthem , Ġand Ġmap Ġthe Ġpredicted Ġanswer Ġspans Ġback Ġto Ġthe Ġtarget Ġlanguage . </s> Both Ġfor ĠNL I Ġand ĠQ A , Ġwe Ġrun Ġeach Ġsystem Ġ5 Ġtimes Ġwith Ġdifferent Ġrandom Ġseeds Ġand Ġreport Ġthe Ġaverage Ġresults . ĠSpace Ġpermitting , Ġwe Ġalso Ġreport Ġthe Ġstandard Ġdeviation Ġacross Ġthe Ġ5 Ġruns . </s> N LI Ġexperiments </s> We Ġnext Ġdiscuss Ġour Ġmain Ġresults Ġin Ġthe ĠX N LI Ġdevelopment Ġset Ġ( Â§ SEC REF 15 , ĠÂ§ SEC REF 16 ), Ġrun Ġadditional Ġexperiments Ġto Ġbetter Ġunderstand Ġthe Ġbehavior Ġof Ġour Ġdifferent Ġvariants Ġ( Â§ SEC REF 17 , ĠÂ§ SEC REF 22 , ĠÂ§ SEC REF 25 ), Ġand Ġcompare Ġour Ġresults Ġto Ġprevious Ġwork Ġin Ġthe ĠX N LI Ġtest Ġset Ġ( Â§ SEC REF 30 ). </s> N LI Ġexperiments Ġ: :: ĠTrans late - Test Ġresults </s> We Ġstart Ġby Ġanalyzing ĠX N LI Ġdevelopment Ġresults Ġfor ĠTrans late - Test . ĠRecall Ġthat , Ġin Ġthis Ġapproach , Ġthe Ġtest Ġset Ġis Ġmachine Ġtranslated Ġinto ĠEnglish , Ġbut Ġtraining Ġis Ġtypically Ġdone Ġon Ġoriginal ĠEnglish Ġdata . ĠOur ĠBT - ES Ġand ĠBT - FI Ġvariants Ġclose Ġthis Ġgap Ġby Ġtraining Ġon Ġa Ġmachine Ġtranslated ĠEnglish Ġversion Ġof Ġthe Ġtraining Ġset Ġgenerated Ġthrough Ġback - translation . ĠAs Ġshown Ġin ĠTable ĠT AB REF 9 , Ġthis Ġbrings Ġsubstantial Ġgains Ġfor Ġboth ĠRobert a Ġand ĠXL M - R , Ġwith Ġan Ġaverage Ġimprovement Ġof Ġ4 . 6 Ġpoints Ġin Ġthe Ġbest Ġcase . ĠQuite Ġremarkably , ĠMT - ES Ġand ĠMT - FI Ġalso Ġoutper form ĠOrig Ġby Ġa Ġsubstantial Ġmargin , Ġand Ġare Ġonly Ġ0 . 8 Ġpoints Ġbelow Ġtheir ĠBT - ES Ġand ĠBT - FI Ġcounterparts . ĠRecall Ġthat , Ġfor Ġthese Ġtwo Ġsystems , Ġtraining Ġis Ġdone Ġin Ġmachine Ġtranslated ĠSpanish Ġor ĠFinnish , Ġwhile Ġinference Ġis Ġdone Ġin Ġmachine Ġtranslated ĠEnglish . ĠThis Ġshows Ġthat Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġoriginal Ġdata Ġto Ġmachine Ġtranslated Ġdata Ġis Ġsubstantially Ġlarger Ġthan Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġone Ġlanguage Ġto Ġanother . </s> N LI Ġexperiments Ġ: :: ĠZero - Shot Ġresults </s> We Ġnext Ġanalyze Ġthe Ġresults Ġfor Ġthe ĠZero - Shot Ġapproach . ĠIn Ġthis Ġcase , Ġinference Ġis Ġdone Ġin Ġthe Ġtest Ġset Ġin Ġeach Ġtarget Ġlanguage Ġwhich , Ġin Ġthe Ġcase Ġof ĠX N LI , Ġwas Ġhuman Ġtranslated Ġfrom ĠEnglish . ĠAs Ġsuch , Ġdifferent Ġfrom Ġthe ĠTrans late - Test Ġapproach , Ġneither Ġtraining Ġon Ġoriginal Ġdata Ġ( Orig ) Ġnor Ġtraining Ġon Ġmachine Ġtranslated Ġdata Ġ( BT - XX Ġand ĠMT - XX ) Ġmakes Ġuse Ġof Ġthe Ġexact Ġsame Ġtype Ġof Ġtext Ġthat Ġthe Ġsystem Ġis Ġexposed Ġto Ġat Ġtest Ġtime . ĠHowever , Ġas Ġshown Ġin ĠTable ĠT AB REF 9 , Ġboth ĠBT - XX Ġand ĠMT - XX Ġoutper form ĠOrig Ġby Ġapproximately Ġ2 Ġpoints , Ġwhich Ġsuggests Ġthat Ġour Ġ( back -) trans lated Ġversions Ġof Ġthe Ġtraining Ġset Ġare Ġmore Ġsimilar Ġto Ġthe Ġhuman Ġtranslated Ġtest Ġsets Ġthan Ġthe Ġoriginal Ġone . ĠThis Ġalso Ġprovides Ġa Ġnew Ġperspective Ġon Ġthe ĠTrans late - Train Ġapproach , Ġwhich Ġwas Ġreported Ġto Ġoutper form ĠOrig Ġin Ġprevious Ġwork ĠB IB REF 5 : Ġwhile Ġthe Ġoriginal Ġmotivation Ġwas Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġsame Ġlanguage Ġthat Ġit Ġis Ġtested Ġon , Ġour Ġresults Ġshow Ġthat Ġmachine Ġtranslating Ġthe Ġtraining Ġset Ġis Ġbeneficial Ġeven Ġwhen Ġthe Ġtarget Ġlanguage Ġis Ġdifferent . </s> N LI Ġexperiments Ġ: :: ĠOriginal Ġvs . Ġtranslated Ġtest Ġsets </s> So Ġas Ġto Ġunderstand Ġwhether Ġthe Ġimprovements Ġobserved Ġso Ġfar Ġare Ġlimited Ġto Ġtranslated Ġtest Ġsets Ġor Ġapply Ġmore Ġgenerally , Ġwe Ġconduct Ġadditional Ġexperiments Ġcomparing Ġtranslated Ġtest Ġsets Ġto Ġoriginal Ġones . ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġall Ġexisting Ġnon - English ĠNL I Ġbenchmarks Ġwere Ġcreated Ġthrough Ġtranslation . ĠFor Ġthat Ġreason , Ġwe Ġbuild Ġa Ġnew Ġtest Ġset Ġthat Ġmim ics ĠX N LI , Ġbut Ġis Ġannot ated Ġin ĠSpanish Ġrather Ġthan ĠEnglish . ĠWe Ġfirst Ġcollect Ġthe Ġpremises Ġfrom Ġa Ġfiltered Ġversion Ġof ĠCommon C rawl ĠB IB REF 42 , Ġtaking Ġa Ġsubset Ġof Ġ5 Ġwebsites Ġthat Ġrepresent Ġa Ġdiverse Ġset Ġof Ġgenres : Ġa Ġnewspaper , Ġan Ġeconomy Ġforum , Ġa Ġcelebrity Ġmagazine , Ġa Ġliterature Ġblog , Ġand Ġa Ġconsumer Ġmagazine . ĠWe Ġthen Ġask Ġnative ĠSpanish Ġannot ators Ġto Ġgenerate Ġan Ġentail ment , Ġa Ġneutral Ġand Ġa Ġcontradiction Ġhypothesis Ġfor Ġeach Ġpremise . ĠWe Ġcollect Ġa Ġtotal Ġof Ġ24 90 Ġexamples Ġusing Ġthis Ġprocedure , Ġwhich Ġis Ġthe Ġsame Ġsize Ġas Ġthe ĠX N LI Ġdevelopment Ġset . ĠFinally , Ġwe Ġcreate Ġa Ġhuman Ġtranslated Ġand Ġa Ġmachine Ġtranslated ĠEnglish Ġversion Ġof Ġthe Ġdataset Ġusing Ġprofessional Ġtransl ators Ġfrom ĠG eng o Ġand Ġour Ġmachine Ġtranslation Ġsystem Ġdescribed Ġin ĠÂ§ SEC REF 8 , Ġrespectively . ĠWe Ġreport Ġresults Ġfor Ġthe Ġbest Ġepoch Ġcheckpoint Ġon Ġeach Ġset . </s> As Ġshown Ġin ĠTable ĠT AB REF 18 , Ġboth ĠBT - XX Ġand ĠMT - XX Ġclearly Ġoutper form ĠOrig Ġin Ġall Ġtest Ġsets Ġcreated Ġthrough Ġtranslation , Ġwhich Ġis Ġconsistent Ġwith Ġour Ġprevious Ġresults . ĠIn Ġcontrast , Ġthe Ġbest Ġresults Ġon Ġthe Ġoriginal ĠEnglish Ġset Ġare Ġobtained Ġby ĠOrig , Ġand Ġneither ĠBT - XX Ġnor ĠMT - XX Ġobtain Ġany Ġclear Ġimprovement Ġon Ġthe Ġone Ġin ĠSpanish Ġeither . ĠThis Ġconfirms Ġthat Ġthe Ġunderlying Ġphenomenon Ġis Ġlimited Ġto Ġtranslated Ġtest Ġsets . ĠIn Ġaddition , Ġit Ġis Ġworth Ġmentioning Ġthat Ġthe Ġresults Ġfor Ġthe Ġmachine Ġtranslated Ġtest Ġset Ġin ĠEnglish Ġare Ġslightly Ġbetter Ġthan Ġthose Ġfor Ġthe Ġhuman Ġtranslated Ġone , Ġwhich Ġsuggests Ġthat Ġthe Ġdifficulty Ġof Ġthe Ġtask Ġdoes Ġnot Ġonly Ġdepend Ġon Ġthe Ġtranslation Ġquality . ĠFinally , Ġit Ġis Ġalso Ġinteresting Ġthat ĠMT - ES Ġis Ġonly Ġmarginally Ġbetter Ġthan ĠMT - FI Ġin Ġboth ĠSpanish Ġtest Ġsets , Ġeven Ġif Ġit Ġcorresponds Ġto Ġthe ĠTrans late - Train Ġapproach , Ġwhereas ĠMT - FI Ġneeds Ġto ĠZero - Shot Ġtransfer Ġfrom ĠFinnish Ġinto ĠSpanish . ĠThis Ġreinforces Ġthe Ġidea Ġthat Ġit Ġis Ġtraining Ġon Ġtranslated Ġdata Ġrather Ġthan Ġtraining Ġon Ġthe Ġtarget Ġlanguage Ġthat Ġis Ġkey Ġin ĠTrans late - Train . </s> N LI Ġexperiments Ġ: :: ĠStress Ġtests </s> In Ġorder Ġto Ġbetter Ġunderstand Ġhow Ġsystems Ġtrained Ġon Ġoriginal Ġand Ġtranslated Ġdata Ġdiffer , Ġwe Ġrun Ġadditional Ġexperiments Ġon Ġthe ĠNL I ĠStress ĠTests ĠB IB REF 19 , Ġwhich Ġwere Ġdesigned Ġto Ġtest Ġthe Ġrobust ness Ġof ĠNL I Ġmodels Ġto Ġspecific Ġlinguistic Ġphenomena Ġin ĠEnglish . ĠThe Ġbenchmark Ġconsists Ġof Ġa Ġcompetence Ġtest , Ġwhich Ġevaluates Ġthe Ġability Ġto Ġunderstand Ġan ton y my Ġrelation Ġand Ġperform Ġnumerical Ġreasoning , Ġa Ġdistraction Ġtest , Ġwhich Ġevaluates Ġthe Ġrobust ness Ġto Ġshallow Ġpatterns Ġlike Ġlex ical Ġoverlap Ġand Ġthe Ġpresence Ġof Ġneg ation Ġwords , Ġand Ġa Ġnoise Ġtest , Ġwhich Ġevaluates Ġrobust ness Ġto Ġspelling Ġerrors . ĠJust Ġas Ġwith Ġprevious Ġexperiments , Ġwe Ġreport Ġresults Ġfor Ġthe Ġbest Ġepoch Ġcheckpoint Ġin Ġeach Ġtest Ġset . </s> As Ġshown Ġin ĠTable ĠT AB REF 23 , ĠOrig Ġoutper forms ĠBT - FI Ġand ĠMT - FI Ġon Ġthe Ġcompetence Ġtest Ġby Ġa Ġlarge Ġmargin , Ġbut Ġthe Ġopposite Ġis Ġtrue Ġon Ġthe Ġdistraction Ġtest . ĠIn Ġparticular , Ġour Ġresults Ġshow Ġthat ĠBT - FI Ġand ĠMT - FI Ġare Ġless Ġreliant Ġon Ġlex ical Ġoverlap Ġand Ġthe Ġpresence Ġof Ġnegative Ġwords . ĠThis Ġfeels Ġintuitive , Ġas Ġtranslating Ġthe Ġpremise Ġand Ġhypothesis Ġindependently âĢĶ as ĠBT - FI Ġand ĠMT - FI Ġdo âĢĶ is Ġlikely Ġto Ġreduce Ġthe Ġlex ical Ġoverlap Ġbetween Ġthem . ĠMore Ġgenerally , Ġthe Ġtranslation Ġprocess Ġcan Ġalter Ġsimilar Ġsuperficial Ġpatterns Ġin Ġthe Ġdata , Ġwhich ĠNL I Ġmodels Ġare Ġsensitive Ġto Ġ( Â§ SEC REF 2 ). ĠThis Ġwould Ġexplain Ġwhy Ġthe Ġresulting Ġmodels Ġhave Ġa Ġdifferent Ġbehavior Ġon Ġdifferent Ġstress Ġtests . </s> N LI Ġexperiments Ġ: :: ĠOutput Ġclass Ġdistribution </s> With Ġthe Ġaim Ġto Ġunderstand Ġthe Ġeffect Ġof Ġthe Ġprevious Ġphenomenon Ġin Ġcross - ling ual Ġsettings , Ġwe Ġlook Ġat Ġthe Ġoutput Ġclass Ġdistribution Ġof Ġour Ġdifferent Ġmodels Ġin Ġthe ĠX N LI Ġdevelopment Ġset . ĠAs Ġshown Ġin ĠTable ĠT AB REF 28 , Ġthe Ġpredictions Ġof Ġall Ġsystems Ġare Ġclose Ġto Ġthe Ġtrue Ġclass Ġdistribution Ġin Ġthe Ġcase Ġof ĠEnglish . ĠNevertheless , ĠOrig Ġis Ġstrongly Ġbiased Ġfor Ġthe Ġrest Ġof Ġlanguages , Ġand Ġtends Ġto Ġunder p redict Ġentail ment Ġand Ġover p redict Ġneutral . ĠThis Ġcan Ġagain Ġbe Ġattributed Ġto Ġthe Ġfact Ġthat Ġthe ĠEnglish Ġtest Ġset Ġis Ġoriginal , Ġwhereas Ġthe Ġrest Ġare Ġhuman Ġtranslated . ĠIn Ġparticular , Ġit Ġis Ġwell - known Ġthat ĠNL I Ġmodels Ġtend Ġto Ġpredict Ġentail ment Ġwhen Ġthere Ġis Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġ( Â§ SEC REF 2 ). ĠHowever , Ġthe Ġdegree Ġof Ġoverlap Ġwill Ġbe Ġsmaller Ġin Ġthe Ġhuman Ġtranslated Ġtest Ġsets Ġgiven Ġthat Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġwere Ġtranslated Ġindependently , Ġwhich Ġexplains Ġwhy Ġentail ment Ġis Ġunder pred icted . ĠIn Ġcontrast , ĠBT - FI Ġand ĠMT - FI Ġare Ġexposed Ġto Ġthe Ġexact Ġsame Ġphenomenon Ġduring Ġtraining , Ġwhich Ġexplains Ġwhy Ġthey Ġare Ġnot Ġthat Ġheavily Ġaffected . </s> So Ġas Ġto Ġmeasure Ġthe Ġimpact Ġof Ġthis Ġphenomenon , Ġwe Ġexplore Ġa Ġsimple Ġapproach Ġto Ġcorrect Ġthis Ġbias : Ġhaving Ġfine - tun ed Ġeach Ġmodel , Ġwe Ġadjust Ġthe Ġbias Ġterm Ġadded Ġto Ġthe Ġlog it Ġof Ġeach Ġclass Ġso Ġthe Ġmodel Ġpredictions Ġmatch Ġthe Ġtrue Ġclass Ġdistribution Ġfor Ġeach Ġlanguage . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 , Ġthis Ġbrings Ġlarge Ġimprovements Ġfor ĠOrig , Ġbut Ġis Ġless Ġeffective Ġfor ĠBT - FI Ġand ĠMT - FI . ĠThis Ġshows Ġthat Ġthe Ġperformance Ġof ĠOrig Ġwas Ġconsiderably Ġhindered Ġby Ġthis Ġbias , Ġwhich ĠBT - FI Ġand ĠMT - FI Ġeffectively Ġmitigate . </s> N LI Ġexperiments Ġ: :: ĠComparison Ġwith Ġthe Ġstate - of - the - art </s> So Ġas Ġto Ġput Ġour Ġresults Ġinto Ġperspective , Ġwe Ġcompare Ġour Ġbest Ġvariant Ġto Ġprevious Ġwork Ġon Ġthe ĠX N LI Ġtest Ġset . ĠAs Ġshown Ġin ĠTable ĠT AB REF 31 , Ġour Ġmethod Ġimproves Ġthe Ġstate - of - the - art Ġfor Ġboth Ġthe ĠTrans late - Test Ġand Ġthe ĠZero - Shot Ġapproaches Ġby Ġ4 . 3 Ġand Ġ2 . 8 Ġpoints , Ġrespectively . ĠIt Ġalso Ġob tains Ġthe Ġbest Ġoverall Ġresults Ġpublished Ġto Ġdate , Ġwith Ġthe Ġadditional Ġadvantage Ġthat Ġthe Ġprevious Ġstate - of - the - art Ġrequired Ġa Ġmachine Ġtranslation Ġsystem Ġbetween ĠEnglish Ġand Ġeach Ġof Ġthe Ġ14 Ġtarget Ġlanguages , Ġwhereas Ġour Ġmethod Ġuses Ġa Ġsingle Ġmachine Ġtranslation Ġsystem Ġbetween ĠEnglish Ġand ĠFinnish Ġ( which Ġis Ġnot Ġone Ġof Ġthe Ġtarget Ġlanguages ). ĠWhile Ġthe Ġmain Ġgoal Ġof Ġour Ġwork Ġis Ġnot Ġto Ġdesign Ġbetter Ġcross - ling ual Ġmodels , Ġbut Ġto Ġanalyze Ġtheir Ġbehavior Ġin Ġconnection Ġto Ġtranslation , Ġthis Ġshows Ġthat Ġthe Ġphenomenon Ġunder Ġstudy Ġis Ġhighly Ġrelevant , Ġto Ġthe Ġextent Ġthat Ġit Ġcan Ġbe Ġexploited Ġto Ġimprove Ġthe Ġstate - of - the - art . </s> Q A Ġexperiments </s> So Ġas Ġto Ġunderstand Ġwhether Ġour Ġprevious Ġfindings Ġapply Ġto Ġother Ġtasks Ġbesides ĠNL I , Ġwe Ġrun Ġadditional Ġexperiments Ġon ĠQ A . ĠAs Ġshown Ġin ĠTable ĠT AB REF 32 , ĠBT - FI Ġand ĠBT - ES Ġdo Ġindeed Ġoutper form ĠOrig Ġfor Ġthe ĠTrans late - Test Ġapproach Ġon ĠML Q A . ĠThe Ġimprovement Ġis Ġmodest , Ġbut Ġvery Ġconsistent Ġacross Ġdifferent Ġlanguages , Ġmodels Ġand Ġruns . ĠThe Ġresults Ġfor ĠMT - ES Ġand ĠMT - FI Ġare Ġless Ġconclusive , Ġpresumably Ġbecause Ġmapping Ġthe Ġanswer Ġspans Ġacross Ġlanguages Ġmight Ġintroduce Ġsome Ġnoise . ĠIn Ġcontrast , Ġwe Ġdo Ġnot Ġobserve Ġany Ġclear Ġimprovement Ġfor Ġthe ĠZero - Shot Ġapproach Ġon Ġthis Ġdataset . ĠOur ĠX Qu AD Ġresults Ġin ĠTable ĠT AB REF 33 Ġare Ġmore Ġpositive , Ġbut Ġstill Ġincon clusive . </s> These Ġresults Ġcan Ġpartly Ġbe Ġexplained Ġby Ġthe Ġtranslation Ġprocedure Ġused Ġto Ġcreate Ġthe Ġdifferent Ġbenchmarks : Ġthe Ġpremises Ġand Ġhypotheses Ġof ĠX N LI Ġwere Ġtranslated Ġindependently , Ġwhereas Ġthe Ġquestions Ġand Ġcontext Ġparagraphs Ġof ĠX Qu AD Ġwere Ġtranslated Ġtogether . ĠSimilarly , ĠML Q A Ġmade Ġuse Ġof Ġparallel Ġcontexts , Ġand Ġtransl ators Ġwere Ġshown Ġthe Ġsentence Ġcontaining Ġeach Ġanswer Ġwhen Ġtranslating Ġthe Ġcorresponding Ġquestion . ĠAs Ġa Ġresult , Ġone Ġcan Ġexpect Ġboth ĠQ A Ġbenchmarks Ġto Ġhave Ġmore Ġconsistent Ġtranslations Ġthan ĠX N LI , Ġwhich Ġwould Ġin Ġturn Ġdiminish Ġthis Ġphenomenon . ĠIn Ġcontrast , Ġthe Ġquestions Ġand Ġcontext Ġparagraphs Ġare Ġindependently Ġtranslated Ġwhen Ġusing Ġmachine Ġtranslation , Ġwhich Ġexplains Ġwhy ĠBT - ES Ġand ĠBT - FI Ġoutper form ĠOrig Ġfor Ġthe ĠTrans late - Test Ġapproach . ĠWe Ġconclude Ġthat Ġthe Ġtranslation Ġartifacts Ġrevealed Ġby Ġour Ġanalysis Ġare Ġnot Ġexclusive Ġto ĠNL I , Ġas Ġthey Ġalso Ġshow Ġup Ġon ĠQ A Ġfor Ġthe ĠTrans late - Test Ġapproach , Ġbut Ġtheir Ġactual Ġimpact Ġcan Ġbe Ġhighly Ġdependent Ġon Ġthe Ġtranslation Ġprocedure Ġused Ġand Ġthe Ġnature Ġof Ġthe Ġtask . </s> Discussion </s> Our Ġanalysis Ġprompts Ġto Ġreconsider Ġprevious Ġfindings Ġin Ġcross - ling ual Ġtransfer Ġlearning Ġas Ġfollows : </s> Discussion Ġ: :: ĠThe Ġcross - ling ual Ġtransfer Ġgap Ġon ĠX N LI Ġwas Ġoverest imated . </s> Given Ġthe Ġparallel Ġnature Ġof ĠX N LI , Ġaccuracy Ġdifferences Ġacross Ġlanguages Ġare Ġcommonly Ġinterpreted Ġas Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom ĠEnglish Ġto Ġthe Ġrest Ġof Ġlanguages . ĠHowever , Ġour Ġwork Ġshows Ġthat Ġthere Ġis Ġanother Ġfactor Ġthat Ġcan Ġhave Ġa Ġmuch Ġlarger Ġimpact : Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġoriginal Ġto Ġtranslated Ġdata . ĠOur Ġresults Ġsuggest Ġthat Ġthe Ġreal Ġcross - ling ual Ġgeneral ization Ġability Ġof ĠXL M - R Ġis Ġconsiderably Ġbetter Ġthan Ġwhat Ġthe Ġaccuracy Ġnumbers Ġin ĠX N LI Ġreflect . </s> Discussion Ġ: :: ĠOver coming Ġthe Ġcross - ling ual Ġgap Ġis Ġnot Ġwhat Ġmakes ĠTrans late - Train Ġwork . </s> The Ġoriginal Ġmotivation Ġfor ĠTrans late - Train Ġwas Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġsame Ġlanguage Ġit Ġis Ġtested Ġon . ĠHowever , Ġwe Ġshow Ġthat Ġit Ġis Ġtraining Ġon Ġtranslated Ġdata , Ġrather Ġthan Ġtraining Ġon Ġthe Ġtarget Ġlanguage , Ġthat Ġis Ġkey Ġfor Ġthis Ġapproach Ġto Ġoutper form ĠZero - Shot Ġas Ġreported Ġby Ġprevious Ġauthors . </s> Discussion Ġ: :: ĠImprovements Ġpreviously Ġattributed Ġto Ġdata Ġaug mentation Ġshould Ġbe Ġrecons idered . </s> The Ġmethod Ġby ĠB IB REF 4 Ġcombines Ġmachine Ġtranslated Ġpremises Ġand Ġhypotheses Ġin Ġdifferent Ġlanguages Ġ( Â§ SEC REF 2 ), Ġresulting Ġin Ġan Ġeffect Ġsimilar Ġto ĠBT - XX Ġand ĠMT - XX . ĠAs Ġsuch , Ġwe Ġbelieve Ġthat Ġthis Ġmethod Ġshould Ġbe Ġanalyzed Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġdataset Ġartifacts Ġrather Ġthan Ġdata Ġaug mentation , Ġas Ġthe Ġauthors Ġdo . ĠFrom Ġthis Ġperspective , Ġhaving Ġthe Ġpremise Ġand Ġthe Ġhypotheses Ġin Ġdifferent Ġlanguages Ġcan Ġreduce Ġthe Ġsuperficial Ġpatterns Ġbetween Ġthem , Ġwhich Ġwould Ġexplain Ġwhy Ġthis Ġapproach Ġis Ġbetter Ġthan Ġusing Ġexamples Ġin Ġa Ġsingle Ġlanguage . </s> Discussion Ġ: :: ĠThe Ġpotential Ġof ĠTrans late - Test Ġwas Ġunderestimated . </s> The Ġprevious Ġbest Ġresults Ġfor ĠTrans late - Test Ġon ĠX N LI Ġl agged Ġbehind Ġthe Ġstate - of - the - art Ġby Ġ4 . 6 Ġpoints . ĠOur Ġwork Ġreduces Ġthis Ġgap Ġto Ġonly Ġ0 . 8 Ġpoints Ġby Ġaddressing Ġthe Ġunderlying Ġtranslation Ġartifacts . ĠThe Ġreason Ġwhy ĠTrans late - Test Ġis Ġmore Ġseverely Ġaffected Ġby Ġthis Ġphenomenon Ġis Ġtwo fold : Ġ( i ) Ġthe Ġeffect Ġis Ġdoubled Ġby Ġfirst Ġusing Ġhuman Ġtranslation Ġto Ġcreate Ġthe Ġtest Ġset Ġand Ġthen Ġmachine Ġtranslation Ġto Ġtranslate Ġit Ġback Ġto ĠEnglish , Ġand Ġ( ii ) ĠTrans late - Train Ġwas Ġinadvertently Ġmitigating Ġthis Ġissue Ġ( see Ġabove ), Ġbut Ġequivalent Ġtechniques Ġwere Ġnever Ġapplied Ġto ĠTrans late - Test . </s> Discussion Ġ: :: ĠFuture Ġevaluation Ġshould Ġbetter Ġaccount Ġfor Ġtranslation Ġartifacts . </s> The Ġevaluation Ġissues Ġraised Ġby Ġour Ġanalysis Ġdo Ġnot Ġhave Ġa Ġsimple Ġsolution . ĠIn Ġfact , Ġwhile Ġwe Ġuse Ġthe Ġterm Ġtranslation Ġartifacts Ġto Ġhighlight Ġthat Ġthey Ġare Ġan Ġunintended Ġeffect Ġof Ġtranslation Ġthat Ġimpacts Ġfinal Ġevaluation , Ġone Ġcould Ġalso Ġargue Ġthat Ġit Ġis Ġthe Ġoriginal Ġdatasets Ġthat Ġcontain Ġthe Ġartifacts , Ġwhich Ġtranslation Ġsimply Ġalters Ġor Ġeven Ġmit ig ates . ĠIn Ġany Ġcase , Ġthis Ġis Ġa Ġmore Ġgeneral Ġissue Ġthat Ġfalls Ġbeyond Ġthe Ġscope Ġof Ġcross - ling ual Ġtransfer Ġlearning , Ġso Ġwe Ġargue Ġthat Ġit Ġshould Ġbe Ġcarefully Ġcontrolled Ġwhen Ġevaluating Ġcross - ling ual Ġmodels . ĠIn Ġthe Ġabsence Ġof Ġmore Ġrobust Ġdatasets , Ġwe Ġrecommend Ġthat Ġfuture Ġmult ilingual Ġbenchmarks Ġshould Ġat Ġleast Ġprovide Ġconsistent Ġtest Ġsets Ġfor ĠEnglish Ġand Ġthe Ġrest Ġof Ġlanguages . ĠThis Ġcan Ġbe Ġachieved Ġby Ġ( i ) Ġusing Ġoriginal Ġannotations Ġin Ġall Ġlanguages , Ġ( ii ) Ġusing Ġoriginal Ġannotations Ġin Ġa Ġnon - English Ġlanguage Ġand Ġtranslating Ġthem Ġinto ĠEnglish Ġand Ġother Ġlanguages , Ġor Ġ( iii ) Ġif Ġtranslating Ġfrom ĠEnglish , Ġdoing Ġso Ġat Ġthe Ġdocument Ġlevel Ġto Ġminimize Ġtranslation Ġinconsistencies . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġhave Ġshown Ġthat Ġboth Ġhuman Ġand Ġmachine Ġtranslation Ġcan Ġalter Ġsuperficial Ġpatterns Ġin Ġdata , Ġwhich Ġrequires Ġreconsider ing Ġprevious Ġfindings Ġin Ġcross - ling ual Ġtransfer Ġlearning . ĠBased Ġon Ġthe Ġgained Ġinsights , Ġwe Ġhave Ġimproved Ġthe Ġstate - of - the - art Ġin ĠX N LI Ġfor Ġthe ĠTrans late - Test Ġand ĠZero - Shot Ġapproaches Ġby Ġa Ġsubstantial Ġmargin . ĠFinally , Ġwe Ġhave Ġshown Ġthat Ġthe Ġphenomenon Ġis Ġnot Ġspecific Ġto ĠNL I Ġbut Ġalso Ġaffects ĠQ A , Ġalthough Ġit Ġis Ġless Ġpronounced Ġthere Ġthanks Ġto Ġthe Ġtranslation Ġprocedure Ġused Ġin Ġthe Ġcorresponding Ġbenchmarks . ĠSo Ġas Ġto Ġfacilitate Ġsimilar Ġstudies Ġin Ġthe Ġfuture , Ġwe Ġrelease Ġour ĠNL I Ġdataset , Ġwhich , Ġunlike Ġprevious Ġbenchmarks , Ġwas Ġannot ated Ġin Ġa Ġnon - English Ġlanguage Ġand Ġhuman Ġtranslated Ġinto ĠEnglish . </s> Acknowled gments </s> We Ġthank ĠNora ĠAr an ber ri Ġand ĠU x oa ĠI Ã± ur ri eta Ġfor Ġhelpful Ġdiscussion Ġduring Ġthe Ġdevelopment Ġof Ġthis Ġwork , Ġas Ġwell Ġas Ġthe Ġrest Ġof Ġour Ġcolleagues Ġfrom Ġthe ĠIX A Ġgroup Ġthat Ġworked Ġas Ġannot ators Ġfor Ġour ĠNL I Ġdataset . </s> This Ġresearch Ġwas Ġpartially Ġfunded Ġby Ġa ĠFacebook ĠFellowship , Ġthe ĠBas que ĠGovernment Ġexcellence Ġresearch Ġgroup Ġ( IT 13 43 - 19 ), Ġthe ĠSpanish ĠMIN EC O Ġ( Un sup MT ĠT IN 2017 âĢĲ 916 92 âĢĲ EXP ĠMC IU / AE I / F ED ER , ĠUE ), ĠProject ĠBig Know ledge Ġ( Ay ud as ĠFund aci Ã³n ĠBB VA Ġa Ġequip os Ġde Ġinvestig aci Ã³n Ġc ient ÃŃ f ica Ġ2018 ), Ġand Ġthe ĠNVIDIA ĠGPU Ġgrant Ġprogram .
Introduction While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks. One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2. This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models. The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training. Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages. Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot). The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models. In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets. Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models. By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI). While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones. Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself. In particular, we show that translating different parts of each instance separately (e.g. the premise and the hypothesis in NLI) can alter superficial patterns in the data (e.g. the degree of lexical overlap between them), which severely affects the generalization ability of current models. Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon. Related work ::: Cross-lingual transfer learning. Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages. Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8. Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful. However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide. Related work ::: Multilingual benchmarks. Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA). A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages. Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition. Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it. Related work ::: Annotation artifacts. Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions. For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length. Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap. Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21. Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23. While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings. Related work ::: Translationese. Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24. This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26. For instance, back-translation brings large BLEU gains for reversed test sets (i.e. when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27. While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation. Experimental design Our goal is to analyze the effect of both human and machine translation in cross-lingual models. For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data. We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10). Experimental design ::: Models and transfer methods We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages. In both cases, we use the large models released by the authors under the fairseq repository. As discussed next, we explore different variants of the training set to fine-tune each model on different tasks. At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot). Experimental design ::: Training variants We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI). For sentences occurring multiple times in the training set (e.g. premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method. In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords. For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5.0 BIBREF33. For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5.0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35. In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1.5, or for which langid.py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively. We use sampling decoding with a temperature of 0.5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments. Experimental design ::: Tasks and evaluation procedure We use the following tasks for our experiments: Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI). Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them. We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28. In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages. These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators. For the Translate-Test approach, we use the machine translated versions from the authors. Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set. Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA). Given a context paragraph and a question, the task is to identify the span answering the question in the context. We fine-tune our models on SQuAD v1.1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch. We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages. In both cases, the translation was done by professional translators at the document level (i.e. when translating a question, the text answering it was also shown). For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39. For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language. Both for NLI and QA, we run each system 5 times with different random seeds and report the average results. Space permitting, we also report the standard deviation across the 5 runs. NLI experiments We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30). NLI experiments ::: Translate-Test results We start by analyzing XNLI development results for Translate-Test. Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data. Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation. As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4.6 points in the best case. Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0.8 points below their BT-ES and BT-FI counterparts. Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English. This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another. NLI experiments ::: Zero-Shot results We next analyze the results for the Zero-Shot approach. In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English. As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time. However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one. This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different. NLI experiments ::: Original vs. translated test sets So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones. However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation. For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English. We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine. We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise. We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set. Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively. We report results for the best epoch checkpoint on each set. As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results. In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either. This confirms that the underlying phenomenon is limited to translated test sets. In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality. Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish. This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train. NLI experiments ::: Stress tests In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English. The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors. Just as with previous experiments, we report results for the best epoch checkpoint in each test set. As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test. In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words. This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them. More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2). This would explain why the resulting models have a different behavior on different stress tests. NLI experiments ::: Output class distribution With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set. As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English. Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral. This can again be attributed to the fact that the English test set is original, whereas the rest are human translated. In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2). However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted. In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected. So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language. As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI. This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate. NLI experiments ::: Comparison with the state-of-the-art So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set. As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4.3 and 2.8 points, respectively. It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages). While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art. QA experiments So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA. As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA. The improvement is modest, but very consistent across different languages, models and runs. The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise. In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset. Our XQuAD results in Table TABREF33 are more positive, but still inconclusive. These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together. Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question. As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon. In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach. We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task. Discussion Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows: Discussion ::: The cross-lingual transfer gap on XNLI was overestimated. Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages. However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data. Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect. Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work. The original motivation for Translate-Train was to train the model on the same language it is tested on. However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors. Discussion ::: Improvements previously attributed to data augmentation should be reconsidered. The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX. As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do. From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language. Discussion ::: The potential of Translate-Test was underestimated. The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4.6 points. Our work reduces this gap to only 0.8 points by addressing the underlying translation artifacts. The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test. Discussion ::: Future evaluation should better account for translation artifacts. The evaluation issues raised by our analysis do not have a simple solution. In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates. In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models. In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages. This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies. Conclusions In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning. Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin. Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks. So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English. Acknowledgments We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset. This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program.
What are the languages they use in their experiment?
5bc1dc6ebcb88fd0310b21d2a74939e35a4c1a11
What Ġare Ġthe Ġlanguages Ġthey Ġuse Ġin Ġtheir Ġexperiment ?
English Ċ French Ċ Spanish Ċ German Ċ Greek Ċ Bul g arian Ċ Russian Ċ Turkish Ċ Arab ic Ċ V iet namese Ċ Th ai Ċ Chinese Ċ H ind i Ċ Sw ah ili Ċ Ur du Ċ F inn ish
<s> What Ġare Ġthe Ġlanguages Ġthey Ġuse Ġin Ġtheir Ġexperiment ? </s> Introduction </s> While Ġmost ĠN LP Ġresources Ġare ĠEnglish - specific , Ġthere Ġhave Ġbeen Ġseveral Ġrecent Ġefforts Ġto Ġbuild Ġmult ilingual Ġbenchmarks . ĠOne Ġpossibility Ġis Ġto Ġcollect Ġand Ġannot ate Ġdata Ġin Ġmultiple Ġlanguages Ġseparately ĠB IB REF 0 , Ġbut Ġmost Ġexisting Ġdatasets Ġhave Ġbeen Ġcreated Ġthrough Ġtranslation ĠB IB REF 1 , ĠB IB REF 2 . ĠThis Ġapproach Ġhas Ġtwo Ġdesirable Ġproperties : Ġit Ġrelies Ġon Ġexisting Ġprofessional Ġtranslation Ġservices Ġrather Ġthan Ġrequiring Ġexpertise Ġin Ġmultiple Ġlanguages , Ġand Ġit Ġresults Ġin Ġparallel Ġevaluation Ġsets Ġthat Ġoffer Ġa Ġmeaningful Ġmeasure Ġof Ġthe Ġcross - ling ual Ġtransfer Ġgap Ġof Ġdifferent Ġmodels . ĠThe Ġresulting Ġmult ilingual Ġdatasets Ġare Ġgenerally Ġused Ġfor Ġevaluation Ġonly , Ġrelying Ġon Ġexisting ĠEnglish Ġdatasets Ġfor Ġtraining . </s> Close ly Ġrelated Ġto Ġthat , Ġcross - ling ual Ġtransfer Ġlearning Ġaims Ġto Ġleverage Ġlarge Ġdatasets Ġavailable Ġin Ġone Ġlanguage âĢĶ typically ĠEnglish âĢĶ to Ġbuild Ġmult ilingual Ġmodels Ġthat Ġcan Ġgeneral ize Ġto Ġother Ġlanguages . ĠPrevious Ġwork Ġhas Ġexplored Ġ3 Ġmain Ġapproaches Ġto Ġthat Ġend : Ġmachine Ġtranslating Ġthe Ġtest Ġset Ġinto ĠEnglish Ġand Ġusing Ġa Ġmon oling ual ĠEnglish Ġmodel Ġ( Trans late - Test ), Ġmachine Ġtranslating Ġthe Ġtraining Ġset Ġinto Ġeach Ġtarget Ġlanguage Ġand Ġtraining Ġthe Ġmodels Ġon Ġtheir Ġrespective Ġlanguages Ġ( Trans late - Train ), Ġor Ġusing ĠEnglish Ġdata Ġto Ġfine - t une Ġa Ġmult ilingual Ġmodel Ġthat Ġis Ġthen Ġtransferred Ġto Ġthe Ġrest Ġof Ġlanguages Ġ( Zero - Shot ). </s> The Ġdataset Ġcreation Ġand Ġtransfer Ġprocedures Ġdescribed Ġabove Ġresult Ġin Ġa Ġmixture Ġof Ġoriginal , Ġhuman Ġtranslated Ġand Ġmachine Ġtranslated Ġdata Ġwhen Ġdealing Ġwith Ġcross - ling ual Ġmodels . ĠIn Ġfact , Ġthe Ġtype Ġof Ġtext Ġa Ġsystem Ġis Ġtrained Ġon Ġdoes Ġnot Ġtypically Ġmatch Ġthe Ġtype Ġof Ġtext Ġit Ġis Ġexposed Ġto Ġat Ġtest Ġtime : ĠTrans late - Test Ġsystems Ġare Ġtrained Ġon Ġoriginal Ġdata Ġand Ġevaluated Ġon Ġmachine Ġtranslated Ġtest Ġsets , ĠZero - Shot Ġsystems Ġare Ġtrained Ġon Ġoriginal Ġdata Ġand Ġevaluated Ġon Ġhuman Ġtranslated Ġtest Ġsets , Ġand ĠTrans late - Train Ġsystems Ġare Ġtrained Ġon Ġmachine Ġtranslated Ġdata Ġand Ġevaluated Ġon Ġhuman Ġtranslated Ġtest Ġsets . </s> Despite Ġoverlooked Ġto Ġdate , Ġwe Ġshow Ġthat Ġsuch Ġmismatch Ġhas Ġa Ġnotable Ġimpact Ġin Ġthe Ġperformance Ġof Ġexisting Ġcross - ling ual Ġmodels . ĠBy Ġusing Ġback - translation ĠB IB REF 3 Ġto Ġparaph rase Ġeach Ġtraining Ġinstance , Ġwe Ġobtain Ġanother ĠEnglish Ġversion Ġof Ġthe Ġtraining Ġset Ġthat Ġbetter Ġresembles Ġthe Ġtest Ġset , Ġobtaining Ġsubstantial Ġimprovements Ġfor Ġthe ĠTrans late - Test Ġand ĠZero - Shot Ġapproaches Ġin Ġcross - ling ual ĠNatural ĠLanguage ĠIn ference Ġ( N LI ). ĠWhile Ġimprovements Ġbrought Ġby Ġmachine Ġtranslation Ġhave Ġpreviously Ġbeen Ġattributed Ġto Ġdata Ġaug mentation ĠB IB REF 4 , Ġwe Ġreject Ġthis Ġhypothesis Ġand Ġshow Ġthat Ġthe Ġphenomenon Ġis Ġonly Ġpresent Ġin Ġtranslated Ġtest Ġsets , Ġbut Ġnot Ġin Ġoriginal Ġones . ĠInstead , Ġour Ġanalysis Ġreveals Ġthat Ġthis Ġbehavior Ġis Ġcaused Ġby Ġsubtle Ġartifacts Ġarising Ġfrom Ġthe Ġtranslation Ġprocess Ġitself . ĠIn Ġparticular , Ġwe Ġshow Ġthat Ġtranslating Ġdifferent Ġparts Ġof Ġeach Ġinstance Ġseparately Ġ( e . g . Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġin ĠNL I ) Ġcan Ġalter Ġsuperficial Ġpatterns Ġin Ġthe Ġdata Ġ( e . g . Ġthe Ġdegree Ġof Ġlex ical Ġoverlap Ġbetween Ġthem ), Ġwhich Ġseverely Ġaffects Ġthe Ġgeneral ization Ġability Ġof Ġcurrent Ġmodels . ĠBased Ġon Ġthe Ġgained Ġinsights , Ġwe Ġimprove Ġthe Ġstate - of - the - art Ġin ĠX N LI , Ġand Ġshow Ġthat Ġsome Ġprevious Ġfindings Ġneed Ġto Ġbe Ġrecons idered Ġin Ġthe Ġlight Ġof Ġthis Ġphenomenon . </s> Related Ġwork Ġ: :: ĠCross - ling ual Ġtransfer Ġlearning . </s> Current Ġcross - ling ual Ġmodels Ġwork Ġby Ġpre - training Ġmult ilingual Ġrepresentations Ġusing Ġsome Ġform Ġof Ġlanguage Ġmodeling , Ġwhich Ġare Ġthen Ġfine - tun ed Ġon Ġthe Ġrelevant Ġtask Ġand Ġtransferred Ġto Ġdifferent Ġlanguages . ĠSome Ġauthors Ġleverage Ġparallel Ġdata Ġto Ġthat Ġend ĠB IB REF 5 , ĠB IB REF 6 , Ġbut Ġtraining Ġa Ġmodel Ġakin Ġto ĠB ERT ĠB IB REF 7 Ġon Ġthe Ġcombination Ġof Ġmon oling ual Ġcorpor a Ġin Ġmultiple Ġlanguages Ġis Ġalso Ġeffective ĠB IB REF 8 . ĠClose ly Ġrelated Ġto Ġour Ġwork , ĠB IB REF 4 Ġshowed Ġthat Ġreplacing Ġsegments Ġof Ġthe Ġtraining Ġdata Ġwith Ġtheir Ġtranslation Ġduring Ġfine - tun ing Ġis Ġhelpful . ĠHowever , Ġthey Ġattribute Ġthis Ġbehavior Ġto Ġa Ġdata Ġaug mentation Ġeffect , Ġwhich Ġwe Ġbelieve Ġshould Ġbe Ġrecons idered Ġgiven Ġthe Ġnew Ġevidence Ġwe Ġprovide . </s> Related Ġwork Ġ: :: ĠMult ilingual Ġbenchmarks . </s> Most Ġbenchmarks Ġcovering Ġa Ġwide Ġset Ġof Ġlanguages Ġhave Ġbeen Ġcreated Ġthrough Ġtranslation , Ġas Ġit Ġis Ġthe Ġcase Ġof ĠX N LI ĠB IB REF 1 Ġfor ĠNL I , ĠPA WS - X ĠB IB REF 9 Ġfor Ġadvers arial Ġparaph rase Ġidentification , Ġand ĠX Qu AD ĠB IB REF 2 Ġand ĠML Q A ĠB IB REF 10 Ġfor ĠQuestion ĠAn sw ering Ġ( Q A ). ĠA Ġnotable Ġexception Ġis ĠTy Di ĠQ A ĠB IB REF 0 , Ġa Ġcontempor aneous ĠQ A Ġdataset Ġthat Ġwas Ġseparately Ġannot ated Ġin Ġ11 Ġlanguages . ĠOther Ġcross - ling ual Ġdatasets Ġleverage Ġexisting Ġmult ilingual Ġresources , Ġas Ġit Ġis Ġthe Ġcase Ġof ĠML Doc ĠB IB REF 11 Ġfor Ġdocument Ġclassification Ġand ĠWik iann ĠB IB REF 12 Ġfor Ġnamed Ġentity Ġrecognition . ĠConc urrent Ġto Ġour Ġwork , ĠB IB REF 13 Ġcombine Ġsome Ġof Ġthese Ġdatasets Ġinto Ġa Ġsingle Ġmult ilingual Ġbenchmark , Ġand Ġevaluate Ġsome Ġwell - known Ġmethods Ġon Ġit . </s> Related Ġwork Ġ: :: ĠAn notation Ġartifacts . </s> Several Ġstudies Ġhave Ġshown Ġthat ĠNL I Ġdatasets Ġlike ĠSN LI ĠB IB REF 14 Ġand ĠMulti N LI ĠB IB REF 15 Ġcontain Ġspurious Ġpatterns Ġthat Ġcan Ġbe Ġexploited Ġto Ġobtain Ġstrong Ġresults Ġwithout Ġmaking Ġreal Ġinf erential Ġdecisions . ĠFor Ġinstance , ĠB IB REF 16 Ġand ĠB IB REF 17 Ġshowed Ġthat Ġa Ġhypothesis - only Ġbaseline Ġperforms Ġbetter Ġthan Ġchance Ġdue Ġto Ġcues Ġon Ġtheir Ġlex ical Ġchoice Ġand Ġsentence Ġlength . ĠSimilarly , ĠB IB REF 18 Ġshowed Ġthat ĠNL I Ġmodels Ġtend Ġto Ġpredict Ġentail ment Ġfor Ġsentence Ġpairs Ġwith Ġa Ġhigh Ġlex ical Ġoverlap . ĠSeveral Ġauthors Ġhave Ġworked Ġon Ġadvers arial Ġdatasets Ġto Ġdiagnose Ġthese Ġissues Ġand Ġprovide Ġa Ġmore Ġchallenging Ġbenchmark ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . ĠBesides ĠNL I , Ġother Ġtasks Ġlike ĠQ A Ġhave Ġalso Ġbeen Ġfound Ġto Ġbe Ġsusceptible Ġto Ġannotation Ġartifacts ĠB IB REF 22 , ĠB IB REF 23 . ĠWhile Ġprevious Ġwork Ġhas Ġfocused Ġon Ġthe Ġmon oling ual Ġscenario , Ġwe Ġshow Ġthat Ġtranslation Ġcan Ġinterfere Ġwith Ġthese Ġartifacts Ġin Ġmult ilingual Ġsettings . </s> Related Ġwork Ġ: :: ĠTranslation ese . </s> Trans lated Ġtexts Ġare Ġknown Ġto Ġhave Ġunique Ġfeatures Ġlike Ġsimpl ification , Ġexplicit ation , Ġnormal ization Ġand Ġinterference , Ġwhich Ġare Ġrefer Ġto Ġas Ġtranslation ese ĠB IB REF 24 . ĠThis Ġphenomenon Ġhas Ġbeen Ġreported Ġto Ġhave Ġa Ġnotable Ġimpact Ġin Ġmachine Ġtranslation Ġevaluation ĠB IB REF 25 , ĠB IB REF 26 . ĠFor Ġinstance , Ġback - translation Ġbrings Ġlarge ĠB LE U Ġgains Ġfor Ġreversed Ġtest Ġsets Ġ( i . e . Ġwhen Ġtranslation ese Ġis Ġon Ġthe Ġsource Ġside Ġand Ġoriginal Ġtext Ġis Ġused Ġas Ġreference ), Ġbut Ġits Ġeffect Ġdimin ishes Ġin Ġthe Ġnatural Ġdirection ĠB IB REF 27 . ĠWhile Ġconnected , Ġthe Ġphenomenon Ġwe Ġanalyze Ġis Ġdifferent Ġin Ġthat Ġit Ġarises Ġfrom Ġtranslation Ġinconsistencies Ġdue Ġto Ġthe Ġlack Ġof Ġcontext , Ġand Ġaffects Ġcross - ling ual Ġtransfer Ġlearning Ġrather Ġthan Ġmachine Ġtranslation . </s> Exper imental Ġdesign </s> Our Ġgoal Ġis Ġto Ġanalyze Ġthe Ġeffect Ġof Ġboth Ġhuman Ġand Ġmachine Ġtranslation Ġin Ġcross - ling ual Ġmodels . ĠFor Ġthat Ġpurpose , Ġthe Ġcore Ġidea Ġof Ġour Ġwork Ġis Ġto Ġ( i ) Ġuse Ġmachine Ġtranslation Ġto Ġeither Ġtranslate Ġthe Ġtraining Ġset Ġinto Ġother Ġlanguages , Ġor Ġgenerate ĠEnglish Ġparaph r ases Ġof Ġit Ġthrough Ġback - translation , Ġand Ġ( ii ) Ġevaluate Ġthe Ġresulting Ġsystems Ġon Ġoriginal , Ġhuman Ġtranslated Ġand Ġmachine Ġtranslated Ġtest Ġsets Ġin Ġcomparison Ġwith Ġsystems Ġtrained Ġon Ġoriginal Ġdata . ĠWe Ġnext Ġdescribe Ġthe Ġmodels Ġused Ġin Ġour Ġexperiments Ġ( Â§ SEC REF 6 ), Ġthe Ġspecific Ġtraining Ġvariants Ġexplored Ġ( Â§ SEC REF 8 ), Ġand Ġthe Ġevaluation Ġprocedure Ġfollowed Ġ( Â§ SEC REF 10 ). </s> Exper imental Ġdesign Ġ: :: ĠModels Ġand Ġtransfer Ġmethods </s> We Ġexperiment Ġwith Ġtwo Ġmodels Ġthat Ġare Ġrepresentative Ġof Ġthe Ġstate - of - the - art Ġin Ġmon oling ual Ġand Ġcross - ling ual Ġpre - training : Ġ( i ) ĠRobert a ĠB IB REF 28 , Ġwhich Ġis Ġan Ġimproved Ġversion Ġof ĠB ERT Ġthat Ġuses Ġmasked Ġlanguage Ġmodeling Ġto Ġpre - train Ġan ĠEnglish ĠTrans former Ġmodel , Ġand Ġ( ii ) ĠXL M - R ĠB IB REF 8 , Ġwhich Ġis Ġa Ġmult ilingual Ġextension Ġof Ġthe Ġformer Ġpre - trained Ġon Ġ100 Ġlanguages . ĠIn Ġboth Ġcases , Ġwe Ġuse Ġthe Ġlarge Ġmodels Ġreleased Ġby Ġthe Ġauthors Ġunder Ġthe Ġfair seq Ġrepository . ĠAs Ġdiscussed Ġnext , Ġwe Ġexplore Ġdifferent Ġvariants Ġof Ġthe Ġtraining Ġset Ġto Ġfine - t une Ġeach Ġmodel Ġon Ġdifferent Ġtasks . ĠAt Ġtest Ġtime , Ġwe Ġtry Ġboth Ġmachine Ġtranslating Ġthe Ġtest Ġset Ġinto ĠEnglish Ġ( Trans late - Test ) Ġand , Ġin Ġthe Ġcase Ġof ĠXL M - R , Ġusing Ġthe Ġactual Ġtest Ġset Ġin Ġthe Ġtarget Ġlanguage Ġ( Zero - Shot ). </s> Exper imental Ġdesign Ġ: :: ĠTraining Ġvariants </s> We Ġtry Ġ3 Ġvariants Ġof Ġeach Ġtraining Ġset Ġto Ġfine - t une Ġour Ġmodels : Ġ( i ) Ġthe Ġoriginal Ġone Ġin ĠEnglish Ġ( Orig ), Ġ( ii ) Ġan ĠEnglish Ġparaph rase Ġof Ġit Ġgenerated Ġthrough Ġback - translation Ġusing ĠSpanish Ġor ĠFinnish Ġas Ġpivot Ġ( BT - ES Ġand ĠBT - FI ), Ġand Ġ( iii ) Ġa Ġmachine Ġtranslated Ġversion Ġin ĠSpanish Ġor ĠFinnish Ġ( MT - ES Ġand ĠMT - FI ). ĠFor Ġsentences Ġoccurring Ġmultiple Ġtimes Ġin Ġthe Ġtraining Ġset Ġ( e . g . Ġpremises Ġrepeated Ġfor Ġmultiple Ġhypotheses ), Ġwe Ġuse Ġthe Ġexact Ġsame Ġtranslation Ġfor Ġall Ġoccurrences , Ġas Ġour Ġgoal Ġis Ġto Ġunderstand Ġthe Ġinherent Ġeffect Ġof Ġtranslation Ġrather Ġthan Ġits Ġpotential Ġapplication Ġas Ġa Ġdata Ġaug mentation Ġmethod . </s> In Ġorder Ġto Ġtrain Ġthe Ġmachine Ġtranslation Ġsystems Ġfor ĠMT - XX Ġand ĠBT - XX , Ġwe Ġuse Ġthe Ġbig ĠTrans former Ġmodel ĠB IB REF 29 Ġwith Ġthe Ġsame Ġsettings Ġas ĠB IB REF 30 Ġand ĠSent ence P iece Ġtoken ization ĠB IB REF 31 Ġwith Ġa Ġjoint Ġvocabulary Ġof Ġ32 k Ġsub words . ĠFor ĠEnglish - Spanish , Ġwe Ġtrain Ġfor Ġ10 Ġepoch s Ġon Ġall Ġparallel Ġdata Ġfrom ĠW MT Ġ2013 ĠB IB REF 32 Ġand ĠPar a C rawl Ġv 5 . 0 ĠB IB REF 33 . ĠFor ĠEnglish - F inn ish , Ġwe Ġtrain Ġfor Ġ40 Ġepoch s Ġon ĠEuro par l Ġand ĠWiki ĠTit les Ġfrom ĠW MT Ġ2019 ĠB IB REF 34 , ĠPar a C rawl Ġv 5 . 0 , Ġand ĠD GT , ĠEU book shop Ġand ĠT ilde MOD EL Ġfrom ĠOP US ĠB IB REF 35 . ĠIn Ġboth Ġcases , Ġwe Ġremove Ġsentences Ġlonger Ġthan Ġ250 Ġtokens , Ġwith Ġa Ġsource / target Ġratio Ġexceeding Ġ1 . 5 , Ġor Ġfor Ġwhich Ġlang id . py ĠB IB REF 36 Ġpredicts Ġa Ġdifferent Ġlanguage , Ġresulting Ġin Ġa Ġfinal Ġcorpus Ġsize Ġof Ġ48 M Ġand Ġ7 M Ġsentence Ġpairs , Ġrespectively . ĠWe Ġuse Ġsampling Ġdecoding Ġwith Ġa Ġtemperature Ġof Ġ0 . 5 Ġfor Ġinference , Ġwhich Ġproduces Ġmore Ġdiverse Ġtranslations Ġthan Ġbeam Ġsearch ĠB IB REF 37 Ġand Ġperformed Ġbetter Ġin Ġour Ġpreliminary Ġexperiments . </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure </s> We Ġuse Ġthe Ġfollowing Ġtasks Ġfor Ġour Ġexperiments : </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure Ġ: :: ĠNatural ĠLanguage ĠIn ference Ġ( N LI ). </s> Given Ġa Ġpremise Ġand Ġa Ġhypothesis , Ġthe Ġtask Ġis Ġto Ġdetermine Ġwhether Ġthere Ġis Ġan Ġentail ment , Ġneutral Ġor Ġcontradiction Ġrelation Ġbetween Ġthem . ĠWe Ġfine - t une Ġour Ġmodels Ġon ĠMulti N LI ĠB IB REF 15 Ġfor Ġ10 Ġepoch s Ġusing Ġthe Ġsame Ġsettings Ġas ĠB IB REF 28 . ĠIn Ġmost Ġof Ġour Ġexperiments , Ġwe Ġevaluate Ġon ĠX N LI ĠB IB REF 1 , Ġwhich Ġcomprises Ġ24 90 Ġdevelopment Ġand Ġ5 010 Ġtest Ġinstances Ġin Ġ15 Ġlanguages . ĠThese Ġwere Ġoriginally Ġannot ated Ġin ĠEnglish , Ġand Ġthe Ġresulting Ġpremises Ġand Ġhypotheses Ġwere Ġindependently Ġtranslated Ġinto Ġthe Ġrest Ġof Ġthe Ġlanguages Ġby Ġprofessional Ġtransl ators . ĠFor Ġthe ĠTrans late - Test Ġapproach , Ġwe Ġuse Ġthe Ġmachine Ġtranslated Ġversions Ġfrom Ġthe Ġauthors . ĠFollowing ĠB IB REF 8 , Ġwe Ġselect Ġthe Ġbest Ġepoch Ġcheckpoint Ġaccording Ġto Ġthe Ġaverage Ġaccuracy Ġin Ġthe Ġdevelopment Ġset . </s> Exper imental Ġdesign Ġ: :: ĠT asks Ġand Ġevaluation Ġprocedure Ġ: :: ĠQuestion ĠAn sw ering Ġ( Q A ). </s> Given Ġa Ġcontext Ġparagraph Ġand Ġa Ġquestion , Ġthe Ġtask Ġis Ġto Ġidentify Ġthe Ġspan Ġanswering Ġthe Ġquestion Ġin Ġthe Ġcontext . ĠWe Ġfine - t une Ġour Ġmodels Ġon ĠS Qu AD Ġv 1 . 1 ĠB IB REF 38 Ġfor Ġ2 Ġepoch s Ġusing Ġthe Ġsame Ġsettings Ġas ĠB IB REF 28 , Ġand Ġreport Ġtest Ġresults Ġfor Ġthe Ġlast Ġepoch . ĠWe Ġuse Ġtwo Ġdatasets Ġfor Ġevaluation : ĠX Qu AD ĠB IB REF 2 , Ġa Ġsubset Ġof Ġthe ĠS Qu AD Ġdevelopment Ġset Ġtranslated Ġinto Ġ10 Ġother Ġlanguages , Ġand ĠML Q A ĠB IB REF 10 Ġa Ġdataset Ġconsisting Ġof Ġparallel Ġcontext Ġparagraphs Ġplus Ġthe Ġcorresponding Ġquestions Ġannot ated Ġin ĠEnglish Ġand Ġtranslated Ġinto Ġ6 Ġother Ġlanguages . ĠIn Ġboth Ġcases , Ġthe Ġtranslation Ġwas Ġdone Ġby Ġprofessional Ġtransl ators Ġat Ġthe Ġdocument Ġlevel Ġ( i . e . Ġwhen Ġtranslating Ġa Ġquestion , Ġthe Ġtext Ġanswering Ġit Ġwas Ġalso Ġshown ). ĠFor Ġour ĠBT - XX Ġand ĠMT - XX Ġvariants , Ġwe Ġtranslate Ġthe Ġcontext Ġparagraph Ġand Ġthe Ġquestions Ġindependently , Ġand Ġmap Ġthe Ġanswer Ġspans Ġusing Ġthe Ġsame Ġprocedure Ġas ĠB IB REF 39 . ĠFor Ġthe ĠTrans late - Test Ġapproach , Ġwe Ġuse Ġthe Ġofficial Ġmachine Ġtranslated Ġversions Ġof ĠML Q A , Ġrun Ġinference Ġover Ġthem , Ġand Ġmap Ġthe Ġpredicted Ġanswer Ġspans Ġback Ġto Ġthe Ġtarget Ġlanguage . </s> Both Ġfor ĠNL I Ġand ĠQ A , Ġwe Ġrun Ġeach Ġsystem Ġ5 Ġtimes Ġwith Ġdifferent Ġrandom Ġseeds Ġand Ġreport Ġthe Ġaverage Ġresults . ĠSpace Ġpermitting , Ġwe Ġalso Ġreport Ġthe Ġstandard Ġdeviation Ġacross Ġthe Ġ5 Ġruns . </s> N LI Ġexperiments </s> We Ġnext Ġdiscuss Ġour Ġmain Ġresults Ġin Ġthe ĠX N LI Ġdevelopment Ġset Ġ( Â§ SEC REF 15 , ĠÂ§ SEC REF 16 ), Ġrun Ġadditional Ġexperiments Ġto Ġbetter Ġunderstand Ġthe Ġbehavior Ġof Ġour Ġdifferent Ġvariants Ġ( Â§ SEC REF 17 , ĠÂ§ SEC REF 22 , ĠÂ§ SEC REF 25 ), Ġand Ġcompare Ġour Ġresults Ġto Ġprevious Ġwork Ġin Ġthe ĠX N LI Ġtest Ġset Ġ( Â§ SEC REF 30 ). </s> N LI Ġexperiments Ġ: :: ĠTrans late - Test Ġresults </s> We Ġstart Ġby Ġanalyzing ĠX N LI Ġdevelopment Ġresults Ġfor ĠTrans late - Test . ĠRecall Ġthat , Ġin Ġthis Ġapproach , Ġthe Ġtest Ġset Ġis Ġmachine Ġtranslated Ġinto ĠEnglish , Ġbut Ġtraining Ġis Ġtypically Ġdone Ġon Ġoriginal ĠEnglish Ġdata . ĠOur ĠBT - ES Ġand ĠBT - FI Ġvariants Ġclose Ġthis Ġgap Ġby Ġtraining Ġon Ġa Ġmachine Ġtranslated ĠEnglish Ġversion Ġof Ġthe Ġtraining Ġset Ġgenerated Ġthrough Ġback - translation . ĠAs Ġshown Ġin ĠTable ĠT AB REF 9 , Ġthis Ġbrings Ġsubstantial Ġgains Ġfor Ġboth ĠRobert a Ġand ĠXL M - R , Ġwith Ġan Ġaverage Ġimprovement Ġof Ġ4 . 6 Ġpoints Ġin Ġthe Ġbest Ġcase . ĠQuite Ġremarkably , ĠMT - ES Ġand ĠMT - FI Ġalso Ġoutper form ĠOrig Ġby Ġa Ġsubstantial Ġmargin , Ġand Ġare Ġonly Ġ0 . 8 Ġpoints Ġbelow Ġtheir ĠBT - ES Ġand ĠBT - FI Ġcounterparts . ĠRecall Ġthat , Ġfor Ġthese Ġtwo Ġsystems , Ġtraining Ġis Ġdone Ġin Ġmachine Ġtranslated ĠSpanish Ġor ĠFinnish , Ġwhile Ġinference Ġis Ġdone Ġin Ġmachine Ġtranslated ĠEnglish . ĠThis Ġshows Ġthat Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġoriginal Ġdata Ġto Ġmachine Ġtranslated Ġdata Ġis Ġsubstantially Ġlarger Ġthan Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġone Ġlanguage Ġto Ġanother . </s> N LI Ġexperiments Ġ: :: ĠZero - Shot Ġresults </s> We Ġnext Ġanalyze Ġthe Ġresults Ġfor Ġthe ĠZero - Shot Ġapproach . ĠIn Ġthis Ġcase , Ġinference Ġis Ġdone Ġin Ġthe Ġtest Ġset Ġin Ġeach Ġtarget Ġlanguage Ġwhich , Ġin Ġthe Ġcase Ġof ĠX N LI , Ġwas Ġhuman Ġtranslated Ġfrom ĠEnglish . ĠAs Ġsuch , Ġdifferent Ġfrom Ġthe ĠTrans late - Test Ġapproach , Ġneither Ġtraining Ġon Ġoriginal Ġdata Ġ( Orig ) Ġnor Ġtraining Ġon Ġmachine Ġtranslated Ġdata Ġ( BT - XX Ġand ĠMT - XX ) Ġmakes Ġuse Ġof Ġthe Ġexact Ġsame Ġtype Ġof Ġtext Ġthat Ġthe Ġsystem Ġis Ġexposed Ġto Ġat Ġtest Ġtime . ĠHowever , Ġas Ġshown Ġin ĠTable ĠT AB REF 9 , Ġboth ĠBT - XX Ġand ĠMT - XX Ġoutper form ĠOrig Ġby Ġapproximately Ġ2 Ġpoints , Ġwhich Ġsuggests Ġthat Ġour Ġ( back -) trans lated Ġversions Ġof Ġthe Ġtraining Ġset Ġare Ġmore Ġsimilar Ġto Ġthe Ġhuman Ġtranslated Ġtest Ġsets Ġthan Ġthe Ġoriginal Ġone . ĠThis Ġalso Ġprovides Ġa Ġnew Ġperspective Ġon Ġthe ĠTrans late - Train Ġapproach , Ġwhich Ġwas Ġreported Ġto Ġoutper form ĠOrig Ġin Ġprevious Ġwork ĠB IB REF 5 : Ġwhile Ġthe Ġoriginal Ġmotivation Ġwas Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġsame Ġlanguage Ġthat Ġit Ġis Ġtested Ġon , Ġour Ġresults Ġshow Ġthat Ġmachine Ġtranslating Ġthe Ġtraining Ġset Ġis Ġbeneficial Ġeven Ġwhen Ġthe Ġtarget Ġlanguage Ġis Ġdifferent . </s> N LI Ġexperiments Ġ: :: ĠOriginal Ġvs . Ġtranslated Ġtest Ġsets </s> So Ġas Ġto Ġunderstand Ġwhether Ġthe Ġimprovements Ġobserved Ġso Ġfar Ġare Ġlimited Ġto Ġtranslated Ġtest Ġsets Ġor Ġapply Ġmore Ġgenerally , Ġwe Ġconduct Ġadditional Ġexperiments Ġcomparing Ġtranslated Ġtest Ġsets Ġto Ġoriginal Ġones . ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġall Ġexisting Ġnon - English ĠNL I Ġbenchmarks Ġwere Ġcreated Ġthrough Ġtranslation . ĠFor Ġthat Ġreason , Ġwe Ġbuild Ġa Ġnew Ġtest Ġset Ġthat Ġmim ics ĠX N LI , Ġbut Ġis Ġannot ated Ġin ĠSpanish Ġrather Ġthan ĠEnglish . ĠWe Ġfirst Ġcollect Ġthe Ġpremises Ġfrom Ġa Ġfiltered Ġversion Ġof ĠCommon C rawl ĠB IB REF 42 , Ġtaking Ġa Ġsubset Ġof Ġ5 Ġwebsites Ġthat Ġrepresent Ġa Ġdiverse Ġset Ġof Ġgenres : Ġa Ġnewspaper , Ġan Ġeconomy Ġforum , Ġa Ġcelebrity Ġmagazine , Ġa Ġliterature Ġblog , Ġand Ġa Ġconsumer Ġmagazine . ĠWe Ġthen Ġask Ġnative ĠSpanish Ġannot ators Ġto Ġgenerate Ġan Ġentail ment , Ġa Ġneutral Ġand Ġa Ġcontradiction Ġhypothesis Ġfor Ġeach Ġpremise . ĠWe Ġcollect Ġa Ġtotal Ġof Ġ24 90 Ġexamples Ġusing Ġthis Ġprocedure , Ġwhich Ġis Ġthe Ġsame Ġsize Ġas Ġthe ĠX N LI Ġdevelopment Ġset . ĠFinally , Ġwe Ġcreate Ġa Ġhuman Ġtranslated Ġand Ġa Ġmachine Ġtranslated ĠEnglish Ġversion Ġof Ġthe Ġdataset Ġusing Ġprofessional Ġtransl ators Ġfrom ĠG eng o Ġand Ġour Ġmachine Ġtranslation Ġsystem Ġdescribed Ġin ĠÂ§ SEC REF 8 , Ġrespectively . ĠWe Ġreport Ġresults Ġfor Ġthe Ġbest Ġepoch Ġcheckpoint Ġon Ġeach Ġset . </s> As Ġshown Ġin ĠTable ĠT AB REF 18 , Ġboth ĠBT - XX Ġand ĠMT - XX Ġclearly Ġoutper form ĠOrig Ġin Ġall Ġtest Ġsets Ġcreated Ġthrough Ġtranslation , Ġwhich Ġis Ġconsistent Ġwith Ġour Ġprevious Ġresults . ĠIn Ġcontrast , Ġthe Ġbest Ġresults Ġon Ġthe Ġoriginal ĠEnglish Ġset Ġare Ġobtained Ġby ĠOrig , Ġand Ġneither ĠBT - XX Ġnor ĠMT - XX Ġobtain Ġany Ġclear Ġimprovement Ġon Ġthe Ġone Ġin ĠSpanish Ġeither . ĠThis Ġconfirms Ġthat Ġthe Ġunderlying Ġphenomenon Ġis Ġlimited Ġto Ġtranslated Ġtest Ġsets . ĠIn Ġaddition , Ġit Ġis Ġworth Ġmentioning Ġthat Ġthe Ġresults Ġfor Ġthe Ġmachine Ġtranslated Ġtest Ġset Ġin ĠEnglish Ġare Ġslightly Ġbetter Ġthan Ġthose Ġfor Ġthe Ġhuman Ġtranslated Ġone , Ġwhich Ġsuggests Ġthat Ġthe Ġdifficulty Ġof Ġthe Ġtask Ġdoes Ġnot Ġonly Ġdepend Ġon Ġthe Ġtranslation Ġquality . ĠFinally , Ġit Ġis Ġalso Ġinteresting Ġthat ĠMT - ES Ġis Ġonly Ġmarginally Ġbetter Ġthan ĠMT - FI Ġin Ġboth ĠSpanish Ġtest Ġsets , Ġeven Ġif Ġit Ġcorresponds Ġto Ġthe ĠTrans late - Train Ġapproach , Ġwhereas ĠMT - FI Ġneeds Ġto ĠZero - Shot Ġtransfer Ġfrom ĠFinnish Ġinto ĠSpanish . ĠThis Ġreinforces Ġthe Ġidea Ġthat Ġit Ġis Ġtraining Ġon Ġtranslated Ġdata Ġrather Ġthan Ġtraining Ġon Ġthe Ġtarget Ġlanguage Ġthat Ġis Ġkey Ġin ĠTrans late - Train . </s> N LI Ġexperiments Ġ: :: ĠStress Ġtests </s> In Ġorder Ġto Ġbetter Ġunderstand Ġhow Ġsystems Ġtrained Ġon Ġoriginal Ġand Ġtranslated Ġdata Ġdiffer , Ġwe Ġrun Ġadditional Ġexperiments Ġon Ġthe ĠNL I ĠStress ĠTests ĠB IB REF 19 , Ġwhich Ġwere Ġdesigned Ġto Ġtest Ġthe Ġrobust ness Ġof ĠNL I Ġmodels Ġto Ġspecific Ġlinguistic Ġphenomena Ġin ĠEnglish . ĠThe Ġbenchmark Ġconsists Ġof Ġa Ġcompetence Ġtest , Ġwhich Ġevaluates Ġthe Ġability Ġto Ġunderstand Ġan ton y my Ġrelation Ġand Ġperform Ġnumerical Ġreasoning , Ġa Ġdistraction Ġtest , Ġwhich Ġevaluates Ġthe Ġrobust ness Ġto Ġshallow Ġpatterns Ġlike Ġlex ical Ġoverlap Ġand Ġthe Ġpresence Ġof Ġneg ation Ġwords , Ġand Ġa Ġnoise Ġtest , Ġwhich Ġevaluates Ġrobust ness Ġto Ġspelling Ġerrors . ĠJust Ġas Ġwith Ġprevious Ġexperiments , Ġwe Ġreport Ġresults Ġfor Ġthe Ġbest Ġepoch Ġcheckpoint Ġin Ġeach Ġtest Ġset . </s> As Ġshown Ġin ĠTable ĠT AB REF 23 , ĠOrig Ġoutper forms ĠBT - FI Ġand ĠMT - FI Ġon Ġthe Ġcompetence Ġtest Ġby Ġa Ġlarge Ġmargin , Ġbut Ġthe Ġopposite Ġis Ġtrue Ġon Ġthe Ġdistraction Ġtest . ĠIn Ġparticular , Ġour Ġresults Ġshow Ġthat ĠBT - FI Ġand ĠMT - FI Ġare Ġless Ġreliant Ġon Ġlex ical Ġoverlap Ġand Ġthe Ġpresence Ġof Ġnegative Ġwords . ĠThis Ġfeels Ġintuitive , Ġas Ġtranslating Ġthe Ġpremise Ġand Ġhypothesis Ġindependently âĢĶ as ĠBT - FI Ġand ĠMT - FI Ġdo âĢĶ is Ġlikely Ġto Ġreduce Ġthe Ġlex ical Ġoverlap Ġbetween Ġthem . ĠMore Ġgenerally , Ġthe Ġtranslation Ġprocess Ġcan Ġalter Ġsimilar Ġsuperficial Ġpatterns Ġin Ġthe Ġdata , Ġwhich ĠNL I Ġmodels Ġare Ġsensitive Ġto Ġ( Â§ SEC REF 2 ). ĠThis Ġwould Ġexplain Ġwhy Ġthe Ġresulting Ġmodels Ġhave Ġa Ġdifferent Ġbehavior Ġon Ġdifferent Ġstress Ġtests . </s> N LI Ġexperiments Ġ: :: ĠOutput Ġclass Ġdistribution </s> With Ġthe Ġaim Ġto Ġunderstand Ġthe Ġeffect Ġof Ġthe Ġprevious Ġphenomenon Ġin Ġcross - ling ual Ġsettings , Ġwe Ġlook Ġat Ġthe Ġoutput Ġclass Ġdistribution Ġof Ġour Ġdifferent Ġmodels Ġin Ġthe ĠX N LI Ġdevelopment Ġset . ĠAs Ġshown Ġin ĠTable ĠT AB REF 28 , Ġthe Ġpredictions Ġof Ġall Ġsystems Ġare Ġclose Ġto Ġthe Ġtrue Ġclass Ġdistribution Ġin Ġthe Ġcase Ġof ĠEnglish . ĠNevertheless , ĠOrig Ġis Ġstrongly Ġbiased Ġfor Ġthe Ġrest Ġof Ġlanguages , Ġand Ġtends Ġto Ġunder p redict Ġentail ment Ġand Ġover p redict Ġneutral . ĠThis Ġcan Ġagain Ġbe Ġattributed Ġto Ġthe Ġfact Ġthat Ġthe ĠEnglish Ġtest Ġset Ġis Ġoriginal , Ġwhereas Ġthe Ġrest Ġare Ġhuman Ġtranslated . ĠIn Ġparticular , Ġit Ġis Ġwell - known Ġthat ĠNL I Ġmodels Ġtend Ġto Ġpredict Ġentail ment Ġwhen Ġthere Ġis Ġa Ġhigh Ġlex ical Ġoverlap Ġbetween Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġ( Â§ SEC REF 2 ). ĠHowever , Ġthe Ġdegree Ġof Ġoverlap Ġwill Ġbe Ġsmaller Ġin Ġthe Ġhuman Ġtranslated Ġtest Ġsets Ġgiven Ġthat Ġthe Ġpremise Ġand Ġthe Ġhypothesis Ġwere Ġtranslated Ġindependently , Ġwhich Ġexplains Ġwhy Ġentail ment Ġis Ġunder pred icted . ĠIn Ġcontrast , ĠBT - FI Ġand ĠMT - FI Ġare Ġexposed Ġto Ġthe Ġexact Ġsame Ġphenomenon Ġduring Ġtraining , Ġwhich Ġexplains Ġwhy Ġthey Ġare Ġnot Ġthat Ġheavily Ġaffected . </s> So Ġas Ġto Ġmeasure Ġthe Ġimpact Ġof Ġthis Ġphenomenon , Ġwe Ġexplore Ġa Ġsimple Ġapproach Ġto Ġcorrect Ġthis Ġbias : Ġhaving Ġfine - tun ed Ġeach Ġmodel , Ġwe Ġadjust Ġthe Ġbias Ġterm Ġadded Ġto Ġthe Ġlog it Ġof Ġeach Ġclass Ġso Ġthe Ġmodel Ġpredictions Ġmatch Ġthe Ġtrue Ġclass Ġdistribution Ġfor Ġeach Ġlanguage . ĠAs Ġshown Ġin ĠTable ĠT AB REF 29 , Ġthis Ġbrings Ġlarge Ġimprovements Ġfor ĠOrig , Ġbut Ġis Ġless Ġeffective Ġfor ĠBT - FI Ġand ĠMT - FI . ĠThis Ġshows Ġthat Ġthe Ġperformance Ġof ĠOrig Ġwas Ġconsiderably Ġhindered Ġby Ġthis Ġbias , Ġwhich ĠBT - FI Ġand ĠMT - FI Ġeffectively Ġmitigate . </s> N LI Ġexperiments Ġ: :: ĠComparison Ġwith Ġthe Ġstate - of - the - art </s> So Ġas Ġto Ġput Ġour Ġresults Ġinto Ġperspective , Ġwe Ġcompare Ġour Ġbest Ġvariant Ġto Ġprevious Ġwork Ġon Ġthe ĠX N LI Ġtest Ġset . ĠAs Ġshown Ġin ĠTable ĠT AB REF 31 , Ġour Ġmethod Ġimproves Ġthe Ġstate - of - the - art Ġfor Ġboth Ġthe ĠTrans late - Test Ġand Ġthe ĠZero - Shot Ġapproaches Ġby Ġ4 . 3 Ġand Ġ2 . 8 Ġpoints , Ġrespectively . ĠIt Ġalso Ġob tains Ġthe Ġbest Ġoverall Ġresults Ġpublished Ġto Ġdate , Ġwith Ġthe Ġadditional Ġadvantage Ġthat Ġthe Ġprevious Ġstate - of - the - art Ġrequired Ġa Ġmachine Ġtranslation Ġsystem Ġbetween ĠEnglish Ġand Ġeach Ġof Ġthe Ġ14 Ġtarget Ġlanguages , Ġwhereas Ġour Ġmethod Ġuses Ġa Ġsingle Ġmachine Ġtranslation Ġsystem Ġbetween ĠEnglish Ġand ĠFinnish Ġ( which Ġis Ġnot Ġone Ġof Ġthe Ġtarget Ġlanguages ). ĠWhile Ġthe Ġmain Ġgoal Ġof Ġour Ġwork Ġis Ġnot Ġto Ġdesign Ġbetter Ġcross - ling ual Ġmodels , Ġbut Ġto Ġanalyze Ġtheir Ġbehavior Ġin Ġconnection Ġto Ġtranslation , Ġthis Ġshows Ġthat Ġthe Ġphenomenon Ġunder Ġstudy Ġis Ġhighly Ġrelevant , Ġto Ġthe Ġextent Ġthat Ġit Ġcan Ġbe Ġexploited Ġto Ġimprove Ġthe Ġstate - of - the - art . </s> Q A Ġexperiments </s> So Ġas Ġto Ġunderstand Ġwhether Ġour Ġprevious Ġfindings Ġapply Ġto Ġother Ġtasks Ġbesides ĠNL I , Ġwe Ġrun Ġadditional Ġexperiments Ġon ĠQ A . ĠAs Ġshown Ġin ĠTable ĠT AB REF 32 , ĠBT - FI Ġand ĠBT - ES Ġdo Ġindeed Ġoutper form ĠOrig Ġfor Ġthe ĠTrans late - Test Ġapproach Ġon ĠML Q A . ĠThe Ġimprovement Ġis Ġmodest , Ġbut Ġvery Ġconsistent Ġacross Ġdifferent Ġlanguages , Ġmodels Ġand Ġruns . ĠThe Ġresults Ġfor ĠMT - ES Ġand ĠMT - FI Ġare Ġless Ġconclusive , Ġpresumably Ġbecause Ġmapping Ġthe Ġanswer Ġspans Ġacross Ġlanguages Ġmight Ġintroduce Ġsome Ġnoise . ĠIn Ġcontrast , Ġwe Ġdo Ġnot Ġobserve Ġany Ġclear Ġimprovement Ġfor Ġthe ĠZero - Shot Ġapproach Ġon Ġthis Ġdataset . ĠOur ĠX Qu AD Ġresults Ġin ĠTable ĠT AB REF 33 Ġare Ġmore Ġpositive , Ġbut Ġstill Ġincon clusive . </s> These Ġresults Ġcan Ġpartly Ġbe Ġexplained Ġby Ġthe Ġtranslation Ġprocedure Ġused Ġto Ġcreate Ġthe Ġdifferent Ġbenchmarks : Ġthe Ġpremises Ġand Ġhypotheses Ġof ĠX N LI Ġwere Ġtranslated Ġindependently , Ġwhereas Ġthe Ġquestions Ġand Ġcontext Ġparagraphs Ġof ĠX Qu AD Ġwere Ġtranslated Ġtogether . ĠSimilarly , ĠML Q A Ġmade Ġuse Ġof Ġparallel Ġcontexts , Ġand Ġtransl ators Ġwere Ġshown Ġthe Ġsentence Ġcontaining Ġeach Ġanswer Ġwhen Ġtranslating Ġthe Ġcorresponding Ġquestion . ĠAs Ġa Ġresult , Ġone Ġcan Ġexpect Ġboth ĠQ A Ġbenchmarks Ġto Ġhave Ġmore Ġconsistent Ġtranslations Ġthan ĠX N LI , Ġwhich Ġwould Ġin Ġturn Ġdiminish Ġthis Ġphenomenon . ĠIn Ġcontrast , Ġthe Ġquestions Ġand Ġcontext Ġparagraphs Ġare Ġindependently Ġtranslated Ġwhen Ġusing Ġmachine Ġtranslation , Ġwhich Ġexplains Ġwhy ĠBT - ES Ġand ĠBT - FI Ġoutper form ĠOrig Ġfor Ġthe ĠTrans late - Test Ġapproach . ĠWe Ġconclude Ġthat Ġthe Ġtranslation Ġartifacts Ġrevealed Ġby Ġour Ġanalysis Ġare Ġnot Ġexclusive Ġto ĠNL I , Ġas Ġthey Ġalso Ġshow Ġup Ġon ĠQ A Ġfor Ġthe ĠTrans late - Test Ġapproach , Ġbut Ġtheir Ġactual Ġimpact Ġcan Ġbe Ġhighly Ġdependent Ġon Ġthe Ġtranslation Ġprocedure Ġused Ġand Ġthe Ġnature Ġof Ġthe Ġtask . </s> Discussion </s> Our Ġanalysis Ġprompts Ġto Ġreconsider Ġprevious Ġfindings Ġin Ġcross - ling ual Ġtransfer Ġlearning Ġas Ġfollows : </s> Discussion Ġ: :: ĠThe Ġcross - ling ual Ġtransfer Ġgap Ġon ĠX N LI Ġwas Ġoverest imated . </s> Given Ġthe Ġparallel Ġnature Ġof ĠX N LI , Ġaccuracy Ġdifferences Ġacross Ġlanguages Ġare Ġcommonly Ġinterpreted Ġas Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom ĠEnglish Ġto Ġthe Ġrest Ġof Ġlanguages . ĠHowever , Ġour Ġwork Ġshows Ġthat Ġthere Ġis Ġanother Ġfactor Ġthat Ġcan Ġhave Ġa Ġmuch Ġlarger Ġimpact : Ġthe Ġloss Ġof Ġperformance Ġwhen Ġgeneral izing Ġfrom Ġoriginal Ġto Ġtranslated Ġdata . ĠOur Ġresults Ġsuggest Ġthat Ġthe Ġreal Ġcross - ling ual Ġgeneral ization Ġability Ġof ĠXL M - R Ġis Ġconsiderably Ġbetter Ġthan Ġwhat Ġthe Ġaccuracy Ġnumbers Ġin ĠX N LI Ġreflect . </s> Discussion Ġ: :: ĠOver coming Ġthe Ġcross - ling ual Ġgap Ġis Ġnot Ġwhat Ġmakes ĠTrans late - Train Ġwork . </s> The Ġoriginal Ġmotivation Ġfor ĠTrans late - Train Ġwas Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġsame Ġlanguage Ġit Ġis Ġtested Ġon . ĠHowever , Ġwe Ġshow Ġthat Ġit Ġis Ġtraining Ġon Ġtranslated Ġdata , Ġrather Ġthan Ġtraining Ġon Ġthe Ġtarget Ġlanguage , Ġthat Ġis Ġkey Ġfor Ġthis Ġapproach Ġto Ġoutper form ĠZero - Shot Ġas Ġreported Ġby Ġprevious Ġauthors . </s> Discussion Ġ: :: ĠImprovements Ġpreviously Ġattributed Ġto Ġdata Ġaug mentation Ġshould Ġbe Ġrecons idered . </s> The Ġmethod Ġby ĠB IB REF 4 Ġcombines Ġmachine Ġtranslated Ġpremises Ġand Ġhypotheses Ġin Ġdifferent Ġlanguages Ġ( Â§ SEC REF 2 ), Ġresulting Ġin Ġan Ġeffect Ġsimilar Ġto ĠBT - XX Ġand ĠMT - XX . ĠAs Ġsuch , Ġwe Ġbelieve Ġthat Ġthis Ġmethod Ġshould Ġbe Ġanalyzed Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġdataset Ġartifacts Ġrather Ġthan Ġdata Ġaug mentation , Ġas Ġthe Ġauthors Ġdo . ĠFrom Ġthis Ġperspective , Ġhaving Ġthe Ġpremise Ġand Ġthe Ġhypotheses Ġin Ġdifferent Ġlanguages Ġcan Ġreduce Ġthe Ġsuperficial Ġpatterns Ġbetween Ġthem , Ġwhich Ġwould Ġexplain Ġwhy Ġthis Ġapproach Ġis Ġbetter Ġthan Ġusing Ġexamples Ġin Ġa Ġsingle Ġlanguage . </s> Discussion Ġ: :: ĠThe Ġpotential Ġof ĠTrans late - Test Ġwas Ġunderestimated . </s> The Ġprevious Ġbest Ġresults Ġfor ĠTrans late - Test Ġon ĠX N LI Ġl agged Ġbehind Ġthe Ġstate - of - the - art Ġby Ġ4 . 6 Ġpoints . ĠOur Ġwork Ġreduces Ġthis Ġgap Ġto Ġonly Ġ0 . 8 Ġpoints Ġby Ġaddressing Ġthe Ġunderlying Ġtranslation Ġartifacts . ĠThe Ġreason Ġwhy ĠTrans late - Test Ġis Ġmore Ġseverely Ġaffected Ġby Ġthis Ġphenomenon Ġis Ġtwo fold : Ġ( i ) Ġthe Ġeffect Ġis Ġdoubled Ġby Ġfirst Ġusing Ġhuman Ġtranslation Ġto Ġcreate Ġthe Ġtest Ġset Ġand Ġthen Ġmachine Ġtranslation Ġto Ġtranslate Ġit Ġback Ġto ĠEnglish , Ġand Ġ( ii ) ĠTrans late - Train Ġwas Ġinadvertently Ġmitigating Ġthis Ġissue Ġ( see Ġabove ), Ġbut Ġequivalent Ġtechniques Ġwere Ġnever Ġapplied Ġto ĠTrans late - Test . </s> Discussion Ġ: :: ĠFuture Ġevaluation Ġshould Ġbetter Ġaccount Ġfor Ġtranslation Ġartifacts . </s> The Ġevaluation Ġissues Ġraised Ġby Ġour Ġanalysis Ġdo Ġnot Ġhave Ġa Ġsimple Ġsolution . ĠIn Ġfact , Ġwhile Ġwe Ġuse Ġthe Ġterm Ġtranslation Ġartifacts Ġto Ġhighlight Ġthat Ġthey Ġare Ġan Ġunintended Ġeffect Ġof Ġtranslation Ġthat Ġimpacts Ġfinal Ġevaluation , Ġone Ġcould Ġalso Ġargue Ġthat Ġit Ġis Ġthe Ġoriginal Ġdatasets Ġthat Ġcontain Ġthe Ġartifacts , Ġwhich Ġtranslation Ġsimply Ġalters Ġor Ġeven Ġmit ig ates . ĠIn Ġany Ġcase , Ġthis Ġis Ġa Ġmore Ġgeneral Ġissue Ġthat Ġfalls Ġbeyond Ġthe Ġscope Ġof Ġcross - ling ual Ġtransfer Ġlearning , Ġso Ġwe Ġargue Ġthat Ġit Ġshould Ġbe Ġcarefully Ġcontrolled Ġwhen Ġevaluating Ġcross - ling ual Ġmodels . ĠIn Ġthe Ġabsence Ġof Ġmore Ġrobust Ġdatasets , Ġwe Ġrecommend Ġthat Ġfuture Ġmult ilingual Ġbenchmarks Ġshould Ġat Ġleast Ġprovide Ġconsistent Ġtest Ġsets Ġfor ĠEnglish Ġand Ġthe Ġrest Ġof Ġlanguages . ĠThis Ġcan Ġbe Ġachieved Ġby Ġ( i ) Ġusing Ġoriginal Ġannotations Ġin Ġall Ġlanguages , Ġ( ii ) Ġusing Ġoriginal Ġannotations Ġin Ġa Ġnon - English Ġlanguage Ġand Ġtranslating Ġthem Ġinto ĠEnglish Ġand Ġother Ġlanguages , Ġor Ġ( iii ) Ġif Ġtranslating Ġfrom ĠEnglish , Ġdoing Ġso Ġat Ġthe Ġdocument Ġlevel Ġto Ġminimize Ġtranslation Ġinconsistencies . </s> Con clusions </s> In Ġthis Ġpaper , Ġwe Ġhave Ġshown Ġthat Ġboth Ġhuman Ġand Ġmachine Ġtranslation Ġcan Ġalter Ġsuperficial Ġpatterns Ġin Ġdata , Ġwhich Ġrequires Ġreconsider ing Ġprevious Ġfindings Ġin Ġcross - ling ual Ġtransfer Ġlearning . ĠBased Ġon Ġthe Ġgained Ġinsights , Ġwe Ġhave Ġimproved Ġthe Ġstate - of - the - art Ġin ĠX N LI Ġfor Ġthe ĠTrans late - Test Ġand ĠZero - Shot Ġapproaches Ġby Ġa Ġsubstantial Ġmargin . ĠFinally , Ġwe Ġhave Ġshown Ġthat Ġthe Ġphenomenon Ġis Ġnot Ġspecific Ġto ĠNL I Ġbut Ġalso Ġaffects ĠQ A , Ġalthough Ġit Ġis Ġless Ġpronounced Ġthere Ġthanks Ġto Ġthe Ġtranslation Ġprocedure Ġused Ġin Ġthe Ġcorresponding Ġbenchmarks . ĠSo Ġas Ġto Ġfacilitate Ġsimilar Ġstudies Ġin Ġthe Ġfuture , Ġwe Ġrelease Ġour ĠNL I Ġdataset , Ġwhich , Ġunlike Ġprevious Ġbenchmarks , Ġwas Ġannot ated Ġin Ġa Ġnon - English Ġlanguage Ġand Ġhuman Ġtranslated Ġinto ĠEnglish . </s> Acknowled gments </s> We Ġthank ĠNora ĠAr an ber ri Ġand ĠU x oa ĠI Ã± ur ri eta Ġfor Ġhelpful Ġdiscussion Ġduring Ġthe Ġdevelopment Ġof Ġthis Ġwork , Ġas Ġwell Ġas Ġthe Ġrest Ġof Ġour Ġcolleagues Ġfrom Ġthe ĠIX A Ġgroup Ġthat Ġworked Ġas Ġannot ators Ġfor Ġour ĠNL I Ġdataset . </s> This Ġresearch Ġwas Ġpartially Ġfunded Ġby Ġa ĠFacebook ĠFellowship , Ġthe ĠBas que ĠGovernment Ġexcellence Ġresearch Ġgroup Ġ( IT 13 43 - 19 ), Ġthe ĠSpanish ĠMIN EC O Ġ( Un sup MT ĠT IN 2017 âĢĲ 916 92 âĢĲ EXP ĠMC IU / AE I / F ED ER , ĠUE ), ĠProject ĠBig Know ledge Ġ( Ay ud as ĠFund aci Ã³n ĠBB VA Ġa Ġequip os Ġde Ġinvestig aci Ã³n Ġc ient ÃŃ f ica Ġ2018 ), Ġand Ġthe ĠNVIDIA ĠGPU Ġgrant Ġprogram .