English-German, English-French.
ABSTRACTIVE
English-German, English-French
ABSTRACTIVE
We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to (i) to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the effect of BT in a well-defined domain-adaptation scenario. For both language pairs, we use the Europarl tests from 2007 and 2008 for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain performance, we will use the WMT newstest 2014.
We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to (i) to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the effect of BT in a well-defined domain-adaptation scenario. For both language pairs, we use the Europarl tests from 2007 and 2008 for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain performance, we will use the WMT newstest 2014.
1903.11437
Introduction Ġ </s> The Ġnew Ġgeneration Ġof ĠNeural ĠMachine ĠTranslation Ġ( N MT ) Ġsystems Ġis Ġknown Ġto Ġbe Ġextremely Ġdata Ġhungry ĠB IB REF 0 Ġ. ĠYet , Ġmost Ġexisting ĠN MT Ġtraining Ġpipelines Ġfail Ġto Ġfully Ġtake Ġadvantage Ġof Ġthe Ġvery Ġlarge Ġvolume Ġof Ġmon oling ual Ġsource Ġand / or Ġparallel Ġdata Ġthat Ġis Ġoften Ġavailable . ĠMaking Ġa Ġbetter Ġuse Ġof Ġdata Ġis Ġparticularly Ġcritical Ġin Ġdomain Ġadaptation Ġscenarios , Ġwhere Ġparallel Ġadaptation Ġdata Ġis Ġusually Ġassumed Ġto Ġbe Ġsmall Ġin Ġcomparison Ġto Ġout - of - domain Ġparallel Ġdata , Ġor Ġto Ġin - domain Ġmon oling ual Ġtexts . ĠThis Ġsituation Ġsharply Ġcontrasts Ġwith Ġthe Ġprevious Ġgeneration Ġof Ġstatistical ĠMT Ġengines ĠB IB REF 1 Ġ, Ġwhich Ġcould Ġseamlessly Ġintegrate Ġvery Ġlarge Ġamounts Ġof Ġnon - par allel Ġdocuments , Ġusually Ġwith Ġa Ġlarge Ġpositive Ġeffect Ġon Ġtranslation Ġquality . </s> Such Ġobservations Ġhave Ġbeen Ġmade Ġrepeatedly Ġand Ġhave Ġled Ġto Ġmany Ġinnovative Ġtechniques Ġto Ġintegrate Ġmon oling ual Ġdata Ġin ĠN MT , Ġthat Ġwe Ġreview Ġshortly . ĠThe Ġmost Ġsuccessful Ġapproach Ġto Ġdate Ġis Ġthe Ġproposal Ġof ĠB IB REF 2 Ġ, Ġwho Ġuse Ġmon oling ual Ġtarget Ġtexts Ġto Ġgenerate Ġartificial Ġparallel Ġdata Ġvia Ġbackward Ġtranslation Ġ( BT ). ĠThis Ġtechnique Ġhas Ġsince Ġproven Ġeffective Ġin Ġmany Ġsubsequent Ġstudies . ĠIt Ġis Ġhowever Ġvery Ġcomput ationally Ġcostly , Ġtypically Ġrequiring Ġto Ġtranslate Ġlarge Ġsets Ġof Ġdata . ĠD etermin ing Ġthe ĠâĢ ľ right âĢ Ŀ Ġamount Ġ( and Ġquality ) Ġof ĠBT Ġdata Ġis Ġanother Ġopen Ġissue , Ġbut Ġwe Ġobserve Ġthat Ġexperiments Ġreported Ġin Ġthe Ġliterature Ġonly Ġuse Ġa Ġsubset Ġof Ġthe Ġavailable Ġmon oling ual Ġresources . ĠThis Ġsuggests Ġthat Ġstandard Ġrecipes Ġfor ĠBT Ġmight Ġbe Ġsub - opt imal . </s> This Ġpaper Ġaims Ġto Ġbetter Ġunderstand Ġthe Ġstrengths Ġand Ġweaknesses Ġof ĠBT Ġand Ġto Ġdesign Ġmore Ġprincipled Ġtechniques Ġto Ġimprove Ġits Ġeffects . ĠMore Ġspecifically , Ġwe Ġseek Ġto Ġanswer Ġthe Ġfollowing Ġquestions : Ġsince Ġthere Ġare Ġmany Ġways Ġto Ġgenerate Ġpseudo Ġparallel Ġcorpor a , Ġhow Ġimportant Ġis Ġthe Ġquality Ġof Ġthis Ġdata Ġfor ĠMT Ġperformance ? ĠWhich Ġproperties Ġof Ġback - trans lated Ġsentences Ġactually Ġmatter Ġfor ĠMT Ġquality ? ĠDoes ĠBT Ġact Ġas Ġsome Ġkind Ġof Ġregular izer ĠB IB REF 3 Ġ? ĠCan ĠBT Ġbe Ġefficiently Ġsimulated ? ĠDoes ĠBT Ġdata Ġplay Ġthe Ġsame Ġrole Ġas Ġa Ġtarget - side Ġlanguage Ġmodeling , Ġor Ġare Ġthey Ġcomplementary ? ĠBT Ġis Ġoften Ġused Ġfor Ġdomain Ġadaptation : Ġcan Ġthe Ġeffect Ġof Ġhaving Ġmore Ġin - domain Ġdata Ġbe Ġsorted Ġout Ġfrom Ġthe Ġmere Ġincrease Ġof Ġtraining Ġmaterial ĠB IB REF 2 Ġ? ĠFor Ġstudies Ġrelated Ġto Ġthe Ġimpact Ġof Ġvarying Ġthe Ġsize Ġof ĠBT Ġdata , Ġwe Ġrefer Ġthe Ġreaders Ġto Ġthe Ġrecent Ġwork Ġof ĠB IB REF 4 Ġ. </s> To Ġanswer Ġthese Ġquestions , Ġwe Ġhave Ġreim ple mented Ġseveral Ġstrategies Ġto Ġuse Ġmon oling ual Ġdata Ġin ĠN MT Ġand Ġhave Ġrun Ġexperiments Ġon Ġtwo Ġlanguage Ġpairs Ġin Ġa Ġvery Ġcontrolled Ġsetting Ġ( see ĠÂ§ ĠSEC REF 2 Ġ). ĠOur Ġmain Ġresults Ġ( see ĠÂ§ ĠSEC REF 4 Ġand ĠÂ§ ĠSEC REF 5 Ġ) Ġsuggest Ġpromising Ġdirections Ġfor Ġefficient Ġdomain Ġadaptation Ġwith Ġcheaper Ġtechniques Ġthan Ġconventional ĠBT . </s> In - domain Ġand Ġout - of - domain Ġdata Ġ </s> We Ġare Ġmostly Ġinterested Ġwith Ġthe Ġfollowing Ġtraining Ġscenario : Ġa Ġlarge Ġout - of - domain Ġparallel Ġcorpus , Ġand Ġlimited Ġmon oling ual Ġin - domain Ġdata . ĠWe Ġfocus Ġhere Ġon Ġthe ĠEuro par l Ġdomain , Ġfor Ġwhich Ġwe Ġhave Ġample Ġdata Ġin Ġseveral Ġlanguages , Ġand Ġuse Ġas Ġin - domain Ġtraining Ġdata Ġthe ĠEuro par l Ġcorpus ĠB IB REF 5 Ġfor Ġtwo Ġtranslation Ġdirections : ĠEnglish ĠIN LINE FORM 0 ĠGerman Ġand ĠEnglish ĠIN LINE FORM 1 ĠFrench . ĠAs Ġwe Ġstudy Ġthe Ġbenefits Ġof Ġmon oling ual Ġdata , Ġmost Ġof Ġour Ġexperiments Ġonly Ġuse Ġthe Ġtarget Ġside Ġof Ġthis Ġcorpus . ĠThe Ġrationale Ġfor Ġchoosing Ġthis Ġdomain Ġis Ġto Ġ( i ) Ġto Ġperform Ġlarge Ġscale Ġcomparisons Ġof Ġsynthetic Ġand Ġnatural Ġparallel Ġcorpor a ; Ġ( ii ) Ġto Ġstudy Ġthe Ġeffect Ġof ĠBT Ġin Ġa Ġwell - defined Ġdomain - adapt ation Ġscenario . ĠFor Ġboth Ġlanguage Ġpairs , Ġwe Ġuse Ġthe ĠEuro par l Ġtests Ġfrom Ġ2007 Ġand Ġ2008 Ġfor Ġevaluation Ġpurposes , Ġkeeping Ġtest Ġ2006 Ġfor Ġdevelopment . ĠWhen Ġmeasuring Ġout - of - domain Ġperformance , Ġwe Ġwill Ġuse Ġthe ĠW MT Ġnew st est Ġ2014 . </s> N MT Ġsetups Ġand Ġperformance Ġ </s> Our Ġbaseline ĠN MT Ġsystem Ġimplements Ġthe Ġattention al Ġenc oder - dec oder Ġapproach ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġas Ġimplemented Ġin ĠNem atus ĠB IB REF 8 Ġon Ġ4 Ġmillion Ġout - of - domain Ġparallel Ġsentences . ĠFor ĠFrench Ġwe Ġuse Ġsamples Ġfrom ĠNews - Comment ary - 11 Ġand ĠWikipedia Ġfrom ĠW MT Ġ2014 Ġshared Ġtranslation Ġtask , Ġas Ġwell Ġas Ġthe ĠMulti - UN ĠB IB REF 9 Ġand ĠEU - Book shop ĠB IB REF 10 Ġcorpor a . ĠFor ĠGerman , Ġwe Ġuse Ġsamples Ġfrom ĠNews - Comment ary - 11 , ĠRapid , ĠCommon - C rawl Ġ( W MT Ġ2017 ) Ġand ĠMulti - UN Ġ( see Ġtable ĠT AB REF 5 Ġ). ĠB ilingual ĠB PE Ġunits ĠB IB REF 11 Ġare Ġlearned Ġwith Ġ50 k Ġmerge Ġoperations , Ġyielding Ġvoc ab ular ies Ġof Ġabout Ġrespectively Ġ32 k Ġand Ġ36 k Ġfor ĠEnglish ĠIN LINE FORM 0 ĠFrench Ġand Ġ32 k Ġand Ġ44 k Ġfor ĠEnglish ĠIN LINE FORM 1 ĠGerman . </s> Both Ġsystems Ġuse Ġ512 - dimensional Ġword Ġembed d ings Ġand Ġa Ġsingle Ġhidden Ġlayer Ġwith Ġ1024 Ġcells . ĠThey Ġare Ġoptimized Ġusing ĠAdam ĠB IB REF 12 Ġand Ġearly Ġstopped Ġaccording Ġto Ġthe Ġvalidation Ġperformance . ĠTraining Ġlasted Ġfor Ġabout Ġthree Ġweeks Ġon Ġan ĠNvidia ĠK 80 ĠGPU Ġcard . </s> System s Ġgenerating Ġback - trans lated Ġdata Ġare Ġtrained Ġusing Ġthe Ġsame Ġout - of - domain Ġcorpus , Ġwhere Ġwe Ġsimply Ġexchange Ġthe Ġsource Ġand Ġtarget Ġsides . ĠThey Ġare Ġfurther Ġdocumented Ġin ĠÂ§ ĠSEC REF 8 Ġ. ĠFor Ġthe Ġsake Ġof Ġcomparison , Ġwe Ġalso Ġtrain Ġa Ġsystem Ġthat Ġhas Ġaccess Ġto Ġa Ġlarge Ġbatch Ġof Ġin - domain Ġparallel Ġdata Ġfollowing Ġthe Ġstrategy Ġoften Ġreferred Ġto Ġas ĠâĢ ľ fine - tun ing âĢ Ŀ : Ġupon Ġconvergence Ġof Ġthe Ġbaseline Ġmodel , Ġwe Ġresume Ġtraining Ġwith Ġa Ġ2 M Ġsentence Ġin - domain Ġcorpus Ġmixed Ġwith Ġan Ġequal Ġamount Ġof Ġrandomly Ġselected Ġout - of - domain Ġnatural Ġsentences , Ġwith Ġthe Ġsame Ġarchitecture Ġand Ġtraining Ġparameters , Ġrunning Ġvalidation Ġevery Ġ2000 Ġupdates Ġwith Ġa Ġpatience Ġof Ġ10 . ĠSince ĠB PE Ġunits Ġare Ġselected Ġbased Ġonly Ġon Ġthe Ġout - of - domain Ġstatistics , Ġfine - tun ing Ġis Ġperformed Ġon Ġsentences Ġthat Ġare Ġslightly Ġlonger Ġ( ie . Ġthey Ġcontain Ġmore Ġunits ) Ġthan Ġfor Ġthe Ġinitial Ġtraining . ĠThis Ġsystem Ġdefines Ġan Ġupper - bound Ġof Ġthe Ġtranslation Ġperformance Ġand Ġis Ġden oted Ġbelow Ġas Ġnatural . </s> Our Ġbaseline Ġand Ġto pl ine Ġresults Ġare Ġin ĠTable ĠT AB REF 6 Ġ, Ġwhere Ġwe Ġmeasure Ġtranslation Ġperformance Ġusing ĠB LE U ĠB IB REF 13 Ġ, ĠBE ER ĠB IB REF 14 Ġ( higher Ġis Ġbetter ) Ġand Ġchar ac TER ĠB IB REF 15 Ġ( small er Ġis Ġbetter ). ĠAs Ġthey Ġare Ġtrained Ġfrom Ġmuch Ġsmaller Ġamounts Ġof Ġdata Ġthan Ġcurrent Ġsystems , Ġthese Ġbas elines Ġare Ġnot Ġquite Ġcompetitive Ġto Ġtoday 's Ġbest Ġsystem , Ġbut Ġstill Ġrepresent Ġserious Ġbas elines Ġfor Ġthese Ġdatasets . ĠGiven Ġour Ġsetups , Ġfine - tun ing Ġwith Ġin - domain Ġnatural Ġdata Ġimproves ĠB LE U Ġby Ġalmost Ġ4 Ġpoints Ġfor Ġboth Ġtranslation Ġdirections Ġon Ġin - domain Ġtests ; Ġit Ġalso Ġimproves , Ġalbeit Ġby Ġa Ġsmaller Ġmargin , Ġthe ĠB LE U Ġscore Ġof Ġthe Ġout - of - domain Ġtests . </s> Using Ġartificial Ġparallel Ġdata Ġin ĠN MT Ġ </s> A Ġsimple Ġway Ġto Ġuse Ġmon oling ual Ġdata Ġin ĠMT Ġis Ġto Ġturn Ġit Ġinto Ġsynthetic Ġparallel Ġdata Ġand Ġlet Ġthe Ġtraining Ġprocedure Ġrun Ġas Ġusual ĠB IB REF 16 Ġ. ĠIn Ġthis Ġsection , Ġwe Ġexplore Ġvarious Ġways Ġto Ġimplement Ġthis Ġstrategy . ĠWe Ġfirst Ġreproduce Ġresults Ġof ĠB IB REF 2 Ġwith ĠBT Ġof Ġvarious Ġqualities , Ġthat Ġwe Ġthen Ġanalyze Ġthoroughly . </s> The Ġquality Ġof ĠBack - Translation Ġ </s> BT Ġrequires Ġthe Ġavailability Ġof Ġan ĠMT Ġsystem Ġin Ġthe Ġreverse Ġtranslation Ġdirection . ĠWe Ġconsider Ġhere Ġthree ĠMT Ġsystems Ġof Ġincreasing Ġquality : </s> back trans - bad : Ġthis Ġis Ġa Ġvery Ġpoor ĠSM T Ġsystem Ġtrained Ġusing Ġonly Ġ50 k Ġparallel Ġsentences Ġfrom Ġthe Ġout - of - domain Ġdata , Ġand Ġno Ġadditional Ġmon oling ual Ġdata . ĠFor Ġthis Ġsystem Ġas Ġfor Ġthe Ġnext Ġone , Ġwe Ġuse ĠMoses ĠB IB REF 17 Ġout - of - the - box , Ġcomputing Ġalign ments Ġwith ĠFast align ĠB IB REF 18 Ġ, Ġwith Ġa Ġminimal Ġpre - processing Ġ( basic Ġtoken ization ). ĠThis Ġsetting Ġprovides Ġus Ġwith Ġa Ġpessimistic Ġestimate Ġof Ġwhat Ġwe Ġcould Ġget Ġin Ġlow - resource Ġconditions . </s> back trans - good : Ġthese Ġare Ġmuch Ġlarger ĠSM T Ġsystems , Ġwhich Ġuse Ġthe Ġsame Ġparallel Ġdata Ġas Ġthe Ġbaseline ĠN MT s Ġ( see ĠÂ§ ĠSEC REF 4 Ġ) Ġand Ġall Ġthe ĠEnglish Ġmon oling ual Ġdata Ġavailable Ġfor Ġthe ĠW MT Ġ2017 Ġshared Ġtasks , Ġtot alling Ġapproximately Ġ174 M Ġsentences . ĠThese Ġsystems Ġare Ġstrong , Ġyet Ġrelatively Ġcheap Ġto Ġbuild . </s> back trans - n mt : Ġthese Ġare Ġthe Ġbest ĠN MT Ġsystems Ġwe Ġcould Ġtrain , Ġusing Ġsettings Ġthat Ġreplicate Ġthe Ġforward Ġtranslation ĠN MT s . </s> Note Ġthat Ġwe Ġdo Ġnot Ġuse Ġany Ġin - domain Ġ( Euro par l ) Ġdata Ġto Ġtrain Ġthese Ġsystems . ĠTheir Ġperformance Ġis Ġreported Ġin ĠTable ĠT AB REF 7 Ġ, Ġwhere Ġwe Ġobserve Ġa Ġ12 ĠB LE U Ġpoints Ġgap Ġbetween Ġthe Ġworst Ġand Ġbest Ġsystems Ġ( for Ġboth Ġlanguages ). </s> As Ġnoted Ġeg . Ġin ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, Ġartificial Ġparallel Ġdata Ġobtained Ġthrough Ġforward - translation Ġ( FT ) Ġcan Ġalso Ġprove Ġadvantageous Ġand Ġwe Ġalso Ġconsider Ġa ĠFT Ġsystem Ġ( f wd trans - n mt ): Ġin Ġthis Ġcase Ġthe Ġtarget Ġside Ġof Ġthe Ġcorpus Ġis Ġartificial Ġand Ġis Ġgenerated Ġusing Ġthe Ġbaseline ĠN MT Ġapplied Ġto Ġa Ġnatural Ġsource . </s> Our Ġresults Ġ( see ĠTable ĠT AB REF 6 Ġ) Ġreplicate Ġthe Ġfindings Ġof ĠB IB REF 2 Ġ: Ġlarge Ġgains Ġcan Ġbe Ġobtained Ġfrom ĠBT Ġ( n early ĠIN LINE FORM 0 ĠB LE U Ġin ĠFrench Ġand ĠGerman ); Ġbetter Ġartificial Ġdata Ġyields Ġbetter Ġtranslation Ġsystems . ĠInterestingly , Ġour Ġbest ĠMoses Ġsystem Ġis Ġalmost Ġas Ġgood Ġas Ġthe ĠN MT Ġand Ġan Ġorder Ġof Ġmagnitude Ġfaster Ġto Ġtrain . ĠImprovements Ġobtained Ġwith Ġthe Ġbad Ġsystem Ġare Ġmuch Ġsmaller ; Ġcontrary Ġto Ġthe Ġbetter ĠMT s , Ġthis Ġsystem Ġis Ġeven Ġdetrimental Ġfor Ġthe Ġout - of - domain Ġtest . </s> G ains Ġwith Ġforward Ġtranslation Ġare Ġsignificant , Ġas Ġin ĠB IB REF 21 Ġ, Ġalbeit Ġabout Ġhalf Ġas Ġgood Ġas Ġwith ĠBT , Ġand Ġresult Ġin Ġsmall Ġimprovements Ġfor Ġthe Ġin - domain Ġand Ġfor Ġthe Ġout - of - domain Ġtests . ĠExper iments Ġcombining Ġforward Ġand Ġbackward Ġtranslation Ġ( back f wd trans - n mt ), Ġeach Ġusing Ġa Ġhalf Ġof Ġthe Ġavailable Ġartificial Ġdata , Ġdo Ġnot Ġoutper form Ġthe Ġbest ĠBT Ġresults . </s> We Ġfinally Ġnote Ġthe Ġlarge Ġremaining Ġdifference Ġbetween ĠBT Ġdata Ġand Ġnatural Ġdata , Ġeven Ġthough Ġthey Ġonly Ġdiffer Ġin Ġtheir Ġsource Ġside . ĠThis Ġshows Ġthat Ġat Ġleast Ġin Ġour Ġdomain - adapt ation Ġsettings , ĠBT Ġdoes Ġnot Ġreally Ġact Ġas Ġa Ġregular izer , Ġcontr arily Ġto Ġthe Ġfindings Ġof ĠB IB REF 4 Ġ, ĠB IB REF 11 Ġ. ĠFigure ĠFIG REF 13 Ġdisplays Ġthe Ġlearning Ġcurves Ġof Ġthese Ġtwo Ġsystems . ĠWe Ġobserve Ġthat Ġback trans - n mt Ġimproves Ġquickly Ġin Ġthe Ġearliest Ġupdates Ġand Ġthen Ġstays Ġhorizontal , Ġwhereas Ġnatural Ġcontinues Ġimproving , Ġeven Ġafter Ġ400 k Ġupdates . ĠTherefore ĠBT Ġdoes Ġnot Ġhelp Ġto Ġavoid Ġover fitting , Ġit Ġactually Ġencourages Ġit , Ġwhich Ġmay Ġbe Ġdue ĠâĢ ľ eas ier âĢ Ŀ Ġtraining Ġexamples Ġ( cf . ĠÂ§ ĠSEC REF 15 Ġ). </s> Pro perties Ġof Ġback - trans lated Ġdata Ġ </s> Comp aring Ġthe Ġnatural Ġand Ġartificial Ġsources Ġof Ġour Ġparallel Ġdata Ġwr t . Ġseveral Ġlinguistic Ġand Ġdistribution al Ġproperties , Ġwe Ġobserve Ġthat Ġ( see ĠFig . ĠFIG REF 21 Ġ- ĠFIG REF 22 Ġ): </s> art ificial Ġsources Ġare Ġon Ġaverage Ġshorter Ġthan Ġnatural Ġones : Ġwhen Ġusing ĠBT , Ġcases Ġwhere Ġthe Ġsource Ġis Ġshorter Ġthan Ġthe Ġtarget Ġare Ġrare r ; Ġcases Ġwhen Ġthey Ġhave Ġthe Ġsame Ġlength Ġare Ġmore Ġfrequent . </s> automatic Ġword Ġalign ments Ġbetween Ġartificial Ġsources Ġtend Ġto Ġbe Ġmore Ġmon ot onic Ġthan Ġwhen Ġusing Ġnatural Ġsources , Ġas Ġmeasured Ġby Ġthe Ġaverage ĠKendall ĠIN LINE FORM 0 Ġof Ġsource - target Ġalign ments ĠB IB REF 22 Ġ: Ġfor ĠFrench - English Ġthe Ġrespective Ġnumbers Ġare Ġ0 . 048 Ġ( natural ) Ġand Ġ0 . 018 Ġ( art ificial ); Ġfor ĠGerman - English Ġ0 . 0 68 Ġand Ġ0 . 05 3 . ĠUsing Ġmore Ġmon ot onic Ġsentence Ġpairs Ġturns Ġout Ġto Ġbe Ġa Ġfacilitating Ġfactor Ġfor ĠN MT , Ġas Ġalso Ġnoted Ġby ĠB IB REF 20 Ġ. </s> sy nt act ically , Ġartificial Ġsources Ġare Ġsimpler Ġthan Ġreal Ġdata ; ĠWe Ġobserve Ġsignificant Ġdifferences Ġin Ġthe Ġdistributions Ġof Ġtree Ġdepths . </s> dist ribution ally , Ġplain Ġword Ġoccurrences Ġin Ġartificial Ġsources Ġare Ġmore Ġconcentrated ; Ġthis Ġalso Ġtranslates Ġinto Ġboth Ġa Ġslower Ġincrease Ġof Ġthe Ġnumber Ġof Ġtypes Ġwr t . Ġthe Ġnumber Ġof Ġsentences Ġand Ġa Ġsmaller Ġnumber Ġof Ġrare Ġevents . </s> The Ġintuition Ġis Ġthat Ġproperties Ġ( i ) Ġand Ġ( ii ) Ġshould Ġhelp Ġtranslation Ġas Ġcompared Ġto Ġnatural Ġsource , Ġwhile Ġproperty Ġ( iv ) Ġshould Ġbe Ġdetrimental . ĠWe Ġchecked Ġ( ii ) Ġby Ġbuilding Ġsystems Ġwith Ġonly Ġ10 M Ġwords Ġfrom Ġthe Ġnatural Ġparallel Ġdata Ġselecting Ġthese Ġdata Ġeither Ġrandomly Ġor Ġbased Ġon Ġthe Ġregular ity Ġof Ġtheir Ġword Ġalign ments . ĠResults Ġin ĠTable ĠT AB REF 23 Ġshow Ġthat Ġthe Ġlatter Ġis Ġmuch Ġpreferable Ġfor Ġthe Ġoverall Ġperformance . ĠThis Ġmight Ġexplain Ġthat Ġthe Ġmostly Ġmon ot onic ĠBT Ġfrom ĠMoses Ġare Ġalmost Ġas Ġgood Ġas Ġthe Ġfluid ĠBT Ġfrom ĠN MT Ġand Ġthat Ġboth Ġboost Ġthe Ġbaseline . </s> St upid ĠBack - Translation Ġ </s> We Ġnow Ġanalyze Ġthe Ġeffect Ġof Ġusing Ġmuch Ġsimpler Ġdata Ġgeneration Ġschemes , Ġwhich Ġdo Ġnot Ġrequire Ġthe Ġavailability Ġof Ġa Ġbackward Ġtranslation Ġengine . </s> Set ups </s> We Ġuse Ġthe Ġfollowing Ġcheap Ġways Ġto Ġgenerate Ġpseudo - source Ġtexts : </s> copy : Ġin Ġthis Ġsetting , Ġthe Ġsource Ġside Ġis Ġa Ġmere Ġcopy Ġof Ġthe Ġtarget - side Ġdata . ĠSince Ġthe Ġsource Ġvocabulary Ġof Ġthe ĠN MT Ġis Ġfixed , Ġcopying Ġthe Ġtarget Ġsentences Ġcan Ġcause Ġthe Ġoccurrence Ġof ĠO OV s . ĠTo Ġavoid Ġthis Ġsituation , ĠB IB REF 24 Ġdecom pose Ġthe Ġtarget Ġwords Ġinto Ġsource - side Ġunits Ġto Ġmake Ġthe Ġcopy Ġlook Ġlike Ġsource Ġsentences . ĠEach ĠO OV Ġfound Ġin Ġthe Ġcopy Ġis Ġsplit Ġinto Ġsmaller Ġunits Ġuntil Ġall Ġthe Ġresulting Ġchunks Ġare Ġin Ġthe Ġsource Ġvocabulary . </s> copy - marked : Ġanother Ġway Ġto Ġintegrate Ġcopies Ġwithout Ġhaving Ġto Ġdeal Ġwith ĠO OV s Ġis Ġto Ġaugment Ġthe Ġsource Ġvocabulary Ġwith Ġa Ġcopy Ġof Ġthe Ġtarget Ġvocabulary . ĠIn Ġthis Ġsetup , ĠB IB REF 25 Ġensure Ġthat Ġboth Ġvoc ab ular ies Ġnever Ġoverlap Ġby Ġmarking Ġthe Ġtarget Ġword Ġcopies Ġwith Ġa Ġspecial Ġlanguage Ġidentifier . ĠTherefore Ġthe ĠEnglish Ġword Ġresume Ġcannot Ġbe Ġconfused Ġwith Ġthe Ġhom ographic ĠFrench Ġword , Ġwhich Ġis Ġmarked Ġ@ fr @ res ume . </s> copy - d ummies : Ġinstead Ġof Ġusing Ġactual Ġcopies , Ġwe Ġreplace Ġeach Ġword Ġwith ĠâĢ ľ d ummy âĢ Ŀ Ġtokens . ĠWe Ġuse Ġthis Ġunrealistic Ġsetup Ġto Ġobserve Ġthe Ġtraining Ġover Ġnoisy Ġand Ġhardly Ġinformative Ġsource Ġsentences . </s> We Ġthen Ġuse Ġthe Ġprocedures Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġ, Ġexcept Ġthat Ġthe Ġpseudo - source Ġembed d ings Ġin Ġthe Ġcopy - marked Ġsetup Ġare Ġpret rained Ġfor Ġthree Ġepoch s Ġon Ġthe Ġin - domain Ġdata , Ġwhile Ġall Ġremaining Ġparameters Ġare Ġfrozen . ĠThis Ġprevents Ġrandom Ġparameters Ġfrom Ġhurting Ġthe Ġalready Ġtrained Ġmodel . </s> Copy + mark ing + no ise Ġis Ġnot Ġso Ġstupid </s> We Ġobserve Ġthat Ġthe Ġcopy Ġsetup Ġhas Ġonly Ġa Ġsmall Ġimpact Ġon Ġthe ĠEnglish - French Ġsystem , Ġfor Ġwhich Ġthe Ġbaseline Ġis Ġalready Ġstrong . ĠThis Ġis Ġless Ġtrue Ġfor ĠEnglish - German Ġwhere Ġsimple Ġcopies Ġyield Ġa Ġsignificant Ġimprovement . ĠPerformance Ġdrops Ġfor Ġboth Ġlanguage Ġpairs Ġin Ġthe Ġcopy - d ummies Ġsetup . </s> We Ġachieve Ġour Ġbest Ġgains Ġwith Ġthe Ġcopy - marked Ġsetup , Ġwhich Ġis Ġthe Ġbest Ġway Ġto Ġuse Ġa Ġcopy Ġof Ġthe Ġtarget Ġ( although Ġthe Ġperformance Ġon Ġthe Ġout - of - domain Ġtests Ġis Ġat Ġmost Ġthe Ġsame Ġas Ġthe Ġbaseline ). ĠSuch Ġgains Ġmay Ġlook Ġsurprising , Ġsince Ġthe ĠN MT Ġmodel Ġdoes Ġnot Ġneed Ġto Ġlearn Ġto Ġtranslate Ġbut Ġonly Ġto Ġcopy Ġthe Ġsource . ĠThis Ġis Ġindeed Ġwhat Ġhappens : Ġto Ġconfirm Ġthis , Ġwe Ġbuilt Ġa Ġfake Ġtest Ġset Ġhaving Ġidentical Ġsource Ġand Ġtarget Ġside Ġ( in ĠFrench ). ĠThe Ġaverage Ġcross - ent ropy Ġfor Ġthis Ġtest Ġset Ġis Ġ0 . 33 , Ġvery Ġclose Ġto Ġ0 , Ġto Ġbe Ġcompared Ġwith Ġan Ġaverage Ġcost Ġof Ġ58 . 52 Ġwhen Ġwe Ġprocess Ġan Ġactual Ġsource Ġ( in ĠEnglish ). ĠThis Ġmeans Ġthat Ġthe Ġmodel Ġhas Ġlearned Ġto Ġcopy Ġwords Ġfrom Ġsource Ġto Ġtarget Ġwith Ġno Ġdifficulty , Ġeven Ġfor Ġsentences Ġnot Ġseen Ġin Ġtraining . ĠA Ġfollow - up Ġquestion Ġis Ġwhether Ġtraining Ġa Ġcopying Ġtask Ġinstead Ġof Ġa Ġtranslation Ġtask Ġlimits Ġthe Ġimprovement : Ġwould Ġthe ĠN MT Ġlearn Ġbetter Ġif Ġthe Ġtask Ġwas Ġharder ? ĠTo Ġmeasure Ġthis , Ġwe Ġintroduce Ġnoise Ġin Ġthe Ġtarget Ġsentences Ġcopied Ġonto Ġthe Ġsource , Ġfollowing Ġthe Ġprocedure Ġof ĠB IB REF 26 Ġ: Ġit Ġdelet es Ġrandom Ġwords Ġand Ġperforms Ġa Ġsmall Ġrandom Ġperm utation Ġof Ġthe Ġremaining Ġwords . ĠResults Ġ(+ ĠSource Ġnoise ) Ġshow Ġno Ġdifference Ġfor Ġthe ĠFrench Ġin - domain Ġtest Ġsets , Ġbut Ġbring Ġthe Ġout - of - domain Ġscore Ġto Ġthe Ġlevel Ġof Ġthe Ġbaseline . ĠFinally , Ġwe Ġobserve Ġa Ġsignificant Ġimprovement Ġon ĠGerman Ġin - domain Ġtest Ġsets , Ġcompared Ġto Ġthe Ġbaseline Ġ( about Ġ+ 1 . 5 ĠB LE U ). ĠThis Ġlast Ġsetup Ġis Ġeven Ġalmost Ġas Ġgood Ġas Ġthe Ġback trans - n mt Ġcondition Ġ( see ĠÂ§ ĠSEC REF 8 Ġ) Ġfor ĠGerman . ĠThis Ġshows Ġthat Ġlearning Ġto Ġre order Ġand Ġpredict Ġmissing Ġwords Ġcan Ġmore Ġeffectively Ġserve Ġour Ġpurposes Ġthan Ġsimply Ġlearning Ġto Ġcopy . </s> T ow ards Ġmore Ġnatural Ġpseudo - s ources Ġ </s> Integ rating Ġmon oling ual Ġdata Ġinto ĠN MT Ġcan Ġbe Ġas Ġeasy Ġas Ġcopying Ġthe Ġtarget Ġinto Ġthe Ġsource , Ġwhich Ġalready Ġgives Ġsome Ġimprovement ; Ġadding Ġnoise Ġmakes Ġthings Ġeven Ġbetter . ĠWe Ġnow Ġstudy Ġways Ġto Ġmake Ġpseudo - s ources Ġlook Ġmore Ġlike Ġnatural Ġdata , Ġusing Ġthe Ġframework Ġof ĠGener ative ĠAd vers arial ĠNetworks Ġ( GAN s ) ĠB IB REF 27 Ġ, Ġan Ġidea Ġborrowed Ġfrom ĠB IB REF 26 Ġ. </s> GAN Ġsetups </s> In Ġour Ġsetups , Ġwe Ġuse Ġa Ġmarked Ġtarget Ġcopy , Ġviewed Ġas Ġa Ġfake Ġsource , Ġwhich Ġa Ġgenerator Ġenc odes Ġso Ġas Ġto Ġfool Ġa Ġdiscrim inator Ġtrained Ġto Ġdistinguish Ġa Ġfake Ġfrom Ġa Ġnatural Ġsource . ĠOur Ġarchitecture Ġcontains Ġtwo Ġdistinct Ġenc od ers , Ġone Ġfor Ġthe Ġnatural Ġsource Ġand Ġone Ġfor Ġthe Ġpseudo - source . ĠThe Ġlatter Ġacts Ġas Ġthe Ġgenerator Ġ( ĠIN LINE FORM 0 Ġ) Ġin Ġthe ĠG AN Ġframework , Ġcomputing Ġa Ġrepresentation Ġof Ġthe Ġpseudo - source Ġthat Ġis Ġthen Ġinput Ġto Ġa Ġdiscrim inator Ġ( ĠIN LINE FORM 1 Ġ), Ġwhich Ġhas Ġto Ġsort Ġnatural Ġfrom Ġartificial Ġenc od ings . ĠIN LINE FORM 2 Ġassigns Ġa Ġprobability Ġof Ġa Ġsentence Ġbeing Ġnatural . </s> During Ġtraining , Ġthe Ġcost Ġof Ġthe Ġdiscrim inator Ġis Ġcomputed Ġover Ġtwo Ġbatches , Ġone Ġwith Ġnatural Ġ( out - of - domain ) Ġsentences ĠIN LINE FORM 0 Ġand Ġone Ġwith Ġ( in - domain ) Ġpseudo - sent ences ĠIN LINE FORM 1 Ġ. ĠThe Ġdiscrim inator Ġis Ġa Ġbid irection al - Rec urrent ĠNeural ĠNetwork Ġ( R NN ) Ġof Ġdimension Ġ1024 . ĠA ver aged Ġstates Ġare Ġpassed Ġto Ġa Ġsingle Ġfeed - forward Ġlayer , Ġto Ġwhich Ġa Ġs igm oid Ġis Ġapplied . ĠIt Ġinputs Ġenc od ings Ġof Ġnatural Ġ( ĠIN LINE FORM 2 Ġ) Ġand Ġpseudo - sent ences Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġis Ġtrained Ġto Ġoptimize : </s> IN LINE FORM 0 </s> IN LINE FORM 0 Ġ' s Ġparameters Ġare Ġupdated Ġto Ġmaxim ally Ġfool ĠIN LINE FORM 1 Ġ, Ġthus Ġthe Ġloss ĠIN LINE FORM 2 Ġ: ĠIN LINE FORM 3 </s> Finally , Ġwe Ġkeep Ġthe Ġusual ĠMT Ġobjective . Ġ( ĠIN LINE FORM 0 Ġis Ġa Ġreal Ġor Ġpseudo - sent ence ): ĠIN LINE FORM 1 </s> We Ġthus Ġneed Ġto Ġtrain Ġthree Ġsets Ġof Ġparameters : ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ( MT Ġparameters ), Ġwith ĠIN LINE FORM 2 Ġ. ĠThe Ġpseudo - source Ġenc oder Ġand Ġembed d ings Ġare Ġupdated Ġwr t . Ġboth ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. ĠFollowing ĠB IB REF 28 Ġ, ĠIN LINE FORM 5 Ġis Ġupdated Ġonly Ġwhen ĠIN LINE FORM 6 Ġ' s Ġaccuracy Ġexceeds ĠIN LINE FORM 7 Ġ. ĠOn Ġthe Ġother Ġhand , ĠIN LINE FORM 8 Ġis Ġnot Ġupdated Ġwhen Ġits Ġaccuracy Ġexceeds ĠIN LINE FORM 9 Ġ. ĠAt Ġeach Ġupdate , Ġtwo Ġbatches Ġare Ġgenerated Ġfor Ġeach Ġtype Ġof Ġdata , Ġwhich Ġare Ġencoded Ġwith Ġthe Ġreal Ġor Ġpseudo - enc oder . ĠThe Ġenc oder Ġoutputs Ġserve Ġto Ġcompute ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. ĠFinally , Ġthe Ġpseudo - source Ġis Ġencoded Ġagain Ġ( once ĠIN LINE FORM 12 Ġis Ġupdated ), Ġboth Ġenc od ers Ġare Ġplugged Ġinto Ġthe Ġtranslation Ġmodel Ġand Ġthe ĠMT Ġcost Ġis Ġback - prop ag ated Ġdown Ġto Ġthe Ġreal Ġand Ġpseudo - word Ġembed d ings . ĠPse udo - enc oder Ġand Ġdiscrim inator Ġparameters Ġare Ġpre - trained Ġfor Ġ10 k Ġupdates . ĠAt Ġtest Ġtime , Ġthe Ġpseudo - enc oder Ġis Ġignored Ġand Ġinference Ġis Ġrun Ġas Ġusual . </s> GAN s Ġcan Ġhelp </s> Results Ġare Ġin ĠTable ĠT AB REF 32 Ġ, Ġassuming Ġthe Ġsame Ġfine - tun ing Ġprocedure Ġas Ġabove . ĠOn Ġtop Ġof Ġthe Ġcopy - marked Ġsetup , Ġour ĠG AN s Ġdo Ġnot Ġprovide Ġany Ġimprovement Ġin Ġboth Ġlanguage Ġpairs , Ġwith Ġthe Ġexception Ġof Ġa Ġsmall Ġimprovement Ġfor ĠEnglish - French Ġon Ġthe Ġout - of - domain Ġtest , Ġwhich Ġwe Ġunderstand Ġas Ġa Ġsign Ġthat Ġthe Ġmodel Ġis Ġmore Ġrobust Ġto Ġdomain Ġvariations , Ġjust Ġlike Ġwhen Ġadding Ġpseudo - source Ġnoise . ĠWhen Ġcombined Ġwith Ġnoise , Ġthe ĠFrench Ġmodel Ġyields Ġthe Ġbest Ġperformance Ġwe Ġcould Ġobtain Ġwith Ġstupid ĠBT Ġon Ġthe Ġin - domain Ġtests , Ġat Ġleast Ġin Ġterms Ġof ĠB LE U Ġand ĠBE ER . ĠOn Ġthe ĠNews Ġdomain , Ġwe Ġremain Ġclose Ġto Ġthe Ġbaseline Ġlevel , Ġwith Ġslight Ġimprovements Ġin ĠGerman . </s> A Ġfirst Ġobservation Ġis Ġthat Ġthis Ġmethod Ġbrings Ġstupid ĠBT Ġmodels Ġcloser Ġto Ġconventional ĠBT , Ġat Ġa Ġgreatly Ġreduced Ġcomputational Ġcost . ĠWhile ĠFrench Ġstill Ġremains Ġ0 . 4 Ġto Ġ1 . 0 ĠB LE U Ġbelow Ġvery Ġgood Ġback translation , Ġboth Ġapproaches Ġare Ġin Ġthe Ġsame Ġballpark Ġfor ĠGerman Ġ- Ġmay Ġbe Ġbecause ĠBT s Ġare Ġbetter Ġfor Ġthe Ġformer Ġsystem Ġthan Ġfor Ġthe Ġlatter . </s> Finally Ġnote Ġthat Ġthe ĠG AN Ġarchitecture Ġhas Ġtwo Ġdifferences Ġwith Ġbasic Ġcopy - marked : Ġ( a ) Ġa Ġdistinct Ġenc oder Ġfor Ġreal Ġand Ġpseudo - sent ence ; Ġ( b ) Ġa Ġdifferent Ġtraining Ġregime Ġfor Ġthese Ġenc od ers . ĠTo Ġsort Ġout Ġthe Ġeffects Ġof Ġ( a ) Ġand Ġ( b ), Ġwe Ġreproduce Ġthe ĠG AN Ġsetup Ġwith ĠBT Ġsentences , Ġinstead Ġof Ġcopies . ĠUsing Ġa Ġseparate Ġenc oder Ġfor Ġthe Ġpseudo - source Ġin Ġthe Ġback trans - n mt Ġsetup Ġcan Ġbe Ġdetrimental Ġto Ġperformance Ġ( see ĠTable ĠT AB REF 32 Ġ): Ġtranslation Ġde grades Ġin ĠFrench Ġfor Ġall Ġmetrics . ĠAdding ĠG AN s Ġon Ġtop Ġof Ġthe Ġpseudo - enc oder Ġwas Ġnot Ġable Ġto Ġmake Ġup Ġfor Ġthe Ġdegradation Ġobserved Ġin ĠFrench , Ġbut Ġallowed Ġthe ĠGerman Ġsystem Ġto Ġslightly Ġoutper form Ġback trans - n mt . ĠEven Ġthough Ġthis Ġsetup Ġis Ġunrealistic Ġand Ġoverly Ġcostly , Ġit Ġshows Ġthat ĠG AN s Ġare Ġactually Ġhelping Ġeven Ġgood Ġsystems . </s> Using ĠTarget ĠLanguage ĠModels Ġ </s> In Ġthis Ġsection , Ġwe Ġcompare Ġthe Ġprevious Ġmethods Ġwith Ġthe Ġuse Ġof Ġa Ġtarget Ġside ĠLanguage ĠModel Ġ( LM ). ĠSeveral Ġproposals Ġexist Ġin Ġthe Ġliterature Ġto Ġintegrate ĠL Ms Ġin ĠN MT : Ġfor Ġinstance , ĠB IB REF 3 Ġstrengthen Ġthe Ġdec oder Ġby Ġintegrating Ġan Ġextra , Ġsource Ġindependent , ĠR NN Ġlayer Ġin Ġa Ġconventional ĠN MT Ġarchitecture . ĠTraining Ġis Ġperformed Ġeither Ġwith Ġparallel , Ġor Ġmon oling ual Ġdata . ĠIn Ġthe Ġlatter Ġcase , Ġword Ġprediction Ġonly Ġrelies Ġon Ġthe Ġsource Ġindependent Ġpart Ġof Ġthe Ġnetwork . </s> LM ĠSetup </s> We Ġhave Ġfollowed ĠB IB REF 29 Ġand Ġreim ple mented Ġtheir Ġdeep - f usion Ġtechnique . ĠIt Ġrequires Ġto Ġfirst Ġindependently Ġlearn Ġa ĠR NN - LM Ġon Ġthe Ġin - domain Ġtarget Ġdata Ġwith Ġa Ġcross - ent ropy Ġobjective ; Ġthen Ġto Ġtrain Ġthe Ġoptimal Ġcombination Ġof Ġthe Ġtranslation Ġand Ġthe Ġlanguage Ġmodels Ġby Ġadding Ġthe Ġhidden Ġstate Ġof Ġthe ĠR NN - LM Ġas Ġan Ġadditional Ġinput Ġto Ġthe Ġsoft max Ġlayer Ġof Ġthe Ġdec oder . ĠOur ĠR NN - L Ms Ġare Ġtrained Ġusing Ġd l 4 mt Ġwith Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġdata Ġand Ġthe ĠEuro par l Ġcorpus Ġ( about Ġ6 M Ġsentences Ġfor Ġboth ĠFrench Ġand ĠGerman ), Ġusing Ġa Ġone - layer ĠGR U Ġwith Ġthe Ġsame Ġdimension Ġas Ġthe ĠMT Ġdec oder Ġ( 1024 ). </s> LM ĠResults </s> Results Ġare Ġin ĠTable ĠT AB REF 33 Ġ. ĠThey Ġshow Ġthat Ġdeep - f usion Ġhardly Ġimproves Ġthe ĠEuro par l Ġresults , Ġwhile Ġwe Ġobtain Ġabout Ġ+ 0 . 6 ĠB LE U Ġover Ġthe Ġbaseline Ġon Ġnew st est - 2014 Ġfor Ġboth Ġlanguages . Ġdeep - f usion Ġdiffers Ġfrom Ġstupid ĠBT Ġin Ġthat Ġthe Ġmodel Ġis Ġnot Ġdirectly Ġoptimized Ġon Ġthe Ġin - domain Ġdata , Ġbut Ġuses Ġthe ĠLM Ġtrained Ġon ĠEuro par l Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġout - of - domain Ġtraining Ġdata . ĠTherefore , Ġno Ġspecific Ġimprovement Ġis Ġto Ġbe Ġexpected Ġin Ġterms Ġof Ġdomain Ġadaptation , Ġand Ġthe Ġperformance Ġincreases Ġin Ġthe Ġmore Ġgeneral Ġdomain . ĠComb ining Ġdeep - f usion Ġand Ġcopy - marked Ġ+ Ġnoise Ġ+ ĠG AN s Ġbrings Ġslight Ġimprovements Ġon Ġthe ĠGerman Ġin - domain Ġtest Ġsets , Ġand Ġperformance Ġout Ġof Ġthe Ġdomain Ġremains Ġnear Ġthe Ġbaseline Ġlevel . </s> Re - analy zing Ġthe Ġeffects Ġof ĠBT Ġ </s> As Ġa Ġfollow Ġup Ġof Ġprevious Ġdiscussions , Ġwe Ġanalyze Ġthe Ġeffect Ġof ĠBT Ġon Ġthe Ġintern als Ġof Ġthe Ġnetwork . ĠAr gu ably , Ġusing Ġa Ġcopy Ġof Ġthe Ġtarget Ġsentence Ġinstead Ġof Ġa Ġnatural Ġsource Ġshould Ġnot Ġbe Ġhelpful Ġfor Ġthe Ġenc oder , Ġbut Ġis Ġit Ġalso Ġthe Ġcase Ġwith Ġa Ġstrong ĠBT ? ĠWhat Ġare Ġthe Ġeffects Ġon Ġthe Ġattention Ġmodel ? </s> Parameter Ġfreezing Ġprotocol </s> To Ġinvestigate Ġthese Ġquestions , Ġwe Ġrun Ġthe Ġsame Ġfine - tun ing Ġusing Ġthe Ġcopy - marked , Ġback trans - n mt Ġand Ġback trans - n mt Ġsetups . ĠNote Ġthat Ġexcept Ġfor Ġthe Ġlast Ġone , Ġall Ġtraining Ġscenarios Ġhave Ġaccess Ġto Ġsame Ġtarget Ġtraining Ġdata . ĠWe Ġintend Ġto Ġsee Ġwhether Ġthe Ġoverall Ġperformance Ġof Ġthe ĠN MT Ġsystem Ġde grades Ġwhen Ġwe Ġselectively Ġfreeze Ġcertain Ġsets Ġof Ġparameters , Ġmeaning Ġthat Ġthey Ġare Ġnot Ġupdated Ġduring Ġfine - tun ing . </s> Results </s> BLE U Ġscores Ġare Ġin ĠTable ĠT AB REF 39 Ġ. ĠThe Ġback trans - n mt Ġsetup Ġis Ġhardly Ġimpacted Ġby Ġselective Ġupdates : Ġupdating Ġthe Ġonly Ġdec oder Ġleads Ġto Ġa Ġdegradation Ġof Ġat Ġmost Ġ0 . 2 ĠB LE U . ĠFor Ġcopy - marked , Ġwe Ġwere Ġnot Ġable Ġto Ġfreeze Ġthe Ġsource Ġembed d ings , Ġsince Ġthese Ġare Ġinitialized Ġwhen Ġfine - tun ing Ġbegins Ġand Ġtherefore Ġneed Ġto Ġbe Ġtrained . ĠWe Ġobserve Ġthat Ġfreezing Ġthe Ġenc oder Ġand / or Ġthe Ġattention Ġparameters Ġhas Ġno Ġimpact Ġon Ġthe ĠEnglish - German Ġsystem , Ġwhereas Ġit Ġslightly Ġde grades Ġthe ĠEnglish - French Ġone . ĠThis Ġsuggests Ġthat Ġusing Ġartificial Ġsources , Ġeven Ġof Ġthe Ġpoorest Ġquality , Ġhas Ġa Ġpositive Ġimpact Ġon Ġall Ġthe Ġcomponents Ġof Ġthe Ġnetwork , Ġwhich Ġmakes Ġanother Ġbig Ġdifference Ġwith Ġthe ĠLM Ġintegration Ġscenario . </s> The Ġlargest Ġdegradation Ġis Ġfor Ġnatural , Ġwhere Ġthe Ġmodel Ġis Ġprevented Ġfrom Ġlearning Ġfrom Ġinformative Ġsource Ġsentences , Ġwhich Ġleads Ġto Ġa Ġdecrease Ġof Ġ0 . 4 Ġto Ġover Ġ1 . 0 ĠB LE U . ĠWe Ġassume Ġfrom Ġthese Ġexperiments Ġthat ĠBT Ġimpacts Ġmost Ġof Ġall Ġthe Ġdec oder , Ġand Ġlearning Ġto Ġencode Ġa Ġpseudo - source , Ġbe Ġit Ġa Ġcopy Ġor Ġan Ġactual Ġback - translation , Ġonly Ġmarginally Ġhelps Ġto Ġsignificantly Ġimprove Ġthe Ġquality . ĠFinally , Ġin Ġthe Ġf wd trans - n mt Ġsetup , Ġfreezing Ġthe Ġdec oder Ġdoes Ġnot Ġseem Ġto Ġharm Ġlearning Ġwith Ġa Ġnatural Ġsource . </s> Related Ġwork </s> The Ġliterature Ġdevoted Ġto Ġthe Ġuse Ġof Ġmon oling ual Ġdata Ġis Ġlarge , Ġand Ġquickly Ġexpanding . ĠWe Ġalready Ġalluded Ġto Ġseveral Ġpossible Ġways Ġto Ġuse Ġsuch Ġdata : Ġusing Ġback - Ġor Ġforward - translation Ġor Ġusing Ġa Ġtarget Ġlanguage Ġmodel . ĠThe Ġformer Ġapproach Ġis Ġmostly Ġdocumented Ġin ĠB IB REF 2 Ġ, Ġand Ġrecently Ġanalyzed Ġin ĠB IB REF 19 Ġ, Ġwhich Ġfocus Ġon Ġfully Ġartificial Ġsettings Ġas Ġwell Ġas Ġpivot - based Ġartificial Ġdata ; Ġand ĠB IB REF 4 Ġ, Ġwhich Ġstudies Ġthe Ġeffects Ġof Ġincreasing Ġthe Ġsize Ġof ĠBT Ġdata . ĠThe Ġstudies Ġof ĠB IB REF 20 Ġ, ĠB IB REF 19 Ġalso Ġconsider Ġforward Ġtranslation Ġand ĠB IB REF 21 Ġexpand Ġthese Ġresults Ġto Ġdomain Ġadaptation Ġscenarios . ĠOur Ġresults Ġare Ġcomplementary Ġto Ġthese Ġearlier Ġstudies . </s> As Ġshown Ġabove , Ġmany Ġalternatives Ġto ĠBT Ġexist . ĠThe Ġmost Ġobvious Ġis Ġto Ġuse Ġtarget ĠL Ms ĠB IB REF 3 Ġ, ĠB IB REF 29 Ġ, Ġas Ġwe Ġhave Ġalso Ġdone Ġhere ; Ġbut Ġattempts Ġto Ġimprove Ġthe Ġenc oder Ġusing Ġmulti - task Ġlearning Ġalso Ġexist ĠB IB REF 30 Ġ. </s> This Ġinvestigation Ġis Ġalso Ġrelated Ġto Ġrecent Ġattempts Ġto Ġconsider Ġsupplementary Ġdata Ġwith Ġa Ġvalid Ġtarget Ġside , Ġsuch Ġas Ġmulti - ling ual ĠN MT ĠB IB REF 31 Ġ, Ġwhere Ġsource Ġtexts Ġin Ġseveral Ġlanguages Ġare Ġfed Ġin Ġthe Ġsame Ġenc oder - dec oder Ġarchitecture , Ġwith Ġpartial Ġsharing Ġof Ġthe Ġlayers . ĠThis Ġis Ġanother Ġrealistic Ġscenario Ġwhere Ġadditional Ġresources Ġcan Ġbe Ġused Ġto Ġselectively Ġimprove Ġparts Ġof Ġthe Ġmodel . </s> Round Ġtrip Ġtraining Ġis Ġanother Ġimportant Ġsource Ġof Ġinspiration , Ġas Ġit Ġcan Ġbe Ġviewed Ġas Ġa Ġway Ġto Ġuse ĠBT Ġto Ġperform Ġsemi - un super vised ĠB IB REF 32 Ġor Ġun super vised ĠB IB REF 33 Ġtraining Ġof ĠN MT . ĠThe Ġmost Ġconvincing Ġattempt Ġto Ġdate Ġalong Ġthese Ġlines Ġhas Ġbeen Ġproposed Ġby ĠB IB REF 26 Ġ, Ġwho Ġpropose Ġto Ġuse ĠG AN s Ġto Ġmitigate Ġthe Ġdifference Ġbetween Ġartificial Ġand Ġnatural Ġdata . </s> Conclusion Ġ </s> In Ġthis Ġpaper , Ġwe Ġhave Ġanalyzed Ġvarious Ġways Ġto Ġintegrate Ġmon oling ual Ġdata Ġin Ġan ĠN MT Ġframework , Ġfocusing Ġon Ġtheir Ġimpact Ġon Ġquality Ġand Ġdomain Ġadaptation . ĠWhile Ġconfirming Ġthe Ġeffectiveness Ġof ĠBT , Ġour Ġstudy Ġalso Ġproposed Ġsignificantly Ġcheaper Ġways Ġto Ġimprove Ġthe Ġbaseline Ġperformance , Ġusing Ġa Ġslightly Ġmodified Ġcopy Ġof Ġthe Ġtarget , Ġinstead Ġof Ġits Ġfull ĠBT . ĠWhen Ġno Ġhigh Ġquality ĠBT Ġis Ġavailable , Ġusing ĠG AN s Ġto Ġmake Ġthe Ġpseudo - source Ġsentences Ġcloser Ġto Ġnatural Ġsource Ġsentences Ġis Ġan Ġefficient Ġsolution Ġfor Ġdomain Ġadaptation . </s> To Ġrecap Ġour Ġanswers Ġto Ġour Ġinitial Ġquestions : Ġthe Ġquality Ġof ĠBT Ġactually Ġmatters Ġfor ĠN MT Ġ( cf . ĠÂ§ ĠSEC REF 8 Ġ) Ġand Ġit Ġseems Ġthat , Ġeven Ġthough Ġartificial Ġsource Ġare Ġlex ically Ġless Ġdiverse Ġand Ġsynt act ically Ġcomplex Ġthan Ġreal Ġsentence , Ġtheir Ġmon oton icity Ġis Ġa Ġfacilitating Ġfactor . ĠWe Ġhave Ġstudied Ġcheaper Ġalternatives Ġand Ġfound Ġout Ġthat Ġcopies Ġof Ġthe Ġtarget , Ġif Ġproperly Ġno ised Ġ( Â§ ĠSEC REF 4 Ġ), Ġand Ġeven Ġbetter , Ġif Ġused Ġwith ĠG AN s , Ġcould Ġbe Ġalmost Ġas Ġgood Ġas Ġlow Ġquality ĠBT s Ġ( Â§ ĠSEC REF 5 Ġ): ĠBT Ġis Ġonly Ġworth Ġits Ġcost Ġwhen Ġgood ĠBT Ġcan Ġbe Ġgenerated . ĠFinally , ĠBT Ġseems Ġpreferable Ġto Ġintegrating Ġexternal ĠLM Ġ- Ġat Ġleast Ġin Ġour Ġdata Ġcondition Ġ( Â§ ĠSEC REF 6 Ġ). ĠFurther Ġexperiments Ġwith Ġlarger ĠL Ms Ġare Ġneeded Ġto Ġconfirm Ġthis Ġobservation , Ġand Ġalso Ġto Ġevaluate Ġthe Ġcomplement arity Ġof Ġboth Ġstrategies . ĠMore Ġwork Ġis Ġneeded Ġto Ġbetter Ġunderstand Ġthe Ġimpact Ġof ĠBT Ġon Ġsub parts Ġof Ġthe Ġnetwork Ġ( Â§ ĠSEC REF 7 Ġ). </s> In Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġother Ġcheap Ġways Ġto Ġgenerate Ġartificial Ġdata . ĠThe Ġexperimental Ġsetup Ġwe Ġproposed Ġmay Ġalso Ġbenefit Ġfrom Ġa Ġrefining Ġof Ġthe Ġdata Ġselection Ġstrategies Ġto Ġfocus Ġon Ġthe Ġmost Ġuseful Ġmon oling ual Ġsentences .
Introduction  The new generation of Neural Machine Translation (NMT) systems is known to be extremely data hungry BIBREF0 . Yet, most existing NMT training pipelines fail to fully take advantage of the very large volume of monolingual source and/or parallel data that is often available. Making a better use of data is particularly critical in domain adaptation scenarios, where parallel adaptation data is usually assumed to be small in comparison to out-of-domain parallel data, or to in-domain monolingual texts. This situation sharply contrasts with the previous generation of statistical MT engines BIBREF1 , which could seamlessly integrate very large amounts of non-parallel documents, usually with a large positive effect on translation quality. Such observations have been made repeatedly and have led to many innovative techniques to integrate monolingual data in NMT, that we review shortly. The most successful approach to date is the proposal of BIBREF2 , who use monolingual target texts to generate artificial parallel data via backward translation (BT). This technique has since proven effective in many subsequent studies. It is however very computationally costly, typically requiring to translate large sets of data. Determining the “right” amount (and quality) of BT data is another open issue, but we observe that experiments reported in the literature only use a subset of the available monolingual resources. This suggests that standard recipes for BT might be sub-optimal. This paper aims to better understand the strengths and weaknesses of BT and to design more principled techniques to improve its effects. More specifically, we seek to answer the following questions: since there are many ways to generate pseudo parallel corpora, how important is the quality of this data for MT performance? Which properties of back-translated sentences actually matter for MT quality? Does BT act as some kind of regularizer BIBREF3 ? Can BT be efficiently simulated? Does BT data play the same role as a target-side language modeling, or are they complementary? BT is often used for domain adaptation: can the effect of having more in-domain data be sorted out from the mere increase of training material BIBREF2 ? For studies related to the impact of varying the size of BT data, we refer the readers to the recent work of BIBREF4 . To answer these questions, we have reimplemented several strategies to use monolingual data in NMT and have run experiments on two language pairs in a very controlled setting (see § SECREF2 ). Our main results (see § SECREF4 and § SECREF5 ) suggest promising directions for efficient domain adaptation with cheaper techniques than conventional BT. In-domain and out-of-domain data  We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to (i) to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the effect of BT in a well-defined domain-adaptation scenario. For both language pairs, we use the Europarl tests from 2007 and 2008 for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain performance, we will use the WMT newstest 2014. NMT setups and performance  Our baseline NMT system implements the attentional encoder-decoder approach BIBREF6 , BIBREF7 as implemented in Nematus BIBREF8 on 4 million out-of-domain parallel sentences. For French we use samples from News-Commentary-11 and Wikipedia from WMT 2014 shared translation task, as well as the Multi-UN BIBREF9 and EU-Bookshop BIBREF10 corpora. For German, we use samples from News-Commentary-11, Rapid, Common-Crawl (WMT 2017) and Multi-UN (see table TABREF5 ). Bilingual BPE units BIBREF11 are learned with 50k merge operations, yielding vocabularies of about respectively 32k and 36k for English INLINEFORM0 French and 32k and 44k for English INLINEFORM1 German. Both systems use 512-dimensional word embeddings and a single hidden layer with 1024 cells. They are optimized using Adam BIBREF12 and early stopped according to the validation performance. Training lasted for about three weeks on an Nvidia K80 GPU card. Systems generating back-translated data are trained using the same out-of-domain corpus, where we simply exchange the source and target sides. They are further documented in § SECREF8 . For the sake of comparison, we also train a system that has access to a large batch of in-domain parallel data following the strategy often referred to as “fine-tuning”: upon convergence of the baseline model, we resume training with a 2M sentence in-domain corpus mixed with an equal amount of randomly selected out-of-domain natural sentences, with the same architecture and training parameters, running validation every 2000 updates with a patience of 10. Since BPE units are selected based only on the out-of-domain statistics, fine-tuning is performed on sentences that are slightly longer (ie. they contain more units) than for the initial training. This system defines an upper-bound of the translation performance and is denoted below as natural. Our baseline and topline results are in Table TABREF6 , where we measure translation performance using BLEU BIBREF13 , BEER BIBREF14 (higher is better) and characTER BIBREF15 (smaller is better). As they are trained from much smaller amounts of data than current systems, these baselines are not quite competitive to today's best system, but still represent serious baselines for these datasets. Given our setups, fine-tuning with in-domain natural data improves BLEU by almost 4 points for both translation directions on in-domain tests; it also improves, albeit by a smaller margin, the BLEU score of the out-of-domain tests. Using artificial parallel data in NMT  A simple way to use monolingual data in MT is to turn it into synthetic parallel data and let the training procedure run as usual BIBREF16 . In this section, we explore various ways to implement this strategy. We first reproduce results of BIBREF2 with BT of various qualities, that we then analyze thoroughly. The quality of Back-Translation  BT requires the availability of an MT system in the reverse translation direction. We consider here three MT systems of increasing quality: backtrans-bad: this is a very poor SMT system trained using only 50k parallel sentences from the out-of-domain data, and no additional monolingual data. For this system as for the next one, we use Moses BIBREF17 out-of-the-box, computing alignments with Fastalign BIBREF18 , with a minimal pre-processing (basic tokenization). This setting provides us with a pessimistic estimate of what we could get in low-resource conditions. backtrans-good: these are much larger SMT systems, which use the same parallel data as the baseline NMTs (see § SECREF4 ) and all the English monolingual data available for the WMT 2017 shared tasks, totalling approximately 174M sentences. These systems are strong, yet relatively cheap to build. backtrans-nmt: these are the best NMT systems we could train, using settings that replicate the forward translation NMTs. Note that we do not use any in-domain (Europarl) data to train these systems. Their performance is reported in Table TABREF7 , where we observe a 12 BLEU points gap between the worst and best systems (for both languages). As noted eg. in BIBREF19 , BIBREF20 , artificial parallel data obtained through forward-translation (FT) can also prove advantageous and we also consider a FT system (fwdtrans-nmt): in this case the target side of the corpus is artificial and is generated using the baseline NMT applied to a natural source. Our results (see Table TABREF6 ) replicate the findings of BIBREF2 : large gains can be obtained from BT (nearly INLINEFORM0 BLEU in French and German); better artificial data yields better translation systems. Interestingly, our best Moses system is almost as good as the NMT and an order of magnitude faster to train. Improvements obtained with the bad system are much smaller; contrary to the better MTs, this system is even detrimental for the out-of-domain test. Gains with forward translation are significant, as in BIBREF21 , albeit about half as good as with BT, and result in small improvements for the in-domain and for the out-of-domain tests. Experiments combining forward and backward translation (backfwdtrans-nmt), each using a half of the available artificial data, do not outperform the best BT results. We finally note the large remaining difference between BT data and natural data, even though they only differ in their source side. This shows that at least in our domain-adaptation settings, BT does not really act as a regularizer, contrarily to the findings of BIBREF4 , BIBREF11 . Figure FIGREF13 displays the learning curves of these two systems. We observe that backtrans-nmt improves quickly in the earliest updates and then stays horizontal, whereas natural continues improving, even after 400k updates. Therefore BT does not help to avoid overfitting, it actually encourages it, which may be due “easier” training examples (cf. § SECREF15 ). Properties of back-translated data  Comparing the natural and artificial sources of our parallel data wrt. several linguistic and distributional properties, we observe that (see Fig. FIGREF21 - FIGREF22 ): artificial sources are on average shorter than natural ones: when using BT, cases where the source is shorter than the target are rarer; cases when they have the same length are more frequent. automatic word alignments between artificial sources tend to be more monotonic than when using natural sources, as measured by the average Kendall INLINEFORM0 of source-target alignments BIBREF22 : for French-English the respective numbers are 0.048 (natural) and 0.018 (artificial); for German-English 0.068 and 0.053. Using more monotonic sentence pairs turns out to be a facilitating factor for NMT, as also noted by BIBREF20 . syntactically, artificial sources are simpler than real data; We observe significant differences in the distributions of tree depths. distributionally, plain word occurrences in artificial sources are more concentrated; this also translates into both a slower increase of the number of types wrt. the number of sentences and a smaller number of rare events. The intuition is that properties (i) and (ii) should help translation as compared to natural source, while property (iv) should be detrimental. We checked (ii) by building systems with only 10M words from the natural parallel data selecting these data either randomly or based on the regularity of their word alignments. Results in Table TABREF23 show that the latter is much preferable for the overall performance. This might explain that the mostly monotonic BT from Moses are almost as good as the fluid BT from NMT and that both boost the baseline. Stupid Back-Translation  We now analyze the effect of using much simpler data generation schemes, which do not require the availability of a backward translation engine. Setups We use the following cheap ways to generate pseudo-source texts: copy: in this setting, the source side is a mere copy of the target-side data. Since the source vocabulary of the NMT is fixed, copying the target sentences can cause the occurrence of OOVs. To avoid this situation, BIBREF24 decompose the target words into source-side units to make the copy look like source sentences. Each OOV found in the copy is split into smaller units until all the resulting chunks are in the source vocabulary. copy-marked: another way to integrate copies without having to deal with OOVs is to augment the source vocabulary with a copy of the target vocabulary. In this setup, BIBREF25 ensure that both vocabularies never overlap by marking the target word copies with a special language identifier. Therefore the English word resume cannot be confused with the homographic French word, which is marked @fr@resume. copy-dummies: instead of using actual copies, we replace each word with “dummy” tokens. We use this unrealistic setup to observe the training over noisy and hardly informative source sentences. We then use the procedures described in § SECREF4 , except that the pseudo-source embeddings in the copy-marked setup are pretrained for three epochs on the in-domain data, while all remaining parameters are frozen. This prevents random parameters from hurting the already trained model. Copy+marking+noise is not so stupid We observe that the copy setup has only a small impact on the English-French system, for which the baseline is already strong. This is less true for English-German where simple copies yield a significant improvement. Performance drops for both language pairs in the copy-dummies setup. We achieve our best gains with the copy-marked setup, which is the best way to use a copy of the target (although the performance on the out-of-domain tests is at most the same as the baseline). Such gains may look surprising, since the NMT model does not need to learn to translate but only to copy the source. This is indeed what happens: to confirm this, we built a fake test set having identical source and target side (in French). The average cross-entropy for this test set is 0.33, very close to 0, to be compared with an average cost of 58.52 when we process an actual source (in English). This means that the model has learned to copy words from source to target with no difficulty, even for sentences not seen in training. A follow-up question is whether training a copying task instead of a translation task limits the improvement: would the NMT learn better if the task was harder? To measure this, we introduce noise in the target sentences copied onto the source, following the procedure of BIBREF26 : it deletes random words and performs a small random permutation of the remaining words. Results (+ Source noise) show no difference for the French in-domain test sets, but bring the out-of-domain score to the level of the baseline. Finally, we observe a significant improvement on German in-domain test sets, compared to the baseline (about +1.5 BLEU). This last setup is even almost as good as the backtrans-nmt condition (see § SECREF8 ) for German. This shows that learning to reorder and predict missing words can more effectively serve our purposes than simply learning to copy. Towards more natural pseudo-sources  Integrating monolingual data into NMT can be as easy as copying the target into the source, which already gives some improvement; adding noise makes things even better. We now study ways to make pseudo-sources look more like natural data, using the framework of Generative Adversarial Networks (GANs) BIBREF27 , an idea borrowed from BIBREF26 . GAN setups In our setups, we use a marked target copy, viewed as a fake source, which a generator encodes so as to fool a discriminator trained to distinguish a fake from a natural source. Our architecture contains two distinct encoders, one for the natural source and one for the pseudo-source. The latter acts as the generator ( INLINEFORM0 ) in the GAN framework, computing a representation of the pseudo-source that is then input to a discriminator ( INLINEFORM1 ), which has to sort natural from artificial encodings. INLINEFORM2 assigns a probability of a sentence being natural. During training, the cost of the discriminator is computed over two batches, one with natural (out-of-domain) sentences INLINEFORM0 and one with (in-domain) pseudo-sentences INLINEFORM1 . The discriminator is a bidirectional-Recurrent Neural Network (RNN) of dimension 1024. Averaged states are passed to a single feed-forward layer, to which a sigmoid is applied. It inputs encodings of natural ( INLINEFORM2 ) and pseudo-sentences ( INLINEFORM3 ) and is trained to optimize: INLINEFORM0 INLINEFORM0 's parameters are updated to maximally fool INLINEFORM1 , thus the loss INLINEFORM2 : INLINEFORM3 Finally, we keep the usual MT objective. ( INLINEFORM0 is a real or pseudo-sentence): INLINEFORM1 We thus need to train three sets of parameters: INLINEFORM0 and INLINEFORM1 (MT parameters), with INLINEFORM2 . The pseudo-source encoder and embeddings are updated wrt. both INLINEFORM3 and INLINEFORM4 . Following BIBREF28 , INLINEFORM5 is updated only when INLINEFORM6 's accuracy exceeds INLINEFORM7 . On the other hand, INLINEFORM8 is not updated when its accuracy exceeds INLINEFORM9 . At each update, two batches are generated for each type of data, which are encoded with the real or pseudo-encoder. The encoder outputs serve to compute INLINEFORM10 and INLINEFORM11 . Finally, the pseudo-source is encoded again (once INLINEFORM12 is updated), both encoders are plugged into the translation model and the MT cost is back-propagated down to the real and pseudo-word embeddings. Pseudo-encoder and discriminator parameters are pre-trained for 10k updates. At test time, the pseudo-encoder is ignored and inference is run as usual. GANs can help Results are in Table TABREF32 , assuming the same fine-tuning procedure as above. On top of the copy-marked setup, our GANs do not provide any improvement in both language pairs, with the exception of a small improvement for English-French on the out-of-domain test, which we understand as a sign that the model is more robust to domain variations, just like when adding pseudo-source noise. When combined with noise, the French model yields the best performance we could obtain with stupid BT on the in-domain tests, at least in terms of BLEU and BEER. On the News domain, we remain close to the baseline level, with slight improvements in German. A first observation is that this method brings stupid BT models closer to conventional BT, at a greatly reduced computational cost. While French still remains 0.4 to 1.0 BLEU below very good backtranslation, both approaches are in the same ballpark for German - may be because BTs are better for the former system than for the latter. Finally note that the GAN architecture has two differences with basic copy-marked: (a) a distinct encoder for real and pseudo-sentence; (b) a different training regime for these encoders. To sort out the effects of (a) and (b), we reproduce the GAN setup with BT sentences, instead of copies. Using a separate encoder for the pseudo-source in the backtrans-nmt setup can be detrimental to performance (see Table TABREF32 ): translation degrades in French for all metrics. Adding GANs on top of the pseudo-encoder was not able to make up for the degradation observed in French, but allowed the German system to slightly outperform backtrans-nmt. Even though this setup is unrealistic and overly costly, it shows that GANs are actually helping even good systems. Using Target Language Models  In this section, we compare the previous methods with the use of a target side Language Model (LM). Several proposals exist in the literature to integrate LMs in NMT: for instance, BIBREF3 strengthen the decoder by integrating an extra, source independent, RNN layer in a conventional NMT architecture. Training is performed either with parallel, or monolingual data. In the latter case, word prediction only relies on the source independent part of the network. LM Setup We have followed BIBREF29 and reimplemented their deep-fusion technique. It requires to first independently learn a RNN-LM on the in-domain target data with a cross-entropy objective; then to train the optimal combination of the translation and the language models by adding the hidden state of the RNN-LM as an additional input to the softmax layer of the decoder. Our RNN-LMs are trained using dl4mt with the target side of the parallel data and the Europarl corpus (about 6M sentences for both French and German), using a one-layer GRU with the same dimension as the MT decoder (1024). LM Results Results are in Table TABREF33 . They show that deep-fusion hardly improves the Europarl results, while we obtain about +0.6 BLEU over the baseline on newstest-2014 for both languages. deep-fusion differs from stupid BT in that the model is not directly optimized on the in-domain data, but uses the LM trained on Europarl to maximize the likelihood of the out-of-domain training data. Therefore, no specific improvement is to be expected in terms of domain adaptation, and the performance increases in the more general domain. Combining deep-fusion and copy-marked + noise + GANs brings slight improvements on the German in-domain test sets, and performance out of the domain remains near the baseline level. Re-analyzing the effects of BT  As a follow up of previous discussions, we analyze the effect of BT on the internals of the network. Arguably, using a copy of the target sentence instead of a natural source should not be helpful for the encoder, but is it also the case with a strong BT? What are the effects on the attention model? Parameter freezing protocol To investigate these questions, we run the same fine-tuning using the copy-marked, backtrans-nmt and backtrans-nmt setups. Note that except for the last one, all training scenarios have access to same target training data. We intend to see whether the overall performance of the NMT system degrades when we selectively freeze certain sets of parameters, meaning that they are not updated during fine-tuning. Results BLEU scores are in Table TABREF39 . The backtrans-nmt setup is hardly impacted by selective updates: updating the only decoder leads to a degradation of at most 0.2 BLEU. For copy-marked, we were not able to freeze the source embeddings, since these are initialized when fine-tuning begins and therefore need to be trained. We observe that freezing the encoder and/or the attention parameters has no impact on the English-German system, whereas it slightly degrades the English-French one. This suggests that using artificial sources, even of the poorest quality, has a positive impact on all the components of the network, which makes another big difference with the LM integration scenario. The largest degradation is for natural, where the model is prevented from learning from informative source sentences, which leads to a decrease of 0.4 to over 1.0 BLEU. We assume from these experiments that BT impacts most of all the decoder, and learning to encode a pseudo-source, be it a copy or an actual back-translation, only marginally helps to significantly improve the quality. Finally, in the fwdtrans-nmt setup, freezing the decoder does not seem to harm learning with a natural source. Related work The literature devoted to the use of monolingual data is large, and quickly expanding. We already alluded to several possible ways to use such data: using back- or forward-translation or using a target language model. The former approach is mostly documented in BIBREF2 , and recently analyzed in BIBREF19 , which focus on fully artificial settings as well as pivot-based artificial data; and BIBREF4 , which studies the effects of increasing the size of BT data. The studies of BIBREF20 , BIBREF19 also consider forward translation and BIBREF21 expand these results to domain adaptation scenarios. Our results are complementary to these earlier studies. As shown above, many alternatives to BT exist. The most obvious is to use target LMs BIBREF3 , BIBREF29 , as we have also done here; but attempts to improve the encoder using multi-task learning also exist BIBREF30 . This investigation is also related to recent attempts to consider supplementary data with a valid target side, such as multi-lingual NMT BIBREF31 , where source texts in several languages are fed in the same encoder-decoder architecture, with partial sharing of the layers. This is another realistic scenario where additional resources can be used to selectively improve parts of the model. Round trip training is another important source of inspiration, as it can be viewed as a way to use BT to perform semi-unsupervised BIBREF32 or unsupervised BIBREF33 training of NMT. The most convincing attempt to date along these lines has been proposed by BIBREF26 , who propose to use GANs to mitigate the difference between artificial and natural data. Conclusion  In this paper, we have analyzed various ways to integrate monolingual data in an NMT framework, focusing on their impact on quality and domain adaptation. While confirming the effectiveness of BT, our study also proposed significantly cheaper ways to improve the baseline performance, using a slightly modified copy of the target, instead of its full BT. When no high quality BT is available, using GANs to make the pseudo-source sentences closer to natural source sentences is an efficient solution for domain adaptation. To recap our answers to our initial questions: the quality of BT actually matters for NMT (cf. § SECREF8 ) and it seems that, even though artificial source are lexically less diverse and syntactically complex than real sentence, their monotonicity is a facilitating factor. We have studied cheaper alternatives and found out that copies of the target, if properly noised (§ SECREF4 ), and even better, if used with GANs, could be almost as good as low quality BTs (§ SECREF5 ): BT is only worth its cost when good BT can be generated. Finally, BT seems preferable to integrating external LM - at least in our data condition (§ SECREF6 ). Further experiments with larger LMs are needed to confirm this observation, and also to evaluate the complementarity of both strategies. More work is needed to better understand the impact of BT on subparts of the network (§ SECREF7 ). In future work, we plan to investigate other cheap ways to generate artificial data. The experimental setup we proposed may also benefit from a refining of the data selection strategies to focus on the most useful monolingual sentences.
what language pairs are explored?
b6f7fadaa1bb828530c2d6780289f12740229d84
what Ġlanguage Ġpairs Ġare Ġexplored ?
English - German , ĠEnglish - French .
<s> what Ġlanguage Ġpairs Ġare Ġexplored ? </s> Introduction Ġ </s> The Ġnew Ġgeneration Ġof ĠNeural ĠMachine ĠTranslation Ġ( N MT ) Ġsystems Ġis Ġknown Ġto Ġbe Ġextremely Ġdata Ġhungry ĠB IB REF 0 Ġ. ĠYet , Ġmost Ġexisting ĠN MT Ġtraining Ġpipelines Ġfail Ġto Ġfully Ġtake Ġadvantage Ġof Ġthe Ġvery Ġlarge Ġvolume Ġof Ġmon oling ual Ġsource Ġand / or Ġparallel Ġdata Ġthat Ġis Ġoften Ġavailable . ĠMaking Ġa Ġbetter Ġuse Ġof Ġdata Ġis Ġparticularly Ġcritical Ġin Ġdomain Ġadaptation Ġscenarios , Ġwhere Ġparallel Ġadaptation Ġdata Ġis Ġusually Ġassumed Ġto Ġbe Ġsmall Ġin Ġcomparison Ġto Ġout - of - domain Ġparallel Ġdata , Ġor Ġto Ġin - domain Ġmon oling ual Ġtexts . ĠThis Ġsituation Ġsharply Ġcontrasts Ġwith Ġthe Ġprevious Ġgeneration Ġof Ġstatistical ĠMT Ġengines ĠB IB REF 1 Ġ, Ġwhich Ġcould Ġseamlessly Ġintegrate Ġvery Ġlarge Ġamounts Ġof Ġnon - par allel Ġdocuments , Ġusually Ġwith Ġa Ġlarge Ġpositive Ġeffect Ġon Ġtranslation Ġquality . </s> Such Ġobservations Ġhave Ġbeen Ġmade Ġrepeatedly Ġand Ġhave Ġled Ġto Ġmany Ġinnovative Ġtechniques Ġto Ġintegrate Ġmon oling ual Ġdata Ġin ĠN MT , Ġthat Ġwe Ġreview Ġshortly . ĠThe Ġmost Ġsuccessful Ġapproach Ġto Ġdate Ġis Ġthe Ġproposal Ġof ĠB IB REF 2 Ġ, Ġwho Ġuse Ġmon oling ual Ġtarget Ġtexts Ġto Ġgenerate Ġartificial Ġparallel Ġdata Ġvia Ġbackward Ġtranslation Ġ( BT ). ĠThis Ġtechnique Ġhas Ġsince Ġproven Ġeffective Ġin Ġmany Ġsubsequent Ġstudies . ĠIt Ġis Ġhowever Ġvery Ġcomput ationally Ġcostly , Ġtypically Ġrequiring Ġto Ġtranslate Ġlarge Ġsets Ġof Ġdata . ĠD etermin ing Ġthe ĠâĢ ľ right âĢ Ŀ Ġamount Ġ( and Ġquality ) Ġof ĠBT Ġdata Ġis Ġanother Ġopen Ġissue , Ġbut Ġwe Ġobserve Ġthat Ġexperiments Ġreported Ġin Ġthe Ġliterature Ġonly Ġuse Ġa Ġsubset Ġof Ġthe Ġavailable Ġmon oling ual Ġresources . ĠThis Ġsuggests Ġthat Ġstandard Ġrecipes Ġfor ĠBT Ġmight Ġbe Ġsub - opt imal . </s> This Ġpaper Ġaims Ġto Ġbetter Ġunderstand Ġthe Ġstrengths Ġand Ġweaknesses Ġof ĠBT Ġand Ġto Ġdesign Ġmore Ġprincipled Ġtechniques Ġto Ġimprove Ġits Ġeffects . ĠMore Ġspecifically , Ġwe Ġseek Ġto Ġanswer Ġthe Ġfollowing Ġquestions : Ġsince Ġthere Ġare Ġmany Ġways Ġto Ġgenerate Ġpseudo Ġparallel Ġcorpor a , Ġhow Ġimportant Ġis Ġthe Ġquality Ġof Ġthis Ġdata Ġfor ĠMT Ġperformance ? ĠWhich Ġproperties Ġof Ġback - trans lated Ġsentences Ġactually Ġmatter Ġfor ĠMT Ġquality ? ĠDoes ĠBT Ġact Ġas Ġsome Ġkind Ġof Ġregular izer ĠB IB REF 3 Ġ? ĠCan ĠBT Ġbe Ġefficiently Ġsimulated ? ĠDoes ĠBT Ġdata Ġplay Ġthe Ġsame Ġrole Ġas Ġa Ġtarget - side Ġlanguage Ġmodeling , Ġor Ġare Ġthey Ġcomplementary ? ĠBT Ġis Ġoften Ġused Ġfor Ġdomain Ġadaptation : Ġcan Ġthe Ġeffect Ġof Ġhaving Ġmore Ġin - domain Ġdata Ġbe Ġsorted Ġout Ġfrom Ġthe Ġmere Ġincrease Ġof Ġtraining Ġmaterial ĠB IB REF 2 Ġ? ĠFor Ġstudies Ġrelated Ġto Ġthe Ġimpact Ġof Ġvarying Ġthe Ġsize Ġof ĠBT Ġdata , Ġwe Ġrefer Ġthe Ġreaders Ġto Ġthe Ġrecent Ġwork Ġof ĠB IB REF 4 Ġ. </s> To Ġanswer Ġthese Ġquestions , Ġwe Ġhave Ġreim ple mented Ġseveral Ġstrategies Ġto Ġuse Ġmon oling ual Ġdata Ġin ĠN MT Ġand Ġhave Ġrun Ġexperiments Ġon Ġtwo Ġlanguage Ġpairs Ġin Ġa Ġvery Ġcontrolled Ġsetting Ġ( see ĠÂ§ ĠSEC REF 2 Ġ). ĠOur Ġmain Ġresults Ġ( see ĠÂ§ ĠSEC REF 4 Ġand ĠÂ§ ĠSEC REF 5 Ġ) Ġsuggest Ġpromising Ġdirections Ġfor Ġefficient Ġdomain Ġadaptation Ġwith Ġcheaper Ġtechniques Ġthan Ġconventional ĠBT . </s> In - domain Ġand Ġout - of - domain Ġdata Ġ </s> We Ġare Ġmostly Ġinterested Ġwith Ġthe Ġfollowing Ġtraining Ġscenario : Ġa Ġlarge Ġout - of - domain Ġparallel Ġcorpus , Ġand Ġlimited Ġmon oling ual Ġin - domain Ġdata . ĠWe Ġfocus Ġhere Ġon Ġthe ĠEuro par l Ġdomain , Ġfor Ġwhich Ġwe Ġhave Ġample Ġdata Ġin Ġseveral Ġlanguages , Ġand Ġuse Ġas Ġin - domain Ġtraining Ġdata Ġthe ĠEuro par l Ġcorpus ĠB IB REF 5 Ġfor Ġtwo Ġtranslation Ġdirections : ĠEnglish ĠIN LINE FORM 0 ĠGerman Ġand ĠEnglish ĠIN LINE FORM 1 ĠFrench . ĠAs Ġwe Ġstudy Ġthe Ġbenefits Ġof Ġmon oling ual Ġdata , Ġmost Ġof Ġour Ġexperiments Ġonly Ġuse Ġthe Ġtarget Ġside Ġof Ġthis Ġcorpus . ĠThe Ġrationale Ġfor Ġchoosing Ġthis Ġdomain Ġis Ġto Ġ( i ) Ġto Ġperform Ġlarge Ġscale Ġcomparisons Ġof Ġsynthetic Ġand Ġnatural Ġparallel Ġcorpor a ; Ġ( ii ) Ġto Ġstudy Ġthe Ġeffect Ġof ĠBT Ġin Ġa Ġwell - defined Ġdomain - adapt ation Ġscenario . ĠFor Ġboth Ġlanguage Ġpairs , Ġwe Ġuse Ġthe ĠEuro par l Ġtests Ġfrom Ġ2007 Ġand Ġ2008 Ġfor Ġevaluation Ġpurposes , Ġkeeping Ġtest Ġ2006 Ġfor Ġdevelopment . ĠWhen Ġmeasuring Ġout - of - domain Ġperformance , Ġwe Ġwill Ġuse Ġthe ĠW MT Ġnew st est Ġ2014 . </s> N MT Ġsetups Ġand Ġperformance Ġ </s> Our Ġbaseline ĠN MT Ġsystem Ġimplements Ġthe Ġattention al Ġenc oder - dec oder Ġapproach ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġas Ġimplemented Ġin ĠNem atus ĠB IB REF 8 Ġon Ġ4 Ġmillion Ġout - of - domain Ġparallel Ġsentences . ĠFor ĠFrench Ġwe Ġuse Ġsamples Ġfrom ĠNews - Comment ary - 11 Ġand ĠWikipedia Ġfrom ĠW MT Ġ2014 Ġshared Ġtranslation Ġtask , Ġas Ġwell Ġas Ġthe ĠMulti - UN ĠB IB REF 9 Ġand ĠEU - Book shop ĠB IB REF 10 Ġcorpor a . ĠFor ĠGerman , Ġwe Ġuse Ġsamples Ġfrom ĠNews - Comment ary - 11 , ĠRapid , ĠCommon - C rawl Ġ( W MT Ġ2017 ) Ġand ĠMulti - UN Ġ( see Ġtable ĠT AB REF 5 Ġ). ĠB ilingual ĠB PE Ġunits ĠB IB REF 11 Ġare Ġlearned Ġwith Ġ50 k Ġmerge Ġoperations , Ġyielding Ġvoc ab ular ies Ġof Ġabout Ġrespectively Ġ32 k Ġand Ġ36 k Ġfor ĠEnglish ĠIN LINE FORM 0 ĠFrench Ġand Ġ32 k Ġand Ġ44 k Ġfor ĠEnglish ĠIN LINE FORM 1 ĠGerman . </s> Both Ġsystems Ġuse Ġ512 - dimensional Ġword Ġembed d ings Ġand Ġa Ġsingle Ġhidden Ġlayer Ġwith Ġ1024 Ġcells . ĠThey Ġare Ġoptimized Ġusing ĠAdam ĠB IB REF 12 Ġand Ġearly Ġstopped Ġaccording Ġto Ġthe Ġvalidation Ġperformance . ĠTraining Ġlasted Ġfor Ġabout Ġthree Ġweeks Ġon Ġan ĠNvidia ĠK 80 ĠGPU Ġcard . </s> System s Ġgenerating Ġback - trans lated Ġdata Ġare Ġtrained Ġusing Ġthe Ġsame Ġout - of - domain Ġcorpus , Ġwhere Ġwe Ġsimply Ġexchange Ġthe Ġsource Ġand Ġtarget Ġsides . ĠThey Ġare Ġfurther Ġdocumented Ġin ĠÂ§ ĠSEC REF 8 Ġ. ĠFor Ġthe Ġsake Ġof Ġcomparison , Ġwe Ġalso Ġtrain Ġa Ġsystem Ġthat Ġhas Ġaccess Ġto Ġa Ġlarge Ġbatch Ġof Ġin - domain Ġparallel Ġdata Ġfollowing Ġthe Ġstrategy Ġoften Ġreferred Ġto Ġas ĠâĢ ľ fine - tun ing âĢ Ŀ : Ġupon Ġconvergence Ġof Ġthe Ġbaseline Ġmodel , Ġwe Ġresume Ġtraining Ġwith Ġa Ġ2 M Ġsentence Ġin - domain Ġcorpus Ġmixed Ġwith Ġan Ġequal Ġamount Ġof Ġrandomly Ġselected Ġout - of - domain Ġnatural Ġsentences , Ġwith Ġthe Ġsame Ġarchitecture Ġand Ġtraining Ġparameters , Ġrunning Ġvalidation Ġevery Ġ2000 Ġupdates Ġwith Ġa Ġpatience Ġof Ġ10 . ĠSince ĠB PE Ġunits Ġare Ġselected Ġbased Ġonly Ġon Ġthe Ġout - of - domain Ġstatistics , Ġfine - tun ing Ġis Ġperformed Ġon Ġsentences Ġthat Ġare Ġslightly Ġlonger Ġ( ie . Ġthey Ġcontain Ġmore Ġunits ) Ġthan Ġfor Ġthe Ġinitial Ġtraining . ĠThis Ġsystem Ġdefines Ġan Ġupper - bound Ġof Ġthe Ġtranslation Ġperformance Ġand Ġis Ġden oted Ġbelow Ġas Ġnatural . </s> Our Ġbaseline Ġand Ġto pl ine Ġresults Ġare Ġin ĠTable ĠT AB REF 6 Ġ, Ġwhere Ġwe Ġmeasure Ġtranslation Ġperformance Ġusing ĠB LE U ĠB IB REF 13 Ġ, ĠBE ER ĠB IB REF 14 Ġ( higher Ġis Ġbetter ) Ġand Ġchar ac TER ĠB IB REF 15 Ġ( small er Ġis Ġbetter ). ĠAs Ġthey Ġare Ġtrained Ġfrom Ġmuch Ġsmaller Ġamounts Ġof Ġdata Ġthan Ġcurrent Ġsystems , Ġthese Ġbas elines Ġare Ġnot Ġquite Ġcompetitive Ġto Ġtoday 's Ġbest Ġsystem , Ġbut Ġstill Ġrepresent Ġserious Ġbas elines Ġfor Ġthese Ġdatasets . ĠGiven Ġour Ġsetups , Ġfine - tun ing Ġwith Ġin - domain Ġnatural Ġdata Ġimproves ĠB LE U Ġby Ġalmost Ġ4 Ġpoints Ġfor Ġboth Ġtranslation Ġdirections Ġon Ġin - domain Ġtests ; Ġit Ġalso Ġimproves , Ġalbeit Ġby Ġa Ġsmaller Ġmargin , Ġthe ĠB LE U Ġscore Ġof Ġthe Ġout - of - domain Ġtests . </s> Using Ġartificial Ġparallel Ġdata Ġin ĠN MT Ġ </s> A Ġsimple Ġway Ġto Ġuse Ġmon oling ual Ġdata Ġin ĠMT Ġis Ġto Ġturn Ġit Ġinto Ġsynthetic Ġparallel Ġdata Ġand Ġlet Ġthe Ġtraining Ġprocedure Ġrun Ġas Ġusual ĠB IB REF 16 Ġ. ĠIn Ġthis Ġsection , Ġwe Ġexplore Ġvarious Ġways Ġto Ġimplement Ġthis Ġstrategy . ĠWe Ġfirst Ġreproduce Ġresults Ġof ĠB IB REF 2 Ġwith ĠBT Ġof Ġvarious Ġqualities , Ġthat Ġwe Ġthen Ġanalyze Ġthoroughly . </s> The Ġquality Ġof ĠBack - Translation Ġ </s> BT Ġrequires Ġthe Ġavailability Ġof Ġan ĠMT Ġsystem Ġin Ġthe Ġreverse Ġtranslation Ġdirection . ĠWe Ġconsider Ġhere Ġthree ĠMT Ġsystems Ġof Ġincreasing Ġquality : </s> back trans - bad : Ġthis Ġis Ġa Ġvery Ġpoor ĠSM T Ġsystem Ġtrained Ġusing Ġonly Ġ50 k Ġparallel Ġsentences Ġfrom Ġthe Ġout - of - domain Ġdata , Ġand Ġno Ġadditional Ġmon oling ual Ġdata . ĠFor Ġthis Ġsystem Ġas Ġfor Ġthe Ġnext Ġone , Ġwe Ġuse ĠMoses ĠB IB REF 17 Ġout - of - the - box , Ġcomputing Ġalign ments Ġwith ĠFast align ĠB IB REF 18 Ġ, Ġwith Ġa Ġminimal Ġpre - processing Ġ( basic Ġtoken ization ). ĠThis Ġsetting Ġprovides Ġus Ġwith Ġa Ġpessimistic Ġestimate Ġof Ġwhat Ġwe Ġcould Ġget Ġin Ġlow - resource Ġconditions . </s> back trans - good : Ġthese Ġare Ġmuch Ġlarger ĠSM T Ġsystems , Ġwhich Ġuse Ġthe Ġsame Ġparallel Ġdata Ġas Ġthe Ġbaseline ĠN MT s Ġ( see ĠÂ§ ĠSEC REF 4 Ġ) Ġand Ġall Ġthe ĠEnglish Ġmon oling ual Ġdata Ġavailable Ġfor Ġthe ĠW MT Ġ2017 Ġshared Ġtasks , Ġtot alling Ġapproximately Ġ174 M Ġsentences . ĠThese Ġsystems Ġare Ġstrong , Ġyet Ġrelatively Ġcheap Ġto Ġbuild . </s> back trans - n mt : Ġthese Ġare Ġthe Ġbest ĠN MT Ġsystems Ġwe Ġcould Ġtrain , Ġusing Ġsettings Ġthat Ġreplicate Ġthe Ġforward Ġtranslation ĠN MT s . </s> Note Ġthat Ġwe Ġdo Ġnot Ġuse Ġany Ġin - domain Ġ( Euro par l ) Ġdata Ġto Ġtrain Ġthese Ġsystems . ĠTheir Ġperformance Ġis Ġreported Ġin ĠTable ĠT AB REF 7 Ġ, Ġwhere Ġwe Ġobserve Ġa Ġ12 ĠB LE U Ġpoints Ġgap Ġbetween Ġthe Ġworst Ġand Ġbest Ġsystems Ġ( for Ġboth Ġlanguages ). </s> As Ġnoted Ġeg . Ġin ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, Ġartificial Ġparallel Ġdata Ġobtained Ġthrough Ġforward - translation Ġ( FT ) Ġcan Ġalso Ġprove Ġadvantageous Ġand Ġwe Ġalso Ġconsider Ġa ĠFT Ġsystem Ġ( f wd trans - n mt ): Ġin Ġthis Ġcase Ġthe Ġtarget Ġside Ġof Ġthe Ġcorpus Ġis Ġartificial Ġand Ġis Ġgenerated Ġusing Ġthe Ġbaseline ĠN MT Ġapplied Ġto Ġa Ġnatural Ġsource . </s> Our Ġresults Ġ( see ĠTable ĠT AB REF 6 Ġ) Ġreplicate Ġthe Ġfindings Ġof ĠB IB REF 2 Ġ: Ġlarge Ġgains Ġcan Ġbe Ġobtained Ġfrom ĠBT Ġ( n early ĠIN LINE FORM 0 ĠB LE U Ġin ĠFrench Ġand ĠGerman ); Ġbetter Ġartificial Ġdata Ġyields Ġbetter Ġtranslation Ġsystems . ĠInterestingly , Ġour Ġbest ĠMoses Ġsystem Ġis Ġalmost Ġas Ġgood Ġas Ġthe ĠN MT Ġand Ġan Ġorder Ġof Ġmagnitude Ġfaster Ġto Ġtrain . ĠImprovements Ġobtained Ġwith Ġthe Ġbad Ġsystem Ġare Ġmuch Ġsmaller ; Ġcontrary Ġto Ġthe Ġbetter ĠMT s , Ġthis Ġsystem Ġis Ġeven Ġdetrimental Ġfor Ġthe Ġout - of - domain Ġtest . </s> G ains Ġwith Ġforward Ġtranslation Ġare Ġsignificant , Ġas Ġin ĠB IB REF 21 Ġ, Ġalbeit Ġabout Ġhalf Ġas Ġgood Ġas Ġwith ĠBT , Ġand Ġresult Ġin Ġsmall Ġimprovements Ġfor Ġthe Ġin - domain Ġand Ġfor Ġthe Ġout - of - domain Ġtests . ĠExper iments Ġcombining Ġforward Ġand Ġbackward Ġtranslation Ġ( back f wd trans - n mt ), Ġeach Ġusing Ġa Ġhalf Ġof Ġthe Ġavailable Ġartificial Ġdata , Ġdo Ġnot Ġoutper form Ġthe Ġbest ĠBT Ġresults . </s> We Ġfinally Ġnote Ġthe Ġlarge Ġremaining Ġdifference Ġbetween ĠBT Ġdata Ġand Ġnatural Ġdata , Ġeven Ġthough Ġthey Ġonly Ġdiffer Ġin Ġtheir Ġsource Ġside . ĠThis Ġshows Ġthat Ġat Ġleast Ġin Ġour Ġdomain - adapt ation Ġsettings , ĠBT Ġdoes Ġnot Ġreally Ġact Ġas Ġa Ġregular izer , Ġcontr arily Ġto Ġthe Ġfindings Ġof ĠB IB REF 4 Ġ, ĠB IB REF 11 Ġ. ĠFigure ĠFIG REF 13 Ġdisplays Ġthe Ġlearning Ġcurves Ġof Ġthese Ġtwo Ġsystems . ĠWe Ġobserve Ġthat Ġback trans - n mt Ġimproves Ġquickly Ġin Ġthe Ġearliest Ġupdates Ġand Ġthen Ġstays Ġhorizontal , Ġwhereas Ġnatural Ġcontinues Ġimproving , Ġeven Ġafter Ġ400 k Ġupdates . ĠTherefore ĠBT Ġdoes Ġnot Ġhelp Ġto Ġavoid Ġover fitting , Ġit Ġactually Ġencourages Ġit , Ġwhich Ġmay Ġbe Ġdue ĠâĢ ľ eas ier âĢ Ŀ Ġtraining Ġexamples Ġ( cf . ĠÂ§ ĠSEC REF 15 Ġ). </s> Pro perties Ġof Ġback - trans lated Ġdata Ġ </s> Comp aring Ġthe Ġnatural Ġand Ġartificial Ġsources Ġof Ġour Ġparallel Ġdata Ġwr t . Ġseveral Ġlinguistic Ġand Ġdistribution al Ġproperties , Ġwe Ġobserve Ġthat Ġ( see ĠFig . ĠFIG REF 21 Ġ- ĠFIG REF 22 Ġ): </s> art ificial Ġsources Ġare Ġon Ġaverage Ġshorter Ġthan Ġnatural Ġones : Ġwhen Ġusing ĠBT , Ġcases Ġwhere Ġthe Ġsource Ġis Ġshorter Ġthan Ġthe Ġtarget Ġare Ġrare r ; Ġcases Ġwhen Ġthey Ġhave Ġthe Ġsame Ġlength Ġare Ġmore Ġfrequent . </s> automatic Ġword Ġalign ments Ġbetween Ġartificial Ġsources Ġtend Ġto Ġbe Ġmore Ġmon ot onic Ġthan Ġwhen Ġusing Ġnatural Ġsources , Ġas Ġmeasured Ġby Ġthe Ġaverage ĠKendall ĠIN LINE FORM 0 Ġof Ġsource - target Ġalign ments ĠB IB REF 22 Ġ: Ġfor ĠFrench - English Ġthe Ġrespective Ġnumbers Ġare Ġ0 . 048 Ġ( natural ) Ġand Ġ0 . 018 Ġ( art ificial ); Ġfor ĠGerman - English Ġ0 . 0 68 Ġand Ġ0 . 05 3 . ĠUsing Ġmore Ġmon ot onic Ġsentence Ġpairs Ġturns Ġout Ġto Ġbe Ġa Ġfacilitating Ġfactor Ġfor ĠN MT , Ġas Ġalso Ġnoted Ġby ĠB IB REF 20 Ġ. </s> sy nt act ically , Ġartificial Ġsources Ġare Ġsimpler Ġthan Ġreal Ġdata ; ĠWe Ġobserve Ġsignificant Ġdifferences Ġin Ġthe Ġdistributions Ġof Ġtree Ġdepths . </s> dist ribution ally , Ġplain Ġword Ġoccurrences Ġin Ġartificial Ġsources Ġare Ġmore Ġconcentrated ; Ġthis Ġalso Ġtranslates Ġinto Ġboth Ġa Ġslower Ġincrease Ġof Ġthe Ġnumber Ġof Ġtypes Ġwr t . Ġthe Ġnumber Ġof Ġsentences Ġand Ġa Ġsmaller Ġnumber Ġof Ġrare Ġevents . </s> The Ġintuition Ġis Ġthat Ġproperties Ġ( i ) Ġand Ġ( ii ) Ġshould Ġhelp Ġtranslation Ġas Ġcompared Ġto Ġnatural Ġsource , Ġwhile Ġproperty Ġ( iv ) Ġshould Ġbe Ġdetrimental . ĠWe Ġchecked Ġ( ii ) Ġby Ġbuilding Ġsystems Ġwith Ġonly Ġ10 M Ġwords Ġfrom Ġthe Ġnatural Ġparallel Ġdata Ġselecting Ġthese Ġdata Ġeither Ġrandomly Ġor Ġbased Ġon Ġthe Ġregular ity Ġof Ġtheir Ġword Ġalign ments . ĠResults Ġin ĠTable ĠT AB REF 23 Ġshow Ġthat Ġthe Ġlatter Ġis Ġmuch Ġpreferable Ġfor Ġthe Ġoverall Ġperformance . ĠThis Ġmight Ġexplain Ġthat Ġthe Ġmostly Ġmon ot onic ĠBT Ġfrom ĠMoses Ġare Ġalmost Ġas Ġgood Ġas Ġthe Ġfluid ĠBT Ġfrom ĠN MT Ġand Ġthat Ġboth Ġboost Ġthe Ġbaseline . </s> St upid ĠBack - Translation Ġ </s> We Ġnow Ġanalyze Ġthe Ġeffect Ġof Ġusing Ġmuch Ġsimpler Ġdata Ġgeneration Ġschemes , Ġwhich Ġdo Ġnot Ġrequire Ġthe Ġavailability Ġof Ġa Ġbackward Ġtranslation Ġengine . </s> Set ups </s> We Ġuse Ġthe Ġfollowing Ġcheap Ġways Ġto Ġgenerate Ġpseudo - source Ġtexts : </s> copy : Ġin Ġthis Ġsetting , Ġthe Ġsource Ġside Ġis Ġa Ġmere Ġcopy Ġof Ġthe Ġtarget - side Ġdata . ĠSince Ġthe Ġsource Ġvocabulary Ġof Ġthe ĠN MT Ġis Ġfixed , Ġcopying Ġthe Ġtarget Ġsentences Ġcan Ġcause Ġthe Ġoccurrence Ġof ĠO OV s . ĠTo Ġavoid Ġthis Ġsituation , ĠB IB REF 24 Ġdecom pose Ġthe Ġtarget Ġwords Ġinto Ġsource - side Ġunits Ġto Ġmake Ġthe Ġcopy Ġlook Ġlike Ġsource Ġsentences . ĠEach ĠO OV Ġfound Ġin Ġthe Ġcopy Ġis Ġsplit Ġinto Ġsmaller Ġunits Ġuntil Ġall Ġthe Ġresulting Ġchunks Ġare Ġin Ġthe Ġsource Ġvocabulary . </s> copy - marked : Ġanother Ġway Ġto Ġintegrate Ġcopies Ġwithout Ġhaving Ġto Ġdeal Ġwith ĠO OV s Ġis Ġto Ġaugment Ġthe Ġsource Ġvocabulary Ġwith Ġa Ġcopy Ġof Ġthe Ġtarget Ġvocabulary . ĠIn Ġthis Ġsetup , ĠB IB REF 25 Ġensure Ġthat Ġboth Ġvoc ab ular ies Ġnever Ġoverlap Ġby Ġmarking Ġthe Ġtarget Ġword Ġcopies Ġwith Ġa Ġspecial Ġlanguage Ġidentifier . ĠTherefore Ġthe ĠEnglish Ġword Ġresume Ġcannot Ġbe Ġconfused Ġwith Ġthe Ġhom ographic ĠFrench Ġword , Ġwhich Ġis Ġmarked Ġ@ fr @ res ume . </s> copy - d ummies : Ġinstead Ġof Ġusing Ġactual Ġcopies , Ġwe Ġreplace Ġeach Ġword Ġwith ĠâĢ ľ d ummy âĢ Ŀ Ġtokens . ĠWe Ġuse Ġthis Ġunrealistic Ġsetup Ġto Ġobserve Ġthe Ġtraining Ġover Ġnoisy Ġand Ġhardly Ġinformative Ġsource Ġsentences . </s> We Ġthen Ġuse Ġthe Ġprocedures Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġ, Ġexcept Ġthat Ġthe Ġpseudo - source Ġembed d ings Ġin Ġthe Ġcopy - marked Ġsetup Ġare Ġpret rained Ġfor Ġthree Ġepoch s Ġon Ġthe Ġin - domain Ġdata , Ġwhile Ġall Ġremaining Ġparameters Ġare Ġfrozen . ĠThis Ġprevents Ġrandom Ġparameters Ġfrom Ġhurting Ġthe Ġalready Ġtrained Ġmodel . </s> Copy + mark ing + no ise Ġis Ġnot Ġso Ġstupid </s> We Ġobserve Ġthat Ġthe Ġcopy Ġsetup Ġhas Ġonly Ġa Ġsmall Ġimpact Ġon Ġthe ĠEnglish - French Ġsystem , Ġfor Ġwhich Ġthe Ġbaseline Ġis Ġalready Ġstrong . ĠThis Ġis Ġless Ġtrue Ġfor ĠEnglish - German Ġwhere Ġsimple Ġcopies Ġyield Ġa Ġsignificant Ġimprovement . ĠPerformance Ġdrops Ġfor Ġboth Ġlanguage Ġpairs Ġin Ġthe Ġcopy - d ummies Ġsetup . </s> We Ġachieve Ġour Ġbest Ġgains Ġwith Ġthe Ġcopy - marked Ġsetup , Ġwhich Ġis Ġthe Ġbest Ġway Ġto Ġuse Ġa Ġcopy Ġof Ġthe Ġtarget Ġ( although Ġthe Ġperformance Ġon Ġthe Ġout - of - domain Ġtests Ġis Ġat Ġmost Ġthe Ġsame Ġas Ġthe Ġbaseline ). ĠSuch Ġgains Ġmay Ġlook Ġsurprising , Ġsince Ġthe ĠN MT Ġmodel Ġdoes Ġnot Ġneed Ġto Ġlearn Ġto Ġtranslate Ġbut Ġonly Ġto Ġcopy Ġthe Ġsource . ĠThis Ġis Ġindeed Ġwhat Ġhappens : Ġto Ġconfirm Ġthis , Ġwe Ġbuilt Ġa Ġfake Ġtest Ġset Ġhaving Ġidentical Ġsource Ġand Ġtarget Ġside Ġ( in ĠFrench ). ĠThe Ġaverage Ġcross - ent ropy Ġfor Ġthis Ġtest Ġset Ġis Ġ0 . 33 , Ġvery Ġclose Ġto Ġ0 , Ġto Ġbe Ġcompared Ġwith Ġan Ġaverage Ġcost Ġof Ġ58 . 52 Ġwhen Ġwe Ġprocess Ġan Ġactual Ġsource Ġ( in ĠEnglish ). ĠThis Ġmeans Ġthat Ġthe Ġmodel Ġhas Ġlearned Ġto Ġcopy Ġwords Ġfrom Ġsource Ġto Ġtarget Ġwith Ġno Ġdifficulty , Ġeven Ġfor Ġsentences Ġnot Ġseen Ġin Ġtraining . ĠA Ġfollow - up Ġquestion Ġis Ġwhether Ġtraining Ġa Ġcopying Ġtask Ġinstead Ġof Ġa Ġtranslation Ġtask Ġlimits Ġthe Ġimprovement : Ġwould Ġthe ĠN MT Ġlearn Ġbetter Ġif Ġthe Ġtask Ġwas Ġharder ? ĠTo Ġmeasure Ġthis , Ġwe Ġintroduce Ġnoise Ġin Ġthe Ġtarget Ġsentences Ġcopied Ġonto Ġthe Ġsource , Ġfollowing Ġthe Ġprocedure Ġof ĠB IB REF 26 Ġ: Ġit Ġdelet es Ġrandom Ġwords Ġand Ġperforms Ġa Ġsmall Ġrandom Ġperm utation Ġof Ġthe Ġremaining Ġwords . ĠResults Ġ(+ ĠSource Ġnoise ) Ġshow Ġno Ġdifference Ġfor Ġthe ĠFrench Ġin - domain Ġtest Ġsets , Ġbut Ġbring Ġthe Ġout - of - domain Ġscore Ġto Ġthe Ġlevel Ġof Ġthe Ġbaseline . ĠFinally , Ġwe Ġobserve Ġa Ġsignificant Ġimprovement Ġon ĠGerman Ġin - domain Ġtest Ġsets , Ġcompared Ġto Ġthe Ġbaseline Ġ( about Ġ+ 1 . 5 ĠB LE U ). ĠThis Ġlast Ġsetup Ġis Ġeven Ġalmost Ġas Ġgood Ġas Ġthe Ġback trans - n mt Ġcondition Ġ( see ĠÂ§ ĠSEC REF 8 Ġ) Ġfor ĠGerman . ĠThis Ġshows Ġthat Ġlearning Ġto Ġre order Ġand Ġpredict Ġmissing Ġwords Ġcan Ġmore Ġeffectively Ġserve Ġour Ġpurposes Ġthan Ġsimply Ġlearning Ġto Ġcopy . </s> T ow ards Ġmore Ġnatural Ġpseudo - s ources Ġ </s> Integ rating Ġmon oling ual Ġdata Ġinto ĠN MT Ġcan Ġbe Ġas Ġeasy Ġas Ġcopying Ġthe Ġtarget Ġinto Ġthe Ġsource , Ġwhich Ġalready Ġgives Ġsome Ġimprovement ; Ġadding Ġnoise Ġmakes Ġthings Ġeven Ġbetter . ĠWe Ġnow Ġstudy Ġways Ġto Ġmake Ġpseudo - s ources Ġlook Ġmore Ġlike Ġnatural Ġdata , Ġusing Ġthe Ġframework Ġof ĠGener ative ĠAd vers arial ĠNetworks Ġ( GAN s ) ĠB IB REF 27 Ġ, Ġan Ġidea Ġborrowed Ġfrom ĠB IB REF 26 Ġ. </s> GAN Ġsetups </s> In Ġour Ġsetups , Ġwe Ġuse Ġa Ġmarked Ġtarget Ġcopy , Ġviewed Ġas Ġa Ġfake Ġsource , Ġwhich Ġa Ġgenerator Ġenc odes Ġso Ġas Ġto Ġfool Ġa Ġdiscrim inator Ġtrained Ġto Ġdistinguish Ġa Ġfake Ġfrom Ġa Ġnatural Ġsource . ĠOur Ġarchitecture Ġcontains Ġtwo Ġdistinct Ġenc od ers , Ġone Ġfor Ġthe Ġnatural Ġsource Ġand Ġone Ġfor Ġthe Ġpseudo - source . ĠThe Ġlatter Ġacts Ġas Ġthe Ġgenerator Ġ( ĠIN LINE FORM 0 Ġ) Ġin Ġthe ĠG AN Ġframework , Ġcomputing Ġa Ġrepresentation Ġof Ġthe Ġpseudo - source Ġthat Ġis Ġthen Ġinput Ġto Ġa Ġdiscrim inator Ġ( ĠIN LINE FORM 1 Ġ), Ġwhich Ġhas Ġto Ġsort Ġnatural Ġfrom Ġartificial Ġenc od ings . ĠIN LINE FORM 2 Ġassigns Ġa Ġprobability Ġof Ġa Ġsentence Ġbeing Ġnatural . </s> During Ġtraining , Ġthe Ġcost Ġof Ġthe Ġdiscrim inator Ġis Ġcomputed Ġover Ġtwo Ġbatches , Ġone Ġwith Ġnatural Ġ( out - of - domain ) Ġsentences ĠIN LINE FORM 0 Ġand Ġone Ġwith Ġ( in - domain ) Ġpseudo - sent ences ĠIN LINE FORM 1 Ġ. ĠThe Ġdiscrim inator Ġis Ġa Ġbid irection al - Rec urrent ĠNeural ĠNetwork Ġ( R NN ) Ġof Ġdimension Ġ1024 . ĠA ver aged Ġstates Ġare Ġpassed Ġto Ġa Ġsingle Ġfeed - forward Ġlayer , Ġto Ġwhich Ġa Ġs igm oid Ġis Ġapplied . ĠIt Ġinputs Ġenc od ings Ġof Ġnatural Ġ( ĠIN LINE FORM 2 Ġ) Ġand Ġpseudo - sent ences Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġis Ġtrained Ġto Ġoptimize : </s> IN LINE FORM 0 </s> IN LINE FORM 0 Ġ' s Ġparameters Ġare Ġupdated Ġto Ġmaxim ally Ġfool ĠIN LINE FORM 1 Ġ, Ġthus Ġthe Ġloss ĠIN LINE FORM 2 Ġ: ĠIN LINE FORM 3 </s> Finally , Ġwe Ġkeep Ġthe Ġusual ĠMT Ġobjective . Ġ( ĠIN LINE FORM 0 Ġis Ġa Ġreal Ġor Ġpseudo - sent ence ): ĠIN LINE FORM 1 </s> We Ġthus Ġneed Ġto Ġtrain Ġthree Ġsets Ġof Ġparameters : ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ( MT Ġparameters ), Ġwith ĠIN LINE FORM 2 Ġ. ĠThe Ġpseudo - source Ġenc oder Ġand Ġembed d ings Ġare Ġupdated Ġwr t . Ġboth ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. ĠFollowing ĠB IB REF 28 Ġ, ĠIN LINE FORM 5 Ġis Ġupdated Ġonly Ġwhen ĠIN LINE FORM 6 Ġ' s Ġaccuracy Ġexceeds ĠIN LINE FORM 7 Ġ. ĠOn Ġthe Ġother Ġhand , ĠIN LINE FORM 8 Ġis Ġnot Ġupdated Ġwhen Ġits Ġaccuracy Ġexceeds ĠIN LINE FORM 9 Ġ. ĠAt Ġeach Ġupdate , Ġtwo Ġbatches Ġare Ġgenerated Ġfor Ġeach Ġtype Ġof Ġdata , Ġwhich Ġare Ġencoded Ġwith Ġthe Ġreal Ġor Ġpseudo - enc oder . ĠThe Ġenc oder Ġoutputs Ġserve Ġto Ġcompute ĠIN LINE FORM 10 Ġand ĠIN LINE FORM 11 Ġ. ĠFinally , Ġthe Ġpseudo - source Ġis Ġencoded Ġagain Ġ( once ĠIN LINE FORM 12 Ġis Ġupdated ), Ġboth Ġenc od ers Ġare Ġplugged Ġinto Ġthe Ġtranslation Ġmodel Ġand Ġthe ĠMT Ġcost Ġis Ġback - prop ag ated Ġdown Ġto Ġthe Ġreal Ġand Ġpseudo - word Ġembed d ings . ĠPse udo - enc oder Ġand Ġdiscrim inator Ġparameters Ġare Ġpre - trained Ġfor Ġ10 k Ġupdates . ĠAt Ġtest Ġtime , Ġthe Ġpseudo - enc oder Ġis Ġignored Ġand Ġinference Ġis Ġrun Ġas Ġusual . </s> GAN s Ġcan Ġhelp </s> Results Ġare Ġin ĠTable ĠT AB REF 32 Ġ, Ġassuming Ġthe Ġsame Ġfine - tun ing Ġprocedure Ġas Ġabove . ĠOn Ġtop Ġof Ġthe Ġcopy - marked Ġsetup , Ġour ĠG AN s Ġdo Ġnot Ġprovide Ġany Ġimprovement Ġin Ġboth Ġlanguage Ġpairs , Ġwith Ġthe Ġexception Ġof Ġa Ġsmall Ġimprovement Ġfor ĠEnglish - French Ġon Ġthe Ġout - of - domain Ġtest , Ġwhich Ġwe Ġunderstand Ġas Ġa Ġsign Ġthat Ġthe Ġmodel Ġis Ġmore Ġrobust Ġto Ġdomain Ġvariations , Ġjust Ġlike Ġwhen Ġadding Ġpseudo - source Ġnoise . ĠWhen Ġcombined Ġwith Ġnoise , Ġthe ĠFrench Ġmodel Ġyields Ġthe Ġbest Ġperformance Ġwe Ġcould Ġobtain Ġwith Ġstupid ĠBT Ġon Ġthe Ġin - domain Ġtests , Ġat Ġleast Ġin Ġterms Ġof ĠB LE U Ġand ĠBE ER . ĠOn Ġthe ĠNews Ġdomain , Ġwe Ġremain Ġclose Ġto Ġthe Ġbaseline Ġlevel , Ġwith Ġslight Ġimprovements Ġin ĠGerman . </s> A Ġfirst Ġobservation Ġis Ġthat Ġthis Ġmethod Ġbrings Ġstupid ĠBT Ġmodels Ġcloser Ġto Ġconventional ĠBT , Ġat Ġa Ġgreatly Ġreduced Ġcomputational Ġcost . ĠWhile ĠFrench Ġstill Ġremains Ġ0 . 4 Ġto Ġ1 . 0 ĠB LE U Ġbelow Ġvery Ġgood Ġback translation , Ġboth Ġapproaches Ġare Ġin Ġthe Ġsame Ġballpark Ġfor ĠGerman Ġ- Ġmay Ġbe Ġbecause ĠBT s Ġare Ġbetter Ġfor Ġthe Ġformer Ġsystem Ġthan Ġfor Ġthe Ġlatter . </s> Finally Ġnote Ġthat Ġthe ĠG AN Ġarchitecture Ġhas Ġtwo Ġdifferences Ġwith Ġbasic Ġcopy - marked : Ġ( a ) Ġa Ġdistinct Ġenc oder Ġfor Ġreal Ġand Ġpseudo - sent ence ; Ġ( b ) Ġa Ġdifferent Ġtraining Ġregime Ġfor Ġthese Ġenc od ers . ĠTo Ġsort Ġout Ġthe Ġeffects Ġof Ġ( a ) Ġand Ġ( b ), Ġwe Ġreproduce Ġthe ĠG AN Ġsetup Ġwith ĠBT Ġsentences , Ġinstead Ġof Ġcopies . ĠUsing Ġa Ġseparate Ġenc oder Ġfor Ġthe Ġpseudo - source Ġin Ġthe Ġback trans - n mt Ġsetup Ġcan Ġbe Ġdetrimental Ġto Ġperformance Ġ( see ĠTable ĠT AB REF 32 Ġ): Ġtranslation Ġde grades Ġin ĠFrench Ġfor Ġall Ġmetrics . ĠAdding ĠG AN s Ġon Ġtop Ġof Ġthe Ġpseudo - enc oder Ġwas Ġnot Ġable Ġto Ġmake Ġup Ġfor Ġthe Ġdegradation Ġobserved Ġin ĠFrench , Ġbut Ġallowed Ġthe ĠGerman Ġsystem Ġto Ġslightly Ġoutper form Ġback trans - n mt . ĠEven Ġthough Ġthis Ġsetup Ġis Ġunrealistic Ġand Ġoverly Ġcostly , Ġit Ġshows Ġthat ĠG AN s Ġare Ġactually Ġhelping Ġeven Ġgood Ġsystems . </s> Using ĠTarget ĠLanguage ĠModels Ġ </s> In Ġthis Ġsection , Ġwe Ġcompare Ġthe Ġprevious Ġmethods Ġwith Ġthe Ġuse Ġof Ġa Ġtarget Ġside ĠLanguage ĠModel Ġ( LM ). ĠSeveral Ġproposals Ġexist Ġin Ġthe Ġliterature Ġto Ġintegrate ĠL Ms Ġin ĠN MT : Ġfor Ġinstance , ĠB IB REF 3 Ġstrengthen Ġthe Ġdec oder Ġby Ġintegrating Ġan Ġextra , Ġsource Ġindependent , ĠR NN Ġlayer Ġin Ġa Ġconventional ĠN MT Ġarchitecture . ĠTraining Ġis Ġperformed Ġeither Ġwith Ġparallel , Ġor Ġmon oling ual Ġdata . ĠIn Ġthe Ġlatter Ġcase , Ġword Ġprediction Ġonly Ġrelies Ġon Ġthe Ġsource Ġindependent Ġpart Ġof Ġthe Ġnetwork . </s> LM ĠSetup </s> We Ġhave Ġfollowed ĠB IB REF 29 Ġand Ġreim ple mented Ġtheir Ġdeep - f usion Ġtechnique . ĠIt Ġrequires Ġto Ġfirst Ġindependently Ġlearn Ġa ĠR NN - LM Ġon Ġthe Ġin - domain Ġtarget Ġdata Ġwith Ġa Ġcross - ent ropy Ġobjective ; Ġthen Ġto Ġtrain Ġthe Ġoptimal Ġcombination Ġof Ġthe Ġtranslation Ġand Ġthe Ġlanguage Ġmodels Ġby Ġadding Ġthe Ġhidden Ġstate Ġof Ġthe ĠR NN - LM Ġas Ġan Ġadditional Ġinput Ġto Ġthe Ġsoft max Ġlayer Ġof Ġthe Ġdec oder . ĠOur ĠR NN - L Ms Ġare Ġtrained Ġusing Ġd l 4 mt Ġwith Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġdata Ġand Ġthe ĠEuro par l Ġcorpus Ġ( about Ġ6 M Ġsentences Ġfor Ġboth ĠFrench Ġand ĠGerman ), Ġusing Ġa Ġone - layer ĠGR U Ġwith Ġthe Ġsame Ġdimension Ġas Ġthe ĠMT Ġdec oder Ġ( 1024 ). </s> LM ĠResults </s> Results Ġare Ġin ĠTable ĠT AB REF 33 Ġ. ĠThey Ġshow Ġthat Ġdeep - f usion Ġhardly Ġimproves Ġthe ĠEuro par l Ġresults , Ġwhile Ġwe Ġobtain Ġabout Ġ+ 0 . 6 ĠB LE U Ġover Ġthe Ġbaseline Ġon Ġnew st est - 2014 Ġfor Ġboth Ġlanguages . Ġdeep - f usion Ġdiffers Ġfrom Ġstupid ĠBT Ġin Ġthat Ġthe Ġmodel Ġis Ġnot Ġdirectly Ġoptimized Ġon Ġthe Ġin - domain Ġdata , Ġbut Ġuses Ġthe ĠLM Ġtrained Ġon ĠEuro par l Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġout - of - domain Ġtraining Ġdata . ĠTherefore , Ġno Ġspecific Ġimprovement Ġis Ġto Ġbe Ġexpected Ġin Ġterms Ġof Ġdomain Ġadaptation , Ġand Ġthe Ġperformance Ġincreases Ġin Ġthe Ġmore Ġgeneral Ġdomain . ĠComb ining Ġdeep - f usion Ġand Ġcopy - marked Ġ+ Ġnoise Ġ+ ĠG AN s Ġbrings Ġslight Ġimprovements Ġon Ġthe ĠGerman Ġin - domain Ġtest Ġsets , Ġand Ġperformance Ġout Ġof Ġthe Ġdomain Ġremains Ġnear Ġthe Ġbaseline Ġlevel . </s> Re - analy zing Ġthe Ġeffects Ġof ĠBT Ġ </s> As Ġa Ġfollow Ġup Ġof Ġprevious Ġdiscussions , Ġwe Ġanalyze Ġthe Ġeffect Ġof ĠBT Ġon Ġthe Ġintern als Ġof Ġthe Ġnetwork . ĠAr gu ably , Ġusing Ġa Ġcopy Ġof Ġthe Ġtarget Ġsentence Ġinstead Ġof Ġa Ġnatural Ġsource Ġshould Ġnot Ġbe Ġhelpful Ġfor Ġthe Ġenc oder , Ġbut Ġis Ġit Ġalso Ġthe Ġcase Ġwith Ġa Ġstrong ĠBT ? ĠWhat Ġare Ġthe Ġeffects Ġon Ġthe Ġattention Ġmodel ? </s> Parameter Ġfreezing Ġprotocol </s> To Ġinvestigate Ġthese Ġquestions , Ġwe Ġrun Ġthe Ġsame Ġfine - tun ing Ġusing Ġthe Ġcopy - marked , Ġback trans - n mt Ġand Ġback trans - n mt Ġsetups . ĠNote Ġthat Ġexcept Ġfor Ġthe Ġlast Ġone , Ġall Ġtraining Ġscenarios Ġhave Ġaccess Ġto Ġsame Ġtarget Ġtraining Ġdata . ĠWe Ġintend Ġto Ġsee Ġwhether Ġthe Ġoverall Ġperformance Ġof Ġthe ĠN MT Ġsystem Ġde grades Ġwhen Ġwe Ġselectively Ġfreeze Ġcertain Ġsets Ġof Ġparameters , Ġmeaning Ġthat Ġthey Ġare Ġnot Ġupdated Ġduring Ġfine - tun ing . </s> Results </s> BLE U Ġscores Ġare Ġin ĠTable ĠT AB REF 39 Ġ. ĠThe Ġback trans - n mt Ġsetup Ġis Ġhardly Ġimpacted Ġby Ġselective Ġupdates : Ġupdating Ġthe Ġonly Ġdec oder Ġleads Ġto Ġa Ġdegradation Ġof Ġat Ġmost Ġ0 . 2 ĠB LE U . ĠFor Ġcopy - marked , Ġwe Ġwere Ġnot Ġable Ġto Ġfreeze Ġthe Ġsource Ġembed d ings , Ġsince Ġthese Ġare Ġinitialized Ġwhen Ġfine - tun ing Ġbegins Ġand Ġtherefore Ġneed Ġto Ġbe Ġtrained . ĠWe Ġobserve Ġthat Ġfreezing Ġthe Ġenc oder Ġand / or Ġthe Ġattention Ġparameters Ġhas Ġno Ġimpact Ġon Ġthe ĠEnglish - German Ġsystem , Ġwhereas Ġit Ġslightly Ġde grades Ġthe ĠEnglish - French Ġone . ĠThis Ġsuggests Ġthat Ġusing Ġartificial Ġsources , Ġeven Ġof Ġthe Ġpoorest Ġquality , Ġhas Ġa Ġpositive Ġimpact Ġon Ġall Ġthe Ġcomponents Ġof Ġthe Ġnetwork , Ġwhich Ġmakes Ġanother Ġbig Ġdifference Ġwith Ġthe ĠLM Ġintegration Ġscenario . </s> The Ġlargest Ġdegradation Ġis Ġfor Ġnatural , Ġwhere Ġthe Ġmodel Ġis Ġprevented Ġfrom Ġlearning Ġfrom Ġinformative Ġsource Ġsentences , Ġwhich Ġleads Ġto Ġa Ġdecrease Ġof Ġ0 . 4 Ġto Ġover Ġ1 . 0 ĠB LE U . ĠWe Ġassume Ġfrom Ġthese Ġexperiments Ġthat ĠBT Ġimpacts Ġmost Ġof Ġall Ġthe Ġdec oder , Ġand Ġlearning Ġto Ġencode Ġa Ġpseudo - source , Ġbe Ġit Ġa Ġcopy Ġor Ġan Ġactual Ġback - translation , Ġonly Ġmarginally Ġhelps Ġto Ġsignificantly Ġimprove Ġthe Ġquality . ĠFinally , Ġin Ġthe Ġf wd trans - n mt Ġsetup , Ġfreezing Ġthe Ġdec oder Ġdoes Ġnot Ġseem Ġto Ġharm Ġlearning Ġwith Ġa Ġnatural Ġsource . </s> Related Ġwork </s> The Ġliterature Ġdevoted Ġto Ġthe Ġuse Ġof Ġmon oling ual Ġdata Ġis Ġlarge , Ġand Ġquickly Ġexpanding . ĠWe Ġalready Ġalluded Ġto Ġseveral Ġpossible Ġways Ġto Ġuse Ġsuch Ġdata : Ġusing Ġback - Ġor Ġforward - translation Ġor Ġusing Ġa Ġtarget Ġlanguage Ġmodel . ĠThe Ġformer Ġapproach Ġis Ġmostly Ġdocumented Ġin ĠB IB REF 2 Ġ, Ġand Ġrecently Ġanalyzed Ġin ĠB IB REF 19 Ġ, Ġwhich Ġfocus Ġon Ġfully Ġartificial Ġsettings Ġas Ġwell Ġas Ġpivot - based Ġartificial Ġdata ; Ġand ĠB IB REF 4 Ġ, Ġwhich Ġstudies Ġthe Ġeffects Ġof Ġincreasing Ġthe Ġsize Ġof ĠBT Ġdata . ĠThe Ġstudies Ġof ĠB IB REF 20 Ġ, ĠB IB REF 19 Ġalso Ġconsider Ġforward Ġtranslation Ġand ĠB IB REF 21 Ġexpand Ġthese Ġresults Ġto Ġdomain Ġadaptation Ġscenarios . ĠOur Ġresults Ġare Ġcomplementary Ġto Ġthese Ġearlier Ġstudies . </s> As Ġshown Ġabove , Ġmany Ġalternatives Ġto ĠBT Ġexist . ĠThe Ġmost Ġobvious Ġis Ġto Ġuse Ġtarget ĠL Ms ĠB IB REF 3 Ġ, ĠB IB REF 29 Ġ, Ġas Ġwe Ġhave Ġalso Ġdone Ġhere ; Ġbut Ġattempts Ġto Ġimprove Ġthe Ġenc oder Ġusing Ġmulti - task Ġlearning Ġalso Ġexist ĠB IB REF 30 Ġ. </s> This Ġinvestigation Ġis Ġalso Ġrelated Ġto Ġrecent Ġattempts Ġto Ġconsider Ġsupplementary Ġdata Ġwith Ġa Ġvalid Ġtarget Ġside , Ġsuch Ġas Ġmulti - ling ual ĠN MT ĠB IB REF 31 Ġ, Ġwhere Ġsource Ġtexts Ġin Ġseveral Ġlanguages Ġare Ġfed Ġin Ġthe Ġsame Ġenc oder - dec oder Ġarchitecture , Ġwith Ġpartial Ġsharing Ġof Ġthe Ġlayers . ĠThis Ġis Ġanother Ġrealistic Ġscenario Ġwhere Ġadditional Ġresources Ġcan Ġbe Ġused Ġto Ġselectively Ġimprove Ġparts Ġof Ġthe Ġmodel . </s> Round Ġtrip Ġtraining Ġis Ġanother Ġimportant Ġsource Ġof Ġinspiration , Ġas Ġit Ġcan Ġbe Ġviewed Ġas Ġa Ġway Ġto Ġuse ĠBT Ġto Ġperform Ġsemi - un super vised ĠB IB REF 32 Ġor Ġun super vised ĠB IB REF 33 Ġtraining Ġof ĠN MT . ĠThe Ġmost Ġconvincing Ġattempt Ġto Ġdate Ġalong Ġthese Ġlines Ġhas Ġbeen Ġproposed Ġby ĠB IB REF 26 Ġ, Ġwho Ġpropose Ġto Ġuse ĠG AN s Ġto Ġmitigate Ġthe Ġdifference Ġbetween Ġartificial Ġand Ġnatural Ġdata . </s> Conclusion Ġ </s> In Ġthis Ġpaper , Ġwe Ġhave Ġanalyzed Ġvarious Ġways Ġto Ġintegrate Ġmon oling ual Ġdata Ġin Ġan ĠN MT Ġframework , Ġfocusing Ġon Ġtheir Ġimpact Ġon Ġquality Ġand Ġdomain Ġadaptation . ĠWhile Ġconfirming Ġthe Ġeffectiveness Ġof ĠBT , Ġour Ġstudy Ġalso Ġproposed Ġsignificantly Ġcheaper Ġways Ġto Ġimprove Ġthe Ġbaseline Ġperformance , Ġusing Ġa Ġslightly Ġmodified Ġcopy Ġof Ġthe Ġtarget , Ġinstead Ġof Ġits Ġfull ĠBT . ĠWhen Ġno Ġhigh Ġquality ĠBT Ġis Ġavailable , Ġusing ĠG AN s Ġto Ġmake Ġthe Ġpseudo - source Ġsentences Ġcloser Ġto Ġnatural Ġsource Ġsentences Ġis Ġan Ġefficient Ġsolution Ġfor Ġdomain Ġadaptation . </s> To Ġrecap Ġour Ġanswers Ġto Ġour Ġinitial Ġquestions : Ġthe Ġquality Ġof ĠBT Ġactually Ġmatters Ġfor ĠN MT Ġ( cf . ĠÂ§ ĠSEC REF 8 Ġ) Ġand Ġit Ġseems Ġthat , Ġeven Ġthough Ġartificial Ġsource Ġare Ġlex ically Ġless Ġdiverse Ġand Ġsynt act ically Ġcomplex Ġthan Ġreal Ġsentence , Ġtheir Ġmon oton icity Ġis Ġa Ġfacilitating Ġfactor . ĠWe Ġhave Ġstudied Ġcheaper Ġalternatives Ġand Ġfound Ġout Ġthat Ġcopies Ġof Ġthe Ġtarget , Ġif Ġproperly Ġno ised Ġ( Â§ ĠSEC REF 4 Ġ), Ġand Ġeven Ġbetter , Ġif Ġused Ġwith ĠG AN s , Ġcould Ġbe Ġalmost Ġas Ġgood Ġas Ġlow Ġquality ĠBT s Ġ( Â§ ĠSEC REF 5 Ġ): ĠBT Ġis Ġonly Ġworth Ġits Ġcost Ġwhen Ġgood ĠBT Ġcan Ġbe Ġgenerated . ĠFinally , ĠBT Ġseems Ġpreferable Ġto Ġintegrating Ġexternal ĠLM Ġ- Ġat Ġleast Ġin Ġour Ġdata Ġcondition Ġ( Â§ ĠSEC REF 6 Ġ). ĠFurther Ġexperiments Ġwith Ġlarger ĠL Ms Ġare Ġneeded Ġto Ġconfirm Ġthis Ġobservation , Ġand Ġalso Ġto Ġevaluate Ġthe Ġcomplement arity Ġof Ġboth Ġstrategies . ĠMore Ġwork Ġis Ġneeded Ġto Ġbetter Ġunderstand Ġthe Ġimpact Ġof ĠBT Ġon Ġsub parts Ġof Ġthe Ġnetwork Ġ( Â§ ĠSEC REF 7 Ġ). </s> In Ġfuture Ġwork , Ġwe Ġplan Ġto Ġinvestigate Ġother Ġcheap Ġways Ġto Ġgenerate Ġartificial Ġdata . ĠThe Ġexperimental Ġsetup Ġwe Ġproposed Ġmay Ġalso Ġbenefit Ġfrom Ġa Ġrefining Ġof Ġthe Ġdata Ġselection Ġstrategies Ġto Ġfocus Ġon Ġthe Ġmost Ġuseful Ġmon oling ual Ġsentences .