attentional encoder-decoder networks BIBREF0
EXTRACTIVE
 the dl4mt-tutorial
EXTRACTIVE
Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout.
Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout. We use minibatches of size 80, a maximum sentence length of 50, word embeddings of size 500, and hidden layers of size 1024. We clip the gradient norm to 1.0 BIBREF4 . We train the models with Adadelta BIBREF5 , reshuffling the training corpus between epochs. We validate the model every 10000 minibatches via Bleu on a validation set (newstest2013, newstest2014, or half of newsdev2016 for EN INLINEFORM0 RO). We perform early stopping for single models, and use the 4 last saved models (with models saved every 30000 minibatches) for the ensemble results. Note that ensemble scores are the result of a single training run. Due to resource limitations, we did not train ensemble components independently, which could result in more diverse models and better ensembles. Decoding is performed with beam search with a beam size of 12. For some language pairs, we used the AmuNMT C++ decoder as a more efficient alternative to the theano implementation of the dl4mt tutorial.
1606.02891
Introduction </s> We Ġparticipated Ġin Ġthe ĠW MT Ġ2016 Ġshared Ġnews Ġtranslation Ġtask Ġby Ġbuilding Ġneural Ġtranslation Ġsystems Ġfor Ġfour Ġlanguage Ġpairs : ĠEnglish ĠIN LINE FORM 0 ĠCzech , ĠEnglish ĠIN LINE FORM 1 ĠGerman , ĠEnglish ĠIN LINE FORM 2 ĠRomanian Ġand ĠEnglish ĠIN LINE FORM 3 ĠRussian . ĠOur Ġsystems Ġare Ġbased Ġon Ġan Ġattention al Ġenc oder - dec oder ĠB IB REF 0 Ġ, Ġusing ĠB PE Ġsub word Ġsegment ation Ġfor Ġopen - voc abulary Ġtranslation Ġwith Ġa Ġfixed Ġvocabulary ĠB IB REF 1 Ġ. ĠWe Ġexperimented Ġwith Ġusing Ġautomatic Ġback - trans lations Ġof Ġthe Ġmon oling ual ĠNews Ġcorpus Ġas Ġadditional Ġtraining Ġdata ĠB IB REF 2 Ġ, Ġpervasive Ġdrop out ĠB IB REF 3 Ġ, Ġand Ġtarget - bid irection al Ġmodels . </s> Bas eline ĠSystem </s> Our Ġsystems Ġare Ġattention al Ġenc oder - dec oder Ġnetworks ĠB IB REF 0 Ġ. ĠWe Ġbase Ġour Ġimplementation Ġon Ġthe Ġd l 4 mt - t utorial , Ġwhich Ġwe Ġenhanced Ġwith Ġnew Ġfeatures Ġsuch Ġas Ġensemble Ġdecoding Ġand Ġpervasive Ġdrop out . </s> We Ġuse Ġmin ib atches Ġof Ġsize Ġ80 , Ġa Ġmaximum Ġsentence Ġlength Ġof Ġ50 , Ġword Ġembed d ings Ġof Ġsize Ġ500 , Ġand Ġhidden Ġlayers Ġof Ġsize Ġ1024 . ĠWe Ġclip Ġthe Ġgradient Ġnorm Ġto Ġ1 . 0 ĠB IB REF 4 Ġ. ĠWe Ġtrain Ġthe Ġmodels Ġwith ĠAd ad elta ĠB IB REF 5 Ġ, Ġresh uff ling Ġthe Ġtraining Ġcorpus Ġbetween Ġepoch s . ĠWe Ġvalidate Ġthe Ġmodel Ġevery Ġ10000 Ġmin ib atches Ġvia ĠBle u Ġon Ġa Ġvalidation Ġset Ġ( new st est 2013 , Ġnew st est 2014 , Ġor Ġhalf Ġof Ġnews dev 2016 Ġfor ĠEN ĠIN LINE FORM 0 ĠRO ). ĠWe Ġperform Ġearly Ġstopping Ġfor Ġsingle Ġmodels , Ġand Ġuse Ġthe Ġ4 Ġlast Ġsaved Ġmodels Ġ( with Ġmodels Ġsaved Ġevery Ġ3 0000 Ġmin ib atches ) Ġfor Ġthe Ġensemble Ġresults . ĠNote Ġthat Ġensemble Ġscores Ġare Ġthe Ġresult Ġof Ġa Ġsingle Ġtraining Ġrun . ĠDue Ġto Ġresource Ġlimitations , Ġwe Ġdid Ġnot Ġtrain Ġensemble Ġcomponents Ġindependently , Ġwhich Ġcould Ġresult Ġin Ġmore Ġdiverse Ġmodels Ġand Ġbetter Ġen semb les . </s> Dec oding Ġis Ġperformed Ġwith Ġbeam Ġsearch Ġwith Ġa Ġbeam Ġsize Ġof Ġ12 . ĠFor Ġsome Ġlanguage Ġpairs , Ġwe Ġused Ġthe ĠAm u N MT ĠC ++ Ġdec oder Ġas Ġa Ġmore Ġefficient Ġalternative Ġto Ġthe Ġthe ano Ġimplementation Ġof Ġthe Ġd l 4 mt Ġtutorial . </s> Byte - pair Ġencoding Ġ( B PE ) </s> To Ġenable Ġopen - voc abulary Ġtranslation , Ġwe Ġsegment Ġwords Ġvia Ġbyte - pair Ġencoding Ġ( B PE ) ĠB IB REF 1 Ġ. ĠB PE , Ġoriginally Ġdevised Ġas Ġa Ġcompression Ġalgorithm ĠB IB REF 6 Ġ, Ġis Ġadapted Ġto Ġword Ġsegment ation Ġas Ġfollows : </s> First , Ġeach Ġword Ġin Ġthe Ġtraining Ġvocabulary Ġis Ġrepresented Ġas Ġa Ġsequence Ġof Ġcharacters , Ġplus Ġan Ġend - of - word Ġsymbol . ĠAll Ġcharacters Ġare Ġadded Ġto Ġthe Ġsymbol Ġvocabulary . ĠThen , Ġthe Ġmost Ġfrequent Ġsymbol Ġpair Ġis Ġidentified , Ġand Ġall Ġits Ġoccurrences Ġare Ġmerged , Ġproducing Ġa Ġnew Ġsymbol Ġthat Ġis Ġadded Ġto Ġthe Ġvocabulary . ĠThe Ġprevious Ġstep Ġis Ġrepeated Ġuntil Ġa Ġset Ġnumber Ġof Ġmerge Ġoperations Ġhave Ġbeen Ġlearned . </s> B PE Ġstarts Ġfrom Ġa Ġcharacter - level Ġsegment ation , Ġbut Ġas Ġwe Ġincrease Ġthe Ġnumber Ġof Ġmerge Ġoperations , Ġit Ġbecomes Ġmore Ġand Ġmore Ġdifferent Ġfrom Ġa Ġpure Ġcharacter - level Ġmodel Ġin Ġthat Ġfrequent Ġcharacter Ġsequences , Ġand Ġeven Ġfull Ġwords , Ġare Ġencoded Ġas Ġa Ġsingle Ġsymbol . ĠThis Ġallows Ġfor Ġa Ġtrade - off Ġbetween Ġthe Ġsize Ġof Ġthe Ġmodel Ġvocabulary Ġand Ġthe Ġlength Ġof Ġtraining Ġsequences . ĠThe Ġordered Ġlist Ġof Ġmerge Ġoperations , Ġlearned Ġon Ġthe Ġtraining Ġset , Ġcan Ġbe Ġapplied Ġto Ġany Ġtext Ġto Ġsegment Ġwords Ġinto Ġsub word Ġunits Ġthat Ġare Ġin - voc abulary Ġin Ġrespect Ġto Ġthe Ġtraining Ġset Ġ( except Ġfor Ġunseen Ġcharacters ). </s> To Ġincrease Ġconsistency Ġin Ġthe Ġsegment ation Ġof Ġthe Ġsource Ġand Ġtarget Ġtext , Ġwe Ġcombine Ġthe Ġsource Ġand Ġtarget Ġside Ġof Ġthe Ġtraining Ġset Ġfor Ġlearning ĠB PE . ĠFor Ġeach Ġlanguage Ġpair , Ġwe Ġlearn Ġ8 95 00 Ġmerge Ġoperations . </s> Sy nt hetic ĠTraining ĠData </s> W MT Ġprovides Ġtask Ġparticipants Ġwith Ġlarge Ġamounts Ġof Ġmon oling ual Ġdata , Ġboth Ġin - domain Ġand Ġout - of - domain . ĠWe Ġexploit Ġthis Ġmon oling ual Ġdata Ġfor Ġtraining Ġas Ġdescribed Ġin ĠB IB REF 2 Ġ. ĠSpecifically , Ġwe Ġsample Ġa Ġsubset Ġof Ġthe Ġavailable Ġtarget - side Ġmon oling ual Ġcorpor a , Ġtranslate Ġit Ġautomatically Ġinto Ġthe Ġsource Ġside Ġof Ġthe Ġrespective Ġlanguage Ġpair , Ġand Ġthen Ġuse Ġthis Ġsynthetic Ġparallel Ġdata Ġfor Ġtraining . ĠFor Ġexample , Ġfor ĠEN ĠIN LINE FORM 0 ĠRO , Ġthe Ġback - translation Ġis Ġperformed Ġwith Ġa ĠRO ĠIN LINE FORM 1 ĠEN Ġsystem , Ġand Ġvice - vers a . </s> 2015 ar X iv 15 110 6 709 S Ġmotivate Ġthe Ġuse Ġof Ġmon oling ual Ġdata Ġwith Ġdomain Ġadaptation , Ġreducing Ġover fitting , Ġand Ġbetter Ġmodelling Ġof Ġflu ency . ĠWe Ġsample Ġmon oling ual Ġdata Ġfrom Ġthe ĠNews ĠC rawl Ġcorpor a , Ġwhich Ġis Ġin - domain Ġwith Ġrespect Ġto Ġthe Ġtest Ġset . </s> The Ġamount Ġof Ġmon oling ual Ġdata Ġback - trans lated Ġfor Ġeach Ġtranslation Ġdirection Ġranges Ġfrom Ġ2 Ġmillion Ġto Ġ10 Ġmillion Ġsentences . ĠStatistics Ġabout Ġthe Ġamount Ġof Ġparallel Ġand Ġsynthetic Ġtraining Ġdata Ġare Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. ĠWith Ġd l 4 mt , Ġwe Ġobserved Ġa Ġtranslation Ġspeed Ġof Ġabout Ġ200 000 Ġsentences Ġper Ġday Ġ( on Ġa Ġsingle ĠTitan ĠX ĠGPU ). </s> P erv asive ĠDrop out </s> For ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġwe Ġobserved Ġpoor Ġperformance Ġbecause Ġof Ġover fitting . ĠTo Ġmitigate Ġthis , Ġwe Ġapply Ġdrop out Ġto Ġall Ġlayers Ġin Ġthe Ġnetwork , Ġincluding Ġrecurrent Ġones . </s> Previous Ġwork Ġdropped Ġout Ġdifferent Ġunits Ġat Ġeach Ġtime Ġstep . ĠWhen Ġapplied Ġto Ġrecurrent Ġconnections , Ġthis Ġhas Ġthe Ġdownside Ġthat Ġit Ġimped es Ġthe Ġinformation Ġflow Ġover Ġlong Ġdistances , Ġand ĠD BL P : conf / ic f hr / Ph am B K L 14 Ġpropose Ġto Ġonly Ġapply Ġdrop out Ġto Ġnon - rec urrent Ġconnections . </s> Instead , Ġwe Ġfollow Ġthe Ġapproach Ġsuggested Ġby Ġ2015 ar X iv 15 12 05 287 G , Ġand Ġuse Ġthe Ġsame Ġdrop out Ġmask Ġat Ġeach Ġtime Ġstep . ĠOur Ġimplementation Ġdiffers Ġfrom Ġthe Ġrecommendations Ġby Ġ2015 ar X iv 15 12 05 287 G Ġin Ġone Ġrespect : Ġwe Ġalso Ġdrop Ġwords Ġat Ġrandom , Ġbut Ġwe Ġdo Ġso Ġon Ġa Ġtoken Ġlevel , Ġnot Ġon Ġa Ġtype Ġlevel . ĠIn Ġother Ġwords , Ġif Ġa Ġword Ġoccurs Ġmultiple Ġtimes Ġin Ġa Ġsentence , Ġwe Ġmay Ġdrop Ġout Ġany Ġnumber Ġof Ġits Ġoccurrences , Ġand Ġnot Ġjust Ġnone Ġor Ġall . </s> In Ġour ĠEnglish ĠIN LINE FORM 0 ĠRomanian Ġexperiments , Ġwe Ġdrop Ġout Ġfull Ġwords Ġ( both Ġon Ġthe Ġsource Ġand Ġtarget Ġside ) Ġwith Ġa Ġprobability Ġof Ġ0 . 1 . ĠFor Ġall Ġother Ġlayers , Ġthe Ġdrop out Ġprobability Ġis Ġset Ġto Ġ0 . 2 . </s> Target - bid irection al ĠTranslation </s> We Ġfound Ġthat Ġduring Ġdecoding , Ġthe Ġmodel Ġwould Ġoccasionally Ġassign Ġa Ġhigh Ġprobability Ġto Ġwords Ġbased Ġon Ġthe Ġtarget Ġcontext Ġalone , Ġignoring Ġthe Ġsource Ġsentence . ĠWe Ġspeculate Ġthat Ġthis Ġis Ġan Ġinstance Ġof Ġthe Ġlabel Ġbias Ġproblem ĠB IB REF 7 Ġ. </s> To Ġmitigate Ġthis Ġproblem , Ġwe Ġexperiment Ġwith Ġtraining Ġseparate Ġmodels Ġthat Ġproduce Ġthe Ġtarget Ġtext Ġfrom Ġright - to - left Ġ( r 2 l ), Ġand Ġre - scoring Ġthe Ġn - best Ġlists Ġthat Ġare Ġproduced Ġby Ġthe Ġmain Ġ( left - to - right ) Ġmodels Ġwith Ġthese Ġr 2 l Ġmodels . ĠSince Ġthe Ġright - to - left Ġmodel Ġwill Ġsee Ġa Ġcomplementary Ġtarget Ġcontext Ġat Ġeach Ġtime Ġstep , Ġwe Ġexpect Ġthat Ġthe Ġaveraged Ġprobabilities Ġwill Ġbe Ġmore Ġrobust . ĠIn Ġparallel Ġto Ġour Ġexperiments , Ġthis Ġidea Ġwas Ġpublished Ġby Ġli u 2016 . </s> We Ġincrease Ġthe Ġsize Ġof Ġthe Ġn - best - list Ġto Ġ50 Ġfor Ġthe Ġre ranking Ġexperiments . </s> A Ġpossible Ġcriticism Ġof Ġthe Ġl - r / r - l Ġre ranking Ġapproach Ġis Ġthat Ġthe Ġgains Ġactually Ġcome Ġfrom Ġadding Ġdiversity Ġto Ġthe Ġensemble , Ġsince Ġwe Ġare Ġnow Ġusing Ġtwo Ġindependent Ġruns . ĠHowever Ġexperiments Ġin ĠB IB REF 8 Ġshow Ġthat Ġa Ġl - r / r - l Ġre ranking Ġsystems Ġis Ġstronger Ġthan Ġan Ġensemble Ġcreated Ġfrom Ġtwo Ġindependent Ġl - r Ġruns . </s> English âĨ Ķ \ left right arrow ĠGerman </s> Table ĠT AB REF 13 Ġshows Ġresults Ġfor ĠEnglish ĠIN LINE FORM 0 ĠGerman . ĠWe Ġobserve Ġimprovements Ġof Ġ3 . 4 âĢĵ 5 . 7 ĠBle u Ġfrom Ġtraining Ġwith Ġa Ġmix Ġof Ġparallel Ġand Ġsynthetic Ġdata , Ġcompared Ġto Ġthe Ġbaseline Ġthat Ġis Ġonly Ġtrained Ġon Ġparallel Ġdata . ĠUsing Ġan Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints Ġgives Ġfurther Ġimprovements Ġ( 1 . 3 âĢĵ 1 . 7 ĠBle u ). ĠOur Ġsubmitted Ġsystem Ġincludes Ġre ranking Ġof Ġthe Ġ50 - best Ġoutput Ġof Ġthe Ġleft - to - right Ġmodel Ġwith Ġa Ġright - to - left Ġmodel ĠâĢĵ Ġagain Ġan Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints ĠâĢĵ Ġwith Ġuniform Ġweights . ĠThis Ġyields Ġan Ġimprovements Ġof Ġ0 . 6 âĢĵ 1 . 1 ĠBle u . </s> English âĨ Ķ \ left right arrow ĠCzech </s> For ĠEnglish ĠIN LINE FORM 0 ĠCzech , Ġwe Ġtrained Ġour Ġbaseline Ġmodel Ġon Ġthe Ġcomplete ĠW MT 16 Ġparallel Ġtraining Ġset Ġ( including ĠC z Eng Ġ1 . 6 pre ĠB IB REF 9 Ġ), Ġuntil Ġwe Ġobserved Ġconvergence Ġon Ġour Ġheld out Ġset Ġ( new st est 2014 ). ĠThis Ġtook Ġapproximately Ġ1 M Ġmin ib atches , Ġor Ġ3 Ġweeks . ĠThen Ġwe Ġcontinued Ġtraining Ġthe Ġmodel Ġon Ġa Ġnew Ġparallel Ġcorpus , Ġcomprising Ġ8 . 2 M Ġsentences Ġback - trans lated Ġfrom Ġthe ĠCzech Ġmon oling ual Ġnews 2015 , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġ9 M Ġsentences Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠThe Ġmodel Ġused Ġfor Ġback - translation Ġwas Ġa Ġneural ĠMT Ġmodel Ġfrom Ġearlier Ġexperiments , Ġtrained Ġon ĠW MT 15 Ġdata . ĠThe Ġtraining Ġon Ġthis Ġsynthetic Ġmix Ġcontinued Ġfor Ġa Ġfurther Ġ400 , 000 Ġmin ib atches . </s> The Ġright - left Ġmodel Ġwas Ġtrained Ġusing Ġa Ġsimilar Ġprocess , Ġbut Ġwith Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġcorpus Ġreversed Ġprior Ġto Ġtraining . ĠThe Ġresulting Ġmodel Ġhad Ġa Ġslightly Ġlower ĠBle u Ġscore Ġon Ġthe Ġdev Ġdata Ġthan Ġthe Ġstandard Ġleft - right Ġmodel . ĠWe Ġcan Ġsee Ġin ĠTable ĠT AB REF 15 Ġthat Ġback - translation Ġimproves Ġperformance Ġby Ġ2 . 2 âĢĵ 2 . 8 ĠBle u , Ġand Ġthat Ġthe Ġfinal Ġsystem Ġ(+ r 2 l Ġre ranking ) Ġimproves Ġby Ġ0 . 7 âĢĵ 1 . 0 ĠBle u Ġon Ġthe Ġensemble Ġof Ġ4 , Ġand Ġ4 . 3 âĢĵ 4 . 9 Ġon Ġthe Ġbaseline . </s> For ĠCzech ĠIN LINE FORM 0 ĠEnglish Ġthe Ġtraining Ġprocess Ġwas Ġsimilar Ġto Ġthe Ġabove , Ġexcept Ġthat Ġwe Ġcreated Ġthe Ġsynthetic Ġtraining Ġdata Ġ( back - trans lated Ġfrom Ġsamples Ġof Ġnews 2015 Ġmon oling ual ĠEnglish ) Ġin Ġbatches Ġof Ġ2 . 5 M , Ġand Ġso Ġwere Ġable Ġto Ġobserve Ġthe Ġeffect Ġof Ġincreasing Ġthe Ġamount Ġof Ġsynthetic Ġdata . ĠAfter Ġtraining Ġa Ġbaseline Ġmodel Ġon Ġall Ġthe ĠW MT 16 Ġparallel Ġset , Ġwe Ġcontinued Ġtraining Ġwith Ġa Ġparallel Ġcorpus Ġconsisting Ġof Ġ2 Ġcopies Ġof Ġthe Ġ2 . 5 M Ġsentences Ġof Ġback - trans lated Ġdata , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġa Ġmatching Ġquantity Ġof Ġdata Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠAfter Ġtraining Ġthis Ġto Ġconvergence , Ġwe Ġrestart ed Ġtraining Ġfrom Ġthe Ġbaseline Ġmodel Ġusing Ġ5 M Ġsentences Ġof Ġback - trans lated Ġdata , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġa Ġmatching Ġquantity Ġof Ġdata Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠWe Ġrepeated Ġthis Ġwith Ġ7 . 5 M Ġsentences Ġfrom Ġnews 2015 Ġmon oling ual , Ġand Ġthen Ġwith Ġ10 M Ġsentences Ġof Ġnews 2015 . ĠThe Ġback - trans lations Ġwere , Ġas Ġfor ĠEnglish ĠIN LINE FORM 1 ĠCzech , Ġcreated Ġwith Ġan Ġearlier ĠN MT Ġmodel Ġtrained Ġon ĠW MT 15 Ġdata . ĠOur Ġfinal ĠCzech ĠIN LINE FORM 2 ĠEnglish Ġwas Ġan Ġensemble Ġof Ġ8 Ġsystems ĠâĢĵ Ġthe Ġlast Ġ4 Ġsave - points Ġof Ġthe Ġ10 M Ġsynthetic Ġdata Ġrun , Ġand Ġthe Ġlast Ġ4 Ġsave - points Ġof Ġthe Ġ7 . 5 M Ġrun . ĠWe Ġshow Ġthis Ġas Ġensemble 8 Ġin ĠTable ĠT AB REF 15 Ġ, Ġand Ġthe Ġ+ sy nt hetic Ġresults Ġare Ġon Ġthe Ġlast Ġ( i . e . Ġ10 M ) Ġsynthetic Ġdata Ġrun . </s> We Ġalso Ġshow Ġin ĠTable ĠT AB REF 16 Ġhow Ġincreasing Ġthe Ġamount Ġof Ġback - trans lated Ġdata Ġaffects Ġthe Ġresults . ĠWe Ġsee Ġthat Ġmost Ġof Ġthe Ġgain Ġfrom Ġback - translation Ġcomes Ġwith Ġthe Ġfirst Ġbatch , Ġbut Ġincreasing Ġthe Ġamount Ġof Ġback - trans lated Ġdata Ġdoes Ġgradually Ġimprove Ġperformance . </s> English âĨ Ķ \ left right arrow ĠRomanian </s> The Ġresults Ġof Ġour ĠEnglish ĠIN LINE FORM 0 ĠRomanian Ġexperiments Ġare Ġshown Ġin ĠTable ĠT AB REF 18 Ġ. ĠThis Ġlanguage Ġpair Ġhas Ġthe Ġsmallest Ġamount Ġof Ġparallel Ġtraining Ġdata , Ġand Ġwe Ġfound Ġdrop out Ġto Ġbe Ġvery Ġeffective , Ġyielding Ġimprovements Ġof Ġ4 âĢĵ 5 ĠBle u . </s> We Ġfound Ġthat Ġthe Ġuse Ġof Ġdi ac rit ics Ġwas Ġinconsistent Ġin Ġthe ĠRomanian Ġtraining Ġ( and Ġdevelopment ) Ġdata , Ġso Ġfor ĠRomanian ĠIN LINE FORM 0 ĠEnglish Ġwe Ġremoved Ġdi ac rit ics Ġfrom Ġthe ĠRomanian Ġsource Ġside , Ġobtaining Ġimprovements Ġof Ġ1 . 3 âĢĵ 1 . 4 ĠBle u . </s> Sy nt hetic Ġtraining Ġdata Ġgives Ġimprovements Ġof Ġ4 . 1 âĢĵ 5 . 1 ĠBle u . Ġfor ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġwe Ġfound Ġthat Ġthe Ġbest Ġsingle Ġsystem Ġoutper formed Ġthe Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints Ġon Ġdev , Ġand Ġwe Ġthus Ġsubmitted Ġthe Ġbest Ġsingle Ġsystem Ġas Ġprimary Ġsystem . </s> English âĨ Ķ \ left right arrow ĠRussian </s> For ĠEnglish ĠIN LINE FORM 0 ĠRussian , Ġwe Ġcannot Ġeffectively Ġlearn ĠB PE Ġon Ġthe Ġjoint Ġvocabulary Ġbecause Ġal ph ab ets Ġdiffer . ĠWe Ġthus Ġfollow Ġthe Ġapproach Ġdescribed Ġin ĠB IB REF 1 Ġ, Ġfirst Ġmapping Ġthe ĠRussian Ġtext Ġinto ĠLatin Ġcharacters Ġvia ĠISO - 9 Ġtransl iter ation , Ġthen Ġlearning Ġthe ĠB PE Ġoperations Ġon Ġthe Ġconc aten ation Ġof Ġthe ĠEnglish Ġand Ġlat in ized ĠRussian Ġtraining Ġdata , Ġthen Ġmapping Ġthe ĠB PE Ġoperations Ġback Ġinto ĠCyr ill ic Ġalphabet . ĠWe Ġapply Ġthe ĠLatin ĠB PE Ġoperations Ġto Ġthe ĠEnglish Ġdata Ġ( training Ġdata Ġand Ġinput ), Ġand Ġboth Ġthe ĠCyr ill ic Ġand ĠLatin ĠB PE Ġoperations Ġto Ġthe ĠRussian Ġdata . </s> Translation Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 21 Ġ. ĠAs Ġfor Ġthe Ġother Ġlanguage Ġpairs , Ġwe Ġobserve Ġstrong Ġimprovements Ġfrom Ġsynthetic Ġtraining Ġdata Ġ( 4 âĢĵ 4 . 4 ĠBle u ). ĠEn semb les Ġyield Ġanother Ġ1 . 1 âĢĵ 1 . 7 ĠBle u . </s> Sh ared ĠTask ĠResults </s> Table ĠT AB REF 22 Ġshows Ġthe Ġranking Ġof Ġour Ġsubmitted Ġsystems Ġat Ġthe ĠW MT 16 Ġshared Ġnews Ġtranslation Ġtask . ĠOur Ġsubmissions Ġare Ġranked Ġ( t ied ) Ġfirst Ġfor Ġ5 Ġout Ġof Ġ8 Ġtranslation Ġdirections Ġin Ġwhich Ġwe Ġparticipated : ĠEN ĠIN LINE FORM 0 ĠCS , ĠEN ĠIN LINE FORM 1 ĠDE , Ġand ĠEN ĠIN LINE FORM 2 ĠRO . ĠThey Ġare Ġalso Ġthe Ġ( t ied ) Ġbest Ġconstrained Ġsystem Ġfor ĠEN ĠIN LINE FORM 3 ĠRU Ġand ĠRO ĠIN LINE FORM 4 ĠEN , Ġor Ġ7 Ġout Ġof Ġ8 Ġtranslation Ġdirections Ġin Ġtotal . </s> Our Ġmodels Ġare Ġalso Ġused Ġin ĠQ T 21 - H im L - Sys Comb ĠB IB REF 10 Ġ, Ġranked Ġ1 âĢĵ 2 Ġfor ĠEN ĠIN LINE FORM 0 ĠRO , Ġand Ġin ĠAM U - U ED IN ĠB IB REF 11 Ġ, Ġranked Ġ2 âĢĵ 3 Ġfor ĠEN ĠIN LINE FORM 1 ĠRU , Ġand Ġ1 âĢĵ 2 Ġfor ĠRU ĠIN LINE FORM 2 ĠEN . </s> Conclusion </s> We Ġdescribe ĠEdinburgh 's Ġneural Ġmachine Ġtranslation Ġsystems Ġfor Ġthe ĠW MT 16 Ġshared Ġnews Ġtranslation Ġtask . ĠFor Ġall Ġtranslation Ġdirections , Ġwe Ġobserve Ġlarge Ġimprovements Ġin Ġtranslation Ġquality Ġfrom Ġusing Ġsynthetic Ġparallel Ġtraining Ġdata , Ġobtained Ġby Ġback - trans l ating Ġin - domain Ġmon oling ual Ġtarget - side Ġdata . ĠP erv asive Ġdrop out Ġon Ġall Ġlayers Ġwas Ġused Ġfor ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġand Ġgave Ġsubstantial Ġimprovements . ĠFor ĠEnglish ĠIN LINE FORM 1 ĠGerman Ġand ĠEnglish ĠIN LINE FORM 2 ĠCzech , Ġwe Ġtrained Ġa Ġright - to - left Ġmodel Ġwith Ġreversed Ġtarget Ġside , Ġand Ġwe Ġfound Ġre ranking Ġthe Ġsystem Ġoutput Ġwith Ġthese Ġreversed Ġmodels Ġhelpful . </s> Acknowled gments </s> This Ġproject Ġhas Ġreceived Ġfunding Ġfrom Ġthe ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġunder Ġgrant Ġagreements Ġ6 45 452 Ġ( Q T 21 ), Ġ6 44 333 Ġ( Tra MO OC ) Ġand Ġ6 44 402 Ġ( H im L ).
Introduction We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs: English INLINEFORM0 Czech, English INLINEFORM1 German, English INLINEFORM2 Romanian and English INLINEFORM3 Russian. Our systems are based on an attentional encoder-decoder BIBREF0 , using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary BIBREF1 . We experimented with using automatic back-translations of the monolingual News corpus as additional training data BIBREF2 , pervasive dropout BIBREF3 , and target-bidirectional models. Baseline System Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout. We use minibatches of size 80, a maximum sentence length of 50, word embeddings of size 500, and hidden layers of size 1024. We clip the gradient norm to 1.0 BIBREF4 . We train the models with Adadelta BIBREF5 , reshuffling the training corpus between epochs. We validate the model every 10000 minibatches via Bleu on a validation set (newstest2013, newstest2014, or half of newsdev2016 for EN INLINEFORM0 RO). We perform early stopping for single models, and use the 4 last saved models (with models saved every 30000 minibatches) for the ensemble results. Note that ensemble scores are the result of a single training run. Due to resource limitations, we did not train ensemble components independently, which could result in more diverse models and better ensembles. Decoding is performed with beam search with a beam size of 12. For some language pairs, we used the AmuNMT C++ decoder as a more efficient alternative to the theano implementation of the dl4mt tutorial. Byte-pair encoding (BPE) To enable open-vocabulary translation, we segment words via byte-pair encoding (BPE) BIBREF1 . BPE, originally devised as a compression algorithm BIBREF6 , is adapted to word segmentation as follows: First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol. All characters are added to the symbol vocabulary. Then, the most frequent symbol pair is identified, and all its occurrences are merged, producing a new symbol that is added to the vocabulary. The previous step is repeated until a set number of merge operations have been learned. BPE starts from a character-level segmentation, but as we increase the number of merge operations, it becomes more and more different from a pure character-level model in that frequent character sequences, and even full words, are encoded as a single symbol. This allows for a trade-off between the size of the model vocabulary and the length of training sequences. The ordered list of merge operations, learned on the training set, can be applied to any text to segment words into subword units that are in-vocabulary in respect to the training set (except for unseen characters). To increase consistency in the segmentation of the source and target text, we combine the source and target side of the training set for learning BPE. For each language pair, we learn 89500 merge operations. Synthetic Training Data WMT provides task participants with large amounts of monolingual data, both in-domain and out-of-domain. We exploit this monolingual data for training as described in BIBREF2 . Specifically, we sample a subset of the available target-side monolingual corpora, translate it automatically into the source side of the respective language pair, and then use this synthetic parallel data for training. For example, for EN INLINEFORM0 RO, the back-translation is performed with a RO INLINEFORM1 EN system, and vice-versa. 2015arXiv151106709S motivate the use of monolingual data with domain adaptation, reducing overfitting, and better modelling of fluency. We sample monolingual data from the News Crawl corpora, which is in-domain with respect to the test set. The amount of monolingual data back-translated for each translation direction ranges from 2 million to 10 million sentences. Statistics about the amount of parallel and synthetic training data are shown in Table TABREF9 . With dl4mt, we observed a translation speed of about 200000 sentences per day (on a single Titan X GPU). Pervasive Dropout For English INLINEFORM0 Romanian, we observed poor performance because of overfitting. To mitigate this, we apply dropout to all layers in the network, including recurrent ones. Previous work dropped out different units at each time step. When applied to recurrent connections, this has the downside that it impedes the information flow over long distances, and DBLP:conf/icfhr/PhamBKL14 propose to only apply dropout to non-recurrent connections. Instead, we follow the approach suggested by 2015arXiv151205287G, and use the same dropout mask at each time step. Our implementation differs from the recommendations by 2015arXiv151205287G in one respect: we also drop words at random, but we do so on a token level, not on a type level. In other words, if a word occurs multiple times in a sentence, we may drop out any number of its occurrences, and not just none or all. In our English INLINEFORM0 Romanian experiments, we drop out full words (both on the source and target side) with a probability of 0.1. For all other layers, the dropout probability is set to 0.2. Target-bidirectional Translation We found that during decoding, the model would occasionally assign a high probability to words based on the target context alone, ignoring the source sentence. We speculate that this is an instance of the label bias problem BIBREF7 . To mitigate this problem, we experiment with training separate models that produce the target text from right-to-left (r2l), and re-scoring the n-best lists that are produced by the main (left-to-right) models with these r2l models. Since the right-to-left model will see a complementary target context at each time step, we expect that the averaged probabilities will be more robust. In parallel to our experiments, this idea was published by liu2016. We increase the size of the n-best-list to 50 for the reranking experiments. A possible criticism of the l-r/r-l reranking approach is that the gains actually come from adding diversity to the ensemble, since we are now using two independent runs. However experiments in BIBREF8 show that a l-r/r-l reranking systems is stronger than an ensemble created from two independent l-r runs. English↔\leftrightarrow German Table TABREF13 shows results for English INLINEFORM0 German. We observe improvements of 3.4–5.7 Bleu from training with a mix of parallel and synthetic data, compared to the baseline that is only trained on parallel data. Using an ensemble of the last 4 checkpoints gives further improvements (1.3–1.7 Bleu). Our submitted system includes reranking of the 50-best output of the left-to-right model with a right-to-left model – again an ensemble of the last 4 checkpoints – with uniform weights. This yields an improvements of 0.6–1.1 Bleu. English↔\leftrightarrow Czech For English INLINEFORM0 Czech, we trained our baseline model on the complete WMT16 parallel training set (including CzEng 1.6pre BIBREF9 ), until we observed convergence on our heldout set (newstest2014). This took approximately 1M minibatches, or 3 weeks. Then we continued training the model on a new parallel corpus, comprising 8.2M sentences back-translated from the Czech monolingual news2015, 5 copies of news-commentary v11, and 9M sentences sampled from Czeng 1.6pre. The model used for back-translation was a neural MT model from earlier experiments, trained on WMT15 data. The training on this synthetic mix continued for a further 400,000 minibatches. The right-left model was trained using a similar process, but with the target side of the parallel corpus reversed prior to training. The resulting model had a slightly lower Bleu score on the dev data than the standard left-right model. We can see in Table TABREF15 that back-translation improves performance by 2.2–2.8 Bleu, and that the final system (+r2l reranking) improves by 0.7–1.0 Bleu on the ensemble of 4, and 4.3–4.9 on the baseline. For Czech INLINEFORM0 English the training process was similar to the above, except that we created the synthetic training data (back-translated from samples of news2015 monolingual English) in batches of 2.5M, and so were able to observe the effect of increasing the amount of synthetic data. After training a baseline model on all the WMT16 parallel set, we continued training with a parallel corpus consisting of 2 copies of the 2.5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. After training this to convergence, we restarted training from the baseline model using 5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. We repeated this with 7.5M sentences from news2015 monolingual, and then with 10M sentences of news2015. The back-translations were, as for English INLINEFORM1 Czech, created with an earlier NMT model trained on WMT15 data. Our final Czech INLINEFORM2 English was an ensemble of 8 systems – the last 4 save-points of the 10M synthetic data run, and the last 4 save-points of the 7.5M run. We show this as ensemble8 in Table TABREF15 , and the +synthetic results are on the last (i.e. 10M) synthetic data run. We also show in Table TABREF16 how increasing the amount of back-translated data affects the results. We see that most of the gain from back-translation comes with the first batch, but increasing the amount of back-translated data does gradually improve performance. English↔\leftrightarrow Romanian The results of our English INLINEFORM0 Romanian experiments are shown in Table TABREF18 . This language pair has the smallest amount of parallel training data, and we found dropout to be very effective, yielding improvements of 4–5 Bleu. We found that the use of diacritics was inconsistent in the Romanian training (and development) data, so for Romanian INLINEFORM0 English we removed diacritics from the Romanian source side, obtaining improvements of 1.3–1.4 Bleu. Synthetic training data gives improvements of 4.1–5.1 Bleu. for English INLINEFORM0 Romanian, we found that the best single system outperformed the ensemble of the last 4 checkpoints on dev, and we thus submitted the best single system as primary system. English↔\leftrightarrow Russian For English INLINEFORM0 Russian, we cannot effectively learn BPE on the joint vocabulary because alphabets differ. We thus follow the approach described in BIBREF1 , first mapping the Russian text into Latin characters via ISO-9 transliteration, then learning the BPE operations on the concatenation of the English and latinized Russian training data, then mapping the BPE operations back into Cyrillic alphabet. We apply the Latin BPE operations to the English data (training data and input), and both the Cyrillic and Latin BPE operations to the Russian data. Translation results are shown in Table TABREF21 . As for the other language pairs, we observe strong improvements from synthetic training data (4–4.4 Bleu). Ensembles yield another 1.1–1.7 Bleu. Shared Task Results Table TABREF22 shows the ranking of our submitted systems at the WMT16 shared news translation task. Our submissions are ranked (tied) first for 5 out of 8 translation directions in which we participated: EN INLINEFORM0 CS, EN INLINEFORM1 DE, and EN INLINEFORM2 RO. They are also the (tied) best constrained system for EN INLINEFORM3 RU and RO INLINEFORM4 EN, or 7 out of 8 translation directions in total. Our models are also used in QT21-HimL-SysComb BIBREF10 , ranked 1–2 for EN INLINEFORM0 RO, and in AMU-UEDIN BIBREF11 , ranked 2–3 for EN INLINEFORM1 RU, and 1–2 for RU INLINEFORM2 EN. Conclusion We describe Edinburgh's neural machine translation systems for the WMT16 shared news translation task. For all translation directions, we observe large improvements in translation quality from using synthetic parallel training data, obtained by back-translating in-domain monolingual target-side data. Pervasive dropout on all layers was used for English INLINEFORM0 Romanian, and gave substantial improvements. For English INLINEFORM1 German and English INLINEFORM2 Czech, we trained a right-to-left model with reversed target side, and we found reranking the system output with these reversed models helpful. Acknowledgments This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), 644333 (TraMOOC) and 644402 (HimL).
what are the baseline systems?
642e8cf1d39faa1cd985d16750cdc6696c52db2f
what Ġare Ġthe Ġbaseline Ġsystems ?
att ention al Ġenc oder - dec oder Ġnetworks ĠB IB REF 0
<s> what Ġare Ġthe Ġbaseline Ġsystems ? </s> Introduction </s> We Ġparticipated Ġin Ġthe ĠW MT Ġ2016 Ġshared Ġnews Ġtranslation Ġtask Ġby Ġbuilding Ġneural Ġtranslation Ġsystems Ġfor Ġfour Ġlanguage Ġpairs : ĠEnglish ĠIN LINE FORM 0 ĠCzech , ĠEnglish ĠIN LINE FORM 1 ĠGerman , ĠEnglish ĠIN LINE FORM 2 ĠRomanian Ġand ĠEnglish ĠIN LINE FORM 3 ĠRussian . ĠOur Ġsystems Ġare Ġbased Ġon Ġan Ġattention al Ġenc oder - dec oder ĠB IB REF 0 Ġ, Ġusing ĠB PE Ġsub word Ġsegment ation Ġfor Ġopen - voc abulary Ġtranslation Ġwith Ġa Ġfixed Ġvocabulary ĠB IB REF 1 Ġ. ĠWe Ġexperimented Ġwith Ġusing Ġautomatic Ġback - trans lations Ġof Ġthe Ġmon oling ual ĠNews Ġcorpus Ġas Ġadditional Ġtraining Ġdata ĠB IB REF 2 Ġ, Ġpervasive Ġdrop out ĠB IB REF 3 Ġ, Ġand Ġtarget - bid irection al Ġmodels . </s> Bas eline ĠSystem </s> Our Ġsystems Ġare Ġattention al Ġenc oder - dec oder Ġnetworks ĠB IB REF 0 Ġ. ĠWe Ġbase Ġour Ġimplementation Ġon Ġthe Ġd l 4 mt - t utorial , Ġwhich Ġwe Ġenhanced Ġwith Ġnew Ġfeatures Ġsuch Ġas Ġensemble Ġdecoding Ġand Ġpervasive Ġdrop out . </s> We Ġuse Ġmin ib atches Ġof Ġsize Ġ80 , Ġa Ġmaximum Ġsentence Ġlength Ġof Ġ50 , Ġword Ġembed d ings Ġof Ġsize Ġ500 , Ġand Ġhidden Ġlayers Ġof Ġsize Ġ1024 . ĠWe Ġclip Ġthe Ġgradient Ġnorm Ġto Ġ1 . 0 ĠB IB REF 4 Ġ. ĠWe Ġtrain Ġthe Ġmodels Ġwith ĠAd ad elta ĠB IB REF 5 Ġ, Ġresh uff ling Ġthe Ġtraining Ġcorpus Ġbetween Ġepoch s . ĠWe Ġvalidate Ġthe Ġmodel Ġevery Ġ10000 Ġmin ib atches Ġvia ĠBle u Ġon Ġa Ġvalidation Ġset Ġ( new st est 2013 , Ġnew st est 2014 , Ġor Ġhalf Ġof Ġnews dev 2016 Ġfor ĠEN ĠIN LINE FORM 0 ĠRO ). ĠWe Ġperform Ġearly Ġstopping Ġfor Ġsingle Ġmodels , Ġand Ġuse Ġthe Ġ4 Ġlast Ġsaved Ġmodels Ġ( with Ġmodels Ġsaved Ġevery Ġ3 0000 Ġmin ib atches ) Ġfor Ġthe Ġensemble Ġresults . ĠNote Ġthat Ġensemble Ġscores Ġare Ġthe Ġresult Ġof Ġa Ġsingle Ġtraining Ġrun . ĠDue Ġto Ġresource Ġlimitations , Ġwe Ġdid Ġnot Ġtrain Ġensemble Ġcomponents Ġindependently , Ġwhich Ġcould Ġresult Ġin Ġmore Ġdiverse Ġmodels Ġand Ġbetter Ġen semb les . </s> Dec oding Ġis Ġperformed Ġwith Ġbeam Ġsearch Ġwith Ġa Ġbeam Ġsize Ġof Ġ12 . ĠFor Ġsome Ġlanguage Ġpairs , Ġwe Ġused Ġthe ĠAm u N MT ĠC ++ Ġdec oder Ġas Ġa Ġmore Ġefficient Ġalternative Ġto Ġthe Ġthe ano Ġimplementation Ġof Ġthe Ġd l 4 mt Ġtutorial . </s> Byte - pair Ġencoding Ġ( B PE ) </s> To Ġenable Ġopen - voc abulary Ġtranslation , Ġwe Ġsegment Ġwords Ġvia Ġbyte - pair Ġencoding Ġ( B PE ) ĠB IB REF 1 Ġ. ĠB PE , Ġoriginally Ġdevised Ġas Ġa Ġcompression Ġalgorithm ĠB IB REF 6 Ġ, Ġis Ġadapted Ġto Ġword Ġsegment ation Ġas Ġfollows : </s> First , Ġeach Ġword Ġin Ġthe Ġtraining Ġvocabulary Ġis Ġrepresented Ġas Ġa Ġsequence Ġof Ġcharacters , Ġplus Ġan Ġend - of - word Ġsymbol . ĠAll Ġcharacters Ġare Ġadded Ġto Ġthe Ġsymbol Ġvocabulary . ĠThen , Ġthe Ġmost Ġfrequent Ġsymbol Ġpair Ġis Ġidentified , Ġand Ġall Ġits Ġoccurrences Ġare Ġmerged , Ġproducing Ġa Ġnew Ġsymbol Ġthat Ġis Ġadded Ġto Ġthe Ġvocabulary . ĠThe Ġprevious Ġstep Ġis Ġrepeated Ġuntil Ġa Ġset Ġnumber Ġof Ġmerge Ġoperations Ġhave Ġbeen Ġlearned . </s> B PE Ġstarts Ġfrom Ġa Ġcharacter - level Ġsegment ation , Ġbut Ġas Ġwe Ġincrease Ġthe Ġnumber Ġof Ġmerge Ġoperations , Ġit Ġbecomes Ġmore Ġand Ġmore Ġdifferent Ġfrom Ġa Ġpure Ġcharacter - level Ġmodel Ġin Ġthat Ġfrequent Ġcharacter Ġsequences , Ġand Ġeven Ġfull Ġwords , Ġare Ġencoded Ġas Ġa Ġsingle Ġsymbol . ĠThis Ġallows Ġfor Ġa Ġtrade - off Ġbetween Ġthe Ġsize Ġof Ġthe Ġmodel Ġvocabulary Ġand Ġthe Ġlength Ġof Ġtraining Ġsequences . ĠThe Ġordered Ġlist Ġof Ġmerge Ġoperations , Ġlearned Ġon Ġthe Ġtraining Ġset , Ġcan Ġbe Ġapplied Ġto Ġany Ġtext Ġto Ġsegment Ġwords Ġinto Ġsub word Ġunits Ġthat Ġare Ġin - voc abulary Ġin Ġrespect Ġto Ġthe Ġtraining Ġset Ġ( except Ġfor Ġunseen Ġcharacters ). </s> To Ġincrease Ġconsistency Ġin Ġthe Ġsegment ation Ġof Ġthe Ġsource Ġand Ġtarget Ġtext , Ġwe Ġcombine Ġthe Ġsource Ġand Ġtarget Ġside Ġof Ġthe Ġtraining Ġset Ġfor Ġlearning ĠB PE . ĠFor Ġeach Ġlanguage Ġpair , Ġwe Ġlearn Ġ8 95 00 Ġmerge Ġoperations . </s> Sy nt hetic ĠTraining ĠData </s> W MT Ġprovides Ġtask Ġparticipants Ġwith Ġlarge Ġamounts Ġof Ġmon oling ual Ġdata , Ġboth Ġin - domain Ġand Ġout - of - domain . ĠWe Ġexploit Ġthis Ġmon oling ual Ġdata Ġfor Ġtraining Ġas Ġdescribed Ġin ĠB IB REF 2 Ġ. ĠSpecifically , Ġwe Ġsample Ġa Ġsubset Ġof Ġthe Ġavailable Ġtarget - side Ġmon oling ual Ġcorpor a , Ġtranslate Ġit Ġautomatically Ġinto Ġthe Ġsource Ġside Ġof Ġthe Ġrespective Ġlanguage Ġpair , Ġand Ġthen Ġuse Ġthis Ġsynthetic Ġparallel Ġdata Ġfor Ġtraining . ĠFor Ġexample , Ġfor ĠEN ĠIN LINE FORM 0 ĠRO , Ġthe Ġback - translation Ġis Ġperformed Ġwith Ġa ĠRO ĠIN LINE FORM 1 ĠEN Ġsystem , Ġand Ġvice - vers a . </s> 2015 ar X iv 15 110 6 709 S Ġmotivate Ġthe Ġuse Ġof Ġmon oling ual Ġdata Ġwith Ġdomain Ġadaptation , Ġreducing Ġover fitting , Ġand Ġbetter Ġmodelling Ġof Ġflu ency . ĠWe Ġsample Ġmon oling ual Ġdata Ġfrom Ġthe ĠNews ĠC rawl Ġcorpor a , Ġwhich Ġis Ġin - domain Ġwith Ġrespect Ġto Ġthe Ġtest Ġset . </s> The Ġamount Ġof Ġmon oling ual Ġdata Ġback - trans lated Ġfor Ġeach Ġtranslation Ġdirection Ġranges Ġfrom Ġ2 Ġmillion Ġto Ġ10 Ġmillion Ġsentences . ĠStatistics Ġabout Ġthe Ġamount Ġof Ġparallel Ġand Ġsynthetic Ġtraining Ġdata Ġare Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. ĠWith Ġd l 4 mt , Ġwe Ġobserved Ġa Ġtranslation Ġspeed Ġof Ġabout Ġ200 000 Ġsentences Ġper Ġday Ġ( on Ġa Ġsingle ĠTitan ĠX ĠGPU ). </s> P erv asive ĠDrop out </s> For ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġwe Ġobserved Ġpoor Ġperformance Ġbecause Ġof Ġover fitting . ĠTo Ġmitigate Ġthis , Ġwe Ġapply Ġdrop out Ġto Ġall Ġlayers Ġin Ġthe Ġnetwork , Ġincluding Ġrecurrent Ġones . </s> Previous Ġwork Ġdropped Ġout Ġdifferent Ġunits Ġat Ġeach Ġtime Ġstep . ĠWhen Ġapplied Ġto Ġrecurrent Ġconnections , Ġthis Ġhas Ġthe Ġdownside Ġthat Ġit Ġimped es Ġthe Ġinformation Ġflow Ġover Ġlong Ġdistances , Ġand ĠD BL P : conf / ic f hr / Ph am B K L 14 Ġpropose Ġto Ġonly Ġapply Ġdrop out Ġto Ġnon - rec urrent Ġconnections . </s> Instead , Ġwe Ġfollow Ġthe Ġapproach Ġsuggested Ġby Ġ2015 ar X iv 15 12 05 287 G , Ġand Ġuse Ġthe Ġsame Ġdrop out Ġmask Ġat Ġeach Ġtime Ġstep . ĠOur Ġimplementation Ġdiffers Ġfrom Ġthe Ġrecommendations Ġby Ġ2015 ar X iv 15 12 05 287 G Ġin Ġone Ġrespect : Ġwe Ġalso Ġdrop Ġwords Ġat Ġrandom , Ġbut Ġwe Ġdo Ġso Ġon Ġa Ġtoken Ġlevel , Ġnot Ġon Ġa Ġtype Ġlevel . ĠIn Ġother Ġwords , Ġif Ġa Ġword Ġoccurs Ġmultiple Ġtimes Ġin Ġa Ġsentence , Ġwe Ġmay Ġdrop Ġout Ġany Ġnumber Ġof Ġits Ġoccurrences , Ġand Ġnot Ġjust Ġnone Ġor Ġall . </s> In Ġour ĠEnglish ĠIN LINE FORM 0 ĠRomanian Ġexperiments , Ġwe Ġdrop Ġout Ġfull Ġwords Ġ( both Ġon Ġthe Ġsource Ġand Ġtarget Ġside ) Ġwith Ġa Ġprobability Ġof Ġ0 . 1 . ĠFor Ġall Ġother Ġlayers , Ġthe Ġdrop out Ġprobability Ġis Ġset Ġto Ġ0 . 2 . </s> Target - bid irection al ĠTranslation </s> We Ġfound Ġthat Ġduring Ġdecoding , Ġthe Ġmodel Ġwould Ġoccasionally Ġassign Ġa Ġhigh Ġprobability Ġto Ġwords Ġbased Ġon Ġthe Ġtarget Ġcontext Ġalone , Ġignoring Ġthe Ġsource Ġsentence . ĠWe Ġspeculate Ġthat Ġthis Ġis Ġan Ġinstance Ġof Ġthe Ġlabel Ġbias Ġproblem ĠB IB REF 7 Ġ. </s> To Ġmitigate Ġthis Ġproblem , Ġwe Ġexperiment Ġwith Ġtraining Ġseparate Ġmodels Ġthat Ġproduce Ġthe Ġtarget Ġtext Ġfrom Ġright - to - left Ġ( r 2 l ), Ġand Ġre - scoring Ġthe Ġn - best Ġlists Ġthat Ġare Ġproduced Ġby Ġthe Ġmain Ġ( left - to - right ) Ġmodels Ġwith Ġthese Ġr 2 l Ġmodels . ĠSince Ġthe Ġright - to - left Ġmodel Ġwill Ġsee Ġa Ġcomplementary Ġtarget Ġcontext Ġat Ġeach Ġtime Ġstep , Ġwe Ġexpect Ġthat Ġthe Ġaveraged Ġprobabilities Ġwill Ġbe Ġmore Ġrobust . ĠIn Ġparallel Ġto Ġour Ġexperiments , Ġthis Ġidea Ġwas Ġpublished Ġby Ġli u 2016 . </s> We Ġincrease Ġthe Ġsize Ġof Ġthe Ġn - best - list Ġto Ġ50 Ġfor Ġthe Ġre ranking Ġexperiments . </s> A Ġpossible Ġcriticism Ġof Ġthe Ġl - r / r - l Ġre ranking Ġapproach Ġis Ġthat Ġthe Ġgains Ġactually Ġcome Ġfrom Ġadding Ġdiversity Ġto Ġthe Ġensemble , Ġsince Ġwe Ġare Ġnow Ġusing Ġtwo Ġindependent Ġruns . ĠHowever Ġexperiments Ġin ĠB IB REF 8 Ġshow Ġthat Ġa Ġl - r / r - l Ġre ranking Ġsystems Ġis Ġstronger Ġthan Ġan Ġensemble Ġcreated Ġfrom Ġtwo Ġindependent Ġl - r Ġruns . </s> English âĨ Ķ \ left right arrow ĠGerman </s> Table ĠT AB REF 13 Ġshows Ġresults Ġfor ĠEnglish ĠIN LINE FORM 0 ĠGerman . ĠWe Ġobserve Ġimprovements Ġof Ġ3 . 4 âĢĵ 5 . 7 ĠBle u Ġfrom Ġtraining Ġwith Ġa Ġmix Ġof Ġparallel Ġand Ġsynthetic Ġdata , Ġcompared Ġto Ġthe Ġbaseline Ġthat Ġis Ġonly Ġtrained Ġon Ġparallel Ġdata . ĠUsing Ġan Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints Ġgives Ġfurther Ġimprovements Ġ( 1 . 3 âĢĵ 1 . 7 ĠBle u ). ĠOur Ġsubmitted Ġsystem Ġincludes Ġre ranking Ġof Ġthe Ġ50 - best Ġoutput Ġof Ġthe Ġleft - to - right Ġmodel Ġwith Ġa Ġright - to - left Ġmodel ĠâĢĵ Ġagain Ġan Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints ĠâĢĵ Ġwith Ġuniform Ġweights . ĠThis Ġyields Ġan Ġimprovements Ġof Ġ0 . 6 âĢĵ 1 . 1 ĠBle u . </s> English âĨ Ķ \ left right arrow ĠCzech </s> For ĠEnglish ĠIN LINE FORM 0 ĠCzech , Ġwe Ġtrained Ġour Ġbaseline Ġmodel Ġon Ġthe Ġcomplete ĠW MT 16 Ġparallel Ġtraining Ġset Ġ( including ĠC z Eng Ġ1 . 6 pre ĠB IB REF 9 Ġ), Ġuntil Ġwe Ġobserved Ġconvergence Ġon Ġour Ġheld out Ġset Ġ( new st est 2014 ). ĠThis Ġtook Ġapproximately Ġ1 M Ġmin ib atches , Ġor Ġ3 Ġweeks . ĠThen Ġwe Ġcontinued Ġtraining Ġthe Ġmodel Ġon Ġa Ġnew Ġparallel Ġcorpus , Ġcomprising Ġ8 . 2 M Ġsentences Ġback - trans lated Ġfrom Ġthe ĠCzech Ġmon oling ual Ġnews 2015 , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġ9 M Ġsentences Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠThe Ġmodel Ġused Ġfor Ġback - translation Ġwas Ġa Ġneural ĠMT Ġmodel Ġfrom Ġearlier Ġexperiments , Ġtrained Ġon ĠW MT 15 Ġdata . ĠThe Ġtraining Ġon Ġthis Ġsynthetic Ġmix Ġcontinued Ġfor Ġa Ġfurther Ġ400 , 000 Ġmin ib atches . </s> The Ġright - left Ġmodel Ġwas Ġtrained Ġusing Ġa Ġsimilar Ġprocess , Ġbut Ġwith Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġcorpus Ġreversed Ġprior Ġto Ġtraining . ĠThe Ġresulting Ġmodel Ġhad Ġa Ġslightly Ġlower ĠBle u Ġscore Ġon Ġthe Ġdev Ġdata Ġthan Ġthe Ġstandard Ġleft - right Ġmodel . ĠWe Ġcan Ġsee Ġin ĠTable ĠT AB REF 15 Ġthat Ġback - translation Ġimproves Ġperformance Ġby Ġ2 . 2 âĢĵ 2 . 8 ĠBle u , Ġand Ġthat Ġthe Ġfinal Ġsystem Ġ(+ r 2 l Ġre ranking ) Ġimproves Ġby Ġ0 . 7 âĢĵ 1 . 0 ĠBle u Ġon Ġthe Ġensemble Ġof Ġ4 , Ġand Ġ4 . 3 âĢĵ 4 . 9 Ġon Ġthe Ġbaseline . </s> For ĠCzech ĠIN LINE FORM 0 ĠEnglish Ġthe Ġtraining Ġprocess Ġwas Ġsimilar Ġto Ġthe Ġabove , Ġexcept Ġthat Ġwe Ġcreated Ġthe Ġsynthetic Ġtraining Ġdata Ġ( back - trans lated Ġfrom Ġsamples Ġof Ġnews 2015 Ġmon oling ual ĠEnglish ) Ġin Ġbatches Ġof Ġ2 . 5 M , Ġand Ġso Ġwere Ġable Ġto Ġobserve Ġthe Ġeffect Ġof Ġincreasing Ġthe Ġamount Ġof Ġsynthetic Ġdata . ĠAfter Ġtraining Ġa Ġbaseline Ġmodel Ġon Ġall Ġthe ĠW MT 16 Ġparallel Ġset , Ġwe Ġcontinued Ġtraining Ġwith Ġa Ġparallel Ġcorpus Ġconsisting Ġof Ġ2 Ġcopies Ġof Ġthe Ġ2 . 5 M Ġsentences Ġof Ġback - trans lated Ġdata , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġa Ġmatching Ġquantity Ġof Ġdata Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠAfter Ġtraining Ġthis Ġto Ġconvergence , Ġwe Ġrestart ed Ġtraining Ġfrom Ġthe Ġbaseline Ġmodel Ġusing Ġ5 M Ġsentences Ġof Ġback - trans lated Ġdata , Ġ5 Ġcopies Ġof Ġnews - comment ary Ġv 11 , Ġand Ġa Ġmatching Ġquantity Ġof Ġdata Ġsampled Ġfrom ĠC z eng Ġ1 . 6 pre . ĠWe Ġrepeated Ġthis Ġwith Ġ7 . 5 M Ġsentences Ġfrom Ġnews 2015 Ġmon oling ual , Ġand Ġthen Ġwith Ġ10 M Ġsentences Ġof Ġnews 2015 . ĠThe Ġback - trans lations Ġwere , Ġas Ġfor ĠEnglish ĠIN LINE FORM 1 ĠCzech , Ġcreated Ġwith Ġan Ġearlier ĠN MT Ġmodel Ġtrained Ġon ĠW MT 15 Ġdata . ĠOur Ġfinal ĠCzech ĠIN LINE FORM 2 ĠEnglish Ġwas Ġan Ġensemble Ġof Ġ8 Ġsystems ĠâĢĵ Ġthe Ġlast Ġ4 Ġsave - points Ġof Ġthe Ġ10 M Ġsynthetic Ġdata Ġrun , Ġand Ġthe Ġlast Ġ4 Ġsave - points Ġof Ġthe Ġ7 . 5 M Ġrun . ĠWe Ġshow Ġthis Ġas Ġensemble 8 Ġin ĠTable ĠT AB REF 15 Ġ, Ġand Ġthe Ġ+ sy nt hetic Ġresults Ġare Ġon Ġthe Ġlast Ġ( i . e . Ġ10 M ) Ġsynthetic Ġdata Ġrun . </s> We Ġalso Ġshow Ġin ĠTable ĠT AB REF 16 Ġhow Ġincreasing Ġthe Ġamount Ġof Ġback - trans lated Ġdata Ġaffects Ġthe Ġresults . ĠWe Ġsee Ġthat Ġmost Ġof Ġthe Ġgain Ġfrom Ġback - translation Ġcomes Ġwith Ġthe Ġfirst Ġbatch , Ġbut Ġincreasing Ġthe Ġamount Ġof Ġback - trans lated Ġdata Ġdoes Ġgradually Ġimprove Ġperformance . </s> English âĨ Ķ \ left right arrow ĠRomanian </s> The Ġresults Ġof Ġour ĠEnglish ĠIN LINE FORM 0 ĠRomanian Ġexperiments Ġare Ġshown Ġin ĠTable ĠT AB REF 18 Ġ. ĠThis Ġlanguage Ġpair Ġhas Ġthe Ġsmallest Ġamount Ġof Ġparallel Ġtraining Ġdata , Ġand Ġwe Ġfound Ġdrop out Ġto Ġbe Ġvery Ġeffective , Ġyielding Ġimprovements Ġof Ġ4 âĢĵ 5 ĠBle u . </s> We Ġfound Ġthat Ġthe Ġuse Ġof Ġdi ac rit ics Ġwas Ġinconsistent Ġin Ġthe ĠRomanian Ġtraining Ġ( and Ġdevelopment ) Ġdata , Ġso Ġfor ĠRomanian ĠIN LINE FORM 0 ĠEnglish Ġwe Ġremoved Ġdi ac rit ics Ġfrom Ġthe ĠRomanian Ġsource Ġside , Ġobtaining Ġimprovements Ġof Ġ1 . 3 âĢĵ 1 . 4 ĠBle u . </s> Sy nt hetic Ġtraining Ġdata Ġgives Ġimprovements Ġof Ġ4 . 1 âĢĵ 5 . 1 ĠBle u . Ġfor ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġwe Ġfound Ġthat Ġthe Ġbest Ġsingle Ġsystem Ġoutper formed Ġthe Ġensemble Ġof Ġthe Ġlast Ġ4 Ġcheckpoints Ġon Ġdev , Ġand Ġwe Ġthus Ġsubmitted Ġthe Ġbest Ġsingle Ġsystem Ġas Ġprimary Ġsystem . </s> English âĨ Ķ \ left right arrow ĠRussian </s> For ĠEnglish ĠIN LINE FORM 0 ĠRussian , Ġwe Ġcannot Ġeffectively Ġlearn ĠB PE Ġon Ġthe Ġjoint Ġvocabulary Ġbecause Ġal ph ab ets Ġdiffer . ĠWe Ġthus Ġfollow Ġthe Ġapproach Ġdescribed Ġin ĠB IB REF 1 Ġ, Ġfirst Ġmapping Ġthe ĠRussian Ġtext Ġinto ĠLatin Ġcharacters Ġvia ĠISO - 9 Ġtransl iter ation , Ġthen Ġlearning Ġthe ĠB PE Ġoperations Ġon Ġthe Ġconc aten ation Ġof Ġthe ĠEnglish Ġand Ġlat in ized ĠRussian Ġtraining Ġdata , Ġthen Ġmapping Ġthe ĠB PE Ġoperations Ġback Ġinto ĠCyr ill ic Ġalphabet . ĠWe Ġapply Ġthe ĠLatin ĠB PE Ġoperations Ġto Ġthe ĠEnglish Ġdata Ġ( training Ġdata Ġand Ġinput ), Ġand Ġboth Ġthe ĠCyr ill ic Ġand ĠLatin ĠB PE Ġoperations Ġto Ġthe ĠRussian Ġdata . </s> Translation Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 21 Ġ. ĠAs Ġfor Ġthe Ġother Ġlanguage Ġpairs , Ġwe Ġobserve Ġstrong Ġimprovements Ġfrom Ġsynthetic Ġtraining Ġdata Ġ( 4 âĢĵ 4 . 4 ĠBle u ). ĠEn semb les Ġyield Ġanother Ġ1 . 1 âĢĵ 1 . 7 ĠBle u . </s> Sh ared ĠTask ĠResults </s> Table ĠT AB REF 22 Ġshows Ġthe Ġranking Ġof Ġour Ġsubmitted Ġsystems Ġat Ġthe ĠW MT 16 Ġshared Ġnews Ġtranslation Ġtask . ĠOur Ġsubmissions Ġare Ġranked Ġ( t ied ) Ġfirst Ġfor Ġ5 Ġout Ġof Ġ8 Ġtranslation Ġdirections Ġin Ġwhich Ġwe Ġparticipated : ĠEN ĠIN LINE FORM 0 ĠCS , ĠEN ĠIN LINE FORM 1 ĠDE , Ġand ĠEN ĠIN LINE FORM 2 ĠRO . ĠThey Ġare Ġalso Ġthe Ġ( t ied ) Ġbest Ġconstrained Ġsystem Ġfor ĠEN ĠIN LINE FORM 3 ĠRU Ġand ĠRO ĠIN LINE FORM 4 ĠEN , Ġor Ġ7 Ġout Ġof Ġ8 Ġtranslation Ġdirections Ġin Ġtotal . </s> Our Ġmodels Ġare Ġalso Ġused Ġin ĠQ T 21 - H im L - Sys Comb ĠB IB REF 10 Ġ, Ġranked Ġ1 âĢĵ 2 Ġfor ĠEN ĠIN LINE FORM 0 ĠRO , Ġand Ġin ĠAM U - U ED IN ĠB IB REF 11 Ġ, Ġranked Ġ2 âĢĵ 3 Ġfor ĠEN ĠIN LINE FORM 1 ĠRU , Ġand Ġ1 âĢĵ 2 Ġfor ĠRU ĠIN LINE FORM 2 ĠEN . </s> Conclusion </s> We Ġdescribe ĠEdinburgh 's Ġneural Ġmachine Ġtranslation Ġsystems Ġfor Ġthe ĠW MT 16 Ġshared Ġnews Ġtranslation Ġtask . ĠFor Ġall Ġtranslation Ġdirections , Ġwe Ġobserve Ġlarge Ġimprovements Ġin Ġtranslation Ġquality Ġfrom Ġusing Ġsynthetic Ġparallel Ġtraining Ġdata , Ġobtained Ġby Ġback - trans l ating Ġin - domain Ġmon oling ual Ġtarget - side Ġdata . ĠP erv asive Ġdrop out Ġon Ġall Ġlayers Ġwas Ġused Ġfor ĠEnglish ĠIN LINE FORM 0 ĠRomanian , Ġand Ġgave Ġsubstantial Ġimprovements . ĠFor ĠEnglish ĠIN LINE FORM 1 ĠGerman Ġand ĠEnglish ĠIN LINE FORM 2 ĠCzech , Ġwe Ġtrained Ġa Ġright - to - left Ġmodel Ġwith Ġreversed Ġtarget Ġside , Ġand Ġwe Ġfound Ġre ranking Ġthe Ġsystem Ġoutput Ġwith Ġthese Ġreversed Ġmodels Ġhelpful . </s> Acknowled gments </s> This Ġproject Ġhas Ġreceived Ġfunding Ġfrom Ġthe ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġunder Ġgrant Ġagreements Ġ6 45 452 Ġ( Q T 21 ), Ġ6 44 333 Ġ( Tra MO OC ) Ġand Ġ6 44 402 Ġ( H im L ).