the English-German dataset
EXTRACTIVE
Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German. We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25 . We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information. We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30 . Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ).
1906.07701
Data </s> We Ġbuild Ġand Ġtest Ġour ĠM MT Ġmodels Ġon Ġthe ĠMulti 30 K Ġdataset ĠB IB REF 21 Ġ. ĠEach Ġimage Ġin ĠMulti 30 K Ġcontains Ġone ĠEnglish Ġ( EN ) Ġdescription Ġtaken Ġfrom ĠFlickr 30 K ĠB IB REF 22 Ġand Ġhuman Ġtranslations Ġinto ĠGerman Ġ( DE ), ĠFrench Ġ( FR ) Ġand ĠCzech ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠThe Ġdataset Ġcontains Ġ29 , 000 Ġinstances Ġfor Ġtraining , Ġ1 , 014 Ġfor Ġdevelopment , Ġand Ġ1 , 000 Ġfor Ġtest . ĠWe Ġonly Ġexperiment Ġwith ĠGerman Ġand ĠFrench , Ġwhich Ġare Ġlanguages Ġfor Ġwhich Ġwe Ġhave Ġin - house Ġexpertise Ġfor Ġthe Ġtype Ġof Ġanalysis Ġwe Ġpresent . ĠIn Ġaddition Ġto Ġthe Ġofficial ĠMulti 30 K Ġtest Ġset Ġ( test Ġ2016 ), Ġwe Ġalso Ġuse Ġthe Ġtest Ġset Ġfrom Ġthe Ġlatest ĠW MT Ġevaluation Ġcompetition , Ġtest Ġ2018 ĠB IB REF 25 Ġ. </s> D eg rad ation Ġof Ġsource </s> In Ġaddition Ġto Ġusing Ġthe ĠMulti 30 K Ġdataset Ġas Ġis Ġ( standard Ġsetup ), Ġwe Ġprobe Ġthe Ġability Ġof Ġour Ġmodels Ġto Ġaddress Ġthe Ġthree Ġlinguistic Ġphenomena Ġwhere Ġadditional Ġcontext Ġhas Ġbeen Ġproved Ġimportant Ġ( Section Ġ): Ġambig u ities , Ġgender - neutral Ġwords Ġand Ġnoisy Ġinput . ĠIn Ġa Ġcontrolled Ġexperiment Ġwhere Ġwe Ġaim Ġto Ġremove Ġthe Ġinfluence Ġof Ġfrequency Ġbiases , Ġwe Ġdegrade Ġthe Ġsource Ġsentences Ġby Ġmask ing Ġwords Ġthrough Ġthree Ġstrategies Ġto Ġreplace Ġwords Ġby Ġa Ġplaceholder : Ġrandom Ġsource Ġwords , Ġambiguous Ġsource Ġwords Ġand Ġgender Ġunmarked Ġsource Ġwords . ĠThe Ġprocedure Ġis Ġapplied Ġto Ġthe Ġtrain , Ġvalidation Ġand Ġtest Ġsets . ĠFor Ġthe Ġresulting Ġdataset Ġgenerated Ġfor Ġeach Ġsetting , Ġwe Ġcompare Ġmodels Ġhaving Ġaccess Ġto Ġtext - only Ġcontext Ġversus Ġadditional Ġtext Ġand Ġmultim od al Ġcontexts . ĠWe Ġseek Ġto Ġget Ġinsights Ġinto Ġthe Ġcontribution Ġof Ġeach Ġtype Ġof Ġcontext Ġto Ġaddress Ġeach Ġtype Ġof Ġdegradation . </s> In Ġthis Ġsetting Ġ( R ND ) Ġwe Ġsimulate Ġerroneous Ġsource Ġwords Ġby Ġrandomly Ġdropping Ġsource Ġcontent Ġwords . ĠWe Ġfirst Ġtag Ġthe Ġentire Ġsource Ġsentences Ġusing Ġthe Ġsp acy Ġtool kit ĠB IB REF 26 Ġand Ġthen Ġdrop Ġnoun s , Ġverbs , Ġadject ives Ġand Ġad verbs Ġand Ġreplace Ġthese Ġwith Ġa Ġdefault ĠBL ANK Ġtoken . ĠBy Ġfocusing Ġon Ġcontent Ġwords , Ġwe Ġdiffer Ġfrom Ġprevious Ġwork Ġthat Ġsuggests Ġthat Ġneural Ġmachine Ġtranslation Ġis Ġrobust Ġto Ġnon - content Ġword Ġnoise Ġin Ġthe Ġsource ĠB IB REF 27 Ġ. </s> In Ġthis Ġsetting Ġ( AM B ), Ġwe Ġrely Ġon Ġthe ĠML T Ġdataset ĠB IB REF 11 Ġwhich Ġprovides Ġa Ġlist Ġof Ġsource Ġwords Ġwith Ġmultiple Ġtranslations Ġin Ġthe ĠMulti 30 k Ġtraining Ġset . ĠWe Ġreplace Ġambiguous Ġwords Ġwith Ġthe ĠBL ANK Ġtoken Ġin Ġthe Ġsource Ġlanguage , Ġwhich Ġresults Ġin Ġtwo Ġlanguage - specific Ġdatasets . </s> In Ġthis Ġsetting Ġ( P ERS ), Ġwe Ġuse Ġthe ĠFlickr ĠEnt ities Ġdataset ĠB IB REF 28 Ġto Ġidentify Ġall Ġthe Ġwords Ġthat Ġwere Ġannot ated Ġby Ġhumans Ġas Ġcorresponding Ġto Ġthe Ġcategory Ġperson . ĠWe Ġthen Ġreplace Ġsuch Ġsource Ġwords Ġwith Ġthe ĠBL ANK Ġtoken . </s> The Ġstatistics Ġof Ġthe Ġresulting Ġdatasets Ġfor Ġthe Ġthree Ġdegradation Ġstrategies Ġare Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. ĠWe Ġnote Ġthat ĠR ND Ġand ĠP ERS Ġare Ġthe Ġsame Ġfor Ġlanguage Ġpairs Ġas Ġthe Ġdegradation Ġonly Ġdepends Ġon Ġthe Ġsource Ġside , Ġwhile Ġfor ĠAM B Ġthe Ġwords Ġreplaced Ġdepend Ġon Ġthe Ġtarget Ġlanguage . </s> Mod els </s> Based Ġon Ġthe Ġmodels Ġdescribed Ġin ĠSection Ġwe Ġexperiment Ġwith Ġeight Ġvariants : Ġ( a ) Ġbaseline Ġtransformer Ġmodel Ġ( base ); Ġ( b ) Ġbase Ġwith ĠA IC Ġ( base + sum ); Ġ( c ) Ġbase Ġwith ĠA IF Ġusing Ġsp acial Ġ( base + att ) Ġor Ġobject Ġbased Ġ( base + obj ) Ġimage Ġfeatures ; Ġ( d ) Ġstandard Ġdeliber ation Ġmodel Ġ( del ); Ġ( e ) Ġdeliber ation Ġmodels Ġenriched Ġwith Ġimage Ġinformation : Ġdel + sum , Ġdel + att Ġand Ġdel + obj . </s> Training </s> In Ġall Ġcases , Ġwe Ġoptim ise Ġour Ġmodels Ġwith Ġcross Ġentropy Ġloss . ĠFor Ġdeliber ation Ġnetwork Ġmodels , Ġwe Ġfirst Ġtrain Ġthe Ġstandard Ġtransformer Ġmodel Ġuntil Ġconvergence , Ġand Ġuse Ġit Ġto Ġinitial ise Ġthe Ġenc oder Ġand Ġfirst - pass Ġdec oder . ĠFor Ġeach Ġof Ġthe Ġtraining Ġsamples , Ġwe Ġfollow ĠB IB REF 19 Ġand Ġobtain Ġa Ġset Ġof Ġ10 - best Ġsamples Ġfrom Ġthe Ġfirst Ġpass Ġdec oder , Ġwith Ġa Ġbeam Ġsearch Ġof Ġsize Ġ10 . ĠWe Ġuse Ġthese Ġas Ġthe Ġfirst - pass Ġdec oder Ġsamples . ĠWe Ġuse ĠAdam Ġas Ġoptim iser ĠB IB REF 29 Ġand Ġtrain Ġthe Ġmodel Ġuntil Ġconvergence . </s> Results </s> In Ġthis Ġsection Ġwe Ġpresent Ġresults Ġof Ġour Ġexperiments , Ġfirst Ġin Ġthe Ġoriginal Ġdataset Ġwithout Ġany Ġsource Ġdegradation Ġ( Section ĠSEC REF 18 Ġ) Ġand Ġthen Ġin Ġthe Ġsetup Ġwith Ġvarious Ġsource Ġdegradation Ġstrategies Ġ( Section ĠSEC REF 25 Ġ). </s> Standard Ġsetup </s> Table ĠT AB REF 14 Ġshows Ġthe Ġresults Ġof Ġour Ġmain Ġexperiments Ġon Ġthe Ġ2016 Ġand Ġ2018 Ġtest Ġsets Ġfor ĠFrench Ġand ĠGerman . ĠWe Ġuse ĠMeteor ĠB IB REF 31 Ġas Ġthe Ġmain Ġmetric , Ġas Ġin Ġthe ĠW MT Ġtasks ĠB IB REF 25 Ġ. ĠWe Ġcompare Ġour Ġtransformer Ġbaseline Ġto Ġtransformer Ġmodels Ġenriched Ġwith Ġimage Ġinformation , Ġas Ġwell Ġas Ġto Ġthe Ġdeliber ation Ġmodels , Ġwith Ġor Ġwithout Ġimage Ġinformation . </s> We Ġfirst Ġnote Ġthat Ġour Ġmultim od al Ġmodels Ġachieve Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġfor Ġtransformer Ġnetworks Ġ( con str ained Ġmodels ) Ġon Ġthe ĠEnglish - German Ġdataset , Ġas Ġcompared Ġto ĠB IB REF 30 Ġ. ĠSecond , Ġour Ġdeliber ation Ġmodels Ġlead Ġto Ġsignificant Ġimprovements Ġover Ġthis Ġbaseline Ġacross Ġtest Ġsets Ġ( average ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ). </s> Trans former - based Ġmodels Ġenriched Ġwith Ġimage Ġinformation Ġ( base + sum , Ġbase + att Ġand Ġbase + obj ), Ġon Ġthe Ġother Ġhand , Ġshow Ġno Ġmajor Ġimprovements Ġwith Ġrespect Ġto Ġthe Ġbase Ġperformance . ĠThis Ġis Ġalso Ġthe Ġcase Ġfor Ġdeliber ation Ġmodels Ġwith Ġimage Ġinformation Ġ( del + sum , Ġdel + att , Ġdel + obj ), Ġwhich Ġdo Ġnot Ġshow Ġsignificant Ġimprovement Ġover Ġthe Ġvanilla Ġdeliber ation Ġperformance Ġ( del ). </s> However , Ġas Ġit Ġhas Ġbeen Ġshown Ġin Ġthe ĠW MT Ġshared Ġtasks Ġon ĠM MT ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġautomatic Ġmetrics Ġoften Ġfail Ġto Ġcapture Ġnuances Ġin Ġtranslation Ġquality , Ġsuch Ġas , Ġthe Ġones Ġwe Ġexpect Ġthe Ġvisual Ġmod ality Ġto Ġhelp Ġwith , Ġwhich ĠâĢĵ Ġaccording Ġto Ġhuman Ġperception ĠâĢĵ Ġlead Ġto Ġbetter Ġtranslations . ĠTo Ġtest Ġthis Ġassumption Ġin Ġour Ġsettings , Ġwe Ġperformed Ġhuman Ġevaluation Ġinvolving Ġprofessional Ġtransl ators Ġand Ġnative Ġspeakers Ġof Ġboth ĠFrench Ġand ĠGerman Ġ( three Ġannot ators ). </s> The Ġannot ators Ġwere Ġasked Ġto Ġrank Ġrandomly Ġselected Ġtest Ġsamples Ġaccording Ġto Ġhow Ġwell Ġthey Ġconvey Ġthe Ġmeaning Ġof Ġthe Ġsource , Ġgiven Ġthe Ġimage Ġ( 50 Ġsamples Ġper Ġlanguage Ġpair Ġper Ġannot ator ). ĠFor Ġeach Ġsource Ġsegment , Ġthe Ġannot ator Ġwas Ġshown Ġthe Ġoutputs Ġof Ġthree Ġsystems : Ġbase + att , Ġthe Ġcurrent ĠM MT Ġstate - of - the - art ĠB IB REF 30 Ġ, Ġdel Ġand Ġdel + obj . ĠA Ġrank Ġcould Ġbe Ġassigned Ġfrom Ġ1 Ġto Ġ3 , Ġallowing Ġties ĠB IB REF 32 Ġ. ĠAn not ators Ġcould Ġassign Ġzero Ġrank Ġto Ġall Ġtranslations Ġif Ġthey Ġwere Ġjudged Ġincomprehensible . ĠFollowing Ġthe Ġcommon Ġpractice Ġin ĠW MT ĠB IB REF 32 Ġ, Ġeach Ġsystem Ġwas Ġthen Ġassigned Ġa Ġscore Ġwhich Ġreflects Ġthe Ġproportion Ġof Ġtimes Ġit Ġwas Ġjudged Ġto Ġbe Ġbetter Ġor Ġequal Ġother Ġsystems . </s> Table ĠT AB REF 19 Ġshows Ġthe Ġhuman Ġevaluation Ġresults . ĠThey Ġare Ġconsistent Ġwith Ġthe Ġautomatic Ġevaluation Ġresults Ġwhen Ġit Ġcomes Ġto Ġthe Ġpreference Ġof Ġhumans Ġtowards Ġthe Ġdeliber ation - based Ġsetups , Ġbut Ġshow Ġa Ġmore Ġpositive Ġoutlook Ġregarding Ġthe Ġaddition Ġof Ġvisual Ġinformation Ġ( del + obj Ġover Ġdel ) Ġfor ĠFrench . </s> Man ual Ġinspection Ġof Ġtranslations Ġsuggests Ġthat Ġdeliber ation Ġsetups Ġtend Ġto Ġimprove Ġboth Ġthe Ġgram matical ity Ġand Ġadequ acy Ġof Ġthe Ġfirst Ġpass Ġoutputs . ĠFor ĠGerman , Ġthe Ġmost Ġcommon Ġmodifications Ġperformed Ġby Ġthe Ġsecond - pass Ġdec oder Ġare Ġsubstit utions Ġof Ġadject ives Ġand Ġverbs Ġ( for Ġtest Ġ2016 , Ġ15 % Ġand Ġ12 % Ġrespectively , Ġof Ġall Ġthe Ġedit Ġdistance Ġoperations ). ĠChanges Ġto Ġadject ives Ġare Ġmainly Ġgram matical , Ġchanges Ġto Ġverbs Ġare Ġcontextual Ġ( e . g ., Ġchanging Ġl au fen Ġto Ġren nen , Ġboth Ġverbs Ġmean Ġrun , Ġbut Ġthe Ġsecond Ġrefers Ġto Ġrunning Ġvery Ġfast ). ĠFor ĠFrench , Ġ15 % Ġof Ġall Ġthe Ġchanges Ġare Ġsubstit utions Ġof Ġnoun s Ġ( for Ġtest Ġ2016 ). ĠThese Ġare Ġagain Ġvery Ġcontextual . ĠFor Ġexample , Ġthe ĠFrench Ġword Ġtra v ail le ur Ġ( worker ) Ġis Ġreplaced Ġby Ġo uv rier Ġ( man ual Ġworker ) Ġin Ġthe Ġcontexts Ġwhere Ġtools , Ġmachinery Ġor Ġbuildings Ġare Ġmentioned . ĠFor Ġour Ġanalysis Ġwe Ġused Ġagain Ġsp acy . </s> The Ġinformation Ġon Ġdetected Ġobjects Ġis Ġparticularly Ġhelpful Ġfor Ġspecific Ġadequ acy Ġissues . ĠFigure ĠFIG REF 15 Ġdemonstrates Ġsome Ġsuch Ġcases . ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġbase + att Ġmodel Ġmisses Ġthe Ġtranslation Ġof Ġrace Ġcar : Ġthe ĠGerman Ġword ĠR enn en Ġtranslates Ġonly Ġthe Ġword Ġrace . Ġdel Ġintroduces Ġthe Ġword Ġcar Ġ( Auto ) Ġinto Ġthe Ġtranslation . ĠFinally , Ġdel + obj Ġcorrectly Ġtranslates Ġthe Ġexpression Ġrace Ġcar Ġ( R enn wagen ) Ġby Ġexploiting Ġthe Ġobject Ġinformation . ĠFor ĠFrench , Ġdel Ġtranslates Ġthe Ġsource Ġpart Ġin Ġa Ġbody Ġof Ġwater , Ġmissing Ġfrom Ġthe Ġbase + att Ġtranslation . Ġdel + obj Ġadditionally Ġtranslated Ġthe Ġword Ġpadd ling Ġaccording Ġto Ġthe Ġdetected Ġobject ĠP addle . </s> Source Ġdegradation Ġsetup </s> Results Ġof Ġour Ġsource Ġdegradation Ġexperiments Ġare Ġshown Ġin ĠTable ĠT AB REF 20 Ġ. ĠA Ġfirst Ġobservation Ġis Ġthat ĠâĢĵ Ġas Ġwith Ġthe Ġstandard Ġsetup ĠâĢĵ Ġthe Ġperformance Ġof Ġour Ġdeliber ation Ġmodels Ġis Ġoverall Ġbetter Ġthan Ġthat Ġof Ġthe Ġbase Ġmodels . ĠThe Ġresults Ġof Ġthe Ġmultim od al Ġmodels Ġdiffer Ġfor ĠGerman Ġand ĠFrench . ĠFor ĠGerman , Ġdel + obj Ġis Ġthe Ġmost Ġsuccessful Ġconfiguration Ġand Ġshows Ġstatistically Ġsignificant Ġimprovements Ġover Ġbase Ġfor Ġall Ġsetups . ĠMoreover , Ġfor ĠR ND Ġand ĠAM B , Ġit Ġshows Ġstatistically Ġsignificant Ġimprovements Ġover Ġdel . ĠHowever , Ġespecially Ġfor ĠR ND Ġand ĠAM B , Ġdel Ġand Ġdel + sum Ġare Ġeither Ġthe Ġsame Ġor Ġslightly Ġworse Ġthan Ġbase . </s> For ĠFrench , Ġall Ġthe Ġdeliber ation Ġmodels Ġshow Ġstatistically Ġsignificant Ġimprovements Ġover Ġbase Ġ( average ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ), Ġbut Ġthe Ġimage Ġinformation Ġadded Ġto Ġdel Ġonly Ġimprove Ġscores Ġsignificantly Ġfor Ġtest Ġ2018 ĠR ND . </s> This Ġdifference Ġin Ġperformances Ġfor ĠFrench Ġand ĠGerman Ġis Ġpotentially Ġrelated Ġto Ġthe Ġneed Ġof Ġmore Ġsignificant Ġrestruct ur ings Ġwhile Ġtranslating Ġfrom ĠEnglish Ġinto ĠGerman . ĠThis Ġis Ġwhere Ġa Ġmore Ġcomplex Ġdel + obj Ġarchitecture Ġis Ġmore Ġhelpful . ĠThis Ġis Ġespecially Ġtrue Ġfor ĠR ND Ġand ĠAM B Ġsetups Ġwhere Ġblank ed Ġwords Ġcould Ġalso Ġbe Ġverbs , Ġthe Ġpart - of - speech Ġmost Ġinfluenced Ġby Ġword Ġorder Ġdifferences Ġbetween ĠEnglish Ġand ĠGerman Ġ( see Ġthe Ġdecreasing Ġcomplexity Ġof Ġtranslations Ġfor Ġdel Ġand Ġdel + obj Ġfor Ġthe Ġexample Ġ( c ) Ġin ĠFigure ĠFIG REF 21 Ġ). </s> To Ġget Ġan Ġinsight Ġinto Ġthe Ġcontribution Ġof Ġdifferent Ġcontexts Ġto Ġthe Ġresolution Ġof Ġbl anks , Ġwe Ġperformed Ġmanual Ġanalysis Ġof Ġexamples Ġcoming Ġfrom Ġthe ĠEnglish - German Ġbase , Ġdel Ġand Ġdel + obj Ġsetups Ġ( 50 Ġrandom Ġexamples Ġper Ġsetup ), Ġwhere Ġwe Ġcount Ġcorrectly Ġtranslated Ġbl anks Ġper Ġsystem . </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 27 Ġ. ĠAs Ġexpected , Ġthey Ġshow Ġthat Ġthe ĠR ND Ġand ĠAM B Ġbl anks Ġare Ġmore Ġdifficult Ġto Ġresolve Ġ( at Ġmost Ġ40 % Ġresolved Ġas Ġcompared Ġto Ġ61 % Ġfor ĠP ERS ). ĠTrans lations Ġof Ġthe Ġmajority Ġof Ġthose Ġbl anks Ġtend Ġto Ġbe Ġguessed Ġby Ġthe Ġtextual Ġcontext Ġalone Ġ( especially Ġfor Ġverbs ). ĠImage Ġinformation Ġis Ġmore Ġhelpful Ġfor ĠP ERS : Ġwe Ġobserve Ġan Ġincrease Ġof Ġ10 % Ġin Ġresolved Ġbl anks Ġfor Ġdel + obj Ġas Ġcompared Ġto Ġdel . ĠHowever , Ġfor ĠP ERS Ġthe Ġtextual Ġcontext Ġis Ġstill Ġenough Ġin Ġthe Ġmajority Ġof Ġthe Ġcases : Ġmodels Ġtend Ġto Ġassociate Ġmen Ġwith Ġsports Ġor Ġwomen Ġwith Ġcooking Ġand Ġare Ġusually Ġright Ġ( see ĠFigure ĠFIG REF 21 Ġexample Ġ( c )). </s> The Ġcases Ġwhere Ġimage Ġhelps Ġseem Ġto Ġbe Ġthose Ġwith Ġrather Ġgeneric Ġcontexts : Ġsee ĠFigure ĠFIG REF 21 Ġ( b ) Ġwhere Ġenjoying Ġa Ġsummer Ġday Ġis Ġnot Ġassociated Ġwith Ġany Ġparticular Ġgender Ġand Ġmake Ġother Ġmodels Ġchoose Ġhom me Ġ( man ) Ġor Ġfem me Ġ( woman ), Ġand Ġonly Ġbase + obj Ġchooses Ġen f ant Ġ( child ) Ġ( the Ġoption Ġclosest Ġto Ġthe Ġreference ). </s> In Ġsome Ġcases Ġdetected Ġobjects Ġare Ġinaccurate Ġor Ġnot Ġprecise Ġenough Ġto Ġbe Ġhelpful Ġ( e . g ., Ġwhen Ġan Ġobject ĠPerson Ġis Ġdetected ) Ġand Ġcan Ġeven Ġharm Ġcorrect Ġtranslations . </s> Con clusions </s> We Ġhave Ġproposed Ġa Ġnovel Ġapproach Ġto Ġmultim od al Ġmachine Ġtranslation Ġwhich Ġmakes Ġbetter Ġuse Ġof Ġcontext , Ġboth Ġtextual Ġand Ġvisual . ĠOur Ġresults Ġshow Ġthat Ġfurther Ġexploring Ġtextual Ġcontext Ġthrough Ġdeliber ation Ġnetworks Ġalready Ġleads Ġto Ġbetter Ġresults Ġthan Ġthe Ġprevious Ġstate Ġof Ġthe Ġart . ĠAdding Ġvisual Ġinformation , Ġand Ġin Ġparticular Ġstructural Ġrepresentations Ġof Ġthis Ġinformation , Ġproved Ġbeneficial Ġwhen Ġinput Ġtext Ġcontains Ġnoise Ġand Ġthe Ġlanguage Ġpair Ġrequires Ġsubstantial Ġrestructuring Ġfrom Ġsource Ġto Ġtarget . ĠOur Ġfindings Ġsuggest Ġthat Ġthe Ġcombination Ġof Ġa Ġdeliber ation Ġapproach Ġand Ġinformation Ġfrom Ġadditional Ġmod alities Ġis Ġa Ġpromising Ġdirection Ġfor Ġmachine Ġtranslation Ġthat Ġis Ġrobust Ġto Ġnoisy Ġinput . ĠOur Ġcode Ġand Ġpre - processing Ġscripts Ġare Ġavailable Ġat Ġhttps :// github . com / Im perial N LP / MM T - Del ib . </s> Acknowled gments </s> The Ġauthors Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġuseful Ġfeedback . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠMulti MT Ġ( H 2020 ĠE RC ĠStarting ĠGrant ĠNo . Ġ6 78 017 ) Ġand ĠMM VC Ġ( New ton ĠFund ĠInst itutional ĠLinks ĠGrant , ĠID Ġ35 234 35 75 ) Ġprojects . ĠWe Ġalso Ġthank Ġthe Ġannot ators Ġfor Ġtheir Ġvaluable Ġhelp .
Data We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 . Degradation of source In addition to using the Multi30K dataset as is (standard setup), we probe the ability of our models to address the three linguistic phenomena where additional context has been proved important (Section ): ambiguities, gender-neutral words and noisy input. In a controlled experiment where we aim to remove the influence of frequency biases, we degrade the source sentences by masking words through three strategies to replace words by a placeholder: random source words, ambiguous source words and gender unmarked source words. The procedure is applied to the train, validation and test sets. For the resulting dataset generated for each setting, we compare models having access to text-only context versus additional text and multimodal contexts. We seek to get insights into the contribution of each type of context to address each type of degradation. In this setting (RND) we simulate erroneous source words by randomly dropping source content words. We first tag the entire source sentences using the spacy toolkit BIBREF26 and then drop nouns, verbs, adjectives and adverbs and replace these with a default BLANK token. By focusing on content words, we differ from previous work that suggests that neural machine translation is robust to non-content word noise in the source BIBREF27 . In this setting (AMB), we rely on the MLT dataset BIBREF11 which provides a list of source words with multiple translations in the Multi30k training set. We replace ambiguous words with the BLANK token in the source language, which results in two language-specific datasets. In this setting (PERS), we use the Flickr Entities dataset BIBREF28 to identify all the words that were annotated by humans as corresponding to the category person. We then replace such source words with the BLANK token. The statistics of the resulting datasets for the three degradation strategies are shown in Table TABREF10 . We note that RND and PERS are the same for language pairs as the degradation only depends on the source side, while for AMB the words replaced depend on the target language. Models Based on the models described in Section we experiment with eight variants: (a) baseline transformer model (base); (b) base with AIC (base+sum); (c) base with AIF using spacial (base+att) or object based (base+obj) image features; (d) standard deliberation model (del); (e) deliberation models enriched with image information: del+sum, del+att and del+obj. Training In all cases, we optimise our models with cross entropy loss. For deliberation network models, we first train the standard transformer model until convergence, and use it to initialise the encoder and first-pass decoder. For each of the training samples, we follow BIBREF19 and obtain a set of 10-best samples from the first pass decoder, with a beam search of size 10. We use these as the first-pass decoder samples. We use Adam as optimiser BIBREF29 and train the model until convergence. Results In this section we present results of our experiments, first in the original dataset without any source degradation (Section SECREF18 ) and then in the setup with various source degradation strategies (Section SECREF25 ). Standard setup Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German. We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25 . We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information. We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30 . Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ). Transformer-based models enriched with image information (base+sum, base+att and base+obj), on the other hand, show no major improvements with respect to the base performance. This is also the case for deliberation models with image information (del+sum, del+att, del+obj), which do not show significant improvement over the vanilla deliberation performance (del). However, as it has been shown in the WMT shared tasks on MMT BIBREF23 , BIBREF24 , BIBREF25 , automatic metrics often fail to capture nuances in translation quality, such as, the ones we expect the visual modality to help with, which – according to human perception – lead to better translations. To test this assumption in our settings, we performed human evaluation involving professional translators and native speakers of both French and German (three annotators). The annotators were asked to rank randomly selected test samples according to how well they convey the meaning of the source, given the image (50 samples per language pair per annotator). For each source segment, the annotator was shown the outputs of three systems: base+att, the current MMT state-of-the-art BIBREF30 , del and del+obj. A rank could be assigned from 1 to 3, allowing ties BIBREF32 . Annotators could assign zero rank to all translations if they were judged incomprehensible. Following the common practice in WMT BIBREF32 , each system was then assigned a score which reflects the proportion of times it was judged to be better or equal other systems. Table TABREF19 shows the human evaluation results. They are consistent with the automatic evaluation results when it comes to the preference of humans towards the deliberation-based setups, but show a more positive outlook regarding the addition of visual information (del+obj over del) for French. Manual inspection of translations suggests that deliberation setups tend to improve both the grammaticality and adequacy of the first pass outputs. For German, the most common modifications performed by the second-pass decoder are substitutions of adjectives and verbs (for test 2016, 15% and 12% respectively, of all the edit distance operations). Changes to adjectives are mainly grammatical, changes to verbs are contextual (e.g., changing laufen to rennen, both verbs mean run, but the second refers to running very fast). For French, 15% of all the changes are substitutions of nouns (for test 2016). These are again very contextual. For example, the French word travailleur (worker) is replaced by ouvrier (manual worker) in the contexts where tools, machinery or buildings are mentioned. For our analysis we used again spacy. The information on detected objects is particularly helpful for specific adequacy issues. Figure FIGREF15 demonstrates some such cases. In the first case, the base+att model misses the translation of race car: the German word Rennen translates only the word race. del introduces the word car (Auto) into the translation. Finally, del+obj correctly translates the expression race car (Rennwagen) by exploiting the object information. For French, del translates the source part in a body of water, missing from the base+att translation. del+obj additionally translated the word paddling according to the detected object Paddle. Source degradation setup Results of our source degradation experiments are shown in Table TABREF20 . A first observation is that – as with the standard setup – the performance of our deliberation models is overall better than that of the base models. The results of the multimodal models differ for German and French. For German, del+obj is the most successful configuration and shows statistically significant improvements over base for all setups. Moreover, for RND and AMB, it shows statistically significant improvements over del. However, especially for RND and AMB, del and del+sum are either the same or slightly worse than base. For French, all the deliberation models show statistically significant improvements over base (average INLINEFORM0 , INLINEFORM1 ), but the image information added to del only improve scores significantly for test 2018 RND. This difference in performances for French and German is potentially related to the need of more significant restructurings while translating from English into German. This is where a more complex del+obj architecture is more helpful. This is especially true for RND and AMB setups where blanked words could also be verbs, the part-of-speech most influenced by word order differences between English and German (see the decreasing complexity of translations for del and del+obj for the example (c) in Figure FIGREF21 ). To get an insight into the contribution of different contexts to the resolution of blanks, we performed manual analysis of examples coming from the English-German base, del and del+obj setups (50 random examples per setup), where we count correctly translated blanks per system. The results are shown in Table TABREF27 . As expected, they show that the RND and AMB blanks are more difficult to resolve (at most 40% resolved as compared to 61% for PERS). Translations of the majority of those blanks tend to be guessed by the textual context alone (especially for verbs). Image information is more helpful for PERS: we observe an increase of 10% in resolved blanks for del+obj as compared to del. However, for PERS the textual context is still enough in the majority of the cases: models tend to associate men with sports or women with cooking and are usually right (see Figure FIGREF21 example (c)). The cases where image helps seem to be those with rather generic contexts: see Figure FIGREF21 (b) where enjoying a summer day is not associated with any particular gender and make other models choose homme (man) or femme (woman), and only base+obj chooses enfant (child) (the option closest to the reference). In some cases detected objects are inaccurate or not precise enough to be helpful (e.g., when an object Person is detected) and can even harm correct translations. Conclusions We have proposed a novel approach to multimodal machine translation which makes better use of context, both textual and visual. Our results show that further exploring textual context through deliberation networks already leads to better results than the previous state of the art. Adding visual information, and in particular structural representations of this information, proved beneficial when input text contains noise and the language pair requires substantial restructuring from source to target. Our findings suggest that the combination of a deliberation approach and information from additional modalities is a promising direction for machine translation that is robust to noisy input. Our code and pre-processing scripts are available at https://github.com/ImperialNLP/MMT-Delib. Acknowledgments The authors thank the anonymous reviewers for their useful feedback. This work was supported by the MultiMT (H2020 ERC Starting Grant No. 678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects. We also thank the annotators for their valuable help.
What dataset does this approach achieve state of the art results on?
1ed49a8c07ef0ac15cfa6b7decbde6604decbd5b
What Ġdataset Ġdoes Ġthis Ġapproach Ġachieve Ġstate Ġof Ġthe Ġart Ġresults Ġon ?
the ĠEnglish - German Ġdataset
<s> What Ġdataset Ġdoes Ġthis Ġapproach Ġachieve Ġstate Ġof Ġthe Ġart Ġresults Ġon ? </s> Data </s> We Ġbuild Ġand Ġtest Ġour ĠM MT Ġmodels Ġon Ġthe ĠMulti 30 K Ġdataset ĠB IB REF 21 Ġ. ĠEach Ġimage Ġin ĠMulti 30 K Ġcontains Ġone ĠEnglish Ġ( EN ) Ġdescription Ġtaken Ġfrom ĠFlickr 30 K ĠB IB REF 22 Ġand Ġhuman Ġtranslations Ġinto ĠGerman Ġ( DE ), ĠFrench Ġ( FR ) Ġand ĠCzech ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ. ĠThe Ġdataset Ġcontains Ġ29 , 000 Ġinstances Ġfor Ġtraining , Ġ1 , 014 Ġfor Ġdevelopment , Ġand Ġ1 , 000 Ġfor Ġtest . ĠWe Ġonly Ġexperiment Ġwith ĠGerman Ġand ĠFrench , Ġwhich Ġare Ġlanguages Ġfor Ġwhich Ġwe Ġhave Ġin - house Ġexpertise Ġfor Ġthe Ġtype Ġof Ġanalysis Ġwe Ġpresent . ĠIn Ġaddition Ġto Ġthe Ġofficial ĠMulti 30 K Ġtest Ġset Ġ( test Ġ2016 ), Ġwe Ġalso Ġuse Ġthe Ġtest Ġset Ġfrom Ġthe Ġlatest ĠW MT Ġevaluation Ġcompetition , Ġtest Ġ2018 ĠB IB REF 25 Ġ. </s> D eg rad ation Ġof Ġsource </s> In Ġaddition Ġto Ġusing Ġthe ĠMulti 30 K Ġdataset Ġas Ġis Ġ( standard Ġsetup ), Ġwe Ġprobe Ġthe Ġability Ġof Ġour Ġmodels Ġto Ġaddress Ġthe Ġthree Ġlinguistic Ġphenomena Ġwhere Ġadditional Ġcontext Ġhas Ġbeen Ġproved Ġimportant Ġ( Section Ġ): Ġambig u ities , Ġgender - neutral Ġwords Ġand Ġnoisy Ġinput . ĠIn Ġa Ġcontrolled Ġexperiment Ġwhere Ġwe Ġaim Ġto Ġremove Ġthe Ġinfluence Ġof Ġfrequency Ġbiases , Ġwe Ġdegrade Ġthe Ġsource Ġsentences Ġby Ġmask ing Ġwords Ġthrough Ġthree Ġstrategies Ġto Ġreplace Ġwords Ġby Ġa Ġplaceholder : Ġrandom Ġsource Ġwords , Ġambiguous Ġsource Ġwords Ġand Ġgender Ġunmarked Ġsource Ġwords . ĠThe Ġprocedure Ġis Ġapplied Ġto Ġthe Ġtrain , Ġvalidation Ġand Ġtest Ġsets . ĠFor Ġthe Ġresulting Ġdataset Ġgenerated Ġfor Ġeach Ġsetting , Ġwe Ġcompare Ġmodels Ġhaving Ġaccess Ġto Ġtext - only Ġcontext Ġversus Ġadditional Ġtext Ġand Ġmultim od al Ġcontexts . ĠWe Ġseek Ġto Ġget Ġinsights Ġinto Ġthe Ġcontribution Ġof Ġeach Ġtype Ġof Ġcontext Ġto Ġaddress Ġeach Ġtype Ġof Ġdegradation . </s> In Ġthis Ġsetting Ġ( R ND ) Ġwe Ġsimulate Ġerroneous Ġsource Ġwords Ġby Ġrandomly Ġdropping Ġsource Ġcontent Ġwords . ĠWe Ġfirst Ġtag Ġthe Ġentire Ġsource Ġsentences Ġusing Ġthe Ġsp acy Ġtool kit ĠB IB REF 26 Ġand Ġthen Ġdrop Ġnoun s , Ġverbs , Ġadject ives Ġand Ġad verbs Ġand Ġreplace Ġthese Ġwith Ġa Ġdefault ĠBL ANK Ġtoken . ĠBy Ġfocusing Ġon Ġcontent Ġwords , Ġwe Ġdiffer Ġfrom Ġprevious Ġwork Ġthat Ġsuggests Ġthat Ġneural Ġmachine Ġtranslation Ġis Ġrobust Ġto Ġnon - content Ġword Ġnoise Ġin Ġthe Ġsource ĠB IB REF 27 Ġ. </s> In Ġthis Ġsetting Ġ( AM B ), Ġwe Ġrely Ġon Ġthe ĠML T Ġdataset ĠB IB REF 11 Ġwhich Ġprovides Ġa Ġlist Ġof Ġsource Ġwords Ġwith Ġmultiple Ġtranslations Ġin Ġthe ĠMulti 30 k Ġtraining Ġset . ĠWe Ġreplace Ġambiguous Ġwords Ġwith Ġthe ĠBL ANK Ġtoken Ġin Ġthe Ġsource Ġlanguage , Ġwhich Ġresults Ġin Ġtwo Ġlanguage - specific Ġdatasets . </s> In Ġthis Ġsetting Ġ( P ERS ), Ġwe Ġuse Ġthe ĠFlickr ĠEnt ities Ġdataset ĠB IB REF 28 Ġto Ġidentify Ġall Ġthe Ġwords Ġthat Ġwere Ġannot ated Ġby Ġhumans Ġas Ġcorresponding Ġto Ġthe Ġcategory Ġperson . ĠWe Ġthen Ġreplace Ġsuch Ġsource Ġwords Ġwith Ġthe ĠBL ANK Ġtoken . </s> The Ġstatistics Ġof Ġthe Ġresulting Ġdatasets Ġfor Ġthe Ġthree Ġdegradation Ġstrategies Ġare Ġshown Ġin ĠTable ĠT AB REF 10 Ġ. ĠWe Ġnote Ġthat ĠR ND Ġand ĠP ERS Ġare Ġthe Ġsame Ġfor Ġlanguage Ġpairs Ġas Ġthe Ġdegradation Ġonly Ġdepends Ġon Ġthe Ġsource Ġside , Ġwhile Ġfor ĠAM B Ġthe Ġwords Ġreplaced Ġdepend Ġon Ġthe Ġtarget Ġlanguage . </s> Mod els </s> Based Ġon Ġthe Ġmodels Ġdescribed Ġin ĠSection Ġwe Ġexperiment Ġwith Ġeight Ġvariants : Ġ( a ) Ġbaseline Ġtransformer Ġmodel Ġ( base ); Ġ( b ) Ġbase Ġwith ĠA IC Ġ( base + sum ); Ġ( c ) Ġbase Ġwith ĠA IF Ġusing Ġsp acial Ġ( base + att ) Ġor Ġobject Ġbased Ġ( base + obj ) Ġimage Ġfeatures ; Ġ( d ) Ġstandard Ġdeliber ation Ġmodel Ġ( del ); Ġ( e ) Ġdeliber ation Ġmodels Ġenriched Ġwith Ġimage Ġinformation : Ġdel + sum , Ġdel + att Ġand Ġdel + obj . </s> Training </s> In Ġall Ġcases , Ġwe Ġoptim ise Ġour Ġmodels Ġwith Ġcross Ġentropy Ġloss . ĠFor Ġdeliber ation Ġnetwork Ġmodels , Ġwe Ġfirst Ġtrain Ġthe Ġstandard Ġtransformer Ġmodel Ġuntil Ġconvergence , Ġand Ġuse Ġit Ġto Ġinitial ise Ġthe Ġenc oder Ġand Ġfirst - pass Ġdec oder . ĠFor Ġeach Ġof Ġthe Ġtraining Ġsamples , Ġwe Ġfollow ĠB IB REF 19 Ġand Ġobtain Ġa Ġset Ġof Ġ10 - best Ġsamples Ġfrom Ġthe Ġfirst Ġpass Ġdec oder , Ġwith Ġa Ġbeam Ġsearch Ġof Ġsize Ġ10 . ĠWe Ġuse Ġthese Ġas Ġthe Ġfirst - pass Ġdec oder Ġsamples . ĠWe Ġuse ĠAdam Ġas Ġoptim iser ĠB IB REF 29 Ġand Ġtrain Ġthe Ġmodel Ġuntil Ġconvergence . </s> Results </s> In Ġthis Ġsection Ġwe Ġpresent Ġresults Ġof Ġour Ġexperiments , Ġfirst Ġin Ġthe Ġoriginal Ġdataset Ġwithout Ġany Ġsource Ġdegradation Ġ( Section ĠSEC REF 18 Ġ) Ġand Ġthen Ġin Ġthe Ġsetup Ġwith Ġvarious Ġsource Ġdegradation Ġstrategies Ġ( Section ĠSEC REF 25 Ġ). </s> Standard Ġsetup </s> Table ĠT AB REF 14 Ġshows Ġthe Ġresults Ġof Ġour Ġmain Ġexperiments Ġon Ġthe Ġ2016 Ġand Ġ2018 Ġtest Ġsets Ġfor ĠFrench Ġand ĠGerman . ĠWe Ġuse ĠMeteor ĠB IB REF 31 Ġas Ġthe Ġmain Ġmetric , Ġas Ġin Ġthe ĠW MT Ġtasks ĠB IB REF 25 Ġ. ĠWe Ġcompare Ġour Ġtransformer Ġbaseline Ġto Ġtransformer Ġmodels Ġenriched Ġwith Ġimage Ġinformation , Ġas Ġwell Ġas Ġto Ġthe Ġdeliber ation Ġmodels , Ġwith Ġor Ġwithout Ġimage Ġinformation . </s> We Ġfirst Ġnote Ġthat Ġour Ġmultim od al Ġmodels Ġachieve Ġthe Ġstate Ġof Ġthe Ġart Ġperformance Ġfor Ġtransformer Ġnetworks Ġ( con str ained Ġmodels ) Ġon Ġthe ĠEnglish - German Ġdataset , Ġas Ġcompared Ġto ĠB IB REF 30 Ġ. ĠSecond , Ġour Ġdeliber ation Ġmodels Ġlead Ġto Ġsignificant Ġimprovements Ġover Ġthis Ġbaseline Ġacross Ġtest Ġsets Ġ( average ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ). </s> Trans former - based Ġmodels Ġenriched Ġwith Ġimage Ġinformation Ġ( base + sum , Ġbase + att Ġand Ġbase + obj ), Ġon Ġthe Ġother Ġhand , Ġshow Ġno Ġmajor Ġimprovements Ġwith Ġrespect Ġto Ġthe Ġbase Ġperformance . ĠThis Ġis Ġalso Ġthe Ġcase Ġfor Ġdeliber ation Ġmodels Ġwith Ġimage Ġinformation Ġ( del + sum , Ġdel + att , Ġdel + obj ), Ġwhich Ġdo Ġnot Ġshow Ġsignificant Ġimprovement Ġover Ġthe Ġvanilla Ġdeliber ation Ġperformance Ġ( del ). </s> However , Ġas Ġit Ġhas Ġbeen Ġshown Ġin Ġthe ĠW MT Ġshared Ġtasks Ġon ĠM MT ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, Ġautomatic Ġmetrics Ġoften Ġfail Ġto Ġcapture Ġnuances Ġin Ġtranslation Ġquality , Ġsuch Ġas , Ġthe Ġones Ġwe Ġexpect Ġthe Ġvisual Ġmod ality Ġto Ġhelp Ġwith , Ġwhich ĠâĢĵ Ġaccording Ġto Ġhuman Ġperception ĠâĢĵ Ġlead Ġto Ġbetter Ġtranslations . ĠTo Ġtest Ġthis Ġassumption Ġin Ġour Ġsettings , Ġwe Ġperformed Ġhuman Ġevaluation Ġinvolving Ġprofessional Ġtransl ators Ġand Ġnative Ġspeakers Ġof Ġboth ĠFrench Ġand ĠGerman Ġ( three Ġannot ators ). </s> The Ġannot ators Ġwere Ġasked Ġto Ġrank Ġrandomly Ġselected Ġtest Ġsamples Ġaccording Ġto Ġhow Ġwell Ġthey Ġconvey Ġthe Ġmeaning Ġof Ġthe Ġsource , Ġgiven Ġthe Ġimage Ġ( 50 Ġsamples Ġper Ġlanguage Ġpair Ġper Ġannot ator ). ĠFor Ġeach Ġsource Ġsegment , Ġthe Ġannot ator Ġwas Ġshown Ġthe Ġoutputs Ġof Ġthree Ġsystems : Ġbase + att , Ġthe Ġcurrent ĠM MT Ġstate - of - the - art ĠB IB REF 30 Ġ, Ġdel Ġand Ġdel + obj . ĠA Ġrank Ġcould Ġbe Ġassigned Ġfrom Ġ1 Ġto Ġ3 , Ġallowing Ġties ĠB IB REF 32 Ġ. ĠAn not ators Ġcould Ġassign Ġzero Ġrank Ġto Ġall Ġtranslations Ġif Ġthey Ġwere Ġjudged Ġincomprehensible . ĠFollowing Ġthe Ġcommon Ġpractice Ġin ĠW MT ĠB IB REF 32 Ġ, Ġeach Ġsystem Ġwas Ġthen Ġassigned Ġa Ġscore Ġwhich Ġreflects Ġthe Ġproportion Ġof Ġtimes Ġit Ġwas Ġjudged Ġto Ġbe Ġbetter Ġor Ġequal Ġother Ġsystems . </s> Table ĠT AB REF 19 Ġshows Ġthe Ġhuman Ġevaluation Ġresults . ĠThey Ġare Ġconsistent Ġwith Ġthe Ġautomatic Ġevaluation Ġresults Ġwhen Ġit Ġcomes Ġto Ġthe Ġpreference Ġof Ġhumans Ġtowards Ġthe Ġdeliber ation - based Ġsetups , Ġbut Ġshow Ġa Ġmore Ġpositive Ġoutlook Ġregarding Ġthe Ġaddition Ġof Ġvisual Ġinformation Ġ( del + obj Ġover Ġdel ) Ġfor ĠFrench . </s> Man ual Ġinspection Ġof Ġtranslations Ġsuggests Ġthat Ġdeliber ation Ġsetups Ġtend Ġto Ġimprove Ġboth Ġthe Ġgram matical ity Ġand Ġadequ acy Ġof Ġthe Ġfirst Ġpass Ġoutputs . ĠFor ĠGerman , Ġthe Ġmost Ġcommon Ġmodifications Ġperformed Ġby Ġthe Ġsecond - pass Ġdec oder Ġare Ġsubstit utions Ġof Ġadject ives Ġand Ġverbs Ġ( for Ġtest Ġ2016 , Ġ15 % Ġand Ġ12 % Ġrespectively , Ġof Ġall Ġthe Ġedit Ġdistance Ġoperations ). ĠChanges Ġto Ġadject ives Ġare Ġmainly Ġgram matical , Ġchanges Ġto Ġverbs Ġare Ġcontextual Ġ( e . g ., Ġchanging Ġl au fen Ġto Ġren nen , Ġboth Ġverbs Ġmean Ġrun , Ġbut Ġthe Ġsecond Ġrefers Ġto Ġrunning Ġvery Ġfast ). ĠFor ĠFrench , Ġ15 % Ġof Ġall Ġthe Ġchanges Ġare Ġsubstit utions Ġof Ġnoun s Ġ( for Ġtest Ġ2016 ). ĠThese Ġare Ġagain Ġvery Ġcontextual . ĠFor Ġexample , Ġthe ĠFrench Ġword Ġtra v ail le ur Ġ( worker ) Ġis Ġreplaced Ġby Ġo uv rier Ġ( man ual Ġworker ) Ġin Ġthe Ġcontexts Ġwhere Ġtools , Ġmachinery Ġor Ġbuildings Ġare Ġmentioned . ĠFor Ġour Ġanalysis Ġwe Ġused Ġagain Ġsp acy . </s> The Ġinformation Ġon Ġdetected Ġobjects Ġis Ġparticularly Ġhelpful Ġfor Ġspecific Ġadequ acy Ġissues . ĠFigure ĠFIG REF 15 Ġdemonstrates Ġsome Ġsuch Ġcases . ĠIn Ġthe Ġfirst Ġcase , Ġthe Ġbase + att Ġmodel Ġmisses Ġthe Ġtranslation Ġof Ġrace Ġcar : Ġthe ĠGerman Ġword ĠR enn en Ġtranslates Ġonly Ġthe Ġword Ġrace . Ġdel Ġintroduces Ġthe Ġword Ġcar Ġ( Auto ) Ġinto Ġthe Ġtranslation . ĠFinally , Ġdel + obj Ġcorrectly Ġtranslates Ġthe Ġexpression Ġrace Ġcar Ġ( R enn wagen ) Ġby Ġexploiting Ġthe Ġobject Ġinformation . ĠFor ĠFrench , Ġdel Ġtranslates Ġthe Ġsource Ġpart Ġin Ġa Ġbody Ġof Ġwater , Ġmissing Ġfrom Ġthe Ġbase + att Ġtranslation . Ġdel + obj Ġadditionally Ġtranslated Ġthe Ġword Ġpadd ling Ġaccording Ġto Ġthe Ġdetected Ġobject ĠP addle . </s> Source Ġdegradation Ġsetup </s> Results Ġof Ġour Ġsource Ġdegradation Ġexperiments Ġare Ġshown Ġin ĠTable ĠT AB REF 20 Ġ. ĠA Ġfirst Ġobservation Ġis Ġthat ĠâĢĵ Ġas Ġwith Ġthe Ġstandard Ġsetup ĠâĢĵ Ġthe Ġperformance Ġof Ġour Ġdeliber ation Ġmodels Ġis Ġoverall Ġbetter Ġthan Ġthat Ġof Ġthe Ġbase Ġmodels . ĠThe Ġresults Ġof Ġthe Ġmultim od al Ġmodels Ġdiffer Ġfor ĠGerman Ġand ĠFrench . ĠFor ĠGerman , Ġdel + obj Ġis Ġthe Ġmost Ġsuccessful Ġconfiguration Ġand Ġshows Ġstatistically Ġsignificant Ġimprovements Ġover Ġbase Ġfor Ġall Ġsetups . ĠMoreover , Ġfor ĠR ND Ġand ĠAM B , Ġit Ġshows Ġstatistically Ġsignificant Ġimprovements Ġover Ġdel . ĠHowever , Ġespecially Ġfor ĠR ND Ġand ĠAM B , Ġdel Ġand Ġdel + sum Ġare Ġeither Ġthe Ġsame Ġor Ġslightly Ġworse Ġthan Ġbase . </s> For ĠFrench , Ġall Ġthe Ġdeliber ation Ġmodels Ġshow Ġstatistically Ġsignificant Ġimprovements Ġover Ġbase Ġ( average ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ), Ġbut Ġthe Ġimage Ġinformation Ġadded Ġto Ġdel Ġonly Ġimprove Ġscores Ġsignificantly Ġfor Ġtest Ġ2018 ĠR ND . </s> This Ġdifference Ġin Ġperformances Ġfor ĠFrench Ġand ĠGerman Ġis Ġpotentially Ġrelated Ġto Ġthe Ġneed Ġof Ġmore Ġsignificant Ġrestruct ur ings Ġwhile Ġtranslating Ġfrom ĠEnglish Ġinto ĠGerman . ĠThis Ġis Ġwhere Ġa Ġmore Ġcomplex Ġdel + obj Ġarchitecture Ġis Ġmore Ġhelpful . ĠThis Ġis Ġespecially Ġtrue Ġfor ĠR ND Ġand ĠAM B Ġsetups Ġwhere Ġblank ed Ġwords Ġcould Ġalso Ġbe Ġverbs , Ġthe Ġpart - of - speech Ġmost Ġinfluenced Ġby Ġword Ġorder Ġdifferences Ġbetween ĠEnglish Ġand ĠGerman Ġ( see Ġthe Ġdecreasing Ġcomplexity Ġof Ġtranslations Ġfor Ġdel Ġand Ġdel + obj Ġfor Ġthe Ġexample Ġ( c ) Ġin ĠFigure ĠFIG REF 21 Ġ). </s> To Ġget Ġan Ġinsight Ġinto Ġthe Ġcontribution Ġof Ġdifferent Ġcontexts Ġto Ġthe Ġresolution Ġof Ġbl anks , Ġwe Ġperformed Ġmanual Ġanalysis Ġof Ġexamples Ġcoming Ġfrom Ġthe ĠEnglish - German Ġbase , Ġdel Ġand Ġdel + obj Ġsetups Ġ( 50 Ġrandom Ġexamples Ġper Ġsetup ), Ġwhere Ġwe Ġcount Ġcorrectly Ġtranslated Ġbl anks Ġper Ġsystem . </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 27 Ġ. ĠAs Ġexpected , Ġthey Ġshow Ġthat Ġthe ĠR ND Ġand ĠAM B Ġbl anks Ġare Ġmore Ġdifficult Ġto Ġresolve Ġ( at Ġmost Ġ40 % Ġresolved Ġas Ġcompared Ġto Ġ61 % Ġfor ĠP ERS ). ĠTrans lations Ġof Ġthe Ġmajority Ġof Ġthose Ġbl anks Ġtend Ġto Ġbe Ġguessed Ġby Ġthe Ġtextual Ġcontext Ġalone Ġ( especially Ġfor Ġverbs ). ĠImage Ġinformation Ġis Ġmore Ġhelpful Ġfor ĠP ERS : Ġwe Ġobserve Ġan Ġincrease Ġof Ġ10 % Ġin Ġresolved Ġbl anks Ġfor Ġdel + obj Ġas Ġcompared Ġto Ġdel . ĠHowever , Ġfor ĠP ERS Ġthe Ġtextual Ġcontext Ġis Ġstill Ġenough Ġin Ġthe Ġmajority Ġof Ġthe Ġcases : Ġmodels Ġtend Ġto Ġassociate Ġmen Ġwith Ġsports Ġor Ġwomen Ġwith Ġcooking Ġand Ġare Ġusually Ġright Ġ( see ĠFigure ĠFIG REF 21 Ġexample Ġ( c )). </s> The Ġcases Ġwhere Ġimage Ġhelps Ġseem Ġto Ġbe Ġthose Ġwith Ġrather Ġgeneric Ġcontexts : Ġsee ĠFigure ĠFIG REF 21 Ġ( b ) Ġwhere Ġenjoying Ġa Ġsummer Ġday Ġis Ġnot Ġassociated Ġwith Ġany Ġparticular Ġgender Ġand Ġmake Ġother Ġmodels Ġchoose Ġhom me Ġ( man ) Ġor Ġfem me Ġ( woman ), Ġand Ġonly Ġbase + obj Ġchooses Ġen f ant Ġ( child ) Ġ( the Ġoption Ġclosest Ġto Ġthe Ġreference ). </s> In Ġsome Ġcases Ġdetected Ġobjects Ġare Ġinaccurate Ġor Ġnot Ġprecise Ġenough Ġto Ġbe Ġhelpful Ġ( e . g ., Ġwhen Ġan Ġobject ĠPerson Ġis Ġdetected ) Ġand Ġcan Ġeven Ġharm Ġcorrect Ġtranslations . </s> Con clusions </s> We Ġhave Ġproposed Ġa Ġnovel Ġapproach Ġto Ġmultim od al Ġmachine Ġtranslation Ġwhich Ġmakes Ġbetter Ġuse Ġof Ġcontext , Ġboth Ġtextual Ġand Ġvisual . ĠOur Ġresults Ġshow Ġthat Ġfurther Ġexploring Ġtextual Ġcontext Ġthrough Ġdeliber ation Ġnetworks Ġalready Ġleads Ġto Ġbetter Ġresults Ġthan Ġthe Ġprevious Ġstate Ġof Ġthe Ġart . ĠAdding Ġvisual Ġinformation , Ġand Ġin Ġparticular Ġstructural Ġrepresentations Ġof Ġthis Ġinformation , Ġproved Ġbeneficial Ġwhen Ġinput Ġtext Ġcontains Ġnoise Ġand Ġthe Ġlanguage Ġpair Ġrequires Ġsubstantial Ġrestructuring Ġfrom Ġsource Ġto Ġtarget . ĠOur Ġfindings Ġsuggest Ġthat Ġthe Ġcombination Ġof Ġa Ġdeliber ation Ġapproach Ġand Ġinformation Ġfrom Ġadditional Ġmod alities Ġis Ġa Ġpromising Ġdirection Ġfor Ġmachine Ġtranslation Ġthat Ġis Ġrobust Ġto Ġnoisy Ġinput . ĠOur Ġcode Ġand Ġpre - processing Ġscripts Ġare Ġavailable Ġat Ġhttps :// github . com / Im perial N LP / MM T - Del ib . </s> Acknowled gments </s> The Ġauthors Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġuseful Ġfeedback . ĠThis Ġwork Ġwas Ġsupported Ġby Ġthe ĠMulti MT Ġ( H 2020 ĠE RC ĠStarting ĠGrant ĠNo . Ġ6 78 017 ) Ġand ĠMM VC Ġ( New ton ĠFund ĠInst itutional ĠLinks ĠGrant , ĠID Ġ35 234 35 75 ) Ġprojects . ĠWe Ġalso Ġthank Ġthe Ġannot ators Ġfor Ġtheir Ġvaluable Ġhelp .