Unanswerable
NONE
Unanswerable
NONE
1909.01383
Introduction </s> Machine Ġtranslation Ġhas Ġmade Ġremarkable Ġprogress , Ġand Ġstudies Ġclaiming Ġit Ġto Ġreach Ġa Ġhuman Ġparity Ġare Ġstarting Ġto Ġappear ĠB IB REF 0 . ĠHowever , Ġwhen Ġevaluating Ġtranslations Ġof Ġthe Ġwhole Ġdocuments Ġrather Ġthan Ġisolated Ġsentences , Ġhuman Ġrat ers Ġshow Ġa Ġstronger Ġpreference Ġfor Ġhuman Ġover Ġmachine Ġtranslation ĠB IB REF 1 . ĠThese Ġfindings Ġemphasize Ġthe Ġneed Ġto Ġshift Ġtowards Ġcontext - aware Ġmachine Ġtranslation Ġboth Ġfrom Ġmodeling Ġand Ġevaluation Ġperspective . </s> Most Ġprevious Ġwork Ġon Ġcontext - aware ĠN MT Ġassumed Ġthat Ġeither Ġall Ġthe Ġbilingual Ġdata Ġis Ġavailable Ġat Ġthe Ġdocument Ġlevel ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 Ġor Ġat Ġleast Ġits Ġfraction ĠB IB REF 11 . ĠBut Ġin Ġpractical Ġscenarios , Ġdocument - level Ġparallel Ġdata Ġis Ġoften Ġscarce , Ġwhich Ġis Ġone Ġof Ġthe Ġchallenges Ġwhen Ġbuilding Ġa Ġcontext - aware Ġsystem . </s> We Ġintroduce Ġan Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠIn Ġour Ġsetting , Ġa Ġseparate Ġmon oling ual Ġsequence - to - sequence Ġmodel Ġ( Doc Rep air ) Ġis Ġused Ġto Ġcorrect Ġsentence - level Ġtranslations Ġof Ġadjacent Ġsentences . ĠThe Ġkey Ġidea Ġis Ġto Ġuse Ġmon oling ual Ġdata Ġto Ġimitate Ġtypical Ġinconsistencies Ġbetween Ġcontext - agn ostic Ġtranslations Ġof Ġisolated Ġsentences . ĠThe ĠDoc Rep air Ġmodel Ġis Ġtrained Ġto Ġmap Ġinconsistent Ġgroups Ġof Ġsentences Ġinto Ġconsistent Ġones . ĠThe Ġconsistent Ġgroups Ġcome Ġfrom Ġthe Ġoriginal Ġtraining Ġdata ; Ġthe Ġinconsistent Ġgroups Ġare Ġobtained Ġby Ġsampling Ġround - trip Ġtranslations Ġfor Ġeach Ġisolated Ġsentence . </s> To Ġvalidate Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġuse Ġthree Ġkinds Ġof Ġevaluation : Ġthe ĠB LE U Ġscore , Ġcontrast ive Ġevaluation Ġof Ġtranslation Ġof Ġseveral Ġdiscourse Ġphenomena ĠB IB REF 11 , Ġand Ġhuman Ġevaluation . ĠWe Ġshow Ġstrong Ġimprovements Ġfor Ġall Ġmetrics . </s> We Ġanalyze Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġmon oling ual Ġdata Ġonly . ĠUsing Ġcontrast ive Ġtest Ġsets Ġfor Ġtargeted Ġevaluation Ġof Ġseveral Ġcontextual Ġphenomena , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġthe Ġmodels Ġtrained Ġon Ġround - trip Ġtranslations Ġand Ġgenuine Ġdocument - level Ġparallel Ġdata . ĠAmong Ġthe Ġfour Ġphenomena Ġin Ġthe Ġtest Ġsets Ġwe Ġuse Ġ( de ix is , Ġlex ical Ġcohesion , ĠVP Ġell ips is Ġand Ġell ips is Ġwhich Ġaffects ĠNP Ġinf lection ) Ġwe Ġfind ĠVP Ġell ips is Ġto Ġbe Ġthe Ġhardest Ġphenomenon Ġto Ġbe Ġcaptured Ġusing Ġround - trip Ġtranslations . </s> Our Ġkey Ġcontributions Ġare Ġas Ġfollows : </s> we Ġintroduce Ġthe Ġfirst Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata ; </s> our Ġapproach Ġshows Ġsubstantial Ġimprovements Ġin Ġtranslation Ġquality Ġas Ġmeasured Ġby ĠB LE U , Ġtargeted Ġcontrast ive Ġevaluation Ġof Ġseveral Ġdiscourse Ġphenomena Ġand Ġhuman Ġevaluation ; </s> we Ġshow Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġmon oling ual Ġdata Ġonly . </s> Our ĠApproach : ĠDocument - level ĠRepair </s> We Ġpropose Ġa Ġmon oling ual ĠDoc Rep air Ġmodel Ġto Ġcorrect Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations Ġof Ġa Ġcontext - agn ostic ĠMT Ġsystem . ĠIt Ġdoes Ġnot Ġuse Ġany Ġstates Ġof Ġa Ġtrained ĠMT Ġmodel Ġwhose Ġoutputs Ġit Ġcorrect s Ġand Ġtherefore Ġcan Ġin Ġprinciple Ġbe Ġtrained Ġto Ġcorrect Ġtranslations Ġfrom Ġany Ġblack - box ĠMT Ġsystem . </s> The ĠDoc Rep air Ġmodel Ġrequires Ġonly Ġmon oling ual Ġdocument - level Ġdata Ġin Ġthe Ġtarget Ġlanguage . ĠIt Ġis Ġa Ġmon oling ual Ġsequence - to - sequence Ġmodel Ġthat Ġmaps Ġinconsistent Ġgroups Ġof Ġsentences Ġinto Ġconsistent Ġones . ĠCons istent Ġgroups Ġcome Ġfrom Ġmon oling ual Ġdocument - level Ġdata . ĠTo Ġobtain Ġinconsistent Ġgroups , Ġeach Ġsentence Ġin Ġa Ġgroup Ġis Ġreplaced Ġwith Ġits Ġround - trip Ġtranslation Ġproduced Ġin Ġisolation Ġfrom Ġcontext . ĠMore Ġformally , Ġforming Ġa Ġtraining Ġmin ib atch Ġfor Ġthe ĠDoc Rep air Ġmodel Ġinvolves Ġthe Ġfollowing Ġsteps Ġ( see Ġalso ĠFigure ĠFIG REF 9 ): </s> sample Ġseveral Ġgroups Ġof Ġsentences Ġfrom Ġthe Ġmon oling ual Ġdata ; </s> for Ġeach Ġsentence Ġin Ġa Ġgroup , Ġ( i ) Ġtranslate Ġit Ġusing Ġa Ġtarget - to - source ĠMT Ġmodel , Ġ( ii ) Ġsample Ġa Ġtranslation Ġof Ġthis Ġback - trans lated Ġsentence Ġin Ġthe Ġsource Ġlanguage Ġusing Ġa Ġsource - to - target ĠMT Ġmodel ; </s> using Ġthese Ġround - trip Ġtranslations Ġof Ġisolated Ġsentences , Ġform Ġan Ġinconsistent Ġversion Ġof Ġthe Ġinitial Ġgroups ; </s> use Ġinconsistent Ġgroups Ġas Ġinput Ġfor Ġthe ĠDoc Rep air Ġmodel , Ġconsistent Ġones Ġas Ġoutput . </s> At Ġtest Ġtime , Ġthe Ġprocess Ġof Ġgetting Ġdocument - level Ġtranslations Ġis Ġtwo - step Ġ( Figure ĠFIG REF 10 ): </s> produ ce Ġtranslations Ġof Ġisolated Ġsentences Ġusing Ġa Ġcontext - agn ostic ĠMT Ġmodel ; </s> apply Ġthe ĠDoc Rep air Ġmodel Ġto Ġa Ġsequence Ġof Ġcontext - agn ostic Ġtranslations Ġto Ġcorrect Ġinconsistencies Ġbetween Ġtranslations . </s> In Ġthe Ġscope Ġof Ġthe Ġcurrent Ġwork , Ġthe ĠDoc Rep air Ġmodel Ġis Ġthe Ġstandard Ġsequence - to - sequence ĠTrans former . ĠSent ences Ġin Ġa Ġgroup Ġare Ġconc aten ated Ġusing Ġa Ġreserved Ġtoken - separ ator Ġbetween Ġsentences . ĠThe ĠTrans former Ġis Ġtrained Ġto Ġcorrect Ġthese Ġlong Ġinconsistent Ġpseudo - sent ences Ġinto Ġconsistent Ġones . ĠThe Ġtoken - separ ator Ġis Ġthen Ġremoved Ġfrom Ġcorrected Ġtranslations . </s> E val uation Ġof ĠContext ual ĠPhen omen a </s> We Ġuse Ġcontrast ive Ġtest Ġsets Ġfor Ġevaluation Ġof Ġdiscourse Ġphenomena Ġfor ĠEnglish - Russian Ġby ĠB IB REF 11 . ĠThese Ġtest Ġsets Ġallow Ġfor Ġtesting Ġdifferent Ġkinds Ġof Ġphenomena Ġwhich , Ġas Ġwe Ġshow , Ġcan Ġbe Ġcaptured Ġfrom Ġmon oling ual Ġdata Ġwith Ġvarying Ġsuccess . ĠIn Ġthis Ġsection , Ġwe Ġprovide Ġtest Ġsets Ġstatistics Ġand Ġbriefly Ġdescribe Ġthe Ġtested Ġphenomena . ĠFor Ġmore Ġdetails , Ġthe Ġreader Ġis Ġreferred Ġto ĠB IB REF 11 . </s> E val uation Ġof ĠContext ual ĠPhen omen a Ġ: :: ĠTest Ġsets </s> There Ġare Ġfour Ġtest Ġsets Ġin Ġthe Ġsuite . ĠEach Ġtest Ġset Ġcontains Ġcontrast ive Ġexamples . ĠIt Ġis Ġspecifically Ġdesigned Ġto Ġtest Ġthe Ġability Ġof Ġa Ġsystem Ġto Ġadapt Ġto Ġcontextual Ġinformation Ġand Ġhandle Ġthe Ġphenomenon Ġunder Ġconsideration . ĠEach Ġtest Ġinstance Ġconsists Ġof Ġa Ġtrue Ġexample Ġ( a Ġsequence Ġof Ġsentences Ġand Ġtheir Ġreference Ġtranslation Ġfrom Ġthe Ġdata ) Ġand Ġseveral Ġcontrast ive Ġtranslations Ġwhich Ġdiffer Ġfrom Ġthe Ġtrue Ġone Ġonly Ġin Ġone Ġspecific Ġaspect . ĠAll Ġcontrast ive Ġtranslations Ġare Ġcorrect Ġand Ġplausible Ġtranslations Ġat Ġthe Ġsentence Ġlevel , Ġand Ġonly Ġcontext Ġreveals Ġthe Ġinconsistencies Ġbetween Ġthem . ĠThe Ġsystem Ġis Ġasked Ġto Ġscore Ġeach Ġcandidate Ġtranslation , Ġand Ġwe Ġcompute Ġthe Ġsystem Ġaccuracy Ġas Ġthe Ġproportion Ġof Ġtimes Ġthe Ġtrue Ġtranslation Ġis Ġpreferred Ġto Ġthe Ġcontrast ive Ġones . ĠTest Ġset Ġstatistics Ġare Ġshown Ġin ĠTable ĠT AB REF 15 . ĠThe Ġsuites Ġfor Ġde ix is Ġand Ġlex ical Ġcohesion Ġare Ġsplit Ġinto Ġdevelopment Ġand Ġtest Ġsets , Ġwith Ġ500 Ġexamples Ġfrom Ġeach Ġused Ġfor Ġvalidation Ġpurposes Ġand Ġthe Ġrest Ġfor Ġtesting . ĠConver gence Ġof Ġboth Ġconsistency Ġscores Ġon Ġthese Ġdevelopment Ġsets Ġand ĠB LE U Ġscore Ġon Ġa Ġgeneral Ġdevelopment Ġset Ġare Ġused Ġas Ġearly Ġstopping Ġcriteria Ġin Ġmodels Ġtraining . ĠFor Ġell ips is , Ġthere Ġis Ġno Ġdedicated Ġdevelopment Ġset , Ġso Ġwe Ġevaluate Ġon Ġall Ġthe Ġell ips is Ġdata Ġand Ġdo Ġnot Ġuse Ġit Ġfor Ġdevelopment . </s> E val uation Ġof ĠContext ual ĠPhen omen a Ġ: :: ĠPhen omen a Ġoverview </s> De ix is ĠDe ict ic Ġwords Ġor Ġphrases , Ġare Ġrefere nt ial Ġexpressions Ġwhose Ġden otation Ġdepends Ġon Ġcontext . ĠThis Ġincludes Ġpersonal Ġde ix is Ġ( âĢ ľ I âĢ Ŀ , ĠâĢ ľ you âĢ Ŀ ), Ġplace Ġde ix is Ġ( âĢ ľ here âĢ Ŀ , ĠâĢ ľ there âĢ Ŀ ), Ġand Ġdiscourse Ġde ix is , Ġwhere Ġparts Ġof Ġthe Ġdiscourse Ġare Ġreferenced Ġ( âĢ ľ that 's Ġa Ġgood Ġquestion âĢ Ŀ ). ĠThe Ġtest Ġset Ġexamples Ġare Ġall Ġrelated Ġto Ġperson Ġde ix is , Ġspecifically Ġthe ĠT - V Ġdistinction Ġbetween Ġinformal Ġand Ġformal Ġyou Ġ( Latin ĠâĢ ľ tu âĢ Ŀ Ġand ĠâĢ ľ v os âĢ Ŀ ) Ġin Ġthe ĠRussian Ġtranslations , Ġand Ġtest Ġfor Ġconsistency Ġin Ġthis Ġrespect . </s> Ell ips is ĠEll ips is Ġis Ġthe Ġomission Ġfrom Ġa Ġclause Ġof Ġone Ġor Ġmore Ġwords Ġthat Ġare Ġnevertheless Ġunderstood Ġin Ġthe Ġcontext Ġof Ġthe Ġremaining Ġelements . ĠIn Ġmachine Ġtranslation , Ġellipt ical Ġconstruct ions Ġin Ġthe Ġsource Ġlanguage Ġpose Ġa Ġproblem Ġin Ġtwo Ġsituations . ĠFirst , Ġif Ġthe Ġtarget Ġlanguage Ġdoes Ġnot Ġallow Ġthe Ġsame Ġtypes Ġof Ġell ips is , Ġrequiring Ġthe Ġel ided Ġmaterial Ġto Ġbe Ġpredicted Ġfrom Ġcontext . ĠSecond , Ġif Ġthe Ġel ided Ġmaterial Ġaffects Ġthe Ġsyntax Ġof Ġthe Ġsentence . ĠFor Ġexample , Ġin ĠRussian Ġthe Ġgram matical Ġfunction Ġof Ġa Ġnoun Ġphrase , Ġand Ġthus Ġits Ġinf lection , Ġmay Ġdepend Ġon Ġthe Ġel ided Ġverb , Ġor , Ġcon versely , Ġthe Ġverb Ġinf lection Ġmay Ġdepend Ġon Ġthe Ġel ided Ġsubject . </s> There Ġare Ġtwo Ġdifferent Ġtest Ġsets Ġfor Ġell ips is . ĠOne Ġcontains Ġexamples Ġwhere Ġa Ġmorph ological Ġform Ġof Ġa Ġnoun Ġgroup Ġin Ġthe Ġlast Ġsentence Ġcan Ġnot Ġbe Ġunderstood Ġwithout Ġcontext Ġbeyond Ġthe Ġsentence Ġlevel Ġ( âĢ ľ ell ips is Ġ( in fl .) âĢ Ŀ Ġin ĠTable ĠT AB REF 15 ). ĠAnother Ġincludes Ġcases Ġof Ġverb Ġphrase Ġell ips is Ġin ĠEnglish , Ġwhich Ġdoes Ġnot Ġexist Ġin ĠRussian , Ġthus Ġrequires Ġpredicting Ġthe Ġverb Ġwhen Ġtranslating Ġinto ĠRussian Ġ( âĢ ľ ell ips is Ġ( VP ) âĢ Ŀ Ġin ĠTable ĠT AB REF 15 ). </s> Lex ical Ġcohesion ĠThe Ġtest Ġset Ġfocuses Ġon Ġreiter ation Ġof Ġnamed Ġentities . ĠWhere Ġseveral Ġtranslations Ġof Ġa Ġnamed Ġentity Ġare Ġpossible , Ġa Ġmodel Ġhas Ġto Ġprefer Ġconsistent Ġtranslations Ġover Ġinconsistent Ġones . </s> Exper imental ĠSetup Ġ: :: ĠData Ġpre processing </s> We Ġuse Ġthe Ġpublicly Ġavailable ĠOpen Sub t itles 2018 Ġcorpus ĠB IB REF 12 Ġfor ĠEnglish Ġand ĠRussian . ĠFor Ġa Ġfair Ġcomparison Ġwith Ġprevious Ġwork , Ġwe Ġtrain Ġthe Ġbaseline ĠMT Ġsystem Ġon Ġthe Ġdata Ġreleased Ġby ĠB IB REF 11 . ĠName ly , Ġour ĠMT Ġsystem Ġis Ġtrained Ġon Ġ6 m Ġinstances . ĠThese Ġare Ġsentence Ġpairs Ġwith Ġa Ġrelative Ġtime Ġoverlap Ġof Ġsubtitle Ġframes Ġbetween Ġsource Ġand Ġtarget Ġlanguage Ġsubtitles Ġof Ġat Ġleast Ġ$ 0 . 9 $. </s> We Ġgathered Ġ30 m Ġgroups Ġof Ġ4 Ġconsecutive Ġsentences Ġas Ġour Ġmon oling ual Ġdata . ĠWe Ġused Ġonly Ġdocuments Ġnot Ġcontaining Ġgroups Ġof Ġsentences Ġfrom Ġgeneral Ġdevelopment Ġand Ġtest Ġsets Ġas Ġwell Ġas Ġfrom Ġcontrast ive Ġtest Ġsets . ĠThe Ġmain Ġresults Ġwe Ġreport Ġare Ġfor Ġthe Ġmodel Ġtrained Ġon Ġall Ġ30 m Ġfragments . </s> We Ġuse Ġthe Ġtoken ization Ġprovided Ġby Ġthe Ġcorpus Ġand Ġuse Ġmulti - ble u . per l Ġon Ġlower c ased Ġdata Ġto Ġcompute ĠB LE U Ġscore . ĠWe Ġuse Ġbeam Ġsearch Ġwith Ġa Ġbeam Ġof Ġ4 . </s> Sent ences Ġwere Ġencoded Ġusing Ġbyte - pair Ġencoding ĠB IB REF 13 , Ġwith Ġsource Ġand Ġtarget Ġvoc ab ular ies Ġof Ġabout Ġ32 000 Ġtokens . ĠTranslation Ġpairs Ġwere Ġbat ched Ġtogether Ġby Ġapproximate Ġsequence Ġlength . ĠEach Ġtraining Ġbatch Ġcontained Ġa Ġset Ġof Ġtranslation Ġpairs Ġcontaining Ġapproximately Ġ15 000 Ġsource Ġtokens . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠTrans former 's Ġperformance Ġdepends Ġheavily Ġon Ġbatch Ġsize ĠB IB REF 14 , Ġand Ġwe Ġchose Ġa Ġlarge Ġbatch Ġsize Ġto Ġensure Ġthe Ġbest Ġperformance . ĠIn Ġtraining Ġcontext - aware Ġmodels , Ġfor Ġearly Ġstopping Ġwe Ġuse Ġboth Ġconvergence Ġin ĠB LE U Ġscore Ġon Ġthe Ġgeneral Ġdevelopment Ġset Ġand Ġscores Ġon Ġthe Ġconsistency Ġdevelopment Ġsets . ĠAfter Ġtraining , Ġwe Ġaverage Ġthe Ġ5 Ġlatest Ġcheckpoints . </s> Exper imental ĠSetup Ġ: :: ĠModels </s> The Ġbaseline Ġmodel , Ġthe Ġmodel Ġused Ġfor Ġback - translation , Ġand Ġthe ĠDoc Rep air Ġmodel Ġare Ġall ĠTrans former Ġbase Ġmodels ĠB IB REF 15 . ĠMore Ġprecisely , Ġthe Ġnumber Ġof Ġlayers Ġis Ġ$ N = 6 $ Ġwith Ġ$ h Ġ= Ġ8 $ Ġparallel Ġattention Ġlayers , Ġor Ġheads . ĠThe Ġdimension ality Ġof Ġinput Ġand Ġoutput Ġis Ġ$ d _{ model } Ġ= Ġ512 $, Ġand Ġthe Ġinner - layer Ġof Ġa Ġfeed - forward Ġnetworks Ġhas Ġdimension ality Ġ$ d _{ ff } = 20 48 $. ĠWe Ġuse Ġregular ization Ġas Ġdescribed Ġin ĠB IB REF 15 . </s> As Ġa Ġsecond Ġbaseline , Ġwe Ġuse Ġthe Ġtwo - pass ĠCAD ec Ġmodel ĠB IB REF 11 . ĠThe Ġfirst Ġpass Ġproduces Ġsentence - level Ġtranslations . ĠThe Ġsecond Ġpass Ġtakes Ġboth Ġthe Ġfirst - pass Ġtranslation Ġand Ġrepresentations Ġof Ġthe Ġcontext Ġsentences Ġas Ġinput Ġand Ġreturns Ġcontextual ized Ġtranslations . ĠCAD ec Ġrequires Ġdocument - level Ġparallel Ġtraining Ġdata , Ġwhile ĠDoc Rep air Ġonly Ġneeds Ġmon oling ual Ġtraining Ġdata . </s> Exper imental ĠSetup Ġ: :: ĠGener ating Ġround - trip Ġtranslations </s> On Ġthe Ġselected Ġ6 m Ġinstances Ġwe Ġtrain Ġsentence - level Ġtranslation Ġmodels Ġin Ġboth Ġdirections . ĠTo Ġcreate Ġtraining Ġdata Ġfor ĠDoc Rep air , Ġwe Ġproceed Ġas Ġfollows . ĠThe ĠRussian Ġmon oling ual Ġdata Ġis Ġfirst Ġtranslated Ġinto ĠEnglish , Ġusing Ġthe ĠRussian $ \ right arrow Ġ$ English Ġmodel Ġand Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ4 . ĠThen , Ġwe Ġuse Ġthe ĠEnglish $ \ right arrow Ġ$ Russian Ġmodel Ġto Ġsample Ġtranslations Ġwith Ġtemperature Ġof Ġ$ 0 { .} 5 $. ĠFor Ġeach Ġsentence , Ġwe Ġpre comp ute Ġ20 Ġsampled Ġtranslations Ġand Ġrandomly Ġchoose Ġone Ġof Ġthem Ġwhen Ġforming Ġa Ġtraining Ġmin ib atch Ġfor ĠDoc Rep air . ĠAlso , Ġin Ġtraining , Ġwe Ġreplace Ġeach Ġtoken Ġin Ġthe Ġinput Ġwith Ġa Ġrandom Ġone Ġwith Ġthe Ġprobability Ġof Ġ$ 10 \ % $. </s> Exper imental ĠSetup Ġ: :: ĠOptim izer </s> As Ġin ĠB IB REF 15 , Ġwe Ġuse Ġthe ĠAdam Ġoptim izer ĠB IB REF 16 , Ġthe Ġparameters Ġare Ġ$\ beta Ġ_ 1 Ġ= Ġ0 { .} 9 $, Ġ$\ beta Ġ_ 2 Ġ= Ġ0 { .} 98 $ Ġand Ġ$\ v are ps ilon Ġ= Ġ10 ^{ - 9 } $. ĠWe Ġvary Ġthe Ġlearning Ġrate Ġover Ġthe Ġcourse Ġof Ġtraining Ġusing Ġthe Ġformula : </s> where Ġ$ warm up \ _ steps Ġ= Ġ16 000 $ Ġand Ġ$ scale = 4 $. </s> Results Ġ: :: ĠGeneral Ġresults </s> The ĠB LE U Ġscores Ġare Ġprovided Ġin ĠTable ĠT AB REF 24 Ġ( we Ġevaluate Ġtranslations Ġof Ġ4 - sent ence Ġfragments ). ĠTo Ġsee Ġwhich Ġpart Ġof Ġthe Ġimprovement Ġis Ġdue Ġto Ġfixing Ġagreement Ġbetween Ġsentences Ġrather Ġthan Ġsimply Ġsentence - level Ġpost - ed iting , Ġwe Ġtrain Ġthe Ġsame Ġrepair Ġmodel Ġat Ġthe Ġsentence Ġlevel . ĠEach Ġsentence Ġin Ġa Ġgroup Ġis Ġnow Ġcorrected Ġseparately , Ġthen Ġthey Ġare Ġput Ġback Ġtogether Ġin Ġa Ġgroup . ĠOne Ġcan Ġsee Ġthat Ġmost Ġof Ġthe Ġimprovement Ġcomes Ġfrom Ġaccounting Ġfor Ġextra - sent ential Ġdependencies . ĠDoc Rep air Ġoutper forms Ġthe Ġbaseline Ġand ĠCAD ec Ġby Ġ0 . 7 ĠB LE U , Ġand Ġits Ġsentence - level Ġrepair Ġversion Ġby Ġ0 . 5 ĠB LE U . </s> Results Ġ: :: ĠCons ist ency Ġresults </s> Sc ores Ġon Ġthe Ġphenomena Ġtest Ġsets Ġare Ġprovided Ġin ĠTable ĠT AB REF 26 . ĠFor Ġde ix is , Ġlex ical Ġcohesion Ġand Ġell ips is Ġ( in fl .) Ġwe Ġsee Ġsubstantial Ġimprovements Ġover Ġboth Ġthe Ġbaseline Ġand ĠCAD ec . ĠThe Ġlargest Ġimprovement Ġover ĠCAD ec Ġ( 22 . 5 Ġpercentage Ġpoints ) Ġis Ġfor Ġlex ical Ġcohesion . ĠHowever , Ġthere Ġis Ġa Ġdrop Ġof Ġalmost Ġ5 Ġpercentage Ġpoints Ġfor ĠVP Ġell ips is . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġbecause Ġit Ġis Ġhard Ġto Ġlearn Ġto Ġcorrect Ġinconsistencies Ġin Ġtranslations Ġcaused Ġby ĠVP Ġell ips is Ġrelying Ġon Ġmon oling ual Ġdata Ġalone . ĠFigure ĠFIG REF 27 ( a ) Ġshows Ġan Ġexample Ġof Ġinconsistency Ġcaused Ġby ĠVP Ġell ips is Ġin ĠEnglish . ĠThere Ġis Ġno ĠVP Ġell ips is Ġin ĠRussian , Ġand Ġwhen Ġtranslating Ġauxiliary ĠâĢ ľ did âĢ Ŀ Ġthe Ġmodel Ġhas Ġto Ġguess Ġthe Ġmain Ġverb . ĠFigure ĠFIG REF 27 ( b ) Ġshows Ġsteps Ġof Ġgenerating Ġround - trip Ġtranslations Ġfor Ġthe Ġtarget Ġside Ġof Ġthe Ġprevious Ġexample . ĠWhen Ġtranslating Ġfrom ĠRussian , Ġmain Ġverbs Ġare Ġunlikely Ġto Ġbe Ġtranslated Ġas Ġthe Ġauxiliary ĠâĢ ľ do âĢ Ŀ Ġin ĠEnglish , Ġand Ġhence Ġthe ĠVP Ġell ips is Ġis Ġrarely Ġpresent Ġon Ġthe ĠEnglish Ġside . ĠThis Ġimplies Ġthe Ġmodel Ġtrained Ġusing Ġthe Ġround - trip Ġtranslations Ġwill Ġnot Ġbe Ġexposed Ġto Ġmany ĠVP Ġell ips is Ġexamples Ġin Ġtraining . ĠWe Ġdiscuss Ġthis Ġfurther Ġin ĠSection ĠSEC REF 34 . </s> Table ĠT AB REF 28 Ġprovides Ġscores Ġfor Ġde ix is Ġand Ġlex ical Ġcohesion Ġseparately Ġfor Ġdifferent Ġdistances Ġbetween Ġsentences Ġrequiring Ġconsistency . ĠIt Ġcan Ġbe Ġseen , Ġthat Ġthe Ġperformance Ġof ĠDoc Rep air Ġde grades Ġless Ġthan Ġthat Ġof ĠCAD ec Ġwhen Ġthe Ġdistance Ġbetween Ġsentences Ġrequiring Ġconsistency Ġgets Ġlarger . </s> Results Ġ: :: ĠHuman Ġevaluation </s> We Ġconduct Ġa Ġhuman Ġevaluation Ġon Ġrandom Ġ700 Ġexamples Ġfrom Ġour Ġgeneral Ġtest Ġset . ĠWe Ġpicked Ġonly Ġexamples Ġwhere Ġa ĠDoc Rep air Ġtranslation Ġis Ġnot Ġa Ġfull Ġcopy Ġof Ġthe Ġbaseline Ġone . </s> The Ġannot ators Ġwere Ġprovided Ġan Ġoriginal Ġgroup Ġof Ġsentences Ġin ĠEnglish Ġand Ġtwo Ġtranslations : Ġbaseline Ġcontext - agn ostic Ġone Ġand Ġthe Ġone Ġcorrected Ġby Ġthe ĠDoc Rep air Ġmodel . ĠTrans lations Ġwere Ġpresented Ġin Ġrandom Ġorder Ġwith Ġno Ġindication Ġwhich Ġmodel Ġthey Ġcame Ġfrom . ĠThe Ġtask Ġis Ġto Ġpick Ġone Ġof Ġthe Ġthree Ġoptions : Ġ( 1 ) Ġthe Ġfirst Ġtranslation Ġis Ġbetter , Ġ( 2 ) Ġthe Ġsecond Ġtranslation Ġis Ġbetter , Ġ( 3 ) Ġthe Ġtranslations Ġare Ġof Ġequal Ġquality . ĠThe Ġannot ators Ġwere Ġasked Ġto Ġavoid Ġthe Ġthird Ġanswer Ġif Ġthey Ġare Ġable Ġto Ġgive Ġpreference Ġto Ġone Ġof Ġthe Ġtranslations . ĠNo Ġother Ġguidelines Ġwere Ġgiven . </s> The Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 30 . ĠIn Ġabout Ġ$ 52 \ % $ Ġof Ġthe Ġcases Ġannot ators Ġmarked Ġtranslations Ġas Ġhaving Ġequal Ġquality . ĠAmong Ġthe Ġcases Ġwhere Ġone Ġof Ġthe Ġtranslations Ġwas Ġmarked Ġbetter Ġthan Ġthe Ġother , Ġthe ĠDoc Rep air Ġtranslation Ġwas Ġmarked Ġbetter Ġin Ġ$ 73 \ % $ Ġof Ġthe Ġcases . ĠThis Ġshows Ġa Ġstrong Ġpreference Ġof Ġthe Ġannot ators Ġfor Ġcorrected Ġtranslations Ġover Ġthe Ġbaseline Ġones . </s> V ary ing ĠTraining ĠData </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġinfluence Ġof Ġthe Ġtraining Ġdata Ġchosen Ġfor Ġdocument - level Ġmodels . ĠIn Ġall Ġexperiments , Ġwe Ġused Ġthe ĠDoc Rep air Ġmodel . </s> V ary ing ĠTraining ĠData Ġ: :: ĠThe Ġamount Ġof Ġtraining Ġdata </s> Table ĠT AB REF 33 Ġprovides ĠB LE U Ġand Ġconsistency Ġscores Ġfor Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġdifferent Ġamount Ġof Ġdata . ĠWe Ġsee Ġthat Ġeven Ġwhen Ġusing Ġa Ġdataset Ġof Ġmoderate Ġsize Ġ( e . g ., Ġ5 m Ġfragments ) Ġwe Ġcan Ġachieve Ġperformance Ġcomparable Ġto Ġthe Ġmodel Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġdata Ġ( 30 m Ġfragments ). ĠMoreover , Ġwe Ġnotice Ġthat Ġde ix is Ġscores Ġare Ġless Ġsensitive Ġto Ġthe Ġamount Ġof Ġtraining Ġdata Ġthan Ġlex ical Ġcohesion Ġand Ġell ips is Ġscores . ĠThe Ġreason Ġmight Ġbe Ġthat , Ġas Ġwe Ġobserved Ġin Ġour Ġprevious Ġwork ĠB IB REF 11 , Ġinconsistencies Ġin Ġtranslations Ġdue Ġto Ġthe Ġpresence Ġof Ġde ict ic Ġwords Ġand Ġphrases Ġare Ġmore Ġfrequent Ġin Ġthis Ġdataset Ġthan Ġother Ġtypes Ġof Ġinconsistencies . ĠAlso , Ġas Ġwe Ġshow Ġin ĠSection ĠSEC REF 7 , Ġthis Ġis Ġthe Ġphenomenon Ġthe Ġmodel Ġlearns Ġfaster Ġin Ġtraining . </s> V ary ing ĠTraining ĠData Ġ: :: ĠOne - way Ġvs Ġround - trip Ġtranslations </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġlimitations Ġof Ġusing Ġonly Ġmon oling ual Ġdata Ġto Ġmodel Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations . ĠIn ĠSection ĠSEC REF 25 Ġwe Ġobserved Ġa Ġdrop Ġin Ġperformance Ġon ĠVP Ġell ips is Ġfor ĠDoc Rep air Ġcompared Ġto ĠCAD ec , Ġwhich Ġwas Ġtrained Ġon Ġparallel Ġdata . ĠWe Ġhypothesized Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġdifferences Ġbetween Ġone - way Ġand Ġround - trip Ġtranslations , Ġand Ġnow Ġwe Ġtest Ġthis Ġhypothesis . ĠTo Ġdo Ġso , Ġwe Ġfix Ġthe Ġdataset Ġand Ġvary Ġthe Ġway Ġin Ġwhich Ġthe Ġinput Ġfor ĠDoc Rep air Ġis Ġgenerated : Ġround - trip Ġor Ġone - way Ġtranslations . ĠThe Ġlatter Ġassumes Ġthat Ġdocument - level Ġdata Ġis Ġparallel , Ġand Ġtranslations Ġare Ġsampled Ġfrom Ġthe Ġsource Ġside Ġof Ġthe Ġsentences Ġin Ġa Ġgroup Ġrather Ġthan Ġfrom Ġtheir Ġback - trans lations . ĠFor Ġparallel Ġdata , Ġwe Ġtake Ġ1 . 5 m Ġparallel Ġinstances Ġwhich Ġwere Ġused Ġfor ĠCAD ec Ġtraining Ġand Ġadd Ġ1 m Ġinstances Ġfrom Ġour Ġmon oling ual Ġdata . ĠFor Ġsegments Ġin Ġthe Ġparallel Ġpart , Ġwe Ġeither Ġsample Ġtranslations Ġfrom Ġthe Ġsource Ġside Ġor Ġuse Ġround - trip Ġtranslations . ĠThe Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 35 . </s> The Ġmodel Ġtrained Ġon Ġone - way Ġtranslations Ġis Ġslightly Ġbetter Ġthan Ġthe Ġone Ġtrained Ġon Ġround - trip Ġtranslations . ĠAs Ġexpected , ĠVP Ġell ips is Ġis Ġthe Ġhardest Ġphenomena Ġto Ġbe Ġcaptured Ġusing Ġround - trip Ġtranslations , Ġand Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġone - way Ġtranslated Ġdata Ġgains Ġ6 % Ġaccuracy Ġon Ġthis Ġtest Ġset . ĠThis Ġshows Ġthat Ġthe ĠDoc Rep air Ġmodel Ġbenefits Ġfrom Ġhaving Ġaccess Ġto Ġnon - sy nt hetic ĠEnglish Ġdata . ĠThis Ġresults Ġin Ġexposing ĠDoc Rep air Ġat Ġtraining Ġtime Ġto ĠRussian Ġtranslations Ġwhich Ġsuffer Ġfrom Ġthe Ġsame Ġinconsistencies Ġas Ġthe Ġones Ġit Ġwill Ġhave Ġto Ġcorrect Ġat Ġtest Ġtime . </s> V ary ing ĠTraining ĠData Ġ: :: ĠFil tering : Ġmon oling ual Ġ( no Ġfiltering ) Ġor Ġparallel </s> Note Ġthat Ġthe Ġscores Ġof Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġ2 . 5 m Ġinstances Ġrandomly Ġchosen Ġfrom Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 33 ) Ġare Ġdifferent Ġfrom Ġthe Ġones Ġfor Ġthe Ġmodel Ġtrained Ġon Ġ2 . 5 m Ġinstances Ġcombined Ġfrom Ġparallel Ġand Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 35 ). ĠFor Ġconvenience , Ġwe Ġshow Ġthese Ġtwo Ġin ĠTable ĠT AB REF 36 . </s> The Ġdomain , Ġthe Ġdataset Ġthese Ġtwo Ġdata Ġsamples Ġwere Ġgathered Ġfrom , Ġand Ġthe Ġway Ġwe Ġgenerated Ġtraining Ġdata Ġfor ĠDoc Rep air Ġ( round - trip Ġtranslations ) Ġare Ġall Ġthe Ġsame . ĠThe Ġonly Ġdifference Ġlies Ġin Ġhow Ġthe Ġdata Ġwas Ġfiltered . ĠFor Ġparallel Ġdata , Ġas Ġin Ġthe Ġprevious Ġwork ĠB IB REF 6 , Ġwe Ġpicked Ġonly Ġsentence Ġpairs Ġwith Ġlarge Ġrelative Ġtime Ġoverlap Ġof Ġsubtitle Ġframes Ġbetween Ġsource - language Ġand Ġtarget - language Ġsubtitles . ĠThis Ġis Ġnecessary Ġto Ġensure Ġthe Ġquality Ġof Ġtranslation Ġdata : Ġone Ġneeds Ġgroups Ġof Ġconsecutive Ġsentences Ġin Ġthe Ġtarget Ġlanguage Ġwhere Ġevery Ġsentence Ġhas Ġa Ġreliable Ġtranslation . </s> Table ĠT AB REF 36 Ġshows Ġthat Ġthe Ġquality Ġof Ġthe Ġmodel Ġtrained Ġon Ġdata Ġwhich Ġcame Ġfrom Ġthe Ġparallel Ġpart Ġis Ġworse Ġthan Ġthe Ġone Ġtrained Ġon Ġmon oling ual Ġdata . ĠThis Ġindicates Ġthat Ġrequiring Ġeach Ġsentence Ġin Ġa Ġgroup Ġto Ġhave Ġa Ġreliable Ġtranslation Ġchanges Ġthe Ġdistribution Ġof Ġthe Ġdata , Ġwhich Ġmight Ġbe Ġnot Ġbeneficial Ġfor Ġtranslation Ġquality Ġand Ġprovides Ġextra Ġmotivation Ġfor Ġusing Ġmon oling ual Ġdata . </s> Learning ĠDynamics </s> Let Ġus Ġnow Ġlook Ġinto Ġhow Ġthe Ġprocess Ġof ĠDoc Rep air Ġtraining Ġprogresses . ĠFigure ĠFIG REF 38 Ġshows Ġhow Ġthe ĠB LE U Ġscores Ġwith Ġthe Ġreference Ġtranslation Ġand Ġwith Ġthe Ġbaseline Ġcontext - agn ostic Ġtranslation Ġ( i . e . Ġthe Ġinput Ġfor Ġthe ĠDoc Rep air Ġmodel ) Ġare Ġchanging Ġduring Ġtraining . ĠFirst , Ġthe Ġmodel Ġquickly Ġlearns Ġto Ġcopy Ġbaseline Ġtranslations : Ġthe ĠB LE U Ġscore Ġwith Ġthe Ġbaseline Ġis Ġvery Ġhigh . ĠThen Ġit Ġgradually Ġlearns Ġto Ġchange Ġthem , Ġwhich Ġleads Ġto Ġan Ġimprovement Ġin ĠB LE U Ġwith Ġthe Ġreference Ġtranslation Ġand Ġa Ġdrop Ġin ĠB LE U Ġwith Ġthe Ġbaseline . ĠImport antly , Ġthe Ġmodel Ġis Ġreluctant Ġto Ġmake Ġchanges : Ġthe ĠB LE U Ġscore Ġbetween Ġtranslations Ġof Ġthe Ġconver ged Ġmodel Ġand Ġthe Ġbaseline Ġis Ġ82 . 5 . ĠWe Ġcount Ġthe Ġnumber Ġof Ġchanged Ġsentences Ġin Ġevery Ġ4 - sent ence Ġfragment Ġin Ġthe Ġtest Ġset Ġand Ġplot Ġthe Ġhist ogram Ġin ĠFigure ĠFIG REF 38 . ĠIn Ġover Ġthan Ġ20 $ \ % $ Ġof Ġthe Ġcases Ġthe Ġmodel Ġhas Ġnot Ġchanged Ġbase Ġtranslations Ġat Ġall . ĠIn Ġalmost Ġ$ 40 \ % $, Ġit Ġmodified Ġonly Ġone Ġsentence Ġand Ġleft Ġthe Ġremaining Ġ3 Ġsentences Ġunchanged . ĠThe Ġmodel Ġchanged Ġmore Ġthan Ġhalf Ġsentences Ġin Ġa Ġgroup Ġin Ġonly Ġ$ 14 \ % $ Ġof Ġthe Ġcases . ĠSeveral Ġexamples Ġof Ġthe ĠDoc Rep air Ġtranslations Ġare Ġshown Ġin ĠFigure ĠFIG REF 43 . </s> Figure ĠFIG REF 42 Ġshows Ġhow Ġconsistency Ġscores Ġare Ġchanging Ġin Ġtraining . ĠFor Ġde ix is , Ġthe Ġmodel Ġachieves Ġthe Ġfinal Ġquality Ġquite Ġquickly ; Ġfor Ġthe Ġrest , Ġit Ġneeds Ġa Ġlarge Ġnumber Ġof Ġtraining Ġsteps Ġto Ġconverge . </s> Related ĠWork </s> Our Ġwork Ġis Ġmost Ġclosely Ġrelated Ġto Ġtwo Ġlines Ġof Ġresearch : Ġautomatic Ġpost - ed iting Ġ( APE ) Ġand Ġdocument - level Ġmachine Ġtranslation . </s> Related ĠWork Ġ: :: ĠAutomatic Ġpost - ed iting </s> Our Ġmodel Ġcan Ġbe Ġregarded Ġas Ġan Ġautomatic Ġpost - ed iting Ġsystem ĠâĢĵ Ġa Ġsystem Ġdesigned Ġto Ġfix Ġsystematic ĠMT Ġerrors Ġthat Ġis Ġdec ou pled Ġfrom Ġthe Ġmain ĠMT Ġsystem . ĠAutomatic Ġpost - ed iting Ġhas Ġa Ġlong Ġhistory , Ġincluding Ġrule - based ĠB IB REF 17 , Ġstatistical ĠB IB REF 18 Ġand Ġneural Ġapproaches ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . </s> In Ġterms Ġof Ġarchitectures , Ġmodern Ġapproaches Ġuse Ġneural Ġsequence - to - sequence Ġmodels , Ġeither Ġmulti - source Ġarchitectures Ġthat Ġconsider Ġboth Ġthe Ġoriginal Ġsource Ġand Ġthe Ġbaseline Ġtranslation ĠB IB REF 19 , ĠB IB REF 20 , Ġor Ġmon oling ual Ġrepair Ġsystems , Ġas Ġin ĠB IB REF 21 , Ġwhich Ġis Ġconcurrent Ġwork Ġto Ġours . ĠTrue Ġpost - ed iting Ġdatasets Ġare Ġtypically Ġsmall Ġand Ġexpensive Ġto Ġcreate ĠB IB REF 22 , Ġhence Ġsynthetic Ġtraining Ġdata Ġhas Ġbeen Ġcreated Ġthat Ġuses Ġoriginal Ġmon oling ual Ġdata Ġas Ġoutput Ġfor Ġthe Ġsequence - to - sequence Ġmodel , Ġpaired Ġwith Ġan Ġautomatic Ġback - translation ĠB IB REF 23 Ġand / or Ġround - trip Ġtranslation Ġas Ġits Ġinput ( s ) ĠB IB REF 19 , ĠB IB REF 21 . </s> While Ġprevious Ġwork Ġon Ġautomatic Ġpost - ed iting Ġoperated Ġon Ġthe Ġsentence Ġlevel , Ġthe Ġmain Ġnovelty Ġof Ġthis Ġwork Ġis Ġthat Ġour ĠDoc Rep air Ġmodel Ġoperates Ġon Ġgroups Ġof Ġsentences Ġand Ġis Ġthus Ġable Ġto Ġfix Ġconsistency Ġerrors Ġcaused Ġby Ġthe Ġcontext - agn ostic Ġbaseline ĠMT Ġsystem . ĠWe Ġconsider Ġthis Ġstrategy Ġof Ġsentence - level Ġbaseline Ġtranslation Ġand Ġcontext - aware Ġmon oling ual Ġrepair Ġattractive Ġwhen Ġparallel Ġdocument - level Ġdata Ġis Ġscarce . </s> For Ġtraining , Ġthe ĠDoc Rep air Ġmodel Ġonly Ġrequires Ġmon oling ual Ġdocument - level Ġdata . ĠWhile Ġwe Ġcreate Ġsynthetic Ġtraining Ġdata Ġvia Ġround - trip Ġtranslation Ġsimilarly Ġto Ġearlier Ġwork ĠB IB REF 19 , ĠB IB REF 21 , Ġnote Ġthat Ġwe Ġpurposefully Ġuse Ġsentence - level ĠMT Ġsystems Ġfor Ġthis Ġto Ġcreate Ġthe Ġtypes Ġof Ġconsistency Ġerrors Ġthat Ġwe Ġaim Ġto Ġfix Ġwith Ġthe Ġcontext - aware ĠDoc Rep air Ġmodel . ĠNot Ġall Ġtypes Ġof Ġconsistency Ġerrors Ġthat Ġwe Ġwant Ġto Ġfix Ġemerge Ġfrom Ġa Ġround - trip Ġtranslation , Ġso Ġaccess Ġto Ġparallel Ġdocument - level Ġdata Ġcan Ġbe Ġuseful Ġ( Section ĠSEC REF 34 ). </s> Related ĠWork Ġ: :: ĠDocument - level ĠN MT </s> Ne ural Ġmodels Ġof ĠMT Ġthat Ġgo Ġbeyond Ġthe Ġsentence - level Ġare Ġan Ġactive Ġresearch Ġarea ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 10 , ĠB IB REF 9 , ĠB IB REF 11 . ĠTypically , Ġthe Ġmain ĠMT Ġsystem Ġis Ġmodified Ġto Ġtake Ġadditional Ġcontext Ġas Ġits Ġinput . ĠOne Ġlimitation Ġof Ġthese Ġapproaches Ġis Ġthat Ġthey Ġassume Ġthat Ġparallel Ġdocument - level Ġtraining Ġdata Ġis Ġavailable . </s> Cl os est Ġto Ġour Ġwork Ġare Ġtwo - pass Ġmodels Ġfor Ġdocument - level ĠN MT ĠB IB REF 24 , ĠB IB REF 11 , Ġwhere Ġa Ġsecond , Ġcontext - aware Ġmodel Ġtakes Ġthe Ġtranslation Ġand Ġhidden Ġrepresentations Ġof Ġthe Ġsentence - level Ġfirst - pass Ġmodel Ġas Ġits Ġinput . ĠThe Ġsecond - pass Ġmodel Ġcan Ġin Ġprinciple Ġbe Ġtrained Ġon Ġa Ġsubset Ġof Ġthe Ġparallel Ġtraining Ġdata ĠB IB REF 11 , Ġsomewhat Ġrelaxing Ġthe Ġassumption Ġthat Ġall Ġtraining Ġdata Ġis Ġat Ġthe Ġdocument Ġlevel . </s> Our Ġwork Ġis Ġdifferent Ġfrom Ġthis Ġprevious Ġwork Ġin Ġtwo Ġmain Ġrespects . ĠFirstly , Ġwe Ġshow Ġthat Ġconsistency Ġcan Ġbe Ġimproved Ġwith Ġonly Ġmon oling ual Ġdocument - level Ġtraining Ġdata . ĠSecondly , Ġthe ĠDoc Rep air Ġmodel Ġis Ġdec ou pled Ġfrom Ġthe Ġfirst - pass ĠMT Ġsystem , Ġwhich Ġimproves Ġits Ġport ability . </s> Con clusions </s> We Ġintroduce Ġthe Ġfirst Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠWe Ġpropose Ġa Ġmon oling ual ĠDoc Rep air Ġmodel Ġto Ġcorrect Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations . ĠThe Ġmodel Ġperforms Ġautomatic Ġpost - ed iting Ġon Ġa Ġsequence Ġof Ġsentence - level Ġtranslations , Ġrefining Ġtranslations Ġof Ġsentences Ġin Ġcontext Ġof Ġeach Ġother . ĠOur Ġapproach Ġresults Ġin Ġsubstantial Ġimprovements Ġin Ġtranslation Ġquality Ġas Ġmeasured Ġby ĠB LE U , Ġtargeted Ġcontrast ive Ġevaluation Ġof Ġseveral Ġdiscourse Ġphenomena Ġand Ġhuman Ġevaluation . ĠMoreover , Ġwe Ġperform Ġerror Ġanalysis Ġand Ġdetect Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠWhile Ġin Ġthe Ġcurrent Ġwork Ġwe Ġused Ġtext Ġfragments Ġof Ġ4 Ġsentences , Ġin Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġconsider Ġlonger Ġcontexts . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcomments . ĠThe Ġauthors Ġalso Ġthank ĠDavid ĠTal bot Ġand ĠY and ex ĠMachine ĠTranslation Ġteam Ġfor Ġhelpful Ġdiscussions Ġand Ġinspiration . ĠIvan ĠTit ov Ġacknowledges Ġsupport Ġof Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ĠSt G ĠBroad Sem Ġ6 78 254 ) Ġand Ġthe ĠDutch ĠNational ĠScience ĠFoundation Ġ( NW O ĠVID I Ġ6 39 . 022 . 518 ). ĠRico ĠS enn rich Ġacknowledges Ġsupport Ġfrom Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( 10 52 12 _ 169 888 ), Ġthe ĠEuropean ĠUnion âĢ Ļ s ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġ( gr ant Ġagreement Ġno Ġ8 254 60 ), Ġand Ġthe ĠRoyal ĠSociety Ġ( NA F \ R 1 \ 18 01 22 ).
Introduction Machine translation has made remarkable progress, and studies claiming it to reach a human parity are starting to appear BIBREF0. However, when evaluating translations of the whole documents rather than isolated sentences, human raters show a stronger preference for human over machine translation BIBREF1. These findings emphasize the need to shift towards context-aware machine translation both from modeling and evaluation perspective. Most previous work on context-aware NMT assumed that either all the bilingual data is available at the document level BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 or at least its fraction BIBREF11. But in practical scenarios, document-level parallel data is often scarce, which is one of the challenges when building a context-aware system. We introduce an approach to context-aware machine translation using only monolingual document-level data. In our setting, a separate monolingual sequence-to-sequence model (DocRepair) is used to correct sentence-level translations of adjacent sentences. The key idea is to use monolingual data to imitate typical inconsistencies between context-agnostic translations of isolated sentences. The DocRepair model is trained to map inconsistent groups of sentences into consistent ones. The consistent groups come from the original training data; the inconsistent groups are obtained by sampling round-trip translations for each isolated sentence. To validate the performance of our model, we use three kinds of evaluation: the BLEU score, contrastive evaluation of translation of several discourse phenomena BIBREF11, and human evaluation. We show strong improvements for all metrics. We analyze which discourse phenomena are hard to capture using monolingual data only. Using contrastive test sets for targeted evaluation of several contextual phenomena, we compare the performance of the models trained on round-trip translations and genuine document-level parallel data. Among the four phenomena in the test sets we use (deixis, lexical cohesion, VP ellipsis and ellipsis which affects NP inflection) we find VP ellipsis to be the hardest phenomenon to be captured using round-trip translations. Our key contributions are as follows: we introduce the first approach to context-aware machine translation using only monolingual document-level data; our approach shows substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation; we show which discourse phenomena are hard to capture using monolingual data only. Our Approach: Document-level Repair We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations of a context-agnostic MT system. It does not use any states of a trained MT model whose outputs it corrects and therefore can in principle be trained to correct translations from any black-box MT system. The DocRepair model requires only monolingual document-level data in the target language. It is a monolingual sequence-to-sequence model that maps inconsistent groups of sentences into consistent ones. Consistent groups come from monolingual document-level data. To obtain inconsistent groups, each sentence in a group is replaced with its round-trip translation produced in isolation from context. More formally, forming a training minibatch for the DocRepair model involves the following steps (see also Figure FIGREF9): sample several groups of sentences from the monolingual data; for each sentence in a group, (i) translate it using a target-to-source MT model, (ii) sample a translation of this back-translated sentence in the source language using a source-to-target MT model; using these round-trip translations of isolated sentences, form an inconsistent version of the initial groups; use inconsistent groups as input for the DocRepair model, consistent ones as output. At test time, the process of getting document-level translations is two-step (Figure FIGREF10): produce translations of isolated sentences using a context-agnostic MT model; apply the DocRepair model to a sequence of context-agnostic translations to correct inconsistencies between translations. In the scope of the current work, the DocRepair model is the standard sequence-to-sequence Transformer. Sentences in a group are concatenated using a reserved token-separator between sentences. The Transformer is trained to correct these long inconsistent pseudo-sentences into consistent ones. The token-separator is then removed from corrected translations. Evaluation of Contextual Phenomena We use contrastive test sets for evaluation of discourse phenomena for English-Russian by BIBREF11. These test sets allow for testing different kinds of phenomena which, as we show, can be captured from monolingual data with varying success. In this section, we provide test sets statistics and briefly describe the tested phenomena. For more details, the reader is referred to BIBREF11. Evaluation of Contextual Phenomena ::: Test sets There are four test sets in the suite. Each test set contains contrastive examples. It is specifically designed to test the ability of a system to adapt to contextual information and handle the phenomenon under consideration. Each test instance consists of a true example (a sequence of sentences and their reference translation from the data) and several contrastive translations which differ from the true one only in one specific aspect. All contrastive translations are correct and plausible translations at the sentence level, and only context reveals the inconsistencies between them. The system is asked to score each candidate translation, and we compute the system accuracy as the proportion of times the true translation is preferred to the contrastive ones. Test set statistics are shown in Table TABREF15. The suites for deixis and lexical cohesion are split into development and test sets, with 500 examples from each used for validation purposes and the rest for testing. Convergence of both consistency scores on these development sets and BLEU score on a general development set are used as early stopping criteria in models training. For ellipsis, there is no dedicated development set, so we evaluate on all the ellipsis data and do not use it for development. Evaluation of Contextual Phenomena ::: Phenomena overview Deixis Deictic words or phrases, are referential expressions whose denotation depends on context. This includes personal deixis (“I”, “you”), place deixis (“here”, “there”), and discourse deixis, where parts of the discourse are referenced (“that's a good question”). The test set examples are all related to person deixis, specifically the T-V distinction between informal and formal you (Latin “tu” and “vos”) in the Russian translations, and test for consistency in this respect. Ellipsis Ellipsis is the omission from a clause of one or more words that are nevertheless understood in the context of the remaining elements. In machine translation, elliptical constructions in the source language pose a problem in two situations. First, if the target language does not allow the same types of ellipsis, requiring the elided material to be predicted from context. Second, if the elided material affects the syntax of the sentence. For example, in Russian the grammatical function of a noun phrase, and thus its inflection, may depend on the elided verb, or, conversely, the verb inflection may depend on the elided subject. There are two different test sets for ellipsis. One contains examples where a morphological form of a noun group in the last sentence can not be understood without context beyond the sentence level (“ellipsis (infl.)” in Table TABREF15). Another includes cases of verb phrase ellipsis in English, which does not exist in Russian, thus requires predicting the verb when translating into Russian (“ellipsis (VP)” in Table TABREF15). Lexical cohesion The test set focuses on reiteration of named entities. Where several translations of a named entity are possible, a model has to prefer consistent translations over inconsistent ones. Experimental Setup ::: Data preprocessing We use the publicly available OpenSubtitles2018 corpus BIBREF12 for English and Russian. For a fair comparison with previous work, we train the baseline MT system on the data released by BIBREF11. Namely, our MT system is trained on 6m instances. These are sentence pairs with a relative time overlap of subtitle frames between source and target language subtitles of at least $0.9$. We gathered 30m groups of 4 consecutive sentences as our monolingual data. We used only documents not containing groups of sentences from general development and test sets as well as from contrastive test sets. The main results we report are for the model trained on all 30m fragments. We use the tokenization provided by the corpus and use multi-bleu.perl on lowercased data to compute BLEU score. We use beam search with a beam of 4. Sentences were encoded using byte-pair encoding BIBREF13, with source and target vocabularies of about 32000 tokens. Translation pairs were batched together by approximate sequence length. Each training batch contained a set of translation pairs containing approximately 15000 source tokens. It has been shown that Transformer's performance depends heavily on batch size BIBREF14, and we chose a large batch size to ensure the best performance. In training context-aware models, for early stopping we use both convergence in BLEU score on the general development set and scores on the consistency development sets. After training, we average the 5 latest checkpoints. Experimental Setup ::: Models The baseline model, the model used for back-translation, and the DocRepair model are all Transformer base models BIBREF15. More precisely, the number of layers is $N=6$ with $h = 8$ parallel attention layers, or heads. The dimensionality of input and output is $d_{model} = 512$, and the inner-layer of a feed-forward networks has dimensionality $d_{ff}=2048$. We use regularization as described in BIBREF15. As a second baseline, we use the two-pass CADec model BIBREF11. The first pass produces sentence-level translations. The second pass takes both the first-pass translation and representations of the context sentences as input and returns contextualized translations. CADec requires document-level parallel training data, while DocRepair only needs monolingual training data. Experimental Setup ::: Generating round-trip translations On the selected 6m instances we train sentence-level translation models in both directions. To create training data for DocRepair, we proceed as follows. The Russian monolingual data is first translated into English, using the Russian$\rightarrow $English model and beam search with beam size of 4. Then, we use the English$\rightarrow $Russian model to sample translations with temperature of $0{.}5$. For each sentence, we precompute 20 sampled translations and randomly choose one of them when forming a training minibatch for DocRepair. Also, in training, we replace each token in the input with a random one with the probability of $10\%$. Experimental Setup ::: Optimizer As in BIBREF15, we use the Adam optimizer BIBREF16, the parameters are $\beta _1 = 0{.}9$, $\beta _2 = 0{.}98$ and $\varepsilon = 10^{-9}$. We vary the learning rate over the course of training using the formula: where $warmup\_steps = 16000$ and $scale=4$. Results ::: General results The BLEU scores are provided in Table TABREF24 (we evaluate translations of 4-sentence fragments). To see which part of the improvement is due to fixing agreement between sentences rather than simply sentence-level post-editing, we train the same repair model at the sentence level. Each sentence in a group is now corrected separately, then they are put back together in a group. One can see that most of the improvement comes from accounting for extra-sentential dependencies. DocRepair outperforms the baseline and CADec by 0.7 BLEU, and its sentence-level repair version by 0.5 BLEU. Results ::: Consistency results Scores on the phenomena test sets are provided in Table TABREF26. For deixis, lexical cohesion and ellipsis (infl.) we see substantial improvements over both the baseline and CADec. The largest improvement over CADec (22.5 percentage points) is for lexical cohesion. However, there is a drop of almost 5 percentage points for VP ellipsis. We hypothesize that this is because it is hard to learn to correct inconsistencies in translations caused by VP ellipsis relying on monolingual data alone. Figure FIGREF27(a) shows an example of inconsistency caused by VP ellipsis in English. There is no VP ellipsis in Russian, and when translating auxiliary “did” the model has to guess the main verb. Figure FIGREF27(b) shows steps of generating round-trip translations for the target side of the previous example. When translating from Russian, main verbs are unlikely to be translated as the auxiliary “do” in English, and hence the VP ellipsis is rarely present on the English side. This implies the model trained using the round-trip translations will not be exposed to many VP ellipsis examples in training. We discuss this further in Section SECREF34. Table TABREF28 provides scores for deixis and lexical cohesion separately for different distances between sentences requiring consistency. It can be seen, that the performance of DocRepair degrades less than that of CADec when the distance between sentences requiring consistency gets larger. Results ::: Human evaluation We conduct a human evaluation on random 700 examples from our general test set. We picked only examples where a DocRepair translation is not a full copy of the baseline one. The annotators were provided an original group of sentences in English and two translations: baseline context-agnostic one and the one corrected by the DocRepair model. Translations were presented in random order with no indication which model they came from. The task is to pick one of the three options: (1) the first translation is better, (2) the second translation is better, (3) the translations are of equal quality. The annotators were asked to avoid the third answer if they are able to give preference to one of the translations. No other guidelines were given. The results are provided in Table TABREF30. In about $52\%$ of the cases annotators marked translations as having equal quality. Among the cases where one of the translations was marked better than the other, the DocRepair translation was marked better in $73\%$ of the cases. This shows a strong preference of the annotators for corrected translations over the baseline ones. Varying Training Data In this section, we discuss the influence of the training data chosen for document-level models. In all experiments, we used the DocRepair model. Varying Training Data ::: The amount of training data Table TABREF33 provides BLEU and consistency scores for the DocRepair model trained on different amount of data. We see that even when using a dataset of moderate size (e.g., 5m fragments) we can achieve performance comparable to the model trained on a large amount of data (30m fragments). Moreover, we notice that deixis scores are less sensitive to the amount of training data than lexical cohesion and ellipsis scores. The reason might be that, as we observed in our previous work BIBREF11, inconsistencies in translations due to the presence of deictic words and phrases are more frequent in this dataset than other types of inconsistencies. Also, as we show in Section SECREF7, this is the phenomenon the model learns faster in training. Varying Training Data ::: One-way vs round-trip translations In this section, we discuss the limitations of using only monolingual data to model inconsistencies between sentence-level translations. In Section SECREF25 we observed a drop in performance on VP ellipsis for DocRepair compared to CADec, which was trained on parallel data. We hypothesized that this is due to the differences between one-way and round-trip translations, and now we test this hypothesis. To do so, we fix the dataset and vary the way in which the input for DocRepair is generated: round-trip or one-way translations. The latter assumes that document-level data is parallel, and translations are sampled from the source side of the sentences in a group rather than from their back-translations. For parallel data, we take 1.5m parallel instances which were used for CADec training and add 1m instances from our monolingual data. For segments in the parallel part, we either sample translations from the source side or use round-trip translations. The results are provided in Table TABREF35. The model trained on one-way translations is slightly better than the one trained on round-trip translations. As expected, VP ellipsis is the hardest phenomena to be captured using round-trip translations, and the DocRepair model trained on one-way translated data gains 6% accuracy on this test set. This shows that the DocRepair model benefits from having access to non-synthetic English data. This results in exposing DocRepair at training time to Russian translations which suffer from the same inconsistencies as the ones it will have to correct at test time. Varying Training Data ::: Filtering: monolingual (no filtering) or parallel Note that the scores of the DocRepair model trained on 2.5m instances randomly chosen from monolingual data (Table TABREF33) are different from the ones for the model trained on 2.5m instances combined from parallel and monolingual data (Table TABREF35). For convenience, we show these two in Table TABREF36. The domain, the dataset these two data samples were gathered from, and the way we generated training data for DocRepair (round-trip translations) are all the same. The only difference lies in how the data was filtered. For parallel data, as in the previous work BIBREF6, we picked only sentence pairs with large relative time overlap of subtitle frames between source-language and target-language subtitles. This is necessary to ensure the quality of translation data: one needs groups of consecutive sentences in the target language where every sentence has a reliable translation. Table TABREF36 shows that the quality of the model trained on data which came from the parallel part is worse than the one trained on monolingual data. This indicates that requiring each sentence in a group to have a reliable translation changes the distribution of the data, which might be not beneficial for translation quality and provides extra motivation for using monolingual data. Learning Dynamics Let us now look into how the process of DocRepair training progresses. Figure FIGREF38 shows how the BLEU scores with the reference translation and with the baseline context-agnostic translation (i.e. the input for the DocRepair model) are changing during training. First, the model quickly learns to copy baseline translations: the BLEU score with the baseline is very high. Then it gradually learns to change them, which leads to an improvement in BLEU with the reference translation and a drop in BLEU with the baseline. Importantly, the model is reluctant to make changes: the BLEU score between translations of the converged model and the baseline is 82.5. We count the number of changed sentences in every 4-sentence fragment in the test set and plot the histogram in Figure FIGREF38. In over than 20$\%$ of the cases the model has not changed base translations at all. In almost $40\%$, it modified only one sentence and left the remaining 3 sentences unchanged. The model changed more than half sentences in a group in only $14\%$ of the cases. Several examples of the DocRepair translations are shown in Figure FIGREF43. Figure FIGREF42 shows how consistency scores are changing in training. For deixis, the model achieves the final quality quite quickly; for the rest, it needs a large number of training steps to converge. Related Work Our work is most closely related to two lines of research: automatic post-editing (APE) and document-level machine translation. Related Work ::: Automatic post-editing Our model can be regarded as an automatic post-editing system – a system designed to fix systematic MT errors that is decoupled from the main MT system. Automatic post-editing has a long history, including rule-based BIBREF17, statistical BIBREF18 and neural approaches BIBREF19, BIBREF20, BIBREF21. In terms of architectures, modern approaches use neural sequence-to-sequence models, either multi-source architectures that consider both the original source and the baseline translation BIBREF19, BIBREF20, or monolingual repair systems, as in BIBREF21, which is concurrent work to ours. True post-editing datasets are typically small and expensive to create BIBREF22, hence synthetic training data has been created that uses original monolingual data as output for the sequence-to-sequence model, paired with an automatic back-translation BIBREF23 and/or round-trip translation as its input(s) BIBREF19, BIBREF21. While previous work on automatic post-editing operated on the sentence level, the main novelty of this work is that our DocRepair model operates on groups of sentences and is thus able to fix consistency errors caused by the context-agnostic baseline MT system. We consider this strategy of sentence-level baseline translation and context-aware monolingual repair attractive when parallel document-level data is scarce. For training, the DocRepair model only requires monolingual document-level data. While we create synthetic training data via round-trip translation similarly to earlier work BIBREF19, BIBREF21, note that we purposefully use sentence-level MT systems for this to create the types of consistency errors that we aim to fix with the context-aware DocRepair model. Not all types of consistency errors that we want to fix emerge from a round-trip translation, so access to parallel document-level data can be useful (Section SECREF34). Related Work ::: Document-level NMT Neural models of MT that go beyond the sentence-level are an active research area BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF10, BIBREF9, BIBREF11. Typically, the main MT system is modified to take additional context as its input. One limitation of these approaches is that they assume that parallel document-level training data is available. Closest to our work are two-pass models for document-level NMT BIBREF24, BIBREF11, where a second, context-aware model takes the translation and hidden representations of the sentence-level first-pass model as its input. The second-pass model can in principle be trained on a subset of the parallel training data BIBREF11, somewhat relaxing the assumption that all training data is at the document level. Our work is different from this previous work in two main respects. Firstly, we show that consistency can be improved with only monolingual document-level training data. Secondly, the DocRepair model is decoupled from the first-pass MT system, which improves its portability. Conclusions We introduce the first approach to context-aware machine translation using only monolingual document-level data. We propose a monolingual DocRepair model to correct inconsistencies between sentence-level translations. The model performs automatic post-editing on a sequence of sentence-level translations, refining translations of sentences in context of each other. Our approach results in substantial improvements in translation quality as measured by BLEU, targeted contrastive evaluation of several discourse phenomena and human evaluation. Moreover, we perform error analysis and detect which discourse phenomena are hard to capture using only monolingual document-level data. While in the current work we used text fragments of 4 sentences, in future work we would like to consider longer contexts. Acknowledgments We would like to thank the anonymous reviewers for their comments. The authors also thank David Talbot and Yandex Machine Translation team for helpful discussions and inspiration. Ivan Titov acknowledges support of the European Research Council (ERC StG BroadSem 678254) and the Dutch National Science Foundation (NWO VIDI 639.022.518). Rico Sennrich acknowledges support from the Swiss National Science Foundation (105212_169888), the European Union’s Horizon 2020 research and innovation programme (grant agreement no 825460), and the Royal Society (NAF\R1\180122).
how many humans evaluated the results?
627b8d7b5b985394428c974aca5ba0c1bbbba377
how Ġmany Ġhumans Ġevaluated Ġthe Ġresults ?
Un answer able
<s> how Ġmany Ġhumans Ġevaluated Ġthe Ġresults ? </s> Introduction </s> Machine Ġtranslation Ġhas Ġmade Ġremarkable Ġprogress , Ġand Ġstudies Ġclaiming Ġit Ġto Ġreach Ġa Ġhuman Ġparity Ġare Ġstarting Ġto Ġappear ĠB IB REF 0 . ĠHowever , Ġwhen Ġevaluating Ġtranslations Ġof Ġthe Ġwhole Ġdocuments Ġrather Ġthan Ġisolated Ġsentences , Ġhuman Ġrat ers Ġshow Ġa Ġstronger Ġpreference Ġfor Ġhuman Ġover Ġmachine Ġtranslation ĠB IB REF 1 . ĠThese Ġfindings Ġemphasize Ġthe Ġneed Ġto Ġshift Ġtowards Ġcontext - aware Ġmachine Ġtranslation Ġboth Ġfrom Ġmodeling Ġand Ġevaluation Ġperspective . </s> Most Ġprevious Ġwork Ġon Ġcontext - aware ĠN MT Ġassumed Ġthat Ġeither Ġall Ġthe Ġbilingual Ġdata Ġis Ġavailable Ġat Ġthe Ġdocument Ġlevel ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 Ġor Ġat Ġleast Ġits Ġfraction ĠB IB REF 11 . ĠBut Ġin Ġpractical Ġscenarios , Ġdocument - level Ġparallel Ġdata Ġis Ġoften Ġscarce , Ġwhich Ġis Ġone Ġof Ġthe Ġchallenges Ġwhen Ġbuilding Ġa Ġcontext - aware Ġsystem . </s> We Ġintroduce Ġan Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠIn Ġour Ġsetting , Ġa Ġseparate Ġmon oling ual Ġsequence - to - sequence Ġmodel Ġ( Doc Rep air ) Ġis Ġused Ġto Ġcorrect Ġsentence - level Ġtranslations Ġof Ġadjacent Ġsentences . ĠThe Ġkey Ġidea Ġis Ġto Ġuse Ġmon oling ual Ġdata Ġto Ġimitate Ġtypical Ġinconsistencies Ġbetween Ġcontext - agn ostic Ġtranslations Ġof Ġisolated Ġsentences . ĠThe ĠDoc Rep air Ġmodel Ġis Ġtrained Ġto Ġmap Ġinconsistent Ġgroups Ġof Ġsentences Ġinto Ġconsistent Ġones . ĠThe Ġconsistent Ġgroups Ġcome Ġfrom Ġthe Ġoriginal Ġtraining Ġdata ; Ġthe Ġinconsistent Ġgroups Ġare Ġobtained Ġby Ġsampling Ġround - trip Ġtranslations Ġfor Ġeach Ġisolated Ġsentence . </s> To Ġvalidate Ġthe Ġperformance Ġof Ġour Ġmodel , Ġwe Ġuse Ġthree Ġkinds Ġof Ġevaluation : Ġthe ĠB LE U Ġscore , Ġcontrast ive Ġevaluation Ġof Ġtranslation Ġof Ġseveral Ġdiscourse Ġphenomena ĠB IB REF 11 , Ġand Ġhuman Ġevaluation . ĠWe Ġshow Ġstrong Ġimprovements Ġfor Ġall Ġmetrics . </s> We Ġanalyze Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġmon oling ual Ġdata Ġonly . ĠUsing Ġcontrast ive Ġtest Ġsets Ġfor Ġtargeted Ġevaluation Ġof Ġseveral Ġcontextual Ġphenomena , Ġwe Ġcompare Ġthe Ġperformance Ġof Ġthe Ġmodels Ġtrained Ġon Ġround - trip Ġtranslations Ġand Ġgenuine Ġdocument - level Ġparallel Ġdata . ĠAmong Ġthe Ġfour Ġphenomena Ġin Ġthe Ġtest Ġsets Ġwe Ġuse Ġ( de ix is , Ġlex ical Ġcohesion , ĠVP Ġell ips is Ġand Ġell ips is Ġwhich Ġaffects ĠNP Ġinf lection ) Ġwe Ġfind ĠVP Ġell ips is Ġto Ġbe Ġthe Ġhardest Ġphenomenon Ġto Ġbe Ġcaptured Ġusing Ġround - trip Ġtranslations . </s> Our Ġkey Ġcontributions Ġare Ġas Ġfollows : </s> we Ġintroduce Ġthe Ġfirst Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata ; </s> our Ġapproach Ġshows Ġsubstantial Ġimprovements Ġin Ġtranslation Ġquality Ġas Ġmeasured Ġby ĠB LE U , Ġtargeted Ġcontrast ive Ġevaluation Ġof Ġseveral Ġdiscourse Ġphenomena Ġand Ġhuman Ġevaluation ; </s> we Ġshow Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġmon oling ual Ġdata Ġonly . </s> Our ĠApproach : ĠDocument - level ĠRepair </s> We Ġpropose Ġa Ġmon oling ual ĠDoc Rep air Ġmodel Ġto Ġcorrect Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations Ġof Ġa Ġcontext - agn ostic ĠMT Ġsystem . ĠIt Ġdoes Ġnot Ġuse Ġany Ġstates Ġof Ġa Ġtrained ĠMT Ġmodel Ġwhose Ġoutputs Ġit Ġcorrect s Ġand Ġtherefore Ġcan Ġin Ġprinciple Ġbe Ġtrained Ġto Ġcorrect Ġtranslations Ġfrom Ġany Ġblack - box ĠMT Ġsystem . </s> The ĠDoc Rep air Ġmodel Ġrequires Ġonly Ġmon oling ual Ġdocument - level Ġdata Ġin Ġthe Ġtarget Ġlanguage . ĠIt Ġis Ġa Ġmon oling ual Ġsequence - to - sequence Ġmodel Ġthat Ġmaps Ġinconsistent Ġgroups Ġof Ġsentences Ġinto Ġconsistent Ġones . ĠCons istent Ġgroups Ġcome Ġfrom Ġmon oling ual Ġdocument - level Ġdata . ĠTo Ġobtain Ġinconsistent Ġgroups , Ġeach Ġsentence Ġin Ġa Ġgroup Ġis Ġreplaced Ġwith Ġits Ġround - trip Ġtranslation Ġproduced Ġin Ġisolation Ġfrom Ġcontext . ĠMore Ġformally , Ġforming Ġa Ġtraining Ġmin ib atch Ġfor Ġthe ĠDoc Rep air Ġmodel Ġinvolves Ġthe Ġfollowing Ġsteps Ġ( see Ġalso ĠFigure ĠFIG REF 9 ): </s> sample Ġseveral Ġgroups Ġof Ġsentences Ġfrom Ġthe Ġmon oling ual Ġdata ; </s> for Ġeach Ġsentence Ġin Ġa Ġgroup , Ġ( i ) Ġtranslate Ġit Ġusing Ġa Ġtarget - to - source ĠMT Ġmodel , Ġ( ii ) Ġsample Ġa Ġtranslation Ġof Ġthis Ġback - trans lated Ġsentence Ġin Ġthe Ġsource Ġlanguage Ġusing Ġa Ġsource - to - target ĠMT Ġmodel ; </s> using Ġthese Ġround - trip Ġtranslations Ġof Ġisolated Ġsentences , Ġform Ġan Ġinconsistent Ġversion Ġof Ġthe Ġinitial Ġgroups ; </s> use Ġinconsistent Ġgroups Ġas Ġinput Ġfor Ġthe ĠDoc Rep air Ġmodel , Ġconsistent Ġones Ġas Ġoutput . </s> At Ġtest Ġtime , Ġthe Ġprocess Ġof Ġgetting Ġdocument - level Ġtranslations Ġis Ġtwo - step Ġ( Figure ĠFIG REF 10 ): </s> produ ce Ġtranslations Ġof Ġisolated Ġsentences Ġusing Ġa Ġcontext - agn ostic ĠMT Ġmodel ; </s> apply Ġthe ĠDoc Rep air Ġmodel Ġto Ġa Ġsequence Ġof Ġcontext - agn ostic Ġtranslations Ġto Ġcorrect Ġinconsistencies Ġbetween Ġtranslations . </s> In Ġthe Ġscope Ġof Ġthe Ġcurrent Ġwork , Ġthe ĠDoc Rep air Ġmodel Ġis Ġthe Ġstandard Ġsequence - to - sequence ĠTrans former . ĠSent ences Ġin Ġa Ġgroup Ġare Ġconc aten ated Ġusing Ġa Ġreserved Ġtoken - separ ator Ġbetween Ġsentences . ĠThe ĠTrans former Ġis Ġtrained Ġto Ġcorrect Ġthese Ġlong Ġinconsistent Ġpseudo - sent ences Ġinto Ġconsistent Ġones . ĠThe Ġtoken - separ ator Ġis Ġthen Ġremoved Ġfrom Ġcorrected Ġtranslations . </s> E val uation Ġof ĠContext ual ĠPhen omen a </s> We Ġuse Ġcontrast ive Ġtest Ġsets Ġfor Ġevaluation Ġof Ġdiscourse Ġphenomena Ġfor ĠEnglish - Russian Ġby ĠB IB REF 11 . ĠThese Ġtest Ġsets Ġallow Ġfor Ġtesting Ġdifferent Ġkinds Ġof Ġphenomena Ġwhich , Ġas Ġwe Ġshow , Ġcan Ġbe Ġcaptured Ġfrom Ġmon oling ual Ġdata Ġwith Ġvarying Ġsuccess . ĠIn Ġthis Ġsection , Ġwe Ġprovide Ġtest Ġsets Ġstatistics Ġand Ġbriefly Ġdescribe Ġthe Ġtested Ġphenomena . ĠFor Ġmore Ġdetails , Ġthe Ġreader Ġis Ġreferred Ġto ĠB IB REF 11 . </s> E val uation Ġof ĠContext ual ĠPhen omen a Ġ: :: ĠTest Ġsets </s> There Ġare Ġfour Ġtest Ġsets Ġin Ġthe Ġsuite . ĠEach Ġtest Ġset Ġcontains Ġcontrast ive Ġexamples . ĠIt Ġis Ġspecifically Ġdesigned Ġto Ġtest Ġthe Ġability Ġof Ġa Ġsystem Ġto Ġadapt Ġto Ġcontextual Ġinformation Ġand Ġhandle Ġthe Ġphenomenon Ġunder Ġconsideration . ĠEach Ġtest Ġinstance Ġconsists Ġof Ġa Ġtrue Ġexample Ġ( a Ġsequence Ġof Ġsentences Ġand Ġtheir Ġreference Ġtranslation Ġfrom Ġthe Ġdata ) Ġand Ġseveral Ġcontrast ive Ġtranslations Ġwhich Ġdiffer Ġfrom Ġthe Ġtrue Ġone Ġonly Ġin Ġone Ġspecific Ġaspect . ĠAll Ġcontrast ive Ġtranslations Ġare Ġcorrect Ġand Ġplausible Ġtranslations Ġat Ġthe Ġsentence Ġlevel , Ġand Ġonly Ġcontext Ġreveals Ġthe Ġinconsistencies Ġbetween Ġthem . ĠThe Ġsystem Ġis Ġasked Ġto Ġscore Ġeach Ġcandidate Ġtranslation , Ġand Ġwe Ġcompute Ġthe Ġsystem Ġaccuracy Ġas Ġthe Ġproportion Ġof Ġtimes Ġthe Ġtrue Ġtranslation Ġis Ġpreferred Ġto Ġthe Ġcontrast ive Ġones . ĠTest Ġset Ġstatistics Ġare Ġshown Ġin ĠTable ĠT AB REF 15 . ĠThe Ġsuites Ġfor Ġde ix is Ġand Ġlex ical Ġcohesion Ġare Ġsplit Ġinto Ġdevelopment Ġand Ġtest Ġsets , Ġwith Ġ500 Ġexamples Ġfrom Ġeach Ġused Ġfor Ġvalidation Ġpurposes Ġand Ġthe Ġrest Ġfor Ġtesting . ĠConver gence Ġof Ġboth Ġconsistency Ġscores Ġon Ġthese Ġdevelopment Ġsets Ġand ĠB LE U Ġscore Ġon Ġa Ġgeneral Ġdevelopment Ġset Ġare Ġused Ġas Ġearly Ġstopping Ġcriteria Ġin Ġmodels Ġtraining . ĠFor Ġell ips is , Ġthere Ġis Ġno Ġdedicated Ġdevelopment Ġset , Ġso Ġwe Ġevaluate Ġon Ġall Ġthe Ġell ips is Ġdata Ġand Ġdo Ġnot Ġuse Ġit Ġfor Ġdevelopment . </s> E val uation Ġof ĠContext ual ĠPhen omen a Ġ: :: ĠPhen omen a Ġoverview </s> De ix is ĠDe ict ic Ġwords Ġor Ġphrases , Ġare Ġrefere nt ial Ġexpressions Ġwhose Ġden otation Ġdepends Ġon Ġcontext . ĠThis Ġincludes Ġpersonal Ġde ix is Ġ( âĢ ľ I âĢ Ŀ , ĠâĢ ľ you âĢ Ŀ ), Ġplace Ġde ix is Ġ( âĢ ľ here âĢ Ŀ , ĠâĢ ľ there âĢ Ŀ ), Ġand Ġdiscourse Ġde ix is , Ġwhere Ġparts Ġof Ġthe Ġdiscourse Ġare Ġreferenced Ġ( âĢ ľ that 's Ġa Ġgood Ġquestion âĢ Ŀ ). ĠThe Ġtest Ġset Ġexamples Ġare Ġall Ġrelated Ġto Ġperson Ġde ix is , Ġspecifically Ġthe ĠT - V Ġdistinction Ġbetween Ġinformal Ġand Ġformal Ġyou Ġ( Latin ĠâĢ ľ tu âĢ Ŀ Ġand ĠâĢ ľ v os âĢ Ŀ ) Ġin Ġthe ĠRussian Ġtranslations , Ġand Ġtest Ġfor Ġconsistency Ġin Ġthis Ġrespect . </s> Ell ips is ĠEll ips is Ġis Ġthe Ġomission Ġfrom Ġa Ġclause Ġof Ġone Ġor Ġmore Ġwords Ġthat Ġare Ġnevertheless Ġunderstood Ġin Ġthe Ġcontext Ġof Ġthe Ġremaining Ġelements . ĠIn Ġmachine Ġtranslation , Ġellipt ical Ġconstruct ions Ġin Ġthe Ġsource Ġlanguage Ġpose Ġa Ġproblem Ġin Ġtwo Ġsituations . ĠFirst , Ġif Ġthe Ġtarget Ġlanguage Ġdoes Ġnot Ġallow Ġthe Ġsame Ġtypes Ġof Ġell ips is , Ġrequiring Ġthe Ġel ided Ġmaterial Ġto Ġbe Ġpredicted Ġfrom Ġcontext . ĠSecond , Ġif Ġthe Ġel ided Ġmaterial Ġaffects Ġthe Ġsyntax Ġof Ġthe Ġsentence . ĠFor Ġexample , Ġin ĠRussian Ġthe Ġgram matical Ġfunction Ġof Ġa Ġnoun Ġphrase , Ġand Ġthus Ġits Ġinf lection , Ġmay Ġdepend Ġon Ġthe Ġel ided Ġverb , Ġor , Ġcon versely , Ġthe Ġverb Ġinf lection Ġmay Ġdepend Ġon Ġthe Ġel ided Ġsubject . </s> There Ġare Ġtwo Ġdifferent Ġtest Ġsets Ġfor Ġell ips is . ĠOne Ġcontains Ġexamples Ġwhere Ġa Ġmorph ological Ġform Ġof Ġa Ġnoun Ġgroup Ġin Ġthe Ġlast Ġsentence Ġcan Ġnot Ġbe Ġunderstood Ġwithout Ġcontext Ġbeyond Ġthe Ġsentence Ġlevel Ġ( âĢ ľ ell ips is Ġ( in fl .) âĢ Ŀ Ġin ĠTable ĠT AB REF 15 ). ĠAnother Ġincludes Ġcases Ġof Ġverb Ġphrase Ġell ips is Ġin ĠEnglish , Ġwhich Ġdoes Ġnot Ġexist Ġin ĠRussian , Ġthus Ġrequires Ġpredicting Ġthe Ġverb Ġwhen Ġtranslating Ġinto ĠRussian Ġ( âĢ ľ ell ips is Ġ( VP ) âĢ Ŀ Ġin ĠTable ĠT AB REF 15 ). </s> Lex ical Ġcohesion ĠThe Ġtest Ġset Ġfocuses Ġon Ġreiter ation Ġof Ġnamed Ġentities . ĠWhere Ġseveral Ġtranslations Ġof Ġa Ġnamed Ġentity Ġare Ġpossible , Ġa Ġmodel Ġhas Ġto Ġprefer Ġconsistent Ġtranslations Ġover Ġinconsistent Ġones . </s> Exper imental ĠSetup Ġ: :: ĠData Ġpre processing </s> We Ġuse Ġthe Ġpublicly Ġavailable ĠOpen Sub t itles 2018 Ġcorpus ĠB IB REF 12 Ġfor ĠEnglish Ġand ĠRussian . ĠFor Ġa Ġfair Ġcomparison Ġwith Ġprevious Ġwork , Ġwe Ġtrain Ġthe Ġbaseline ĠMT Ġsystem Ġon Ġthe Ġdata Ġreleased Ġby ĠB IB REF 11 . ĠName ly , Ġour ĠMT Ġsystem Ġis Ġtrained Ġon Ġ6 m Ġinstances . ĠThese Ġare Ġsentence Ġpairs Ġwith Ġa Ġrelative Ġtime Ġoverlap Ġof Ġsubtitle Ġframes Ġbetween Ġsource Ġand Ġtarget Ġlanguage Ġsubtitles Ġof Ġat Ġleast Ġ$ 0 . 9 $. </s> We Ġgathered Ġ30 m Ġgroups Ġof Ġ4 Ġconsecutive Ġsentences Ġas Ġour Ġmon oling ual Ġdata . ĠWe Ġused Ġonly Ġdocuments Ġnot Ġcontaining Ġgroups Ġof Ġsentences Ġfrom Ġgeneral Ġdevelopment Ġand Ġtest Ġsets Ġas Ġwell Ġas Ġfrom Ġcontrast ive Ġtest Ġsets . ĠThe Ġmain Ġresults Ġwe Ġreport Ġare Ġfor Ġthe Ġmodel Ġtrained Ġon Ġall Ġ30 m Ġfragments . </s> We Ġuse Ġthe Ġtoken ization Ġprovided Ġby Ġthe Ġcorpus Ġand Ġuse Ġmulti - ble u . per l Ġon Ġlower c ased Ġdata Ġto Ġcompute ĠB LE U Ġscore . ĠWe Ġuse Ġbeam Ġsearch Ġwith Ġa Ġbeam Ġof Ġ4 . </s> Sent ences Ġwere Ġencoded Ġusing Ġbyte - pair Ġencoding ĠB IB REF 13 , Ġwith Ġsource Ġand Ġtarget Ġvoc ab ular ies Ġof Ġabout Ġ32 000 Ġtokens . ĠTranslation Ġpairs Ġwere Ġbat ched Ġtogether Ġby Ġapproximate Ġsequence Ġlength . ĠEach Ġtraining Ġbatch Ġcontained Ġa Ġset Ġof Ġtranslation Ġpairs Ġcontaining Ġapproximately Ġ15 000 Ġsource Ġtokens . ĠIt Ġhas Ġbeen Ġshown Ġthat ĠTrans former 's Ġperformance Ġdepends Ġheavily Ġon Ġbatch Ġsize ĠB IB REF 14 , Ġand Ġwe Ġchose Ġa Ġlarge Ġbatch Ġsize Ġto Ġensure Ġthe Ġbest Ġperformance . ĠIn Ġtraining Ġcontext - aware Ġmodels , Ġfor Ġearly Ġstopping Ġwe Ġuse Ġboth Ġconvergence Ġin ĠB LE U Ġscore Ġon Ġthe Ġgeneral Ġdevelopment Ġset Ġand Ġscores Ġon Ġthe Ġconsistency Ġdevelopment Ġsets . ĠAfter Ġtraining , Ġwe Ġaverage Ġthe Ġ5 Ġlatest Ġcheckpoints . </s> Exper imental ĠSetup Ġ: :: ĠModels </s> The Ġbaseline Ġmodel , Ġthe Ġmodel Ġused Ġfor Ġback - translation , Ġand Ġthe ĠDoc Rep air Ġmodel Ġare Ġall ĠTrans former Ġbase Ġmodels ĠB IB REF 15 . ĠMore Ġprecisely , Ġthe Ġnumber Ġof Ġlayers Ġis Ġ$ N = 6 $ Ġwith Ġ$ h Ġ= Ġ8 $ Ġparallel Ġattention Ġlayers , Ġor Ġheads . ĠThe Ġdimension ality Ġof Ġinput Ġand Ġoutput Ġis Ġ$ d _{ model } Ġ= Ġ512 $, Ġand Ġthe Ġinner - layer Ġof Ġa Ġfeed - forward Ġnetworks Ġhas Ġdimension ality Ġ$ d _{ ff } = 20 48 $. ĠWe Ġuse Ġregular ization Ġas Ġdescribed Ġin ĠB IB REF 15 . </s> As Ġa Ġsecond Ġbaseline , Ġwe Ġuse Ġthe Ġtwo - pass ĠCAD ec Ġmodel ĠB IB REF 11 . ĠThe Ġfirst Ġpass Ġproduces Ġsentence - level Ġtranslations . ĠThe Ġsecond Ġpass Ġtakes Ġboth Ġthe Ġfirst - pass Ġtranslation Ġand Ġrepresentations Ġof Ġthe Ġcontext Ġsentences Ġas Ġinput Ġand Ġreturns Ġcontextual ized Ġtranslations . ĠCAD ec Ġrequires Ġdocument - level Ġparallel Ġtraining Ġdata , Ġwhile ĠDoc Rep air Ġonly Ġneeds Ġmon oling ual Ġtraining Ġdata . </s> Exper imental ĠSetup Ġ: :: ĠGener ating Ġround - trip Ġtranslations </s> On Ġthe Ġselected Ġ6 m Ġinstances Ġwe Ġtrain Ġsentence - level Ġtranslation Ġmodels Ġin Ġboth Ġdirections . ĠTo Ġcreate Ġtraining Ġdata Ġfor ĠDoc Rep air , Ġwe Ġproceed Ġas Ġfollows . ĠThe ĠRussian Ġmon oling ual Ġdata Ġis Ġfirst Ġtranslated Ġinto ĠEnglish , Ġusing Ġthe ĠRussian $ \ right arrow Ġ$ English Ġmodel Ġand Ġbeam Ġsearch Ġwith Ġbeam Ġsize Ġof Ġ4 . ĠThen , Ġwe Ġuse Ġthe ĠEnglish $ \ right arrow Ġ$ Russian Ġmodel Ġto Ġsample Ġtranslations Ġwith Ġtemperature Ġof Ġ$ 0 { .} 5 $. ĠFor Ġeach Ġsentence , Ġwe Ġpre comp ute Ġ20 Ġsampled Ġtranslations Ġand Ġrandomly Ġchoose Ġone Ġof Ġthem Ġwhen Ġforming Ġa Ġtraining Ġmin ib atch Ġfor ĠDoc Rep air . ĠAlso , Ġin Ġtraining , Ġwe Ġreplace Ġeach Ġtoken Ġin Ġthe Ġinput Ġwith Ġa Ġrandom Ġone Ġwith Ġthe Ġprobability Ġof Ġ$ 10 \ % $. </s> Exper imental ĠSetup Ġ: :: ĠOptim izer </s> As Ġin ĠB IB REF 15 , Ġwe Ġuse Ġthe ĠAdam Ġoptim izer ĠB IB REF 16 , Ġthe Ġparameters Ġare Ġ$\ beta Ġ_ 1 Ġ= Ġ0 { .} 9 $, Ġ$\ beta Ġ_ 2 Ġ= Ġ0 { .} 98 $ Ġand Ġ$\ v are ps ilon Ġ= Ġ10 ^{ - 9 } $. ĠWe Ġvary Ġthe Ġlearning Ġrate Ġover Ġthe Ġcourse Ġof Ġtraining Ġusing Ġthe Ġformula : </s> where Ġ$ warm up \ _ steps Ġ= Ġ16 000 $ Ġand Ġ$ scale = 4 $. </s> Results Ġ: :: ĠGeneral Ġresults </s> The ĠB LE U Ġscores Ġare Ġprovided Ġin ĠTable ĠT AB REF 24 Ġ( we Ġevaluate Ġtranslations Ġof Ġ4 - sent ence Ġfragments ). ĠTo Ġsee Ġwhich Ġpart Ġof Ġthe Ġimprovement Ġis Ġdue Ġto Ġfixing Ġagreement Ġbetween Ġsentences Ġrather Ġthan Ġsimply Ġsentence - level Ġpost - ed iting , Ġwe Ġtrain Ġthe Ġsame Ġrepair Ġmodel Ġat Ġthe Ġsentence Ġlevel . ĠEach Ġsentence Ġin Ġa Ġgroup Ġis Ġnow Ġcorrected Ġseparately , Ġthen Ġthey Ġare Ġput Ġback Ġtogether Ġin Ġa Ġgroup . ĠOne Ġcan Ġsee Ġthat Ġmost Ġof Ġthe Ġimprovement Ġcomes Ġfrom Ġaccounting Ġfor Ġextra - sent ential Ġdependencies . ĠDoc Rep air Ġoutper forms Ġthe Ġbaseline Ġand ĠCAD ec Ġby Ġ0 . 7 ĠB LE U , Ġand Ġits Ġsentence - level Ġrepair Ġversion Ġby Ġ0 . 5 ĠB LE U . </s> Results Ġ: :: ĠCons ist ency Ġresults </s> Sc ores Ġon Ġthe Ġphenomena Ġtest Ġsets Ġare Ġprovided Ġin ĠTable ĠT AB REF 26 . ĠFor Ġde ix is , Ġlex ical Ġcohesion Ġand Ġell ips is Ġ( in fl .) Ġwe Ġsee Ġsubstantial Ġimprovements Ġover Ġboth Ġthe Ġbaseline Ġand ĠCAD ec . ĠThe Ġlargest Ġimprovement Ġover ĠCAD ec Ġ( 22 . 5 Ġpercentage Ġpoints ) Ġis Ġfor Ġlex ical Ġcohesion . ĠHowever , Ġthere Ġis Ġa Ġdrop Ġof Ġalmost Ġ5 Ġpercentage Ġpoints Ġfor ĠVP Ġell ips is . ĠWe Ġhypothes ize Ġthat Ġthis Ġis Ġbecause Ġit Ġis Ġhard Ġto Ġlearn Ġto Ġcorrect Ġinconsistencies Ġin Ġtranslations Ġcaused Ġby ĠVP Ġell ips is Ġrelying Ġon Ġmon oling ual Ġdata Ġalone . ĠFigure ĠFIG REF 27 ( a ) Ġshows Ġan Ġexample Ġof Ġinconsistency Ġcaused Ġby ĠVP Ġell ips is Ġin ĠEnglish . ĠThere Ġis Ġno ĠVP Ġell ips is Ġin ĠRussian , Ġand Ġwhen Ġtranslating Ġauxiliary ĠâĢ ľ did âĢ Ŀ Ġthe Ġmodel Ġhas Ġto Ġguess Ġthe Ġmain Ġverb . ĠFigure ĠFIG REF 27 ( b ) Ġshows Ġsteps Ġof Ġgenerating Ġround - trip Ġtranslations Ġfor Ġthe Ġtarget Ġside Ġof Ġthe Ġprevious Ġexample . ĠWhen Ġtranslating Ġfrom ĠRussian , Ġmain Ġverbs Ġare Ġunlikely Ġto Ġbe Ġtranslated Ġas Ġthe Ġauxiliary ĠâĢ ľ do âĢ Ŀ Ġin ĠEnglish , Ġand Ġhence Ġthe ĠVP Ġell ips is Ġis Ġrarely Ġpresent Ġon Ġthe ĠEnglish Ġside . ĠThis Ġimplies Ġthe Ġmodel Ġtrained Ġusing Ġthe Ġround - trip Ġtranslations Ġwill Ġnot Ġbe Ġexposed Ġto Ġmany ĠVP Ġell ips is Ġexamples Ġin Ġtraining . ĠWe Ġdiscuss Ġthis Ġfurther Ġin ĠSection ĠSEC REF 34 . </s> Table ĠT AB REF 28 Ġprovides Ġscores Ġfor Ġde ix is Ġand Ġlex ical Ġcohesion Ġseparately Ġfor Ġdifferent Ġdistances Ġbetween Ġsentences Ġrequiring Ġconsistency . ĠIt Ġcan Ġbe Ġseen , Ġthat Ġthe Ġperformance Ġof ĠDoc Rep air Ġde grades Ġless Ġthan Ġthat Ġof ĠCAD ec Ġwhen Ġthe Ġdistance Ġbetween Ġsentences Ġrequiring Ġconsistency Ġgets Ġlarger . </s> Results Ġ: :: ĠHuman Ġevaluation </s> We Ġconduct Ġa Ġhuman Ġevaluation Ġon Ġrandom Ġ700 Ġexamples Ġfrom Ġour Ġgeneral Ġtest Ġset . ĠWe Ġpicked Ġonly Ġexamples Ġwhere Ġa ĠDoc Rep air Ġtranslation Ġis Ġnot Ġa Ġfull Ġcopy Ġof Ġthe Ġbaseline Ġone . </s> The Ġannot ators Ġwere Ġprovided Ġan Ġoriginal Ġgroup Ġof Ġsentences Ġin ĠEnglish Ġand Ġtwo Ġtranslations : Ġbaseline Ġcontext - agn ostic Ġone Ġand Ġthe Ġone Ġcorrected Ġby Ġthe ĠDoc Rep air Ġmodel . ĠTrans lations Ġwere Ġpresented Ġin Ġrandom Ġorder Ġwith Ġno Ġindication Ġwhich Ġmodel Ġthey Ġcame Ġfrom . ĠThe Ġtask Ġis Ġto Ġpick Ġone Ġof Ġthe Ġthree Ġoptions : Ġ( 1 ) Ġthe Ġfirst Ġtranslation Ġis Ġbetter , Ġ( 2 ) Ġthe Ġsecond Ġtranslation Ġis Ġbetter , Ġ( 3 ) Ġthe Ġtranslations Ġare Ġof Ġequal Ġquality . ĠThe Ġannot ators Ġwere Ġasked Ġto Ġavoid Ġthe Ġthird Ġanswer Ġif Ġthey Ġare Ġable Ġto Ġgive Ġpreference Ġto Ġone Ġof Ġthe Ġtranslations . ĠNo Ġother Ġguidelines Ġwere Ġgiven . </s> The Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 30 . ĠIn Ġabout Ġ$ 52 \ % $ Ġof Ġthe Ġcases Ġannot ators Ġmarked Ġtranslations Ġas Ġhaving Ġequal Ġquality . ĠAmong Ġthe Ġcases Ġwhere Ġone Ġof Ġthe Ġtranslations Ġwas Ġmarked Ġbetter Ġthan Ġthe Ġother , Ġthe ĠDoc Rep air Ġtranslation Ġwas Ġmarked Ġbetter Ġin Ġ$ 73 \ % $ Ġof Ġthe Ġcases . ĠThis Ġshows Ġa Ġstrong Ġpreference Ġof Ġthe Ġannot ators Ġfor Ġcorrected Ġtranslations Ġover Ġthe Ġbaseline Ġones . </s> V ary ing ĠTraining ĠData </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġinfluence Ġof Ġthe Ġtraining Ġdata Ġchosen Ġfor Ġdocument - level Ġmodels . ĠIn Ġall Ġexperiments , Ġwe Ġused Ġthe ĠDoc Rep air Ġmodel . </s> V ary ing ĠTraining ĠData Ġ: :: ĠThe Ġamount Ġof Ġtraining Ġdata </s> Table ĠT AB REF 33 Ġprovides ĠB LE U Ġand Ġconsistency Ġscores Ġfor Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġdifferent Ġamount Ġof Ġdata . ĠWe Ġsee Ġthat Ġeven Ġwhen Ġusing Ġa Ġdataset Ġof Ġmoderate Ġsize Ġ( e . g ., Ġ5 m Ġfragments ) Ġwe Ġcan Ġachieve Ġperformance Ġcomparable Ġto Ġthe Ġmodel Ġtrained Ġon Ġa Ġlarge Ġamount Ġof Ġdata Ġ( 30 m Ġfragments ). ĠMoreover , Ġwe Ġnotice Ġthat Ġde ix is Ġscores Ġare Ġless Ġsensitive Ġto Ġthe Ġamount Ġof Ġtraining Ġdata Ġthan Ġlex ical Ġcohesion Ġand Ġell ips is Ġscores . ĠThe Ġreason Ġmight Ġbe Ġthat , Ġas Ġwe Ġobserved Ġin Ġour Ġprevious Ġwork ĠB IB REF 11 , Ġinconsistencies Ġin Ġtranslations Ġdue Ġto Ġthe Ġpresence Ġof Ġde ict ic Ġwords Ġand Ġphrases Ġare Ġmore Ġfrequent Ġin Ġthis Ġdataset Ġthan Ġother Ġtypes Ġof Ġinconsistencies . ĠAlso , Ġas Ġwe Ġshow Ġin ĠSection ĠSEC REF 7 , Ġthis Ġis Ġthe Ġphenomenon Ġthe Ġmodel Ġlearns Ġfaster Ġin Ġtraining . </s> V ary ing ĠTraining ĠData Ġ: :: ĠOne - way Ġvs Ġround - trip Ġtranslations </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġlimitations Ġof Ġusing Ġonly Ġmon oling ual Ġdata Ġto Ġmodel Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations . ĠIn ĠSection ĠSEC REF 25 Ġwe Ġobserved Ġa Ġdrop Ġin Ġperformance Ġon ĠVP Ġell ips is Ġfor ĠDoc Rep air Ġcompared Ġto ĠCAD ec , Ġwhich Ġwas Ġtrained Ġon Ġparallel Ġdata . ĠWe Ġhypothesized Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġdifferences Ġbetween Ġone - way Ġand Ġround - trip Ġtranslations , Ġand Ġnow Ġwe Ġtest Ġthis Ġhypothesis . ĠTo Ġdo Ġso , Ġwe Ġfix Ġthe Ġdataset Ġand Ġvary Ġthe Ġway Ġin Ġwhich Ġthe Ġinput Ġfor ĠDoc Rep air Ġis Ġgenerated : Ġround - trip Ġor Ġone - way Ġtranslations . ĠThe Ġlatter Ġassumes Ġthat Ġdocument - level Ġdata Ġis Ġparallel , Ġand Ġtranslations Ġare Ġsampled Ġfrom Ġthe Ġsource Ġside Ġof Ġthe Ġsentences Ġin Ġa Ġgroup Ġrather Ġthan Ġfrom Ġtheir Ġback - trans lations . ĠFor Ġparallel Ġdata , Ġwe Ġtake Ġ1 . 5 m Ġparallel Ġinstances Ġwhich Ġwere Ġused Ġfor ĠCAD ec Ġtraining Ġand Ġadd Ġ1 m Ġinstances Ġfrom Ġour Ġmon oling ual Ġdata . ĠFor Ġsegments Ġin Ġthe Ġparallel Ġpart , Ġwe Ġeither Ġsample Ġtranslations Ġfrom Ġthe Ġsource Ġside Ġor Ġuse Ġround - trip Ġtranslations . ĠThe Ġresults Ġare Ġprovided Ġin ĠTable ĠT AB REF 35 . </s> The Ġmodel Ġtrained Ġon Ġone - way Ġtranslations Ġis Ġslightly Ġbetter Ġthan Ġthe Ġone Ġtrained Ġon Ġround - trip Ġtranslations . ĠAs Ġexpected , ĠVP Ġell ips is Ġis Ġthe Ġhardest Ġphenomena Ġto Ġbe Ġcaptured Ġusing Ġround - trip Ġtranslations , Ġand Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġone - way Ġtranslated Ġdata Ġgains Ġ6 % Ġaccuracy Ġon Ġthis Ġtest Ġset . ĠThis Ġshows Ġthat Ġthe ĠDoc Rep air Ġmodel Ġbenefits Ġfrom Ġhaving Ġaccess Ġto Ġnon - sy nt hetic ĠEnglish Ġdata . ĠThis Ġresults Ġin Ġexposing ĠDoc Rep air Ġat Ġtraining Ġtime Ġto ĠRussian Ġtranslations Ġwhich Ġsuffer Ġfrom Ġthe Ġsame Ġinconsistencies Ġas Ġthe Ġones Ġit Ġwill Ġhave Ġto Ġcorrect Ġat Ġtest Ġtime . </s> V ary ing ĠTraining ĠData Ġ: :: ĠFil tering : Ġmon oling ual Ġ( no Ġfiltering ) Ġor Ġparallel </s> Note Ġthat Ġthe Ġscores Ġof Ġthe ĠDoc Rep air Ġmodel Ġtrained Ġon Ġ2 . 5 m Ġinstances Ġrandomly Ġchosen Ġfrom Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 33 ) Ġare Ġdifferent Ġfrom Ġthe Ġones Ġfor Ġthe Ġmodel Ġtrained Ġon Ġ2 . 5 m Ġinstances Ġcombined Ġfrom Ġparallel Ġand Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 35 ). ĠFor Ġconvenience , Ġwe Ġshow Ġthese Ġtwo Ġin ĠTable ĠT AB REF 36 . </s> The Ġdomain , Ġthe Ġdataset Ġthese Ġtwo Ġdata Ġsamples Ġwere Ġgathered Ġfrom , Ġand Ġthe Ġway Ġwe Ġgenerated Ġtraining Ġdata Ġfor ĠDoc Rep air Ġ( round - trip Ġtranslations ) Ġare Ġall Ġthe Ġsame . ĠThe Ġonly Ġdifference Ġlies Ġin Ġhow Ġthe Ġdata Ġwas Ġfiltered . ĠFor Ġparallel Ġdata , Ġas Ġin Ġthe Ġprevious Ġwork ĠB IB REF 6 , Ġwe Ġpicked Ġonly Ġsentence Ġpairs Ġwith Ġlarge Ġrelative Ġtime Ġoverlap Ġof Ġsubtitle Ġframes Ġbetween Ġsource - language Ġand Ġtarget - language Ġsubtitles . ĠThis Ġis Ġnecessary Ġto Ġensure Ġthe Ġquality Ġof Ġtranslation Ġdata : Ġone Ġneeds Ġgroups Ġof Ġconsecutive Ġsentences Ġin Ġthe Ġtarget Ġlanguage Ġwhere Ġevery Ġsentence Ġhas Ġa Ġreliable Ġtranslation . </s> Table ĠT AB REF 36 Ġshows Ġthat Ġthe Ġquality Ġof Ġthe Ġmodel Ġtrained Ġon Ġdata Ġwhich Ġcame Ġfrom Ġthe Ġparallel Ġpart Ġis Ġworse Ġthan Ġthe Ġone Ġtrained Ġon Ġmon oling ual Ġdata . ĠThis Ġindicates Ġthat Ġrequiring Ġeach Ġsentence Ġin Ġa Ġgroup Ġto Ġhave Ġa Ġreliable Ġtranslation Ġchanges Ġthe Ġdistribution Ġof Ġthe Ġdata , Ġwhich Ġmight Ġbe Ġnot Ġbeneficial Ġfor Ġtranslation Ġquality Ġand Ġprovides Ġextra Ġmotivation Ġfor Ġusing Ġmon oling ual Ġdata . </s> Learning ĠDynamics </s> Let Ġus Ġnow Ġlook Ġinto Ġhow Ġthe Ġprocess Ġof ĠDoc Rep air Ġtraining Ġprogresses . ĠFigure ĠFIG REF 38 Ġshows Ġhow Ġthe ĠB LE U Ġscores Ġwith Ġthe Ġreference Ġtranslation Ġand Ġwith Ġthe Ġbaseline Ġcontext - agn ostic Ġtranslation Ġ( i . e . Ġthe Ġinput Ġfor Ġthe ĠDoc Rep air Ġmodel ) Ġare Ġchanging Ġduring Ġtraining . ĠFirst , Ġthe Ġmodel Ġquickly Ġlearns Ġto Ġcopy Ġbaseline Ġtranslations : Ġthe ĠB LE U Ġscore Ġwith Ġthe Ġbaseline Ġis Ġvery Ġhigh . ĠThen Ġit Ġgradually Ġlearns Ġto Ġchange Ġthem , Ġwhich Ġleads Ġto Ġan Ġimprovement Ġin ĠB LE U Ġwith Ġthe Ġreference Ġtranslation Ġand Ġa Ġdrop Ġin ĠB LE U Ġwith Ġthe Ġbaseline . ĠImport antly , Ġthe Ġmodel Ġis Ġreluctant Ġto Ġmake Ġchanges : Ġthe ĠB LE U Ġscore Ġbetween Ġtranslations Ġof Ġthe Ġconver ged Ġmodel Ġand Ġthe Ġbaseline Ġis Ġ82 . 5 . ĠWe Ġcount Ġthe Ġnumber Ġof Ġchanged Ġsentences Ġin Ġevery Ġ4 - sent ence Ġfragment Ġin Ġthe Ġtest Ġset Ġand Ġplot Ġthe Ġhist ogram Ġin ĠFigure ĠFIG REF 38 . ĠIn Ġover Ġthan Ġ20 $ \ % $ Ġof Ġthe Ġcases Ġthe Ġmodel Ġhas Ġnot Ġchanged Ġbase Ġtranslations Ġat Ġall . ĠIn Ġalmost Ġ$ 40 \ % $, Ġit Ġmodified Ġonly Ġone Ġsentence Ġand Ġleft Ġthe Ġremaining Ġ3 Ġsentences Ġunchanged . ĠThe Ġmodel Ġchanged Ġmore Ġthan Ġhalf Ġsentences Ġin Ġa Ġgroup Ġin Ġonly Ġ$ 14 \ % $ Ġof Ġthe Ġcases . ĠSeveral Ġexamples Ġof Ġthe ĠDoc Rep air Ġtranslations Ġare Ġshown Ġin ĠFigure ĠFIG REF 43 . </s> Figure ĠFIG REF 42 Ġshows Ġhow Ġconsistency Ġscores Ġare Ġchanging Ġin Ġtraining . ĠFor Ġde ix is , Ġthe Ġmodel Ġachieves Ġthe Ġfinal Ġquality Ġquite Ġquickly ; Ġfor Ġthe Ġrest , Ġit Ġneeds Ġa Ġlarge Ġnumber Ġof Ġtraining Ġsteps Ġto Ġconverge . </s> Related ĠWork </s> Our Ġwork Ġis Ġmost Ġclosely Ġrelated Ġto Ġtwo Ġlines Ġof Ġresearch : Ġautomatic Ġpost - ed iting Ġ( APE ) Ġand Ġdocument - level Ġmachine Ġtranslation . </s> Related ĠWork Ġ: :: ĠAutomatic Ġpost - ed iting </s> Our Ġmodel Ġcan Ġbe Ġregarded Ġas Ġan Ġautomatic Ġpost - ed iting Ġsystem ĠâĢĵ Ġa Ġsystem Ġdesigned Ġto Ġfix Ġsystematic ĠMT Ġerrors Ġthat Ġis Ġdec ou pled Ġfrom Ġthe Ġmain ĠMT Ġsystem . ĠAutomatic Ġpost - ed iting Ġhas Ġa Ġlong Ġhistory , Ġincluding Ġrule - based ĠB IB REF 17 , Ġstatistical ĠB IB REF 18 Ġand Ġneural Ġapproaches ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 . </s> In Ġterms Ġof Ġarchitectures , Ġmodern Ġapproaches Ġuse Ġneural Ġsequence - to - sequence Ġmodels , Ġeither Ġmulti - source Ġarchitectures Ġthat Ġconsider Ġboth Ġthe Ġoriginal Ġsource Ġand Ġthe Ġbaseline Ġtranslation ĠB IB REF 19 , ĠB IB REF 20 , Ġor Ġmon oling ual Ġrepair Ġsystems , Ġas Ġin ĠB IB REF 21 , Ġwhich Ġis Ġconcurrent Ġwork Ġto Ġours . ĠTrue Ġpost - ed iting Ġdatasets Ġare Ġtypically Ġsmall Ġand Ġexpensive Ġto Ġcreate ĠB IB REF 22 , Ġhence Ġsynthetic Ġtraining Ġdata Ġhas Ġbeen Ġcreated Ġthat Ġuses Ġoriginal Ġmon oling ual Ġdata Ġas Ġoutput Ġfor Ġthe Ġsequence - to - sequence Ġmodel , Ġpaired Ġwith Ġan Ġautomatic Ġback - translation ĠB IB REF 23 Ġand / or Ġround - trip Ġtranslation Ġas Ġits Ġinput ( s ) ĠB IB REF 19 , ĠB IB REF 21 . </s> While Ġprevious Ġwork Ġon Ġautomatic Ġpost - ed iting Ġoperated Ġon Ġthe Ġsentence Ġlevel , Ġthe Ġmain Ġnovelty Ġof Ġthis Ġwork Ġis Ġthat Ġour ĠDoc Rep air Ġmodel Ġoperates Ġon Ġgroups Ġof Ġsentences Ġand Ġis Ġthus Ġable Ġto Ġfix Ġconsistency Ġerrors Ġcaused Ġby Ġthe Ġcontext - agn ostic Ġbaseline ĠMT Ġsystem . ĠWe Ġconsider Ġthis Ġstrategy Ġof Ġsentence - level Ġbaseline Ġtranslation Ġand Ġcontext - aware Ġmon oling ual Ġrepair Ġattractive Ġwhen Ġparallel Ġdocument - level Ġdata Ġis Ġscarce . </s> For Ġtraining , Ġthe ĠDoc Rep air Ġmodel Ġonly Ġrequires Ġmon oling ual Ġdocument - level Ġdata . ĠWhile Ġwe Ġcreate Ġsynthetic Ġtraining Ġdata Ġvia Ġround - trip Ġtranslation Ġsimilarly Ġto Ġearlier Ġwork ĠB IB REF 19 , ĠB IB REF 21 , Ġnote Ġthat Ġwe Ġpurposefully Ġuse Ġsentence - level ĠMT Ġsystems Ġfor Ġthis Ġto Ġcreate Ġthe Ġtypes Ġof Ġconsistency Ġerrors Ġthat Ġwe Ġaim Ġto Ġfix Ġwith Ġthe Ġcontext - aware ĠDoc Rep air Ġmodel . ĠNot Ġall Ġtypes Ġof Ġconsistency Ġerrors Ġthat Ġwe Ġwant Ġto Ġfix Ġemerge Ġfrom Ġa Ġround - trip Ġtranslation , Ġso Ġaccess Ġto Ġparallel Ġdocument - level Ġdata Ġcan Ġbe Ġuseful Ġ( Section ĠSEC REF 34 ). </s> Related ĠWork Ġ: :: ĠDocument - level ĠN MT </s> Ne ural Ġmodels Ġof ĠMT Ġthat Ġgo Ġbeyond Ġthe Ġsentence - level Ġare Ġan Ġactive Ġresearch Ġarea ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 10 , ĠB IB REF 9 , ĠB IB REF 11 . ĠTypically , Ġthe Ġmain ĠMT Ġsystem Ġis Ġmodified Ġto Ġtake Ġadditional Ġcontext Ġas Ġits Ġinput . ĠOne Ġlimitation Ġof Ġthese Ġapproaches Ġis Ġthat Ġthey Ġassume Ġthat Ġparallel Ġdocument - level Ġtraining Ġdata Ġis Ġavailable . </s> Cl os est Ġto Ġour Ġwork Ġare Ġtwo - pass Ġmodels Ġfor Ġdocument - level ĠN MT ĠB IB REF 24 , ĠB IB REF 11 , Ġwhere Ġa Ġsecond , Ġcontext - aware Ġmodel Ġtakes Ġthe Ġtranslation Ġand Ġhidden Ġrepresentations Ġof Ġthe Ġsentence - level Ġfirst - pass Ġmodel Ġas Ġits Ġinput . ĠThe Ġsecond - pass Ġmodel Ġcan Ġin Ġprinciple Ġbe Ġtrained Ġon Ġa Ġsubset Ġof Ġthe Ġparallel Ġtraining Ġdata ĠB IB REF 11 , Ġsomewhat Ġrelaxing Ġthe Ġassumption Ġthat Ġall Ġtraining Ġdata Ġis Ġat Ġthe Ġdocument Ġlevel . </s> Our Ġwork Ġis Ġdifferent Ġfrom Ġthis Ġprevious Ġwork Ġin Ġtwo Ġmain Ġrespects . ĠFirstly , Ġwe Ġshow Ġthat Ġconsistency Ġcan Ġbe Ġimproved Ġwith Ġonly Ġmon oling ual Ġdocument - level Ġtraining Ġdata . ĠSecondly , Ġthe ĠDoc Rep air Ġmodel Ġis Ġdec ou pled Ġfrom Ġthe Ġfirst - pass ĠMT Ġsystem , Ġwhich Ġimproves Ġits Ġport ability . </s> Con clusions </s> We Ġintroduce Ġthe Ġfirst Ġapproach Ġto Ġcontext - aware Ġmachine Ġtranslation Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠWe Ġpropose Ġa Ġmon oling ual ĠDoc Rep air Ġmodel Ġto Ġcorrect Ġinconsistencies Ġbetween Ġsentence - level Ġtranslations . ĠThe Ġmodel Ġperforms Ġautomatic Ġpost - ed iting Ġon Ġa Ġsequence Ġof Ġsentence - level Ġtranslations , Ġrefining Ġtranslations Ġof Ġsentences Ġin Ġcontext Ġof Ġeach Ġother . ĠOur Ġapproach Ġresults Ġin Ġsubstantial Ġimprovements Ġin Ġtranslation Ġquality Ġas Ġmeasured Ġby ĠB LE U , Ġtargeted Ġcontrast ive Ġevaluation Ġof Ġseveral Ġdiscourse Ġphenomena Ġand Ġhuman Ġevaluation . ĠMoreover , Ġwe Ġperform Ġerror Ġanalysis Ġand Ġdetect Ġwhich Ġdiscourse Ġphenomena Ġare Ġhard Ġto Ġcapture Ġusing Ġonly Ġmon oling ual Ġdocument - level Ġdata . ĠWhile Ġin Ġthe Ġcurrent Ġwork Ġwe Ġused Ġtext Ġfragments Ġof Ġ4 Ġsentences , Ġin Ġfuture Ġwork Ġwe Ġwould Ġlike Ġto Ġconsider Ġlonger Ġcontexts . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcomments . ĠThe Ġauthors Ġalso Ġthank ĠDavid ĠTal bot Ġand ĠY and ex ĠMachine ĠTranslation Ġteam Ġfor Ġhelpful Ġdiscussions Ġand Ġinspiration . ĠIvan ĠTit ov Ġacknowledges Ġsupport Ġof Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ĠSt G ĠBroad Sem Ġ6 78 254 ) Ġand Ġthe ĠDutch ĠNational ĠScience ĠFoundation Ġ( NW O ĠVID I Ġ6 39 . 022 . 518 ). ĠRico ĠS enn rich Ġacknowledges Ġsupport Ġfrom Ġthe ĠSwiss ĠNational ĠScience ĠFoundation Ġ( 10 52 12 _ 169 888 ), Ġthe ĠEuropean ĠUnion âĢ Ļ s ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġ( gr ant Ġagreement Ġno Ġ8 254 60 ), Ġand Ġthe ĠRoyal ĠSociety Ġ( NA F \ R 1 \ 18 01 22 ).