grammatical, spelling and word order errors
EXTRACTIVE
spelling, word order and grammatical errors
EXTRACTIVE
To conclude this section, our model reliable corrects grammatical, spelling and word order errors on , with more mixed performance on lexical choice errors and some unnecessary paraphrasing of the input. The error types that the model manages well can be traced back to having a strong monolingual language model, a common trait of a good NMT model. As the model operates on the level of word parts and its vocabulary is limited, this leads to combining wrong word parts, sometimes across languages. This could be fixed by either using character-based NMT or doing automatic spelling correction prior to applying our current model.
We showed that for GEC our approach reliably corrects spelling, word order and grammatical errors, while being less reliable on lexical choice errors. Applied to style transfer our model is very good at meaning preservation and output fluency, while reliably transferring style for English contractions, lexical choice and grammatical constructions. The main benefit is that no annotated data is used to train the model, thus making it very easy to train it for other (especially under-resourced) languages.
1903.11283
Introduction </s> Sequ ence - to - sequence Ġ( seq 2 seq ) Ġtransformations Ġhave Ġrecently Ġproven Ġto Ġbe Ġa Ġsuccessful Ġframework Ġfor Ġseveral Ġnatural Ġlanguage Ġprocessing Ġtasks , Ġlike : Ġmachine Ġtranslation Ġ( MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġspeech Ġrecognition ĠB IB REF 2 Ġ, Ġspeech Ġsynthesis ĠB IB REF 3 Ġ, Ġnatural Ġlanguage Ġinference ĠB IB REF 4 Ġand Ġothers . ĠHowever , Ġthe Ġsuccess Ġof Ġthese Ġmodels Ġdepends Ġon Ġthe Ġavailability Ġof Ġlarge Ġamounts Ġof Ġdirectly Ġannot ated Ġdata Ġfor Ġthe Ġtask Ġat Ġhand Ġ( like Ġtranslation Ġexamples , Ġtext Ġsegments Ġand Ġtheir Ġspeech Ġrecordings , Ġetc .). ĠThis Ġis Ġa Ġsevere Ġlimitation Ġfor Ġtasks Ġwhere Ġdata Ġis Ġnot Ġabundantly Ġavailable Ġas Ġwell Ġas Ġfor Ġlow - resource Ġlanguages . </s> Here Ġwe Ġfocus Ġon Ġtwo Ġsuch Ġtasks : Ġgram matical Ġerror Ġcorrection Ġ( G EC ) Ġand Ġstyle Ġtransfer . ĠModern Ġapproaches Ġto ĠG EC Ġlearn Ġfrom Ġparallel Ġcorpor a Ġof Ġerroneous Ġsegments Ġand Ġtheir Ġmanual Ġcorrections ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ; Ġtext Ġstyle Ġtransfer Ġalso Ġrelies Ġon Ġsupervised Ġapproaches Ġthat Ġrequire Ġtexts Ġof Ġthe Ġsame Ġmeaning Ġand Ġdifferent Ġstyles ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġor Ġimp rec ise Ġun super vised Ġmethods ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> In Ġthis Ġpaper Ġwe Ġintroduce Ġan Ġapproach Ġto Ġperforming Ġboth ĠG EC Ġand Ġstyle Ġtransfer Ġwith Ġthe Ġsame Ġtrained Ġmodel , Ġwhile Ġnot Ġusing Ġany Ġsupervised Ġtraining Ġdata Ġfor Ġeither Ġtask . ĠIt Ġis Ġbased Ġon Ġzero - shot Ġneural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 11 Ġ, Ġand Ġas Ġsuch , Ġthe Ġonly Ġkind Ġof Ġdata Ġit Ġuses Ġis Ġregular Ġparallel Ġcorpor a Ġ( with Ġtexts Ġand Ġtheir Ġtranslations ). ĠHowever , Ġwe Ġapply Ġthe Ġmodel Ġto Ġdo Ġmon oling ual Ġtransfer , Ġasking Ġto Ġtranslate Ġthe Ġinput Ġsegment Ġinto Ġthe Ġsame Ġlanguage . ĠWe Ġshow , Ġthat Ġthis ĠâĢ ľ mon oling ual Ġtranslation âĢ Ŀ Ġis Ġwhat Ġenables Ġthe Ġmodel Ġto Ġcorrect Ġthe Ġerrors Ġin Ġthe Ġinput Ġas Ġwell Ġas Ġadapt Ġthe Ġoutput Ġinto Ġa Ġdesired Ġstyle . ĠMoreover , Ġthe Ġsame Ġtrained Ġmodel Ġperforms Ġboth Ġtasks Ġon Ġseveral Ġlanguages . </s> Our Ġmain Ġcontributions Ġare Ġthus : Ġ( i ) Ġa Ġsingle Ġmethod Ġfor Ġboth Ġstyle Ġtransfer Ġand Ġgram matical Ġerror Ġcorrection , Ġwithout Ġusing Ġannot ated Ġdata Ġfor Ġeither Ġtask , Ġ( ii ) Ġsupport Ġfor Ġboth Ġtasks Ġon Ġmultiple Ġlanguages Ġwithin Ġthe Ġsame Ġmodel , Ġ( iii ) Ġa Ġthorough Ġquantitative Ġand Ġqualitative Ġmanual Ġevaluation Ġof Ġthe Ġmodel Ġon Ġboth Ġtasks , Ġand Ġ( iv ) Ġhighlighting Ġof Ġthe Ġmodel 's Ġreliability Ġaspects Ġon Ġboth Ġtasks . ĠWe Ġused Ġpublicly Ġavailable Ġsoftware Ġand Ġdata ; Ġan Ġonline Ġdemo Ġof Ġour Ġresults Ġis Ġavailable , Ġbut Ġconcealed Ġfor Ġanonym ization Ġpurposes . </s> We Ġdescribe Ġthe Ġdetails Ġof Ġour Ġapproach Ġin ĠSection ĠSEC REF 2 Ġ, Ġthen Ġevaluate Ġit Ġin Ġterms Ġof Ġperformance Ġin Ġgram matical Ġerror Ġcorrection Ġin ĠSection ĠSEC REF 3 Ġand Ġin Ġstyle Ġtransfer Ġin ĠSection ĠSEC REF 4 Ġ. ĠThe Ġpaper Ġends Ġwith Ġa Ġreview Ġof Ġrelated Ġwork Ġin ĠSection ĠSEC REF 5 Ġand Ġconclusions Ġin ĠSection ĠSEC REF 6 Ġ. </s> Method </s> As Ġmentioned Ġin Ġthe Ġintroduction , Ġour Ġapproach Ġis Ġbased Ġon Ġthe Ġidea Ġof Ġzero - shot ĠMT ĠB IB REF 11 Ġ. ĠThere Ġthe Ġauthors Ġshow Ġthat Ġafter Ġtraining Ġa Ġsingle Ġmodel Ġto Ġtranslate Ġfrom ĠPortuguese Ġto ĠEnglish Ġas Ġwell Ġas Ġfrom ĠEnglish Ġto ĠSpanish , Ġit Ġcan Ġalso Ġtranslate ĠPortuguese Ġinto ĠSpanish , Ġwithout Ġseeing Ġany Ġtranslation Ġexamples Ġfor Ġthis Ġlanguage Ġpair . ĠWe Ġuse Ġthe Ġzero - shot Ġeffect Ġto Ġachieve Ġmon oling ual Ġtranslation Ġby Ġtraining Ġthe Ġmodel Ġon Ġbilingual Ġexamples Ġin Ġboth Ġdirections , Ġand Ġthen Ġdoing Ġtranslation Ġinto Ġthe Ġsame Ġlanguage Ġas Ġthe Ġinput : Ġillustrated Ġon ĠFigure ĠFIG REF 1 Ġ. </s> With Ġregular Ġsentences Ġmon oling ual Ġtranslation Ġdoes Ġnot Ġseem Ġuseful , Ġas Ġits Ġbehaviour Ġmainly Ġconsists Ġof Ġcopying . ĠHowever , Ġwhen Ġthe Ġinput Ġsentence Ġhas Ġcharacteristics Ġunseen Ġor Ġrarely Ġseen Ġby Ġthe Ġmodel Ġat Ġtraining Ġtime Ġ( like Ġgram matical Ġerrors Ġor Ġdifferent Ġstyl istic Ġchoices ) ĠâĢĵ Ġthe Ġdec oder Ġstill Ġgenerates Ġthe Ġmore Ġregular Ġversion Ġof Ġthe Ġsentence Ġ( thus Ġfixing Ġthe Ġerrors Ġor Ġadapting Ġthe Ġstyle ). ĠFurthermore , Ġin Ġcase Ġof Ġmult ilingual Ġmulti - domain ĠN MT ĠB IB REF 12 Ġ, Ġit Ġis Ġpossible Ġto Ġswitch Ġbetween Ġdifferent Ġdomains Ġor Ġstyles Ġat Ġruntime , Ġthus Ġperforming ĠâĢ ľ mon oling ual Ġdomain Ġadaptation âĢ Ŀ Ġor Ġstyle Ġtransfer . </s> To Ġcreate Ġa Ġmult ilingual Ġmulti - domain ĠN MT Ġsystem Ġwe Ġuse Ġthe Ġself - att ention Ġarchitecture ĠB IB REF 13 Ġ. ĠInstead Ġof Ġspecifying Ġthe Ġoutput Ġlanguage Ġwith Ġa Ġtoken Ġinside Ġthe Ġinput Ġsequence , Ġas ĠB IB REF 11 Ġdid , Ġwe Ġfollow ĠB IB REF 12 Ġand Ġuse Ġword Ġfeatures Ġ( or Ġfactors ). ĠOn Ġone Ġhand , Ġthis Ġprovides Ġa Ġstronger Ġsignal Ġfor Ġthe Ġmodel , Ġand Ġon Ġthe Ġother ĠâĢĵ Ġallows Ġfor Ġadditional Ġparam et riz ation , Ġwhich Ġin Ġour Ġcase Ġis Ġthe Ġtext Ġdomain / style Ġof Ġthe Ġcorpus . </s> As Ġa Ġresult , Ġa Ġpre - process ed ĠEnglish - Lat v ian Ġtraining Ġset Ġsentence Ġpair ĠâĢ ľ Hello ! âĢ Ŀ âĢĵ âĢ ľ S ve iki ! âĢ Ŀ Ġlooks Ġlike : </s> Here Ġ2 lv Ġand Ġ2 os Ġspecify ĠLat v ian Ġand ĠOpen Sub t itles Ġas Ġthe Ġoutput Ġlanguage Ġand Ġdomain ; Ġthe Ġoutput Ġtext Ġhas Ġno Ġfactors Ġto Ġpredict . ĠAt Ġapplication Ġtime Ġwe Ġsimply Ġuse Ġthe Ġsame Ġinput Ġand Ġoutput Ġlanguages , Ġfor Ġexample Ġthe Ġgram matically Ġincorrect Ġinput ĠâĢ ľ we Ġis âĢ Ŀ Ġlooks Ġlike Ġthe Ġfollowing , Ġafter Ġpre - processing : </s> The Ġintuition Ġbehind Ġour Ġapproach Ġis Ġthat Ġa Ġmult ilingual Ġshared Ġenc oder Ġproduces Ġsem antically Ġrich Ġlatent Ġsentence Ġrepresentations ĠB IB REF 14 Ġ, Ġwhich Ġprovide Ġa Ġsolid Ġground Ġfor Ġthe Ġeffective Ġstyle Ġtransfer Ġon Ġtop . </s> Next Ġwe Ġpresent Ġthe Ġtechnical Ġdetails , Ġthe Ġexperiment Ġsetup Ġand Ġthe Ġdata Ġwe Ġused Ġfor Ġtraining Ġthe Ġmodel Ġused Ġin Ġthe Ġexperiments . </s> L anguages Ġand ĠData </s> We Ġuse Ġthree Ġlanguages Ġin Ġour Ġexperiments : ĠEnglish , ĠEston ian Ġand ĠLat v ian . ĠAll Ġthree Ġhave Ġdifferent Ġcharacteristics , Ġfor Ġexample ĠLat v ian Ġand Ġ( especially ) ĠEston ian Ġare Ġmorph ologically Ġcomplex Ġand Ġhave Ġloose Ġword Ġorder , Ġwhile ĠEnglish Ġhas Ġa Ġstrict Ġword Ġorder Ġand Ġthe Ġmorphology Ġis Ġmuch Ġsimpler . ĠMost Ġimportantly , Ġall Ġthree Ġlanguages Ġhave Ġerror - correct ed Ġcorpor a Ġfor Ġtesting Ġpurposes , Ġthough Ġwork Ġon Ġtheir Ġautomatic Ġgram matical Ġerror Ġcorrection Ġis Ġextremely Ġlimited Ġ( see ĠSection ĠSEC REF 3 Ġ). </s> The Ġcorpor a Ġwe Ġuse Ġfor Ġtraining Ġthe Ġmodel Ġare ĠOpen Sub t itles 2018 ĠB IB REF 15 Ġ, ĠEuro par l ĠB IB REF 16 Ġ, ĠJ RC - Ac qu is Ġand ĠE ME A ĠB IB REF 17 Ġ. ĠWe Ġassume Ġthat Ġthere Ġshould Ġbe Ġsufficient Ġstyl istic Ġdifference Ġbetween Ġthese Ġcorpor a , Ġespecially Ġbetween Ġthe Ġmore Ġinformal ĠOpen Sub t itles 2018 Ġ( com pr ised Ġof Ġmovie Ġand ĠTV Ġsubtitles ) Ġon Ġone Ġhand Ġand ĠEuro par l Ġand ĠJ RC - Ac qu is Ġ( pro ceed ings Ġand Ġdocuments Ġof Ġthe ĠEuropean ĠParliament ) Ġon Ġthe Ġother . </s> Technical ĠDetails </s> For ĠEuro par l , ĠJ RC - Ac qu is Ġand ĠE ME A Ġwe Ġuse Ġall Ġdata Ġavailable Ġfor ĠEnglish - E ston ian , ĠEnglish - Lat v ian Ġand ĠEston ian - Lat v ian Ġlanguage Ġpairs . ĠFrom ĠOpen Sub t itles 2018 Ġwe Ġtake Ġa Ġrandom Ġsubset Ġof Ġ3 M Ġsentence Ġpairs Ġfor ĠEnglish - E ston ian , Ġwhich Ġis Ġstill Ġmore Ġthan ĠEnglish - Lat v ian Ġand ĠEston ian - Lat v ian Ġ( below Ġ1 M ; Ġthere Ġwe Ġuse Ġthe Ġwhole Ġcorpus ). ĠThis Ġis Ġdone Ġto Ġbalance Ġthe Ġcorpor a Ġrepresentation Ġand Ġto Ġlimit Ġthe Ġsize Ġof Ġtraining Ġdata . </s> Details Ġon Ġthe Ġmodel Ġhyper - param eters , Ġdata Ġpre - processing Ġand Ġtraining Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 7 Ġ. </s> E val uation </s> First , Ġwe Ġevaluate Ġour Ġmodel Ġin Ġthe Ġcontext Ġof ĠMT , Ġas Ġthe Ġtranslation Ġquality Ġcan Ġbe Ġexpected Ġto Ġhave Ġinfluence Ġon Ġthe Ġother Ġtasks Ġthat Ġthe Ġmodel Ġperforms . ĠWe Ġuse Ġpublic Ġbenchmarks Ġfor ĠEston ian - English Ġand ĠLat v ian - English Ġtranslations Ġfrom Ġthe Ġnews Ġtranslation Ġshared Ġtasks Ġof ĠW MT Ġ2017 Ġand Ġ2018 ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThe ĠB LE U Ġscores Ġfor Ġeach Ġtranslation Ġdirection Ġand Ġall Ġincluded Ġstyles / dom ains Ġare Ġshown Ġin ĠTable ĠT AB REF 6 Ġ. </s> Some Ġsurface Ġnotes Ġon Ġthese Ġresults : Ġthe ĠB LE U Ġscores Ġfor Ġtranslation Ġfrom Ġand Ġinto ĠLat v ian Ġare Ġbelow ĠEnglish - E ston ian Ġscores , Ġwhich Ġis Ġlikely Ġexplained Ġby Ġsmaller Ġdatasets Ġthat Ġinclude ĠLat v ian . ĠAlso , Ġtranslation Ġinto ĠEnglish Ġhas Ġhigher Ġscores Ġthan Ġinto ĠEston ian / Lat v ian , Ġwhich Ġis Ġalso Ġexpected . </s> An Ġinteresting Ġside - effect Ġwe Ġhave Ġobserved Ġis Ġthe Ġmodel 's Ġresilience Ġto Ġcode - sw itching Ġin Ġthe Ġinput Ġtext . ĠThe Ġreason Ġis Ġthat Ġthe Ġmodel Ġis Ġtrained Ġwith Ġonly Ġthe Ġtarget Ġlanguage Ġ( and Ġdomain ), Ġand Ġnot Ġthe Ġsource Ġlanguage , Ġas Ġa Ġresult Ġof Ġwhich Ġit Ġlearns Ġlanguage Ġnormal ization Ġof Ġsorts . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ Ma Ġt ahan Ġtwo Ġs ald Äĵ j um us . âĢ Ŀ Ġ( âĢ ľ Ma Ġt ahan âĢ Ŀ Ġ/ ĠâĢ ľ I Ġwant âĢ Ŀ Ġin ĠEston ian , ĠâĢ ľ two âĢ Ŀ Ġand ĠâĢ ľ s ald Äĵ j um us âĢ Ŀ Ġ/ ĠâĢ ľ ice - cream s âĢ Ŀ Ġin Ġgen itive , Ġplural Ġin ĠLat v ian ) Ġis Ġcorrectly Ġtranslated Ġinto ĠEnglish Ġas ĠâĢ ľ I Ġwant Ġtwo Ġice Ġcre ams . âĢ Ŀ . ĠSee Ġmore Ġexamples Ġin ĠAppendix ĠSEC REF 8 Ġ. </s> G ram matical ĠError ĠCorrection </s> In Ġthis Ġsection Ġwe Ġevaluate Ġour Ġmodel 's Ġperformance Ġin Ġthe ĠG EC Ġtask : Ġfor Ġexample , Ġfor Ġthe ĠEnglish Ġinput ĠâĢ ľ huge Ġfan ĠI Ġare âĢ Ŀ , Ġour Ġmodel 's Ġoutput Ġis ĠâĢ ľ I Ġam Ġa Ġhuge Ġfan âĢ Ŀ ; Ġthis Ġsection 's Ġgoal Ġis Ġto Ġsystematically Ġcheck , Ġhow Ġreliable Ġthe Ġcorrections Ġare . </s> Although ĠG EC Ġdoes Ġnot Ġrequire Ġany Ġdistinction Ġin Ġtext Ġstyle , Ġthe Ġcore Ġidea Ġof Ġthis Ġarticle Ġis Ġto Ġalso Ġperform Ġstyle Ġtransfer Ġwith Ġthe Ġsame Ġmult ilingual Ġmulti - domain Ġmodel . ĠThat Ġonly Ġmeans Ġthat Ġfor ĠG EC Ġwe Ġhave Ġto Ġselect Ġan Ġoutput Ġdomain / style Ġwhen Ġproducing Ġerror Ġcorrections . </s> Naturally , Ġthe Ġmodel Ġonly Ġcop es Ġwith Ġsome Ġkinds Ġof Ġerrors Ġand Ġfails Ġon Ġothers ĠâĢĵ Ġfor Ġinstance , Ġword Ġorder Ġis Ġrestored , Ġas Ġlong Ġas Ġit Ġdoes Ġnot Ġaffect Ġthe Ġperception Ġof Ġthe Ġmeaning . ĠOn Ġthe Ġother Ġhand , Ġwe Ġdo Ġnot Ġexpect Ġorth ographic Ġvariations Ġlike Ġty pos Ġto Ġbe Ġfixed Ġreliably , Ġsince Ġthey Ġaffect Ġthe Ġsub - word Ġsegment ation Ġof Ġthe Ġinput Ġand Ġthus Ġcan Ġhinder Ġthe Ġtranslation . </s> Below Ġwe Ġpresent Ġqualitative Ġand Ġquantitative Ġanalysis Ġof Ġour Ġmodel 's ĠG EC Ġresults , Ġshowing Ġits Ġoverall Ġperformance , Ġas Ġwell Ġas Ġwhich Ġkinds Ġof Ġerrors Ġare Ġhandled Ġreliably Ġand Ġwhich Ġare Ġnot . </s> Test ĠData Ġand ĠMet rics </s> We Ġuse Ġthe Ġfollowing Ġerror - correct ed Ġcorpor a Ġboth Ġfor Ġscoring Ġand Ġas Ġbasis Ġfor Ġmanual Ġanalysis : </s> for ĠEnglish : ĠCo N LL - 2014 ĠB IB REF 5 Ġand ĠJ F LE G ĠB IB REF 20 Ġcorpor a </s> for ĠEston ian : Ġthe ĠLear ner ĠLanguage ĠCorpus ĠB IB REF 21 </s> for ĠLat v ian : Ġthe ĠError - annot ated ĠCorpus Ġof ĠLat v ian ĠB IB REF 22 </s> All Ġof Ġthese Ġare Ġbased Ġon Ġlanguage Ġlear ner Ġ( L 2 ) Ġessays Ġand Ġtheir Ġmanual Ġcorrections . </s> To Ġevaluate Ġthe Ġmodel Ġquant itatively Ġwe Ġused Ġtwo Ġmetrics : Ġthe ĠMax - Match Ġ( M ĠIN LINE FORM 0 Ġ) Ġmetric Ġfrom Ġthe ĠCo N LL - 2014 Ġshared Ġtask Ġscorer , Ġand Ġthe ĠG LE U Ġscore ĠB IB REF 23 Ġfor Ġthe Ġother Ġcorpor a . ĠThe Ġmain Ġdifference Ġis Ġthat ĠM ĠIN LINE FORM 1 Ġis Ġbased Ġon Ġthe Ġannotation Ġof Ġerror Ġcategories , Ġwhile Ġthe ĠG LE U Ġscore Ġcompares Ġthe Ġautomatic Ġcorrection Ġto Ġa Ġreference Ġwithout Ġany Ġerror Ġcategor ization . </s> Results </s> The ĠM ĠIN LINE FORM 0 Ġscores Ġare Ġcomputed Ġbased Ġon Ġerror - annot ated Ġcorpor a . ĠSince Ġerror Ġannotations Ġwere Ġonly Ġavailable Ġfor ĠEnglish , Ġwe Ġcalculated Ġthe Ġscores Ġon ĠEnglish ĠCo N LL Ġcorpus , Ġsee ĠTable ĠT AB REF 12 Ġ). </s> Our Ġmodel Ġgets Ġthe ĠM ĠIN LINE FORM 0 Ġscore Ġof Ġ32 . 1 . ĠWhile Ġit Ġdoes Ġnot Ġreach Ġthe Ġscore Ġof Ġthe Ġbest ĠCo N LL Ġmodel ĠB IB REF 24 Ġor Ġthe Ġstate - of - the - art ĠB IB REF 25 Ġ, Ġthese Ġuse Ġannot ated Ġcorpor a Ġto Ġtrain . ĠOur Ġresults Ġcount Ġas Ġrestricted Ġin ĠCo N LL Ġdefinitions Ġand Ġare Ġmore Ġdirectly Ġcomparable Ġto Ġthe Ġclass ifier - based Ġapproach Ġtrained Ġon Ġun annot ated Ġcorpor a Ġby ĠB IB REF 26 Ġ, Ġwhile Ġrequiring Ġeven Ġless Ġeffort . </s> The ĠG LE U Ġscores Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 13 Ġ. ĠWe Ġcalculated ĠG LE U Ġfor Ġboth Ġformal Ġand Ġinformal Ġstyle Ġmodels Ġfor Ġall Ġthree Ġlanguages . ĠFor ĠEnglish Ġour Ġmodel 's Ġbest Ġscore Ġwas Ġ45 . 9 Ġand Ġfor ĠEston ian Ġit Ġwas Ġ38 . 1 . ĠLat v ian Ġcorrected Ġoutput Ġin Ġfact Ġget Ġworse Ġscores Ġthan Ġthe Ġoriginal Ġunc orrect ed Ġcorpus , Ġwhich Ġcan Ġbe Ġexplained Ġby Ġsmaller Ġtraining Ġcorpor a Ġand Ġworse ĠMT Ġquality Ġfor ĠLat v ian Ġ( see ĠTable ĠT AB REF 6 Ġ). </s> Qual itative ĠAnalysis </s> We Ġlooked Ġat Ġthe Ġautomatic Ġcorrections Ġfor Ġ100 Ġerroneous Ġsentences Ġof ĠEnglish Ġand ĠEston ian Ġeach Ġas Ġwell Ġas Ġ80 Ġsentences Ġof ĠLat v ian . ĠThe Ġoverall Ġaim Ġwas Ġto Ġfind Ġthe Ġratio Ġof Ġsentences Ġwhere Ġ( 1 ) Ġall Ġerrors Ġhave Ġbeen Ġcorrected Ġ( 2 ) Ġonly Ġsome Ġare Ġcorrected Ġ( 3 ) Ġonly Ġsome Ġare Ġcorrected Ġand Ġpart Ġof Ġthe Ġmeaning Ġis Ġchanged Ġand Ġ( 4 ) Ġall Ġmeaning Ġis Ġlost . </s> The Ġanalysis Ġwas Ġdone Ġseparately Ġfor Ġfour Ġerror Ġtypes : Ġspelling Ġand Ġgram matical Ġerrors , Ġword Ġchoice Ġand Ġword Ġorder . ĠIn Ġcase Ġa Ġsentence Ġincluded Ġmore Ġthat Ġone Ġerror Ġtype Ġit Ġwas Ġcounted Ġonce Ġfor Ġeach Ġerror Ġtype . ĠFor ĠEnglish Ġthe Ġfirst Ġtwo Ġtypes Ġwere Ġannot ated Ġin Ġthe Ġcorpus , Ġthe Ġrest Ġwere Ġannot ated Ġby Ġus , Ġseparating Ġthe Ġoriginal Ġthird Ġerror Ġcategory Ġinto Ġtwo Ġnew Ġones . ĠThe Ġresults Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 15 Ġ. </s> Not Ġall ĠEnglish Ġsentences Ġincluded Ġerrors . Ġ30 Ġsentences Ġremained Ġunchanged , Ġout Ġof Ġwhich Ġ17 Ġhad Ġno Ġmistakes Ġin Ġthem . ĠFor Ġthe Ġchanged Ġsentences Ġ87 % Ġwere Ġfully Ġor Ġpartially Ġcorrected . ĠIn Ġcase Ġof ĠEston ian , Ġwhere Ġall Ġsentences Ġhad Ġmistakes , Ġ61 Ġout Ġof Ġthe Ġ100 Ġsentences Ġwere Ġfully Ġor Ġpartially Ġcorrected Ġwithout Ġloss Ġof Ġinformation . Ġ12 Ġsentences Ġbecame Ġnonsense , Ġall Ġof Ġwhich Ġoriginally Ġhad Ġsome Ġlex ical Ġmistakes . ĠFor ĠEnglish Ġthe Ġresults Ġare Ġsimilar : Ġthe Ġmost Ġconfusing Ġtype Ġof Ġerrors Ġthat Ġleads Ġto Ġcomplete Ġloss Ġof Ġmeaning Ġis Ġword Ġchoice . ĠOn Ġthe Ġother Ġhand , Ġthis Ġwas Ġalso Ġthe Ġmost Ġcommon Ġerror Ġtype Ġfor Ġboth Ġlanguages Ġand Ġerrors Ġof Ġthat Ġtype Ġwere Ġfully Ġcorrected Ġin Ġ45 % Ġof Ġcases Ġfor ĠEston ian Ġand Ġ72 % Ġfor ĠEnglish . ĠUsing Ġwords Ġin Ġthe Ġwrong Ġorder Ġis Ġa Ġtypical Ġlear ner 's Ġerror Ġfor ĠEston ian Ġthat Ġhas Ġrather Ġfree Ġword Ġorder . ĠIt Ġis Ġalso Ġdifficult Ġto Ġdescribe Ġor Ġset Ġrules Ġfor Ġthis Ġtype Ġof Ġerror . ĠOur Ġmodel Ġmanages Ġthis Ġtype Ġrather Ġwell , Ġcorrecting Ġ79 % Ġof Ġsentences Ġaccept ably , Ġonly Ġlosing Ġsome Ġmeaning Ġin Ġ2 Ġsentences Ġincluding Ġthis Ġerror Ġtype . </s> A Ġsimilar Ġexperiment Ġusing Ġ80 ĠLat v ian Ġsentences Ġyielded Ġ17 Ġfully Ġcorrected Ġsentences , Ġ15 , Ġ22 Ġand Ġ26 Ġrespectively Ġfor Ġthe Ġother Ġcategories . ĠAs Ġthe ĠLat v ian Ġmodel Ġis Ġweaker Ġin Ġgeneral , Ġthis Ġalso Ġleads Ġto Ġmore Ġchances Ġof Ġlosing Ġsome Ġof Ġthe Ġmeaning ; Ġwe Ġexclude Ġit Ġfrom Ġthe Ġmore Ġdetailed Ġanalysis Ġand Ġfocus Ġon ĠEnglish Ġand ĠEston ian . </s> Our Ġmodel Ġhandles Ġpunct uation , Ġword Ġorder Ġmistakes Ġand Ġgram matical Ġerrors Ġwell . ĠFor Ġexample Ġthe Ġsubject - verb Ġdisagreement Ġin ĠEnglish ĠUID 16 Ġand Ġverb - object Ġdisagreement Ġin ĠEston ian ĠUID 19 Ġhave Ġbeen Ġcorrected . </s> âĢ ľ When Ġprice Ġof Ġgas Ġgoes Ġup Ġ, Ġthe Ġconsumer Ġdo Ġnot Ġwant Ġbuy Ġgas Ġfor Ġfuels âĢ Ŀ </s> âĢ ľ When Ġthe Ġprice Ġof Ġgas Ġgoes Ġup , Ġthe Ġconsumer Ġdoesn 't Ġwant Ġto Ġbuy Ġgas Ġfor Ġfuels âĢ Ŀ </s> âĢ ľ S elle p Ã¤ rast ĠÃ ¼ tle b Ġta Ġfilm i Ġl Ã µ p us , Ġet Ġt ah ab Ġo ma Ġun ist use Ġtag asi âĢ Ŀ </s> âĢ ľ S elle p Ã¤ rast ĠÃ ¼ tle b Ġta Ġfilm i Ġl Ã µ p us , Ġet Ġt ah ab Ġo ma Ġun ist ust Ġtag asi âĢ Ŀ </s> that 's - why Ġsays Ġhe Ġfilm Ġat - end , Ġthat Ġ( he )- w ants Ġhis - own Ġdream ĠIN LINE FORM 0 </s> Sent ences Ġthat Ġinclude Ġseveral Ġerror Ġtypes Ġare Ġgenerally Ġnoticeably Ġmore Ġdifficult Ġto Ġcorrect . ĠDepending Ġon Ġthe Ġerror Ġtypes Ġthat Ġhave Ġbeen Ġcombined Ġour Ġmodel Ġmanages Ġquite Ġwell Ġand Ġcorrect s Ġall Ġor Ġseveral Ġerrors Ġpresent . ĠThe Ġsentence ĠUID 23 Ġincludes Ġmistakes Ġwith Ġword Ġorder Ġand Ġword Ġchoice : Ġthe Ġargument Ġ" v aba ain et ele " Ġ( to Ġelect ive Ġcourses ) Ġhere Ġshould Ġpreced e Ġthe Ġverb Ġand Ġthe Ġverb Ġ" reg ist re er uma " Ġ( register Ġoneself ) Ġtakes Ġno Ġsuch Ġargument . ĠOur Ġmodel Ġcorrect s Ġboth Ġmistakes Ġwhile Ġalso Ġreplacing Ġthe Ġword Ġ" see j Ã¤ rel " Ġ( after Ġthat ) Ġwith Ġits Ġsyn onym . </s> See j Ã¤ rel Ġp idi Ġ iga Ã¼ ks Ġen n ast Ġregist re er uma Ġv aba ain et ele . </s> then Ġhad - to Ġeveryone Ġoneself Ġregister - ones elf Ġto - free - c ourses </s> Si is Ġp idi Ġ iga Ã¼ ks Ġend Ġv aba ain et ele Ġregist re er ima . </s> then Ġhad - to Ġeveryone Ġoneself Ġto - free - c ourses Ġregister </s> The Ġmodel Ġfixes Ġty pos , Ġbut Ġit Ġmainly Ġmanages Ġcases Ġwhere Ġtwo Ġletters Ġare Ġneeded Ġbut Ġone Ġis Ġwritten Ġand Ġvice Ġversa , Ġfor Ġexample Ġ" det a illed " Ġis Ġcorrected Ġto Ġ" det ailed " Ġand ĠâĢ ĺ pl aning âĢ Ļ Ġto Ġ" plan ning ". ĠMore Ġcomplicated Ġmistakes Ġare Ġmissed , Ġespecially Ġif Ġcombined Ġwith Ġother Ġerror Ġtypes , Ġand Ġin Ġsome Ġsentences Ġa Ġmiss pelled Ġword Ġis Ġchanged Ġinto Ġan Ġincorrect Ġform Ġthat Ġhas Ġa Ġcommon Ġending , Ġlike Ġ" mis und rest ood " Ġto Ġ" mis und rest ed ". ĠThe Ġresults Ġget Ġbetter Ġif Ġthe Ġinput Ġhas Ġbeen Ġautomatically Ġspell - checked . </s> The Ġsystem Ġdoes Ġmore Ġchanges Ġthan Ġare Ġstrictly Ġnecessary Ġand Ġoften Ġreplaces Ġcorrect Ġwords Ġand Ġphrases , Ġfor Ġexample Ġ" f requently " Ġwas Ġchanged Ġto Ġto Ġ" often " Ġor Ġin ĠEston ian ĠâĢ ľ Ã¶ e lda âĢ Ŀ Ġ(" say ") Ġto ĠâĢ ľ av ald ada âĢ Ŀ Ġ(" pub lish "). ĠSometimes Ġthis Ġalso Ġconf uses Ġthe Ġmeaning : Ġ" s iblings " Ġwas Ġchanged Ġto Ġ" friends ". </s> To Ġconclude Ġthis Ġsection , Ġour Ġmodel Ġreliable Ġcorrect s Ġgram matical , Ġspelling Ġand Ġword Ġorder Ġerrors Ġon Ġ, Ġwith Ġmore Ġmixed Ġperformance Ġon Ġlex ical Ġchoice Ġerrors Ġand Ġsome Ġunnecessary Ġparaph r asing Ġof Ġthe Ġinput . ĠThe Ġerror Ġtypes Ġthat Ġthe Ġmodel Ġmanages Ġwell Ġcan Ġbe Ġtraced Ġback Ġto Ġhaving Ġa Ġstrong Ġmon oling ual Ġlanguage Ġmodel , Ġa Ġcommon Ġtrait Ġof Ġa Ġgood ĠN MT Ġmodel . ĠAs Ġthe Ġmodel Ġoperates Ġon Ġthe Ġlevel Ġof Ġword Ġparts Ġand Ġits Ġvocabulary Ġis Ġlimited , Ġthis Ġleads Ġto Ġcombining Ġwrong Ġword Ġparts , Ġsometimes Ġacross Ġlanguages . ĠThis Ġcould Ġbe Ġfixed Ġby Ġeither Ġusing Ġcharacter - based ĠN MT Ġor Ġdoing Ġautomatic Ġspelling Ġcorrection Ġprior Ġto Ġapplying Ġour Ġcurrent Ġmodel . </s> We Ġlimit Ġfurther Ġcomparisons Ġto Ġtwo Ġstyles , Ġtranslating Ġsentences Ġof Ġthe ĠOpen Sub t itles Ġtest Ġset Ġinto Ġthe Ġstyle Ġof ĠEuro par l Ġand Ġvice Ġversa . ĠOur Ġassumption Ġis Ġthat , Ġgenerally , Ġmovie Ġsubtitles Ġgrav itate Ġtowards Ġthe Ġmore Ġinformal Ġstyle , Ġand Ġparliament Ġproceedings Ġtowards Ġthe Ġmore Ġformal Ġ( see Ġexamples Ġof Ġtranslations Ġinto Ġthose Ġstyles Ġin ĠTable ĠT AB REF 30 Ġ). ĠPrel iminary Ġtests Ġshowed Ġthat ĠJ RC - Ac qu is Ġand ĠE ME A Ġtexts Ġresulted Ġin Ġpractically Ġthe Ġsame Ġstyle Ġas ĠEuro par l . ĠWe Ġalso Ġleave ĠLat v ian Ġout Ġof Ġthe Ġevaluations , Ġassuming Ġthat Ġits Ġperformance Ġis Ġweaker , Ġsimilarly Ġto ĠG EC Ġresults . </s> Human Ġevaluation Ġwas Ġperformed Ġon Ġa Ġsubset Ġof Ġ100 Ġsentences , Ġ50 Ġof Ġthem Ġselected Ġrandomly Ġfrom Ġthe ĠOpen Sub t itles Ġtest Ġset Ġand Ġthe Ġother Ġ50 Ġfrom ĠEuro par l . ĠEach Ġsentence Ġwas Ġtranslated Ġinto Ġthe Ġopposite Ġstyle . ĠThe Ġresulting Ġpairs Ġwere Ġpresented Ġto Ġparticipants , Ġwho Ġwere Ġasked Ġthe Ġfollowing Ġquestions Ġabout Ġeach Ġof Ġthem : Ġ( 1 ) ĠDo Ġthe Ġsentences Ġdiffer Ġin Ġany Ġway ? Ġ( 2 ) ĠHow Ġfluent Ġis Ġthe Ġtranslated Ġsentence ? Ġ( On Ġa Ġscale Ġof Ġ1 Ġto Ġ4 , Ġwhere Ġ1 Ġis Ġun readable , Ġand Ġ4 Ġis Ġperfectly Ġfluent ); Ġ( 3 ) ĠHow Ġsimilar Ġare Ġthe Ġsentences Ġin Ġmeaning ? Ġ( With Ġoptions Ġ" ex actly Ġthe Ġsame ", Ġ" the Ġsame Ġwith Ġminor Ġchanges ", Ġ" more Ġor Ġless Ġthe Ġsame ", Ġ" completely Ġdifferent Ġor Ġnonsensical "); Ġ( 4 ) ĠDoes Ġthe Ġtranslated Ġsentence Ġsound Ġmore Ġformal Ġthan Ġthe Ġoriginal , Ġmore Ġinformal , Ġor Ġneither ? Ġ( 5 ) ĠWhat Ġdifferences Ġare Ġthere Ġbetween Ġthe Ġsentences ? Ġ( E . g . Ġgram matical , Ġlex ical , Ġmissing Ġwords Ġor Ġphrases , Ġword Ġorder , Ġcontract ions , Ġthe Ġuse Ġof Ġformal Ġ" you "). </s> Two Ġsuch Ġsurveys Ġwere Ġconducted , Ġone Ġin ĠEnglish Ġand Ġone Ġin ĠEston ian . Ġ3 Ġpeople Ġparticipated Ġin Ġeach Ġof Ġthem , Ġeach Ġof Ġthe Ġthree Ġeval u ators Ġpresented Ġwith Ġthe Ġsame Ġset Ġof Ġexamples . </s> In Ġevaluation Ġof Ġflu ency , Ġall Ġthree Ġhuman Ġeval u ators Ġgave Ġthe Ġtranslated Ġsentences Ġthe Ġsame Ġscore Ġin Ġ41 Ġout Ġof Ġ55 Ġcases Ġin ĠEnglish Ġ( not Ġtaking Ġinto Ġaccount Ġsentences Ġwhich Ġwere Ġsimply Ġcopied Ġfrom Ġtheir Ġoriginals ), Ġand Ġin Ġ51 Ġout Ġof Ġ68 Ġcases Ġin ĠEston ian . ĠIn Ġevaluation Ġof Ġdirection Ġof Ġstyle Ġtransfer , Ġall Ġthree Ġeval u ators Ġagreed Ġin Ġ16 Ġcases Ġand Ġat Ġleast Ġtwo Ġagreed Ġin Ġ43 Ġcases Ġin ĠEnglish , Ġand Ġin ĠEston ian Ġin Ġ19 Ġcases Ġall Ġthree Ġagreed Ġand Ġin Ġ59 Ġat Ġleast Ġtwo . </s> Of Ġthe Ġ100 Ġtranslated Ġsentences , Ġ45 Ġwere Ġmarked Ġby Ġall Ġparticipants Ġas Ġbeing Ġthe Ġsame Ġas Ġtheir Ġoriginal Ġsentences Ġin Ġthe ĠEnglish Ġset Ġand Ġ32 Ġin ĠEston ian . ĠThe Ġremaining Ġ55 Ġand Ġ68 , Ġrespectively , Ġwere Ġused Ġto Ġquantify Ġstyle Ġtransfer Ġquality . </s> Being Ġa Ġreasonably Ġstrong ĠMT Ġsystem , Ġour Ġmodel Ġscores Ġquite Ġhigh Ġon Ġflu ency Ġ( 3 . 84 Ġfor ĠEnglish , Ġ3 . 64 Ġfor ĠEston ian ) Ġand Ġmeaning Ġpreservation Ġ( 3 . 67 Ġfor ĠEnglish , Ġ3 . 35 Ġfor ĠEston ian ). ĠFor Ġmeaning Ġpreservation , Ġthe Ġjudgments Ġwere Ġconverted Ġinto Ġa Ġscale Ġof Ġ1 - 4 , Ġwhere Ġ1 Ġstands Ġfor Ġcompletely Ġdifferent Ġmeanings Ġor Ġnonsensical Ġsentences , Ġand Ġ4 Ġfor Ġthe Ġexact Ġsame Ġmeaning . </s> We Ġevaluated Ġthe Ġstyle Ġtransfer Ġitself Ġin Ġthe Ġfollowing Ġway . ĠFor Ġeach Ġpair Ġof Ġsentences , Ġthe Ġaverage Ġscore Ġgiven Ġby Ġthree Ġeval u ators Ġwas Ġcalculated , Ġin Ġwhich Ġthe Ġanswer Ġthat Ġthe Ġtranslated Ġsentence Ġis Ġmore Ġformal Ġcounts Ġas Ġ+ 1 , Ġmore Ġinformal Ġas Ġ- 1 , Ġand Ġneither Ġas Ġ0 . ĠWe Ġcalculated Ġthe Ġroot Ġmean Ġsquare Ġerror Ġbetween Ġthese Ġscores Ġand Ġdesired Ġvalues Ġ(+ 1 Ġif Ġwe Ġaimed Ġto Ġmake Ġthe Ġsentence Ġmore Ġformal , Ġ- 1 Ġif Ġmore Ġinformal ). ĠRM SE Ġof Ġ0 Ġwould Ġstand Ġfor Ġalways Ġtransferring Ġstyle Ġin Ġthe Ġright Ġdirection Ġas Ġjudged Ġby Ġall Ġeval u ators , Ġand Ġ2 Ġfor Ġalways Ġtransferring Ġstyle Ġin Ġthe Ġwrong Ġdirection . </s> On Ġthe ĠEnglish Ġset , Ġthe ĠRM SE Ġis Ġ0 . 78 , Ġand Ġon ĠEston ian Ġ0 . 89 . ĠThese Ġnumbers Ġshow Ġthat Ġstyle Ġtransfer Ġgenerally Ġhappens Ġin Ġthe Ġright Ġdirection , Ġbut Ġis Ġnot Ġvery Ġstrong . ĠOf Ġthe Ġ55 Ġsentences Ġin ĠEnglish Ġthat Ġwere Ġdifferent Ġfrom Ġtheir Ġsource Ġsentences , Ġin Ġ33 Ġcases Ġthe Ġsign Ġof Ġthe Ġaverage Ġhuman Ġscore Ġmatched Ġthe Ġdesired Ġone , Ġin Ġ7 Ġit Ġdid Ġnot , Ġand Ġin Ġ15 Ġno Ġchange Ġin Ġstyle Ġwas Ġobserved Ġby Ġhumans . ĠIn ĠEston ian Ġ36 Ġsentences Ġshowed Ġthe Ġright Ġdirection Ġof Ġstyle Ġtransfer , Ġ10 Ġwrong , Ġand Ġ22 Ġno Ġchange . </s> In ĠEnglish Ġsentences Ġwhere Ġthe Ġdirection Ġof Ġstyle Ġtransfer Ġwas Ġfound Ġto Ġbe Ġcorrect Ġ( Figure ĠFIG REF 32 Ġ), Ġchanges Ġin Ġuse Ġof Ġcontract ions Ġwere Ġreported Ġin Ġ19 Ġcases Ġ( e . g . ĠI Ġhave Ġjust Ġbeen Ġvs . ĠI 've Ġjust Ġbeen ), Ġlex ical Ġchanges Ġin Ġ15 Ġcases Ġ( e . g . Ġ' cause Ġvs . Ġbecause , Ġor Ġsure Ġvs . Ġcertainly ), Ġgram matical Ġin Ġ13 Ġ( e . g . Ġreplacing Ġno Ġone 's Ġgonna Ġwith Ġno Ġone Ġwill , Ġor Ġmethod Ġof Ġproduction Ġwith Ġproduction Ġmethod ), Ġmissing Ġor Ġadded Ġwords Ġor Ġphrases Ġin Ġ8 Ġcases . </s> In ĠEston ian Ġcorrectly Ġtransferred Ġsentences Ġ( Figure ĠFIG REF 32 Ġ), Ġthe Ġmost Ġfrequently Ġreported Ġwere Ġlex ical Ġsubstit utions Ġ( 30 Ġcases ), Ġfollowed Ġby Ġmissing Ġof Ġadded Ġwords Ġor Ġphrases Ġ( 24 Ġcases ), Ġchanges Ġin Ġgrammar Ġ( 22 Ġcases ) Ġand Ġin Ġword Ġorder Ġ( 16 Ġcases ). </s> To Ġconclude Ġthis Ġsection , Ġunlike Ġmany Ġstyle Ġtransfer Ġmodels Ġwhich Ġproduce Ġtext Ġwith Ġstrong Ġstyle Ġcharacteristics Ġ( e . g . Ġwith Ġstrong Ġpositive Ġor Ġnegative Ġsentiment ), Ġoften Ġat Ġthe Ġcost Ġof Ġpreserving Ġmeaning Ġand Ġflu ency , Ġour Ġmodel Ġgrav itates Ġtowards Ġkeeping Ġthe Ġmeaning Ġand Ġflu ency Ġof Ġthe Ġoriginal Ġsentence Ġintact Ġand Ġmim icking Ġsome Ġof Ġthe Ġdesired Ġstyl istic Ġtraits . </s> Style ĠTransfer </s> Next Ġwe Ġmove Ġon Ġto Ġevaluating Ġthe Ġsame Ġmodel Ġin Ġterms Ġof Ġits Ġperformance Ġin Ġthe Ġcontext Ġof Ġstyle Ġtransfer . </s> At Ġfirst , Ġwe Ġexamined Ġhow Ġoften Ġthe Ġsentences Ġchange Ġwhen Ġtranslated Ġmon oling ually . ĠThe Ġassumption Ġis Ġthat Ġpassing Ġmodified Ġstyle Ġfactors Ġshould Ġprevent Ġthe Ġmodel Ġfrom Ġsimply Ġcopying Ġthe Ġsource Ġsequences Ġwhen Ġtranslating Ġinside Ġa Ġsingle Ġlanguage , Ġand Ġincentiv ize Ġit Ġto Ġmatch Ġits Ġoutput Ġto Ġcertain Ġstyle Ġcharacteristics Ġtypical Ġfor Ġdifferent Ġcorpor a . ĠFigure ĠFIG REF 28 Ġshows Ġthe Ġproportions Ġof Ġsentence Ġpairs Ġin Ġthe Ġ1000 - sent ence Ġtest Ġsets Ġwhere Ġthere Ġwas Ġa Ġsignificant Ġdifference Ġbetween Ġtranslations Ġinto Ġdifferent Ġstyles . ĠWe Ġcan Ġobserve Ġthat ĠEnglish Ġtexts Ġchange Ġless Ġoften Ġthan ĠEston ian Ġor ĠLat v ian , Ġwhile ĠEuro par l Ġsentences Ġare Ġchanged Ġmore Ġoften Ġthan Ġthose Ġof Ġother Ġcorpor a . </s> To Ġassess Ġwhether Ġthese Ġchanges Ġactually Ġcorrespond Ġto Ġthe Ġmodel 's Ġcapability Ġfor Ġtransferring Ġstyle , Ġwe Ġturned Ġto Ġhelp Ġof Ġhuman Ġeval u ators . </s> Cross - ling ual ĠStyle ĠTransfer </s> Being Ġable Ġto Ġtranslate Ġbetween Ġlanguages Ġand Ġalso Ġto Ġmodify Ġthe Ġoutput Ġto Ġmatch Ġthe Ġdesired Ġstyle Ġallows Ġthe Ġmodel Ġto Ġessentially Ġperform Ġdomain Ġadaptation . ĠWhen Ġtranslating Ġfrom Ġa Ġlanguage Ġwhich Ġhas Ġno Ġformal Ġ" you " Ġ( English ) Ġinto Ġone Ġthat Ġdoes Ġ( E ston ian Ġor ĠLat v ian ), Ġit Ġwill Ġquite Ġconsistently Ġuse Ġthe Ġinformal Ġvariant Ġwhen Ġthe Ġtarget Ġstyle Ġis ĠOpen Sub t itles Ġand Ġthe Ġformal Ġwhen Ġthe Ġtarget Ġstyle Ġis ĠEuro par l Ġ( you Ġrock ĠIN LINE FORM 0 Ġsa Ġro kid / te Ġro k ite ). ĠThe Ġmodel Ġis Ġalso Ġquite Ġconsistent Ġin Ġuse Ġof Ġcontract ions Ġin ĠEnglish Ġ( es Ġes mu ĠÅ ¡ e it ĠIN LINE FORM 1 ĠI Ġam Ġhere / I 'm Ġhere ). ĠSome Ġlex ical Ġsubstit utions Ġoccur : Ġneed Ġon ĠMatt i Ġlaps ed . ĠIN LINE FORM 2 Ġthose Ġare ĠMatt 's Ġkids ./ these Ġare ĠMatt 's Ġchildren . ĠWord Ġorder Ġmay Ġchange : ĠWhere Ġis ĠAnna 's Ġbag ? Ġis ĠKus Ġon ĠAnna Ġk ott ? Ġin Ġthe Ġmore Ġformal Ġvariant , Ġand ĠKus ĠAnna Ġk ott Ġon ? Ġin Ġthe Ġmore Ġinformal . ĠThis Ġfeature Ġis Ġuseful , Ġbut Ġout Ġof Ġscope Ġof Ġthis Ġarticle , Ġas Ġwe Ġfocus Ġon Ġmon oling ual Ġapplications . </s> Related ĠWork </s> G ram matical Ġerror Ġcorrection : Ġthere Ġhave Ġbeen Ġfour Ġshared Ġtasks Ġfor ĠG EC Ġwith Ġprepared Ġerror - tag ged Ġdatasets Ġfor ĠL 2 Ġlearners Ġof ĠEnglish Ġin Ġthe Ġlast Ġdecade : ĠH OO ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġand ĠCo N LL ĠB IB REF 29 Ġ, ĠB IB REF 5 Ġ. ĠThis Ġhas Ġgiven Ġan Ġopportunity Ġto Ġtrain Ġnew Ġmodels Ġon Ġthe Ġshared Ġdatasets Ġand Ġget Ġan Ġobjective Ġcomparison Ġof Ġresults . ĠThe Ġgeneral Ġapproach Ġfor Ġgram matical Ġerror Ġcorrection Ġhas Ġbeen Ġto Ġuse Ġeither Ġrule - based Ġapproach , Ġmachine Ġlearning Ġon Ġerror - tag ged Ġcorpor a , ĠMT Ġmodels Ġon Ġparallel Ġdata Ġof Ġerroneous Ġand Ġcorrected Ġsentences , Ġor Ġa Ġcombination Ġof Ġthese ĠB IB REF 5 Ġ. ĠThe Ġtop Ġmodel Ġof Ġthe ĠCON LL Ġshared Ġtask Ġin Ġ2014 Ġused Ġa Ġcombined Ġmodel Ġof Ġrule - based Ġapproach Ġand ĠMT ĠB IB REF 24 Ġ. ĠAll Ġof Ġthese Ġrequire Ġannot ated Ġdata Ġor Ġconsiderable Ġeffort Ġto Ġcreate , Ġwhereas Ġour Ġmodel Ġis Ġmuch Ġmore Ġresource - independent . ĠAnother Ġfocus Ġof Ġthe Ġnewer Ġresearch Ġis Ġon Ġcreating ĠG EC Ġmodels Ġwithout Ġhuman - annot ated Ġresources . ĠFor Ġexample ĠB IB REF 26 Ġcombine Ġstatistical ĠMT Ġwith Ġun super vised Ġclassification Ġusing Ġun annot ated Ġparallel Ġdata Ġfor ĠMT Ġand Ġun annot ated Ġnative Ġdata Ġfor Ġthe Ġclassification Ġmodel . ĠIn Ġthis Ġcase Ġparallel Ġdata Ġof Ġerroneous Ġand Ġcorrected Ġsentences Ġis Ġstill Ġnecessary Ġfor ĠMT ; Ġthe Ġclass ifier Ġuses Ġnative Ġdata , Ġbut Ġstill Ġneeds Ġdefinitions Ġof Ġpossible Ġerror Ġtypes Ġto Ġclassify ĠâĢĵ Ġthis Ġwork Ġneeds Ġto Ġbe Ġdone Ġby Ġa Ġhuman Ġand Ġis Ġdifficult Ġfor Ġsome Ġless Ġclear Ġerror Ġtypes . ĠIn Ġour Ġapproach Ġthere Ġis Ġno Ġneed Ġfor Ġparallel Ġdata Ġnor Ġto Ġspecify Ġerror Ġtypes , Ġonly Ġfor Ġnative Ġdata . </s> There Ġhas Ġbeen Ġlittle Ġwork Ġon ĠEston ian Ġand ĠLat v ian ĠG EC , Ġall Ġlimited Ġwith Ġrule - based Ġapproaches ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ. ĠFor Ġboth Ġlanguages , Ġas Ġwell Ġas Ġany Ġlow - res ourced Ġlanguages , Ġour Ġapproach Ġgives Ġa Ġfeasible Ġway Ġto Ġdo Ġgram matical Ġerror Ġcorrection Ġwithout Ġneeding Ġneither Ġparallel Ġnor Ġerror Ġtagged Ġcorpor a . ĠStyle Ġtransfer : ĠSeveral Ġapproaches Ġuse Ġdirectly Ġannot ated Ġdata : Ġfor Ġexample , ĠB IB REF 7 Ġand ĠB IB REF 8 Ġtrain ĠMT Ġsystems Ġon Ġthe Ġcorpus Ġof Ġmodern ĠEnglish ĠShakespeare Ġto Ġoriginal ĠShakespeare . ĠB IB REF 32 Ġcollect Ġa Ġdataset Ġof Ġ110 K Ġinformal / form al Ġsentence Ġpairs Ġand Ġtrain Ġrule - based , Ġphrase - based , Ġand Ġneural ĠMT Ġsystems Ġusing Ġthis Ġdata . </s> One Ġline Ġof Ġwork Ġaims Ġat Ġlearning Ġa Ġstyle - independent Ġlatent Ġrepresentation Ġof Ġcontent Ġwhile Ġbuilding Ġdec od ers Ġthat Ġcan Ġgenerate Ġsentences Ġin Ġthe Ġstyle Ġof Ġchoice ĠB IB REF 9 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ, ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġ. ĠUn super vised ĠMT Ġhas Ġalso Ġbeen Ġadapted Ġfor Ġthe Ġtask ĠB IB REF 10 Ġ, ĠB IB REF 40 Ġ. ĠOur Ġsystem Ġalso Ġdoes Ġnot Ġrequire Ġparallel Ġdata Ġbetween Ġstyles , Ġbut Ġlever ages Ġthe Ġstability Ġof Ġthe Ġoff - the - she lf Ġsupervised ĠN MT Ġto Ġavoid Ġthe Ġhassle Ġof Ġtraining Ġun super vised ĠN MT Ġsystems Ġand Ġmaking ĠG AN s Ġconverge . ĠAnother Ġproblem Ġwith Ġmany Ġcurrent Ġ( both Ġsupervised Ġand Ġun super vised ) Ġstyle Ġtransfer Ġmethods Ġis Ġthat Ġthey Ġare Ġbounded Ġto Ġsolve Ġa Ġbinary Ġtask , Ġwhere Ġonly Ġtwo Ġstyles Ġare Ġincluded Ġ( whether Ġbecause Ġof Ġdata Ġor Ġrestrictions Ġof Ġthe Ġapproach ). ĠOur Ġmethod , Ġon Ġthe Ġother Ġhand , Ġcan Ġbe Ġextended Ġto Ġas Ġmany Ġstyles Ġas Ġneeded Ġas Ġlong Ġas Ġthere Ġare Ġparallel ĠMT Ġcorpor a Ġin Ġthese Ġstyles Ġavailable . </s> Not ably , ĠB IB REF 41 Ġuse Ġside Ġconst r ains Ġin Ġorder Ġto Ġtranslate Ġin Ġpolite / imp ol ite ĠGerman , Ġwhile Ġwe Ġrely Ġon Ġmult ilingual Ġenc oder Ġrepresentations Ġand Ġuse Ġthe Ġsystem Ġmon oling ually Ġat Ġinference Ġtime . </s> Finally , Ġthe Ġmost Ġsimilar Ġto Ġour Ġwork Ġconcept ually Ġis Ġthe Ġapproach Ġof ĠB IB REF 42 Ġ, Ġwhere Ġthey Ġtranslate Ġa Ġsentence Ġinto Ġanother Ġlanguage , Ġhoping Ġthat Ġit Ġwill Ġlose Ġsome Ġstyle Ġindicators , Ġand Ġthen Ġtranslate Ġit Ġback Ġinto Ġthe Ġoriginal Ġlanguage Ġwith Ġa Ġdesired Ġstyle Ġtag Ġattached Ġto Ġthe Ġenc oder Ġlatent Ġspace . ĠWe Ġalso Ġuse Ġthe ĠMT Ġenc oder Ġto Ġobtain Ġrich Ġsentence Ġrepresentations , Ġbut Ġlearn Ġthem Ġdirectly Ġas Ġa Ġpart Ġof Ġa Ġsingle Ġmult ilingual Ġtranslation Ġsystem . </s> Con clusions </s> We Ġpresented Ġa Ġsimple Ġapproach Ġwhere Ġa Ġsingle Ġmult ilingual ĠN MT Ġmodel Ġis Ġadapted Ġto Ġmon oling ual Ġtransfer Ġand Ġperforms Ġgram matical Ġerror Ġcorrection Ġand Ġstyle Ġtransfer . ĠWe Ġexperimented Ġwith Ġthree Ġlanguages Ġand Ġpresented Ġextensive Ġevaluation Ġof Ġthe Ġmodel Ġon Ġboth Ġtasks . ĠWe Ġused Ġpublicly Ġavailable Ġsoftware Ġand Ġdata Ġand Ġbelieve Ġthat Ġour Ġwork Ġcan Ġbe Ġeasily Ġreproduced . </s> We Ġshowed Ġthat Ġfor ĠG EC Ġour Ġapproach Ġreliably Ġcorrect s Ġspelling , Ġword Ġorder Ġand Ġgram matical Ġerrors , Ġwhile Ġbeing Ġless Ġreliable Ġon Ġlex ical Ġchoice Ġerrors . ĠApplied Ġto Ġstyle Ġtransfer Ġour Ġmodel Ġis Ġvery Ġgood Ġat Ġmeaning Ġpreservation Ġand Ġoutput Ġflu ency , Ġwhile Ġreliably Ġtransferring Ġstyle Ġfor ĠEnglish Ġcontract ions , Ġlex ical Ġchoice Ġand Ġgram matical Ġconstruct ions . ĠThe Ġmain Ġbenefit Ġis Ġthat Ġno Ġannot ated Ġdata Ġis Ġused Ġto Ġtrain Ġthe Ġmodel , Ġthus Ġmaking Ġit Ġvery Ġeasy Ġto Ġtrain Ġit Ġfor Ġother Ġ( especially Ġunder - res ourced ) Ġlanguages . </s> Future Ġwork Ġincludes Ġexploring Ġadaptations Ġof Ġthis Ġapproach Ġto Ġboth Ġtasks Ġseparately , Ġwhile Ġkeeping Ġthe Ġlow Ġcost Ġof Ġcreating Ġsuch Ġmodels . </s> Model ĠTraining : ĠTechnical ĠDetails </s> After Ġrudimentary Ġcleaning Ġ( rem oving Ġpairs Ġwhere Ġat Ġleast Ġone Ġsentence Ġis Ġlonger Ġthat Ġ100 Ġtokens , Ġat Ġleast Ġone Ġsentence Ġis Ġan Ġempty Ġstring Ġor Ġdoes Ġnot Ġcontain Ġany Ġal ph abetic Ġcharacters , Ġand Ġpairs Ġwith Ġlength Ġratio Ġover Ġ9 ) Ġand Ġduplication Ġto Ġaccommodate Ġboth Ġtranslation Ġdirections Ġin Ġeach Ġlanguage Ġpair , Ġthe Ġtotal Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġis Ġ22 . 9 M Ġsentence Ġpairs ; Ġtraining Ġset Ġsizes Ġper Ġlanguage Ġand Ġcorpus Ġare Ġgiven Ġin ĠTable ĠT AB REF 36 Ġ. ĠVal idation Ġset Ġconsists Ġof Ġ12 K Ġsentence Ġpairs , Ġ500 Ġfor Ġeach Ġcombination Ġof Ġtranslation Ġdirection Ġand Ġcorpus . ĠWe Ġalso Ġkeep Ġa Ġtest Ġset Ġof Ġ24 K Ġsentence Ġpairs , Ġ1000 Ġfor Ġeach Ġtranslation Ġdirection Ġand Ġcorpus . </s> The Ġdata Ġpre processing Ġpipeline Ġconsists Ġof Ġtoken ization Ġwith ĠMoses Ġtoken izer ĠB IB REF 44 Ġ, Ġtrue - c asing , Ġand Ġsegment ation Ġwith ĠSent ence P iece ĠB IB REF 45 Ġwith Ġa Ġjoint Ġvocabulary Ġof Ġsize Ġ32 Ġ000 . </s> We Ġtrained Ġa ĠTrans former ĠN MT Ġmodel Ġusing Ġthe ĠSoc keye Ġframework ĠB IB REF 43 Ġ, Ġmostly Ġfollowing Ġthe Ġso - called ĠTrans former Ġbase Ġmodel : Ġwe Ġused Ġ6 Ġlayers , Ġ512 Ġpositions , Ġ8 Ġattention Ġheads Ġand ĠRe LU Ġactiv ations Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder ; ĠAdam Ġoptim izer Ġwas Ġused . ĠSource Ġand Ġtarget Ġtoken Ġembed d ings Ġwere Ġboth Ġof Ġsize Ġ512 , Ġand Ġfactors Ġdetermining Ġtarget Ġlanguage Ġand Ġstyle Ġhad Ġembed d ings Ġof Ġsize Ġ4 . ĠB atch Ġsize Ġwas Ġset Ġto Ġ2048 Ġwords , Ġinitial Ġlearning Ġrate Ġto Ġ0 . 0002 , Ġreducing Ġby Ġa Ġfactor Ġof Ġ0 . 7 Ġevery Ġtime Ġthe Ġvalidation Ġperplex ity Ġhad Ġnot Ġimproved Ġfor Ġ8 Ġcheckpoints , Ġwhich Ġhappened Ġevery Ġ4000 Ġupdates . ĠThe Ġmodel Ġconver ged Ġduring Ġthe Ġ17 th Ġepoch , Ġwhen Ġvalidation Ġperplex ity Ġhas Ġnot Ġimproved Ġfor Ġ32 Ġconsecutive Ġcheckpoints . ĠThe Ġparameters Ġof Ġa Ġsingle Ġbest Ġcheckpoint Ġwere Ġused Ġfor Ġall Ġtranslations , Ġwith Ġbeam Ġsize Ġset Ġto Ġ5 . </s> Output ĠExamples </s> We Ġpresent Ġmore Ġexamples Ġof Ġtranslation Ġof Ġcode - sw itched Ġinput Ġsegments , Ġerror Ġcorrection Ġand Ġstyle Ġtransfer Ġin ĠEnglish , ĠEston ian Ġand ĠLat v ian , Ġinformal Ġ( inf ) Ġand Ġformal Ġ( fml ) Ġoutput Ġstyle :
Introduction Sequence-to-sequence (seq2seq) transformations have recently proven to be a successful framework for several natural language processing tasks, like: machine translation (MT) BIBREF0 , BIBREF1 , speech recognition BIBREF2 , speech synthesis BIBREF3 , natural language inference BIBREF4 and others. However, the success of these models depends on the availability of large amounts of directly annotated data for the task at hand (like translation examples, text segments and their speech recordings, etc.). This is a severe limitation for tasks where data is not abundantly available as well as for low-resource languages. Here we focus on two such tasks: grammatical error correction (GEC) and style transfer. Modern approaches to GEC learn from parallel corpora of erroneous segments and their manual corrections BIBREF5 , BIBREF6 ; text style transfer also relies on supervised approaches that require texts of the same meaning and different styles BIBREF7 , BIBREF8 or imprecise unsupervised methods BIBREF9 , BIBREF10 . In this paper we introduce an approach to performing both GEC and style transfer with the same trained model, while not using any supervised training data for either task. It is based on zero-shot neural machine translation (NMT) BIBREF11 , and as such, the only kind of data it uses is regular parallel corpora (with texts and their translations). However, we apply the model to do monolingual transfer, asking to translate the input segment into the same language. We show, that this “monolingual translation” is what enables the model to correct the errors in the input as well as adapt the output into a desired style. Moreover, the same trained model performs both tasks on several languages. Our main contributions are thus: (i) a single method for both style transfer and grammatical error correction, without using annotated data for either task, (ii) support for both tasks on multiple languages within the same model, (iii) a thorough quantitative and qualitative manual evaluation of the model on both tasks, and (iv) highlighting of the model's reliability aspects on both tasks. We used publicly available software and data; an online demo of our results is available, but concealed for anonymization purposes. We describe the details of our approach in Section SECREF2 , then evaluate it in terms of performance in grammatical error correction in Section SECREF3 and in style transfer in Section SECREF4 . The paper ends with a review of related work in Section SECREF5 and conclusions in Section SECREF6 . Method As mentioned in the introduction, our approach is based on the idea of zero-shot MT BIBREF11 . There the authors show that after training a single model to translate from Portuguese to English as well as from English to Spanish, it can also translate Portuguese into Spanish, without seeing any translation examples for this language pair. We use the zero-shot effect to achieve monolingual translation by training the model on bilingual examples in both directions, and then doing translation into the same language as the input: illustrated on Figure FIGREF1 . With regular sentences monolingual translation does not seem useful, as its behaviour mainly consists of copying. However, when the input sentence has characteristics unseen or rarely seen by the model at training time (like grammatical errors or different stylistic choices) – the decoder still generates the more regular version of the sentence (thus fixing the errors or adapting the style). Furthermore, in case of multilingual multi-domain NMT BIBREF12 , it is possible to switch between different domains or styles at runtime, thus performing “monolingual domain adaptation” or style transfer. To create a multilingual multi-domain NMT system we use the self-attention architecture BIBREF13 . Instead of specifying the output language with a token inside the input sequence, as BIBREF11 did, we follow BIBREF12 and use word features (or factors). On one hand, this provides a stronger signal for the model, and on the other – allows for additional parametrization, which in our case is the text domain/style of the corpus. As a result, a pre-processed English-Latvian training set sentence pair “Hello!”–“Sveiki!” looks like: Here 2lv and 2os specify Latvian and OpenSubtitles as the output language and domain; the output text has no factors to predict. At application time we simply use the same input and output languages, for example the grammatically incorrect input “we is” looks like the following, after pre-processing: The intuition behind our approach is that a multilingual shared encoder produces semantically rich latent sentence representations BIBREF14 , which provide a solid ground for the effective style transfer on top. Next we present the technical details, the experiment setup and the data we used for training the model used in the experiments. Languages and Data We use three languages in our experiments: English, Estonian and Latvian. All three have different characteristics, for example Latvian and (especially) Estonian are morphologically complex and have loose word order, while English has a strict word order and the morphology is much simpler. Most importantly, all three languages have error-corrected corpora for testing purposes, though work on their automatic grammatical error correction is extremely limited (see Section SECREF3 ). The corpora we use for training the model are OpenSubtitles2018 BIBREF15 , Europarl BIBREF16 , JRC-Acquis and EMEA BIBREF17 . We assume that there should be sufficient stylistic difference between these corpora, especially between the more informal OpenSubtitles2018 (comprised of movie and TV subtitles) on one hand and Europarl and JRC-Acquis (proceedings and documents of the European Parliament) on the other. Technical Details For Europarl, JRC-Acquis and EMEA we use all data available for English-Estonian, English-Latvian and Estonian-Latvian language pairs. From OpenSubtitles2018 we take a random subset of 3M sentence pairs for English-Estonian, which is still more than English-Latvian and Estonian-Latvian (below 1M; there we use the whole corpus). This is done to balance the corpora representation and to limit the size of training data. Details on the model hyper-parameters, data pre-processing and training can be found in Appendix SECREF7 . Evaluation First, we evaluate our model in the context of MT, as the translation quality can be expected to have influence on the other tasks that the model performs. We use public benchmarks for Estonian-English and Latvian-English translations from the news translation shared tasks of WMT 2017 and 2018 BIBREF18 , BIBREF19 . The BLEU scores for each translation direction and all included styles/domains are shown in Table TABREF6 . Some surface notes on these results: the BLEU scores for translation from and into Latvian are below English-Estonian scores, which is likely explained by smaller datasets that include Latvian. Also, translation into English has higher scores than into Estonian/Latvian, which is also expected. An interesting side-effect we have observed is the model's resilience to code-switching in the input text. The reason is that the model is trained with only the target language (and domain), and not the source language, as a result of which it learns language normalization of sorts. For example, the sentence “Ma tahan two saldējumus.” (“Ma tahan” / “I want” in Estonian, “two” and “saldējumus” / “ice-creams” in genitive, plural in Latvian) is correctly translated into English as “I want two ice creams.”. See more examples in Appendix SECREF8 . Grammatical Error Correction In this section we evaluate our model's performance in the GEC task: for example, for the English input “huge fan I are”, our model's output is “I am a huge fan”; this section's goal is to systematically check, how reliable the corrections are. Although GEC does not require any distinction in text style, the core idea of this article is to also perform style transfer with the same multilingual multi-domain model. That only means that for GEC we have to select an output domain/style when producing error corrections. Naturally, the model only copes with some kinds of errors and fails on others – for instance, word order is restored, as long as it does not affect the perception of the meaning. On the other hand, we do not expect orthographic variations like typos to be fixed reliably, since they affect the sub-word segmentation of the input and thus can hinder the translation. Below we present qualitative and quantitative analysis of our model's GEC results, showing its overall performance, as well as which kinds of errors are handled reliably and which are not. Test Data and Metrics We use the following error-corrected corpora both for scoring and as basis for manual analysis: for English: CoNLL-2014 BIBREF5 and JFLEG BIBREF20 corpora for Estonian: the Learner Language Corpus BIBREF21 for Latvian: the Error-annotated Corpus of Latvian BIBREF22 All of these are based on language learner (L2) essays and their manual corrections. To evaluate the model quantitatively we used two metrics: the Max-Match (M INLINEFORM0 ) metric from the CoNLL-2014 shared task scorer, and the GLEU score BIBREF23 for the other corpora. The main difference is that M INLINEFORM1 is based on the annotation of error categories, while the GLEU score compares the automatic correction to a reference without any error categorization. Results The M INLINEFORM0 scores are computed based on error-annotated corpora. Since error annotations were only available for English, we calculated the scores on English CoNLL corpus, see Table TABREF12 ). Our model gets the M INLINEFORM0 score of 32.1. While it does not reach the score of the best CoNLL model BIBREF24 or the state-of-the-art BIBREF25 , these use annotated corpora to train. Our results count as restricted in CoNLL definitions and are more directly comparable to the classifier-based approach trained on unannotated corpora by BIBREF26 , while requiring even less effort. The GLEU scores can be seen in Table TABREF13 . We calculated GLEU for both formal and informal style models for all three languages. For English our model's best score was 45.9 and for Estonian it was 38.1. Latvian corrected output in fact get worse scores than the original uncorrected corpus, which can be explained by smaller training corpora and worse MT quality for Latvian (see Table TABREF6 ). Qualitative Analysis We looked at the automatic corrections for 100 erroneous sentences of English and Estonian each as well as 80 sentences of Latvian. The overall aim was to find the ratio of sentences where (1) all errors have been corrected (2) only some are corrected (3) only some are corrected and part of the meaning is changed and (4) all meaning is lost. The analysis was done separately for four error types: spelling and grammatical errors, word choice and word order. In case a sentence included more that one error type it was counted once for each error type. For English the first two types were annotated in the corpus, the rest were annotated by us, separating the original third error category into two new ones. The results can be seen in Table TABREF15 . Not all English sentences included errors. 30 sentences remained unchanged, out of which 17 had no mistakes in them. For the changed sentences 87% were fully or partially corrected. In case of Estonian, where all sentences had mistakes, 61 out of the 100 sentences were fully or partially corrected without loss of information. 12 sentences became nonsense, all of which originally had some lexical mistakes. For English the results are similar: the most confusing type of errors that leads to complete loss of meaning is word choice. On the other hand, this was also the most common error type for both languages and errors of that type were fully corrected in 45% of cases for Estonian and 72% for English. Using words in the wrong order is a typical learner's error for Estonian that has rather free word order. It is also difficult to describe or set rules for this type of error. Our model manages this type rather well, correcting 79% of sentences acceptably, only losing some meaning in 2 sentences including this error type. A similar experiment using 80 Latvian sentences yielded 17 fully corrected sentences, 15, 22 and 26 respectively for the other categories. As the Latvian model is weaker in general, this also leads to more chances of losing some of the meaning; we exclude it from the more detailed analysis and focus on English and Estonian. Our model handles punctuation, word order mistakes and grammatical errors well. For example the subject-verb disagreement in English UID16 and verb-object disagreement in Estonian UID19 have been corrected. “When price of gas goes up , the consumer do not want buy gas for fuels” “When the price of gas goes up, the consumer doesn't want to buy gas for fuels” “Sellepärast ütleb ta filmi lõpus, et tahab oma unistuse tagasi” “Sellepärast ütleb ta filmi lõpus, et tahab oma unistust tagasi” that's-why says he film at-end, that (he)-wants his-own dream INLINEFORM0 Sentences that include several error types are generally noticeably more difficult to correct. Depending on the error types that have been combined our model manages quite well and corrects all or several errors present. The sentence UID23 includes mistakes with word order and word choice: the argument "vabaainetele" (to elective courses) here should precede the verb and the verb "registreeruma" (register oneself) takes no such argument. Our model corrects both mistakes while also replacing the word "seejärel" (after that) with its synonym. Seejärel pidi igaüks ennast registreeruma vabaainetele. then had-to everyone oneself register-oneself to-free-courses Siis pidi igaüks end vabaainetele registreerima. then had-to everyone oneself to-free-courses register The model fixes typos, but it mainly manages cases where two letters are needed but one is written and vice versa, for example "detailled" is corrected to "detailed" and ‘planing’ to "planning". More complicated mistakes are missed, especially if combined with other error types, and in some sentences a misspelled word is changed into an incorrect form that has a common ending, like "misundrestood" to "misundrested". The results get better if the input has been automatically spell-checked. The system does more changes than are strictly necessary and often replaces correct words and phrases, for example "frequently" was changed to to "often" or in Estonian “öelda” ("say") to “avaldada” ("publish"). Sometimes this also confuses the meaning: "siblings" was changed to "friends". To conclude this section, our model reliable corrects grammatical, spelling and word order errors on , with more mixed performance on lexical choice errors and some unnecessary paraphrasing of the input. The error types that the model manages well can be traced back to having a strong monolingual language model, a common trait of a good NMT model. As the model operates on the level of word parts and its vocabulary is limited, this leads to combining wrong word parts, sometimes across languages. This could be fixed by either using character-based NMT or doing automatic spelling correction prior to applying our current model. We limit further comparisons to two styles, translating sentences of the OpenSubtitles test set into the style of Europarl and vice versa. Our assumption is that, generally, movie subtitles gravitate towards the more informal style, and parliament proceedings towards the more formal (see examples of translations into those styles in Table TABREF30 ). Preliminary tests showed that JRC-Acquis and EMEA texts resulted in practically the same style as Europarl. We also leave Latvian out of the evaluations, assuming that its performance is weaker, similarly to GEC results. Human evaluation was performed on a subset of 100 sentences, 50 of them selected randomly from the OpenSubtitles test set and the other 50 from Europarl. Each sentence was translated into the opposite style. The resulting pairs were presented to participants, who were asked the following questions about each of them: (1) Do the sentences differ in any way? (2) How fluent is the translated sentence? (On a scale of 1 to 4, where 1 is unreadable, and 4 is perfectly fluent); (3) How similar are the sentences in meaning? (With options "exactly the same", "the same with minor changes", "more or less the same", "completely different or nonsensical"); (4) Does the translated sentence sound more formal than the original, more informal, or neither? (5) What differences are there between the sentences? (E.g. grammatical, lexical, missing words or phrases, word order, contractions, the use of formal "you"). Two such surveys were conducted, one in English and one in Estonian. 3 people participated in each of them, each of the three evaluators presented with the same set of examples. In evaluation of fluency, all three human evaluators gave the translated sentences the same score in 41 out of 55 cases in English (not taking into account sentences which were simply copied from their originals), and in 51 out of 68 cases in Estonian. In evaluation of direction of style transfer, all three evaluators agreed in 16 cases and at least two agreed in 43 cases in English, and in Estonian in 19 cases all three agreed and in 59 at least two. Of the 100 translated sentences, 45 were marked by all participants as being the same as their original sentences in the English set and 32 in Estonian. The remaining 55 and 68, respectively, were used to quantify style transfer quality. Being a reasonably strong MT system, our model scores quite high on fluency (3.84 for English, 3.64 for Estonian) and meaning preservation (3.67 for English, 3.35 for Estonian). For meaning preservation, the judgments were converted into a scale of 1-4, where 1 stands for completely different meanings or nonsensical sentences, and 4 for the exact same meaning. We evaluated the style transfer itself in the following way. For each pair of sentences, the average score given by three evaluators was calculated, in which the answer that the translated sentence is more formal counts as +1, more informal as -1, and neither as 0. We calculated the root mean square error between these scores and desired values (+1 if we aimed to make the sentence more formal, -1 if more informal). RMSE of 0 would stand for always transferring style in the right direction as judged by all evaluators, and 2 for always transferring style in the wrong direction. On the English set, the RMSE is 0.78, and on Estonian 0.89. These numbers show that style transfer generally happens in the right direction, but is not very strong. Of the 55 sentences in English that were different from their source sentences, in 33 cases the sign of the average human score matched the desired one, in 7 it did not, and in 15 no change in style was observed by humans. In Estonian 36 sentences showed the right direction of style transfer, 10 wrong, and 22 no change. In English sentences where the direction of style transfer was found to be correct (Figure FIGREF32 ), changes in use of contractions were reported in 19 cases (e.g. I have just been vs. I've just been), lexical changes in 15 cases (e.g. 'cause vs. because, or sure vs. certainly), grammatical in 13 (e.g. replacing no one's gonna with no one will, or method of production with production method), missing or added words or phrases in 8 cases. In Estonian correctly transferred sentences (Figure FIGREF32 ), the most frequently reported were lexical substitutions (30 cases), followed by missing of added words or phrases (24 cases), changes in grammar (22 cases) and in word order (16 cases). To conclude this section, unlike many style transfer models which produce text with strong style characteristics (e.g. with strong positive or negative sentiment), often at the cost of preserving meaning and fluency, our model gravitates towards keeping the meaning and fluency of the original sentence intact and mimicking some of the desired stylistic traits. Style Transfer Next we move on to evaluating the same model in terms of its performance in the context of style transfer. At first, we examined how often the sentences change when translated monolingually. The assumption is that passing modified style factors should prevent the model from simply copying the source sequences when translating inside a single language, and incentivize it to match its output to certain style characteristics typical for different corpora. Figure FIGREF28 shows the proportions of sentence pairs in the 1000-sentence test sets where there was a significant difference between translations into different styles. We can observe that English texts change less often than Estonian or Latvian, while Europarl sentences are changed more often than those of other corpora. To assess whether these changes actually correspond to the model's capability for transferring style, we turned to help of human evaluators. Cross-lingual Style Transfer Being able to translate between languages and also to modify the output to match the desired style allows the model to essentially perform domain adaptation. When translating from a language which has no formal "you" (English) into one that does (Estonian or Latvian), it will quite consistently use the informal variant when the target style is OpenSubtitles and the formal when the target style is Europarl (you rock INLINEFORM0 sa rokid/te rokite). The model is also quite consistent in use of contractions in English (es esmu šeit INLINEFORM1 I am here/I'm here). Some lexical substitutions occur: need on Matti lapsed. INLINEFORM2 those are Matt's kids./these are Matt's children. Word order may change: Where is Anna's bag? is Kus on Anna kott? in the more formal variant, and Kus Anna kott on? in the more informal. This feature is useful, but out of scope of this article, as we focus on monolingual applications. Related Work Grammatical error correction: there have been four shared tasks for GEC with prepared error-tagged datasets for L2 learners of English in the last decade: HOO BIBREF27 , BIBREF28 and CoNLL BIBREF29 , BIBREF5 . This has given an opportunity to train new models on the shared datasets and get an objective comparison of results. The general approach for grammatical error correction has been to use either rule-based approach, machine learning on error-tagged corpora, MT models on parallel data of erroneous and corrected sentences, or a combination of these BIBREF5 . The top model of the CONLL shared task in 2014 used a combined model of rule-based approach and MT BIBREF24 . All of these require annotated data or considerable effort to create, whereas our model is much more resource-independent. Another focus of the newer research is on creating GEC models without human-annotated resources. For example BIBREF26 combine statistical MT with unsupervised classification using unannotated parallel data for MT and unannotated native data for the classification model. In this case parallel data of erroneous and corrected sentences is still necessary for MT; the classifier uses native data, but still needs definitions of possible error types to classify – this work needs to be done by a human and is difficult for some less clear error types. In our approach there is no need for parallel data nor to specify error types, only for native data. There has been little work on Estonian and Latvian GEC, all limited with rule-based approaches BIBREF30 , BIBREF31 . For both languages, as well as any low-resourced languages, our approach gives a feasible way to do grammatical error correction without needing neither parallel nor error tagged corpora. Style transfer: Several approaches use directly annotated data: for example, BIBREF7 and BIBREF8 train MT systems on the corpus of modern English Shakespeare to original Shakespeare. BIBREF32 collect a dataset of 110K informal/formal sentence pairs and train rule-based, phrase-based, and neural MT systems using this data. One line of work aims at learning a style-independent latent representation of content while building decoders that can generate sentences in the style of choice BIBREF9 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 , BIBREF38 , BIBREF39 . Unsupervised MT has also been adapted for the task BIBREF10 , BIBREF40 . Our system also does not require parallel data between styles, but leverages the stability of the off-the-shelf supervised NMT to avoid the hassle of training unsupervised NMT systems and making GANs converge. Another problem with many current (both supervised and unsupervised) style transfer methods is that they are bounded to solve a binary task, where only two styles are included (whether because of data or restrictions of the approach). Our method, on the other hand, can be extended to as many styles as needed as long as there are parallel MT corpora in these styles available. Notably, BIBREF41 use side constrains in order to translate in polite/impolite German, while we rely on multilingual encoder representations and use the system monolingually at inference time. Finally, the most similar to our work conceptually is the approach of BIBREF42 , where they translate a sentence into another language, hoping that it will lose some style indicators, and then translate it back into the original language with a desired style tag attached to the encoder latent space. We also use the MT encoder to obtain rich sentence representations, but learn them directly as a part of a single multilingual translation system. Conclusions We presented a simple approach where a single multilingual NMT model is adapted to monolingual transfer and performs grammatical error correction and style transfer. We experimented with three languages and presented extensive evaluation of the model on both tasks. We used publicly available software and data and believe that our work can be easily reproduced. We showed that for GEC our approach reliably corrects spelling, word order and grammatical errors, while being less reliable on lexical choice errors. Applied to style transfer our model is very good at meaning preservation and output fluency, while reliably transferring style for English contractions, lexical choice and grammatical constructions. The main benefit is that no annotated data is used to train the model, thus making it very easy to train it for other (especially under-resourced) languages. Future work includes exploring adaptations of this approach to both tasks separately, while keeping the low cost of creating such models. Model Training: Technical Details After rudimentary cleaning (removing pairs where at least one sentence is longer that 100 tokens, at least one sentence is an empty string or does not contain any alphabetic characters, and pairs with length ratio over 9) and duplication to accommodate both translation directions in each language pair, the total size of the training corpus is 22.9M sentence pairs; training set sizes per language and corpus are given in Table TABREF36 . Validation set consists of 12K sentence pairs, 500 for each combination of translation direction and corpus. We also keep a test set of 24K sentence pairs, 1000 for each translation direction and corpus. The data preprocessing pipeline consists of tokenization with Moses tokenizer BIBREF44 , true-casing, and segmentation with SentencePiece BIBREF45 with a joint vocabulary of size 32 000. We trained a Transformer NMT model using the Sockeye framework BIBREF43 , mostly following the so-called Transformer base model: we used 6 layers, 512 positions, 8 attention heads and ReLU activations for both the encoder and decoder; Adam optimizer was used. Source and target token embeddings were both of size 512, and factors determining target language and style had embeddings of size 4. Batch size was set to 2048 words, initial learning rate to 0.0002, reducing by a factor of 0.7 every time the validation perplexity had not improved for 8 checkpoints, which happened every 4000 updates. The model converged during the 17th epoch, when validation perplexity has not improved for 32 consecutive checkpoints. The parameters of a single best checkpoint were used for all translations, with beam size set to 5. Output Examples We present more examples of translation of code-switched input segments, error correction and style transfer in English, Estonian and Latvian, informal (inf) and formal (fml) output style:
What error types is their model more reliable for?
62ea141d0fb342dfb97c69b49d1c978665b93b3c
What Ġerror Ġtypes Ġis Ġtheir Ġmodel Ġmore Ġreliable Ġfor ?
gram matical , Ġspelling Ġand Ġword Ġorder Ġerrors
<s> What Ġerror Ġtypes Ġis Ġtheir Ġmodel Ġmore Ġreliable Ġfor ? </s> Introduction </s> Sequ ence - to - sequence Ġ( seq 2 seq ) Ġtransformations Ġhave Ġrecently Ġproven Ġto Ġbe Ġa Ġsuccessful Ġframework Ġfor Ġseveral Ġnatural Ġlanguage Ġprocessing Ġtasks , Ġlike : Ġmachine Ġtranslation Ġ( MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġspeech Ġrecognition ĠB IB REF 2 Ġ, Ġspeech Ġsynthesis ĠB IB REF 3 Ġ, Ġnatural Ġlanguage Ġinference ĠB IB REF 4 Ġand Ġothers . ĠHowever , Ġthe Ġsuccess Ġof Ġthese Ġmodels Ġdepends Ġon Ġthe Ġavailability Ġof Ġlarge Ġamounts Ġof Ġdirectly Ġannot ated Ġdata Ġfor Ġthe Ġtask Ġat Ġhand Ġ( like Ġtranslation Ġexamples , Ġtext Ġsegments Ġand Ġtheir Ġspeech Ġrecordings , Ġetc .). ĠThis Ġis Ġa Ġsevere Ġlimitation Ġfor Ġtasks Ġwhere Ġdata Ġis Ġnot Ġabundantly Ġavailable Ġas Ġwell Ġas Ġfor Ġlow - resource Ġlanguages . </s> Here Ġwe Ġfocus Ġon Ġtwo Ġsuch Ġtasks : Ġgram matical Ġerror Ġcorrection Ġ( G EC ) Ġand Ġstyle Ġtransfer . ĠModern Ġapproaches Ġto ĠG EC Ġlearn Ġfrom Ġparallel Ġcorpor a Ġof Ġerroneous Ġsegments Ġand Ġtheir Ġmanual Ġcorrections ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ; Ġtext Ġstyle Ġtransfer Ġalso Ġrelies Ġon Ġsupervised Ġapproaches Ġthat Ġrequire Ġtexts Ġof Ġthe Ġsame Ġmeaning Ġand Ġdifferent Ġstyles ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġor Ġimp rec ise Ġun super vised Ġmethods ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> In Ġthis Ġpaper Ġwe Ġintroduce Ġan Ġapproach Ġto Ġperforming Ġboth ĠG EC Ġand Ġstyle Ġtransfer Ġwith Ġthe Ġsame Ġtrained Ġmodel , Ġwhile Ġnot Ġusing Ġany Ġsupervised Ġtraining Ġdata Ġfor Ġeither Ġtask . ĠIt Ġis Ġbased Ġon Ġzero - shot Ġneural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 11 Ġ, Ġand Ġas Ġsuch , Ġthe Ġonly Ġkind Ġof Ġdata Ġit Ġuses Ġis Ġregular Ġparallel Ġcorpor a Ġ( with Ġtexts Ġand Ġtheir Ġtranslations ). ĠHowever , Ġwe Ġapply Ġthe Ġmodel Ġto Ġdo Ġmon oling ual Ġtransfer , Ġasking Ġto Ġtranslate Ġthe Ġinput Ġsegment Ġinto Ġthe Ġsame Ġlanguage . ĠWe Ġshow , Ġthat Ġthis ĠâĢ ľ mon oling ual Ġtranslation âĢ Ŀ Ġis Ġwhat Ġenables Ġthe Ġmodel Ġto Ġcorrect Ġthe Ġerrors Ġin Ġthe Ġinput Ġas Ġwell Ġas Ġadapt Ġthe Ġoutput Ġinto Ġa Ġdesired Ġstyle . ĠMoreover , Ġthe Ġsame Ġtrained Ġmodel Ġperforms Ġboth Ġtasks Ġon Ġseveral Ġlanguages . </s> Our Ġmain Ġcontributions Ġare Ġthus : Ġ( i ) Ġa Ġsingle Ġmethod Ġfor Ġboth Ġstyle Ġtransfer Ġand Ġgram matical Ġerror Ġcorrection , Ġwithout Ġusing Ġannot ated Ġdata Ġfor Ġeither Ġtask , Ġ( ii ) Ġsupport Ġfor Ġboth Ġtasks Ġon Ġmultiple Ġlanguages Ġwithin Ġthe Ġsame Ġmodel , Ġ( iii ) Ġa Ġthorough Ġquantitative Ġand Ġqualitative Ġmanual Ġevaluation Ġof Ġthe Ġmodel Ġon Ġboth Ġtasks , Ġand Ġ( iv ) Ġhighlighting Ġof Ġthe Ġmodel 's Ġreliability Ġaspects Ġon Ġboth Ġtasks . ĠWe Ġused Ġpublicly Ġavailable Ġsoftware Ġand Ġdata ; Ġan Ġonline Ġdemo Ġof Ġour Ġresults Ġis Ġavailable , Ġbut Ġconcealed Ġfor Ġanonym ization Ġpurposes . </s> We Ġdescribe Ġthe Ġdetails Ġof Ġour Ġapproach Ġin ĠSection ĠSEC REF 2 Ġ, Ġthen Ġevaluate Ġit Ġin Ġterms Ġof Ġperformance Ġin Ġgram matical Ġerror Ġcorrection Ġin ĠSection ĠSEC REF 3 Ġand Ġin Ġstyle Ġtransfer Ġin ĠSection ĠSEC REF 4 Ġ. ĠThe Ġpaper Ġends Ġwith Ġa Ġreview Ġof Ġrelated Ġwork Ġin ĠSection ĠSEC REF 5 Ġand Ġconclusions Ġin ĠSection ĠSEC REF 6 Ġ. </s> Method </s> As Ġmentioned Ġin Ġthe Ġintroduction , Ġour Ġapproach Ġis Ġbased Ġon Ġthe Ġidea Ġof Ġzero - shot ĠMT ĠB IB REF 11 Ġ. ĠThere Ġthe Ġauthors Ġshow Ġthat Ġafter Ġtraining Ġa Ġsingle Ġmodel Ġto Ġtranslate Ġfrom ĠPortuguese Ġto ĠEnglish Ġas Ġwell Ġas Ġfrom ĠEnglish Ġto ĠSpanish , Ġit Ġcan Ġalso Ġtranslate ĠPortuguese Ġinto ĠSpanish , Ġwithout Ġseeing Ġany Ġtranslation Ġexamples Ġfor Ġthis Ġlanguage Ġpair . ĠWe Ġuse Ġthe Ġzero - shot Ġeffect Ġto Ġachieve Ġmon oling ual Ġtranslation Ġby Ġtraining Ġthe Ġmodel Ġon Ġbilingual Ġexamples Ġin Ġboth Ġdirections , Ġand Ġthen Ġdoing Ġtranslation Ġinto Ġthe Ġsame Ġlanguage Ġas Ġthe Ġinput : Ġillustrated Ġon ĠFigure ĠFIG REF 1 Ġ. </s> With Ġregular Ġsentences Ġmon oling ual Ġtranslation Ġdoes Ġnot Ġseem Ġuseful , Ġas Ġits Ġbehaviour Ġmainly Ġconsists Ġof Ġcopying . ĠHowever , Ġwhen Ġthe Ġinput Ġsentence Ġhas Ġcharacteristics Ġunseen Ġor Ġrarely Ġseen Ġby Ġthe Ġmodel Ġat Ġtraining Ġtime Ġ( like Ġgram matical Ġerrors Ġor Ġdifferent Ġstyl istic Ġchoices ) ĠâĢĵ Ġthe Ġdec oder Ġstill Ġgenerates Ġthe Ġmore Ġregular Ġversion Ġof Ġthe Ġsentence Ġ( thus Ġfixing Ġthe Ġerrors Ġor Ġadapting Ġthe Ġstyle ). ĠFurthermore , Ġin Ġcase Ġof Ġmult ilingual Ġmulti - domain ĠN MT ĠB IB REF 12 Ġ, Ġit Ġis Ġpossible Ġto Ġswitch Ġbetween Ġdifferent Ġdomains Ġor Ġstyles Ġat Ġruntime , Ġthus Ġperforming ĠâĢ ľ mon oling ual Ġdomain Ġadaptation âĢ Ŀ Ġor Ġstyle Ġtransfer . </s> To Ġcreate Ġa Ġmult ilingual Ġmulti - domain ĠN MT Ġsystem Ġwe Ġuse Ġthe Ġself - att ention Ġarchitecture ĠB IB REF 13 Ġ. ĠInstead Ġof Ġspecifying Ġthe Ġoutput Ġlanguage Ġwith Ġa Ġtoken Ġinside Ġthe Ġinput Ġsequence , Ġas ĠB IB REF 11 Ġdid , Ġwe Ġfollow ĠB IB REF 12 Ġand Ġuse Ġword Ġfeatures Ġ( or Ġfactors ). ĠOn Ġone Ġhand , Ġthis Ġprovides Ġa Ġstronger Ġsignal Ġfor Ġthe Ġmodel , Ġand Ġon Ġthe Ġother ĠâĢĵ Ġallows Ġfor Ġadditional Ġparam et riz ation , Ġwhich Ġin Ġour Ġcase Ġis Ġthe Ġtext Ġdomain / style Ġof Ġthe Ġcorpus . </s> As Ġa Ġresult , Ġa Ġpre - process ed ĠEnglish - Lat v ian Ġtraining Ġset Ġsentence Ġpair ĠâĢ ľ Hello ! âĢ Ŀ âĢĵ âĢ ľ S ve iki ! âĢ Ŀ Ġlooks Ġlike : </s> Here Ġ2 lv Ġand Ġ2 os Ġspecify ĠLat v ian Ġand ĠOpen Sub t itles Ġas Ġthe Ġoutput Ġlanguage Ġand Ġdomain ; Ġthe Ġoutput Ġtext Ġhas Ġno Ġfactors Ġto Ġpredict . ĠAt Ġapplication Ġtime Ġwe Ġsimply Ġuse Ġthe Ġsame Ġinput Ġand Ġoutput Ġlanguages , Ġfor Ġexample Ġthe Ġgram matically Ġincorrect Ġinput ĠâĢ ľ we Ġis âĢ Ŀ Ġlooks Ġlike Ġthe Ġfollowing , Ġafter Ġpre - processing : </s> The Ġintuition Ġbehind Ġour Ġapproach Ġis Ġthat Ġa Ġmult ilingual Ġshared Ġenc oder Ġproduces Ġsem antically Ġrich Ġlatent Ġsentence Ġrepresentations ĠB IB REF 14 Ġ, Ġwhich Ġprovide Ġa Ġsolid Ġground Ġfor Ġthe Ġeffective Ġstyle Ġtransfer Ġon Ġtop . </s> Next Ġwe Ġpresent Ġthe Ġtechnical Ġdetails , Ġthe Ġexperiment Ġsetup Ġand Ġthe Ġdata Ġwe Ġused Ġfor Ġtraining Ġthe Ġmodel Ġused Ġin Ġthe Ġexperiments . </s> L anguages Ġand ĠData </s> We Ġuse Ġthree Ġlanguages Ġin Ġour Ġexperiments : ĠEnglish , ĠEston ian Ġand ĠLat v ian . ĠAll Ġthree Ġhave Ġdifferent Ġcharacteristics , Ġfor Ġexample ĠLat v ian Ġand Ġ( especially ) ĠEston ian Ġare Ġmorph ologically Ġcomplex Ġand Ġhave Ġloose Ġword Ġorder , Ġwhile ĠEnglish Ġhas Ġa Ġstrict Ġword Ġorder Ġand Ġthe Ġmorphology Ġis Ġmuch Ġsimpler . ĠMost Ġimportantly , Ġall Ġthree Ġlanguages Ġhave Ġerror - correct ed Ġcorpor a Ġfor Ġtesting Ġpurposes , Ġthough Ġwork Ġon Ġtheir Ġautomatic Ġgram matical Ġerror Ġcorrection Ġis Ġextremely Ġlimited Ġ( see ĠSection ĠSEC REF 3 Ġ). </s> The Ġcorpor a Ġwe Ġuse Ġfor Ġtraining Ġthe Ġmodel Ġare ĠOpen Sub t itles 2018 ĠB IB REF 15 Ġ, ĠEuro par l ĠB IB REF 16 Ġ, ĠJ RC - Ac qu is Ġand ĠE ME A ĠB IB REF 17 Ġ. ĠWe Ġassume Ġthat Ġthere Ġshould Ġbe Ġsufficient Ġstyl istic Ġdifference Ġbetween Ġthese Ġcorpor a , Ġespecially Ġbetween Ġthe Ġmore Ġinformal ĠOpen Sub t itles 2018 Ġ( com pr ised Ġof Ġmovie Ġand ĠTV Ġsubtitles ) Ġon Ġone Ġhand Ġand ĠEuro par l Ġand ĠJ RC - Ac qu is Ġ( pro ceed ings Ġand Ġdocuments Ġof Ġthe ĠEuropean ĠParliament ) Ġon Ġthe Ġother . </s> Technical ĠDetails </s> For ĠEuro par l , ĠJ RC - Ac qu is Ġand ĠE ME A Ġwe Ġuse Ġall Ġdata Ġavailable Ġfor ĠEnglish - E ston ian , ĠEnglish - Lat v ian Ġand ĠEston ian - Lat v ian Ġlanguage Ġpairs . ĠFrom ĠOpen Sub t itles 2018 Ġwe Ġtake Ġa Ġrandom Ġsubset Ġof Ġ3 M Ġsentence Ġpairs Ġfor ĠEnglish - E ston ian , Ġwhich Ġis Ġstill Ġmore Ġthan ĠEnglish - Lat v ian Ġand ĠEston ian - Lat v ian Ġ( below Ġ1 M ; Ġthere Ġwe Ġuse Ġthe Ġwhole Ġcorpus ). ĠThis Ġis Ġdone Ġto Ġbalance Ġthe Ġcorpor a Ġrepresentation Ġand Ġto Ġlimit Ġthe Ġsize Ġof Ġtraining Ġdata . </s> Details Ġon Ġthe Ġmodel Ġhyper - param eters , Ġdata Ġpre - processing Ġand Ġtraining Ġcan Ġbe Ġfound Ġin ĠAppendix ĠSEC REF 7 Ġ. </s> E val uation </s> First , Ġwe Ġevaluate Ġour Ġmodel Ġin Ġthe Ġcontext Ġof ĠMT , Ġas Ġthe Ġtranslation Ġquality Ġcan Ġbe Ġexpected Ġto Ġhave Ġinfluence Ġon Ġthe Ġother Ġtasks Ġthat Ġthe Ġmodel Ġperforms . ĠWe Ġuse Ġpublic Ġbenchmarks Ġfor ĠEston ian - English Ġand ĠLat v ian - English Ġtranslations Ġfrom Ġthe Ġnews Ġtranslation Ġshared Ġtasks Ġof ĠW MT Ġ2017 Ġand Ġ2018 ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThe ĠB LE U Ġscores Ġfor Ġeach Ġtranslation Ġdirection Ġand Ġall Ġincluded Ġstyles / dom ains Ġare Ġshown Ġin ĠTable ĠT AB REF 6 Ġ. </s> Some Ġsurface Ġnotes Ġon Ġthese Ġresults : Ġthe ĠB LE U Ġscores Ġfor Ġtranslation Ġfrom Ġand Ġinto ĠLat v ian Ġare Ġbelow ĠEnglish - E ston ian Ġscores , Ġwhich Ġis Ġlikely Ġexplained Ġby Ġsmaller Ġdatasets Ġthat Ġinclude ĠLat v ian . ĠAlso , Ġtranslation Ġinto ĠEnglish Ġhas Ġhigher Ġscores Ġthan Ġinto ĠEston ian / Lat v ian , Ġwhich Ġis Ġalso Ġexpected . </s> An Ġinteresting Ġside - effect Ġwe Ġhave Ġobserved Ġis Ġthe Ġmodel 's Ġresilience Ġto Ġcode - sw itching Ġin Ġthe Ġinput Ġtext . ĠThe Ġreason Ġis Ġthat Ġthe Ġmodel Ġis Ġtrained Ġwith Ġonly Ġthe Ġtarget Ġlanguage Ġ( and Ġdomain ), Ġand Ġnot Ġthe Ġsource Ġlanguage , Ġas Ġa Ġresult Ġof Ġwhich Ġit Ġlearns Ġlanguage Ġnormal ization Ġof Ġsorts . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ Ma Ġt ahan Ġtwo Ġs ald Äĵ j um us . âĢ Ŀ Ġ( âĢ ľ Ma Ġt ahan âĢ Ŀ Ġ/ ĠâĢ ľ I Ġwant âĢ Ŀ Ġin ĠEston ian , ĠâĢ ľ two âĢ Ŀ Ġand ĠâĢ ľ s ald Äĵ j um us âĢ Ŀ Ġ/ ĠâĢ ľ ice - cream s âĢ Ŀ Ġin Ġgen itive , Ġplural Ġin ĠLat v ian ) Ġis Ġcorrectly Ġtranslated Ġinto ĠEnglish Ġas ĠâĢ ľ I Ġwant Ġtwo Ġice Ġcre ams . âĢ Ŀ . ĠSee Ġmore Ġexamples Ġin ĠAppendix ĠSEC REF 8 Ġ. </s> G ram matical ĠError ĠCorrection </s> In Ġthis Ġsection Ġwe Ġevaluate Ġour Ġmodel 's Ġperformance Ġin Ġthe ĠG EC Ġtask : Ġfor Ġexample , Ġfor Ġthe ĠEnglish Ġinput ĠâĢ ľ huge Ġfan ĠI Ġare âĢ Ŀ , Ġour Ġmodel 's Ġoutput Ġis ĠâĢ ľ I Ġam Ġa Ġhuge Ġfan âĢ Ŀ ; Ġthis Ġsection 's Ġgoal Ġis Ġto Ġsystematically Ġcheck , Ġhow Ġreliable Ġthe Ġcorrections Ġare . </s> Although ĠG EC Ġdoes Ġnot Ġrequire Ġany Ġdistinction Ġin Ġtext Ġstyle , Ġthe Ġcore Ġidea Ġof Ġthis Ġarticle Ġis Ġto Ġalso Ġperform Ġstyle Ġtransfer Ġwith Ġthe Ġsame Ġmult ilingual Ġmulti - domain Ġmodel . ĠThat Ġonly Ġmeans Ġthat Ġfor ĠG EC Ġwe Ġhave Ġto Ġselect Ġan Ġoutput Ġdomain / style Ġwhen Ġproducing Ġerror Ġcorrections . </s> Naturally , Ġthe Ġmodel Ġonly Ġcop es Ġwith Ġsome Ġkinds Ġof Ġerrors Ġand Ġfails Ġon Ġothers ĠâĢĵ Ġfor Ġinstance , Ġword Ġorder Ġis Ġrestored , Ġas Ġlong Ġas Ġit Ġdoes Ġnot Ġaffect Ġthe Ġperception Ġof Ġthe Ġmeaning . ĠOn Ġthe Ġother Ġhand , Ġwe Ġdo Ġnot Ġexpect Ġorth ographic Ġvariations Ġlike Ġty pos Ġto Ġbe Ġfixed Ġreliably , Ġsince Ġthey Ġaffect Ġthe Ġsub - word Ġsegment ation Ġof Ġthe Ġinput Ġand Ġthus Ġcan Ġhinder Ġthe Ġtranslation . </s> Below Ġwe Ġpresent Ġqualitative Ġand Ġquantitative Ġanalysis Ġof Ġour Ġmodel 's ĠG EC Ġresults , Ġshowing Ġits Ġoverall Ġperformance , Ġas Ġwell Ġas Ġwhich Ġkinds Ġof Ġerrors Ġare Ġhandled Ġreliably Ġand Ġwhich Ġare Ġnot . </s> Test ĠData Ġand ĠMet rics </s> We Ġuse Ġthe Ġfollowing Ġerror - correct ed Ġcorpor a Ġboth Ġfor Ġscoring Ġand Ġas Ġbasis Ġfor Ġmanual Ġanalysis : </s> for ĠEnglish : ĠCo N LL - 2014 ĠB IB REF 5 Ġand ĠJ F LE G ĠB IB REF 20 Ġcorpor a </s> for ĠEston ian : Ġthe ĠLear ner ĠLanguage ĠCorpus ĠB IB REF 21 </s> for ĠLat v ian : Ġthe ĠError - annot ated ĠCorpus Ġof ĠLat v ian ĠB IB REF 22 </s> All Ġof Ġthese Ġare Ġbased Ġon Ġlanguage Ġlear ner Ġ( L 2 ) Ġessays Ġand Ġtheir Ġmanual Ġcorrections . </s> To Ġevaluate Ġthe Ġmodel Ġquant itatively Ġwe Ġused Ġtwo Ġmetrics : Ġthe ĠMax - Match Ġ( M ĠIN LINE FORM 0 Ġ) Ġmetric Ġfrom Ġthe ĠCo N LL - 2014 Ġshared Ġtask Ġscorer , Ġand Ġthe ĠG LE U Ġscore ĠB IB REF 23 Ġfor Ġthe Ġother Ġcorpor a . ĠThe Ġmain Ġdifference Ġis Ġthat ĠM ĠIN LINE FORM 1 Ġis Ġbased Ġon Ġthe Ġannotation Ġof Ġerror Ġcategories , Ġwhile Ġthe ĠG LE U Ġscore Ġcompares Ġthe Ġautomatic Ġcorrection Ġto Ġa Ġreference Ġwithout Ġany Ġerror Ġcategor ization . </s> Results </s> The ĠM ĠIN LINE FORM 0 Ġscores Ġare Ġcomputed Ġbased Ġon Ġerror - annot ated Ġcorpor a . ĠSince Ġerror Ġannotations Ġwere Ġonly Ġavailable Ġfor ĠEnglish , Ġwe Ġcalculated Ġthe Ġscores Ġon ĠEnglish ĠCo N LL Ġcorpus , Ġsee ĠTable ĠT AB REF 12 Ġ). </s> Our Ġmodel Ġgets Ġthe ĠM ĠIN LINE FORM 0 Ġscore Ġof Ġ32 . 1 . ĠWhile Ġit Ġdoes Ġnot Ġreach Ġthe Ġscore Ġof Ġthe Ġbest ĠCo N LL Ġmodel ĠB IB REF 24 Ġor Ġthe Ġstate - of - the - art ĠB IB REF 25 Ġ, Ġthese Ġuse Ġannot ated Ġcorpor a Ġto Ġtrain . ĠOur Ġresults Ġcount Ġas Ġrestricted Ġin ĠCo N LL Ġdefinitions Ġand Ġare Ġmore Ġdirectly Ġcomparable Ġto Ġthe Ġclass ifier - based Ġapproach Ġtrained Ġon Ġun annot ated Ġcorpor a Ġby ĠB IB REF 26 Ġ, Ġwhile Ġrequiring Ġeven Ġless Ġeffort . </s> The ĠG LE U Ġscores Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 13 Ġ. ĠWe Ġcalculated ĠG LE U Ġfor Ġboth Ġformal Ġand Ġinformal Ġstyle Ġmodels Ġfor Ġall Ġthree Ġlanguages . ĠFor ĠEnglish Ġour Ġmodel 's Ġbest Ġscore Ġwas Ġ45 . 9 Ġand Ġfor ĠEston ian Ġit Ġwas Ġ38 . 1 . ĠLat v ian Ġcorrected Ġoutput Ġin Ġfact Ġget Ġworse Ġscores Ġthan Ġthe Ġoriginal Ġunc orrect ed Ġcorpus , Ġwhich Ġcan Ġbe Ġexplained Ġby Ġsmaller Ġtraining Ġcorpor a Ġand Ġworse ĠMT Ġquality Ġfor ĠLat v ian Ġ( see ĠTable ĠT AB REF 6 Ġ). </s> Qual itative ĠAnalysis </s> We Ġlooked Ġat Ġthe Ġautomatic Ġcorrections Ġfor Ġ100 Ġerroneous Ġsentences Ġof ĠEnglish Ġand ĠEston ian Ġeach Ġas Ġwell Ġas Ġ80 Ġsentences Ġof ĠLat v ian . ĠThe Ġoverall Ġaim Ġwas Ġto Ġfind Ġthe Ġratio Ġof Ġsentences Ġwhere Ġ( 1 ) Ġall Ġerrors Ġhave Ġbeen Ġcorrected Ġ( 2 ) Ġonly Ġsome Ġare Ġcorrected Ġ( 3 ) Ġonly Ġsome Ġare Ġcorrected Ġand Ġpart Ġof Ġthe Ġmeaning Ġis Ġchanged Ġand Ġ( 4 ) Ġall Ġmeaning Ġis Ġlost . </s> The Ġanalysis Ġwas Ġdone Ġseparately Ġfor Ġfour Ġerror Ġtypes : Ġspelling Ġand Ġgram matical Ġerrors , Ġword Ġchoice Ġand Ġword Ġorder . ĠIn Ġcase Ġa Ġsentence Ġincluded Ġmore Ġthat Ġone Ġerror Ġtype Ġit Ġwas Ġcounted Ġonce Ġfor Ġeach Ġerror Ġtype . ĠFor ĠEnglish Ġthe Ġfirst Ġtwo Ġtypes Ġwere Ġannot ated Ġin Ġthe Ġcorpus , Ġthe Ġrest Ġwere Ġannot ated Ġby Ġus , Ġseparating Ġthe Ġoriginal Ġthird Ġerror Ġcategory Ġinto Ġtwo Ġnew Ġones . ĠThe Ġresults Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 15 Ġ. </s> Not Ġall ĠEnglish Ġsentences Ġincluded Ġerrors . Ġ30 Ġsentences Ġremained Ġunchanged , Ġout Ġof Ġwhich Ġ17 Ġhad Ġno Ġmistakes Ġin Ġthem . ĠFor Ġthe Ġchanged Ġsentences Ġ87 % Ġwere Ġfully Ġor Ġpartially Ġcorrected . ĠIn Ġcase Ġof ĠEston ian , Ġwhere Ġall Ġsentences Ġhad Ġmistakes , Ġ61 Ġout Ġof Ġthe Ġ100 Ġsentences Ġwere Ġfully Ġor Ġpartially Ġcorrected Ġwithout Ġloss Ġof Ġinformation . Ġ12 Ġsentences Ġbecame Ġnonsense , Ġall Ġof Ġwhich Ġoriginally Ġhad Ġsome Ġlex ical Ġmistakes . ĠFor ĠEnglish Ġthe Ġresults Ġare Ġsimilar : Ġthe Ġmost Ġconfusing Ġtype Ġof Ġerrors Ġthat Ġleads Ġto Ġcomplete Ġloss Ġof Ġmeaning Ġis Ġword Ġchoice . ĠOn Ġthe Ġother Ġhand , Ġthis Ġwas Ġalso Ġthe Ġmost Ġcommon Ġerror Ġtype Ġfor Ġboth Ġlanguages Ġand Ġerrors Ġof Ġthat Ġtype Ġwere Ġfully Ġcorrected Ġin Ġ45 % Ġof Ġcases Ġfor ĠEston ian Ġand Ġ72 % Ġfor ĠEnglish . ĠUsing Ġwords Ġin Ġthe Ġwrong Ġorder Ġis Ġa Ġtypical Ġlear ner 's Ġerror Ġfor ĠEston ian Ġthat Ġhas Ġrather Ġfree Ġword Ġorder . ĠIt Ġis Ġalso Ġdifficult Ġto Ġdescribe Ġor Ġset Ġrules Ġfor Ġthis Ġtype Ġof Ġerror . ĠOur Ġmodel Ġmanages Ġthis Ġtype Ġrather Ġwell , Ġcorrecting Ġ79 % Ġof Ġsentences Ġaccept ably , Ġonly Ġlosing Ġsome Ġmeaning Ġin Ġ2 Ġsentences Ġincluding Ġthis Ġerror Ġtype . </s> A Ġsimilar Ġexperiment Ġusing Ġ80 ĠLat v ian Ġsentences Ġyielded Ġ17 Ġfully Ġcorrected Ġsentences , Ġ15 , Ġ22 Ġand Ġ26 Ġrespectively Ġfor Ġthe Ġother Ġcategories . ĠAs Ġthe ĠLat v ian Ġmodel Ġis Ġweaker Ġin Ġgeneral , Ġthis Ġalso Ġleads Ġto Ġmore Ġchances Ġof Ġlosing Ġsome Ġof Ġthe Ġmeaning ; Ġwe Ġexclude Ġit Ġfrom Ġthe Ġmore Ġdetailed Ġanalysis Ġand Ġfocus Ġon ĠEnglish Ġand ĠEston ian . </s> Our Ġmodel Ġhandles Ġpunct uation , Ġword Ġorder Ġmistakes Ġand Ġgram matical Ġerrors Ġwell . ĠFor Ġexample Ġthe Ġsubject - verb Ġdisagreement Ġin ĠEnglish ĠUID 16 Ġand Ġverb - object Ġdisagreement Ġin ĠEston ian ĠUID 19 Ġhave Ġbeen Ġcorrected . </s> âĢ ľ When Ġprice Ġof Ġgas Ġgoes Ġup Ġ, Ġthe Ġconsumer Ġdo Ġnot Ġwant Ġbuy Ġgas Ġfor Ġfuels âĢ Ŀ </s> âĢ ľ When Ġthe Ġprice Ġof Ġgas Ġgoes Ġup , Ġthe Ġconsumer Ġdoesn 't Ġwant Ġto Ġbuy Ġgas Ġfor Ġfuels âĢ Ŀ </s> âĢ ľ S elle p Ã¤ rast ĠÃ ¼ tle b Ġta Ġfilm i Ġl Ã µ p us , Ġet Ġt ah ab Ġo ma Ġun ist use Ġtag asi âĢ Ŀ </s> âĢ ľ S elle p Ã¤ rast ĠÃ ¼ tle b Ġta Ġfilm i Ġl Ã µ p us , Ġet Ġt ah ab Ġo ma Ġun ist ust Ġtag asi âĢ Ŀ </s> that 's - why Ġsays Ġhe Ġfilm Ġat - end , Ġthat Ġ( he )- w ants Ġhis - own Ġdream ĠIN LINE FORM 0 </s> Sent ences Ġthat Ġinclude Ġseveral Ġerror Ġtypes Ġare Ġgenerally Ġnoticeably Ġmore Ġdifficult Ġto Ġcorrect . ĠDepending Ġon Ġthe Ġerror Ġtypes Ġthat Ġhave Ġbeen Ġcombined Ġour Ġmodel Ġmanages Ġquite Ġwell Ġand Ġcorrect s Ġall Ġor Ġseveral Ġerrors Ġpresent . ĠThe Ġsentence ĠUID 23 Ġincludes Ġmistakes Ġwith Ġword Ġorder Ġand Ġword Ġchoice : Ġthe Ġargument Ġ" v aba ain et ele " Ġ( to Ġelect ive Ġcourses ) Ġhere Ġshould Ġpreced e Ġthe Ġverb Ġand Ġthe Ġverb Ġ" reg ist re er uma " Ġ( register Ġoneself ) Ġtakes Ġno Ġsuch Ġargument . ĠOur Ġmodel Ġcorrect s Ġboth Ġmistakes Ġwhile Ġalso Ġreplacing Ġthe Ġword Ġ" see j Ã¤ rel " Ġ( after Ġthat ) Ġwith Ġits Ġsyn onym . </s> See j Ã¤ rel Ġp idi Ġ iga Ã¼ ks Ġen n ast Ġregist re er uma Ġv aba ain et ele . </s> then Ġhad - to Ġeveryone Ġoneself Ġregister - ones elf Ġto - free - c ourses </s> Si is Ġp idi Ġ iga Ã¼ ks Ġend Ġv aba ain et ele Ġregist re er ima . </s> then Ġhad - to Ġeveryone Ġoneself Ġto - free - c ourses Ġregister </s> The Ġmodel Ġfixes Ġty pos , Ġbut Ġit Ġmainly Ġmanages Ġcases Ġwhere Ġtwo Ġletters Ġare Ġneeded Ġbut Ġone Ġis Ġwritten Ġand Ġvice Ġversa , Ġfor Ġexample Ġ" det a illed " Ġis Ġcorrected Ġto Ġ" det ailed " Ġand ĠâĢ ĺ pl aning âĢ Ļ Ġto Ġ" plan ning ". ĠMore Ġcomplicated Ġmistakes Ġare Ġmissed , Ġespecially Ġif Ġcombined Ġwith Ġother Ġerror Ġtypes , Ġand Ġin Ġsome Ġsentences Ġa Ġmiss pelled Ġword Ġis Ġchanged Ġinto Ġan Ġincorrect Ġform Ġthat Ġhas Ġa Ġcommon Ġending , Ġlike Ġ" mis und rest ood " Ġto Ġ" mis und rest ed ". ĠThe Ġresults Ġget Ġbetter Ġif Ġthe Ġinput Ġhas Ġbeen Ġautomatically Ġspell - checked . </s> The Ġsystem Ġdoes Ġmore Ġchanges Ġthan Ġare Ġstrictly Ġnecessary Ġand Ġoften Ġreplaces Ġcorrect Ġwords Ġand Ġphrases , Ġfor Ġexample Ġ" f requently " Ġwas Ġchanged Ġto Ġto Ġ" often " Ġor Ġin ĠEston ian ĠâĢ ľ Ã¶ e lda âĢ Ŀ Ġ(" say ") Ġto ĠâĢ ľ av ald ada âĢ Ŀ Ġ(" pub lish "). ĠSometimes Ġthis Ġalso Ġconf uses Ġthe Ġmeaning : Ġ" s iblings " Ġwas Ġchanged Ġto Ġ" friends ". </s> To Ġconclude Ġthis Ġsection , Ġour Ġmodel Ġreliable Ġcorrect s Ġgram matical , Ġspelling Ġand Ġword Ġorder Ġerrors Ġon Ġ, Ġwith Ġmore Ġmixed Ġperformance Ġon Ġlex ical Ġchoice Ġerrors Ġand Ġsome Ġunnecessary Ġparaph r asing Ġof Ġthe Ġinput . ĠThe Ġerror Ġtypes Ġthat Ġthe Ġmodel Ġmanages Ġwell Ġcan Ġbe Ġtraced Ġback Ġto Ġhaving Ġa Ġstrong Ġmon oling ual Ġlanguage Ġmodel , Ġa Ġcommon Ġtrait Ġof Ġa Ġgood ĠN MT Ġmodel . ĠAs Ġthe Ġmodel Ġoperates Ġon Ġthe Ġlevel Ġof Ġword Ġparts Ġand Ġits Ġvocabulary Ġis Ġlimited , Ġthis Ġleads Ġto Ġcombining Ġwrong Ġword Ġparts , Ġsometimes Ġacross Ġlanguages . ĠThis Ġcould Ġbe Ġfixed Ġby Ġeither Ġusing Ġcharacter - based ĠN MT Ġor Ġdoing Ġautomatic Ġspelling Ġcorrection Ġprior Ġto Ġapplying Ġour Ġcurrent Ġmodel . </s> We Ġlimit Ġfurther Ġcomparisons Ġto Ġtwo Ġstyles , Ġtranslating Ġsentences Ġof Ġthe ĠOpen Sub t itles Ġtest Ġset Ġinto Ġthe Ġstyle Ġof ĠEuro par l Ġand Ġvice Ġversa . ĠOur Ġassumption Ġis Ġthat , Ġgenerally , Ġmovie Ġsubtitles Ġgrav itate Ġtowards Ġthe Ġmore Ġinformal Ġstyle , Ġand Ġparliament Ġproceedings Ġtowards Ġthe Ġmore Ġformal Ġ( see Ġexamples Ġof Ġtranslations Ġinto Ġthose Ġstyles Ġin ĠTable ĠT AB REF 30 Ġ). ĠPrel iminary Ġtests Ġshowed Ġthat ĠJ RC - Ac qu is Ġand ĠE ME A Ġtexts Ġresulted Ġin Ġpractically Ġthe Ġsame Ġstyle Ġas ĠEuro par l . ĠWe Ġalso Ġleave ĠLat v ian Ġout Ġof Ġthe Ġevaluations , Ġassuming Ġthat Ġits Ġperformance Ġis Ġweaker , Ġsimilarly Ġto ĠG EC Ġresults . </s> Human Ġevaluation Ġwas Ġperformed Ġon Ġa Ġsubset Ġof Ġ100 Ġsentences , Ġ50 Ġof Ġthem Ġselected Ġrandomly Ġfrom Ġthe ĠOpen Sub t itles Ġtest Ġset Ġand Ġthe Ġother Ġ50 Ġfrom ĠEuro par l . ĠEach Ġsentence Ġwas Ġtranslated Ġinto Ġthe Ġopposite Ġstyle . ĠThe Ġresulting Ġpairs Ġwere Ġpresented Ġto Ġparticipants , Ġwho Ġwere Ġasked Ġthe Ġfollowing Ġquestions Ġabout Ġeach Ġof Ġthem : Ġ( 1 ) ĠDo Ġthe Ġsentences Ġdiffer Ġin Ġany Ġway ? Ġ( 2 ) ĠHow Ġfluent Ġis Ġthe Ġtranslated Ġsentence ? Ġ( On Ġa Ġscale Ġof Ġ1 Ġto Ġ4 , Ġwhere Ġ1 Ġis Ġun readable , Ġand Ġ4 Ġis Ġperfectly Ġfluent ); Ġ( 3 ) ĠHow Ġsimilar Ġare Ġthe Ġsentences Ġin Ġmeaning ? Ġ( With Ġoptions Ġ" ex actly Ġthe Ġsame ", Ġ" the Ġsame Ġwith Ġminor Ġchanges ", Ġ" more Ġor Ġless Ġthe Ġsame ", Ġ" completely Ġdifferent Ġor Ġnonsensical "); Ġ( 4 ) ĠDoes Ġthe Ġtranslated Ġsentence Ġsound Ġmore Ġformal Ġthan Ġthe Ġoriginal , Ġmore Ġinformal , Ġor Ġneither ? Ġ( 5 ) ĠWhat Ġdifferences Ġare Ġthere Ġbetween Ġthe Ġsentences ? Ġ( E . g . Ġgram matical , Ġlex ical , Ġmissing Ġwords Ġor Ġphrases , Ġword Ġorder , Ġcontract ions , Ġthe Ġuse Ġof Ġformal Ġ" you "). </s> Two Ġsuch Ġsurveys Ġwere Ġconducted , Ġone Ġin ĠEnglish Ġand Ġone Ġin ĠEston ian . Ġ3 Ġpeople Ġparticipated Ġin Ġeach Ġof Ġthem , Ġeach Ġof Ġthe Ġthree Ġeval u ators Ġpresented Ġwith Ġthe Ġsame Ġset Ġof Ġexamples . </s> In Ġevaluation Ġof Ġflu ency , Ġall Ġthree Ġhuman Ġeval u ators Ġgave Ġthe Ġtranslated Ġsentences Ġthe Ġsame Ġscore Ġin Ġ41 Ġout Ġof Ġ55 Ġcases Ġin ĠEnglish Ġ( not Ġtaking Ġinto Ġaccount Ġsentences Ġwhich Ġwere Ġsimply Ġcopied Ġfrom Ġtheir Ġoriginals ), Ġand Ġin Ġ51 Ġout Ġof Ġ68 Ġcases Ġin ĠEston ian . ĠIn Ġevaluation Ġof Ġdirection Ġof Ġstyle Ġtransfer , Ġall Ġthree Ġeval u ators Ġagreed Ġin Ġ16 Ġcases Ġand Ġat Ġleast Ġtwo Ġagreed Ġin Ġ43 Ġcases Ġin ĠEnglish , Ġand Ġin ĠEston ian Ġin Ġ19 Ġcases Ġall Ġthree Ġagreed Ġand Ġin Ġ59 Ġat Ġleast Ġtwo . </s> Of Ġthe Ġ100 Ġtranslated Ġsentences , Ġ45 Ġwere Ġmarked Ġby Ġall Ġparticipants Ġas Ġbeing Ġthe Ġsame Ġas Ġtheir Ġoriginal Ġsentences Ġin Ġthe ĠEnglish Ġset Ġand Ġ32 Ġin ĠEston ian . ĠThe Ġremaining Ġ55 Ġand Ġ68 , Ġrespectively , Ġwere Ġused Ġto Ġquantify Ġstyle Ġtransfer Ġquality . </s> Being Ġa Ġreasonably Ġstrong ĠMT Ġsystem , Ġour Ġmodel Ġscores Ġquite Ġhigh Ġon Ġflu ency Ġ( 3 . 84 Ġfor ĠEnglish , Ġ3 . 64 Ġfor ĠEston ian ) Ġand Ġmeaning Ġpreservation Ġ( 3 . 67 Ġfor ĠEnglish , Ġ3 . 35 Ġfor ĠEston ian ). ĠFor Ġmeaning Ġpreservation , Ġthe Ġjudgments Ġwere Ġconverted Ġinto Ġa Ġscale Ġof Ġ1 - 4 , Ġwhere Ġ1 Ġstands Ġfor Ġcompletely Ġdifferent Ġmeanings Ġor Ġnonsensical Ġsentences , Ġand Ġ4 Ġfor Ġthe Ġexact Ġsame Ġmeaning . </s> We Ġevaluated Ġthe Ġstyle Ġtransfer Ġitself Ġin Ġthe Ġfollowing Ġway . ĠFor Ġeach Ġpair Ġof Ġsentences , Ġthe Ġaverage Ġscore Ġgiven Ġby Ġthree Ġeval u ators Ġwas Ġcalculated , Ġin Ġwhich Ġthe Ġanswer Ġthat Ġthe Ġtranslated Ġsentence Ġis Ġmore Ġformal Ġcounts Ġas Ġ+ 1 , Ġmore Ġinformal Ġas Ġ- 1 , Ġand Ġneither Ġas Ġ0 . ĠWe Ġcalculated Ġthe Ġroot Ġmean Ġsquare Ġerror Ġbetween Ġthese Ġscores Ġand Ġdesired Ġvalues Ġ(+ 1 Ġif Ġwe Ġaimed Ġto Ġmake Ġthe Ġsentence Ġmore Ġformal , Ġ- 1 Ġif Ġmore Ġinformal ). ĠRM SE Ġof Ġ0 Ġwould Ġstand Ġfor Ġalways Ġtransferring Ġstyle Ġin Ġthe Ġright Ġdirection Ġas Ġjudged Ġby Ġall Ġeval u ators , Ġand Ġ2 Ġfor Ġalways Ġtransferring Ġstyle Ġin Ġthe Ġwrong Ġdirection . </s> On Ġthe ĠEnglish Ġset , Ġthe ĠRM SE Ġis Ġ0 . 78 , Ġand Ġon ĠEston ian Ġ0 . 89 . ĠThese Ġnumbers Ġshow Ġthat Ġstyle Ġtransfer Ġgenerally Ġhappens Ġin Ġthe Ġright Ġdirection , Ġbut Ġis Ġnot Ġvery Ġstrong . ĠOf Ġthe Ġ55 Ġsentences Ġin ĠEnglish Ġthat Ġwere Ġdifferent Ġfrom Ġtheir Ġsource Ġsentences , Ġin Ġ33 Ġcases Ġthe Ġsign Ġof Ġthe Ġaverage Ġhuman Ġscore Ġmatched Ġthe Ġdesired Ġone , Ġin Ġ7 Ġit Ġdid Ġnot , Ġand Ġin Ġ15 Ġno Ġchange Ġin Ġstyle Ġwas Ġobserved Ġby Ġhumans . ĠIn ĠEston ian Ġ36 Ġsentences Ġshowed Ġthe Ġright Ġdirection Ġof Ġstyle Ġtransfer , Ġ10 Ġwrong , Ġand Ġ22 Ġno Ġchange . </s> In ĠEnglish Ġsentences Ġwhere Ġthe Ġdirection Ġof Ġstyle Ġtransfer Ġwas Ġfound Ġto Ġbe Ġcorrect Ġ( Figure ĠFIG REF 32 Ġ), Ġchanges Ġin Ġuse Ġof Ġcontract ions Ġwere Ġreported Ġin Ġ19 Ġcases Ġ( e . g . ĠI Ġhave Ġjust Ġbeen Ġvs . ĠI 've Ġjust Ġbeen ), Ġlex ical Ġchanges Ġin Ġ15 Ġcases Ġ( e . g . Ġ' cause Ġvs . Ġbecause , Ġor Ġsure Ġvs . Ġcertainly ), Ġgram matical Ġin Ġ13 Ġ( e . g . Ġreplacing Ġno Ġone 's Ġgonna Ġwith Ġno Ġone Ġwill , Ġor Ġmethod Ġof Ġproduction Ġwith Ġproduction Ġmethod ), Ġmissing Ġor Ġadded Ġwords Ġor Ġphrases Ġin Ġ8 Ġcases . </s> In ĠEston ian Ġcorrectly Ġtransferred Ġsentences Ġ( Figure ĠFIG REF 32 Ġ), Ġthe Ġmost Ġfrequently Ġreported Ġwere Ġlex ical Ġsubstit utions Ġ( 30 Ġcases ), Ġfollowed Ġby Ġmissing Ġof Ġadded Ġwords Ġor Ġphrases Ġ( 24 Ġcases ), Ġchanges Ġin Ġgrammar Ġ( 22 Ġcases ) Ġand Ġin Ġword Ġorder Ġ( 16 Ġcases ). </s> To Ġconclude Ġthis Ġsection , Ġunlike Ġmany Ġstyle Ġtransfer Ġmodels Ġwhich Ġproduce Ġtext Ġwith Ġstrong Ġstyle Ġcharacteristics Ġ( e . g . Ġwith Ġstrong Ġpositive Ġor Ġnegative Ġsentiment ), Ġoften Ġat Ġthe Ġcost Ġof Ġpreserving Ġmeaning Ġand Ġflu ency , Ġour Ġmodel Ġgrav itates Ġtowards Ġkeeping Ġthe Ġmeaning Ġand Ġflu ency Ġof Ġthe Ġoriginal Ġsentence Ġintact Ġand Ġmim icking Ġsome Ġof Ġthe Ġdesired Ġstyl istic Ġtraits . </s> Style ĠTransfer </s> Next Ġwe Ġmove Ġon Ġto Ġevaluating Ġthe Ġsame Ġmodel Ġin Ġterms Ġof Ġits Ġperformance Ġin Ġthe Ġcontext Ġof Ġstyle Ġtransfer . </s> At Ġfirst , Ġwe Ġexamined Ġhow Ġoften Ġthe Ġsentences Ġchange Ġwhen Ġtranslated Ġmon oling ually . ĠThe Ġassumption Ġis Ġthat Ġpassing Ġmodified Ġstyle Ġfactors Ġshould Ġprevent Ġthe Ġmodel Ġfrom Ġsimply Ġcopying Ġthe Ġsource Ġsequences Ġwhen Ġtranslating Ġinside Ġa Ġsingle Ġlanguage , Ġand Ġincentiv ize Ġit Ġto Ġmatch Ġits Ġoutput Ġto Ġcertain Ġstyle Ġcharacteristics Ġtypical Ġfor Ġdifferent Ġcorpor a . ĠFigure ĠFIG REF 28 Ġshows Ġthe Ġproportions Ġof Ġsentence Ġpairs Ġin Ġthe Ġ1000 - sent ence Ġtest Ġsets Ġwhere Ġthere Ġwas Ġa Ġsignificant Ġdifference Ġbetween Ġtranslations Ġinto Ġdifferent Ġstyles . ĠWe Ġcan Ġobserve Ġthat ĠEnglish Ġtexts Ġchange Ġless Ġoften Ġthan ĠEston ian Ġor ĠLat v ian , Ġwhile ĠEuro par l Ġsentences Ġare Ġchanged Ġmore Ġoften Ġthan Ġthose Ġof Ġother Ġcorpor a . </s> To Ġassess Ġwhether Ġthese Ġchanges Ġactually Ġcorrespond Ġto Ġthe Ġmodel 's Ġcapability Ġfor Ġtransferring Ġstyle , Ġwe Ġturned Ġto Ġhelp Ġof Ġhuman Ġeval u ators . </s> Cross - ling ual ĠStyle ĠTransfer </s> Being Ġable Ġto Ġtranslate Ġbetween Ġlanguages Ġand Ġalso Ġto Ġmodify Ġthe Ġoutput Ġto Ġmatch Ġthe Ġdesired Ġstyle Ġallows Ġthe Ġmodel Ġto Ġessentially Ġperform Ġdomain Ġadaptation . ĠWhen Ġtranslating Ġfrom Ġa Ġlanguage Ġwhich Ġhas Ġno Ġformal Ġ" you " Ġ( English ) Ġinto Ġone Ġthat Ġdoes Ġ( E ston ian Ġor ĠLat v ian ), Ġit Ġwill Ġquite Ġconsistently Ġuse Ġthe Ġinformal Ġvariant Ġwhen Ġthe Ġtarget Ġstyle Ġis ĠOpen Sub t itles Ġand Ġthe Ġformal Ġwhen Ġthe Ġtarget Ġstyle Ġis ĠEuro par l Ġ( you Ġrock ĠIN LINE FORM 0 Ġsa Ġro kid / te Ġro k ite ). ĠThe Ġmodel Ġis Ġalso Ġquite Ġconsistent Ġin Ġuse Ġof Ġcontract ions Ġin ĠEnglish Ġ( es Ġes mu ĠÅ ¡ e it ĠIN LINE FORM 1 ĠI Ġam Ġhere / I 'm Ġhere ). ĠSome Ġlex ical Ġsubstit utions Ġoccur : Ġneed Ġon ĠMatt i Ġlaps ed . ĠIN LINE FORM 2 Ġthose Ġare ĠMatt 's Ġkids ./ these Ġare ĠMatt 's Ġchildren . ĠWord Ġorder Ġmay Ġchange : ĠWhere Ġis ĠAnna 's Ġbag ? Ġis ĠKus Ġon ĠAnna Ġk ott ? Ġin Ġthe Ġmore Ġformal Ġvariant , Ġand ĠKus ĠAnna Ġk ott Ġon ? Ġin Ġthe Ġmore Ġinformal . ĠThis Ġfeature Ġis Ġuseful , Ġbut Ġout Ġof Ġscope Ġof Ġthis Ġarticle , Ġas Ġwe Ġfocus Ġon Ġmon oling ual Ġapplications . </s> Related ĠWork </s> G ram matical Ġerror Ġcorrection : Ġthere Ġhave Ġbeen Ġfour Ġshared Ġtasks Ġfor ĠG EC Ġwith Ġprepared Ġerror - tag ged Ġdatasets Ġfor ĠL 2 Ġlearners Ġof ĠEnglish Ġin Ġthe Ġlast Ġdecade : ĠH OO ĠB IB REF 27 Ġ, ĠB IB REF 28 Ġand ĠCo N LL ĠB IB REF 29 Ġ, ĠB IB REF 5 Ġ. ĠThis Ġhas Ġgiven Ġan Ġopportunity Ġto Ġtrain Ġnew Ġmodels Ġon Ġthe Ġshared Ġdatasets Ġand Ġget Ġan Ġobjective Ġcomparison Ġof Ġresults . ĠThe Ġgeneral Ġapproach Ġfor Ġgram matical Ġerror Ġcorrection Ġhas Ġbeen Ġto Ġuse Ġeither Ġrule - based Ġapproach , Ġmachine Ġlearning Ġon Ġerror - tag ged Ġcorpor a , ĠMT Ġmodels Ġon Ġparallel Ġdata Ġof Ġerroneous Ġand Ġcorrected Ġsentences , Ġor Ġa Ġcombination Ġof Ġthese ĠB IB REF 5 Ġ. ĠThe Ġtop Ġmodel Ġof Ġthe ĠCON LL Ġshared Ġtask Ġin Ġ2014 Ġused Ġa Ġcombined Ġmodel Ġof Ġrule - based Ġapproach Ġand ĠMT ĠB IB REF 24 Ġ. ĠAll Ġof Ġthese Ġrequire Ġannot ated Ġdata Ġor Ġconsiderable Ġeffort Ġto Ġcreate , Ġwhereas Ġour Ġmodel Ġis Ġmuch Ġmore Ġresource - independent . ĠAnother Ġfocus Ġof Ġthe Ġnewer Ġresearch Ġis Ġon Ġcreating ĠG EC Ġmodels Ġwithout Ġhuman - annot ated Ġresources . ĠFor Ġexample ĠB IB REF 26 Ġcombine Ġstatistical ĠMT Ġwith Ġun super vised Ġclassification Ġusing Ġun annot ated Ġparallel Ġdata Ġfor ĠMT Ġand Ġun annot ated Ġnative Ġdata Ġfor Ġthe Ġclassification Ġmodel . ĠIn Ġthis Ġcase Ġparallel Ġdata Ġof Ġerroneous Ġand Ġcorrected Ġsentences Ġis Ġstill Ġnecessary Ġfor ĠMT ; Ġthe Ġclass ifier Ġuses Ġnative Ġdata , Ġbut Ġstill Ġneeds Ġdefinitions Ġof Ġpossible Ġerror Ġtypes Ġto Ġclassify ĠâĢĵ Ġthis Ġwork Ġneeds Ġto Ġbe Ġdone Ġby Ġa Ġhuman Ġand Ġis Ġdifficult Ġfor Ġsome Ġless Ġclear Ġerror Ġtypes . ĠIn Ġour Ġapproach Ġthere Ġis Ġno Ġneed Ġfor Ġparallel Ġdata Ġnor Ġto Ġspecify Ġerror Ġtypes , Ġonly Ġfor Ġnative Ġdata . </s> There Ġhas Ġbeen Ġlittle Ġwork Ġon ĠEston ian Ġand ĠLat v ian ĠG EC , Ġall Ġlimited Ġwith Ġrule - based Ġapproaches ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ. ĠFor Ġboth Ġlanguages , Ġas Ġwell Ġas Ġany Ġlow - res ourced Ġlanguages , Ġour Ġapproach Ġgives Ġa Ġfeasible Ġway Ġto Ġdo Ġgram matical Ġerror Ġcorrection Ġwithout Ġneeding Ġneither Ġparallel Ġnor Ġerror Ġtagged Ġcorpor a . ĠStyle Ġtransfer : ĠSeveral Ġapproaches Ġuse Ġdirectly Ġannot ated Ġdata : Ġfor Ġexample , ĠB IB REF 7 Ġand ĠB IB REF 8 Ġtrain ĠMT Ġsystems Ġon Ġthe Ġcorpus Ġof Ġmodern ĠEnglish ĠShakespeare Ġto Ġoriginal ĠShakespeare . ĠB IB REF 32 Ġcollect Ġa Ġdataset Ġof Ġ110 K Ġinformal / form al Ġsentence Ġpairs Ġand Ġtrain Ġrule - based , Ġphrase - based , Ġand Ġneural ĠMT Ġsystems Ġusing Ġthis Ġdata . </s> One Ġline Ġof Ġwork Ġaims Ġat Ġlearning Ġa Ġstyle - independent Ġlatent Ġrepresentation Ġof Ġcontent Ġwhile Ġbuilding Ġdec od ers Ġthat Ġcan Ġgenerate Ġsentences Ġin Ġthe Ġstyle Ġof Ġchoice ĠB IB REF 9 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ, ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġ. ĠUn super vised ĠMT Ġhas Ġalso Ġbeen Ġadapted Ġfor Ġthe Ġtask ĠB IB REF 10 Ġ, ĠB IB REF 40 Ġ. ĠOur Ġsystem Ġalso Ġdoes Ġnot Ġrequire Ġparallel Ġdata Ġbetween Ġstyles , Ġbut Ġlever ages Ġthe Ġstability Ġof Ġthe Ġoff - the - she lf Ġsupervised ĠN MT Ġto Ġavoid Ġthe Ġhassle Ġof Ġtraining Ġun super vised ĠN MT Ġsystems Ġand Ġmaking ĠG AN s Ġconverge . ĠAnother Ġproblem Ġwith Ġmany Ġcurrent Ġ( both Ġsupervised Ġand Ġun super vised ) Ġstyle Ġtransfer Ġmethods Ġis Ġthat Ġthey Ġare Ġbounded Ġto Ġsolve Ġa Ġbinary Ġtask , Ġwhere Ġonly Ġtwo Ġstyles Ġare Ġincluded Ġ( whether Ġbecause Ġof Ġdata Ġor Ġrestrictions Ġof Ġthe Ġapproach ). ĠOur Ġmethod , Ġon Ġthe Ġother Ġhand , Ġcan Ġbe Ġextended Ġto Ġas Ġmany Ġstyles Ġas Ġneeded Ġas Ġlong Ġas Ġthere Ġare Ġparallel ĠMT Ġcorpor a Ġin Ġthese Ġstyles Ġavailable . </s> Not ably , ĠB IB REF 41 Ġuse Ġside Ġconst r ains Ġin Ġorder Ġto Ġtranslate Ġin Ġpolite / imp ol ite ĠGerman , Ġwhile Ġwe Ġrely Ġon Ġmult ilingual Ġenc oder Ġrepresentations Ġand Ġuse Ġthe Ġsystem Ġmon oling ually Ġat Ġinference Ġtime . </s> Finally , Ġthe Ġmost Ġsimilar Ġto Ġour Ġwork Ġconcept ually Ġis Ġthe Ġapproach Ġof ĠB IB REF 42 Ġ, Ġwhere Ġthey Ġtranslate Ġa Ġsentence Ġinto Ġanother Ġlanguage , Ġhoping Ġthat Ġit Ġwill Ġlose Ġsome Ġstyle Ġindicators , Ġand Ġthen Ġtranslate Ġit Ġback Ġinto Ġthe Ġoriginal Ġlanguage Ġwith Ġa Ġdesired Ġstyle Ġtag Ġattached Ġto Ġthe Ġenc oder Ġlatent Ġspace . ĠWe Ġalso Ġuse Ġthe ĠMT Ġenc oder Ġto Ġobtain Ġrich Ġsentence Ġrepresentations , Ġbut Ġlearn Ġthem Ġdirectly Ġas Ġa Ġpart Ġof Ġa Ġsingle Ġmult ilingual Ġtranslation Ġsystem . </s> Con clusions </s> We Ġpresented Ġa Ġsimple Ġapproach Ġwhere Ġa Ġsingle Ġmult ilingual ĠN MT Ġmodel Ġis Ġadapted Ġto Ġmon oling ual Ġtransfer Ġand Ġperforms Ġgram matical Ġerror Ġcorrection Ġand Ġstyle Ġtransfer . ĠWe Ġexperimented Ġwith Ġthree Ġlanguages Ġand Ġpresented Ġextensive Ġevaluation Ġof Ġthe Ġmodel Ġon Ġboth Ġtasks . ĠWe Ġused Ġpublicly Ġavailable Ġsoftware Ġand Ġdata Ġand Ġbelieve Ġthat Ġour Ġwork Ġcan Ġbe Ġeasily Ġreproduced . </s> We Ġshowed Ġthat Ġfor ĠG EC Ġour Ġapproach Ġreliably Ġcorrect s Ġspelling , Ġword Ġorder Ġand Ġgram matical Ġerrors , Ġwhile Ġbeing Ġless Ġreliable Ġon Ġlex ical Ġchoice Ġerrors . ĠApplied Ġto Ġstyle Ġtransfer Ġour Ġmodel Ġis Ġvery Ġgood Ġat Ġmeaning Ġpreservation Ġand Ġoutput Ġflu ency , Ġwhile Ġreliably Ġtransferring Ġstyle Ġfor ĠEnglish Ġcontract ions , Ġlex ical Ġchoice Ġand Ġgram matical Ġconstruct ions . ĠThe Ġmain Ġbenefit Ġis Ġthat Ġno Ġannot ated Ġdata Ġis Ġused Ġto Ġtrain Ġthe Ġmodel , Ġthus Ġmaking Ġit Ġvery Ġeasy Ġto Ġtrain Ġit Ġfor Ġother Ġ( especially Ġunder - res ourced ) Ġlanguages . </s> Future Ġwork Ġincludes Ġexploring Ġadaptations Ġof Ġthis Ġapproach Ġto Ġboth Ġtasks Ġseparately , Ġwhile Ġkeeping Ġthe Ġlow Ġcost Ġof Ġcreating Ġsuch Ġmodels . </s> Model ĠTraining : ĠTechnical ĠDetails </s> After Ġrudimentary Ġcleaning Ġ( rem oving Ġpairs Ġwhere Ġat Ġleast Ġone Ġsentence Ġis Ġlonger Ġthat Ġ100 Ġtokens , Ġat Ġleast Ġone Ġsentence Ġis Ġan Ġempty Ġstring Ġor Ġdoes Ġnot Ġcontain Ġany Ġal ph abetic Ġcharacters , Ġand Ġpairs Ġwith Ġlength Ġratio Ġover Ġ9 ) Ġand Ġduplication Ġto Ġaccommodate Ġboth Ġtranslation Ġdirections Ġin Ġeach Ġlanguage Ġpair , Ġthe Ġtotal Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġis Ġ22 . 9 M Ġsentence Ġpairs ; Ġtraining Ġset Ġsizes Ġper Ġlanguage Ġand Ġcorpus Ġare Ġgiven Ġin ĠTable ĠT AB REF 36 Ġ. ĠVal idation Ġset Ġconsists Ġof Ġ12 K Ġsentence Ġpairs , Ġ500 Ġfor Ġeach Ġcombination Ġof Ġtranslation Ġdirection Ġand Ġcorpus . ĠWe Ġalso Ġkeep Ġa Ġtest Ġset Ġof Ġ24 K Ġsentence Ġpairs , Ġ1000 Ġfor Ġeach Ġtranslation Ġdirection Ġand Ġcorpus . </s> The Ġdata Ġpre processing Ġpipeline Ġconsists Ġof Ġtoken ization Ġwith ĠMoses Ġtoken izer ĠB IB REF 44 Ġ, Ġtrue - c asing , Ġand Ġsegment ation Ġwith ĠSent ence P iece ĠB IB REF 45 Ġwith Ġa Ġjoint Ġvocabulary Ġof Ġsize Ġ32 Ġ000 . </s> We Ġtrained Ġa ĠTrans former ĠN MT Ġmodel Ġusing Ġthe ĠSoc keye Ġframework ĠB IB REF 43 Ġ, Ġmostly Ġfollowing Ġthe Ġso - called ĠTrans former Ġbase Ġmodel : Ġwe Ġused Ġ6 Ġlayers , Ġ512 Ġpositions , Ġ8 Ġattention Ġheads Ġand ĠRe LU Ġactiv ations Ġfor Ġboth Ġthe Ġenc oder Ġand Ġdec oder ; ĠAdam Ġoptim izer Ġwas Ġused . ĠSource Ġand Ġtarget Ġtoken Ġembed d ings Ġwere Ġboth Ġof Ġsize Ġ512 , Ġand Ġfactors Ġdetermining Ġtarget Ġlanguage Ġand Ġstyle Ġhad Ġembed d ings Ġof Ġsize Ġ4 . ĠB atch Ġsize Ġwas Ġset Ġto Ġ2048 Ġwords , Ġinitial Ġlearning Ġrate Ġto Ġ0 . 0002 , Ġreducing Ġby Ġa Ġfactor Ġof Ġ0 . 7 Ġevery Ġtime Ġthe Ġvalidation Ġperplex ity Ġhad Ġnot Ġimproved Ġfor Ġ8 Ġcheckpoints , Ġwhich Ġhappened Ġevery Ġ4000 Ġupdates . ĠThe Ġmodel Ġconver ged Ġduring Ġthe Ġ17 th Ġepoch , Ġwhen Ġvalidation Ġperplex ity Ġhas Ġnot Ġimproved Ġfor Ġ32 Ġconsecutive Ġcheckpoints . ĠThe Ġparameters Ġof Ġa Ġsingle Ġbest Ġcheckpoint Ġwere Ġused Ġfor Ġall Ġtranslations , Ġwith Ġbeam Ġsize Ġset Ġto Ġ5 . </s> Output ĠExamples </s> We Ġpresent Ġmore Ġexamples Ġof Ġtranslation Ġof Ġcode - sw itched Ġinput Ġsegments , Ġerror Ġcorrection Ġand Ġstyle Ġtransfer Ġin ĠEnglish , ĠEston ian Ġand ĠLat v ian , Ġinformal Ġ( inf ) Ġand Ġformal Ġ( fml ) Ġoutput Ġstyle :