Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF0, NTCIR PatentMT Parallel Corpus BIBREF1
EXTRACTIVE
Asian Scientific Paper Excerpt Corpus, NTCIR PatentMT Parallel Corpus 
EXTRACTIVE
We used two parallel corpora: Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF0 and NTCIR PatentMT Parallel Corpus BIBREF1 . Regarding the training data of ASPEC, we used only the first 1 million sentences sorted by sentence-alignment similarity. Japanese sentences were segmented by the morphological analyzer MeCab (version 0.996, IPADIC), and English sentences were tokenized by tokenizer.perl of Moses. Table TABREF14 shows the numbers of the sentences in each corpus. Note that sentences with more than 40 words were excluded from the training data.
We used two parallel corpora: Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF0 and NTCIR PatentMT Parallel Corpus BIBREF1 . Regarding the training data of ASPEC, we used only the first 1 million sentences sorted by sentence-alignment similarity. Japanese sentences were segmented by the morphological analyzer MeCab (version 0.996, IPADIC), and English sentences were tokenized by tokenizer.perl of Moses. Table TABREF14 shows the numbers of the sentences in each corpus. Note that sentences with more than 40 words were excluded from the training data.
1706.08198
Introduction </s> Recently , Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġgained Ġpopularity Ġin Ġthe Ġfield Ġof Ġmachine Ġtranslation . ĠThe Ġconventional Ġenc oder - dec oder ĠN MT Ġproposed Ġby ĠCho 2014 Ġuses Ġtwo Ġrecurrent Ġneural Ġnetworks Ġ( R NN ): Ġone Ġis Ġan Ġenc oder , Ġwhich Ġenc odes Ġa Ġsource Ġsequence Ġinto Ġa Ġfixed - length Ġvector , Ġand Ġthe Ġother Ġis Ġa Ġdec oder , Ġwhich Ġdec odes Ġthe Ġvector Ġinto Ġa Ġtarget Ġsequence . ĠA Ġnewly Ġproposed Ġattention - based ĠN MT Ġby ĠD z mit ry Bah d ana 2014 Ġcan Ġpredict Ġoutput Ġwords Ġusing Ġthe Ġweights Ġof Ġeach Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġby Ġthe Ġattention Ġmechanism , Ġimproving Ġthe Ġadequ acy Ġof Ġtranslation . </s> Even Ġwith Ġthe Ġsuccess Ġof Ġattention - based Ġmodels , Ġa Ġnumber Ġof Ġopen Ġquestions Ġremain Ġin ĠN MT . ĠTu 2016 Ġargued Ġtwo Ġof Ġthe Ġcommon Ġproblems Ġare Ġover - translation : Ġsome Ġwords Ġare Ġrepeatedly Ġtranslated Ġunnecessary Ġand Ġunder - translation : Ġsome Ġwords Ġare Ġmistakenly Ġunt rans lated . ĠThis Ġis Ġdue Ġto Ġthe Ġfact Ġthat ĠN MT Ġcan Ġnot Ġcompletely Ġconvert Ġthe Ġinformation Ġfrom Ġthe Ġsource Ġsentence Ġto Ġthe Ġtarget Ġsentence . ĠMi 2016 a Ġand ĠFeng 2016 Ġpointed Ġout Ġthat ĠN MT Ġlacks Ġthe Ġnotion Ġof Ġcoverage Ġvector Ġin Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( P BS MT ), Ġso Ġunless Ġotherwise Ġspecified , Ġthere Ġis Ġno Ġway Ġto Ġprevent Ġmissing Ġtranslations . </s> Another Ġproblem Ġin ĠN MT Ġis Ġan Ġobjective Ġfunction . ĠN MT Ġis Ġoptimized Ġby Ġcross - ent ropy ; Ġtherefore , Ġit Ġdoes Ġnot Ġdirectly Ġmaximize Ġthe Ġtranslation Ġaccuracy . ĠShen 2016 Ġpointed Ġout Ġthat Ġoptimization Ġby Ġcross - ent ropy Ġis Ġnot Ġappropriate Ġand Ġproposed Ġa Ġmethod Ġof Ġoptimization Ġbased Ġon Ġa Ġtranslation Ġaccuracy Ġscore , Ġsuch Ġas Ġexpected ĠB LE U , Ġwhich Ġled Ġto Ġimprovement Ġof Ġtranslation Ġaccuracy . ĠHowever , ĠB LE U Ġis Ġan Ġevaluation Ġmetric Ġbased Ġon Ġn - gram Ġprecision ; Ġtherefore , Ġrepetition Ġof Ġsome Ġwords Ġmay Ġbe Ġpresent Ġin Ġthe Ġtranslation Ġeven Ġthough Ġthe ĠB LE U Ġscore Ġis Ġimproved . </s> To Ġaddress Ġto Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation , Ġtu 2016 ne ural Ġintroduce Ġan Ġenc oder - dec oder - re construct or Ġframework Ġthat Ġoptim izes ĠN MT Ġby Ġback - translation Ġfrom Ġthe Ġoutput Ġsentences Ġinto Ġthe Ġoriginal Ġsource Ġsentences . ĠIn Ġtheir Ġmethod , Ġafter Ġtraining Ġthe Ġforward Ġtranslation Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT , Ġthey Ġtrain Ġa Ġback - translation Ġmodel Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġinto Ġthe Ġsource Ġsequence Ġby Ġa Ġnew Ġdec oder Ġto Ġenforce Ġagreement Ġbetween Ġsource Ġand Ġtarget Ġsentences . </s> In Ġorder Ġto Ġconfirm Ġthe Ġlanguage Ġindependence Ġof Ġthe Ġframework , Ġwe Ġexperiment Ġon Ġtwo Ġparallel Ġcorpor a Ġof ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks Ġusing Ġencode - dec oder - re construct or . ĠOur Ġexperiments Ġshow Ġthat Ġtheir Ġmethod Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon ĠEnglish - Japanese Ġtranslation Ġtask , Ġthough Ġthe Ġdifference Ġis Ġnot Ġsignificant Ġon ĠJapanese - English Ġtranslation Ġtask . </s> In Ġaddition , Ġwe Ġjointly Ġtrain Ġa Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training , Ġand Ġthen Ġevaluate Ġthis Ġmodel . ĠAs Ġa Ġresult , Ġthe Ġenc oder - dec oder - re construct or Ġcan Ġnot Ġbe Ġtrained Ġwell Ġwithout Ġpre - training , Ġso Ġit Ġproves Ġthat Ġwe Ġhave Ġto Ġtrain Ġthe Ġforward Ġtranslation Ġmodel Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġas Ġpre - training . </s> The Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> Related ĠWorks </s> Several Ġstudies Ġhave Ġaddressed Ġthe ĠN MT - specific Ġproblem Ġof Ġmissing Ġor Ġrepeating Ġwords . ĠN ie h ues 2016 Ġoptimized ĠN MT Ġby Ġadding Ġthe Ġoutputs Ġof ĠPBS MT Ġto Ġthe Ġinput Ġof ĠN MT . ĠMi 2016 a Ġand ĠFeng 2016 Ġintroduced Ġa Ġdistributed Ġversion Ġof Ġcoverage Ġvector Ġtaken Ġfrom ĠPBS MT Ġto Ġconsider Ġwhich Ġwords Ġhave Ġbeen Ġalready Ġtranslated . ĠAll Ġthese Ġmethods , Ġincluding Ġours , Ġemploy Ġinformation Ġof Ġthe Ġsource Ġsentence Ġto Ġimprove Ġthe Ġquality Ġof Ġtranslation , Ġbut Ġour Ġmethod Ġuses Ġback - translation Ġto Ġensure Ġthat Ġthere Ġis Ġno Ġinconsistency . ĠUnlike Ġother Ġmethods , Ġonce Ġlearned , Ġour Ġmethod Ġis Ġidentical Ġto Ġthe Ġconventional ĠN MT Ġmodel , Ġso Ġit Ġdoes Ġnot Ġneed Ġany Ġadditional Ġparameters Ġsuch Ġas Ġcoverage Ġvector Ġor Ġa ĠPBS MT Ġsystem Ġfor Ġtesting . </s> The Ġattention Ġmechanism Ġproposed Ġby ĠMeng 2016 Ġconsiders Ġnot Ġonly Ġthe Ġhidden Ġstates Ġof Ġthe Ġenc oder Ġbut Ġalso Ġthe Ġhidden Ġstates Ġof Ġthe Ġdec oder Ġso Ġthat Ġover - translation Ġcan Ġbe Ġrelaxed . ĠIn Ġaddition , Ġthe Ġattention Ġmechanism Ġproposed Ġby ĠFeng 2016 Ġcomp utes Ġa Ġcontext Ġvector Ġby Ġconsidering Ġthe Ġprevious Ġcontext Ġvector Ġto Ġprevent Ġover - translation . ĠThese Ġworks Ġindirectly Ġreduce Ġrepeating Ġand Ġmissing Ġwords , Ġwhile Ġwe Ġdirectly Ġpenal ize Ġtranslation Ġmismatch Ġby Ġconsidering Ġback - translation . </s> The Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġproposed Ġby Ġtu 2016 ne ural Ġoptim izes ĠN MT Ġby Ġreconstruct or Ġusing Ġback - translation . ĠThey Ġconsider Ġlikelihood Ġof Ġboth Ġof Ġforward Ġtranslation Ġand Ġback - translation , Ġand Ġthen Ġthis Ġframework Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon Ġa ĠChinese - English Ġtranslation Ġtask . </s> Ne ural ĠMachine ĠTranslation </s> Here , Ġwe Ġdescribe Ġthe Ġattention - based ĠN MT Ġproposed Ġby ĠD z mit ry Bah d ana 2014 Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. </s> The Ġinput Ġsequence Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġconverted Ġinto Ġa Ġfixed - length Ġvector Ġby Ġthe Ġenc oder Ġusing Ġan ĠR NN . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 1 Ġ, Ġthe Ġhidden Ġstate ĠIN LINE FORM 2 Ġof Ġthe Ġenc oder Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġa Ġbid irection al ĠR NN . ĠThe Ġforward Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġbackward Ġstate ĠIN LINE FORM 1 Ġare Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġnon linear Ġfunctions . ĠThe Ġhidden Ġstates ĠIN LINE FORM 2 Ġare Ġconverted Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . </s> The Ġfixed - length Ġvector ĠIN LINE FORM 0 Ġgenerated Ġby Ġthe Ġenc oder Ġis Ġconverted Ġinto Ġthe Ġtarget Ġsequence Ġ( ĠIN LINE FORM 1 Ġ) Ġby Ġthe Ġdec oder Ġusing Ġan ĠR NN . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 2 Ġ, Ġthe Ġconditional Ġprobability Ġof Ġthe Ġoutput Ġword ĠIN LINE FORM 3 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . ĠThe Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġdec oder Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġthe Ġhidden Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġtarget Ġword ĠIN LINE FORM 1 Ġat Ġthe Ġprevious Ġtime Ġstep Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 2 Ġ. </s> The Ġcontext Ġvector ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġsum Ġof Ġeach Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġenc oder . ĠIt Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> and Ġits Ġweight ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġprobability Ġdistribution . ĠIt Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweight Ġvector Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . </s> The Ġobjective Ġfunction Ġis Ġdefined Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġdata Ġand ĠIN LINE FORM 1 Ġis Ġa Ġmodel Ġparameter . </s> Inc identally , Ġas Ġa Ġnon linear Ġfunction , Ġthe Ġhyper b olic Ġtang ent Ġfunction Ġor Ġthe Ġrect ified Ġlinear Ġunit Ġare Ġgenerally Ġused . </s> Arch itect ure </s> Next , Ġwe Ġdescribe Ġthe Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġproposed Ġby Ġtu 2016 ne ural Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. ĠThe Ġenc oder - dec oder - re construct or Ġconsists Ġof Ġtwo Ġcomponents : Ġthe Ġstandard Ġenc oder - dec oder Ġas Ġan Ġattention - based ĠN MT Ġproposed Ġby ĠD z mit ry Bah d ana 2014 Ġand Ġthe Ġreconstruct or Ġwhich Ġback - trans l ates Ġfrom Ġthe Ġhidden Ġstates Ġof Ġdec oder Ġto Ġthe Ġsource Ġsentence . </s> In Ġtheir Ġmethod , Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġis Ġback - trans lated Ġinto Ġthe Ġsource Ġsequence Ġ( ĠIN LINE FORM 0 Ġ) Ġby Ġthe Ġreconstruct or Ġfor Ġthe Ġback - translation . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 1 Ġ, Ġthe Ġconditional Ġprobability Ġof Ġthe Ġoutput Ġword ĠIN LINE FORM 2 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . ĠThe Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġreconstruct or Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġthe Ġhidden Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġsource Ġword ĠIN LINE FORM 1 Ġat Ġthe Ġprevious Ġtime Ġstep Ġand Ġthe Ġnew Ġcontext Ġvector Ġ( in verse Ġcontext Ġvector ) ĠIN LINE FORM 2 Ġ. </s> The Ġinverse Ġcontext Ġvector ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġsum Ġof Ġeach Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġdec oder Ġ( on Ġforward Ġtranslation ). ĠIt Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> and Ġits Ġweight ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġprobability Ġdistribution . ĠIt Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweight Ġvector Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . </s> The Ġobjective Ġfunction Ġis Ġdefined Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġdata , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġmodel Ġparameters Ġand ĠIN LINE FORM 3 Ġis Ġa Ġhyper - param eter Ġwhich Ġcan Ġconsider Ġthe Ġweight Ġbetween Ġforward Ġtranslation Ġand Ġback - translation . </s> This Ġobjective Ġfunction Ġconsists Ġof Ġtwo Ġparts : Ġforward Ġmeasures Ġtranslation Ġflu ency , Ġand Ġbackward Ġmeasures Ġtranslation Ġadequ acy . ĠThus , Ġthe Ġcombined Ġobjective Ġfunction Ġis Ġmore Ġconsistent Ġwith Ġthe Ġgoal Ġof Ġenhancing Ġoverall Ġtranslation Ġquality , Ġand Ġcan Ġmore Ġeffectively Ġguide Ġthe Ġparameter Ġtraining Ġfor Ġmaking Ġbetter Ġtranslation . </s> Training </s> The Ġenc oder - dec oder - re construct or Ġis Ġtrained Ġwith Ġlikelihood Ġof Ġboth Ġthe Ġenc oder - dec oder Ġand Ġthe Ġreconstruct or Ġon Ġa Ġset Ġof Ġtraining Ġdatasets . Ġtu 2016 ne ural Ġtrained Ġa Ġback - translation Ġmodel Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġinto Ġthe Ġsource Ġsequence Ġby Ġreconstruct or Ġto Ġenforce Ġagreement Ġbetween Ġsource Ġand Ġtarget Ġsentences Ġusing ĠEqu ation ĠEQ REF 23 Ġafter Ġtraining Ġthe Ġforward Ġtranslation Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġusing ĠEqu ation ĠEQ REF 13 Ġ. </s> In Ġaddition , Ġwe Ġexperiment Ġto Ġjointly Ġtrain Ġa Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training . ĠIt Ġmay Ġlearn Ġa Ġglobally Ġoptimal Ġmodel Ġcompared Ġto Ġlocally Ġoptimal Ġmodel Ġpre - trained Ġusing Ġthe Ġforward Ġtranslation . </s> Testing </s> tu 2016 ne ural Ġused Ġa Ġbeam Ġsearch Ġto Ġpredict Ġtarget Ġsentences Ġthat Ġapproximately Ġmaxim izes Ġboth Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġon Ġtesting . ĠIn Ġthis Ġpaper , Ġhowever , Ġwe Ġdo Ġnot Ġuse Ġa Ġbeam Ġsearch Ġfor Ġsimplicity Ġand Ġeffectiveness . </s> Exper iments </s> We Ġevaluated Ġthe Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġon ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks . </s> Dat as ets </s> We Ġused Ġtwo Ġparallel Ġcorpor a : ĠAsian ĠScientific ĠPaper ĠEx cerpt ĠCorpus Ġ( AS P EC ) ĠB IB REF 0 Ġand ĠN TC IR ĠPatent MT ĠParallel ĠCorpus ĠB IB REF 1 Ġ. ĠRegarding Ġthe Ġtraining Ġdata Ġof ĠASP EC , Ġwe Ġused Ġonly Ġthe Ġfirst Ġ1 Ġmillion Ġsentences Ġsorted Ġby Ġsentence - al ignment Ġsimilarity . ĠJapanese Ġsentences Ġwere Ġsegment ed Ġby Ġthe Ġmorph ological Ġanaly zer ĠMe C ab Ġ( version Ġ0 . 996 , ĠIP AD IC ), Ġand ĠEnglish Ġsentences Ġwere Ġtoken ized Ġby Ġtoken izer . per l Ġof ĠMoses . ĠTable ĠT AB REF 14 Ġshows Ġthe Ġnumbers Ġof Ġthe Ġsentences Ġin Ġeach Ġcorpus . ĠNote Ġthat Ġsentences Ġwith Ġmore Ġthan Ġ40 Ġwords Ġwere Ġexcluded Ġfrom Ġthe Ġtraining Ġdata . </s> Mod els </s> We Ġused Ġthe Ġattention - based ĠN MT ĠB IB REF 2 Ġas Ġa Ġbaseline - N MT , Ġthe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġand Ġthe Ġenc oder - dec oder - re construct or Ġthat Ġjointly Ġtrained Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training . ĠThe ĠR NN Ġused Ġin Ġthe Ġexperiments Ġhad Ġ512 Ġhidden Ġunits , Ġ512 Ġembed ding Ġunits , Ġ30 , 000 Ġvocabulary Ġsize Ġand Ġ64 Ġbatch Ġsize . ĠWe Ġused ĠAd ag rad Ġ( initial Ġlearning Ġrate Ġ0 . 01 ) Ġfor Ġoptimizing Ġmodel Ġparameters . ĠWe Ġtrained Ġour Ġmodel Ġon ĠGeForce ĠGTX ĠTIT AN ĠX ĠGPU . ĠNote Ġthat Ġwe Ġset Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġon Ġthe Ġenc oder - dec oder - re construct or Ġsame Ġas Ġtu 2016 ne ural . </s> Results </s> T ables ĠT AB REF 15 Ġand ĠT AB REF 16 Ġshow Ġthe Ġtranslation Ġaccuracy Ġin ĠB LE U Ġscores , Ġthe ĠIN LINE FORM 0 Ġ- value Ġof Ġthe Ġsignificance Ġtest Ġby Ġboot strap Ġres am pling ĠB IB REF 4 Ġand Ġtraining Ġtime Ġin Ġhours Ġuntil Ġconvergence . ĠThe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġrequires Ġslightly Ġlonger Ġtime Ġto Ġtrain Ġthan Ġthe Ġbaseline ĠN MT , Ġbut Ġwe Ġemphasize Ġthat Ġdecoding Ġtime Ġremains Ġthe Ġsame Ġwith Ġthe Ġenc oder - dec oder - re construct or Ġand Ġbaseline - N MT . ĠThe Ġresults Ġshow Ġthat Ġthe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġsignificantly Ġimproves Ġtranslation Ġaccuracy Ġby Ġ1 . 01 Ġpoints Ġon ĠASP EC Ġand Ġ1 . 37 Ġpoints Ġon ĠN TC IR Ġin ĠEnglish - Japanese Ġtranslation Ġ( ĠIN LINE FORM 1 Ġ). ĠHowever , Ġit Ġdoes Ġnot Ġsignificantly Ġimprove Ġtranslation Ġaccuracy Ġin ĠJapanese - English Ġtranslation . ĠIn Ġaddition , Ġit Ġis Ġproved Ġthat Ġthe Ġenc oder - dec oder - re construct or Ġwithout Ġpre - training Ġwors ens Ġrather Ġthan Ġimproves Ġtranslation Ġaccuracy . </s> Table ĠT AB REF 24 Ġshows Ġexamples Ġof Ġoutputs Ġof ĠEnglish - Japanese Ġtranslations . ĠIn ĠExample Ġ1 , ĠâĢ ľ UTF 8 min ä¹ ± Ġæ µ ģ Ġç ² ĺ æĢ § Ġãģ ¨ Ġæ ķ ° å Ģ ¤ Ġç ² ĺ æĢ § Ġ ãģ® Ġå¤ § å° ı Ġé ĸ ¢ ä¿ Ĥ Ġãģ « ãĤ Ī ãĤĬ Ġ, âĢ Ŀ Ġ( on Ġthe Ġbasis Ġof Ġthe Ġrelation Ġbetween Ġturbulent Ġvis cos ity Ġand Ġnumerical Ġvis cos ity Ġin Ġsize ) Ġis Ġmissing Ġin Ġthe Ġoutput Ġof Ġbaseline - N MT , Ġbut ĠâĢ ľ UTF 8 min ä¹ ± Ġæ µ ģ Ġç ² ĺ æĢ § Ġãģ ¨ Ġæ ķ ° å Ģ ¤ Ġç ļ Ħ Ġç ² ĺ æĢ § Ġ ãģ® Ġé ĸ ¢ ä¿ Ĥ ĠãĤ Ĵ Ġå Ł º Ġãģ « âĢ Ŀ Ġ( on Ġthe Ġbasis Ġof Ġthe Ġrelation Ġbetween Ġturbulent Ġvis cos ity Ġand Ġnumerical Ġvis cos ity ) Ġis Ġpresent Ġin Ġthe Ġoutput Ġof Ġenc oder - dec oder - re construct or . ĠIn ĠExample Ġ2 , ĠâĢ ľ UTF 8 min æĸ ° çĶŁ åħ Ĳ âĢ Ŀ Ġ( new born Ġinfant ) Ġand ĠâĢ ľ UTF 8 min 30 æŃ ³ ä» ¥ ä¸Ĭ ãģ® âĢ Ŀ Ġ( of Ġ30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore ) Ġare Ġrepeated Ġin Ġthe Ġoutput Ġof Ġbaseline - N MT , Ġbut Ġthey Ġappear Ġonly Ġonce Ġin Ġthe Ġoutput Ġof Ġenc oder - dec oder - re construct or . </s> In Ġaddition , ĠFigures ĠFIG REF 25 Ġand ĠFIG REF 25 Ġshow Ġthe Ġattention Ġlayer Ġon Ġbaseline - N MT Ġand Ġenc oder - dec oder - re construct or Ġin Ġeach Ġexample . ĠIn ĠFigure ĠFIG REF 25 Ġ, Ġalthough Ġthe Ġattention Ġlayer Ġof Ġbaseline ĠN MT Ġattends Ġinput Ġword ĠâĢ ľ t urb ulent âĢ Ŀ , Ġthe Ġdec oder Ġdoes Ġnot Ġoutput ĠâĢ ľ UTF 8 min ä¹ ± æµ ģ âĢ Ŀ Ġ( t urb ulent ) Ġbut ĠâĢ ľ UTF 8 min æ ¤ ľ è ¨ İ âĢ Ŀ Ġ( ex am ined ) Ġat Ġthe Ġ13 th Ġword . ĠThus , Ġunder - translation Ġmay Ġbe Ġresulted Ġfrom Ġthe Ġhidden Ġlayer Ġor Ġthe Ġembed ding Ġlayer Ġinstead Ġof Ġthe Ġattention Ġlayer . ĠIn ĠFigure ĠFIG REF 25 Ġ, Ġit Ġis Ġfound Ġthat Ġthe Ġattention Ġlayer Ġof Ġbaseline - N MT Ġrepeatedly Ġattends Ġinput Ġwords ĠâĢ ľ new born Ġinfant âĢ Ŀ Ġand ĠâĢ ľ 30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore âĢ Ŀ . ĠConsequently , Ġthe Ġdec oder Ġrepeatedly Ġoutputs ĠâĢ ľ UTF 8 min æĸ ° çĶŁ åħ Ĳ âĢ Ŀ Ġ( new born Ġinfant ) Ġand ĠâĢ ľ UTF 8 min 30 æŃ ³ ä» ¥ ä¸Ĭ ãģ® âĢ Ŀ Ġ( of Ġ30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore ). ĠOn Ġthe Ġother Ġhand , Ġthe Ġattention Ġlayer Ġof Ġenc oder - dec oder - re construct or Ġalmost Ġcorrectly Ġattends Ġinput Ġwords . </s> Table ĠT AB REF 28 Ġshows Ġa Ġcomparison Ġof Ġthe Ġnumber Ġof Ġword Ġoccurrences Ġfor Ġeach Ġcorpus Ġand Ġmodel . ĠThe Ġcolumns Ġshow Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġthat Ġappear Ġmore Ġfrequently Ġthan Ġthe Ġcounterparts Ġin Ġthe Ġreference , Ġand Ġ( ii ) Ġthe Ġnumber Ġof Ġwords Ġthat Ġappear Ġmore Ġthan Ġonce Ġbut Ġare Ġnot Ġincluded Ġin Ġthe Ġreference . ĠNote Ġthat Ġthese Ġnumbers Ġdo Ġnot Ġinclude Ġunknown Ġwords , Ġso Ġ( iii ) Ġshows Ġthe Ġnumber Ġof Ġunknown Ġwords . ĠIn Ġall Ġthe Ġcases , Ġthe Ġnumber Ġof Ġoccurrence Ġof Ġredundant Ġwords Ġis Ġreduced Ġin Ġenc oder - dec oder - re construct or . ĠThus , Ġwe Ġconfirmed Ġthat Ġenc oder - dec oder - re construct or Ġachieves Ġreduction Ġof Ġrepeating Ġand Ġmissing Ġwords Ġwhile Ġmaintaining Ġthe Ġquality Ġof Ġtranslation . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġevaluated Ġthe Ġenc oder - dec oder - re construct or Ġon ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks . ĠIn Ġaddition , Ġwe Ġevaluate Ġthe Ġeffectiveness Ġof Ġpre - training Ġby Ġcomparing Ġit Ġwith Ġa Ġjointly - trained Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation . ĠExperimental Ġresults Ġshow Ġthat Ġthe Ġenc oder - dec oder - re construct or Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon ĠEnglish - Japanese Ġtranslation Ġtask , Ġand Ġthe Ġenc oder - dec oder - re construct or Ġcan Ġnot Ġbe Ġtrained Ġwell Ġwithout Ġpre - training , Ġso Ġit Ġproves Ġthat Ġwe Ġhave Ġto Ġtrain Ġthe Ġforward Ġtranslation Ġmodel Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġas Ġpre - training .
Introduction Recently, neural machine translation (NMT) has gained popularity in the field of machine translation. The conventional encoder-decoder NMT proposed by Cho2014 uses two recurrent neural networks (RNN): one is an encoder, which encodes a source sequence into a fixed-length vector, and the other is a decoder, which decodes the vector into a target sequence. A newly proposed attention-based NMT by DzmitryBahdana2014 can predict output words using the weights of each hidden state of the encoder by the attention mechanism, improving the adequacy of translation. Even with the success of attention-based models, a number of open questions remain in NMT. Tu2016 argued two of the common problems are over-translation: some words are repeatedly translated unnecessary and under-translation: some words are mistakenly untranslated. This is due to the fact that NMT can not completely convert the information from the source sentence to the target sentence. Mi2016a and Feng2016 pointed out that NMT lacks the notion of coverage vector in phrase-based statistical machine translation (PBSMT), so unless otherwise specified, there is no way to prevent missing translations. Another problem in NMT is an objective function. NMT is optimized by cross-entropy; therefore, it does not directly maximize the translation accuracy. Shen2016 pointed out that optimization by cross-entropy is not appropriate and proposed a method of optimization based on a translation accuracy score, such as expected BLEU, which led to improvement of translation accuracy. However, BLEU is an evaluation metric based on n-gram precision; therefore, repetition of some words may be present in the translation even though the BLEU score is improved. To address to problem of repeating and missing words in the translation, tu2016neural introduce an encoder-decoder-reconstructor framework that optimizes NMT by back-translation from the output sentences into the original source sentences. In their method, after training the forward translation in a manner similar to the conventional attention-based NMT, they train a back-translation model from the hidden state of the decoder into the source sequence by a new decoder to enforce agreement between source and target sentences. In order to confirm the language independence of the framework, we experiment on two parallel corpora of English-Japanese and Japanese-English translation tasks using encode-decoder-reconstructor. Our experiments show that their method offers significant improvement in BLEU scores and alleviates the problem of repeating and missing words in the translation on English-Japanese translation task, though the difference is not significant on Japanese-English translation task. In addition, we jointly train a model of forward translation and back-translation without pre-training, and then evaluate this model. As a result, the encoder-decoder-reconstructor can not be trained well without pre-training, so it proves that we have to train the forward translation model in a manner similar to the conventional attention-based NMT as pre-training. The main contributions of this paper are as follows: Related Works Several studies have addressed the NMT-specific problem of missing or repeating words. Niehues2016 optimized NMT by adding the outputs of PBSMT to the input of NMT. Mi2016a and Feng2016 introduced a distributed version of coverage vector taken from PBSMT to consider which words have been already translated. All these methods, including ours, employ information of the source sentence to improve the quality of translation, but our method uses back-translation to ensure that there is no inconsistency. Unlike other methods, once learned, our method is identical to the conventional NMT model, so it does not need any additional parameters such as coverage vector or a PBSMT system for testing. The attention mechanism proposed by Meng2016 considers not only the hidden states of the encoder but also the hidden states of the decoder so that over-translation can be relaxed. In addition, the attention mechanism proposed by Feng2016 computes a context vector by considering the previous context vector to prevent over-translation. These works indirectly reduce repeating and missing words, while we directly penalize translation mismatch by considering back-translation. The encoder-decoder-reconstructor framework for NMT proposed by tu2016neural optimizes NMT by reconstructor using back-translation. They consider likelihood of both of forward translation and back-translation, and then this framework offers significant improvement in BLEU scores and alleviates the problem of repeating and missing words in the translation on a Chinese-English translation task. Neural Machine Translation Here, we describe the attention-based NMT proposed by DzmitryBahdana2014 as shown in Figure FIGREF1 . The input sequence ( INLINEFORM0 ) is converted into a fixed-length vector by the encoder using an RNN. At each time step INLINEFORM1 , the hidden state INLINEFORM2 of the encoder is presented as DISPLAYFORM0 using a bidirectional RNN. The forward state INLINEFORM0 and the backward state INLINEFORM1 are computed by DISPLAYFORM0 and DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 are nonlinear functions. The hidden states INLINEFORM2 are converted into a fixed-length vector INLINEFORM3 as DISPLAYFORM0 where INLINEFORM0 is a nonlinear function. The fixed-length vector INLINEFORM0 generated by the encoder is converted into the target sequence ( INLINEFORM1 ) by the decoder using an RNN. At each time step INLINEFORM2 , the conditional probability of the output word INLINEFORM3 is computed by DISPLAYFORM0 where INLINEFORM0 is a nonlinear function. The hidden state INLINEFORM1 of the decoder is presented as DISPLAYFORM0 using the hidden state INLINEFORM0 and the target word INLINEFORM1 at the previous time step and the context vector INLINEFORM2 . The context vector INLINEFORM0 is a weighted sum of each hidden state INLINEFORM1 of the encoder. It is presented as DISPLAYFORM0 and its weight INLINEFORM0 is a normalized probability distribution. It is computed by DISPLAYFORM0 and DISPLAYFORM0 where INLINEFORM0 is a weight vector and INLINEFORM1 and INLINEFORM2 are weight matrices. The objective function is defined by DISPLAYFORM0 where INLINEFORM0 is the number of data and INLINEFORM1 is a model parameter. Incidentally, as a nonlinear function, the hyperbolic tangent function or the rectified linear unit are generally used. Architecture Next, we describe the encoder-decoder-reconstructor framework for NMT proposed by tu2016neural as shown in Figure FIGREF1 . The encoder-decoder-reconstructor consists of two components: the standard encoder-decoder as an attention-based NMT proposed by DzmitryBahdana2014 and the reconstructor which back-translates from the hidden states of decoder to the source sentence. In their method, the hidden state of the decoder is back-translated into the source sequence ( INLINEFORM0 ) by the reconstructor for the back-translation. At each time step INLINEFORM1 , the conditional probability of the output word INLINEFORM2 is computed by DISPLAYFORM0 where INLINEFORM0 is a nonlinear function. The hidden state INLINEFORM1 of the reconstructor is presented as DISPLAYFORM0 using the hidden state INLINEFORM0 and the source word INLINEFORM1 at the previous time step and the new context vector (inverse context vector) INLINEFORM2 . The inverse context vector INLINEFORM0 is a weighted sum of each hidden state INLINEFORM1 of the decoder (on forward translation). It is presented as DISPLAYFORM0 and its weight INLINEFORM0 is a normalized probability distribution. It is computed by DISPLAYFORM0 and DISPLAYFORM0 where INLINEFORM0 is a weight vector and INLINEFORM1 and INLINEFORM2 are weight matrices. The objective function is defined by DISPLAYFORM0 where INLINEFORM0 is the number of data, INLINEFORM1 and INLINEFORM2 are model parameters and INLINEFORM3 is a hyper-parameter which can consider the weight between forward translation and back-translation. This objective function consists of two parts: forward measures translation fluency, and backward measures translation adequacy. Thus, the combined objective function is more consistent with the goal of enhancing overall translation quality, and can more effectively guide the parameter training for making better translation. Training The encoder-decoder-reconstructor is trained with likelihood of both the encoder-decoder and the reconstructor on a set of training datasets. tu2016neural trained a back-translation model from the hidden state of the decoder into the source sequence by reconstructor to enforce agreement between source and target sentences using Equation EQREF23 after training the forward translation in a manner similar to the conventional attention-based NMT using Equation EQREF13 . In addition, we experiment to jointly train a model of forward translation and back-translation without pre-training. It may learn a globally optimal model compared to locally optimal model pre-trained using the forward translation. Testing tu2016neural used a beam search to predict target sentences that approximately maximizes both of forward translation and back-translation on testing. In this paper, however, we do not use a beam search for simplicity and effectiveness. Experiments We evaluated the encoder-decoder-reconstructor framework for NMT on English-Japanese and Japanese-English translation tasks. Datasets We used two parallel corpora: Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF0 and NTCIR PatentMT Parallel Corpus BIBREF1 . Regarding the training data of ASPEC, we used only the first 1 million sentences sorted by sentence-alignment similarity. Japanese sentences were segmented by the morphological analyzer MeCab (version 0.996, IPADIC), and English sentences were tokenized by tokenizer.perl of Moses. Table TABREF14 shows the numbers of the sentences in each corpus. Note that sentences with more than 40 words were excluded from the training data. Models We used the attention-based NMT BIBREF2 as a baseline-NMT, the encoder-decoder-reconstructor BIBREF3 and the encoder-decoder-reconstructor that jointly trained forward translation and back-translation without pre-training. The RNN used in the experiments had 512 hidden units, 512 embedding units, 30,000 vocabulary size and 64 batch size. We used Adagrad (initial learning rate 0.01) for optimizing model parameters. We trained our model on GeForce GTX TITAN X GPU. Note that we set the hyper-parameter INLINEFORM0 on the encoder-decoder-reconstructor same as tu2016neural. Results Tables TABREF15 and TABREF16 show the translation accuracy in BLEU scores, the INLINEFORM0 -value of the significance test by bootstrap resampling BIBREF4 and training time in hours until convergence. The encoder-decoder-reconstructor BIBREF3 requires slightly longer time to train than the baseline NMT, but we emphasize that decoding time remains the same with the encoder-decoder-reconstructor and baseline-NMT. The results show that the encoder-decoder-reconstructor BIBREF3 significantly improves translation accuracy by 1.01 points on ASPEC and 1.37 points on NTCIR in English-Japanese translation ( INLINEFORM1 ). However, it does not significantly improve translation accuracy in Japanese-English translation. In addition, it is proved that the encoder-decoder-reconstructor without pre-training worsens rather than improves translation accuracy. Table TABREF24 shows examples of outputs of English-Japanese translations. In Example 1, “UTF8min乱 流 粘性 と 数値 粘性 の 大小 関係 により ,” (on the basis of the relation between turbulent viscosity and numerical viscosity in size) is missing in the output of baseline-NMT, but “UTF8min乱 流 粘性 と 数値 的 粘性 の 関係 を 基 に” (on the basis of the relation between turbulent viscosity and numerical viscosity) is present in the output of encoder-decoder-reconstructor. In Example 2, “UTF8min新生児” (newborn infant) and “UTF8min30歳以上の” (of 30 ‐ year ‐ old or more) are repeated in the output of baseline-NMT, but they appear only once in the output of encoder-decoder-reconstructor. In addition, Figures FIGREF25 and FIGREF25 show the attention layer on baseline-NMT and encoder-decoder-reconstructor in each example. In Figure FIGREF25 , although the attention layer of baseline NMT attends input word “turbulent”, the decoder does not output “UTF8min乱流” (turbulent) but “UTF8min検討” (examined) at the 13th word. Thus, under-translation may be resulted from the hidden layer or the embedding layer instead of the attention layer. In Figure FIGREF25 , it is found that the attention layer of baseline-NMT repeatedly attends input words “newborn infant” and “30 ‐ year ‐ old or more”. Consequently, the decoder repeatedly outputs “UTF8min新生児” (newborn infant) and “UTF8min30歳以上の” (of 30 ‐ year ‐ old or more). On the other hand, the attention layer of encoder-decoder-reconstructor almost correctly attends input words. Table TABREF28 shows a comparison of the number of word occurrences for each corpus and model. The columns show (i) the number of words that appear more frequently than the counterparts in the reference, and (ii) the number of words that appear more than once but are not included in the reference. Note that these numbers do not include unknown words, so (iii) shows the number of unknown words. In all the cases, the number of occurrence of redundant words is reduced in encoder-decoder-reconstructor. Thus, we confirmed that encoder-decoder-reconstructor achieves reduction of repeating and missing words while maintaining the quality of translation. Conclusion In this paper, we evaluated the encoder-decoder-reconstructor on English-Japanese and Japanese-English translation tasks. In addition, we evaluate the effectiveness of pre-training by comparing it with a jointly-trained model of forward translation and back-translation. Experimental results show that the encoder-decoder-reconstructor offers significant improvement in BLEU scores and alleviates the problem of repeating and missing words in the translation on English-Japanese translation task, and the encoder-decoder-reconstructor can not be trained well without pre-training, so it proves that we have to train the forward translation model in a manner similar to the conventional attention-based NMT as pre-training.
What parallel corpus did they use?
6024039bbd1118c5dab86c41cce1175d99f10a25
What Ġparallel Ġcorpus Ġdid Ġthey Ġuse ?
Asian ĠScientific ĠPaper ĠEx cerpt ĠCorpus Ġ( AS P EC ) ĠB IB REF 0 , ĠN TC IR ĠPatent MT ĠParallel ĠCorpus ĠB IB REF 1
<s> What Ġparallel Ġcorpus Ġdid Ġthey Ġuse ? </s> Introduction </s> Recently , Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġgained Ġpopularity Ġin Ġthe Ġfield Ġof Ġmachine Ġtranslation . ĠThe Ġconventional Ġenc oder - dec oder ĠN MT Ġproposed Ġby ĠCho 2014 Ġuses Ġtwo Ġrecurrent Ġneural Ġnetworks Ġ( R NN ): Ġone Ġis Ġan Ġenc oder , Ġwhich Ġenc odes Ġa Ġsource Ġsequence Ġinto Ġa Ġfixed - length Ġvector , Ġand Ġthe Ġother Ġis Ġa Ġdec oder , Ġwhich Ġdec odes Ġthe Ġvector Ġinto Ġa Ġtarget Ġsequence . ĠA Ġnewly Ġproposed Ġattention - based ĠN MT Ġby ĠD z mit ry Bah d ana 2014 Ġcan Ġpredict Ġoutput Ġwords Ġusing Ġthe Ġweights Ġof Ġeach Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġby Ġthe Ġattention Ġmechanism , Ġimproving Ġthe Ġadequ acy Ġof Ġtranslation . </s> Even Ġwith Ġthe Ġsuccess Ġof Ġattention - based Ġmodels , Ġa Ġnumber Ġof Ġopen Ġquestions Ġremain Ġin ĠN MT . ĠTu 2016 Ġargued Ġtwo Ġof Ġthe Ġcommon Ġproblems Ġare Ġover - translation : Ġsome Ġwords Ġare Ġrepeatedly Ġtranslated Ġunnecessary Ġand Ġunder - translation : Ġsome Ġwords Ġare Ġmistakenly Ġunt rans lated . ĠThis Ġis Ġdue Ġto Ġthe Ġfact Ġthat ĠN MT Ġcan Ġnot Ġcompletely Ġconvert Ġthe Ġinformation Ġfrom Ġthe Ġsource Ġsentence Ġto Ġthe Ġtarget Ġsentence . ĠMi 2016 a Ġand ĠFeng 2016 Ġpointed Ġout Ġthat ĠN MT Ġlacks Ġthe Ġnotion Ġof Ġcoverage Ġvector Ġin Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( P BS MT ), Ġso Ġunless Ġotherwise Ġspecified , Ġthere Ġis Ġno Ġway Ġto Ġprevent Ġmissing Ġtranslations . </s> Another Ġproblem Ġin ĠN MT Ġis Ġan Ġobjective Ġfunction . ĠN MT Ġis Ġoptimized Ġby Ġcross - ent ropy ; Ġtherefore , Ġit Ġdoes Ġnot Ġdirectly Ġmaximize Ġthe Ġtranslation Ġaccuracy . ĠShen 2016 Ġpointed Ġout Ġthat Ġoptimization Ġby Ġcross - ent ropy Ġis Ġnot Ġappropriate Ġand Ġproposed Ġa Ġmethod Ġof Ġoptimization Ġbased Ġon Ġa Ġtranslation Ġaccuracy Ġscore , Ġsuch Ġas Ġexpected ĠB LE U , Ġwhich Ġled Ġto Ġimprovement Ġof Ġtranslation Ġaccuracy . ĠHowever , ĠB LE U Ġis Ġan Ġevaluation Ġmetric Ġbased Ġon Ġn - gram Ġprecision ; Ġtherefore , Ġrepetition Ġof Ġsome Ġwords Ġmay Ġbe Ġpresent Ġin Ġthe Ġtranslation Ġeven Ġthough Ġthe ĠB LE U Ġscore Ġis Ġimproved . </s> To Ġaddress Ġto Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation , Ġtu 2016 ne ural Ġintroduce Ġan Ġenc oder - dec oder - re construct or Ġframework Ġthat Ġoptim izes ĠN MT Ġby Ġback - translation Ġfrom Ġthe Ġoutput Ġsentences Ġinto Ġthe Ġoriginal Ġsource Ġsentences . ĠIn Ġtheir Ġmethod , Ġafter Ġtraining Ġthe Ġforward Ġtranslation Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT , Ġthey Ġtrain Ġa Ġback - translation Ġmodel Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġinto Ġthe Ġsource Ġsequence Ġby Ġa Ġnew Ġdec oder Ġto Ġenforce Ġagreement Ġbetween Ġsource Ġand Ġtarget Ġsentences . </s> In Ġorder Ġto Ġconfirm Ġthe Ġlanguage Ġindependence Ġof Ġthe Ġframework , Ġwe Ġexperiment Ġon Ġtwo Ġparallel Ġcorpor a Ġof ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks Ġusing Ġencode - dec oder - re construct or . ĠOur Ġexperiments Ġshow Ġthat Ġtheir Ġmethod Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon ĠEnglish - Japanese Ġtranslation Ġtask , Ġthough Ġthe Ġdifference Ġis Ġnot Ġsignificant Ġon ĠJapanese - English Ġtranslation Ġtask . </s> In Ġaddition , Ġwe Ġjointly Ġtrain Ġa Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training , Ġand Ġthen Ġevaluate Ġthis Ġmodel . ĠAs Ġa Ġresult , Ġthe Ġenc oder - dec oder - re construct or Ġcan Ġnot Ġbe Ġtrained Ġwell Ġwithout Ġpre - training , Ġso Ġit Ġproves Ġthat Ġwe Ġhave Ġto Ġtrain Ġthe Ġforward Ġtranslation Ġmodel Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġas Ġpre - training . </s> The Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> Related ĠWorks </s> Several Ġstudies Ġhave Ġaddressed Ġthe ĠN MT - specific Ġproblem Ġof Ġmissing Ġor Ġrepeating Ġwords . ĠN ie h ues 2016 Ġoptimized ĠN MT Ġby Ġadding Ġthe Ġoutputs Ġof ĠPBS MT Ġto Ġthe Ġinput Ġof ĠN MT . ĠMi 2016 a Ġand ĠFeng 2016 Ġintroduced Ġa Ġdistributed Ġversion Ġof Ġcoverage Ġvector Ġtaken Ġfrom ĠPBS MT Ġto Ġconsider Ġwhich Ġwords Ġhave Ġbeen Ġalready Ġtranslated . ĠAll Ġthese Ġmethods , Ġincluding Ġours , Ġemploy Ġinformation Ġof Ġthe Ġsource Ġsentence Ġto Ġimprove Ġthe Ġquality Ġof Ġtranslation , Ġbut Ġour Ġmethod Ġuses Ġback - translation Ġto Ġensure Ġthat Ġthere Ġis Ġno Ġinconsistency . ĠUnlike Ġother Ġmethods , Ġonce Ġlearned , Ġour Ġmethod Ġis Ġidentical Ġto Ġthe Ġconventional ĠN MT Ġmodel , Ġso Ġit Ġdoes Ġnot Ġneed Ġany Ġadditional Ġparameters Ġsuch Ġas Ġcoverage Ġvector Ġor Ġa ĠPBS MT Ġsystem Ġfor Ġtesting . </s> The Ġattention Ġmechanism Ġproposed Ġby ĠMeng 2016 Ġconsiders Ġnot Ġonly Ġthe Ġhidden Ġstates Ġof Ġthe Ġenc oder Ġbut Ġalso Ġthe Ġhidden Ġstates Ġof Ġthe Ġdec oder Ġso Ġthat Ġover - translation Ġcan Ġbe Ġrelaxed . ĠIn Ġaddition , Ġthe Ġattention Ġmechanism Ġproposed Ġby ĠFeng 2016 Ġcomp utes Ġa Ġcontext Ġvector Ġby Ġconsidering Ġthe Ġprevious Ġcontext Ġvector Ġto Ġprevent Ġover - translation . ĠThese Ġworks Ġindirectly Ġreduce Ġrepeating Ġand Ġmissing Ġwords , Ġwhile Ġwe Ġdirectly Ġpenal ize Ġtranslation Ġmismatch Ġby Ġconsidering Ġback - translation . </s> The Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġproposed Ġby Ġtu 2016 ne ural Ġoptim izes ĠN MT Ġby Ġreconstruct or Ġusing Ġback - translation . ĠThey Ġconsider Ġlikelihood Ġof Ġboth Ġof Ġforward Ġtranslation Ġand Ġback - translation , Ġand Ġthen Ġthis Ġframework Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon Ġa ĠChinese - English Ġtranslation Ġtask . </s> Ne ural ĠMachine ĠTranslation </s> Here , Ġwe Ġdescribe Ġthe Ġattention - based ĠN MT Ġproposed Ġby ĠD z mit ry Bah d ana 2014 Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. </s> The Ġinput Ġsequence Ġ( ĠIN LINE FORM 0 Ġ) Ġis Ġconverted Ġinto Ġa Ġfixed - length Ġvector Ġby Ġthe Ġenc oder Ġusing Ġan ĠR NN . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 1 Ġ, Ġthe Ġhidden Ġstate ĠIN LINE FORM 2 Ġof Ġthe Ġenc oder Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġa Ġbid irection al ĠR NN . ĠThe Ġforward Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġbackward Ġstate ĠIN LINE FORM 1 Ġare Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġnon linear Ġfunctions . ĠThe Ġhidden Ġstates ĠIN LINE FORM 2 Ġare Ġconverted Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġas ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . </s> The Ġfixed - length Ġvector ĠIN LINE FORM 0 Ġgenerated Ġby Ġthe Ġenc oder Ġis Ġconverted Ġinto Ġthe Ġtarget Ġsequence Ġ( ĠIN LINE FORM 1 Ġ) Ġby Ġthe Ġdec oder Ġusing Ġan ĠR NN . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 2 Ġ, Ġthe Ġconditional Ġprobability Ġof Ġthe Ġoutput Ġword ĠIN LINE FORM 3 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . ĠThe Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġdec oder Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġthe Ġhidden Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġtarget Ġword ĠIN LINE FORM 1 Ġat Ġthe Ġprevious Ġtime Ġstep Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 2 Ġ. </s> The Ġcontext Ġvector ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġsum Ġof Ġeach Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġenc oder . ĠIt Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> and Ġits Ġweight ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġprobability Ġdistribution . ĠIt Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweight Ġvector Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . </s> The Ġobjective Ġfunction Ġis Ġdefined Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġdata Ġand ĠIN LINE FORM 1 Ġis Ġa Ġmodel Ġparameter . </s> Inc identally , Ġas Ġa Ġnon linear Ġfunction , Ġthe Ġhyper b olic Ġtang ent Ġfunction Ġor Ġthe Ġrect ified Ġlinear Ġunit Ġare Ġgenerally Ġused . </s> Arch itect ure </s> Next , Ġwe Ġdescribe Ġthe Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġproposed Ġby Ġtu 2016 ne ural Ġas Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ. ĠThe Ġenc oder - dec oder - re construct or Ġconsists Ġof Ġtwo Ġcomponents : Ġthe Ġstandard Ġenc oder - dec oder Ġas Ġan Ġattention - based ĠN MT Ġproposed Ġby ĠD z mit ry Bah d ana 2014 Ġand Ġthe Ġreconstruct or Ġwhich Ġback - trans l ates Ġfrom Ġthe Ġhidden Ġstates Ġof Ġdec oder Ġto Ġthe Ġsource Ġsentence . </s> In Ġtheir Ġmethod , Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġis Ġback - trans lated Ġinto Ġthe Ġsource Ġsequence Ġ( ĠIN LINE FORM 0 Ġ) Ġby Ġthe Ġreconstruct or Ġfor Ġthe Ġback - translation . ĠAt Ġeach Ġtime Ġstep ĠIN LINE FORM 1 Ġ, Ġthe Ġconditional Ġprobability Ġof Ġthe Ġoutput Ġword ĠIN LINE FORM 2 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon linear Ġfunction . ĠThe Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġreconstruct or Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> using Ġthe Ġhidden Ġstate ĠIN LINE FORM 0 Ġand Ġthe Ġsource Ġword ĠIN LINE FORM 1 Ġat Ġthe Ġprevious Ġtime Ġstep Ġand Ġthe Ġnew Ġcontext Ġvector Ġ( in verse Ġcontext Ġvector ) ĠIN LINE FORM 2 Ġ. </s> The Ġinverse Ġcontext Ġvector ĠIN LINE FORM 0 Ġis Ġa Ġweighted Ġsum Ġof Ġeach Ġhidden Ġstate ĠIN LINE FORM 1 Ġof Ġthe Ġdec oder Ġ( on Ġforward Ġtranslation ). ĠIt Ġis Ġpresented Ġas ĠDIS PLAY FORM 0 </s> and Ġits Ġweight ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġprobability Ġdistribution . ĠIt Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> and ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġweight Ġvector Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . </s> The Ġobjective Ġfunction Ġis Ġdefined Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġdata , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġmodel Ġparameters Ġand ĠIN LINE FORM 3 Ġis Ġa Ġhyper - param eter Ġwhich Ġcan Ġconsider Ġthe Ġweight Ġbetween Ġforward Ġtranslation Ġand Ġback - translation . </s> This Ġobjective Ġfunction Ġconsists Ġof Ġtwo Ġparts : Ġforward Ġmeasures Ġtranslation Ġflu ency , Ġand Ġbackward Ġmeasures Ġtranslation Ġadequ acy . ĠThus , Ġthe Ġcombined Ġobjective Ġfunction Ġis Ġmore Ġconsistent Ġwith Ġthe Ġgoal Ġof Ġenhancing Ġoverall Ġtranslation Ġquality , Ġand Ġcan Ġmore Ġeffectively Ġguide Ġthe Ġparameter Ġtraining Ġfor Ġmaking Ġbetter Ġtranslation . </s> Training </s> The Ġenc oder - dec oder - re construct or Ġis Ġtrained Ġwith Ġlikelihood Ġof Ġboth Ġthe Ġenc oder - dec oder Ġand Ġthe Ġreconstruct or Ġon Ġa Ġset Ġof Ġtraining Ġdatasets . Ġtu 2016 ne ural Ġtrained Ġa Ġback - translation Ġmodel Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġinto Ġthe Ġsource Ġsequence Ġby Ġreconstruct or Ġto Ġenforce Ġagreement Ġbetween Ġsource Ġand Ġtarget Ġsentences Ġusing ĠEqu ation ĠEQ REF 23 Ġafter Ġtraining Ġthe Ġforward Ġtranslation Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġusing ĠEqu ation ĠEQ REF 13 Ġ. </s> In Ġaddition , Ġwe Ġexperiment Ġto Ġjointly Ġtrain Ġa Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training . ĠIt Ġmay Ġlearn Ġa Ġglobally Ġoptimal Ġmodel Ġcompared Ġto Ġlocally Ġoptimal Ġmodel Ġpre - trained Ġusing Ġthe Ġforward Ġtranslation . </s> Testing </s> tu 2016 ne ural Ġused Ġa Ġbeam Ġsearch Ġto Ġpredict Ġtarget Ġsentences Ġthat Ġapproximately Ġmaxim izes Ġboth Ġof Ġforward Ġtranslation Ġand Ġback - translation Ġon Ġtesting . ĠIn Ġthis Ġpaper , Ġhowever , Ġwe Ġdo Ġnot Ġuse Ġa Ġbeam Ġsearch Ġfor Ġsimplicity Ġand Ġeffectiveness . </s> Exper iments </s> We Ġevaluated Ġthe Ġenc oder - dec oder - re construct or Ġframework Ġfor ĠN MT Ġon ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks . </s> Dat as ets </s> We Ġused Ġtwo Ġparallel Ġcorpor a : ĠAsian ĠScientific ĠPaper ĠEx cerpt ĠCorpus Ġ( AS P EC ) ĠB IB REF 0 Ġand ĠN TC IR ĠPatent MT ĠParallel ĠCorpus ĠB IB REF 1 Ġ. ĠRegarding Ġthe Ġtraining Ġdata Ġof ĠASP EC , Ġwe Ġused Ġonly Ġthe Ġfirst Ġ1 Ġmillion Ġsentences Ġsorted Ġby Ġsentence - al ignment Ġsimilarity . ĠJapanese Ġsentences Ġwere Ġsegment ed Ġby Ġthe Ġmorph ological Ġanaly zer ĠMe C ab Ġ( version Ġ0 . 996 , ĠIP AD IC ), Ġand ĠEnglish Ġsentences Ġwere Ġtoken ized Ġby Ġtoken izer . per l Ġof ĠMoses . ĠTable ĠT AB REF 14 Ġshows Ġthe Ġnumbers Ġof Ġthe Ġsentences Ġin Ġeach Ġcorpus . ĠNote Ġthat Ġsentences Ġwith Ġmore Ġthan Ġ40 Ġwords Ġwere Ġexcluded Ġfrom Ġthe Ġtraining Ġdata . </s> Mod els </s> We Ġused Ġthe Ġattention - based ĠN MT ĠB IB REF 2 Ġas Ġa Ġbaseline - N MT , Ġthe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġand Ġthe Ġenc oder - dec oder - re construct or Ġthat Ġjointly Ġtrained Ġforward Ġtranslation Ġand Ġback - translation Ġwithout Ġpre - training . ĠThe ĠR NN Ġused Ġin Ġthe Ġexperiments Ġhad Ġ512 Ġhidden Ġunits , Ġ512 Ġembed ding Ġunits , Ġ30 , 000 Ġvocabulary Ġsize Ġand Ġ64 Ġbatch Ġsize . ĠWe Ġused ĠAd ag rad Ġ( initial Ġlearning Ġrate Ġ0 . 01 ) Ġfor Ġoptimizing Ġmodel Ġparameters . ĠWe Ġtrained Ġour Ġmodel Ġon ĠGeForce ĠGTX ĠTIT AN ĠX ĠGPU . ĠNote Ġthat Ġwe Ġset Ġthe Ġhyper - param eter ĠIN LINE FORM 0 Ġon Ġthe Ġenc oder - dec oder - re construct or Ġsame Ġas Ġtu 2016 ne ural . </s> Results </s> T ables ĠT AB REF 15 Ġand ĠT AB REF 16 Ġshow Ġthe Ġtranslation Ġaccuracy Ġin ĠB LE U Ġscores , Ġthe ĠIN LINE FORM 0 Ġ- value Ġof Ġthe Ġsignificance Ġtest Ġby Ġboot strap Ġres am pling ĠB IB REF 4 Ġand Ġtraining Ġtime Ġin Ġhours Ġuntil Ġconvergence . ĠThe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġrequires Ġslightly Ġlonger Ġtime Ġto Ġtrain Ġthan Ġthe Ġbaseline ĠN MT , Ġbut Ġwe Ġemphasize Ġthat Ġdecoding Ġtime Ġremains Ġthe Ġsame Ġwith Ġthe Ġenc oder - dec oder - re construct or Ġand Ġbaseline - N MT . ĠThe Ġresults Ġshow Ġthat Ġthe Ġenc oder - dec oder - re construct or ĠB IB REF 3 Ġsignificantly Ġimproves Ġtranslation Ġaccuracy Ġby Ġ1 . 01 Ġpoints Ġon ĠASP EC Ġand Ġ1 . 37 Ġpoints Ġon ĠN TC IR Ġin ĠEnglish - Japanese Ġtranslation Ġ( ĠIN LINE FORM 1 Ġ). ĠHowever , Ġit Ġdoes Ġnot Ġsignificantly Ġimprove Ġtranslation Ġaccuracy Ġin ĠJapanese - English Ġtranslation . ĠIn Ġaddition , Ġit Ġis Ġproved Ġthat Ġthe Ġenc oder - dec oder - re construct or Ġwithout Ġpre - training Ġwors ens Ġrather Ġthan Ġimproves Ġtranslation Ġaccuracy . </s> Table ĠT AB REF 24 Ġshows Ġexamples Ġof Ġoutputs Ġof ĠEnglish - Japanese Ġtranslations . ĠIn ĠExample Ġ1 , ĠâĢ ľ UTF 8 min ä¹ ± Ġæ µ ģ Ġç ² ĺ æĢ § Ġãģ ¨ Ġæ ķ ° å Ģ ¤ Ġç ² ĺ æĢ § Ġ ãģ® Ġå¤ § å° ı Ġé ĸ ¢ ä¿ Ĥ Ġãģ « ãĤ Ī ãĤĬ Ġ, âĢ Ŀ Ġ( on Ġthe Ġbasis Ġof Ġthe Ġrelation Ġbetween Ġturbulent Ġvis cos ity Ġand Ġnumerical Ġvis cos ity Ġin Ġsize ) Ġis Ġmissing Ġin Ġthe Ġoutput Ġof Ġbaseline - N MT , Ġbut ĠâĢ ľ UTF 8 min ä¹ ± Ġæ µ ģ Ġç ² ĺ æĢ § Ġãģ ¨ Ġæ ķ ° å Ģ ¤ Ġç ļ Ħ Ġç ² ĺ æĢ § Ġ ãģ® Ġé ĸ ¢ ä¿ Ĥ ĠãĤ Ĵ Ġå Ł º Ġãģ « âĢ Ŀ Ġ( on Ġthe Ġbasis Ġof Ġthe Ġrelation Ġbetween Ġturbulent Ġvis cos ity Ġand Ġnumerical Ġvis cos ity ) Ġis Ġpresent Ġin Ġthe Ġoutput Ġof Ġenc oder - dec oder - re construct or . ĠIn ĠExample Ġ2 , ĠâĢ ľ UTF 8 min æĸ ° çĶŁ åħ Ĳ âĢ Ŀ Ġ( new born Ġinfant ) Ġand ĠâĢ ľ UTF 8 min 30 æŃ ³ ä» ¥ ä¸Ĭ ãģ® âĢ Ŀ Ġ( of Ġ30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore ) Ġare Ġrepeated Ġin Ġthe Ġoutput Ġof Ġbaseline - N MT , Ġbut Ġthey Ġappear Ġonly Ġonce Ġin Ġthe Ġoutput Ġof Ġenc oder - dec oder - re construct or . </s> In Ġaddition , ĠFigures ĠFIG REF 25 Ġand ĠFIG REF 25 Ġshow Ġthe Ġattention Ġlayer Ġon Ġbaseline - N MT Ġand Ġenc oder - dec oder - re construct or Ġin Ġeach Ġexample . ĠIn ĠFigure ĠFIG REF 25 Ġ, Ġalthough Ġthe Ġattention Ġlayer Ġof Ġbaseline ĠN MT Ġattends Ġinput Ġword ĠâĢ ľ t urb ulent âĢ Ŀ , Ġthe Ġdec oder Ġdoes Ġnot Ġoutput ĠâĢ ľ UTF 8 min ä¹ ± æµ ģ âĢ Ŀ Ġ( t urb ulent ) Ġbut ĠâĢ ľ UTF 8 min æ ¤ ľ è ¨ İ âĢ Ŀ Ġ( ex am ined ) Ġat Ġthe Ġ13 th Ġword . ĠThus , Ġunder - translation Ġmay Ġbe Ġresulted Ġfrom Ġthe Ġhidden Ġlayer Ġor Ġthe Ġembed ding Ġlayer Ġinstead Ġof Ġthe Ġattention Ġlayer . ĠIn ĠFigure ĠFIG REF 25 Ġ, Ġit Ġis Ġfound Ġthat Ġthe Ġattention Ġlayer Ġof Ġbaseline - N MT Ġrepeatedly Ġattends Ġinput Ġwords ĠâĢ ľ new born Ġinfant âĢ Ŀ Ġand ĠâĢ ľ 30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore âĢ Ŀ . ĠConsequently , Ġthe Ġdec oder Ġrepeatedly Ġoutputs ĠâĢ ľ UTF 8 min æĸ ° çĶŁ åħ Ĳ âĢ Ŀ Ġ( new born Ġinfant ) Ġand ĠâĢ ľ UTF 8 min 30 æŃ ³ ä» ¥ ä¸Ĭ ãģ® âĢ Ŀ Ġ( of Ġ30 ĠâĢ Ĳ Ġyear ĠâĢ Ĳ Ġold Ġor Ġmore ). ĠOn Ġthe Ġother Ġhand , Ġthe Ġattention Ġlayer Ġof Ġenc oder - dec oder - re construct or Ġalmost Ġcorrectly Ġattends Ġinput Ġwords . </s> Table ĠT AB REF 28 Ġshows Ġa Ġcomparison Ġof Ġthe Ġnumber Ġof Ġword Ġoccurrences Ġfor Ġeach Ġcorpus Ġand Ġmodel . ĠThe Ġcolumns Ġshow Ġ( i ) Ġthe Ġnumber Ġof Ġwords Ġthat Ġappear Ġmore Ġfrequently Ġthan Ġthe Ġcounterparts Ġin Ġthe Ġreference , Ġand Ġ( ii ) Ġthe Ġnumber Ġof Ġwords Ġthat Ġappear Ġmore Ġthan Ġonce Ġbut Ġare Ġnot Ġincluded Ġin Ġthe Ġreference . ĠNote Ġthat Ġthese Ġnumbers Ġdo Ġnot Ġinclude Ġunknown Ġwords , Ġso Ġ( iii ) Ġshows Ġthe Ġnumber Ġof Ġunknown Ġwords . ĠIn Ġall Ġthe Ġcases , Ġthe Ġnumber Ġof Ġoccurrence Ġof Ġredundant Ġwords Ġis Ġreduced Ġin Ġenc oder - dec oder - re construct or . ĠThus , Ġwe Ġconfirmed Ġthat Ġenc oder - dec oder - re construct or Ġachieves Ġreduction Ġof Ġrepeating Ġand Ġmissing Ġwords Ġwhile Ġmaintaining Ġthe Ġquality Ġof Ġtranslation . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġevaluated Ġthe Ġenc oder - dec oder - re construct or Ġon ĠEnglish - Japanese Ġand ĠJapanese - English Ġtranslation Ġtasks . ĠIn Ġaddition , Ġwe Ġevaluate Ġthe Ġeffectiveness Ġof Ġpre - training Ġby Ġcomparing Ġit Ġwith Ġa Ġjointly - trained Ġmodel Ġof Ġforward Ġtranslation Ġand Ġback - translation . ĠExperimental Ġresults Ġshow Ġthat Ġthe Ġenc oder - dec oder - re construct or Ġoffers Ġsignificant Ġimprovement Ġin ĠB LE U Ġscores Ġand Ġallev iates Ġthe Ġproblem Ġof Ġrepeating Ġand Ġmissing Ġwords Ġin Ġthe Ġtranslation Ġon ĠEnglish - Japanese Ġtranslation Ġtask , Ġand Ġthe Ġenc oder - dec oder - re construct or Ġcan Ġnot Ġbe Ġtrained Ġwell Ġwithout Ġpre - training , Ġso Ġit Ġproves Ġthat Ġwe Ġhave Ġto Ġtrain Ġthe Ġforward Ġtranslation Ġmodel Ġin Ġa Ġmanner Ġsimilar Ġto Ġthe Ġconventional Ġattention - based ĠN MT Ġas Ġpre - training .