Yes
BOOLEAN
Yes
BOOLEAN
FLOAT SELECTED: Table 1: Results of the English→German systems in a simulated under-resourced scenario.
A standard NMT system employs parallel data only. While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial. To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data. gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion. Both methods requires to train separate systems with possibly different hyperparameters for each. Conversely, by applying mix-source method to the big monolingual data, we need to train only one network. We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data.
1611.04798
Introduction </s> Ne ural ĠMachine ĠTranslation Ġ( N MT ) Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġtranslation Ġtasks Ġwhen ĠN MT Ġsystems Ġperform Ġbest Ġin Ġrecent Ġmachine Ġtranslation Ġcampaigns ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠCompared Ġto Ġphrase - based ĠStatistical ĠMachine ĠTranslation Ġ( SM T ) Ġwhich Ġis Ġbasically Ġan Ġensemble Ġof Ġdifferent Ġfeatures Ġtrained Ġand Ġtuned Ġseparately , ĠN MT Ġdirectly Ġmodeling Ġthe Ġtranslation Ġrelationship Ġbetween Ġsource Ġand Ġtarget Ġsentences . ĠUnlike ĠSM T , ĠN MT Ġdoes Ġnot Ġrequire Ġmuch Ġlinguistic Ġinformation Ġand Ġlarge Ġmon oling ual Ġdata Ġto Ġachieve Ġgood Ġperformances . </s> An ĠN MT Ġconsists Ġof Ġan Ġenc oder Ġwhich Ġrec urs ively Ġreads Ġand Ġrepresents Ġthe Ġwhole Ġsource Ġsentence Ġinto Ġa Ġcontext Ġvector Ġand Ġa Ġrecurrent Ġdec oder Ġwhich Ġtakes Ġthe Ġcontext Ġvector Ġand Ġits Ġprevious Ġstate Ġto Ġpredict Ġthe Ġnext Ġtarget Ġword . ĠIt Ġis Ġthen Ġtrained Ġin Ġan Ġend - to - end Ġfashion Ġto Ġlearn Ġparameters Ġwhich Ġmaxim izes Ġthe Ġlikelihood Ġbetween Ġthe Ġoutputs Ġand Ġthe Ġreferences . ĠRecently , Ġattention - based ĠN MT Ġhas Ġbeen Ġfeatured Ġin Ġmost Ġstate - of - the - art Ġsystems . ĠFirst Ġintroduced Ġby ĠB IB REF 2 Ġ, Ġattention Ġmechanism Ġis Ġintegrated Ġin Ġdec oder Ġside Ġas Ġfeed forward Ġlayers . ĠIt Ġallows Ġthe ĠN MT Ġto Ġdecide Ġwhich Ġsource Ġwords Ġshould Ġtake Ġpart Ġin Ġthe Ġpredicting Ġprocess Ġof Ġthe Ġnext Ġtarget Ġwords . ĠIt Ġhelps Ġto Ġimprove ĠN MT s Ġsignificantly . ĠNevertheless , Ġsince Ġthe Ġattention Ġmechanism Ġis Ġspecific Ġto Ġa Ġparticular Ġsource Ġsentence Ġand Ġthe Ġconsidering Ġtarget Ġword , Ġit Ġis Ġalso Ġspecific Ġto Ġparticular Ġlanguage Ġpairs . </s> Some Ġrecent Ġwork Ġhas Ġfocused Ġon Ġextending Ġthe ĠN MT Ġframework Ġto Ġmult ilingual Ġscenarios . ĠBy Ġtraining Ġsuch Ġnetwork Ġusing Ġparallel Ġcorpor a Ġin Ġnumber Ġof Ġdifferent Ġlanguages , ĠN MT Ġcould Ġbenefit Ġfrom Ġadditional Ġinformation Ġembedded Ġin Ġa Ġcommon Ġsemantic Ġspace Ġacross Ġlanguages . ĠBasically , Ġthe Ġproposed ĠN MT Ġare Ġrequired Ġto Ġemploy Ġmultiple Ġenc od ers Ġor Ġmultiple Ġdec od ers Ġto Ġdeal Ġwith Ġmult iling uality . ĠFurthermore , Ġin Ġorder Ġto Ġavoid Ġthe Ġtight Ġdependency Ġof Ġthe Ġattention Ġmechanism Ġto Ġspecific Ġlanguage Ġpairs , Ġthey Ġalso Ġneed Ġto Ġmodify Ġtheir Ġarchitecture Ġto Ġcombine Ġeither Ġthe Ġenc od ers Ġor Ġthe Ġattention Ġlayers . ĠThese Ġmodifications Ġare Ġspecific Ġto Ġthe Ġpurpose Ġof Ġthe Ġtasks Ġas Ġwell . ĠThus , Ġthose Ġmult ilingual ĠN MT s Ġare Ġmore Ġcomplicated , Ġmuch Ġmore Ġfree Ġparameters Ġto Ġlearn Ġand Ġmore Ġdifficult Ġto Ġperform Ġstandard Ġtrain ings Ġcompared Ġto Ġthe Ġoriginal ĠN MT . </s> In Ġthis Ġpaper , Ġwe Ġintroduce Ġa Ġunified Ġapproach Ġto Ġseamlessly Ġextend Ġthe Ġoriginal ĠN MT Ġto Ġmult ilingual Ġsettings . ĠOur Ġapproach Ġallows Ġus Ġto Ġintegrate Ġany Ġlanguage Ġin Ġany Ġside Ġof Ġthe Ġenc oder - dec oder Ġarchitecture Ġwith Ġonly Ġone Ġenc oder Ġand Ġone Ġdec oder Ġfor Ġall Ġthe Ġlanguages Ġinvolved . ĠMoreover , Ġit Ġis Ġnot Ġnecessary Ġto Ġdo Ġany Ġnetwork Ġmodification Ġto Ġenable Ġattention Ġmechanism Ġin Ġour ĠN MT Ġsystems . ĠWe Ġthen Ġapply Ġour Ġpro pro sed Ġframework Ġin Ġtwo Ġdemanding Ġscenarios : Ġunder - res ourced Ġtranslation Ġand Ġzero - res ourced Ġtranslation . ĠThe Ġresults Ġshow Ġthat Ġbringing Ġmult iling uality Ġto ĠN MT Ġhelps Ġto Ġimprove Ġindividual Ġtranslations . ĠWith Ġsome Ġinsightful Ġanalyses Ġof Ġthe Ġresults , Ġwe Ġset Ġour Ġgoal Ġtoward Ġa Ġfully Ġmult ilingual ĠN MT Ġframework . </s> The Ġpaper Ġstarts Ġwith Ġa Ġdetailed Ġintroduction Ġto Ġattention - based ĠN MT . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġrelated Ġwork Ġabout Ġmulti - task ĠN MT Ġis Ġreviewed . ĠSection ĠSEC REF 5 Ġdescribes Ġour Ġproposed Ġapproach Ġand Ġthorough Ġcomparisons Ġto Ġthe Ġrelated Ġwork . ĠIt Ġis Ġfollowed Ġby Ġa Ġsection Ġof Ġevaluating Ġour Ġsystems Ġin Ġtwo Ġaforementioned Ġscenarios , Ġin Ġwhich Ġdifferent Ġstrategies Ġhave Ġbeen Ġemployed Ġunder Ġa Ġunified Ġapproach Ġ( Section ĠSEC REF 4 Ġ). ĠFinally , Ġthe Ġpaper Ġends Ġwith Ġconclusion Ġand Ġfuture Ġwork . </s> This Ġwork Ġis Ġlic enced Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Ne ural ĠMachine ĠTranslation : ĠBackground </s> An ĠN MT Ġsystem Ġconsists Ġof Ġan Ġenc oder Ġwhich Ġautomatically Ġlearns Ġthe Ġcharacteristics Ġof Ġa Ġsource Ġsentence Ġinto Ġfix - length Ġcontext Ġvectors Ġand Ġa Ġdec oder Ġthat Ġrec urs ively Ġcombines Ġthe Ġproduced Ġcontext Ġvectors Ġwith Ġthe Ġprevious Ġtarget Ġword Ġto Ġgenerate Ġthe Ġmost Ġprobable Ġword Ġfrom Ġa Ġtarget Ġvocabulary . </s> More Ġspecifically , Ġa Ġbid irection al Ġrecurrent Ġenc oder Ġreads Ġevery Ġwords ĠIN LINE FORM 0 Ġof Ġa Ġsource Ġsentence ĠIN LINE FORM 1 Ġand Ġenc odes Ġa Ġrepresentation ĠIN LINE FORM 2 Ġof Ġthe Ġsentence Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġconc at inated Ġfrom Ġthose Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : ĠIN LINE FORM 4 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġone - hot Ġvector Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġword Ġembed ding Ġmatrix Ġwhich Ġis Ġshared Ġacross Ġthe Ġsource Ġwords . ĠIN LINE FORM 3 Ġis Ġthe Ġrecurrent Ġunit Ġcomputing Ġthe Ġcurrent Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġbased Ġon Ġthe Ġprevious Ġhidden Ġstate . ĠIN LINE FORM 4 Ġis Ġthen Ġcalled Ġan Ġannotation Ġvector , Ġwhich Ġenc odes Ġthe Ġsource Ġsentence Ġup Ġto Ġthe Ġtime ĠIN LINE FORM 5 Ġfrom Ġboth Ġforward Ġand Ġbackward Ġdirections . ĠRec urrent Ġunits Ġin ĠN MT Ġcan Ġbe Ġa Ġsimple Ġrecurrent Ġneural Ġnetwork Ġunit Ġ( R NN ), Ġa ĠLong ĠShort - Term ĠMemory Ġunit Ġ( L ST M ) ĠB IB REF 3 Ġor Ġa ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 4 </s> Similar Ġto Ġthe Ġenc oder , Ġthe Ġrecurrent Ġdec oder Ġgenerates Ġone Ġtarget Ġword ĠIN LINE FORM 0 Ġto Ġform Ġa Ġtranslated Ġtarget Ġsentence ĠIN LINE FORM 1 Ġin Ġthe Ġend . ĠAt Ġthe Ġtime ĠIN LINE FORM 2 Ġ, Ġit Ġtakes Ġthe Ġprevious Ġhidden Ġstate Ġof Ġthe Ġdec oder ĠIN LINE FORM 3 Ġ, Ġthe Ġprevious Ġembedded Ġword Ġrepresentation ĠIN LINE FORM 4 Ġand Ġa Ġtime - specific Ġcontext Ġvector ĠIN LINE FORM 5 Ġas Ġinputs Ġto Ġcalculate Ġthe Ġcurrent Ġhidden Ġstate ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 </s> Again , ĠIN LINE FORM 0 Ġis Ġthe Ġrecurrent Ġactivation Ġfunction Ġof Ġthe Ġdec oder Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġshared Ġword Ġembed ding Ġmatrix Ġof Ġthe Ġtarget Ġsentences . ĠThe Ġcontext Ġvector ĠIN LINE FORM 2 Ġis Ġcalculated Ġbased Ġon Ġthe Ġannotation Ġvectors Ġfrom Ġthe Ġenc oder . ĠBefore Ġfeeding Ġthe Ġannotation Ġvectors Ġinto Ġthe Ġdec oder , Ġan Ġattention Ġmechanism Ġis Ġset Ġup Ġin Ġbetween , Ġin Ġorder Ġto Ġchoose Ġwhich Ġannotation Ġvectors Ġshould Ġcontribute Ġto Ġthe Ġpredicting Ġdecision Ġof Ġthe Ġnext Ġtarget Ġword . ĠInt uitive ly , Ġa Ġrelevance Ġbetween Ġthe Ġprevious Ġtarget Ġword Ġand Ġthe Ġannotation Ġvectors Ġcan Ġbe Ġused Ġto Ġform Ġsome Ġattention Ġscenario . ĠThere Ġexists Ġseveral Ġways Ġto Ġcalculate Ġthe Ġrelevance Ġas Ġshown Ġin ĠB IB REF 5 Ġ, Ġbut Ġwhat Ġwe Ġdescribe Ġhere Ġfollows Ġthe Ġproposed Ġmethod Ġof ĠB IB REF 2 ĠDIS PLAY FORM 0 </s> In ĠB IB REF 2 Ġ, Ġthis Ġattention Ġmechanism , Ġoriginally Ġcalled Ġalignment Ġmodel , Ġhas Ġbeen Ġemployed Ġas Ġa Ġsimple Ġfeed forward Ġnetwork Ġwith Ġthe Ġfirst Ġlayer Ġis Ġa Ġlearn able Ġlayer Ġvia ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġrelevance Ġscores ĠIN LINE FORM 3 Ġare Ġthen Ġnormalized Ġinto Ġattention Ġweights ĠIN LINE FORM 4 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 5 Ġis Ġcalculated Ġas Ġthe Ġweighted Ġsum Ġof Ġall Ġannotation Ġvectors ĠIN LINE FORM 6 Ġ. ĠDepending Ġon Ġhow Ġmuch Ġattention Ġthe Ġtarget Ġword Ġat Ġtime ĠIN LINE FORM 7 Ġput Ġon Ġthe Ġsource Ġstates ĠIN LINE FORM 8 Ġ, Ġa Ġsoft Ġalignment Ġis Ġlearned . ĠBy Ġbeing Ġemployed Ġthis Ġway , Ġword Ġalignment Ġis Ġnot Ġa Ġlatent Ġvariable Ġbut Ġa Ġparam et ri zed Ġfunction , Ġmaking Ġthe Ġalignment Ġmodel Ġdifferent iable . ĠThus , Ġit Ġcould Ġbe Ġtrained Ġtogether Ġwith Ġthe Ġwhole Ġarchitecture Ġusing Ġback prop ag ation . </s> One Ġof Ġthe Ġmost Ġsevere Ġproblems Ġof ĠN MT Ġis Ġhandling Ġof Ġthe Ġrare Ġwords , Ġwhich Ġare Ġnot Ġin Ġthe Ġshort Ġlists Ġof Ġthe Ġvoc ab ular ies , Ġi . e . Ġout - of - voc abulary Ġ( OO V ) Ġwords , Ġor Ġdo Ġnot Ġappear Ġin Ġthe Ġtraining Ġset Ġat Ġall . ĠIn ĠB IB REF 6 Ġ, Ġthe Ġrare Ġtarget Ġwords Ġare Ġcopied Ġfrom Ġtheir Ġaligned Ġsource Ġwords Ġafter Ġthe Ġtranslation . ĠThis Ġhe uristic Ġworks Ġwell Ġwith ĠO OV Ġwords Ġand Ġnamed Ġentities Ġbut Ġunable Ġto Ġtranslate Ġunseen Ġwords . ĠIn ĠB IB REF 7 Ġ, Ġtheir Ġproposed ĠN MT Ġmodels Ġhave Ġbeen Ġshown Ġto Ġnot Ġonly Ġbe Ġeffective Ġon Ġreducing Ġvocabulary Ġsizes Ġbut Ġalso Ġhave Ġthe Ġability Ġto Ġgenerate Ġunseen Ġwords . ĠThis Ġis Ġachieved Ġby Ġsegment ing Ġthe Ġrare Ġwords Ġinto Ġsub word Ġunits Ġand Ġtranslating Ġthem . ĠThe Ġstate - of - the - art Ġtranslation Ġsystems Ġessentially Ġemploy Ġsub word ĠN MT ĠB IB REF 7 Ġ. </s> Universal ĠEnc oder Ġand ĠDecoder Ġfor ĠMult ilingual ĠNeural ĠMachine ĠTranslation </s> While Ġthe Ġmajority Ġof Ġprevious Ġresearch Ġhas Ġfocused Ġon Ġimproving Ġthe Ġperformance Ġof ĠN MT Ġon Ġindividual Ġlanguage Ġpairs Ġwith Ġindividual ĠN MT Ġsystems , Ġrecent Ġwork Ġhas Ġstarted Ġinvestigating Ġpotential Ġways Ġto Ġconduct Ġthe Ġtranslation Ġinvolved Ġin Ġmultiple Ġlanguages Ġusing Ġa Ġsingle ĠN MT Ġsystem . ĠThe Ġpossible Ġreason Ġexplaining Ġthese Ġefforts Ġlies Ġon Ġthe Ġunique Ġarchitecture Ġof ĠN MT . ĠUnlike ĠSM T , ĠN MT Ġconsists Ġof Ġseparated Ġneural Ġnetworks Ġfor Ġthe Ġsource Ġand Ġtarget Ġsides , Ġor Ġthe Ġenc oder Ġand Ġdec oder , Ġrespectively . ĠThis Ġallows Ġthese Ġcomponents Ġto Ġmap Ġa Ġsentence Ġin Ġany Ġlanguage Ġto Ġa Ġrepresentation Ġin Ġan Ġembed ding Ġspace Ġwhich Ġis Ġbelieved Ġto Ġshare Ġcommon Ġsemantics Ġamong Ġthe Ġsource Ġlanguages Ġinvolved . ĠFrom Ġthat Ġshared Ġspace , Ġthe Ġdec oder , Ġwith Ġsome Ġimplicit Ġor Ġexplicit Ġrelevant Ġconstraints , Ġcould Ġtransform Ġthe Ġrepresentation Ġinto Ġa Ġconcrete Ġsentence Ġin Ġany Ġdesired Ġlanguage . ĠIn Ġthis Ġsection , Ġwe Ġreview Ġsome Ġrelated Ġwork Ġon Ġthis Ġmatter . ĠWe Ġthen Ġdescribe Ġa Ġunified Ġapproach Ġtoward Ġan Ġuniversal Ġattention - based ĠN MT Ġscheme . ĠOur Ġapproach Ġdoes Ġnot Ġrequire Ġany Ġarchitecture Ġmodification Ġand Ġit Ġcan Ġbe Ġtrained Ġto Ġlearn Ġa Ġminimal Ġnumber Ġof Ġparameters Ġcompared Ġto Ġthe Ġother Ġwork . </s> Related ĠWork </s> By Ġextending Ġthe Ġsolution Ġof Ġsequence - to - sequence Ġmodeling Ġusing Ġenc oder - dec oder Ġarchitectures Ġto Ġmulti - task Ġlearning , ĠLu ong 2016 Ġmanaged Ġto Ġachieve Ġbetter Ġperformance Ġon Ġsome ĠIN LINE FORM 0 Ġtasks Ġsuch Ġas Ġtranslation , Ġparsing Ġand Ġimage Ġcaption ing Ġcompared Ġto Ġindividual Ġtasks . ĠSpecifically Ġin Ġtranslation , Ġthe Ġwork Ġutilizes Ġmultiple Ġenc od ers Ġto Ġtranslate Ġfrom Ġmultiple Ġlanguages , Ġand Ġmultiple Ġdec od ers Ġto Ġtranslate Ġto Ġmultiple Ġlanguages . ĠIn Ġthis Ġview Ġof Ġmult ilingual Ġtranslation , Ġeach Ġlanguage Ġin Ġsource Ġor Ġtarget Ġside Ġis Ġmodeled Ġby Ġone Ġenc oder Ġor Ġdec oder , Ġdepending Ġon Ġthe Ġside Ġof Ġthe Ġtranslation . ĠDue Ġto Ġthe Ġnatural Ġdiversity Ġbetween Ġtwo Ġtasks Ġin Ġthat Ġmulti - task Ġlearning Ġscenario , Ġe . g . Ġtranslation Ġand Ġparsing , Ġit Ġcould Ġnot Ġfeature Ġthe Ġattention Ġmechanism Ġalthough Ġit Ġhas Ġproven Ġits Ġeffectiveness Ġin ĠN MT . ĠThere Ġexists Ġtwo Ġdirections Ġwhich Ġproposed Ġfor Ġmult ilingual Ġtranslation Ġscenarios Ġwhere Ġthey Ġleverage Ġthe Ġattention Ġmechanism . ĠThe Ġfirst Ġone Ġis Ġindicated Ġin Ġthe Ġwork Ġfrom ĠB IB REF 8 Ġ, Ġwhere Ġit Ġintroduce Ġan Ġone - to - many Ġmult ilingual ĠN MT Ġsystem Ġto Ġtranslates Ġfrom Ġone Ġsource Ġlanguage Ġinto Ġmultiple Ġtarget Ġlanguages . ĠHaving Ġone Ġsource Ġlanguage , Ġthe Ġattention Ġmechanism Ġis Ġthen Ġhanded Ġover Ġto Ġthe Ġcorresponding Ġdec oder . ĠThe Ġobjective Ġfunction Ġis Ġchanged Ġto Ġadapt Ġto Ġmult ilingual Ġsettings . ĠIn Ġtesting Ġtime , Ġthe Ġparameters Ġspecific Ġto Ġa Ġdesired Ġlanguage Ġpair Ġare Ġused Ġto Ġperform Ġthe Ġtranslation . </s> F ir at 2016 Ġproposed Ġanother Ġapproach Ġwhich Ġgenuinely Ġdelivers Ġattention - based ĠN MT Ġto Ġmult ilingual Ġtranslation . ĠAs Ġin ĠB IB REF 9 Ġ, Ġtheir Ġapproach Ġutilizes Ġone Ġenc oder Ġper Ġsource Ġlanguage Ġand Ġone Ġdec oder Ġper Ġtarget Ġlanguage Ġfor Ġmany - to - many Ġtranslation Ġtasks . ĠInstead Ġof Ġa Ġquad r atic Ġnumber Ġof Ġindependent Ġattention Ġlayers , Ġhowever , Ġone Ġsingle Ġattention Ġmechanism Ġis Ġintegrated Ġinto Ġtheir ĠN MT , Ġperforming Ġan Ġaff ine Ġtransformation Ġbetween Ġthe Ġhidden Ġlayer Ġof ĠIN LINE FORM 0 Ġsource Ġlanguages Ġand Ġthat Ġone Ġof ĠIN LINE FORM 1 Ġtarget Ġlanguages . ĠIt Ġis Ġrequired Ġto Ġchange Ġtheir Ġarchitecture Ġto Ġacc om od ate Ġsuch Ġa Ġcomplicated Ġshared Ġattention Ġmechanism . </s> In Ġa Ġseparate Ġeffort Ġto Ġachieve Ġmult ilingual ĠN MT , Ġthe Ġwork Ġof ĠZ oph 2016 Ġlever ages Ġavailable Ġparallel Ġdata Ġfrom Ġother Ġlanguage Ġpairs Ġto Ġhelp Ġreducing Ġpossible Ġambig u ities Ġin Ġthe Ġtranslation Ġprocess Ġinto Ġa Ġsingle Ġtarget Ġlanguage . ĠThey Ġemployed Ġthe Ġmulti - source Ġattention - based ĠN MT Ġin Ġa Ġway Ġthat Ġonly Ġone Ġattention Ġmechanism Ġis Ġrequired Ġdespite Ġhaving Ġmultiple Ġenc od ers . ĠTo Ġachieve Ġthis , Ġthe Ġoutputs Ġof Ġthe Ġenc od ers Ġwere Ġcombined Ġbefore Ġfeeding Ġto Ġthe Ġattention Ġlayer . ĠThey Ġimplemented Ġtwo Ġtypes Ġof Ġenc oder Ġcombination ; ĠOne Ġis Ġadding Ġa Ġnon - linear Ġlayer Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġenc od ers ' Ġhidden Ġstates . ĠThe Ġother Ġis Ġusing Ġa Ġvariant Ġof ĠL ST M Ġtaking Ġthe Ġrespective Ġgate Ġvalues Ġfrom Ġthe Ġindividual ĠL ST M Ġunits Ġof Ġthe Ġenc od ers . ĠAs Ġa Ġresult , Ġthe Ġcombined Ġhidden Ġstates Ġcontain Ġinformation Ġfrom Ġboth Ġenc od ers Ġ, Ġthus Ġencode Ġthe Ġcommon Ġsemantic Ġof Ġthe Ġtwo Ġsource Ġlanguages . </s> Universal ĠEnc oder Ġand ĠDecoder </s> Insp ired Ġby Ġthe Ġmulti - source ĠN MT Ġas Ġadditional Ġparallel Ġdata Ġin Ġseveral Ġlanguages Ġare Ġexpected Ġto Ġbenefit Ġsingle Ġtranslations , Ġwe Ġaim Ġto Ġdevelop Ġa ĠN MT - based Ġapproach Ġtoward Ġan Ġuniversal Ġframework Ġto Ġperform Ġmult ilingual Ġtranslation . ĠOur Ġsolution Ġfeatures Ġtwo Ġtreatments : Ġ1 ) ĠC oding Ġthe Ġwords Ġin Ġdifferent Ġlanguages Ġas Ġdifferent Ġwords Ġin Ġthe Ġlanguage - m ixed Ġvoc ab ular ies Ġand Ġ2 ) ĠFor cing Ġthe ĠN MT Ġto Ġtranslating Ġa Ġrepresentation Ġof Ġsource Ġsentences Ġinto Ġthe Ġsentences Ġin Ġa Ġdesired Ġtarget Ġlanguage . </s> Language - specific ĠC oding . ĠWhen Ġthe Ġenc oder Ġof Ġa ĠN MT Ġsystem Ġconsiders Ġwords Ġacross Ġlanguages Ġas Ġdifferent Ġwords , Ġwith Ġa Ġwell - ch osen Ġarchitecture , Ġit Ġis Ġexpected Ġto Ġbe Ġable Ġto Ġlearn Ġa Ġgood Ġrepresentation Ġof Ġthe Ġsource Ġwords Ġin Ġan Ġembed ding Ġspace Ġin Ġwhich Ġwords Ġcarrying Ġsimilar Ġmeaning Ġwould Ġhave Ġa Ġcloser Ġdistance Ġto Ġeach Ġothers Ġthan Ġthose Ġare Ġsem antically Ġdifferent . ĠThis Ġshould Ġhold Ġtrue Ġwhen Ġthe Ġwords Ġhave Ġthe Ġsame Ġor Ġsimilar Ġsurface Ġform , Ġsuch Ġas Ġ(@ de @ Obama ; Ġ@ en @ Obama ) Ġor Ġ(@ de @ Pro j ek tion ; Ġ@ en @ project ion ). ĠThis Ġshould Ġalso Ġhold Ġtrue Ġwhen Ġthe Ġwords Ġhave Ġthe Ġsame Ġor Ġsimilar Ġmeaning Ġacross Ġlanguages , Ġsuch Ġas Ġ(@ en @ car ; Ġ@ en @ aut om obile ) Ġor Ġ(@ de @ Fl uss u fer ; Ġ@ en @ bank ). ĠOur Ġenc oder Ġthen Ġacts Ġsimilarly Ġto Ġthe Ġone Ġof Ġmulti - source Ġapproach ĠB IB REF 10 Ġ, Ġcollecting Ġadditional Ġinformation Ġfrom Ġother Ġsources Ġfor Ġbetter Ġtranslations , Ġbut Ġwith Ġa Ġmuch Ġsimpler Ġembed ding Ġfunction . ĠUnlike Ġthem , Ġwe Ġneed Ġonly Ġone Ġenc oder , Ġso Ġwe Ġcould Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġto Ġlearn . ĠFurthermore , Ġwe Ġneither Ġneed Ġto Ġchange Ġthe Ġnetwork Ġarchitecture Ġnor Ġdepend Ġon Ġwhich Ġrecurrent Ġunit Ġ( GR U , ĠL ST M Ġor Ġsimple ĠR NN ) Ġis Ġcurrently Ġusing Ġin Ġthe Ġenc oder . </s> We Ġcould Ġapply Ġthe Ġsame Ġtrick Ġto Ġthe Ġtarget Ġsentences Ġand Ġthus Ġenable Ġmany - to - many Ġtranslation Ġcapability Ġof Ġour ĠN MT Ġsystem . ĠSimilar Ġto Ġthe Ġmulti - target Ġtranslation ĠB IB REF 8 Ġ, Ġwe Ġexploit Ġfurther Ġthe Ġcorrelation Ġin Ġsemantics Ġof Ġthose Ġtarget Ġsentences Ġacross Ġdifferent Ġlanguages . ĠThe Ġmain Ġdifference Ġbetween Ġour Ġapproach Ġand Ġthe Ġwork Ġof ĠB IB REF 8 Ġis Ġthat Ġwe Ġneed Ġonly Ġone Ġdec oder Ġfor Ġall Ġtarget Ġlanguages . ĠGiven Ġone Ġenc oder Ġfor Ġmultiple Ġsource Ġlanguages Ġand Ġone Ġdec oder Ġfor Ġmultiple Ġtarget Ġlanguages , Ġit Ġis Ġtrivial Ġto Ġincorporate Ġthe Ġattention Ġmechanism Ġas Ġin Ġthe Ġcase Ġof Ġa Ġregular ĠN MT Ġfor Ġsingle Ġlanguage Ġtranslation . ĠIn Ġtraining , Ġthe Ġattention Ġlayers Ġwere Ġdirected Ġto Ġlearn Ġrelevant Ġalign ments Ġbetween Ġwords Ġin Ġspecific Ġlanguage Ġpair Ġand Ġforward Ġthe Ġproduced Ġcontext Ġvector Ġto Ġthe Ġdec oder . ĠNow Ġwe Ġrely Ġtotally Ġon Ġthe Ġnetwork Ġto Ġlearn Ġgood Ġalign ments Ġbetween Ġsource Ġand Ġtarget Ġsides . ĠIn Ġfact , Ġgiving Ġmore Ġinformation , Ġour Ġsystem Ġare Ġable Ġto Ġform Ġnice Ġalign ments . </s> In Ġcomparison Ġto Ġother Ġresearch Ġthat Ġcould Ġperform Ġcomplete Ġmulti - task Ġlearning , Ġe . g . Ġthe Ġwork Ġfrom ĠB IB REF 9 Ġor Ġthe Ġapproach Ġproposed Ġby ĠB IB REF 11 Ġ, Ġour Ġmethod Ġis Ġable Ġto Ġaccommodate Ġthe Ġattention Ġlayers Ġseem lessly Ġand Ġeasily . ĠIt Ġalso Ġdraws Ġa Ġclear Ġdistinction Ġfrom Ġthose Ġworks Ġin Ġterm Ġof Ġthe Ġcomplexity Ġof Ġthe Ġwhole Ġnetwork : Ġconsiderably Ġless Ġparameters Ġto Ġlearn , Ġthus Ġreduces Ġover fitting , Ġwith Ġa Ġconventional Ġattention Ġmechanism Ġand Ġa Ġstandard Ġtraining Ġprocedure . </s> Target ĠFor cing . ĠWhile Ġlanguage - specific Ġcoding Ġallows Ġus Ġto Ġimplement Ġa Ġmult ilingual Ġattention - based ĠN MT , Ġthere Ġare Ġtwo Ġissues Ġwe Ġhave Ġto Ġconsider Ġbefore Ġtraining Ġthe Ġnetwork . ĠThe Ġfirst Ġis Ġthat Ġthe Ġnumber Ġof Ġrare Ġwords Ġwould Ġincrease Ġin Ġproportion Ġwith Ġthe Ġnumber Ġof Ġlanguages Ġinvolved . ĠThis Ġmight Ġbe Ġsolved Ġby Ġapplying Ġa Ġrare Ġword Ġtreatment Ġmethod Ġwith Ġappropriate Ġawareness Ġof Ġthe Ġvoc ab ular ies ' Ġsize . ĠThe Ġsecond Ġone Ġis Ġmore Ġproblematic : ĠAmb ig uity Ġlevel Ġin Ġthe Ġtranslation Ġprocess Ġdefinitely Ġincreases Ġdue Ġto Ġthe Ġadditional Ġintroduction Ġof Ġwords Ġhaving Ġthe Ġsame Ġor Ġsimilar Ġmeaning Ġacross Ġlanguages Ġat Ġboth Ġsource Ġand Ġtarget Ġsides . ĠWe Ġdeal Ġwith Ġthe Ġproblem Ġby Ġexplicitly Ġforcing Ġthe Ġattention Ġand Ġtranslation Ġto Ġthe Ġdirection Ġthat Ġwe Ġprefer , Ġexpecting Ġthe Ġinformation Ġwould Ġlimit Ġthe Ġambiguity Ġto Ġthe Ġscope Ġof Ġone Ġlanguage Ġinstead Ġof Ġall Ġtarget Ġlanguages . ĠWe Ġrealize Ġthis Ġidea Ġby Ġadding Ġat Ġthe Ġbeginning Ġand Ġat Ġthe Ġend Ġof Ġevery Ġsource Ġsentences Ġa Ġspecial Ġsymbol Ġindicating Ġthe Ġlanguage Ġthey Ġwould Ġbe Ġtranslated Ġinto . ĠFor Ġexample , Ġin Ġa Ġmult ilingual ĠN MT , Ġwhen Ġa Ġsource Ġsentence Ġis ĠGerman Ġand Ġthe Ġtarget Ġlanguage Ġis ĠEnglish , Ġthe Ġoriginal Ġsentence Ġ( al ready Ġlanguage - specific Ġcoded ) Ġis : </s> @ de @ dar um Ġ@ de @ ge ht Ġ@ de @ es Ġ@ de @ in Ġ@ de @ me inem Ġ@ de @ V ort rag </s> Now Ġwhen Ġwe Ġforce Ġit Ġto Ġbe Ġtranslated Ġinto ĠEnglish , Ġthe Ġtarget - forced Ġsentence Ġbecomes : </s> < E > Ġ@ de @ dar um Ġ@ de @ ge ht Ġ@ de @ es Ġ@ de @ in Ġ@ de @ me inem Ġ@ de @ V ort rag Ġ< E > </s> Due Ġto Ġthe Ġnature Ġof Ġrecurrent Ġunits Ġused Ġin Ġthe Ġenc oder Ġand Ġdec oder , Ġin Ġtraining , Ġthose Ġstarting Ġsymbols Ġencourage Ġthe Ġnetwork Ġlearning Ġthe Ġtranslation Ġof Ġfollowing Ġtarget Ġwords Ġin Ġa Ġparticular Ġlanguage Ġpair . ĠIn Ġtesting Ġtime , Ġinformation Ġof Ġthe Ġtarget Ġlanguage Ġwe Ġprovided Ġhelp Ġto Ġlimit Ġthe Ġtranslated Ġcandidates , Ġhence Ġforming Ġthe Ġtranslation Ġin Ġthe Ġdesired Ġlanguage . </s> Figure ĠFIG REF 8 Ġillustrates Ġthe Ġessence Ġof Ġour Ġapproach . ĠWith Ġtwo Ġsteps Ġin Ġthe Ġpre processing Ġphase , Ġnamely Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing , Ġwe Ġare Ġable Ġto Ġemploy Ġmult ilingual Ġattention - based ĠN MT Ġwithout Ġany Ġspecial Ġtreatment Ġin Ġtraining Ġsuch Ġa Ġstandard Ġarchitecture . ĠOur Ġenc oder Ġand Ġattention - enable Ġdec oder Ġcan Ġbe Ġseen Ġas Ġa Ġshared Ġenc oder Ġand Ġdec oder Ġacross Ġlanguages , Ġor Ġan Ġuniversal Ġenc oder Ġand Ġdec oder . ĠThe Ġflex ib it ily Ġof Ġour Ġapproach Ġallow Ġus Ġto Ġintegrate Ġany Ġlanguage Ġinto Ġsource Ġor Ġtarget Ġside . ĠAs Ġwe Ġwill Ġsee Ġin ĠSection ĠSEC REF 4 Ġ, Ġit Ġhas Ġproven Ġto Ġbe Ġextremely Ġhelpful Ġnot Ġonly Ġin Ġlow - res ourced Ġscenarios Ġbut Ġalso Ġin Ġtranslation Ġof Ġwell - res ourced Ġlanguage Ġpairs Ġas Ġit Ġprovides Ġa Ġnovel Ġway Ġto Ġmake Ġuse Ġof Ġlarge Ġmon oling ual Ġcorpor a Ġin ĠN MT . </s> E val uation </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġevaluation Ġof Ġour Ġproposed Ġapproach Ġin Ġcomparisons Ġwith Ġthe Ġstrong Ġbas elines Ġusing ĠN MT Ġin Ġtwo Ġscenarios : Ġthe Ġtranslation Ġof Ġan Ġunder - resource Ġlanguage Ġpair Ġand Ġthe Ġtranslation Ġof Ġa Ġlanguage Ġpair Ġthat Ġdoes Ġnot Ġexist Ġany Ġpar alled Ġdata Ġat Ġall . </s> Exper imental ĠSettings </s> Training ĠData . ĠWe Ġchoose ĠW IT 3 's ĠTED Ġcorpus ĠB IB REF 12 Ġas Ġthe Ġbasis Ġof Ġour Ġexperiments Ġsince Ġit Ġmight Ġbe Ġthe Ġonly Ġhigh - quality Ġparallel Ġdata Ġof Ġmany Ġlow - res ourced Ġlanguage Ġpairs . ĠTED Ġis Ġalso Ġmult ilingual Ġin Ġa Ġsense Ġthat Ġit Ġincludes Ġnumbers Ġof Ġtalks Ġwhich Ġare Ġcommonly Ġtranslated Ġinto Ġmany Ġlanguages . ĠIn Ġaddition , Ġwe Ġuse Ġa Ġmuch Ġlarger Ġcorpus Ġprovided Ġfreely Ġby ĠW MT Ġorganizers Ġwhen Ġwe Ġevaluate Ġthe Ġimpact Ġof Ġour Ġapproach Ġin Ġa Ġreal Ġmachine Ġtranslation Ġcampaign . ĠIt Ġincludes Ġthe Ġpar alled Ġcorpus Ġextracted Ġfrom Ġthe Ġdigital Ġcorpus Ġof ĠEuropean ĠParliament Ġ( EP PS ), Ġthe ĠNews ĠCommentary Ġ( NC ) Ġand Ġthe Ġweb - c rawled Ġparallel Ġdata Ġ( Common C rawl ). ĠWhile Ġthe Ġnumber Ġof Ġsentences Ġin Ġpopular ĠTED Ġcorpor a Ġvaries Ġfrom Ġ13 Ġthousands Ġto Ġ17 Ġthousands , Ġthe Ġtotal Ġnumber Ġof Ġsentences Ġin Ġthose Ġlarger Ġcorpus Ġis Ġapproximately Ġ3 Ġmillion Ġsentences . </s> Ne ural ĠMachine ĠTranslation ĠSetup . ĠAll Ġexperiments Ġhave Ġbeen Ġconducted Ġusing ĠN MT Ġframework ĠNem atus , ĠFollowing Ġthe Ġwork Ġof ĠS enn rich 2016 a , Ġsub word Ġsegment ation Ġis Ġhandled Ġin Ġthe Ġprep oc essing Ġphase Ġusing ĠByte - P air ĠEnc oding Ġ( B PE ). ĠExcept s Ġstated Ġclearly Ġin Ġsome Ġexperiments , Ġwe Ġset Ġthe Ġnumber Ġof ĠB PE Ġmerging Ġoperations Ġat Ġ395 00 Ġon Ġthe Ġjoint Ġof Ġsource Ġand Ġtarget Ġdata . ĠWhen Ġtraining Ġall ĠN MT Ġsystems , Ġwe Ġtake Ġout Ġthe Ġsentence Ġpairs Ġexceeding Ġ50 - word Ġlength Ġand Ġshuffle Ġthem Ġinside Ġevery Ġmin ib atch . ĠOur Ġshort - list Ġvoc ab ular ies Ġcontain Ġ40 , 000 Ġmost Ġfrequent Ġwords Ġwhile Ġthe Ġothers Ġare Ġconsidered Ġas Ġrare Ġwords Ġand Ġapplied Ġthe Ġsub word Ġtranslation . ĠWe Ġuse Ġan Ġ1024 - cell ĠGR U Ġlayer Ġand Ġ1000 - dimensional Ġembed d ings Ġwith Ġdrop out Ġat Ġevery Ġlayer Ġwith Ġthe Ġprobability Ġof Ġ0 . 2 Ġin Ġthe Ġembed ding Ġand Ġhidden Ġlayers Ġand Ġ0 . 1 Ġin Ġthe Ġinput Ġand Ġour put Ġlayers . ĠWe Ġtrained Ġour Ġsystems Ġusing Ġgradient Ġdescent Ġoptimization Ġwith ĠAd ad elta ĠB IB REF 13 Ġon Ġmin ib atches Ġof Ġsize Ġ80 Ġand Ġthe Ġgradient Ġis Ġresc aled Ġwhenever Ġits Ġnorm Ġexceed Ġ1 . 0 . ĠAll Ġthe Ġtrain ings Ġlast Ġapproximately Ġseven Ġdays Ġif Ġthe Ġearly - st opping Ġcondition Ġcould Ġnot Ġbe Ġreached . ĠAt Ġa Ġcertain Ġtime , Ġan Ġexternal Ġevaluation Ġscript Ġon ĠB LE U ĠB IB REF 14 Ġis Ġconducted Ġon Ġa Ġdevelopment Ġset Ġto Ġdecide Ġthe Ġearly - st opping Ġcondition . ĠThis Ġevaluation Ġscript Ġhas Ġalso Ġbeing Ġused Ġto Ġchoose Ġthe Ġmodel Ġarch iving Ġthe Ġbest ĠB LE U Ġon Ġthe Ġdevelopment Ġset Ġinstead Ġof Ġthe Ġmaximal Ġlog like lihood Ġbetween Ġthe Ġtranslations Ġand Ġtarget Ġsentences Ġwhile Ġtraining . ĠIn Ġtranslation , Ġthe Ġframework Ġproduces ĠIN LINE FORM 0 Ġ- best Ġcandidates Ġand Ġwe Ġthen Ġuse Ġa Ġbeam Ġsearch Ġwith Ġthe Ġbeam Ġsize Ġof Ġ12 Ġto Ġget Ġthe Ġbest Ġtranslation . </s> Under - res ourced ĠTranslation </s> First , Ġwe Ġconsider Ġthe Ġtranslation Ġfor Ġan Ġunder - res ourced Ġpair Ġof Ġlanguages . ĠHere Ġa Ġsmall Ġportion Ġof Ġthe Ġlarge Ġparallel Ġcorpus Ġfor ĠEnglish - German Ġis Ġused Ġas Ġa Ġsimulation Ġfor Ġthe Ġscenario Ġwhere Ġwe Ġdo Ġnot Ġhave Ġmuch Ġparallel Ġdata : ĠTrans l ating Ġtexts Ġin ĠEnglish Ġto ĠGerman . ĠWe Ġperform Ġlanguage - specific Ġcoding Ġin Ġboth Ġsource Ġand Ġtarget Ġsides . ĠBy Ġaccommodating Ġthe ĠGerman Ġmon oling ual Ġdata Ġas Ġan Ġadditional Ġinput Ġ( German ĠIN LINE FORM 0 ĠGerman ), Ġwhich Ġwe Ġcalled Ġthe Ġmix - source Ġapproach , Ġwe Ġcould Ġenrich Ġthe Ġtraining Ġdata Ġin Ġa Ġsimple , Ġnatural Ġway . ĠGiven Ġthis Ġunder - res ourced Ġsituation , Ġit Ġcould Ġhelp Ġour ĠN MT Ġobtain Ġa Ġbetter Ġrepresentation Ġof Ġthe Ġsource Ġside , Ġhence , Ġable Ġto Ġlearn Ġthe Ġtranslation Ġrelationship Ġbetter . ĠIncluding Ġmon oling ual Ġdata Ġin Ġthis Ġway Ġmight Ġalso Ġimprove Ġthe Ġtranslation Ġof Ġsome Ġrare Ġword Ġtypes Ġsuch Ġas Ġnamed Ġentities . ĠFurthermore , Ġas Ġthe Ġultimate Ġgoal Ġof Ġour Ġwork , Ġwe Ġwould Ġlike Ġto Ġinvestigate Ġthe Ġadvantages Ġof Ġmult iling uality Ġin ĠN MT . ĠWe Ġincorporate Ġa Ġsimilar Ġportion Ġof ĠFrench - German Ġparallel Ġcorpus Ġinto Ġthe ĠEnglish - German Ġone . ĠAs Ġdiscussed Ġin ĠSection ĠSEC REF 5 Ġ, Ġit Ġis Ġexpected Ġto Ġhelp Ġreducing Ġthe Ġambiguity Ġin Ġtranslation Ġbetween Ġone Ġlanguage Ġpair Ġsince Ġit Ġutilizes Ġthe Ġsemantic Ġcontext Ġprovided Ġby Ġthe Ġother Ġsource Ġlanguage . ĠWe Ġname Ġthis Ġmix - multi - source . </s> Table ĠT AB REF 16 Ġsummarizes Ġthe Ġperformance Ġof Ġour Ġsystems Ġmeasured Ġin ĠB LE U Ġon Ġtwo Ġtest Ġsets , Ġt st 2013 Ġand Ġt st 2014 . ĠCompared Ġto Ġthe Ġbaseline ĠN MT Ġsystem Ġwhich Ġis Ġsolely Ġtrained Ġon ĠTED ĠEnglish - German Ġdata , Ġour Ġmix - source Ġsystem Ġachieves Ġa Ġconsiderable Ġimprovement Ġof Ġ2 . 6 ĠB LE U Ġpoints Ġon Ġt st 2013 Ġand Ġ2 . 1 ĠB LE U Ġpoints Ġon Ġand Ġt st 2014 Ġ. ĠAdding ĠFrench Ġdata Ġto Ġthe Ġsource Ġside Ġand Ġtheir Ġcorresponding ĠGerman Ġdata Ġto Ġthe Ġtarget Ġside Ġin Ġour Ġmix - multi - source Ġsystem Ġalso Ġhelp Ġto Ġgain Ġ2 . 2 Ġand Ġ1 . 6 ĠB LE U Ġpoints Ġmore Ġon Ġt st 2013 Ġt st 2014 , Ġrespectively . ĠWe Ġobserve Ġa Ġbetter Ġimprovement Ġfrom Ġour Ġmix - source Ġsystem Ġcompared Ġto Ġour Ġmix - multi - source Ġsystem . ĠWe Ġspeculate Ġthe Ġreason Ġthat Ġthe Ġmix - source Ġenc oder Ġutilize Ġthe Ġsame Ġinformation Ġshared Ġin Ġtwo Ġlanguages Ġwhile Ġthe Ġmix - multi - source Ġreceives Ġand Ġprocesses Ġsimilar Ġinformation Ġin Ġthe Ġother Ġlanguage Ġbut Ġnot Ġnecessarily Ġthe Ġsame . ĠWe Ġmight Ġvalidate Ġthis Ġhypothesis Ġby Ġcomparing Ġtwo Ġsystems Ġtrained Ġon Ġa Ġcommon ĠEnglish - German - French Ġcorpus Ġof ĠTED . ĠWe Ġput Ġit Ġin Ġour Ġfuture Ġwork 's Ġplan . </s> As Ġwe Ġexpected ĠFigure ĠFIG REF 19 Ġshows Ġhow Ġdifferent Ġwords Ġin Ġdifferent Ġlanguages Ġcan Ġbe Ġclose Ġin Ġthe Ġshared Ġspace Ġafter Ġbeing Ġlearned Ġto Ġtranslate Ġinto Ġa Ġcommon Ġlanguage . ĠWe Ġextract Ġthe Ġword Ġembed d ings Ġfrom Ġthe Ġenc oder Ġof Ġthe Ġmix - multi - source Ġ( En , Fr ĠIN LINE FORM 0 ĠDe , De ) Ġafter Ġtraining , Ġremove Ġthe Ġlanguage - specific Ġcodes Ġ(@ en @ Ġand Ġ@ fr @ ) and Ġproject Ġthe Ġword Ġvectors Ġto Ġthe Ġ2 D Ġspace Ġusing Ġt - S NE ĠB IB REF 15 Ġ. </s> Using Ġlarge Ġmon oling ual Ġdata Ġin ĠN MT . </s> A Ġstandard ĠN MT Ġsystem Ġemploys Ġparallel Ġdata Ġonly . ĠWhile Ġgood Ġparallel Ġcorpor a Ġare Ġlimited Ġin Ġnumber , Ġgetting Ġmon oling ual Ġdata Ġof Ġan Ġarbitrary Ġlanguage Ġis Ġtrivial . ĠTo Ġmake Ġuse Ġof ĠGerman Ġmon oling ual Ġcorpus Ġin Ġan ĠEnglish ĠIN LINE FORM 0 ĠGerman ĠN MT Ġsystem , Ġs enn rich 2016 b Ġbuilt Ġa Ġseparate ĠGerman ĠIN LINE FORM 1 ĠEnglish ĠN MT Ġusing Ġthe Ġsame Ġparallel Ġcorpus , Ġthen Ġthey Ġused Ġthat Ġsystem Ġto Ġtranslate Ġthe ĠGerman Ġmon oling ual Ġcorpus Ġback Ġto ĠEnglish , Ġforming Ġa Ġsynthesis Ġparallel Ġdata . Ġgul ce h re 2015 Ġtrained Ġanother ĠR NN - based Ġlanguage Ġmodel Ġto Ġscore Ġthe Ġmon oling ual Ġcorpus Ġand Ġintegrate Ġit Ġto Ġthe ĠN MT Ġsystem Ġthrough Ġshallow Ġor Ġdeep Ġfusion . ĠBoth Ġmethods Ġrequires Ġto Ġtrain Ġseparate Ġsystems Ġwith Ġpossibly Ġdifferent Ġhyper param eters Ġfor Ġeach . ĠConversely , Ġby Ġapplying Ġmix - source Ġmethod Ġto Ġthe Ġbig Ġmon oling ual Ġdata , Ġwe Ġneed Ġto Ġtrain Ġonly Ġone Ġnetwork . ĠWe Ġmix Ġthe ĠTED Ġparallel Ġcorpus Ġand Ġthe Ġsubstantial Ġmon oling ual Ġcorpus Ġ( EP PS + NC + Common C rawl ) Ġand Ġtrain Ġa Ġmix - source ĠN MT Ġsystem Ġfrom Ġthose Ġdata . </s> The Ġfirst Ġresult Ġis Ġnot Ġencouraging Ġwhen Ġits Ġperformance Ġis Ġeven Ġworse Ġthan Ġthe Ġbaseline ĠN MT Ġwhich Ġis Ġtrained Ġon Ġthe Ġsmall Ġparallel Ġdata Ġonly . ĠNot Ġusing Ġthe Ġsame Ġinformation Ġin Ġthe Ġsource Ġside , Ġas Ġwe Ġdiscussed Ġin Ġcase Ġof Ġmix - multi - source Ġstrategy , Ġcould Ġexplain Ġthe Ġdegrading Ġin Ġperformance Ġof Ġsuch Ġa Ġsystem . ĠBut Ġwe Ġbelieve Ġthat Ġthe Ġmagnitude Ġand Ġun bal ancing Ġof Ġthe Ġcorpus Ġare Ġthe Ġmain Ġreasons . ĠThe Ġdata Ġcontains Ġnearly Ġfour Ġmillions Ġsentences Ġbut Ġonly Ġaround Ġtwenty Ġthousands Ġof Ġthem Ġ( 0 . 5 %) Ġare Ġthe Ġgenuine Ġparallel Ġdata . ĠAs Ġa Ġquick Ġattempt , Ġafter Ġwe Ġget Ġthe Ġmodel Ġwith Ġthat Ġbig Ġdata , Ġwe Ġcontinue Ġtraining Ġon Ġthe Ġreal Ġparallel Ġcorpus Ġfor Ġsome Ġmore Ġepoch s . ĠWhen Ġthis Ġadaptation Ġis Ġapplied , Ġour Ġsystem Ġbrings Ġan Ġimprovement Ġof Ġ+ 1 . 52 ĠB LE U Ġon Ġt st 2013 Ġand Ġ+ 1 . 06 ĠB LE U Ġon Ġt st 2014 Ġ( Table ĠT AB REF 21 Ġ). </s> Zero - res ourced ĠTranslation </s> Among Ġlow - res ourced Ġscenarios , Ġzero - res ourced Ġtranslation Ġtask Ġstands Ġin Ġan Ġextreme Ġlevel . ĠA Ġzero - res ourced Ġtranslation Ġtask Ġis Ġone Ġof Ġthe Ġmost Ġdifficult Ġsituation Ġwhen Ġthere Ġis Ġno Ġparallel Ġdata Ġbetween Ġthe Ġtranslating Ġlanguage Ġpair . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthere Ġhave Ġbeen Ġyet Ġexisted Ġa Ġpublished Ġwork Ġabout Ġusing ĠN MT Ġfor Ġzero - res ourced Ġtranslation Ġtasks Ġup Ġto Ġnow . ĠIn Ġthis Ġsection , Ġwe Ġextend Ġour Ġstrategies Ġusing Ġthe Ġproposed Ġmult ilingual ĠN MT Ġapproach Ġas Ġfirst Ġattempts Ġto Ġthis Ġextreme Ġsituation . </s> We Ġemploy Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing Ġin Ġa Ġstrategy Ġcalled Ġbridge . ĠUnlike Ġthe Ġstrategies Ġused Ġin Ġunder - res ourced Ġtranslation Ġtask , Ġbridge Ġis Ġan Ġentire Ġmany - to - many Ġmult ilingual ĠN MT . ĠSim ulating Ġa Ġzero - res ourced ĠGerman ĠIN LINE FORM 0 ĠFrench Ġtranslation Ġtask Ġgiven Ġthe Ġavailable ĠGerman - English Ġand ĠEnglish - French Ġparallel Ġcorpor a , Ġafter Ġapplying Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing Ġfor Ġeach Ġcorpus , Ġwe Ġmix Ġthose Ġdata Ġwith Ġan ĠEnglish - English Ġdata Ġas Ġa ĠâĢ ľ bridge âĢ Ŀ Ġcreating Ġsome Ġconnection Ġbetween ĠGerman Ġand ĠFrench . ĠWe Ġalso Ġpropose Ġa Ġvariant Ġof Ġthis Ġstrategy Ġthat Ġwe Ġincorporate ĠFrench - French Ġdata . ĠAnd Ġwe Ġcall Ġit Ġuniversal . </s> We Ġevaluate Ġbridge Ġand Ġuniversal Ġsystems Ġon Ġtwo ĠGerman ĠIN LINE FORM 0 ĠFrench Ġtest Ġsets . ĠThey Ġare Ġcompared Ġto Ġa Ġdirect Ġsystem , Ġwhich Ġis Ġan ĠN MT Ġtrained Ġon ĠGerman ĠIN LINE FORM 1 ĠFrench Ġdata , Ġand Ġto Ġa Ġpivot Ġsystem , Ġwhich Ġessentially Ġconsists Ġof Ġtwo Ġseparate ĠN MT s Ġtrained Ġto Ġtranslate Ġfrom ĠGerman Ġto ĠEnglish Ġand ĠEnglish Ġto ĠFrench . ĠThe Ġdirect Ġsystem Ġshould Ġnot Ġexist Ġin Ġa Ġreal Ġzero - res ourced Ġsituation . ĠWe Ġrefer Ġit Ġas Ġthe Ġperfect Ġsystem Ġfor Ġcomparison Ġpurpose Ġonly . ĠIn Ġcase Ġof Ġthe Ġpivot Ġsystem , Ġto Ġgenerate Ġa Ġtranslated Ġtext Ġin ĠFrench Ġfrom Ġa ĠGerman Ġsentence , Ġwe Ġfirst Ġtranslate Ġit Ġto ĠEnglish , Ġthen Ġthe Ġoutput Ġsentence Ġis Ġfed Ġto Ġthe ĠEnglish ĠIN LINE FORM 2 ĠGerman ĠN MT Ġsystem Ġto Ġobtain Ġthe ĠFrench Ġtranslation . ĠSince Ġthere Ġare Ġmore Ġthan Ġtwo Ġlanguages Ġinvolved Ġin Ġthose Ġsystems , Ġwe Ġincrease Ġthe Ġnumber Ġof ĠB PE Ġmerging Ġoperations Ġproportion ally Ġin Ġorder Ġto Ġreduce Ġthe Ġnumber Ġof Ġrare Ġwords Ġin Ġsuch Ġsystems . ĠWe Ġdo Ġnot Ġexpect Ġour Ġproposed Ġsystems Ġto Ġperform Ġwell Ġwith Ġthis Ġprimitive Ġway Ġof Ġbuilding Ġdirect Ġtranslating Ġconnections Ġsince Ġthis Ġis Ġessentially Ġa Ġdifficult Ġtask . ĠWe Ġreport Ġthe Ġperformance Ġof Ġthose Ġsystems Ġin ĠTable ĠT AB REF 23 Ġ. </s> Un supp rising ly , Ġboth Ġbridge Ġand Ġuniversal Ġsystems Ġperform Ġworse Ġthan Ġthe Ġpivot Ġone . ĠWe Ġconsider Ġtwo Ġpossible Ġreasons : </s> Our Ġtarget Ġforcing Ġmechanism Ġis Ġmoderately Ġprimitive . ĠSince Ġthe Ġprocess Ġis Ġapplied Ġafter Ġlanguage - specific Ġcoding , Ġthe Ġtarget Ġforcing Ġsymbol Ġis Ġthe Ġsame Ġfor Ġall Ġsource Ġsentences Ġin Ġevery Ġlanguages . ĠThus , Ġthe Ġforcing Ġstrength Ġmight Ġnot Ġbe Ġenough Ġto Ġguide Ġthe Ġdecision Ġof Ġthe Ġnext Ġwords . ĠOnce Ġthe Ġvery Ġfirst Ġword Ġis Ġtranslated Ġinto Ġa Ġword Ġin Ġwrong Ġlanguage , Ġthe Ġfollowing Ġwords Ġtend Ġto Ġbe Ġtranslated Ġinto Ġthat Ġwrong Ġlanguage Ġagain . ĠTable ĠT AB REF 24 Ġshows Ġsome Ġstatistics Ġof Ġthe Ġtranslated Ġwords Ġand Ġsentences Ġin Ġwrong Ġlanguage . </s> Bal ancing Ġof Ġthe Ġtraining Ġcorpus . ĠAlthough Ġit Ġis Ġnot Ġsevere Ġas Ġin Ġthe Ġcase Ġof Ġmix - source Ġsystem Ġfor Ġlarge Ġmon oling ual Ġdata , Ġthe Ġlimited Ġnumber Ġof Ġsentences Ġin Ġtarget Ġlanguage Ġcan Ġaffect Ġthe Ġtraining . ĠThe Ġdifference Ġof Ġ1 . 07 ĠB LE U Ġpoints Ġbetween Ġbridge Ġand Ġuniversal Ġmight Ġexplain Ġthis Ġassumption Ġas Ġwe Ġadded Ġmore Ġtarget Ġdata Ġ( French ) Ġin Ġuniversal Ġstrategy , Ġthus Ġreducing Ġthe Ġun balance Ġin Ġtraining . </s> Those Ġissues Ġwould Ġbe Ġaddressed Ġin Ġour Ġfollowing Ġfuture Ġwork Ġtoward Ġthe Ġmult ilingual Ġattention - based ĠN MT . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġfirst Ġattempts Ġin Ġbuilding Ġa Ġmult ilingual ĠNeural ĠMachine ĠTranslation Ġframework . ĠBy Ġtreating Ġwords Ġin Ġdifferent Ġlanguages Ġas Ġdifferent Ġwords Ġand Ġforce Ġthe Ġattention Ġand Ġtranslation Ġto Ġthe Ġdirection Ġof Ġdesired Ġtarget Ġlanguage , Ġwe Ġare Ġable Ġto Ġemploy Ġattention - enable ĠN MT Ġtoward Ġa Ġmult ilingual Ġtranslation Ġsystem . ĠOur Ġproposed Ġapproach Ġallev iates Ġthe Ġneed Ġof Ġcomplicated Ġarchitecture Ġre - design ing Ġwhen Ġaccommodating Ġattention Ġmechanism . ĠIn Ġaddition , Ġthe Ġnumber Ġof Ġfree Ġparameters Ġto Ġlearn Ġin Ġour Ġnetwork Ġdoes Ġnot Ġgo Ġbeyond Ġthat Ġmagn itute Ġof Ġa Ġsingle ĠN MT Ġsystem . ĠWith Ġits Ġunivers ality , Ġour Ġapproach Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġan Ġunder - res ourced Ġtranslation Ġtask Ġwith Ġconsiderable Ġimprovements . ĠIn Ġaddition , Ġthe Ġapproach Ġhas Ġachieved Ġinteresting Ġand Ġpromising Ġresults Ġwhen Ġapplied Ġin Ġthe Ġtranslation Ġtask Ġthat Ġthere Ġis Ġno Ġdirect Ġparallel Ġcorpus Ġbetween Ġsource Ġand Ġtarget Ġlanguages . </s> Nevertheless , Ġthere Ġare Ġissues Ġthat Ġwe Ġcan Ġcontinue Ġworking Ġon Ġto Ġaddress Ġin Ġfuture Ġwork . ĠA Ġmore Ġbalancing Ġdata Ġwould Ġbe Ġhelpful Ġfor Ġthis Ġframework . ĠThe Ġmechanism Ġof Ġforcing Ġthe ĠN MT Ġsystem Ġto Ġthe Ġright Ġtarget Ġlanguage Ġcould Ġbe Ġimproved . ĠWe Ġcould Ġconduct Ġmore Ġdetailed Ġanalyses Ġof Ġthe Ġvarious Ġstrategies Ġunder Ġthe Ġframework Ġto Ġshow Ġits Ġunivers arity .
Introduction Neural Machine Translation (NMT) has shown its effectiveness in translation tasks when NMT systems perform best in recent machine translation campaigns BIBREF0 , BIBREF1 . Compared to phrase-based Statistical Machine Translation (SMT) which is basically an ensemble of different features trained and tuned separately, NMT directly modeling the translation relationship between source and target sentences. Unlike SMT, NMT does not require much linguistic information and large monolingual data to achieve good performances. An NMT consists of an encoder which recursively reads and represents the whole source sentence into a context vector and a recurrent decoder which takes the context vector and its previous state to predict the next target word. It is then trained in an end-to-end fashion to learn parameters which maximizes the likelihood between the outputs and the references. Recently, attention-based NMT has been featured in most state-of-the-art systems. First introduced by BIBREF2 , attention mechanism is integrated in decoder side as feedforward layers. It allows the NMT to decide which source words should take part in the predicting process of the next target words. It helps to improve NMTs significantly. Nevertheless, since the attention mechanism is specific to a particular source sentence and the considering target word, it is also specific to particular language pairs. Some recent work has focused on extending the NMT framework to multilingual scenarios. By training such network using parallel corpora in number of different languages, NMT could benefit from additional information embedded in a common semantic space across languages. Basically, the proposed NMT are required to employ multiple encoders or multiple decoders to deal with multilinguality. Furthermore, in order to avoid the tight dependency of the attention mechanism to specific language pairs, they also need to modify their architecture to combine either the encoders or the attention layers. These modifications are specific to the purpose of the tasks as well. Thus, those multilingual NMTs are more complicated, much more free parameters to learn and more difficult to perform standard trainings compared to the original NMT. In this paper, we introduce a unified approach to seamlessly extend the original NMT to multilingual settings. Our approach allows us to integrate any language in any side of the encoder-decoder architecture with only one encoder and one decoder for all the languages involved. Moreover, it is not necessary to do any network modification to enable attention mechanism in our NMT systems. We then apply our proprosed framework in two demanding scenarios: under-resourced translation and zero-resourced translation. The results show that bringing multilinguality to NMT helps to improve individual translations. With some insightful analyses of the results, we set our goal toward a fully multilingual NMT framework. The paper starts with a detailed introduction to attention-based NMT. In Section SECREF3 , related work about multi-task NMT is reviewed. Section SECREF5 describes our proposed approach and thorough comparisons to the related work. It is followed by a section of evaluating our systems in two aforementioned scenarios, in which different strategies have been employed under a unified approach (Section SECREF4 ). Finally, the paper ends with conclusion and future work. This work is licenced under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ Neural Machine Translation: Background An NMT system consists of an encoder which automatically learns the characteristics of a source sentence into fix-length context vectors and a decoder that recursively combines the produced context vectors with the previous target word to generate the most probable word from a target vocabulary. More specifically, a bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatinated from those of the forward and backward directions: INLINEFORM4 Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words. INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. INLINEFORM4 is then called an annotation vector, which encodes the source sentence up to the time INLINEFORM5 from both forward and backward directions. Recurrent units in NMT can be a simple recurrent neural network unit (RNN), a Long Short-Term Memory unit (LSTM) BIBREF3 or a Gated Recurrent Unit (GRU) BIBREF4 Similar to the encoder, the recurrent decoder generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end. At the time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and a time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7 Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences. The context vector INLINEFORM2 is calculated based on the annotation vectors from the encoder. Before feeding the annotation vectors into the decoder, an attention mechanism is set up in between, in order to choose which annotation vectors should contribute to the predicting decision of the next target word. Intuitively, a relevance between the previous target word and the annotation vectors can be used to form some attention scenario. There exists several ways to calculate the relevance as shown in BIBREF5 , but what we describe here follows the proposed method of BIBREF2 DISPLAYFORM0 In BIBREF2 , this attention mechanism, originally called alignment model, has been employed as a simple feedforward network with the first layer is a learnable layer via INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6 . Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned. By being employed this way, word alignment is not a latent variable but a parametrized function, making the alignment model differentiable. Thus, it could be trained together with the whole architecture using backpropagation. One of the most severe problems of NMT is handling of the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. In BIBREF6 , the rare target words are copied from their aligned source words after the translation. This heuristic works well with OOV words and named entities but unable to translate unseen words. In BIBREF7 , their proposed NMT models have been shown to not only be effective on reducing vocabulary sizes but also have the ability to generate unseen words. This is achieved by segmenting the rare words into subword units and translating them. The state-of-the-art translation systems essentially employ subword NMT BIBREF7 . Universal Encoder and Decoder for Multilingual Neural Machine Translation While the majority of previous research has focused on improving the performance of NMT on individual language pairs with individual NMT systems, recent work has started investigating potential ways to conduct the translation involved in multiple languages using a single NMT system. The possible reason explaining these efforts lies on the unique architecture of NMT. Unlike SMT, NMT consists of separated neural networks for the source and target sides, or the encoder and decoder, respectively. This allows these components to map a sentence in any language to a representation in an embedding space which is believed to share common semantics among the source languages involved. From that shared space, the decoder, with some implicit or explicit relevant constraints, could transform the representation into a concrete sentence in any desired language. In this section, we review some related work on this matter. We then describe a unified approach toward an universal attention-based NMT scheme. Our approach does not require any architecture modification and it can be trained to learn a minimal number of parameters compared to the other work. Related Work By extending the solution of sequence-to-sequence modeling using encoder-decoder architectures to multi-task learning, Luong2016 managed to achieve better performance on some INLINEFORM0 tasks such as translation, parsing and image captioning compared to individual tasks. Specifically in translation, the work utilizes multiple encoders to translate from multiple languages, and multiple decoders to translate to multiple languages. In this view of multilingual translation, each language in source or target side is modeled by one encoder or decoder, depending on the side of the translation. Due to the natural diversity between two tasks in that multi-task learning scenario, e.g. translation and parsing, it could not feature the attention mechanism although it has proven its effectiveness in NMT. There exists two directions which proposed for multilingual translation scenarios where they leverage the attention mechanism. The first one is indicated in the work from BIBREF8 , where it introduce an one-to-many multilingual NMT system to translates from one source language into multiple target languages. Having one source language, the attention mechanism is then handed over to the corresponding decoder. The objective function is changed to adapt to multilingual settings. In testing time, the parameters specific to a desired language pair are used to perform the translation. Firat2016 proposed another approach which genuinely delivers attention-based NMT to multilingual translation. As in BIBREF9 , their approach utilizes one encoder per source language and one decoder per target language for many-to-many translation tasks. Instead of a quadratic number of independent attention layers, however, one single attention mechanism is integrated into their NMT, performing an affine transformation between the hidden layer of INLINEFORM0 source languages and that one of INLINEFORM1 target languages. It is required to change their architecture to accomodate such a complicated shared attention mechanism. In a separate effort to achieve multilingual NMT, the work of Zoph2016 leverages available parallel data from other language pairs to help reducing possible ambiguities in the translation process into a single target language. They employed the multi-source attention-based NMT in a way that only one attention mechanism is required despite having multiple encoders. To achieve this, the outputs of the encoders were combined before feeding to the attention layer. They implemented two types of encoder combination; One is adding a non-linear layer on the concatenation of the encoders' hidden states. The other is using a variant of LSTM taking the respective gate values from the individual LSTM units of the encoders. As a result, the combined hidden states contain information from both encoders , thus encode the common semantic of the two source languages. Universal Encoder and Decoder Inspired by the multi-source NMT as additional parallel data in several languages are expected to benefit single translations, we aim to develop a NMT-based approach toward an universal framework to perform multilingual translation. Our solution features two treatments: 1) Coding the words in different languages as different words in the language-mixed vocabularies and 2) Forcing the NMT to translating a representation of source sentences into the sentences in a desired target language. Language-specific Coding. When the encoder of a NMT system considers words across languages as different words, with a well-chosen architecture, it is expected to be able to learn a good representation of the source words in an embedding space in which words carrying similar meaning would have a closer distance to each others than those are semantically different. This should hold true when the words have the same or similar surface form, such as (@de@Obama; @en@Obama) or (@de@Projektion; @en@projection). This should also hold true when the words have the same or similar meaning across languages, such as (@en@car; @en@automobile) or (@de@Flussufer; @en@bank). Our encoder then acts similarly to the one of multi-source approach BIBREF10 , collecting additional information from other sources for better translations, but with a much simpler embedding function. Unlike them, we need only one encoder, so we could reduce the number of parameters to learn. Furthermore, we neither need to change the network architecture nor depend on which recurrent unit (GRU, LSTM or simple RNN) is currently using in the encoder. We could apply the same trick to the target sentences and thus enable many-to-many translation capability of our NMT system. Similar to the multi-target translation BIBREF8 , we exploit further the correlation in semantics of those target sentences across different languages. The main difference between our approach and the work of BIBREF8 is that we need only one decoder for all target languages. Given one encoder for multiple source languages and one decoder for multiple target languages, it is trivial to incorporate the attention mechanism as in the case of a regular NMT for single language translation. In training, the attention layers were directed to learn relevant alignments between words in specific language pair and forward the produced context vector to the decoder. Now we rely totally on the network to learn good alignments between source and target sides. In fact, giving more information, our system are able to form nice alignments. In comparison to other research that could perform complete multi-task learning, e.g. the work from BIBREF9 or the approach proposed by BIBREF11 , our method is able to accommodate the attention layers seemlessly and easily. It also draws a clear distinction from those works in term of the complexity of the whole network: considerably less parameters to learn, thus reduces overfitting, with a conventional attention mechanism and a standard training procedure. Target Forcing. While language-specific coding allows us to implement a multilingual attention-based NMT, there are two issues we have to consider before training the network. The first is that the number of rare words would increase in proportion with the number of languages involved. This might be solved by applying a rare word treatment method with appropriate awareness of the vocabularies' size. The second one is more problematic: Ambiguity level in the translation process definitely increases due to the additional introduction of words having the same or similar meaning across languages at both source and target sides. We deal with the problem by explicitly forcing the attention and translation to the direction that we prefer, expecting the information would limit the ambiguity to the scope of one language instead of all target languages. We realize this idea by adding at the beginning and at the end of every source sentences a special symbol indicating the language they would be translated into. For example, in a multilingual NMT, when a source sentence is German and the target language is English, the original sentence (already language-specific coded) is: @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag Now when we force it to be translated into English, the target-forced sentence becomes: <E> @de@darum @de@geht @de@es @de@in @de@meinem @de@Vortrag <E> Due to the nature of recurrent units used in the encoder and decoder, in training, those starting symbols encourage the network learning the translation of following target words in a particular language pair. In testing time, information of the target language we provided help to limit the translated candidates, hence forming the translation in the desired language. Figure FIGREF8 illustrates the essence of our approach. With two steps in the preprocessing phase, namely language-specific coding and target forcing, we are able to employ multilingual attention-based NMT without any special treatment in training such a standard architecture. Our encoder and attention-enable decoder can be seen as a shared encoder and decoder across languages, or an universal encoder and decoder. The flexibitily of our approach allow us to integrate any language into source or target side. As we will see in Section SECREF4 , it has proven to be extremely helpful not only in low-resourced scenarios but also in translation of well-resourced language pairs as it provides a novel way to make use of large monolingual corpora in NMT. Evaluation In this section, we describe the evaluation of our proposed approach in comparisons with the strong baselines using NMT in two scenarios: the translation of an under-resource language pair and the translation of a language pair that does not exist any paralled data at all. Experimental Settings Training Data. We choose WIT3's TED corpus BIBREF12 as the basis of our experiments since it might be the only high-quality parallel data of many low-resourced language pairs. TED is also multilingual in a sense that it includes numbers of talks which are commonly translated into many languages. In addition, we use a much larger corpus provided freely by WMT organizers when we evaluate the impact of our approach in a real machine translation campaign. It includes the paralled corpus extracted from the digital corpus of European Parliament (EPPS), the News Commentary (NC) and the web-crawled parallel data (CommonCrawl). While the number of sentences in popular TED corpora varies from 13 thousands to 17 thousands, the total number of sentences in those larger corpus is approximately 3 million sentences. Neural Machine Translation Setup. All experiments have been conducted using NMT framework Nematus, Following the work of Sennrich2016a, subword segmentation is handled in the prepocessing phase using Byte-Pair Encoding (BPE). Excepts stated clearly in some experiments, we set the number of BPE merging operations at 39500 on the joint of source and target data. When training all NMT systems, we take out the sentence pairs exceeding 50-word length and shuffle them inside every minibatch. Our short-list vocabularies contain 40,000 most frequent words while the others are considered as rare words and applied the subword translation. We use an 1024-cell GRU layer and 1000-dimensional embeddings with dropout at every layer with the probability of 0.2 in the embedding and hidden layers and 0.1 in the input and ourput layers. We trained our systems using gradient descent optimization with Adadelta BIBREF13 on minibatches of size 80 and the gradient is rescaled whenever its norm exceed 1.0. All the trainings last approximately seven days if the early-stopping condition could not be reached. At a certain time, an external evaluation script on BLEU BIBREF14 is conducted on a development set to decide the early-stopping condition. This evaluation script has also being used to choose the model archiving the best BLEU on the development set instead of the maximal loglikelihood between the translations and target sentences while training. In translation, the framework produces INLINEFORM0 -best candidates and we then use a beam search with the beam size of 12 to get the best translation. Under-resourced Translation First, we consider the translation for an under-resourced pair of languages. Here a small portion of the large parallel corpus for English-German is used as a simulation for the scenario where we do not have much parallel data: Translating texts in English to German. We perform language-specific coding in both source and target sides. By accommodating the German monolingual data as an additional input (German INLINEFORM0 German), which we called the mix-source approach, we could enrich the training data in a simple, natural way. Given this under-resourced situation, it could help our NMT obtain a better representation of the source side, hence, able to learn the translation relationship better. Including monolingual data in this way might also improve the translation of some rare word types such as named entities. Furthermore, as the ultimate goal of our work, we would like to investigate the advantages of multilinguality in NMT. We incorporate a similar portion of French-German parallel corpus into the English-German one. As discussed in Section SECREF5 , it is expected to help reducing the ambiguity in translation between one language pair since it utilizes the semantic context provided by the other source language. We name this mix-multi-source. Table TABREF16 summarizes the performance of our systems measured in BLEU on two test sets, tst2013 and tst2014. Compared to the baseline NMT system which is solely trained on TED English-German data, our mix-source system achieves a considerable improvement of 2.6 BLEU points on tst2013 and 2.1 BLEU points on and tst2014 . Adding French data to the source side and their corresponding German data to the target side in our mix-multi-source system also help to gain 2.2 and 1.6 BLEU points more on tst2013 tst2014, respectively. We observe a better improvement from our mix-source system compared to our mix-multi-source system. We speculate the reason that the mix-source encoder utilize the same information shared in two languages while the mix-multi-source receives and processes similar information in the other language but not necessarily the same. We might validate this hypothesis by comparing two systems trained on a common English-German-French corpus of TED. We put it in our future work's plan. As we expected Figure FIGREF19 shows how different words in different languages can be close in the shared space after being learned to translate into a common language. We extract the word embeddings from the encoder of the mix-multi-source (En,Fr INLINEFORM0 De,De) after training, remove the language-specific codes (@en@ and @fr@)and project the word vectors to the 2D space using t-SNE BIBREF15 . Using large monolingual data in NMT. A standard NMT system employs parallel data only. While good parallel corpora are limited in number, getting monolingual data of an arbitrary language is trivial. To make use of German monolingual corpus in an English INLINEFORM0 German NMT system, sennrich2016b built a separate German INLINEFORM1 English NMT using the same parallel corpus, then they used that system to translate the German monolingual corpus back to English, forming a synthesis parallel data. gulcehre2015 trained another RNN-based language model to score the monolingual corpus and integrate it to the NMT system through shallow or deep fusion. Both methods requires to train separate systems with possibly different hyperparameters for each. Conversely, by applying mix-source method to the big monolingual data, we need to train only one network. We mix the TED parallel corpus and the substantial monolingual corpus (EPPS+NC+CommonCrawl) and train a mix-source NMT system from those data. The first result is not encouraging when its performance is even worse than the baseline NMT which is trained on the small parallel data only. Not using the same information in the source side, as we discussed in case of mix-multi-source strategy, could explain the degrading in performance of such a system. But we believe that the magnitude and unbalancing of the corpus are the main reasons. The data contains nearly four millions sentences but only around twenty thousands of them (0.5%) are the genuine parallel data. As a quick attempt, after we get the model with that big data, we continue training on the real parallel corpus for some more epochs. When this adaptation is applied, our system brings an improvement of +1.52 BLEU on tst2013 and +1.06 BLEU on tst2014 (Table TABREF21 ). Zero-resourced Translation Among low-resourced scenarios, zero-resourced translation task stands in an extreme level. A zero-resourced translation task is one of the most difficult situation when there is no parallel data between the translating language pair. To the best of our knowledge, there have been yet existed a published work about using NMT for zero-resourced translation tasks up to now. In this section, we extend our strategies using the proposed multilingual NMT approach as first attempts to this extreme situation. We employ language-specific coding and target forcing in a strategy called bridge. Unlike the strategies used in under-resourced translation task, bridge is an entire many-to-many multilingual NMT. Simulating a zero-resourced German INLINEFORM0 French translation task given the available German-English and English-French parallel corpora, after applying language-specific coding and target forcing for each corpus, we mix those data with an English-English data as a “bridge” creating some connection between German and French. We also propose a variant of this strategy that we incorporate French-French data. And we call it universal. We evaluate bridge and universal systems on two German INLINEFORM0 French test sets. They are compared to a direct system, which is an NMT trained on German INLINEFORM1 French data, and to a pivot system, which essentially consists of two separate NMTs trained to translate from German to English and English to French. The direct system should not exist in a real zero-resourced situation. We refer it as the perfect system for comparison purpose only. In case of the pivot system, to generate a translated text in French from a German sentence, we first translate it to English, then the output sentence is fed to the English INLINEFORM2 German NMT system to obtain the French translation. Since there are more than two languages involved in those systems, we increase the number of BPE merging operations proportionally in order to reduce the number of rare words in such systems. We do not expect our proposed systems to perform well with this primitive way of building direct translating connections since this is essentially a difficult task. We report the performance of those systems in Table TABREF23 . Unsupprisingly, both bridge and universal systems perform worse than the pivot one. We consider two possible reasons: Our target forcing mechanism is moderately primitive. Since the process is applied after language-specific coding, the target forcing symbol is the same for all source sentences in every languages. Thus, the forcing strength might not be enough to guide the decision of the next words. Once the very first word is translated into a word in wrong language, the following words tend to be translated into that wrong language again. Table TABREF24 shows some statistics of the translated words and sentences in wrong language. Balancing of the training corpus. Although it is not severe as in the case of mix-source system for large monolingual data, the limited number of sentences in target language can affect the training. The difference of 1.07 BLEU points between bridge and universal might explain this assumption as we added more target data (French) in universal strategy, thus reducing the unbalance in training. Those issues would be addressed in our following future work toward the multilingual attention-based NMT. Conclusion and Future Work In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework. By treating words in different languages as different words and force the attention and translation to the direction of desired target language, we are able to employ attention-enable NMT toward a multilingual translation system. Our proposed approach alleviates the need of complicated architecture re-designing when accommodating attention mechanism. In addition, the number of free parameters to learn in our network does not go beyond that magnitute of a single NMT system. With its universality, our approach has shown its effectiveness in an under-resourced translation task with considerable improvements. In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages. Nevertheless, there are issues that we can continue working on to address in future work. A more balancing data would be helpful for this framework. The mechanism of forcing the NMT system to the right target language could be improved. We could conduct more detailed analyses of the various strategies under the framework to show its universarity.
Do they test their framework performance on commonly used language pairs, such as English-to-German?
897ba53ef44f658c128125edd26abf605060fb13
Do Ġthey Ġtest Ġtheir Ġframework Ġperformance Ġon Ġcommonly Ġused Ġlanguage Ġpairs , Ġsuch Ġas ĠEnglish - to - German ?
Yes
<s> Do Ġthey Ġtest Ġtheir Ġframework Ġperformance Ġon Ġcommonly Ġused Ġlanguage Ġpairs , Ġsuch Ġas ĠEnglish - to - German ? </s> Introduction </s> Ne ural ĠMachine ĠTranslation Ġ( N MT ) Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġtranslation Ġtasks Ġwhen ĠN MT Ġsystems Ġperform Ġbest Ġin Ġrecent Ġmachine Ġtranslation Ġcampaigns ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠCompared Ġto Ġphrase - based ĠStatistical ĠMachine ĠTranslation Ġ( SM T ) Ġwhich Ġis Ġbasically Ġan Ġensemble Ġof Ġdifferent Ġfeatures Ġtrained Ġand Ġtuned Ġseparately , ĠN MT Ġdirectly Ġmodeling Ġthe Ġtranslation Ġrelationship Ġbetween Ġsource Ġand Ġtarget Ġsentences . ĠUnlike ĠSM T , ĠN MT Ġdoes Ġnot Ġrequire Ġmuch Ġlinguistic Ġinformation Ġand Ġlarge Ġmon oling ual Ġdata Ġto Ġachieve Ġgood Ġperformances . </s> An ĠN MT Ġconsists Ġof Ġan Ġenc oder Ġwhich Ġrec urs ively Ġreads Ġand Ġrepresents Ġthe Ġwhole Ġsource Ġsentence Ġinto Ġa Ġcontext Ġvector Ġand Ġa Ġrecurrent Ġdec oder Ġwhich Ġtakes Ġthe Ġcontext Ġvector Ġand Ġits Ġprevious Ġstate Ġto Ġpredict Ġthe Ġnext Ġtarget Ġword . ĠIt Ġis Ġthen Ġtrained Ġin Ġan Ġend - to - end Ġfashion Ġto Ġlearn Ġparameters Ġwhich Ġmaxim izes Ġthe Ġlikelihood Ġbetween Ġthe Ġoutputs Ġand Ġthe Ġreferences . ĠRecently , Ġattention - based ĠN MT Ġhas Ġbeen Ġfeatured Ġin Ġmost Ġstate - of - the - art Ġsystems . ĠFirst Ġintroduced Ġby ĠB IB REF 2 Ġ, Ġattention Ġmechanism Ġis Ġintegrated Ġin Ġdec oder Ġside Ġas Ġfeed forward Ġlayers . ĠIt Ġallows Ġthe ĠN MT Ġto Ġdecide Ġwhich Ġsource Ġwords Ġshould Ġtake Ġpart Ġin Ġthe Ġpredicting Ġprocess Ġof Ġthe Ġnext Ġtarget Ġwords . ĠIt Ġhelps Ġto Ġimprove ĠN MT s Ġsignificantly . ĠNevertheless , Ġsince Ġthe Ġattention Ġmechanism Ġis Ġspecific Ġto Ġa Ġparticular Ġsource Ġsentence Ġand Ġthe Ġconsidering Ġtarget Ġword , Ġit Ġis Ġalso Ġspecific Ġto Ġparticular Ġlanguage Ġpairs . </s> Some Ġrecent Ġwork Ġhas Ġfocused Ġon Ġextending Ġthe ĠN MT Ġframework Ġto Ġmult ilingual Ġscenarios . ĠBy Ġtraining Ġsuch Ġnetwork Ġusing Ġparallel Ġcorpor a Ġin Ġnumber Ġof Ġdifferent Ġlanguages , ĠN MT Ġcould Ġbenefit Ġfrom Ġadditional Ġinformation Ġembedded Ġin Ġa Ġcommon Ġsemantic Ġspace Ġacross Ġlanguages . ĠBasically , Ġthe Ġproposed ĠN MT Ġare Ġrequired Ġto Ġemploy Ġmultiple Ġenc od ers Ġor Ġmultiple Ġdec od ers Ġto Ġdeal Ġwith Ġmult iling uality . ĠFurthermore , Ġin Ġorder Ġto Ġavoid Ġthe Ġtight Ġdependency Ġof Ġthe Ġattention Ġmechanism Ġto Ġspecific Ġlanguage Ġpairs , Ġthey Ġalso Ġneed Ġto Ġmodify Ġtheir Ġarchitecture Ġto Ġcombine Ġeither Ġthe Ġenc od ers Ġor Ġthe Ġattention Ġlayers . ĠThese Ġmodifications Ġare Ġspecific Ġto Ġthe Ġpurpose Ġof Ġthe Ġtasks Ġas Ġwell . ĠThus , Ġthose Ġmult ilingual ĠN MT s Ġare Ġmore Ġcomplicated , Ġmuch Ġmore Ġfree Ġparameters Ġto Ġlearn Ġand Ġmore Ġdifficult Ġto Ġperform Ġstandard Ġtrain ings Ġcompared Ġto Ġthe Ġoriginal ĠN MT . </s> In Ġthis Ġpaper , Ġwe Ġintroduce Ġa Ġunified Ġapproach Ġto Ġseamlessly Ġextend Ġthe Ġoriginal ĠN MT Ġto Ġmult ilingual Ġsettings . ĠOur Ġapproach Ġallows Ġus Ġto Ġintegrate Ġany Ġlanguage Ġin Ġany Ġside Ġof Ġthe Ġenc oder - dec oder Ġarchitecture Ġwith Ġonly Ġone Ġenc oder Ġand Ġone Ġdec oder Ġfor Ġall Ġthe Ġlanguages Ġinvolved . ĠMoreover , Ġit Ġis Ġnot Ġnecessary Ġto Ġdo Ġany Ġnetwork Ġmodification Ġto Ġenable Ġattention Ġmechanism Ġin Ġour ĠN MT Ġsystems . ĠWe Ġthen Ġapply Ġour Ġpro pro sed Ġframework Ġin Ġtwo Ġdemanding Ġscenarios : Ġunder - res ourced Ġtranslation Ġand Ġzero - res ourced Ġtranslation . ĠThe Ġresults Ġshow Ġthat Ġbringing Ġmult iling uality Ġto ĠN MT Ġhelps Ġto Ġimprove Ġindividual Ġtranslations . ĠWith Ġsome Ġinsightful Ġanalyses Ġof Ġthe Ġresults , Ġwe Ġset Ġour Ġgoal Ġtoward Ġa Ġfully Ġmult ilingual ĠN MT Ġframework . </s> The Ġpaper Ġstarts Ġwith Ġa Ġdetailed Ġintroduction Ġto Ġattention - based ĠN MT . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġrelated Ġwork Ġabout Ġmulti - task ĠN MT Ġis Ġreviewed . ĠSection ĠSEC REF 5 Ġdescribes Ġour Ġproposed Ġapproach Ġand Ġthorough Ġcomparisons Ġto Ġthe Ġrelated Ġwork . ĠIt Ġis Ġfollowed Ġby Ġa Ġsection Ġof Ġevaluating Ġour Ġsystems Ġin Ġtwo Ġaforementioned Ġscenarios , Ġin Ġwhich Ġdifferent Ġstrategies Ġhave Ġbeen Ġemployed Ġunder Ġa Ġunified Ġapproach Ġ( Section ĠSEC REF 4 Ġ). ĠFinally , Ġthe Ġpaper Ġends Ġwith Ġconclusion Ġand Ġfuture Ġwork . </s> This Ġwork Ġis Ġlic enced Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Ne ural ĠMachine ĠTranslation : ĠBackground </s> An ĠN MT Ġsystem Ġconsists Ġof Ġan Ġenc oder Ġwhich Ġautomatically Ġlearns Ġthe Ġcharacteristics Ġof Ġa Ġsource Ġsentence Ġinto Ġfix - length Ġcontext Ġvectors Ġand Ġa Ġdec oder Ġthat Ġrec urs ively Ġcombines Ġthe Ġproduced Ġcontext Ġvectors Ġwith Ġthe Ġprevious Ġtarget Ġword Ġto Ġgenerate Ġthe Ġmost Ġprobable Ġword Ġfrom Ġa Ġtarget Ġvocabulary . </s> More Ġspecifically , Ġa Ġbid irection al Ġrecurrent Ġenc oder Ġreads Ġevery Ġwords ĠIN LINE FORM 0 Ġof Ġa Ġsource Ġsentence ĠIN LINE FORM 1 Ġand Ġenc odes Ġa Ġrepresentation ĠIN LINE FORM 2 Ġof Ġthe Ġsentence Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġconc at inated Ġfrom Ġthose Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : ĠIN LINE FORM 4 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġone - hot Ġvector Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġword Ġembed ding Ġmatrix Ġwhich Ġis Ġshared Ġacross Ġthe Ġsource Ġwords . ĠIN LINE FORM 3 Ġis Ġthe Ġrecurrent Ġunit Ġcomputing Ġthe Ġcurrent Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġbased Ġon Ġthe Ġprevious Ġhidden Ġstate . ĠIN LINE FORM 4 Ġis Ġthen Ġcalled Ġan Ġannotation Ġvector , Ġwhich Ġenc odes Ġthe Ġsource Ġsentence Ġup Ġto Ġthe Ġtime ĠIN LINE FORM 5 Ġfrom Ġboth Ġforward Ġand Ġbackward Ġdirections . ĠRec urrent Ġunits Ġin ĠN MT Ġcan Ġbe Ġa Ġsimple Ġrecurrent Ġneural Ġnetwork Ġunit Ġ( R NN ), Ġa ĠLong ĠShort - Term ĠMemory Ġunit Ġ( L ST M ) ĠB IB REF 3 Ġor Ġa ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 4 </s> Similar Ġto Ġthe Ġenc oder , Ġthe Ġrecurrent Ġdec oder Ġgenerates Ġone Ġtarget Ġword ĠIN LINE FORM 0 Ġto Ġform Ġa Ġtranslated Ġtarget Ġsentence ĠIN LINE FORM 1 Ġin Ġthe Ġend . ĠAt Ġthe Ġtime ĠIN LINE FORM 2 Ġ, Ġit Ġtakes Ġthe Ġprevious Ġhidden Ġstate Ġof Ġthe Ġdec oder ĠIN LINE FORM 3 Ġ, Ġthe Ġprevious Ġembedded Ġword Ġrepresentation ĠIN LINE FORM 4 Ġand Ġa Ġtime - specific Ġcontext Ġvector ĠIN LINE FORM 5 Ġas Ġinputs Ġto Ġcalculate Ġthe Ġcurrent Ġhidden Ġstate ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 </s> Again , ĠIN LINE FORM 0 Ġis Ġthe Ġrecurrent Ġactivation Ġfunction Ġof Ġthe Ġdec oder Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġshared Ġword Ġembed ding Ġmatrix Ġof Ġthe Ġtarget Ġsentences . ĠThe Ġcontext Ġvector ĠIN LINE FORM 2 Ġis Ġcalculated Ġbased Ġon Ġthe Ġannotation Ġvectors Ġfrom Ġthe Ġenc oder . ĠBefore Ġfeeding Ġthe Ġannotation Ġvectors Ġinto Ġthe Ġdec oder , Ġan Ġattention Ġmechanism Ġis Ġset Ġup Ġin Ġbetween , Ġin Ġorder Ġto Ġchoose Ġwhich Ġannotation Ġvectors Ġshould Ġcontribute Ġto Ġthe Ġpredicting Ġdecision Ġof Ġthe Ġnext Ġtarget Ġword . ĠInt uitive ly , Ġa Ġrelevance Ġbetween Ġthe Ġprevious Ġtarget Ġword Ġand Ġthe Ġannotation Ġvectors Ġcan Ġbe Ġused Ġto Ġform Ġsome Ġattention Ġscenario . ĠThere Ġexists Ġseveral Ġways Ġto Ġcalculate Ġthe Ġrelevance Ġas Ġshown Ġin ĠB IB REF 5 Ġ, Ġbut Ġwhat Ġwe Ġdescribe Ġhere Ġfollows Ġthe Ġproposed Ġmethod Ġof ĠB IB REF 2 ĠDIS PLAY FORM 0 </s> In ĠB IB REF 2 Ġ, Ġthis Ġattention Ġmechanism , Ġoriginally Ġcalled Ġalignment Ġmodel , Ġhas Ġbeen Ġemployed Ġas Ġa Ġsimple Ġfeed forward Ġnetwork Ġwith Ġthe Ġfirst Ġlayer Ġis Ġa Ġlearn able Ġlayer Ġvia ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġrelevance Ġscores ĠIN LINE FORM 3 Ġare Ġthen Ġnormalized Ġinto Ġattention Ġweights ĠIN LINE FORM 4 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 5 Ġis Ġcalculated Ġas Ġthe Ġweighted Ġsum Ġof Ġall Ġannotation Ġvectors ĠIN LINE FORM 6 Ġ. ĠDepending Ġon Ġhow Ġmuch Ġattention Ġthe Ġtarget Ġword Ġat Ġtime ĠIN LINE FORM 7 Ġput Ġon Ġthe Ġsource Ġstates ĠIN LINE FORM 8 Ġ, Ġa Ġsoft Ġalignment Ġis Ġlearned . ĠBy Ġbeing Ġemployed Ġthis Ġway , Ġword Ġalignment Ġis Ġnot Ġa Ġlatent Ġvariable Ġbut Ġa Ġparam et ri zed Ġfunction , Ġmaking Ġthe Ġalignment Ġmodel Ġdifferent iable . ĠThus , Ġit Ġcould Ġbe Ġtrained Ġtogether Ġwith Ġthe Ġwhole Ġarchitecture Ġusing Ġback prop ag ation . </s> One Ġof Ġthe Ġmost Ġsevere Ġproblems Ġof ĠN MT Ġis Ġhandling Ġof Ġthe Ġrare Ġwords , Ġwhich Ġare Ġnot Ġin Ġthe Ġshort Ġlists Ġof Ġthe Ġvoc ab ular ies , Ġi . e . Ġout - of - voc abulary Ġ( OO V ) Ġwords , Ġor Ġdo Ġnot Ġappear Ġin Ġthe Ġtraining Ġset Ġat Ġall . ĠIn ĠB IB REF 6 Ġ, Ġthe Ġrare Ġtarget Ġwords Ġare Ġcopied Ġfrom Ġtheir Ġaligned Ġsource Ġwords Ġafter Ġthe Ġtranslation . ĠThis Ġhe uristic Ġworks Ġwell Ġwith ĠO OV Ġwords Ġand Ġnamed Ġentities Ġbut Ġunable Ġto Ġtranslate Ġunseen Ġwords . ĠIn ĠB IB REF 7 Ġ, Ġtheir Ġproposed ĠN MT Ġmodels Ġhave Ġbeen Ġshown Ġto Ġnot Ġonly Ġbe Ġeffective Ġon Ġreducing Ġvocabulary Ġsizes Ġbut Ġalso Ġhave Ġthe Ġability Ġto Ġgenerate Ġunseen Ġwords . ĠThis Ġis Ġachieved Ġby Ġsegment ing Ġthe Ġrare Ġwords Ġinto Ġsub word Ġunits Ġand Ġtranslating Ġthem . ĠThe Ġstate - of - the - art Ġtranslation Ġsystems Ġessentially Ġemploy Ġsub word ĠN MT ĠB IB REF 7 Ġ. </s> Universal ĠEnc oder Ġand ĠDecoder Ġfor ĠMult ilingual ĠNeural ĠMachine ĠTranslation </s> While Ġthe Ġmajority Ġof Ġprevious Ġresearch Ġhas Ġfocused Ġon Ġimproving Ġthe Ġperformance Ġof ĠN MT Ġon Ġindividual Ġlanguage Ġpairs Ġwith Ġindividual ĠN MT Ġsystems , Ġrecent Ġwork Ġhas Ġstarted Ġinvestigating Ġpotential Ġways Ġto Ġconduct Ġthe Ġtranslation Ġinvolved Ġin Ġmultiple Ġlanguages Ġusing Ġa Ġsingle ĠN MT Ġsystem . ĠThe Ġpossible Ġreason Ġexplaining Ġthese Ġefforts Ġlies Ġon Ġthe Ġunique Ġarchitecture Ġof ĠN MT . ĠUnlike ĠSM T , ĠN MT Ġconsists Ġof Ġseparated Ġneural Ġnetworks Ġfor Ġthe Ġsource Ġand Ġtarget Ġsides , Ġor Ġthe Ġenc oder Ġand Ġdec oder , Ġrespectively . ĠThis Ġallows Ġthese Ġcomponents Ġto Ġmap Ġa Ġsentence Ġin Ġany Ġlanguage Ġto Ġa Ġrepresentation Ġin Ġan Ġembed ding Ġspace Ġwhich Ġis Ġbelieved Ġto Ġshare Ġcommon Ġsemantics Ġamong Ġthe Ġsource Ġlanguages Ġinvolved . ĠFrom Ġthat Ġshared Ġspace , Ġthe Ġdec oder , Ġwith Ġsome Ġimplicit Ġor Ġexplicit Ġrelevant Ġconstraints , Ġcould Ġtransform Ġthe Ġrepresentation Ġinto Ġa Ġconcrete Ġsentence Ġin Ġany Ġdesired Ġlanguage . ĠIn Ġthis Ġsection , Ġwe Ġreview Ġsome Ġrelated Ġwork Ġon Ġthis Ġmatter . ĠWe Ġthen Ġdescribe Ġa Ġunified Ġapproach Ġtoward Ġan Ġuniversal Ġattention - based ĠN MT Ġscheme . ĠOur Ġapproach Ġdoes Ġnot Ġrequire Ġany Ġarchitecture Ġmodification Ġand Ġit Ġcan Ġbe Ġtrained Ġto Ġlearn Ġa Ġminimal Ġnumber Ġof Ġparameters Ġcompared Ġto Ġthe Ġother Ġwork . </s> Related ĠWork </s> By Ġextending Ġthe Ġsolution Ġof Ġsequence - to - sequence Ġmodeling Ġusing Ġenc oder - dec oder Ġarchitectures Ġto Ġmulti - task Ġlearning , ĠLu ong 2016 Ġmanaged Ġto Ġachieve Ġbetter Ġperformance Ġon Ġsome ĠIN LINE FORM 0 Ġtasks Ġsuch Ġas Ġtranslation , Ġparsing Ġand Ġimage Ġcaption ing Ġcompared Ġto Ġindividual Ġtasks . ĠSpecifically Ġin Ġtranslation , Ġthe Ġwork Ġutilizes Ġmultiple Ġenc od ers Ġto Ġtranslate Ġfrom Ġmultiple Ġlanguages , Ġand Ġmultiple Ġdec od ers Ġto Ġtranslate Ġto Ġmultiple Ġlanguages . ĠIn Ġthis Ġview Ġof Ġmult ilingual Ġtranslation , Ġeach Ġlanguage Ġin Ġsource Ġor Ġtarget Ġside Ġis Ġmodeled Ġby Ġone Ġenc oder Ġor Ġdec oder , Ġdepending Ġon Ġthe Ġside Ġof Ġthe Ġtranslation . ĠDue Ġto Ġthe Ġnatural Ġdiversity Ġbetween Ġtwo Ġtasks Ġin Ġthat Ġmulti - task Ġlearning Ġscenario , Ġe . g . Ġtranslation Ġand Ġparsing , Ġit Ġcould Ġnot Ġfeature Ġthe Ġattention Ġmechanism Ġalthough Ġit Ġhas Ġproven Ġits Ġeffectiveness Ġin ĠN MT . ĠThere Ġexists Ġtwo Ġdirections Ġwhich Ġproposed Ġfor Ġmult ilingual Ġtranslation Ġscenarios Ġwhere Ġthey Ġleverage Ġthe Ġattention Ġmechanism . ĠThe Ġfirst Ġone Ġis Ġindicated Ġin Ġthe Ġwork Ġfrom ĠB IB REF 8 Ġ, Ġwhere Ġit Ġintroduce Ġan Ġone - to - many Ġmult ilingual ĠN MT Ġsystem Ġto Ġtranslates Ġfrom Ġone Ġsource Ġlanguage Ġinto Ġmultiple Ġtarget Ġlanguages . ĠHaving Ġone Ġsource Ġlanguage , Ġthe Ġattention Ġmechanism Ġis Ġthen Ġhanded Ġover Ġto Ġthe Ġcorresponding Ġdec oder . ĠThe Ġobjective Ġfunction Ġis Ġchanged Ġto Ġadapt Ġto Ġmult ilingual Ġsettings . ĠIn Ġtesting Ġtime , Ġthe Ġparameters Ġspecific Ġto Ġa Ġdesired Ġlanguage Ġpair Ġare Ġused Ġto Ġperform Ġthe Ġtranslation . </s> F ir at 2016 Ġproposed Ġanother Ġapproach Ġwhich Ġgenuinely Ġdelivers Ġattention - based ĠN MT Ġto Ġmult ilingual Ġtranslation . ĠAs Ġin ĠB IB REF 9 Ġ, Ġtheir Ġapproach Ġutilizes Ġone Ġenc oder Ġper Ġsource Ġlanguage Ġand Ġone Ġdec oder Ġper Ġtarget Ġlanguage Ġfor Ġmany - to - many Ġtranslation Ġtasks . ĠInstead Ġof Ġa Ġquad r atic Ġnumber Ġof Ġindependent Ġattention Ġlayers , Ġhowever , Ġone Ġsingle Ġattention Ġmechanism Ġis Ġintegrated Ġinto Ġtheir ĠN MT , Ġperforming Ġan Ġaff ine Ġtransformation Ġbetween Ġthe Ġhidden Ġlayer Ġof ĠIN LINE FORM 0 Ġsource Ġlanguages Ġand Ġthat Ġone Ġof ĠIN LINE FORM 1 Ġtarget Ġlanguages . ĠIt Ġis Ġrequired Ġto Ġchange Ġtheir Ġarchitecture Ġto Ġacc om od ate Ġsuch Ġa Ġcomplicated Ġshared Ġattention Ġmechanism . </s> In Ġa Ġseparate Ġeffort Ġto Ġachieve Ġmult ilingual ĠN MT , Ġthe Ġwork Ġof ĠZ oph 2016 Ġlever ages Ġavailable Ġparallel Ġdata Ġfrom Ġother Ġlanguage Ġpairs Ġto Ġhelp Ġreducing Ġpossible Ġambig u ities Ġin Ġthe Ġtranslation Ġprocess Ġinto Ġa Ġsingle Ġtarget Ġlanguage . ĠThey Ġemployed Ġthe Ġmulti - source Ġattention - based ĠN MT Ġin Ġa Ġway Ġthat Ġonly Ġone Ġattention Ġmechanism Ġis Ġrequired Ġdespite Ġhaving Ġmultiple Ġenc od ers . ĠTo Ġachieve Ġthis , Ġthe Ġoutputs Ġof Ġthe Ġenc od ers Ġwere Ġcombined Ġbefore Ġfeeding Ġto Ġthe Ġattention Ġlayer . ĠThey Ġimplemented Ġtwo Ġtypes Ġof Ġenc oder Ġcombination ; ĠOne Ġis Ġadding Ġa Ġnon - linear Ġlayer Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġenc od ers ' Ġhidden Ġstates . ĠThe Ġother Ġis Ġusing Ġa Ġvariant Ġof ĠL ST M Ġtaking Ġthe Ġrespective Ġgate Ġvalues Ġfrom Ġthe Ġindividual ĠL ST M Ġunits Ġof Ġthe Ġenc od ers . ĠAs Ġa Ġresult , Ġthe Ġcombined Ġhidden Ġstates Ġcontain Ġinformation Ġfrom Ġboth Ġenc od ers Ġ, Ġthus Ġencode Ġthe Ġcommon Ġsemantic Ġof Ġthe Ġtwo Ġsource Ġlanguages . </s> Universal ĠEnc oder Ġand ĠDecoder </s> Insp ired Ġby Ġthe Ġmulti - source ĠN MT Ġas Ġadditional Ġparallel Ġdata Ġin Ġseveral Ġlanguages Ġare Ġexpected Ġto Ġbenefit Ġsingle Ġtranslations , Ġwe Ġaim Ġto Ġdevelop Ġa ĠN MT - based Ġapproach Ġtoward Ġan Ġuniversal Ġframework Ġto Ġperform Ġmult ilingual Ġtranslation . ĠOur Ġsolution Ġfeatures Ġtwo Ġtreatments : Ġ1 ) ĠC oding Ġthe Ġwords Ġin Ġdifferent Ġlanguages Ġas Ġdifferent Ġwords Ġin Ġthe Ġlanguage - m ixed Ġvoc ab ular ies Ġand Ġ2 ) ĠFor cing Ġthe ĠN MT Ġto Ġtranslating Ġa Ġrepresentation Ġof Ġsource Ġsentences Ġinto Ġthe Ġsentences Ġin Ġa Ġdesired Ġtarget Ġlanguage . </s> Language - specific ĠC oding . ĠWhen Ġthe Ġenc oder Ġof Ġa ĠN MT Ġsystem Ġconsiders Ġwords Ġacross Ġlanguages Ġas Ġdifferent Ġwords , Ġwith Ġa Ġwell - ch osen Ġarchitecture , Ġit Ġis Ġexpected Ġto Ġbe Ġable Ġto Ġlearn Ġa Ġgood Ġrepresentation Ġof Ġthe Ġsource Ġwords Ġin Ġan Ġembed ding Ġspace Ġin Ġwhich Ġwords Ġcarrying Ġsimilar Ġmeaning Ġwould Ġhave Ġa Ġcloser Ġdistance Ġto Ġeach Ġothers Ġthan Ġthose Ġare Ġsem antically Ġdifferent . ĠThis Ġshould Ġhold Ġtrue Ġwhen Ġthe Ġwords Ġhave Ġthe Ġsame Ġor Ġsimilar Ġsurface Ġform , Ġsuch Ġas Ġ(@ de @ Obama ; Ġ@ en @ Obama ) Ġor Ġ(@ de @ Pro j ek tion ; Ġ@ en @ project ion ). ĠThis Ġshould Ġalso Ġhold Ġtrue Ġwhen Ġthe Ġwords Ġhave Ġthe Ġsame Ġor Ġsimilar Ġmeaning Ġacross Ġlanguages , Ġsuch Ġas Ġ(@ en @ car ; Ġ@ en @ aut om obile ) Ġor Ġ(@ de @ Fl uss u fer ; Ġ@ en @ bank ). ĠOur Ġenc oder Ġthen Ġacts Ġsimilarly Ġto Ġthe Ġone Ġof Ġmulti - source Ġapproach ĠB IB REF 10 Ġ, Ġcollecting Ġadditional Ġinformation Ġfrom Ġother Ġsources Ġfor Ġbetter Ġtranslations , Ġbut Ġwith Ġa Ġmuch Ġsimpler Ġembed ding Ġfunction . ĠUnlike Ġthem , Ġwe Ġneed Ġonly Ġone Ġenc oder , Ġso Ġwe Ġcould Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġto Ġlearn . ĠFurthermore , Ġwe Ġneither Ġneed Ġto Ġchange Ġthe Ġnetwork Ġarchitecture Ġnor Ġdepend Ġon Ġwhich Ġrecurrent Ġunit Ġ( GR U , ĠL ST M Ġor Ġsimple ĠR NN ) Ġis Ġcurrently Ġusing Ġin Ġthe Ġenc oder . </s> We Ġcould Ġapply Ġthe Ġsame Ġtrick Ġto Ġthe Ġtarget Ġsentences Ġand Ġthus Ġenable Ġmany - to - many Ġtranslation Ġcapability Ġof Ġour ĠN MT Ġsystem . ĠSimilar Ġto Ġthe Ġmulti - target Ġtranslation ĠB IB REF 8 Ġ, Ġwe Ġexploit Ġfurther Ġthe Ġcorrelation Ġin Ġsemantics Ġof Ġthose Ġtarget Ġsentences Ġacross Ġdifferent Ġlanguages . ĠThe Ġmain Ġdifference Ġbetween Ġour Ġapproach Ġand Ġthe Ġwork Ġof ĠB IB REF 8 Ġis Ġthat Ġwe Ġneed Ġonly Ġone Ġdec oder Ġfor Ġall Ġtarget Ġlanguages . ĠGiven Ġone Ġenc oder Ġfor Ġmultiple Ġsource Ġlanguages Ġand Ġone Ġdec oder Ġfor Ġmultiple Ġtarget Ġlanguages , Ġit Ġis Ġtrivial Ġto Ġincorporate Ġthe Ġattention Ġmechanism Ġas Ġin Ġthe Ġcase Ġof Ġa Ġregular ĠN MT Ġfor Ġsingle Ġlanguage Ġtranslation . ĠIn Ġtraining , Ġthe Ġattention Ġlayers Ġwere Ġdirected Ġto Ġlearn Ġrelevant Ġalign ments Ġbetween Ġwords Ġin Ġspecific Ġlanguage Ġpair Ġand Ġforward Ġthe Ġproduced Ġcontext Ġvector Ġto Ġthe Ġdec oder . ĠNow Ġwe Ġrely Ġtotally Ġon Ġthe Ġnetwork Ġto Ġlearn Ġgood Ġalign ments Ġbetween Ġsource Ġand Ġtarget Ġsides . ĠIn Ġfact , Ġgiving Ġmore Ġinformation , Ġour Ġsystem Ġare Ġable Ġto Ġform Ġnice Ġalign ments . </s> In Ġcomparison Ġto Ġother Ġresearch Ġthat Ġcould Ġperform Ġcomplete Ġmulti - task Ġlearning , Ġe . g . Ġthe Ġwork Ġfrom ĠB IB REF 9 Ġor Ġthe Ġapproach Ġproposed Ġby ĠB IB REF 11 Ġ, Ġour Ġmethod Ġis Ġable Ġto Ġaccommodate Ġthe Ġattention Ġlayers Ġseem lessly Ġand Ġeasily . ĠIt Ġalso Ġdraws Ġa Ġclear Ġdistinction Ġfrom Ġthose Ġworks Ġin Ġterm Ġof Ġthe Ġcomplexity Ġof Ġthe Ġwhole Ġnetwork : Ġconsiderably Ġless Ġparameters Ġto Ġlearn , Ġthus Ġreduces Ġover fitting , Ġwith Ġa Ġconventional Ġattention Ġmechanism Ġand Ġa Ġstandard Ġtraining Ġprocedure . </s> Target ĠFor cing . ĠWhile Ġlanguage - specific Ġcoding Ġallows Ġus Ġto Ġimplement Ġa Ġmult ilingual Ġattention - based ĠN MT , Ġthere Ġare Ġtwo Ġissues Ġwe Ġhave Ġto Ġconsider Ġbefore Ġtraining Ġthe Ġnetwork . ĠThe Ġfirst Ġis Ġthat Ġthe Ġnumber Ġof Ġrare Ġwords Ġwould Ġincrease Ġin Ġproportion Ġwith Ġthe Ġnumber Ġof Ġlanguages Ġinvolved . ĠThis Ġmight Ġbe Ġsolved Ġby Ġapplying Ġa Ġrare Ġword Ġtreatment Ġmethod Ġwith Ġappropriate Ġawareness Ġof Ġthe Ġvoc ab ular ies ' Ġsize . ĠThe Ġsecond Ġone Ġis Ġmore Ġproblematic : ĠAmb ig uity Ġlevel Ġin Ġthe Ġtranslation Ġprocess Ġdefinitely Ġincreases Ġdue Ġto Ġthe Ġadditional Ġintroduction Ġof Ġwords Ġhaving Ġthe Ġsame Ġor Ġsimilar Ġmeaning Ġacross Ġlanguages Ġat Ġboth Ġsource Ġand Ġtarget Ġsides . ĠWe Ġdeal Ġwith Ġthe Ġproblem Ġby Ġexplicitly Ġforcing Ġthe Ġattention Ġand Ġtranslation Ġto Ġthe Ġdirection Ġthat Ġwe Ġprefer , Ġexpecting Ġthe Ġinformation Ġwould Ġlimit Ġthe Ġambiguity Ġto Ġthe Ġscope Ġof Ġone Ġlanguage Ġinstead Ġof Ġall Ġtarget Ġlanguages . ĠWe Ġrealize Ġthis Ġidea Ġby Ġadding Ġat Ġthe Ġbeginning Ġand Ġat Ġthe Ġend Ġof Ġevery Ġsource Ġsentences Ġa Ġspecial Ġsymbol Ġindicating Ġthe Ġlanguage Ġthey Ġwould Ġbe Ġtranslated Ġinto . ĠFor Ġexample , Ġin Ġa Ġmult ilingual ĠN MT , Ġwhen Ġa Ġsource Ġsentence Ġis ĠGerman Ġand Ġthe Ġtarget Ġlanguage Ġis ĠEnglish , Ġthe Ġoriginal Ġsentence Ġ( al ready Ġlanguage - specific Ġcoded ) Ġis : </s> @ de @ dar um Ġ@ de @ ge ht Ġ@ de @ es Ġ@ de @ in Ġ@ de @ me inem Ġ@ de @ V ort rag </s> Now Ġwhen Ġwe Ġforce Ġit Ġto Ġbe Ġtranslated Ġinto ĠEnglish , Ġthe Ġtarget - forced Ġsentence Ġbecomes : </s> < E > Ġ@ de @ dar um Ġ@ de @ ge ht Ġ@ de @ es Ġ@ de @ in Ġ@ de @ me inem Ġ@ de @ V ort rag Ġ< E > </s> Due Ġto Ġthe Ġnature Ġof Ġrecurrent Ġunits Ġused Ġin Ġthe Ġenc oder Ġand Ġdec oder , Ġin Ġtraining , Ġthose Ġstarting Ġsymbols Ġencourage Ġthe Ġnetwork Ġlearning Ġthe Ġtranslation Ġof Ġfollowing Ġtarget Ġwords Ġin Ġa Ġparticular Ġlanguage Ġpair . ĠIn Ġtesting Ġtime , Ġinformation Ġof Ġthe Ġtarget Ġlanguage Ġwe Ġprovided Ġhelp Ġto Ġlimit Ġthe Ġtranslated Ġcandidates , Ġhence Ġforming Ġthe Ġtranslation Ġin Ġthe Ġdesired Ġlanguage . </s> Figure ĠFIG REF 8 Ġillustrates Ġthe Ġessence Ġof Ġour Ġapproach . ĠWith Ġtwo Ġsteps Ġin Ġthe Ġpre processing Ġphase , Ġnamely Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing , Ġwe Ġare Ġable Ġto Ġemploy Ġmult ilingual Ġattention - based ĠN MT Ġwithout Ġany Ġspecial Ġtreatment Ġin Ġtraining Ġsuch Ġa Ġstandard Ġarchitecture . ĠOur Ġenc oder Ġand Ġattention - enable Ġdec oder Ġcan Ġbe Ġseen Ġas Ġa Ġshared Ġenc oder Ġand Ġdec oder Ġacross Ġlanguages , Ġor Ġan Ġuniversal Ġenc oder Ġand Ġdec oder . ĠThe Ġflex ib it ily Ġof Ġour Ġapproach Ġallow Ġus Ġto Ġintegrate Ġany Ġlanguage Ġinto Ġsource Ġor Ġtarget Ġside . ĠAs Ġwe Ġwill Ġsee Ġin ĠSection ĠSEC REF 4 Ġ, Ġit Ġhas Ġproven Ġto Ġbe Ġextremely Ġhelpful Ġnot Ġonly Ġin Ġlow - res ourced Ġscenarios Ġbut Ġalso Ġin Ġtranslation Ġof Ġwell - res ourced Ġlanguage Ġpairs Ġas Ġit Ġprovides Ġa Ġnovel Ġway Ġto Ġmake Ġuse Ġof Ġlarge Ġmon oling ual Ġcorpor a Ġin ĠN MT . </s> E val uation </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġevaluation Ġof Ġour Ġproposed Ġapproach Ġin Ġcomparisons Ġwith Ġthe Ġstrong Ġbas elines Ġusing ĠN MT Ġin Ġtwo Ġscenarios : Ġthe Ġtranslation Ġof Ġan Ġunder - resource Ġlanguage Ġpair Ġand Ġthe Ġtranslation Ġof Ġa Ġlanguage Ġpair Ġthat Ġdoes Ġnot Ġexist Ġany Ġpar alled Ġdata Ġat Ġall . </s> Exper imental ĠSettings </s> Training ĠData . ĠWe Ġchoose ĠW IT 3 's ĠTED Ġcorpus ĠB IB REF 12 Ġas Ġthe Ġbasis Ġof Ġour Ġexperiments Ġsince Ġit Ġmight Ġbe Ġthe Ġonly Ġhigh - quality Ġparallel Ġdata Ġof Ġmany Ġlow - res ourced Ġlanguage Ġpairs . ĠTED Ġis Ġalso Ġmult ilingual Ġin Ġa Ġsense Ġthat Ġit Ġincludes Ġnumbers Ġof Ġtalks Ġwhich Ġare Ġcommonly Ġtranslated Ġinto Ġmany Ġlanguages . ĠIn Ġaddition , Ġwe Ġuse Ġa Ġmuch Ġlarger Ġcorpus Ġprovided Ġfreely Ġby ĠW MT Ġorganizers Ġwhen Ġwe Ġevaluate Ġthe Ġimpact Ġof Ġour Ġapproach Ġin Ġa Ġreal Ġmachine Ġtranslation Ġcampaign . ĠIt Ġincludes Ġthe Ġpar alled Ġcorpus Ġextracted Ġfrom Ġthe Ġdigital Ġcorpus Ġof ĠEuropean ĠParliament Ġ( EP PS ), Ġthe ĠNews ĠCommentary Ġ( NC ) Ġand Ġthe Ġweb - c rawled Ġparallel Ġdata Ġ( Common C rawl ). ĠWhile Ġthe Ġnumber Ġof Ġsentences Ġin Ġpopular ĠTED Ġcorpor a Ġvaries Ġfrom Ġ13 Ġthousands Ġto Ġ17 Ġthousands , Ġthe Ġtotal Ġnumber Ġof Ġsentences Ġin Ġthose Ġlarger Ġcorpus Ġis Ġapproximately Ġ3 Ġmillion Ġsentences . </s> Ne ural ĠMachine ĠTranslation ĠSetup . ĠAll Ġexperiments Ġhave Ġbeen Ġconducted Ġusing ĠN MT Ġframework ĠNem atus , ĠFollowing Ġthe Ġwork Ġof ĠS enn rich 2016 a , Ġsub word Ġsegment ation Ġis Ġhandled Ġin Ġthe Ġprep oc essing Ġphase Ġusing ĠByte - P air ĠEnc oding Ġ( B PE ). ĠExcept s Ġstated Ġclearly Ġin Ġsome Ġexperiments , Ġwe Ġset Ġthe Ġnumber Ġof ĠB PE Ġmerging Ġoperations Ġat Ġ395 00 Ġon Ġthe Ġjoint Ġof Ġsource Ġand Ġtarget Ġdata . ĠWhen Ġtraining Ġall ĠN MT Ġsystems , Ġwe Ġtake Ġout Ġthe Ġsentence Ġpairs Ġexceeding Ġ50 - word Ġlength Ġand Ġshuffle Ġthem Ġinside Ġevery Ġmin ib atch . ĠOur Ġshort - list Ġvoc ab ular ies Ġcontain Ġ40 , 000 Ġmost Ġfrequent Ġwords Ġwhile Ġthe Ġothers Ġare Ġconsidered Ġas Ġrare Ġwords Ġand Ġapplied Ġthe Ġsub word Ġtranslation . ĠWe Ġuse Ġan Ġ1024 - cell ĠGR U Ġlayer Ġand Ġ1000 - dimensional Ġembed d ings Ġwith Ġdrop out Ġat Ġevery Ġlayer Ġwith Ġthe Ġprobability Ġof Ġ0 . 2 Ġin Ġthe Ġembed ding Ġand Ġhidden Ġlayers Ġand Ġ0 . 1 Ġin Ġthe Ġinput Ġand Ġour put Ġlayers . ĠWe Ġtrained Ġour Ġsystems Ġusing Ġgradient Ġdescent Ġoptimization Ġwith ĠAd ad elta ĠB IB REF 13 Ġon Ġmin ib atches Ġof Ġsize Ġ80 Ġand Ġthe Ġgradient Ġis Ġresc aled Ġwhenever Ġits Ġnorm Ġexceed Ġ1 . 0 . ĠAll Ġthe Ġtrain ings Ġlast Ġapproximately Ġseven Ġdays Ġif Ġthe Ġearly - st opping Ġcondition Ġcould Ġnot Ġbe Ġreached . ĠAt Ġa Ġcertain Ġtime , Ġan Ġexternal Ġevaluation Ġscript Ġon ĠB LE U ĠB IB REF 14 Ġis Ġconducted Ġon Ġa Ġdevelopment Ġset Ġto Ġdecide Ġthe Ġearly - st opping Ġcondition . ĠThis Ġevaluation Ġscript Ġhas Ġalso Ġbeing Ġused Ġto Ġchoose Ġthe Ġmodel Ġarch iving Ġthe Ġbest ĠB LE U Ġon Ġthe Ġdevelopment Ġset Ġinstead Ġof Ġthe Ġmaximal Ġlog like lihood Ġbetween Ġthe Ġtranslations Ġand Ġtarget Ġsentences Ġwhile Ġtraining . ĠIn Ġtranslation , Ġthe Ġframework Ġproduces ĠIN LINE FORM 0 Ġ- best Ġcandidates Ġand Ġwe Ġthen Ġuse Ġa Ġbeam Ġsearch Ġwith Ġthe Ġbeam Ġsize Ġof Ġ12 Ġto Ġget Ġthe Ġbest Ġtranslation . </s> Under - res ourced ĠTranslation </s> First , Ġwe Ġconsider Ġthe Ġtranslation Ġfor Ġan Ġunder - res ourced Ġpair Ġof Ġlanguages . ĠHere Ġa Ġsmall Ġportion Ġof Ġthe Ġlarge Ġparallel Ġcorpus Ġfor ĠEnglish - German Ġis Ġused Ġas Ġa Ġsimulation Ġfor Ġthe Ġscenario Ġwhere Ġwe Ġdo Ġnot Ġhave Ġmuch Ġparallel Ġdata : ĠTrans l ating Ġtexts Ġin ĠEnglish Ġto ĠGerman . ĠWe Ġperform Ġlanguage - specific Ġcoding Ġin Ġboth Ġsource Ġand Ġtarget Ġsides . ĠBy Ġaccommodating Ġthe ĠGerman Ġmon oling ual Ġdata Ġas Ġan Ġadditional Ġinput Ġ( German ĠIN LINE FORM 0 ĠGerman ), Ġwhich Ġwe Ġcalled Ġthe Ġmix - source Ġapproach , Ġwe Ġcould Ġenrich Ġthe Ġtraining Ġdata Ġin Ġa Ġsimple , Ġnatural Ġway . ĠGiven Ġthis Ġunder - res ourced Ġsituation , Ġit Ġcould Ġhelp Ġour ĠN MT Ġobtain Ġa Ġbetter Ġrepresentation Ġof Ġthe Ġsource Ġside , Ġhence , Ġable Ġto Ġlearn Ġthe Ġtranslation Ġrelationship Ġbetter . ĠIncluding Ġmon oling ual Ġdata Ġin Ġthis Ġway Ġmight Ġalso Ġimprove Ġthe Ġtranslation Ġof Ġsome Ġrare Ġword Ġtypes Ġsuch Ġas Ġnamed Ġentities . ĠFurthermore , Ġas Ġthe Ġultimate Ġgoal Ġof Ġour Ġwork , Ġwe Ġwould Ġlike Ġto Ġinvestigate Ġthe Ġadvantages Ġof Ġmult iling uality Ġin ĠN MT . ĠWe Ġincorporate Ġa Ġsimilar Ġportion Ġof ĠFrench - German Ġparallel Ġcorpus Ġinto Ġthe ĠEnglish - German Ġone . ĠAs Ġdiscussed Ġin ĠSection ĠSEC REF 5 Ġ, Ġit Ġis Ġexpected Ġto Ġhelp Ġreducing Ġthe Ġambiguity Ġin Ġtranslation Ġbetween Ġone Ġlanguage Ġpair Ġsince Ġit Ġutilizes Ġthe Ġsemantic Ġcontext Ġprovided Ġby Ġthe Ġother Ġsource Ġlanguage . ĠWe Ġname Ġthis Ġmix - multi - source . </s> Table ĠT AB REF 16 Ġsummarizes Ġthe Ġperformance Ġof Ġour Ġsystems Ġmeasured Ġin ĠB LE U Ġon Ġtwo Ġtest Ġsets , Ġt st 2013 Ġand Ġt st 2014 . ĠCompared Ġto Ġthe Ġbaseline ĠN MT Ġsystem Ġwhich Ġis Ġsolely Ġtrained Ġon ĠTED ĠEnglish - German Ġdata , Ġour Ġmix - source Ġsystem Ġachieves Ġa Ġconsiderable Ġimprovement Ġof Ġ2 . 6 ĠB LE U Ġpoints Ġon Ġt st 2013 Ġand Ġ2 . 1 ĠB LE U Ġpoints Ġon Ġand Ġt st 2014 Ġ. ĠAdding ĠFrench Ġdata Ġto Ġthe Ġsource Ġside Ġand Ġtheir Ġcorresponding ĠGerman Ġdata Ġto Ġthe Ġtarget Ġside Ġin Ġour Ġmix - multi - source Ġsystem Ġalso Ġhelp Ġto Ġgain Ġ2 . 2 Ġand Ġ1 . 6 ĠB LE U Ġpoints Ġmore Ġon Ġt st 2013 Ġt st 2014 , Ġrespectively . ĠWe Ġobserve Ġa Ġbetter Ġimprovement Ġfrom Ġour Ġmix - source Ġsystem Ġcompared Ġto Ġour Ġmix - multi - source Ġsystem . ĠWe Ġspeculate Ġthe Ġreason Ġthat Ġthe Ġmix - source Ġenc oder Ġutilize Ġthe Ġsame Ġinformation Ġshared Ġin Ġtwo Ġlanguages Ġwhile Ġthe Ġmix - multi - source Ġreceives Ġand Ġprocesses Ġsimilar Ġinformation Ġin Ġthe Ġother Ġlanguage Ġbut Ġnot Ġnecessarily Ġthe Ġsame . ĠWe Ġmight Ġvalidate Ġthis Ġhypothesis Ġby Ġcomparing Ġtwo Ġsystems Ġtrained Ġon Ġa Ġcommon ĠEnglish - German - French Ġcorpus Ġof ĠTED . ĠWe Ġput Ġit Ġin Ġour Ġfuture Ġwork 's Ġplan . </s> As Ġwe Ġexpected ĠFigure ĠFIG REF 19 Ġshows Ġhow Ġdifferent Ġwords Ġin Ġdifferent Ġlanguages Ġcan Ġbe Ġclose Ġin Ġthe Ġshared Ġspace Ġafter Ġbeing Ġlearned Ġto Ġtranslate Ġinto Ġa Ġcommon Ġlanguage . ĠWe Ġextract Ġthe Ġword Ġembed d ings Ġfrom Ġthe Ġenc oder Ġof Ġthe Ġmix - multi - source Ġ( En , Fr ĠIN LINE FORM 0 ĠDe , De ) Ġafter Ġtraining , Ġremove Ġthe Ġlanguage - specific Ġcodes Ġ(@ en @ Ġand Ġ@ fr @ ) and Ġproject Ġthe Ġword Ġvectors Ġto Ġthe Ġ2 D Ġspace Ġusing Ġt - S NE ĠB IB REF 15 Ġ. </s> Using Ġlarge Ġmon oling ual Ġdata Ġin ĠN MT . </s> A Ġstandard ĠN MT Ġsystem Ġemploys Ġparallel Ġdata Ġonly . ĠWhile Ġgood Ġparallel Ġcorpor a Ġare Ġlimited Ġin Ġnumber , Ġgetting Ġmon oling ual Ġdata Ġof Ġan Ġarbitrary Ġlanguage Ġis Ġtrivial . ĠTo Ġmake Ġuse Ġof ĠGerman Ġmon oling ual Ġcorpus Ġin Ġan ĠEnglish ĠIN LINE FORM 0 ĠGerman ĠN MT Ġsystem , Ġs enn rich 2016 b Ġbuilt Ġa Ġseparate ĠGerman ĠIN LINE FORM 1 ĠEnglish ĠN MT Ġusing Ġthe Ġsame Ġparallel Ġcorpus , Ġthen Ġthey Ġused Ġthat Ġsystem Ġto Ġtranslate Ġthe ĠGerman Ġmon oling ual Ġcorpus Ġback Ġto ĠEnglish , Ġforming Ġa Ġsynthesis Ġparallel Ġdata . Ġgul ce h re 2015 Ġtrained Ġanother ĠR NN - based Ġlanguage Ġmodel Ġto Ġscore Ġthe Ġmon oling ual Ġcorpus Ġand Ġintegrate Ġit Ġto Ġthe ĠN MT Ġsystem Ġthrough Ġshallow Ġor Ġdeep Ġfusion . ĠBoth Ġmethods Ġrequires Ġto Ġtrain Ġseparate Ġsystems Ġwith Ġpossibly Ġdifferent Ġhyper param eters Ġfor Ġeach . ĠConversely , Ġby Ġapplying Ġmix - source Ġmethod Ġto Ġthe Ġbig Ġmon oling ual Ġdata , Ġwe Ġneed Ġto Ġtrain Ġonly Ġone Ġnetwork . ĠWe Ġmix Ġthe ĠTED Ġparallel Ġcorpus Ġand Ġthe Ġsubstantial Ġmon oling ual Ġcorpus Ġ( EP PS + NC + Common C rawl ) Ġand Ġtrain Ġa Ġmix - source ĠN MT Ġsystem Ġfrom Ġthose Ġdata . </s> The Ġfirst Ġresult Ġis Ġnot Ġencouraging Ġwhen Ġits Ġperformance Ġis Ġeven Ġworse Ġthan Ġthe Ġbaseline ĠN MT Ġwhich Ġis Ġtrained Ġon Ġthe Ġsmall Ġparallel Ġdata Ġonly . ĠNot Ġusing Ġthe Ġsame Ġinformation Ġin Ġthe Ġsource Ġside , Ġas Ġwe Ġdiscussed Ġin Ġcase Ġof Ġmix - multi - source Ġstrategy , Ġcould Ġexplain Ġthe Ġdegrading Ġin Ġperformance Ġof Ġsuch Ġa Ġsystem . ĠBut Ġwe Ġbelieve Ġthat Ġthe Ġmagnitude Ġand Ġun bal ancing Ġof Ġthe Ġcorpus Ġare Ġthe Ġmain Ġreasons . ĠThe Ġdata Ġcontains Ġnearly Ġfour Ġmillions Ġsentences Ġbut Ġonly Ġaround Ġtwenty Ġthousands Ġof Ġthem Ġ( 0 . 5 %) Ġare Ġthe Ġgenuine Ġparallel Ġdata . ĠAs Ġa Ġquick Ġattempt , Ġafter Ġwe Ġget Ġthe Ġmodel Ġwith Ġthat Ġbig Ġdata , Ġwe Ġcontinue Ġtraining Ġon Ġthe Ġreal Ġparallel Ġcorpus Ġfor Ġsome Ġmore Ġepoch s . ĠWhen Ġthis Ġadaptation Ġis Ġapplied , Ġour Ġsystem Ġbrings Ġan Ġimprovement Ġof Ġ+ 1 . 52 ĠB LE U Ġon Ġt st 2013 Ġand Ġ+ 1 . 06 ĠB LE U Ġon Ġt st 2014 Ġ( Table ĠT AB REF 21 Ġ). </s> Zero - res ourced ĠTranslation </s> Among Ġlow - res ourced Ġscenarios , Ġzero - res ourced Ġtranslation Ġtask Ġstands Ġin Ġan Ġextreme Ġlevel . ĠA Ġzero - res ourced Ġtranslation Ġtask Ġis Ġone Ġof Ġthe Ġmost Ġdifficult Ġsituation Ġwhen Ġthere Ġis Ġno Ġparallel Ġdata Ġbetween Ġthe Ġtranslating Ġlanguage Ġpair . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthere Ġhave Ġbeen Ġyet Ġexisted Ġa Ġpublished Ġwork Ġabout Ġusing ĠN MT Ġfor Ġzero - res ourced Ġtranslation Ġtasks Ġup Ġto Ġnow . ĠIn Ġthis Ġsection , Ġwe Ġextend Ġour Ġstrategies Ġusing Ġthe Ġproposed Ġmult ilingual ĠN MT Ġapproach Ġas Ġfirst Ġattempts Ġto Ġthis Ġextreme Ġsituation . </s> We Ġemploy Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing Ġin Ġa Ġstrategy Ġcalled Ġbridge . ĠUnlike Ġthe Ġstrategies Ġused Ġin Ġunder - res ourced Ġtranslation Ġtask , Ġbridge Ġis Ġan Ġentire Ġmany - to - many Ġmult ilingual ĠN MT . ĠSim ulating Ġa Ġzero - res ourced ĠGerman ĠIN LINE FORM 0 ĠFrench Ġtranslation Ġtask Ġgiven Ġthe Ġavailable ĠGerman - English Ġand ĠEnglish - French Ġparallel Ġcorpor a , Ġafter Ġapplying Ġlanguage - specific Ġcoding Ġand Ġtarget Ġforcing Ġfor Ġeach Ġcorpus , Ġwe Ġmix Ġthose Ġdata Ġwith Ġan ĠEnglish - English Ġdata Ġas Ġa ĠâĢ ľ bridge âĢ Ŀ Ġcreating Ġsome Ġconnection Ġbetween ĠGerman Ġand ĠFrench . ĠWe Ġalso Ġpropose Ġa Ġvariant Ġof Ġthis Ġstrategy Ġthat Ġwe Ġincorporate ĠFrench - French Ġdata . ĠAnd Ġwe Ġcall Ġit Ġuniversal . </s> We Ġevaluate Ġbridge Ġand Ġuniversal Ġsystems Ġon Ġtwo ĠGerman ĠIN LINE FORM 0 ĠFrench Ġtest Ġsets . ĠThey Ġare Ġcompared Ġto Ġa Ġdirect Ġsystem , Ġwhich Ġis Ġan ĠN MT Ġtrained Ġon ĠGerman ĠIN LINE FORM 1 ĠFrench Ġdata , Ġand Ġto Ġa Ġpivot Ġsystem , Ġwhich Ġessentially Ġconsists Ġof Ġtwo Ġseparate ĠN MT s Ġtrained Ġto Ġtranslate Ġfrom ĠGerman Ġto ĠEnglish Ġand ĠEnglish Ġto ĠFrench . ĠThe Ġdirect Ġsystem Ġshould Ġnot Ġexist Ġin Ġa Ġreal Ġzero - res ourced Ġsituation . ĠWe Ġrefer Ġit Ġas Ġthe Ġperfect Ġsystem Ġfor Ġcomparison Ġpurpose Ġonly . ĠIn Ġcase Ġof Ġthe Ġpivot Ġsystem , Ġto Ġgenerate Ġa Ġtranslated Ġtext Ġin ĠFrench Ġfrom Ġa ĠGerman Ġsentence , Ġwe Ġfirst Ġtranslate Ġit Ġto ĠEnglish , Ġthen Ġthe Ġoutput Ġsentence Ġis Ġfed Ġto Ġthe ĠEnglish ĠIN LINE FORM 2 ĠGerman ĠN MT Ġsystem Ġto Ġobtain Ġthe ĠFrench Ġtranslation . ĠSince Ġthere Ġare Ġmore Ġthan Ġtwo Ġlanguages Ġinvolved Ġin Ġthose Ġsystems , Ġwe Ġincrease Ġthe Ġnumber Ġof ĠB PE Ġmerging Ġoperations Ġproportion ally Ġin Ġorder Ġto Ġreduce Ġthe Ġnumber Ġof Ġrare Ġwords Ġin Ġsuch Ġsystems . ĠWe Ġdo Ġnot Ġexpect Ġour Ġproposed Ġsystems Ġto Ġperform Ġwell Ġwith Ġthis Ġprimitive Ġway Ġof Ġbuilding Ġdirect Ġtranslating Ġconnections Ġsince Ġthis Ġis Ġessentially Ġa Ġdifficult Ġtask . ĠWe Ġreport Ġthe Ġperformance Ġof Ġthose Ġsystems Ġin ĠTable ĠT AB REF 23 Ġ. </s> Un supp rising ly , Ġboth Ġbridge Ġand Ġuniversal Ġsystems Ġperform Ġworse Ġthan Ġthe Ġpivot Ġone . ĠWe Ġconsider Ġtwo Ġpossible Ġreasons : </s> Our Ġtarget Ġforcing Ġmechanism Ġis Ġmoderately Ġprimitive . ĠSince Ġthe Ġprocess Ġis Ġapplied Ġafter Ġlanguage - specific Ġcoding , Ġthe Ġtarget Ġforcing Ġsymbol Ġis Ġthe Ġsame Ġfor Ġall Ġsource Ġsentences Ġin Ġevery Ġlanguages . ĠThus , Ġthe Ġforcing Ġstrength Ġmight Ġnot Ġbe Ġenough Ġto Ġguide Ġthe Ġdecision Ġof Ġthe Ġnext Ġwords . ĠOnce Ġthe Ġvery Ġfirst Ġword Ġis Ġtranslated Ġinto Ġa Ġword Ġin Ġwrong Ġlanguage , Ġthe Ġfollowing Ġwords Ġtend Ġto Ġbe Ġtranslated Ġinto Ġthat Ġwrong Ġlanguage Ġagain . ĠTable ĠT AB REF 24 Ġshows Ġsome Ġstatistics Ġof Ġthe Ġtranslated Ġwords Ġand Ġsentences Ġin Ġwrong Ġlanguage . </s> Bal ancing Ġof Ġthe Ġtraining Ġcorpus . ĠAlthough Ġit Ġis Ġnot Ġsevere Ġas Ġin Ġthe Ġcase Ġof Ġmix - source Ġsystem Ġfor Ġlarge Ġmon oling ual Ġdata , Ġthe Ġlimited Ġnumber Ġof Ġsentences Ġin Ġtarget Ġlanguage Ġcan Ġaffect Ġthe Ġtraining . ĠThe Ġdifference Ġof Ġ1 . 07 ĠB LE U Ġpoints Ġbetween Ġbridge Ġand Ġuniversal Ġmight Ġexplain Ġthis Ġassumption Ġas Ġwe Ġadded Ġmore Ġtarget Ġdata Ġ( French ) Ġin Ġuniversal Ġstrategy , Ġthus Ġreducing Ġthe Ġun balance Ġin Ġtraining . </s> Those Ġissues Ġwould Ġbe Ġaddressed Ġin Ġour Ġfollowing Ġfuture Ġwork Ġtoward Ġthe Ġmult ilingual Ġattention - based ĠN MT . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġfirst Ġattempts Ġin Ġbuilding Ġa Ġmult ilingual ĠNeural ĠMachine ĠTranslation Ġframework . ĠBy Ġtreating Ġwords Ġin Ġdifferent Ġlanguages Ġas Ġdifferent Ġwords Ġand Ġforce Ġthe Ġattention Ġand Ġtranslation Ġto Ġthe Ġdirection Ġof Ġdesired Ġtarget Ġlanguage , Ġwe Ġare Ġable Ġto Ġemploy Ġattention - enable ĠN MT Ġtoward Ġa Ġmult ilingual Ġtranslation Ġsystem . ĠOur Ġproposed Ġapproach Ġallev iates Ġthe Ġneed Ġof Ġcomplicated Ġarchitecture Ġre - design ing Ġwhen Ġaccommodating Ġattention Ġmechanism . ĠIn Ġaddition , Ġthe Ġnumber Ġof Ġfree Ġparameters Ġto Ġlearn Ġin Ġour Ġnetwork Ġdoes Ġnot Ġgo Ġbeyond Ġthat Ġmagn itute Ġof Ġa Ġsingle ĠN MT Ġsystem . ĠWith Ġits Ġunivers ality , Ġour Ġapproach Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġan Ġunder - res ourced Ġtranslation Ġtask Ġwith Ġconsiderable Ġimprovements . ĠIn Ġaddition , Ġthe Ġapproach Ġhas Ġachieved Ġinteresting Ġand Ġpromising Ġresults Ġwhen Ġapplied Ġin Ġthe Ġtranslation Ġtask Ġthat Ġthere Ġis Ġno Ġdirect Ġparallel Ġcorpus Ġbetween Ġsource Ġand Ġtarget Ġlanguages . </s> Nevertheless , Ġthere Ġare Ġissues Ġthat Ġwe Ġcan Ġcontinue Ġworking Ġon Ġto Ġaddress Ġin Ġfuture Ġwork . ĠA Ġmore Ġbalancing Ġdata Ġwould Ġbe Ġhelpful Ġfor Ġthis Ġframework . ĠThe Ġmechanism Ġof Ġforcing Ġthe ĠN MT Ġsystem Ġto Ġthe Ġright Ġtarget Ġlanguage Ġcould Ġbe Ġimproved . ĠWe Ġcould Ġconduct Ġmore Ġdetailed Ġanalyses Ġof Ġthe Ġvarious Ġstrategies Ġunder Ġthe Ġframework Ġto Ġshow Ġits Ġunivers arity .