No
BOOLEAN
We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3's corpus BIBREF15 . After removing blank and duplicate lines we obtained 106758 pairs of sentences. The validation set used in all experiments is dev2010 and the test set is tst2010.
1805.07133
Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġis Ġwidely Ġapplied Ġfor Ġmachine Ġtranslation Ġ( MT ) Ġin Ġrecent Ġyears Ġand Ġfocuses Ġon Ġpopular Ġlanguage Ġpairs Ġsuch Ġas ĠEnglish ĠIN LINE FORM 0 ĠFrench , ĠEnglish ĠIN LINE FORM 1 ĠGerman , ĠEnglish ĠIN LINE FORM 2 ĠChinese Ġor ĠEnglish ĠIN LINE FORM 3 ĠJapanese . ĠN MT Ġhas Ġobtained Ġstate - of - the - art Ġperformance Ġon Ġthose Ġlanguage Ġpairs Ġcompared Ġto Ġthe Ġtraditional Ġstatistical Ġmachine Ġtranslation Ġ( SM T ) Ġwhen Ġgiven Ġenough Ġdata ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFurthermore , Ġdue Ġto Ġthe Ġability Ġof Ġfeature Ġlearning , ĠN MT Ġsystems Ġcan Ġbe Ġtrained Ġend - to - end Ġwith Ġpure Ġparallel Ġtexts Ġand Ġminimal Ġlinguistic Ġknowledge Ġof Ġthe Ġlanguages Ġinvolved . ĠThus Ġit Ġmakes Ġtraining ĠN MT Ġfor Ġa Ġnew Ġlanguage Ġpair Ġmuch Ġeasier , Ġmore Ġscalable Ġand Ġrobust . ĠNevertheless , ĠN MT Ġhas Ġnot Ġbeen Ġemployed Ġin Ġmany Ġlow - res ourced Ġlanguage Ġpairs Ġsince Ġin Ġthose Ġscenarios , Ġdata Ġscarcity Ġoften Ġlimits Ġthe Ġlearning Ġability Ġof Ġneural Ġmethods . ĠIn Ġcontrast , Ġcomb inating Ġcomplicated Ġlinguistic - driven Ġfeatures Ġin Ġa Ġtypical Ġlog - linear Ġframework Ġstill Ġkeeps ĠSM T Ġthe Ġbest Ġapproach Ġin Ġmany Ġtranslation Ġdirections Ġbut Ġalso Ġhard Ġto Ġapply Ġto Ġnew Ġdomains Ġor Ġto Ġother Ġlanguage Ġpairs . </s> In Ġthis Ġpaper Ġwe Ġattempt Ġto Ġbuild ĠN MT Ġsystems Ġfor Ġsuch Ġa Ġlow - res ourced Ġlanguage Ġpair : ĠJapanese ĠIN LINE FORM 0 ĠVietnamese . ĠOur Ġaim Ġis Ġto Ġset Ġthe Ġfirst Ġand Ġreasonable ĠN MT Ġsystems Ġthat Ġcan Ġbe Ġreprodu cible Ġin Ġorder Ġto Ġserve Ġas Ġthe Ġbaseline Ġfor Ġfurther Ġresear ches Ġin Ġthe Ġdirection . ĠFurthermore , Ġwe Ġconduct Ġexperiments Ġusing Ġsome Ġadvanced Ġmethods Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġsystems . ĠAn Ġimportant Ġcriteria Ġfor Ġthose Ġmethods Ġis Ġthat Ġthey Ġmust Ġbe Ġscalable Ġand Ġlanguage - independent Ġas Ġmuch Ġas Ġpossible . ĠThe Ġcriteria Ġensures Ġthe Ġbasic Ġprinciple Ġof ĠN MT Ġas Ġwell Ġas Ġthe Ġreprodu c ibility Ġof Ġthe Ġsystems . ĠOn Ġthe Ġother Ġhand , Ġthe Ġmethods Ġare Ġchosen Ġin Ġthe Ġdirection Ġthat Ġthey Ġwould Ġhelp Ġalleviate Ġthe Ġdata Ġsp arsity Ġproblem Ġof ĠN MT Ġwhen Ġbeing Ġapplied Ġin Ġthis Ġlow - res ourced Ġsetting . </s> Specifically , Ġto Ġdeal Ġwith Ġrare - word Ġtranslation Ġproblems , Ġwe Ġexperiment Ġwith Ġtranslation Ġunits Ġin Ġdifferent Ġlevels : Ġsub word , Ġword Ġand Ġbeyond . ĠIn Ġmorph ological - rich Ġlanguages Ġsuch Ġas ĠEnglish Ġor ĠGerman , Ġusing Ġsub word Ġas Ġthe Ġtranslation Ġunits Ġis Ġoften Ġsuitable Ġsince Ġneural Ġmethods Ġare Ġable Ġto Ġinduce Ġmeaning Ġor Ġlinguistic Ġfunction Ġof Ġsub - comp onents Ġconstit uting Ġa Ġword . ĠByte - P air ĠEnc oding Ġ( B PE ) ĠB IB REF 4 Ġis Ġa Ġsimple Ġun super vised Ġtechnique Ġto Ġdo Ġsub word Ġsegment ation Ġand Ġit Ġhas Ġgreat Ġeffects Ġwhen Ġapplied Ġto ĠN MT Ġtraining . ĠJapanese Ġand ĠVietnamese Ġ( and Ġsome Ġother ĠAsian Ġlanguages ), Ġhowever , Ġhave Ġdifferent Ġword Ġsegment ation Ġissues . ĠHence , Ġit Ġwould Ġbe Ġdifficult Ġto Ġapply ĠB PE Ġdirectly Ġto Ġthe Ġtexts Ġin Ġthose Ġlanguages Ġand Ġbuild ĠN MT Ġsystems Ġfor Ġsub word Ġtranslation Ġwithout Ġany Ġmodification . ĠIn Ġthis Ġpaper , Ġwe Ġexperiment Ġdifferent Ġsegment ation Ġmethods Ġfor Ġboth Ġlanguages Ġand Ġalso Ġpropose Ġa Ġvariant Ġof ĠB PE Ġalgorithm Ġto Ġlearn Ġtranslation Ġunits Ġfor ĠVietnamese Ġin Ġan Ġun super vised Ġway . </s> We Ġalso Ġattempt Ġto Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata Ġby Ġusing Ġback - trans lated Ġtexts Ġor Ġmix - source Ġdata Ġjust Ġfrom Ġour Ġsmall Ġavailable Ġcorpus . ĠThose Ġdata Ġaug mentation Ġapproaches Ġhave Ġshown Ġtheir Ġeffectiveness Ġon Ġvarious ĠN MT Ġsystems , Ġespecially Ġin Ġunder - res ourced Ġscenarios . ĠWhile Ġback Ġtranslation Ġtechnique Ġis Ġused Ġto Ġgenerate Ġsynthetic Ġdata Ġfrom Ġmon oling ual Ġcorpor a , Ġmix - source Ġtechnique Ġutilizes Ġhuman - quality Ġcorpor a Ġin Ġa Ġmult ilingual Ġsetting , Ġleveraging Ġthe Ġtransfer Ġlearning Ġability Ġacross Ġlanguages . ĠBoth Ġare Ġsimple Ġbut Ġeleg antly Ġmodel Ġthe Ġrelevant Ġnoise Ġneeded Ġin Ġtraining Ġneural Ġarchitectures Ġin Ġsuch Ġlow - res ourced Ġsituations . </s> The Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> Ne ural ĠMachine ĠTranslation </s> In Ġthis Ġsection , Ġwe Ġwill Ġdescribe Ġthe Ġgeneral Ġarchitecture Ġof ĠN MT Ġas Ġa Ġkind Ġof Ġsequence - to - sequence Ġmodeling Ġframework . ĠIn Ġthis Ġkind Ġof Ġsequence - to - sequence Ġmodeling Ġframework , Ġoften Ġthere Ġis Ġan Ġenc oder Ġtrying Ġto Ġencode Ġcontext Ġinformation Ġfrom Ġthe Ġinput Ġsequence Ġand Ġa Ġdec oder Ġto Ġgenerate Ġone Ġitem Ġof Ġthe Ġoutput Ġsequence Ġat Ġa Ġtime Ġbased Ġon Ġthe Ġcontext Ġof Ġboth Ġinput Ġand Ġoutput Ġsequences . ĠBesides , Ġan Ġadditional Ġcomponent , Ġnamed Ġattention , Ġexists Ġin Ġbetween , Ġdeciding Ġwhich Ġparts Ġof Ġthe Ġinput Ġsequence Ġthe Ġdec oder Ġshould Ġpay Ġattention Ġin Ġorder Ġto Ġchoose Ġwhich Ġto Ġoutput Ġnext . ĠIn Ġother Ġwords , Ġthis Ġattention Ġcomponent Ġcalculates Ġthe Ġcontext Ġrelevant Ġto Ġthe Ġdecision Ġof Ġthe Ġdec oder Ġat Ġthe Ġconsidering Ġtime . ĠThose Ġcomponents Ġas Ġa Ġwhole Ġconstitute Ġa Ġlarge Ġtrain able Ġneural Ġarchitecture Ġcalled Ġthe Ġfamous Ġattention - based Ġenc oder - dec oder Ġframework . ĠThis Ġframework Ġbecomes Ġpopular Ġin Ġmany Ġsequence - to - sequence Ġtasks . </s> In Ġthe Ġfield Ġof Ġmachine Ġtranslation , Ġusing Ġthe Ġattention - based Ġenc oder - dec oder Ġframework Ġis Ġreferred Ġto Ġas ĠNeural ĠMachine ĠTranslation Ġapproach . ĠFirst Ġpresented Ġin ĠB IB REF 0 Ġ, Ġthe Ġenc oder Ġand Ġdec oder Ġin ĠN MT Ġare Ġrecurrent - based , Ġwhich Ġeach Ġhidden Ġunit Ġin Ġthose Ġcomponents Ġis Ġa Ġrecurrent Ġunit Ġlike ĠLong ĠShort - term ĠMemory Ġ( L ST M ) ĠB IB REF 5 Ġor ĠG ated ĠRec urrent ĠUnit ( GR U ) ĠB IB REF 6 Ġ. ĠLater , Ġthe Ġenc oder Ġor Ġdec oder Ġcan Ġalso Ġbe Ġa Ġconv olution al Ġarchitecture , Ġas Ġin ĠB IB REF 7 Ġ. ĠRecently , ĠB IB REF 8 Ġintroduces Ġtransformer Ġarchitecture , Ġin Ġwhich Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġa Ġspecial Ġvariant Ġof Ġattention Ġmechanism , Ġcalled Ġself Ġattention . ĠIn Ġthis Ġpaper , Ġwe Ġwill Ġbriefly Ġexplain Ġthe ĠRec urrent ĠN MT Ġas Ġwe Ġutilize Ġit Ġin Ġour Ġexperiments . </s> The ĠRec urrent ĠN MT Ġmodel Ġfollows Ġthe Ġattention - based Ġarchitecture Ġproposed Ġby ĠB IB REF 0 Ġ. ĠThe Ġbid irection al Ġrecurrent Ġenc oder Ġreads Ġevery Ġwords ĠIN LINE FORM 0 Ġof Ġa Ġsource Ġsentence ĠIN LINE FORM 1 Ġand Ġenc odes Ġa Ġrepresentation ĠIN LINE FORM 2 Ġof Ġthe Ġsentence Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġconc aten ated Ġfrom Ġthose Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : ĠIN LINE FORM 4 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġone - hot Ġvector Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġword Ġembed ding Ġmatrix Ġwhich Ġis Ġshared Ġacross Ġthe Ġsource Ġwords . ĠIN LINE FORM 3 Ġis Ġthe Ġrecurrent Ġunit Ġcomputing Ġthe Ġcurrent Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġbased Ġon Ġthe Ġprevious Ġhidden Ġstate . ĠIN LINE FORM 4 Ġis Ġthen Ġcalled Ġan Ġannotation Ġvector , Ġwhich Ġrepresent Ġthe Ġinformation Ġof Ġthe Ġsource Ġsentence Ġup Ġto Ġthe Ġtime ĠIN LINE FORM 5 Ġfrom Ġboth Ġforward Ġand Ġbackward Ġdirections . </s> Those Ġannotation Ġvectors Ġof Ġthe Ġsource Ġsentences Ġare Ġcombined Ġin Ġthe Ġattention Ġlayers Ġin Ġa Ġway Ġthat Ġthe Ġresulted Ġvector Ġenc odes Ġthe Ġsource Ġcontext Ġrelevant Ġto Ġthe Ġconsidering Ġtarget Ġword Ġthe Ġdec oder Ġshould Ġproduce . ĠInt uitive ly , Ġa Ġrelevance Ġbetween Ġthe Ġprevious Ġtarget Ġword ĠIN LINE FORM 0 Ġand Ġthe Ġannotation Ġvectors ĠIN LINE FORM 1 Ġcorresponding Ġto Ġthe Ġsource Ġwords ĠIN LINE FORM 2 Ġcan Ġbe Ġused Ġto Ġform Ġsome Ġattention Ġscenario : ĠDIS PLAY FORM 0 </s> This Ġspecific Ġattention Ġmechanism , Ġoriginally Ġcalled Ġalignment Ġmodel Ġin ĠB IB REF 0 Ġ, Ġhas Ġbeen Ġemployed Ġas Ġa Ġsimple Ġfeed forward Ġnetwork Ġwith Ġthe Ġfirst Ġlayer Ġis Ġa Ġlearn able Ġlayer Ġvia Ġadaptation Ġfactors ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġrelevance Ġscores ĠIN LINE FORM 3 Ġare Ġthen Ġnormalized Ġinto Ġattention Ġweights ĠIN LINE FORM 4 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 5 Ġis Ġcalculated Ġas Ġthe Ġweighted Ġsum Ġof Ġall Ġannotation Ġvectors ĠIN LINE FORM 6 Ġ. ĠDepending Ġon Ġhow Ġmuch Ġattention Ġthe Ġtarget Ġword Ġat Ġtime ĠIN LINE FORM 7 Ġput Ġon Ġthe Ġsource Ġstates ĠIN LINE FORM 8 Ġ, Ġa Ġsoft Ġalignment Ġis Ġlearned Ġand Ġa Ġsource Ġcontext ĠIN LINE FORM 9 Ġat Ġtime ĠIN LINE FORM 10 Ġis Ġcalculated Ġprior Ġto Ġthe Ġprediction Ġof Ġthe Ġdec oder . </s> Similar Ġto Ġthe Ġenc oder , Ġthe Ġrecurrent Ġdec oder Ġrec urs ively Ġgenerates Ġone Ġtarget Ġword ĠIN LINE FORM 0 Ġto Ġform Ġa Ġtranslated Ġtarget Ġsentence ĠIN LINE FORM 1 Ġin Ġthe Ġend . ĠAt Ġtime ĠIN LINE FORM 2 Ġ, Ġit Ġtakes Ġthe Ġprevious Ġhidden Ġstate Ġof Ġthe Ġdec oder ĠIN LINE FORM 3 Ġ, Ġthe Ġprevious Ġembedded Ġword Ġrepresentation ĠIN LINE FORM 4 Ġand Ġthe Ġtime - specific Ġcontext Ġvector ĠIN LINE FORM 5 Ġas Ġinputs Ġto Ġcalculate Ġthe Ġcurrent Ġhidden Ġstate ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 </s> Again , ĠIN LINE FORM 0 Ġis Ġthe Ġrecurrent Ġactivation Ġfunction Ġof Ġthe Ġdec oder Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġshared Ġword Ġembed ding Ġmatrix Ġof Ġthe Ġtarget Ġsentences . </s> Given Ġthe Ġparallel Ġcorpus Ġconsisting Ġof ĠIN LINE FORM 0 Ġtraining Ġexamples ĠIN LINE FORM 1 Ġ, Ġthe Ġobjective Ġis Ġto Ġmaximize Ġthe Ġconditional Ġlog - pro b ability Ġof Ġthe Ġcorrect Ġtranslation Ġgiven Ġa Ġsource Ġsentence Ġwith Ġrespect Ġto Ġthe Ġparameters ĠIN LINE FORM 2 Ġof Ġthe Ġwhole Ġmodel : ĠIN LINE FORM 3 </s> Sub word ĠTranslation </s> One Ġof Ġthe Ġmost Ġsevere Ġproblems Ġof ĠN MT Ġis Ġdealing Ġwith Ġthe Ġrare Ġwords , Ġwhich Ġare Ġnot Ġin Ġthe Ġshort Ġlists Ġof Ġthe Ġvoc ab ular ies , Ġi . e . Ġout - of - voc abulary Ġ( OO V ) Ġwords , Ġor Ġdo Ġnot Ġappear Ġin Ġthe Ġtraining Ġset Ġat Ġall . ĠOn Ġone Ġhand , Ġwe Ġwould Ġlike Ġto Ġhave Ġfewer ĠO OV Ġwords Ġby Ġincreasing Ġthe Ġsize Ġof Ġthe Ġshort Ġlists . ĠOn Ġthe Ġother Ġhand , Ġwe Ġneed Ġour Ġneural Ġnetwork Ġto Ġlearn Ġfast Ġand Ġhas Ġa Ġgood Ġgeneral ization Ġcapability Ġon Ġthe Ġunseen Ġwords Ġas Ġwell . </s> As Ġexplained Ġshortly Ġin Ġthe Ġintroduction , Ġfor Ġmany Ġlanguages , Ġusing Ġsub word Ġinstead Ġof Ġword Ġas Ġa Ġtranslation Ġunit Ġ( T U ) Ġhas Ġbeen Ġshown Ġthat Ġit Ġis Ġnot Ġonly Ġeffective Ġon Ġreducing Ġvocabulary Ġsizes , Ġthus Ġallev iating Ġthe Ġcomputing Ġburden Ġon Ġthe Ġlarge Ġsoft - max Ġlayer Ġas Ġwell Ġas Ġreducing Ġa Ġsubstantial Ġnumber Ġof Ġparameters Ġto Ġbe Ġlearnt , Ġbut Ġalso Ġhas Ġthe Ġability Ġto Ġgenerate Ġunseen Ġwords . ĠIn Ġthose Ġlanguages , Ġa Ġword Ġcan Ġbe Ġa Ġcompound Ġword Ġor Ġcomprised Ġby Ġsub - comp onents , Ġeach Ġhas Ġits Ġown Ġraw Ġmeaning Ġor Ġcontains Ġmorph ological Ġinformation . ĠSe gment ing Ġwords Ġinto Ġsub - comp onents Ġallows ĠN MT Ġto Ġlearn Ġto Ġtranslate Ġthem Ġwith Ġconsiderably Ġfewer Ġdata . ĠFor Ġexample , Ġit Ġis Ġdefinitely Ġless Ġchance Ġto Ġsee Ġthis Ġpopular ĠGerman Ġword ĠâĢ ľ W ohn ung s re in ig ung " Ġ( English Ġequival ence : ĠâĢ ľ house Ġcleaning âĢ Ŀ ) Ġthan Ġits Ġsub - comp onents ĠâĢ ľ W ohn ung âĢ Ŀ Ġ( i . e . ĠâĢ ľ house âĢ Ŀ Ġor ĠâĢ ľ flat âĢ Ŀ ) Ġand ĠâĢ ľ re in ig ung âĢ Ŀ Ġ( i . e . ĠâĢ ľ cle aning âĢ Ŀ ) Ġin Ġa Ġmiddle - sized ĠGerman - English Ġparallel Ġcorpus . ĠInstead , ĠN MT Ġcan Ġobserve Ġand Ġtranslate Ġthose Ġsub - comp onents Ġ( âĢ ľ W ohn ung âĢ Ŀ Ġand ĠâĢ ľ re in ig ung âĢ Ŀ ) Ġand Ġcombine Ġtheir Ġtranslations Ġto Ġgenerate Ġthe Ġunseen Ġwords Ġ( âĢ ľ house Ġcleaning âĢ Ŀ ). ĠThis Ġis Ġachieved Ġby Ġsegment ing Ġwords Ġinto Ġsub word Ġunits Ġusing Ġsegment ation Ġtechniques Ġin Ġthe Ġpre processing Ġphase Ġprior Ġto Ġtranslation . ĠThere Ġare Ġseveral Ġsegment ation Ġmethods ; ĠSome Ġare Ġthe Ġcomplicate Ġones Ġwhich Ġrequire Ġlinguistic Ġresources Ġor Ġhuman - crafted Ġrules . ĠThus , Ġthey Ġare Ġnot Ġlanguage - independent Ġand Ġexpensive Ġto Ġobtain Ġfor Ġlow - res ourced Ġlanguages . </s> Byte - P air ĠEnc oding , Ġotherwise , Ġis Ġa Ġsimple Ġbut Ġrobust Ġtechnique Ġto Ġdo Ġsub word Ġsegment ation . ĠSince Ġit Ġis Ġan Ġun super vised Ġand Ġfast Ġtechnique , Ġit Ġhas Ġgreat Ġeffects Ġwhen Ġapplied Ġto Ġbuild ĠN MT Ġsystems Ġfor Ġmorph ologically Ġrich Ġlanguages . ĠB PE Ġis Ġoriginally Ġproposed Ġin ĠB IB REF 9 Ġas Ġa Ġdata Ġcompression Ġtechnique Ġby Ġiter atively Ġreplaces Ġthe Ġmost Ġfrequent Ġpair Ġof Ġbytes Ġin Ġa Ġsequence Ġwith Ġa Ġsingle , Ġunused Ġbyte . ĠB IB REF 4 Ġrealized ĠG age 's Ġalgorithm Ġfor Ġword Ġsegment ation Ġby Ġmerging Ġfrequent Ġcharacters Ġinside Ġa Ġword Ġinstead Ġof Ġmerging Ġfrequent Ġpairs Ġof Ġbytes Ġin Ġa Ġwhole Ġfile Ġ( sequence Ġof Ġbytes ). ĠBeing Ġapplied Ġin Ġtranslation Ġfrom Ġand Ġto Ġmorph ological Ġrich Ġlanguages , Ġit Ġcan Ġautomatically Ġinduce Ġsub - comp onents Ġof Ġa Ġword Ġ( i . e . Ġsequence Ġof Ġcharacters ) Ġwhich Ġbear Ġsome Ġmeaning Ġor Ġmorph ological Ġfunction Ġwithout Ġknowing Ġmuch Ġabout Ġlinguistic Ġcharacteristics Ġof Ġthose Ġlanguages . ĠHence , Ġthe ĠT U Ġof Ġthe ĠN MT Ġused Ġin Ġthose Ġlanguages Ġis Ġat Ġa Ġsmaller Ġlevel Ġof Ġword , Ġi . e . Ġsub word . </s> On Ġthe Ġother Ġhand , ĠJapanese Ġand ĠVietnamese Ġare Ġdifferent Ġto Ġthose Ġlanguages Ġin Ġthe Ġway Ġof Ġhow Ġwe Ġconsider Ġa Ġword . ĠIn ĠVietnamese , Ġthe ĠT U , Ġnormally Ġconsidered Ġas Ġa Ġword , Ġwhich Ġoften Ġseparated Ġto Ġeach Ġothers Ġby Ġwhite Ġspaces , Ġis Ġnot Ġa Ġword Ġin Ġlinguistic Ġterm Ġsince Ġit Ġdoes Ġnot Ġreally Ġhave Ġits Ġown Ġmeaning . ĠTherefore , Ġapplying ĠB PE Ġto Ġsegment Ġthose ĠT Us Ġinto Ġsmaller Ġunits Ġwithout Ġany Ġmodification Ġis Ġnot Ġsuitable Ġfor ĠVietnamese . ĠIn Ġcase Ġof ĠJapanese , Ġthere Ġare Ġno Ġspace Ġto Ġseparate Ġwritten Ġtexts Ġinto ĠT Us . ĠThus , Ġfor ĠVietnamese Ġand ĠJapanese Ġas Ġwell Ġas Ġfor Ġthe Ġlanguages Ġhaving Ġsimilar Ġproblem , Ġbefore Ġapplying ĠB PE , Ġit Ġis Ġnecessary Ġto Ġhave Ġsome Ġpre processing Ġstep Ġto Ġtoken ize Ġthe Ġtexts Ġinto Ġwords . </s> V iet namese ĠToken ization </s> V iet namese ĠFrom Ġthe Ġlinguistic Ġpoint Ġof Ġview , Ġeach Ġsequence Ġof Ġcharacters Ġbetween Ġtwo Ġwhite Ġspaces Ġin ĠVietnamese Ġtexts Ġcannot Ġbe Ġconsidered Ġas Ġa Ġword Ġsince Ġit Ġdoes Ġnot Ġalways Ġhave Ġa Ġfull Ġmeaning Ġto Ġstand Ġalone . ĠFor Ġexample , Ġin Ġthe Ġsentence ĠâĢ ľ h Ã´ m Ġn ay Ġl Ãł Ġsin h Ġn h á º Ń t Ġc á » § a Ġt Ã´ i âĢ Ŀ Ġ( English Ġequival ence : ĠâĢ ľ Today Ġis Ġmy Ġbirthday âĢ Ŀ ), ĠâĢ ľ h Ã´ m âĢ Ŀ Ġand ĠâĢ ľ n ay âĢ Ŀ Ġare Ġnot Ġtwo Ġwords , Ġthey Ġtogether Ġform Ġa Ġword , Ġwhich Ġmeans ĠâĢ ľ today âĢ Ŀ . ĠNevertheless , ĠâĢ ľ h Ã´ m âĢ Ŀ Ġand ĠâĢ ľ n ay âĢ Ŀ Ġsomehow Ġstill Ġbear Ġsome Ġmeaning : ĠâĢ ľ h Ã´ m âĢ Ŀ - âĢ ľ day âĢ Ŀ , ĠâĢ ľ n ay âĢ Ŀ - âĢ ľ now âĢ Ŀ . ĠSimilarly , ĠâĢ ľ sin h âĢ Ŀ - âĢ ľ birth âĢ Ŀ Ġand ĠâĢ ľ n h á º Ń t âĢ Ŀ - âĢ ľ date âĢ Ŀ Ġalso Ġform Ġthe Ġword ĠâĢ ľ sin h Ġn h á º Ń t âĢ Ŀ - âĢ ľ birth day âĢ Ŀ Ġbut Ġthey Ġare Ġnot Ġtwo Ġdistinct Ġwords . ĠWe Ġcould Ġalso Ġcall Ġthem Ġsub words . </s> In Ġmany ĠVietnamese Ġprocessing Ġtasks Ġsuch Ġas ĠPart - of - Spe ech ĠT agging , ĠSynt ax ĠPars ing Ġor ĠCh unk ing , Ġthere Ġrequires Ġa Ġstep Ġto Ġconc aten ate Ġthose Ġsub words Ġto Ġmake Ġa Ġword Ġsince Ġin Ġthose Ġtasks , Ġword Ġis Ġnecessarily Ġconsidered Ġas Ġthe Ġsmallest Ġunit Ġto Ġbe Ġprocessed . ĠThis Ġstep Ġis Ġnormally Ġreferred Ġto Ġas Ġword Ġsegment ation . ĠThere Ġare Ġvarious Ġword Ġsegment ation Ġmethods ; ĠThe Ġbest Ġones Ġare Ġusing Ġmachine Ġlearning Ġapproaches Ġto Ġlearn Ġfrom Ġa Ġlabeled Ġcorpus . ĠIt Ġmakes Ġthe Ġtasks Ġhard Ġand Ġexpensive Ġto Ġbe Ġapplied Ġin Ġother Ġdomains . ĠFurthermore , Ġthe Ġtranslation Ġunit Ġin ĠMachine ĠTranslation Ġdoes Ġnot Ġneed Ġto Ġbe Ġa Ġword Ġbut Ġcan Ġbe Ġa Ġsub word Ġor Ġsequence Ġof Ġsub words Ġif Ġit Ġhave Ġits Ġown Ġmeaning . </s> With Ġthis Ġobservation Ġin Ġmind , Ġif Ġwe Ġconsider Ġa Ġsub word , Ġi . e . Ġthe Ġsequence Ġof Ġcharacters Ġbetween Ġtwo Ġwhite Ġspaces , Ġas Ġa Ġbyte Ġand Ġa Ġsentence Ġas Ġa Ġsequence Ġof Ġbytes , Ġwe Ġcan Ġapply Ġthe ĠB PE Ġalgorithm Ġstraight - forward : ĠWe Ġiter atively Ġfind Ġand Ġconc aten ate Ġthe Ġmost Ġfrequent Ġpair Ġof Ġsub words Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġand Ġreplace Ġit Ġby Ġan Ġunseen Ġsub words ĠIN LINE FORM 2 Ġ. ĠWe Ġdo Ġnot Ġmerge ĠIN LINE FORM 3 Ġwith ĠIN LINE FORM 4 Ġif Ġone Ġof Ġthem Ġare Ġdigits Ġor Ġpunct uations Ġor Ġother Ġspecial Ġsymbols . ĠThe ĠB PE Ġlearning Ġalgorithm Ġhas Ġan Ġarguments Ġwhich Ġis Ġthe Ġminimum Ġvalue Ġof Ġfrequency . ĠIn Ġpractice , Ġwe Ġset Ġthe Ġminimum Ġfrequency Ġis Ġ2 . ĠList ing Ġ1 Ġpresents Ġthis Ġvariant Ġof ĠB PE , Ġwhich Ġwe Ġcall ĠV NB PE . </s> my style Ġdef Ġget _ most _ fre q _ pair ( text , min _ fre q ): Ġdict Ġ= Ġ{} Ġdump Word Ġ= Ġ{ set _ of _ separ ate _ sy mb ols } Ġfor Ġline Ġin Ġtext : Ġw 1 Ġ= Ġthe _ first _ word _ in _ the _ line Ġfor Ġw 2 Ġ= Ġeach _ word _ in _ the _ line : Ġif Ġw 1 , w 2 Ġnot Ġin Ġdump Word Ġand Ġw 1 Ġis Ġnot Ġa Ġnumber : Ġdict [ w 1 , w 2 ] Ġ+= Ġ1 Ġget _ next _ pair _ in _ the _ line () Ġw 1 Ġ= Ġw 2 Ġreturn Ġ( all Ġpairs Ġhas Ġfre q Ġ> Ġmin _ fre q ) Ġdef Ġupdate _ p airs ( pair , text , codes _ file ): Ġoriginal _ word Ġ= Ġpair [ 0 ] Ġ+ Ġ" Ġ" Ġ+ Ġpair [ 1 ] Ġreplaced _ word Ġ= Ġpair [ 0 ] Ġ+ Ġ"_ " Ġ+ Ġpair [ 1 ] Ġinput _ file . replace ( rew , or w ) Ġwrite _ re placed _ word _ to _ codes _ file () Ġreturn Ġinput _ file Ġ### ĠMA IN ĠPART Ġ### Ġmin _ fre q = 2 Ġopen ( input _ file ) Ġto Ġread Ġopen ( codes _ file ) Ġand Ġ( output _ file ) Ġto Ġwrite Ġpairs Ġ= Ġget _ most _ fre q _ pair ( input _ file , min _ fre q ) Ġarrange _ p airs _ for _ dec re asing _ of _ frequency () Ġfor Ġeach Ġ( fre q , pair ) Ġin Ġpairs : Ġupdate _ p airs ( pair , input _ file , codes _ file ) Ġoutput _ file = input _ file Ġclose _ all _ files () ĠCompared Ġto Ġother Ġword Ġsegment ation Ġmethods Ġwhich Ġrequire Ġtraining Ġon Ġlabeled Ġdata , Ġour ĠV NB PE Ġis Ġa Ġsimple Ġun super vised Ġmethod , Ġalike Ġto Ġits Ġoriginal ĠB PE Ġalgorithm . ĠTable ĠT AB REF 7 Ġshows Ġthe Ġoutputs Ġof Ġone Ġdecent Ġword Ġsegment ation Ġmethod Ġand Ġour ĠV NB PE . </s> Japanese ĠToken ization </s> In Ġa ĠJapanese Ġwritten Ġtext , Ġthere Ġcould Ġbe Ġa Ġmixture Ġof Ġthree Ġdifferent Ġtypes Ġof Ġscripts : ĠChinese Ġcharacters Ġ( kan ji ) Ġand Ġthe Ġother Ġtwo Ġsyll ab ic Ġscripts : Ġh ir ag ana Ġand Ġk at ak ana . ĠEach Ġof Ġkan ji Ġcharacters Ġcan Ġbe Ġloosely Ġconsidered Ġas Ġa Ġsub word Ġthat Ġwe Ġmentioned Ġin Ġthe Ġprevious Ġsection . ĠIn Ġthe Ġmeanwhile , Ġeach Ġof Ġh ir ag ana Ġor Ġk at ak ana Ġcharacters Ġcan Ġbe Ġconsidered Ġas Ġa Ġlat in Ġcharacter Ġin ĠEnglish Ġor ĠVietnamese . ĠIn Ġaddition , Ġthere Ġis Ġno Ġspace Ġin Ġthe ĠJapanese Ġwritten Ġtexts Ġto Ġseparate Ġthe Ġcharacters , Ġeither Ġkan ji , Ġ, h ir ag ana Ġor Ġk at ak ana . ĠSo Ġwe Ġcannot Ġlearn Ġgood Ġsub words Ġfrom Ġa Ġsmall Ġcorpus Ġby Ġdirectly Ġapplying ĠB PE Ġor Ġthe Ġvariant ĠV NB PE Ġon ĠJapanese Ġwritten Ġtexts . ĠIn Ġorder Ġto Ġlearn Ġgood Ġsub words Ġand Ġwith Ġa Ġlittle Ġknowledge Ġabout ĠJapanese , Ġwe Ġdecided Ġto Ġuse ĠKy Tea ĠB IB REF 10 Ġto Ġdo ĠJapanese Ġword Ġsegment ation Ġand Ġthen Ġapply ĠS enn rich 's ĠB PE Ġon Ġthose Ġword - se gment ed Ġtexts . ĠSome Ġexamples Ġof Ġthe ĠJapanese Ġwords Ġgoing Ġthrough Ġthe Ġword Ġsegment ation Ġand ĠB PE Ġare Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. </s> Data ĠAug mentation </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġdata Ġaug mentation Ġmethods Ġwe Ġuse Ġto Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġorder Ġto Ġmake Ġour ĠN MT Ġsystems Ġsuffer Ġless Ġfrom Ġthe Ġlow - res ourced Ġsituation Ġin ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġtranslation . ĠAlthough ĠN MT Ġsystems Ġcan Ġpredict Ġand Ġgenerate Ġthe Ġtranslation Ġof Ġunseen Ġwords Ġon Ġtheir Ġvoc ab ular ies , Ġbut Ġthey Ġonly Ġperform Ġthis Ġwell Ġif Ġthe Ġparallel Ġcorpus Ġfor Ġtraining Ġare Ġsufficiently Ġlarge . ĠFor Ġmany Ġunder - res ourced Ġlanguages , Ġunfortunately , Ġit Ġhardly Ġpresents . ĠIn Ġreality , Ġalthough Ġthe Ġmon oling ual Ġdata Ġof ĠVietnamese Ġand ĠJapanese Ġare Ġimmensely Ġavailable Ġdue Ġto Ġthe Ġpopularity Ġof Ġtheir Ġspeakers , Ġthe Ġbilingual ĠJapanese - V iet namese Ġcorpor a Ġare Ġvery Ġlimited Ġand Ġoften Ġin Ġlow Ġquality Ġor Ġin Ġnarrowly Ġtechnical Ġdomains . ĠTherefore , Ġdata Ġaug mentation Ġmethods Ġto Ġexploit Ġmon oling ual Ġdata Ġfor ĠN MT Ġsystems Ġare Ġnecessary Ġto Ġobtain Ġmore Ġbilingual Ġdata , Ġthus Ġupgrading Ġthe Ġtranslating Ġquality . </s> Back ĠTranslation </s> One Ġof Ġthe Ġapproaches Ġto Ġleverage Ġthe Ġmon oling ual Ġdata Ġis Ġto Ġuse Ġa Ġmachine Ġtranslation Ġsystem Ġto Ġtranslate Ġthose Ġdata Ġin Ġorder Ġto Ġcreate Ġa Ġsynthetic Ġparallel Ġdata . ĠNormally , Ġthe Ġmon oling ual Ġdata Ġin Ġthe Ġtarget Ġlanguage Ġis Ġtranslated , Ġthus Ġthe Ġname Ġof Ġthe Ġmethod : ĠBack ĠTranslation ĠB IB REF 11 Ġ. </s> More Ġspecific , Ġto Ġgenerate Ġthe Ġdata Ġfor Ġan ĠX ĠIN LINE FORM 0 ĠY ĠN MT Ġsystem , Ġwe Ġuse Ġthe Ġbest ĠY ĠIN LINE FORM 1 ĠX Ġtranslation Ġsystem Ġwe Ġhave Ġto Ġtranslate Ġevery Ġsentence ĠIN LINE FORM 2 Ġin Ġthe Ġmon oling ual Ġdata Ġof Ġlanguage ĠY Ġinto Ġsentences ĠIN LINE FORM 3 Ġin Ġthe Ġsource Ġlanguage ĠX . ĠThen Ġwe Ġpair ĠIN LINE FORM 4 Ġto Ġget Ġthe Ġsynthetic Ġdata . ĠFinally , Ġoriginal Ġbilingual Ġdata Ġand Ġsynthetic Ġdata Ġare Ġmixed Ġto Ġtrain Ġour ĠN MT Ġfrom Ġthe Ġstart . </s> Back ĠTranslation Ġcan Ġimprove Ġthe Ġestimate Ġconditional Ġprobability Ġof Ġthe Ġtarget Ġword Ġon Ġthe Ġprevious Ġcontext Ġwords Ġthrough Ġadding Ġa Ġbilingual Ġdata Ġwith Ġapproximate Ġtranslations Ġto Ġthe Ġsource Ġlanguages . ĠFurthermore , Ġthe Ġsynthetic Ġdata Ġmight Ġcontain Ġsome Ġtranslation Ġnoise Ġfrom Ġthe ĠBack ĠTranslation Ġsystem , Ġand Ġif Ġthis Ġnoise Ġis Ġrelevant , Ġour ĠN MT Ġcan Ġbe Ġmore Ġrobust Ġin Ġlearning Ġhow Ġto Ġtranslate Ġnoisy Ġinputs . ĠOne Ġthe Ġother Ġhand , Ġif Ġthe Ġquality Ġof Ġthe ĠBack ĠTranslation Ġsystem Ġis Ġnot Ġadequate , Ġusing Ġthe Ġsynthesis Ġdata Ġmight Ġbring Ġadverse Ġeffects Ġto Ġour ĠN MT . </s> In Ġthis Ġpaper , Ġwe Ġsubs ample Ġan Ġamount Ġof ĠVietnamese Ġmon oling ual Ġdata Ġso Ġthat Ġwe Ġcan Ġcreate Ġa Ġsynthetic Ġcorpus Ġhaving Ġthe Ġsame Ġsize Ġwith Ġthe ĠJapanese - V iet namese Ġparallel Ġcorpus . ĠIn Ġthe Ġend , Ġthe Ġdata Ġwe Ġhave Ġis Ġdouble Ġin Ġsize Ġcompared Ġto Ġthe Ġoriginal Ġone . </s> Mix - Source ĠApproach </s> Another Ġdata Ġaug mentation Ġmethod Ġconsidered Ġuseful Ġin Ġthis Ġlow - res ourced Ġsetting Ġis Ġthe Ġmix - source Ġmethod ĠB IB REF 12 Ġ. ĠIn Ġthis Ġmethod , Ġwe Ġcan Ġutilize Ġthe Ġmon oling ual Ġdata Ġof Ġthe Ġtarget Ġlanguage Ġin Ġa Ġmult ilingual ĠN MT Ġsystem Ġby Ġmixing Ġthe Ġoriginal Ġsource Ġsentences Ġwith Ġthose Ġtarget Ġmon oling ual Ġdata . ĠThe Ġmult ilingual Ġframework Ġthen Ġuses Ġthe Ġshare Ġinformation Ġacross Ġsource Ġand Ġtarget Ġlanguages Ġto Ġimprove Ġthe Ġdecision Ġof Ġthe Ġtarget Ġwords Ġto Ġbe Ġchosen . </s> Specifically , Ġthere Ġare Ġa Ġsmall Ġparallel Ġcorpus ĠIN LINE FORM 0 Ġof Ġthe Ġlanguage Ġpair ĠX - Y Ġwhich Ġhas ĠIN LINE FORM 1 Ġsentence Ġpairs ĠIN LINE FORM 2 Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġa Ġbig Ġmon oling ual Ġcorpus ĠIN LINE FORM 4 Ġof Ġthe Ġlanguage ĠY Ġwhich Ġhas ĠIN LINE FORM 5 Ġsentences ĠIN LINE FORM 6 Ġ( ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ). ĠNow Ġfrom Ġthe Ġmon oling ual Ġcorpus ĠIN LINE FORM 9 Ġwe Ġcan Ġgenerate Ġa Ġparallel Ġcorpus ĠIN LINE FORM 10 Ġwhere Ġwe Ġtry Ġto Ġmodel Ġthe Ġidentical Ġtranslation ĠY - Y : ĠIN LINE FORM 11 Ġ( ĠIN LINE FORM 12 Ġ). ĠThen Ġwe Ġmix ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġto Ġget Ġa Ġparallel Ġcorpus Ġof Ġthe Ġsize ĠIN LINE FORM 15 Ġ. ĠSimilar Ġto Ġthe ĠBack ĠTranslation , Ġwe Ġsubs ample ĠIN LINE FORM 16 ĠVietnamese Ġsentence Ġpairs Ġfrom Ġthe Ġcorpus ĠIN LINE FORM 17 Ġthen Ġthe Ġsize Ġof Ġthe Ġparallel Ġdata Ġwe Ġhave Ġis Ġalso Ġdoubled . </s> To Ġlet Ġthe ĠN MT Ġknows Ġwhich Ġlanguage Ġa Ġcertain Ġsource Ġsentence Ġis Ġin Ġand Ġthen Ġcan Ġmodel Ġthe Ġlanguage Ġinformation , Ġwe Ġfollows Ġthe Ġconventions Ġfrom ĠB IB REF 12 Ġ. ĠWe Ġappend Ġlanguage Ġtags Ġto Ġevery Ġword Ġin Ġboth Ġsource Ġand Ġtarget Ġsentences Ġof Ġthe Ġmixed ĠIN LINE FORM 0 Ġcorpus Ġto Ġindicate Ġthe Ġlanguage Ġof Ġthe Ġwords . ĠThis Ġtechnique Ġshows Ġthe Ġeffectiveness Ġin Ġlow - res ourced Ġscenarios ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġour ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġis Ġsuch Ġa Ġscenario . </s> Data ĠPrepar ation </s> We Ġcollected ĠJapanese - V iet namese Ġparallel Ġdata Ġfrom ĠTED Ġtalks Ġextracted Ġfrom ĠW IT 3 's Ġcorpus ĠB IB REF 15 Ġ. ĠAfter Ġremoving Ġblank Ġand Ġduplicate Ġlines Ġwe Ġobtained Ġ106 758 Ġpairs Ġof Ġsentences . ĠThe Ġvalidation Ġset Ġused Ġin Ġall Ġexperiments Ġis Ġdev 2010 Ġand Ġthe Ġtest Ġset Ġis Ġt st 2010 . </s> The Ġdata Ġaug mentation Ġmethods Ġhas Ġbeen Ġapplied Ġonly Ġfor Ġthe ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġdirection . ĠFor ĠBack ĠTranslation , Ġwe Ġuse ĠVietnamese Ġmon oling ual Ġdata Ġfrom ĠV N ESE cor p us Ġof ĠDong Du Ġwhich Ġincludes Ġ349 578 Ġsentences . ĠWe Ġshuffle Ġthe Ġlines Ġof ĠV N ESE cor p us Ġcorpus Ġand Ġtake Ġout Ġthe Ġfirst Ġ106 758 Ġsentences Ġ( the Ġsame Ġas Ġthe Ġnumber Ġof Ġsentence Ġpairs Ġin Ġthe Ġoriginal Ġparallel Ġcorpus ). ĠFor ĠMix - Source , Ġinstead Ġof Ġusing Ġa Ġsubs am pled Ġmon oling ual Ġcorpus , Ġwe Ġuse Ġthe ĠVietnamese Ġpart Ġof Ġthe ĠJapanese - V iet namese Ġparallel Ġcorpus Ġin Ġorder Ġto Ġlearn Ġthe Ġmult ilingual Ġinformation Ġin Ġthe Ġsame Ġdomain . ĠOur Ġdatasets Ġare Ġlisted Ġin ĠTable ĠT AB REF 14 Ġ. </s> Pre processing </s> After Ġusing ĠKy Tea Ġto Ġtoken ize Ġthe ĠJapanese Ġtexts , Ġwe Ġlearn Ġand Ġapply ĠS enn rich 's ĠB PE Ġfrom Ġthe Ġtoken ized Ġtexts . ĠFor ĠVietnamese Ġtexts , Ġfirst Ġwe Ġuse ĠMoses Ġscripts Ġto Ġnormal ize Ġthe Ġtexts Ġfrom Ġdigits , Ġpunct uations Ġand Ġspecial Ġsymbols . ĠWe Ġuse Ġpy vi Ġfor ĠVietnamese Ġword Ġsegment ation Ġsince Ġit Ġis Ġone Ġof Ġthe Ġbest Ġtools Ġfor Ġthis Ġtask Ġin Ġterm Ġof Ġspeed , Ġrobust ness Ġand Ġperformance . ĠOn Ġthe Ġother Ġhand , Ġwe Ġuse V NB PE Ġas Ġan Ġalternative Ġway Ġof Ġdoing Ġword Ġsegment ation . ĠThose Ġtwo Ġapproaches Ġare Ġcompared Ġin Ġan Ġextr ins ic Ġevaluation Ġof Ġthe ĠN MT Ġsystems Ġemploying Ġthem . </s> System ĠArchitecture Ġand ĠTraining </s> We Ġimplement Ġthe Ġtranslation Ġsystems Ġusing ĠOpen N MT - py Ġframework ĠB IB REF 16 Ġ. ĠOur Ġsystem Ġarchitecture Ġincludes Ġtwo Ġbi - direction al ĠL ST M Ġlayers Ġfor Ġthe Ġenc oder Ġand Ġtwo ĠL ST M Ġlayers Ġfor Ġthe Ġdec oder , Ġeach Ġlayer Ġhas Ġthe Ġsize Ġof Ġ512 Ġhidden Ġunits . ĠThe Ġsize Ġof Ġsource Ġand Ġtarget Ġembed ding Ġlayers Ġis Ġalso Ġ512 . ĠWe Ġuse ĠAdam Ġoptim izer ĠB IB REF 17 Ġand Ġlearning Ġrate Ġan ne aling Ġscheme Ġwith Ġthe Ġinitial Ġlearning Ġrate Ġat ĠIN LINE FORM 0 Ġ. ĠWe Ġtrain Ġeach Ġsystems Ġfor Ġ15 Ġepoch s Ġwith Ġthe Ġbatch Ġsize Ġof Ġ64 . ĠThe Ġbest Ġmodel Ġin Ġterm Ġof Ġthe Ġun ig ram Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġis Ġusually Ġused Ġto Ġtranslate Ġthe Ġtest Ġset Ġwith Ġbeam Ġsize Ġof Ġ16 . ĠOther Ġsettings Ġare Ġthe Ġdefault Ġsettings Ġof ĠOpen N MT - py , Ġotherwise Ġalready Ġnoted . </s> Results </s> We Ġevaluate Ġthe Ġquality Ġof Ġtranslation Ġof Ġsystems Ġbased Ġon Ġdifferent Ġapproaches Ġmentioned Ġin Ġprevious Ġsections . ĠThe Ġmulti - BLE U Ġfrom ĠMoses Ġscripts Ġis Ġused . ĠThe Ġresults Ġhave Ġshown Ġin Ġthe ĠTable ĠT AB REF 21 Ġ. </s> Bas eline . ĠFor Ġthe Ġbaseline Ġsystems , Ġthe Ġtraining Ġdata Ġincludes ĠKy Tea - se gment ed ĠJapanese Ġtexts Ġand Ġpy vi - se gment ed ĠVietnamese Ġtexts . ĠFor Ġcomparison Ġpurpose , Ġwe Ġbuild Ġtwo Ġbaseline Ġsystems Ġfor Ġeach Ġdirection : Ġone Ġis Ġuse Ġthe Ġtraditional Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( SM T ), Ġthe Ġother Ġone Ġis Ġthe ĠN MT Ġsystem . ĠAlthough Ġour Ġtraining Ġset Ġis Ġsmall Ġbut Ġwe Ġfind Ġthat Ġthe ĠN MT Ġsystems Ġ( 2 ) Ġare Ġstill Ġmore Ġeffective Ġthan Ġthe Ġphrase - based ĠSM T Ġmodels Ġ( 1 ) Ġin Ġboth Ġtranslation Ġdirections . </s> Sub word ĠN MT . ĠWe Ġapplied ĠV NB PE Ġand ĠJP B PE Ġto Ġthe Ġbaseline 's Ġdata Ġand Ġtrained ĠN MT Ġsystems . ĠOn ĠVietnamese ĠIN LINE FORM 0 ĠJapanese , Ġwe Ġobserved Ġan Ġimprovement Ġof Ġ0 . 6 ĠB LE U Ġpoints Ġwhen Ġwe Ġused Ġour ĠV NB PE Ġ( 3 ) Ġinstead Ġof Ġthe Ġpy vi 's Ġword Ġsegment ation Ġ( 2 ). ĠFurthermore , Ġwhen Ġwe Ġtrained Ġour ĠN MT Ġmodels Ġusing Ġboth ĠB PE Ġmethods Ġ( 4 ), Ġwe Ġobtained Ġa Ġbigger Ġgain Ġof Ġ1 . 15 ĠB LE U Ġpoints . ĠThe Ġsimilar Ġimprovements Ġcan Ġbe Ġfound Ġin Ġthe ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġas Ġwell : Ġ0 . 29 ĠB LE U Ġpoints Ġbetween Ġ( 3 ) Ġand Ġ( 2 ) Ġand Ġ0 . 57 ĠB LE U Ġpoints Ġbetween Ġ( 4 ) Ġand Ġ( 3 ). ĠThis Ġdraws Ġtwo Ġconclusions : Ġ( i ), Ġdespite Ġusing Ġan Ġun super vised ĠVietnamese Ġword Ġsegment ation Ġwhich Ġis Ġfast , Ġrobust Ġand Ġdoes Ġnot Ġrequire Ġlinguistic Ġresources , Ġour ĠN MT Ġsystems Ġperformed Ġbetter Ġthan Ġthose Ġsystems Ġemploying Ġa Ġcomplicate Ġword Ġsegment ation Ġmethod , Ġ( ii ) ĠB PE Ġworks Ġsignificantly Ġwell Ġfor ĠJapanese Ġtexts Ġafter Ġwe Ġtoken ized Ġthe Ġtexts . </s> Data ĠAug mentation . ĠWe Ġuse Ġthe Ġbest Ġsystem Ġfor ĠVietnamese ĠIN LINE FORM 0 ĠJapanese , Ġwhich Ġis Ġthe ĠN MT Ġsystems Ġtrained Ġon ĠB PE - process ed Ġtexts , Ġto Ġgenerate Ġthe Ġsynthetic Ġdata Ġfor ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġtranslation . ĠAlthough Ġwe Ġachieved Ġsome Ġgain Ġ( from Ġ9 . 04 Ġto Ġ9 . 39 ĠB LE U Ġpoints ), Ġthe Ġeffectiveness Ġof ĠBack ĠTranslation Ġis Ġnot Ġon Ġpar Ġwith Ġits Ġapplication Ġon Ġthe Ġtranslation Ġsystems Ġof Ġother Ġlanguage Ġpairs . ĠLooking Ġinto Ġthe ĠVietnamese ĠIN LINE FORM 2 ĠJapanese Ġtranslation Ġof ĠDong Du Ġcorpus Ġand Ġits ĠB LE U Ġscore , Ġwe Ġspeculate Ġthat Ġit Ġis Ġbecause Ġthe ĠVietnamese ĠIN LINE FORM 3 ĠJapanese Ġsystem Ġis Ġnot Ġgood Ġenough Ġto Ġproduce Ġreasonable Ġsynthetic Ġdata . ĠIn Ġthe Ġmeanwhile , Ġcombining ĠBack ĠTranslation Ġand ĠMix - Source Ġbrings Ġa Ġconsiderable Ġimprovements Ġof Ġ0 . 6 ĠB LE U Ġpoints Ġcompared Ġto Ġnot Ġusing Ġthem . </s> Related ĠWorks </s> Japanese - V iet namese ĠMT Ġis Ġfirst ly Ġmentioned Ġin Ġ2005 ĠB IB REF 18 Ġ. ĠThe Ġauthors Ġfocused Ġon Ġthe Ġdifference Ġfrom Ġembed ding Ġstructures Ġbetween ĠJapanese Ġand ĠVietnamese , Ġand Ġthen Ġproposed Ġrules Ġfor ĠMT Ġsystem Ġand Ġexperiment Ġon Ġvery Ġsmall Ġdataset Ġ( 714 ĠJapanese Ġembed ding Ġsentences ). ĠThis Ġapproach Ġis Ġsuitable Ġfor Ġsmall Ġsystem Ġapplied Ġin Ġa Ġspecific Ġdomain Ġor Ġlanguage , Ġbut Ġit Ġis Ġnot Ġeasily Ġextend able Ġto Ġother Ġdomains Ġor Ġlanguages Ġdue Ġto Ġthe Ġexp ens iveness Ġof Ġbuilding Ġsuch Ġrules . </s> The Ġother Ġprevious Ġwork Ġfor ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġuses ĠSM T ĠB IB REF 19 Ġ. ĠThey Ġalso Ġconducted Ġthe Ġexperiments Ġon Ġparallel Ġcorpor a Ġcollected Ġfrom ĠTED Ġtalks . ĠThey Ġused Ġphrase - based Ġand Ġtree - to - string Ġmodels Ġand Ġhave Ġshown Ġthat Ġthe ĠSM T Ġsystem Ġtrained Ġon ĠFrench ĠIN LINE FORM 1 ĠVietnamese Ġob tains Ġbetter Ġresults Ġthan Ġthe Ġsystem Ġof ĠJapanese ĠIN LINE FORM 2 ĠVietnamese Ġbecause ĠFrench Ġand ĠVietnamese Ġhave Ġmore Ġsimilarities Ġin Ġthe Ġstructures Ġof Ġsentences Ġthan Ġbetween ĠJapanese Ġand ĠVietnamese . ĠWe Ġalso Ġbuilt Ġphrase - based Ġsystems Ġon Ġthe ĠTED Ġdata Ġand Ġachieved Ġbetter ĠB LE U Ġscores Ġwhen Ġusing ĠN MT . </s> Recently , Ġsome Ġworks Ġuse Ġmon oling ual Ġdata Ġto Ġimprove Ġthe Ġaccuracy Ġof ĠN MT Ġsystems . ĠB IB REF 11 Ġhave Ġshown Ġsignificant Ġimprovements Ġby Ġusing Ġmon oling ual Ġdata Ġon Ġtarget - side Ġto Ġgenerate Ġsynthetic Ġdata Ġand Ġthen Ġadd Ġthem Ġto Ġoriginal Ġtraining Ġdata . ĠB IB REF 20 Ġhave Ġshown Ġsignificant Ġimprovements Ġby Ġ" self - learning " Ġmethod Ġto Ġgenerate Ġthe Ġtarget Ġsentences Ġbased Ġon Ġmon oling ual Ġdata Ġon Ġthe Ġsource - side Ġand Ġthen Ġcombined Ġthem Ġwith Ġoriginal Ġbilingual Ġdata Ġto Ġtrain . ĠB IB REF 12 Ġconvert Ġmon oling ual Ġcorpus Ġon Ġthe Ġtarget - side Ġinto Ġa Ġbite xt s Ġby Ġcopying Ġtarget Ġsentences Ġto Ġthe Ġsource Ġsentence Ġand Ġthen Ġcombined Ġoriginal Ġbilingual Ġdata Ġtogether Ġon Ġtraining . ĠOur Ġsystems Ġemploy Ġthose Ġapproaches Ġto Ġexploit Ġmon oling ual Ġdata Ġand Ġshow Ġthe Ġimproved Ġperformance Ġfor ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġtranslations . </s> Conclusion </s> We Ġhas Ġbuilt Ġthe Ġfirst ĠJapanese ĠIN LINE FORM 0 ĠVietnamese ĠN MT Ġsystems Ġand Ġreleased Ġthe Ġdataset Ġas Ġwell Ġas Ġthe Ġassociated Ġtraining Ġscripts . ĠWe Ġhave Ġalso Ġshown Ġthat Ġthe Ġproposed ĠV NB PE Ġalgorithm Ġcan Ġbe Ġused Ġfor ĠVietnamese Ġword Ġsegment ation Ġin Ġorder Ġto Ġconduct Ġneural Ġmachine Ġtranslation . ĠFurthermore , Ġby Ġadapting ĠBack ĠTranslation Ġand ĠMix - Source , Ġour ĠN MT Ġsystems Ġachieved Ġthe Ġbest Ġimprovement Ġon Ġthe Ġdataset . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġexploit Ġmore Ġdomain Ġand Ġmult ilingual Ġinformation Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġsystems . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġcenter Ġof ĠHigh - performance Ġcomputing Ġ( H PC ), ĠUniversity Ġof ĠEngineering Ġand ĠTechnology , ĠV N U , ĠVietnam Ġfor Ġallowing Ġus Ġto Ġuse Ġtheir ĠGPUs Ġto Ġperform Ġthe Ġexperiments Ġmentioned Ġin Ġthe Ġpaper . ĠWe Ġalso Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcareful Ġreading Ġof Ġour Ġpaper Ġand Ġinsightful Ġcomments .
Introduction Neural machine translation (NMT) BIBREF0 , BIBREF1 is widely applied for machine translation (MT) in recent years and focuses on popular language pairs such as English INLINEFORM0 French, English INLINEFORM1 German, English INLINEFORM2 Chinese or English INLINEFORM3 Japanese. NMT has obtained state-of-the-art performance on those language pairs compared to the traditional statistical machine translation (SMT) when given enough data BIBREF2 , BIBREF3 . Furthermore, due to the ability of feature learning, NMT systems can be trained end-to-end with pure parallel texts and minimal linguistic knowledge of the languages involved. Thus it makes training NMT for a new language pair much easier, more scalable and robust. Nevertheless, NMT has not been employed in many low-resourced language pairs since in those scenarios, data scarcity often limits the learning ability of neural methods. In contrast, combinating complicated linguistic-driven features in a typical log-linear framework still keeps SMT the best approach in many translation directions but also hard to apply to new domains or to other language pairs. In this paper we attempt to build NMT systems for such a low-resourced language pair: Japanese INLINEFORM0 Vietnamese. Our aim is to set the first and reasonable NMT systems that can be reproducible in order to serve as the baseline for further researches in the direction. Furthermore, we conduct experiments using some advanced methods to improve the quality of the systems. An important criteria for those methods is that they must be scalable and language-independent as much as possible. The criteria ensures the basic principle of NMT as well as the reproducibility of the systems. On the other hand, the methods are chosen in the direction that they would help alleviate the data sparsity problem of NMT when being applied in this low-resourced setting. Specifically, to deal with rare-word translation problems, we experiment with translation units in different levels: subword, word and beyond. In morphological-rich languages such as English or German, using subword as the translation units is often suitable since neural methods are able to induce meaning or linguistic function of sub-components constituting a word. Byte-Pair Encoding (BPE) BIBREF4 is a simple unsupervised technique to do subword segmentation and it has great effects when applied to NMT training. Japanese and Vietnamese (and some other Asian languages), however, have different word segmentation issues. Hence, it would be difficult to apply BPE directly to the texts in those languages and build NMT systems for subword translation without any modification. In this paper, we experiment different segmentation methods for both languages and also propose a variant of BPE algorithm to learn translation units for Vietnamese in an unsupervised way. We also attempt to increase the amount of training data by using back-translated texts or mix-source data just from our small available corpus. Those data augmentation approaches have shown their effectiveness on various NMT systems, especially in under-resourced scenarios. While back translation technique is used to generate synthetic data from monolingual corpora, mix-source technique utilizes human-quality corpora in a multilingual setting, leveraging the transfer learning ability across languages. Both are simple but elegantly model the relevant noise needed in training neural architectures in such low-resourced situations. The main contributions of this paper are: Neural Machine Translation In this section, we will describe the general architecture of NMT as a kind of sequence-to-sequence modeling framework. In this kind of sequence-to-sequence modeling framework, often there is an encoder trying to encode context information from the input sequence and a decoder to generate one item of the output sequence at a time based on the context of both input and output sequences. Besides, an additional component, named attention, exists in between, deciding which parts of the input sequence the decoder should pay attention in order to choose which to output next. In other words, this attention component calculates the context relevant to the decision of the decoder at the considering time. Those components as a whole constitute a large trainable neural architecture called the famous attention-based encoder-decoder framework. This framework becomes popular in many sequence-to-sequence tasks. In the field of machine translation, using the attention-based encoder-decoder framework is referred to as Neural Machine Translation approach. First presented in BIBREF0 , the encoder and decoder in NMT are recurrent-based, which each hidden unit in those components is a recurrent unit like Long Short-term Memory (LSTM) BIBREF5 or Gated Recurrent Unit(GRU) BIBREF6 . Later, the encoder or decoder can also be a convolutional architecture, as in BIBREF7 . Recently, BIBREF8 introduces transformer architecture, in which both the encoder and decoder are a special variant of attention mechanism, called self attention. In this paper, we will briefly explain the Recurrent NMT as we utilize it in our experiments. The Recurrent NMT model follows the attention-based architecture proposed by BIBREF0 . The bidirectional recurrent encoder reads every words INLINEFORM0 of a source sentence INLINEFORM1 and encodes a representation INLINEFORM2 of the sentence into a fixed-length vector INLINEFORM3 concatenated from those of the forward and backward directions: INLINEFORM4 Here INLINEFORM0 is the one-hot vector of the word INLINEFORM1 and INLINEFORM2 is the word embedding matrix which is shared across the source words. INLINEFORM3 is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. INLINEFORM4 is then called an annotation vector, which represent the information of the source sentence up to the time INLINEFORM5 from both forward and backward directions. Those annotation vectors of the source sentences are combined in the attention layers in a way that the resulted vector encodes the source context relevant to the considering target word the decoder should produce. Intuitively, a relevance between the previous target word INLINEFORM0 and the annotation vectors INLINEFORM1 corresponding to the source words INLINEFORM2 can be used to form some attention scenario: DISPLAYFORM0 This specific attention mechanism, originally called alignment model in BIBREF0 , has been employed as a simple feedforward network with the first layer is a learnable layer via adaptation factors INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The relevance scores INLINEFORM3 are then normalized into attention weights INLINEFORM4 and the context vector INLINEFORM5 is calculated as the weighted sum of all annotation vectors INLINEFORM6 . Depending on how much attention the target word at time INLINEFORM7 put on the source states INLINEFORM8 , a soft alignment is learned and a source context INLINEFORM9 at time INLINEFORM10 is calculated prior to the prediction of the decoder. Similar to the encoder, the recurrent decoder recursively generates one target word INLINEFORM0 to form a translated target sentence INLINEFORM1 in the end. At time INLINEFORM2 , it takes the previous hidden state of the decoder INLINEFORM3 , the previous embedded word representation INLINEFORM4 and the time-specific context vector INLINEFORM5 as inputs to calculate the current hidden state INLINEFORM6 : INLINEFORM7 Again, INLINEFORM0 is the recurrent activation function of the decoder and INLINEFORM1 is the shared word embedding matrix of the target sentences. Given the parallel corpus consisting of INLINEFORM0 training examples INLINEFORM1 , the objective is to maximize the conditional log-probability of the correct translation given a source sentence with respect to the parameters INLINEFORM2 of the whole model: INLINEFORM3 Subword Translation One of the most severe problems of NMT is dealing with the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. On one hand, we would like to have fewer OOV words by increasing the size of the short lists. On the other hand, we need our neural network to learn fast and has a good generalization capability on the unseen words as well. As explained shortly in the introduction, for many languages, using subword instead of word as a translation unit (TU) has been shown that it is not only effective on reducing vocabulary sizes, thus alleviating the computing burden on the large soft-max layer as well as reducing a substantial number of parameters to be learnt, but also has the ability to generate unseen words. In those languages, a word can be a compound word or comprised by sub-components, each has its own raw meaning or contains morphological information. Segmenting words into sub-components allows NMT to learn to translate them with considerably fewer data. For example, it is definitely less chance to see this popular German word “Wohnungsreinigung" (English equivalence: “house cleaning”) than its sub-components “Wohnung” (i.e. “house” or “flat”) and “reinigung” (i.e. “cleaning”) in a middle-sized German-English parallel corpus. Instead, NMT can observe and translate those sub-components (“Wohnung” and “reinigung”) and combine their translations to generate the unseen words (“house cleaning”). This is achieved by segmenting words into subword units using segmentation techniques in the preprocessing phase prior to translation. There are several segmentation methods; Some are the complicate ones which require linguistic resources or human-crafted rules. Thus, they are not language-independent and expensive to obtain for low-resourced languages. Byte-Pair Encoding, otherwise, is a simple but robust technique to do subword segmentation. Since it is an unsupervised and fast technique, it has great effects when applied to build NMT systems for morphologically rich languages. BPE is originally proposed in BIBREF9 as a data compression technique by iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. BIBREF4 realized Gage's algorithm for word segmentation by merging frequent characters inside a word instead of merging frequent pairs of bytes in a whole file (sequence of bytes). Being applied in translation from and to morphological rich languages, it can automatically induce sub-components of a word (i.e. sequence of characters) which bear some meaning or morphological function without knowing much about linguistic characteristics of those languages. Hence, the TU of the NMT used in those languages is at a smaller level of word, i.e. subword. On the other hand, Japanese and Vietnamese are different to those languages in the way of how we consider a word. In Vietnamese, the TU, normally considered as a word, which often separated to each others by white spaces, is not a word in linguistic term since it does not really have its own meaning. Therefore, applying BPE to segment those TUs into smaller units without any modification is not suitable for Vietnamese. In case of Japanese, there are no space to separate written texts into TUs. Thus, for Vietnamese and Japanese as well as for the languages having similar problem, before applying BPE, it is necessary to have some preprocessing step to tokenize the texts into words. Vietnamese Tokenization Vietnamese From the linguistic point of view, each sequence of characters between two white spaces in Vietnamese texts cannot be considered as a word since it does not always have a full meaning to stand alone. For example, in the sentence “hôm nay là sinh nhật của tôi” (English equivalence: “Today is my birthday”), “hôm” and “nay” are not two words, they together form a word, which means “today”. Nevertheless, “hôm” and “nay” somehow still bear some meaning: “hôm”-“day”, “nay”-“now”. Similarly, “sinh”-“birth” and “nhật”-“date” also form the word “sinh nhật”-“birthday” but they are not two distinct words. We could also call them subwords. In many Vietnamese processing tasks such as Part-of-Speech Tagging, Syntax Parsing or Chunking, there requires a step to concatenate those subwords to make a word since in those tasks, word is necessarily considered as the smallest unit to be processed. This step is normally referred to as word segmentation. There are various word segmentation methods; The best ones are using machine learning approaches to learn from a labeled corpus. It makes the tasks hard and expensive to be applied in other domains. Furthermore, the translation unit in Machine Translation does not need to be a word but can be a subword or sequence of subwords if it have its own meaning. With this observation in mind, if we consider a subword, i.e. the sequence of characters between two white spaces, as a byte and a sentence as a sequence of bytes, we can apply the BPE algorithm straight-forward: We iteratively find and concatenate the most frequent pair of subwords ( INLINEFORM0 , INLINEFORM1 ) and replace it by an unseen subwords INLINEFORM2 . We do not merge INLINEFORM3 with INLINEFORM4 if one of them are digits or punctuations or other special symbols. The BPE learning algorithm has an arguments which is the minimum value of frequency. In practice, we set the minimum frequency is 2. Listing 1 presents this variant of BPE, which we call VNBPE. mystyle def get_most_freq_pair(text,min_freq): dict = {} dumpWord = {set_of_separate_symbols} for line in text: w1 = the_first_word_in_the_line for w2 = each_word_in_the_line: if w1,w2 not in dumpWord and w1 is not a number: dict[w1,w2] += 1 get_next_pair_in_the_line() w1 = w2 return (all pairs has freq > min_freq) def update_pairs(pair,text,codes_file): original_word = pair[0] + " " + pair[1] replaced_word = pair[0] + "_" + pair[1] input_file.replace(rew,orw) write_replaced_word_to_codes_file() return input_file ### MAIN PART ### min_freq=2 open(input_file) to read open(codes_file) and (output_file) to write pairs = get_most_freq_pair(input_file,min_freq) arrange_pairs_for_decreasing_of_frequency() for each (freq,pair) in pairs: update_pairs(pair,input_file,codes_file) output_file=input_file close_all_files() Compared to other word segmentation methods which require training on labeled data, our VNBPE is a simple unsupervised method, alike to its original BPE algorithm. Table TABREF7 shows the outputs of one decent word segmentation method and our VNBPE. Japanese Tokenization In a Japanese written text, there could be a mixture of three different types of scripts: Chinese characters (kanji) and the other two syllabic scripts: hiragana and katakana. Each of kanji characters can be loosely considered as a subword that we mentioned in the previous section. In the meanwhile, each of hiragana or katakana characters can be considered as a latin character in English or Vietnamese. In addition, there is no space in the Japanese written texts to separate the characters, either kanji, ,hiragana or katakana. So we cannot learn good subwords from a small corpus by directly applying BPE or the variant VNBPE on Japanese written texts. In order to learn good subwords and with a little knowledge about Japanese, we decided to use KyTea BIBREF10 to do Japanese word segmentation and then apply Sennrich's BPE on those word-segmented texts. Some examples of the Japanese words going through the word segmentation and BPE are shown in Table TABREF9 . Data Augmentation In this section, we describe the data augmentation methods we use to increase the amount of training data in order to make our NMT systems suffer less from the low-resourced situation in Japanese INLINEFORM0 Vietnamese translation. Although NMT systems can predict and generate the translation of unseen words on their vocabularies, but they only perform this well if the parallel corpus for training are sufficiently large. For many under-resourced languages, unfortunately, it hardly presents. In reality, although the monolingual data of Vietnamese and Japanese are immensely available due to the popularity of their speakers, the bilingual Japanese-Vietnamese corpora are very limited and often in low quality or in narrowly technical domains. Therefore, data augmentation methods to exploit monolingual data for NMT systems are necessary to obtain more bilingual data, thus upgrading the translating quality. Back Translation One of the approaches to leverage the monolingual data is to use a machine translation system to translate those data in order to create a synthetic parallel data. Normally, the monolingual data in the target language is translated, thus the name of the method: Back Translation BIBREF11 . More specific, to generate the data for an X INLINEFORM0 Y NMT system, we use the best Y INLINEFORM1 X translation system we have to translate every sentence INLINEFORM2 in the monolingual data of language Y into sentences INLINEFORM3 in the source language X. Then we pair INLINEFORM4 to get the synthetic data. Finally, original bilingual data and synthetic data are mixed to train our NMT from the start. Back Translation can improve the estimate conditional probability of the target word on the previous context words through adding a bilingual data with approximate translations to the source languages. Furthermore, the synthetic data might contain some translation noise from the Back Translation system, and if this noise is relevant, our NMT can be more robust in learning how to translate noisy inputs. One the other hand, if the quality of the Back Translation system is not adequate, using the synthesis data might bring adverse effects to our NMT. In this paper, we subsample an amount of Vietnamese monolingual data so that we can create a synthetic corpus having the same size with the Japanese-Vietnamese parallel corpus. In the end, the data we have is double in size compared to the original one. Mix-Source Approach Another data augmentation method considered useful in this low-resourced setting is the mix-source method BIBREF12 . In this method, we can utilize the monolingual data of the target language in a multilingual NMT system by mixing the original source sentences with those target monolingual data. The multilingual framework then uses the share information across source and target languages to improve the decision of the target words to be chosen. Specifically, there are a small parallel corpus INLINEFORM0 of the language pair X-Y which has INLINEFORM1 sentence pairs INLINEFORM2 ( INLINEFORM3 ) and a big monolingual corpus INLINEFORM4 of the language Y which has INLINEFORM5 sentences INLINEFORM6 ( INLINEFORM7 , INLINEFORM8 ). Now from the monolingual corpus INLINEFORM9 we can generate a parallel corpus INLINEFORM10 where we try to model the identical translation Y-Y: INLINEFORM11 ( INLINEFORM12 ). Then we mix INLINEFORM13 and INLINEFORM14 to get a parallel corpus of the size INLINEFORM15 . Similar to the Back Translation, we subsample INLINEFORM16 Vietnamese sentence pairs from the corpus INLINEFORM17 then the size of the parallel data we have is also doubled. To let the NMT knows which language a certain source sentence is in and then can model the language information, we follows the conventions from BIBREF12 . We append language tags to every word in both source and target sentences of the mixed INLINEFORM0 corpus to indicate the language of the words. This technique shows the effectiveness in low-resourced scenarios BIBREF13 , BIBREF14 and our Japanese INLINEFORM1 Vietnamese is such a scenario. Data Preparation We collected Japanese-Vietnamese parallel data from TED talks extracted from WIT3's corpus BIBREF15 . After removing blank and duplicate lines we obtained 106758 pairs of sentences. The validation set used in all experiments is dev2010 and the test set is tst2010. The data augmentation methods has been applied only for the Japanese INLINEFORM0 Vietnamese direction. For Back Translation, we use Vietnamese monolingual data from VNESEcorpus of DongDu which includes 349578 sentences. We shuffle the lines of VNESEcorpus corpus and take out the first 106758 sentences (the same as the number of sentence pairs in the original parallel corpus). For Mix-Source, instead of using a subsampled monolingual corpus, we use the Vietnamese part of the Japanese-Vietnamese parallel corpus in order to learn the multilingual information in the same domain. Our datasets are listed in Table TABREF14 . Preprocessing After using KyTea to tokenize the Japanese texts, we learn and apply Sennrich's BPE from the tokenized texts. For Vietnamese texts, first we use Moses scripts to normalize the texts from digits, punctuations and special symbols. We use pyvi for Vietnamese word segmentation since it is one of the best tools for this task in term of speed, robustness and performance. On the other hand, we useVNBPE as an alternative way of doing word segmentation. Those two approaches are compared in an extrinsic evaluation of the NMT systems employing them. System Architecture and Training We implement the translation systems using OpenNMT-py framework BIBREF16 . Our system architecture includes two bi-directional LSTM layers for the encoder and two LSTM layers for the decoder, each layer has the size of 512 hidden units. The size of source and target embedding layers is also 512. We use Adam optimizer BIBREF17 and learning rate annealing scheme with the initial learning rate at INLINEFORM0 . We train each systems for 15 epochs with the batch size of 64. The best model in term of the unigram accuracy on the validation set is usually used to translate the test set with beam size of 16. Other settings are the default settings of OpenNMT-py, otherwise already noted. Results We evaluate the quality of translation of systems based on different approaches mentioned in previous sections. The multi-BLEU from Moses scripts is used. The results have shown in the Table TABREF21 . Baseline. For the baseline systems, the training data includes KyTea-segmented Japanese texts and pyvi-segmented Vietnamese texts. For comparison purpose, we build two baseline systems for each direction: one is use the traditional phrase-based statistical machine translation (SMT), the other one is the NMT system. Although our training set is small but we find that the NMT systems (2) are still more effective than the phrase-based SMT models (1) in both translation directions. Subword NMT. We applied VNBPE and JPBPE to the baseline's data and trained NMT systems. On Vietnamese INLINEFORM0 Japanese, we observed an improvement of 0.6 BLEU points when we used our VNBPE (3) instead of the pyvi's word segmentation (2). Furthermore, when we trained our NMT models using both BPE methods (4), we obtained a bigger gain of 1.15 BLEU points. The similar improvements can be found in the Japanese INLINEFORM1 Vietnamese as well: 0.29 BLEU points between (3) and (2) and 0.57 BLEU points between (4) and (3). This draws two conclusions: (i), despite using an unsupervised Vietnamese word segmentation which is fast, robust and does not require linguistic resources, our NMT systems performed better than those systems employing a complicate word segmentation method, (ii) BPE works significantly well for Japanese texts after we tokenized the texts. Data Augmentation. We use the best system for Vietnamese INLINEFORM0 Japanese, which is the NMT systems trained on BPE-processed texts, to generate the synthetic data for Japanese INLINEFORM1 Vietnamese translation. Although we achieved some gain (from 9.04 to 9.39 BLEU points), the effectiveness of Back Translation is not on par with its application on the translation systems of other language pairs. Looking into the Vietnamese INLINEFORM2 Japanese translation of DongDu corpus and its BLEU score, we speculate that it is because the Vietnamese INLINEFORM3 Japanese system is not good enough to produce reasonable synthetic data. In the meanwhile, combining Back Translation and Mix-Source brings a considerable improvements of 0.6 BLEU points compared to not using them. Related Works Japanese-Vietnamese MT is firstly mentioned in 2005 BIBREF18 . The authors focused on the difference from embedding structures between Japanese and Vietnamese, and then proposed rules for MT system and experiment on very small dataset (714 Japanese embedding sentences). This approach is suitable for small system applied in a specific domain or language, but it is not easily extendable to other domains or languages due to the expensiveness of building such rules. The other previous work for Japanese INLINEFORM0 Vietnamese uses SMT BIBREF19 . They also conducted the experiments on parallel corpora collected from TED talks. They used phrase-based and tree-to-string models and have shown that the SMT system trained on French INLINEFORM1 Vietnamese obtains better results than the system of Japanese INLINEFORM2 Vietnamese because French and Vietnamese have more similarities in the structures of sentences than between Japanese and Vietnamese. We also built phrase-based systems on the TED data and achieved better BLEU scores when using NMT. Recently, some works use monolingual data to improve the accuracy of NMT systems. BIBREF11 have shown significant improvements by using monolingual data on target-side to generate synthetic data and then add them to original training data. BIBREF20 have shown significant improvements by "self-learning" method to generate the target sentences based on monolingual data on the source-side and then combined them with original bilingual data to train. BIBREF12 convert monolingual corpus on the target-side into a bitexts by copying target sentences to the source sentence and then combined original bilingual data together on training. Our systems employ those approaches to exploit monolingual data and show the improved performance for Japanese INLINEFORM0 Vietnamese translations. Conclusion We has built the first Japanese INLINEFORM0 Vietnamese NMT systems and released the dataset as well as the associated training scripts. We have also shown that the proposed VNBPE algorithm can be used for Vietnamese word segmentation in order to conduct neural machine translation. Furthermore, by adapting Back Translation and Mix-Source, our NMT systems achieved the best improvement on the dataset. In the future, we will exploit more domain and multilingual information to improve the quality of the systems. Acknowledgments We would like to thank the center of High-performance computing (HPC), University of Engineering and Technology, VNU, Vietnam for allowing us to use their GPUs to perform the experiments mentioned in the paper. We also thank the anonymous reviewers for their careful reading of our paper and insightful comments.
did they collect their own data?
b9ea841b817ba23281c95c7a769873b840dee8d5
did Ġthey Ġcollect Ġtheir Ġown Ġdata ?
No
<s> did Ġthey Ġcollect Ġtheir Ġown Ġdata ? </s> Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġis Ġwidely Ġapplied Ġfor Ġmachine Ġtranslation Ġ( MT ) Ġin Ġrecent Ġyears Ġand Ġfocuses Ġon Ġpopular Ġlanguage Ġpairs Ġsuch Ġas ĠEnglish ĠIN LINE FORM 0 ĠFrench , ĠEnglish ĠIN LINE FORM 1 ĠGerman , ĠEnglish ĠIN LINE FORM 2 ĠChinese Ġor ĠEnglish ĠIN LINE FORM 3 ĠJapanese . ĠN MT Ġhas Ġobtained Ġstate - of - the - art Ġperformance Ġon Ġthose Ġlanguage Ġpairs Ġcompared Ġto Ġthe Ġtraditional Ġstatistical Ġmachine Ġtranslation Ġ( SM T ) Ġwhen Ġgiven Ġenough Ġdata ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFurthermore , Ġdue Ġto Ġthe Ġability Ġof Ġfeature Ġlearning , ĠN MT Ġsystems Ġcan Ġbe Ġtrained Ġend - to - end Ġwith Ġpure Ġparallel Ġtexts Ġand Ġminimal Ġlinguistic Ġknowledge Ġof Ġthe Ġlanguages Ġinvolved . ĠThus Ġit Ġmakes Ġtraining ĠN MT Ġfor Ġa Ġnew Ġlanguage Ġpair Ġmuch Ġeasier , Ġmore Ġscalable Ġand Ġrobust . ĠNevertheless , ĠN MT Ġhas Ġnot Ġbeen Ġemployed Ġin Ġmany Ġlow - res ourced Ġlanguage Ġpairs Ġsince Ġin Ġthose Ġscenarios , Ġdata Ġscarcity Ġoften Ġlimits Ġthe Ġlearning Ġability Ġof Ġneural Ġmethods . ĠIn Ġcontrast , Ġcomb inating Ġcomplicated Ġlinguistic - driven Ġfeatures Ġin Ġa Ġtypical Ġlog - linear Ġframework Ġstill Ġkeeps ĠSM T Ġthe Ġbest Ġapproach Ġin Ġmany Ġtranslation Ġdirections Ġbut Ġalso Ġhard Ġto Ġapply Ġto Ġnew Ġdomains Ġor Ġto Ġother Ġlanguage Ġpairs . </s> In Ġthis Ġpaper Ġwe Ġattempt Ġto Ġbuild ĠN MT Ġsystems Ġfor Ġsuch Ġa Ġlow - res ourced Ġlanguage Ġpair : ĠJapanese ĠIN LINE FORM 0 ĠVietnamese . ĠOur Ġaim Ġis Ġto Ġset Ġthe Ġfirst Ġand Ġreasonable ĠN MT Ġsystems Ġthat Ġcan Ġbe Ġreprodu cible Ġin Ġorder Ġto Ġserve Ġas Ġthe Ġbaseline Ġfor Ġfurther Ġresear ches Ġin Ġthe Ġdirection . ĠFurthermore , Ġwe Ġconduct Ġexperiments Ġusing Ġsome Ġadvanced Ġmethods Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġsystems . ĠAn Ġimportant Ġcriteria Ġfor Ġthose Ġmethods Ġis Ġthat Ġthey Ġmust Ġbe Ġscalable Ġand Ġlanguage - independent Ġas Ġmuch Ġas Ġpossible . ĠThe Ġcriteria Ġensures Ġthe Ġbasic Ġprinciple Ġof ĠN MT Ġas Ġwell Ġas Ġthe Ġreprodu c ibility Ġof Ġthe Ġsystems . ĠOn Ġthe Ġother Ġhand , Ġthe Ġmethods Ġare Ġchosen Ġin Ġthe Ġdirection Ġthat Ġthey Ġwould Ġhelp Ġalleviate Ġthe Ġdata Ġsp arsity Ġproblem Ġof ĠN MT Ġwhen Ġbeing Ġapplied Ġin Ġthis Ġlow - res ourced Ġsetting . </s> Specifically , Ġto Ġdeal Ġwith Ġrare - word Ġtranslation Ġproblems , Ġwe Ġexperiment Ġwith Ġtranslation Ġunits Ġin Ġdifferent Ġlevels : Ġsub word , Ġword Ġand Ġbeyond . ĠIn Ġmorph ological - rich Ġlanguages Ġsuch Ġas ĠEnglish Ġor ĠGerman , Ġusing Ġsub word Ġas Ġthe Ġtranslation Ġunits Ġis Ġoften Ġsuitable Ġsince Ġneural Ġmethods Ġare Ġable Ġto Ġinduce Ġmeaning Ġor Ġlinguistic Ġfunction Ġof Ġsub - comp onents Ġconstit uting Ġa Ġword . ĠByte - P air ĠEnc oding Ġ( B PE ) ĠB IB REF 4 Ġis Ġa Ġsimple Ġun super vised Ġtechnique Ġto Ġdo Ġsub word Ġsegment ation Ġand Ġit Ġhas Ġgreat Ġeffects Ġwhen Ġapplied Ġto ĠN MT Ġtraining . ĠJapanese Ġand ĠVietnamese Ġ( and Ġsome Ġother ĠAsian Ġlanguages ), Ġhowever , Ġhave Ġdifferent Ġword Ġsegment ation Ġissues . ĠHence , Ġit Ġwould Ġbe Ġdifficult Ġto Ġapply ĠB PE Ġdirectly Ġto Ġthe Ġtexts Ġin Ġthose Ġlanguages Ġand Ġbuild ĠN MT Ġsystems Ġfor Ġsub word Ġtranslation Ġwithout Ġany Ġmodification . ĠIn Ġthis Ġpaper , Ġwe Ġexperiment Ġdifferent Ġsegment ation Ġmethods Ġfor Ġboth Ġlanguages Ġand Ġalso Ġpropose Ġa Ġvariant Ġof ĠB PE Ġalgorithm Ġto Ġlearn Ġtranslation Ġunits Ġfor ĠVietnamese Ġin Ġan Ġun super vised Ġway . </s> We Ġalso Ġattempt Ġto Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata Ġby Ġusing Ġback - trans lated Ġtexts Ġor Ġmix - source Ġdata Ġjust Ġfrom Ġour Ġsmall Ġavailable Ġcorpus . ĠThose Ġdata Ġaug mentation Ġapproaches Ġhave Ġshown Ġtheir Ġeffectiveness Ġon Ġvarious ĠN MT Ġsystems , Ġespecially Ġin Ġunder - res ourced Ġscenarios . ĠWhile Ġback Ġtranslation Ġtechnique Ġis Ġused Ġto Ġgenerate Ġsynthetic Ġdata Ġfrom Ġmon oling ual Ġcorpor a , Ġmix - source Ġtechnique Ġutilizes Ġhuman - quality Ġcorpor a Ġin Ġa Ġmult ilingual Ġsetting , Ġleveraging Ġthe Ġtransfer Ġlearning Ġability Ġacross Ġlanguages . ĠBoth Ġare Ġsimple Ġbut Ġeleg antly Ġmodel Ġthe Ġrelevant Ġnoise Ġneeded Ġin Ġtraining Ġneural Ġarchitectures Ġin Ġsuch Ġlow - res ourced Ġsituations . </s> The Ġmain Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> Ne ural ĠMachine ĠTranslation </s> In Ġthis Ġsection , Ġwe Ġwill Ġdescribe Ġthe Ġgeneral Ġarchitecture Ġof ĠN MT Ġas Ġa Ġkind Ġof Ġsequence - to - sequence Ġmodeling Ġframework . ĠIn Ġthis Ġkind Ġof Ġsequence - to - sequence Ġmodeling Ġframework , Ġoften Ġthere Ġis Ġan Ġenc oder Ġtrying Ġto Ġencode Ġcontext Ġinformation Ġfrom Ġthe Ġinput Ġsequence Ġand Ġa Ġdec oder Ġto Ġgenerate Ġone Ġitem Ġof Ġthe Ġoutput Ġsequence Ġat Ġa Ġtime Ġbased Ġon Ġthe Ġcontext Ġof Ġboth Ġinput Ġand Ġoutput Ġsequences . ĠBesides , Ġan Ġadditional Ġcomponent , Ġnamed Ġattention , Ġexists Ġin Ġbetween , Ġdeciding Ġwhich Ġparts Ġof Ġthe Ġinput Ġsequence Ġthe Ġdec oder Ġshould Ġpay Ġattention Ġin Ġorder Ġto Ġchoose Ġwhich Ġto Ġoutput Ġnext . ĠIn Ġother Ġwords , Ġthis Ġattention Ġcomponent Ġcalculates Ġthe Ġcontext Ġrelevant Ġto Ġthe Ġdecision Ġof Ġthe Ġdec oder Ġat Ġthe Ġconsidering Ġtime . ĠThose Ġcomponents Ġas Ġa Ġwhole Ġconstitute Ġa Ġlarge Ġtrain able Ġneural Ġarchitecture Ġcalled Ġthe Ġfamous Ġattention - based Ġenc oder - dec oder Ġframework . ĠThis Ġframework Ġbecomes Ġpopular Ġin Ġmany Ġsequence - to - sequence Ġtasks . </s> In Ġthe Ġfield Ġof Ġmachine Ġtranslation , Ġusing Ġthe Ġattention - based Ġenc oder - dec oder Ġframework Ġis Ġreferred Ġto Ġas ĠNeural ĠMachine ĠTranslation Ġapproach . ĠFirst Ġpresented Ġin ĠB IB REF 0 Ġ, Ġthe Ġenc oder Ġand Ġdec oder Ġin ĠN MT Ġare Ġrecurrent - based , Ġwhich Ġeach Ġhidden Ġunit Ġin Ġthose Ġcomponents Ġis Ġa Ġrecurrent Ġunit Ġlike ĠLong ĠShort - term ĠMemory Ġ( L ST M ) ĠB IB REF 5 Ġor ĠG ated ĠRec urrent ĠUnit ( GR U ) ĠB IB REF 6 Ġ. ĠLater , Ġthe Ġenc oder Ġor Ġdec oder Ġcan Ġalso Ġbe Ġa Ġconv olution al Ġarchitecture , Ġas Ġin ĠB IB REF 7 Ġ. ĠRecently , ĠB IB REF 8 Ġintroduces Ġtransformer Ġarchitecture , Ġin Ġwhich Ġboth Ġthe Ġenc oder Ġand Ġdec oder Ġare Ġa Ġspecial Ġvariant Ġof Ġattention Ġmechanism , Ġcalled Ġself Ġattention . ĠIn Ġthis Ġpaper , Ġwe Ġwill Ġbriefly Ġexplain Ġthe ĠRec urrent ĠN MT Ġas Ġwe Ġutilize Ġit Ġin Ġour Ġexperiments . </s> The ĠRec urrent ĠN MT Ġmodel Ġfollows Ġthe Ġattention - based Ġarchitecture Ġproposed Ġby ĠB IB REF 0 Ġ. ĠThe Ġbid irection al Ġrecurrent Ġenc oder Ġreads Ġevery Ġwords ĠIN LINE FORM 0 Ġof Ġa Ġsource Ġsentence ĠIN LINE FORM 1 Ġand Ġenc odes Ġa Ġrepresentation ĠIN LINE FORM 2 Ġof Ġthe Ġsentence Ġinto Ġa Ġfixed - length Ġvector ĠIN LINE FORM 3 Ġconc aten ated Ġfrom Ġthose Ġof Ġthe Ġforward Ġand Ġbackward Ġdirections : ĠIN LINE FORM 4 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġone - hot Ġvector Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġword Ġembed ding Ġmatrix Ġwhich Ġis Ġshared Ġacross Ġthe Ġsource Ġwords . ĠIN LINE FORM 3 Ġis Ġthe Ġrecurrent Ġunit Ġcomputing Ġthe Ġcurrent Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġbased Ġon Ġthe Ġprevious Ġhidden Ġstate . ĠIN LINE FORM 4 Ġis Ġthen Ġcalled Ġan Ġannotation Ġvector , Ġwhich Ġrepresent Ġthe Ġinformation Ġof Ġthe Ġsource Ġsentence Ġup Ġto Ġthe Ġtime ĠIN LINE FORM 5 Ġfrom Ġboth Ġforward Ġand Ġbackward Ġdirections . </s> Those Ġannotation Ġvectors Ġof Ġthe Ġsource Ġsentences Ġare Ġcombined Ġin Ġthe Ġattention Ġlayers Ġin Ġa Ġway Ġthat Ġthe Ġresulted Ġvector Ġenc odes Ġthe Ġsource Ġcontext Ġrelevant Ġto Ġthe Ġconsidering Ġtarget Ġword Ġthe Ġdec oder Ġshould Ġproduce . ĠInt uitive ly , Ġa Ġrelevance Ġbetween Ġthe Ġprevious Ġtarget Ġword ĠIN LINE FORM 0 Ġand Ġthe Ġannotation Ġvectors ĠIN LINE FORM 1 Ġcorresponding Ġto Ġthe Ġsource Ġwords ĠIN LINE FORM 2 Ġcan Ġbe Ġused Ġto Ġform Ġsome Ġattention Ġscenario : ĠDIS PLAY FORM 0 </s> This Ġspecific Ġattention Ġmechanism , Ġoriginally Ġcalled Ġalignment Ġmodel Ġin ĠB IB REF 0 Ġ, Ġhas Ġbeen Ġemployed Ġas Ġa Ġsimple Ġfeed forward Ġnetwork Ġwith Ġthe Ġfirst Ġlayer Ġis Ġa Ġlearn able Ġlayer Ġvia Ġadaptation Ġfactors ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ. ĠThe Ġrelevance Ġscores ĠIN LINE FORM 3 Ġare Ġthen Ġnormalized Ġinto Ġattention Ġweights ĠIN LINE FORM 4 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 5 Ġis Ġcalculated Ġas Ġthe Ġweighted Ġsum Ġof Ġall Ġannotation Ġvectors ĠIN LINE FORM 6 Ġ. ĠDepending Ġon Ġhow Ġmuch Ġattention Ġthe Ġtarget Ġword Ġat Ġtime ĠIN LINE FORM 7 Ġput Ġon Ġthe Ġsource Ġstates ĠIN LINE FORM 8 Ġ, Ġa Ġsoft Ġalignment Ġis Ġlearned Ġand Ġa Ġsource Ġcontext ĠIN LINE FORM 9 Ġat Ġtime ĠIN LINE FORM 10 Ġis Ġcalculated Ġprior Ġto Ġthe Ġprediction Ġof Ġthe Ġdec oder . </s> Similar Ġto Ġthe Ġenc oder , Ġthe Ġrecurrent Ġdec oder Ġrec urs ively Ġgenerates Ġone Ġtarget Ġword ĠIN LINE FORM 0 Ġto Ġform Ġa Ġtranslated Ġtarget Ġsentence ĠIN LINE FORM 1 Ġin Ġthe Ġend . ĠAt Ġtime ĠIN LINE FORM 2 Ġ, Ġit Ġtakes Ġthe Ġprevious Ġhidden Ġstate Ġof Ġthe Ġdec oder ĠIN LINE FORM 3 Ġ, Ġthe Ġprevious Ġembedded Ġword Ġrepresentation ĠIN LINE FORM 4 Ġand Ġthe Ġtime - specific Ġcontext Ġvector ĠIN LINE FORM 5 Ġas Ġinputs Ġto Ġcalculate Ġthe Ġcurrent Ġhidden Ġstate ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 </s> Again , ĠIN LINE FORM 0 Ġis Ġthe Ġrecurrent Ġactivation Ġfunction Ġof Ġthe Ġdec oder Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġshared Ġword Ġembed ding Ġmatrix Ġof Ġthe Ġtarget Ġsentences . </s> Given Ġthe Ġparallel Ġcorpus Ġconsisting Ġof ĠIN LINE FORM 0 Ġtraining Ġexamples ĠIN LINE FORM 1 Ġ, Ġthe Ġobjective Ġis Ġto Ġmaximize Ġthe Ġconditional Ġlog - pro b ability Ġof Ġthe Ġcorrect Ġtranslation Ġgiven Ġa Ġsource Ġsentence Ġwith Ġrespect Ġto Ġthe Ġparameters ĠIN LINE FORM 2 Ġof Ġthe Ġwhole Ġmodel : ĠIN LINE FORM 3 </s> Sub word ĠTranslation </s> One Ġof Ġthe Ġmost Ġsevere Ġproblems Ġof ĠN MT Ġis Ġdealing Ġwith Ġthe Ġrare Ġwords , Ġwhich Ġare Ġnot Ġin Ġthe Ġshort Ġlists Ġof Ġthe Ġvoc ab ular ies , Ġi . e . Ġout - of - voc abulary Ġ( OO V ) Ġwords , Ġor Ġdo Ġnot Ġappear Ġin Ġthe Ġtraining Ġset Ġat Ġall . ĠOn Ġone Ġhand , Ġwe Ġwould Ġlike Ġto Ġhave Ġfewer ĠO OV Ġwords Ġby Ġincreasing Ġthe Ġsize Ġof Ġthe Ġshort Ġlists . ĠOn Ġthe Ġother Ġhand , Ġwe Ġneed Ġour Ġneural Ġnetwork Ġto Ġlearn Ġfast Ġand Ġhas Ġa Ġgood Ġgeneral ization Ġcapability Ġon Ġthe Ġunseen Ġwords Ġas Ġwell . </s> As Ġexplained Ġshortly Ġin Ġthe Ġintroduction , Ġfor Ġmany Ġlanguages , Ġusing Ġsub word Ġinstead Ġof Ġword Ġas Ġa Ġtranslation Ġunit Ġ( T U ) Ġhas Ġbeen Ġshown Ġthat Ġit Ġis Ġnot Ġonly Ġeffective Ġon Ġreducing Ġvocabulary Ġsizes , Ġthus Ġallev iating Ġthe Ġcomputing Ġburden Ġon Ġthe Ġlarge Ġsoft - max Ġlayer Ġas Ġwell Ġas Ġreducing Ġa Ġsubstantial Ġnumber Ġof Ġparameters Ġto Ġbe Ġlearnt , Ġbut Ġalso Ġhas Ġthe Ġability Ġto Ġgenerate Ġunseen Ġwords . ĠIn Ġthose Ġlanguages , Ġa Ġword Ġcan Ġbe Ġa Ġcompound Ġword Ġor Ġcomprised Ġby Ġsub - comp onents , Ġeach Ġhas Ġits Ġown Ġraw Ġmeaning Ġor Ġcontains Ġmorph ological Ġinformation . ĠSe gment ing Ġwords Ġinto Ġsub - comp onents Ġallows ĠN MT Ġto Ġlearn Ġto Ġtranslate Ġthem Ġwith Ġconsiderably Ġfewer Ġdata . ĠFor Ġexample , Ġit Ġis Ġdefinitely Ġless Ġchance Ġto Ġsee Ġthis Ġpopular ĠGerman Ġword ĠâĢ ľ W ohn ung s re in ig ung " Ġ( English Ġequival ence : ĠâĢ ľ house Ġcleaning âĢ Ŀ ) Ġthan Ġits Ġsub - comp onents ĠâĢ ľ W ohn ung âĢ Ŀ Ġ( i . e . ĠâĢ ľ house âĢ Ŀ Ġor ĠâĢ ľ flat âĢ Ŀ ) Ġand ĠâĢ ľ re in ig ung âĢ Ŀ Ġ( i . e . ĠâĢ ľ cle aning âĢ Ŀ ) Ġin Ġa Ġmiddle - sized ĠGerman - English Ġparallel Ġcorpus . ĠInstead , ĠN MT Ġcan Ġobserve Ġand Ġtranslate Ġthose Ġsub - comp onents Ġ( âĢ ľ W ohn ung âĢ Ŀ Ġand ĠâĢ ľ re in ig ung âĢ Ŀ ) Ġand Ġcombine Ġtheir Ġtranslations Ġto Ġgenerate Ġthe Ġunseen Ġwords Ġ( âĢ ľ house Ġcleaning âĢ Ŀ ). ĠThis Ġis Ġachieved Ġby Ġsegment ing Ġwords Ġinto Ġsub word Ġunits Ġusing Ġsegment ation Ġtechniques Ġin Ġthe Ġpre processing Ġphase Ġprior Ġto Ġtranslation . ĠThere Ġare Ġseveral Ġsegment ation Ġmethods ; ĠSome Ġare Ġthe Ġcomplicate Ġones Ġwhich Ġrequire Ġlinguistic Ġresources Ġor Ġhuman - crafted Ġrules . ĠThus , Ġthey Ġare Ġnot Ġlanguage - independent Ġand Ġexpensive Ġto Ġobtain Ġfor Ġlow - res ourced Ġlanguages . </s> Byte - P air ĠEnc oding , Ġotherwise , Ġis Ġa Ġsimple Ġbut Ġrobust Ġtechnique Ġto Ġdo Ġsub word Ġsegment ation . ĠSince Ġit Ġis Ġan Ġun super vised Ġand Ġfast Ġtechnique , Ġit Ġhas Ġgreat Ġeffects Ġwhen Ġapplied Ġto Ġbuild ĠN MT Ġsystems Ġfor Ġmorph ologically Ġrich Ġlanguages . ĠB PE Ġis Ġoriginally Ġproposed Ġin ĠB IB REF 9 Ġas Ġa Ġdata Ġcompression Ġtechnique Ġby Ġiter atively Ġreplaces Ġthe Ġmost Ġfrequent Ġpair Ġof Ġbytes Ġin Ġa Ġsequence Ġwith Ġa Ġsingle , Ġunused Ġbyte . ĠB IB REF 4 Ġrealized ĠG age 's Ġalgorithm Ġfor Ġword Ġsegment ation Ġby Ġmerging Ġfrequent Ġcharacters Ġinside Ġa Ġword Ġinstead Ġof Ġmerging Ġfrequent Ġpairs Ġof Ġbytes Ġin Ġa Ġwhole Ġfile Ġ( sequence Ġof Ġbytes ). ĠBeing Ġapplied Ġin Ġtranslation Ġfrom Ġand Ġto Ġmorph ological Ġrich Ġlanguages , Ġit Ġcan Ġautomatically Ġinduce Ġsub - comp onents Ġof Ġa Ġword Ġ( i . e . Ġsequence Ġof Ġcharacters ) Ġwhich Ġbear Ġsome Ġmeaning Ġor Ġmorph ological Ġfunction Ġwithout Ġknowing Ġmuch Ġabout Ġlinguistic Ġcharacteristics Ġof Ġthose Ġlanguages . ĠHence , Ġthe ĠT U Ġof Ġthe ĠN MT Ġused Ġin Ġthose Ġlanguages Ġis Ġat Ġa Ġsmaller Ġlevel Ġof Ġword , Ġi . e . Ġsub word . </s> On Ġthe Ġother Ġhand , ĠJapanese Ġand ĠVietnamese Ġare Ġdifferent Ġto Ġthose Ġlanguages Ġin Ġthe Ġway Ġof Ġhow Ġwe Ġconsider Ġa Ġword . ĠIn ĠVietnamese , Ġthe ĠT U , Ġnormally Ġconsidered Ġas Ġa Ġword , Ġwhich Ġoften Ġseparated Ġto Ġeach Ġothers Ġby Ġwhite Ġspaces , Ġis Ġnot Ġa Ġword Ġin Ġlinguistic Ġterm Ġsince Ġit Ġdoes Ġnot Ġreally Ġhave Ġits Ġown Ġmeaning . ĠTherefore , Ġapplying ĠB PE Ġto Ġsegment Ġthose ĠT Us Ġinto Ġsmaller Ġunits Ġwithout Ġany Ġmodification Ġis Ġnot Ġsuitable Ġfor ĠVietnamese . ĠIn Ġcase Ġof ĠJapanese , Ġthere Ġare Ġno Ġspace Ġto Ġseparate Ġwritten Ġtexts Ġinto ĠT Us . ĠThus , Ġfor ĠVietnamese Ġand ĠJapanese Ġas Ġwell Ġas Ġfor Ġthe Ġlanguages Ġhaving Ġsimilar Ġproblem , Ġbefore Ġapplying ĠB PE , Ġit Ġis Ġnecessary Ġto Ġhave Ġsome Ġpre processing Ġstep Ġto Ġtoken ize Ġthe Ġtexts Ġinto Ġwords . </s> V iet namese ĠToken ization </s> V iet namese ĠFrom Ġthe Ġlinguistic Ġpoint Ġof Ġview , Ġeach Ġsequence Ġof Ġcharacters Ġbetween Ġtwo Ġwhite Ġspaces Ġin ĠVietnamese Ġtexts Ġcannot Ġbe Ġconsidered Ġas Ġa Ġword Ġsince Ġit Ġdoes Ġnot Ġalways Ġhave Ġa Ġfull Ġmeaning Ġto Ġstand Ġalone . ĠFor Ġexample , Ġin Ġthe Ġsentence ĠâĢ ľ h Ã´ m Ġn ay Ġl Ãł Ġsin h Ġn h á º Ń t Ġc á » § a Ġt Ã´ i âĢ Ŀ Ġ( English Ġequival ence : ĠâĢ ľ Today Ġis Ġmy Ġbirthday âĢ Ŀ ), ĠâĢ ľ h Ã´ m âĢ Ŀ Ġand ĠâĢ ľ n ay âĢ Ŀ Ġare Ġnot Ġtwo Ġwords , Ġthey Ġtogether Ġform Ġa Ġword , Ġwhich Ġmeans ĠâĢ ľ today âĢ Ŀ . ĠNevertheless , ĠâĢ ľ h Ã´ m âĢ Ŀ Ġand ĠâĢ ľ n ay âĢ Ŀ Ġsomehow Ġstill Ġbear Ġsome Ġmeaning : ĠâĢ ľ h Ã´ m âĢ Ŀ - âĢ ľ day âĢ Ŀ , ĠâĢ ľ n ay âĢ Ŀ - âĢ ľ now âĢ Ŀ . ĠSimilarly , ĠâĢ ľ sin h âĢ Ŀ - âĢ ľ birth âĢ Ŀ Ġand ĠâĢ ľ n h á º Ń t âĢ Ŀ - âĢ ľ date âĢ Ŀ Ġalso Ġform Ġthe Ġword ĠâĢ ľ sin h Ġn h á º Ń t âĢ Ŀ - âĢ ľ birth day âĢ Ŀ Ġbut Ġthey Ġare Ġnot Ġtwo Ġdistinct Ġwords . ĠWe Ġcould Ġalso Ġcall Ġthem Ġsub words . </s> In Ġmany ĠVietnamese Ġprocessing Ġtasks Ġsuch Ġas ĠPart - of - Spe ech ĠT agging , ĠSynt ax ĠPars ing Ġor ĠCh unk ing , Ġthere Ġrequires Ġa Ġstep Ġto Ġconc aten ate Ġthose Ġsub words Ġto Ġmake Ġa Ġword Ġsince Ġin Ġthose Ġtasks , Ġword Ġis Ġnecessarily Ġconsidered Ġas Ġthe Ġsmallest Ġunit Ġto Ġbe Ġprocessed . ĠThis Ġstep Ġis Ġnormally Ġreferred Ġto Ġas Ġword Ġsegment ation . ĠThere Ġare Ġvarious Ġword Ġsegment ation Ġmethods ; ĠThe Ġbest Ġones Ġare Ġusing Ġmachine Ġlearning Ġapproaches Ġto Ġlearn Ġfrom Ġa Ġlabeled Ġcorpus . ĠIt Ġmakes Ġthe Ġtasks Ġhard Ġand Ġexpensive Ġto Ġbe Ġapplied Ġin Ġother Ġdomains . ĠFurthermore , Ġthe Ġtranslation Ġunit Ġin ĠMachine ĠTranslation Ġdoes Ġnot Ġneed Ġto Ġbe Ġa Ġword Ġbut Ġcan Ġbe Ġa Ġsub word Ġor Ġsequence Ġof Ġsub words Ġif Ġit Ġhave Ġits Ġown Ġmeaning . </s> With Ġthis Ġobservation Ġin Ġmind , Ġif Ġwe Ġconsider Ġa Ġsub word , Ġi . e . Ġthe Ġsequence Ġof Ġcharacters Ġbetween Ġtwo Ġwhite Ġspaces , Ġas Ġa Ġbyte Ġand Ġa Ġsentence Ġas Ġa Ġsequence Ġof Ġbytes , Ġwe Ġcan Ġapply Ġthe ĠB PE Ġalgorithm Ġstraight - forward : ĠWe Ġiter atively Ġfind Ġand Ġconc aten ate Ġthe Ġmost Ġfrequent Ġpair Ġof Ġsub words Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġand Ġreplace Ġit Ġby Ġan Ġunseen Ġsub words ĠIN LINE FORM 2 Ġ. ĠWe Ġdo Ġnot Ġmerge ĠIN LINE FORM 3 Ġwith ĠIN LINE FORM 4 Ġif Ġone Ġof Ġthem Ġare Ġdigits Ġor Ġpunct uations Ġor Ġother Ġspecial Ġsymbols . ĠThe ĠB PE Ġlearning Ġalgorithm Ġhas Ġan Ġarguments Ġwhich Ġis Ġthe Ġminimum Ġvalue Ġof Ġfrequency . ĠIn Ġpractice , Ġwe Ġset Ġthe Ġminimum Ġfrequency Ġis Ġ2 . ĠList ing Ġ1 Ġpresents Ġthis Ġvariant Ġof ĠB PE , Ġwhich Ġwe Ġcall ĠV NB PE . </s> my style Ġdef Ġget _ most _ fre q _ pair ( text , min _ fre q ): Ġdict Ġ= Ġ{} Ġdump Word Ġ= Ġ{ set _ of _ separ ate _ sy mb ols } Ġfor Ġline Ġin Ġtext : Ġw 1 Ġ= Ġthe _ first _ word _ in _ the _ line Ġfor Ġw 2 Ġ= Ġeach _ word _ in _ the _ line : Ġif Ġw 1 , w 2 Ġnot Ġin Ġdump Word Ġand Ġw 1 Ġis Ġnot Ġa Ġnumber : Ġdict [ w 1 , w 2 ] Ġ+= Ġ1 Ġget _ next _ pair _ in _ the _ line () Ġw 1 Ġ= Ġw 2 Ġreturn Ġ( all Ġpairs Ġhas Ġfre q Ġ> Ġmin _ fre q ) Ġdef Ġupdate _ p airs ( pair , text , codes _ file ): Ġoriginal _ word Ġ= Ġpair [ 0 ] Ġ+ Ġ" Ġ" Ġ+ Ġpair [ 1 ] Ġreplaced _ word Ġ= Ġpair [ 0 ] Ġ+ Ġ"_ " Ġ+ Ġpair [ 1 ] Ġinput _ file . replace ( rew , or w ) Ġwrite _ re placed _ word _ to _ codes _ file () Ġreturn Ġinput _ file Ġ### ĠMA IN ĠPART Ġ### Ġmin _ fre q = 2 Ġopen ( input _ file ) Ġto Ġread Ġopen ( codes _ file ) Ġand Ġ( output _ file ) Ġto Ġwrite Ġpairs Ġ= Ġget _ most _ fre q _ pair ( input _ file , min _ fre q ) Ġarrange _ p airs _ for _ dec re asing _ of _ frequency () Ġfor Ġeach Ġ( fre q , pair ) Ġin Ġpairs : Ġupdate _ p airs ( pair , input _ file , codes _ file ) Ġoutput _ file = input _ file Ġclose _ all _ files () ĠCompared Ġto Ġother Ġword Ġsegment ation Ġmethods Ġwhich Ġrequire Ġtraining Ġon Ġlabeled Ġdata , Ġour ĠV NB PE Ġis Ġa Ġsimple Ġun super vised Ġmethod , Ġalike Ġto Ġits Ġoriginal ĠB PE Ġalgorithm . ĠTable ĠT AB REF 7 Ġshows Ġthe Ġoutputs Ġof Ġone Ġdecent Ġword Ġsegment ation Ġmethod Ġand Ġour ĠV NB PE . </s> Japanese ĠToken ization </s> In Ġa ĠJapanese Ġwritten Ġtext , Ġthere Ġcould Ġbe Ġa Ġmixture Ġof Ġthree Ġdifferent Ġtypes Ġof Ġscripts : ĠChinese Ġcharacters Ġ( kan ji ) Ġand Ġthe Ġother Ġtwo Ġsyll ab ic Ġscripts : Ġh ir ag ana Ġand Ġk at ak ana . ĠEach Ġof Ġkan ji Ġcharacters Ġcan Ġbe Ġloosely Ġconsidered Ġas Ġa Ġsub word Ġthat Ġwe Ġmentioned Ġin Ġthe Ġprevious Ġsection . ĠIn Ġthe Ġmeanwhile , Ġeach Ġof Ġh ir ag ana Ġor Ġk at ak ana Ġcharacters Ġcan Ġbe Ġconsidered Ġas Ġa Ġlat in Ġcharacter Ġin ĠEnglish Ġor ĠVietnamese . ĠIn Ġaddition , Ġthere Ġis Ġno Ġspace Ġin Ġthe ĠJapanese Ġwritten Ġtexts Ġto Ġseparate Ġthe Ġcharacters , Ġeither Ġkan ji , Ġ, h ir ag ana Ġor Ġk at ak ana . ĠSo Ġwe Ġcannot Ġlearn Ġgood Ġsub words Ġfrom Ġa Ġsmall Ġcorpus Ġby Ġdirectly Ġapplying ĠB PE Ġor Ġthe Ġvariant ĠV NB PE Ġon ĠJapanese Ġwritten Ġtexts . ĠIn Ġorder Ġto Ġlearn Ġgood Ġsub words Ġand Ġwith Ġa Ġlittle Ġknowledge Ġabout ĠJapanese , Ġwe Ġdecided Ġto Ġuse ĠKy Tea ĠB IB REF 10 Ġto Ġdo ĠJapanese Ġword Ġsegment ation Ġand Ġthen Ġapply ĠS enn rich 's ĠB PE Ġon Ġthose Ġword - se gment ed Ġtexts . ĠSome Ġexamples Ġof Ġthe ĠJapanese Ġwords Ġgoing Ġthrough Ġthe Ġword Ġsegment ation Ġand ĠB PE Ġare Ġshown Ġin ĠTable ĠT AB REF 9 Ġ. </s> Data ĠAug mentation </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġdata Ġaug mentation Ġmethods Ġwe Ġuse Ġto Ġincrease Ġthe Ġamount Ġof Ġtraining Ġdata Ġin Ġorder Ġto Ġmake Ġour ĠN MT Ġsystems Ġsuffer Ġless Ġfrom Ġthe Ġlow - res ourced Ġsituation Ġin ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġtranslation . ĠAlthough ĠN MT Ġsystems Ġcan Ġpredict Ġand Ġgenerate Ġthe Ġtranslation Ġof Ġunseen Ġwords Ġon Ġtheir Ġvoc ab ular ies , Ġbut Ġthey Ġonly Ġperform Ġthis Ġwell Ġif Ġthe Ġparallel Ġcorpus Ġfor Ġtraining Ġare Ġsufficiently Ġlarge . ĠFor Ġmany Ġunder - res ourced Ġlanguages , Ġunfortunately , Ġit Ġhardly Ġpresents . ĠIn Ġreality , Ġalthough Ġthe Ġmon oling ual Ġdata Ġof ĠVietnamese Ġand ĠJapanese Ġare Ġimmensely Ġavailable Ġdue Ġto Ġthe Ġpopularity Ġof Ġtheir Ġspeakers , Ġthe Ġbilingual ĠJapanese - V iet namese Ġcorpor a Ġare Ġvery Ġlimited Ġand Ġoften Ġin Ġlow Ġquality Ġor Ġin Ġnarrowly Ġtechnical Ġdomains . ĠTherefore , Ġdata Ġaug mentation Ġmethods Ġto Ġexploit Ġmon oling ual Ġdata Ġfor ĠN MT Ġsystems Ġare Ġnecessary Ġto Ġobtain Ġmore Ġbilingual Ġdata , Ġthus Ġupgrading Ġthe Ġtranslating Ġquality . </s> Back ĠTranslation </s> One Ġof Ġthe Ġapproaches Ġto Ġleverage Ġthe Ġmon oling ual Ġdata Ġis Ġto Ġuse Ġa Ġmachine Ġtranslation Ġsystem Ġto Ġtranslate Ġthose Ġdata Ġin Ġorder Ġto Ġcreate Ġa Ġsynthetic Ġparallel Ġdata . ĠNormally , Ġthe Ġmon oling ual Ġdata Ġin Ġthe Ġtarget Ġlanguage Ġis Ġtranslated , Ġthus Ġthe Ġname Ġof Ġthe Ġmethod : ĠBack ĠTranslation ĠB IB REF 11 Ġ. </s> More Ġspecific , Ġto Ġgenerate Ġthe Ġdata Ġfor Ġan ĠX ĠIN LINE FORM 0 ĠY ĠN MT Ġsystem , Ġwe Ġuse Ġthe Ġbest ĠY ĠIN LINE FORM 1 ĠX Ġtranslation Ġsystem Ġwe Ġhave Ġto Ġtranslate Ġevery Ġsentence ĠIN LINE FORM 2 Ġin Ġthe Ġmon oling ual Ġdata Ġof Ġlanguage ĠY Ġinto Ġsentences ĠIN LINE FORM 3 Ġin Ġthe Ġsource Ġlanguage ĠX . ĠThen Ġwe Ġpair ĠIN LINE FORM 4 Ġto Ġget Ġthe Ġsynthetic Ġdata . ĠFinally , Ġoriginal Ġbilingual Ġdata Ġand Ġsynthetic Ġdata Ġare Ġmixed Ġto Ġtrain Ġour ĠN MT Ġfrom Ġthe Ġstart . </s> Back ĠTranslation Ġcan Ġimprove Ġthe Ġestimate Ġconditional Ġprobability Ġof Ġthe Ġtarget Ġword Ġon Ġthe Ġprevious Ġcontext Ġwords Ġthrough Ġadding Ġa Ġbilingual Ġdata Ġwith Ġapproximate Ġtranslations Ġto Ġthe Ġsource Ġlanguages . ĠFurthermore , Ġthe Ġsynthetic Ġdata Ġmight Ġcontain Ġsome Ġtranslation Ġnoise Ġfrom Ġthe ĠBack ĠTranslation Ġsystem , Ġand Ġif Ġthis Ġnoise Ġis Ġrelevant , Ġour ĠN MT Ġcan Ġbe Ġmore Ġrobust Ġin Ġlearning Ġhow Ġto Ġtranslate Ġnoisy Ġinputs . ĠOne Ġthe Ġother Ġhand , Ġif Ġthe Ġquality Ġof Ġthe ĠBack ĠTranslation Ġsystem Ġis Ġnot Ġadequate , Ġusing Ġthe Ġsynthesis Ġdata Ġmight Ġbring Ġadverse Ġeffects Ġto Ġour ĠN MT . </s> In Ġthis Ġpaper , Ġwe Ġsubs ample Ġan Ġamount Ġof ĠVietnamese Ġmon oling ual Ġdata Ġso Ġthat Ġwe Ġcan Ġcreate Ġa Ġsynthetic Ġcorpus Ġhaving Ġthe Ġsame Ġsize Ġwith Ġthe ĠJapanese - V iet namese Ġparallel Ġcorpus . ĠIn Ġthe Ġend , Ġthe Ġdata Ġwe Ġhave Ġis Ġdouble Ġin Ġsize Ġcompared Ġto Ġthe Ġoriginal Ġone . </s> Mix - Source ĠApproach </s> Another Ġdata Ġaug mentation Ġmethod Ġconsidered Ġuseful Ġin Ġthis Ġlow - res ourced Ġsetting Ġis Ġthe Ġmix - source Ġmethod ĠB IB REF 12 Ġ. ĠIn Ġthis Ġmethod , Ġwe Ġcan Ġutilize Ġthe Ġmon oling ual Ġdata Ġof Ġthe Ġtarget Ġlanguage Ġin Ġa Ġmult ilingual ĠN MT Ġsystem Ġby Ġmixing Ġthe Ġoriginal Ġsource Ġsentences Ġwith Ġthose Ġtarget Ġmon oling ual Ġdata . ĠThe Ġmult ilingual Ġframework Ġthen Ġuses Ġthe Ġshare Ġinformation Ġacross Ġsource Ġand Ġtarget Ġlanguages Ġto Ġimprove Ġthe Ġdecision Ġof Ġthe Ġtarget Ġwords Ġto Ġbe Ġchosen . </s> Specifically , Ġthere Ġare Ġa Ġsmall Ġparallel Ġcorpus ĠIN LINE FORM 0 Ġof Ġthe Ġlanguage Ġpair ĠX - Y Ġwhich Ġhas ĠIN LINE FORM 1 Ġsentence Ġpairs ĠIN LINE FORM 2 Ġ( ĠIN LINE FORM 3 Ġ) Ġand Ġa Ġbig Ġmon oling ual Ġcorpus ĠIN LINE FORM 4 Ġof Ġthe Ġlanguage ĠY Ġwhich Ġhas ĠIN LINE FORM 5 Ġsentences ĠIN LINE FORM 6 Ġ( ĠIN LINE FORM 7 Ġ, ĠIN LINE FORM 8 Ġ). ĠNow Ġfrom Ġthe Ġmon oling ual Ġcorpus ĠIN LINE FORM 9 Ġwe Ġcan Ġgenerate Ġa Ġparallel Ġcorpus ĠIN LINE FORM 10 Ġwhere Ġwe Ġtry Ġto Ġmodel Ġthe Ġidentical Ġtranslation ĠY - Y : ĠIN LINE FORM 11 Ġ( ĠIN LINE FORM 12 Ġ). ĠThen Ġwe Ġmix ĠIN LINE FORM 13 Ġand ĠIN LINE FORM 14 Ġto Ġget Ġa Ġparallel Ġcorpus Ġof Ġthe Ġsize ĠIN LINE FORM 15 Ġ. ĠSimilar Ġto Ġthe ĠBack ĠTranslation , Ġwe Ġsubs ample ĠIN LINE FORM 16 ĠVietnamese Ġsentence Ġpairs Ġfrom Ġthe Ġcorpus ĠIN LINE FORM 17 Ġthen Ġthe Ġsize Ġof Ġthe Ġparallel Ġdata Ġwe Ġhave Ġis Ġalso Ġdoubled . </s> To Ġlet Ġthe ĠN MT Ġknows Ġwhich Ġlanguage Ġa Ġcertain Ġsource Ġsentence Ġis Ġin Ġand Ġthen Ġcan Ġmodel Ġthe Ġlanguage Ġinformation , Ġwe Ġfollows Ġthe Ġconventions Ġfrom ĠB IB REF 12 Ġ. ĠWe Ġappend Ġlanguage Ġtags Ġto Ġevery Ġword Ġin Ġboth Ġsource Ġand Ġtarget Ġsentences Ġof Ġthe Ġmixed ĠIN LINE FORM 0 Ġcorpus Ġto Ġindicate Ġthe Ġlanguage Ġof Ġthe Ġwords . ĠThis Ġtechnique Ġshows Ġthe Ġeffectiveness Ġin Ġlow - res ourced Ġscenarios ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġour ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġis Ġsuch Ġa Ġscenario . </s> Data ĠPrepar ation </s> We Ġcollected ĠJapanese - V iet namese Ġparallel Ġdata Ġfrom ĠTED Ġtalks Ġextracted Ġfrom ĠW IT 3 's Ġcorpus ĠB IB REF 15 Ġ. ĠAfter Ġremoving Ġblank Ġand Ġduplicate Ġlines Ġwe Ġobtained Ġ106 758 Ġpairs Ġof Ġsentences . ĠThe Ġvalidation Ġset Ġused Ġin Ġall Ġexperiments Ġis Ġdev 2010 Ġand Ġthe Ġtest Ġset Ġis Ġt st 2010 . </s> The Ġdata Ġaug mentation Ġmethods Ġhas Ġbeen Ġapplied Ġonly Ġfor Ġthe ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġdirection . ĠFor ĠBack ĠTranslation , Ġwe Ġuse ĠVietnamese Ġmon oling ual Ġdata Ġfrom ĠV N ESE cor p us Ġof ĠDong Du Ġwhich Ġincludes Ġ349 578 Ġsentences . ĠWe Ġshuffle Ġthe Ġlines Ġof ĠV N ESE cor p us Ġcorpus Ġand Ġtake Ġout Ġthe Ġfirst Ġ106 758 Ġsentences Ġ( the Ġsame Ġas Ġthe Ġnumber Ġof Ġsentence Ġpairs Ġin Ġthe Ġoriginal Ġparallel Ġcorpus ). ĠFor ĠMix - Source , Ġinstead Ġof Ġusing Ġa Ġsubs am pled Ġmon oling ual Ġcorpus , Ġwe Ġuse Ġthe ĠVietnamese Ġpart Ġof Ġthe ĠJapanese - V iet namese Ġparallel Ġcorpus Ġin Ġorder Ġto Ġlearn Ġthe Ġmult ilingual Ġinformation Ġin Ġthe Ġsame Ġdomain . ĠOur Ġdatasets Ġare Ġlisted Ġin ĠTable ĠT AB REF 14 Ġ. </s> Pre processing </s> After Ġusing ĠKy Tea Ġto Ġtoken ize Ġthe ĠJapanese Ġtexts , Ġwe Ġlearn Ġand Ġapply ĠS enn rich 's ĠB PE Ġfrom Ġthe Ġtoken ized Ġtexts . ĠFor ĠVietnamese Ġtexts , Ġfirst Ġwe Ġuse ĠMoses Ġscripts Ġto Ġnormal ize Ġthe Ġtexts Ġfrom Ġdigits , Ġpunct uations Ġand Ġspecial Ġsymbols . ĠWe Ġuse Ġpy vi Ġfor ĠVietnamese Ġword Ġsegment ation Ġsince Ġit Ġis Ġone Ġof Ġthe Ġbest Ġtools Ġfor Ġthis Ġtask Ġin Ġterm Ġof Ġspeed , Ġrobust ness Ġand Ġperformance . ĠOn Ġthe Ġother Ġhand , Ġwe Ġuse V NB PE Ġas Ġan Ġalternative Ġway Ġof Ġdoing Ġword Ġsegment ation . ĠThose Ġtwo Ġapproaches Ġare Ġcompared Ġin Ġan Ġextr ins ic Ġevaluation Ġof Ġthe ĠN MT Ġsystems Ġemploying Ġthem . </s> System ĠArchitecture Ġand ĠTraining </s> We Ġimplement Ġthe Ġtranslation Ġsystems Ġusing ĠOpen N MT - py Ġframework ĠB IB REF 16 Ġ. ĠOur Ġsystem Ġarchitecture Ġincludes Ġtwo Ġbi - direction al ĠL ST M Ġlayers Ġfor Ġthe Ġenc oder Ġand Ġtwo ĠL ST M Ġlayers Ġfor Ġthe Ġdec oder , Ġeach Ġlayer Ġhas Ġthe Ġsize Ġof Ġ512 Ġhidden Ġunits . ĠThe Ġsize Ġof Ġsource Ġand Ġtarget Ġembed ding Ġlayers Ġis Ġalso Ġ512 . ĠWe Ġuse ĠAdam Ġoptim izer ĠB IB REF 17 Ġand Ġlearning Ġrate Ġan ne aling Ġscheme Ġwith Ġthe Ġinitial Ġlearning Ġrate Ġat ĠIN LINE FORM 0 Ġ. ĠWe Ġtrain Ġeach Ġsystems Ġfor Ġ15 Ġepoch s Ġwith Ġthe Ġbatch Ġsize Ġof Ġ64 . ĠThe Ġbest Ġmodel Ġin Ġterm Ġof Ġthe Ġun ig ram Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġis Ġusually Ġused Ġto Ġtranslate Ġthe Ġtest Ġset Ġwith Ġbeam Ġsize Ġof Ġ16 . ĠOther Ġsettings Ġare Ġthe Ġdefault Ġsettings Ġof ĠOpen N MT - py , Ġotherwise Ġalready Ġnoted . </s> Results </s> We Ġevaluate Ġthe Ġquality Ġof Ġtranslation Ġof Ġsystems Ġbased Ġon Ġdifferent Ġapproaches Ġmentioned Ġin Ġprevious Ġsections . ĠThe Ġmulti - BLE U Ġfrom ĠMoses Ġscripts Ġis Ġused . ĠThe Ġresults Ġhave Ġshown Ġin Ġthe ĠTable ĠT AB REF 21 Ġ. </s> Bas eline . ĠFor Ġthe Ġbaseline Ġsystems , Ġthe Ġtraining Ġdata Ġincludes ĠKy Tea - se gment ed ĠJapanese Ġtexts Ġand Ġpy vi - se gment ed ĠVietnamese Ġtexts . ĠFor Ġcomparison Ġpurpose , Ġwe Ġbuild Ġtwo Ġbaseline Ġsystems Ġfor Ġeach Ġdirection : Ġone Ġis Ġuse Ġthe Ġtraditional Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( SM T ), Ġthe Ġother Ġone Ġis Ġthe ĠN MT Ġsystem . ĠAlthough Ġour Ġtraining Ġset Ġis Ġsmall Ġbut Ġwe Ġfind Ġthat Ġthe ĠN MT Ġsystems Ġ( 2 ) Ġare Ġstill Ġmore Ġeffective Ġthan Ġthe Ġphrase - based ĠSM T Ġmodels Ġ( 1 ) Ġin Ġboth Ġtranslation Ġdirections . </s> Sub word ĠN MT . ĠWe Ġapplied ĠV NB PE Ġand ĠJP B PE Ġto Ġthe Ġbaseline 's Ġdata Ġand Ġtrained ĠN MT Ġsystems . ĠOn ĠVietnamese ĠIN LINE FORM 0 ĠJapanese , Ġwe Ġobserved Ġan Ġimprovement Ġof Ġ0 . 6 ĠB LE U Ġpoints Ġwhen Ġwe Ġused Ġour ĠV NB PE Ġ( 3 ) Ġinstead Ġof Ġthe Ġpy vi 's Ġword Ġsegment ation Ġ( 2 ). ĠFurthermore , Ġwhen Ġwe Ġtrained Ġour ĠN MT Ġmodels Ġusing Ġboth ĠB PE Ġmethods Ġ( 4 ), Ġwe Ġobtained Ġa Ġbigger Ġgain Ġof Ġ1 . 15 ĠB LE U Ġpoints . ĠThe Ġsimilar Ġimprovements Ġcan Ġbe Ġfound Ġin Ġthe ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġas Ġwell : Ġ0 . 29 ĠB LE U Ġpoints Ġbetween Ġ( 3 ) Ġand Ġ( 2 ) Ġand Ġ0 . 57 ĠB LE U Ġpoints Ġbetween Ġ( 4 ) Ġand Ġ( 3 ). ĠThis Ġdraws Ġtwo Ġconclusions : Ġ( i ), Ġdespite Ġusing Ġan Ġun super vised ĠVietnamese Ġword Ġsegment ation Ġwhich Ġis Ġfast , Ġrobust Ġand Ġdoes Ġnot Ġrequire Ġlinguistic Ġresources , Ġour ĠN MT Ġsystems Ġperformed Ġbetter Ġthan Ġthose Ġsystems Ġemploying Ġa Ġcomplicate Ġword Ġsegment ation Ġmethod , Ġ( ii ) ĠB PE Ġworks Ġsignificantly Ġwell Ġfor ĠJapanese Ġtexts Ġafter Ġwe Ġtoken ized Ġthe Ġtexts . </s> Data ĠAug mentation . ĠWe Ġuse Ġthe Ġbest Ġsystem Ġfor ĠVietnamese ĠIN LINE FORM 0 ĠJapanese , Ġwhich Ġis Ġthe ĠN MT Ġsystems Ġtrained Ġon ĠB PE - process ed Ġtexts , Ġto Ġgenerate Ġthe Ġsynthetic Ġdata Ġfor ĠJapanese ĠIN LINE FORM 1 ĠVietnamese Ġtranslation . ĠAlthough Ġwe Ġachieved Ġsome Ġgain Ġ( from Ġ9 . 04 Ġto Ġ9 . 39 ĠB LE U Ġpoints ), Ġthe Ġeffectiveness Ġof ĠBack ĠTranslation Ġis Ġnot Ġon Ġpar Ġwith Ġits Ġapplication Ġon Ġthe Ġtranslation Ġsystems Ġof Ġother Ġlanguage Ġpairs . ĠLooking Ġinto Ġthe ĠVietnamese ĠIN LINE FORM 2 ĠJapanese Ġtranslation Ġof ĠDong Du Ġcorpus Ġand Ġits ĠB LE U Ġscore , Ġwe Ġspeculate Ġthat Ġit Ġis Ġbecause Ġthe ĠVietnamese ĠIN LINE FORM 3 ĠJapanese Ġsystem Ġis Ġnot Ġgood Ġenough Ġto Ġproduce Ġreasonable Ġsynthetic Ġdata . ĠIn Ġthe Ġmeanwhile , Ġcombining ĠBack ĠTranslation Ġand ĠMix - Source Ġbrings Ġa Ġconsiderable Ġimprovements Ġof Ġ0 . 6 ĠB LE U Ġpoints Ġcompared Ġto Ġnot Ġusing Ġthem . </s> Related ĠWorks </s> Japanese - V iet namese ĠMT Ġis Ġfirst ly Ġmentioned Ġin Ġ2005 ĠB IB REF 18 Ġ. ĠThe Ġauthors Ġfocused Ġon Ġthe Ġdifference Ġfrom Ġembed ding Ġstructures Ġbetween ĠJapanese Ġand ĠVietnamese , Ġand Ġthen Ġproposed Ġrules Ġfor ĠMT Ġsystem Ġand Ġexperiment Ġon Ġvery Ġsmall Ġdataset Ġ( 714 ĠJapanese Ġembed ding Ġsentences ). ĠThis Ġapproach Ġis Ġsuitable Ġfor Ġsmall Ġsystem Ġapplied Ġin Ġa Ġspecific Ġdomain Ġor Ġlanguage , Ġbut Ġit Ġis Ġnot Ġeasily Ġextend able Ġto Ġother Ġdomains Ġor Ġlanguages Ġdue Ġto Ġthe Ġexp ens iveness Ġof Ġbuilding Ġsuch Ġrules . </s> The Ġother Ġprevious Ġwork Ġfor ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġuses ĠSM T ĠB IB REF 19 Ġ. ĠThey Ġalso Ġconducted Ġthe Ġexperiments Ġon Ġparallel Ġcorpor a Ġcollected Ġfrom ĠTED Ġtalks . ĠThey Ġused Ġphrase - based Ġand Ġtree - to - string Ġmodels Ġand Ġhave Ġshown Ġthat Ġthe ĠSM T Ġsystem Ġtrained Ġon ĠFrench ĠIN LINE FORM 1 ĠVietnamese Ġob tains Ġbetter Ġresults Ġthan Ġthe Ġsystem Ġof ĠJapanese ĠIN LINE FORM 2 ĠVietnamese Ġbecause ĠFrench Ġand ĠVietnamese Ġhave Ġmore Ġsimilarities Ġin Ġthe Ġstructures Ġof Ġsentences Ġthan Ġbetween ĠJapanese Ġand ĠVietnamese . ĠWe Ġalso Ġbuilt Ġphrase - based Ġsystems Ġon Ġthe ĠTED Ġdata Ġand Ġachieved Ġbetter ĠB LE U Ġscores Ġwhen Ġusing ĠN MT . </s> Recently , Ġsome Ġworks Ġuse Ġmon oling ual Ġdata Ġto Ġimprove Ġthe Ġaccuracy Ġof ĠN MT Ġsystems . ĠB IB REF 11 Ġhave Ġshown Ġsignificant Ġimprovements Ġby Ġusing Ġmon oling ual Ġdata Ġon Ġtarget - side Ġto Ġgenerate Ġsynthetic Ġdata Ġand Ġthen Ġadd Ġthem Ġto Ġoriginal Ġtraining Ġdata . ĠB IB REF 20 Ġhave Ġshown Ġsignificant Ġimprovements Ġby Ġ" self - learning " Ġmethod Ġto Ġgenerate Ġthe Ġtarget Ġsentences Ġbased Ġon Ġmon oling ual Ġdata Ġon Ġthe Ġsource - side Ġand Ġthen Ġcombined Ġthem Ġwith Ġoriginal Ġbilingual Ġdata Ġto Ġtrain . ĠB IB REF 12 Ġconvert Ġmon oling ual Ġcorpus Ġon Ġthe Ġtarget - side Ġinto Ġa Ġbite xt s Ġby Ġcopying Ġtarget Ġsentences Ġto Ġthe Ġsource Ġsentence Ġand Ġthen Ġcombined Ġoriginal Ġbilingual Ġdata Ġtogether Ġon Ġtraining . ĠOur Ġsystems Ġemploy Ġthose Ġapproaches Ġto Ġexploit Ġmon oling ual Ġdata Ġand Ġshow Ġthe Ġimproved Ġperformance Ġfor ĠJapanese ĠIN LINE FORM 0 ĠVietnamese Ġtranslations . </s> Conclusion </s> We Ġhas Ġbuilt Ġthe Ġfirst ĠJapanese ĠIN LINE FORM 0 ĠVietnamese ĠN MT Ġsystems Ġand Ġreleased Ġthe Ġdataset Ġas Ġwell Ġas Ġthe Ġassociated Ġtraining Ġscripts . ĠWe Ġhave Ġalso Ġshown Ġthat Ġthe Ġproposed ĠV NB PE Ġalgorithm Ġcan Ġbe Ġused Ġfor ĠVietnamese Ġword Ġsegment ation Ġin Ġorder Ġto Ġconduct Ġneural Ġmachine Ġtranslation . ĠFurthermore , Ġby Ġadapting ĠBack ĠTranslation Ġand ĠMix - Source , Ġour ĠN MT Ġsystems Ġachieved Ġthe Ġbest Ġimprovement Ġon Ġthe Ġdataset . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġexploit Ġmore Ġdomain Ġand Ġmult ilingual Ġinformation Ġto Ġimprove Ġthe Ġquality Ġof Ġthe Ġsystems . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġcenter Ġof ĠHigh - performance Ġcomputing Ġ( H PC ), ĠUniversity Ġof ĠEngineering Ġand ĠTechnology , ĠV N U , ĠVietnam Ġfor Ġallowing Ġus Ġto Ġuse Ġtheir ĠGPUs Ġto Ġperform Ġthe Ġexperiments Ġmentioned Ġin Ġthe Ġpaper . ĠWe Ġalso Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġcareful Ġreading Ġof Ġour Ġpaper Ġand Ġinsightful Ġcomments .