pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17
EXTRACTIVE
M2M Transformer
ABSTRACTIVE
We began with evaluating standard MT paradigms, i.e., PBSMT BIBREF3 and NMT BIBREF1 . As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14 . As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18 . In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8 . After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17 .
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting. We empirically confirmed the limited success of well-established solutions when restricted to in-domain data. Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3.7 BLEU points compared to a strong baseline, as summarized in Table TABREF53 . This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation. FLOAT SELECTED: Table 13: Summary of our investigation: BLEU scores of the best NMT systems at each step.
1907.03060
Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġhas Ġenabled Ġend - to - end Ġtraining Ġof Ġa Ġtranslation Ġsystem Ġwithout Ġneeding Ġto Ġdeal Ġwith Ġword Ġalign ments , Ġtranslation Ġrules , Ġand Ġcomplicated Ġdecoding Ġalgorithms , Ġwhich Ġare Ġthe Ġcharacteristics Ġof Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( P BS MT ) ĠB IB REF 3 Ġ. ĠAlthough ĠN MT Ġcan Ġbe Ġsignificantly Ġbetter Ġthan ĠPBS MT Ġin Ġresource - rich Ġscenarios , ĠPBS MT Ġperforms Ġbetter Ġin Ġlow - resource Ġscenarios ĠB IB REF 4 Ġ. ĠOnly Ġby Ġexploiting Ġcross - ling ual Ġtransfer Ġlearning Ġtechniques ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġcan Ġthe ĠN MT Ġperformance Ġapproach ĠPBS MT Ġperformance Ġin Ġlow - resource Ġscenarios . </s> However , Ġsuch Ġmethods Ġusually Ġrequire Ġan ĠN MT Ġmodel Ġtrained Ġon Ġa Ġresource - rich Ġlanguage Ġpair Ġlike ĠFrench ĠIN LINE FORM 0 ĠEnglish Ġ( parent ), Ġwhich Ġis Ġto Ġbe Ġfine - tun ed Ġfor Ġa Ġlow - resource Ġlanguage Ġpair Ġlike ĠUzbek ĠIN LINE FORM 1 ĠEnglish Ġ( child ). ĠOn Ġthe Ġother Ġhand , Ġmult ilingual Ġapproaches ĠB IB REF 8 Ġpropose Ġto Ġtrain Ġa Ġsingle Ġmodel Ġto Ġtranslate Ġmultiple Ġlanguage Ġpairs . ĠHowever , Ġthese Ġapproaches Ġare Ġeffective Ġonly Ġwhen Ġthe Ġparent Ġtarget Ġor Ġsource Ġlanguage Ġis Ġrelatively Ġresource - rich Ġlike ĠEnglish Ġ( En ). ĠFurthermore , Ġthe Ġparents Ġand Ġchildren Ġmodels Ġshould Ġbe Ġtrained Ġon Ġsimilar Ġdomains ; Ġotherwise , Ġone Ġhas Ġto Ġtake Ġinto Ġaccount Ġan Ġadditional Ġproblem Ġof Ġdomain Ġadaptation ĠB IB REF 9 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġwork Ġon Ġa Ġlingu istically Ġdistant Ġand Ġthus Ġchallenging Ġlanguage Ġpair ĠJapanese ĠIN LINE FORM 0 ĠRussian Ġ( Ja ĠIN LINE FORM 1 ĠRu ) Ġwhich Ġhas Ġonly Ġ12 k Ġlines Ġof Ġnews Ġdomain Ġparallel Ġcorpus Ġand Ġhence Ġis Ġextremely Ġresource - poor . ĠFurthermore , Ġthe Ġamount Ġof Ġindirect Ġin - domain Ġparallel Ġcorpor a , Ġi . e ., ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn , Ġare Ġalso Ġsmall . ĠAs Ġwe Ġdemonstrate Ġin ĠSection ĠSEC REF 4 Ġ, Ġthis Ġseverely Ġlimits Ġthe Ġperformance Ġof Ġprominent Ġlow - resource Ġtechniques , Ġsuch Ġas Ġmult ilingual Ġmodeling , Ġback - translation , Ġand Ġpivot - based ĠPBS MT . ĠTo Ġremedy Ġthis , Ġwe Ġpropose Ġa Ġnovel Ġmult ist age Ġfine - tun ing Ġmethod Ġfor ĠN MT Ġthat Ġcombines Ġmult ilingual Ġmodeling ĠB IB REF 8 Ġand Ġdomain Ġadaptation ĠB IB REF 9 Ġ. </s> We Ġhave Ġaddressed Ġtwo Ġimportant Ġresearch Ġquestions Ġ( R Q s ) Ġin Ġthe Ġcontext Ġof Ġextremely Ġlow - resource Ġmachine Ġtranslation Ġ( MT ) Ġand Ġour Ġexplor ations Ġhave Ġderived Ġrational Ġcontributions Ġ( CT s ) Ġas Ġfollows : </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġperform Ġsuch Ġan Ġextensive Ġevaluation Ġof Ġextremely Ġlow - resource ĠMT Ġproblem Ġand Ġpropose Ġa Ġnovel Ġmult ilingual Ġmult ist age Ġfine - tun ing Ġapproach Ġinvolving Ġmult ilingual Ġmodeling Ġand Ġdomain Ġadaptation Ġto Ġaddress Ġit . </s> Our ĠJapanese âĢĵ Russian ĠSetting </s> In Ġthis Ġpaper , Ġwe Ġdeal Ġwith ĠJa ĠIN LINE FORM 0 ĠRu Ġnews Ġtranslation . ĠThis Ġlanguage Ġpair Ġis Ġvery Ġchallenging Ġbecause Ġthe Ġlanguages Ġinvolved Ġhave Ġcompletely Ġdifferent Ġwriting Ġsystem , Ġphon ology , Ġmorphology , Ġgrammar , Ġand Ġsyntax . ĠAmong Ġvarious Ġdomains , Ġwe Ġexperimented Ġwith Ġtranslations Ġin Ġthe Ġnews Ġdomain , Ġconsidering Ġthe Ġimportance Ġof Ġsharing Ġnews Ġbetween Ġdifferent Ġlanguage Ġspeakers . ĠMoreover , Ġnews Ġdomain Ġis Ġone Ġof Ġthe Ġmost Ġchallenging Ġtasks , Ġdue Ġto Ġlarge Ġpresence Ġof Ġout - of - voc abulary Ġ( OO V ) Ġtokens Ġand Ġlong Ġsentences . ĠTo Ġestablish Ġand Ġevaluate Ġexisting Ġmethods , Ġwe Ġalso Ġinvolved ĠEnglish Ġas Ġthe Ġthird Ġlanguage . ĠAs Ġdirect Ġparallel Ġcorpor a Ġare Ġscarce , Ġinvolving Ġa Ġlanguage Ġsuch Ġas ĠEnglish Ġfor Ġpiv oting Ġis Ġquite Ġcommon ĠB IB REF 10 Ġ. </s> There Ġhas Ġbeen Ġno Ġclean Ġheld - out Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠJa ĠIN LINE FORM 1 ĠEn Ġnews Ġtranslation . ĠTherefore , Ġwe Ġmanually Ġcompiled Ġdevelopment Ġand Ġtest Ġsets Ġusing ĠNews ĠCommentary Ġdata Ġas Ġa Ġsource . ĠSince Ġthe Ġgiven ĠJa ĠIN LINE FORM 2 ĠRu Ġand ĠJa ĠIN LINE FORM 3 ĠEn Ġdata Ġshare Ġmany Ġlines Ġin Ġthe ĠJapanese Ġside , Ġwe Ġfirst Ġcompiled Ġtri - text Ġdata . ĠThen , Ġfrom Ġeach Ġline , Ġcorresponding Ġparts Ġacross Ġlanguages Ġwere Ġmanually Ġidentified , Ġand Ġun aligned Ġparts Ġwere Ġsplit Ġoff Ġinto Ġa Ġnew Ġline . ĠNote Ġthat Ġwe Ġhave Ġnever Ġmerged Ġtwo Ġor Ġmore Ġlines . ĠAs Ġa Ġresult , Ġwe Ġobtained Ġ1 , 654 Ġlines Ġof Ġdata Ġcomprising Ġtr ilingual , Ġbilingual , Ġand Ġmon oling ual Ġsegments Ġ( main ly Ġsentences ) Ġas Ġsummarized Ġin ĠTable ĠT AB REF 8 Ġ. ĠFinally , Ġfor Ġthe Ġsake Ġof Ġcompar ability , Ġwe Ġrandomly Ġchose Ġ600 Ġtr ilingual Ġsentences Ġto Ġcreate Ġa Ġtest Ġset , Ġand Ġconc aten ated Ġthe Ġrest Ġof Ġthem Ġand Ġbilingual Ġsentences Ġto Ġform Ġdevelopment Ġsets . </s> Our Ġmanually Ġaligned Ġdevelopment Ġand Ġtest Ġsets Ġare Ġpublicly Ġavailable . </s> Related ĠWork </s> k oe hn - know les : 2017 : N MT Ġshowed Ġthat ĠN MT Ġis Ġunable Ġto Ġhandle Ġlow - resource Ġlanguage Ġpairs Ġas Ġopposed Ġto ĠPBS MT . ĠTransfer Ġlearning Ġapproaches ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġwork Ġwell Ġwhen Ġa Ġlarge Ġhelping Ġparallel Ġcorpus Ġis Ġavailable . ĠThis Ġrestricts Ġone Ġof Ġthe Ġsource Ġor Ġthe Ġtarget Ġlanguages Ġto Ġbe ĠEnglish Ġwhich , Ġin Ġour Ġcase , Ġis Ġnot Ġpossible . ĠAppro aches Ġinvolving Ġbi - direction al ĠN MT Ġmodeling Ġis Ġshown Ġto Ġdrastically Ġimprove Ġlow - resource Ġtranslation ĠB IB REF 11 Ġ. ĠHowever , Ġlike Ġmost Ġother , Ġthis Ġwork Ġfocuses Ġon Ġtranslation Ġfrom Ġand Ġinto ĠEnglish . </s> Rem aining Ġoptions Ġinclude Ġ( a ) Ġun super vised ĠMT ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, Ġ( b ) Ġparallel Ġsentence Ġmining Ġfrom Ġnon - par allel Ġor Ġcomparable Ġcorpor a ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, Ġ( c ) Ġgenerating Ġpseudo - par allel Ġdata ĠB IB REF 17 Ġ, Ġand Ġ( d ) ĠMT Ġbased Ġon Ġpivot Ġlanguages ĠB IB REF 10 Ġ. ĠThe Ġlinguistic Ġdistance Ġbetween ĠJapanese Ġand ĠRussian Ġmakes Ġit Ġextremely Ġdifficult Ġto Ġlearn Ġbilingual Ġknowledge , Ġsuch Ġas Ġbilingual Ġlex icons Ġand Ġbilingual Ġword Ġembed d ings . ĠUn super vised ĠMT Ġis Ġthus Ġnot Ġpromising Ġyet , Ġdue Ġto Ġits Ġheavy Ġreliance Ġon Ġaccurate Ġbilingual Ġword Ġembed d ings . ĠNeither Ġdoes Ġparallel Ġsentence Ġmining , Ġdue Ġto Ġthe Ġdifficulty Ġof Ġobtaining Ġaccurate Ġbilingual Ġlex icons . ĠPse udo - par allel Ġdata Ġcan Ġbe Ġused Ġto Ġaugment Ġexisting Ġparallel Ġcorpor a Ġfor Ġtraining , Ġand Ġprevious Ġwork Ġhas Ġreported Ġthat Ġsuch Ġdata Ġgenerated Ġby Ġso - called Ġback - translation Ġcan Ġsubstantially Ġimprove Ġthe Ġquality Ġof ĠN MT . ĠHowever , Ġthis Ġapproach Ġrequires Ġbase ĠMT Ġsystems Ġthat Ġcan Ġgenerate Ġsomewhat Ġaccurate Ġtranslations . ĠIt Ġis Ġthus Ġinf eas ible Ġin Ġour Ġscenario , Ġbecause Ġwe Ġcan Ġobtain Ġonly Ġa Ġweak Ġsystem Ġwhich Ġis Ġthe Ġconsequence Ġof Ġan Ġextremely Ġlow - resource Ġsituation . ĠMT Ġbased Ġon Ġpivot Ġlanguages Ġrequires Ġlarge Ġin - domain Ġparallel Ġcorpor a Ġinvolving Ġthe Ġpivot Ġlanguages . ĠThis Ġtechnique Ġis Ġthus Ġinf eas ible , Ġbecause Ġthe Ġin - domain Ġparallel Ġcorpor a Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġpairs Ġare Ġalso Ġextremely Ġlimited , Ġwhereas Ġthere Ġare Ġlarge Ġparallel Ġcorpor a Ġin Ġother Ġdomains . ĠSection ĠSEC REF 4 Ġempir ically Ġconfirms Ġthe Ġlimit Ġof Ġthese Ġexisting Ġapproaches . </s> Fortunately , Ġthere Ġare Ġtwo Ġuseful Ġtransfer Ġlearning Ġsolutions Ġusing ĠN MT : Ġ( e ) Ġmult ilingual Ġmodeling Ġto Ġincorporate Ġmultiple Ġlanguage Ġpairs Ġinto Ġa Ġsingle Ġmodel ĠB IB REF 8 Ġand Ġ( f ) Ġdomain Ġadaptation Ġto Ġincorporate Ġout - of - domain Ġdata ĠB IB REF 9 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġexplore Ġa Ġnovel Ġmethod Ġinvolving Ġstep - wise Ġfine - tun ing Ġto Ġcombine Ġthese Ġtwo Ġmethods . ĠBy Ġimproving Ġthe Ġtranslation Ġquality Ġin Ġthis Ġway , Ġwe Ġcan Ġalso Ġincrease Ġthe Ġlikelihood Ġof Ġpseudo - par allel Ġdata Ġbeing Ġuseful Ġto Ġfurther Ġimprove Ġtranslation Ġquality . </s> Limit Ġof ĠUsing Ġonly ĠIn - domain ĠData </s> This Ġsection Ġanswers Ġour Ġfirst Ġresearch Ġquestion , Ġ[ R Q 1 ], Ġabout Ġthe Ġtranslation Ġquality Ġthat Ġwe Ġcan Ġachieve Ġusing Ġexisting Ġmethods Ġand Ġin - domain Ġparallel Ġand Ġmon oling ual Ġdata . ĠWe Ġthen Ġuse Ġthe Ġstrongest Ġmodel Ġto Ġconduct Ġexperiments Ġon Ġgenerating Ġand Ġutilizing Ġback - trans lated Ġpseudo - par allel Ġdata Ġfor Ġaugment ing ĠN MT . ĠOur Ġintention Ġis Ġto Ġempir ically Ġidentify Ġthe Ġmost Ġeffective Ġpractices Ġas Ġwell Ġas Ġrecognize Ġthe Ġlimitations Ġof Ġrelying Ġonly Ġon Ġin - domain Ġparallel Ġcorpor a . </s> Data </s> To Ġtrain ĠMT Ġsystems Ġamong Ġthe Ġthree Ġlanguages , Ġi . e ., ĠJapanese , ĠRussian , Ġand ĠEnglish , Ġwe Ġused Ġall Ġthe Ġparallel Ġdata Ġprovided Ġby ĠGlobal ĠVoices , Ġmore Ġspecifically Ġthose Ġavailable Ġat ĠOP US . ĠTable ĠT AB REF 9 Ġsummarizes Ġthe Ġsize Ġof Ġtrain / development / test Ġsplits Ġused Ġin Ġour Ġexperiments . ĠThe Ġnumber Ġof Ġparallel Ġsentences Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġis Ġ12 k , Ġfor ĠJa ĠIN LINE FORM 1 ĠEn Ġis Ġ47 k , Ġand Ġfor ĠRu ĠIN LINE FORM 2 ĠEn Ġis Ġ82 k . ĠNote Ġthat Ġthe Ġthree Ġcorpor a Ġare Ġnot Ġmutually Ġexclusive : Ġ9 k Ġout Ġof Ġ12 k Ġsentences Ġin Ġthe ĠJa ĠIN LINE FORM 3 ĠRu Ġcorpus Ġwere Ġalso Ġincluded Ġin Ġthe Ġother Ġtwo Ġparallel Ġcorpor a , Ġassociated Ġwith Ġidentical ĠEnglish Ġtranslations . ĠThis Ġputs Ġa Ġlimit Ġon Ġthe Ġpositive Ġimpact Ġthat Ġthe Ġhelping Ġcorpor a Ġcan Ġhave Ġon Ġthe Ġtranslation Ġquality . </s> Even Ġwhen Ġone Ġfocuses Ġon Ġlow - resource Ġlanguage Ġpairs , Ġwe Ġoften Ġhave Ġaccess Ġto Ġlarger Ġquantities Ġof Ġin - domain Ġmon oling ual Ġdata Ġof Ġeach Ġlanguage . ĠSuch Ġmon oling ual Ġdata Ġare Ġuseful Ġto Ġimprove Ġquality Ġof ĠMT , Ġfor Ġexample , Ġas Ġthe Ġsource Ġof Ġpseudo - par allel Ġdata Ġfor Ġaugment ing Ġtraining Ġdata Ġfor ĠN MT ĠB IB REF 17 Ġand Ġas Ġthe Ġtraining Ġdata Ġfor Ġlarge Ġand Ġsmoot hed Ġlanguage Ġmodels Ġfor ĠPBS MT ĠB IB REF 4 Ġ. ĠTable ĠT AB REF 13 Ġsummarizes Ġthe Ġstatistics Ġon Ġour Ġmon oling ual Ġcorpor a Ġfor Ġseveral Ġdomains Ġincluding Ġthe Ġnews Ġdomain . ĠNote Ġthat Ġwe Ġremoved Ġfrom Ġthe ĠGlobal ĠVoices Ġmon oling ual Ġcorpor a Ġthose Ġsentences Ġthat Ġare Ġalready Ġpresent Ġin Ġthe Ġparallel Ġcorpus . </s> https :// d umps . wik imedia . org / back up - index . html Ġ( 201 805 01 ) Ġhttp :// www . stat mt . org / w mt 18 / translation - task . html Ġhttps :// www . y omi uri . co . jp / database / gl oss ary / Ġhttp :// www . cs . j hu . edu / ~ ke vind uh / a / mult it arget - ted t alks / Ġhttp :// opus . nl pl . eu / T ato eb a - v 2 . php </s> We Ġtoken ized ĠEnglish Ġand ĠRussian Ġsentences Ġusing Ġtoken izer . per l Ġof ĠMoses ĠB IB REF 3 Ġ. ĠTo Ġtoken ize ĠJapanese Ġsentences , Ġwe Ġused ĠMe C ab Ġwith Ġthe ĠIPA Ġdictionary . ĠAfter Ġtoken ization , Ġwe Ġeliminated Ġdupl icated Ġsentence Ġpairs Ġand Ġsentences Ġwith Ġmore Ġthan Ġ100 Ġtokens Ġfor Ġall Ġthe Ġlanguages . </s> MT ĠMethods ĠExam ined </s> We Ġbegan Ġwith Ġevaluating Ġstandard ĠMT Ġparad ig ms , Ġi . e ., ĠPBS MT ĠB IB REF 3 Ġand ĠN MT ĠB IB REF 1 Ġ. ĠAs Ġfor ĠPBS MT , Ġwe Ġalso Ġexamined Ġtwo Ġadvanced Ġmethods : Ġpivot - based Ġtranslation Ġrelying Ġon Ġa Ġhelping Ġlanguage ĠB IB REF 10 Ġand Ġinduction Ġof Ġphrase Ġtables Ġfrom Ġmon oling ual Ġdata ĠB IB REF 14 Ġ. </s> As Ġfor ĠN MT , Ġwe Ġcompared Ġtwo Ġtypes Ġof Ġenc oder - dec oder Ġarchitectures : Ġattention al ĠR NN - based Ġmodel Ġ( RN MT ) ĠB IB REF 2 Ġand Ġthe ĠTrans former Ġmodel ĠB IB REF 18 Ġ. ĠIn Ġaddition Ġto Ġstandard Ġun i - direction al Ġmodeling , Ġto Ġcope Ġwith Ġthe Ġlow - resource Ġproblem , Ġwe Ġexamined Ġtwo Ġmulti - direction al Ġmodels : Ġbi - direction al Ġmodel ĠB IB REF 11 Ġand Ġmulti - to - multi Ġ( M 2 M ) Ġmodel ĠB IB REF 8 Ġ. </s> After Ġidentifying Ġthe Ġbest Ġmodel , Ġwe Ġalso Ġexamined Ġthe Ġusefulness Ġof Ġa Ġdata Ġaug mentation Ġmethod Ġbased Ġon Ġback - translation ĠB IB REF 17 Ġ. </s> First , Ġwe Ġbuilt Ġa ĠPBS MT Ġsystem Ġfor Ġeach Ġof Ġthe Ġsix Ġtranslation Ġdirections . ĠWe Ġobtained Ġphrase Ġtables Ġfrom Ġparallel Ġcorpus Ġusing ĠSy MG IZ A ++ Ġwith Ġthe Ġgrow - di ag - final Ġhe ur istics Ġfor Ġword Ġalignment , Ġand ĠMoses Ġfor Ġphrase Ġpair Ġextraction . ĠThen , Ġwe Ġtrained Ġa Ġbi - direction al ĠMS D Ġ( mon ot one , Ġswap , Ġand Ġdiscontin uous ) Ġlex ical ized Ġre ordering Ġmodel . ĠWe Ġalso Ġtrained Ġthree Ġ5 - gram Ġlanguage Ġmodels , Ġusing ĠKen LM Ġon Ġthe Ġfollowing Ġmon oling ual Ġdata : Ġ( 1 ) Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġdata , Ġ( 2 ) Ġthe Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġthe Ġmon oling ual Ġdata Ġfrom ĠGlobal ĠVoices , Ġand Ġ( 3 ) Ġthe Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġall Ġmon oling ual Ġdata Ġin Ġthe Ġnews Ġdomain Ġin ĠTable ĠT AB REF 13 Ġ. </s> Sub sequently , Ġusing ĠEnglish Ġas Ġthe Ġpivot Ġlanguage , Ġwe Ġexamined Ġthe Ġfollowing Ġthree Ġtypes Ġof Ġpivot - based ĠPBS MT Ġsystems ĠB IB REF 10 Ġ, ĠB IB REF 19 Ġfor Ġeach Ġof ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠRu ĠIN LINE FORM 1 ĠJa . </s> 2 - step Ġdecoding Ġusing Ġthe Ġsource - to - English Ġand ĠEnglish - to - target Ġsystems . </s> Ob tain Ġa Ġnew Ġphrase Ġtable Ġfrom Ġsynthetic Ġparallel Ġdata Ġgenerated Ġby Ġtranslating ĠEnglish Ġside Ġof Ġthe Ġtarget âĢĵ English Ġtraining Ġparallel Ġdata Ġto Ġthe Ġsource Ġlanguage Ġwith Ġthe ĠEnglish - to - source Ġsystem . </s> Comp ile Ġa Ġnew Ġphrase Ġtable Ġcombining Ġthose Ġfor Ġthe Ġsource - to - English Ġand ĠEnglish - to - target Ġsystems . </s> Among Ġthese Ġthree , Ġtri ang ulation Ġis Ġthe Ġmost Ġcomput ationally Ġexpensive Ġmethod . ĠAlthough Ġwe Ġhad Ġfiltered Ġthe Ġcomponent Ġphrase Ġtables Ġusing Ġthe Ġstatistical Ġsignificance Ġpr uning Ġmethod ĠB IB REF 20 Ġ, Ġtri ang ulation Ġcan Ġgenerate Ġan Ġenormous Ġnumber Ġof Ġphrase Ġpairs . ĠTo Ġreduce Ġthe Ġcomputational Ġcost Ġduring Ġdecoding Ġand Ġthe Ġnegative Ġeffects Ġof Ġpotentially Ġnoisy Ġphrase Ġpairs , Ġwe Ġretained Ġfor Ġeach Ġsource Ġphrase ĠIN LINE FORM 0 Ġonly Ġthe ĠIN LINE FORM 1 Ġ- best Ġtranslations ĠIN LINE FORM 2 Ġaccording Ġto Ġthe Ġforward Ġtranslation Ġprobability ĠIN LINE FORM 3 Ġcalculated Ġfrom Ġthe Ġconditional Ġprobabilities Ġin Ġthe Ġcomponent Ġmodels Ġas Ġdefined Ġin Ġut iy ama : 07 . ĠFor Ġeach Ġof Ġthe Ġretained Ġphrase Ġpairs , Ġwe Ġalso Ġcalculated Ġthe Ġbackward Ġtranslation Ġprobability , ĠIN LINE FORM 4 Ġ, Ġand Ġlex ical Ġtranslation Ġprobabilities , ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ, Ġin Ġthe Ġsame Ġmanner Ġas ĠIN LINE FORM 7 Ġ. </s> We Ġalso Ġinvestigated Ġthe Ġutility Ġof Ġrecent Ġadvances Ġin Ġun super vised ĠMT . ĠEven Ġthough Ġwe Ġbegan Ġwith Ġa Ġpublicly Ġavailable Ġimplementation Ġof Ġun super vised ĠPBS MT ĠB IB REF 13 Ġ, Ġit Ġcrashed Ġdue Ġto Ġunknown Ġreasons . ĠWe Ġtherefore Ġfollowed Ġanother Ġmethod Ġdescribed Ġin Ġmar ie : us mt - un mt . ĠInstead Ġof Ġshort ĠIN LINE FORM 0 Ġ- gram s ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġwe Ġcollected Ġa Ġset Ġof Ġphrases Ġin ĠJapanese Ġand ĠRussian Ġfrom Ġrespective Ġmon oling ual Ġdata Ġusing Ġthe Ġword 2 phrase Ġalgorithm ĠB IB REF 21 Ġ, Ġas Ġin Ġmar ie : us mt - un mt . ĠTo Ġreduce Ġthe Ġcomplexity , Ġwe Ġused Ġrandomly Ġselected Ġ10 M Ġmon oling ual Ġsentences , Ġand Ġ300 k Ġmost Ġfrequent Ġphrases Ġmade Ġof Ġwords Ġamong Ġthe Ġ300 k Ġmost Ġfrequent Ġwords . ĠFor Ġeach Ġsource Ġphrase ĠIN LINE FORM 1 Ġ, Ġwe Ġselected Ġ300 - best Ġtarget Ġphrases ĠIN LINE FORM 2 Ġaccording Ġto Ġthe Ġtranslation Ġprobability Ġas Ġin ĠD 18 - 15 49 : ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġstands Ġfor Ġa Ġbilingual Ġembed ding Ġof Ġa Ġgiven Ġphrase , Ġobtained Ġthrough Ġaveraging Ġbilingual Ġembed d ings Ġof Ġconstituent Ġwords Ġlearned Ġfrom Ġthe Ġtwo Ġmon oling ual Ġdata Ġusing Ġfast Text Ġand Ġvec map . ĠFor Ġeach Ġof Ġthe Ġretained Ġphrase Ġpair , ĠIN LINE FORM 5 Ġwas Ġcomputed Ġanalog ously . ĠWe Ġalso Ġcomputed Ġlex ical Ġtranslation Ġprobabilities Ġrelying Ġon Ġthose Ġlearned Ġfrom Ġthe Ġgiven Ġsmall Ġparallel Ġcorpus . </s> Up Ġto Ġfour Ġphrase Ġtables Ġwere Ġjointly Ġexploited Ġby Ġthe Ġmultiple Ġdecoding Ġpath Ġability Ġof ĠMoses . ĠWe ights Ġfor Ġthe Ġfeatures Ġwere Ġtuned Ġusing ĠKB - MI RA ĠB IB REF 22 Ġon Ġthe Ġdevelopment Ġset ; Ġwe Ġtook Ġthe Ġbest Ġweights Ġafter Ġ15 Ġiterations . ĠTwo Ġhyper - param eters , Ġnamely , ĠIN LINE FORM 0 Ġfor Ġthe Ġnumber Ġof Ġpivot - based Ġphrase Ġpairs Ġper Ġsource Ġphrase Ġand ĠIN LINE FORM 1 Ġfor Ġdistortion Ġlimit , Ġwere Ġdetermined Ġby Ġa Ġgrid Ġsearch Ġon ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠIn Ġcontrast , Ġwe Ġused Ġpredetermined Ġhyper - param eters Ġfor Ġphrase Ġtable Ġinduction Ġfrom Ġmon oling ual Ġdata , Ġfollowing Ġthe Ġconvention : Ġ200 Ġfor Ġthe Ġdimension Ġof Ġword Ġand Ġphrase Ġembed d ings Ġand ĠIN LINE FORM 4 Ġ. </s> We Ġused Ġthe Ġopen - source Ġimplementation Ġof Ġthe ĠRN MT Ġand Ġthe ĠTrans former Ġmodels Ġin Ġtens or 2 t ensor . ĠA Ġun i - direction al Ġmodel Ġfor Ġeach Ġof Ġthe Ġsix Ġtranslation Ġdirections Ġwas Ġtrained Ġon Ġthe Ġcorresponding Ġparallel Ġcorpus . ĠBi - direction al Ġand ĠM 2 M Ġmodels Ġwere Ġrealized Ġby Ġadding Ġan Ġartificial Ġtoken Ġthat Ġspecifies Ġthe Ġtarget Ġlanguage Ġto Ġthe Ġbeginning Ġof Ġeach Ġsource Ġsentence Ġand Ġshuff ling Ġthe Ġentire Ġtraining Ġdata ĠB IB REF 8 Ġ. </s> Table ĠT AB REF 22 Ġcontains Ġsome Ġspecific Ġhyper - param eters Ġfor Ġour Ġbaseline ĠN MT Ġmodels . ĠThe Ġhyper - param eters Ġnot Ġmentioned Ġin Ġthis Ġtable Ġused Ġthe Ġdefault Ġvalues Ġin Ġtens or 2 t ensor . ĠFor ĠM 2 M Ġsystems , Ġwe Ġover - sam pled ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠJa ĠIN LINE FORM 1 ĠEn Ġtraining Ġdata Ġso Ġthat Ġtheir Ġsizes Ġmatch Ġthe Ġlargest ĠRu ĠIN LINE FORM 2 ĠEn Ġdata . ĠTo Ġreduce Ġthe Ġnumber Ġof Ġunknown Ġwords , Ġwe Ġused Ġtens or 2 t ensor 's Ġinternal Ġsub - word Ġsegment ation Ġmechanism . ĠSince Ġwe Ġwork Ġin Ġa Ġlow - resource Ġsetting , Ġwe Ġused Ġshared Ġsub - word Ġvoc ab ular ies Ġof Ġsize Ġ16 k Ġfor Ġthe Ġun i - Ġand Ġbi - direction al Ġmodels Ġand Ġ32 k Ġfor Ġthe ĠM 2 M Ġmodels . ĠThe Ġnumber Ġof Ġtraining Ġiterations Ġwas Ġdetermined Ġby Ġearly - st opping : Ġwe Ġevaluated Ġour Ġmodels Ġon Ġthe Ġdevelopment Ġset Ġevery Ġ1 , 000 Ġupdates , Ġand Ġstopped Ġtraining Ġif ĠB LE U Ġscore Ġfor Ġthe Ġdevelopment Ġset Ġwas Ġnot Ġimproved Ġfor Ġ10 , 000 Ġupdates Ġ( 10 Ġcheck - points ). ĠNote Ġthat Ġthe Ġdevelopment Ġset Ġwas Ġcreated Ġby Ġconc aten ating Ġthose Ġfor Ġthe Ġindividual Ġtranslation Ġdirections Ġwithout Ġany Ġover - sam pling . </s> Having Ġtrained Ġthe Ġmodels , Ġwe Ġaveraged Ġthe Ġlast Ġ10 Ġcheck - points Ġand Ġdec oded Ġthe Ġtest Ġsets Ġwith Ġa Ġbeam Ġsize Ġof Ġ4 Ġand Ġa Ġlength Ġpenalty Ġwhich Ġwas Ġtuned Ġby Ġa Ġlinear Ġsearch Ġon Ġthe ĠB LE U Ġscore Ġfor Ġthe Ġdevelopment Ġset . </s> Similarly Ġto ĠPBS MT , Ġwe Ġalso Ġevaluated ĠâĢ ľ C ascade âĢ Ŀ Ġand ĠâĢ ľ Sy nt hes ize âĢ Ŀ Ġmethods Ġwith Ġun i - direction al ĠN MT Ġmodels . </s> Results </s> We Ġevaluated ĠMT Ġmodels Ġusing Ġcase - sensitive Ġand Ġtoken ized ĠB LE U ĠB IB REF 23 Ġon Ġtest Ġsets , Ġusing ĠMoses 's Ġmulti - ble u . per l . ĠStatistical Ġsignificance Ġ( ĠIN LINE FORM 0 Ġ) Ġon Ġthe Ġdifference Ġof ĠB LE U Ġscores Ġwas Ġtested Ġby ĠMoses 's Ġboot strap - hyp ot hesis - diff erence - sign ific ance . pl . </s> T ables ĠT AB REF 27 Ġand ĠT AB REF 31 Ġshow ĠB LE U Ġscores Ġof Ġall Ġthe Ġmodels , Ġexcept Ġthe ĠN MT Ġsystems Ġaugmented Ġwith Ġback - trans lations . ĠWhereas Ġsome Ġmodels Ġachieved Ġreasonable ĠB LE U Ġscores Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġtranslation , Ġall Ġthe Ġresults Ġfor ĠJa ĠIN LINE FORM 2 ĠRu , Ġwhich Ġis Ġour Ġmain Ġconcern , Ġwere Ġab ys mal . </s> Among Ġthe ĠN MT Ġmodels , ĠTrans former Ġmodels Ġ( b ĠIN LINE FORM 0 Ġ) Ġwere Ġproven Ġto Ġbe Ġbetter Ġthan ĠRN MT Ġmodels Ġ( a ĠIN LINE FORM 1 Ġ). ĠRN MT Ġmodels Ġcould Ġnot Ġeven Ġoutper form Ġthe Ġun i - direction al ĠPBS MT Ġmodels Ġ( c 1 ). ĠM 2 M Ġmodels Ġ( a 3 ) Ġand Ġ( b 3 ) Ġoutper formed Ġtheir Ġcorresponding Ġun i - Ġand Ġbi - direction al Ġmodels Ġin Ġmost Ġcases . ĠIt Ġis Ġworth Ġnoting Ġthat Ġin Ġthis Ġextremely Ġlow - resource Ġscenario , ĠB LE U Ġscores Ġof Ġthe ĠM 2 M ĠRN MT Ġmodel Ġfor Ġthe Ġlargest Ġlanguage Ġpair , Ġi . e ., ĠRu ĠIN LINE FORM 2 ĠEn , Ġwere Ġlower Ġthan Ġthose Ġof Ġthe Ġun i - Ġand Ġbi - direction al ĠRN MT Ġmodels Ġas Ġin ĠT AC L 10 81 . ĠIn Ġcontrast , Ġwith Ġthe ĠM 2 M ĠTrans former Ġmodel , ĠRu ĠIN LINE FORM 3 ĠEn Ġalso Ġbenefited Ġfrom Ġmult ilingual ism . </s> Standard ĠPBS MT Ġmodels Ġ( c 1 ) Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġun i - direction al ĠN MT Ġmodels Ġ( a 1 ) Ġand Ġ( b 1 ), Ġas Ġreported Ġby Ġk oe hn - know les : 2017 : N MT , Ġwhereas Ġthey Ġunder per form Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ). ĠAs Ġshown Ġin ĠTable ĠT AB REF 31 Ġ, Ġpivot - based ĠPBS MT Ġsystems Ġalways Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġ( c 1 ). ĠThe Ġbest Ġmodel Ġwith Ġthree Ġphrase Ġtables , Ġlabeled ĠâĢ ľ Sy nt hes ize Ġ/ ĠTri ang ulate Ġ/ ĠGold , âĢ Ŀ Ġbrought Ġvisible ĠB LE U Ġgains Ġwith Ġsubstantial Ġreduction Ġof ĠO OV Ġtokens Ġ( 30 47 ĠIN LINE FORM 0 Ġ11 80 Ġfor ĠJa ĠIN LINE FORM 1 ĠRu , Ġ4 463 ĠIN LINE FORM 2 Ġ18 12 Ġfor ĠRu ĠIN LINE FORM 3 ĠJa ). ĠHowever , Ġfurther Ġextension Ġwith Ġphrase Ġtables Ġinduced Ġfrom Ġmon oling ual Ġdata Ġdid Ġnot Ġpush Ġthe Ġlimit , Ġdespite Ġtheir Ġhigh Ġcoverage ; Ġonly Ġ336 Ġand Ġ6 77 ĠO OV Ġtokens Ġwere Ġleft Ġfor Ġthe Ġtwo Ġtranslation Ġdirections , Ġrespectively . ĠThis Ġis Ġdue Ġto Ġthe Ġpoor Ġquality Ġof Ġthe Ġbilingual Ġword Ġembed d ings Ġused Ġto Ġextract Ġthe Ġphrase Ġtable , Ġas Ġenvis aged Ġin ĠSection ĠSEC REF 3 Ġ. </s> None Ġof Ġpivot - based Ġapproaches Ġwith Ġun i - direction al ĠN MT Ġmodels Ġcould Ġeven Ġremotely Ġrival Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ). </s> Table ĠT AB REF 46 Ġshows Ġthe Ġresults Ġof Ġour Ġmult ist age Ġfine - tun ing , Ġwhere Ġthe ĠIDs Ġof Ġeach Ġrow Ġrefer Ġto Ġthose Ġdescribed Ġin ĠSection ĠSEC REF 41 Ġ. ĠFirst Ġof Ġall , Ġthe Ġfinal Ġmodels Ġof Ġour Ġmult ist age Ġfine - tun ing , Ġi . e ., ĠV Ġand ĠVII , Ġachieved Ġsignificantly Ġhigher ĠB LE U Ġscores Ġthan Ġ( b 3 ) Ġin ĠTable ĠT AB REF 27 Ġ, Ġa Ġweak Ġbaseline Ġwithout Ġusing Ġany Ġmon oling ual Ġdata , Ġand Ġ# 10 Ġin ĠTable ĠT AB REF 33 Ġ, Ġa Ġstrong Ġbaseline Ġestablished Ġwith Ġmon oling ual Ġdata . </s> The Ġperformance Ġof Ġthe Ġinitial Ġmodel Ġ( I ) Ġdepends Ġon Ġthe Ġlanguage Ġpair . ĠFor ĠJa ĠIN LINE FORM 0 ĠRu Ġpair , Ġit Ġcannot Ġachieve Ġminimum Ġlevel Ġof Ġquality Ġsince Ġthe Ġmodel Ġhas Ġnever Ġseen Ġparallel Ġdata Ġfor Ġthis Ġpair . ĠThe Ġperformance Ġon ĠJa ĠIN LINE FORM 1 ĠEn Ġpair Ġwas Ġmuch Ġlower Ġthan Ġthe Ġtwo Ġbaseline Ġmodels , Ġreflecting Ġthe Ġcrucial Ġmismatch Ġbetween Ġtraining Ġand Ġtesting Ġdomains . ĠIn Ġcontrast , ĠRu ĠIN LINE FORM 2 ĠEn Ġpair Ġbenefited Ġthe Ġmost Ġand Ġachieved Ġsurprisingly Ġhigh ĠB LE U Ġscores . ĠThe Ġreason Ġmight Ġbe Ġdue Ġto Ġthe Ġproximity Ġof Ġout - of - domain Ġtraining Ġdata Ġand Ġin - domain Ġtest Ġdata . </s> The Ġfirst Ġfine - tun ing Ġstage Ġsignificantly Ġpushed Ġup Ġthe Ġtranslation Ġquality Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġpairs , Ġin Ġboth Ġcases Ġwith Ġfine - tun ing Ġ( II ) Ġand Ġmixed Ġfine - tun ing Ġ( III ). ĠAt Ġthis Ġstage , Ġboth Ġmodels Ġperformed Ġonly Ġpoorly Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġpair Ġas Ġthey Ġhave Ġnot Ġyet Ġseen ĠJa ĠIN LINE FORM 3 ĠRu Ġparallel Ġdata . ĠEither Ġmodel Ġhad Ġa Ġconsistent Ġadvantage Ġto Ġthe Ġother . </s> When Ġthese Ġmodels Ġwere Ġfurther Ġfine - tun ed Ġonly Ġon Ġthe Ġin - domain ĠJa ĠIN LINE FORM 0 ĠRu Ġparallel Ġdata Ġ( IV Ġand ĠVI ), Ġwe Ġobtained Ġtranslations Ġof Ġbetter Ġquality Ġthan Ġthe Ġtwo Ġbas elines Ġfor ĠJa ĠIN LINE FORM 1 ĠRu Ġpair . ĠHowever , Ġas Ġa Ġresult Ġof Ġcomplete Ġignorance Ġof ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġpairs , Ġthe Ġmodels Ġonly Ġproduced Ġtranslations Ġof Ġpoor Ġquality Ġfor Ġthese Ġlanguage Ġpairs . ĠIn Ġcontrast , Ġmixed Ġfine - tun ing Ġfor Ġthe Ġsecond Ġfine - tun ing Ġstage Ġ( V Ġand ĠVII ) Ġresulted Ġin Ġconsistently Ġbetter Ġmodels Ġthan Ġconventional Ġfine - tun ing Ġ( IV Ġand ĠVI ), Ġirrespective Ġof Ġthe Ġchoice Ġat Ġthe Ġfirst Ġstage , Ġthanks Ġto Ġthe Ġgradual Ġshift Ġof Ġparameters Ġrealized Ġby Ġin - domain ĠJa ĠIN LINE FORM 4 ĠEn Ġand ĠRu ĠIN LINE FORM 5 ĠEn Ġparallel Ġdata . ĠUnfortunately , Ġthe Ġtranslation Ġquality Ġfor ĠJa ĠIN LINE FORM 6 ĠEn Ġand ĠRu ĠIN LINE FORM 7 ĠEn Ġpairs Ġsometimes Ġdegraded Ġfrom ĠII Ġand ĠIII . ĠNevertheless , Ġthe ĠB LE U Ġscores Ġstill Ġretain Ġthe Ġlarge Ġmargin Ġagainst Ġtwo Ġbas elines . </s> The Ġlast Ġthree Ġrows Ġin ĠTable ĠT AB REF 46 Ġpresent ĠB LE U Ġscores Ġobtained Ġby Ġthe Ġmethods Ġwith Ġfewer Ġfine - tun ing Ġsteps . ĠThe Ġmost Ġnaive Ġmodel ĠI ', Ġtrained Ġon Ġthe Ġbalanced Ġmixture Ġof Ġwhole Ġfive Ġtypes Ġof Ġcorpor a Ġfrom Ġscratch , Ġand Ġthe Ġmodel ĠII ', Ġobtained Ġthrough Ġa Ġsingle - step Ġconventional Ġfine - tun ing Ġof ĠI Ġon Ġall Ġthe Ġin - domain Ġdata , Ġachieved Ġonly ĠB LE U Ġscores Ġconsistently Ġworse Ġthan ĠVII . ĠIn Ġcontrast , Ġwhen Ġwe Ġmerged Ġour Ġtwo Ġfine - tun ing Ġsteps Ġinto Ġa Ġsingle Ġmixed Ġfine - tun ing Ġon ĠI , Ġwe Ġobtained Ġa Ġmodel ĠIII ' Ġwhich Ġis Ġbetter Ġfor Ġthe ĠJa ĠIN LINE FORM 0 ĠRu Ġpair Ġthan ĠVII . ĠNevertheless , Ġthey Ġare Ġstill Ġcomparable Ġto Ġthose Ġof ĠVII Ġand Ġthe ĠB LE U Ġscores Ġfor Ġthe Ġother Ġtwo Ġlanguage Ġpairs Ġare Ġmuch Ġlower Ġthan ĠVII . ĠAs Ġsuch , Ġwe Ġconclude Ġthat Ġour Ġmult ist age Ġfine - tun ing Ġleads Ġto Ġa Ġmore Ġrobust Ġin - domain Ġmult ilingual Ġmodel . </s> Aug mentation Ġwith ĠBack - translation </s> Given Ġthat Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ) Ġachieved Ġbest Ġresults Ġfor Ġmost Ġof Ġthe Ġtranslation Ġdirections Ġand Ġcompetitive Ġresults Ġfor Ġthe Ġrest , Ġwe Ġfurther Ġexplored Ġit Ġthrough Ġback - translation . </s> We Ġexamined Ġthe Ġutility Ġof Ġpseudo - par allel Ġdata Ġfor Ġall Ġthe Ġsix Ġtranslation Ġdirections , Ġunlike Ġthe Ġwork Ġof Ġlake w 2017 impro ving Ġand Ġlake w 2018 com par ison , Ġwhich Ġconcentrate Ġonly Ġon Ġthe Ġzero - shot Ġlanguage Ġpair , Ġand Ġthe Ġwork Ġof ĠW 18 - 27 10 , Ġwhich Ġcompares Ġonly Ġun i - Ġor Ġbi - direction al Ġmodels . ĠWe Ġinvestigated Ġwhether Ġeach Ġtranslation Ġdirection Ġin ĠM 2 M Ġmodels Ġwill Ġbenefit Ġfrom Ġpseudo - par allel Ġdata Ġand Ġif Ġso , Ġwhat Ġkind Ġof Ġimprovement Ġtakes Ġplace . </s> First , Ġwe Ġselected Ġsentences Ġto Ġbe Ġback - trans lated Ġfrom Ġin - domain Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 13 Ġ), Ġrelying Ġon Ġthe Ġscore Ġproposed Ġby Ġmo ore : int elligent Ġvia Ġthe Ġfollowing Ġprocedure . </s> For Ġeach Ġlanguage , Ġtrain Ġtwo Ġ4 - gram Ġlanguage Ġmodels , Ġusing ĠKen LM : Ġan Ġin - domain Ġone Ġon Ġall Ġthe ĠGlobal ĠVoices Ġdata , Ġi . e ., Ġboth Ġparallel Ġand Ġmon oling ual Ġdata , Ġand Ġa Ġgeneral - domain Ġone Ġon Ġthe Ġconc aten ation Ġof ĠGlobal ĠVoices , ĠI W SL T , Ġand ĠT ato eb a Ġdata . </s> For Ġeach Ġlanguage , Ġdiscard Ġsentences Ġcontaining ĠO OV s Ġaccording Ġto Ġthe Ġin - domain Ġlanguage Ġmodel . </s> For Ġeach Ġtranslation Ġdirection , Ġselect Ġthe ĠIN LINE FORM 0 Ġ- best Ġmon oling ual Ġsentences Ġin Ġthe Ġnews Ġdomain , Ġaccording Ġto Ġthe Ġdifference Ġbetween Ġcross - ent ropy Ġscores Ġgiven Ġby Ġthe Ġin - domain Ġand Ġgeneral - domain Ġlanguage Ġmodels . </s> Whereas ĠW 18 - 27 10 Ġexploited Ġmon oling ual Ġdata Ġmuch Ġlarger Ġthan Ġparallel Ġdata , Ġwe Ġmaintained Ġa Ġ1 : 1 Ġratio Ġbetween Ġthem ĠB IB REF 8 Ġ, Ġsetting ĠIN LINE FORM 0 Ġto Ġthe Ġnumber Ġof Ġlines Ġof Ġparallel Ġdata Ġof Ġgiven Ġlanguage Ġpair . </s> Se lected Ġmon oling ual Ġsentences Ġwere Ġthen Ġtranslated Ġusing Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ) Ġto Ġcompose Ġpseudo - par allel Ġdata . ĠThen , Ġthe Ġpseudo - par allel Ġdata Ġwere Ġenlarged Ġby Ġover - sam pling Ġas Ġsummarized Ġin ĠTable ĠT AB REF 32 Ġ. ĠFinally , Ġnew ĠN MT Ġmodels Ġwere Ġtrained Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġoriginal Ġparallel Ġand Ġpseudo - par allel Ġdata Ġfrom Ġscratch Ġin Ġthe Ġsame Ġmanner Ġas Ġthe Ġprevious ĠN MT Ġmodels Ġwith Ġthe Ġsame Ġhyper - param eters . </s> Table ĠT AB REF 33 Ġshows Ġthe ĠB LE U Ġscores Ġachieved Ġby Ġseveral Ġreasonable Ġcombinations Ġof Ġsix - way Ġpseudo - par allel Ġdata . ĠWe Ġobserved Ġthat Ġthe Ġuse Ġof Ġall Ġsix - way Ġpseudo - par allel Ġdata Ġ(# 10 ) Ġsignificantly Ġimproved Ġthe Ġbase Ġmodel Ġfor Ġall Ġthe Ġtranslation Ġdirections , Ġexcept ĠEn ĠIN LINE FORM 0 ĠRu . ĠA Ġtranslation Ġdirection Ġoften Ġbenefited Ġwhen Ġthe Ġpseudo - par allel Ġdata Ġfor Ġthat Ġspecific Ġdirection Ġwas Ġused . </s> Summary </s> We Ġhave Ġevaluated Ġan Ġextensive Ġvariation Ġof ĠMT Ġmodels Ġthat Ġrely Ġonly Ġon Ġin - domain Ġparallel Ġand Ġmon oling ual Ġdata . ĠHowever , Ġthe Ġresulting ĠB LE U Ġscores Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġand ĠRu ĠIN LINE FORM 3 ĠJa Ġtasks Ġdo Ġnot Ġexceed Ġ10 ĠB LE U Ġpoints , Ġimplying Ġthe Ġinherent Ġlimitation Ġof Ġthe Ġin - domain Ġdata Ġas Ġwell Ġas Ġthe Ġdifficulty Ġof Ġthese Ġtranslation Ġdirections . </s> Expl o iting ĠLarge ĠOut - of - Domain ĠData ĠInv olving Ġa ĠHelp ing ĠLanguage </s> The Ġlimitation Ġof Ġrelying Ġonly Ġon Ġin - domain Ġdata Ġdemonstrated Ġin ĠSection ĠSEC REF 4 Ġmotiv ates Ġus Ġto Ġexplore Ġother Ġtypes Ġof Ġparallel Ġdata . ĠAs Ġraised Ġin Ġour Ġsecond Ġresearch Ġquestion , Ġ[ R Q 2 ], Ġwe Ġconsidered Ġthe Ġeffective Ġways Ġto Ġexploit Ġout - of - domain Ġdata . </s> According Ġto Ġlanguage Ġpair Ġand Ġdomain , Ġparallel Ġdata Ġcan Ġbe Ġclassified Ġinto Ġfour Ġcategories Ġin ĠTable ĠT AB REF 40 Ġ. ĠAmong Ġall Ġthe Ġcategories , Ġout - of - domain Ġdata Ġfor Ġthe Ġlanguage Ġpair Ġof Ġinterest Ġhave Ġbeen Ġexploited Ġin Ġthe Ġdomain Ġadaptation Ġscenarios Ġ( C ĠIN LINE FORM 0 ĠA ) ĠB IB REF 9 Ġ. ĠHowever , Ġfor ĠJa ĠIN LINE FORM 1 ĠRu , Ġno Ġout - of - domain Ġdata Ġis Ġavailable . ĠTo Ġexploit Ġout - of - domain Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġpairs Ġinstead , Ġwe Ġpropose Ġa Ġmult ist age Ġfine - tun ing Ġmethod , Ġwhich Ġcombines Ġtwo Ġtypes Ġof Ġtransfer Ġlearning , Ġi . e ., Ġdomain Ġadaptation Ġfor ĠJa ĠIN LINE FORM 4 ĠEn Ġand ĠRu ĠIN LINE FORM 5 ĠEn Ġ( D ĠIN LINE FORM 6 ĠB ) Ġand Ġmult ilingual Ġtransfer Ġ( B ĠIN LINE FORM 7 ĠA ), Ġrelying Ġon Ġthe ĠM 2 M Ġmodel Ġexamined Ġin ĠSection ĠSEC REF 4 Ġ. ĠWe Ġalso Ġexamined Ġthe Ġutility Ġof Ġfine - tun ing Ġfor Ġiter atively Ġgenerating Ġand Ġusing Ġpseudo - par allel Ġdata . </s> Mult ist age ĠFine - tun ing </s> Simply Ġusing ĠN MT Ġsystems Ġtrained Ġon Ġout - of - domain Ġdata Ġfor Ġin - domain Ġtranslation Ġis Ġknown Ġto Ġperform Ġbadly . ĠIn Ġorder Ġto Ġeffectively Ġuse Ġlarge - scale Ġout - of - domain Ġdata Ġfor Ġour Ġextremely Ġlow - resource Ġtask , Ġwe Ġpropose Ġto Ġperform Ġdomain Ġadaptation Ġthrough Ġeither Ġ( a ) Ġconventional Ġfine - tun ing , Ġwhere Ġan ĠN MT Ġsystem Ġtrained Ġon Ġout - of - domain Ġdata Ġis Ġfine - tun ed Ġonly Ġon Ġin - domain Ġdata , Ġor Ġ( b ) Ġmixed Ġfine - tun ing ĠB IB REF 9 Ġ, Ġwhere Ġpre - trained Ġout - of - domain ĠN MT Ġsystem Ġis Ġfine - tun ed Ġusing Ġa Ġmixture Ġof Ġin - domain Ġand Ġout - of - domain Ġdata . ĠThe Ġsame Ġoptions Ġare Ġavailable Ġfor Ġtransferring Ġfrom ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġto ĠJa ĠIN LINE FORM 2 ĠRu . </s> We Ġinevitably Ġinvolve Ġtwo Ġtypes Ġof Ġtransfer Ġlearning , Ġi . e ., Ġdomain Ġadaptation Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġand Ġmult ilingual Ġtransfer Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġpair . ĠAmong Ġseveral Ġconceivable Ġoptions Ġfor Ġmanaging Ġthese Ġtwo Ġproblems , Ġwe Ġexamined Ġthe Ġfollowing Ġmult ist age Ġfine - tun ing . </s> Pre - train Ġa Ġmult ilingual Ġmodel Ġonly Ġon Ġthe ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġout - of - domain Ġparallel Ġdata Ġ( I ), Ġwhere Ġthe Ġvocabulary Ġof Ġthe Ġmodel Ġis Ġdetermined Ġon Ġthe Ġbasis Ġof Ġthe Ġin - domain Ġparallel Ġdata Ġin Ġthe Ġsame Ġmanner Ġas Ġthe ĠM 2 M ĠN MT Ġmodels Ġexamined Ġin ĠSection ĠSEC REF 4 Ġ. </s> Fine - t une Ġthe Ġpre - trained Ġmodel Ġ( I ) Ġon Ġthe Ġin - domain ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġparallel Ġdata Ġ( fine - tun ing , ĠII ) Ġor Ġon Ġthe Ġmixture Ġof Ġin - domain Ġand Ġout - of - domain ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġparallel Ġdata Ġ( m ixed Ġfine - tun ing , ĠIII ). </s> Further Ġfine - t une Ġthe Ġmodels Ġ( each Ġof ĠII Ġand ĠIII ) Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġon Ġin - domain Ġparallel Ġdata Ġfor Ġthis Ġlanguage Ġpair Ġonly Ġ( fine - tun ing , ĠIV Ġand ĠVI ) Ġor Ġon Ġall Ġthe Ġin - domain Ġparallel Ġdata Ġ( m ixed Ġfine - tun ing , ĠV Ġand ĠVII ). </s> We Ġchose Ġthis Ġway Ġdue Ġto Ġthe Ġfollowing Ġtwo Ġreasons . ĠFirst , Ġwe Ġneed Ġto Ġtake Ġa Ġbalance Ġbetween Ġseveral Ġdifferent Ġparallel Ġcorpor a Ġsizes . ĠThe Ġother Ġreason Ġis Ġdivision Ġof Ġlabor ; Ġwe Ġassume Ġthat Ġsolving Ġeach Ġsub - problem Ġone Ġby Ġone Ġshould Ġenable Ġgradual Ġshift Ġof Ġparameters . </s> Data ĠSelection </s> As Ġan Ġadditional Ġlarge - scale Ġout - of - domain Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 0 ĠEn , Ġwe Ġused Ġthe Ġclean est Ġ1 . 5 M Ġsentences Ġfrom Ġthe ĠAsian ĠScientific ĠPaper ĠEx cerpt ĠCorpus Ġ( AS P EC ) ĠB IB REF 24 Ġ. ĠAs Ġfor ĠRu ĠIN LINE FORM 1 ĠEn , Ġwe Ġused Ġthe ĠUN Ġand ĠY and ex Ġcorpor a Ġreleased Ġfor Ġthe ĠW MT Ġ2018 ĠNews ĠTranslation ĠTask . ĠWe Ġretained ĠRu ĠIN LINE FORM 2 ĠEn Ġsentence Ġpairs Ġthat Ġcontain Ġat Ġleast Ġone ĠO OV Ġtoken Ġin Ġboth Ġsides , Ġaccording Ġto Ġthe Ġin - domain Ġlanguage Ġmodel Ġtrained Ġin ĠSection ĠSEC REF 34 Ġ. ĠTable ĠT AB REF 45 Ġsummarizes Ġthe Ġstatistics Ġon Ġthe Ġremaining Ġout - of - domain Ġparallel Ġdata . </s> Further ĠAug mentation Ġwith ĠBack - translation </s> Having Ġobtained Ġa Ġbetter Ġmodel , Ġwe Ġexamined Ġagain Ġthe Ġutility Ġof Ġback - translation . ĠMore Ġprecisely , Ġwe Ġinvestigated Ġ( a ) Ġwhether Ġthe Ġpseudo - par allel Ġdata Ġgenerated Ġby Ġan Ġimproved ĠN MT Ġmodel Ġleads Ġto Ġa Ġfurther Ġimprovement , Ġand Ġ( b ) Ġwhether Ġone Ġmore Ġstage Ġof Ġfine - tun ing Ġon Ġthe Ġmixture Ġof Ġoriginal Ġparallel Ġand Ġpseudo - par allel Ġdata Ġwill Ġresult Ġin Ġa Ġmodel Ġbetter Ġthan Ġtraining Ġa Ġnew Ġmodel Ġfrom Ġscratch Ġas Ġexamined Ġin ĠSection ĠSEC REF 34 Ġ. </s> Given Ġan ĠN MT Ġmodel , Ġwe Ġfirst Ġgenerated Ġsix - way Ġpseudo - par allel Ġdata Ġby Ġtranslating Ġmon oling ual Ġdata . ĠFor Ġthe Ġsake Ġof Ġcompar ability , Ġwe Ġused Ġthe Ġidentical Ġmon oling ual Ġsentences Ġsampled Ġin ĠSection ĠSEC REF 34 Ġ. ĠThen , Ġwe Ġfurther Ġfine - tun ed Ġthe Ġgiven Ġmodel Ġon Ġthe Ġmixture Ġof Ġthe Ġgenerated Ġpseudo - par allel Ġdata Ġand Ġthe Ġoriginal Ġparallel Ġdata , Ġfollowing Ġthe Ġsame Ġover - sam pling Ġprocedure Ġin ĠSection ĠSEC REF 34 Ġ. ĠWe Ġrepeated Ġthese Ġsteps Ġfive Ġtimes . </s> Table ĠT AB REF 51 Ġshows Ġthe Ġresults . ĠâĢ ľ new Ġ# 10 âĢ Ŀ Ġin Ġthe Ġsecond Ġrow Ġindicates Ġan ĠM 2 M ĠTrans former Ġmodel Ġtrained Ġfrom Ġscratch Ġon Ġthe Ġmixture Ġof Ġsix - way Ġpseudo - par allel Ġdata Ġgenerated Ġby ĠVII Ġand Ġthe Ġoriginal Ġparallel Ġdata . ĠIt Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġ# 10 Ġin ĠTable ĠT AB REF 33 Ġthanks Ġto Ġthe Ġpseudo - par allel Ġdata Ġof Ġbetter Ġquality , Ġbut Ġunder per formed Ġthe Ġbase ĠN MT Ġmodel ĠVII . ĠIn Ġcontrast , Ġour Ġfine - tun ed Ġmodel ĠVIII Ġsuccessfully Ġsurpassed ĠVII , Ġand Ġone Ġmore Ġiteration Ġ( IX ) Ġfurther Ġimproved ĠB LE U Ġscores Ġfor Ġall Ġtranslation Ġdirections , Ġexcept ĠRu ĠIN LINE FORM 0 ĠEn . ĠAlthough Ġfurther Ġiterations Ġdid Ġnot Ġnecessarily Ġgain ĠB LE U Ġscores , Ġwe Ġcame Ġto Ġa Ġmuch Ġhigher Ġplateau Ġcompared Ġto Ġthe Ġresults Ġin ĠSection ĠSEC REF 4 Ġ. </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġchallenged Ġthe Ġdifficult Ġtask Ġof ĠJa ĠIN LINE FORM 0 ĠRu Ġnews Ġdomain Ġtranslation Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWe Ġempir ically Ġconfirmed Ġthe Ġlimited Ġsuccess Ġof Ġwell - established Ġsolutions Ġwhen Ġrestricted Ġto Ġin - domain Ġdata . ĠThen , Ġto Ġincorporate Ġout - of - domain Ġdata , Ġwe Ġproposed Ġa Ġmult ilingual Ġmult ist age Ġfine - tun ing Ġapproach Ġand Ġobserved Ġthat Ġit Ġsubstantially Ġimproves ĠJa ĠIN LINE FORM 1 ĠRu Ġtranslation Ġby Ġover Ġ3 . 7 ĠB LE U Ġpoints Ġcompared Ġto Ġa Ġstrong Ġbaseline , Ġas Ġsummarized Ġin ĠTable ĠT AB REF 53 Ġ. ĠThis Ġpaper Ġcontains Ġan Ġempirical Ġcomparison Ġof Ġseveral Ġexisting Ġapproaches Ġand Ġhence Ġwe Ġhope Ġthat Ġour Ġpaper Ġcan Ġact Ġas Ġa Ġguideline Ġto Ġresearchers Ġattempting Ġto Ġtackle Ġextremely Ġlow - resource Ġtranslation . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġconfirm Ġfurther Ġfine - tun ing Ġfor Ġeach Ġof Ġspecific Ġtranslation Ġdirections . ĠWe Ġwill Ġalso Ġexplore Ġthe Ġway Ġto Ġexploit Ġout - of - domain Ġpseudo - par allel Ġdata , Ġbetter Ġdomain - adapt ation Ġapproaches , Ġand Ġadditional Ġchallenging Ġlanguage Ġpairs . </s> Acknowled gments </s> This Ġwork Ġwas Ġcarried Ġout Ġwhen ĠA iz han ĠI man k ul ova Ġwas Ġtaking Ġup Ġan Ġinternship Ġat ĠN ICT , ĠJapan . ĠWe Ġwould Ġlike Ġto Ġthank Ġthe Ġreviewers Ġfor Ġtheir Ġinsightful Ġcomments . ĠA Ġpart Ġof Ġthis Ġwork Ġwas Ġconducted Ġunder Ġthe Ġprogram ĠâĢ ľ Prom otion Ġof ĠGlobal ĠCommunications ĠPlan : ĠResearch , ĠDevelopment , Ġand ĠSocial ĠDemon stration Ġof ĠMult ilingual ĠSpeech ĠTranslation ĠTechnology âĢ Ŀ Ġof Ġthe ĠMinistry Ġof ĠInternal ĠAffairs Ġand ĠCommunications Ġ( MIC ), ĠJapan .
Introduction Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3 . Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4 . Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios. However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child). On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs. However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En). Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9 . In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor. Furthermore, the amount of indirect in-domain parallel corpora, i.e., Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small. As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT. To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9 . We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows: To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it. Our Japanese–Russian Setting In this paper, we deal with Ja INLINEFORM0 Ru news translation. This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax. Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers. Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences. To establish and evaluate existing methods, we also involved English as the third language. As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10 . There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation. Therefore, we manually compiled development and test sets using News Commentary data as a source. Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data. Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line. Note that we have never merged two or more lines. As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8 . Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets. Our manually aligned development and test sets are publicly available. Related Work koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT. Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available. This restricts one of the source or the target languages to be English which, in our case, is not possible. Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11 . However, like most other, this work focuses on translation from and into English. Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10 . The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings. Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings. Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons. Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT. However, this approach requires base MT systems that can generate somewhat accurate translations. It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation. MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages. This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains. Section SECREF4 empirically confirms the limit of these existing approaches. Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9 . In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods. By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality. Limit of Using only In-domain Data This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data. We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT. Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora. Data To train MT systems among the three languages, i.e., Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS. Table TABREF9 summarizes the size of train/development/test splits used in our experiments. The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k. Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations. This puts a limit on the positive impact that the helping corpora can have on the translation quality. Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language. Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4 . Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain. Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus. https://dumps.wikimedia.org/backup-index.html (20180501) http://www.statmt.org/wmt18/translation-task.html https://www.yomiuri.co.jp/database/glossary/ http://www.cs.jhu.edu/~kevinduh/a/multitarget-tedtalks/ http://opus.nlpl.eu/Tatoeba-v2.php We tokenized English and Russian sentences using tokenizer.perl of Moses BIBREF3 . To tokenize Japanese sentences, we used MeCab with the IPA dictionary. After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages. MT Methods Examined We began with evaluating standard MT paradigms, i.e., PBSMT BIBREF3 and NMT BIBREF1 . As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14 . As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18 . In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8 . After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17 . First, we built a PBSMT system for each of the six translation directions. We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction. Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model. We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13 . Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja. 2-step decoding using the source-to-English and English-to-target systems. Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system. Compile a new phrase table combining those for the source-to-English and English-to-target systems. Among these three, triangulation is the most computationally expensive method. Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs. To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07. For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7 . We also investigated the utility of recent advances in unsupervised MT. Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons. We therefore followed another method described in marie:usmt-unmt. Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt. To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words. For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap. For each of the retained phrase pair, INLINEFORM5 was computed analogously. We also computed lexical translation probabilities relying on those learned from the given small parallel corpus. Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses. Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations. Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3 . In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4 . We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor. A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus. Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8 . Table TABREF22 contains some specific hyper-parameters for our baseline NMT models. The hyper-parameters not mentioned in this table used the default values in tensor2tensor. For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data. To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism. Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models. The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points). Note that the development set was created by concatenating those for the individual translation directions without any over-sampling. Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set. Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models. Results We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu.perl. Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance.pl. Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations. Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal. Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 ). RNMT models could not even outperform the uni-directional PBSMT models (c1). M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases. It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i.e., Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081. In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism. Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3). As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1). The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja). However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively. This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3 . None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3). Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41 . First of all, the final models of our multistage fine-tuning, i.e., V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data. The performance of the initial model (I) depends on the language pair. For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair. The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains. In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores. The reason might be due to the proximity of out-of-domain training data and in-domain test data. The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III). At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data. Either model had a consistent advantage to the other. When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair. However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs. In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data. Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III. Nevertheless, the BLEU scores still retain the large margin against two baselines. The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps. The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII. In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII. Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII. As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model. Augmentation with Back-translation Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation. We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models. We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place. First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure. For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i.e., both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data. For each language, discard sentences containing OOVs according to the in-domain language model. For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models. Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair. Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data. Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32 . Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters. Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data. We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru. A translation direction often benefited when the pseudo-parallel data for that specific direction was used. Summary We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data. However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions. Exploiting Large Out-of-Domain Data Involving a Helping Language The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data. As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data. According to language pair and domain, parallel data can be classified into four categories in Table TABREF40 . Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9 . However, for Ja INLINEFORM1 Ru, no out-of-domain data is available. To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i.e., domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4 . We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data. Multistage Fine-tuning Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly. In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data. The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru. We inevitably involve two types of transfer learning, i.e., domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair. Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning. Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4 . Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III). Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII). We chose this way due to the following two reasons. First, we need to take a balance between several different parallel corpora sizes. The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters. Data Selection As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1.5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24 . As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task. We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34 . Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data. Further Augmentation with Back-translation Having obtained a better model, we examined again the utility of back-translation. More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34 . Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data. For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34 . Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34 . We repeated these steps five times. Table TABREF51 shows the results. “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data. It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII. In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En. Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4 . Conclusion In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting. We empirically confirmed the limited success of well-established solutions when restricted to in-domain data. Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3.7 BLEU points compared to a strong baseline, as summarized in Table TABREF53 . This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation. In the future, we plan to confirm further fine-tuning for each of specific translation directions. We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs. Acknowledgments This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan. We would like to thank the reviewers for their insightful comments. A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan.
what was the baseline?
761de1610e934189850e8fda707dc5239dd58092
what Ġwas Ġthe Ġbaseline ?
p ivot - based Ġtranslation Ġrelying Ġon Ġa Ġhelping Ġlanguage ĠB IB REF 10 , Ġn duction Ġof Ġphrase Ġtables Ġfrom Ġmon oling ual Ġdata ĠB IB REF 14 Ġ, Ġattention al ĠR NN - based Ġmodel Ġ( RN MT ) ĠB IB REF 2 , ĠTrans former Ġmodel ĠB IB REF 18 , Ġbi - direction al Ġmodel ĠB IB REF 11 , Ġmulti - to - multi Ġ( M 2 M ) Ġmodel ĠB IB REF 8 , Ġback - translation ĠB IB REF 17
<s> what Ġwas Ġthe Ġbaseline ? </s> Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġhas Ġenabled Ġend - to - end Ġtraining Ġof Ġa Ġtranslation Ġsystem Ġwithout Ġneeding Ġto Ġdeal Ġwith Ġword Ġalign ments , Ġtranslation Ġrules , Ġand Ġcomplicated Ġdecoding Ġalgorithms , Ġwhich Ġare Ġthe Ġcharacteristics Ġof Ġphrase - based Ġstatistical Ġmachine Ġtranslation Ġ( P BS MT ) ĠB IB REF 3 Ġ. ĠAlthough ĠN MT Ġcan Ġbe Ġsignificantly Ġbetter Ġthan ĠPBS MT Ġin Ġresource - rich Ġscenarios , ĠPBS MT Ġperforms Ġbetter Ġin Ġlow - resource Ġscenarios ĠB IB REF 4 Ġ. ĠOnly Ġby Ġexploiting Ġcross - ling ual Ġtransfer Ġlearning Ġtechniques ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, Ġcan Ġthe ĠN MT Ġperformance Ġapproach ĠPBS MT Ġperformance Ġin Ġlow - resource Ġscenarios . </s> However , Ġsuch Ġmethods Ġusually Ġrequire Ġan ĠN MT Ġmodel Ġtrained Ġon Ġa Ġresource - rich Ġlanguage Ġpair Ġlike ĠFrench ĠIN LINE FORM 0 ĠEnglish Ġ( parent ), Ġwhich Ġis Ġto Ġbe Ġfine - tun ed Ġfor Ġa Ġlow - resource Ġlanguage Ġpair Ġlike ĠUzbek ĠIN LINE FORM 1 ĠEnglish Ġ( child ). ĠOn Ġthe Ġother Ġhand , Ġmult ilingual Ġapproaches ĠB IB REF 8 Ġpropose Ġto Ġtrain Ġa Ġsingle Ġmodel Ġto Ġtranslate Ġmultiple Ġlanguage Ġpairs . ĠHowever , Ġthese Ġapproaches Ġare Ġeffective Ġonly Ġwhen Ġthe Ġparent Ġtarget Ġor Ġsource Ġlanguage Ġis Ġrelatively Ġresource - rich Ġlike ĠEnglish Ġ( En ). ĠFurthermore , Ġthe Ġparents Ġand Ġchildren Ġmodels Ġshould Ġbe Ġtrained Ġon Ġsimilar Ġdomains ; Ġotherwise , Ġone Ġhas Ġto Ġtake Ġinto Ġaccount Ġan Ġadditional Ġproblem Ġof Ġdomain Ġadaptation ĠB IB REF 9 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġwork Ġon Ġa Ġlingu istically Ġdistant Ġand Ġthus Ġchallenging Ġlanguage Ġpair ĠJapanese ĠIN LINE FORM 0 ĠRussian Ġ( Ja ĠIN LINE FORM 1 ĠRu ) Ġwhich Ġhas Ġonly Ġ12 k Ġlines Ġof Ġnews Ġdomain Ġparallel Ġcorpus Ġand Ġhence Ġis Ġextremely Ġresource - poor . ĠFurthermore , Ġthe Ġamount Ġof Ġindirect Ġin - domain Ġparallel Ġcorpor a , Ġi . e ., ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn , Ġare Ġalso Ġsmall . ĠAs Ġwe Ġdemonstrate Ġin ĠSection ĠSEC REF 4 Ġ, Ġthis Ġseverely Ġlimits Ġthe Ġperformance Ġof Ġprominent Ġlow - resource Ġtechniques , Ġsuch Ġas Ġmult ilingual Ġmodeling , Ġback - translation , Ġand Ġpivot - based ĠPBS MT . ĠTo Ġremedy Ġthis , Ġwe Ġpropose Ġa Ġnovel Ġmult ist age Ġfine - tun ing Ġmethod Ġfor ĠN MT Ġthat Ġcombines Ġmult ilingual Ġmodeling ĠB IB REF 8 Ġand Ġdomain Ġadaptation ĠB IB REF 9 Ġ. </s> We Ġhave Ġaddressed Ġtwo Ġimportant Ġresearch Ġquestions Ġ( R Q s ) Ġin Ġthe Ġcontext Ġof Ġextremely Ġlow - resource Ġmachine Ġtranslation Ġ( MT ) Ġand Ġour Ġexplor ations Ġhave Ġderived Ġrational Ġcontributions Ġ( CT s ) Ġas Ġfollows : </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġthe Ġfirst Ġto Ġperform Ġsuch Ġan Ġextensive Ġevaluation Ġof Ġextremely Ġlow - resource ĠMT Ġproblem Ġand Ġpropose Ġa Ġnovel Ġmult ilingual Ġmult ist age Ġfine - tun ing Ġapproach Ġinvolving Ġmult ilingual Ġmodeling Ġand Ġdomain Ġadaptation Ġto Ġaddress Ġit . </s> Our ĠJapanese âĢĵ Russian ĠSetting </s> In Ġthis Ġpaper , Ġwe Ġdeal Ġwith ĠJa ĠIN LINE FORM 0 ĠRu Ġnews Ġtranslation . ĠThis Ġlanguage Ġpair Ġis Ġvery Ġchallenging Ġbecause Ġthe Ġlanguages Ġinvolved Ġhave Ġcompletely Ġdifferent Ġwriting Ġsystem , Ġphon ology , Ġmorphology , Ġgrammar , Ġand Ġsyntax . ĠAmong Ġvarious Ġdomains , Ġwe Ġexperimented Ġwith Ġtranslations Ġin Ġthe Ġnews Ġdomain , Ġconsidering Ġthe Ġimportance Ġof Ġsharing Ġnews Ġbetween Ġdifferent Ġlanguage Ġspeakers . ĠMoreover , Ġnews Ġdomain Ġis Ġone Ġof Ġthe Ġmost Ġchallenging Ġtasks , Ġdue Ġto Ġlarge Ġpresence Ġof Ġout - of - voc abulary Ġ( OO V ) Ġtokens Ġand Ġlong Ġsentences . ĠTo Ġestablish Ġand Ġevaluate Ġexisting Ġmethods , Ġwe Ġalso Ġinvolved ĠEnglish Ġas Ġthe Ġthird Ġlanguage . ĠAs Ġdirect Ġparallel Ġcorpor a Ġare Ġscarce , Ġinvolving Ġa Ġlanguage Ġsuch Ġas ĠEnglish Ġfor Ġpiv oting Ġis Ġquite Ġcommon ĠB IB REF 10 Ġ. </s> There Ġhas Ġbeen Ġno Ġclean Ġheld - out Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠJa ĠIN LINE FORM 1 ĠEn Ġnews Ġtranslation . ĠTherefore , Ġwe Ġmanually Ġcompiled Ġdevelopment Ġand Ġtest Ġsets Ġusing ĠNews ĠCommentary Ġdata Ġas Ġa Ġsource . ĠSince Ġthe Ġgiven ĠJa ĠIN LINE FORM 2 ĠRu Ġand ĠJa ĠIN LINE FORM 3 ĠEn Ġdata Ġshare Ġmany Ġlines Ġin Ġthe ĠJapanese Ġside , Ġwe Ġfirst Ġcompiled Ġtri - text Ġdata . ĠThen , Ġfrom Ġeach Ġline , Ġcorresponding Ġparts Ġacross Ġlanguages Ġwere Ġmanually Ġidentified , Ġand Ġun aligned Ġparts Ġwere Ġsplit Ġoff Ġinto Ġa Ġnew Ġline . ĠNote Ġthat Ġwe Ġhave Ġnever Ġmerged Ġtwo Ġor Ġmore Ġlines . ĠAs Ġa Ġresult , Ġwe Ġobtained Ġ1 , 654 Ġlines Ġof Ġdata Ġcomprising Ġtr ilingual , Ġbilingual , Ġand Ġmon oling ual Ġsegments Ġ( main ly Ġsentences ) Ġas Ġsummarized Ġin ĠTable ĠT AB REF 8 Ġ. ĠFinally , Ġfor Ġthe Ġsake Ġof Ġcompar ability , Ġwe Ġrandomly Ġchose Ġ600 Ġtr ilingual Ġsentences Ġto Ġcreate Ġa Ġtest Ġset , Ġand Ġconc aten ated Ġthe Ġrest Ġof Ġthem Ġand Ġbilingual Ġsentences Ġto Ġform Ġdevelopment Ġsets . </s> Our Ġmanually Ġaligned Ġdevelopment Ġand Ġtest Ġsets Ġare Ġpublicly Ġavailable . </s> Related ĠWork </s> k oe hn - know les : 2017 : N MT Ġshowed Ġthat ĠN MT Ġis Ġunable Ġto Ġhandle Ġlow - resource Ġlanguage Ġpairs Ġas Ġopposed Ġto ĠPBS MT . ĠTransfer Ġlearning Ġapproaches ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġwork Ġwell Ġwhen Ġa Ġlarge Ġhelping Ġparallel Ġcorpus Ġis Ġavailable . ĠThis Ġrestricts Ġone Ġof Ġthe Ġsource Ġor Ġthe Ġtarget Ġlanguages Ġto Ġbe ĠEnglish Ġwhich , Ġin Ġour Ġcase , Ġis Ġnot Ġpossible . ĠAppro aches Ġinvolving Ġbi - direction al ĠN MT Ġmodeling Ġis Ġshown Ġto Ġdrastically Ġimprove Ġlow - resource Ġtranslation ĠB IB REF 11 Ġ. ĠHowever , Ġlike Ġmost Ġother , Ġthis Ġwork Ġfocuses Ġon Ġtranslation Ġfrom Ġand Ġinto ĠEnglish . </s> Rem aining Ġoptions Ġinclude Ġ( a ) Ġun super vised ĠMT ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, Ġ( b ) Ġparallel Ġsentence Ġmining Ġfrom Ġnon - par allel Ġor Ġcomparable Ġcorpor a ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, Ġ( c ) Ġgenerating Ġpseudo - par allel Ġdata ĠB IB REF 17 Ġ, Ġand Ġ( d ) ĠMT Ġbased Ġon Ġpivot Ġlanguages ĠB IB REF 10 Ġ. ĠThe Ġlinguistic Ġdistance Ġbetween ĠJapanese Ġand ĠRussian Ġmakes Ġit Ġextremely Ġdifficult Ġto Ġlearn Ġbilingual Ġknowledge , Ġsuch Ġas Ġbilingual Ġlex icons Ġand Ġbilingual Ġword Ġembed d ings . ĠUn super vised ĠMT Ġis Ġthus Ġnot Ġpromising Ġyet , Ġdue Ġto Ġits Ġheavy Ġreliance Ġon Ġaccurate Ġbilingual Ġword Ġembed d ings . ĠNeither Ġdoes Ġparallel Ġsentence Ġmining , Ġdue Ġto Ġthe Ġdifficulty Ġof Ġobtaining Ġaccurate Ġbilingual Ġlex icons . ĠPse udo - par allel Ġdata Ġcan Ġbe Ġused Ġto Ġaugment Ġexisting Ġparallel Ġcorpor a Ġfor Ġtraining , Ġand Ġprevious Ġwork Ġhas Ġreported Ġthat Ġsuch Ġdata Ġgenerated Ġby Ġso - called Ġback - translation Ġcan Ġsubstantially Ġimprove Ġthe Ġquality Ġof ĠN MT . ĠHowever , Ġthis Ġapproach Ġrequires Ġbase ĠMT Ġsystems Ġthat Ġcan Ġgenerate Ġsomewhat Ġaccurate Ġtranslations . ĠIt Ġis Ġthus Ġinf eas ible Ġin Ġour Ġscenario , Ġbecause Ġwe Ġcan Ġobtain Ġonly Ġa Ġweak Ġsystem Ġwhich Ġis Ġthe Ġconsequence Ġof Ġan Ġextremely Ġlow - resource Ġsituation . ĠMT Ġbased Ġon Ġpivot Ġlanguages Ġrequires Ġlarge Ġin - domain Ġparallel Ġcorpor a Ġinvolving Ġthe Ġpivot Ġlanguages . ĠThis Ġtechnique Ġis Ġthus Ġinf eas ible , Ġbecause Ġthe Ġin - domain Ġparallel Ġcorpor a Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġpairs Ġare Ġalso Ġextremely Ġlimited , Ġwhereas Ġthere Ġare Ġlarge Ġparallel Ġcorpor a Ġin Ġother Ġdomains . ĠSection ĠSEC REF 4 Ġempir ically Ġconfirms Ġthe Ġlimit Ġof Ġthese Ġexisting Ġapproaches . </s> Fortunately , Ġthere Ġare Ġtwo Ġuseful Ġtransfer Ġlearning Ġsolutions Ġusing ĠN MT : Ġ( e ) Ġmult ilingual Ġmodeling Ġto Ġincorporate Ġmultiple Ġlanguage Ġpairs Ġinto Ġa Ġsingle Ġmodel ĠB IB REF 8 Ġand Ġ( f ) Ġdomain Ġadaptation Ġto Ġincorporate Ġout - of - domain Ġdata ĠB IB REF 9 Ġ. ĠIn Ġthis Ġpaper , Ġwe Ġexplore Ġa Ġnovel Ġmethod Ġinvolving Ġstep - wise Ġfine - tun ing Ġto Ġcombine Ġthese Ġtwo Ġmethods . ĠBy Ġimproving Ġthe Ġtranslation Ġquality Ġin Ġthis Ġway , Ġwe Ġcan Ġalso Ġincrease Ġthe Ġlikelihood Ġof Ġpseudo - par allel Ġdata Ġbeing Ġuseful Ġto Ġfurther Ġimprove Ġtranslation Ġquality . </s> Limit Ġof ĠUsing Ġonly ĠIn - domain ĠData </s> This Ġsection Ġanswers Ġour Ġfirst Ġresearch Ġquestion , Ġ[ R Q 1 ], Ġabout Ġthe Ġtranslation Ġquality Ġthat Ġwe Ġcan Ġachieve Ġusing Ġexisting Ġmethods Ġand Ġin - domain Ġparallel Ġand Ġmon oling ual Ġdata . ĠWe Ġthen Ġuse Ġthe Ġstrongest Ġmodel Ġto Ġconduct Ġexperiments Ġon Ġgenerating Ġand Ġutilizing Ġback - trans lated Ġpseudo - par allel Ġdata Ġfor Ġaugment ing ĠN MT . ĠOur Ġintention Ġis Ġto Ġempir ically Ġidentify Ġthe Ġmost Ġeffective Ġpractices Ġas Ġwell Ġas Ġrecognize Ġthe Ġlimitations Ġof Ġrelying Ġonly Ġon Ġin - domain Ġparallel Ġcorpor a . </s> Data </s> To Ġtrain ĠMT Ġsystems Ġamong Ġthe Ġthree Ġlanguages , Ġi . e ., ĠJapanese , ĠRussian , Ġand ĠEnglish , Ġwe Ġused Ġall Ġthe Ġparallel Ġdata Ġprovided Ġby ĠGlobal ĠVoices , Ġmore Ġspecifically Ġthose Ġavailable Ġat ĠOP US . ĠTable ĠT AB REF 9 Ġsummarizes Ġthe Ġsize Ġof Ġtrain / development / test Ġsplits Ġused Ġin Ġour Ġexperiments . ĠThe Ġnumber Ġof Ġparallel Ġsentences Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġis Ġ12 k , Ġfor ĠJa ĠIN LINE FORM 1 ĠEn Ġis Ġ47 k , Ġand Ġfor ĠRu ĠIN LINE FORM 2 ĠEn Ġis Ġ82 k . ĠNote Ġthat Ġthe Ġthree Ġcorpor a Ġare Ġnot Ġmutually Ġexclusive : Ġ9 k Ġout Ġof Ġ12 k Ġsentences Ġin Ġthe ĠJa ĠIN LINE FORM 3 ĠRu Ġcorpus Ġwere Ġalso Ġincluded Ġin Ġthe Ġother Ġtwo Ġparallel Ġcorpor a , Ġassociated Ġwith Ġidentical ĠEnglish Ġtranslations . ĠThis Ġputs Ġa Ġlimit Ġon Ġthe Ġpositive Ġimpact Ġthat Ġthe Ġhelping Ġcorpor a Ġcan Ġhave Ġon Ġthe Ġtranslation Ġquality . </s> Even Ġwhen Ġone Ġfocuses Ġon Ġlow - resource Ġlanguage Ġpairs , Ġwe Ġoften Ġhave Ġaccess Ġto Ġlarger Ġquantities Ġof Ġin - domain Ġmon oling ual Ġdata Ġof Ġeach Ġlanguage . ĠSuch Ġmon oling ual Ġdata Ġare Ġuseful Ġto Ġimprove Ġquality Ġof ĠMT , Ġfor Ġexample , Ġas Ġthe Ġsource Ġof Ġpseudo - par allel Ġdata Ġfor Ġaugment ing Ġtraining Ġdata Ġfor ĠN MT ĠB IB REF 17 Ġand Ġas Ġthe Ġtraining Ġdata Ġfor Ġlarge Ġand Ġsmoot hed Ġlanguage Ġmodels Ġfor ĠPBS MT ĠB IB REF 4 Ġ. ĠTable ĠT AB REF 13 Ġsummarizes Ġthe Ġstatistics Ġon Ġour Ġmon oling ual Ġcorpor a Ġfor Ġseveral Ġdomains Ġincluding Ġthe Ġnews Ġdomain . ĠNote Ġthat Ġwe Ġremoved Ġfrom Ġthe ĠGlobal ĠVoices Ġmon oling ual Ġcorpor a Ġthose Ġsentences Ġthat Ġare Ġalready Ġpresent Ġin Ġthe Ġparallel Ġcorpus . </s> https :// d umps . wik imedia . org / back up - index . html Ġ( 201 805 01 ) Ġhttp :// www . stat mt . org / w mt 18 / translation - task . html Ġhttps :// www . y omi uri . co . jp / database / gl oss ary / Ġhttp :// www . cs . j hu . edu / ~ ke vind uh / a / mult it arget - ted t alks / Ġhttp :// opus . nl pl . eu / T ato eb a - v 2 . php </s> We Ġtoken ized ĠEnglish Ġand ĠRussian Ġsentences Ġusing Ġtoken izer . per l Ġof ĠMoses ĠB IB REF 3 Ġ. ĠTo Ġtoken ize ĠJapanese Ġsentences , Ġwe Ġused ĠMe C ab Ġwith Ġthe ĠIPA Ġdictionary . ĠAfter Ġtoken ization , Ġwe Ġeliminated Ġdupl icated Ġsentence Ġpairs Ġand Ġsentences Ġwith Ġmore Ġthan Ġ100 Ġtokens Ġfor Ġall Ġthe Ġlanguages . </s> MT ĠMethods ĠExam ined </s> We Ġbegan Ġwith Ġevaluating Ġstandard ĠMT Ġparad ig ms , Ġi . e ., ĠPBS MT ĠB IB REF 3 Ġand ĠN MT ĠB IB REF 1 Ġ. ĠAs Ġfor ĠPBS MT , Ġwe Ġalso Ġexamined Ġtwo Ġadvanced Ġmethods : Ġpivot - based Ġtranslation Ġrelying Ġon Ġa Ġhelping Ġlanguage ĠB IB REF 10 Ġand Ġinduction Ġof Ġphrase Ġtables Ġfrom Ġmon oling ual Ġdata ĠB IB REF 14 Ġ. </s> As Ġfor ĠN MT , Ġwe Ġcompared Ġtwo Ġtypes Ġof Ġenc oder - dec oder Ġarchitectures : Ġattention al ĠR NN - based Ġmodel Ġ( RN MT ) ĠB IB REF 2 Ġand Ġthe ĠTrans former Ġmodel ĠB IB REF 18 Ġ. ĠIn Ġaddition Ġto Ġstandard Ġun i - direction al Ġmodeling , Ġto Ġcope Ġwith Ġthe Ġlow - resource Ġproblem , Ġwe Ġexamined Ġtwo Ġmulti - direction al Ġmodels : Ġbi - direction al Ġmodel ĠB IB REF 11 Ġand Ġmulti - to - multi Ġ( M 2 M ) Ġmodel ĠB IB REF 8 Ġ. </s> After Ġidentifying Ġthe Ġbest Ġmodel , Ġwe Ġalso Ġexamined Ġthe Ġusefulness Ġof Ġa Ġdata Ġaug mentation Ġmethod Ġbased Ġon Ġback - translation ĠB IB REF 17 Ġ. </s> First , Ġwe Ġbuilt Ġa ĠPBS MT Ġsystem Ġfor Ġeach Ġof Ġthe Ġsix Ġtranslation Ġdirections . ĠWe Ġobtained Ġphrase Ġtables Ġfrom Ġparallel Ġcorpus Ġusing ĠSy MG IZ A ++ Ġwith Ġthe Ġgrow - di ag - final Ġhe ur istics Ġfor Ġword Ġalignment , Ġand ĠMoses Ġfor Ġphrase Ġpair Ġextraction . ĠThen , Ġwe Ġtrained Ġa Ġbi - direction al ĠMS D Ġ( mon ot one , Ġswap , Ġand Ġdiscontin uous ) Ġlex ical ized Ġre ordering Ġmodel . ĠWe Ġalso Ġtrained Ġthree Ġ5 - gram Ġlanguage Ġmodels , Ġusing ĠKen LM Ġon Ġthe Ġfollowing Ġmon oling ual Ġdata : Ġ( 1 ) Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġdata , Ġ( 2 ) Ġthe Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġthe Ġmon oling ual Ġdata Ġfrom ĠGlobal ĠVoices , Ġand Ġ( 3 ) Ġthe Ġconc aten ation Ġof Ġ( 1 ) Ġand Ġall Ġmon oling ual Ġdata Ġin Ġthe Ġnews Ġdomain Ġin ĠTable ĠT AB REF 13 Ġ. </s> Sub sequently , Ġusing ĠEnglish Ġas Ġthe Ġpivot Ġlanguage , Ġwe Ġexamined Ġthe Ġfollowing Ġthree Ġtypes Ġof Ġpivot - based ĠPBS MT Ġsystems ĠB IB REF 10 Ġ, ĠB IB REF 19 Ġfor Ġeach Ġof ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠRu ĠIN LINE FORM 1 ĠJa . </s> 2 - step Ġdecoding Ġusing Ġthe Ġsource - to - English Ġand ĠEnglish - to - target Ġsystems . </s> Ob tain Ġa Ġnew Ġphrase Ġtable Ġfrom Ġsynthetic Ġparallel Ġdata Ġgenerated Ġby Ġtranslating ĠEnglish Ġside Ġof Ġthe Ġtarget âĢĵ English Ġtraining Ġparallel Ġdata Ġto Ġthe Ġsource Ġlanguage Ġwith Ġthe ĠEnglish - to - source Ġsystem . </s> Comp ile Ġa Ġnew Ġphrase Ġtable Ġcombining Ġthose Ġfor Ġthe Ġsource - to - English Ġand ĠEnglish - to - target Ġsystems . </s> Among Ġthese Ġthree , Ġtri ang ulation Ġis Ġthe Ġmost Ġcomput ationally Ġexpensive Ġmethod . ĠAlthough Ġwe Ġhad Ġfiltered Ġthe Ġcomponent Ġphrase Ġtables Ġusing Ġthe Ġstatistical Ġsignificance Ġpr uning Ġmethod ĠB IB REF 20 Ġ, Ġtri ang ulation Ġcan Ġgenerate Ġan Ġenormous Ġnumber Ġof Ġphrase Ġpairs . ĠTo Ġreduce Ġthe Ġcomputational Ġcost Ġduring Ġdecoding Ġand Ġthe Ġnegative Ġeffects Ġof Ġpotentially Ġnoisy Ġphrase Ġpairs , Ġwe Ġretained Ġfor Ġeach Ġsource Ġphrase ĠIN LINE FORM 0 Ġonly Ġthe ĠIN LINE FORM 1 Ġ- best Ġtranslations ĠIN LINE FORM 2 Ġaccording Ġto Ġthe Ġforward Ġtranslation Ġprobability ĠIN LINE FORM 3 Ġcalculated Ġfrom Ġthe Ġconditional Ġprobabilities Ġin Ġthe Ġcomponent Ġmodels Ġas Ġdefined Ġin Ġut iy ama : 07 . ĠFor Ġeach Ġof Ġthe Ġretained Ġphrase Ġpairs , Ġwe Ġalso Ġcalculated Ġthe Ġbackward Ġtranslation Ġprobability , ĠIN LINE FORM 4 Ġ, Ġand Ġlex ical Ġtranslation Ġprobabilities , ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ, Ġin Ġthe Ġsame Ġmanner Ġas ĠIN LINE FORM 7 Ġ. </s> We Ġalso Ġinvestigated Ġthe Ġutility Ġof Ġrecent Ġadvances Ġin Ġun super vised ĠMT . ĠEven Ġthough Ġwe Ġbegan Ġwith Ġa Ġpublicly Ġavailable Ġimplementation Ġof Ġun super vised ĠPBS MT ĠB IB REF 13 Ġ, Ġit Ġcrashed Ġdue Ġto Ġunknown Ġreasons . ĠWe Ġtherefore Ġfollowed Ġanother Ġmethod Ġdescribed Ġin Ġmar ie : us mt - un mt . ĠInstead Ġof Ġshort ĠIN LINE FORM 0 Ġ- gram s ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, Ġwe Ġcollected Ġa Ġset Ġof Ġphrases Ġin ĠJapanese Ġand ĠRussian Ġfrom Ġrespective Ġmon oling ual Ġdata Ġusing Ġthe Ġword 2 phrase Ġalgorithm ĠB IB REF 21 Ġ, Ġas Ġin Ġmar ie : us mt - un mt . ĠTo Ġreduce Ġthe Ġcomplexity , Ġwe Ġused Ġrandomly Ġselected Ġ10 M Ġmon oling ual Ġsentences , Ġand Ġ300 k Ġmost Ġfrequent Ġphrases Ġmade Ġof Ġwords Ġamong Ġthe Ġ300 k Ġmost Ġfrequent Ġwords . ĠFor Ġeach Ġsource Ġphrase ĠIN LINE FORM 1 Ġ, Ġwe Ġselected Ġ300 - best Ġtarget Ġphrases ĠIN LINE FORM 2 Ġaccording Ġto Ġthe Ġtranslation Ġprobability Ġas Ġin ĠD 18 - 15 49 : ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġstands Ġfor Ġa Ġbilingual Ġembed ding Ġof Ġa Ġgiven Ġphrase , Ġobtained Ġthrough Ġaveraging Ġbilingual Ġembed d ings Ġof Ġconstituent Ġwords Ġlearned Ġfrom Ġthe Ġtwo Ġmon oling ual Ġdata Ġusing Ġfast Text Ġand Ġvec map . ĠFor Ġeach Ġof Ġthe Ġretained Ġphrase Ġpair , ĠIN LINE FORM 5 Ġwas Ġcomputed Ġanalog ously . ĠWe Ġalso Ġcomputed Ġlex ical Ġtranslation Ġprobabilities Ġrelying Ġon Ġthose Ġlearned Ġfrom Ġthe Ġgiven Ġsmall Ġparallel Ġcorpus . </s> Up Ġto Ġfour Ġphrase Ġtables Ġwere Ġjointly Ġexploited Ġby Ġthe Ġmultiple Ġdecoding Ġpath Ġability Ġof ĠMoses . ĠWe ights Ġfor Ġthe Ġfeatures Ġwere Ġtuned Ġusing ĠKB - MI RA ĠB IB REF 22 Ġon Ġthe Ġdevelopment Ġset ; Ġwe Ġtook Ġthe Ġbest Ġweights Ġafter Ġ15 Ġiterations . ĠTwo Ġhyper - param eters , Ġnamely , ĠIN LINE FORM 0 Ġfor Ġthe Ġnumber Ġof Ġpivot - based Ġphrase Ġpairs Ġper Ġsource Ġphrase Ġand ĠIN LINE FORM 1 Ġfor Ġdistortion Ġlimit , Ġwere Ġdetermined Ġby Ġa Ġgrid Ġsearch Ġon ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠIn Ġcontrast , Ġwe Ġused Ġpredetermined Ġhyper - param eters Ġfor Ġphrase Ġtable Ġinduction Ġfrom Ġmon oling ual Ġdata , Ġfollowing Ġthe Ġconvention : Ġ200 Ġfor Ġthe Ġdimension Ġof Ġword Ġand Ġphrase Ġembed d ings Ġand ĠIN LINE FORM 4 Ġ. </s> We Ġused Ġthe Ġopen - source Ġimplementation Ġof Ġthe ĠRN MT Ġand Ġthe ĠTrans former Ġmodels Ġin Ġtens or 2 t ensor . ĠA Ġun i - direction al Ġmodel Ġfor Ġeach Ġof Ġthe Ġsix Ġtranslation Ġdirections Ġwas Ġtrained Ġon Ġthe Ġcorresponding Ġparallel Ġcorpus . ĠBi - direction al Ġand ĠM 2 M Ġmodels Ġwere Ġrealized Ġby Ġadding Ġan Ġartificial Ġtoken Ġthat Ġspecifies Ġthe Ġtarget Ġlanguage Ġto Ġthe Ġbeginning Ġof Ġeach Ġsource Ġsentence Ġand Ġshuff ling Ġthe Ġentire Ġtraining Ġdata ĠB IB REF 8 Ġ. </s> Table ĠT AB REF 22 Ġcontains Ġsome Ġspecific Ġhyper - param eters Ġfor Ġour Ġbaseline ĠN MT Ġmodels . ĠThe Ġhyper - param eters Ġnot Ġmentioned Ġin Ġthis Ġtable Ġused Ġthe Ġdefault Ġvalues Ġin Ġtens or 2 t ensor . ĠFor ĠM 2 M Ġsystems , Ġwe Ġover - sam pled ĠJa ĠIN LINE FORM 0 ĠRu Ġand ĠJa ĠIN LINE FORM 1 ĠEn Ġtraining Ġdata Ġso Ġthat Ġtheir Ġsizes Ġmatch Ġthe Ġlargest ĠRu ĠIN LINE FORM 2 ĠEn Ġdata . ĠTo Ġreduce Ġthe Ġnumber Ġof Ġunknown Ġwords , Ġwe Ġused Ġtens or 2 t ensor 's Ġinternal Ġsub - word Ġsegment ation Ġmechanism . ĠSince Ġwe Ġwork Ġin Ġa Ġlow - resource Ġsetting , Ġwe Ġused Ġshared Ġsub - word Ġvoc ab ular ies Ġof Ġsize Ġ16 k Ġfor Ġthe Ġun i - Ġand Ġbi - direction al Ġmodels Ġand Ġ32 k Ġfor Ġthe ĠM 2 M Ġmodels . ĠThe Ġnumber Ġof Ġtraining Ġiterations Ġwas Ġdetermined Ġby Ġearly - st opping : Ġwe Ġevaluated Ġour Ġmodels Ġon Ġthe Ġdevelopment Ġset Ġevery Ġ1 , 000 Ġupdates , Ġand Ġstopped Ġtraining Ġif ĠB LE U Ġscore Ġfor Ġthe Ġdevelopment Ġset Ġwas Ġnot Ġimproved Ġfor Ġ10 , 000 Ġupdates Ġ( 10 Ġcheck - points ). ĠNote Ġthat Ġthe Ġdevelopment Ġset Ġwas Ġcreated Ġby Ġconc aten ating Ġthose Ġfor Ġthe Ġindividual Ġtranslation Ġdirections Ġwithout Ġany Ġover - sam pling . </s> Having Ġtrained Ġthe Ġmodels , Ġwe Ġaveraged Ġthe Ġlast Ġ10 Ġcheck - points Ġand Ġdec oded Ġthe Ġtest Ġsets Ġwith Ġa Ġbeam Ġsize Ġof Ġ4 Ġand Ġa Ġlength Ġpenalty Ġwhich Ġwas Ġtuned Ġby Ġa Ġlinear Ġsearch Ġon Ġthe ĠB LE U Ġscore Ġfor Ġthe Ġdevelopment Ġset . </s> Similarly Ġto ĠPBS MT , Ġwe Ġalso Ġevaluated ĠâĢ ľ C ascade âĢ Ŀ Ġand ĠâĢ ľ Sy nt hes ize âĢ Ŀ Ġmethods Ġwith Ġun i - direction al ĠN MT Ġmodels . </s> Results </s> We Ġevaluated ĠMT Ġmodels Ġusing Ġcase - sensitive Ġand Ġtoken ized ĠB LE U ĠB IB REF 23 Ġon Ġtest Ġsets , Ġusing ĠMoses 's Ġmulti - ble u . per l . ĠStatistical Ġsignificance Ġ( ĠIN LINE FORM 0 Ġ) Ġon Ġthe Ġdifference Ġof ĠB LE U Ġscores Ġwas Ġtested Ġby ĠMoses 's Ġboot strap - hyp ot hesis - diff erence - sign ific ance . pl . </s> T ables ĠT AB REF 27 Ġand ĠT AB REF 31 Ġshow ĠB LE U Ġscores Ġof Ġall Ġthe Ġmodels , Ġexcept Ġthe ĠN MT Ġsystems Ġaugmented Ġwith Ġback - trans lations . ĠWhereas Ġsome Ġmodels Ġachieved Ġreasonable ĠB LE U Ġscores Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġtranslation , Ġall Ġthe Ġresults Ġfor ĠJa ĠIN LINE FORM 2 ĠRu , Ġwhich Ġis Ġour Ġmain Ġconcern , Ġwere Ġab ys mal . </s> Among Ġthe ĠN MT Ġmodels , ĠTrans former Ġmodels Ġ( b ĠIN LINE FORM 0 Ġ) Ġwere Ġproven Ġto Ġbe Ġbetter Ġthan ĠRN MT Ġmodels Ġ( a ĠIN LINE FORM 1 Ġ). ĠRN MT Ġmodels Ġcould Ġnot Ġeven Ġoutper form Ġthe Ġun i - direction al ĠPBS MT Ġmodels Ġ( c 1 ). ĠM 2 M Ġmodels Ġ( a 3 ) Ġand Ġ( b 3 ) Ġoutper formed Ġtheir Ġcorresponding Ġun i - Ġand Ġbi - direction al Ġmodels Ġin Ġmost Ġcases . ĠIt Ġis Ġworth Ġnoting Ġthat Ġin Ġthis Ġextremely Ġlow - resource Ġscenario , ĠB LE U Ġscores Ġof Ġthe ĠM 2 M ĠRN MT Ġmodel Ġfor Ġthe Ġlargest Ġlanguage Ġpair , Ġi . e ., ĠRu ĠIN LINE FORM 2 ĠEn , Ġwere Ġlower Ġthan Ġthose Ġof Ġthe Ġun i - Ġand Ġbi - direction al ĠRN MT Ġmodels Ġas Ġin ĠT AC L 10 81 . ĠIn Ġcontrast , Ġwith Ġthe ĠM 2 M ĠTrans former Ġmodel , ĠRu ĠIN LINE FORM 3 ĠEn Ġalso Ġbenefited Ġfrom Ġmult ilingual ism . </s> Standard ĠPBS MT Ġmodels Ġ( c 1 ) Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġun i - direction al ĠN MT Ġmodels Ġ( a 1 ) Ġand Ġ( b 1 ), Ġas Ġreported Ġby Ġk oe hn - know les : 2017 : N MT , Ġwhereas Ġthey Ġunder per form Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ). ĠAs Ġshown Ġin ĠTable ĠT AB REF 31 Ġ, Ġpivot - based ĠPBS MT Ġsystems Ġalways Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġ( c 1 ). ĠThe Ġbest Ġmodel Ġwith Ġthree Ġphrase Ġtables , Ġlabeled ĠâĢ ľ Sy nt hes ize Ġ/ ĠTri ang ulate Ġ/ ĠGold , âĢ Ŀ Ġbrought Ġvisible ĠB LE U Ġgains Ġwith Ġsubstantial Ġreduction Ġof ĠO OV Ġtokens Ġ( 30 47 ĠIN LINE FORM 0 Ġ11 80 Ġfor ĠJa ĠIN LINE FORM 1 ĠRu , Ġ4 463 ĠIN LINE FORM 2 Ġ18 12 Ġfor ĠRu ĠIN LINE FORM 3 ĠJa ). ĠHowever , Ġfurther Ġextension Ġwith Ġphrase Ġtables Ġinduced Ġfrom Ġmon oling ual Ġdata Ġdid Ġnot Ġpush Ġthe Ġlimit , Ġdespite Ġtheir Ġhigh Ġcoverage ; Ġonly Ġ336 Ġand Ġ6 77 ĠO OV Ġtokens Ġwere Ġleft Ġfor Ġthe Ġtwo Ġtranslation Ġdirections , Ġrespectively . ĠThis Ġis Ġdue Ġto Ġthe Ġpoor Ġquality Ġof Ġthe Ġbilingual Ġword Ġembed d ings Ġused Ġto Ġextract Ġthe Ġphrase Ġtable , Ġas Ġenvis aged Ġin ĠSection ĠSEC REF 3 Ġ. </s> None Ġof Ġpivot - based Ġapproaches Ġwith Ġun i - direction al ĠN MT Ġmodels Ġcould Ġeven Ġremotely Ġrival Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ). </s> Table ĠT AB REF 46 Ġshows Ġthe Ġresults Ġof Ġour Ġmult ist age Ġfine - tun ing , Ġwhere Ġthe ĠIDs Ġof Ġeach Ġrow Ġrefer Ġto Ġthose Ġdescribed Ġin ĠSection ĠSEC REF 41 Ġ. ĠFirst Ġof Ġall , Ġthe Ġfinal Ġmodels Ġof Ġour Ġmult ist age Ġfine - tun ing , Ġi . e ., ĠV Ġand ĠVII , Ġachieved Ġsignificantly Ġhigher ĠB LE U Ġscores Ġthan Ġ( b 3 ) Ġin ĠTable ĠT AB REF 27 Ġ, Ġa Ġweak Ġbaseline Ġwithout Ġusing Ġany Ġmon oling ual Ġdata , Ġand Ġ# 10 Ġin ĠTable ĠT AB REF 33 Ġ, Ġa Ġstrong Ġbaseline Ġestablished Ġwith Ġmon oling ual Ġdata . </s> The Ġperformance Ġof Ġthe Ġinitial Ġmodel Ġ( I ) Ġdepends Ġon Ġthe Ġlanguage Ġpair . ĠFor ĠJa ĠIN LINE FORM 0 ĠRu Ġpair , Ġit Ġcannot Ġachieve Ġminimum Ġlevel Ġof Ġquality Ġsince Ġthe Ġmodel Ġhas Ġnever Ġseen Ġparallel Ġdata Ġfor Ġthis Ġpair . ĠThe Ġperformance Ġon ĠJa ĠIN LINE FORM 1 ĠEn Ġpair Ġwas Ġmuch Ġlower Ġthan Ġthe Ġtwo Ġbaseline Ġmodels , Ġreflecting Ġthe Ġcrucial Ġmismatch Ġbetween Ġtraining Ġand Ġtesting Ġdomains . ĠIn Ġcontrast , ĠRu ĠIN LINE FORM 2 ĠEn Ġpair Ġbenefited Ġthe Ġmost Ġand Ġachieved Ġsurprisingly Ġhigh ĠB LE U Ġscores . ĠThe Ġreason Ġmight Ġbe Ġdue Ġto Ġthe Ġproximity Ġof Ġout - of - domain Ġtraining Ġdata Ġand Ġin - domain Ġtest Ġdata . </s> The Ġfirst Ġfine - tun ing Ġstage Ġsignificantly Ġpushed Ġup Ġthe Ġtranslation Ġquality Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġpairs , Ġin Ġboth Ġcases Ġwith Ġfine - tun ing Ġ( II ) Ġand Ġmixed Ġfine - tun ing Ġ( III ). ĠAt Ġthis Ġstage , Ġboth Ġmodels Ġperformed Ġonly Ġpoorly Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġpair Ġas Ġthey Ġhave Ġnot Ġyet Ġseen ĠJa ĠIN LINE FORM 3 ĠRu Ġparallel Ġdata . ĠEither Ġmodel Ġhad Ġa Ġconsistent Ġadvantage Ġto Ġthe Ġother . </s> When Ġthese Ġmodels Ġwere Ġfurther Ġfine - tun ed Ġonly Ġon Ġthe Ġin - domain ĠJa ĠIN LINE FORM 0 ĠRu Ġparallel Ġdata Ġ( IV Ġand ĠVI ), Ġwe Ġobtained Ġtranslations Ġof Ġbetter Ġquality Ġthan Ġthe Ġtwo Ġbas elines Ġfor ĠJa ĠIN LINE FORM 1 ĠRu Ġpair . ĠHowever , Ġas Ġa Ġresult Ġof Ġcomplete Ġignorance Ġof ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġpairs , Ġthe Ġmodels Ġonly Ġproduced Ġtranslations Ġof Ġpoor Ġquality Ġfor Ġthese Ġlanguage Ġpairs . ĠIn Ġcontrast , Ġmixed Ġfine - tun ing Ġfor Ġthe Ġsecond Ġfine - tun ing Ġstage Ġ( V Ġand ĠVII ) Ġresulted Ġin Ġconsistently Ġbetter Ġmodels Ġthan Ġconventional Ġfine - tun ing Ġ( IV Ġand ĠVI ), Ġirrespective Ġof Ġthe Ġchoice Ġat Ġthe Ġfirst Ġstage , Ġthanks Ġto Ġthe Ġgradual Ġshift Ġof Ġparameters Ġrealized Ġby Ġin - domain ĠJa ĠIN LINE FORM 4 ĠEn Ġand ĠRu ĠIN LINE FORM 5 ĠEn Ġparallel Ġdata . ĠUnfortunately , Ġthe Ġtranslation Ġquality Ġfor ĠJa ĠIN LINE FORM 6 ĠEn Ġand ĠRu ĠIN LINE FORM 7 ĠEn Ġpairs Ġsometimes Ġdegraded Ġfrom ĠII Ġand ĠIII . ĠNevertheless , Ġthe ĠB LE U Ġscores Ġstill Ġretain Ġthe Ġlarge Ġmargin Ġagainst Ġtwo Ġbas elines . </s> The Ġlast Ġthree Ġrows Ġin ĠTable ĠT AB REF 46 Ġpresent ĠB LE U Ġscores Ġobtained Ġby Ġthe Ġmethods Ġwith Ġfewer Ġfine - tun ing Ġsteps . ĠThe Ġmost Ġnaive Ġmodel ĠI ', Ġtrained Ġon Ġthe Ġbalanced Ġmixture Ġof Ġwhole Ġfive Ġtypes Ġof Ġcorpor a Ġfrom Ġscratch , Ġand Ġthe Ġmodel ĠII ', Ġobtained Ġthrough Ġa Ġsingle - step Ġconventional Ġfine - tun ing Ġof ĠI Ġon Ġall Ġthe Ġin - domain Ġdata , Ġachieved Ġonly ĠB LE U Ġscores Ġconsistently Ġworse Ġthan ĠVII . ĠIn Ġcontrast , Ġwhen Ġwe Ġmerged Ġour Ġtwo Ġfine - tun ing Ġsteps Ġinto Ġa Ġsingle Ġmixed Ġfine - tun ing Ġon ĠI , Ġwe Ġobtained Ġa Ġmodel ĠIII ' Ġwhich Ġis Ġbetter Ġfor Ġthe ĠJa ĠIN LINE FORM 0 ĠRu Ġpair Ġthan ĠVII . ĠNevertheless , Ġthey Ġare Ġstill Ġcomparable Ġto Ġthose Ġof ĠVII Ġand Ġthe ĠB LE U Ġscores Ġfor Ġthe Ġother Ġtwo Ġlanguage Ġpairs Ġare Ġmuch Ġlower Ġthan ĠVII . ĠAs Ġsuch , Ġwe Ġconclude Ġthat Ġour Ġmult ist age Ġfine - tun ing Ġleads Ġto Ġa Ġmore Ġrobust Ġin - domain Ġmult ilingual Ġmodel . </s> Aug mentation Ġwith ĠBack - translation </s> Given Ġthat Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ) Ġachieved Ġbest Ġresults Ġfor Ġmost Ġof Ġthe Ġtranslation Ġdirections Ġand Ġcompetitive Ġresults Ġfor Ġthe Ġrest , Ġwe Ġfurther Ġexplored Ġit Ġthrough Ġback - translation . </s> We Ġexamined Ġthe Ġutility Ġof Ġpseudo - par allel Ġdata Ġfor Ġall Ġthe Ġsix Ġtranslation Ġdirections , Ġunlike Ġthe Ġwork Ġof Ġlake w 2017 impro ving Ġand Ġlake w 2018 com par ison , Ġwhich Ġconcentrate Ġonly Ġon Ġthe Ġzero - shot Ġlanguage Ġpair , Ġand Ġthe Ġwork Ġof ĠW 18 - 27 10 , Ġwhich Ġcompares Ġonly Ġun i - Ġor Ġbi - direction al Ġmodels . ĠWe Ġinvestigated Ġwhether Ġeach Ġtranslation Ġdirection Ġin ĠM 2 M Ġmodels Ġwill Ġbenefit Ġfrom Ġpseudo - par allel Ġdata Ġand Ġif Ġso , Ġwhat Ġkind Ġof Ġimprovement Ġtakes Ġplace . </s> First , Ġwe Ġselected Ġsentences Ġto Ġbe Ġback - trans lated Ġfrom Ġin - domain Ġmon oling ual Ġdata Ġ( Table ĠT AB REF 13 Ġ), Ġrelying Ġon Ġthe Ġscore Ġproposed Ġby Ġmo ore : int elligent Ġvia Ġthe Ġfollowing Ġprocedure . </s> For Ġeach Ġlanguage , Ġtrain Ġtwo Ġ4 - gram Ġlanguage Ġmodels , Ġusing ĠKen LM : Ġan Ġin - domain Ġone Ġon Ġall Ġthe ĠGlobal ĠVoices Ġdata , Ġi . e ., Ġboth Ġparallel Ġand Ġmon oling ual Ġdata , Ġand Ġa Ġgeneral - domain Ġone Ġon Ġthe Ġconc aten ation Ġof ĠGlobal ĠVoices , ĠI W SL T , Ġand ĠT ato eb a Ġdata . </s> For Ġeach Ġlanguage , Ġdiscard Ġsentences Ġcontaining ĠO OV s Ġaccording Ġto Ġthe Ġin - domain Ġlanguage Ġmodel . </s> For Ġeach Ġtranslation Ġdirection , Ġselect Ġthe ĠIN LINE FORM 0 Ġ- best Ġmon oling ual Ġsentences Ġin Ġthe Ġnews Ġdomain , Ġaccording Ġto Ġthe Ġdifference Ġbetween Ġcross - ent ropy Ġscores Ġgiven Ġby Ġthe Ġin - domain Ġand Ġgeneral - domain Ġlanguage Ġmodels . </s> Whereas ĠW 18 - 27 10 Ġexploited Ġmon oling ual Ġdata Ġmuch Ġlarger Ġthan Ġparallel Ġdata , Ġwe Ġmaintained Ġa Ġ1 : 1 Ġratio Ġbetween Ġthem ĠB IB REF 8 Ġ, Ġsetting ĠIN LINE FORM 0 Ġto Ġthe Ġnumber Ġof Ġlines Ġof Ġparallel Ġdata Ġof Ġgiven Ġlanguage Ġpair . </s> Se lected Ġmon oling ual Ġsentences Ġwere Ġthen Ġtranslated Ġusing Ġthe ĠM 2 M ĠTrans former ĠN MT Ġmodel Ġ( b 3 ) Ġto Ġcompose Ġpseudo - par allel Ġdata . ĠThen , Ġthe Ġpseudo - par allel Ġdata Ġwere Ġenlarged Ġby Ġover - sam pling Ġas Ġsummarized Ġin ĠTable ĠT AB REF 32 Ġ. ĠFinally , Ġnew ĠN MT Ġmodels Ġwere Ġtrained Ġon Ġthe Ġconc aten ation Ġof Ġthe Ġoriginal Ġparallel Ġand Ġpseudo - par allel Ġdata Ġfrom Ġscratch Ġin Ġthe Ġsame Ġmanner Ġas Ġthe Ġprevious ĠN MT Ġmodels Ġwith Ġthe Ġsame Ġhyper - param eters . </s> Table ĠT AB REF 33 Ġshows Ġthe ĠB LE U Ġscores Ġachieved Ġby Ġseveral Ġreasonable Ġcombinations Ġof Ġsix - way Ġpseudo - par allel Ġdata . ĠWe Ġobserved Ġthat Ġthe Ġuse Ġof Ġall Ġsix - way Ġpseudo - par allel Ġdata Ġ(# 10 ) Ġsignificantly Ġimproved Ġthe Ġbase Ġmodel Ġfor Ġall Ġthe Ġtranslation Ġdirections , Ġexcept ĠEn ĠIN LINE FORM 0 ĠRu . ĠA Ġtranslation Ġdirection Ġoften Ġbenefited Ġwhen Ġthe Ġpseudo - par allel Ġdata Ġfor Ġthat Ġspecific Ġdirection Ġwas Ġused . </s> Summary </s> We Ġhave Ġevaluated Ġan Ġextensive Ġvariation Ġof ĠMT Ġmodels Ġthat Ġrely Ġonly Ġon Ġin - domain Ġparallel Ġand Ġmon oling ual Ġdata . ĠHowever , Ġthe Ġresulting ĠB LE U Ġscores Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġand ĠRu ĠIN LINE FORM 3 ĠJa Ġtasks Ġdo Ġnot Ġexceed Ġ10 ĠB LE U Ġpoints , Ġimplying Ġthe Ġinherent Ġlimitation Ġof Ġthe Ġin - domain Ġdata Ġas Ġwell Ġas Ġthe Ġdifficulty Ġof Ġthese Ġtranslation Ġdirections . </s> Expl o iting ĠLarge ĠOut - of - Domain ĠData ĠInv olving Ġa ĠHelp ing ĠLanguage </s> The Ġlimitation Ġof Ġrelying Ġonly Ġon Ġin - domain Ġdata Ġdemonstrated Ġin ĠSection ĠSEC REF 4 Ġmotiv ates Ġus Ġto Ġexplore Ġother Ġtypes Ġof Ġparallel Ġdata . ĠAs Ġraised Ġin Ġour Ġsecond Ġresearch Ġquestion , Ġ[ R Q 2 ], Ġwe Ġconsidered Ġthe Ġeffective Ġways Ġto Ġexploit Ġout - of - domain Ġdata . </s> According Ġto Ġlanguage Ġpair Ġand Ġdomain , Ġparallel Ġdata Ġcan Ġbe Ġclassified Ġinto Ġfour Ġcategories Ġin ĠTable ĠT AB REF 40 Ġ. ĠAmong Ġall Ġthe Ġcategories , Ġout - of - domain Ġdata Ġfor Ġthe Ġlanguage Ġpair Ġof Ġinterest Ġhave Ġbeen Ġexploited Ġin Ġthe Ġdomain Ġadaptation Ġscenarios Ġ( C ĠIN LINE FORM 0 ĠA ) ĠB IB REF 9 Ġ. ĠHowever , Ġfor ĠJa ĠIN LINE FORM 1 ĠRu , Ġno Ġout - of - domain Ġdata Ġis Ġavailable . ĠTo Ġexploit Ġout - of - domain Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġpairs Ġinstead , Ġwe Ġpropose Ġa Ġmult ist age Ġfine - tun ing Ġmethod , Ġwhich Ġcombines Ġtwo Ġtypes Ġof Ġtransfer Ġlearning , Ġi . e ., Ġdomain Ġadaptation Ġfor ĠJa ĠIN LINE FORM 4 ĠEn Ġand ĠRu ĠIN LINE FORM 5 ĠEn Ġ( D ĠIN LINE FORM 6 ĠB ) Ġand Ġmult ilingual Ġtransfer Ġ( B ĠIN LINE FORM 7 ĠA ), Ġrelying Ġon Ġthe ĠM 2 M Ġmodel Ġexamined Ġin ĠSection ĠSEC REF 4 Ġ. ĠWe Ġalso Ġexamined Ġthe Ġutility Ġof Ġfine - tun ing Ġfor Ġiter atively Ġgenerating Ġand Ġusing Ġpseudo - par allel Ġdata . </s> Mult ist age ĠFine - tun ing </s> Simply Ġusing ĠN MT Ġsystems Ġtrained Ġon Ġout - of - domain Ġdata Ġfor Ġin - domain Ġtranslation Ġis Ġknown Ġto Ġperform Ġbadly . ĠIn Ġorder Ġto Ġeffectively Ġuse Ġlarge - scale Ġout - of - domain Ġdata Ġfor Ġour Ġextremely Ġlow - resource Ġtask , Ġwe Ġpropose Ġto Ġperform Ġdomain Ġadaptation Ġthrough Ġeither Ġ( a ) Ġconventional Ġfine - tun ing , Ġwhere Ġan ĠN MT Ġsystem Ġtrained Ġon Ġout - of - domain Ġdata Ġis Ġfine - tun ed Ġonly Ġon Ġin - domain Ġdata , Ġor Ġ( b ) Ġmixed Ġfine - tun ing ĠB IB REF 9 Ġ, Ġwhere Ġpre - trained Ġout - of - domain ĠN MT Ġsystem Ġis Ġfine - tun ed Ġusing Ġa Ġmixture Ġof Ġin - domain Ġand Ġout - of - domain Ġdata . ĠThe Ġsame Ġoptions Ġare Ġavailable Ġfor Ġtransferring Ġfrom ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġto ĠJa ĠIN LINE FORM 2 ĠRu . </s> We Ġinevitably Ġinvolve Ġtwo Ġtypes Ġof Ġtransfer Ġlearning , Ġi . e ., Ġdomain Ġadaptation Ġfor ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġand Ġmult ilingual Ġtransfer Ġfor ĠJa ĠIN LINE FORM 2 ĠRu Ġpair . ĠAmong Ġseveral Ġconceivable Ġoptions Ġfor Ġmanaging Ġthese Ġtwo Ġproblems , Ġwe Ġexamined Ġthe Ġfollowing Ġmult ist age Ġfine - tun ing . </s> Pre - train Ġa Ġmult ilingual Ġmodel Ġonly Ġon Ġthe ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġout - of - domain Ġparallel Ġdata Ġ( I ), Ġwhere Ġthe Ġvocabulary Ġof Ġthe Ġmodel Ġis Ġdetermined Ġon Ġthe Ġbasis Ġof Ġthe Ġin - domain Ġparallel Ġdata Ġin Ġthe Ġsame Ġmanner Ġas Ġthe ĠM 2 M ĠN MT Ġmodels Ġexamined Ġin ĠSection ĠSEC REF 4 Ġ. </s> Fine - t une Ġthe Ġpre - trained Ġmodel Ġ( I ) Ġon Ġthe Ġin - domain ĠJa ĠIN LINE FORM 0 ĠEn Ġand ĠRu ĠIN LINE FORM 1 ĠEn Ġparallel Ġdata Ġ( fine - tun ing , ĠII ) Ġor Ġon Ġthe Ġmixture Ġof Ġin - domain Ġand Ġout - of - domain ĠJa ĠIN LINE FORM 2 ĠEn Ġand ĠRu ĠIN LINE FORM 3 ĠEn Ġparallel Ġdata Ġ( m ixed Ġfine - tun ing , ĠIII ). </s> Further Ġfine - t une Ġthe Ġmodels Ġ( each Ġof ĠII Ġand ĠIII ) Ġfor ĠJa ĠIN LINE FORM 0 ĠRu Ġon Ġin - domain Ġparallel Ġdata Ġfor Ġthis Ġlanguage Ġpair Ġonly Ġ( fine - tun ing , ĠIV Ġand ĠVI ) Ġor Ġon Ġall Ġthe Ġin - domain Ġparallel Ġdata Ġ( m ixed Ġfine - tun ing , ĠV Ġand ĠVII ). </s> We Ġchose Ġthis Ġway Ġdue Ġto Ġthe Ġfollowing Ġtwo Ġreasons . ĠFirst , Ġwe Ġneed Ġto Ġtake Ġa Ġbalance Ġbetween Ġseveral Ġdifferent Ġparallel Ġcorpor a Ġsizes . ĠThe Ġother Ġreason Ġis Ġdivision Ġof Ġlabor ; Ġwe Ġassume Ġthat Ġsolving Ġeach Ġsub - problem Ġone Ġby Ġone Ġshould Ġenable Ġgradual Ġshift Ġof Ġparameters . </s> Data ĠSelection </s> As Ġan Ġadditional Ġlarge - scale Ġout - of - domain Ġparallel Ġdata Ġfor ĠJa ĠIN LINE FORM 0 ĠEn , Ġwe Ġused Ġthe Ġclean est Ġ1 . 5 M Ġsentences Ġfrom Ġthe ĠAsian ĠScientific ĠPaper ĠEx cerpt ĠCorpus Ġ( AS P EC ) ĠB IB REF 24 Ġ. ĠAs Ġfor ĠRu ĠIN LINE FORM 1 ĠEn , Ġwe Ġused Ġthe ĠUN Ġand ĠY and ex Ġcorpor a Ġreleased Ġfor Ġthe ĠW MT Ġ2018 ĠNews ĠTranslation ĠTask . ĠWe Ġretained ĠRu ĠIN LINE FORM 2 ĠEn Ġsentence Ġpairs Ġthat Ġcontain Ġat Ġleast Ġone ĠO OV Ġtoken Ġin Ġboth Ġsides , Ġaccording Ġto Ġthe Ġin - domain Ġlanguage Ġmodel Ġtrained Ġin ĠSection ĠSEC REF 34 Ġ. ĠTable ĠT AB REF 45 Ġsummarizes Ġthe Ġstatistics Ġon Ġthe Ġremaining Ġout - of - domain Ġparallel Ġdata . </s> Further ĠAug mentation Ġwith ĠBack - translation </s> Having Ġobtained Ġa Ġbetter Ġmodel , Ġwe Ġexamined Ġagain Ġthe Ġutility Ġof Ġback - translation . ĠMore Ġprecisely , Ġwe Ġinvestigated Ġ( a ) Ġwhether Ġthe Ġpseudo - par allel Ġdata Ġgenerated Ġby Ġan Ġimproved ĠN MT Ġmodel Ġleads Ġto Ġa Ġfurther Ġimprovement , Ġand Ġ( b ) Ġwhether Ġone Ġmore Ġstage Ġof Ġfine - tun ing Ġon Ġthe Ġmixture Ġof Ġoriginal Ġparallel Ġand Ġpseudo - par allel Ġdata Ġwill Ġresult Ġin Ġa Ġmodel Ġbetter Ġthan Ġtraining Ġa Ġnew Ġmodel Ġfrom Ġscratch Ġas Ġexamined Ġin ĠSection ĠSEC REF 34 Ġ. </s> Given Ġan ĠN MT Ġmodel , Ġwe Ġfirst Ġgenerated Ġsix - way Ġpseudo - par allel Ġdata Ġby Ġtranslating Ġmon oling ual Ġdata . ĠFor Ġthe Ġsake Ġof Ġcompar ability , Ġwe Ġused Ġthe Ġidentical Ġmon oling ual Ġsentences Ġsampled Ġin ĠSection ĠSEC REF 34 Ġ. ĠThen , Ġwe Ġfurther Ġfine - tun ed Ġthe Ġgiven Ġmodel Ġon Ġthe Ġmixture Ġof Ġthe Ġgenerated Ġpseudo - par allel Ġdata Ġand Ġthe Ġoriginal Ġparallel Ġdata , Ġfollowing Ġthe Ġsame Ġover - sam pling Ġprocedure Ġin ĠSection ĠSEC REF 34 Ġ. ĠWe Ġrepeated Ġthese Ġsteps Ġfive Ġtimes . </s> Table ĠT AB REF 51 Ġshows Ġthe Ġresults . ĠâĢ ľ new Ġ# 10 âĢ Ŀ Ġin Ġthe Ġsecond Ġrow Ġindicates Ġan ĠM 2 M ĠTrans former Ġmodel Ġtrained Ġfrom Ġscratch Ġon Ġthe Ġmixture Ġof Ġsix - way Ġpseudo - par allel Ġdata Ġgenerated Ġby ĠVII Ġand Ġthe Ġoriginal Ġparallel Ġdata . ĠIt Ġachieved Ġhigher ĠB LE U Ġscores Ġthan Ġ# 10 Ġin ĠTable ĠT AB REF 33 Ġthanks Ġto Ġthe Ġpseudo - par allel Ġdata Ġof Ġbetter Ġquality , Ġbut Ġunder per formed Ġthe Ġbase ĠN MT Ġmodel ĠVII . ĠIn Ġcontrast , Ġour Ġfine - tun ed Ġmodel ĠVIII Ġsuccessfully Ġsurpassed ĠVII , Ġand Ġone Ġmore Ġiteration Ġ( IX ) Ġfurther Ġimproved ĠB LE U Ġscores Ġfor Ġall Ġtranslation Ġdirections , Ġexcept ĠRu ĠIN LINE FORM 0 ĠEn . ĠAlthough Ġfurther Ġiterations Ġdid Ġnot Ġnecessarily Ġgain ĠB LE U Ġscores , Ġwe Ġcame Ġto Ġa Ġmuch Ġhigher Ġplateau Ġcompared Ġto Ġthe Ġresults Ġin ĠSection ĠSEC REF 4 Ġ. </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġchallenged Ġthe Ġdifficult Ġtask Ġof ĠJa ĠIN LINE FORM 0 ĠRu Ġnews Ġdomain Ġtranslation Ġin Ġan Ġextremely Ġlow - resource Ġsetting . ĠWe Ġempir ically Ġconfirmed Ġthe Ġlimited Ġsuccess Ġof Ġwell - established Ġsolutions Ġwhen Ġrestricted Ġto Ġin - domain Ġdata . ĠThen , Ġto Ġincorporate Ġout - of - domain Ġdata , Ġwe Ġproposed Ġa Ġmult ilingual Ġmult ist age Ġfine - tun ing Ġapproach Ġand Ġobserved Ġthat Ġit Ġsubstantially Ġimproves ĠJa ĠIN LINE FORM 1 ĠRu Ġtranslation Ġby Ġover Ġ3 . 7 ĠB LE U Ġpoints Ġcompared Ġto Ġa Ġstrong Ġbaseline , Ġas Ġsummarized Ġin ĠTable ĠT AB REF 53 Ġ. ĠThis Ġpaper Ġcontains Ġan Ġempirical Ġcomparison Ġof Ġseveral Ġexisting Ġapproaches Ġand Ġhence Ġwe Ġhope Ġthat Ġour Ġpaper Ġcan Ġact Ġas Ġa Ġguideline Ġto Ġresearchers Ġattempting Ġto Ġtackle Ġextremely Ġlow - resource Ġtranslation . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġconfirm Ġfurther Ġfine - tun ing Ġfor Ġeach Ġof Ġspecific Ġtranslation Ġdirections . ĠWe Ġwill Ġalso Ġexplore Ġthe Ġway Ġto Ġexploit Ġout - of - domain Ġpseudo - par allel Ġdata , Ġbetter Ġdomain - adapt ation Ġapproaches , Ġand Ġadditional Ġchallenging Ġlanguage Ġpairs . </s> Acknowled gments </s> This Ġwork Ġwas Ġcarried Ġout Ġwhen ĠA iz han ĠI man k ul ova Ġwas Ġtaking Ġup Ġan Ġinternship Ġat ĠN ICT , ĠJapan . ĠWe Ġwould Ġlike Ġto Ġthank Ġthe Ġreviewers Ġfor Ġtheir Ġinsightful Ġcomments . ĠA Ġpart Ġof Ġthis Ġwork Ġwas Ġconducted Ġunder Ġthe Ġprogram ĠâĢ ľ Prom otion Ġof ĠGlobal ĠCommunications ĠPlan : ĠResearch , ĠDevelopment , Ġand ĠSocial ĠDemon stration Ġof ĠMult ilingual ĠSpeech ĠTranslation ĠTechnology âĢ Ŀ Ġof Ġthe ĠMinistry Ġof ĠInternal ĠAffairs Ġand ĠCommunications Ġ( MIC ), ĠJapan .