Answer with content missing: (Evaluation section) Given that in CLIR the primary goal is to get a better ranked list of documents against a translated query, we only report Mean Average Precision (MAP).
ABSTRACTIVE
Results and Analysis
1906.06849
Introduction </s> CL IR Ġsystems Ġretrieve Ġdocuments Ġwritten Ġin Ġa Ġlanguage Ġthat Ġis Ġdifferent Ġfrom Ġsearch Ġquery Ġlanguage ĠB IB REF 0 Ġ. ĠThe Ġprimary Ġobjective Ġof ĠCL IR Ġis Ġto Ġtranslate Ġor Ġproject Ġa Ġquery Ġinto Ġthe Ġlanguage Ġof Ġthe Ġdocument Ġrepository ĠB IB REF 1 Ġ, Ġwhich Ġwe Ġrefer Ġto Ġas ĠRet ri eval ĠCorpus Ġ( RC ). ĠTo Ġthis Ġend , Ġcommon ĠCL IR Ġapproaches Ġtranslate Ġsearch Ġqueries Ġusing Ġa ĠMachine ĠTranslation Ġ( MT ) Ġmodel Ġand Ġthen Ġuse Ġa Ġmon oling ual ĠIR Ġsystem Ġto Ġretrieve Ġfrom ĠRC . ĠIn Ġthis Ġprocess , Ġa Ġtranslation Ġmodel Ġis Ġtreated Ġas Ġa Ġblack Ġbox ĠB IB REF 2 Ġ, Ġand Ġit Ġis Ġusually Ġtrained Ġon Ġa Ġsentence Ġlevel Ġparallel Ġcorpus , Ġwhich Ġwe Ġrefer Ġto Ġas ĠTranslation ĠCorpus Ġ( TC ). </s> We Ġaddress Ġa Ġpit fall Ġof Ġusing Ġexisting ĠMT Ġmodels Ġfor Ġquery Ġtranslation ĠB IB REF 1 Ġ. ĠAn ĠMT Ġmodel Ġtrained Ġon ĠTC Ġdoes Ġnot Ġhave Ġany Ġknowledge Ġof ĠRC . ĠIn Ġan Ġextreme Ġsetting , Ġwhere Ġthere Ġare Ġno Ġcommon Ġterms Ġbetween Ġthe Ġtarget Ġside Ġof ĠTC Ġand ĠRC , Ġa Ġwell Ġtrained Ġand Ġtested Ġtranslation Ġmodel Ġwould Ġfail Ġbecause Ġof Ġvocabulary Ġmismatch Ġbetween Ġthe Ġtranslated Ġquery Ġand Ġdocuments Ġof ĠRC . ĠAssuming Ġa Ġrelaxed Ġscenario Ġwhere Ġsome Ġcommon ality Ġexists Ġbetween Ġtwo Ġcorpor a , Ġa Ġtranslation Ġmodel Ġmight Ġstill Ġperform Ġpoorly , Ġfavoring Ġterms Ġthat Ġare Ġmore Ġlikely Ġin ĠTC Ġbut Ġrare Ġin ĠRC . ĠOur Ġhypothesis Ġis Ġthat Ġa Ġsearch Ġquery Ġtranslation Ġmodel Ġwould Ġperform Ġbetter Ġif Ġa Ġtranslated Ġquery Ġterm Ġis Ġlikely Ġto Ġappear Ġin Ġthe Ġboth Ġretrieval Ġand Ġtranslation Ġcorpor a , Ġa Ġproperty Ġwe Ġcall Ġbalanced Ġtranslation . </s> To Ġachieve Ġbalanced Ġtranslations , Ġit Ġis Ġdesired Ġto Ġconstruct Ġan ĠMT Ġmodel Ġthat Ġis Ġaware Ġof ĠRC Ġvocabulary . ĠDifferent Ġtypes Ġof ĠMT Ġapproaches Ġhave Ġbeen Ġadopted Ġfor ĠCL IR Ġtask , Ġsuch Ġas Ġdictionary - based ĠMT , Ġrule - based ĠMT , Ġstatistical ĠMT Ġetc . ĠB IB REF 3 Ġ. ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġa Ġneural Ġsearch Ġquery Ġtranslation Ġapproach Ġhas Ġyet Ġto Ġbe Ġtaken Ġby Ġthe Ġcommunity . ĠN MT Ġmodels Ġwith Ġattention Ġbased Ġenc oder - dec oder Ġtechniques Ġhave Ġachieved Ġstate - of - the - art Ġperformance Ġfor Ġseveral Ġlanguage Ġpairs ĠB IB REF 4 Ġ. ĠWe Ġpropose Ġa Ġmulti - task Ġlearning ĠN MT Ġarchitecture Ġthat Ġtakes ĠRC Ġvocabulary Ġinto Ġaccount Ġby Ġlearning ĠRe lev ance - based ĠAux iliary ĠTask Ġ( R AT ). ĠR AT Ġis Ġinspired Ġfrom Ġtwo Ġword Ġembed ding Ġlearning Ġapproaches : ĠRe lev ance - based ĠWord ĠEmb edd ing Ġ( R WE ) ĠB IB REF 5 Ġand ĠContinuous ĠBag Ġof ĠWords Ġ( CB OW ) Ġembed ding ĠB IB REF 6 Ġ. ĠWe Ġshow Ġthat Ġlearning ĠN MT Ġwith ĠR AT Ġenables Ġit Ġto Ġgenerate Ġbalanced Ġtranslation . </s> N MT Ġmodels Ġlearn Ġto Ġencode Ġthe Ġmeaning Ġof Ġa Ġsource Ġsentence Ġand Ġdecode Ġthe Ġmeaning Ġto Ġgenerate Ġwords Ġin Ġa Ġtarget Ġlanguage ĠB IB REF 7 Ġ. ĠIn Ġthe Ġproposed Ġmulti - task Ġlearning Ġmodel , ĠR AT Ġshares Ġthe Ġdec oder Ġembed ding Ġand Ġfinal Ġrepresentation Ġlayer Ġwith ĠN MT . ĠOur Ġarchitecture Ġanswers Ġthe Ġfollowing Ġquestion : ĠIn Ġthe Ġdecoding Ġstage , Ġcan Ġwe Ġrestrict Ġan ĠN MT Ġmodel Ġso Ġthat Ġit Ġdoes Ġnot Ġonly Ġgenerate Ġterms Ġthat Ġare Ġhighly Ġlikely Ġin ĠTC ? . ĠWe Ġshow Ġthat Ġtraining Ġa Ġstrong Ġbaseline ĠN MT Ġwith ĠR AT Ġroughly Ġachieves Ġ16 % Ġimprovement Ġover Ġthe Ġbaseline . ĠUsing Ġa Ġqualitative Ġanalysis , Ġwe Ġfurther Ġshow Ġthat ĠR AT Ġworks Ġas Ġa Ġregular izer Ġand Ġprohibits ĠN MT Ġto Ġover fit Ġto ĠTC Ġvocabulary . </s> Bal anced ĠTranslation ĠApproach </s> We Ġtrain ĠN MT Ġwith ĠR AT Ġto Ġachieve Ġbetter Ġquery Ġtranslations . ĠWe Ġimprove Ġa Ġrecently Ġproposed ĠN MT Ġbaseline , ĠTrans former , Ġthat Ġachieves Ġstate - of - the - art Ġresults Ġfor Ġsentence Ġpairs Ġin Ġsome Ġlanguages ĠB IB REF 8 Ġ. ĠWe Ġdiscuss ĠTrans former , ĠR AT , Ġand Ġour Ġmulti - task Ġlearning Ġarchitecture Ġthat Ġachieves Ġbalanced Ġtranslation . </s> N MT Ġand ĠTrans former </s> In Ġprinciple , Ġwe Ġcould Ġadopt Ġany ĠN MT Ġand Ġcombine Ġit Ġwith ĠR AT . ĠAn ĠN MT Ġsystem Ġdirectly Ġmodels Ġthe Ġconditional Ġprobability ĠIN LINE FORM 0 Ġof Ġtranslating Ġa Ġsource Ġsentence , ĠIN LINE FORM 1 Ġ, Ġto Ġa Ġtarget Ġsentence ĠIN LINE FORM 2 Ġ. ĠA Ġbasic Ġform Ġof ĠN MT Ġcomprises Ġtwo Ġcomponents : Ġ( a ) Ġan Ġenc oder Ġthat Ġcomp utes Ġthe Ġrepresentations Ġor Ġmeaning Ġof ĠIN LINE FORM 3 Ġand Ġ( b ) Ġa Ġdec oder Ġthat Ġgenerates Ġone Ġtarget Ġword Ġat Ġa Ġtime . ĠState - of - the - art ĠN MT Ġmodels Ġhave Ġan Ġattention Ġcomponent Ġthat ĠâĢ ľ se ar ches Ġfor Ġa Ġset Ġof Ġpositions Ġin Ġa Ġsource Ġsentence Ġwhere Ġthe Ġmost Ġrelevant Ġinformation Ġis Ġconcentrated âĢ Ŀ ĠB IB REF 4 Ġ. </s> For Ġthis Ġstudy , Ġwe Ġuse Ġa Ġstate - of - the - art ĠN MT Ġmodel , ĠTrans former ĠB IB REF 8 Ġ, Ġthat Ġuses Ġpositional Ġencoding Ġand Ġself Ġattention Ġmechanism Ġto Ġachieve Ġthree Ġbenefits Ġover Ġthe Ġexisting Ġconv olution al Ġor Ġrecurrent Ġneural Ġnetwork Ġbased Ġmodels : Ġ( a ) Ġreduced Ġcomputational Ġcomplexity Ġof Ġeach Ġlayer , Ġ( b ) Ġparallel Ġcomputation , Ġand Ġ( c ) Ġpath Ġlength Ġbetween Ġlong - range Ġdependencies . </s> Re lev ance - based ĠAux iliary ĠTask Ġ( R AT ) </s> We Ġdefine ĠR AT Ġa Ġvariant Ġof Ġword Ġembed ding Ġtask ĠB IB REF 6 Ġ. ĠWord Ġembed ding Ġapproaches Ġlearn Ġhigh Ġdimensional Ġdense Ġrepresentations Ġfor Ġwords Ġand Ġtheir Ġobjective Ġfunctions Ġaim Ġto Ġcapture Ġcontextual Ġinformation Ġaround Ġa Ġword . ĠB IB REF 5 Ġproposed Ġa Ġmodel Ġthat Ġlearns Ġword Ġvectors Ġby Ġpredicting Ġwords Ġin Ġrelevant Ġdocuments Ġretrieved Ġagainst Ġa Ġsearch Ġquery . ĠWe Ġfollow Ġthe Ġsame Ġidea Ġbut Ġuse Ġa Ġsimpler Ġlearning Ġapproach Ġthat Ġis Ġsuitable Ġfor Ġour Ġtask . ĠThey Ġtried Ġto Ġpredict Ġwords Ġfrom Ġthe Ġrelevance Ġmodel ĠB IB REF 9 Ġcomputed Ġfrom Ġa Ġquery , Ġwhich Ġdoes Ġnot Ġwork Ġfor Ġour Ġtask Ġbecause Ġthe Ġconnection Ġbetween Ġa Ġquery Ġand Ġranked Ġsentences Ġfalls Ġrapidly Ġafter Ġthe Ġtop Ġone Ġ( see Ġbelow ). </s> We Ġconsider Ġtwo Ġdata Ġsources Ġfor Ġlearning ĠN MT Ġand ĠR AT Ġjointly . ĠThe Ġfirst Ġone Ġis Ġa Ġsentence - level Ġparallel Ġcorpus , Ġwhich Ġwe Ġrefer Ġto Ġas Ġtranslation Ġcorpus , ĠIN LINE FORM 0 Ġ. ĠThe Ġsecond Ġone Ġis Ġthe Ġretrieval Ġcorpus , Ġwhich Ġis Ġa Ġcollection Ġof ĠIN LINE FORM 1 Ġdocuments ĠIN LINE FORM 2 Ġin Ġthe Ġsame Ġlanguage Ġas ĠIN LINE FORM 3 Ġ. ĠOur Ġword - embed ding Ġapproach Ġtakes Ġeach ĠIN LINE FORM 4 Ġ, Ġuses Ġit Ġas Ġa Ġquery Ġto Ġretrieve Ġthe Ġtop Ġdocument ĠIN LINE FORM 5 Ġ. ĠAfter Ġthat Ġwe Ġobtain ĠIN LINE FORM 6 Ġby Ġconc aten ating ĠIN LINE FORM 7 Ġwith ĠIN LINE FORM 8 Ġand Ġrandomly Ġshuff ling Ġthe Ġwords Ġin Ġthe Ġcombined Ġsequence . ĠWe Ġthen Ġaugment ĠIN LINE FORM 9 Ġusing ĠIN LINE FORM 10 Ġand Ġobtain Ġa Ġdataset , ĠIN LINE FORM 11 Ġ. ĠWe Ġuse ĠIN LINE FORM 12 Ġto Ġlearn Ġa Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġembed ding Ġas Ġproposed Ġby ĠB IB REF 6 Ġ. ĠThis Ġlearning Ġcomponent Ġshares Ġtwo Ġlayers Ġwith Ġthe ĠN MT Ġmodel . ĠThe Ġgoal Ġis Ġto Ġexpose Ġthe Ġretrieval Ġcorpus ' Ġvocabulary Ġto Ġthe ĠN MT Ġmodel . ĠWe Ġdiscuss Ġlayer Ġsharing Ġin Ġthe Ġnext Ġsection . </s> We Ġselect Ġthe Ġsingle Ġtop Ġdocument Ġretrieved Ġagainst Ġa Ġsentence ĠIN LINE FORM 0 Ġbecause Ġa Ġsentence Ġis Ġa Ġweak Ġrepresentation Ġof Ġinformation Ġneed . ĠAs Ġa Ġresult , Ġdocuments Ġat Ġlower Ġranks Ġshow Ġheavy Ġshift Ġfrom Ġthe Ġcontext Ġof Ġthe Ġsentence Ġquery . ĠWe Ġverified Ġthis Ġby Ġobserving Ġthat Ġa Ġrelevance Ġmodel Ġconstructed Ġfrom Ġthe Ġtop ĠIN LINE FORM 1 Ġdocuments Ġdoes Ġnot Ġperform Ġwell Ġin Ġthis Ġsetting . ĠWe Ġthus Ġdev iate Ġfrom Ġthe Ġrelevance Ġmodel Ġbased Ġapproach Ġtaken Ġby ĠB IB REF 5 Ġand Ġlearn Ġover Ġthe Ġrandom Ġshuff ling Ġof ĠIN LINE FORM 2 Ġand Ġa Ġsingle Ġdocument . ĠRandom Ġshuff ling Ġhas Ġshown Ġreasonable Ġeffectiveness Ġfor Ġword Ġembed ding Ġconstruction Ġfor Ġcomparable Ġcorpus ĠB IB REF 10 Ġ. </s> Multi - task ĠN MT ĠArchitecture </s> Our Ġbalanced Ġtranslation Ġarchitecture Ġis Ġpresented Ġin ĠFigure ĠFIG REF 3 Ġ. ĠThis Ġarchitecture Ġis ĠN MT - model Ġag nostic Ġas Ġwe Ġonly Ġpropose Ġto Ġshare Ġtwo Ġlayers Ġcommon Ġto Ġmost ĠN MT s : Ġthe Ġtrain able Ġtarget Ġembed ding Ġlayer Ġand Ġthe Ġtransformation Ġfunction ĠB IB REF 7 Ġthat Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġthe Ġunion Ġof Ġthe Ġvocabulary Ġof ĠTC Ġand ĠRC . ĠHence , Ġthe Ġsize Ġof Ġthe Ġvocabulary , ĠIN LINE FORM 0 Ġ, Ġis Ġmuch Ġlarger Ġcompared Ġto ĠTC Ġand Ġit Ġenables Ġthe Ġmodel Ġto Ġaccess ĠRC . ĠIn Ġorder Ġto Ġshow Ġtask Ġsharing Ġclearly Ġwe Ġplaced Ġtwo Ġshared Ġlayers Ġbetween ĠN MT Ġand ĠR AT Ġin ĠFigure ĠFIG REF 3 Ġ. ĠWe Ġalso Ġshow Ġthe Ġtwo Ġdifferent Ġpaths Ġtaken Ġby Ġtwo Ġdifferent Ġtasks Ġat Ġtraining Ġtime : Ġthe ĠN MT Ġpath Ġin Ġshown Ġwith Ġred Ġarrows Ġwhile Ġthe ĠR AT Ġpath Ġis Ġshown Ġin Ġgreen Ġarrows . </s> On ĠN MT Ġpath Ġtraining Ġloss Ġis Ġcomputed Ġas Ġthe Ġsum Ġof Ġterm - wise Ġsoft max Ġwith Ġcross - ent ropy Ġloss Ġof Ġthe Ġpredicted Ġtranslation Ġand Ġthe Ġhuman Ġtranslation Ġand Ġit Ġsummed Ġover Ġa Ġbatch Ġof Ġsentence Ġpairs , ĠIN LINE FORM 0 Ġ. ĠWe Ġalso Ġuse Ġa Ġsimilar Ġloss Ġfunction Ġto Ġtrain Ġword Ġembed ding Ġover Ġa Ġset Ġof Ġcontext Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġpivot Ġ( ĠIN LINE FORM 2 Ġ) Ġpairs Ġformed Ġusing ĠIN LINE FORM 3 Ġas Ġquery Ġto Ġretrieve ĠIN LINE FORM 4 Ġusing ĠQuery ĠLike lihood Ġ( QL ) Ġrank er , ĠIN LINE FORM 5 Ġ. ĠThis Ġobjective Ġis Ġsimilar Ġto ĠCB OW Ġword Ġembed ding Ġas Ġcontext Ġis Ġused Ġto Ġpredict Ġpivot Ġword ĠHere , Ġwe Ġuse Ġa Ġscaling Ġfactor ĠIN LINE FORM 6 Ġ, Ġto Ġhave Ġa Ġbalance Ġbetween Ġthe Ġgrad ients Ġfrom Ġthe ĠN MT Ġloss Ġand ĠR AT Ġloss . ĠFor ĠR AT , Ġthe Ġcontext Ġis Ġdrawn Ġfrom Ġa Ġcontext Ġwindow Ġfollowing ĠB IB REF 6 Ġ. </s> In Ġthe Ġfigure , ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrepresents Ġthe Ġtop Ġdocument Ġretrieved Ġagainst ĠIN LINE FORM 2 Ġ. ĠThe Ġshuff ler Ġcomponent Ġshuff les ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġand Ġcreates Ġ( context , Ġpivot ) Ġpairs . ĠAfter Ġthat Ġthose Ġdata Ġpoints Ġare Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġlinear Ġprojection Ġlayer Ġand Ġeventually Ġto Ġthe Ġtransformation Ġfunction . ĠInt uitive ly , Ġthe Ġword Ġembed ding Ġtask Ġis Ġsimilar Ġto ĠN MT Ġas Ġit Ġtries Ġto Ġassign Ġa Ġlarge Ġprobability Ġmass Ġto Ġa Ġtarget Ġword Ġgiven Ġa Ġcontext . ĠHowever , Ġit Ġenables Ġthe Ġtransformation Ġfunction Ġand Ġdecoding Ġlayer Ġto Ġassign Ġprobability Ġmass Ġnot Ġonly Ġto Ġterms Ġfrom ĠTC , Ġbut Ġalso Ġto Ġterms Ġfrom ĠRC . ĠThis Ġimplicitly Ġprohibits ĠN MT Ġto Ġover fit Ġand Ġprovides Ġa Ġregular ization Ġeffect . ĠA Ġsimilar Ġtechnique Ġwas Ġproposed Ġby ĠB IB REF 11 Ġto Ġhandle Ġout - of - voc abulary Ġor Ġless Ġfrequent Ġwords Ġfor ĠN MT . ĠFor Ġthese Ġterms Ġthey Ġenabled Ġthe Ġtransformation Ġ( also Ġcalled Ġthe Ġsoft max Ġcross - ent ropy Ġlayer ) Ġto Ġfairly Ġdistribute Ġprobability Ġmass Ġamong Ġsimilar Ġwords . ĠIn Ġcontrast , Ġwe Ġfocus Ġon Ġrelevant Ġterms Ġrather Ġthan Ġsimilar Ġterms . </s> Results Ġand ĠAnalysis </s> Table ĠT AB REF 14 Ġshows Ġthe Ġeffectiveness Ġof Ġour Ġmodel Ġ( multi - task Ġtransformer ) Ġover Ġthe Ġbaseline Ġtransformer ĠB IB REF 8 Ġ. ĠOur Ġmodel Ġachieves Ġsignificant Ġperformance Ġgains Ġin Ġthe Ġtest Ġsets Ġover Ġthe Ġbaseline Ġfor Ġboth ĠItalian Ġand ĠFinnish Ġquery Ġtranslation . ĠThe Ġoverall Ġlow ĠMAP Ġfor ĠN MT Ġcan Ġpossibly Ġbe Ġimproved Ġwith Ġlarger ĠTC . ĠMoreover , Ġour Ġmodel Ġvalidation Ġapproach Ġrequires Ġaccess Ġto ĠRC Ġindex , Ġand Ġit Ġslows Ġdown Ġoverall Ġtraining Ġprocess . ĠHence , Ġwe Ġcould Ġnot Ġtrain Ġour Ġmodel Ġfor Ġa Ġlarge Ġnumber Ġof Ġepoch s Ġ- Ġit Ġmay Ġbe Ġanother Ġcause Ġof Ġthe Ġlow Ġperformance . </s> We Ġwant Ġto Ġshow Ġthat Ġtranslation Ġterms Ġgenerated Ġby Ġour Ġmulti - task Ġtransformer Ġare Ġroughly Ġequally Ġlikely Ġto Ġbe Ġseen Ġin Ġthe ĠEuro par l Ġcorpus Ġ( TC ) Ġor Ġthe ĠCLE F Ġcorpus Ġ( RC ). ĠGiven Ġa Ġtranslation Ġterm ĠIN LINE FORM 0 Ġ, Ġwe Ġcompute Ġthe Ġratio Ġof Ġthe Ġprobability Ġof Ġseeing ĠIN LINE FORM 1 Ġin ĠTC Ġand ĠRC , ĠIN LINE FORM 2 Ġ. ĠHere , ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġis Ġcalculated Ġsimilarly . ĠGiven Ġa Ġquery ĠIN LINE FORM 5 Ġand Ġits Ġtranslation ĠIN LINE FORM 6 Ġprovided Ġby Ġmodel ĠIN LINE FORM 7 Ġ, Ġwe Ġcalculate Ġthe Ġbalance Ġof ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ. ĠIf ĠIN LINE FORM 10 Ġis Ġclose Ġto Ġ1 , Ġthe Ġtranslation Ġterms Ġare Ġas Ġlikely Ġin ĠTC Ġas Ġin ĠRC . ĠFigure ĠFIG REF 15 Ġshows Ġthe Ġbalance Ġvalues Ġfor Ġtransformer Ġand Ġour Ġmodel Ġfor Ġa Ġrandom Ġsample Ġof Ġ20 Ġqueries Ġfrom Ġthe Ġvalidation Ġset Ġof ĠItalian Ġqueries , Ġrespectively . ĠFigure ĠFIG REF 16 Ġshows Ġthe Ġbalance Ġvalues Ġfor Ġtransformer Ġand Ġour Ġmodel Ġfor Ġa Ġrandom Ġsample Ġof Ġ20 Ġqueries Ġfrom Ġthe Ġtest Ġset Ġof ĠItalian Ġqueries , Ġrespectively . ĠIt Ġis Ġevident Ġthat Ġour Ġmodel Ġachieves Ġbetter Ġbalance Ġcompared Ġto Ġbaseline Ġtransformer , Ġexcept Ġfor Ġa Ġvery Ġfew Ġcases . </s> Given Ġa Ġquery ĠIN LINE FORM 0 Ġ, Ġconsider ĠIN LINE FORM 1 Ġas Ġthe Ġset Ġof Ġterms Ġfrom Ġhuman Ġtranslation Ġof ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġas Ġthe Ġset Ġof Ġtranslation Ġterms Ġgenerated Ġby Ġmodel ĠIN LINE FORM 4 Ġ. ĠWe Ġdefine ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġas Ġprecision Ġand Ġrecall Ġof ĠIN LINE FORM 7 Ġfor Ġmodel ĠIN LINE FORM 8 Ġ. ĠIn ĠTable ĠT AB REF 19 Ġ, Ġwe Ġreport Ġaverage Ġprecision Ġand Ġrecall Ġfor Ġboth Ġtransformer Ġand Ġour Ġmodel Ġacross Ġour Ġtrain Ġand Ġvalidation Ġquery Ġset Ġover Ġtwo Ġlanguage Ġpairs . ĠOur Ġmodel Ġgenerates Ġprecise Ġtranslation , Ġi . e . Ġit Ġavoids Ġterms Ġthat Ġmight Ġbe Ġuseless Ġor Ġeven Ġharmful Ġfor Ġretrieval . ĠGenerally , Ġfrom Ġour Ġobservation , Ġavoided Ġterms Ġare Ġhighly Ġlikely Ġterms Ġfrom ĠTC Ġand Ġthey Ġare Ġgenerated Ġbecause Ġof Ġtranslation Ġmodel Ġover fitting . ĠOur Ġmodel Ġachieves Ġa Ġregular ization Ġeffect Ġthrough Ġan Ġauxiliary Ġtask . ĠThis Ġconfirms Ġresults Ġfrom Ġexisting Ġmulti - t asking Ġliterature ĠB IB REF 15 Ġ. </s> To Ġexplore Ġtranslation Ġquality , Ġconsider Ġpair Ġof Ġsample Ġtranslations Ġprovided Ġby Ġtwo Ġmodels . ĠFor Ġexample , Ġagainst Ġan ĠItalian Ġquery , Ġmed ag lia Ġo ro Ġsuper Ġv inse Ġmed ag lia Ġo ro Ġsuper Ġo lim pi adi Ġin vern ali Ġl ille hammer , Ġtranslated Ġterm Ġset Ġfrom Ġour Ġmodel Ġis Ġ{ gold , Ġcoin , Ġsuper , Ġfree , Ġharmon ising , Ġwon , Ġwinter , Ġo lymp ics }, Ġwhile Ġtransformer Ġoutput Ġis Ġ{ oly mp ic , Ġgold , Ġone , Ġcoin , Ġsuper , Ġyears , Ġwon , Ġparliament , Ġalso , Ġtwo , Ġwinter }. ĠTerm Ġset Ġfrom Ġhuman Ġtranslation Ġis : Ġ{ super , Ġgold , Ġmedal , Ġwon , Ġl ille hammer , Ġo ly pm ic , Ġwinter , Ġgames }. ĠTrans former Ġcomes Ġup Ġwith Ġterms Ġlike Ġparliament , Ġalso , Ġtwo Ġand Ġyears Ġthat Ġnever Ġappears Ġin Ġhuman Ġtranslation . ĠWe Ġfound Ġthat Ġthese Ġterms Ġare Ġvery Ġlikely Ġin ĠEuro par l Ġand Ġrare Ġin ĠCLE F . ĠOur Ġmodel Ġalso Ġgenerates Ġterms Ġsuch Ġas Ġharmon ising , Ġfree , Ġo lymp ics Ġthat Ġnot Ġgenerated Ġby Ġtransformer . ĠHowever , Ġwe Ġfound Ġthat Ġthese Ġterms Ġare Ġequally Ġlikely Ġin ĠEuro par l Ġand ĠCLE F . </s> Conclusion </s> We Ġpresent Ġa Ġmulti - task Ġlearning Ġarchitecture Ġto Ġlearn ĠN MT Ġfor Ġsearch Ġquery Ġtranslation . ĠAs Ġthe Ġmotivating Ġtask Ġis ĠCL IR , Ġwe Ġevaluated Ġthe Ġranking Ġeffectiveness Ġof Ġour Ġproposed Ġarchitecture . ĠWe Ġused Ġsentences Ġfrom Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġcorpus Ġas Ġqueries Ġto Ġretrieve Ġrelevant Ġdocument Ġand Ġuse Ġterms Ġfrom Ġthose Ġdocuments Ġto Ġtrain Ġa Ġword Ġembed ding Ġmodel Ġalong Ġwith ĠN MT . ĠOne Ġbig Ġchallenge Ġin Ġthis Ġlandscape Ġis Ġto Ġsample Ġmeaningful Ġqueries Ġfrom Ġsentences Ġas Ġsentences Ġdo Ġnot Ġdirectly Ġconvey Ġinformation Ġneed . ĠIn Ġthe Ġfuture , Ġwe Ġhope Ġto Ġlearn Ġmodels Ġthat Ġare Ġable Ġto Ġsample Ġsearch Ġqueries Ġor Ġinformation Ġneeds Ġfrom Ġsentences Ġand Ġuse Ġthe Ġoutput Ġof Ġthat Ġmodel Ġto Ġget Ġrelevant Ġdocuments . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby Ġthe ĠCenter Ġfor ĠIntelligent ĠInformation ĠRet ri eval Ġand Ġin Ġpart Ġby Ġthe ĠAir ĠForce ĠResearch ĠLaboratory Ġ( AF RL ) Ġand ĠI AR PA Ġunder Ġcontract Ġ# FA 86 50 - 17 - C - 9 118 Ġunder Ġsubcontract Ġ# 147 75 Ġfrom ĠRay the on ĠBB N ĠTechnologies ĠCorporation . ĠAny Ġopinions , Ġfindings Ġand Ġconclusions Ġor Ġrecommendations Ġexpressed Ġin Ġthis Ġmaterial Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġdo Ġnot Ġnecessarily Ġreflect Ġthose Ġof Ġthe Ġsponsor . </s> L oss ĠFunction Ġand ĠVal idation ĠPerformance ĠAnalysis </s> We Ġshow Ġthe Ġloss Ġfunction Ġanalysis Ġof Ġtransformer Ġand Ġour Ġmodel . ĠFigure ĠFIG REF 23 Ġshows Ġthe Ġvalidation Ġperformance Ġof Ġtransformer Ġagainst Ġglobal Ġtraining Ġsteps . ĠFigure ĠFIG REF 21 Ġshow Ġthe Ġvalidation Ġperformance Ġof Ġour Ġmodel Ġfor Ġthe Ġsame Ġnumber Ġof Ġglobal Ġsteps . ĠFigure ĠFIG REF 22 Ġshows Ġthat ĠN MT Ġloss Ġis Ġgoing Ġdown Ġwith Ġthe Ġnumber Ġof Ġsteps , Ġwhile ĠFigure ĠFIG REF 20 Ġshows Ġthe Ġdegradation Ġof Ġthe Ġloss Ġof Ġour Ġproposed ĠR AT Ġtask .
Introduction CLIR systems retrieve documents written in a language that is different from search query language BIBREF0 . The primary objective of CLIR is to translate or project a query into the language of the document repository BIBREF1 , which we refer to as Retrieval Corpus (RC). To this end, common CLIR approaches translate search queries using a Machine Translation (MT) model and then use a monolingual IR system to retrieve from RC. In this process, a translation model is treated as a black box BIBREF2 , and it is usually trained on a sentence level parallel corpus, which we refer to as Translation Corpus (TC). We address a pitfall of using existing MT models for query translation BIBREF1 . An MT model trained on TC does not have any knowledge of RC. In an extreme setting, where there are no common terms between the target side of TC and RC, a well trained and tested translation model would fail because of vocabulary mismatch between the translated query and documents of RC. Assuming a relaxed scenario where some commonality exists between two corpora, a translation model might still perform poorly, favoring terms that are more likely in TC but rare in RC. Our hypothesis is that a search query translation model would perform better if a translated query term is likely to appear in the both retrieval and translation corpora, a property we call balanced translation. To achieve balanced translations, it is desired to construct an MT model that is aware of RC vocabulary. Different types of MT approaches have been adopted for CLIR task, such as dictionary-based MT, rule-based MT, statistical MT etc. BIBREF3 . However, to the best of our knowledge, a neural search query translation approach has yet to be taken by the community. NMT models with attention based encoder-decoder techniques have achieved state-of-the-art performance for several language pairs BIBREF4 . We propose a multi-task learning NMT architecture that takes RC vocabulary into account by learning Relevance-based Auxiliary Task (RAT). RAT is inspired from two word embedding learning approaches: Relevance-based Word Embedding (RWE) BIBREF5 and Continuous Bag of Words (CBOW) embedding BIBREF6 . We show that learning NMT with RAT enables it to generate balanced translation. NMT models learn to encode the meaning of a source sentence and decode the meaning to generate words in a target language BIBREF7 . In the proposed multi-task learning model, RAT shares the decoder embedding and final representation layer with NMT. Our architecture answers the following question: In the decoding stage, can we restrict an NMT model so that it does not only generate terms that are highly likely in TC?. We show that training a strong baseline NMT with RAT roughly achieves 16% improvement over the baseline. Using a qualitative analysis, we further show that RAT works as a regularizer and prohibits NMT to overfit to TC vocabulary. Balanced Translation Approach We train NMT with RAT to achieve better query translations. We improve a recently proposed NMT baseline, Transformer, that achieves state-of-the-art results for sentence pairs in some languages BIBREF8 . We discuss Transformer, RAT, and our multi-task learning architecture that achieves balanced translation. NMT and Transformer In principle, we could adopt any NMT and combine it with RAT. An NMT system directly models the conditional probability INLINEFORM0 of translating a source sentence, INLINEFORM1 , to a target sentence INLINEFORM2 . A basic form of NMT comprises two components: (a) an encoder that computes the representations or meaning of INLINEFORM3 and (b) a decoder that generates one target word at a time. State-of-the-art NMT models have an attention component that “searches for a set of positions in a source sentence where the most relevant information is concentrated” BIBREF4 . For this study, we use a state-of-the-art NMT model, Transformer BIBREF8 , that uses positional encoding and self attention mechanism to achieve three benefits over the existing convolutional or recurrent neural network based models: (a) reduced computational complexity of each layer, (b) parallel computation, and (c) path length between long-range dependencies. Relevance-based Auxiliary Task (RAT) We define RAT a variant of word embedding task BIBREF6 . Word embedding approaches learn high dimensional dense representations for words and their objective functions aim to capture contextual information around a word. BIBREF5 proposed a model that learns word vectors by predicting words in relevant documents retrieved against a search query. We follow the same idea but use a simpler learning approach that is suitable for our task. They tried to predict words from the relevance model BIBREF9 computed from a query, which does not work for our task because the connection between a query and ranked sentences falls rapidly after the top one (see below). We consider two data sources for learning NMT and RAT jointly. The first one is a sentence-level parallel corpus, which we refer to as translation corpus, INLINEFORM0 . The second one is the retrieval corpus, which is a collection of INLINEFORM1 documents INLINEFORM2 in the same language as INLINEFORM3 . Our word-embedding approach takes each INLINEFORM4 , uses it as a query to retrieve the top document INLINEFORM5 . After that we obtain INLINEFORM6 by concatenating INLINEFORM7 with INLINEFORM8 and randomly shuffling the words in the combined sequence. We then augment INLINEFORM9 using INLINEFORM10 and obtain a dataset, INLINEFORM11 . We use INLINEFORM12 to learn a continuous bag of words (CBOW) embedding as proposed by BIBREF6 . This learning component shares two layers with the NMT model. The goal is to expose the retrieval corpus' vocabulary to the NMT model. We discuss layer sharing in the next section. We select the single top document retrieved against a sentence INLINEFORM0 because a sentence is a weak representation of information need. As a result, documents at lower ranks show heavy shift from the context of the sentence query. We verified this by observing that a relevance model constructed from the top INLINEFORM1 documents does not perform well in this setting. We thus deviate from the relevance model based approach taken by BIBREF5 and learn over the random shuffling of INLINEFORM2 and a single document. Random shuffling has shown reasonable effectiveness for word embedding construction for comparable corpus BIBREF10 . Multi-task NMT Architecture Our balanced translation architecture is presented in Figure FIGREF3 . This architecture is NMT-model agnostic as we only propose to share two layers common to most NMTs: the trainable target embedding layer and the transformation function BIBREF7 that outputs a probability distribution over the union of the vocabulary of TC and RC. Hence, the size of the vocabulary, INLINEFORM0 , is much larger compared to TC and it enables the model to access RC. In order to show task sharing clearly we placed two shared layers between NMT and RAT in Figure FIGREF3 . We also show the two different paths taken by two different tasks at training time: the NMT path in shown with red arrows while the RAT path is shown in green arrows. On NMT path training loss is computed as the sum of term-wise softmax with cross-entropy loss of the predicted translation and the human translation and it summed over a batch of sentence pairs, INLINEFORM0 . We also use a similar loss function to train word embedding over a set of context ( INLINEFORM1 ) and pivot ( INLINEFORM2 ) pairs formed using INLINEFORM3 as query to retrieve INLINEFORM4 using Query Likelihood (QL) ranker, INLINEFORM5 . This objective is similar to CBOW word embedding as context is used to predict pivot word Here, we use a scaling factor INLINEFORM6 , to have a balance between the gradients from the NMT loss and RAT loss. For RAT, the context is drawn from a context window following BIBREF6 . In the figure, INLINEFORM0 and INLINEFORM1 represents the top document retrieved against INLINEFORM2 . The shuffler component shuffles INLINEFORM3 and INLINEFORM4 and creates (context, pivot) pairs. After that those data points are passed through a fully connected linear projection layer and eventually to the transformation function. Intuitively, the word embedding task is similar to NMT as it tries to assign a large probability mass to a target word given a context. However, it enables the transformation function and decoding layer to assign probability mass not only to terms from TC, but also to terms from RC. This implicitly prohibits NMT to overfit and provides a regularization effect. A similar technique was proposed by BIBREF11 to handle out-of-vocabulary or less frequent words for NMT. For these terms they enabled the transformation (also called the softmax cross-entropy layer) to fairly distribute probability mass among similar words. In contrast, we focus on relevant terms rather than similar terms. Results and Analysis Table TABREF14 shows the effectiveness of our model (multi-task transformer) over the baseline transformer BIBREF8 . Our model achieves significant performance gains in the test sets over the baseline for both Italian and Finnish query translation. The overall low MAP for NMT can possibly be improved with larger TC. Moreover, our model validation approach requires access to RC index, and it slows down overall training process. Hence, we could not train our model for a large number of epochs - it may be another cause of the low performance. We want to show that translation terms generated by our multi-task transformer are roughly equally likely to be seen in the Europarl corpus (TC) or the CLEF corpus (RC). Given a translation term INLINEFORM0 , we compute the ratio of the probability of seeing INLINEFORM1 in TC and RC, INLINEFORM2 . Here, INLINEFORM3 and INLINEFORM4 is calculated similarly. Given a query INLINEFORM5 and its translation INLINEFORM6 provided by model INLINEFORM7 , we calculate the balance of INLINEFORM8 , INLINEFORM9 . If INLINEFORM10 is close to 1, the translation terms are as likely in TC as in RC. Figure FIGREF15 shows the balance values for transformer and our model for a random sample of 20 queries from the validation set of Italian queries, respectively. Figure FIGREF16 shows the balance values for transformer and our model for a random sample of 20 queries from the test set of Italian queries, respectively. It is evident that our model achieves better balance compared to baseline transformer, except for a very few cases. Given a query INLINEFORM0 , consider INLINEFORM1 as the set of terms from human translation of INLINEFORM2 and INLINEFORM3 as the set of translation terms generated by model INLINEFORM4 . We define INLINEFORM5 and INLINEFORM6 as precision and recall of INLINEFORM7 for model INLINEFORM8 . In Table TABREF19 , we report average precision and recall for both transformer and our model across our train and validation query set over two language pairs. Our model generates precise translation, i.e. it avoids terms that might be useless or even harmful for retrieval. Generally, from our observation, avoided terms are highly likely terms from TC and they are generated because of translation model overfitting. Our model achieves a regularization effect through an auxiliary task. This confirms results from existing multi-tasking literature BIBREF15 . To explore translation quality, consider pair of sample translations provided by two models. For example, against an Italian query, medaglia oro super vinse medaglia oro super olimpiadi invernali lillehammer, translated term set from our model is {gold, coin, super, free, harmonising, won, winter, olympics}, while transformer output is {olympic, gold, one, coin, super, years, won, parliament, also, two, winter}. Term set from human translation is: {super, gold, medal, won, lillehammer, olypmic, winter, games}. Transformer comes up with terms like parliament, also, two and years that never appears in human translation. We found that these terms are very likely in Europarl and rare in CLEF. Our model also generates terms such as harmonising, free, olympics that not generated by transformer. However, we found that these terms are equally likely in Europarl and CLEF. Conclusion We present a multi-task learning architecture to learn NMT for search query translation. As the motivating task is CLIR, we evaluated the ranking effectiveness of our proposed architecture. We used sentences from the target side of the parallel corpus as queries to retrieve relevant document and use terms from those documents to train a word embedding model along with NMT. One big challenge in this landscape is to sample meaningful queries from sentences as sentences do not directly convey information need. In the future, we hope to learn models that are able to sample search queries or information needs from sentences and use the output of that model to get relevant documents. Acknowledgments This work was supported in part by the Center for Intelligent Information Retrieval and in part by the Air Force Research Laboratory (AFRL) and IARPA under contract #FA8650-17-C-9118 under subcontract #14775 from Raytheon BBN Technologies Corporation. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor. Loss Function and Validation Performance Analysis We show the loss function analysis of transformer and our model. Figure FIGREF23 shows the validation performance of transformer against global training steps. Figure FIGREF21 show the validation performance of our model for the same number of global steps. Figure FIGREF22 shows that NMT loss is going down with the number of steps, while Figure FIGREF20 shows the degradation of the loss of our proposed RAT task.
what quantitative analysis is done?
5529f26f72ce47440c2a64248063a6d5892b9fde
what Ġquantitative Ġanalysis Ġis Ġdone ?
Answer Ġwith Ġcontent Ġmissing : Ġ( E val uation Ġsection ) ĠGiven Ġthat Ġin ĠCL IR Ġthe Ġprimary Ġgoal Ġis Ġto Ġget Ġa Ġbetter Ġranked Ġlist Ġof Ġdocuments Ġagainst Ġa Ġtranslated Ġquery , Ġwe Ġonly Ġreport ĠMean ĠAverage ĠPrecision Ġ( MAP ).
<s> what Ġquantitative Ġanalysis Ġis Ġdone ? </s> Introduction </s> CL IR Ġsystems Ġretrieve Ġdocuments Ġwritten Ġin Ġa Ġlanguage Ġthat Ġis Ġdifferent Ġfrom Ġsearch Ġquery Ġlanguage ĠB IB REF 0 Ġ. ĠThe Ġprimary Ġobjective Ġof ĠCL IR Ġis Ġto Ġtranslate Ġor Ġproject Ġa Ġquery Ġinto Ġthe Ġlanguage Ġof Ġthe Ġdocument Ġrepository ĠB IB REF 1 Ġ, Ġwhich Ġwe Ġrefer Ġto Ġas ĠRet ri eval ĠCorpus Ġ( RC ). ĠTo Ġthis Ġend , Ġcommon ĠCL IR Ġapproaches Ġtranslate Ġsearch Ġqueries Ġusing Ġa ĠMachine ĠTranslation Ġ( MT ) Ġmodel Ġand Ġthen Ġuse Ġa Ġmon oling ual ĠIR Ġsystem Ġto Ġretrieve Ġfrom ĠRC . ĠIn Ġthis Ġprocess , Ġa Ġtranslation Ġmodel Ġis Ġtreated Ġas Ġa Ġblack Ġbox ĠB IB REF 2 Ġ, Ġand Ġit Ġis Ġusually Ġtrained Ġon Ġa Ġsentence Ġlevel Ġparallel Ġcorpus , Ġwhich Ġwe Ġrefer Ġto Ġas ĠTranslation ĠCorpus Ġ( TC ). </s> We Ġaddress Ġa Ġpit fall Ġof Ġusing Ġexisting ĠMT Ġmodels Ġfor Ġquery Ġtranslation ĠB IB REF 1 Ġ. ĠAn ĠMT Ġmodel Ġtrained Ġon ĠTC Ġdoes Ġnot Ġhave Ġany Ġknowledge Ġof ĠRC . ĠIn Ġan Ġextreme Ġsetting , Ġwhere Ġthere Ġare Ġno Ġcommon Ġterms Ġbetween Ġthe Ġtarget Ġside Ġof ĠTC Ġand ĠRC , Ġa Ġwell Ġtrained Ġand Ġtested Ġtranslation Ġmodel Ġwould Ġfail Ġbecause Ġof Ġvocabulary Ġmismatch Ġbetween Ġthe Ġtranslated Ġquery Ġand Ġdocuments Ġof ĠRC . ĠAssuming Ġa Ġrelaxed Ġscenario Ġwhere Ġsome Ġcommon ality Ġexists Ġbetween Ġtwo Ġcorpor a , Ġa Ġtranslation Ġmodel Ġmight Ġstill Ġperform Ġpoorly , Ġfavoring Ġterms Ġthat Ġare Ġmore Ġlikely Ġin ĠTC Ġbut Ġrare Ġin ĠRC . ĠOur Ġhypothesis Ġis Ġthat Ġa Ġsearch Ġquery Ġtranslation Ġmodel Ġwould Ġperform Ġbetter Ġif Ġa Ġtranslated Ġquery Ġterm Ġis Ġlikely Ġto Ġappear Ġin Ġthe Ġboth Ġretrieval Ġand Ġtranslation Ġcorpor a , Ġa Ġproperty Ġwe Ġcall Ġbalanced Ġtranslation . </s> To Ġachieve Ġbalanced Ġtranslations , Ġit Ġis Ġdesired Ġto Ġconstruct Ġan ĠMT Ġmodel Ġthat Ġis Ġaware Ġof ĠRC Ġvocabulary . ĠDifferent Ġtypes Ġof ĠMT Ġapproaches Ġhave Ġbeen Ġadopted Ġfor ĠCL IR Ġtask , Ġsuch Ġas Ġdictionary - based ĠMT , Ġrule - based ĠMT , Ġstatistical ĠMT Ġetc . ĠB IB REF 3 Ġ. ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġa Ġneural Ġsearch Ġquery Ġtranslation Ġapproach Ġhas Ġyet Ġto Ġbe Ġtaken Ġby Ġthe Ġcommunity . ĠN MT Ġmodels Ġwith Ġattention Ġbased Ġenc oder - dec oder Ġtechniques Ġhave Ġachieved Ġstate - of - the - art Ġperformance Ġfor Ġseveral Ġlanguage Ġpairs ĠB IB REF 4 Ġ. ĠWe Ġpropose Ġa Ġmulti - task Ġlearning ĠN MT Ġarchitecture Ġthat Ġtakes ĠRC Ġvocabulary Ġinto Ġaccount Ġby Ġlearning ĠRe lev ance - based ĠAux iliary ĠTask Ġ( R AT ). ĠR AT Ġis Ġinspired Ġfrom Ġtwo Ġword Ġembed ding Ġlearning Ġapproaches : ĠRe lev ance - based ĠWord ĠEmb edd ing Ġ( R WE ) ĠB IB REF 5 Ġand ĠContinuous ĠBag Ġof ĠWords Ġ( CB OW ) Ġembed ding ĠB IB REF 6 Ġ. ĠWe Ġshow Ġthat Ġlearning ĠN MT Ġwith ĠR AT Ġenables Ġit Ġto Ġgenerate Ġbalanced Ġtranslation . </s> N MT Ġmodels Ġlearn Ġto Ġencode Ġthe Ġmeaning Ġof Ġa Ġsource Ġsentence Ġand Ġdecode Ġthe Ġmeaning Ġto Ġgenerate Ġwords Ġin Ġa Ġtarget Ġlanguage ĠB IB REF 7 Ġ. ĠIn Ġthe Ġproposed Ġmulti - task Ġlearning Ġmodel , ĠR AT Ġshares Ġthe Ġdec oder Ġembed ding Ġand Ġfinal Ġrepresentation Ġlayer Ġwith ĠN MT . ĠOur Ġarchitecture Ġanswers Ġthe Ġfollowing Ġquestion : ĠIn Ġthe Ġdecoding Ġstage , Ġcan Ġwe Ġrestrict Ġan ĠN MT Ġmodel Ġso Ġthat Ġit Ġdoes Ġnot Ġonly Ġgenerate Ġterms Ġthat Ġare Ġhighly Ġlikely Ġin ĠTC ? . ĠWe Ġshow Ġthat Ġtraining Ġa Ġstrong Ġbaseline ĠN MT Ġwith ĠR AT Ġroughly Ġachieves Ġ16 % Ġimprovement Ġover Ġthe Ġbaseline . ĠUsing Ġa Ġqualitative Ġanalysis , Ġwe Ġfurther Ġshow Ġthat ĠR AT Ġworks Ġas Ġa Ġregular izer Ġand Ġprohibits ĠN MT Ġto Ġover fit Ġto ĠTC Ġvocabulary . </s> Bal anced ĠTranslation ĠApproach </s> We Ġtrain ĠN MT Ġwith ĠR AT Ġto Ġachieve Ġbetter Ġquery Ġtranslations . ĠWe Ġimprove Ġa Ġrecently Ġproposed ĠN MT Ġbaseline , ĠTrans former , Ġthat Ġachieves Ġstate - of - the - art Ġresults Ġfor Ġsentence Ġpairs Ġin Ġsome Ġlanguages ĠB IB REF 8 Ġ. ĠWe Ġdiscuss ĠTrans former , ĠR AT , Ġand Ġour Ġmulti - task Ġlearning Ġarchitecture Ġthat Ġachieves Ġbalanced Ġtranslation . </s> N MT Ġand ĠTrans former </s> In Ġprinciple , Ġwe Ġcould Ġadopt Ġany ĠN MT Ġand Ġcombine Ġit Ġwith ĠR AT . ĠAn ĠN MT Ġsystem Ġdirectly Ġmodels Ġthe Ġconditional Ġprobability ĠIN LINE FORM 0 Ġof Ġtranslating Ġa Ġsource Ġsentence , ĠIN LINE FORM 1 Ġ, Ġto Ġa Ġtarget Ġsentence ĠIN LINE FORM 2 Ġ. ĠA Ġbasic Ġform Ġof ĠN MT Ġcomprises Ġtwo Ġcomponents : Ġ( a ) Ġan Ġenc oder Ġthat Ġcomp utes Ġthe Ġrepresentations Ġor Ġmeaning Ġof ĠIN LINE FORM 3 Ġand Ġ( b ) Ġa Ġdec oder Ġthat Ġgenerates Ġone Ġtarget Ġword Ġat Ġa Ġtime . ĠState - of - the - art ĠN MT Ġmodels Ġhave Ġan Ġattention Ġcomponent Ġthat ĠâĢ ľ se ar ches Ġfor Ġa Ġset Ġof Ġpositions Ġin Ġa Ġsource Ġsentence Ġwhere Ġthe Ġmost Ġrelevant Ġinformation Ġis Ġconcentrated âĢ Ŀ ĠB IB REF 4 Ġ. </s> For Ġthis Ġstudy , Ġwe Ġuse Ġa Ġstate - of - the - art ĠN MT Ġmodel , ĠTrans former ĠB IB REF 8 Ġ, Ġthat Ġuses Ġpositional Ġencoding Ġand Ġself Ġattention Ġmechanism Ġto Ġachieve Ġthree Ġbenefits Ġover Ġthe Ġexisting Ġconv olution al Ġor Ġrecurrent Ġneural Ġnetwork Ġbased Ġmodels : Ġ( a ) Ġreduced Ġcomputational Ġcomplexity Ġof Ġeach Ġlayer , Ġ( b ) Ġparallel Ġcomputation , Ġand Ġ( c ) Ġpath Ġlength Ġbetween Ġlong - range Ġdependencies . </s> Re lev ance - based ĠAux iliary ĠTask Ġ( R AT ) </s> We Ġdefine ĠR AT Ġa Ġvariant Ġof Ġword Ġembed ding Ġtask ĠB IB REF 6 Ġ. ĠWord Ġembed ding Ġapproaches Ġlearn Ġhigh Ġdimensional Ġdense Ġrepresentations Ġfor Ġwords Ġand Ġtheir Ġobjective Ġfunctions Ġaim Ġto Ġcapture Ġcontextual Ġinformation Ġaround Ġa Ġword . ĠB IB REF 5 Ġproposed Ġa Ġmodel Ġthat Ġlearns Ġword Ġvectors Ġby Ġpredicting Ġwords Ġin Ġrelevant Ġdocuments Ġretrieved Ġagainst Ġa Ġsearch Ġquery . ĠWe Ġfollow Ġthe Ġsame Ġidea Ġbut Ġuse Ġa Ġsimpler Ġlearning Ġapproach Ġthat Ġis Ġsuitable Ġfor Ġour Ġtask . ĠThey Ġtried Ġto Ġpredict Ġwords Ġfrom Ġthe Ġrelevance Ġmodel ĠB IB REF 9 Ġcomputed Ġfrom Ġa Ġquery , Ġwhich Ġdoes Ġnot Ġwork Ġfor Ġour Ġtask Ġbecause Ġthe Ġconnection Ġbetween Ġa Ġquery Ġand Ġranked Ġsentences Ġfalls Ġrapidly Ġafter Ġthe Ġtop Ġone Ġ( see Ġbelow ). </s> We Ġconsider Ġtwo Ġdata Ġsources Ġfor Ġlearning ĠN MT Ġand ĠR AT Ġjointly . ĠThe Ġfirst Ġone Ġis Ġa Ġsentence - level Ġparallel Ġcorpus , Ġwhich Ġwe Ġrefer Ġto Ġas Ġtranslation Ġcorpus , ĠIN LINE FORM 0 Ġ. ĠThe Ġsecond Ġone Ġis Ġthe Ġretrieval Ġcorpus , Ġwhich Ġis Ġa Ġcollection Ġof ĠIN LINE FORM 1 Ġdocuments ĠIN LINE FORM 2 Ġin Ġthe Ġsame Ġlanguage Ġas ĠIN LINE FORM 3 Ġ. ĠOur Ġword - embed ding Ġapproach Ġtakes Ġeach ĠIN LINE FORM 4 Ġ, Ġuses Ġit Ġas Ġa Ġquery Ġto Ġretrieve Ġthe Ġtop Ġdocument ĠIN LINE FORM 5 Ġ. ĠAfter Ġthat Ġwe Ġobtain ĠIN LINE FORM 6 Ġby Ġconc aten ating ĠIN LINE FORM 7 Ġwith ĠIN LINE FORM 8 Ġand Ġrandomly Ġshuff ling Ġthe Ġwords Ġin Ġthe Ġcombined Ġsequence . ĠWe Ġthen Ġaugment ĠIN LINE FORM 9 Ġusing ĠIN LINE FORM 10 Ġand Ġobtain Ġa Ġdataset , ĠIN LINE FORM 11 Ġ. ĠWe Ġuse ĠIN LINE FORM 12 Ġto Ġlearn Ġa Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġembed ding Ġas Ġproposed Ġby ĠB IB REF 6 Ġ. ĠThis Ġlearning Ġcomponent Ġshares Ġtwo Ġlayers Ġwith Ġthe ĠN MT Ġmodel . ĠThe Ġgoal Ġis Ġto Ġexpose Ġthe Ġretrieval Ġcorpus ' Ġvocabulary Ġto Ġthe ĠN MT Ġmodel . ĠWe Ġdiscuss Ġlayer Ġsharing Ġin Ġthe Ġnext Ġsection . </s> We Ġselect Ġthe Ġsingle Ġtop Ġdocument Ġretrieved Ġagainst Ġa Ġsentence ĠIN LINE FORM 0 Ġbecause Ġa Ġsentence Ġis Ġa Ġweak Ġrepresentation Ġof Ġinformation Ġneed . ĠAs Ġa Ġresult , Ġdocuments Ġat Ġlower Ġranks Ġshow Ġheavy Ġshift Ġfrom Ġthe Ġcontext Ġof Ġthe Ġsentence Ġquery . ĠWe Ġverified Ġthis Ġby Ġobserving Ġthat Ġa Ġrelevance Ġmodel Ġconstructed Ġfrom Ġthe Ġtop ĠIN LINE FORM 1 Ġdocuments Ġdoes Ġnot Ġperform Ġwell Ġin Ġthis Ġsetting . ĠWe Ġthus Ġdev iate Ġfrom Ġthe Ġrelevance Ġmodel Ġbased Ġapproach Ġtaken Ġby ĠB IB REF 5 Ġand Ġlearn Ġover Ġthe Ġrandom Ġshuff ling Ġof ĠIN LINE FORM 2 Ġand Ġa Ġsingle Ġdocument . ĠRandom Ġshuff ling Ġhas Ġshown Ġreasonable Ġeffectiveness Ġfor Ġword Ġembed ding Ġconstruction Ġfor Ġcomparable Ġcorpus ĠB IB REF 10 Ġ. </s> Multi - task ĠN MT ĠArchitecture </s> Our Ġbalanced Ġtranslation Ġarchitecture Ġis Ġpresented Ġin ĠFigure ĠFIG REF 3 Ġ. ĠThis Ġarchitecture Ġis ĠN MT - model Ġag nostic Ġas Ġwe Ġonly Ġpropose Ġto Ġshare Ġtwo Ġlayers Ġcommon Ġto Ġmost ĠN MT s : Ġthe Ġtrain able Ġtarget Ġembed ding Ġlayer Ġand Ġthe Ġtransformation Ġfunction ĠB IB REF 7 Ġthat Ġoutputs Ġa Ġprobability Ġdistribution Ġover Ġthe Ġunion Ġof Ġthe Ġvocabulary Ġof ĠTC Ġand ĠRC . ĠHence , Ġthe Ġsize Ġof Ġthe Ġvocabulary , ĠIN LINE FORM 0 Ġ, Ġis Ġmuch Ġlarger Ġcompared Ġto ĠTC Ġand Ġit Ġenables Ġthe Ġmodel Ġto Ġaccess ĠRC . ĠIn Ġorder Ġto Ġshow Ġtask Ġsharing Ġclearly Ġwe Ġplaced Ġtwo Ġshared Ġlayers Ġbetween ĠN MT Ġand ĠR AT Ġin ĠFigure ĠFIG REF 3 Ġ. ĠWe Ġalso Ġshow Ġthe Ġtwo Ġdifferent Ġpaths Ġtaken Ġby Ġtwo Ġdifferent Ġtasks Ġat Ġtraining Ġtime : Ġthe ĠN MT Ġpath Ġin Ġshown Ġwith Ġred Ġarrows Ġwhile Ġthe ĠR AT Ġpath Ġis Ġshown Ġin Ġgreen Ġarrows . </s> On ĠN MT Ġpath Ġtraining Ġloss Ġis Ġcomputed Ġas Ġthe Ġsum Ġof Ġterm - wise Ġsoft max Ġwith Ġcross - ent ropy Ġloss Ġof Ġthe Ġpredicted Ġtranslation Ġand Ġthe Ġhuman Ġtranslation Ġand Ġit Ġsummed Ġover Ġa Ġbatch Ġof Ġsentence Ġpairs , ĠIN LINE FORM 0 Ġ. ĠWe Ġalso Ġuse Ġa Ġsimilar Ġloss Ġfunction Ġto Ġtrain Ġword Ġembed ding Ġover Ġa Ġset Ġof Ġcontext Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġpivot Ġ( ĠIN LINE FORM 2 Ġ) Ġpairs Ġformed Ġusing ĠIN LINE FORM 3 Ġas Ġquery Ġto Ġretrieve ĠIN LINE FORM 4 Ġusing ĠQuery ĠLike lihood Ġ( QL ) Ġrank er , ĠIN LINE FORM 5 Ġ. ĠThis Ġobjective Ġis Ġsimilar Ġto ĠCB OW Ġword Ġembed ding Ġas Ġcontext Ġis Ġused Ġto Ġpredict Ġpivot Ġword ĠHere , Ġwe Ġuse Ġa Ġscaling Ġfactor ĠIN LINE FORM 6 Ġ, Ġto Ġhave Ġa Ġbalance Ġbetween Ġthe Ġgrad ients Ġfrom Ġthe ĠN MT Ġloss Ġand ĠR AT Ġloss . ĠFor ĠR AT , Ġthe Ġcontext Ġis Ġdrawn Ġfrom Ġa Ġcontext Ġwindow Ġfollowing ĠB IB REF 6 Ġ. </s> In Ġthe Ġfigure , ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrepresents Ġthe Ġtop Ġdocument Ġretrieved Ġagainst ĠIN LINE FORM 2 Ġ. ĠThe Ġshuff ler Ġcomponent Ġshuff les ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġand Ġcreates Ġ( context , Ġpivot ) Ġpairs . ĠAfter Ġthat Ġthose Ġdata Ġpoints Ġare Ġpassed Ġthrough Ġa Ġfully Ġconnected Ġlinear Ġprojection Ġlayer Ġand Ġeventually Ġto Ġthe Ġtransformation Ġfunction . ĠInt uitive ly , Ġthe Ġword Ġembed ding Ġtask Ġis Ġsimilar Ġto ĠN MT Ġas Ġit Ġtries Ġto Ġassign Ġa Ġlarge Ġprobability Ġmass Ġto Ġa Ġtarget Ġword Ġgiven Ġa Ġcontext . ĠHowever , Ġit Ġenables Ġthe Ġtransformation Ġfunction Ġand Ġdecoding Ġlayer Ġto Ġassign Ġprobability Ġmass Ġnot Ġonly Ġto Ġterms Ġfrom ĠTC , Ġbut Ġalso Ġto Ġterms Ġfrom ĠRC . ĠThis Ġimplicitly Ġprohibits ĠN MT Ġto Ġover fit Ġand Ġprovides Ġa Ġregular ization Ġeffect . ĠA Ġsimilar Ġtechnique Ġwas Ġproposed Ġby ĠB IB REF 11 Ġto Ġhandle Ġout - of - voc abulary Ġor Ġless Ġfrequent Ġwords Ġfor ĠN MT . ĠFor Ġthese Ġterms Ġthey Ġenabled Ġthe Ġtransformation Ġ( also Ġcalled Ġthe Ġsoft max Ġcross - ent ropy Ġlayer ) Ġto Ġfairly Ġdistribute Ġprobability Ġmass Ġamong Ġsimilar Ġwords . ĠIn Ġcontrast , Ġwe Ġfocus Ġon Ġrelevant Ġterms Ġrather Ġthan Ġsimilar Ġterms . </s> Results Ġand ĠAnalysis </s> Table ĠT AB REF 14 Ġshows Ġthe Ġeffectiveness Ġof Ġour Ġmodel Ġ( multi - task Ġtransformer ) Ġover Ġthe Ġbaseline Ġtransformer ĠB IB REF 8 Ġ. ĠOur Ġmodel Ġachieves Ġsignificant Ġperformance Ġgains Ġin Ġthe Ġtest Ġsets Ġover Ġthe Ġbaseline Ġfor Ġboth ĠItalian Ġand ĠFinnish Ġquery Ġtranslation . ĠThe Ġoverall Ġlow ĠMAP Ġfor ĠN MT Ġcan Ġpossibly Ġbe Ġimproved Ġwith Ġlarger ĠTC . ĠMoreover , Ġour Ġmodel Ġvalidation Ġapproach Ġrequires Ġaccess Ġto ĠRC Ġindex , Ġand Ġit Ġslows Ġdown Ġoverall Ġtraining Ġprocess . ĠHence , Ġwe Ġcould Ġnot Ġtrain Ġour Ġmodel Ġfor Ġa Ġlarge Ġnumber Ġof Ġepoch s Ġ- Ġit Ġmay Ġbe Ġanother Ġcause Ġof Ġthe Ġlow Ġperformance . </s> We Ġwant Ġto Ġshow Ġthat Ġtranslation Ġterms Ġgenerated Ġby Ġour Ġmulti - task Ġtransformer Ġare Ġroughly Ġequally Ġlikely Ġto Ġbe Ġseen Ġin Ġthe ĠEuro par l Ġcorpus Ġ( TC ) Ġor Ġthe ĠCLE F Ġcorpus Ġ( RC ). ĠGiven Ġa Ġtranslation Ġterm ĠIN LINE FORM 0 Ġ, Ġwe Ġcompute Ġthe Ġratio Ġof Ġthe Ġprobability Ġof Ġseeing ĠIN LINE FORM 1 Ġin ĠTC Ġand ĠRC , ĠIN LINE FORM 2 Ġ. ĠHere , ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġis Ġcalculated Ġsimilarly . ĠGiven Ġa Ġquery ĠIN LINE FORM 5 Ġand Ġits Ġtranslation ĠIN LINE FORM 6 Ġprovided Ġby Ġmodel ĠIN LINE FORM 7 Ġ, Ġwe Ġcalculate Ġthe Ġbalance Ġof ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ. ĠIf ĠIN LINE FORM 10 Ġis Ġclose Ġto Ġ1 , Ġthe Ġtranslation Ġterms Ġare Ġas Ġlikely Ġin ĠTC Ġas Ġin ĠRC . ĠFigure ĠFIG REF 15 Ġshows Ġthe Ġbalance Ġvalues Ġfor Ġtransformer Ġand Ġour Ġmodel Ġfor Ġa Ġrandom Ġsample Ġof Ġ20 Ġqueries Ġfrom Ġthe Ġvalidation Ġset Ġof ĠItalian Ġqueries , Ġrespectively . ĠFigure ĠFIG REF 16 Ġshows Ġthe Ġbalance Ġvalues Ġfor Ġtransformer Ġand Ġour Ġmodel Ġfor Ġa Ġrandom Ġsample Ġof Ġ20 Ġqueries Ġfrom Ġthe Ġtest Ġset Ġof ĠItalian Ġqueries , Ġrespectively . ĠIt Ġis Ġevident Ġthat Ġour Ġmodel Ġachieves Ġbetter Ġbalance Ġcompared Ġto Ġbaseline Ġtransformer , Ġexcept Ġfor Ġa Ġvery Ġfew Ġcases . </s> Given Ġa Ġquery ĠIN LINE FORM 0 Ġ, Ġconsider ĠIN LINE FORM 1 Ġas Ġthe Ġset Ġof Ġterms Ġfrom Ġhuman Ġtranslation Ġof ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġas Ġthe Ġset Ġof Ġtranslation Ġterms Ġgenerated Ġby Ġmodel ĠIN LINE FORM 4 Ġ. ĠWe Ġdefine ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġas Ġprecision Ġand Ġrecall Ġof ĠIN LINE FORM 7 Ġfor Ġmodel ĠIN LINE FORM 8 Ġ. ĠIn ĠTable ĠT AB REF 19 Ġ, Ġwe Ġreport Ġaverage Ġprecision Ġand Ġrecall Ġfor Ġboth Ġtransformer Ġand Ġour Ġmodel Ġacross Ġour Ġtrain Ġand Ġvalidation Ġquery Ġset Ġover Ġtwo Ġlanguage Ġpairs . ĠOur Ġmodel Ġgenerates Ġprecise Ġtranslation , Ġi . e . Ġit Ġavoids Ġterms Ġthat Ġmight Ġbe Ġuseless Ġor Ġeven Ġharmful Ġfor Ġretrieval . ĠGenerally , Ġfrom Ġour Ġobservation , Ġavoided Ġterms Ġare Ġhighly Ġlikely Ġterms Ġfrom ĠTC Ġand Ġthey Ġare Ġgenerated Ġbecause Ġof Ġtranslation Ġmodel Ġover fitting . ĠOur Ġmodel Ġachieves Ġa Ġregular ization Ġeffect Ġthrough Ġan Ġauxiliary Ġtask . ĠThis Ġconfirms Ġresults Ġfrom Ġexisting Ġmulti - t asking Ġliterature ĠB IB REF 15 Ġ. </s> To Ġexplore Ġtranslation Ġquality , Ġconsider Ġpair Ġof Ġsample Ġtranslations Ġprovided Ġby Ġtwo Ġmodels . ĠFor Ġexample , Ġagainst Ġan ĠItalian Ġquery , Ġmed ag lia Ġo ro Ġsuper Ġv inse Ġmed ag lia Ġo ro Ġsuper Ġo lim pi adi Ġin vern ali Ġl ille hammer , Ġtranslated Ġterm Ġset Ġfrom Ġour Ġmodel Ġis Ġ{ gold , Ġcoin , Ġsuper , Ġfree , Ġharmon ising , Ġwon , Ġwinter , Ġo lymp ics }, Ġwhile Ġtransformer Ġoutput Ġis Ġ{ oly mp ic , Ġgold , Ġone , Ġcoin , Ġsuper , Ġyears , Ġwon , Ġparliament , Ġalso , Ġtwo , Ġwinter }. ĠTerm Ġset Ġfrom Ġhuman Ġtranslation Ġis : Ġ{ super , Ġgold , Ġmedal , Ġwon , Ġl ille hammer , Ġo ly pm ic , Ġwinter , Ġgames }. ĠTrans former Ġcomes Ġup Ġwith Ġterms Ġlike Ġparliament , Ġalso , Ġtwo Ġand Ġyears Ġthat Ġnever Ġappears Ġin Ġhuman Ġtranslation . ĠWe Ġfound Ġthat Ġthese Ġterms Ġare Ġvery Ġlikely Ġin ĠEuro par l Ġand Ġrare Ġin ĠCLE F . ĠOur Ġmodel Ġalso Ġgenerates Ġterms Ġsuch Ġas Ġharmon ising , Ġfree , Ġo lymp ics Ġthat Ġnot Ġgenerated Ġby Ġtransformer . ĠHowever , Ġwe Ġfound Ġthat Ġthese Ġterms Ġare Ġequally Ġlikely Ġin ĠEuro par l Ġand ĠCLE F . </s> Conclusion </s> We Ġpresent Ġa Ġmulti - task Ġlearning Ġarchitecture Ġto Ġlearn ĠN MT Ġfor Ġsearch Ġquery Ġtranslation . ĠAs Ġthe Ġmotivating Ġtask Ġis ĠCL IR , Ġwe Ġevaluated Ġthe Ġranking Ġeffectiveness Ġof Ġour Ġproposed Ġarchitecture . ĠWe Ġused Ġsentences Ġfrom Ġthe Ġtarget Ġside Ġof Ġthe Ġparallel Ġcorpus Ġas Ġqueries Ġto Ġretrieve Ġrelevant Ġdocument Ġand Ġuse Ġterms Ġfrom Ġthose Ġdocuments Ġto Ġtrain Ġa Ġword Ġembed ding Ġmodel Ġalong Ġwith ĠN MT . ĠOne Ġbig Ġchallenge Ġin Ġthis Ġlandscape Ġis Ġto Ġsample Ġmeaningful Ġqueries Ġfrom Ġsentences Ġas Ġsentences Ġdo Ġnot Ġdirectly Ġconvey Ġinformation Ġneed . ĠIn Ġthe Ġfuture , Ġwe Ġhope Ġto Ġlearn Ġmodels Ġthat Ġare Ġable Ġto Ġsample Ġsearch Ġqueries Ġor Ġinformation Ġneeds Ġfrom Ġsentences Ġand Ġuse Ġthe Ġoutput Ġof Ġthat Ġmodel Ġto Ġget Ġrelevant Ġdocuments . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġin Ġpart Ġby Ġthe ĠCenter Ġfor ĠIntelligent ĠInformation ĠRet ri eval Ġand Ġin Ġpart Ġby Ġthe ĠAir ĠForce ĠResearch ĠLaboratory Ġ( AF RL ) Ġand ĠI AR PA Ġunder Ġcontract Ġ# FA 86 50 - 17 - C - 9 118 Ġunder Ġsubcontract Ġ# 147 75 Ġfrom ĠRay the on ĠBB N ĠTechnologies ĠCorporation . ĠAny Ġopinions , Ġfindings Ġand Ġconclusions Ġor Ġrecommendations Ġexpressed Ġin Ġthis Ġmaterial Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġdo Ġnot Ġnecessarily Ġreflect Ġthose Ġof Ġthe Ġsponsor . </s> L oss ĠFunction Ġand ĠVal idation ĠPerformance ĠAnalysis </s> We Ġshow Ġthe Ġloss Ġfunction Ġanalysis Ġof Ġtransformer Ġand Ġour Ġmodel . ĠFigure ĠFIG REF 23 Ġshows Ġthe Ġvalidation Ġperformance Ġof Ġtransformer Ġagainst Ġglobal Ġtraining Ġsteps . ĠFigure ĠFIG REF 21 Ġshow Ġthe Ġvalidation Ġperformance Ġof Ġour Ġmodel Ġfor Ġthe Ġsame Ġnumber Ġof Ġglobal Ġsteps . ĠFigure ĠFIG REF 22 Ġshows Ġthat ĠN MT Ġloss Ġis Ġgoing Ġdown Ġwith Ġthe Ġnumber Ġof Ġsteps , Ġwhile ĠFigure ĠFIG REF 20 Ġshows Ġthe Ġdegradation Ġof Ġthe Ġloss Ġof Ġour Ġproposed ĠR AT Ġtask .