For certain POS tags, e.g. VERB, PRON.
ABSTRACTIVE
most word alignments only involve one or a few words, attention can be distributed more freely
EXTRACTIVE
To better understand how attention accuracy affects translation quality, we analyse the relationship between attention loss and word prediction loss for individual part-of-speech classes. Figure FIGREF22 shows how attention loss differs when generating different POS tags. One can see that attention loss varies substantially across different POS tags. In particular, we focus on the cases of NOUN and VERB which are the most frequent POS tags in the dataset. As shown, the attention of NOUN is the closest to alignments on average. But the average attention loss for VERB is almost two times larger than the loss for NOUN. FLOAT SELECTED: Figure 6: Correlation of attention entropy and word prediction loss for the input-feeding system.
As another informative variable in our analysis, we look into the attention concentration. While most word alignments only involve one or a few words, attention can be distributed more freely. We measure the concentration of attention by computing the entropy of the attention distribution: DISPLAYFORM0
1710.03348
Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġgained Ġa Ġlot Ġof Ġattention Ġrecently Ġdue Ġto Ġits Ġsubstantial Ġimprovements Ġin Ġmachine Ġtranslation Ġquality Ġachieving Ġstate - of - the - art Ġperformance Ġfor Ġseveral Ġlanguages ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠThe Ġcore Ġarchitecture Ġof Ġneural Ġmachine Ġtranslation Ġmodels Ġis Ġbased Ġon Ġthe Ġgeneral Ġenc oder - dec oder Ġapproach ĠB IB REF 3 Ġ. ĠNeural Ġmachine Ġtranslation Ġis Ġan Ġend - to - end Ġapproach Ġthat Ġlearns Ġto Ġencode Ġsource Ġsentences Ġinto Ġdistributed Ġrepresentations Ġand Ġdecode Ġthese Ġrepresentations Ġinto Ġsentences Ġin Ġthe Ġtarget Ġlanguage . ĠAmong Ġthe Ġdifferent Ġneural ĠMT Ġmodels , Ġattention al ĠN MT ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġhas Ġbecome Ġpopular Ġdue Ġto Ġits Ġcapability Ġto Ġuse Ġthe Ġmost Ġrelevant Ġparts Ġof Ġthe Ġsource Ġsentence Ġat Ġeach Ġtranslation Ġstep . ĠThis Ġcapability Ġalso Ġmakes Ġthe Ġattention al Ġmodel Ġsuperior Ġin Ġtranslating Ġlonger Ġsentences ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> Figure ĠFIG REF 1 Ġshows Ġan Ġexample Ġof Ġhow Ġattention Ġuses Ġthe Ġmost Ġrelevant Ġsource Ġwords Ġto Ġgenerate Ġa Ġtarget Ġword Ġat Ġeach Ġstep Ġof Ġthe Ġtranslation . ĠIn Ġthis Ġpaper Ġwe Ġfocus Ġon Ġstudying Ġthe Ġrelevance Ġof Ġthe Ġattended Ġparts , Ġespecially Ġcases Ġwhere Ġattention Ġis Ġ` s me ared Ġout ' Ġover Ġmultiple Ġsource Ġwords Ġwhere Ġtheir Ġrelevance Ġis Ġnot Ġentirely Ġobvious , Ġsee , Ġe . g ., ĠâĢ ľ would " Ġand ĠâĢ ľ like " Ġin ĠFigure ĠFIG REF 1 Ġ. ĠHere , Ġwe Ġask Ġwhether Ġthese Ġare Ġdue Ġto Ġerrors Ġof Ġthe Ġattention Ġmechanism Ġor Ġare Ġa Ġdesired Ġbehavior Ġof Ġthe Ġmodel . </s> Since Ġthe Ġintroduction Ġof Ġattention Ġmodels Ġin Ġneural Ġmachine Ġtranslation ĠB IB REF 4 Ġvarious Ġmodifications Ġhave Ġbeen Ġproposed ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġthere Ġis Ġno Ġstudy Ġthat Ġprovides Ġan Ġanalysis Ġof Ġwhat Ġkind Ġof Ġphenomena Ġis Ġbeing Ġcaptured Ġby Ġattention . ĠThere Ġare Ġsome Ġworks Ġthat Ġhave Ġlooked Ġto Ġattention Ġas Ġbeing Ġsimilar Ġto Ġtraditional Ġword Ġalignment ĠB IB REF 8 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 9 Ġ. ĠSome Ġof Ġthese Ġapproaches Ġalso Ġexperimented Ġwith Ġtraining Ġthe Ġattention Ġmodel Ġusing Ġtraditional Ġalign ments ĠB IB REF 8 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 9 Ġ. Ġli u - E t Al : 2016 : COL ING Ġhave Ġshown Ġthat Ġattention Ġcould Ġbe Ġseen Ġas Ġa Ġre ordering Ġmodel Ġas Ġwell Ġas Ġan Ġalignment Ġmodel . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġinvestigating Ġthe Ġdifferences Ġbetween Ġattention Ġand Ġalignment Ġand Ġwhat Ġis Ġbeing Ġcaptured Ġby Ġthe Ġattention Ġmechanism Ġin Ġgeneral . ĠThe Ġquestions Ġthat Ġwe Ġare Ġaiming Ġto Ġanswer Ġinclude : ĠIs Ġthe Ġattention Ġmodel Ġonly Ġcapable Ġof Ġmodelling Ġalignment ? ĠAnd Ġhow Ġsimilar Ġis Ġattention Ġto Ġalignment Ġin Ġdifferent Ġsynt actic Ġphenomena ? </s> Our Ġanalysis Ġshows Ġthat Ġattention Ġmodels Ġtraditional Ġalignment Ġin Ġsome Ġcases Ġmore Ġclosely Ġwhile Ġit Ġcaptures Ġinformation Ġbeyond Ġalignment Ġin Ġothers . ĠFor Ġinstance , Ġattention Ġagrees Ġwith Ġtraditional Ġalign ments Ġto Ġa Ġhigh Ġdegree Ġin Ġthe Ġcase Ġof Ġnoun s . ĠHowever , Ġit Ġcaptures Ġother Ġinformation Ġrather Ġthan Ġonly Ġthe Ġtransl ational Ġequivalent Ġin Ġthe Ġcase Ġof Ġverbs . </s> This Ġpaper Ġmakes Ġthe Ġfollowing Ġcontributions : Ġ1 ) ĠWe Ġprovide Ġa Ġdetailed Ġcomparison Ġof Ġattention Ġin ĠN MT Ġand Ġword Ġalignment . Ġ2 ) ĠWe Ġshow Ġthat Ġwhile Ġdifferent Ġattention Ġmechanisms Ġcan Ġlead Ġto Ġdifferent Ġdegrees Ġof Ġcompliance Ġwith Ġrespect Ġto Ġword Ġalign ments , Ġglobal Ġcompliance Ġis Ġnot Ġalways Ġhelpful Ġfor Ġword Ġprediction . Ġ3 ) ĠWe Ġshow Ġthat Ġattention Ġfollows Ġdifferent Ġpatterns Ġdepending Ġon Ġthe Ġtype Ġof Ġthe Ġword Ġbeing Ġgenerated . Ġ4 ) ĠWe Ġdemonstrate Ġthat Ġattention Ġdoes Ġnot Ġalways Ġcomply Ġwith Ġalignment . ĠWe Ġprovide Ġevidence Ġshowing Ġthat Ġthe Ġdifference Ġbetween Ġattention Ġand Ġalignment Ġis Ġdue Ġto Ġattention Ġmodel Ġcapability Ġto Ġattend Ġthe Ġcontext Ġwords Ġinfluencing Ġthe Ġcurrent Ġword Ġtranslation . </s> Related ĠWork </s> li u - E t Al : 2016 : COL ING Ġinvestigate Ġhow Ġtraining Ġthe Ġattention Ġmodel Ġin Ġa Ġsupervised Ġmanner Ġcan Ġbenefit Ġmachine Ġtranslation Ġquality . ĠTo Ġthis Ġend Ġthey Ġuse Ġtraditional Ġalign ments Ġobtained Ġby Ġrunning Ġautomatic Ġalignment Ġtools Ġ( G IZ A ++ ĠB IB REF 10 Ġand Ġfast _ align ĠB IB REF 11 Ġ) Ġon Ġthe Ġtraining Ġdata Ġand Ġfeed Ġit Ġas Ġground Ġtruth Ġto Ġthe Ġattention Ġnetwork . ĠThey Ġreport Ġsome Ġimprovements Ġin Ġtranslation Ġquality Ġarguing Ġthat Ġthe Ġattention Ġmodel Ġhas Ġlearned Ġto Ġbetter Ġalign Ġsource Ġand Ġtarget Ġwords . ĠThe Ġapproach Ġof Ġtraining Ġattention Ġusing Ġtraditional Ġalign ments Ġhas Ġalso Ġbeen Ġproposed Ġby Ġothers ĠB IB REF 9 Ġ, ĠB IB REF 8 Ġ. Ġc hen 2016 guided Ġshow Ġthat Ġguided Ġattention Ġwith Ġtraditional Ġalignment Ġhelps Ġin Ġthe Ġdomain Ġof Ġe - commerce Ġdata Ġwhich Ġincludes Ġlots Ġof Ġout Ġof Ġvocabulary Ġ( OO V ) Ġproduct Ġnames Ġand Ġplace holders , Ġbut Ġnot Ġmuch Ġin Ġthe Ġother Ġdomains . Ġal kh oul i - E t Al : 2016 : W MT Ġhave Ġseparated Ġthe Ġalignment Ġmodel Ġand Ġtranslation Ġmodel , Ġreasoning Ġthat Ġthis Ġavoids Ġpropagation Ġof Ġerrors Ġfrom Ġone Ġmodel Ġto Ġthe Ġother Ġas Ġwell Ġas Ġproviding Ġmore Ġflexibility Ġin Ġthe Ġmodel Ġtypes Ġand Ġtraining Ġof Ġthe Ġmodels . ĠThey Ġuse Ġa Ġfeed - forward Ġneural Ġnetwork Ġas Ġtheir Ġalignment Ġmodel Ġthat Ġlearns Ġto Ġmodel Ġjumps Ġin Ġthe Ġsource Ġside Ġusing ĠH MM / IB M Ġalign ments Ġobtained Ġby Ġusing ĠG IZ A ++ . </s> shi - pad hi - k night : 2016 : EM N LP 2016 Ġshow Ġthat Ġvarious Ġkinds Ġof Ġsynt actic Ġinformation Ġare Ġbeing Ġlearned Ġand Ġencoded Ġin Ġthe Ġoutput Ġhidden Ġstates Ġof Ġthe Ġenc oder . ĠThe Ġneural Ġsystem Ġfor Ġtheir Ġexperimental Ġanalysis Ġis Ġnot Ġan Ġattention al Ġmodel Ġand Ġthey Ġargue Ġthat Ġattention Ġdoes Ġnot Ġhave Ġany Ġimpact Ġfor Ġlearning Ġsynt actic Ġinformation . ĠHowever , Ġperforming Ġthe Ġsame Ġanalysis Ġfor Ġmorph ological Ġinformation , Ġbel ink ov 2017 ne ural Ġshow Ġthat Ġattention Ġhas Ġalso Ġsome Ġeffect Ġon Ġthe Ġinformation Ġthat Ġthe Ġenc oder Ġof Ġneural Ġmachine Ġtranslation Ġsystem Ġenc odes Ġin Ġits Ġoutput Ġhidden Ġstates . ĠAs Ġpart Ġof Ġtheir Ġanalysis Ġthey Ġshow Ġthat Ġa Ġneural Ġmachine Ġtranslation Ġsystem Ġthat Ġhas Ġan Ġattention Ġmodel Ġcan Ġlearn Ġthe ĠPOS Ġtags Ġof Ġthe Ġsource Ġside Ġmore Ġefficiently Ġthan Ġa Ġsystem Ġwithout Ġattention . </s> Recently , Ġk oe hn 2017 six Ġcarried Ġout Ġa Ġbrief Ġanalysis Ġof Ġhow Ġmuch Ġattention Ġand Ġalignment Ġmatch Ġin Ġdifferent Ġlanguages Ġby Ġmeasuring Ġthe Ġprobability Ġmass Ġthat Ġattention Ġgives Ġto Ġalign ments Ġobtained Ġfrom Ġan Ġautomatic Ġalignment Ġtool . ĠThey Ġalso Ġreport Ġdifferences Ġbased Ġon Ġthe Ġmost Ġattended Ġwords . </s> The Ġmixed Ġresults Ġreported Ġby Ġc hen 2016 guided , Ġal kh oul i - E t Al : 2016 : W MT , Ġli u - E t Al : 2016 : COL ING Ġon Ġoptimizing Ġattention Ġwith Ġrespect Ġto Ġalign ments Ġmotiv ates Ġa Ġmore Ġthorough Ġanalysis Ġof Ġattention Ġmodels Ġin ĠN MT . </s> Att ention ĠModels </s> This Ġsection Ġprovides Ġa Ġshort Ġbackground Ġon Ġattention Ġand Ġdiscusses Ġtwo Ġmost Ġpopular Ġattention Ġmodels Ġwhich Ġare Ġalso Ġused Ġin Ġthis Ġpaper . ĠThe Ġfirst Ġmodel Ġis Ġa Ġnon - rec urrent Ġattention Ġmodel Ġwhich Ġis Ġequivalent Ġto Ġthe ĠâĢ ľ global Ġattention " Ġmethod Ġproposed Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠThe Ġsecond Ġattention Ġmodel Ġthat Ġwe Ġuse Ġin Ġour Ġinvestigation Ġis Ġan Ġinput - feeding Ġmodel Ġsimilar Ġto Ġthe Ġattention Ġmodel Ġfirst Ġproposed Ġby Ġb ah dan au - E t Al : 2015 : IC LR Ġand Ġturned Ġto Ġa Ġmore Ġgeneral Ġone Ġand Ġcalled Ġinput - feeding Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠBelow Ġwe Ġdescribe Ġthe Ġdetails Ġof Ġboth Ġmodels . </s> Both Ġnon - rec urrent Ġand Ġinput - feeding Ġmodels Ġcompute Ġa Ġcontext Ġvector ĠIN LINE FORM 0 Ġat Ġeach Ġtime Ġstep . ĠSub sequently , Ġthey Ġconc aten ate Ġthe Ġcontext Ġvector Ġto Ġthe Ġhidden Ġstate Ġof Ġdec oder Ġand Ġpass Ġit Ġthrough Ġa Ġnon - linear ity Ġbefore Ġit Ġis Ġfed Ġinto Ġthe Ġsoft max Ġoutput Ġlayer Ġof Ġthe Ġtranslation Ġnetwork . ĠDIS PLAY FORM 0 </s> The Ġdifference Ġof Ġthe Ġtwo Ġmodels Ġlays Ġin Ġthe Ġway Ġthey Ġcompute Ġthe Ġcontext Ġvector . ĠIn Ġthe Ġnon - rec urrent Ġmodel , Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġis Ġcompared Ġto Ġeach Ġhidden Ġstate Ġof Ġthe Ġenc oder . ĠOften , Ġthis Ġcomparison Ġis Ġrealized Ġas Ġthe Ġdot Ġproduct Ġof Ġvectors . ĠThen Ġthe Ġcomparison Ġresult Ġis Ġfed Ġto Ġa Ġsoft max Ġlayer Ġto Ġcompute Ġthe Ġattention Ġweight . ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġis ĠIN LINE FORM 3 Ġth Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġlength Ġof Ġthe Ġsource Ġsentence . ĠThen Ġthe Ġcomputed Ġalignment Ġweights Ġare Ġused Ġto Ġcompute Ġa Ġweighted Ġsum Ġover Ġthe Ġenc oder Ġhidden Ġstates Ġwhich Ġresults Ġin Ġthe Ġcontext Ġvector Ġmentioned Ġabove : ĠDIS PLAY FORM 0 </s> The Ġinput - feeding Ġmodel Ġchanges Ġthe Ġcontext Ġvector Ġcomputation Ġin Ġa Ġway Ġthat Ġat Ġeach Ġstep ĠIN LINE FORM 0 Ġthe Ġcontext Ġvector Ġis Ġaware Ġof Ġthe Ġpreviously Ġcomputed Ġcontext ĠIN LINE FORM 1 Ġ. ĠTo Ġthis Ġend , Ġthe Ġinput - feeding Ġmodel Ġfeeds Ġback Ġits Ġown ĠIN LINE FORM 2 Ġto Ġthe Ġnetwork Ġand Ġuses Ġthe Ġresulting Ġhidden Ġstate Ġinstead Ġof Ġthe Ġcontext - independent ĠIN LINE FORM 3 Ġ, Ġto Ġcompare Ġto Ġthe Ġhidden Ġstates Ġof Ġthe Ġenc oder . ĠThis Ġis Ġdefined Ġin Ġthe Ġfollowing Ġequations : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> Here , ĠIN LINE FORM 0 Ġis Ġthe Ġfunction Ġthat Ġthe Ġstacked ĠL ST M Ġapplies Ġto Ġthe Ġinput , ĠIN LINE FORM 1 Ġis Ġthe Ġlast Ġgenerated Ġtarget Ġword , Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġoutput Ġof Ġprevious Ġtime Ġstep Ġof Ġthe Ġinput - feeding Ġnetwork Ġitself , Ġmeaning Ġthe Ġoutput Ġof ĠEqu ation ĠEQ REF 2 Ġin Ġthe Ġcase Ġthat Ġcontext Ġvector Ġhas Ġbeen Ġcomputed Ġusing ĠIN LINE FORM 3 Ġfrom ĠEqu ation ĠEQ REF 7 Ġ. </s> Comp aring ĠAttention Ġwith ĠAl ignment </s> As Ġmentioned Ġabove , Ġit Ġis Ġa Ġcommonly Ġheld Ġassumption Ġthat Ġattention Ġcorresponds Ġto Ġword Ġalign ments . ĠTo Ġverify Ġthis , Ġwe Ġinvestigate Ġwhether Ġhigher Ġconsistency Ġbetween Ġattention Ġand Ġalignment Ġleads Ġto Ġbetter Ġtranslations . </s> Me asuring ĠAttention - Al ignment ĠAccuracy </s> In Ġorder Ġto Ġcompare Ġattent ions Ġof Ġmultiple Ġsystems Ġas Ġwell Ġas Ġto Ġmeasure Ġthe Ġdifference Ġbetween Ġattention Ġand Ġword Ġalignment , Ġwe Ġconvert Ġthe Ġhard Ġword Ġalign ments Ġinto Ġsoft Ġones Ġand Ġuse Ġcross Ġentropy Ġbetween Ġattention Ġand Ġsoft Ġalignment Ġas Ġa Ġloss Ġfunction . ĠFor Ġthis Ġpurpose , Ġwe Ġuse Ġmanual Ġalign ments Ġprovided Ġby ĠRW TH ĠGerman - English Ġdataset Ġas Ġthe Ġhard Ġalign ments . ĠThe Ġstatistics Ġof Ġthe Ġdata Ġare Ġgiven Ġin ĠTable ĠT AB REF 8 Ġ. ĠWe Ġconvert Ġthe Ġhard Ġalign ments Ġto Ġsoft Ġalign ments Ġusing ĠEqu ation ĠEQ REF 10 Ġ. ĠFor Ġun aligned Ġwords , Ġwe Ġfirst Ġassume Ġthat Ġthey Ġhave Ġbeen Ġaligned Ġto Ġall Ġthe Ġwords Ġin Ġthe Ġsource Ġside Ġand Ġthen Ġdo Ġthe Ġconversion . ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġset Ġof Ġsource Ġwords Ġaligned Ġto Ġtarget Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġsource Ġwords Ġin Ġthe Ġset . </s> After Ġconversion Ġof Ġthe Ġhard Ġalign ments Ġto Ġsoft Ġones , Ġwe Ġcompute Ġthe Ġattention Ġloss Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġsource Ġsentence Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġweight Ġof Ġthe Ġalignment Ġlink Ġbetween Ġsource Ġword ĠIN LINE FORM 2 Ġand Ġthe Ġtarget Ġword Ġ( see ĠEqu ation ĠEQ REF 10 Ġ). ĠIN LINE FORM 3 Ġis Ġthe Ġattention Ġweight ĠIN LINE FORM 4 Ġ( see ĠEqu ation ĠEQ REF 4 Ġ) Ġof Ġthe Ġsource Ġword ĠIN LINE FORM 5 Ġ, Ġwhen Ġgenerating Ġthe Ġtarget Ġword ĠIN LINE FORM 6 Ġ. </s> In Ġour Ġanalysis , Ġwe Ġalso Ġlook Ġinto Ġthe Ġrelation Ġbetween Ġtranslation Ġquality Ġand Ġthe Ġquality Ġof Ġthe Ġattention Ġwith Ġrespect Ġto Ġthe Ġalign ments . ĠFor Ġmeasuring Ġthe Ġquality Ġof Ġattention , Ġwe Ġuse Ġthe Ġattention Ġloss Ġdefined Ġin ĠEqu ation ĠEQ REF 11 Ġ. ĠAs Ġa Ġmeasure Ġof Ġtranslation Ġquality , Ġwe Ġchoose Ġthe Ġloss Ġbetween Ġthe Ġoutput Ġof Ġour ĠN MT Ġsystem Ġand Ġthe Ġreference Ġtranslation Ġat Ġeach Ġtranslation Ġstep , Ġwhich Ġwe Ġcall Ġword Ġprediction Ġloss . ĠThe Ġword Ġprediction Ġloss Ġfor Ġword ĠIN LINE FORM 0 Ġis Ġlog ar ith m Ġof Ġthe Ġprobability Ġgiven Ġin ĠEqu ation ĠEQ REF 12 Ġ. ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġsource Ġsentence , ĠIN LINE FORM 1 Ġis Ġtarget Ġword Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġhistory Ġgiven Ġby Ġthe Ġreference Ġtranslation Ġand ĠIN LINE FORM 4 Ġis Ġgiven Ġby ĠEqu ation ĠEQ REF 2 Ġfor Ġeither Ġnon - rec urrent Ġor Ġinput - feeding Ġattention Ġmodels . </s> Spe ar man 's Ġrank Ġcorrelation Ġis Ġused Ġto Ġcompute Ġthe Ġcorrelation Ġbetween Ġattention Ġloss Ġand Ġword Ġprediction Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġranks Ġof Ġthe Ġattention Ġlosses Ġand Ġword Ġprediction Ġlosses , Ġrespectively , ĠIN LINE FORM 2 Ġis Ġthe Ġcovari ance Ġbetween Ġtwo Ġinput Ġvariables , Ġand ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġthe Ġstandard Ġdeviations Ġof ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ. </s> If Ġthere Ġis Ġa Ġclose Ġrelationship Ġbetween Ġword Ġprediction Ġquality Ġand Ġconsistency Ġof Ġattention Ġversus Ġalignment , Ġthen Ġthere Ġshould Ġbe Ġhigh Ġcorrelation Ġbetween Ġword Ġprediction Ġloss Ġand Ġattention Ġloss . ĠFigure ĠFIG REF 13 Ġshows Ġan Ġexample Ġwith Ġdifferent Ġlevels Ġof Ġconsistency Ġbetween Ġattention Ġand Ġword Ġalign ments . ĠFor Ġthe Ġtarget Ġwords ĠâĢ ľ will " Ġand ĠâĢ ľ come " Ġthe Ġattention Ġis Ġnot Ġfocused Ġon Ġthe Ġmanually Ġaligned Ġword Ġbut Ġdistributed Ġbetween Ġthe Ġaligned Ġword Ġand Ġother Ġwords . ĠThe Ġfocus Ġof Ġthis Ġpaper Ġis Ġexamining Ġcases Ġwhere Ġattention Ġdoes Ġnot Ġfollow Ġalignment , Ġanswering Ġthe Ġquestions Ġwhether Ġthose Ġcases Ġrepresent Ġerrors Ġor Ġdesirable Ġbehavior Ġof Ġthe Ġattention Ġmodel . </s> Me asuring ĠAttention ĠConcent ration </s> As Ġanother Ġinformative Ġvariable Ġin Ġour Ġanalysis , Ġwe Ġlook Ġinto Ġthe Ġattention Ġconcentration . ĠWhile Ġmost Ġword Ġalign ments Ġonly Ġinvolve Ġone Ġor Ġa Ġfew Ġwords , Ġattention Ġcan Ġbe Ġdistributed Ġmore Ġfreely . ĠWe Ġmeasure Ġthe Ġconcentration Ġof Ġattention Ġby Ġcomputing Ġthe Ġentropy Ġof Ġthe Ġattention Ġdistribution : ĠDIS PLAY FORM 0 </s> E mp ir ical ĠAnalysis Ġof ĠAttention ĠBeh aviour </s> We Ġconduct Ġour Ġanalysis Ġusing Ġthe Ġtwo Ġdifferent Ġattention Ġmodels Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ. ĠOur Ġfirst Ġattention Ġmodel Ġis Ġthe Ġglobal Ġmodel Ġwithout Ġinput - feeding Ġas Ġintroduced Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠThe Ġsecond Ġmodel Ġis Ġthe Ġinput - feeding Ġmodel ĠB IB REF 5 Ġ, Ġwhich Ġuses Ġrecurrent Ġattention . ĠOur ĠN MT Ġsystem Ġis Ġa Ġun id irection al Ġenc oder - dec oder Ġsystem Ġas Ġdescribed Ġin ĠB IB REF 5 Ġ, Ġusing Ġ4 Ġrecurrent Ġlayers . </s> We Ġtrained Ġthe Ġsystems Ġwith Ġdimension Ġsize Ġof Ġ1 , 000 Ġand Ġbatch Ġsize Ġof Ġ80 Ġfor Ġ20 Ġepoch s . ĠThe Ġvocabulary Ġfor Ġboth Ġsource Ġand Ġtarget Ġside Ġis Ġset Ġto Ġbe Ġthe Ġ30 K Ġmost Ġcommon Ġwords . ĠThe Ġlearning Ġrate Ġis Ġset Ġto Ġbe Ġ1 Ġand Ġa Ġmaximum Ġgradient Ġnorm Ġof Ġ5 Ġhas Ġbeen Ġused . ĠWe Ġalso Ġuse Ġa Ġdrop out Ġrate Ġof Ġ0 . 3 Ġto Ġavoid Ġover fitting . </s> Imp act Ġof ĠAttention ĠMechan ism </s> We Ġtrain Ġboth Ġof Ġthe Ġsystems Ġon Ġthe ĠW MT 15 ĠGerman - to - English Ġtraining Ġdata , Ġsee ĠTable ĠT AB REF 18 Ġfor Ġsome Ġstatistics . ĠTable ĠT AB REF 17 Ġshows Ġthe ĠB LE U Ġscores ĠB IB REF 12 Ġfor Ġboth Ġsystems Ġon Ġdifferent Ġtest Ġsets . </s> Since Ġwe Ġuse ĠPOS Ġtags Ġand Ġdependency Ġroles Ġin Ġour Ġanalysis , Ġboth Ġof Ġwhich Ġare Ġbased Ġon Ġwords , Ġwe Ġchose Ġnot Ġto Ġuse ĠB PE ĠB IB REF 13 Ġwhich Ġoperates Ġat Ġthe Ġsub - word Ġlevel . </s> We Ġreport Ġalignment Ġerror Ġrate Ġ( A ER ) ĠB IB REF 14 Ġ, Ġwhich Ġis Ġcommonly Ġused Ġto Ġmeasure Ġalignment Ġquality , Ġin ĠTable ĠT AB REF 20 Ġto Ġshow Ġthe Ġdifference Ġbetween Ġattent ions Ġand Ġhuman Ġalign ments Ġprovided Ġby ĠRW TH ĠGerman - English Ġdataset . ĠTo Ġcompute ĠA ER Ġover Ġattent ions , Ġwe Ġfollow ĠD BL P j ournals cor r Lu ong PM 15 Ġto Ġproduce Ġhard Ġalign ments Ġfrom Ġattent ions Ġby Ġchoosing Ġthe Ġmost Ġattended Ġsource Ġword Ġfor Ġeach Ġtarget Ġword . ĠWe Ġalso Ġuse ĠG IZ A ++ ĠB IB REF 10 Ġto Ġproduce Ġautomatic Ġalign ments Ġover Ġthe Ġdata Ġset Ġto Ġallow Ġfor Ġa Ġcomparison Ġbetween Ġautomatically Ġgenerated Ġalign ments Ġand Ġthe Ġattent ions Ġgenerated Ġby Ġour Ġsystems . ĠG IZ A ++ Ġis Ġrun Ġin Ġboth Ġdirections Ġand Ġalign ments Ġare Ġsymm et ri zed Ġusing Ġthe Ġgrow - di ag - final - and Ġrefined Ġalignment Ġhe uristic . </s> As Ġshown Ġin ĠTable ĠT AB REF 20 Ġ, Ġthe Ġinput - feeding Ġsystem Ġnot Ġonly Ġachieves Ġa Ġhigher ĠB LE U Ġscore , Ġbut Ġalso Ġuses Ġattent ions Ġthat Ġare Ġcloser Ġto Ġthe Ġhuman Ġalign ments . </s> Table ĠT AB REF 21 Ġcompares Ġinput - feeding Ġand Ġnon - rec urrent Ġattention Ġin Ġterms Ġof Ġattention Ġloss Ġcomputed Ġusing ĠEqu ation ĠEQ REF 11 Ġ. ĠHere Ġthe Ġlosses Ġbetween Ġthe Ġattention Ġproduced Ġby Ġeach Ġsystem Ġand Ġthe Ġhuman Ġalign ments Ġis Ġreported . ĠAs Ġexpected , Ġthe Ġdifference Ġin Ġattention Ġlosses Ġare Ġin Ġline Ġwith ĠA ER . </s> The Ġdifference Ġbetween Ġthese Ġcomparisons Ġis Ġthat ĠA ER Ġonly Ġtakes Ġthe Ġmost Ġattended Ġword Ġinto Ġaccount Ġwhile Ġattention Ġloss Ġconsiders Ġthe Ġentire Ġattention Ġdistribution . </s> Al ignment ĠQuality ĠImpact Ġon ĠTranslation </s> Based Ġon Ġthe Ġresults Ġin ĠSection ĠSEC REF 19 Ġ, Ġone Ġmight Ġbe Ġinclined Ġto Ġconclude Ġthat Ġthe Ġcloser Ġthe Ġattention Ġis Ġto Ġthe Ġword Ġalign ments Ġthe Ġbetter Ġthe Ġtranslation . ĠHowever , Ġc hen 2016 guided , Ġli u - E t Al : 2016 : COL ING , Ġal kh oul i - E t Al : 2016 : W MT Ġreport Ġmixed Ġresults Ġby Ġoptimizing Ġtheir ĠN MT Ġsystem Ġwith Ġrespect Ġto Ġword Ġprediction Ġand Ġalignment Ġquality . ĠThese Ġfindings Ġwarrant Ġa Ġmore Ġfine - gr ained Ġanalysis Ġof Ġattention . ĠTo Ġthis Ġend , Ġwe Ġinclude ĠPOS Ġtags Ġin Ġour Ġanalysis Ġand Ġstudy Ġthe Ġpatterns Ġof Ġattention Ġbased Ġon ĠPOS Ġtags Ġof Ġthe Ġtarget Ġwords . ĠWe Ġchoose ĠPOS Ġtags Ġbecause Ġthey Ġexhibit Ġsome Ġsimple Ġsynt actic Ġcharacteristics . ĠWe Ġuse Ġthe Ġcoarse Ġgr ained Ġuniversal ĠPOS Ġtags ĠB IB REF 15 Ġgiven Ġin ĠTable ĠT AB REF 25 Ġ. </s> To Ġbetter Ġunderstand Ġhow Ġattention Ġaccuracy Ġaffects Ġtranslation Ġquality , Ġwe Ġanalyse Ġthe Ġrelationship Ġbetween Ġattention Ġloss Ġand Ġword Ġprediction Ġloss Ġfor Ġindividual Ġpart - of - speech Ġclasses . ĠFigure ĠFIG REF 22 Ġshows Ġhow Ġattention Ġloss Ġdiffers Ġwhen Ġgenerating Ġdifferent ĠPOS Ġtags . ĠOne Ġcan Ġsee Ġthat Ġattention Ġloss Ġvaries Ġsubstantially Ġacross Ġdifferent ĠPOS Ġtags . ĠIn Ġparticular , Ġwe Ġfocus Ġon Ġthe Ġcases Ġof ĠN OUN Ġand ĠVER B Ġwhich Ġare Ġthe Ġmost Ġfrequent ĠPOS Ġtags Ġin Ġthe Ġdataset . ĠAs Ġshown , Ġthe Ġattention Ġof ĠN OUN Ġis Ġthe Ġclosest Ġto Ġalign ments Ġon Ġaverage . ĠBut Ġthe Ġaverage Ġattention Ġloss Ġfor ĠVER B Ġis Ġalmost Ġtwo Ġtimes Ġlarger Ġthan Ġthe Ġloss Ġfor ĠN OUN . </s> Considering Ġthis Ġdifference Ġand Ġthe Ġobservations Ġin ĠSection ĠSEC REF 19 Ġ, Ġa Ġnatural Ġfollow - up Ġwould Ġbe Ġto Ġfocus Ġon Ġgetting Ġthe Ġattention Ġof Ġverbs Ġto Ġbe Ġcloser Ġto Ġalign ments . ĠHowever , ĠFigure ĠFIG REF 22 Ġshows Ġthat Ġthe Ġaverage Ġword Ġprediction Ġloss Ġfor Ġverbs Ġis Ġactually Ġsmaller Ġcompared Ġto Ġthe Ġloss Ġfor Ġnoun s . ĠIn Ġother Ġwords , Ġalthough Ġthe Ġattention Ġfor Ġverbs Ġis Ġsubstantially Ġmore Ġinconsistent Ġwith Ġthe Ġword Ġalign ments Ġthan Ġfor Ġnoun s , Ġthe ĠN MT Ġsystem Ġtranslates Ġverbs Ġmore Ġaccurately Ġthan Ġnoun s Ġon Ġaverage . </s> To Ġformal ize Ġthis Ġrelationship Ġwe Ġcompute ĠSpear man 's Ġrank Ġcorrelation Ġbetween Ġword Ġprediction Ġloss Ġand Ġattention Ġloss , Ġbased Ġon Ġthe ĠPOS Ġtags Ġof Ġthe Ġtarget Ġside , Ġfor Ġthe Ġinput - feeding Ġmodel , Ġsee ĠFigure ĠFIG REF 27 Ġ. </s> The Ġlow Ġcorrelation Ġfor Ġverbs Ġconfirms Ġthat Ġattention Ġto Ġother Ġparts Ġof Ġsource Ġsentence Ġrather Ġthan Ġthe Ġaligned Ġword Ġis Ġnecessary Ġfor Ġtranslating Ġverbs Ġand Ġthat Ġattention Ġdoes Ġnot Ġnecessarily Ġhave Ġto Ġfollow Ġalign ments . ĠHowever , Ġthe Ġhigher Ġcorrelation Ġfor Ġnoun s Ġmeans Ġthat Ġconsistency Ġof Ġattention Ġwith Ġalign ments Ġis Ġmore Ġdesirable . ĠThis Ġcould , Ġin Ġa Ġway , Ġexplain Ġthe Ġmixed Ġresult Ġreported Ġfor Ġtraining Ġattention Ġusing Ġalign ments ĠB IB REF 9 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠEspecially Ġthe Ġresults Ġby Ġc hen 2016 guided Ġin Ġwhich Ġlarge Ġimprovements Ġare Ġachieved Ġfor Ġthe Ġe - commerce Ġdomain Ġwhich Ġcontains Ġmany ĠO OV Ġproduct Ġnames Ġand Ġplace holders , Ġbut Ġno Ġor Ġvery Ġweak Ġimprovements Ġwere Ġachieved Ġover Ġcommon Ġdomains . </s> Att ention ĠConcent ration </s> In Ġword Ġalignment , Ġmost Ġtarget Ġwords Ġare Ġaligned Ġto Ġone Ġsource Ġword . ĠThe Ġaverage Ġnumber Ġof Ġsource Ġwords Ġaligned Ġto Ġnoun s Ġand Ġverbs Ġis Ġ1 . 1 Ġand Ġ1 . 2 Ġrespectively . ĠTo Ġinvestigate Ġto Ġwhat Ġextent Ġthis Ġalso Ġholds Ġfor Ġattention Ġwe Ġmeasure Ġthe Ġattention Ġconcentration Ġby Ġcomputing Ġthe Ġentropy Ġof Ġthe Ġattention Ġdistribution , Ġsee ĠEqu ation ĠEQ REF 16 Ġ. </s> Figure ĠFIG REF 28 Ġshows Ġthe Ġaverage Ġentropy Ġof Ġattention Ġbased Ġon ĠPOS Ġtags . ĠAs Ġshown , Ġnoun s Ġhave Ġone Ġof Ġthe Ġlowest Ġent rop ies Ġmeaning Ġthat Ġon Ġaverage Ġthe Ġattention Ġfor Ġnoun s Ġtends Ġto Ġbe Ġconcentrated . ĠThis Ġalso Ġexplains Ġthe Ġclos eness Ġof Ġthe Ġattention Ġto Ġalign ments Ġfor Ġnoun s . ĠIn Ġaddition , Ġthe Ġcorrelation Ġbetween Ġattention Ġentropy Ġand Ġattention Ġloss Ġin Ġcase Ġof Ġnoun s Ġis Ġhigh Ġas Ġshown Ġin ĠFigure ĠFIG REF 28 Ġ. ĠThis Ġmeans Ġthat Ġattention Ġentropy Ġcan Ġbe Ġused Ġas Ġa Ġmeasure Ġof Ġclos eness Ġof Ġattention Ġto Ġalignment Ġin Ġthe Ġcase Ġof Ġnoun s . </s> The Ġhigher Ġattention Ġentropy Ġfor Ġverbs , Ġin ĠFigure ĠFIG REF 28 Ġ, Ġshows Ġthat Ġthe Ġattention Ġis Ġmore Ġdistributed Ġcompared Ġto Ġnoun s . ĠThe Ġlow Ġcorrelation Ġbetween Ġattention Ġentropy Ġand Ġword Ġprediction Ġloss Ġ( see ĠFigure ĠFIG REF 32 Ġ) Ġshows Ġthat Ġattention Ġconcentration Ġis Ġnot Ġrequired Ġwhen Ġtranslating Ġinto Ġverbs . ĠThis Ġalso Ġconfirms Ġthat Ġthe Ġcorrect Ġtranslation Ġof Ġverbs Ġrequires Ġthe Ġsystems Ġto Ġpay Ġattention Ġto Ġdifferent Ġparts Ġof Ġthe Ġsource Ġsentence . </s> Another Ġinteresting Ġobservation Ġhere Ġis Ġthe Ġlow Ġcorrelation Ġfor Ġpronouns Ġ( PR ON ) Ġand Ġparticles Ġ( PR T ), Ġsee ĠFigure ĠFIG REF 32 Ġ. ĠAs Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 28 Ġ, Ġthese Ġtags Ġhave Ġmore Ġdistributed Ġattention Ġcomparing Ġto Ġnoun s , Ġfor Ġexample . ĠThis Ġcould Ġeither Ġmean Ġthat Ġthe Ġattention Ġmodel Ġdoes Ġnot Ġknow Ġwhere Ġto Ġfocus Ġor Ġit Ġdeliberately Ġpays Ġattention Ġto Ġmultiple , Ġsomehow Ġrelevant , Ġplaces Ġto Ġbe Ġable Ġto Ġproduce Ġa Ġbetter Ġtranslation . ĠThe Ġlatter Ġis Ġsupported Ġby Ġthe Ġrelatively Ġlow Ġword Ġprediction Ġlosses , Ġshown Ġin Ġthe ĠFigure ĠFIG REF 22 Ġ. </s> Att ention ĠDistribution </s> To Ġfurther Ġunderstand Ġunder Ġwhich Ġconditions Ġattention Ġis Ġpaid Ġto Ġwords Ġother Ġthan Ġthe Ġaligned Ġwords , Ġwe Ġstudy Ġthe Ġdistribution Ġof Ġattention Ġover Ġthe Ġsource Ġwords . ĠFirst , Ġwe Ġmeasure Ġhow Ġmuch Ġattention Ġis Ġpaid Ġto Ġthe Ġaligned Ġwords Ġfor Ġeach ĠPOS Ġtag , Ġon Ġaverage . ĠTo Ġthis Ġend , Ġwe Ġcompute Ġthe Ġpercentage Ġof Ġthe Ġprobability Ġmass Ġthat Ġthe Ġattention Ġmodel Ġhas Ġassigned Ġto Ġaligned Ġwords Ġfor Ġeach ĠPOS Ġtag , Ġsee ĠTable ĠT AB REF 35 Ġ. </s> One Ġcan Ġnotice Ġthat Ġless Ġthan Ġhalf Ġof Ġthe Ġattention Ġis Ġpaid Ġto Ġalignment Ġpoints Ġfor Ġmost Ġof Ġthe ĠPOS Ġtags . ĠTo Ġexamine Ġhow Ġthe Ġrest Ġof Ġattention Ġin Ġeach Ġcase Ġhas Ġbeen Ġdistributed Ġover Ġthe Ġsource Ġsentence Ġwe Ġmeasure Ġthe Ġattention Ġdistribution Ġover Ġdependency Ġroles Ġin Ġthe Ġsource Ġside . ĠWe Ġfirst Ġparse Ġthe Ġsource Ġside Ġof ĠRW TH Ġdata Ġusing Ġthe ĠPar Z u Ġparser ĠB IB REF 16 Ġ. ĠThen Ġwe Ġcompute Ġhow Ġthe Ġattention Ġprobability Ġmass Ġgiven Ġto Ġthe Ġwords Ġother Ġthan Ġthe Ġalignment Ġpoints , Ġis Ġdistributed Ġover Ġdependency Ġroles . ĠTable ĠT AB REF 33 Ġgives Ġthe Ġmost Ġattended Ġroles Ġfor Ġeach ĠPOS Ġtag . ĠHere , Ġwe Ġfocus Ġon ĠPOS Ġtags Ġdiscussed Ġearlier . ĠOne Ġcan Ġsee Ġthat Ġthe Ġmost Ġattended Ġroles Ġwhen Ġtranslating Ġto Ġnoun s Ġinclude Ġadject ives Ġand Ġdetermin ers Ġand Ġin Ġthe Ġcase Ġof Ġtranslating Ġto Ġverbs , Ġit Ġincludes Ġauxiliary Ġverbs , Ġad verbs Ġ( including Ġneg ation ), Ġsubjects , Ġand Ġobjects . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġhave Ġstudied Ġattention Ġin Ġneural Ġmachine Ġtranslation Ġand Ġprovided Ġan Ġanalysis Ġof Ġthe Ġrelation Ġbetween Ġattention Ġand Ġword Ġalignment . ĠWe Ġhave Ġshown Ġthat Ġattention Ġagrees Ġwith Ġtraditional Ġalignment Ġto Ġa Ġcertain Ġextent . ĠHowever , Ġthis Ġdiffers Ġsubstantially Ġby Ġattention Ġmechanism Ġand Ġthe Ġtype Ġof Ġthe Ġword Ġbeing Ġgenerated . ĠWe Ġhave Ġshown Ġthat Ġattention Ġhas Ġdifferent Ġpatterns Ġbased Ġon Ġthe ĠPOS Ġtag Ġof Ġthe Ġtarget Ġword . ĠThe Ġconcentrated Ġpattern Ġof Ġattention Ġand Ġthe Ġrelatively Ġhigh Ġcorrelations Ġfor Ġnoun s Ġshow Ġthat Ġtraining Ġthe Ġattention Ġwith Ġexplicit Ġalignment Ġlabels Ġis Ġuseful Ġfor Ġgenerating Ġnoun s . ĠHowever , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġverbs , Ġsince Ġthe Ġlarge Ġportion Ġof Ġattention Ġbeing Ġpaid Ġto Ġwords Ġother Ġthan Ġalignment Ġpoints , Ġis Ġalready Ġcapturing Ġother Ġrelevant Ġinformation . ĠTraining Ġattention Ġwith Ġalign ments Ġin Ġthis Ġcase Ġwill Ġforce Ġthe Ġattention Ġmodel Ġto Ġforget Ġthese Ġuseful Ġinformation . ĠThis Ġexplains Ġthe Ġmixed Ġresults Ġreported Ġwhen Ġguiding Ġattention Ġto Ġcomply Ġwith Ġalign ments ĠB IB REF 9 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> Acknowled gments </s> This Ġresearch Ġwas Ġfunded Ġin Ġpart Ġby Ġthe ĠNetherlands ĠOrganization Ġfor ĠScientific ĠResearch Ġ( NW O ) Ġunder Ġproject Ġnumbers Ġ6 39 . 022 . 213 Ġand Ġ6 12 . 001 . 218 .
Introduction Neural machine translation (NMT) has gained a lot of attention recently due to its substantial improvements in machine translation quality achieving state-of-the-art performance for several languages BIBREF0 , BIBREF1 , BIBREF2 . The core architecture of neural machine translation models is based on the general encoder-decoder approach BIBREF3 . Neural machine translation is an end-to-end approach that learns to encode source sentences into distributed representations and decode these representations into sentences in the target language. Among the different neural MT models, attentional NMT BIBREF4 , BIBREF5 has become popular due to its capability to use the most relevant parts of the source sentence at each translation step. This capability also makes the attentional model superior in translating longer sentences BIBREF4 , BIBREF5 . Figure FIGREF1 shows an example of how attention uses the most relevant source words to generate a target word at each step of the translation. In this paper we focus on studying the relevance of the attended parts, especially cases where attention is `smeared out' over multiple source words where their relevance is not entirely obvious, see, e.g., “would" and “like" in Figure FIGREF1 . Here, we ask whether these are due to errors of the attention mechanism or are a desired behavior of the model. Since the introduction of attention models in neural machine translation BIBREF4 various modifications have been proposed BIBREF5 , BIBREF6 , BIBREF7 . However, to the best of our knowledge there is no study that provides an analysis of what kind of phenomena is being captured by attention. There are some works that have looked to attention as being similar to traditional word alignment BIBREF8 , BIBREF6 , BIBREF7 , BIBREF9 . Some of these approaches also experimented with training the attention model using traditional alignments BIBREF8 , BIBREF7 , BIBREF9 . liu-EtAl:2016:COLING have shown that attention could be seen as a reordering model as well as an alignment model. In this paper, we focus on investigating the differences between attention and alignment and what is being captured by the attention mechanism in general. The questions that we are aiming to answer include: Is the attention model only capable of modelling alignment? And how similar is attention to alignment in different syntactic phenomena? Our analysis shows that attention models traditional alignment in some cases more closely while it captures information beyond alignment in others. For instance, attention agrees with traditional alignments to a high degree in the case of nouns. However, it captures other information rather than only the translational equivalent in the case of verbs. This paper makes the following contributions: 1) We provide a detailed comparison of attention in NMT and word alignment. 2) We show that while different attention mechanisms can lead to different degrees of compliance with respect to word alignments, global compliance is not always helpful for word prediction. 3) We show that attention follows different patterns depending on the type of the word being generated. 4) We demonstrate that attention does not always comply with alignment. We provide evidence showing that the difference between attention and alignment is due to attention model capability to attend the context words influencing the current word translation. Related Work liu-EtAl:2016:COLING investigate how training the attention model in a supervised manner can benefit machine translation quality. To this end they use traditional alignments obtained by running automatic alignment tools (GIZA++ BIBREF10 and fast_align BIBREF11 ) on the training data and feed it as ground truth to the attention network. They report some improvements in translation quality arguing that the attention model has learned to better align source and target words. The approach of training attention using traditional alignments has also been proposed by others BIBREF9 , BIBREF8 . chen2016guided show that guided attention with traditional alignment helps in the domain of e-commerce data which includes lots of out of vocabulary (OOV) product names and placeholders, but not much in the other domains. alkhouli-EtAl:2016:WMT have separated the alignment model and translation model, reasoning that this avoids propagation of errors from one model to the other as well as providing more flexibility in the model types and training of the models. They use a feed-forward neural network as their alignment model that learns to model jumps in the source side using HMM/IBM alignments obtained by using GIZA++. shi-padhi-knight:2016:EMNLP2016 show that various kinds of syntactic information are being learned and encoded in the output hidden states of the encoder. The neural system for their experimental analysis is not an attentional model and they argue that attention does not have any impact for learning syntactic information. However, performing the same analysis for morphological information, belinkov2017neural show that attention has also some effect on the information that the encoder of neural machine translation system encodes in its output hidden states. As part of their analysis they show that a neural machine translation system that has an attention model can learn the POS tags of the source side more efficiently than a system without attention. Recently, koehn2017six carried out a brief analysis of how much attention and alignment match in different languages by measuring the probability mass that attention gives to alignments obtained from an automatic alignment tool. They also report differences based on the most attended words. The mixed results reported by chen2016guided, alkhouli-EtAl:2016:WMT, liu-EtAl:2016:COLING on optimizing attention with respect to alignments motivates a more thorough analysis of attention models in NMT. Attention Models This section provides a short background on attention and discusses two most popular attention models which are also used in this paper. The first model is a non-recurrent attention model which is equivalent to the “global attention" method proposed by DBLPjournalscorrLuongPM15. The second attention model that we use in our investigation is an input-feeding model similar to the attention model first proposed by bahdanau-EtAl:2015:ICLR and turned to a more general one and called input-feeding by DBLPjournalscorrLuongPM15. Below we describe the details of both models. Both non-recurrent and input-feeding models compute a context vector INLINEFORM0 at each time step. Subsequently, they concatenate the context vector to the hidden state of decoder and pass it through a non-linearity before it is fed into the softmax output layer of the translation network. DISPLAYFORM0 The difference of the two models lays in the way they compute the context vector. In the non-recurrent model, the hidden state of the decoder is compared to each hidden state of the encoder. Often, this comparison is realized as the dot product of vectors. Then the comparison result is fed to a softmax layer to compute the attention weight. DISPLAYFORM0 DISPLAYFORM1 Here INLINEFORM0 is the hidden state of the decoder at time INLINEFORM1 , INLINEFORM2 is INLINEFORM3 th hidden state of the encoder and INLINEFORM4 is the length of the source sentence. Then the computed alignment weights are used to compute a weighted sum over the encoder hidden states which results in the context vector mentioned above: DISPLAYFORM0 The input-feeding model changes the context vector computation in a way that at each step INLINEFORM0 the context vector is aware of the previously computed context INLINEFORM1 . To this end, the input-feeding model feeds back its own INLINEFORM2 to the network and uses the resulting hidden state instead of the context-independent INLINEFORM3 , to compare to the hidden states of the encoder. This is defined in the following equations: DISPLAYFORM0 DISPLAYFORM1 Here, INLINEFORM0 is the function that the stacked LSTM applies to the input, INLINEFORM1 is the last generated target word, and INLINEFORM2 is the output of previous time step of the input-feeding network itself, meaning the output of Equation EQREF2 in the case that context vector has been computed using INLINEFORM3 from Equation EQREF7 . Comparing Attention with Alignment As mentioned above, it is a commonly held assumption that attention corresponds to word alignments. To verify this, we investigate whether higher consistency between attention and alignment leads to better translations. Measuring Attention-Alignment Accuracy In order to compare attentions of multiple systems as well as to measure the difference between attention and word alignment, we convert the hard word alignments into soft ones and use cross entropy between attention and soft alignment as a loss function. For this purpose, we use manual alignments provided by RWTH German-English dataset as the hard alignments. The statistics of the data are given in Table TABREF8 . We convert the hard alignments to soft alignments using Equation EQREF10 . For unaligned words, we first assume that they have been aligned to all the words in the source side and then do the conversion. DISPLAYFORM0 Here INLINEFORM0 is the set of source words aligned to target word INLINEFORM1 and INLINEFORM2 is the number of source words in the set. After conversion of the hard alignments to soft ones, we compute the attention loss as follows: DISPLAYFORM0 Here INLINEFORM0 is the source sentence and INLINEFORM1 is the weight of the alignment link between source word INLINEFORM2 and the target word (see Equation EQREF10 ). INLINEFORM3 is the attention weight INLINEFORM4 (see Equation EQREF4 ) of the source word INLINEFORM5 , when generating the target word INLINEFORM6 . In our analysis, we also look into the relation between translation quality and the quality of the attention with respect to the alignments. For measuring the quality of attention, we use the attention loss defined in Equation EQREF11 . As a measure of translation quality, we choose the loss between the output of our NMT system and the reference translation at each translation step, which we call word prediction loss. The word prediction loss for word INLINEFORM0 is logarithm of the probability given in Equation EQREF12 . DISPLAYFORM0 Here INLINEFORM0 is the source sentence, INLINEFORM1 is target word at time step INLINEFORM2 , INLINEFORM3 is the target history given by the reference translation and INLINEFORM4 is given by Equation EQREF2 for either non-recurrent or input-feeding attention models. Spearman's rank correlation is used to compute the correlation between attention loss and word prediction loss: DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 are the ranks of the attention losses and word prediction losses, respectively, INLINEFORM2 is the covariance between two input variables, and INLINEFORM3 and INLINEFORM4 are the standard deviations of INLINEFORM5 and INLINEFORM6 . If there is a close relationship between word prediction quality and consistency of attention versus alignment, then there should be high correlation between word prediction loss and attention loss. Figure FIGREF13 shows an example with different levels of consistency between attention and word alignments. For the target words “will" and “come" the attention is not focused on the manually aligned word but distributed between the aligned word and other words. The focus of this paper is examining cases where attention does not follow alignment, answering the questions whether those cases represent errors or desirable behavior of the attention model. Measuring Attention Concentration As another informative variable in our analysis, we look into the attention concentration. While most word alignments only involve one or a few words, attention can be distributed more freely. We measure the concentration of attention by computing the entropy of the attention distribution: DISPLAYFORM0 Empirical Analysis of Attention Behaviour We conduct our analysis using the two different attention models described in Section SECREF3 . Our first attention model is the global model without input-feeding as introduced by DBLPjournalscorrLuongPM15. The second model is the input-feeding model BIBREF5 , which uses recurrent attention. Our NMT system is a unidirectional encoder-decoder system as described in BIBREF5 , using 4 recurrent layers. We trained the systems with dimension size of 1,000 and batch size of 80 for 20 epochs. The vocabulary for both source and target side is set to be the 30K most common words. The learning rate is set to be 1 and a maximum gradient norm of 5 has been used. We also use a dropout rate of 0.3 to avoid overfitting. Impact of Attention Mechanism We train both of the systems on the WMT15 German-to-English training data, see Table TABREF18 for some statistics. Table TABREF17 shows the BLEU scores BIBREF12 for both systems on different test sets. Since we use POS tags and dependency roles in our analysis, both of which are based on words, we chose not to use BPE BIBREF13 which operates at the sub-word level. We report alignment error rate (AER) BIBREF14 , which is commonly used to measure alignment quality, in Table TABREF20 to show the difference between attentions and human alignments provided by RWTH German-English dataset. To compute AER over attentions, we follow DBLPjournalscorrLuongPM15 to produce hard alignments from attentions by choosing the most attended source word for each target word. We also use GIZA++ BIBREF10 to produce automatic alignments over the data set to allow for a comparison between automatically generated alignments and the attentions generated by our systems. GIZA++ is run in both directions and alignments are symmetrized using the grow-diag-final-and refined alignment heuristic. As shown in Table TABREF20 , the input-feeding system not only achieves a higher BLEU score, but also uses attentions that are closer to the human alignments. Table TABREF21 compares input-feeding and non-recurrent attention in terms of attention loss computed using Equation EQREF11 . Here the losses between the attention produced by each system and the human alignments is reported. As expected, the difference in attention losses are in line with AER. The difference between these comparisons is that AER only takes the most attended word into account while attention loss considers the entire attention distribution. Alignment Quality Impact on Translation Based on the results in Section SECREF19 , one might be inclined to conclude that the closer the attention is to the word alignments the better the translation. However, chen2016guided, liu-EtAl:2016:COLING, alkhouli-EtAl:2016:WMT report mixed results by optimizing their NMT system with respect to word prediction and alignment quality. These findings warrant a more fine-grained analysis of attention. To this end, we include POS tags in our analysis and study the patterns of attention based on POS tags of the target words. We choose POS tags because they exhibit some simple syntactic characteristics. We use the coarse grained universal POS tags BIBREF15 given in Table TABREF25 . To better understand how attention accuracy affects translation quality, we analyse the relationship between attention loss and word prediction loss for individual part-of-speech classes. Figure FIGREF22 shows how attention loss differs when generating different POS tags. One can see that attention loss varies substantially across different POS tags. In particular, we focus on the cases of NOUN and VERB which are the most frequent POS tags in the dataset. As shown, the attention of NOUN is the closest to alignments on average. But the average attention loss for VERB is almost two times larger than the loss for NOUN. Considering this difference and the observations in Section SECREF19 , a natural follow-up would be to focus on getting the attention of verbs to be closer to alignments. However, Figure FIGREF22 shows that the average word prediction loss for verbs is actually smaller compared to the loss for nouns. In other words, although the attention for verbs is substantially more inconsistent with the word alignments than for nouns, the NMT system translates verbs more accurately than nouns on average. To formalize this relationship we compute Spearman's rank correlation between word prediction loss and attention loss, based on the POS tags of the target side, for the input-feeding model, see Figure FIGREF27 . The low correlation for verbs confirms that attention to other parts of source sentence rather than the aligned word is necessary for translating verbs and that attention does not necessarily have to follow alignments. However, the higher correlation for nouns means that consistency of attention with alignments is more desirable. This could, in a way, explain the mixed result reported for training attention using alignments BIBREF9 , BIBREF7 , BIBREF8 . Especially the results by chen2016guided in which large improvements are achieved for the e-commerce domain which contains many OOV product names and placeholders, but no or very weak improvements were achieved over common domains. Attention Concentration In word alignment, most target words are aligned to one source word. The average number of source words aligned to nouns and verbs is 1.1 and 1.2 respectively. To investigate to what extent this also holds for attention we measure the attention concentration by computing the entropy of the attention distribution, see Equation EQREF16 . Figure FIGREF28 shows the average entropy of attention based on POS tags. As shown, nouns have one of the lowest entropies meaning that on average the attention for nouns tends to be concentrated. This also explains the closeness of the attention to alignments for nouns. In addition, the correlation between attention entropy and attention loss in case of nouns is high as shown in Figure FIGREF28 . This means that attention entropy can be used as a measure of closeness of attention to alignment in the case of nouns. The higher attention entropy for verbs, in Figure FIGREF28 , shows that the attention is more distributed compared to nouns. The low correlation between attention entropy and word prediction loss (see Figure FIGREF32 ) shows that attention concentration is not required when translating into verbs. This also confirms that the correct translation of verbs requires the systems to pay attention to different parts of the source sentence. Another interesting observation here is the low correlation for pronouns (PRON) and particles (PRT), see Figure FIGREF32 . As can be seen in Figure FIGREF28 , these tags have more distributed attention comparing to nouns, for example. This could either mean that the attention model does not know where to focus or it deliberately pays attention to multiple, somehow relevant, places to be able to produce a better translation. The latter is supported by the relatively low word prediction losses, shown in the Figure FIGREF22 . Attention Distribution To further understand under which conditions attention is paid to words other than the aligned words, we study the distribution of attention over the source words. First, we measure how much attention is paid to the aligned words for each POS tag, on average. To this end, we compute the percentage of the probability mass that the attention model has assigned to aligned words for each POS tag, see Table TABREF35 . One can notice that less than half of the attention is paid to alignment points for most of the POS tags. To examine how the rest of attention in each case has been distributed over the source sentence we measure the attention distribution over dependency roles in the source side. We first parse the source side of RWTH data using the ParZu parser BIBREF16 . Then we compute how the attention probability mass given to the words other than the alignment points, is distributed over dependency roles. Table TABREF33 gives the most attended roles for each POS tag. Here, we focus on POS tags discussed earlier. One can see that the most attended roles when translating to nouns include adjectives and determiners and in the case of translating to verbs, it includes auxiliary verbs, adverbs (including negation), subjects, and objects. Conclusion In this paper, we have studied attention in neural machine translation and provided an analysis of the relation between attention and word alignment. We have shown that attention agrees with traditional alignment to a certain extent. However, this differs substantially by attention mechanism and the type of the word being generated. We have shown that attention has different patterns based on the POS tag of the target word. The concentrated pattern of attention and the relatively high correlations for nouns show that training the attention with explicit alignment labels is useful for generating nouns. However, this is not the case for verbs, since the large portion of attention being paid to words other than alignment points, is already capturing other relevant information. Training attention with alignments in this case will force the attention model to forget these useful information. This explains the mixed results reported when guiding attention to comply with alignments BIBREF9 , BIBREF7 , BIBREF8 . Acknowledgments This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project numbers 639.022.213 and 612.001.218.
In what cases is attention different from alignment?
044f922604b4b3f42ae381419fd5cd5624fa0637
In Ġwhat Ġcases Ġis Ġattention Ġdifferent Ġfrom Ġalignment ?
For Ġcertain ĠPOS Ġtags , Ġe . g . ĠVER B , ĠPR ON .
<s> In Ġwhat Ġcases Ġis Ġattention Ġdifferent Ġfrom Ġalignment ? </s> Introduction </s> Ne ural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġgained Ġa Ġlot Ġof Ġattention Ġrecently Ġdue Ġto Ġits Ġsubstantial Ġimprovements Ġin Ġmachine Ġtranslation Ġquality Ġachieving Ġstate - of - the - art Ġperformance Ġfor Ġseveral Ġlanguages ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠThe Ġcore Ġarchitecture Ġof Ġneural Ġmachine Ġtranslation Ġmodels Ġis Ġbased Ġon Ġthe Ġgeneral Ġenc oder - dec oder Ġapproach ĠB IB REF 3 Ġ. ĠNeural Ġmachine Ġtranslation Ġis Ġan Ġend - to - end Ġapproach Ġthat Ġlearns Ġto Ġencode Ġsource Ġsentences Ġinto Ġdistributed Ġrepresentations Ġand Ġdecode Ġthese Ġrepresentations Ġinto Ġsentences Ġin Ġthe Ġtarget Ġlanguage . ĠAmong Ġthe Ġdifferent Ġneural ĠMT Ġmodels , Ġattention al ĠN MT ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġhas Ġbecome Ġpopular Ġdue Ġto Ġits Ġcapability Ġto Ġuse Ġthe Ġmost Ġrelevant Ġparts Ġof Ġthe Ġsource Ġsentence Ġat Ġeach Ġtranslation Ġstep . ĠThis Ġcapability Ġalso Ġmakes Ġthe Ġattention al Ġmodel Ġsuperior Ġin Ġtranslating Ġlonger Ġsentences ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. </s> Figure ĠFIG REF 1 Ġshows Ġan Ġexample Ġof Ġhow Ġattention Ġuses Ġthe Ġmost Ġrelevant Ġsource Ġwords Ġto Ġgenerate Ġa Ġtarget Ġword Ġat Ġeach Ġstep Ġof Ġthe Ġtranslation . ĠIn Ġthis Ġpaper Ġwe Ġfocus Ġon Ġstudying Ġthe Ġrelevance Ġof Ġthe Ġattended Ġparts , Ġespecially Ġcases Ġwhere Ġattention Ġis Ġ` s me ared Ġout ' Ġover Ġmultiple Ġsource Ġwords Ġwhere Ġtheir Ġrelevance Ġis Ġnot Ġentirely Ġobvious , Ġsee , Ġe . g ., ĠâĢ ľ would " Ġand ĠâĢ ľ like " Ġin ĠFigure ĠFIG REF 1 Ġ. ĠHere , Ġwe Ġask Ġwhether Ġthese Ġare Ġdue Ġto Ġerrors Ġof Ġthe Ġattention Ġmechanism Ġor Ġare Ġa Ġdesired Ġbehavior Ġof Ġthe Ġmodel . </s> Since Ġthe Ġintroduction Ġof Ġattention Ġmodels Ġin Ġneural Ġmachine Ġtranslation ĠB IB REF 4 Ġvarious Ġmodifications Ġhave Ġbeen Ġproposed ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge Ġthere Ġis Ġno Ġstudy Ġthat Ġprovides Ġan Ġanalysis Ġof Ġwhat Ġkind Ġof Ġphenomena Ġis Ġbeing Ġcaptured Ġby Ġattention . ĠThere Ġare Ġsome Ġworks Ġthat Ġhave Ġlooked Ġto Ġattention Ġas Ġbeing Ġsimilar Ġto Ġtraditional Ġword Ġalignment ĠB IB REF 8 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 9 Ġ. ĠSome Ġof Ġthese Ġapproaches Ġalso Ġexperimented Ġwith Ġtraining Ġthe Ġattention Ġmodel Ġusing Ġtraditional Ġalign ments ĠB IB REF 8 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 9 Ġ. Ġli u - E t Al : 2016 : COL ING Ġhave Ġshown Ġthat Ġattention Ġcould Ġbe Ġseen Ġas Ġa Ġre ordering Ġmodel Ġas Ġwell Ġas Ġan Ġalignment Ġmodel . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġinvestigating Ġthe Ġdifferences Ġbetween Ġattention Ġand Ġalignment Ġand Ġwhat Ġis Ġbeing Ġcaptured Ġby Ġthe Ġattention Ġmechanism Ġin Ġgeneral . ĠThe Ġquestions Ġthat Ġwe Ġare Ġaiming Ġto Ġanswer Ġinclude : ĠIs Ġthe Ġattention Ġmodel Ġonly Ġcapable Ġof Ġmodelling Ġalignment ? ĠAnd Ġhow Ġsimilar Ġis Ġattention Ġto Ġalignment Ġin Ġdifferent Ġsynt actic Ġphenomena ? </s> Our Ġanalysis Ġshows Ġthat Ġattention Ġmodels Ġtraditional Ġalignment Ġin Ġsome Ġcases Ġmore Ġclosely Ġwhile Ġit Ġcaptures Ġinformation Ġbeyond Ġalignment Ġin Ġothers . ĠFor Ġinstance , Ġattention Ġagrees Ġwith Ġtraditional Ġalign ments Ġto Ġa Ġhigh Ġdegree Ġin Ġthe Ġcase Ġof Ġnoun s . ĠHowever , Ġit Ġcaptures Ġother Ġinformation Ġrather Ġthan Ġonly Ġthe Ġtransl ational Ġequivalent Ġin Ġthe Ġcase Ġof Ġverbs . </s> This Ġpaper Ġmakes Ġthe Ġfollowing Ġcontributions : Ġ1 ) ĠWe Ġprovide Ġa Ġdetailed Ġcomparison Ġof Ġattention Ġin ĠN MT Ġand Ġword Ġalignment . Ġ2 ) ĠWe Ġshow Ġthat Ġwhile Ġdifferent Ġattention Ġmechanisms Ġcan Ġlead Ġto Ġdifferent Ġdegrees Ġof Ġcompliance Ġwith Ġrespect Ġto Ġword Ġalign ments , Ġglobal Ġcompliance Ġis Ġnot Ġalways Ġhelpful Ġfor Ġword Ġprediction . Ġ3 ) ĠWe Ġshow Ġthat Ġattention Ġfollows Ġdifferent Ġpatterns Ġdepending Ġon Ġthe Ġtype Ġof Ġthe Ġword Ġbeing Ġgenerated . Ġ4 ) ĠWe Ġdemonstrate Ġthat Ġattention Ġdoes Ġnot Ġalways Ġcomply Ġwith Ġalignment . ĠWe Ġprovide Ġevidence Ġshowing Ġthat Ġthe Ġdifference Ġbetween Ġattention Ġand Ġalignment Ġis Ġdue Ġto Ġattention Ġmodel Ġcapability Ġto Ġattend Ġthe Ġcontext Ġwords Ġinfluencing Ġthe Ġcurrent Ġword Ġtranslation . </s> Related ĠWork </s> li u - E t Al : 2016 : COL ING Ġinvestigate Ġhow Ġtraining Ġthe Ġattention Ġmodel Ġin Ġa Ġsupervised Ġmanner Ġcan Ġbenefit Ġmachine Ġtranslation Ġquality . ĠTo Ġthis Ġend Ġthey Ġuse Ġtraditional Ġalign ments Ġobtained Ġby Ġrunning Ġautomatic Ġalignment Ġtools Ġ( G IZ A ++ ĠB IB REF 10 Ġand Ġfast _ align ĠB IB REF 11 Ġ) Ġon Ġthe Ġtraining Ġdata Ġand Ġfeed Ġit Ġas Ġground Ġtruth Ġto Ġthe Ġattention Ġnetwork . ĠThey Ġreport Ġsome Ġimprovements Ġin Ġtranslation Ġquality Ġarguing Ġthat Ġthe Ġattention Ġmodel Ġhas Ġlearned Ġto Ġbetter Ġalign Ġsource Ġand Ġtarget Ġwords . ĠThe Ġapproach Ġof Ġtraining Ġattention Ġusing Ġtraditional Ġalign ments Ġhas Ġalso Ġbeen Ġproposed Ġby Ġothers ĠB IB REF 9 Ġ, ĠB IB REF 8 Ġ. Ġc hen 2016 guided Ġshow Ġthat Ġguided Ġattention Ġwith Ġtraditional Ġalignment Ġhelps Ġin Ġthe Ġdomain Ġof Ġe - commerce Ġdata Ġwhich Ġincludes Ġlots Ġof Ġout Ġof Ġvocabulary Ġ( OO V ) Ġproduct Ġnames Ġand Ġplace holders , Ġbut Ġnot Ġmuch Ġin Ġthe Ġother Ġdomains . Ġal kh oul i - E t Al : 2016 : W MT Ġhave Ġseparated Ġthe Ġalignment Ġmodel Ġand Ġtranslation Ġmodel , Ġreasoning Ġthat Ġthis Ġavoids Ġpropagation Ġof Ġerrors Ġfrom Ġone Ġmodel Ġto Ġthe Ġother Ġas Ġwell Ġas Ġproviding Ġmore Ġflexibility Ġin Ġthe Ġmodel Ġtypes Ġand Ġtraining Ġof Ġthe Ġmodels . ĠThey Ġuse Ġa Ġfeed - forward Ġneural Ġnetwork Ġas Ġtheir Ġalignment Ġmodel Ġthat Ġlearns Ġto Ġmodel Ġjumps Ġin Ġthe Ġsource Ġside Ġusing ĠH MM / IB M Ġalign ments Ġobtained Ġby Ġusing ĠG IZ A ++ . </s> shi - pad hi - k night : 2016 : EM N LP 2016 Ġshow Ġthat Ġvarious Ġkinds Ġof Ġsynt actic Ġinformation Ġare Ġbeing Ġlearned Ġand Ġencoded Ġin Ġthe Ġoutput Ġhidden Ġstates Ġof Ġthe Ġenc oder . ĠThe Ġneural Ġsystem Ġfor Ġtheir Ġexperimental Ġanalysis Ġis Ġnot Ġan Ġattention al Ġmodel Ġand Ġthey Ġargue Ġthat Ġattention Ġdoes Ġnot Ġhave Ġany Ġimpact Ġfor Ġlearning Ġsynt actic Ġinformation . ĠHowever , Ġperforming Ġthe Ġsame Ġanalysis Ġfor Ġmorph ological Ġinformation , Ġbel ink ov 2017 ne ural Ġshow Ġthat Ġattention Ġhas Ġalso Ġsome Ġeffect Ġon Ġthe Ġinformation Ġthat Ġthe Ġenc oder Ġof Ġneural Ġmachine Ġtranslation Ġsystem Ġenc odes Ġin Ġits Ġoutput Ġhidden Ġstates . ĠAs Ġpart Ġof Ġtheir Ġanalysis Ġthey Ġshow Ġthat Ġa Ġneural Ġmachine Ġtranslation Ġsystem Ġthat Ġhas Ġan Ġattention Ġmodel Ġcan Ġlearn Ġthe ĠPOS Ġtags Ġof Ġthe Ġsource Ġside Ġmore Ġefficiently Ġthan Ġa Ġsystem Ġwithout Ġattention . </s> Recently , Ġk oe hn 2017 six Ġcarried Ġout Ġa Ġbrief Ġanalysis Ġof Ġhow Ġmuch Ġattention Ġand Ġalignment Ġmatch Ġin Ġdifferent Ġlanguages Ġby Ġmeasuring Ġthe Ġprobability Ġmass Ġthat Ġattention Ġgives Ġto Ġalign ments Ġobtained Ġfrom Ġan Ġautomatic Ġalignment Ġtool . ĠThey Ġalso Ġreport Ġdifferences Ġbased Ġon Ġthe Ġmost Ġattended Ġwords . </s> The Ġmixed Ġresults Ġreported Ġby Ġc hen 2016 guided , Ġal kh oul i - E t Al : 2016 : W MT , Ġli u - E t Al : 2016 : COL ING Ġon Ġoptimizing Ġattention Ġwith Ġrespect Ġto Ġalign ments Ġmotiv ates Ġa Ġmore Ġthorough Ġanalysis Ġof Ġattention Ġmodels Ġin ĠN MT . </s> Att ention ĠModels </s> This Ġsection Ġprovides Ġa Ġshort Ġbackground Ġon Ġattention Ġand Ġdiscusses Ġtwo Ġmost Ġpopular Ġattention Ġmodels Ġwhich Ġare Ġalso Ġused Ġin Ġthis Ġpaper . ĠThe Ġfirst Ġmodel Ġis Ġa Ġnon - rec urrent Ġattention Ġmodel Ġwhich Ġis Ġequivalent Ġto Ġthe ĠâĢ ľ global Ġattention " Ġmethod Ġproposed Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠThe Ġsecond Ġattention Ġmodel Ġthat Ġwe Ġuse Ġin Ġour Ġinvestigation Ġis Ġan Ġinput - feeding Ġmodel Ġsimilar Ġto Ġthe Ġattention Ġmodel Ġfirst Ġproposed Ġby Ġb ah dan au - E t Al : 2015 : IC LR Ġand Ġturned Ġto Ġa Ġmore Ġgeneral Ġone Ġand Ġcalled Ġinput - feeding Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠBelow Ġwe Ġdescribe Ġthe Ġdetails Ġof Ġboth Ġmodels . </s> Both Ġnon - rec urrent Ġand Ġinput - feeding Ġmodels Ġcompute Ġa Ġcontext Ġvector ĠIN LINE FORM 0 Ġat Ġeach Ġtime Ġstep . ĠSub sequently , Ġthey Ġconc aten ate Ġthe Ġcontext Ġvector Ġto Ġthe Ġhidden Ġstate Ġof Ġdec oder Ġand Ġpass Ġit Ġthrough Ġa Ġnon - linear ity Ġbefore Ġit Ġis Ġfed Ġinto Ġthe Ġsoft max Ġoutput Ġlayer Ġof Ġthe Ġtranslation Ġnetwork . ĠDIS PLAY FORM 0 </s> The Ġdifference Ġof Ġthe Ġtwo Ġmodels Ġlays Ġin Ġthe Ġway Ġthey Ġcompute Ġthe Ġcontext Ġvector . ĠIn Ġthe Ġnon - rec urrent Ġmodel , Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġis Ġcompared Ġto Ġeach Ġhidden Ġstate Ġof Ġthe Ġenc oder . ĠOften , Ġthis Ġcomparison Ġis Ġrealized Ġas Ġthe Ġdot Ġproduct Ġof Ġvectors . ĠThen Ġthe Ġcomparison Ġresult Ġis Ġfed Ġto Ġa Ġsoft max Ġlayer Ġto Ġcompute Ġthe Ġattention Ġweight . ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstate Ġof Ġthe Ġdec oder Ġat Ġtime ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġis ĠIN LINE FORM 3 Ġth Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġlength Ġof Ġthe Ġsource Ġsentence . ĠThen Ġthe Ġcomputed Ġalignment Ġweights Ġare Ġused Ġto Ġcompute Ġa Ġweighted Ġsum Ġover Ġthe Ġenc oder Ġhidden Ġstates Ġwhich Ġresults Ġin Ġthe Ġcontext Ġvector Ġmentioned Ġabove : ĠDIS PLAY FORM 0 </s> The Ġinput - feeding Ġmodel Ġchanges Ġthe Ġcontext Ġvector Ġcomputation Ġin Ġa Ġway Ġthat Ġat Ġeach Ġstep ĠIN LINE FORM 0 Ġthe Ġcontext Ġvector Ġis Ġaware Ġof Ġthe Ġpreviously Ġcomputed Ġcontext ĠIN LINE FORM 1 Ġ. ĠTo Ġthis Ġend , Ġthe Ġinput - feeding Ġmodel Ġfeeds Ġback Ġits Ġown ĠIN LINE FORM 2 Ġto Ġthe Ġnetwork Ġand Ġuses Ġthe Ġresulting Ġhidden Ġstate Ġinstead Ġof Ġthe Ġcontext - independent ĠIN LINE FORM 3 Ġ, Ġto Ġcompare Ġto Ġthe Ġhidden Ġstates Ġof Ġthe Ġenc oder . ĠThis Ġis Ġdefined Ġin Ġthe Ġfollowing Ġequations : ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> Here , ĠIN LINE FORM 0 Ġis Ġthe Ġfunction Ġthat Ġthe Ġstacked ĠL ST M Ġapplies Ġto Ġthe Ġinput , ĠIN LINE FORM 1 Ġis Ġthe Ġlast Ġgenerated Ġtarget Ġword , Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġoutput Ġof Ġprevious Ġtime Ġstep Ġof Ġthe Ġinput - feeding Ġnetwork Ġitself , Ġmeaning Ġthe Ġoutput Ġof ĠEqu ation ĠEQ REF 2 Ġin Ġthe Ġcase Ġthat Ġcontext Ġvector Ġhas Ġbeen Ġcomputed Ġusing ĠIN LINE FORM 3 Ġfrom ĠEqu ation ĠEQ REF 7 Ġ. </s> Comp aring ĠAttention Ġwith ĠAl ignment </s> As Ġmentioned Ġabove , Ġit Ġis Ġa Ġcommonly Ġheld Ġassumption Ġthat Ġattention Ġcorresponds Ġto Ġword Ġalign ments . ĠTo Ġverify Ġthis , Ġwe Ġinvestigate Ġwhether Ġhigher Ġconsistency Ġbetween Ġattention Ġand Ġalignment Ġleads Ġto Ġbetter Ġtranslations . </s> Me asuring ĠAttention - Al ignment ĠAccuracy </s> In Ġorder Ġto Ġcompare Ġattent ions Ġof Ġmultiple Ġsystems Ġas Ġwell Ġas Ġto Ġmeasure Ġthe Ġdifference Ġbetween Ġattention Ġand Ġword Ġalignment , Ġwe Ġconvert Ġthe Ġhard Ġword Ġalign ments Ġinto Ġsoft Ġones Ġand Ġuse Ġcross Ġentropy Ġbetween Ġattention Ġand Ġsoft Ġalignment Ġas Ġa Ġloss Ġfunction . ĠFor Ġthis Ġpurpose , Ġwe Ġuse Ġmanual Ġalign ments Ġprovided Ġby ĠRW TH ĠGerman - English Ġdataset Ġas Ġthe Ġhard Ġalign ments . ĠThe Ġstatistics Ġof Ġthe Ġdata Ġare Ġgiven Ġin ĠTable ĠT AB REF 8 Ġ. ĠWe Ġconvert Ġthe Ġhard Ġalign ments Ġto Ġsoft Ġalign ments Ġusing ĠEqu ation ĠEQ REF 10 Ġ. ĠFor Ġun aligned Ġwords , Ġwe Ġfirst Ġassume Ġthat Ġthey Ġhave Ġbeen Ġaligned Ġto Ġall Ġthe Ġwords Ġin Ġthe Ġsource Ġside Ġand Ġthen Ġdo Ġthe Ġconversion . ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġset Ġof Ġsource Ġwords Ġaligned Ġto Ġtarget Ġword ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġsource Ġwords Ġin Ġthe Ġset . </s> After Ġconversion Ġof Ġthe Ġhard Ġalign ments Ġto Ġsoft Ġones , Ġwe Ġcompute Ġthe Ġattention Ġloss Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġsource Ġsentence Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġweight Ġof Ġthe Ġalignment Ġlink Ġbetween Ġsource Ġword ĠIN LINE FORM 2 Ġand Ġthe Ġtarget Ġword Ġ( see ĠEqu ation ĠEQ REF 10 Ġ). ĠIN LINE FORM 3 Ġis Ġthe Ġattention Ġweight ĠIN LINE FORM 4 Ġ( see ĠEqu ation ĠEQ REF 4 Ġ) Ġof Ġthe Ġsource Ġword ĠIN LINE FORM 5 Ġ, Ġwhen Ġgenerating Ġthe Ġtarget Ġword ĠIN LINE FORM 6 Ġ. </s> In Ġour Ġanalysis , Ġwe Ġalso Ġlook Ġinto Ġthe Ġrelation Ġbetween Ġtranslation Ġquality Ġand Ġthe Ġquality Ġof Ġthe Ġattention Ġwith Ġrespect Ġto Ġthe Ġalign ments . ĠFor Ġmeasuring Ġthe Ġquality Ġof Ġattention , Ġwe Ġuse Ġthe Ġattention Ġloss Ġdefined Ġin ĠEqu ation ĠEQ REF 11 Ġ. ĠAs Ġa Ġmeasure Ġof Ġtranslation Ġquality , Ġwe Ġchoose Ġthe Ġloss Ġbetween Ġthe Ġoutput Ġof Ġour ĠN MT Ġsystem Ġand Ġthe Ġreference Ġtranslation Ġat Ġeach Ġtranslation Ġstep , Ġwhich Ġwe Ġcall Ġword Ġprediction Ġloss . ĠThe Ġword Ġprediction Ġloss Ġfor Ġword ĠIN LINE FORM 0 Ġis Ġlog ar ith m Ġof Ġthe Ġprobability Ġgiven Ġin ĠEqu ation ĠEQ REF 12 Ġ. ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġthe Ġsource Ġsentence , ĠIN LINE FORM 1 Ġis Ġtarget Ġword Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġhistory Ġgiven Ġby Ġthe Ġreference Ġtranslation Ġand ĠIN LINE FORM 4 Ġis Ġgiven Ġby ĠEqu ation ĠEQ REF 2 Ġfor Ġeither Ġnon - rec urrent Ġor Ġinput - feeding Ġattention Ġmodels . </s> Spe ar man 's Ġrank Ġcorrelation Ġis Ġused Ġto Ġcompute Ġthe Ġcorrelation Ġbetween Ġattention Ġloss Ġand Ġword Ġprediction Ġloss : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġranks Ġof Ġthe Ġattention Ġlosses Ġand Ġword Ġprediction Ġlosses , Ġrespectively , ĠIN LINE FORM 2 Ġis Ġthe Ġcovari ance Ġbetween Ġtwo Ġinput Ġvariables , Ġand ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġthe Ġstandard Ġdeviations Ġof ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ. </s> If Ġthere Ġis Ġa Ġclose Ġrelationship Ġbetween Ġword Ġprediction Ġquality Ġand Ġconsistency Ġof Ġattention Ġversus Ġalignment , Ġthen Ġthere Ġshould Ġbe Ġhigh Ġcorrelation Ġbetween Ġword Ġprediction Ġloss Ġand Ġattention Ġloss . ĠFigure ĠFIG REF 13 Ġshows Ġan Ġexample Ġwith Ġdifferent Ġlevels Ġof Ġconsistency Ġbetween Ġattention Ġand Ġword Ġalign ments . ĠFor Ġthe Ġtarget Ġwords ĠâĢ ľ will " Ġand ĠâĢ ľ come " Ġthe Ġattention Ġis Ġnot Ġfocused Ġon Ġthe Ġmanually Ġaligned Ġword Ġbut Ġdistributed Ġbetween Ġthe Ġaligned Ġword Ġand Ġother Ġwords . ĠThe Ġfocus Ġof Ġthis Ġpaper Ġis Ġexamining Ġcases Ġwhere Ġattention Ġdoes Ġnot Ġfollow Ġalignment , Ġanswering Ġthe Ġquestions Ġwhether Ġthose Ġcases Ġrepresent Ġerrors Ġor Ġdesirable Ġbehavior Ġof Ġthe Ġattention Ġmodel . </s> Me asuring ĠAttention ĠConcent ration </s> As Ġanother Ġinformative Ġvariable Ġin Ġour Ġanalysis , Ġwe Ġlook Ġinto Ġthe Ġattention Ġconcentration . ĠWhile Ġmost Ġword Ġalign ments Ġonly Ġinvolve Ġone Ġor Ġa Ġfew Ġwords , Ġattention Ġcan Ġbe Ġdistributed Ġmore Ġfreely . ĠWe Ġmeasure Ġthe Ġconcentration Ġof Ġattention Ġby Ġcomputing Ġthe Ġentropy Ġof Ġthe Ġattention Ġdistribution : ĠDIS PLAY FORM 0 </s> E mp ir ical ĠAnalysis Ġof ĠAttention ĠBeh aviour </s> We Ġconduct Ġour Ġanalysis Ġusing Ġthe Ġtwo Ġdifferent Ġattention Ġmodels Ġdescribed Ġin ĠSection ĠSEC REF 3 Ġ. ĠOur Ġfirst Ġattention Ġmodel Ġis Ġthe Ġglobal Ġmodel Ġwithout Ġinput - feeding Ġas Ġintroduced Ġby ĠD BL P j ournals cor r Lu ong PM 15 . ĠThe Ġsecond Ġmodel Ġis Ġthe Ġinput - feeding Ġmodel ĠB IB REF 5 Ġ, Ġwhich Ġuses Ġrecurrent Ġattention . ĠOur ĠN MT Ġsystem Ġis Ġa Ġun id irection al Ġenc oder - dec oder Ġsystem Ġas Ġdescribed Ġin ĠB IB REF 5 Ġ, Ġusing Ġ4 Ġrecurrent Ġlayers . </s> We Ġtrained Ġthe Ġsystems Ġwith Ġdimension Ġsize Ġof Ġ1 , 000 Ġand Ġbatch Ġsize Ġof Ġ80 Ġfor Ġ20 Ġepoch s . ĠThe Ġvocabulary Ġfor Ġboth Ġsource Ġand Ġtarget Ġside Ġis Ġset Ġto Ġbe Ġthe Ġ30 K Ġmost Ġcommon Ġwords . ĠThe Ġlearning Ġrate Ġis Ġset Ġto Ġbe Ġ1 Ġand Ġa Ġmaximum Ġgradient Ġnorm Ġof Ġ5 Ġhas Ġbeen Ġused . ĠWe Ġalso Ġuse Ġa Ġdrop out Ġrate Ġof Ġ0 . 3 Ġto Ġavoid Ġover fitting . </s> Imp act Ġof ĠAttention ĠMechan ism </s> We Ġtrain Ġboth Ġof Ġthe Ġsystems Ġon Ġthe ĠW MT 15 ĠGerman - to - English Ġtraining Ġdata , Ġsee ĠTable ĠT AB REF 18 Ġfor Ġsome Ġstatistics . ĠTable ĠT AB REF 17 Ġshows Ġthe ĠB LE U Ġscores ĠB IB REF 12 Ġfor Ġboth Ġsystems Ġon Ġdifferent Ġtest Ġsets . </s> Since Ġwe Ġuse ĠPOS Ġtags Ġand Ġdependency Ġroles Ġin Ġour Ġanalysis , Ġboth Ġof Ġwhich Ġare Ġbased Ġon Ġwords , Ġwe Ġchose Ġnot Ġto Ġuse ĠB PE ĠB IB REF 13 Ġwhich Ġoperates Ġat Ġthe Ġsub - word Ġlevel . </s> We Ġreport Ġalignment Ġerror Ġrate Ġ( A ER ) ĠB IB REF 14 Ġ, Ġwhich Ġis Ġcommonly Ġused Ġto Ġmeasure Ġalignment Ġquality , Ġin ĠTable ĠT AB REF 20 Ġto Ġshow Ġthe Ġdifference Ġbetween Ġattent ions Ġand Ġhuman Ġalign ments Ġprovided Ġby ĠRW TH ĠGerman - English Ġdataset . ĠTo Ġcompute ĠA ER Ġover Ġattent ions , Ġwe Ġfollow ĠD BL P j ournals cor r Lu ong PM 15 Ġto Ġproduce Ġhard Ġalign ments Ġfrom Ġattent ions Ġby Ġchoosing Ġthe Ġmost Ġattended Ġsource Ġword Ġfor Ġeach Ġtarget Ġword . ĠWe Ġalso Ġuse ĠG IZ A ++ ĠB IB REF 10 Ġto Ġproduce Ġautomatic Ġalign ments Ġover Ġthe Ġdata Ġset Ġto Ġallow Ġfor Ġa Ġcomparison Ġbetween Ġautomatically Ġgenerated Ġalign ments Ġand Ġthe Ġattent ions Ġgenerated Ġby Ġour Ġsystems . ĠG IZ A ++ Ġis Ġrun Ġin Ġboth Ġdirections Ġand Ġalign ments Ġare Ġsymm et ri zed Ġusing Ġthe Ġgrow - di ag - final - and Ġrefined Ġalignment Ġhe uristic . </s> As Ġshown Ġin ĠTable ĠT AB REF 20 Ġ, Ġthe Ġinput - feeding Ġsystem Ġnot Ġonly Ġachieves Ġa Ġhigher ĠB LE U Ġscore , Ġbut Ġalso Ġuses Ġattent ions Ġthat Ġare Ġcloser Ġto Ġthe Ġhuman Ġalign ments . </s> Table ĠT AB REF 21 Ġcompares Ġinput - feeding Ġand Ġnon - rec urrent Ġattention Ġin Ġterms Ġof Ġattention Ġloss Ġcomputed Ġusing ĠEqu ation ĠEQ REF 11 Ġ. ĠHere Ġthe Ġlosses Ġbetween Ġthe Ġattention Ġproduced Ġby Ġeach Ġsystem Ġand Ġthe Ġhuman Ġalign ments Ġis Ġreported . ĠAs Ġexpected , Ġthe Ġdifference Ġin Ġattention Ġlosses Ġare Ġin Ġline Ġwith ĠA ER . </s> The Ġdifference Ġbetween Ġthese Ġcomparisons Ġis Ġthat ĠA ER Ġonly Ġtakes Ġthe Ġmost Ġattended Ġword Ġinto Ġaccount Ġwhile Ġattention Ġloss Ġconsiders Ġthe Ġentire Ġattention Ġdistribution . </s> Al ignment ĠQuality ĠImpact Ġon ĠTranslation </s> Based Ġon Ġthe Ġresults Ġin ĠSection ĠSEC REF 19 Ġ, Ġone Ġmight Ġbe Ġinclined Ġto Ġconclude Ġthat Ġthe Ġcloser Ġthe Ġattention Ġis Ġto Ġthe Ġword Ġalign ments Ġthe Ġbetter Ġthe Ġtranslation . ĠHowever , Ġc hen 2016 guided , Ġli u - E t Al : 2016 : COL ING , Ġal kh oul i - E t Al : 2016 : W MT Ġreport Ġmixed Ġresults Ġby Ġoptimizing Ġtheir ĠN MT Ġsystem Ġwith Ġrespect Ġto Ġword Ġprediction Ġand Ġalignment Ġquality . ĠThese Ġfindings Ġwarrant Ġa Ġmore Ġfine - gr ained Ġanalysis Ġof Ġattention . ĠTo Ġthis Ġend , Ġwe Ġinclude ĠPOS Ġtags Ġin Ġour Ġanalysis Ġand Ġstudy Ġthe Ġpatterns Ġof Ġattention Ġbased Ġon ĠPOS Ġtags Ġof Ġthe Ġtarget Ġwords . ĠWe Ġchoose ĠPOS Ġtags Ġbecause Ġthey Ġexhibit Ġsome Ġsimple Ġsynt actic Ġcharacteristics . ĠWe Ġuse Ġthe Ġcoarse Ġgr ained Ġuniversal ĠPOS Ġtags ĠB IB REF 15 Ġgiven Ġin ĠTable ĠT AB REF 25 Ġ. </s> To Ġbetter Ġunderstand Ġhow Ġattention Ġaccuracy Ġaffects Ġtranslation Ġquality , Ġwe Ġanalyse Ġthe Ġrelationship Ġbetween Ġattention Ġloss Ġand Ġword Ġprediction Ġloss Ġfor Ġindividual Ġpart - of - speech Ġclasses . ĠFigure ĠFIG REF 22 Ġshows Ġhow Ġattention Ġloss Ġdiffers Ġwhen Ġgenerating Ġdifferent ĠPOS Ġtags . ĠOne Ġcan Ġsee Ġthat Ġattention Ġloss Ġvaries Ġsubstantially Ġacross Ġdifferent ĠPOS Ġtags . ĠIn Ġparticular , Ġwe Ġfocus Ġon Ġthe Ġcases Ġof ĠN OUN Ġand ĠVER B Ġwhich Ġare Ġthe Ġmost Ġfrequent ĠPOS Ġtags Ġin Ġthe Ġdataset . ĠAs Ġshown , Ġthe Ġattention Ġof ĠN OUN Ġis Ġthe Ġclosest Ġto Ġalign ments Ġon Ġaverage . ĠBut Ġthe Ġaverage Ġattention Ġloss Ġfor ĠVER B Ġis Ġalmost Ġtwo Ġtimes Ġlarger Ġthan Ġthe Ġloss Ġfor ĠN OUN . </s> Considering Ġthis Ġdifference Ġand Ġthe Ġobservations Ġin ĠSection ĠSEC REF 19 Ġ, Ġa Ġnatural Ġfollow - up Ġwould Ġbe Ġto Ġfocus Ġon Ġgetting Ġthe Ġattention Ġof Ġverbs Ġto Ġbe Ġcloser Ġto Ġalign ments . ĠHowever , ĠFigure ĠFIG REF 22 Ġshows Ġthat Ġthe Ġaverage Ġword Ġprediction Ġloss Ġfor Ġverbs Ġis Ġactually Ġsmaller Ġcompared Ġto Ġthe Ġloss Ġfor Ġnoun s . ĠIn Ġother Ġwords , Ġalthough Ġthe Ġattention Ġfor Ġverbs Ġis Ġsubstantially Ġmore Ġinconsistent Ġwith Ġthe Ġword Ġalign ments Ġthan Ġfor Ġnoun s , Ġthe ĠN MT Ġsystem Ġtranslates Ġverbs Ġmore Ġaccurately Ġthan Ġnoun s Ġon Ġaverage . </s> To Ġformal ize Ġthis Ġrelationship Ġwe Ġcompute ĠSpear man 's Ġrank Ġcorrelation Ġbetween Ġword Ġprediction Ġloss Ġand Ġattention Ġloss , Ġbased Ġon Ġthe ĠPOS Ġtags Ġof Ġthe Ġtarget Ġside , Ġfor Ġthe Ġinput - feeding Ġmodel , Ġsee ĠFigure ĠFIG REF 27 Ġ. </s> The Ġlow Ġcorrelation Ġfor Ġverbs Ġconfirms Ġthat Ġattention Ġto Ġother Ġparts Ġof Ġsource Ġsentence Ġrather Ġthan Ġthe Ġaligned Ġword Ġis Ġnecessary Ġfor Ġtranslating Ġverbs Ġand Ġthat Ġattention Ġdoes Ġnot Ġnecessarily Ġhave Ġto Ġfollow Ġalign ments . ĠHowever , Ġthe Ġhigher Ġcorrelation Ġfor Ġnoun s Ġmeans Ġthat Ġconsistency Ġof Ġattention Ġwith Ġalign ments Ġis Ġmore Ġdesirable . ĠThis Ġcould , Ġin Ġa Ġway , Ġexplain Ġthe Ġmixed Ġresult Ġreported Ġfor Ġtraining Ġattention Ġusing Ġalign ments ĠB IB REF 9 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠEspecially Ġthe Ġresults Ġby Ġc hen 2016 guided Ġin Ġwhich Ġlarge Ġimprovements Ġare Ġachieved Ġfor Ġthe Ġe - commerce Ġdomain Ġwhich Ġcontains Ġmany ĠO OV Ġproduct Ġnames Ġand Ġplace holders , Ġbut Ġno Ġor Ġvery Ġweak Ġimprovements Ġwere Ġachieved Ġover Ġcommon Ġdomains . </s> Att ention ĠConcent ration </s> In Ġword Ġalignment , Ġmost Ġtarget Ġwords Ġare Ġaligned Ġto Ġone Ġsource Ġword . ĠThe Ġaverage Ġnumber Ġof Ġsource Ġwords Ġaligned Ġto Ġnoun s Ġand Ġverbs Ġis Ġ1 . 1 Ġand Ġ1 . 2 Ġrespectively . ĠTo Ġinvestigate Ġto Ġwhat Ġextent Ġthis Ġalso Ġholds Ġfor Ġattention Ġwe Ġmeasure Ġthe Ġattention Ġconcentration Ġby Ġcomputing Ġthe Ġentropy Ġof Ġthe Ġattention Ġdistribution , Ġsee ĠEqu ation ĠEQ REF 16 Ġ. </s> Figure ĠFIG REF 28 Ġshows Ġthe Ġaverage Ġentropy Ġof Ġattention Ġbased Ġon ĠPOS Ġtags . ĠAs Ġshown , Ġnoun s Ġhave Ġone Ġof Ġthe Ġlowest Ġent rop ies Ġmeaning Ġthat Ġon Ġaverage Ġthe Ġattention Ġfor Ġnoun s Ġtends Ġto Ġbe Ġconcentrated . ĠThis Ġalso Ġexplains Ġthe Ġclos eness Ġof Ġthe Ġattention Ġto Ġalign ments Ġfor Ġnoun s . ĠIn Ġaddition , Ġthe Ġcorrelation Ġbetween Ġattention Ġentropy Ġand Ġattention Ġloss Ġin Ġcase Ġof Ġnoun s Ġis Ġhigh Ġas Ġshown Ġin ĠFigure ĠFIG REF 28 Ġ. ĠThis Ġmeans Ġthat Ġattention Ġentropy Ġcan Ġbe Ġused Ġas Ġa Ġmeasure Ġof Ġclos eness Ġof Ġattention Ġto Ġalignment Ġin Ġthe Ġcase Ġof Ġnoun s . </s> The Ġhigher Ġattention Ġentropy Ġfor Ġverbs , Ġin ĠFigure ĠFIG REF 28 Ġ, Ġshows Ġthat Ġthe Ġattention Ġis Ġmore Ġdistributed Ġcompared Ġto Ġnoun s . ĠThe Ġlow Ġcorrelation Ġbetween Ġattention Ġentropy Ġand Ġword Ġprediction Ġloss Ġ( see ĠFigure ĠFIG REF 32 Ġ) Ġshows Ġthat Ġattention Ġconcentration Ġis Ġnot Ġrequired Ġwhen Ġtranslating Ġinto Ġverbs . ĠThis Ġalso Ġconfirms Ġthat Ġthe Ġcorrect Ġtranslation Ġof Ġverbs Ġrequires Ġthe Ġsystems Ġto Ġpay Ġattention Ġto Ġdifferent Ġparts Ġof Ġthe Ġsource Ġsentence . </s> Another Ġinteresting Ġobservation Ġhere Ġis Ġthe Ġlow Ġcorrelation Ġfor Ġpronouns Ġ( PR ON ) Ġand Ġparticles Ġ( PR T ), Ġsee ĠFigure ĠFIG REF 32 Ġ. ĠAs Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 28 Ġ, Ġthese Ġtags Ġhave Ġmore Ġdistributed Ġattention Ġcomparing Ġto Ġnoun s , Ġfor Ġexample . ĠThis Ġcould Ġeither Ġmean Ġthat Ġthe Ġattention Ġmodel Ġdoes Ġnot Ġknow Ġwhere Ġto Ġfocus Ġor Ġit Ġdeliberately Ġpays Ġattention Ġto Ġmultiple , Ġsomehow Ġrelevant , Ġplaces Ġto Ġbe Ġable Ġto Ġproduce Ġa Ġbetter Ġtranslation . ĠThe Ġlatter Ġis Ġsupported Ġby Ġthe Ġrelatively Ġlow Ġword Ġprediction Ġlosses , Ġshown Ġin Ġthe ĠFigure ĠFIG REF 22 Ġ. </s> Att ention ĠDistribution </s> To Ġfurther Ġunderstand Ġunder Ġwhich Ġconditions Ġattention Ġis Ġpaid Ġto Ġwords Ġother Ġthan Ġthe Ġaligned Ġwords , Ġwe Ġstudy Ġthe Ġdistribution Ġof Ġattention Ġover Ġthe Ġsource Ġwords . ĠFirst , Ġwe Ġmeasure Ġhow Ġmuch Ġattention Ġis Ġpaid Ġto Ġthe Ġaligned Ġwords Ġfor Ġeach ĠPOS Ġtag , Ġon Ġaverage . ĠTo Ġthis Ġend , Ġwe Ġcompute Ġthe Ġpercentage Ġof Ġthe Ġprobability Ġmass Ġthat Ġthe Ġattention Ġmodel Ġhas Ġassigned Ġto Ġaligned Ġwords Ġfor Ġeach ĠPOS Ġtag , Ġsee ĠTable ĠT AB REF 35 Ġ. </s> One Ġcan Ġnotice Ġthat Ġless Ġthan Ġhalf Ġof Ġthe Ġattention Ġis Ġpaid Ġto Ġalignment Ġpoints Ġfor Ġmost Ġof Ġthe ĠPOS Ġtags . ĠTo Ġexamine Ġhow Ġthe Ġrest Ġof Ġattention Ġin Ġeach Ġcase Ġhas Ġbeen Ġdistributed Ġover Ġthe Ġsource Ġsentence Ġwe Ġmeasure Ġthe Ġattention Ġdistribution Ġover Ġdependency Ġroles Ġin Ġthe Ġsource Ġside . ĠWe Ġfirst Ġparse Ġthe Ġsource Ġside Ġof ĠRW TH Ġdata Ġusing Ġthe ĠPar Z u Ġparser ĠB IB REF 16 Ġ. ĠThen Ġwe Ġcompute Ġhow Ġthe Ġattention Ġprobability Ġmass Ġgiven Ġto Ġthe Ġwords Ġother Ġthan Ġthe Ġalignment Ġpoints , Ġis Ġdistributed Ġover Ġdependency Ġroles . ĠTable ĠT AB REF 33 Ġgives Ġthe Ġmost Ġattended Ġroles Ġfor Ġeach ĠPOS Ġtag . ĠHere , Ġwe Ġfocus Ġon ĠPOS Ġtags Ġdiscussed Ġearlier . ĠOne Ġcan Ġsee Ġthat Ġthe Ġmost Ġattended Ġroles Ġwhen Ġtranslating Ġto Ġnoun s Ġinclude Ġadject ives Ġand Ġdetermin ers Ġand Ġin Ġthe Ġcase Ġof Ġtranslating Ġto Ġverbs , Ġit Ġincludes Ġauxiliary Ġverbs , Ġad verbs Ġ( including Ġneg ation ), Ġsubjects , Ġand Ġobjects . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġhave Ġstudied Ġattention Ġin Ġneural Ġmachine Ġtranslation Ġand Ġprovided Ġan Ġanalysis Ġof Ġthe Ġrelation Ġbetween Ġattention Ġand Ġword Ġalignment . ĠWe Ġhave Ġshown Ġthat Ġattention Ġagrees Ġwith Ġtraditional Ġalignment Ġto Ġa Ġcertain Ġextent . ĠHowever , Ġthis Ġdiffers Ġsubstantially Ġby Ġattention Ġmechanism Ġand Ġthe Ġtype Ġof Ġthe Ġword Ġbeing Ġgenerated . ĠWe Ġhave Ġshown Ġthat Ġattention Ġhas Ġdifferent Ġpatterns Ġbased Ġon Ġthe ĠPOS Ġtag Ġof Ġthe Ġtarget Ġword . ĠThe Ġconcentrated Ġpattern Ġof Ġattention Ġand Ġthe Ġrelatively Ġhigh Ġcorrelations Ġfor Ġnoun s Ġshow Ġthat Ġtraining Ġthe Ġattention Ġwith Ġexplicit Ġalignment Ġlabels Ġis Ġuseful Ġfor Ġgenerating Ġnoun s . ĠHowever , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġverbs , Ġsince Ġthe Ġlarge Ġportion Ġof Ġattention Ġbeing Ġpaid Ġto Ġwords Ġother Ġthan Ġalignment Ġpoints , Ġis Ġalready Ġcapturing Ġother Ġrelevant Ġinformation . ĠTraining Ġattention Ġwith Ġalign ments Ġin Ġthis Ġcase Ġwill Ġforce Ġthe Ġattention Ġmodel Ġto Ġforget Ġthese Ġuseful Ġinformation . ĠThis Ġexplains Ġthe Ġmixed Ġresults Ġreported Ġwhen Ġguiding Ġattention Ġto Ġcomply Ġwith Ġalign ments ĠB IB REF 9 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. </s> Acknowled gments </s> This Ġresearch Ġwas Ġfunded Ġin Ġpart Ġby Ġthe ĠNetherlands ĠOrganization Ġfor ĠScientific ĠResearch Ġ( NW O ) Ġunder Ġproject Ġnumbers Ġ6 39 . 022 . 213 Ġand Ġ6 12 . 001 . 218 .