On the datasets DE-EN, JA-EN, RO-EN, and EN-DE, the baseline achieves 29.79, 21.57, 32.70, and 26.02  BLEU score, respectively. The 1.5-entmax achieves  29.83, 22.13, 33.10, and 25.89 BLEU score, which is a difference of +0.04, +0.56, +0.40, and -0.13 BLEU score versus the baseline. The α-entmax achieves 29.90, 21.74, 32.89, and 26.93 BLEU score, which is a difference of +0.11, +0.17, +0.19, +0.91 BLEU score versus the baseline.
ABSTRACTIVE
FLOAT SELECTED: Table 1: Machine translation tokenized BLEU test results on IWSLT 2017 DE EN, KFTT JA EN, WMT 2016 RO EN and WMT 2014 EN DE, respectively. We report test set tokenized BLEU BIBREF32 results in Table TABREF27. We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis. In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads.
1909.00015
Introduction </s> The ĠTrans former Ġarchitecture ĠB IB REF 0 Ġfor Ġdeep Ġneural Ġnetworks Ġhas Ġquickly Ġrisen Ġto Ġprominence Ġin ĠN LP Ġthrough Ġits Ġefficiency Ġand Ġperformance , Ġleading Ġto Ġimprovements Ġin Ġthe Ġstate Ġof Ġthe Ġart Ġof ĠNeural ĠMachine ĠTranslation ĠB IB REF 1 , ĠB IB REF 2 , Ġas Ġwell Ġas Ġinspiring Ġother Ġpowerful Ġgeneral - purpose Ġmodels Ġlike ĠB ERT ĠB IB REF 3 Ġand ĠG PT - 2 ĠB IB REF 4 . ĠAt Ġthe Ġheart Ġof Ġthe ĠTrans former Ġlie Ġmulti - head Ġattention Ġmechanisms : Ġeach Ġword Ġis Ġrepresented Ġby Ġmultiple Ġdifferent Ġweighted Ġaverages Ġof Ġits Ġrelevant Ġcontext . ĠAs Ġsuggested Ġby Ġrecent Ġworks Ġon Ġinterpreting Ġattention Ġhead Ġroles , Ġseparate Ġattention Ġheads Ġmay Ġlearn Ġto Ġlook Ġfor Ġvarious Ġrelationships Ġbetween Ġtokens ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . </s> The Ġattention Ġdistribution Ġof Ġeach Ġhead Ġis Ġpredicted Ġtypically Ġusing Ġthe Ġsoft max Ġnormal izing Ġtransform . ĠAs Ġa Ġresult , Ġall Ġcontext Ġwords Ġhave Ġnon - zero Ġattention Ġweight . ĠRecent Ġwork Ġon Ġsingle Ġattention Ġarchitectures Ġsuggest Ġthat Ġusing Ġsparse Ġnormal izing Ġtransforms Ġin Ġattention Ġmechanisms Ġsuch Ġas Ġsparse max ĠâĢĵ Ġwhich Ġcan Ġyield Ġexactly Ġzero Ġprobabilities Ġfor Ġirrelevant Ġwords ĠâĢĵ Ġmay Ġimprove Ġperformance Ġand Ġinterpret ability ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 . ĠQual itative Ġanalysis Ġof Ġattention Ġheads ĠB IB REF 0 Ġsuggests Ġthat , Ġdepending Ġon Ġwhat Ġphenomena Ġthey Ġcapture , Ġheads Ġtend Ġto Ġfavor Ġfl atter Ġor Ġmore Ġpeaked Ġdistributions . </s> Recent Ġworks Ġhave Ġproposed Ġsparse ĠTransformers ĠB IB REF 10 Ġand Ġadaptive Ġspan ĠTransformers ĠB IB REF 11 . ĠHowever , Ġthe ĠâĢ ľ sp arsity " Ġof Ġthose Ġmodels Ġonly Ġlimits Ġthe Ġattention Ġto Ġa Ġcontiguous Ġspan Ġof Ġpast Ġtokens , Ġwhile Ġin Ġthis Ġwork Ġwe Ġpropose Ġa Ġhighly Ġadaptive ĠTrans former Ġmodel Ġthat Ġis Ġcapable Ġof Ġattending Ġto Ġa Ġsparse Ġset Ġof Ġwords Ġthat Ġare Ġnot Ġnecessarily Ġcontiguous . ĠFigure ĠFIG REF 1 Ġshows Ġthe Ġrelationship Ġof Ġthese Ġmethods Ġwith Ġours . </s> Our Ġcontributions Ġare Ġthe Ġfollowing : </s> We Ġintroduce Ġsparse Ġattention Ġinto Ġthe ĠTrans former Ġarchitecture , Ġshowing Ġthat Ġit Ġe ases Ġinterpret ability Ġand Ġleads Ġto Ġslight Ġaccuracy Ġgains . </s> We Ġpropose Ġan Ġadaptive Ġversion Ġof Ġsparse Ġattention , Ġwhere Ġthe Ġshape Ġof Ġeach Ġattention Ġhead Ġis Ġlearn able Ġand Ġcan Ġvary Ġcontinuously Ġand Ġdynamically Ġbetween Ġthe Ġdense Ġlimit Ġcase Ġof Ġsoft max Ġand Ġthe Ġsparse , Ġpiece wise - linear Ġsparse max Ġcase . </s> We Ġmake Ġan Ġextensive Ġanalysis Ġof Ġthe Ġadded Ġinterpret ability Ġof Ġthese Ġmodels , Ġidentifying Ġboth Ġcris per Ġexamples Ġof Ġattention Ġhead Ġbehavior Ġobserved Ġin Ġprevious Ġwork , Ġas Ġwell Ġas Ġnovel Ġbehaviors Ġunravel ed Ġthanks Ġto Ġthe Ġsp arsity Ġand Ġadapt ivity Ġof Ġour Ġproposed Ġmodel . </s> Background Ġ: :: ĠThe ĠTrans former </s> In ĠN MT , Ġthe ĠTrans former ĠB IB REF 0 Ġis Ġa Ġsequence - to - sequence Ġ( seq 2 seq ) Ġmodel Ġwhich Ġmaps Ġan Ġinput Ġsequence Ġto Ġan Ġoutput Ġsequence Ġthrough Ġhierarchical Ġmulti - head Ġattention Ġmechanisms , Ġyielding Ġa Ġdynamic , Ġcontext - dependent Ġstrategy Ġfor Ġpropag ating Ġinformation Ġwithin Ġand Ġacross Ġsentences . ĠIt Ġcontrasts Ġwith Ġprevious Ġseq 2 seq Ġmodels , Ġwhich Ġusually Ġrely Ġeither Ġon Ġcostly Ġg ated Ġrecurrent Ġoperations ĠB IB REF 15 , ĠB IB REF 16 Ġor Ġstatic Ġconv olutions ĠB IB REF 17 . </s> Given Ġ$ n $ Ġquery Ġcontexts Ġand Ġ$ m $ Ġsequence Ġitems Ġunder Ġconsideration , Ġattention Ġmechanisms Ġcompute , Ġfor Ġeach Ġquery , Ġa Ġweighted Ġrepresentation Ġof Ġthe Ġitems . ĠThe Ġparticular Ġattention Ġmechanism Ġused Ġin ĠB IB REF 0 Ġis Ġcalled Ġscaled Ġdot - product Ġattention , Ġand Ġit Ġis Ġcomputed Ġin Ġthe Ġfollowing Ġway : </s> where Ġ$\ math bf Ġ{ Q } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontains Ġrepresentations Ġof Ġthe Ġqueries , Ġ$\ math bf Ġ{ K }, Ġ\ math bf Ġ{ V } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġd } $ Ġare Ġthe Ġkeys Ġand Ġvalues Ġof Ġthe Ġitems Ġattended Ġover , Ġand Ġ$ d $ Ġis Ġthe Ġdimension ality Ġof Ġthese Ġrepresentations . ĠThe Ġ$\ math bf Ġ{\ pi Ġ} $ Ġmapping Ġnormal izes Ġrow - wise Ġusing Ġsoft max , Ġ$\ math bf Ġ{\ pi Ġ} (\ math bf Ġ{ Z }) _{ ij } Ġ= Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ soft max }} (\ math bf Ġ{ z } _ i ) _ j $, Ġwhere </s> In Ġwords , Ġthe Ġkeys Ġare Ġused Ġto Ġcompute Ġa Ġrelevance Ġscore Ġbetween Ġeach Ġitem Ġand Ġquery . ĠThen , Ġnormalized Ġattention Ġweights Ġare Ġcomputed Ġusing Ġsoft max , Ġand Ġthese Ġare Ġused Ġto Ġweight Ġthe Ġvalues Ġof Ġeach Ġitem Ġat Ġeach Ġquery Ġcontext . </s> However , Ġfor Ġcomplex Ġtasks , Ġdifferent Ġparts Ġof Ġa Ġsequence Ġmay Ġbe Ġrelevant Ġin Ġdifferent Ġways , Ġmotivating Ġmulti - head Ġattention Ġin ĠTransformers . ĠThis Ġis Ġsimply Ġthe Ġapplication Ġof ĠEqu ation ĠDIS PLAY _ FORM 7 Ġin Ġparallel Ġ$ H $ Ġtimes , Ġeach Ġwith Ġa Ġdifferent , Ġlearned Ġlinear Ġtransformation Ġthat Ġallows Ġspecialization : </s> In Ġthe ĠTrans former , Ġthere Ġare Ġthree Ġseparate Ġmulti - head Ġattention Ġmechanisms Ġfor Ġdistinct Ġpurposes : </s> Enc oder Ġself - att ention : Ġbuilds Ġrich , Ġlayered Ġrepresentations Ġof Ġeach Ġinput Ġword , Ġby Ġattending Ġon Ġthe Ġentire Ġinput Ġsentence . </s> Context Ġattention : Ġselects Ġa Ġrepresentative Ġweighted Ġaverage Ġof Ġthe Ġenc od ings Ġof Ġthe Ġinput Ġwords , Ġat Ġeach Ġtime Ġstep Ġof Ġthe Ġdec oder . </s> Dec oder Ġself - att ention : Ġattends Ġover Ġthe Ġpartial Ġoutput Ġsentence Ġfragment Ġproduced Ġso Ġfar . </s> Together , Ġthese Ġmechanisms Ġenable Ġthe Ġcontextual ized Ġflow Ġof Ġinformation Ġbetween Ġthe Ġinput Ġsentence Ġand Ġthe Ġsequential Ġdec oder . </s> Background Ġ: :: ĠSp arse ĠAttention </s> The Ġsoft max Ġmapping Ġ( Equ ation ĠDIS PLAY _ FORM 8 ) Ġis Ġelement wise Ġproportional Ġto Ġ$\ exp Ġ$ , Ġtherefore Ġit Ġcan Ġnever Ġassign Ġa Ġweight Ġof Ġexactly Ġzero . ĠThus , Ġunnecessary Ġitems Ġare Ġstill Ġtaken Ġinto Ġconsideration Ġto Ġsome Ġextent . ĠSince Ġits Ġoutput Ġsums Ġto Ġone , Ġthis Ġinvariably Ġmeans Ġless Ġweight Ġis Ġassigned Ġto Ġthe Ġrelevant Ġitems , Ġpotentially Ġharming Ġperformance Ġand Ġinterpret ability ĠB IB REF 18 . ĠThis Ġhas Ġmotivated Ġa Ġline Ġof Ġresearch Ġon Ġlearning Ġnetworks Ġwith Ġsparse Ġm appings ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 . ĠWe Ġfocus Ġon Ġa Ġrecently - introdu ced Ġflexible Ġfamily Ġof Ġtransformations , Ġ$\ alpha Ġ$ - ent max ĠB IB REF 23 , ĠB IB REF 14 , Ġdefined Ġas : </s> where Ġ$\ tri angle Ġ^ d Ġ\ l brace Ġ\ math bf Ġ{ p }\ in Ġ\ math bb Ġ{ R } ^ d :\ sum Ġ_ { i } Ġp _ i Ġ= Ġ1 \ r brace Ġ$ Ġis Ġthe Ġprobability Ġsimple x , Ġand , Ġfor Ġ$\ alpha Ġ\ ge Ġ1 $, Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġis Ġthe ĠTs all is Ġcontinuous Ġfamily Ġof Ġent rop ies ĠB IB REF 24 : </s> This Ġfamily Ġcontains Ġthe Ġwell - known ĠShannon Ġand ĠG ini Ġent rop ies , Ġcorresponding Ġto Ġthe Ġcases Ġ$\ alpha Ġ= 1 $ Ġand Ġ$\ alpha Ġ= 2 $, Ġrespectively . </s> Equ ation ĠDIS PLAY _ FORM 14 Ġinvolves Ġa Ġconve x Ġoptimization Ġsub problem . ĠUsing Ġthe Ġdefinition Ġof Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ , Ġthe Ġoptim ality Ġconditions Ġmay Ġbe Ġused Ġto Ġderive Ġthe Ġfollowing Ġform Ġfor Ġthe Ġsolution Ġ( App endix ĠSEC REF 83 ): </s> where Ġ$ [ \ cd ot Ġ] _ + $ Ġis Ġthe Ġpositive Ġpart Ġ( Re LU ) Ġfunction , Ġ$\ math bf Ġ{ 1 } $ Ġdenotes Ġthe Ġvector Ġof Ġall Ġones , Ġand Ġ$\ t au Ġ$ ĠâĢĵ Ġwhich Ġacts Ġlike Ġa Ġthreshold ĠâĢĵ Ġis Ġthe ĠLag range Ġmultiplier Ġcorresponding Ġto Ġthe Ġ$\ sum Ġ_ i Ġp _ i = 1 $ Ġconstraint . </s> Background Ġ: :: ĠSp arse ĠAttention Ġ: :: ĠProperties Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max . </s> The Ġappeal Ġof Ġ$\ alpha Ġ$ - ent max Ġfor Ġattention Ġrests Ġon Ġthe Ġfollowing Ġproperties . ĠFor Ġ$\ alpha Ġ= 1 $ Ġ( i . e ., Ġwhen Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġbecomes Ġthe ĠShannon Ġentropy ), Ġit Ġexactly Ġrecovers Ġthe Ġsoft max Ġmapping Ġ( We Ġprovide Ġa Ġshort Ġderiv ation Ġin ĠAppendix ĠSEC REF 89 .). ĠFor Ġall Ġ$\ alpha Ġ> 1 $ Ġit Ġpermits Ġsparse Ġsolutions , Ġin Ġstark Ġcontrast Ġto Ġsoft max . ĠIn Ġparticular , Ġfor Ġ$\ alpha Ġ= 2 $, Ġit Ġrecovers Ġthe Ġsparse max Ġmapping ĠB IB REF 19 , Ġwhich Ġis Ġpiece wise Ġlinear . ĠIn - between , Ġas Ġ$\ alpha Ġ$ Ġincreases , Ġthe Ġmapping Ġcontinuously Ġgets Ġsp arser Ġas Ġits Ġcurv ature Ġchanges . </s> To Ġcompute Ġthe Ġvalue Ġof Ġ$\ alpha Ġ$ - ent max , Ġone Ġmust Ġfind Ġthe Ġthreshold Ġ$\ t au Ġ$ Ġsuch Ġthat Ġthe Ġr . h . s . Ġin ĠEqu ation ĠDIS PLAY _ FORM 16 Ġsums Ġto Ġone . ĠB IB REF 23 Ġpropose Ġa Ġgeneral Ġbis ection Ġalgorithm . ĠB IB REF 14 Ġintroduce Ġa Ġfaster , Ġexact Ġalgorithm Ġfor Ġ$\ alpha Ġ= 1 . 5 $, Ġand Ġenable Ġusing Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġfixed Ġ$\ alpha Ġ$ Ġwithin Ġa Ġneural Ġnetwork Ġby Ġshowing Ġthat Ġthe Ġ$\ alpha Ġ$ - ent max ĠJacob ian Ġw . r . t . Ġ$\ math bf Ġ{ z } $ Ġfor Ġ$\ math bf Ġ{ p } ^ \ star Ġ= Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġis </s> Our Ġwork Ġfurther s Ġthe Ġstudy Ġof Ġ$\ alpha Ġ$ - ent max Ġby Ġproviding Ġa Ġderiv ation Ġof Ġthe ĠJacob ian Ġw . r . t . Ġthe Ġhyper - param eter Ġ$\ alpha Ġ$ Ġ( Section ĠSEC REF 3 ), Ġthereby Ġallowing Ġthe Ġshape Ġand Ġsp arsity Ġof Ġthe Ġmapping Ġto Ġbe Ġlearned Ġautomatically . ĠThis Ġis Ġparticularly Ġappealing Ġin Ġthe Ġcontext Ġof Ġmulti - head Ġattention Ġmechanisms , Ġwhere Ġwe Ġshall Ġshow Ġin ĠSection ĠSEC REF 35 Ġthat Ġdifferent Ġheads Ġtend Ġto Ġlearn Ġdifferent Ġsp arsity Ġbehaviors . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max </s> We Ġnow Ġpropose Ġa Ġnovel ĠTrans former Ġarchitecture Ġwherein Ġwe Ġsimply Ġreplace Ġsoft max Ġwith Ġ$\ alpha Ġ$ - ent max Ġin Ġthe Ġattention Ġheads . ĠCon crete ly , Ġwe Ġreplace Ġthe Ġrow Ġnormal ization Ġ$\ math bf Ġ{\ pi Ġ} $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 7 Ġby </s> This Ġchange Ġleads Ġto Ġsparse Ġattention Ġweights , Ġas Ġlong Ġas Ġ$\ alpha Ġ> 1 $ ; Ġin Ġparticular , Ġ$\ alpha Ġ= 1 . 5 $ Ġis Ġa Ġsensible Ġstarting Ġpoint ĠB IB REF 14 . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġ: :: ĠDifferent Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ Ġper Ġhead . </s> Unlike ĠL ST M - based Ġseq 2 seq Ġmodels , Ġwhere Ġ$\ alpha Ġ$ Ġcan Ġbe Ġmore Ġeasily Ġtuned Ġby Ġgrid Ġsearch , Ġin Ġa ĠTrans former , Ġthere Ġare Ġmany Ġattention Ġheads Ġin Ġmultiple Ġlayers . ĠCru cial Ġto Ġthe Ġpower Ġof Ġsuch Ġmodels , Ġthe Ġdifferent Ġheads Ġcapture Ġdifferent Ġlinguistic Ġphenomena , Ġsome Ġof Ġthem Ġisol ating Ġimportant Ġwords , Ġothers Ġspreading Ġout Ġattention Ġacross Ġphrases ĠB IB REF 0 . ĠThis Ġmotiv ates Ġusing Ġdifferent , Ġadaptive Ġ$\ alpha Ġ$ Ġvalues Ġfor Ġeach Ġattention Ġhead , Ġsuch Ġthat Ġsome Ġheads Ġmay Ġlearn Ġto Ġbe Ġsp arser , Ġand Ġothers Ġmay Ġbecome Ġcloser Ġto Ġsoft max . ĠWe Ġpropose Ġdoing Ġso Ġby Ġtreating Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġas Ġneural Ġnetwork Ġparameters , Ġoptimized Ġvia Ġsto ch astic Ġgrad ients Ġalong Ġwith Ġthe Ġother Ġweights . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġ: :: ĠDer iv atives Ġw . r . t . Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ . </s> In Ġorder Ġto Ġoptimize Ġ$\ alpha Ġ$ Ġautomatically Ġvia Ġgradient Ġmethods , Ġwe Ġmust Ġcompute Ġthe ĠJacob ian Ġof Ġthe Ġent max Ġoutput Ġw . r . t . Ġ$\ alpha Ġ$ . ĠSince Ġent max Ġis Ġdefined Ġthrough Ġan Ġoptimization Ġproblem , Ġthis Ġis Ġnon - t riv ial Ġand Ġcannot Ġbe Ġsimply Ġhandled Ġthrough Ġautomatic Ġdifferentiation ; Ġit Ġfalls Ġwithin Ġthe Ġdomain Ġof Ġarg min Ġdifferentiation , Ġan Ġactive Ġresearch Ġtopic Ġin Ġoptimization ĠB IB REF 25 , ĠB IB REF 26 . </s> One Ġof Ġour Ġkey Ġcontributions Ġis Ġthe Ġderiv ation Ġof Ġa Ġclosed - form Ġexpression Ġfor Ġthis ĠJacob ian . ĠThe Ġnext Ġproposition Ġprovides Ġsuch Ġan Ġexpression , Ġenabling Ġent max Ġlayers Ġwith Ġadaptive Ġ$\ alpha Ġ$ . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġours Ġis Ġthe Ġfirst Ġneural Ġnetwork Ġmodule Ġthat Ġcan Ġautomatically , Ġcontinuously Ġvary Ġin Ġshape Ġaway Ġfrom Ġsoft max Ġand Ġtoward Ġsparse Ġm appings Ġlike Ġsparse max . </s> Prop osition Ġ1 ĠLet Ġ$\ math bf Ġ{ p } ^ \ star Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġbe Ġthe Ġsolution Ġof ĠEqu ation ĠDIS PLAY _ FORM 14 . ĠDen ote Ġthe Ġdistribution Ġ$\ t ilde { p } _ i Ġ{ ( p _ i ^ \ star Ġ) ^{ 2 Ġ- Ġ\ alpha Ġ} }{ Ġ\ sum Ġ_ j ( p _ j ^ \ star Ġ) ^{ 2 - \ alpha Ġ}} $ Ġand Ġlet Ġ$ h _ i Ġ- p ^ \ star Ġ_ i Ġ\ log Ġp ^ \ star Ġ_ i $. ĠThe Ġ$ i $ th Ġcomponent Ġof Ġthe ĠJacob ian Ġ$\ math bf Ġ{ g } Ġ\ frac {\ partial Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) }{ \ partial Ġ\ alpha Ġ} $ Ġis </s> proof Ġuses Ġimplicit Ġfunction Ġdifferentiation Ġand Ġis Ġgiven Ġin ĠAppendix ĠSEC REF 10 . </s> Prop osition ĠUN K REF 22 Ġprovides Ġthe Ġremaining Ġmissing Ġpiece Ġneeded Ġfor Ġtraining Ġadapt ively Ġsparse ĠTransformers . ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġevaluate Ġthis Ġstrategy Ġon Ġneural Ġmachine Ġtranslation , Ġand Ġanalyze Ġthe Ġbehavior Ġof Ġthe Ġlearned Ġattention Ġheads . </s> Exper iments </s> We Ġapply Ġour Ġadapt ively Ġsparse ĠTransformers Ġon Ġfour Ġmachine Ġtranslation Ġtasks . ĠFor Ġcomparison , Ġa Ġnatural Ġbaseline Ġis Ġthe Ġstandard ĠTrans former Ġarchitecture Ġusing Ġthe Ġsoft max Ġtransform Ġin Ġits Ġmulti - head Ġattention Ġmechanisms . ĠWe Ġconsider Ġtwo Ġother Ġmodel Ġvariants Ġin Ġour Ġexperiments Ġthat Ġmake Ġuse Ġof Ġdifferent Ġnormal izing Ġtransformations : </s> 1 . 5 - ent max : Ġa ĠTrans former Ġwith Ġsparse Ġent max Ġattention Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $ Ġfor Ġall Ġheads . ĠThis Ġis Ġa Ġnovel Ġmodel , Ġsince Ġ1 . 5 - ent max Ġhad Ġonly Ġbeen Ġproposed Ġfor ĠR NN - based ĠN MT Ġmodels ĠB IB REF 14 , Ġbut Ġnever Ġin ĠTransformers , Ġwhere Ġattention Ġmodules Ġare Ġnot Ġjust Ġone Ġsingle Ġcomponent Ġof Ġthe Ġseq 2 seq Ġmodel Ġbut Ġrather Ġan Ġintegral Ġpart Ġof Ġall Ġof Ġthe Ġmodel Ġcomponents . </s> $ \ alpha Ġ$ - ent max : Ġan Ġadaptive ĠTrans former Ġwith Ġsparse Ġent max Ġattention Ġwith Ġa Ġdifferent , Ġlearned Ġ$\ alpha Ġ_ { i , j } ^ t $ Ġfor Ġeach Ġhead . </s> The Ġadaptive Ġmodel Ġhas Ġan Ġadditional Ġscal ar Ġparameter Ġper Ġattention Ġhead Ġper Ġlayer Ġfor Ġeach Ġof Ġthe Ġthree Ġattention Ġmechanisms Ġ( enc oder Ġself - att ention , Ġcontext Ġattention , Ġand Ġdec oder Ġself - att ention ), Ġi . e ., </s> and Ġwe Ġset Ġ$\ alpha Ġ_ { i , j } ^ t Ġ= Ġ1 Ġ+ Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ s igm oid }} ( a _{ i , j } ^ t ) Ġ\ in Ġ] 1 , Ġ2 [ $. ĠAll Ġor Ġsome Ġof Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġcan Ġbe Ġtied Ġif Ġdesired , Ġbut Ġwe Ġkeep Ġthem Ġindependent Ġfor Ġanalysis Ġpurposes . </s> Exper iments Ġ: :: ĠDat as ets . </s> Our Ġmodels Ġwere Ġtrained Ġon Ġ4 Ġmachine Ġtranslation Ġdatasets Ġof Ġdifferent Ġtraining Ġsizes : </s> [ item se p = . 5 ex , left margin = 2 ex ] </s> I W SL T Ġ2017 ĠGerman Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 27 : Ġ200 K Ġsentence Ġpairs . </s> K FT T ĠJapanese Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 28 : Ġ300 K Ġsentence Ġpairs . </s> W MT Ġ2016 ĠRomanian Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 29 : Ġ600 K Ġsentence Ġpairs . </s> W MT Ġ2014 ĠEnglish Ġ$\ right arrow Ġ$ ĠGerman ĠB IB REF 30 : Ġ4 . 5 M Ġsentence Ġpairs . </s> All Ġof Ġthese Ġdatasets Ġwere Ġpre process ed Ġwith Ġbyte - pair Ġencoding ĠB IB REF 31 , Ġusing Ġjoint Ġsegment ations Ġof Ġ32 k Ġmerge Ġoperations . </s> Exper iments Ġ: :: ĠTraining . </s> We Ġfollow Ġthe Ġdimensions Ġof Ġthe ĠTrans former - Base Ġmodel Ġof ĠB IB REF 0 : ĠThe Ġnumber Ġof Ġlayers Ġis Ġ$ L = 6 $ Ġand Ġnumber Ġof Ġheads Ġis Ġ$ H = 8 $ Ġin Ġthe Ġenc oder Ġself - att ention , Ġthe Ġcontext Ġattention , Ġand Ġthe Ġdec oder Ġself - att ention . ĠWe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ8 192 Ġtokens Ġand Ġwarm Ġup Ġthe Ġlearning Ġrate Ġlin early Ġuntil Ġ20 k Ġsteps , Ġafter Ġwhich Ġit Ġdec ays Ġaccording Ġto Ġan Ġinverse Ġsquare Ġroot Ġschedule . ĠAll Ġmodels Ġwere Ġtrained Ġuntil Ġconvergence Ġof Ġvalidation Ġaccuracy , Ġand Ġevaluation Ġwas Ġdone Ġat Ġeach Ġ10 k Ġsteps Ġfor Ġro $ \ right arrow Ġ$ en Ġand Ġen $ \ right arrow Ġ$ de Ġand Ġat Ġeach Ġ5 k Ġsteps Ġfor Ġde $ \ right arrow Ġ$ en Ġand Ġja $ \ right arrow Ġ$ en . ĠThe Ġend - to - end Ġcomputational Ġoverhead Ġof Ġour Ġmethods , Ġwhen Ġcompared Ġto Ġstandard Ġsoft max , Ġis Ġrelatively Ġsmall ; Ġin Ġtraining Ġtokens Ġper Ġsecond , Ġthe Ġmodels Ġusing Ġ$\ alpha Ġ$ - ent max Ġand Ġ$ 1 . 5 $ - ent max Ġare , Ġrespectively , Ġ$ 75 \ % $ Ġand Ġ$ 90 \ % $ Ġthe Ġspeed Ġof Ġthe Ġsoft max Ġmodel . </s> Exper iments Ġ: :: ĠResults . </s> We Ġreport Ġtest Ġset Ġtoken ized ĠB LE U ĠB IB REF 32 Ġresults Ġin ĠTable ĠT AB REF 27 . ĠWe Ġcan Ġsee Ġthat Ġreplacing Ġsoft max Ġby Ġent max Ġdoes Ġnot Ġhurt Ġperformance Ġin Ġany Ġof Ġthe Ġdatasets ; Ġindeed , Ġsparse Ġattention ĠTransformers Ġtend Ġto Ġhave Ġslightly Ġhigher ĠB LE U , Ġbut Ġtheir Ġsp arsity Ġleads Ġto Ġa Ġbetter Ġpotential Ġfor Ġanalysis . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġmake Ġuse Ġof Ġthis Ġpotential Ġby Ġexploring Ġthe Ġlearned Ġinternal Ġmechanics Ġof Ġthe Ġself - att ention Ġheads . </s> Analysis </s> We Ġconduct Ġan Ġanalysis Ġfor Ġthe Ġhigher - resource Ġdataset ĠW MT Ġ2014 ĠEnglish Ġ$\ right arrow Ġ$ ĠGerman Ġof Ġthe Ġattention Ġin Ġthe Ġsparse Ġadaptive ĠTrans former Ġmodel Ġ($ \ alpha Ġ$ - ent max ) Ġat Ġmultiple Ġlevels : Ġwe Ġanalyze Ġhigh - level Ġstatistics Ġas Ġwell Ġas Ġindividual Ġhead Ġbehavior . ĠMoreover , Ġwe Ġmake Ġa Ġqualitative Ġanalysis Ġof Ġthe Ġinterpret ability Ġcapabilities Ġof Ġour Ġmodels . </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠWhat Ġkind Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ Ġvalues Ġare Ġlearned ? </s> Figure ĠFIG REF 37 Ġshows Ġthe Ġlearning Ġtraject ories Ġof Ġthe Ġ$\ alpha Ġ$ Ġparameters Ġof Ġa Ġselected Ġsubset Ġof Ġheads . ĠWe Ġgenerally Ġobserve Ġa Ġtendency Ġfor Ġthe Ġrandomly - initialized Ġ$\ alpha Ġ$ Ġparameters Ġto Ġdecrease Ġinitially , Ġsuggesting Ġthat Ġsoft max - like Ġbehavior Ġmay Ġbe Ġpreferable Ġwhile Ġthe Ġmodel Ġis Ġstill Ġvery Ġuncertain . ĠAfter Ġaround Ġone Ġthousand Ġsteps , Ġsome Ġheads Ġchange Ġdirection Ġand Ġbecome Ġsp arser , Ġperhaps Ġas Ġthey Ġbecome Ġmore Ġconfident Ġand Ġspecialized . ĠThis Ġshows Ġthat Ġthe Ġinitialization Ġof Ġ$\ alpha Ġ$ Ġdoes Ġnot Ġpred eter mine Ġits Ġsp arsity Ġlevel Ġor Ġthe Ġrole Ġthe Ġhead Ġwill Ġhave Ġthroughout . ĠIn Ġparticular , Ġhead Ġ8 Ġin Ġthe Ġenc oder Ġself - att ention Ġlayer Ġ2 Ġfirst Ġdrops Ġto Ġaround Ġ$\ alpha Ġ= 1 . 3 $ Ġbefore Ġbecoming Ġone Ġof Ġthe Ġsp ars est Ġheads , Ġwith Ġ$\ alpha Ġ\ app rox Ġ2 $. </s> The Ġoverall Ġdistribution Ġof Ġ$\ alpha Ġ$ Ġvalues Ġat Ġconvergence Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 38 . ĠWe Ġcan Ġobserve Ġthat Ġthe Ġenc oder Ġself - att ention Ġblocks Ġlearn Ġto Ġconcentrate Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġin Ġtwo Ġmodes : Ġa Ġvery Ġsparse Ġone Ġaround Ġ$\ alpha Ġ\ right arrow Ġ2 $, Ġand Ġa Ġdense Ġone Ġbetween Ġsoft max Ġand Ġ1 . 5 - ent max Ġ. ĠHowever , Ġthe Ġdec oder Ġself Ġand Ġcontext Ġattention Ġonly Ġlearn Ġto Ġdistribute Ġthese Ġparameters Ġin Ġa Ġsingle Ġmode . ĠWe Ġshow Ġnext Ġthat Ġthis Ġis Ġreflected Ġin Ġthe Ġaverage Ġdensity Ġof Ġattention Ġweight Ġvectors Ġas Ġwell . </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠAttention Ġweight Ġdensity Ġwhen Ġtranslating . </s> For Ġany Ġ$\ alpha Ġ> 1 $, Ġit Ġwould Ġstill Ġbe Ġpossible Ġfor Ġthe Ġweight Ġmat rices Ġin ĠEqu ation ĠDIS PLAY _ FORM 9 Ġto Ġlearn Ġre - sc al ings Ġso Ġas Ġto Ġmake Ġattention Ġsp arser Ġor Ġdens er . ĠTo Ġvisualize Ġthe Ġimpact Ġof Ġadaptive Ġ$\ alpha Ġ$ Ġvalues , Ġwe Ġcompare Ġthe Ġempirical Ġattention Ġweight Ġdensity Ġ( the Ġaverage Ġnumber Ġof Ġtokens Ġreceiving Ġnon - zero Ġattention ) Ġwithin Ġeach Ġmodule , Ġagainst Ġsparse ĠTransformers Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $. </s> Figure ĠFIG REF 40 Ġshows Ġthat , Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $, Ġheads Ġtend Ġto Ġbe Ġsparse Ġand Ġsimilarly - dist ributed Ġin Ġall Ġthree Ġattention Ġmodules . ĠWith Ġlearned Ġ$\ alpha Ġ$ , Ġthere Ġare Ġtwo Ġnotable Ġchanges : Ġ( i ) Ġa Ġprominent Ġmode Ġcorresponding Ġto Ġfully Ġdense Ġprobabilities , Ġshowing Ġthat Ġour Ġmodels Ġlearn Ġto Ġcombine Ġsparse Ġand Ġdense Ġattention , Ġand Ġ( ii ) Ġa Ġdistinction Ġbetween Ġthe Ġenc oder Ġself - att ention ĠâĢĵ Ġwhose Ġbackground Ġdistribution Ġtends Ġtoward Ġextreme Ġsp arsity ĠâĢĵ Ġand Ġthe Ġother Ġtwo Ġmodules , Ġwho Ġexhibit Ġmore Ġuniform Ġbackground Ġdistributions . ĠThis Ġsuggests Ġthat Ġperhaps Ġentirely Ġsparse ĠTransformers Ġare Ġsub opt imal . </s> The Ġfact Ġthat Ġthe Ġdec oder Ġseems Ġto Ġprefer Ġdens er Ġattention Ġdistributions Ġmight Ġbe Ġattributed Ġto Ġit Ġbeing Ġauto - reg ressive , Ġonly Ġhaving Ġaccess Ġto Ġpast Ġtokens Ġand Ġnot Ġthe Ġfull Ġsentence . ĠWe Ġspeculate Ġthat Ġit Ġmight Ġlose Ġtoo Ġmuch Ġinformation Ġif Ġit Ġassigned Ġweights Ġof Ġzero Ġto Ġtoo Ġmany Ġtokens Ġin Ġthe Ġself - att ention , Ġsince Ġthere Ġare Ġfewer Ġtokens Ġto Ġattend Ġto Ġin Ġthe Ġfirst Ġplace . </s> Te asing Ġthis Ġdown Ġinto Ġseparate Ġlayers , ĠFigure ĠFIG REF 41 Ġshows Ġthe Ġaverage Ġ( s orted ) Ġdensity Ġof Ġeach Ġhead Ġfor Ġeach Ġlayer . ĠWe Ġobserve Ġthat Ġ$\ alpha Ġ$ - ent max Ġis Ġable Ġto Ġlearn Ġdifferent Ġsp arsity Ġpatterns Ġat Ġeach Ġlayer , Ġleading Ġto Ġmore Ġvariance Ġin Ġindividual Ġhead Ġbehavior , Ġto Ġclearly - identified Ġdense Ġand Ġsparse Ġheads , Ġand Ġoverall Ġto Ġdifferent Ġtendencies Ġcompared Ġto Ġthe Ġfixed Ġcase Ġof Ġ$\ alpha Ġ= 1 . 5 $. </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠHead Ġdiversity . </s> To Ġmeasure Ġthe Ġoverall Ġdisagreement Ġbetween Ġattention Ġheads , Ġas Ġa Ġmeasure Ġof Ġhead Ġdiversity , Ġwe Ġuse Ġthe Ġfollowing Ġgeneral ization Ġof Ġthe ĠJensen - Sh annon Ġdivergence : </s> where Ġ$\ math bf Ġ{ p } _ j $ Ġis Ġthe Ġvector Ġof Ġattention Ġweights Ġassigned Ġby Ġhead Ġ$ j $ Ġto Ġeach Ġword Ġin Ġthe Ġsequence , Ġand Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } $ Ġis Ġthe ĠShannon Ġentropy , Ġbase - adjusted Ġbased Ġon Ġthe Ġdimension Ġof Ġ$\ math bf Ġ{ p } $ Ġsuch Ġthat Ġ$ JS Ġ\ le Ġ1 $. ĠWe Ġaverage Ġthis Ġmeasure Ġover Ġthe Ġentire Ġvalidation Ġset . ĠThe Ġhigher Ġthis Ġmetric Ġis , Ġthe Ġmore Ġthe Ġheads Ġare Ġtaking Ġdifferent Ġroles Ġin Ġthe Ġmodel . </s> Figure ĠFIG REF 44 Ġshows Ġthat Ġboth Ġsparse ĠTrans former Ġvariants Ġshow Ġmore Ġdiversity Ġthan Ġthe Ġtraditional Ġsoft max Ġone . ĠInterestingly , Ġdiversity Ġseems Ġto Ġpeak Ġin Ġthe Ġmiddle Ġlayers Ġof Ġthe Ġenc oder Ġself - att ention Ġand Ġcontext Ġattention , Ġwhile Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe Ġdec oder Ġself - att ention . </s> The Ġstatistics Ġshown Ġin Ġthis Ġsection Ġcan Ġbe Ġfound Ġfor Ġthe Ġother Ġlanguage Ġpairs Ġin ĠAppendix ĠSEC REF 8 . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations </s> Previous Ġwork Ġpointed Ġout Ġsome Ġspecific Ġroles Ġplayed Ġby Ġdifferent Ġheads Ġin Ġthe Ġsoft max ĠTrans former Ġmodel ĠB IB REF 33 , ĠB IB REF 5 , ĠB IB REF 9 . ĠIdent ifying Ġthe Ġspecialization Ġof Ġa Ġhead Ġcan Ġbe Ġdone Ġby Ġobserving Ġthe Ġtype Ġof Ġtokens Ġor Ġsequences Ġthat Ġthe Ġhead Ġoften Ġassigns Ġmost Ġof Ġits Ġattention Ġweight ; Ġthis Ġis Ġfacilitated Ġby Ġsp arsity . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠPos itional Ġheads . </s> One Ġparticular Ġtype Ġof Ġhead , Ġas Ġnoted Ġby ĠB IB REF 9 , Ġis Ġthe Ġpositional Ġhead . ĠThese Ġheads Ġtend Ġto Ġfocus Ġtheir Ġattention Ġon Ġeither Ġthe Ġprevious Ġor Ġnext Ġtoken Ġin Ġthe Ġsequence , Ġthus Ġobtaining Ġrepresentations Ġof Ġthe Ġneighborhood Ġof Ġthe Ġcurrent Ġtime Ġstep . ĠIn ĠFigure ĠFIG REF 47 , Ġwe Ġshow Ġattention Ġplots Ġfor Ġsuch Ġheads , Ġfound Ġfor Ġeach Ġof Ġthe Ġstudied Ġmodels . ĠThe Ġsp arsity Ġof Ġour Ġmodels Ġallows Ġthese Ġheads Ġto Ġbe Ġmore Ġconfident Ġin Ġtheir Ġrepresentations , Ġby Ġassigning Ġthe Ġwhole Ġprobability Ġdistribution Ġto Ġa Ġsingle Ġtoken Ġin Ġthe Ġsequence . ĠCon crete ly , Ġwe Ġmay Ġmeasure Ġa Ġpositional Ġhead 's Ġconfidence Ġas Ġthe Ġaverage Ġattention Ġweight Ġassigned Ġto Ġthe Ġprevious Ġtoken . ĠThe Ġsoft max Ġmodel Ġhas Ġthree Ġheads Ġfor Ġposition Ġ$ - 1 $, Ġwith Ġmedian Ġconfidence Ġ$ 93 . 5 \ % $. ĠThe Ġ$ 1 . 5 $ - ent max Ġmodel Ġalso Ġhas Ġthree Ġheads Ġfor Ġthis Ġposition , Ġwith Ġmedian Ġconfidence Ġ$ 94 . 4 \ % $. ĠThe Ġadaptive Ġmodel Ġhas Ġfour Ġheads , Ġwith Ġmedian Ġconf idences Ġ$ 95 . 9 \ % $, Ġthe Ġlowest - confidence Ġhead Ġbeing Ġdense Ġwith Ġ$\ alpha Ġ= 1 . 18 $, Ġwhile Ġthe Ġhighest - confidence Ġhead Ġbeing Ġsparse Ġ($ \ alpha Ġ= 1 . 91 $ ). </s> For Ġposition Ġ$ + 1 $, Ġthe Ġmodels Ġeach Ġdedicate Ġone Ġhead , Ġwith Ġconfidence Ġaround Ġ$ 95 \ % $, Ġslightly Ġhigher Ġfor Ġent max . ĠThe Ġadaptive Ġmodel Ġsets Ġ$\ alpha Ġ= 1 . 96 $ Ġfor Ġthis Ġhead . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠB PE - mer ging Ġhead . </s> Due Ġto Ġthe Ġsp arsity Ġof Ġour Ġmodels , Ġwe Ġare Ġable Ġto Ġidentify Ġother Ġhead Ġspecial izations , Ġeasily Ġidentifying Ġwhich Ġheads Ġshould Ġbe Ġfurther Ġanalysed . ĠIn ĠFigure ĠFIG REF 51 Ġwe Ġshow Ġone Ġsuch Ġhead Ġwhere Ġthe Ġ$\ alpha Ġ$ Ġvalue Ġis Ġparticularly Ġhigh Ġ( in Ġthe Ġenc oder , Ġlayer Ġ1 , Ġhead Ġ4 Ġdepicted Ġin ĠFigure ĠFIG REF 37 ). ĠWe Ġfound Ġthat Ġthis Ġhead Ġmost Ġoften Ġlooks Ġat Ġthe Ġcurrent Ġtime Ġstep Ġwith Ġhigh Ġconfidence , Ġmaking Ġit Ġa Ġpositional Ġhead Ġwith Ġoffset Ġ0 . ĠHowever , Ġthis Ġhead Ġoften Ġspreads Ġweight Ġsparse ly Ġover Ġ2 - 3 Ġneighboring Ġtokens , Ġwhen Ġthe Ġtokens Ġare Ġpart Ġof Ġthe Ġsame ĠB PE Ġcluster Ġor Ġhyp hen ated Ġwords . ĠAs Ġthis Ġhead Ġis Ġin Ġthe Ġfirst Ġlayer , Ġit Ġprovides Ġa Ġuseful Ġservice Ġto Ġthe Ġhigher Ġlayers Ġby Ġcombining Ġinformation Ġevenly Ġwithin Ġsome ĠB PE Ġclusters . </s> For Ġeach ĠB PE Ġcluster Ġor Ġcluster Ġof Ġhyp hen ated Ġwords , Ġwe Ġcomputed Ġa Ġscore Ġbetween Ġ0 Ġand Ġ1 Ġthat Ġcorresponds Ġto Ġthe Ġmaximum Ġattention Ġmass Ġassigned Ġby Ġany Ġtoken Ġto Ġthe Ġrest Ġof Ġthe Ġtokens Ġinside Ġthe Ġcluster Ġin Ġorder Ġto Ġquantify Ġthe ĠB PE - mer ging Ġcapabilities Ġof Ġthese Ġheads . ĠThere Ġare Ġnot Ġany Ġattention Ġheads Ġin Ġthe Ġsoft max Ġmodel Ġthat Ġare Ġable Ġto Ġobtain Ġa Ġscore Ġover Ġ$ 80 \ % $, Ġwhile Ġfor Ġ$ 1 . 5 $ - ent max Ġand Ġ$\ alpha Ġ$ - ent max Ġthere Ġare Ġtwo Ġheads Ġin Ġeach Ġ($ 83 . 3 \ % $ Ġand Ġ$ 85 . 6 \ % $ Ġfor Ġ$ 1 . 5 $ - ent max Ġand Ġ$ 88 . 5 \ % $ Ġand Ġ$ 89 . 8 \ % $ Ġfor Ġ$\ alpha Ġ$ - ent max ). </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠInter rog ation Ġhead . </s> On Ġthe Ġother Ġhand , Ġin ĠFigure ĠFIG REF 52 Ġwe Ġshow Ġa Ġhead Ġfor Ġwhich Ġour Ġadapt ively Ġsparse Ġmodel Ġchose Ġan Ġ$\ alpha Ġ$ Ġclose Ġto Ġ1 , Ġmaking Ġit Ġcloser Ġto Ġsoft max Ġ( also Ġshown Ġin Ġenc oder , Ġlayer Ġ1 , Ġhead Ġ3 Ġdepicted Ġin ĠFigure ĠFIG REF 37 ). ĠWe Ġobserve Ġthat Ġthis Ġhead Ġassigns Ġa Ġhigh Ġprobability Ġto Ġquestion Ġmarks Ġat Ġthe Ġend Ġof Ġthe Ġsentence Ġin Ġtime Ġsteps Ġwhere Ġthe Ġcurrent Ġtoken Ġis Ġinterrog ative , Ġthus Ġmaking Ġit Ġan Ġinterrogation - det ect ing Ġhead . ĠWe Ġalso Ġobserve Ġthis Ġtype Ġof Ġheads Ġin Ġthe Ġother Ġmodels , Ġwhich Ġwe Ġalso Ġdepict Ġin ĠFigure ĠFIG REF 52 . ĠThe Ġaverage Ġattention Ġweight Ġplaced Ġon Ġthe Ġquestion Ġmark Ġwhen Ġthe Ġcurrent Ġtoken Ġis Ġan Ġinterrog ative Ġword Ġis Ġ$ 98 . 5 \ % $ Ġfor Ġsoft max , Ġ$ 97 . 0 \ % $ Ġfor Ġ$ 1 . 5 $ - ent max , Ġand Ġ$ 99 . 5 \ % $ Ġfor Ġ$\ alpha Ġ$ - ent max . </s> Furthermore , Ġwe Ġcan Ġexamine Ġsentences Ġwhere Ġsome Ġtend entially Ġsparse Ġheads Ġbecome Ġless Ġso , Ġthus Ġidentifying Ġsources Ġof Ġambiguity Ġwhere Ġthe Ġhead Ġis Ġless Ġconfident Ġin Ġits Ġprediction . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 55 Ġwhere Ġsp arsity Ġin Ġthe Ġsame Ġhead Ġdiffers Ġfor Ġsentences Ġof Ġsimilar Ġlength . </s> Related ĠWork Ġ: :: ĠSp arse Ġattention . </s> Prior Ġwork Ġhas Ġdeveloped Ġsparse Ġattention Ġmechanisms , Ġincluding Ġapplications Ġto ĠN MT ĠB IB REF 19 , ĠB IB REF 12 , ĠB IB REF 20 , ĠB IB REF 22 , ĠB IB REF 34 . ĠB IB REF 14 Ġintroduced Ġthe Ġent max Ġfunction Ġthis Ġwork Ġbuilds Ġupon . ĠIn Ġtheir Ġwork , Ġthere Ġis Ġa Ġsingle Ġattention Ġmechanism Ġwhich Ġis Ġcontrolled Ġby Ġa Ġfixed Ġ$\ alpha Ġ$ . ĠIn Ġcontrast , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġallow Ġsuch Ġattention Ġm appings Ġto Ġdynamically Ġadapt Ġtheir Ġcurv ature Ġand Ġsp arsity , Ġby Ġautomatically Ġadjusting Ġthe Ġcontinuous Ġ$\ alpha Ġ$ Ġparameter . ĠWe Ġalso Ġprovide Ġthe Ġfirst Ġresults Ġusing Ġsparse Ġattention Ġin Ġa ĠTrans former Ġmodel . </s> Related ĠWork Ġ: :: ĠFixed Ġsp arsity Ġpatterns . </s> Recent Ġresearch Ġimproves Ġthe Ġscal ability Ġof ĠTrans former - like Ġnetworks Ġthrough Ġstatic , Ġfixed Ġsp arsity Ġpatterns ĠB IB REF 10 , ĠB IB REF 35 . ĠOur Ġadapt ively - s parse ĠTrans former Ġcan Ġdynamically Ġselect Ġa Ġsp arsity Ġpattern Ġthat Ġfinds Ġrelevant Ġwords Ġregardless Ġof Ġtheir Ġposition Ġ( e . g ., ĠFigure ĠFIG REF 52 ). ĠMoreover , Ġthe Ġtwo Ġstrategies Ġcould Ġbe Ġcombined . ĠIn Ġa Ġconcurrent Ġline Ġof Ġresearch , ĠB IB REF 11 Ġpropose Ġan Ġadaptive Ġattention Ġspan Ġfor ĠTrans former Ġlanguage Ġmodels . ĠWhile Ġtheir Ġwork Ġhas Ġeach Ġhead Ġlearn Ġa Ġdifferent Ġcontiguous Ġspan Ġof Ġcontext Ġtokens Ġto Ġattend Ġto , Ġour Ġwork Ġfinds Ġdifferent Ġsp arsity Ġpatterns Ġin Ġthe Ġsame Ġspan . ĠInterestingly , Ġsome Ġof Ġtheir Ġfindings Ġmirror Ġours ĠâĢĵ Ġwe Ġfound Ġthat Ġattention Ġheads Ġin Ġthe Ġlast Ġlayers Ġtend Ġto Ġbe Ġdens er Ġon Ġaverage Ġwhen Ġcompared Ġto Ġthe Ġones Ġin Ġthe Ġfirst Ġlayers , Ġwhile Ġtheir Ġwork Ġhas Ġfound Ġthat Ġlower Ġlayers Ġtend Ġto Ġhave Ġa Ġshorter Ġattention Ġspan Ġcompared Ġto Ġhigher Ġlayers . </s> Related ĠWork Ġ: :: ĠTrans former Ġinterpret ability . </s> The Ġoriginal ĠTrans former Ġpaper ĠB IB REF 0 Ġshows Ġattention Ġvisual izations , Ġfrom Ġwhich Ġsome Ġspeculation Ġcan Ġbe Ġmade Ġof Ġthe Ġroles Ġthe Ġseveral Ġattention Ġheads Ġhave . ĠB IB REF 7 Ġstudy Ġthe Ġsynt actic Ġabilities Ġof Ġthe ĠTrans former Ġself - att ention , Ġwhile ĠB IB REF 6 Ġextract Ġdependency Ġrelations Ġfrom Ġthe Ġattention Ġweights . ĠB IB REF 8 Ġfind Ġthat Ġthe Ġself - att ent ions Ġin ĠB ERT ĠB IB REF 3 Ġfollow Ġa Ġsequence Ġof Ġprocesses Ġthat Ġresembles Ġa Ġclassical ĠN LP Ġpipeline . ĠRegarding Ġredundancy Ġof Ġheads , ĠB IB REF 9 Ġdevelop Ġa Ġmethod Ġthat Ġis Ġable Ġto Ġpr une Ġheads Ġof Ġthe Ġmulti - head Ġattention Ġmodule Ġand Ġmake Ġan Ġempirical Ġstudy Ġof Ġthe Ġrole Ġthat Ġeach Ġhead Ġhas Ġin Ġself - att ention Ġ( pos itional , Ġsynt actic Ġand Ġrare Ġwords ). ĠB IB REF 36 Ġalso Ġaim Ġto Ġreduce Ġhead Ġredundancy Ġby Ġadding Ġa Ġregular ization Ġterm Ġto Ġthe Ġloss Ġthat Ġmaxim izes Ġhead Ġdisagreement Ġand Ġobtain Ġimproved Ġresults . ĠWhile Ġnot Ġconsidering ĠTrans former Ġattent ions , ĠB IB REF 18 Ġshow Ġthat Ġtraditional Ġattention Ġmechanisms Ġdo Ġnot Ġnecessarily Ġimprove Ġinterpret ability Ġsince Ġsoft max Ġattention Ġis Ġvulnerable Ġto Ġan Ġadvers arial Ġattack Ġleading Ġto Ġwildly Ġdifferent Ġmodel Ġpredictions Ġfor Ġthe Ġsame Ġattention Ġweights . ĠSp arse Ġattention Ġmay Ġmitigate Ġthese Ġissues ; Ġhowever , Ġour Ġwork Ġfocuses Ġmostly Ġon Ġa Ġmore Ġmechanical Ġaspect Ġof Ġinterpretation Ġby Ġanalyzing Ġhead Ġbehavior , Ġrather Ġthan Ġon Ġexplanations Ġfor Ġpredictions . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġcontribute Ġa Ġnovel Ġstrategy Ġfor Ġadapt ively Ġsparse Ġattention , Ġand , Ġin Ġparticular , Ġfor Ġadapt ively Ġsparse ĠTransformers . ĠWe Ġpresent Ġthe Ġfirst Ġempirical Ġanalysis Ġof ĠTransformers Ġwith Ġsparse Ġattention Ġm appings Ġ( i . e ., Ġent max ), Ġshowing Ġpotential Ġin Ġboth Ġtranslation Ġaccuracy Ġas Ġwell Ġas Ġin Ġmodel Ġinterpret ability . </s> In Ġparticular , Ġwe Ġanalyzed Ġhow Ġthe Ġattention Ġheads Ġin Ġthe Ġproposed Ġadapt ively Ġsparse ĠTrans former Ġcan Ġspecialize Ġmore Ġand Ġwith Ġhigher Ġconfidence . ĠOur Ġadapt ivity Ġstrategy Ġrelies Ġonly Ġon Ġgradient - based Ġoptimization , Ġside - ste pping Ġcostly Ġper - head Ġhyper - param eter Ġsearches . ĠFurther Ġspeed - ups Ġare Ġpossible Ġby Ġleveraging Ġmore Ġparallel ism Ġin Ġthe Ġbis ection Ġalgorithm Ġfor Ġcomputing Ġ$\ alpha Ġ$ - ent max . </s> Finally , Ġsome Ġof Ġthe Ġautomatically - learn ed Ġbehaviors Ġof Ġour Ġadapt ively Ġsparse ĠTransformers ĠâĢĵ Ġfor Ġinstance , Ġthe Ġnear - d eter ministic Ġpositional Ġheads Ġor Ġthe Ġsub word Ġjoining Ġhead ĠâĢĵ Ġmay Ġprovide Ġnew Ġideas Ġfor Ġdesigning Ġstatic Ġvariations Ġof Ġthe ĠTrans former . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ĠSt G ĠDeep SP IN Ġ7 58 969 ), Ġand Ġby Ġthe ĠFund a Ã§ Ã£o Ġpara Ġa ĠCi Ãª nc ia Ġe ĠTec n olog ia Ġthrough Ġcontracts ĠUID / EE A / 5000 8 / 2019 Ġand ĠCM UP ER I / T IC / 00 46 / 2014 Ġ( Go Local ). ĠWe Ġare Ġgrateful Ġto ĠBen ĠPeters Ġfor Ġthe Ġ$\ alpha Ġ$ - ent max Ġcode Ġand ĠEr ick ĠF on sec a , ĠMarcos ĠTre vis o , ĠPedro ĠMart ins , Ġand ĠTs vet om ila ĠM ih ay l ova Ġfor Ġinsightful Ġgroup Ġdiscussion . ĠWe Ġthank ĠMath ieu ĠBl ond el Ġfor Ġthe Ġidea Ġto Ġlearn Ġ$\ alpha Ġ$ . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġfeedback . </s> Supplement ary ĠMaterial </s> Background Ġ: :: ĠRegular ized ĠFen chel - Young Ġprediction Ġfunctions </s> Definition Ġ1 Ġ( BI B REF 23 ) </s> Let Ġ$\ O mega Ġ\ col on Ġ\ tri angle Ġ^ d Ġ\ right arrow Ġ{\ math bb Ġ{ R }} \ cup Ġ\ l brace Ġ\ in fty Ġ\ r brace Ġ$ Ġbe Ġa Ġstrictly Ġconve x Ġregular ization Ġfunction . ĠWe Ġdefine Ġthe Ġprediction Ġfunction Ġ$\ math bf Ġ{\ pi Ġ} _{ \ O mega Ġ} $ Ġas </s> Background Ġ: :: ĠCharacter izing Ġthe Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġmapping </s> L em ma Ġ1 Ġ( BI B REF 14 ) ĠFor Ġany Ġ$\ math bf Ġ{ z } $, Ġthere Ġexists Ġa Ġunique Ġ$\ t au Ġ^ \ star Ġ$ Ġsuch Ġthat </s> Proof : ĠFrom Ġthe Ġdefinition Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $, </s> we Ġmay Ġeasily Ġidentify Ġit Ġwith Ġa Ġregular ized Ġprediction Ġfunction Ġ( Def . ĠUN K REF 81 ): </s> We Ġfirst Ġnote Ġthat Ġfor Ġall Ġ$\ math bf Ġ{ p }\ in Ġ\ tri angle Ġ^ d $, </s> From Ġthe Ġconstant Ġinvari ance Ġand Ġscaling Ġproperties Ġof Ġ$\ math bf Ġ{\ pi Ġ} _{ \ O mega Ġ} $ ĠB IB REF 23 , </s> Using ĠB IB REF 23 , Ġnoting Ġthat Ġ$ g ^ {\ prime Ġ} ( t ) Ġ= Ġt ^ {\ alpha Ġ- Ġ1 } $ Ġand Ġ$( g ^ {\ prime Ġ}) ^{ - 1 } ( u ) Ġ= Ġu ^ {{ 1 }{ \ alpha Ġ- 1 }} $, Ġyields </s> Since Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġis Ġstrictly Ġconve x Ġon Ġthe Ġsimple x , Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġhas Ġa Ġunique Ġsolution Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ . ĠEqu ation ĠDIS PLAY _ FORM 88 Ġimplicitly Ġdefines Ġa Ġone - to - one Ġmapping Ġbetween Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ Ġand Ġ$\ t au Ġ^ \ star Ġ$ Ġas Ġlong Ġas Ġ$\ math bf Ġ{ p } ^ \ star Ġ\ in Ġ\ tri angle Ġ$ , Ġtherefore Ġ$\ t au Ġ^ \ star Ġ$ Ġis Ġalso Ġunique . </s> Background Ġ: :: ĠConnect ions Ġto Ġsoft max Ġand Ġsparse max </s> The ĠEucl ide an Ġprojection Ġonto Ġthe Ġsimple x , Ġsometimes Ġreferred Ġto , Ġin Ġthe Ġcontext Ġof Ġneural Ġattention , Ġas Ġsparse max ĠB IB REF 19 , Ġis Ġdefined Ġas </s> The Ġsolution Ġcan Ġbe Ġcharacterized Ġthrough Ġthe Ġunique Ġthreshold Ġ$\ t au Ġ$ Ġsuch Ġthat Ġ$\ sum Ġ_ i Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ s parse max }} (\ math bf Ġ{ z }) _ i Ġ= Ġ1 $ Ġand ĠB IB REF 38 </s> Thus , Ġeach Ġcoordinate Ġof Ġthe Ġsparse max Ġsolution Ġis Ġa Ġpiece wise - linear Ġfunction . ĠVis ibly , Ġthis Ġexpression Ġis Ġrecovered Ġwhen Ġsetting Ġ$\ alpha Ġ= 2 $ Ġin Ġthe Ġ$\ alpha Ġ$ - ent max Ġexpression Ġ( Equ ation ĠDIS PLAY _ FORM 85 ); Ġfor Ġother Ġvalues Ġof Ġ$\ alpha Ġ$ , Ġthe Ġexponent Ġinduces Ġcurv ature . </s> On Ġthe Ġother Ġhand , Ġthe Ġwell - known Ġsoft max Ġis Ġusually Ġdefined Ġthrough Ġthe Ġexpression </s> which Ġcan Ġbe Ġshown Ġto Ġbe Ġthe Ġunique Ġsolution Ġof Ġthe Ġoptimization Ġproblem </s> where Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } (\ math bf Ġ{ p }) Ġ- \ sum Ġ_ i Ġp _ i Ġ\ log Ġp _ i $ Ġis Ġthe ĠShannon Ġentropy . ĠIndeed , Ġsetting Ġthe Ġgradient Ġto Ġ0 Ġyields Ġthe Ġcondition Ġ$\ log Ġp _ i Ġ= Ġz _ j Ġ- Ġ\ nu Ġ_ i Ġ- Ġ\ t au Ġ- Ġ1 $, Ġwhere Ġ$\ t au Ġ$ Ġand Ġ$\ nu Ġ> Ġ0 $ Ġare ĠLag range Ġmultipl iers Ġfor Ġthe Ġsimple x Ġconstraints Ġ$\ sum Ġ_ i Ġp _ i Ġ= Ġ1 $ Ġand Ġ$ p _ i Ġ\ ge Ġ0 $, Ġrespectively . ĠSince Ġthe Ġl . h . s . Ġis Ġonly Ġfinite Ġfor Ġ$ p _ i > 0 $, Ġwe Ġmust Ġhave Ġ$\ nu Ġ_ i = 0 $ Ġfor Ġall Ġ$ i $, Ġby Ġcomplementary Ġslack ness . ĠThus , Ġthe Ġsolution Ġmust Ġhave Ġthe Ġform Ġ$ p _ i Ġ= Ġ{\ exp Ġ( z _ i ) }{ Z } $, Ġyielding ĠEqu ation ĠDIS PLAY _ FORM 92 . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ </s> Rec all Ġthat Ġthe Ġent max Ġtransformation Ġis Ġdefined Ġas : </s> where Ġ$\ alpha Ġ\ ge Ġ1 $ Ġand Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _{ \ alpha Ġ} $ Ġis Ġthe ĠTs all is Ġentropy , </s> and Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } (\ math bf Ġ{ p } ): = Ġ- \ sum Ġ_ j Ġp _ j Ġ\ log Ġp _ j $ Ġis Ġthe ĠShannon Ġentropy . </s> In Ġthis Ġsection , Ġwe Ġderive Ġthe ĠJacob ian Ġof Ġ$\ oper at orn am ew ith limits {\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġrespect Ġto Ġthe Ġscal ar Ġparameter Ġ$\ alpha Ġ$ . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠGeneral Ġcase Ġof Ġ@ ! ST ART @ $ \ alpha Ġ> 1 $ @ ! END @ </s> From Ġthe ĠK KT Ġconditions Ġassociated Ġwith Ġthe Ġoptimization Ġproblem Ġin ĠE q . ĠDIS PLAY _ FORM 85 , Ġwe Ġhave Ġthat Ġthe Ġsolution Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $ Ġhas Ġthe Ġfollowing Ġform , Ġcoordinate - wise : </s> where Ġ$\ t au Ġ^ {\ star Ġ} $ Ġis Ġa Ġscal ar ĠLag range Ġmultiplier Ġthat Ġensures Ġthat Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $ Ġnormal izes Ġto Ġ1 , Ġi . e ., Ġit Ġis Ġdefined Ġimplicitly Ġby Ġthe Ġcondition : </s> For Ġgeneral Ġvalues Ġof Ġ$\ alpha Ġ$ , ĠE q . ĠDIS PLAY _ FORM 98 Ġlacks Ġa Ġclosed Ġform Ġsolution . ĠThis Ġmakes Ġthe Ġcomputation Ġof Ġthe ĠJacob ian </s> non - t riv ial . ĠFortunately , Ġwe Ġcan Ġuse Ġthe Ġtechnique Ġof Ġimplicit Ġdifferentiation Ġto Ġobtain Ġthis ĠJacob ian . </s> The ĠJacob ian Ġexists Ġalmost Ġeverywhere , Ġand Ġthe Ġexpressions Ġwe Ġderive Ġexpressions Ġyield Ġa Ġgeneralized ĠJacob ian ĠB IB REF 37 Ġat Ġany Ġnon - different iable Ġpoints Ġthat Ġmay Ġoccur Ġfor Ġcertain Ġ($ \ alpha Ġ$ , Ġ$\ math bf Ġ{ z } $ ) Ġpairs . ĠWe Ġbegin Ġby Ġnoting Ġthat Ġ$\ frac {\ partial Ġp _ i ^ {\ star Ġ} }{ \ partial Ġ\ alpha Ġ} Ġ= Ġ0 $ Ġif Ġ$ p _ i ^ {\ star Ġ} Ġ= Ġ0 $, Ġbecause Ġincreasing Ġ$\ alpha Ġ$ Ġkeeps Ġsparse Ġcoordinates Ġsparse . ĠTherefore Ġwe Ġneed Ġto Ġworry Ġonly Ġabout Ġcoordinates Ġthat Ġare Ġin Ġthe Ġsupport Ġof Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ . ĠWe Ġwill Ġassume Ġhereafter Ġthat Ġthe Ġ$ i $ th Ġcoordinate Ġof Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ Ġis Ġnon - zero . ĠWe Ġhave : </s> We Ġcan Ġsee Ġthat Ġthis ĠJacob ian Ġdepends Ġon Ġ$\ frac {\ partial Ġ\ t au Ġ^ {\ star Ġ} }{ \ partial Ġ\ alpha Ġ} $, Ġwhich Ġwe Ġnow Ġcompute Ġusing Ġimplicit Ġdifferentiation . </s> Let Ġ$\ math cal Ġ{ S } Ġ= Ġ\ l brace Ġi : Ġp ^ \ star Ġ_ i Ġ> Ġ0 Ġ\ r brace Ġ$ ). ĠBy Ġdifferent iating Ġboth Ġsides Ġof ĠE q . ĠDIS PLAY _ FORM 98 , Ġre - using Ġsome Ġof Ġthe Ġsteps Ġin ĠE q . ĠDIS PLAY _ FORM 101 , Ġand Ġrecalling ĠE q . ĠDIS PLAY _ FORM 97 , Ġwe Ġget </s> from Ġwhich Ġwe Ġobtain : </s> Finally , Ġplug ging ĠE q . ĠDIS PLAY _ FORM 103 Ġinto ĠE q . ĠDIS PLAY _ FORM 101 , Ġwe Ġget : </s> where Ġwe Ġdenote Ġby </s> The Ġdistribution Ġ$\ t ilde {\ math bf Ġ{ p }} (\ alpha Ġ) $ Ġcan Ġbe Ġinterpreted Ġas Ġa ĠâĢ ľ s ke wed âĢ Ŀ Ġdistribution Ġobtained Ġfrom Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $, Ġwhich Ġappears Ġin Ġthe ĠJacob ian Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġw . r . t . Ġ$\ math bf Ġ{ z } $ Ġas Ġwell ĠB IB REF 14 . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠSol ving Ġthe Ġind etermination Ġfor Ġ@ ! ST ART @ $ \ alpha Ġ= 1 $ @ ! END @ </s> We Ġcan Ġwrite ĠE q . ĠDIS PLAY _ FORM 104 Ġas </s> When Ġ$\ alpha Ġ\ right arrow Ġ1 ^ + $, Ġwe Ġhave Ġ$\ t ilde {\ math bf Ġ{ p }} (\ alpha Ġ) Ġ\ right arrow Ġ\ math bf Ġ{ p } ^ {\ star Ġ} $, Ġwhich Ġleads Ġto Ġa Ġ$\ frac { 0 }{ 0 } $ Ġind etermination . </s> To Ġsolve Ġthis Ġind etermination , Ġwe Ġwill Ġneed Ġto Ġapply ĠL ' H Ã´ p ital 's Ġrule Ġtwice . ĠLet Ġus Ġfirst Ġcompute Ġthe Ġderivative Ġof Ġ$\ t ilde { p } _ i (\ alpha Ġ) $ Ġwith Ġrespect Ġto Ġ$\ alpha Ġ$ . ĠWe Ġhave </s> there fore </s> Different iating Ġthe Ġnumer ator Ġand Ġdenomin ator Ġin ĠE q . ĠDIS PLAY _ FORM 107 , Ġwe Ġget : </s> with </s> and </s> When Ġ$\ alpha Ġ\ right arrow Ġ1 ^ + $, Ġ$ B $ Ġbecomes Ġagain Ġa Ġ$\ frac { 0 }{ 0 } $ Ġind etermination , Ġwhich Ġwe Ġcan Ġsolve Ġby Ġapplying Ġagain ĠL ' H Ã´ p ital 's Ġrule . ĠDifferent iating Ġthe Ġnumer ator Ġand Ġdenomin ator Ġin ĠE q . ĠDIS PLAY _ FORM 112 : </s> Finally , Ġsum ming ĠE q . ĠDIS PLAY _ FORM 111 Ġand ĠE q . ĠDIS PLAY _ FORM 113 , Ġwe Ġget </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠSummary </s> To Ġsum Ġup , Ġwe Ġhave Ġthe Ġfollowing Ġexpression Ġfor Ġthe ĠJacob ian Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġrespect Ġto Ġ$\ alpha Ġ$ :
Introduction The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4. At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context. As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9. The attention distribution of each head is predicted typically using the softmax normalizing transform. As a result, all context words have non-zero attention weight. Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14. Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions. Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11. However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous. Figure FIGREF1 shows the relationship of these methods with ours. Our contributions are the following: We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains. We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case. We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model. Background ::: The Transformer In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences. It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17. Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items. The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way: where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations. The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where In words, the keys are used to compute a relevance score between each item and query. Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context. However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers. This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization: In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes: Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence. Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder. Decoder self-attention: attends over the partial output sentence fragment produced so far. Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder. Background ::: Sparse Attention The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero. Thus, unnecessary items are still taken into consideration to some extent. Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18. This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22. We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as: where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24: This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively. Equation DISPLAY_FORM14 involves a convex optimization subproblem. Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83): where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint. Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax. The appeal of $\alpha $-entmax for attention rests on the following properties. For $\alpha =1$ (i.e., when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89.). For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax. In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear. In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes. To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r.h.s. in Equation DISPLAY_FORM16 sums to one. BIBREF23 propose a general bisection algorithm. BIBREF14 introduce a faster, exact algorithm for $\alpha =1.5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w.r.t. $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w.r.t. the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically. This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors. Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads. Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1.5$ is a sensible starting point BIBREF14. Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head. Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers. Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0. This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax. We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights. Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w.r.t. @!START@$\alpha $@!END@. In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w.r.t. $\alpha $. Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26. One of our key contributions is the derivation of a closed-form expression for this Jacobian. The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $. To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax. Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14. Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$. The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is proof uses implicit function differentiation and is given in Appendix SECREF10. Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers. In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads. Experiments We apply our adaptively sparse Transformers on four machine translation tasks. For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms. We consider two other model variants in our experiments that make use of different normalizing transformations: 1.5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1.5$ for all heads. This is a novel model, since 1.5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components. $\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head. The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i.e., and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$. All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes. Experiments ::: Datasets. Our models were trained on 4 machine translation datasets of different training sizes: [itemsep=.5ex,leftmargin=2ex] IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs. KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs. WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs. WMT 2014 English $\rightarrow $ German BIBREF30: 4.5M sentence pairs. All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations. Experiments ::: Training. We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention. We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule. All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en. The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1.5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model. Experiments ::: Results. We report test set tokenized BLEU BIBREF32 results in Table TABREF27. We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis. In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads. Analysis We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior. Moreover, we make a qualitative analysis of the interpretability capabilities of our models. Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned? Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads. We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain. After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized. This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout. In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1.3$ before becoming one of the sparsest heads, with $\alpha \approx 2$. The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38. We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1.5-entmax . However, the decoder self and context attention only learn to distribute these parameters in a single mode. We show next that this is reflected in the average density of attention weight vectors as well. Analysis ::: High-Level Statistics ::: Attention weight density when translating. For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser. To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1.5$. Figure FIGREF40 shows that, with fixed $\alpha =1.5$, heads tend to be sparse and similarly-distributed in all three attention modules. With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions. This suggests that perhaps entirely sparse Transformers are suboptimal. The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence. We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place. Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer. We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1.5$. Analysis ::: High-Level Statistics ::: Head diversity. To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence: where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$. We average this measure over the entire validation set. The higher this metric is, the more the heads are taking different roles in the model. Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one. Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention. The statistics shown in this section can be found for the other language pairs in Appendix SECREF8. Analysis ::: Identifying Head Specializations Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9. Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity. Analysis ::: Identifying Head Specializations ::: Positional heads. One particular type of head, as noted by BIBREF9, is the positional head. These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step. In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models. The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence. Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token. The softmax model has three heads for position $-1$, with median confidence $93.5\%$. The $1.5$-entmax model also has three heads for this position, with median confidence $94.4\%$. The adaptive model has four heads, with median confidences $95.9\%$, the lowest-confidence head being dense with $\alpha =1.18$, while the highest-confidence head being sparse ($\alpha =1.91$). For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax. The adaptive model sets $\alpha =1.96$ for this head. Analysis ::: Identifying Head Specializations ::: BPE-merging head. Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed. In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37). We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0. However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words. As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters. For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads. There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1.5$-entmax and $\alpha $-entmax there are two heads in each ($83.3\%$ and $85.6\%$ for $1.5$-entmax and $88.5\%$ and $89.8\%$ for $\alpha $-entmax). Analysis ::: Identifying Head Specializations ::: Interrogation head. On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37). We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head. We also observe this type of heads in the other models, which we also depict in Figure FIGREF52. The average attention weight placed on the question mark when the current token is an interrogative word is $98.5\%$ for softmax, $97.0\%$ for $1.5$-entmax, and $99.5\%$ for $\alpha $-entmax. Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction. An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length. Related Work ::: Sparse attention. Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34. BIBREF14 introduced the entmax function this work builds upon. In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $. In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter. We also provide the first results using sparse attention in a Transformer model. Related Work ::: Fixed sparsity patterns. Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35. Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e.g., Figure FIGREF52). Moreover, the two strategies could be combined. In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models. While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span. Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers. Related Work ::: Transformer interpretability. The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have. BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights. BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline. Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words). BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results. While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights. Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions. Conclusion and Future Work We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers. We present the first empirical analysis of Transformers with sparse attention mappings (i.e., entmax), showing potential in both translation accuracy as well as in model interpretability. In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence. Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches. Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax. Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer. Acknowledgments This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal). We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion. We thank Mathieu Blondel for the idea to learn $\alpha $. We would also like to thank the anonymous reviewers for their helpful feedback. Supplementary Material Background ::: Regularized Fenchel-Young prediction functions Definition 1 (BIBREF23) Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function. We define the prediction function $\mathbf {\pi }_{\Omega }$ as Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$, we may easily identify it with a regularized prediction function (Def. UNKREF81): We first note that for all $\mathbf {p}\in \triangle ^d$, From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23, Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $. Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique. Background ::: Connections to softmax and sparsemax The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38 Thus, each coordinate of the sparsemax solution is a piecewise-linear function. Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature. On the other hand, the well-known softmax is usually defined through the expression which can be shown to be the unique solution of the optimization problem where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy. Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively. Since the l.h.s. is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness. Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92. Jacobian of @!START@$\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ Recall that the entmax transformation is defined as: where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy, and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy. In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $. Jacobian of @!START@$\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@ From the KKT conditions associated with the optimization problem in Eq. DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise: where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i.e., it is defined implicitly by the condition: For general values of $\alpha $, Eq. DISPLAY_FORM98 lacks a closed form solution. This makes the computation of the Jacobian non-trivial. Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian. The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs. We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse. Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $. We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero. We have: We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation. Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $). By differentiating both sides of Eq. DISPLAY_FORM98, re-using some of the steps in Eq. DISPLAY_FORM101, and recalling Eq. DISPLAY_FORM97, we get from which we obtain: Finally, plugging Eq. DISPLAY_FORM103 into Eq. DISPLAY_FORM101, we get: where we denote by The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w.r.t. $\mathbf {z}$ as well BIBREF14. Jacobian of @!START@$\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@ We can write Eq. DISPLAY_FORM104 as When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination. To solve this indetermination, we will need to apply L'Hôpital's rule twice. Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $. We have therefore Differentiating the numerator and denominator in Eq. DISPLAY_FORM107, we get: with and When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule. Differentiating the numerator and denominator in Eq. DISPLAY_FORM112: Finally, summing Eq. DISPLAY_FORM111 and Eq. DISPLAY_FORM113, we get Jacobian of @!START@$\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:
HOw does the method perform compared with baselines?
f8c1b17d265a61502347c9a937269b38fc3fcab1
HO w Ġdoes Ġthe Ġmethod Ġperform Ġcompared Ġwith Ġbas elines ?
On Ġthe Ġdatasets ĠDE - EN , ĠJA - EN , ĠRO - EN , Ġand ĠEN - DE , Ġthe Ġbaseline Ġachieves Ġ29 . 79 , Ġ21 . 57 , Ġ32 . 70 , Ġand Ġ26 . 02 Ġ ĠB LE U Ġscore , Ġrespectively . ĠThe Ġ1 . 5 - ent max Ġachieves Ġ Ġ29 . 83 , Ġ22 . 13 , Ġ33 . 10 , Ġand Ġ25 . 89 ĠB LE U Ġscore , Ġwhich Ġis Ġa Ġdifference Ġof Ġ+ 0 . 04 , Ġ+ 0 . 56 , Ġ+ 0 . 40 , Ġand Ġ- 0 . 13 ĠB LE U Ġscore Ġversus Ġthe Ġbaseline . ĠThe ĠÎ± - ent max Ġachieves Ġ29 . 90 , Ġ21 . 74 , Ġ32 . 89 , Ġand Ġ26 . 93 ĠB LE U Ġscore , Ġwhich Ġis Ġa Ġdifference Ġof Ġ+ 0 . 11 , Ġ+ 0 . 17 , Ġ+ 0 . 19 , Ġ+ 0 . 91 ĠB LE U Ġscore Ġversus Ġthe Ġbaseline .
<s> HO w Ġdoes Ġthe Ġmethod Ġperform Ġcompared Ġwith Ġbas elines ? </s> Introduction </s> The ĠTrans former Ġarchitecture ĠB IB REF 0 Ġfor Ġdeep Ġneural Ġnetworks Ġhas Ġquickly Ġrisen Ġto Ġprominence Ġin ĠN LP Ġthrough Ġits Ġefficiency Ġand Ġperformance , Ġleading Ġto Ġimprovements Ġin Ġthe Ġstate Ġof Ġthe Ġart Ġof ĠNeural ĠMachine ĠTranslation ĠB IB REF 1 , ĠB IB REF 2 , Ġas Ġwell Ġas Ġinspiring Ġother Ġpowerful Ġgeneral - purpose Ġmodels Ġlike ĠB ERT ĠB IB REF 3 Ġand ĠG PT - 2 ĠB IB REF 4 . ĠAt Ġthe Ġheart Ġof Ġthe ĠTrans former Ġlie Ġmulti - head Ġattention Ġmechanisms : Ġeach Ġword Ġis Ġrepresented Ġby Ġmultiple Ġdifferent Ġweighted Ġaverages Ġof Ġits Ġrelevant Ġcontext . ĠAs Ġsuggested Ġby Ġrecent Ġworks Ġon Ġinterpreting Ġattention Ġhead Ġroles , Ġseparate Ġattention Ġheads Ġmay Ġlearn Ġto Ġlook Ġfor Ġvarious Ġrelationships Ġbetween Ġtokens ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 . </s> The Ġattention Ġdistribution Ġof Ġeach Ġhead Ġis Ġpredicted Ġtypically Ġusing Ġthe Ġsoft max Ġnormal izing Ġtransform . ĠAs Ġa Ġresult , Ġall Ġcontext Ġwords Ġhave Ġnon - zero Ġattention Ġweight . ĠRecent Ġwork Ġon Ġsingle Ġattention Ġarchitectures Ġsuggest Ġthat Ġusing Ġsparse Ġnormal izing Ġtransforms Ġin Ġattention Ġmechanisms Ġsuch Ġas Ġsparse max ĠâĢĵ Ġwhich Ġcan Ġyield Ġexactly Ġzero Ġprobabilities Ġfor Ġirrelevant Ġwords ĠâĢĵ Ġmay Ġimprove Ġperformance Ġand Ġinterpret ability ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 . ĠQual itative Ġanalysis Ġof Ġattention Ġheads ĠB IB REF 0 Ġsuggests Ġthat , Ġdepending Ġon Ġwhat Ġphenomena Ġthey Ġcapture , Ġheads Ġtend Ġto Ġfavor Ġfl atter Ġor Ġmore Ġpeaked Ġdistributions . </s> Recent Ġworks Ġhave Ġproposed Ġsparse ĠTransformers ĠB IB REF 10 Ġand Ġadaptive Ġspan ĠTransformers ĠB IB REF 11 . ĠHowever , Ġthe ĠâĢ ľ sp arsity " Ġof Ġthose Ġmodels Ġonly Ġlimits Ġthe Ġattention Ġto Ġa Ġcontiguous Ġspan Ġof Ġpast Ġtokens , Ġwhile Ġin Ġthis Ġwork Ġwe Ġpropose Ġa Ġhighly Ġadaptive ĠTrans former Ġmodel Ġthat Ġis Ġcapable Ġof Ġattending Ġto Ġa Ġsparse Ġset Ġof Ġwords Ġthat Ġare Ġnot Ġnecessarily Ġcontiguous . ĠFigure ĠFIG REF 1 Ġshows Ġthe Ġrelationship Ġof Ġthese Ġmethods Ġwith Ġours . </s> Our Ġcontributions Ġare Ġthe Ġfollowing : </s> We Ġintroduce Ġsparse Ġattention Ġinto Ġthe ĠTrans former Ġarchitecture , Ġshowing Ġthat Ġit Ġe ases Ġinterpret ability Ġand Ġleads Ġto Ġslight Ġaccuracy Ġgains . </s> We Ġpropose Ġan Ġadaptive Ġversion Ġof Ġsparse Ġattention , Ġwhere Ġthe Ġshape Ġof Ġeach Ġattention Ġhead Ġis Ġlearn able Ġand Ġcan Ġvary Ġcontinuously Ġand Ġdynamically Ġbetween Ġthe Ġdense Ġlimit Ġcase Ġof Ġsoft max Ġand Ġthe Ġsparse , Ġpiece wise - linear Ġsparse max Ġcase . </s> We Ġmake Ġan Ġextensive Ġanalysis Ġof Ġthe Ġadded Ġinterpret ability Ġof Ġthese Ġmodels , Ġidentifying Ġboth Ġcris per Ġexamples Ġof Ġattention Ġhead Ġbehavior Ġobserved Ġin Ġprevious Ġwork , Ġas Ġwell Ġas Ġnovel Ġbehaviors Ġunravel ed Ġthanks Ġto Ġthe Ġsp arsity Ġand Ġadapt ivity Ġof Ġour Ġproposed Ġmodel . </s> Background Ġ: :: ĠThe ĠTrans former </s> In ĠN MT , Ġthe ĠTrans former ĠB IB REF 0 Ġis Ġa Ġsequence - to - sequence Ġ( seq 2 seq ) Ġmodel Ġwhich Ġmaps Ġan Ġinput Ġsequence Ġto Ġan Ġoutput Ġsequence Ġthrough Ġhierarchical Ġmulti - head Ġattention Ġmechanisms , Ġyielding Ġa Ġdynamic , Ġcontext - dependent Ġstrategy Ġfor Ġpropag ating Ġinformation Ġwithin Ġand Ġacross Ġsentences . ĠIt Ġcontrasts Ġwith Ġprevious Ġseq 2 seq Ġmodels , Ġwhich Ġusually Ġrely Ġeither Ġon Ġcostly Ġg ated Ġrecurrent Ġoperations ĠB IB REF 15 , ĠB IB REF 16 Ġor Ġstatic Ġconv olutions ĠB IB REF 17 . </s> Given Ġ$ n $ Ġquery Ġcontexts Ġand Ġ$ m $ Ġsequence Ġitems Ġunder Ġconsideration , Ġattention Ġmechanisms Ġcompute , Ġfor Ġeach Ġquery , Ġa Ġweighted Ġrepresentation Ġof Ġthe Ġitems . ĠThe Ġparticular Ġattention Ġmechanism Ġused Ġin ĠB IB REF 0 Ġis Ġcalled Ġscaled Ġdot - product Ġattention , Ġand Ġit Ġis Ġcomputed Ġin Ġthe Ġfollowing Ġway : </s> where Ġ$\ math bf Ġ{ Q } Ġ\ in Ġ\ math bb Ġ{ R } ^{ n Ġ\ times Ġd } $ Ġcontains Ġrepresentations Ġof Ġthe Ġqueries , Ġ$\ math bf Ġ{ K }, Ġ\ math bf Ġ{ V } Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġd } $ Ġare Ġthe Ġkeys Ġand Ġvalues Ġof Ġthe Ġitems Ġattended Ġover , Ġand Ġ$ d $ Ġis Ġthe Ġdimension ality Ġof Ġthese Ġrepresentations . ĠThe Ġ$\ math bf Ġ{\ pi Ġ} $ Ġmapping Ġnormal izes Ġrow - wise Ġusing Ġsoft max , Ġ$\ math bf Ġ{\ pi Ġ} (\ math bf Ġ{ Z }) _{ ij } Ġ= Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ soft max }} (\ math bf Ġ{ z } _ i ) _ j $, Ġwhere </s> In Ġwords , Ġthe Ġkeys Ġare Ġused Ġto Ġcompute Ġa Ġrelevance Ġscore Ġbetween Ġeach Ġitem Ġand Ġquery . ĠThen , Ġnormalized Ġattention Ġweights Ġare Ġcomputed Ġusing Ġsoft max , Ġand Ġthese Ġare Ġused Ġto Ġweight Ġthe Ġvalues Ġof Ġeach Ġitem Ġat Ġeach Ġquery Ġcontext . </s> However , Ġfor Ġcomplex Ġtasks , Ġdifferent Ġparts Ġof Ġa Ġsequence Ġmay Ġbe Ġrelevant Ġin Ġdifferent Ġways , Ġmotivating Ġmulti - head Ġattention Ġin ĠTransformers . ĠThis Ġis Ġsimply Ġthe Ġapplication Ġof ĠEqu ation ĠDIS PLAY _ FORM 7 Ġin Ġparallel Ġ$ H $ Ġtimes , Ġeach Ġwith Ġa Ġdifferent , Ġlearned Ġlinear Ġtransformation Ġthat Ġallows Ġspecialization : </s> In Ġthe ĠTrans former , Ġthere Ġare Ġthree Ġseparate Ġmulti - head Ġattention Ġmechanisms Ġfor Ġdistinct Ġpurposes : </s> Enc oder Ġself - att ention : Ġbuilds Ġrich , Ġlayered Ġrepresentations Ġof Ġeach Ġinput Ġword , Ġby Ġattending Ġon Ġthe Ġentire Ġinput Ġsentence . </s> Context Ġattention : Ġselects Ġa Ġrepresentative Ġweighted Ġaverage Ġof Ġthe Ġenc od ings Ġof Ġthe Ġinput Ġwords , Ġat Ġeach Ġtime Ġstep Ġof Ġthe Ġdec oder . </s> Dec oder Ġself - att ention : Ġattends Ġover Ġthe Ġpartial Ġoutput Ġsentence Ġfragment Ġproduced Ġso Ġfar . </s> Together , Ġthese Ġmechanisms Ġenable Ġthe Ġcontextual ized Ġflow Ġof Ġinformation Ġbetween Ġthe Ġinput Ġsentence Ġand Ġthe Ġsequential Ġdec oder . </s> Background Ġ: :: ĠSp arse ĠAttention </s> The Ġsoft max Ġmapping Ġ( Equ ation ĠDIS PLAY _ FORM 8 ) Ġis Ġelement wise Ġproportional Ġto Ġ$\ exp Ġ$ , Ġtherefore Ġit Ġcan Ġnever Ġassign Ġa Ġweight Ġof Ġexactly Ġzero . ĠThus , Ġunnecessary Ġitems Ġare Ġstill Ġtaken Ġinto Ġconsideration Ġto Ġsome Ġextent . ĠSince Ġits Ġoutput Ġsums Ġto Ġone , Ġthis Ġinvariably Ġmeans Ġless Ġweight Ġis Ġassigned Ġto Ġthe Ġrelevant Ġitems , Ġpotentially Ġharming Ġperformance Ġand Ġinterpret ability ĠB IB REF 18 . ĠThis Ġhas Ġmotivated Ġa Ġline Ġof Ġresearch Ġon Ġlearning Ġnetworks Ġwith Ġsparse Ġm appings ĠB IB REF 19 , ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 . ĠWe Ġfocus Ġon Ġa Ġrecently - introdu ced Ġflexible Ġfamily Ġof Ġtransformations , Ġ$\ alpha Ġ$ - ent max ĠB IB REF 23 , ĠB IB REF 14 , Ġdefined Ġas : </s> where Ġ$\ tri angle Ġ^ d Ġ\ l brace Ġ\ math bf Ġ{ p }\ in Ġ\ math bb Ġ{ R } ^ d :\ sum Ġ_ { i } Ġp _ i Ġ= Ġ1 \ r brace Ġ$ Ġis Ġthe Ġprobability Ġsimple x , Ġand , Ġfor Ġ$\ alpha Ġ\ ge Ġ1 $, Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġis Ġthe ĠTs all is Ġcontinuous Ġfamily Ġof Ġent rop ies ĠB IB REF 24 : </s> This Ġfamily Ġcontains Ġthe Ġwell - known ĠShannon Ġand ĠG ini Ġent rop ies , Ġcorresponding Ġto Ġthe Ġcases Ġ$\ alpha Ġ= 1 $ Ġand Ġ$\ alpha Ġ= 2 $, Ġrespectively . </s> Equ ation ĠDIS PLAY _ FORM 14 Ġinvolves Ġa Ġconve x Ġoptimization Ġsub problem . ĠUsing Ġthe Ġdefinition Ġof Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ , Ġthe Ġoptim ality Ġconditions Ġmay Ġbe Ġused Ġto Ġderive Ġthe Ġfollowing Ġform Ġfor Ġthe Ġsolution Ġ( App endix ĠSEC REF 83 ): </s> where Ġ$ [ \ cd ot Ġ] _ + $ Ġis Ġthe Ġpositive Ġpart Ġ( Re LU ) Ġfunction , Ġ$\ math bf Ġ{ 1 } $ Ġdenotes Ġthe Ġvector Ġof Ġall Ġones , Ġand Ġ$\ t au Ġ$ ĠâĢĵ Ġwhich Ġacts Ġlike Ġa Ġthreshold ĠâĢĵ Ġis Ġthe ĠLag range Ġmultiplier Ġcorresponding Ġto Ġthe Ġ$\ sum Ġ_ i Ġp _ i = 1 $ Ġconstraint . </s> Background Ġ: :: ĠSp arse ĠAttention Ġ: :: ĠProperties Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max . </s> The Ġappeal Ġof Ġ$\ alpha Ġ$ - ent max Ġfor Ġattention Ġrests Ġon Ġthe Ġfollowing Ġproperties . ĠFor Ġ$\ alpha Ġ= 1 $ Ġ( i . e ., Ġwhen Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġbecomes Ġthe ĠShannon Ġentropy ), Ġit Ġexactly Ġrecovers Ġthe Ġsoft max Ġmapping Ġ( We Ġprovide Ġa Ġshort Ġderiv ation Ġin ĠAppendix ĠSEC REF 89 .). ĠFor Ġall Ġ$\ alpha Ġ> 1 $ Ġit Ġpermits Ġsparse Ġsolutions , Ġin Ġstark Ġcontrast Ġto Ġsoft max . ĠIn Ġparticular , Ġfor Ġ$\ alpha Ġ= 2 $, Ġit Ġrecovers Ġthe Ġsparse max Ġmapping ĠB IB REF 19 , Ġwhich Ġis Ġpiece wise Ġlinear . ĠIn - between , Ġas Ġ$\ alpha Ġ$ Ġincreases , Ġthe Ġmapping Ġcontinuously Ġgets Ġsp arser Ġas Ġits Ġcurv ature Ġchanges . </s> To Ġcompute Ġthe Ġvalue Ġof Ġ$\ alpha Ġ$ - ent max , Ġone Ġmust Ġfind Ġthe Ġthreshold Ġ$\ t au Ġ$ Ġsuch Ġthat Ġthe Ġr . h . s . Ġin ĠEqu ation ĠDIS PLAY _ FORM 16 Ġsums Ġto Ġone . ĠB IB REF 23 Ġpropose Ġa Ġgeneral Ġbis ection Ġalgorithm . ĠB IB REF 14 Ġintroduce Ġa Ġfaster , Ġexact Ġalgorithm Ġfor Ġ$\ alpha Ġ= 1 . 5 $, Ġand Ġenable Ġusing Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġfixed Ġ$\ alpha Ġ$ Ġwithin Ġa Ġneural Ġnetwork Ġby Ġshowing Ġthat Ġthe Ġ$\ alpha Ġ$ - ent max ĠJacob ian Ġw . r . t . Ġ$\ math bf Ġ{ z } $ Ġfor Ġ$\ math bf Ġ{ p } ^ \ star Ġ= Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġis </s> Our Ġwork Ġfurther s Ġthe Ġstudy Ġof Ġ$\ alpha Ġ$ - ent max Ġby Ġproviding Ġa Ġderiv ation Ġof Ġthe ĠJacob ian Ġw . r . t . Ġthe Ġhyper - param eter Ġ$\ alpha Ġ$ Ġ( Section ĠSEC REF 3 ), Ġthereby Ġallowing Ġthe Ġshape Ġand Ġsp arsity Ġof Ġthe Ġmapping Ġto Ġbe Ġlearned Ġautomatically . ĠThis Ġis Ġparticularly Ġappealing Ġin Ġthe Ġcontext Ġof Ġmulti - head Ġattention Ġmechanisms , Ġwhere Ġwe Ġshall Ġshow Ġin ĠSection ĠSEC REF 35 Ġthat Ġdifferent Ġheads Ġtend Ġto Ġlearn Ġdifferent Ġsp arsity Ġbehaviors . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max </s> We Ġnow Ġpropose Ġa Ġnovel ĠTrans former Ġarchitecture Ġwherein Ġwe Ġsimply Ġreplace Ġsoft max Ġwith Ġ$\ alpha Ġ$ - ent max Ġin Ġthe Ġattention Ġheads . ĠCon crete ly , Ġwe Ġreplace Ġthe Ġrow Ġnormal ization Ġ$\ math bf Ġ{\ pi Ġ} $ Ġin ĠEqu ation ĠDIS PLAY _ FORM 7 Ġby </s> This Ġchange Ġleads Ġto Ġsparse Ġattention Ġweights , Ġas Ġlong Ġas Ġ$\ alpha Ġ> 1 $ ; Ġin Ġparticular , Ġ$\ alpha Ġ= 1 . 5 $ Ġis Ġa Ġsensible Ġstarting Ġpoint ĠB IB REF 14 . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġ: :: ĠDifferent Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ Ġper Ġhead . </s> Unlike ĠL ST M - based Ġseq 2 seq Ġmodels , Ġwhere Ġ$\ alpha Ġ$ Ġcan Ġbe Ġmore Ġeasily Ġtuned Ġby Ġgrid Ġsearch , Ġin Ġa ĠTrans former , Ġthere Ġare Ġmany Ġattention Ġheads Ġin Ġmultiple Ġlayers . ĠCru cial Ġto Ġthe Ġpower Ġof Ġsuch Ġmodels , Ġthe Ġdifferent Ġheads Ġcapture Ġdifferent Ġlinguistic Ġphenomena , Ġsome Ġof Ġthem Ġisol ating Ġimportant Ġwords , Ġothers Ġspreading Ġout Ġattention Ġacross Ġphrases ĠB IB REF 0 . ĠThis Ġmotiv ates Ġusing Ġdifferent , Ġadaptive Ġ$\ alpha Ġ$ Ġvalues Ġfor Ġeach Ġattention Ġhead , Ġsuch Ġthat Ġsome Ġheads Ġmay Ġlearn Ġto Ġbe Ġsp arser , Ġand Ġothers Ġmay Ġbecome Ġcloser Ġto Ġsoft max . ĠWe Ġpropose Ġdoing Ġso Ġby Ġtreating Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġas Ġneural Ġnetwork Ġparameters , Ġoptimized Ġvia Ġsto ch astic Ġgrad ients Ġalong Ġwith Ġthe Ġother Ġweights . </s> Adapt ively ĠSp arse ĠTransformers Ġwith Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġ: :: ĠDer iv atives Ġw . r . t . Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ . </s> In Ġorder Ġto Ġoptimize Ġ$\ alpha Ġ$ Ġautomatically Ġvia Ġgradient Ġmethods , Ġwe Ġmust Ġcompute Ġthe ĠJacob ian Ġof Ġthe Ġent max Ġoutput Ġw . r . t . Ġ$\ alpha Ġ$ . ĠSince Ġent max Ġis Ġdefined Ġthrough Ġan Ġoptimization Ġproblem , Ġthis Ġis Ġnon - t riv ial Ġand Ġcannot Ġbe Ġsimply Ġhandled Ġthrough Ġautomatic Ġdifferentiation ; Ġit Ġfalls Ġwithin Ġthe Ġdomain Ġof Ġarg min Ġdifferentiation , Ġan Ġactive Ġresearch Ġtopic Ġin Ġoptimization ĠB IB REF 25 , ĠB IB REF 26 . </s> One Ġof Ġour Ġkey Ġcontributions Ġis Ġthe Ġderiv ation Ġof Ġa Ġclosed - form Ġexpression Ġfor Ġthis ĠJacob ian . ĠThe Ġnext Ġproposition Ġprovides Ġsuch Ġan Ġexpression , Ġenabling Ġent max Ġlayers Ġwith Ġadaptive Ġ$\ alpha Ġ$ . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġours Ġis Ġthe Ġfirst Ġneural Ġnetwork Ġmodule Ġthat Ġcan Ġautomatically , Ġcontinuously Ġvary Ġin Ġshape Ġaway Ġfrom Ġsoft max Ġand Ġtoward Ġsparse Ġm appings Ġlike Ġsparse max . </s> Prop osition Ġ1 ĠLet Ġ$\ math bf Ġ{ p } ^ \ star Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġbe Ġthe Ġsolution Ġof ĠEqu ation ĠDIS PLAY _ FORM 14 . ĠDen ote Ġthe Ġdistribution Ġ$\ t ilde { p } _ i Ġ{ ( p _ i ^ \ star Ġ) ^{ 2 Ġ- Ġ\ alpha Ġ} }{ Ġ\ sum Ġ_ j ( p _ j ^ \ star Ġ) ^{ 2 - \ alpha Ġ}} $ Ġand Ġlet Ġ$ h _ i Ġ- p ^ \ star Ġ_ i Ġ\ log Ġp ^ \ star Ġ_ i $. ĠThe Ġ$ i $ th Ġcomponent Ġof Ġthe ĠJacob ian Ġ$\ math bf Ġ{ g } Ġ\ frac {\ partial Ġ\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) }{ \ partial Ġ\ alpha Ġ} $ Ġis </s> proof Ġuses Ġimplicit Ġfunction Ġdifferentiation Ġand Ġis Ġgiven Ġin ĠAppendix ĠSEC REF 10 . </s> Prop osition ĠUN K REF 22 Ġprovides Ġthe Ġremaining Ġmissing Ġpiece Ġneeded Ġfor Ġtraining Ġadapt ively Ġsparse ĠTransformers . ĠIn Ġthe Ġfollowing Ġsection , Ġwe Ġevaluate Ġthis Ġstrategy Ġon Ġneural Ġmachine Ġtranslation , Ġand Ġanalyze Ġthe Ġbehavior Ġof Ġthe Ġlearned Ġattention Ġheads . </s> Exper iments </s> We Ġapply Ġour Ġadapt ively Ġsparse ĠTransformers Ġon Ġfour Ġmachine Ġtranslation Ġtasks . ĠFor Ġcomparison , Ġa Ġnatural Ġbaseline Ġis Ġthe Ġstandard ĠTrans former Ġarchitecture Ġusing Ġthe Ġsoft max Ġtransform Ġin Ġits Ġmulti - head Ġattention Ġmechanisms . ĠWe Ġconsider Ġtwo Ġother Ġmodel Ġvariants Ġin Ġour Ġexperiments Ġthat Ġmake Ġuse Ġof Ġdifferent Ġnormal izing Ġtransformations : </s> 1 . 5 - ent max : Ġa ĠTrans former Ġwith Ġsparse Ġent max Ġattention Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $ Ġfor Ġall Ġheads . ĠThis Ġis Ġa Ġnovel Ġmodel , Ġsince Ġ1 . 5 - ent max Ġhad Ġonly Ġbeen Ġproposed Ġfor ĠR NN - based ĠN MT Ġmodels ĠB IB REF 14 , Ġbut Ġnever Ġin ĠTransformers , Ġwhere Ġattention Ġmodules Ġare Ġnot Ġjust Ġone Ġsingle Ġcomponent Ġof Ġthe Ġseq 2 seq Ġmodel Ġbut Ġrather Ġan Ġintegral Ġpart Ġof Ġall Ġof Ġthe Ġmodel Ġcomponents . </s> $ \ alpha Ġ$ - ent max : Ġan Ġadaptive ĠTrans former Ġwith Ġsparse Ġent max Ġattention Ġwith Ġa Ġdifferent , Ġlearned Ġ$\ alpha Ġ_ { i , j } ^ t $ Ġfor Ġeach Ġhead . </s> The Ġadaptive Ġmodel Ġhas Ġan Ġadditional Ġscal ar Ġparameter Ġper Ġattention Ġhead Ġper Ġlayer Ġfor Ġeach Ġof Ġthe Ġthree Ġattention Ġmechanisms Ġ( enc oder Ġself - att ention , Ġcontext Ġattention , Ġand Ġdec oder Ġself - att ention ), Ġi . e ., </s> and Ġwe Ġset Ġ$\ alpha Ġ_ { i , j } ^ t Ġ= Ġ1 Ġ+ Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ s igm oid }} ( a _{ i , j } ^ t ) Ġ\ in Ġ] 1 , Ġ2 [ $. ĠAll Ġor Ġsome Ġof Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġcan Ġbe Ġtied Ġif Ġdesired , Ġbut Ġwe Ġkeep Ġthem Ġindependent Ġfor Ġanalysis Ġpurposes . </s> Exper iments Ġ: :: ĠDat as ets . </s> Our Ġmodels Ġwere Ġtrained Ġon Ġ4 Ġmachine Ġtranslation Ġdatasets Ġof Ġdifferent Ġtraining Ġsizes : </s> [ item se p = . 5 ex , left margin = 2 ex ] </s> I W SL T Ġ2017 ĠGerman Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 27 : Ġ200 K Ġsentence Ġpairs . </s> K FT T ĠJapanese Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 28 : Ġ300 K Ġsentence Ġpairs . </s> W MT Ġ2016 ĠRomanian Ġ$\ right arrow Ġ$ ĠEnglish ĠB IB REF 29 : Ġ600 K Ġsentence Ġpairs . </s> W MT Ġ2014 ĠEnglish Ġ$\ right arrow Ġ$ ĠGerman ĠB IB REF 30 : Ġ4 . 5 M Ġsentence Ġpairs . </s> All Ġof Ġthese Ġdatasets Ġwere Ġpre process ed Ġwith Ġbyte - pair Ġencoding ĠB IB REF 31 , Ġusing Ġjoint Ġsegment ations Ġof Ġ32 k Ġmerge Ġoperations . </s> Exper iments Ġ: :: ĠTraining . </s> We Ġfollow Ġthe Ġdimensions Ġof Ġthe ĠTrans former - Base Ġmodel Ġof ĠB IB REF 0 : ĠThe Ġnumber Ġof Ġlayers Ġis Ġ$ L = 6 $ Ġand Ġnumber Ġof Ġheads Ġis Ġ$ H = 8 $ Ġin Ġthe Ġenc oder Ġself - att ention , Ġthe Ġcontext Ġattention , Ġand Ġthe Ġdec oder Ġself - att ention . ĠWe Ġuse Ġa Ġmini - batch Ġsize Ġof Ġ8 192 Ġtokens Ġand Ġwarm Ġup Ġthe Ġlearning Ġrate Ġlin early Ġuntil Ġ20 k Ġsteps , Ġafter Ġwhich Ġit Ġdec ays Ġaccording Ġto Ġan Ġinverse Ġsquare Ġroot Ġschedule . ĠAll Ġmodels Ġwere Ġtrained Ġuntil Ġconvergence Ġof Ġvalidation Ġaccuracy , Ġand Ġevaluation Ġwas Ġdone Ġat Ġeach Ġ10 k Ġsteps Ġfor Ġro $ \ right arrow Ġ$ en Ġand Ġen $ \ right arrow Ġ$ de Ġand Ġat Ġeach Ġ5 k Ġsteps Ġfor Ġde $ \ right arrow Ġ$ en Ġand Ġja $ \ right arrow Ġ$ en . ĠThe Ġend - to - end Ġcomputational Ġoverhead Ġof Ġour Ġmethods , Ġwhen Ġcompared Ġto Ġstandard Ġsoft max , Ġis Ġrelatively Ġsmall ; Ġin Ġtraining Ġtokens Ġper Ġsecond , Ġthe Ġmodels Ġusing Ġ$\ alpha Ġ$ - ent max Ġand Ġ$ 1 . 5 $ - ent max Ġare , Ġrespectively , Ġ$ 75 \ % $ Ġand Ġ$ 90 \ % $ Ġthe Ġspeed Ġof Ġthe Ġsoft max Ġmodel . </s> Exper iments Ġ: :: ĠResults . </s> We Ġreport Ġtest Ġset Ġtoken ized ĠB LE U ĠB IB REF 32 Ġresults Ġin ĠTable ĠT AB REF 27 . ĠWe Ġcan Ġsee Ġthat Ġreplacing Ġsoft max Ġby Ġent max Ġdoes Ġnot Ġhurt Ġperformance Ġin Ġany Ġof Ġthe Ġdatasets ; Ġindeed , Ġsparse Ġattention ĠTransformers Ġtend Ġto Ġhave Ġslightly Ġhigher ĠB LE U , Ġbut Ġtheir Ġsp arsity Ġleads Ġto Ġa Ġbetter Ġpotential Ġfor Ġanalysis . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġmake Ġuse Ġof Ġthis Ġpotential Ġby Ġexploring Ġthe Ġlearned Ġinternal Ġmechanics Ġof Ġthe Ġself - att ention Ġheads . </s> Analysis </s> We Ġconduct Ġan Ġanalysis Ġfor Ġthe Ġhigher - resource Ġdataset ĠW MT Ġ2014 ĠEnglish Ġ$\ right arrow Ġ$ ĠGerman Ġof Ġthe Ġattention Ġin Ġthe Ġsparse Ġadaptive ĠTrans former Ġmodel Ġ($ \ alpha Ġ$ - ent max ) Ġat Ġmultiple Ġlevels : Ġwe Ġanalyze Ġhigh - level Ġstatistics Ġas Ġwell Ġas Ġindividual Ġhead Ġbehavior . ĠMoreover , Ġwe Ġmake Ġa Ġqualitative Ġanalysis Ġof Ġthe Ġinterpret ability Ġcapabilities Ġof Ġour Ġmodels . </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠWhat Ġkind Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ Ġvalues Ġare Ġlearned ? </s> Figure ĠFIG REF 37 Ġshows Ġthe Ġlearning Ġtraject ories Ġof Ġthe Ġ$\ alpha Ġ$ Ġparameters Ġof Ġa Ġselected Ġsubset Ġof Ġheads . ĠWe Ġgenerally Ġobserve Ġa Ġtendency Ġfor Ġthe Ġrandomly - initialized Ġ$\ alpha Ġ$ Ġparameters Ġto Ġdecrease Ġinitially , Ġsuggesting Ġthat Ġsoft max - like Ġbehavior Ġmay Ġbe Ġpreferable Ġwhile Ġthe Ġmodel Ġis Ġstill Ġvery Ġuncertain . ĠAfter Ġaround Ġone Ġthousand Ġsteps , Ġsome Ġheads Ġchange Ġdirection Ġand Ġbecome Ġsp arser , Ġperhaps Ġas Ġthey Ġbecome Ġmore Ġconfident Ġand Ġspecialized . ĠThis Ġshows Ġthat Ġthe Ġinitialization Ġof Ġ$\ alpha Ġ$ Ġdoes Ġnot Ġpred eter mine Ġits Ġsp arsity Ġlevel Ġor Ġthe Ġrole Ġthe Ġhead Ġwill Ġhave Ġthroughout . ĠIn Ġparticular , Ġhead Ġ8 Ġin Ġthe Ġenc oder Ġself - att ention Ġlayer Ġ2 Ġfirst Ġdrops Ġto Ġaround Ġ$\ alpha Ġ= 1 . 3 $ Ġbefore Ġbecoming Ġone Ġof Ġthe Ġsp ars est Ġheads , Ġwith Ġ$\ alpha Ġ\ app rox Ġ2 $. </s> The Ġoverall Ġdistribution Ġof Ġ$\ alpha Ġ$ Ġvalues Ġat Ġconvergence Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 38 . ĠWe Ġcan Ġobserve Ġthat Ġthe Ġenc oder Ġself - att ention Ġblocks Ġlearn Ġto Ġconcentrate Ġthe Ġ$\ alpha Ġ$ Ġvalues Ġin Ġtwo Ġmodes : Ġa Ġvery Ġsparse Ġone Ġaround Ġ$\ alpha Ġ\ right arrow Ġ2 $, Ġand Ġa Ġdense Ġone Ġbetween Ġsoft max Ġand Ġ1 . 5 - ent max Ġ. ĠHowever , Ġthe Ġdec oder Ġself Ġand Ġcontext Ġattention Ġonly Ġlearn Ġto Ġdistribute Ġthese Ġparameters Ġin Ġa Ġsingle Ġmode . ĠWe Ġshow Ġnext Ġthat Ġthis Ġis Ġreflected Ġin Ġthe Ġaverage Ġdensity Ġof Ġattention Ġweight Ġvectors Ġas Ġwell . </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠAttention Ġweight Ġdensity Ġwhen Ġtranslating . </s> For Ġany Ġ$\ alpha Ġ> 1 $, Ġit Ġwould Ġstill Ġbe Ġpossible Ġfor Ġthe Ġweight Ġmat rices Ġin ĠEqu ation ĠDIS PLAY _ FORM 9 Ġto Ġlearn Ġre - sc al ings Ġso Ġas Ġto Ġmake Ġattention Ġsp arser Ġor Ġdens er . ĠTo Ġvisualize Ġthe Ġimpact Ġof Ġadaptive Ġ$\ alpha Ġ$ Ġvalues , Ġwe Ġcompare Ġthe Ġempirical Ġattention Ġweight Ġdensity Ġ( the Ġaverage Ġnumber Ġof Ġtokens Ġreceiving Ġnon - zero Ġattention ) Ġwithin Ġeach Ġmodule , Ġagainst Ġsparse ĠTransformers Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $. </s> Figure ĠFIG REF 40 Ġshows Ġthat , Ġwith Ġfixed Ġ$\ alpha Ġ= 1 . 5 $, Ġheads Ġtend Ġto Ġbe Ġsparse Ġand Ġsimilarly - dist ributed Ġin Ġall Ġthree Ġattention Ġmodules . ĠWith Ġlearned Ġ$\ alpha Ġ$ , Ġthere Ġare Ġtwo Ġnotable Ġchanges : Ġ( i ) Ġa Ġprominent Ġmode Ġcorresponding Ġto Ġfully Ġdense Ġprobabilities , Ġshowing Ġthat Ġour Ġmodels Ġlearn Ġto Ġcombine Ġsparse Ġand Ġdense Ġattention , Ġand Ġ( ii ) Ġa Ġdistinction Ġbetween Ġthe Ġenc oder Ġself - att ention ĠâĢĵ Ġwhose Ġbackground Ġdistribution Ġtends Ġtoward Ġextreme Ġsp arsity ĠâĢĵ Ġand Ġthe Ġother Ġtwo Ġmodules , Ġwho Ġexhibit Ġmore Ġuniform Ġbackground Ġdistributions . ĠThis Ġsuggests Ġthat Ġperhaps Ġentirely Ġsparse ĠTransformers Ġare Ġsub opt imal . </s> The Ġfact Ġthat Ġthe Ġdec oder Ġseems Ġto Ġprefer Ġdens er Ġattention Ġdistributions Ġmight Ġbe Ġattributed Ġto Ġit Ġbeing Ġauto - reg ressive , Ġonly Ġhaving Ġaccess Ġto Ġpast Ġtokens Ġand Ġnot Ġthe Ġfull Ġsentence . ĠWe Ġspeculate Ġthat Ġit Ġmight Ġlose Ġtoo Ġmuch Ġinformation Ġif Ġit Ġassigned Ġweights Ġof Ġzero Ġto Ġtoo Ġmany Ġtokens Ġin Ġthe Ġself - att ention , Ġsince Ġthere Ġare Ġfewer Ġtokens Ġto Ġattend Ġto Ġin Ġthe Ġfirst Ġplace . </s> Te asing Ġthis Ġdown Ġinto Ġseparate Ġlayers , ĠFigure ĠFIG REF 41 Ġshows Ġthe Ġaverage Ġ( s orted ) Ġdensity Ġof Ġeach Ġhead Ġfor Ġeach Ġlayer . ĠWe Ġobserve Ġthat Ġ$\ alpha Ġ$ - ent max Ġis Ġable Ġto Ġlearn Ġdifferent Ġsp arsity Ġpatterns Ġat Ġeach Ġlayer , Ġleading Ġto Ġmore Ġvariance Ġin Ġindividual Ġhead Ġbehavior , Ġto Ġclearly - identified Ġdense Ġand Ġsparse Ġheads , Ġand Ġoverall Ġto Ġdifferent Ġtendencies Ġcompared Ġto Ġthe Ġfixed Ġcase Ġof Ġ$\ alpha Ġ= 1 . 5 $. </s> Analysis Ġ: :: ĠHigh - Level ĠStatistics Ġ: :: ĠHead Ġdiversity . </s> To Ġmeasure Ġthe Ġoverall Ġdisagreement Ġbetween Ġattention Ġheads , Ġas Ġa Ġmeasure Ġof Ġhead Ġdiversity , Ġwe Ġuse Ġthe Ġfollowing Ġgeneral ization Ġof Ġthe ĠJensen - Sh annon Ġdivergence : </s> where Ġ$\ math bf Ġ{ p } _ j $ Ġis Ġthe Ġvector Ġof Ġattention Ġweights Ġassigned Ġby Ġhead Ġ$ j $ Ġto Ġeach Ġword Ġin Ġthe Ġsequence , Ġand Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } $ Ġis Ġthe ĠShannon Ġentropy , Ġbase - adjusted Ġbased Ġon Ġthe Ġdimension Ġof Ġ$\ math bf Ġ{ p } $ Ġsuch Ġthat Ġ$ JS Ġ\ le Ġ1 $. ĠWe Ġaverage Ġthis Ġmeasure Ġover Ġthe Ġentire Ġvalidation Ġset . ĠThe Ġhigher Ġthis Ġmetric Ġis , Ġthe Ġmore Ġthe Ġheads Ġare Ġtaking Ġdifferent Ġroles Ġin Ġthe Ġmodel . </s> Figure ĠFIG REF 44 Ġshows Ġthat Ġboth Ġsparse ĠTrans former Ġvariants Ġshow Ġmore Ġdiversity Ġthan Ġthe Ġtraditional Ġsoft max Ġone . ĠInterestingly , Ġdiversity Ġseems Ġto Ġpeak Ġin Ġthe Ġmiddle Ġlayers Ġof Ġthe Ġenc oder Ġself - att ention Ġand Ġcontext Ġattention , Ġwhile Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe Ġdec oder Ġself - att ention . </s> The Ġstatistics Ġshown Ġin Ġthis Ġsection Ġcan Ġbe Ġfound Ġfor Ġthe Ġother Ġlanguage Ġpairs Ġin ĠAppendix ĠSEC REF 8 . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations </s> Previous Ġwork Ġpointed Ġout Ġsome Ġspecific Ġroles Ġplayed Ġby Ġdifferent Ġheads Ġin Ġthe Ġsoft max ĠTrans former Ġmodel ĠB IB REF 33 , ĠB IB REF 5 , ĠB IB REF 9 . ĠIdent ifying Ġthe Ġspecialization Ġof Ġa Ġhead Ġcan Ġbe Ġdone Ġby Ġobserving Ġthe Ġtype Ġof Ġtokens Ġor Ġsequences Ġthat Ġthe Ġhead Ġoften Ġassigns Ġmost Ġof Ġits Ġattention Ġweight ; Ġthis Ġis Ġfacilitated Ġby Ġsp arsity . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠPos itional Ġheads . </s> One Ġparticular Ġtype Ġof Ġhead , Ġas Ġnoted Ġby ĠB IB REF 9 , Ġis Ġthe Ġpositional Ġhead . ĠThese Ġheads Ġtend Ġto Ġfocus Ġtheir Ġattention Ġon Ġeither Ġthe Ġprevious Ġor Ġnext Ġtoken Ġin Ġthe Ġsequence , Ġthus Ġobtaining Ġrepresentations Ġof Ġthe Ġneighborhood Ġof Ġthe Ġcurrent Ġtime Ġstep . ĠIn ĠFigure ĠFIG REF 47 , Ġwe Ġshow Ġattention Ġplots Ġfor Ġsuch Ġheads , Ġfound Ġfor Ġeach Ġof Ġthe Ġstudied Ġmodels . ĠThe Ġsp arsity Ġof Ġour Ġmodels Ġallows Ġthese Ġheads Ġto Ġbe Ġmore Ġconfident Ġin Ġtheir Ġrepresentations , Ġby Ġassigning Ġthe Ġwhole Ġprobability Ġdistribution Ġto Ġa Ġsingle Ġtoken Ġin Ġthe Ġsequence . ĠCon crete ly , Ġwe Ġmay Ġmeasure Ġa Ġpositional Ġhead 's Ġconfidence Ġas Ġthe Ġaverage Ġattention Ġweight Ġassigned Ġto Ġthe Ġprevious Ġtoken . ĠThe Ġsoft max Ġmodel Ġhas Ġthree Ġheads Ġfor Ġposition Ġ$ - 1 $, Ġwith Ġmedian Ġconfidence Ġ$ 93 . 5 \ % $. ĠThe Ġ$ 1 . 5 $ - ent max Ġmodel Ġalso Ġhas Ġthree Ġheads Ġfor Ġthis Ġposition , Ġwith Ġmedian Ġconfidence Ġ$ 94 . 4 \ % $. ĠThe Ġadaptive Ġmodel Ġhas Ġfour Ġheads , Ġwith Ġmedian Ġconf idences Ġ$ 95 . 9 \ % $, Ġthe Ġlowest - confidence Ġhead Ġbeing Ġdense Ġwith Ġ$\ alpha Ġ= 1 . 18 $, Ġwhile Ġthe Ġhighest - confidence Ġhead Ġbeing Ġsparse Ġ($ \ alpha Ġ= 1 . 91 $ ). </s> For Ġposition Ġ$ + 1 $, Ġthe Ġmodels Ġeach Ġdedicate Ġone Ġhead , Ġwith Ġconfidence Ġaround Ġ$ 95 \ % $, Ġslightly Ġhigher Ġfor Ġent max . ĠThe Ġadaptive Ġmodel Ġsets Ġ$\ alpha Ġ= 1 . 96 $ Ġfor Ġthis Ġhead . </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠB PE - mer ging Ġhead . </s> Due Ġto Ġthe Ġsp arsity Ġof Ġour Ġmodels , Ġwe Ġare Ġable Ġto Ġidentify Ġother Ġhead Ġspecial izations , Ġeasily Ġidentifying Ġwhich Ġheads Ġshould Ġbe Ġfurther Ġanalysed . ĠIn ĠFigure ĠFIG REF 51 Ġwe Ġshow Ġone Ġsuch Ġhead Ġwhere Ġthe Ġ$\ alpha Ġ$ Ġvalue Ġis Ġparticularly Ġhigh Ġ( in Ġthe Ġenc oder , Ġlayer Ġ1 , Ġhead Ġ4 Ġdepicted Ġin ĠFigure ĠFIG REF 37 ). ĠWe Ġfound Ġthat Ġthis Ġhead Ġmost Ġoften Ġlooks Ġat Ġthe Ġcurrent Ġtime Ġstep Ġwith Ġhigh Ġconfidence , Ġmaking Ġit Ġa Ġpositional Ġhead Ġwith Ġoffset Ġ0 . ĠHowever , Ġthis Ġhead Ġoften Ġspreads Ġweight Ġsparse ly Ġover Ġ2 - 3 Ġneighboring Ġtokens , Ġwhen Ġthe Ġtokens Ġare Ġpart Ġof Ġthe Ġsame ĠB PE Ġcluster Ġor Ġhyp hen ated Ġwords . ĠAs Ġthis Ġhead Ġis Ġin Ġthe Ġfirst Ġlayer , Ġit Ġprovides Ġa Ġuseful Ġservice Ġto Ġthe Ġhigher Ġlayers Ġby Ġcombining Ġinformation Ġevenly Ġwithin Ġsome ĠB PE Ġclusters . </s> For Ġeach ĠB PE Ġcluster Ġor Ġcluster Ġof Ġhyp hen ated Ġwords , Ġwe Ġcomputed Ġa Ġscore Ġbetween Ġ0 Ġand Ġ1 Ġthat Ġcorresponds Ġto Ġthe Ġmaximum Ġattention Ġmass Ġassigned Ġby Ġany Ġtoken Ġto Ġthe Ġrest Ġof Ġthe Ġtokens Ġinside Ġthe Ġcluster Ġin Ġorder Ġto Ġquantify Ġthe ĠB PE - mer ging Ġcapabilities Ġof Ġthese Ġheads . ĠThere Ġare Ġnot Ġany Ġattention Ġheads Ġin Ġthe Ġsoft max Ġmodel Ġthat Ġare Ġable Ġto Ġobtain Ġa Ġscore Ġover Ġ$ 80 \ % $, Ġwhile Ġfor Ġ$ 1 . 5 $ - ent max Ġand Ġ$\ alpha Ġ$ - ent max Ġthere Ġare Ġtwo Ġheads Ġin Ġeach Ġ($ 83 . 3 \ % $ Ġand Ġ$ 85 . 6 \ % $ Ġfor Ġ$ 1 . 5 $ - ent max Ġand Ġ$ 88 . 5 \ % $ Ġand Ġ$ 89 . 8 \ % $ Ġfor Ġ$\ alpha Ġ$ - ent max ). </s> Analysis Ġ: :: ĠIdent ifying ĠHead ĠSpecial izations Ġ: :: ĠInter rog ation Ġhead . </s> On Ġthe Ġother Ġhand , Ġin ĠFigure ĠFIG REF 52 Ġwe Ġshow Ġa Ġhead Ġfor Ġwhich Ġour Ġadapt ively Ġsparse Ġmodel Ġchose Ġan Ġ$\ alpha Ġ$ Ġclose Ġto Ġ1 , Ġmaking Ġit Ġcloser Ġto Ġsoft max Ġ( also Ġshown Ġin Ġenc oder , Ġlayer Ġ1 , Ġhead Ġ3 Ġdepicted Ġin ĠFigure ĠFIG REF 37 ). ĠWe Ġobserve Ġthat Ġthis Ġhead Ġassigns Ġa Ġhigh Ġprobability Ġto Ġquestion Ġmarks Ġat Ġthe Ġend Ġof Ġthe Ġsentence Ġin Ġtime Ġsteps Ġwhere Ġthe Ġcurrent Ġtoken Ġis Ġinterrog ative , Ġthus Ġmaking Ġit Ġan Ġinterrogation - det ect ing Ġhead . ĠWe Ġalso Ġobserve Ġthis Ġtype Ġof Ġheads Ġin Ġthe Ġother Ġmodels , Ġwhich Ġwe Ġalso Ġdepict Ġin ĠFigure ĠFIG REF 52 . ĠThe Ġaverage Ġattention Ġweight Ġplaced Ġon Ġthe Ġquestion Ġmark Ġwhen Ġthe Ġcurrent Ġtoken Ġis Ġan Ġinterrog ative Ġword Ġis Ġ$ 98 . 5 \ % $ Ġfor Ġsoft max , Ġ$ 97 . 0 \ % $ Ġfor Ġ$ 1 . 5 $ - ent max , Ġand Ġ$ 99 . 5 \ % $ Ġfor Ġ$\ alpha Ġ$ - ent max . </s> Furthermore , Ġwe Ġcan Ġexamine Ġsentences Ġwhere Ġsome Ġtend entially Ġsparse Ġheads Ġbecome Ġless Ġso , Ġthus Ġidentifying Ġsources Ġof Ġambiguity Ġwhere Ġthe Ġhead Ġis Ġless Ġconfident Ġin Ġits Ġprediction . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 55 Ġwhere Ġsp arsity Ġin Ġthe Ġsame Ġhead Ġdiffers Ġfor Ġsentences Ġof Ġsimilar Ġlength . </s> Related ĠWork Ġ: :: ĠSp arse Ġattention . </s> Prior Ġwork Ġhas Ġdeveloped Ġsparse Ġattention Ġmechanisms , Ġincluding Ġapplications Ġto ĠN MT ĠB IB REF 19 , ĠB IB REF 12 , ĠB IB REF 20 , ĠB IB REF 22 , ĠB IB REF 34 . ĠB IB REF 14 Ġintroduced Ġthe Ġent max Ġfunction Ġthis Ġwork Ġbuilds Ġupon . ĠIn Ġtheir Ġwork , Ġthere Ġis Ġa Ġsingle Ġattention Ġmechanism Ġwhich Ġis Ġcontrolled Ġby Ġa Ġfixed Ġ$\ alpha Ġ$ . ĠIn Ġcontrast , Ġthis Ġis Ġthe Ġfirst Ġwork Ġto Ġallow Ġsuch Ġattention Ġm appings Ġto Ġdynamically Ġadapt Ġtheir Ġcurv ature Ġand Ġsp arsity , Ġby Ġautomatically Ġadjusting Ġthe Ġcontinuous Ġ$\ alpha Ġ$ Ġparameter . ĠWe Ġalso Ġprovide Ġthe Ġfirst Ġresults Ġusing Ġsparse Ġattention Ġin Ġa ĠTrans former Ġmodel . </s> Related ĠWork Ġ: :: ĠFixed Ġsp arsity Ġpatterns . </s> Recent Ġresearch Ġimproves Ġthe Ġscal ability Ġof ĠTrans former - like Ġnetworks Ġthrough Ġstatic , Ġfixed Ġsp arsity Ġpatterns ĠB IB REF 10 , ĠB IB REF 35 . ĠOur Ġadapt ively - s parse ĠTrans former Ġcan Ġdynamically Ġselect Ġa Ġsp arsity Ġpattern Ġthat Ġfinds Ġrelevant Ġwords Ġregardless Ġof Ġtheir Ġposition Ġ( e . g ., ĠFigure ĠFIG REF 52 ). ĠMoreover , Ġthe Ġtwo Ġstrategies Ġcould Ġbe Ġcombined . ĠIn Ġa Ġconcurrent Ġline Ġof Ġresearch , ĠB IB REF 11 Ġpropose Ġan Ġadaptive Ġattention Ġspan Ġfor ĠTrans former Ġlanguage Ġmodels . ĠWhile Ġtheir Ġwork Ġhas Ġeach Ġhead Ġlearn Ġa Ġdifferent Ġcontiguous Ġspan Ġof Ġcontext Ġtokens Ġto Ġattend Ġto , Ġour Ġwork Ġfinds Ġdifferent Ġsp arsity Ġpatterns Ġin Ġthe Ġsame Ġspan . ĠInterestingly , Ġsome Ġof Ġtheir Ġfindings Ġmirror Ġours ĠâĢĵ Ġwe Ġfound Ġthat Ġattention Ġheads Ġin Ġthe Ġlast Ġlayers Ġtend Ġto Ġbe Ġdens er Ġon Ġaverage Ġwhen Ġcompared Ġto Ġthe Ġones Ġin Ġthe Ġfirst Ġlayers , Ġwhile Ġtheir Ġwork Ġhas Ġfound Ġthat Ġlower Ġlayers Ġtend Ġto Ġhave Ġa Ġshorter Ġattention Ġspan Ġcompared Ġto Ġhigher Ġlayers . </s> Related ĠWork Ġ: :: ĠTrans former Ġinterpret ability . </s> The Ġoriginal ĠTrans former Ġpaper ĠB IB REF 0 Ġshows Ġattention Ġvisual izations , Ġfrom Ġwhich Ġsome Ġspeculation Ġcan Ġbe Ġmade Ġof Ġthe Ġroles Ġthe Ġseveral Ġattention Ġheads Ġhave . ĠB IB REF 7 Ġstudy Ġthe Ġsynt actic Ġabilities Ġof Ġthe ĠTrans former Ġself - att ention , Ġwhile ĠB IB REF 6 Ġextract Ġdependency Ġrelations Ġfrom Ġthe Ġattention Ġweights . ĠB IB REF 8 Ġfind Ġthat Ġthe Ġself - att ent ions Ġin ĠB ERT ĠB IB REF 3 Ġfollow Ġa Ġsequence Ġof Ġprocesses Ġthat Ġresembles Ġa Ġclassical ĠN LP Ġpipeline . ĠRegarding Ġredundancy Ġof Ġheads , ĠB IB REF 9 Ġdevelop Ġa Ġmethod Ġthat Ġis Ġable Ġto Ġpr une Ġheads Ġof Ġthe Ġmulti - head Ġattention Ġmodule Ġand Ġmake Ġan Ġempirical Ġstudy Ġof Ġthe Ġrole Ġthat Ġeach Ġhead Ġhas Ġin Ġself - att ention Ġ( pos itional , Ġsynt actic Ġand Ġrare Ġwords ). ĠB IB REF 36 Ġalso Ġaim Ġto Ġreduce Ġhead Ġredundancy Ġby Ġadding Ġa Ġregular ization Ġterm Ġto Ġthe Ġloss Ġthat Ġmaxim izes Ġhead Ġdisagreement Ġand Ġobtain Ġimproved Ġresults . ĠWhile Ġnot Ġconsidering ĠTrans former Ġattent ions , ĠB IB REF 18 Ġshow Ġthat Ġtraditional Ġattention Ġmechanisms Ġdo Ġnot Ġnecessarily Ġimprove Ġinterpret ability Ġsince Ġsoft max Ġattention Ġis Ġvulnerable Ġto Ġan Ġadvers arial Ġattack Ġleading Ġto Ġwildly Ġdifferent Ġmodel Ġpredictions Ġfor Ġthe Ġsame Ġattention Ġweights . ĠSp arse Ġattention Ġmay Ġmitigate Ġthese Ġissues ; Ġhowever , Ġour Ġwork Ġfocuses Ġmostly Ġon Ġa Ġmore Ġmechanical Ġaspect Ġof Ġinterpretation Ġby Ġanalyzing Ġhead Ġbehavior , Ġrather Ġthan Ġon Ġexplanations Ġfor Ġpredictions . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġcontribute Ġa Ġnovel Ġstrategy Ġfor Ġadapt ively Ġsparse Ġattention , Ġand , Ġin Ġparticular , Ġfor Ġadapt ively Ġsparse ĠTransformers . ĠWe Ġpresent Ġthe Ġfirst Ġempirical Ġanalysis Ġof ĠTransformers Ġwith Ġsparse Ġattention Ġm appings Ġ( i . e ., Ġent max ), Ġshowing Ġpotential Ġin Ġboth Ġtranslation Ġaccuracy Ġas Ġwell Ġas Ġin Ġmodel Ġinterpret ability . </s> In Ġparticular , Ġwe Ġanalyzed Ġhow Ġthe Ġattention Ġheads Ġin Ġthe Ġproposed Ġadapt ively Ġsparse ĠTrans former Ġcan Ġspecialize Ġmore Ġand Ġwith Ġhigher Ġconfidence . ĠOur Ġadapt ivity Ġstrategy Ġrelies Ġonly Ġon Ġgradient - based Ġoptimization , Ġside - ste pping Ġcostly Ġper - head Ġhyper - param eter Ġsearches . ĠFurther Ġspeed - ups Ġare Ġpossible Ġby Ġleveraging Ġmore Ġparallel ism Ġin Ġthe Ġbis ection Ġalgorithm Ġfor Ġcomputing Ġ$\ alpha Ġ$ - ent max . </s> Finally , Ġsome Ġof Ġthe Ġautomatically - learn ed Ġbehaviors Ġof Ġour Ġadapt ively Ġsparse ĠTransformers ĠâĢĵ Ġfor Ġinstance , Ġthe Ġnear - d eter ministic Ġpositional Ġheads Ġor Ġthe Ġsub word Ġjoining Ġhead ĠâĢĵ Ġmay Ġprovide Ġnew Ġideas Ġfor Ġdesigning Ġstatic Ġvariations Ġof Ġthe ĠTrans former . </s> Acknowled gments </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠEuropean ĠResearch ĠCouncil Ġ( ERC ĠSt G ĠDeep SP IN Ġ7 58 969 ), Ġand Ġby Ġthe ĠFund a Ã§ Ã£o Ġpara Ġa ĠCi Ãª nc ia Ġe ĠTec n olog ia Ġthrough Ġcontracts ĠUID / EE A / 5000 8 / 2019 Ġand ĠCM UP ER I / T IC / 00 46 / 2014 Ġ( Go Local ). ĠWe Ġare Ġgrateful Ġto ĠBen ĠPeters Ġfor Ġthe Ġ$\ alpha Ġ$ - ent max Ġcode Ġand ĠEr ick ĠF on sec a , ĠMarcos ĠTre vis o , ĠPedro ĠMart ins , Ġand ĠTs vet om ila ĠM ih ay l ova Ġfor Ġinsightful Ġgroup Ġdiscussion . ĠWe Ġthank ĠMath ieu ĠBl ond el Ġfor Ġthe Ġidea Ġto Ġlearn Ġ$\ alpha Ġ$ . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġfeedback . </s> Supplement ary ĠMaterial </s> Background Ġ: :: ĠRegular ized ĠFen chel - Young Ġprediction Ġfunctions </s> Definition Ġ1 Ġ( BI B REF 23 ) </s> Let Ġ$\ O mega Ġ\ col on Ġ\ tri angle Ġ^ d Ġ\ right arrow Ġ{\ math bb Ġ{ R }} \ cup Ġ\ l brace Ġ\ in fty Ġ\ r brace Ġ$ Ġbe Ġa Ġstrictly Ġconve x Ġregular ization Ġfunction . ĠWe Ġdefine Ġthe Ġprediction Ġfunction Ġ$\ math bf Ġ{\ pi Ġ} _{ \ O mega Ġ} $ Ġas </s> Background Ġ: :: ĠCharacter izing Ġthe Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġmapping </s> L em ma Ġ1 Ġ( BI B REF 14 ) ĠFor Ġany Ġ$\ math bf Ġ{ z } $, Ġthere Ġexists Ġa Ġunique Ġ$\ t au Ġ^ \ star Ġ$ Ġsuch Ġthat </s> Proof : ĠFrom Ġthe Ġdefinition Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $, </s> we Ġmay Ġeasily Ġidentify Ġit Ġwith Ġa Ġregular ized Ġprediction Ġfunction Ġ( Def . ĠUN K REF 81 ): </s> We Ġfirst Ġnote Ġthat Ġfor Ġall Ġ$\ math bf Ġ{ p }\ in Ġ\ tri angle Ġ^ d $, </s> From Ġthe Ġconstant Ġinvari ance Ġand Ġscaling Ġproperties Ġof Ġ$\ math bf Ġ{\ pi Ġ} _{ \ O mega Ġ} $ ĠB IB REF 23 , </s> Using ĠB IB REF 23 , Ġnoting Ġthat Ġ$ g ^ {\ prime Ġ} ( t ) Ġ= Ġt ^ {\ alpha Ġ- Ġ1 } $ Ġand Ġ$( g ^ {\ prime Ġ}) ^{ - 1 } ( u ) Ġ= Ġu ^ {{ 1 }{ \ alpha Ġ- 1 }} $, Ġyields </s> Since Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _ \ alpha Ġ$ Ġis Ġstrictly Ġconve x Ġon Ġthe Ġsimple x , Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġhas Ġa Ġunique Ġsolution Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ . ĠEqu ation ĠDIS PLAY _ FORM 88 Ġimplicitly Ġdefines Ġa Ġone - to - one Ġmapping Ġbetween Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ Ġand Ġ$\ t au Ġ^ \ star Ġ$ Ġas Ġlong Ġas Ġ$\ math bf Ġ{ p } ^ \ star Ġ\ in Ġ\ tri angle Ġ$ , Ġtherefore Ġ$\ t au Ġ^ \ star Ġ$ Ġis Ġalso Ġunique . </s> Background Ġ: :: ĠConnect ions Ġto Ġsoft max Ġand Ġsparse max </s> The ĠEucl ide an Ġprojection Ġonto Ġthe Ġsimple x , Ġsometimes Ġreferred Ġto , Ġin Ġthe Ġcontext Ġof Ġneural Ġattention , Ġas Ġsparse max ĠB IB REF 19 , Ġis Ġdefined Ġas </s> The Ġsolution Ġcan Ġbe Ġcharacterized Ġthrough Ġthe Ġunique Ġthreshold Ġ$\ t au Ġ$ Ġsuch Ġthat Ġ$\ sum Ġ_ i Ġ\ oper at orn am ew ith limits {\ math sf Ġ{ s parse max }} (\ math bf Ġ{ z }) _ i Ġ= Ġ1 $ Ġand ĠB IB REF 38 </s> Thus , Ġeach Ġcoordinate Ġof Ġthe Ġsparse max Ġsolution Ġis Ġa Ġpiece wise - linear Ġfunction . ĠVis ibly , Ġthis Ġexpression Ġis Ġrecovered Ġwhen Ġsetting Ġ$\ alpha Ġ= 2 $ Ġin Ġthe Ġ$\ alpha Ġ$ - ent max Ġexpression Ġ( Equ ation ĠDIS PLAY _ FORM 85 ); Ġfor Ġother Ġvalues Ġof Ġ$\ alpha Ġ$ , Ġthe Ġexponent Ġinduces Ġcurv ature . </s> On Ġthe Ġother Ġhand , Ġthe Ġwell - known Ġsoft max Ġis Ġusually Ġdefined Ġthrough Ġthe Ġexpression </s> which Ġcan Ġbe Ġshown Ġto Ġbe Ġthe Ġunique Ġsolution Ġof Ġthe Ġoptimization Ġproblem </s> where Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } (\ math bf Ġ{ p }) Ġ- \ sum Ġ_ i Ġp _ i Ġ\ log Ġp _ i $ Ġis Ġthe ĠShannon Ġentropy . ĠIndeed , Ġsetting Ġthe Ġgradient Ġto Ġ0 Ġyields Ġthe Ġcondition Ġ$\ log Ġp _ i Ġ= Ġz _ j Ġ- Ġ\ nu Ġ_ i Ġ- Ġ\ t au Ġ- Ġ1 $, Ġwhere Ġ$\ t au Ġ$ Ġand Ġ$\ nu Ġ> Ġ0 $ Ġare ĠLag range Ġmultipl iers Ġfor Ġthe Ġsimple x Ġconstraints Ġ$\ sum Ġ_ i Ġp _ i Ġ= Ġ1 $ Ġand Ġ$ p _ i Ġ\ ge Ġ0 $, Ġrespectively . ĠSince Ġthe Ġl . h . s . Ġis Ġonly Ġfinite Ġfor Ġ$ p _ i > 0 $, Ġwe Ġmust Ġhave Ġ$\ nu Ġ_ i = 0 $ Ġfor Ġall Ġ$ i $, Ġby Ġcomplementary Ġslack ness . ĠThus , Ġthe Ġsolution Ġmust Ġhave Ġthe Ġform Ġ$ p _ i Ġ= Ġ{\ exp Ġ( z _ i ) }{ Z } $, Ġyielding ĠEqu ation ĠDIS PLAY _ FORM 92 . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ </s> Rec all Ġthat Ġthe Ġent max Ġtransformation Ġis Ġdefined Ġas : </s> where Ġ$\ alpha Ġ\ ge Ġ1 $ Ġand Ġ$\ math sf Ġ{ H } ^ {\ text sc Ġ{ T }} _{ \ alpha Ġ} $ Ġis Ġthe ĠTs all is Ġentropy , </s> and Ġ$\ math sf Ġ{ H } ^ \ text sc Ġ{ S } (\ math bf Ġ{ p } ): = Ġ- \ sum Ġ_ j Ġp _ j Ġ\ log Ġp _ j $ Ġis Ġthe ĠShannon Ġentropy . </s> In Ġthis Ġsection , Ġwe Ġderive Ġthe ĠJacob ian Ġof Ġ$\ oper at orn am ew ith limits {\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġrespect Ġto Ġthe Ġscal ar Ġparameter Ġ$\ alpha Ġ$ . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠGeneral Ġcase Ġof Ġ@ ! ST ART @ $ \ alpha Ġ> 1 $ @ ! END @ </s> From Ġthe ĠK KT Ġconditions Ġassociated Ġwith Ġthe Ġoptimization Ġproblem Ġin ĠE q . ĠDIS PLAY _ FORM 85 , Ġwe Ġhave Ġthat Ġthe Ġsolution Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $ Ġhas Ġthe Ġfollowing Ġform , Ġcoordinate - wise : </s> where Ġ$\ t au Ġ^ {\ star Ġ} $ Ġis Ġa Ġscal ar ĠLag range Ġmultiplier Ġthat Ġensures Ġthat Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $ Ġnormal izes Ġto Ġ1 , Ġi . e ., Ġit Ġis Ġdefined Ġimplicitly Ġby Ġthe Ġcondition : </s> For Ġgeneral Ġvalues Ġof Ġ$\ alpha Ġ$ , ĠE q . ĠDIS PLAY _ FORM 98 Ġlacks Ġa Ġclosed Ġform Ġsolution . ĠThis Ġmakes Ġthe Ġcomputation Ġof Ġthe ĠJacob ian </s> non - t riv ial . ĠFortunately , Ġwe Ġcan Ġuse Ġthe Ġtechnique Ġof Ġimplicit Ġdifferentiation Ġto Ġobtain Ġthis ĠJacob ian . </s> The ĠJacob ian Ġexists Ġalmost Ġeverywhere , Ġand Ġthe Ġexpressions Ġwe Ġderive Ġexpressions Ġyield Ġa Ġgeneralized ĠJacob ian ĠB IB REF 37 Ġat Ġany Ġnon - different iable Ġpoints Ġthat Ġmay Ġoccur Ġfor Ġcertain Ġ($ \ alpha Ġ$ , Ġ$\ math bf Ġ{ z } $ ) Ġpairs . ĠWe Ġbegin Ġby Ġnoting Ġthat Ġ$\ frac {\ partial Ġp _ i ^ {\ star Ġ} }{ \ partial Ġ\ alpha Ġ} Ġ= Ġ0 $ Ġif Ġ$ p _ i ^ {\ star Ġ} Ġ= Ġ0 $, Ġbecause Ġincreasing Ġ$\ alpha Ġ$ Ġkeeps Ġsparse Ġcoordinates Ġsparse . ĠTherefore Ġwe Ġneed Ġto Ġworry Ġonly Ġabout Ġcoordinates Ġthat Ġare Ġin Ġthe Ġsupport Ġof Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ . ĠWe Ġwill Ġassume Ġhereafter Ġthat Ġthe Ġ$ i $ th Ġcoordinate Ġof Ġ$\ math bf Ġ{ p } ^ \ star Ġ$ Ġis Ġnon - zero . ĠWe Ġhave : </s> We Ġcan Ġsee Ġthat Ġthis ĠJacob ian Ġdepends Ġon Ġ$\ frac {\ partial Ġ\ t au Ġ^ {\ star Ġ} }{ \ partial Ġ\ alpha Ġ} $, Ġwhich Ġwe Ġnow Ġcompute Ġusing Ġimplicit Ġdifferentiation . </s> Let Ġ$\ math cal Ġ{ S } Ġ= Ġ\ l brace Ġi : Ġp ^ \ star Ġ_ i Ġ> Ġ0 Ġ\ r brace Ġ$ ). ĠBy Ġdifferent iating Ġboth Ġsides Ġof ĠE q . ĠDIS PLAY _ FORM 98 , Ġre - using Ġsome Ġof Ġthe Ġsteps Ġin ĠE q . ĠDIS PLAY _ FORM 101 , Ġand Ġrecalling ĠE q . ĠDIS PLAY _ FORM 97 , Ġwe Ġget </s> from Ġwhich Ġwe Ġobtain : </s> Finally , Ġplug ging ĠE q . ĠDIS PLAY _ FORM 103 Ġinto ĠE q . ĠDIS PLAY _ FORM 101 , Ġwe Ġget : </s> where Ġwe Ġdenote Ġby </s> The Ġdistribution Ġ$\ t ilde {\ math bf Ġ{ p }} (\ alpha Ġ) $ Ġcan Ġbe Ġinterpreted Ġas Ġa ĠâĢ ľ s ke wed âĢ Ŀ Ġdistribution Ġobtained Ġfrom Ġ$\ math bf Ġ{ p } ^ {\ star Ġ} $, Ġwhich Ġappears Ġin Ġthe ĠJacob ian Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} (\ math bf Ġ{ z }) $ Ġw . r . t . Ġ$\ math bf Ġ{ z } $ Ġas Ġwell ĠB IB REF 14 . </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠSol ving Ġthe Ġind etermination Ġfor Ġ@ ! ST ART @ $ \ alpha Ġ= 1 $ @ ! END @ </s> We Ġcan Ġwrite ĠE q . ĠDIS PLAY _ FORM 104 Ġas </s> When Ġ$\ alpha Ġ\ right arrow Ġ1 ^ + $, Ġwe Ġhave Ġ$\ t ilde {\ math bf Ġ{ p }} (\ alpha Ġ) Ġ\ right arrow Ġ\ math bf Ġ{ p } ^ {\ star Ġ} $, Ġwhich Ġleads Ġto Ġa Ġ$\ frac { 0 }{ 0 } $ Ġind etermination . </s> To Ġsolve Ġthis Ġind etermination , Ġwe Ġwill Ġneed Ġto Ġapply ĠL ' H Ã´ p ital 's Ġrule Ġtwice . ĠLet Ġus Ġfirst Ġcompute Ġthe Ġderivative Ġof Ġ$\ t ilde { p } _ i (\ alpha Ġ) $ Ġwith Ġrespect Ġto Ġ$\ alpha Ġ$ . ĠWe Ġhave </s> there fore </s> Different iating Ġthe Ġnumer ator Ġand Ġdenomin ator Ġin ĠE q . ĠDIS PLAY _ FORM 107 , Ġwe Ġget : </s> with </s> and </s> When Ġ$\ alpha Ġ\ right arrow Ġ1 ^ + $, Ġ$ B $ Ġbecomes Ġagain Ġa Ġ$\ frac { 0 }{ 0 } $ Ġind etermination , Ġwhich Ġwe Ġcan Ġsolve Ġby Ġapplying Ġagain ĠL ' H Ã´ p ital 's Ġrule . ĠDifferent iating Ġthe Ġnumer ator Ġand Ġdenomin ator Ġin ĠE q . ĠDIS PLAY _ FORM 112 : </s> Finally , Ġsum ming ĠE q . ĠDIS PLAY _ FORM 111 Ġand ĠE q . ĠDIS PLAY _ FORM 113 , Ġwe Ġget </s> Jacob ian Ġof Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ - ent max Ġw . r . t . Ġthe Ġshape Ġparameter Ġ@ ! ST ART @ $ \ alpha Ġ$ @ ! END @ : ĠProof Ġof ĠProposition Ġ@ ! ST ART @ UID 22 @ ! END @ Ġ: :: ĠSummary </s> To Ġsum Ġup , Ġwe Ġhave Ġthe Ġfollowing Ġexpression Ġfor Ġthe ĠJacob ian Ġof Ġ$\ math op Ġ{\ math sf Ġ{\ alpha Ġ} \ text normal Ġ{ - }\ math sf Ġ{ ent max Ġ}} $ Ġwith Ġrespect Ġto Ġ$\ alpha Ġ$ :