Linked entities may be ambiguous or too common
ABSTRACTIVE
linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness., the linked entities may also be too common to be considered an entity.
EXTRACTIVE
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness. First, the extracted entities may be ambiguous. In the example, the entity “South Korean” is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\%$ of the top 100 entities and $53.6\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems. Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities.
Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness. Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities.
1806.05504
Introduction </s> Text Ġsummar ization Ġis Ġa Ġtask Ġto Ġgenerate Ġa Ġshorter Ġand Ġconcise Ġversion Ġof Ġa Ġtext Ġwhile Ġpreserving Ġthe Ġmeaning Ġof Ġthe Ġoriginal Ġtext . ĠThe Ġtask Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġsubt ask Ġbased Ġon Ġthe Ġapproach : Ġextract ive Ġand Ġabstract ive Ġsummar ization . ĠExtract ive Ġsummar ization Ġis Ġa Ġtask Ġto Ġcreate Ġsumm aries Ġby Ġpulling Ġout Ġsnippets Ġof Ġtext Ġform Ġthe Ġoriginal Ġtext Ġand Ġcombining Ġthem Ġto Ġform Ġa Ġsummary . ĠAbstract ive Ġsummar ization Ġasks Ġto Ġgenerate Ġsumm aries Ġfrom Ġscratch Ġwithout Ġthe Ġrestriction Ġto Ġuse Ġthe Ġavailable Ġwords Ġfrom Ġthe Ġoriginal Ġtext . ĠDue Ġto Ġthe Ġlimitations Ġof Ġextract ive Ġsummar ization Ġon Ġinco herent Ġtexts Ġand Ġunnatural Ġmethodology ĠB IB REF 0 Ġ, Ġthe Ġresearch Ġtrend Ġhas Ġshifted Ġtowards Ġabstract ive Ġsummar ization . </s> Sequ ence - to - sequence Ġmodels ĠB IB REF 1 Ġwith Ġattention Ġmechanism ĠB IB REF 2 Ġhave Ġfound Ġgreat Ġsuccess Ġin Ġgenerating Ġabstract ive Ġsumm aries , Ġboth Ġfrom Ġa Ġsingle Ġsentence ĠB IB REF 3 Ġand Ġfrom Ġa Ġlong Ġdocument Ġwith Ġmultiple Ġsentences ĠB IB REF 4 Ġ. ĠHowever , Ġwhen Ġgenerating Ġsumm aries , Ġit Ġis Ġnecessary Ġto Ġdetermine Ġthe Ġmain Ġtopic Ġand Ġto Ġs ift Ġout Ġunnecessary Ġinformation Ġthat Ġcan Ġbe Ġomitted . ĠSequence - to - sequence Ġmodels Ġhave Ġthe Ġtendency Ġto Ġinclude Ġall Ġthe Ġinformation , Ġrelevant Ġor Ġnot , Ġthat Ġare Ġfound Ġin Ġthe Ġoriginal Ġtext . ĠThis Ġmay Ġresult Ġto Ġuncon cise Ġsumm aries Ġthat Ġconcent rates Ġwrongly Ġon Ġirrelevant Ġtopics . ĠThe Ġproblem Ġis Ġespecially Ġsevere Ġwhen Ġsummar izing Ġlonger Ġtexts . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġuse Ġentities Ġfound Ġin Ġthe Ġoriginal Ġtext Ġto Ġinfer Ġthe Ġsummary Ġtopic , Ġmitigating Ġthe Ġaforementioned Ġproblem . ĠSpecifically , Ġwe Ġleverage Ġon Ġlinked Ġentities Ġextracted Ġby Ġemploying Ġa Ġreadily Ġavailable Ġentity Ġlinking Ġsystem . ĠThe Ġimportance Ġof Ġusing Ġlinked Ġentities Ġin Ġsummar ization Ġis Ġintuitive Ġand Ġcan Ġbe Ġexplained Ġby Ġlooking Ġat ĠFigure Ġ1 Ġas Ġan Ġexample . ĠFirst Ġ( O 1 Ġin Ġthe ĠFigure ), Ġaside Ġfrom Ġauxiliary Ġwords Ġto Ġconstruct Ġa Ġsentence , Ġa Ġsummary Ġis Ġmainly Ġcomposed Ġof Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext . ĠSecond Ġ( O 2 ), Ġwe Ġcan Ġdepict Ġthe Ġmain Ġtopic Ġof Ġthe Ġsummary Ġas Ġa Ġprobability Ġdistribution Ġof Ġrelevant Ġentities Ġfrom Ġthe Ġlist Ġof Ġentities . ĠFinally Ġ( O 3 ), Ġwe Ġcan Ġleverage Ġon Ġentity Ġcommons ense Ġlearned Ġfrom Ġa Ġseparate Ġlarge Ġknowledge Ġbase Ġsuch Ġas ĠWikipedia . </s> To Ġthis Ġend , Ġwe Ġpresent Ġa Ġmethod Ġto Ġeffectively Ġapply Ġlinked Ġentities Ġin Ġsequence - to - sequence Ġmodels , Ġcalled ĠEntity 2 Topic Ġ( E 2 T ). ĠE 2 T Ġis Ġa Ġmodule Ġthat Ġcan Ġbe Ġeasily Ġattached Ġto Ġany Ġsequence - to - sequence Ġbased Ġsummar ization Ġmodel . ĠThe Ġmodule Ġenc odes Ġthe Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext Ġby Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ), Ġconstructs Ġa Ġvector Ġrepresenting Ġthe Ġtopic Ġof Ġthe Ġsummary Ġto Ġbe Ġgenerated , Ġand Ġinforms Ġthe Ġdec oder Ġabout Ġthe Ġconstructed Ġtopic Ġvector . ĠDue Ġto Ġthe Ġimperfect ions Ġof Ġcurrent ĠE LS 's , Ġthe Ġextracted Ġlinked Ġentities Ġmay Ġbe Ġtoo Ġambiguous Ġand Ġcoarse Ġto Ġbe Ġconsidered Ġrelevant Ġto Ġthe Ġsummary . ĠWe Ġsolve Ġthis Ġissue Ġby Ġusing Ġentity Ġenc od ers Ġwith Ġselective Ġdis amb ig uation Ġand Ġby Ġconstructing Ġtopic Ġvectors Ġusing Ġfirm Ġattention . </s> We Ġexperiment Ġon Ġtwo Ġdatasets , ĠGig aw ord Ġand ĠCNN , Ġwith Ġvarying Ġlengths . ĠWe Ġshow Ġthat Ġapplying Ġour Ġmodule Ġto Ġa Ġsequence - to - sequence Ġmodel Ġwith Ġattention Ġmechanism Ġsignificantly Ġincreases Ġits Ġperformance Ġon Ġboth Ġdatasets . ĠMoreover , Ġwhen Ġcompared Ġwith Ġthe Ġstate - of - the - art Ġmodels Ġfor Ġeach Ġdataset , Ġthe Ġmodel Ġob tains Ġa Ġcomparable Ġperformance Ġon Ġthe ĠGig aw ord Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġshort , Ġand Ġoutper forms Ġall Ġcompeting Ġmodels Ġon Ġthe ĠCNN Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġlonger . ĠFurthermore , Ġwe Ġprovide Ġanalysis Ġon Ġhow Ġour Ġmodel Ġeffectively Ġuses Ġthe Ġextracted Ġlinked Ġentities Ġto Ġproduce Ġconcise Ġand Ġbetter Ġsumm aries . </s> Use fulness Ġof Ġlinked Ġentities Ġin Ġsummar ization </s> In Ġthe Ġnext Ġsubsections , Ġwe Ġpresent Ġdetailed Ġarguments Ġwith Ġempirical Ġand Ġpreviously Ġexamined Ġevid ences Ġon Ġthe Ġobservations Ġand Ġpossible Ġissues Ġwhen Ġusing Ġlinked Ġentities Ġextracted Ġby Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ) Ġfor Ġgenerating Ġabstract ive Ġsumm aries . ĠFor Ġthis Ġpurpose , Ġwe Ġuse Ġthe Ġdevelopment Ġsets Ġof Ġthe ĠGig aw ord Ġdataset Ġprovided Ġin ĠB IB REF 5 Ġand Ġof Ġthe ĠCNN Ġdataset Ġprovided Ġin ĠB IB REF 6 Ġas Ġthe Ġexperimental Ġdata Ġfor Ġquantitative Ġevidence Ġand Ġrefer Ġthe Ġreaders Ġto ĠFigure Ġ1 Ġas Ġthe Ġrunning Ġexample . </s> Obs erv ations </s> As Ġdiscussed Ġin ĠSection Ġ" Introduction " Ġ, Ġwe Ġfind Ġthree Ġobservations Ġthat Ġshow Ġthe Ġusefulness Ġof Ġlinked Ġentities Ġfor Ġabstract ive Ġsummar ization . </s> First , Ġsumm aries Ġare Ġmainly Ġcomposed Ġof Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext . ĠIn Ġthe Ġexample , Ġit Ġcan Ġbe Ġseen Ġthat Ġthe Ġsummary Ġcontains Ġfour Ġwords Ġthat Ġrefer Ġto Ġdifferent Ġentities . ĠIn Ġfact , Ġall Ġnoun Ġphrases Ġin Ġthe Ġsummary Ġmention Ġat Ġleast Ġone Ġlinked Ġentity . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġextract Ġlinked Ġentities Ġfrom Ġthe Ġoriginal Ġtext Ġand Ġcompare Ġthem Ġto Ġthe Ġnoun Ġphrases Ġfound Ġin Ġthe Ġsummary . ĠWe Ġreport Ġthat Ġ$ 77 . 1 \ % $ Ġand Ġ$ 75 . 1 \ % $ Ġof Ġthe Ġnoun Ġphrases Ġon Ġthe ĠGig aw ord Ġand ĠCNN Ġdatasets , Ġrespectively , Ġcontain Ġat Ġleast Ġone Ġlinked Ġentity , Ġwhich Ġconfirms Ġour Ġobservation . </s> Second , Ġlinked Ġentities Ġcan Ġbe Ġused Ġto Ġrepresent Ġthe Ġtopic Ġof Ġthe Ġsummary , Ġdefined Ġas Ġa Ġmult in omial Ġdistribution Ġover Ġentities , Ġas Ġgraph ically Ġshown Ġin Ġthe Ġexample , Ġwhere Ġthe Ġprobabilities Ġrefer Ġto Ġthe Ġrelevance Ġof Ġthe Ġentities . ĠEnt ities Ġhave Ġbeen Ġpreviously Ġused Ġto Ġrepresent Ġtopics ĠB IB REF 7 Ġ, Ġas Ġthey Ġcan Ġbe Ġutilized Ġas Ġa Ġcontrolled Ġvocabulary Ġof Ġthe Ġmain Ġtopics Ġin Ġa Ġdocument ĠB IB REF 8 Ġ. ĠIn Ġthe Ġexample , Ġwe Ġsee Ġthat Ġthe Ġentity ĠâĢ ľ J ae ĠSe o âĢ Ŀ Ġis Ġthe Ġmost Ġrelevant Ġbecause Ġit Ġis Ġthe Ġsubject Ġof Ġthe Ġsummary , Ġwhile Ġthe Ġentity ĠâĢ ľ South ĠKorean âĢ Ŀ Ġis Ġless Ġrelevant Ġbecause Ġit Ġis Ġless Ġimportant Ġwhen Ġconstructing Ġthe Ġsummary . </s> Third , Ġwe Ġcan Ġmake Ġuse Ġof Ġthe Ġentity Ġcommons ense Ġthat Ġcan Ġbe Ġlearned Ġas Ġa Ġcontinuous Ġvector Ġrepresentation Ġfrom Ġa Ġseparate Ġlarger Ġcorpus ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠIn Ġthe Ġexample , Ġif Ġwe Ġknow Ġthat Ġthe Ġentities ĠâĢ ľ Los ĠAngeles ĠDodgers âĢ Ŀ Ġand ĠâĢ ľ New ĠYork ĠMets âĢ Ŀ Ġare ĠAmerican Ġbaseball Ġteams Ġand ĠâĢ ľ J ae ĠSe o âĢ Ŀ Ġis Ġa Ġbaseball Ġplayer Ġassociated Ġwith Ġthe Ġteams , Ġthen Ġwe Ġcan Ġuse Ġthis Ġinformation Ġto Ġgenerate Ġmore Ġcoherent Ġsumm aries . ĠWe Ġfind Ġthat Ġ$ 76 . 0 \ % $ Ġof Ġthe Ġextracted Ġlinked Ġentities Ġare Ġcovered Ġby Ġthe Ġpre - trained Ġvectors Ġin Ġour Ġexperimental Ġdata , Ġproving Ġour Ġthird Ġobservation . </s> P ossible Ġissues </s> Despite Ġits Ġusefulness , Ġlinked Ġentities Ġextracted Ġfrom ĠE LS 's Ġhave Ġissues Ġbecause Ġof Ġlow Ġprecision Ġrates ĠB IB REF 11 Ġand Ġdesign Ġchallenges Ġin Ġtraining Ġdatasets ĠB IB REF 12 Ġ. ĠThese Ġissues Ġcan Ġbe Ġsummarized Ġinto Ġtwo Ġparts : Ġambiguity Ġand Ġco ars eness . </s> First , Ġthe Ġextracted Ġentities Ġmay Ġbe Ġambiguous . ĠIn Ġthe Ġexample , Ġthe Ġentity ĠâĢ ľ South ĠKorean âĢ Ŀ Ġis Ġambiguous Ġbecause Ġit Ġcan Ġrefer Ġto Ġboth Ġthe ĠSouth ĠKorean Ġperson Ġand Ġthe ĠSouth ĠKorean Ġlanguage , Ġamong Ġothers . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġextract Ġ( 1 ) Ġthe Ġtop Ġ100 Ġentities Ġbased Ġon Ġfrequency , Ġand Ġ( 2 ) Ġthe Ġentities Ġextracted Ġfrom Ġ100 Ġrandomly Ġselected Ġtexts , Ġand Ġcheck Ġwhether Ġthey Ġhave Ġdis amb ig uation Ġpages Ġin ĠWikipedia Ġor Ġnot . ĠWe Ġdiscover Ġthat Ġ$ 71 . 0 \ % $ Ġof Ġthe Ġtop Ġ100 Ġentities Ġand Ġ$ 53 . 6 \ % $ Ġof Ġthe Ġentities Ġpicked Ġat Ġrandom Ġhave Ġdis amb ig uation Ġpages , Ġwhich Ġshows Ġthat Ġmost Ġentities Ġare Ġprone Ġto Ġambiguity Ġproblems . </s> Second , Ġthe Ġlinked Ġentities Ġmay Ġalso Ġbe Ġtoo Ġcommon Ġto Ġbe Ġconsidered Ġan Ġentity . ĠThis Ġmay Ġintroduce Ġerrors Ġand Ġirre lev ance Ġto Ġthe Ġsummary . ĠIn Ġthe Ġexample , ĠâĢ ľ Wednesday âĢ Ŀ Ġis Ġerroneous Ġbecause Ġit Ġis Ġwrongly Ġlinked Ġto Ġthe Ġentity ĠâĢ ľ Wednesday ĠNight ĠBaseball âĢ Ŀ . ĠAlso , ĠâĢ ľ sw ap âĢ Ŀ Ġis Ġirrelevant Ġbecause Ġalthough Ġit Ġis Ġlinked Ġcorrectly Ġto Ġthe Ġentity ĠâĢ ľ Trade Ġ( Sports ) âĢ Ŀ , Ġit Ġis Ġtoo Ġcommon Ġand Ġirrelevant Ġwhen Ġgenerating Ġthe Ġsumm aries . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġrandomly Ġselect Ġ100 Ġdata Ġinstances Ġand Ġtag Ġthe Ġcorrectness Ġand Ġrelevance Ġof Ġextracted Ġentities Ġinto Ġone Ġof Ġfour Ġlabels : ĠA : Ġcorrect Ġand Ġrelevant , ĠB : Ġcorrect Ġand Ġsomewhat Ġrelevant , ĠC : Ġcorrect Ġbut Ġirrelevant , Ġand ĠD : Ġincorrect . ĠResults Ġshow Ġthat Ġ$ 29 . 4 \ % $ Ġ, Ġ$ 13 . 7 \ % $ Ġ, Ġ$ 30 . 0 \ % $ Ġ, Ġand Ġ$ 26 . 9 \ % $ Ġare Ġtagged Ġwith ĠA , ĠB , ĠC , Ġand ĠD , Ġrespectively , Ġwhich Ġshows Ġthat Ġthere Ġis Ġa Ġlarge Ġamount Ġof Ġincorrect Ġand Ġirrelevant Ġentities . </s> Our Ġmodel </s> To Ġsolve Ġthe Ġissues Ġdescribed Ġabove , Ġwe Ġpresent ĠEntity 2 Topic Ġ( E 2 T ), Ġa Ġmodule Ġthat Ġcan Ġbe Ġeasily Ġattached Ġto Ġany Ġsequence - to - sequence Ġbased Ġabstract ive Ġsummar ization Ġmodel . ĠE 2 T Ġenc odes Ġthe Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġtext Ġand Ġtransforms Ġthem Ġinto Ġa Ġsingle Ġtopic Ġvector . ĠThis Ġvector Ġis Ġultimately Ġconc aten ated Ġto Ġthe Ġdec oder Ġhidden Ġstate Ġvectors . ĠThe Ġmodule Ġcontains Ġtwo Ġsub modules Ġspecifically Ġfor Ġthe Ġissues Ġpresented Ġby Ġthe Ġentity Ġlinking Ġsystems : Ġthe Ġentity Ġencoding Ġsub module Ġwith Ġselective Ġdis amb ig uation Ġand Ġthe Ġpool ing Ġsub module Ġwith Ġfirm Ġattention . </s> Overall , Ġour Ġfull Ġarchitecture Ġcan Ġbe Ġillustrated Ġas Ġin ĠFigure Ġ2 Ġ, Ġwhich Ġconsists Ġof Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ), Ġa Ġsequence - to - sequence Ġwith Ġattention Ġmechanism Ġmodel , Ġand Ġthe ĠE 2 T Ġmodule . ĠWe Ġnote Ġthat Ġour Ġproposed Ġmodule Ġcan Ġbe Ġeasily Ġattached Ġto Ġmore Ġsophisticated Ġabstract ive Ġsummar ization Ġmodels ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġthat Ġare Ġbased Ġon Ġthe Ġtraditional Ġenc oder - dec oder Ġframework Ġand Ġconsequently Ġcan Ġproduce Ġbetter Ġresults . ĠThe Ġcode Ġof Ġthe Ġbase Ġmodel Ġand Ġthe ĠE 2 T Ġare Ġavailable Ġonline . </s> Base Ġmodel </s> As Ġour Ġbase Ġmodel , Ġwe Ġemploy Ġa Ġbasic Ġenc oder - dec oder ĠR NN Ġused Ġin Ġmost Ġneural Ġmachine Ġtranslation ĠB IB REF 2 Ġand Ġtext Ġsummar ization ĠB IB REF 15 Ġtasks . ĠWe Ġemploy Ġa Ġtwo - layer Ġbid irection al ĠGR U Ġ( Bi GR U ) Ġas Ġthe Ġrecurrent Ġunit Ġof Ġthe Ġenc oder . ĠThe ĠBi GR U Ġconsists Ġof Ġa Ġforward Ġand Ġbackward ĠGR U , Ġwhich Ġresults Ġto Ġsequences Ġof Ġforward Ġand Ġbackward Ġhidden Ġstates Ġ$( \ over right arrow { h } _ 1 , Ġ\ over right arrow { h } _ 2 , Ġ... , Ġ\ over right arrow { h } _ n ) $ Ġand Ġ$( \ over left arrow { h } _ 1 , Ġ\ over left arrow { h } _ 2 , Ġ... , Ġ\ over left arrow { h } _ n ) $ Ġ, Ġrespectively : Ġ$ Ġ\ over right arrow { h } _ i Ġ& = ĠGR U ( x _ i , Ġ\ over right arrow { h } _{ i - 1 }) Ġ\\ Ġ\ over left arrow { h } _ i Ġ& = ĠGR U ( x _ i , Ġ\ over left arrow { h } _{ i + 1 }) Ġ\ non umber Ġ$ </s> The Ġforward Ġand Ġbackward Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġget Ġthe Ġhidden Ġstate Ġvectors Ġof Ġthe Ġtokens Ġ( i . e . Ġ$ h _ i Ġ= Ġ[ \ over right arrow { h } _ i ; Ġ\ over left arrow { h } _ i ] $ Ġ). ĠThe Ġfinal Ġstates Ġof Ġthe Ġforward Ġand Ġbackward ĠGR U Ġare Ġalso Ġconc aten ated Ġto Ġcreate Ġthe Ġfinal Ġtext Ġrepresentation Ġvector Ġof Ġthe Ġenc oder Ġ$ s Ġ= Ġ[ \ over right arrow { h } _ n ; Ġ\ over left arrow { h } _ 1 ] $ Ġ. ĠThese Ġvalues Ġare Ġcalculated Ġper Ġlayer , Ġwhere Ġ$ x _ t $ Ġof Ġthe Ġsecond Ġlayer Ġis Ġ$ h _ t $ Ġof Ġthe Ġfirst Ġlayer . ĠThe Ġfinal Ġtext Ġrepresentation Ġvectors Ġare Ġprojected Ġby Ġa Ġfully Ġconnected Ġlayer Ġand Ġare Ġpassed Ġto Ġthe Ġdec oder Ġas Ġthe Ġinitial Ġhidden Ġstates Ġ$ s _ 0 Ġ= Ġs $ Ġ. </s> For Ġthe Ġdec oder , Ġwe Ġuse Ġa Ġtwo - layer Ġun i - direction al ĠGR U Ġwith Ġattention . ĠAt Ġeach Ġtime Ġstep Ġ$ t $ Ġ, Ġthe Ġprevious Ġtoken Ġ$ y _{ t - 1 } $ Ġ, Ġthe Ġprevious Ġhidden Ġstate Ġ$ s _{ t - 1 } $ Ġ, Ġand Ġthe Ġprevious Ġcontext Ġvector Ġ$ c _{ t - 1 } $ Ġare Ġpassed Ġto Ġa ĠGR U Ġto Ġcalculate Ġthe Ġnew Ġhidden Ġstate Ġ$ s _ t $ Ġ, Ġas Ġshown Ġin Ġthe Ġequation Ġbelow . </s> $$ s _ t Ġ= ĠGR U ( w _{ t - 1 }, Ġs _{ t - 1 }, Ġc _{ t - 1 }) Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ9 ) </s> The Ġcontext Ġvector Ġ$ c _ t $ Ġis Ġcomputed Ġusing Ġthe Ġadditive Ġattention Ġmechanism ĠB IB REF 2 Ġ, Ġwhich Ġmatches Ġthe Ġcurrent Ġdec oder Ġstate Ġ$ s _ t $ Ġand Ġeach Ġenc oder Ġstate Ġ$ h _ i $ Ġto Ġget Ġan Ġimportance Ġscore . ĠThe Ġscores Ġare Ġthen Ġpassed Ġto Ġa Ġsoft max Ġand Ġare Ġused Ġto Ġpool Ġthe Ġenc oder Ġstates Ġusing Ġweighted Ġsum . ĠThe Ġfinal Ġpooled Ġvector Ġis Ġthe Ġcontext Ġvector , Ġas Ġshown Ġin Ġthe Ġequations Ġbelow . Ġ$ Ġg _{ t , i } Ġ& = Ġv _ a ^ \ top Ġtan h ( W _ a Ġs _{ t - 1 } Ġ+ ĠU _ a Ġh _ i ) Ġ\\ Ġa _{ t , i } Ġ& = Ġ\ frac { exp ( g _{ t , i }) }{ \ sum Ġ_ i Ġexp ( g _{ t , i }) } Ġ\\ Ġc _ t Ġ& = Ġ\ sum Ġ_ i Ġa _{ t , i } Ġh _ i Ġ\ non umber Ġ$ </s> Finally , Ġthe Ġprevious Ġtoken Ġ$ y _{ t - 1 } $ Ġ, Ġthe Ġcurrent Ġcontext Ġvector Ġ$ c _ t $ Ġ, Ġand Ġthe Ġcurrent Ġdec oder Ġstate Ġ$ s _ t $ Ġare Ġused Ġto Ġgenerate Ġthe Ġcurrent Ġword Ġ$ y _ t $ Ġwith Ġa Ġsoft max Ġlayer Ġover Ġthe Ġdec oder Ġvocabulary , Ġas Ġshown Ġbelow . Ġ$ Ġo _ t Ġ& = ĠW _ w Ġw _{ t - 1 } Ġ+ ĠW _ c Ġc _ t Ġ+ ĠW _ s Ġs _ t Ġ\\ Ġp ( y _ t Ġ| Ġy _{ < t }) Ġ& = Ġsoft max ( W _ o Ġo _ t ) Ġ\ non umber Ġ$ </s> Entity Ġencoding Ġsub module </s> After Ġperforming Ġentity Ġlinking Ġto Ġthe Ġinput Ġtext Ġusing Ġthe ĠE LS , Ġwe Ġreceive Ġa Ġsequential Ġlist Ġof Ġlinked Ġentities , Ġarranged Ġbased Ġon Ġtheir Ġlocation Ġin Ġthe Ġtext . ĠWe Ġembed Ġthese Ġentities Ġto Ġ$ d $ Ġ- dimensional Ġvectors Ġ$ E Ġ= Ġ\ l brace Ġe _ 1 , Ġe _ 2 , Ġ... , Ġe _ m \ r brace Ġ$ Ġwhere Ġ$ e _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġ. ĠSince Ġthese Ġentities Ġmay Ġstill Ġcontain Ġambiguity , Ġit Ġis Ġnecessary Ġto Ġresolve Ġthem Ġbefore Ġapplying Ġthem Ġto Ġthe Ġbase Ġmodel . ĠBased Ġon Ġthe Ġidea Ġthat Ġan Ġambiguous Ġentity Ġcan Ġbe Ġdis amb ig uated Ġusing Ġits Ġneighboring Ġentities , Ġwe Ġintroduce Ġtwo Ġkinds Ġof Ġdis amb ig uating Ġenc od ers Ġbelow . </s> One Ġway Ġto Ġdis amb ig uate Ġan Ġentity Ġis Ġby Ġusing Ġall Ġthe Ġother Ġentities , Ġputting Ġmore Ġimportance Ġto Ġentities Ġthat Ġare Ġnearer . ĠFor Ġthis Ġpurpose , Ġwe Ġemploy Ġan ĠR NN - based Ġmodel Ġto Ġglobally Ġdis amb ig uate Ġthe Ġentities . ĠSpecifically , Ġwe Ġuse ĠBi GR U Ġand Ġconc aten ate Ġthe Ġforward Ġand Ġbackward Ġhidden Ġstate Ġvectors Ġas Ġthe Ġnew Ġentity Ġvector : Ġ$ Ġ\ over right arrow { h } _ i Ġ& = ĠGR U ( e _ i , Ġ\ over right arrow { h } _{ i - 1 }) Ġ\\ Ġ\ over left arrow { h } _ i Ġ& = ĠGR U ( e _ i , Ġ\ over left arrow { h } _{ i + 1 }) Ġ\\ Ġe ^ {\ prime Ġ} _ i Ġ& = Ġ[ \ over right arrow { h } _ i ; Ġ\ over left arrow { h } _ i ] Ġ\ non umber Ġ$ </s> Another Ġway Ġto Ġdis amb ig uate Ġan Ġentity Ġis Ġby Ġusing Ġonly Ġthe Ġdirect Ġneighbors Ġof Ġthe Ġentity , Ġputting Ġno Ġimportance Ġvalue Ġto Ġentities Ġthat Ġare Ġfar . ĠTo Ġdo Ġthis , Ġwe Ġemploy Ġa ĠCNN - based Ġmodel Ġto Ġlocally Ġdis amb ig uate Ġthe Ġentities . ĠSpecifically , Ġwe Ġdo Ġthe Ġconv olution Ġoperation Ġusing Ġfilter Ġmat rices Ġ$ W _ f Ġ\ in Ġ\ math bb Ġ{ R } ^{ h Ġ\ times Ġd } $ Ġwith Ġfilter Ġsize Ġ$ h $ Ġto Ġa Ġwindow Ġof Ġ$ h $ Ġwords . ĠWe Ġdo Ġthis Ġfor Ġdifferent Ġsizes Ġof Ġ$ h $ Ġ. ĠThis Ġproduces Ġnew Ġfeature Ġvectors Ġ$ c _{ i , h } $ Ġas Ġshown Ġbelow , Ġwhere Ġ$ f ( .) $ Ġis Ġa Ġnon - linear Ġfunction : Ġ$ Ġc _{ i , h } Ġ= Ġf ([ e _{ i -( h - 1 )/ 2 }; Ġ... ; Ġe _{ i + h ( + 1 )/ 2 } ] ^ \ top ĠW _ f Ġ+ Ġb _ f ) Ġ\ non umber Ġ$ </s> The Ġconv olution Ġoperation Ġreduces Ġthe Ġnumber Ġof Ġentities Ġdifferently Ġdepending Ġon Ġthe Ġfilter Ġsize Ġ$ h $ Ġ. ĠTo Ġprevent Ġloss Ġof Ġinformation Ġand Ġto Ġproduce Ġthe Ġsame Ġamount Ġof Ġfeature Ġvectors Ġ$ c _{ i , h } $ Ġ, Ġwe Ġpad Ġthe Ġentity Ġlist Ġdynamically Ġsuch Ġthat Ġwhen Ġthe Ġfilter Ġsize Ġis Ġ$ h $ Ġ, Ġthe Ġnumber Ġof Ġpadd ings Ġon Ġeach Ġside Ġis Ġ$( h - 1 )/ 2 $ Ġ. ĠThe Ġfilter Ġsize Ġ$ h $ Ġtherefore Ġrefers Ġto Ġthe Ġnumber Ġof Ġentities Ġused Ġto Ġdis amb ig uate Ġa Ġmiddle Ġentity . ĠFinally , Ġwe Ġconc aten ate Ġall Ġfeature Ġvectors Ġof Ġdifferent Ġ$ h $ Ġ' s Ġfor Ġeach Ġ$ i $ Ġas Ġthe Ġnew Ġentity Ġvector : Ġ$ Ġe ^ {\ prime Ġ} _ i Ġ= Ġ[ c _{ i , h _ 1 }; Ġc _{ i , Ġh _ 2 }; Ġ... ] Ġ\ non umber Ġ$ </s> The Ġquestion Ġon Ġwhich Ġdis amb ig uating Ġenc oder Ġis Ġbetter Ġhas Ġbeen Ġa Ġdebate ; Ġsome Ġargued Ġthat Ġusing Ġonly Ġthe Ġlocal Ġcontext Ġis Ġappropriate ĠB IB REF 16 Ġwhile Ġsome Ġclaimed Ġthat Ġadditionally Ġusing Ġglobal Ġcontext Ġalso Ġhelps ĠB IB REF 17 Ġ. ĠThe ĠR NN - based Ġenc oder Ġis Ġgood Ġas Ġit Ġsmart ly Ġmakes Ġuse Ġof Ġall Ġentities , Ġhowever Ġit Ġmay Ġperform Ġbad Ġwhen Ġthere Ġare Ġmany Ġentities Ġas Ġit Ġintroduces Ġnoise Ġwhen Ġusing Ġa Ġfar Ġentity Ġduring Ġdis amb ig uation . ĠThe ĠCNN - based Ġenc oder Ġis Ġgood Ġas Ġit Ġminim izes Ġthe Ġnoise Ġby Ġtotally Ġignoring Ġfar Ġentities Ġwhen Ġdis amb ig uating , Ġhowever Ġdetermining Ġthe Ġappropriate Ġfilter Ġsizes Ġ$ h $ Ġneeds Ġengineering . ĠOverall , Ġwe Ġargue Ġthat Ġwhen Ġthe Ġinput Ġtext Ġis Ġshort Ġ( e . g . Ġa Ġsentence ), Ġboth Ġenc od ers Ġperform Ġcompar ably , Ġotherwise Ġwhen Ġthe Ġinput Ġtext Ġis Ġlong Ġ( e . g . Ġa Ġdocument ), Ġthe ĠCNN - based Ġenc oder Ġperforms Ġbetter . </s> It Ġis Ġobvious Ġthat Ġnot Ġall Ġentities Ġneed Ġto Ġbe Ġdis amb ig uated . ĠWhen Ġa Ġcorrectly Ġlinked Ġand Ġalready Ġadequately Ġdis amb ig uated Ġentity Ġis Ġdis amb ig uated Ġagain , Ġit Ġwould Ġmake Ġthe Ġentity Ġvery Ġcontext - specific Ġand Ġmight Ġnot Ġbe Ġsuitable Ġfor Ġthe Ġsummar ization Ġtask . ĠOur Ġentity Ġencoding Ġsub module Ġtherefore Ġuses Ġa Ġselective Ġmechanism Ġthat Ġdecides Ġwhether Ġto Ġuse Ġthe Ġdis amb ig uating Ġenc oder Ġor Ġnot . ĠThis Ġis Ġdone Ġby Ġintroducing Ġa Ġselective Ġdis amb ig uation Ġgate Ġ$ d $ Ġ. ĠThe Ġfinal Ġentity Ġvector Ġ$\ t ilde { e } _ i $ Ġis Ġcalculated Ġas Ġthe Ġlinear Ġtransformation Ġof Ġ$ e _ i $ Ġand Ġ$ e ^ {\ prime Ġ} _ i $ Ġ: Ġ$ Ġe ^ {\ prime Ġ} _ i Ġ& = Ġenc oder ( e _ i ) Ġ\\ Ġd Ġ& = Ġ\ s igma Ġ( W _ d Ġe ^ {\ prime Ġ} _ i Ġ+ Ġb _ d ) Ġ\\ Ġ\ t ilde { e } _ i Ġ& = Ġd Ġ\ times Ġf ( W _ x Ġe _ i Ġ+ Ġb _ x ) Ġ+ Ġ\\ Ġ& Ġ\ quad Ġ( 1 - d ) Ġ\ times Ġf ( W _ y Ġe ^ {\ prime Ġ} _ i Ġ+ Ġb _ y ) Ġ\ non umber Ġ$ </s> The Ġfull Ġentity Ġencoding Ġsub module Ġis Ġillustrated Ġin ĠFigure Ġ3 Ġ. ĠUltimately , Ġthe Ġsub module Ġoutputs Ġthe Ġdis amb ig uated Ġentity Ġvectors Ġ$\ t ilde { E } Ġ= Ġ\ l brace Ġ\ t ilde { e } _ 1 , Ġ\ t ilde { e } _ 2 , Ġ... , Ġ\ t ilde { e } _ m \ r brace Ġ$ Ġ. </s> Pool ing Ġsub module </s> The Ġentity Ġvectors Ġ$\ t ilde { E } $ Ġare Ġpooled Ġto Ġcreate Ġa Ġsingle Ġtopic Ġvector Ġ$ t $ Ġthat Ġrepresents Ġthe Ġtopic Ġof Ġthe Ġsummary . ĠOne Ġpossible Ġpool ing Ġtechnique Ġis Ġto Ġuse Ġsoft Ġattention ĠB IB REF 18 Ġon Ġthe Ġvectors Ġto Ġdetermine Ġthe Ġimportance Ġvalue Ġof Ġeach Ġvector , Ġwhich Ġcan Ġbe Ġdone Ġby Ġmatching Ġeach Ġentity Ġvector Ġwith Ġthe Ġtext Ġvector Ġ$ s $ Ġfrom Ġthe Ġtext Ġenc oder Ġas Ġthe Ġcontext Ġvector . ĠThe Ġentity Ġvectors Ġare Ġthen Ġpooled Ġusing Ġweighted Ġsum . ĠOne Ġproblem Ġwith Ġsoft Ġattention Ġis Ġthat Ġit Ġconsiders Ġall Ġentity Ġvectors Ġwhen Ġconstructing Ġthe Ġtopic Ġvector . ĠHowever , Ġnot Ġall Ġentities Ġare Ġimportant Ġand Ġnecessary Ġwhen Ġgenerating Ġsumm aries . ĠMoreover , Ġa Ġnumber Ġof Ġthese Ġentities Ġmay Ġbe Ġerroneous Ġand Ġirrelevant , Ġas Ġreported Ġin ĠSection Ġ" Related Ġwork " Ġ. ĠSoft Ġattention Ġgives Ġnon - neg lig ible Ġimportant Ġscores Ġto Ġthese Ġentities , Ġthus Ġadds Ġunnecessary Ġnoise Ġto Ġthe Ġconstruction Ġof Ġthe Ġtopic Ġvector . </s> Our Ġpool ing Ġsub module Ġinstead Ġuses Ġfirm Ġattention Ġmechanism Ġto Ġconsider Ġonly Ġtop Ġ$ k $ Ġentities Ġwhen Ġconstructing Ġthe Ġtopic Ġvector . ĠThis Ġis Ġdone Ġin Ġa Ġdifferent iable Ġway Ġas Ġfollows : Ġ$ ĠG Ġ& = Ġv _ a ^ \ top Ġtan h ( W _ a Ġ\ t ilde { E } Ġ+ ĠU _ a Ġs ) Ġ\\ ĠK Ġ& = Ġtop \ _ k ( G ) Ġ\\ ĠP Ġ& = Ġsparse \ _ vector ( K , Ġ0 , Ġ- \ in fty Ġ) Ġ\\ Ġg ^ {\ prime Ġ} _ i Ġ& = Ġg _ i Ġ+ Ġp _ i Ġ\\ Ġa _ i Ġ& = Ġ\ frac { exp ( g ^ {\ prime Ġ} _ i ) }{ \ sum Ġ_ i Ġexp ( g ^ {\ prime Ġ} _ i )} Ġ\\ Ġt Ġ& = Ġ\ sum Ġ_ i Ġa _ i Ġ\ t ilde { e } _ i Ġ\ non umber Ġ$ </s> where Ġthe Ġfunctions Ġ$ K Ġ= Ġtop \ _ k ( G ) $ Ġgets Ġthe Ġindices Ġof Ġthe Ġtop Ġ$ k $ Ġvectors Ġin Ġ$ G $ Ġand Ġ$ P Ġ= Ġsparse \ _ vector ( K , 0 ,- \ in fty Ġ) $ Ġcreates Ġa Ġsparse Ġvector Ġwhere Ġthe Ġvalues Ġof Ġ$ K $ Ġis Ġ0 Ġand Ġ$ - \ in fty Ġ$ Ġotherwise . ĠThe Ġsparse Ġvector Ġ$ P $ Ġis Ġadded Ġto Ġthe Ġoriginal Ġimportance Ġscore Ġvector Ġ$ G $ Ġto Ġcreate Ġa Ġnew Ġimportance Ġscore Ġvector . ĠIn Ġthis Ġnew Ġvector , Ġimportant Ġscores Ġof Ġnon - top Ġ$ k $ 0 Ġentities Ġare Ġ$ k $ 1 Ġ. ĠWhen Ġsoft max Ġis Ġapplied , Ġthis Ġgives Ġvery Ġsmall , Ġnegligible , Ġand Ġclose - to - zero Ġvalues Ġto Ġnon - top Ġ$ k $ 2 Ġentities . ĠThe Ġvalue Ġ$ k $ 3 Ġdepends Ġon Ġthe Ġlengths Ġof Ġthe Ġinput Ġtext Ġand Ġsummary . ĠMoreover , Ġwhen Ġ$ k $ 4 Ġincreases Ġtowards Ġinfinity , Ġfirm Ġattention Ġbecomes Ġsoft Ġattention . ĠWe Ġdecide Ġ$ k $ 5 Ġempir ically Ġ( see ĠSection Ġ" Exper imental Ġsettings " Ġ). </s> Ext ending Ġfrom Ġthe Ġbase Ġmodel </s> Entity 2 Topic Ġmodule Ġextends Ġthe Ġbase Ġmodel Ġas Ġfollows . ĠThe Ġfinal Ġtext Ġrepresentation Ġvector Ġ$ s $ Ġis Ġused Ġas Ġa Ġcontext Ġvector Ġwhen Ġconstructing Ġthe Ġtopic Ġvector Ġ$ t $ Ġin Ġthe Ġpool ing Ġsub module . ĠThe Ġtopic Ġvector Ġ$ t $ Ġis Ġthen Ġconc aten ated Ġto Ġthe Ġdec oder Ġhidden Ġstate Ġvectors Ġ$ s _ i $ Ġ, Ġi . e . Ġ$ s ^ {\ prime Ġ} _ i Ġ= Ġ[ s _ i ; Ġt ] $ Ġ. ĠThe Ġconc aten ated Ġvector Ġis Ġfinally Ġused Ġto Ġcreate Ġthe Ġoutput Ġvector : </s> $$ o _ i Ġ= ĠW _ w Ġw _{ i - 1 } Ġ+ ĠW _ c Ġc _ i Ġ+ ĠW _ s Ġs ^ {\ prime Ġ} _ i Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ18 ) </s> Related Ġwork </s> Due Ġto Ġits Ġrecent Ġsuccess , Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġused Ġwith Ġcompetitive Ġresults Ġon Ġabstract ive Ġsummar ization . ĠA Ġneural Ġattention Ġmodel Ġwas Ġfirst Ġapplied Ġto Ġthe Ġtask , Ġeasily Ġachieving Ġstate - of - the - art Ġperformance Ġon Ġmultiple Ġdatasets ĠB IB REF 5 Ġ. ĠThe Ġmodel Ġhas Ġbeen Ġextended Ġto Ġinstead Ġuse Ġrecurrent Ġneural Ġnetwork Ġas Ġdec oder ĠB IB REF 3 Ġ. ĠThe Ġmodel Ġwas Ġfurther Ġextended Ġto Ġuse Ġa Ġfull ĠR NN Ġenc oder - dec oder Ġframework Ġand Ġfurther Ġenhancements Ġthrough Ġlex ical Ġand Ġstatistical Ġfeatures ĠB IB REF 15 Ġ. ĠThe Ġcurrent Ġstate - of - the - art Ġperformance Ġis Ġachieved Ġby Ġselectively Ġencoding Ġwords Ġas Ġa Ġprocess Ġof Ġdist illing Ġsalient Ġinformation ĠB IB REF 13 Ġ. </s> Ne ural Ġabstract ive Ġsummar ization Ġmodels Ġhave Ġalso Ġbeen Ġexplored Ġto Ġsummarize Ġlonger Ġdocuments . ĠWord Ġextraction Ġmodels Ġhave Ġbeen Ġpreviously Ġexplored , Ġperforming Ġworse Ġthan Ġsentence Ġextraction Ġmodels ĠB IB REF 19 Ġ. ĠHier arch ical Ġattention - based Ġrecurrent Ġneural Ġnetworks Ġhave Ġalso Ġbeen Ġapplied Ġto Ġthe Ġtask , Ġowing Ġto Ġthe Ġidea Ġthat Ġthere Ġare Ġmultiple Ġsentences Ġin Ġa Ġdocument ĠB IB REF 15 Ġ. ĠFinally , Ġdistraction - based Ġmodels Ġwere Ġproposed Ġto Ġenable Ġmodels Ġto Ġtraverse Ġthe Ġtext Ġcontent Ġand Ġgrasp Ġthe Ġoverall Ġmeaning ĠB IB REF 4 Ġ. ĠThe Ġcurrent Ġstate - of - the - art Ġperformance Ġis Ġachieved Ġby Ġa Ġgraph - based Ġattention al Ġneural Ġmodel , Ġconsidering Ġthe Ġkey Ġfactors Ġof Ġdocument Ġsummar ization Ġsuch Ġas Ġsal iency , Ġflu ency Ġand Ġnovelty ĠB IB REF 14 Ġ. </s> Previous Ġstudies Ġon Ġthe Ġsummar ization Ġtasks Ġhave Ġonly Ġused Ġentities Ġin Ġthe Ġpre processing Ġstage Ġto Ġanonym ize Ġthe Ġdataset ĠB IB REF 15 Ġand Ġto Ġmitigate Ġout - of - voc abulary Ġproblems ĠB IB REF 14 Ġ. ĠLink ed Ġentities Ġfor Ġsummar ization Ġare Ġstill Ġnot Ġproperly Ġexplored Ġand Ġwe Ġare Ġthe Ġfirst Ġto Ġuse Ġlinked Ġentities Ġto Ġimprove Ġthe Ġperformance Ġof Ġthe Ġsummar izer . </s> Results </s> We Ġreport Ġthe ĠR OU GE ĠF 1 Ġscores Ġfor Ġboth Ġdatasets Ġof Ġall Ġthe Ġcompeting Ġmodels Ġusing ĠR OU GE ĠF 1 Ġscores ĠB IB REF 27 Ġ. ĠWe Ġreport Ġthe Ġresults Ġon Ġthe ĠGig aw ord Ġand Ġthe ĠCNN Ġdataset Ġin ĠTable Ġ2 Ġand ĠTable Ġ3 Ġ, Ġrespectively . ĠIn ĠGig aw ord Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġshort , Ġour Ġbest Ġmodel Ġachieves Ġa Ġcomparable Ġperformance Ġwith Ġthe Ġcurrent Ġstate - of - the - art . ĠIn ĠCNN Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġlonger , Ġour Ġbest Ġmodel Ġoutper forms Ġall Ġthe Ġprevious Ġmodels . ĠWe Ġemphasize Ġthat ĠE 2 T Ġmodule Ġis Ġeasily Ġattach able Ġto Ġbetter Ġmodels , Ġand Ġwe Ġexpect ĠE 2 T Ġto Ġimprove Ġtheir Ġperformance Ġas Ġwell . ĠOverall , ĠE 2 T Ġachieves Ġa Ġsignificant Ġimprovement Ġover Ġthe Ġbaseline Ġmodel Ġbase , Ġwith Ġat Ġleast Ġ2 ĠR OU GE - 1 Ġpoints Ġincrease Ġin Ġthe ĠGig aw ord Ġdataset Ġand Ġ6 ĠR OU GE - 1 Ġpoints Ġincrease Ġin Ġthe ĠCNN Ġdataset . ĠIn Ġfact , Ġall Ġvariants Ġof ĠE 2 T Ġgain Ġimprovements Ġover Ġthe Ġbaseline , Ġimplying Ġthat Ġleveraging Ġon Ġlinked Ġentities Ġimproves Ġthe Ġperformance Ġof Ġthe Ġsummar izer . ĠAmong Ġthe Ġmodel Ġvariants , Ġthe ĠCNN - based Ġenc oder Ġwith Ġselective Ġdis amb ig uation Ġand Ġfirm Ġattention Ġperforms Ġthe Ġbest . </s> Aut omatic Ġevaluation Ġon Ġthe ĠGig aw ord Ġdataset Ġshows Ġthat Ġthe ĠCNN Ġand ĠR NN Ġvariants Ġof Ġbase + E 2 T Ġhave Ġsimilar Ġperformance . ĠTo Ġbreak Ġthe Ġtie Ġbetween Ġboth Ġmodels , Ġwe Ġalso Ġconduct Ġhuman Ġevaluation Ġon Ġthe ĠGig aw ord Ġdataset . ĠWe Ġinstruct Ġtwo Ġannot ators Ġto Ġread Ġthe Ġinput Ġsentence Ġand Ġrank Ġthe Ġcompeting Ġsumm aries Ġfrom Ġfirst Ġto Ġlast Ġaccording Ġto Ġtheir Ġrelevance Ġand Ġflu ency : Ġ( a ) Ġthe Ġoriginal Ġsummary Ġgold , Ġand Ġfrom Ġmodels Ġ( b ) Ġbase , Ġ( c ) Ġbase + E 2 T c nn , Ġand Ġ( d ) Ġbase + E 2 Tr nn . ĠWe Ġthen Ġcompute Ġ( i ) Ġthe Ġproportion Ġof Ġevery Ġranking Ġof Ġeach Ġmodel Ġand Ġ( ii ) Ġthe Ġmean Ġrank Ġof Ġeach Ġmodel . ĠThe Ġresults Ġare Ġreported Ġin ĠTable Ġ4 Ġ. ĠThe Ġmodel Ġwith Ġthe Ġbest Ġmean Ġrank Ġis Ġbase + E 2 T c nn , Ġfollowed Ġby Ġgold , Ġthen Ġby Ġbase + E 2 Tr nn Ġand Ġbase , Ġrespectively . ĠWe Ġalso Ġperform ĠAN OVA Ġand Ġpost - h oc ĠTu key Ġtests Ġto Ġshow Ġthat Ġthe ĠCNN Ġvariant Ġis Ġsignificantly Ġ( Ġ$ p < 0 . 01 $ Ġ) Ġbetter Ġthan Ġthe ĠR NN Ġvariant Ġand Ġthe Ġbase Ġmodel . ĠThe ĠR NN Ġvariant Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġthe ĠCNN Ġvariant , Ġcontrary Ġto Ġthe Ġautomatic ĠR OU GE Ġevaluation Ġabove . ĠInterestingly , Ġthe ĠCNN Ġvariant Ġproduces Ġbetter Ġ( but Ġwith Ġno Ġsignificant Ġdifference ) Ġsumm aries Ġthan Ġthe Ġgold Ġsumm aries . ĠWe Ġposit Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġthe Ġarticle Ġtitle Ġdoes Ġnot Ġcorrespond Ġto Ġthe Ġsummary Ġof Ġthe Ġfirst Ġsentence . </s> Conclusion </s> We Ġproposed Ġto Ġleverage Ġon Ġlinked Ġentities Ġto Ġimprove Ġthe Ġperformance Ġof Ġsequence - to - sequence Ġmodels Ġon Ġneural Ġabstract ive Ġsummar ization Ġtask . ĠLink ed Ġentities Ġare Ġused Ġto Ġguide Ġthe Ġdecoding Ġprocess Ġbased Ġon Ġthe Ġsummary Ġtopic Ġand Ġcommons ense Ġlearned Ġfrom Ġa Ġknowledge Ġbase . ĠWe Ġintroduced ĠEntity 2 Topic Ġ( E 2 T ), Ġa Ġmodule Ġthat Ġis Ġeasily Ġattach able Ġto Ġany Ġmodel Ġusing Ġan Ġenc oder - dec oder Ġframework . ĠE 2 T Ġapplies Ġlinked Ġentities Ġinto Ġthe Ġsummar izer Ġby Ġencoding Ġthe Ġentities Ġwith Ġselective Ġdis amb ig uation Ġand Ġpool ing Ġthem Ġinto Ġone Ġsummary Ġtopic Ġvector Ġwith Ġfirm Ġattention Ġmechanism . ĠWe Ġshowed Ġthat Ġby Ġapplying ĠE 2 T Ġto Ġa Ġbasic Ġsequence - to - sequence Ġmodel , Ġwe Ġachieve Ġsignificant Ġimprovements Ġover Ġthe Ġbase Ġmodel Ġand Ġconsequently Ġachieve Ġa Ġcomparable Ġperformance Ġwith Ġmore Ġcomplex Ġsummar ization Ġmodels . </s> Acknowled gement </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġfeedback . ĠThis Ġwork Ġwas Ġsupported Ġby ĠMicrosoft ĠResearch , Ġand ĠInstitute Ġfor ĠInformation Ġcommunications ĠTechnology ĠPromotion Ġ( I IT P ) Ġgrant Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( No . 2017 - 0 - 01 778 Ġ, ĠDevelopment Ġof ĠExplain able ĠHuman level ĠDeep ĠMachine ĠLearning ĠIn ference ĠFramework ). ĠS . ĠH wang Ġis Ġa Ġcorresponding Ġauthor .
Introduction Text summarization is a task to generate a shorter and concise version of a text while preserving the meaning of the original text. The task can be divided into two subtask based on the approach: extractive and abstractive summarization. Extractive summarization is a task to create summaries by pulling out snippets of text form the original text and combining them to form a summary. Abstractive summarization asks to generate summaries from scratch without the restriction to use the available words from the original text. Due to the limitations of extractive summarization on incoherent texts and unnatural methodology BIBREF0 , the research trend has shifted towards abstractive summarization. Sequence-to-sequence models BIBREF1 with attention mechanism BIBREF2 have found great success in generating abstractive summaries, both from a single sentence BIBREF3 and from a long document with multiple sentences BIBREF4 . However, when generating summaries, it is necessary to determine the main topic and to sift out unnecessary information that can be omitted. Sequence-to-sequence models have the tendency to include all the information, relevant or not, that are found in the original text. This may result to unconcise summaries that concentrates wrongly on irrelevant topics. The problem is especially severe when summarizing longer texts. In this paper, we propose to use entities found in the original text to infer the summary topic, mitigating the aforementioned problem. Specifically, we leverage on linked entities extracted by employing a readily available entity linking system. The importance of using linked entities in summarization is intuitive and can be explained by looking at Figure 1 as an example. First (O1 in the Figure), aside from auxiliary words to construct a sentence, a summary is mainly composed of linked entities extracted from the original text. Second (O2), we can depict the main topic of the summary as a probability distribution of relevant entities from the list of entities. Finally (O3), we can leverage on entity commonsense learned from a separate large knowledge base such as Wikipedia. To this end, we present a method to effectively apply linked entities in sequence-to-sequence models, called Entity2Topic (E2T). E2T is a module that can be easily attached to any sequence-to-sequence based summarization model. The module encodes the entities extracted from the original text by an entity linking system (ELS), constructs a vector representing the topic of the summary to be generated, and informs the decoder about the constructed topic vector. Due to the imperfections of current ELS's, the extracted linked entities may be too ambiguous and coarse to be considered relevant to the summary. We solve this issue by using entity encoders with selective disambiguation and by constructing topic vectors using firm attention. We experiment on two datasets, Gigaword and CNN, with varying lengths. We show that applying our module to a sequence-to-sequence model with attention mechanism significantly increases its performance on both datasets. Moreover, when compared with the state-of-the-art models for each dataset, the model obtains a comparable performance on the Gigaword dataset where the texts are short, and outperforms all competing models on the CNN dataset where the texts are longer. Furthermore, we provide analysis on how our model effectively uses the extracted linked entities to produce concise and better summaries. Usefulness of linked entities in summarization In the next subsections, we present detailed arguments with empirical and previously examined evidences on the observations and possible issues when using linked entities extracted by an entity linking system (ELS) for generating abstractive summaries. For this purpose, we use the development sets of the Gigaword dataset provided in BIBREF5 and of the CNN dataset provided in BIBREF6 as the experimental data for quantitative evidence and refer the readers to Figure 1 as the running example. Observations As discussed in Section "Introduction" , we find three observations that show the usefulness of linked entities for abstractive summarization. First, summaries are mainly composed of linked entities extracted from the original text. In the example, it can be seen that the summary contains four words that refer to different entities. In fact, all noun phrases in the summary mention at least one linked entity. In our experimental data, we extract linked entities from the original text and compare them to the noun phrases found in the summary. We report that $77.1\%$ and $75.1\%$ of the noun phrases on the Gigaword and CNN datasets, respectively, contain at least one linked entity, which confirms our observation. Second, linked entities can be used to represent the topic of the summary, defined as a multinomial distribution over entities, as graphically shown in the example, where the probabilities refer to the relevance of the entities. Entities have been previously used to represent topics BIBREF7 , as they can be utilized as a controlled vocabulary of the main topics in a document BIBREF8 . In the example, we see that the entity “Jae Seo” is the most relevant because it is the subject of the summary, while the entity “South Korean” is less relevant because it is less important when constructing the summary. Third, we can make use of the entity commonsense that can be learned as a continuous vector representation from a separate larger corpus BIBREF9 , BIBREF10 . In the example, if we know that the entities “Los Angeles Dodgers” and “New York Mets” are American baseball teams and “Jae Seo” is a baseball player associated with the teams, then we can use this information to generate more coherent summaries. We find that $76.0\%$ of the extracted linked entities are covered by the pre-trained vectors in our experimental data, proving our third observation. Possible issues Despite its usefulness, linked entities extracted from ELS's have issues because of low precision rates BIBREF11 and design challenges in training datasets BIBREF12 . These issues can be summarized into two parts: ambiguity and coarseness. First, the extracted entities may be ambiguous. In the example, the entity “South Korean” is ambiguous because it can refer to both the South Korean person and the South Korean language, among others. In our experimental data, we extract (1) the top 100 entities based on frequency, and (2) the entities extracted from 100 randomly selected texts, and check whether they have disambiguation pages in Wikipedia or not. We discover that $71.0\%$ of the top 100 entities and $53.6\%$ of the entities picked at random have disambiguation pages, which shows that most entities are prone to ambiguity problems. Second, the linked entities may also be too common to be considered an entity. This may introduce errors and irrelevance to the summary. In the example, “Wednesday” is erroneous because it is wrongly linked to the entity “Wednesday Night Baseball”. Also, “swap” is irrelevant because although it is linked correctly to the entity “Trade (Sports)”, it is too common and irrelevant when generating the summaries. In our experimental data, we randomly select 100 data instances and tag the correctness and relevance of extracted entities into one of four labels: A: correct and relevant, B: correct and somewhat relevant, C: correct but irrelevant, and D: incorrect. Results show that $29.4\%$ , $13.7\%$ , $30.0\%$ , and $26.9\%$ are tagged with A, B, C, and D, respectively, which shows that there is a large amount of incorrect and irrelevant entities. Our model To solve the issues described above, we present Entity2Topic (E2T), a module that can be easily attached to any sequence-to-sequence based abstractive summarization model. E2T encodes the linked entities extracted from the text and transforms them into a single topic vector. This vector is ultimately concatenated to the decoder hidden state vectors. The module contains two submodules specifically for the issues presented by the entity linking systems: the entity encoding submodule with selective disambiguation and the pooling submodule with firm attention. Overall, our full architecture can be illustrated as in Figure 2 , which consists of an entity linking system (ELS), a sequence-to-sequence with attention mechanism model, and the E2T module. We note that our proposed module can be easily attached to more sophisticated abstractive summarization models BIBREF13 , BIBREF14 that are based on the traditional encoder-decoder framework and consequently can produce better results. The code of the base model and the E2T are available online. Base model As our base model, we employ a basic encoder-decoder RNN used in most neural machine translation BIBREF2 and text summarization BIBREF15 tasks. We employ a two-layer bidirectional GRU (BiGRU) as the recurrent unit of the encoder. The BiGRU consists of a forward and backward GRU, which results to sequences of forward and backward hidden states $(\overrightarrow{h}_1, \overrightarrow{h}_2, ..., \overrightarrow{h}_n)$ and $(\overleftarrow{h}_1, \overleftarrow{h}_2, ..., \overleftarrow{h}_n)$ , respectively: $ \overrightarrow{h}_i &= GRU(x_i, \overrightarrow{h}_{i-1}) \\ \overleftarrow{h}_i &= GRU(x_i, \overleftarrow{h}_{i+1}) \nonumber $ The forward and backward hidden states are concatenated to get the hidden state vectors of the tokens (i.e. $h_i = [\overrightarrow{h}_i; \overleftarrow{h}_i]$ ). The final states of the forward and backward GRU are also concatenated to create the final text representation vector of the encoder $s = [\overrightarrow{h}_n; \overleftarrow{h}_1]$ . These values are calculated per layer, where $x_t$ of the second layer is $h_t$ of the first layer. The final text representation vectors are projected by a fully connected layer and are passed to the decoder as the initial hidden states $s_0 = s$ . For the decoder, we use a two-layer uni-directional GRU with attention. At each time step $t$ , the previous token $y_{t-1}$ , the previous hidden state $s_{t-1}$ , and the previous context vector $c_{t-1}$ are passed to a GRU to calculate the new hidden state $s_t$ , as shown in the equation below. $$s_t = GRU(w_{t-1}, s_{t-1}, c_{t-1}) \nonumber $$   (Eq. 9) The context vector $c_t$ is computed using the additive attention mechanism BIBREF2 , which matches the current decoder state $s_t$ and each encoder state $h_i$ to get an importance score. The scores are then passed to a softmax and are used to pool the encoder states using weighted sum. The final pooled vector is the context vector, as shown in the equations below. $ g_{t,i} &= v_a^\top tanh(W_a s_{t-1} + U_a h_i) \\ a_{t,i} &= \frac{exp(g_{t,i})}{\sum _i exp(g_{t,i})} \\ c_t &= \sum _i a_{t,i} h_i \nonumber $ Finally, the previous token $y_{t-1}$ , the current context vector $c_t$ , and the current decoder state $s_t$ are used to generate the current word $y_t$ with a softmax layer over the decoder vocabulary, as shown below. $ o_t &= W_w w_{t-1} + W_c c_t + W_s s_t \\ p(y_t | y_{<t}) &= softmax(W_o o_t) \nonumber $ Entity encoding submodule After performing entity linking to the input text using the ELS, we receive a sequential list of linked entities, arranged based on their location in the text. We embed these entities to $d$ -dimensional vectors $E = \lbrace e_1, e_2, ..., e_m\rbrace $ where $e_i \in \mathbb {R}^d$ . Since these entities may still contain ambiguity, it is necessary to resolve them before applying them to the base model. Based on the idea that an ambiguous entity can be disambiguated using its neighboring entities, we introduce two kinds of disambiguating encoders below. One way to disambiguate an entity is by using all the other entities, putting more importance to entities that are nearer. For this purpose, we employ an RNN-based model to globally disambiguate the entities. Specifically, we use BiGRU and concatenate the forward and backward hidden state vectors as the new entity vector: $ \overrightarrow{h}_i &= GRU(e_i, \overrightarrow{h}_{i-1}) \\ \overleftarrow{h}_i &= GRU(e_i, \overleftarrow{h}_{i+1}) \\ e^{\prime }_i &= [\overrightarrow{h}_i; \overleftarrow{h}_i] \nonumber $ Another way to disambiguate an entity is by using only the direct neighbors of the entity, putting no importance value to entities that are far. To do this, we employ a CNN-based model to locally disambiguate the entities. Specifically, we do the convolution operation using filter matrices $W_f \in \mathbb {R}^{h \times d}$ with filter size $h$ to a window of $h$ words. We do this for different sizes of $h$ . This produces new feature vectors $c_{i,h}$ as shown below, where $f(.)$ is a non-linear function: $ c_{i,h} = f([e_{i-(h-1)/2}; ...; e_{i+h(+1)/2}]^\top W_f + b_f) \nonumber $ The convolution operation reduces the number of entities differently depending on the filter size $h$ . To prevent loss of information and to produce the same amount of feature vectors $c_{i,h}$ , we pad the entity list dynamically such that when the filter size is $h$ , the number of paddings on each side is $(h-1)/2$ . The filter size $h$ therefore refers to the number of entities used to disambiguate a middle entity. Finally, we concatenate all feature vectors of different $h$ 's for each $i$ as the new entity vector: $ e^{\prime }_i = [c_{i,h_1}; c_{i, h_2}; ...] \nonumber $ The question on which disambiguating encoder is better has been a debate; some argued that using only the local context is appropriate BIBREF16 while some claimed that additionally using global context also helps BIBREF17 . The RNN-based encoder is good as it smartly makes use of all entities, however it may perform bad when there are many entities as it introduces noise when using a far entity during disambiguation. The CNN-based encoder is good as it minimizes the noise by totally ignoring far entities when disambiguating, however determining the appropriate filter sizes $h$ needs engineering. Overall, we argue that when the input text is short (e.g. a sentence), both encoders perform comparably, otherwise when the input text is long (e.g. a document), the CNN-based encoder performs better. It is obvious that not all entities need to be disambiguated. When a correctly linked and already adequately disambiguated entity is disambiguated again, it would make the entity very context-specific and might not be suitable for the summarization task. Our entity encoding submodule therefore uses a selective mechanism that decides whether to use the disambiguating encoder or not. This is done by introducing a selective disambiguation gate $d$ . The final entity vector $\tilde{e}_i$ is calculated as the linear transformation of $e_i$ and $e^{\prime }_i$ : $ e^{\prime }_i &= encoder(e_i) \\ d &= \sigma (W_d e^{\prime }_i + b_d) \\ \tilde{e}_i &= d \times f(W_x e_i + b_x) + \\ & \quad (1-d) \times f(W_y e^{\prime }_i + b_y) \nonumber $ The full entity encoding submodule is illustrated in Figure 3 . Ultimately, the submodule outputs the disambiguated entity vectors $\tilde{E} = \lbrace \tilde{e}_1, \tilde{e}_2, ..., \tilde{e}_m\rbrace $ . Pooling submodule The entity vectors $\tilde{E}$ are pooled to create a single topic vector $t$ that represents the topic of the summary. One possible pooling technique is to use soft attention BIBREF18 on the vectors to determine the importance value of each vector, which can be done by matching each entity vector with the text vector $s$ from the text encoder as the context vector. The entity vectors are then pooled using weighted sum. One problem with soft attention is that it considers all entity vectors when constructing the topic vector. However, not all entities are important and necessary when generating summaries. Moreover, a number of these entities may be erroneous and irrelevant, as reported in Section "Related work" . Soft attention gives non-negligible important scores to these entities, thus adds unnecessary noise to the construction of the topic vector. Our pooling submodule instead uses firm attention mechanism to consider only top $k$ entities when constructing the topic vector. This is done in a differentiable way as follows: $ G &= v_a^\top tanh(W_a \tilde{E} + U_a s) \\ K &= top\_k(G) \\ P &= sparse\_vector(K, 0, -\infty ) \\ g^{\prime }_i &= g_i + p_i \\ a_i &= \frac{exp(g^{\prime }_i)}{\sum _i exp(g^{\prime }_i)} \\ t &= \sum _i a_i \tilde{e}_i \nonumber $ where the functions $K = top\_k(G)$ gets the indices of the top $k$ vectors in $G$ and $P = sparse\_vector(K,0,-\infty )$ creates a sparse vector where the values of $K$ is 0 and $-\infty $ otherwise. The sparse vector $P$ is added to the original importance score vector $G$ to create a new importance score vector. In this new vector, important scores of non-top $k$0 entities are $k$1 . When softmax is applied, this gives very small, negligible, and close-to-zero values to non-top $k$2 entities. The value $k$3 depends on the lengths of the input text and summary. Moreover, when $k$4 increases towards infinity, firm attention becomes soft attention. We decide $k$5 empirically (see Section "Experimental settings" ). Extending from the base model Entity2Topic module extends the base model as follows. The final text representation vector $s$ is used as a context vector when constructing the topic vector $t$ in the pooling submodule. The topic vector $t$ is then concatenated to the decoder hidden state vectors $s_i$ , i.e. $s^{\prime }_i = [s_i; t]$ . The concatenated vector is finally used to create the output vector: $$o_i = W_w w_{i-1} + W_c c_i + W_s s^{\prime }_i \nonumber $$   (Eq. 18) Related work Due to its recent success, neural network models have been used with competitive results on abstractive summarization. A neural attention model was first applied to the task, easily achieving state-of-the-art performance on multiple datasets BIBREF5 . The model has been extended to instead use recurrent neural network as decoder BIBREF3 . The model was further extended to use a full RNN encoder-decoder framework and further enhancements through lexical and statistical features BIBREF15 . The current state-of-the-art performance is achieved by selectively encoding words as a process of distilling salient information BIBREF13 . Neural abstractive summarization models have also been explored to summarize longer documents. Word extraction models have been previously explored, performing worse than sentence extraction models BIBREF19 . Hierarchical attention-based recurrent neural networks have also been applied to the task, owing to the idea that there are multiple sentences in a document BIBREF15 . Finally, distraction-based models were proposed to enable models to traverse the text content and grasp the overall meaning BIBREF4 . The current state-of-the-art performance is achieved by a graph-based attentional neural model, considering the key factors of document summarization such as saliency, fluency and novelty BIBREF14 . Previous studies on the summarization tasks have only used entities in the preprocessing stage to anonymize the dataset BIBREF15 and to mitigate out-of-vocabulary problems BIBREF14 . Linked entities for summarization are still not properly explored and we are the first to use linked entities to improve the performance of the summarizer. Results We report the ROUGE F1 scores for both datasets of all the competing models using ROUGE F1 scores BIBREF27 . We report the results on the Gigaword and the CNN dataset in Table 2 and Table 3 , respectively. In Gigaword dataset where the texts are short, our best model achieves a comparable performance with the current state-of-the-art. In CNN dataset where the texts are longer, our best model outperforms all the previous models. We emphasize that E2T module is easily attachable to better models, and we expect E2T to improve their performance as well. Overall, E2T achieves a significant improvement over the baseline model base, with at least 2 ROUGE-1 points increase in the Gigaword dataset and 6 ROUGE-1 points increase in the CNN dataset. In fact, all variants of E2T gain improvements over the baseline, implying that leveraging on linked entities improves the performance of the summarizer. Among the model variants, the CNN-based encoder with selective disambiguation and firm attention performs the best. Automatic evaluation on the Gigaword dataset shows that the CNN and RNN variants of base+E2T have similar performance. To break the tie between both models, we also conduct human evaluation on the Gigaword dataset. We instruct two annotators to read the input sentence and rank the competing summaries from first to last according to their relevance and fluency: (a) the original summary gold, and from models (b) base, (c) base+E2Tcnn, and (d) base+E2Trnn. We then compute (i) the proportion of every ranking of each model and (ii) the mean rank of each model. The results are reported in Table 4 . The model with the best mean rank is base+E2Tcnn, followed by gold, then by base+E2Trnn and base, respectively. We also perform ANOVA and post-hoc Tukey tests to show that the CNN variant is significantly ( $p<0.01$ ) better than the RNN variant and the base model. The RNN variant does not perform as well as the CNN variant, contrary to the automatic ROUGE evaluation above. Interestingly, the CNN variant produces better (but with no significant difference) summaries than the gold summaries. We posit that this is due to the fact that the article title does not correspond to the summary of the first sentence. Conclusion We proposed to leverage on linked entities to improve the performance of sequence-to-sequence models on neural abstractive summarization task. Linked entities are used to guide the decoding process based on the summary topic and commonsense learned from a knowledge base. We introduced Entity2Topic (E2T), a module that is easily attachable to any model using an encoder-decoder framework. E2T applies linked entities into the summarizer by encoding the entities with selective disambiguation and pooling them into one summary topic vector with firm attention mechanism. We showed that by applying E2T to a basic sequence-to-sequence model, we achieve significant improvements over the base model and consequently achieve a comparable performance with more complex summarization models. Acknowledgement We would like to thank the three anonymous reviewers for their valuable feedback. This work was supported by Microsoft Research, and Institute for Information communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2017-0-01778 , Development of Explainable Humanlevel Deep Machine Learning Inference Framework). S. Hwang is a corresponding author.
Why are current ELS's not sufficiently effective?
ef7212075e80bf35b7889dc8dd52fcbae0d1400a
Why Ġare Ġcurrent ĠE LS 's Ġnot Ġsufficiently Ġeffective ?
Link ed Ġentities Ġmay Ġbe Ġambiguous Ġor Ġtoo Ġcommon
<s> Why Ġare Ġcurrent ĠE LS 's Ġnot Ġsufficiently Ġeffective ? </s> Introduction </s> Text Ġsummar ization Ġis Ġa Ġtask Ġto Ġgenerate Ġa Ġshorter Ġand Ġconcise Ġversion Ġof Ġa Ġtext Ġwhile Ġpreserving Ġthe Ġmeaning Ġof Ġthe Ġoriginal Ġtext . ĠThe Ġtask Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġsubt ask Ġbased Ġon Ġthe Ġapproach : Ġextract ive Ġand Ġabstract ive Ġsummar ization . ĠExtract ive Ġsummar ization Ġis Ġa Ġtask Ġto Ġcreate Ġsumm aries Ġby Ġpulling Ġout Ġsnippets Ġof Ġtext Ġform Ġthe Ġoriginal Ġtext Ġand Ġcombining Ġthem Ġto Ġform Ġa Ġsummary . ĠAbstract ive Ġsummar ization Ġasks Ġto Ġgenerate Ġsumm aries Ġfrom Ġscratch Ġwithout Ġthe Ġrestriction Ġto Ġuse Ġthe Ġavailable Ġwords Ġfrom Ġthe Ġoriginal Ġtext . ĠDue Ġto Ġthe Ġlimitations Ġof Ġextract ive Ġsummar ization Ġon Ġinco herent Ġtexts Ġand Ġunnatural Ġmethodology ĠB IB REF 0 Ġ, Ġthe Ġresearch Ġtrend Ġhas Ġshifted Ġtowards Ġabstract ive Ġsummar ization . </s> Sequ ence - to - sequence Ġmodels ĠB IB REF 1 Ġwith Ġattention Ġmechanism ĠB IB REF 2 Ġhave Ġfound Ġgreat Ġsuccess Ġin Ġgenerating Ġabstract ive Ġsumm aries , Ġboth Ġfrom Ġa Ġsingle Ġsentence ĠB IB REF 3 Ġand Ġfrom Ġa Ġlong Ġdocument Ġwith Ġmultiple Ġsentences ĠB IB REF 4 Ġ. ĠHowever , Ġwhen Ġgenerating Ġsumm aries , Ġit Ġis Ġnecessary Ġto Ġdetermine Ġthe Ġmain Ġtopic Ġand Ġto Ġs ift Ġout Ġunnecessary Ġinformation Ġthat Ġcan Ġbe Ġomitted . ĠSequence - to - sequence Ġmodels Ġhave Ġthe Ġtendency Ġto Ġinclude Ġall Ġthe Ġinformation , Ġrelevant Ġor Ġnot , Ġthat Ġare Ġfound Ġin Ġthe Ġoriginal Ġtext . ĠThis Ġmay Ġresult Ġto Ġuncon cise Ġsumm aries Ġthat Ġconcent rates Ġwrongly Ġon Ġirrelevant Ġtopics . ĠThe Ġproblem Ġis Ġespecially Ġsevere Ġwhen Ġsummar izing Ġlonger Ġtexts . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġuse Ġentities Ġfound Ġin Ġthe Ġoriginal Ġtext Ġto Ġinfer Ġthe Ġsummary Ġtopic , Ġmitigating Ġthe Ġaforementioned Ġproblem . ĠSpecifically , Ġwe Ġleverage Ġon Ġlinked Ġentities Ġextracted Ġby Ġemploying Ġa Ġreadily Ġavailable Ġentity Ġlinking Ġsystem . ĠThe Ġimportance Ġof Ġusing Ġlinked Ġentities Ġin Ġsummar ization Ġis Ġintuitive Ġand Ġcan Ġbe Ġexplained Ġby Ġlooking Ġat ĠFigure Ġ1 Ġas Ġan Ġexample . ĠFirst Ġ( O 1 Ġin Ġthe ĠFigure ), Ġaside Ġfrom Ġauxiliary Ġwords Ġto Ġconstruct Ġa Ġsentence , Ġa Ġsummary Ġis Ġmainly Ġcomposed Ġof Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext . ĠSecond Ġ( O 2 ), Ġwe Ġcan Ġdepict Ġthe Ġmain Ġtopic Ġof Ġthe Ġsummary Ġas Ġa Ġprobability Ġdistribution Ġof Ġrelevant Ġentities Ġfrom Ġthe Ġlist Ġof Ġentities . ĠFinally Ġ( O 3 ), Ġwe Ġcan Ġleverage Ġon Ġentity Ġcommons ense Ġlearned Ġfrom Ġa Ġseparate Ġlarge Ġknowledge Ġbase Ġsuch Ġas ĠWikipedia . </s> To Ġthis Ġend , Ġwe Ġpresent Ġa Ġmethod Ġto Ġeffectively Ġapply Ġlinked Ġentities Ġin Ġsequence - to - sequence Ġmodels , Ġcalled ĠEntity 2 Topic Ġ( E 2 T ). ĠE 2 T Ġis Ġa Ġmodule Ġthat Ġcan Ġbe Ġeasily Ġattached Ġto Ġany Ġsequence - to - sequence Ġbased Ġsummar ization Ġmodel . ĠThe Ġmodule Ġenc odes Ġthe Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext Ġby Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ), Ġconstructs Ġa Ġvector Ġrepresenting Ġthe Ġtopic Ġof Ġthe Ġsummary Ġto Ġbe Ġgenerated , Ġand Ġinforms Ġthe Ġdec oder Ġabout Ġthe Ġconstructed Ġtopic Ġvector . ĠDue Ġto Ġthe Ġimperfect ions Ġof Ġcurrent ĠE LS 's , Ġthe Ġextracted Ġlinked Ġentities Ġmay Ġbe Ġtoo Ġambiguous Ġand Ġcoarse Ġto Ġbe Ġconsidered Ġrelevant Ġto Ġthe Ġsummary . ĠWe Ġsolve Ġthis Ġissue Ġby Ġusing Ġentity Ġenc od ers Ġwith Ġselective Ġdis amb ig uation Ġand Ġby Ġconstructing Ġtopic Ġvectors Ġusing Ġfirm Ġattention . </s> We Ġexperiment Ġon Ġtwo Ġdatasets , ĠGig aw ord Ġand ĠCNN , Ġwith Ġvarying Ġlengths . ĠWe Ġshow Ġthat Ġapplying Ġour Ġmodule Ġto Ġa Ġsequence - to - sequence Ġmodel Ġwith Ġattention Ġmechanism Ġsignificantly Ġincreases Ġits Ġperformance Ġon Ġboth Ġdatasets . ĠMoreover , Ġwhen Ġcompared Ġwith Ġthe Ġstate - of - the - art Ġmodels Ġfor Ġeach Ġdataset , Ġthe Ġmodel Ġob tains Ġa Ġcomparable Ġperformance Ġon Ġthe ĠGig aw ord Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġshort , Ġand Ġoutper forms Ġall Ġcompeting Ġmodels Ġon Ġthe ĠCNN Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġlonger . ĠFurthermore , Ġwe Ġprovide Ġanalysis Ġon Ġhow Ġour Ġmodel Ġeffectively Ġuses Ġthe Ġextracted Ġlinked Ġentities Ġto Ġproduce Ġconcise Ġand Ġbetter Ġsumm aries . </s> Use fulness Ġof Ġlinked Ġentities Ġin Ġsummar ization </s> In Ġthe Ġnext Ġsubsections , Ġwe Ġpresent Ġdetailed Ġarguments Ġwith Ġempirical Ġand Ġpreviously Ġexamined Ġevid ences Ġon Ġthe Ġobservations Ġand Ġpossible Ġissues Ġwhen Ġusing Ġlinked Ġentities Ġextracted Ġby Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ) Ġfor Ġgenerating Ġabstract ive Ġsumm aries . ĠFor Ġthis Ġpurpose , Ġwe Ġuse Ġthe Ġdevelopment Ġsets Ġof Ġthe ĠGig aw ord Ġdataset Ġprovided Ġin ĠB IB REF 5 Ġand Ġof Ġthe ĠCNN Ġdataset Ġprovided Ġin ĠB IB REF 6 Ġas Ġthe Ġexperimental Ġdata Ġfor Ġquantitative Ġevidence Ġand Ġrefer Ġthe Ġreaders Ġto ĠFigure Ġ1 Ġas Ġthe Ġrunning Ġexample . </s> Obs erv ations </s> As Ġdiscussed Ġin ĠSection Ġ" Introduction " Ġ, Ġwe Ġfind Ġthree Ġobservations Ġthat Ġshow Ġthe Ġusefulness Ġof Ġlinked Ġentities Ġfor Ġabstract ive Ġsummar ization . </s> First , Ġsumm aries Ġare Ġmainly Ġcomposed Ġof Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġoriginal Ġtext . ĠIn Ġthe Ġexample , Ġit Ġcan Ġbe Ġseen Ġthat Ġthe Ġsummary Ġcontains Ġfour Ġwords Ġthat Ġrefer Ġto Ġdifferent Ġentities . ĠIn Ġfact , Ġall Ġnoun Ġphrases Ġin Ġthe Ġsummary Ġmention Ġat Ġleast Ġone Ġlinked Ġentity . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġextract Ġlinked Ġentities Ġfrom Ġthe Ġoriginal Ġtext Ġand Ġcompare Ġthem Ġto Ġthe Ġnoun Ġphrases Ġfound Ġin Ġthe Ġsummary . ĠWe Ġreport Ġthat Ġ$ 77 . 1 \ % $ Ġand Ġ$ 75 . 1 \ % $ Ġof Ġthe Ġnoun Ġphrases Ġon Ġthe ĠGig aw ord Ġand ĠCNN Ġdatasets , Ġrespectively , Ġcontain Ġat Ġleast Ġone Ġlinked Ġentity , Ġwhich Ġconfirms Ġour Ġobservation . </s> Second , Ġlinked Ġentities Ġcan Ġbe Ġused Ġto Ġrepresent Ġthe Ġtopic Ġof Ġthe Ġsummary , Ġdefined Ġas Ġa Ġmult in omial Ġdistribution Ġover Ġentities , Ġas Ġgraph ically Ġshown Ġin Ġthe Ġexample , Ġwhere Ġthe Ġprobabilities Ġrefer Ġto Ġthe Ġrelevance Ġof Ġthe Ġentities . ĠEnt ities Ġhave Ġbeen Ġpreviously Ġused Ġto Ġrepresent Ġtopics ĠB IB REF 7 Ġ, Ġas Ġthey Ġcan Ġbe Ġutilized Ġas Ġa Ġcontrolled Ġvocabulary Ġof Ġthe Ġmain Ġtopics Ġin Ġa Ġdocument ĠB IB REF 8 Ġ. ĠIn Ġthe Ġexample , Ġwe Ġsee Ġthat Ġthe Ġentity ĠâĢ ľ J ae ĠSe o âĢ Ŀ Ġis Ġthe Ġmost Ġrelevant Ġbecause Ġit Ġis Ġthe Ġsubject Ġof Ġthe Ġsummary , Ġwhile Ġthe Ġentity ĠâĢ ľ South ĠKorean âĢ Ŀ Ġis Ġless Ġrelevant Ġbecause Ġit Ġis Ġless Ġimportant Ġwhen Ġconstructing Ġthe Ġsummary . </s> Third , Ġwe Ġcan Ġmake Ġuse Ġof Ġthe Ġentity Ġcommons ense Ġthat Ġcan Ġbe Ġlearned Ġas Ġa Ġcontinuous Ġvector Ġrepresentation Ġfrom Ġa Ġseparate Ġlarger Ġcorpus ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠIn Ġthe Ġexample , Ġif Ġwe Ġknow Ġthat Ġthe Ġentities ĠâĢ ľ Los ĠAngeles ĠDodgers âĢ Ŀ Ġand ĠâĢ ľ New ĠYork ĠMets âĢ Ŀ Ġare ĠAmerican Ġbaseball Ġteams Ġand ĠâĢ ľ J ae ĠSe o âĢ Ŀ Ġis Ġa Ġbaseball Ġplayer Ġassociated Ġwith Ġthe Ġteams , Ġthen Ġwe Ġcan Ġuse Ġthis Ġinformation Ġto Ġgenerate Ġmore Ġcoherent Ġsumm aries . ĠWe Ġfind Ġthat Ġ$ 76 . 0 \ % $ Ġof Ġthe Ġextracted Ġlinked Ġentities Ġare Ġcovered Ġby Ġthe Ġpre - trained Ġvectors Ġin Ġour Ġexperimental Ġdata , Ġproving Ġour Ġthird Ġobservation . </s> P ossible Ġissues </s> Despite Ġits Ġusefulness , Ġlinked Ġentities Ġextracted Ġfrom ĠE LS 's Ġhave Ġissues Ġbecause Ġof Ġlow Ġprecision Ġrates ĠB IB REF 11 Ġand Ġdesign Ġchallenges Ġin Ġtraining Ġdatasets ĠB IB REF 12 Ġ. ĠThese Ġissues Ġcan Ġbe Ġsummarized Ġinto Ġtwo Ġparts : Ġambiguity Ġand Ġco ars eness . </s> First , Ġthe Ġextracted Ġentities Ġmay Ġbe Ġambiguous . ĠIn Ġthe Ġexample , Ġthe Ġentity ĠâĢ ľ South ĠKorean âĢ Ŀ Ġis Ġambiguous Ġbecause Ġit Ġcan Ġrefer Ġto Ġboth Ġthe ĠSouth ĠKorean Ġperson Ġand Ġthe ĠSouth ĠKorean Ġlanguage , Ġamong Ġothers . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġextract Ġ( 1 ) Ġthe Ġtop Ġ100 Ġentities Ġbased Ġon Ġfrequency , Ġand Ġ( 2 ) Ġthe Ġentities Ġextracted Ġfrom Ġ100 Ġrandomly Ġselected Ġtexts , Ġand Ġcheck Ġwhether Ġthey Ġhave Ġdis amb ig uation Ġpages Ġin ĠWikipedia Ġor Ġnot . ĠWe Ġdiscover Ġthat Ġ$ 71 . 0 \ % $ Ġof Ġthe Ġtop Ġ100 Ġentities Ġand Ġ$ 53 . 6 \ % $ Ġof Ġthe Ġentities Ġpicked Ġat Ġrandom Ġhave Ġdis amb ig uation Ġpages , Ġwhich Ġshows Ġthat Ġmost Ġentities Ġare Ġprone Ġto Ġambiguity Ġproblems . </s> Second , Ġthe Ġlinked Ġentities Ġmay Ġalso Ġbe Ġtoo Ġcommon Ġto Ġbe Ġconsidered Ġan Ġentity . ĠThis Ġmay Ġintroduce Ġerrors Ġand Ġirre lev ance Ġto Ġthe Ġsummary . ĠIn Ġthe Ġexample , ĠâĢ ľ Wednesday âĢ Ŀ Ġis Ġerroneous Ġbecause Ġit Ġis Ġwrongly Ġlinked Ġto Ġthe Ġentity ĠâĢ ľ Wednesday ĠNight ĠBaseball âĢ Ŀ . ĠAlso , ĠâĢ ľ sw ap âĢ Ŀ Ġis Ġirrelevant Ġbecause Ġalthough Ġit Ġis Ġlinked Ġcorrectly Ġto Ġthe Ġentity ĠâĢ ľ Trade Ġ( Sports ) âĢ Ŀ , Ġit Ġis Ġtoo Ġcommon Ġand Ġirrelevant Ġwhen Ġgenerating Ġthe Ġsumm aries . ĠIn Ġour Ġexperimental Ġdata , Ġwe Ġrandomly Ġselect Ġ100 Ġdata Ġinstances Ġand Ġtag Ġthe Ġcorrectness Ġand Ġrelevance Ġof Ġextracted Ġentities Ġinto Ġone Ġof Ġfour Ġlabels : ĠA : Ġcorrect Ġand Ġrelevant , ĠB : Ġcorrect Ġand Ġsomewhat Ġrelevant , ĠC : Ġcorrect Ġbut Ġirrelevant , Ġand ĠD : Ġincorrect . ĠResults Ġshow Ġthat Ġ$ 29 . 4 \ % $ Ġ, Ġ$ 13 . 7 \ % $ Ġ, Ġ$ 30 . 0 \ % $ Ġ, Ġand Ġ$ 26 . 9 \ % $ Ġare Ġtagged Ġwith ĠA , ĠB , ĠC , Ġand ĠD , Ġrespectively , Ġwhich Ġshows Ġthat Ġthere Ġis Ġa Ġlarge Ġamount Ġof Ġincorrect Ġand Ġirrelevant Ġentities . </s> Our Ġmodel </s> To Ġsolve Ġthe Ġissues Ġdescribed Ġabove , Ġwe Ġpresent ĠEntity 2 Topic Ġ( E 2 T ), Ġa Ġmodule Ġthat Ġcan Ġbe Ġeasily Ġattached Ġto Ġany Ġsequence - to - sequence Ġbased Ġabstract ive Ġsummar ization Ġmodel . ĠE 2 T Ġenc odes Ġthe Ġlinked Ġentities Ġextracted Ġfrom Ġthe Ġtext Ġand Ġtransforms Ġthem Ġinto Ġa Ġsingle Ġtopic Ġvector . ĠThis Ġvector Ġis Ġultimately Ġconc aten ated Ġto Ġthe Ġdec oder Ġhidden Ġstate Ġvectors . ĠThe Ġmodule Ġcontains Ġtwo Ġsub modules Ġspecifically Ġfor Ġthe Ġissues Ġpresented Ġby Ġthe Ġentity Ġlinking Ġsystems : Ġthe Ġentity Ġencoding Ġsub module Ġwith Ġselective Ġdis amb ig uation Ġand Ġthe Ġpool ing Ġsub module Ġwith Ġfirm Ġattention . </s> Overall , Ġour Ġfull Ġarchitecture Ġcan Ġbe Ġillustrated Ġas Ġin ĠFigure Ġ2 Ġ, Ġwhich Ġconsists Ġof Ġan Ġentity Ġlinking Ġsystem Ġ( ELS ), Ġa Ġsequence - to - sequence Ġwith Ġattention Ġmechanism Ġmodel , Ġand Ġthe ĠE 2 T Ġmodule . ĠWe Ġnote Ġthat Ġour Ġproposed Ġmodule Ġcan Ġbe Ġeasily Ġattached Ġto Ġmore Ġsophisticated Ġabstract ive Ġsummar ization Ġmodels ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġthat Ġare Ġbased Ġon Ġthe Ġtraditional Ġenc oder - dec oder Ġframework Ġand Ġconsequently Ġcan Ġproduce Ġbetter Ġresults . ĠThe Ġcode Ġof Ġthe Ġbase Ġmodel Ġand Ġthe ĠE 2 T Ġare Ġavailable Ġonline . </s> Base Ġmodel </s> As Ġour Ġbase Ġmodel , Ġwe Ġemploy Ġa Ġbasic Ġenc oder - dec oder ĠR NN Ġused Ġin Ġmost Ġneural Ġmachine Ġtranslation ĠB IB REF 2 Ġand Ġtext Ġsummar ization ĠB IB REF 15 Ġtasks . ĠWe Ġemploy Ġa Ġtwo - layer Ġbid irection al ĠGR U Ġ( Bi GR U ) Ġas Ġthe Ġrecurrent Ġunit Ġof Ġthe Ġenc oder . ĠThe ĠBi GR U Ġconsists Ġof Ġa Ġforward Ġand Ġbackward ĠGR U , Ġwhich Ġresults Ġto Ġsequences Ġof Ġforward Ġand Ġbackward Ġhidden Ġstates Ġ$( \ over right arrow { h } _ 1 , Ġ\ over right arrow { h } _ 2 , Ġ... , Ġ\ over right arrow { h } _ n ) $ Ġand Ġ$( \ over left arrow { h } _ 1 , Ġ\ over left arrow { h } _ 2 , Ġ... , Ġ\ over left arrow { h } _ n ) $ Ġ, Ġrespectively : Ġ$ Ġ\ over right arrow { h } _ i Ġ& = ĠGR U ( x _ i , Ġ\ over right arrow { h } _{ i - 1 }) Ġ\\ Ġ\ over left arrow { h } _ i Ġ& = ĠGR U ( x _ i , Ġ\ over left arrow { h } _{ i + 1 }) Ġ\ non umber Ġ$ </s> The Ġforward Ġand Ġbackward Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġget Ġthe Ġhidden Ġstate Ġvectors Ġof Ġthe Ġtokens Ġ( i . e . Ġ$ h _ i Ġ= Ġ[ \ over right arrow { h } _ i ; Ġ\ over left arrow { h } _ i ] $ Ġ). ĠThe Ġfinal Ġstates Ġof Ġthe Ġforward Ġand Ġbackward ĠGR U Ġare Ġalso Ġconc aten ated Ġto Ġcreate Ġthe Ġfinal Ġtext Ġrepresentation Ġvector Ġof Ġthe Ġenc oder Ġ$ s Ġ= Ġ[ \ over right arrow { h } _ n ; Ġ\ over left arrow { h } _ 1 ] $ Ġ. ĠThese Ġvalues Ġare Ġcalculated Ġper Ġlayer , Ġwhere Ġ$ x _ t $ Ġof Ġthe Ġsecond Ġlayer Ġis Ġ$ h _ t $ Ġof Ġthe Ġfirst Ġlayer . ĠThe Ġfinal Ġtext Ġrepresentation Ġvectors Ġare Ġprojected Ġby Ġa Ġfully Ġconnected Ġlayer Ġand Ġare Ġpassed Ġto Ġthe Ġdec oder Ġas Ġthe Ġinitial Ġhidden Ġstates Ġ$ s _ 0 Ġ= Ġs $ Ġ. </s> For Ġthe Ġdec oder , Ġwe Ġuse Ġa Ġtwo - layer Ġun i - direction al ĠGR U Ġwith Ġattention . ĠAt Ġeach Ġtime Ġstep Ġ$ t $ Ġ, Ġthe Ġprevious Ġtoken Ġ$ y _{ t - 1 } $ Ġ, Ġthe Ġprevious Ġhidden Ġstate Ġ$ s _{ t - 1 } $ Ġ, Ġand Ġthe Ġprevious Ġcontext Ġvector Ġ$ c _{ t - 1 } $ Ġare Ġpassed Ġto Ġa ĠGR U Ġto Ġcalculate Ġthe Ġnew Ġhidden Ġstate Ġ$ s _ t $ Ġ, Ġas Ġshown Ġin Ġthe Ġequation Ġbelow . </s> $$ s _ t Ġ= ĠGR U ( w _{ t - 1 }, Ġs _{ t - 1 }, Ġc _{ t - 1 }) Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ9 ) </s> The Ġcontext Ġvector Ġ$ c _ t $ Ġis Ġcomputed Ġusing Ġthe Ġadditive Ġattention Ġmechanism ĠB IB REF 2 Ġ, Ġwhich Ġmatches Ġthe Ġcurrent Ġdec oder Ġstate Ġ$ s _ t $ Ġand Ġeach Ġenc oder Ġstate Ġ$ h _ i $ Ġto Ġget Ġan Ġimportance Ġscore . ĠThe Ġscores Ġare Ġthen Ġpassed Ġto Ġa Ġsoft max Ġand Ġare Ġused Ġto Ġpool Ġthe Ġenc oder Ġstates Ġusing Ġweighted Ġsum . ĠThe Ġfinal Ġpooled Ġvector Ġis Ġthe Ġcontext Ġvector , Ġas Ġshown Ġin Ġthe Ġequations Ġbelow . Ġ$ Ġg _{ t , i } Ġ& = Ġv _ a ^ \ top Ġtan h ( W _ a Ġs _{ t - 1 } Ġ+ ĠU _ a Ġh _ i ) Ġ\\ Ġa _{ t , i } Ġ& = Ġ\ frac { exp ( g _{ t , i }) }{ \ sum Ġ_ i Ġexp ( g _{ t , i }) } Ġ\\ Ġc _ t Ġ& = Ġ\ sum Ġ_ i Ġa _{ t , i } Ġh _ i Ġ\ non umber Ġ$ </s> Finally , Ġthe Ġprevious Ġtoken Ġ$ y _{ t - 1 } $ Ġ, Ġthe Ġcurrent Ġcontext Ġvector Ġ$ c _ t $ Ġ, Ġand Ġthe Ġcurrent Ġdec oder Ġstate Ġ$ s _ t $ Ġare Ġused Ġto Ġgenerate Ġthe Ġcurrent Ġword Ġ$ y _ t $ Ġwith Ġa Ġsoft max Ġlayer Ġover Ġthe Ġdec oder Ġvocabulary , Ġas Ġshown Ġbelow . Ġ$ Ġo _ t Ġ& = ĠW _ w Ġw _{ t - 1 } Ġ+ ĠW _ c Ġc _ t Ġ+ ĠW _ s Ġs _ t Ġ\\ Ġp ( y _ t Ġ| Ġy _{ < t }) Ġ& = Ġsoft max ( W _ o Ġo _ t ) Ġ\ non umber Ġ$ </s> Entity Ġencoding Ġsub module </s> After Ġperforming Ġentity Ġlinking Ġto Ġthe Ġinput Ġtext Ġusing Ġthe ĠE LS , Ġwe Ġreceive Ġa Ġsequential Ġlist Ġof Ġlinked Ġentities , Ġarranged Ġbased Ġon Ġtheir Ġlocation Ġin Ġthe Ġtext . ĠWe Ġembed Ġthese Ġentities Ġto Ġ$ d $ Ġ- dimensional Ġvectors Ġ$ E Ġ= Ġ\ l brace Ġe _ 1 , Ġe _ 2 , Ġ... , Ġe _ m \ r brace Ġ$ Ġwhere Ġ$ e _ i Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġ. ĠSince Ġthese Ġentities Ġmay Ġstill Ġcontain Ġambiguity , Ġit Ġis Ġnecessary Ġto Ġresolve Ġthem Ġbefore Ġapplying Ġthem Ġto Ġthe Ġbase Ġmodel . ĠBased Ġon Ġthe Ġidea Ġthat Ġan Ġambiguous Ġentity Ġcan Ġbe Ġdis amb ig uated Ġusing Ġits Ġneighboring Ġentities , Ġwe Ġintroduce Ġtwo Ġkinds Ġof Ġdis amb ig uating Ġenc od ers Ġbelow . </s> One Ġway Ġto Ġdis amb ig uate Ġan Ġentity Ġis Ġby Ġusing Ġall Ġthe Ġother Ġentities , Ġputting Ġmore Ġimportance Ġto Ġentities Ġthat Ġare Ġnearer . ĠFor Ġthis Ġpurpose , Ġwe Ġemploy Ġan ĠR NN - based Ġmodel Ġto Ġglobally Ġdis amb ig uate Ġthe Ġentities . ĠSpecifically , Ġwe Ġuse ĠBi GR U Ġand Ġconc aten ate Ġthe Ġforward Ġand Ġbackward Ġhidden Ġstate Ġvectors Ġas Ġthe Ġnew Ġentity Ġvector : Ġ$ Ġ\ over right arrow { h } _ i Ġ& = ĠGR U ( e _ i , Ġ\ over right arrow { h } _{ i - 1 }) Ġ\\ Ġ\ over left arrow { h } _ i Ġ& = ĠGR U ( e _ i , Ġ\ over left arrow { h } _{ i + 1 }) Ġ\\ Ġe ^ {\ prime Ġ} _ i Ġ& = Ġ[ \ over right arrow { h } _ i ; Ġ\ over left arrow { h } _ i ] Ġ\ non umber Ġ$ </s> Another Ġway Ġto Ġdis amb ig uate Ġan Ġentity Ġis Ġby Ġusing Ġonly Ġthe Ġdirect Ġneighbors Ġof Ġthe Ġentity , Ġputting Ġno Ġimportance Ġvalue Ġto Ġentities Ġthat Ġare Ġfar . ĠTo Ġdo Ġthis , Ġwe Ġemploy Ġa ĠCNN - based Ġmodel Ġto Ġlocally Ġdis amb ig uate Ġthe Ġentities . ĠSpecifically , Ġwe Ġdo Ġthe Ġconv olution Ġoperation Ġusing Ġfilter Ġmat rices Ġ$ W _ f Ġ\ in Ġ\ math bb Ġ{ R } ^{ h Ġ\ times Ġd } $ Ġwith Ġfilter Ġsize Ġ$ h $ Ġto Ġa Ġwindow Ġof Ġ$ h $ Ġwords . ĠWe Ġdo Ġthis Ġfor Ġdifferent Ġsizes Ġof Ġ$ h $ Ġ. ĠThis Ġproduces Ġnew Ġfeature Ġvectors Ġ$ c _{ i , h } $ Ġas Ġshown Ġbelow , Ġwhere Ġ$ f ( .) $ Ġis Ġa Ġnon - linear Ġfunction : Ġ$ Ġc _{ i , h } Ġ= Ġf ([ e _{ i -( h - 1 )/ 2 }; Ġ... ; Ġe _{ i + h ( + 1 )/ 2 } ] ^ \ top ĠW _ f Ġ+ Ġb _ f ) Ġ\ non umber Ġ$ </s> The Ġconv olution Ġoperation Ġreduces Ġthe Ġnumber Ġof Ġentities Ġdifferently Ġdepending Ġon Ġthe Ġfilter Ġsize Ġ$ h $ Ġ. ĠTo Ġprevent Ġloss Ġof Ġinformation Ġand Ġto Ġproduce Ġthe Ġsame Ġamount Ġof Ġfeature Ġvectors Ġ$ c _{ i , h } $ Ġ, Ġwe Ġpad Ġthe Ġentity Ġlist Ġdynamically Ġsuch Ġthat Ġwhen Ġthe Ġfilter Ġsize Ġis Ġ$ h $ Ġ, Ġthe Ġnumber Ġof Ġpadd ings Ġon Ġeach Ġside Ġis Ġ$( h - 1 )/ 2 $ Ġ. ĠThe Ġfilter Ġsize Ġ$ h $ Ġtherefore Ġrefers Ġto Ġthe Ġnumber Ġof Ġentities Ġused Ġto Ġdis amb ig uate Ġa Ġmiddle Ġentity . ĠFinally , Ġwe Ġconc aten ate Ġall Ġfeature Ġvectors Ġof Ġdifferent Ġ$ h $ Ġ' s Ġfor Ġeach Ġ$ i $ Ġas Ġthe Ġnew Ġentity Ġvector : Ġ$ Ġe ^ {\ prime Ġ} _ i Ġ= Ġ[ c _{ i , h _ 1 }; Ġc _{ i , Ġh _ 2 }; Ġ... ] Ġ\ non umber Ġ$ </s> The Ġquestion Ġon Ġwhich Ġdis amb ig uating Ġenc oder Ġis Ġbetter Ġhas Ġbeen Ġa Ġdebate ; Ġsome Ġargued Ġthat Ġusing Ġonly Ġthe Ġlocal Ġcontext Ġis Ġappropriate ĠB IB REF 16 Ġwhile Ġsome Ġclaimed Ġthat Ġadditionally Ġusing Ġglobal Ġcontext Ġalso Ġhelps ĠB IB REF 17 Ġ. ĠThe ĠR NN - based Ġenc oder Ġis Ġgood Ġas Ġit Ġsmart ly Ġmakes Ġuse Ġof Ġall Ġentities , Ġhowever Ġit Ġmay Ġperform Ġbad Ġwhen Ġthere Ġare Ġmany Ġentities Ġas Ġit Ġintroduces Ġnoise Ġwhen Ġusing Ġa Ġfar Ġentity Ġduring Ġdis amb ig uation . ĠThe ĠCNN - based Ġenc oder Ġis Ġgood Ġas Ġit Ġminim izes Ġthe Ġnoise Ġby Ġtotally Ġignoring Ġfar Ġentities Ġwhen Ġdis amb ig uating , Ġhowever Ġdetermining Ġthe Ġappropriate Ġfilter Ġsizes Ġ$ h $ Ġneeds Ġengineering . ĠOverall , Ġwe Ġargue Ġthat Ġwhen Ġthe Ġinput Ġtext Ġis Ġshort Ġ( e . g . Ġa Ġsentence ), Ġboth Ġenc od ers Ġperform Ġcompar ably , Ġotherwise Ġwhen Ġthe Ġinput Ġtext Ġis Ġlong Ġ( e . g . Ġa Ġdocument ), Ġthe ĠCNN - based Ġenc oder Ġperforms Ġbetter . </s> It Ġis Ġobvious Ġthat Ġnot Ġall Ġentities Ġneed Ġto Ġbe Ġdis amb ig uated . ĠWhen Ġa Ġcorrectly Ġlinked Ġand Ġalready Ġadequately Ġdis amb ig uated Ġentity Ġis Ġdis amb ig uated Ġagain , Ġit Ġwould Ġmake Ġthe Ġentity Ġvery Ġcontext - specific Ġand Ġmight Ġnot Ġbe Ġsuitable Ġfor Ġthe Ġsummar ization Ġtask . ĠOur Ġentity Ġencoding Ġsub module Ġtherefore Ġuses Ġa Ġselective Ġmechanism Ġthat Ġdecides Ġwhether Ġto Ġuse Ġthe Ġdis amb ig uating Ġenc oder Ġor Ġnot . ĠThis Ġis Ġdone Ġby Ġintroducing Ġa Ġselective Ġdis amb ig uation Ġgate Ġ$ d $ Ġ. ĠThe Ġfinal Ġentity Ġvector Ġ$\ t ilde { e } _ i $ Ġis Ġcalculated Ġas Ġthe Ġlinear Ġtransformation Ġof Ġ$ e _ i $ Ġand Ġ$ e ^ {\ prime Ġ} _ i $ Ġ: Ġ$ Ġe ^ {\ prime Ġ} _ i Ġ& = Ġenc oder ( e _ i ) Ġ\\ Ġd Ġ& = Ġ\ s igma Ġ( W _ d Ġe ^ {\ prime Ġ} _ i Ġ+ Ġb _ d ) Ġ\\ Ġ\ t ilde { e } _ i Ġ& = Ġd Ġ\ times Ġf ( W _ x Ġe _ i Ġ+ Ġb _ x ) Ġ+ Ġ\\ Ġ& Ġ\ quad Ġ( 1 - d ) Ġ\ times Ġf ( W _ y Ġe ^ {\ prime Ġ} _ i Ġ+ Ġb _ y ) Ġ\ non umber Ġ$ </s> The Ġfull Ġentity Ġencoding Ġsub module Ġis Ġillustrated Ġin ĠFigure Ġ3 Ġ. ĠUltimately , Ġthe Ġsub module Ġoutputs Ġthe Ġdis amb ig uated Ġentity Ġvectors Ġ$\ t ilde { E } Ġ= Ġ\ l brace Ġ\ t ilde { e } _ 1 , Ġ\ t ilde { e } _ 2 , Ġ... , Ġ\ t ilde { e } _ m \ r brace Ġ$ Ġ. </s> Pool ing Ġsub module </s> The Ġentity Ġvectors Ġ$\ t ilde { E } $ Ġare Ġpooled Ġto Ġcreate Ġa Ġsingle Ġtopic Ġvector Ġ$ t $ Ġthat Ġrepresents Ġthe Ġtopic Ġof Ġthe Ġsummary . ĠOne Ġpossible Ġpool ing Ġtechnique Ġis Ġto Ġuse Ġsoft Ġattention ĠB IB REF 18 Ġon Ġthe Ġvectors Ġto Ġdetermine Ġthe Ġimportance Ġvalue Ġof Ġeach Ġvector , Ġwhich Ġcan Ġbe Ġdone Ġby Ġmatching Ġeach Ġentity Ġvector Ġwith Ġthe Ġtext Ġvector Ġ$ s $ Ġfrom Ġthe Ġtext Ġenc oder Ġas Ġthe Ġcontext Ġvector . ĠThe Ġentity Ġvectors Ġare Ġthen Ġpooled Ġusing Ġweighted Ġsum . ĠOne Ġproblem Ġwith Ġsoft Ġattention Ġis Ġthat Ġit Ġconsiders Ġall Ġentity Ġvectors Ġwhen Ġconstructing Ġthe Ġtopic Ġvector . ĠHowever , Ġnot Ġall Ġentities Ġare Ġimportant Ġand Ġnecessary Ġwhen Ġgenerating Ġsumm aries . ĠMoreover , Ġa Ġnumber Ġof Ġthese Ġentities Ġmay Ġbe Ġerroneous Ġand Ġirrelevant , Ġas Ġreported Ġin ĠSection Ġ" Related Ġwork " Ġ. ĠSoft Ġattention Ġgives Ġnon - neg lig ible Ġimportant Ġscores Ġto Ġthese Ġentities , Ġthus Ġadds Ġunnecessary Ġnoise Ġto Ġthe Ġconstruction Ġof Ġthe Ġtopic Ġvector . </s> Our Ġpool ing Ġsub module Ġinstead Ġuses Ġfirm Ġattention Ġmechanism Ġto Ġconsider Ġonly Ġtop Ġ$ k $ Ġentities Ġwhen Ġconstructing Ġthe Ġtopic Ġvector . ĠThis Ġis Ġdone Ġin Ġa Ġdifferent iable Ġway Ġas Ġfollows : Ġ$ ĠG Ġ& = Ġv _ a ^ \ top Ġtan h ( W _ a Ġ\ t ilde { E } Ġ+ ĠU _ a Ġs ) Ġ\\ ĠK Ġ& = Ġtop \ _ k ( G ) Ġ\\ ĠP Ġ& = Ġsparse \ _ vector ( K , Ġ0 , Ġ- \ in fty Ġ) Ġ\\ Ġg ^ {\ prime Ġ} _ i Ġ& = Ġg _ i Ġ+ Ġp _ i Ġ\\ Ġa _ i Ġ& = Ġ\ frac { exp ( g ^ {\ prime Ġ} _ i ) }{ \ sum Ġ_ i Ġexp ( g ^ {\ prime Ġ} _ i )} Ġ\\ Ġt Ġ& = Ġ\ sum Ġ_ i Ġa _ i Ġ\ t ilde { e } _ i Ġ\ non umber Ġ$ </s> where Ġthe Ġfunctions Ġ$ K Ġ= Ġtop \ _ k ( G ) $ Ġgets Ġthe Ġindices Ġof Ġthe Ġtop Ġ$ k $ Ġvectors Ġin Ġ$ G $ Ġand Ġ$ P Ġ= Ġsparse \ _ vector ( K , 0 ,- \ in fty Ġ) $ Ġcreates Ġa Ġsparse Ġvector Ġwhere Ġthe Ġvalues Ġof Ġ$ K $ Ġis Ġ0 Ġand Ġ$ - \ in fty Ġ$ Ġotherwise . ĠThe Ġsparse Ġvector Ġ$ P $ Ġis Ġadded Ġto Ġthe Ġoriginal Ġimportance Ġscore Ġvector Ġ$ G $ Ġto Ġcreate Ġa Ġnew Ġimportance Ġscore Ġvector . ĠIn Ġthis Ġnew Ġvector , Ġimportant Ġscores Ġof Ġnon - top Ġ$ k $ 0 Ġentities Ġare Ġ$ k $ 1 Ġ. ĠWhen Ġsoft max Ġis Ġapplied , Ġthis Ġgives Ġvery Ġsmall , Ġnegligible , Ġand Ġclose - to - zero Ġvalues Ġto Ġnon - top Ġ$ k $ 2 Ġentities . ĠThe Ġvalue Ġ$ k $ 3 Ġdepends Ġon Ġthe Ġlengths Ġof Ġthe Ġinput Ġtext Ġand Ġsummary . ĠMoreover , Ġwhen Ġ$ k $ 4 Ġincreases Ġtowards Ġinfinity , Ġfirm Ġattention Ġbecomes Ġsoft Ġattention . ĠWe Ġdecide Ġ$ k $ 5 Ġempir ically Ġ( see ĠSection Ġ" Exper imental Ġsettings " Ġ). </s> Ext ending Ġfrom Ġthe Ġbase Ġmodel </s> Entity 2 Topic Ġmodule Ġextends Ġthe Ġbase Ġmodel Ġas Ġfollows . ĠThe Ġfinal Ġtext Ġrepresentation Ġvector Ġ$ s $ Ġis Ġused Ġas Ġa Ġcontext Ġvector Ġwhen Ġconstructing Ġthe Ġtopic Ġvector Ġ$ t $ Ġin Ġthe Ġpool ing Ġsub module . ĠThe Ġtopic Ġvector Ġ$ t $ Ġis Ġthen Ġconc aten ated Ġto Ġthe Ġdec oder Ġhidden Ġstate Ġvectors Ġ$ s _ i $ Ġ, Ġi . e . Ġ$ s ^ {\ prime Ġ} _ i Ġ= Ġ[ s _ i ; Ġt ] $ Ġ. ĠThe Ġconc aten ated Ġvector Ġis Ġfinally Ġused Ġto Ġcreate Ġthe Ġoutput Ġvector : </s> $$ o _ i Ġ= ĠW _ w Ġw _{ i - 1 } Ġ+ ĠW _ c Ġc _ i Ġ+ ĠW _ s Ġs ^ {\ prime Ġ} _ i Ġ\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ18 ) </s> Related Ġwork </s> Due Ġto Ġits Ġrecent Ġsuccess , Ġneural Ġnetwork Ġmodels Ġhave Ġbeen Ġused Ġwith Ġcompetitive Ġresults Ġon Ġabstract ive Ġsummar ization . ĠA Ġneural Ġattention Ġmodel Ġwas Ġfirst Ġapplied Ġto Ġthe Ġtask , Ġeasily Ġachieving Ġstate - of - the - art Ġperformance Ġon Ġmultiple Ġdatasets ĠB IB REF 5 Ġ. ĠThe Ġmodel Ġhas Ġbeen Ġextended Ġto Ġinstead Ġuse Ġrecurrent Ġneural Ġnetwork Ġas Ġdec oder ĠB IB REF 3 Ġ. ĠThe Ġmodel Ġwas Ġfurther Ġextended Ġto Ġuse Ġa Ġfull ĠR NN Ġenc oder - dec oder Ġframework Ġand Ġfurther Ġenhancements Ġthrough Ġlex ical Ġand Ġstatistical Ġfeatures ĠB IB REF 15 Ġ. ĠThe Ġcurrent Ġstate - of - the - art Ġperformance Ġis Ġachieved Ġby Ġselectively Ġencoding Ġwords Ġas Ġa Ġprocess Ġof Ġdist illing Ġsalient Ġinformation ĠB IB REF 13 Ġ. </s> Ne ural Ġabstract ive Ġsummar ization Ġmodels Ġhave Ġalso Ġbeen Ġexplored Ġto Ġsummarize Ġlonger Ġdocuments . ĠWord Ġextraction Ġmodels Ġhave Ġbeen Ġpreviously Ġexplored , Ġperforming Ġworse Ġthan Ġsentence Ġextraction Ġmodels ĠB IB REF 19 Ġ. ĠHier arch ical Ġattention - based Ġrecurrent Ġneural Ġnetworks Ġhave Ġalso Ġbeen Ġapplied Ġto Ġthe Ġtask , Ġowing Ġto Ġthe Ġidea Ġthat Ġthere Ġare Ġmultiple Ġsentences Ġin Ġa Ġdocument ĠB IB REF 15 Ġ. ĠFinally , Ġdistraction - based Ġmodels Ġwere Ġproposed Ġto Ġenable Ġmodels Ġto Ġtraverse Ġthe Ġtext Ġcontent Ġand Ġgrasp Ġthe Ġoverall Ġmeaning ĠB IB REF 4 Ġ. ĠThe Ġcurrent Ġstate - of - the - art Ġperformance Ġis Ġachieved Ġby Ġa Ġgraph - based Ġattention al Ġneural Ġmodel , Ġconsidering Ġthe Ġkey Ġfactors Ġof Ġdocument Ġsummar ization Ġsuch Ġas Ġsal iency , Ġflu ency Ġand Ġnovelty ĠB IB REF 14 Ġ. </s> Previous Ġstudies Ġon Ġthe Ġsummar ization Ġtasks Ġhave Ġonly Ġused Ġentities Ġin Ġthe Ġpre processing Ġstage Ġto Ġanonym ize Ġthe Ġdataset ĠB IB REF 15 Ġand Ġto Ġmitigate Ġout - of - voc abulary Ġproblems ĠB IB REF 14 Ġ. ĠLink ed Ġentities Ġfor Ġsummar ization Ġare Ġstill Ġnot Ġproperly Ġexplored Ġand Ġwe Ġare Ġthe Ġfirst Ġto Ġuse Ġlinked Ġentities Ġto Ġimprove Ġthe Ġperformance Ġof Ġthe Ġsummar izer . </s> Results </s> We Ġreport Ġthe ĠR OU GE ĠF 1 Ġscores Ġfor Ġboth Ġdatasets Ġof Ġall Ġthe Ġcompeting Ġmodels Ġusing ĠR OU GE ĠF 1 Ġscores ĠB IB REF 27 Ġ. ĠWe Ġreport Ġthe Ġresults Ġon Ġthe ĠGig aw ord Ġand Ġthe ĠCNN Ġdataset Ġin ĠTable Ġ2 Ġand ĠTable Ġ3 Ġ, Ġrespectively . ĠIn ĠGig aw ord Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġshort , Ġour Ġbest Ġmodel Ġachieves Ġa Ġcomparable Ġperformance Ġwith Ġthe Ġcurrent Ġstate - of - the - art . ĠIn ĠCNN Ġdataset Ġwhere Ġthe Ġtexts Ġare Ġlonger , Ġour Ġbest Ġmodel Ġoutper forms Ġall Ġthe Ġprevious Ġmodels . ĠWe Ġemphasize Ġthat ĠE 2 T Ġmodule Ġis Ġeasily Ġattach able Ġto Ġbetter Ġmodels , Ġand Ġwe Ġexpect ĠE 2 T Ġto Ġimprove Ġtheir Ġperformance Ġas Ġwell . ĠOverall , ĠE 2 T Ġachieves Ġa Ġsignificant Ġimprovement Ġover Ġthe Ġbaseline Ġmodel Ġbase , Ġwith Ġat Ġleast Ġ2 ĠR OU GE - 1 Ġpoints Ġincrease Ġin Ġthe ĠGig aw ord Ġdataset Ġand Ġ6 ĠR OU GE - 1 Ġpoints Ġincrease Ġin Ġthe ĠCNN Ġdataset . ĠIn Ġfact , Ġall Ġvariants Ġof ĠE 2 T Ġgain Ġimprovements Ġover Ġthe Ġbaseline , Ġimplying Ġthat Ġleveraging Ġon Ġlinked Ġentities Ġimproves Ġthe Ġperformance Ġof Ġthe Ġsummar izer . ĠAmong Ġthe Ġmodel Ġvariants , Ġthe ĠCNN - based Ġenc oder Ġwith Ġselective Ġdis amb ig uation Ġand Ġfirm Ġattention Ġperforms Ġthe Ġbest . </s> Aut omatic Ġevaluation Ġon Ġthe ĠGig aw ord Ġdataset Ġshows Ġthat Ġthe ĠCNN Ġand ĠR NN Ġvariants Ġof Ġbase + E 2 T Ġhave Ġsimilar Ġperformance . ĠTo Ġbreak Ġthe Ġtie Ġbetween Ġboth Ġmodels , Ġwe Ġalso Ġconduct Ġhuman Ġevaluation Ġon Ġthe ĠGig aw ord Ġdataset . ĠWe Ġinstruct Ġtwo Ġannot ators Ġto Ġread Ġthe Ġinput Ġsentence Ġand Ġrank Ġthe Ġcompeting Ġsumm aries Ġfrom Ġfirst Ġto Ġlast Ġaccording Ġto Ġtheir Ġrelevance Ġand Ġflu ency : Ġ( a ) Ġthe Ġoriginal Ġsummary Ġgold , Ġand Ġfrom Ġmodels Ġ( b ) Ġbase , Ġ( c ) Ġbase + E 2 T c nn , Ġand Ġ( d ) Ġbase + E 2 Tr nn . ĠWe Ġthen Ġcompute Ġ( i ) Ġthe Ġproportion Ġof Ġevery Ġranking Ġof Ġeach Ġmodel Ġand Ġ( ii ) Ġthe Ġmean Ġrank Ġof Ġeach Ġmodel . ĠThe Ġresults Ġare Ġreported Ġin ĠTable Ġ4 Ġ. ĠThe Ġmodel Ġwith Ġthe Ġbest Ġmean Ġrank Ġis Ġbase + E 2 T c nn , Ġfollowed Ġby Ġgold , Ġthen Ġby Ġbase + E 2 Tr nn Ġand Ġbase , Ġrespectively . ĠWe Ġalso Ġperform ĠAN OVA Ġand Ġpost - h oc ĠTu key Ġtests Ġto Ġshow Ġthat Ġthe ĠCNN Ġvariant Ġis Ġsignificantly Ġ( Ġ$ p < 0 . 01 $ Ġ) Ġbetter Ġthan Ġthe ĠR NN Ġvariant Ġand Ġthe Ġbase Ġmodel . ĠThe ĠR NN Ġvariant Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġthe ĠCNN Ġvariant , Ġcontrary Ġto Ġthe Ġautomatic ĠR OU GE Ġevaluation Ġabove . ĠInterestingly , Ġthe ĠCNN Ġvariant Ġproduces Ġbetter Ġ( but Ġwith Ġno Ġsignificant Ġdifference ) Ġsumm aries Ġthan Ġthe Ġgold Ġsumm aries . ĠWe Ġposit Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġthe Ġarticle Ġtitle Ġdoes Ġnot Ġcorrespond Ġto Ġthe Ġsummary Ġof Ġthe Ġfirst Ġsentence . </s> Conclusion </s> We Ġproposed Ġto Ġleverage Ġon Ġlinked Ġentities Ġto Ġimprove Ġthe Ġperformance Ġof Ġsequence - to - sequence Ġmodels Ġon Ġneural Ġabstract ive Ġsummar ization Ġtask . ĠLink ed Ġentities Ġare Ġused Ġto Ġguide Ġthe Ġdecoding Ġprocess Ġbased Ġon Ġthe Ġsummary Ġtopic Ġand Ġcommons ense Ġlearned Ġfrom Ġa Ġknowledge Ġbase . ĠWe Ġintroduced ĠEntity 2 Topic Ġ( E 2 T ), Ġa Ġmodule Ġthat Ġis Ġeasily Ġattach able Ġto Ġany Ġmodel Ġusing Ġan Ġenc oder - dec oder Ġframework . ĠE 2 T Ġapplies Ġlinked Ġentities Ġinto Ġthe Ġsummar izer Ġby Ġencoding Ġthe Ġentities Ġwith Ġselective Ġdis amb ig uation Ġand Ġpool ing Ġthem Ġinto Ġone Ġsummary Ġtopic Ġvector Ġwith Ġfirm Ġattention Ġmechanism . ĠWe Ġshowed Ġthat Ġby Ġapplying ĠE 2 T Ġto Ġa Ġbasic Ġsequence - to - sequence Ġmodel , Ġwe Ġachieve Ġsignificant Ġimprovements Ġover Ġthe Ġbase Ġmodel Ġand Ġconsequently Ġachieve Ġa Ġcomparable Ġperformance Ġwith Ġmore Ġcomplex Ġsummar ization Ġmodels . </s> Acknowled gement </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġfeedback . ĠThis Ġwork Ġwas Ġsupported Ġby ĠMicrosoft ĠResearch , Ġand ĠInstitute Ġfor ĠInformation Ġcommunications ĠTechnology ĠPromotion Ġ( I IT P ) Ġgrant Ġfunded Ġby Ġthe ĠKorea Ġgovernment Ġ( MS IT ) Ġ( No . 2017 - 0 - 01 778 Ġ, ĠDevelopment Ġof ĠExplain able ĠHuman level ĠDeep ĠMachine ĠLearning ĠIn ference ĠFramework ). ĠS . ĠH wang Ġis Ġa Ġcorresponding Ġauthor .