global (the whole document), local context (e.g., the section/topic)
EXTRACTIVE
global (the whole document) and the local context (e.g., the section/topic) 
EXTRACTIVE
In contrast, in order to exploit section information, in this paper we propose to capture a distributed representation of both the global (the whole document) and the local context (e.g., the section/topic) when deciding if a sentence should be included in the summary
In contrast, in order to exploit section information, in this paper we propose to capture a distributed representation of both the global (the whole document) and the local context (e.g., the section/topic) when deciding if a sentence should be included in the summary
1909.08089
Introduction </s> Single - document Ġsummar ization Ġis Ġthe Ġtask Ġof Ġgenerating Ġa Ġshort Ġsummary Ġfor Ġa Ġgiven Ġdocument . ĠIdeally , Ġthe Ġgenerated Ġsumm aries Ġshould Ġbe Ġfluent Ġand Ġcoherent , Ġand Ġshould Ġfaithfully Ġmaintain Ġthe Ġmost Ġimportant Ġinformation Ġin Ġthe Ġsource Ġdocument . Ġpurple This Ġis Ġa Ġvery Ġchallenging Ġtask , Ġbecause Ġit Ġarguably Ġrequires Ġan Ġin - depth Ġunderstanding Ġof Ġthe Ġsource Ġdocument , Ġand Ġcurrent Ġautomatic Ġsolutions Ġare Ġstill Ġfar Ġfrom Ġhuman Ġperformance ĠB IB REF 0 Ġ. </s> Single - document Ġsummar ization Ġcan Ġbe Ġeither Ġextract ive Ġor Ġabstract ive . ĠExtract ive Ġmethods Ġtypically Ġpick Ġsentences Ġdirectly Ġfrom Ġthe Ġoriginal Ġdocument Ġbased Ġon Ġtheir Ġimportance , Ġand Ġform Ġthe Ġsummary Ġas Ġan Ġaggregate Ġof Ġthese Ġsentences . ĠUsually , Ġsumm aries Ġgenerated Ġin Ġthis Ġway Ġhave Ġa Ġbetter Ġperformance Ġon Ġflu ency Ġand Ġgrammar , Ġbut Ġthey Ġmay Ġcontain Ġmuch Ġredundancy Ġand Ġlack Ġin Ġco herence Ġacross Ġsentences . ĠIn Ġcontrast , Ġabstract ive Ġmethods Ġattempt Ġto Ġmimic Ġwhat Ġhumans Ġdo Ġby Ġfirst Ġextracting Ġcontent Ġfrom Ġthe Ġsource Ġdocument Ġand Ġthen Ġproduce Ġnew Ġsentences Ġthat Ġaggregate Ġand Ġorganize Ġthe Ġextracted Ġinformation . ĠSince Ġthe Ġsentences Ġare Ġgenerated Ġfrom Ġscratch Ġthey Ġtend Ġto Ġhave Ġa Ġrelatively Ġworse Ġperformance Ġon Ġflu ency Ġand Ġgrammar . ĠFurthermore , Ġwhile Ġabstract ive Ġsumm aries Ġare Ġtypically Ġless Ġredundant , Ġthey Ġmay Ġend Ġup Ġincluding Ġmisleading Ġor Ġeven Ġutterly Ġfalse Ġstatements , Ġbecause Ġthe Ġmethods Ġto Ġextract Ġand Ġaggregate Ġinformation Ġform Ġthe Ġsource Ġdocument Ġare Ġstill Ġrather Ġnoisy . </s> In Ġthis Ġwork , Ġwe Ġfocus Ġon Ġextracting Ġinformative Ġsentences Ġfrom Ġa Ġgiven Ġdocument Ġ( without Ġdealing Ġwith Ġredundancy ), Ġespecially Ġwhen Ġthe Ġdocument Ġis Ġrelatively Ġlong Ġ( e . g ., Ġscientific Ġarticles ). </s> Most Ġrecent Ġworks Ġon Ġneural Ġextract ive Ġsummar ization Ġhave Ġbeen Ġrather Ġsuccessful Ġin Ġgenerating Ġsumm aries Ġof Ġshort Ġnews Ġdocuments Ġ( around Ġ650 Ġwords / document ) ĠB IB REF 1 Ġby Ġapplying Ġneural ĠSe q 2 Se q Ġmodels ĠB IB REF 2 Ġ. ĠHowever Ġwhen Ġit Ġcomes Ġto Ġlong Ġdocuments , Ġthese Ġmodels Ġtend Ġto Ġstruggle Ġwith Ġlonger Ġsequences Ġbecause Ġat Ġeach Ġdecoding Ġstep , Ġthe Ġdec oder Ġneeds Ġto Ġlearn Ġto Ġconstruct Ġa Ġcontext Ġvector Ġcapturing Ġrelevant Ġinformation Ġfrom Ġall Ġthe Ġtokens Ġin Ġthe Ġsource Ġsequence ĠB IB REF 3 Ġ. </s> Long Ġdocuments Ġtypically Ġcover Ġmultiple Ġtopics . ĠIn Ġgeneral , Ġthe Ġlonger Ġa Ġdocument Ġis , Ġthe Ġmore Ġtopics Ġare Ġdiscussed . ĠAs Ġa Ġmatter Ġof Ġfact , Ġwhen Ġhumans Ġwrite Ġlong Ġdocuments Ġthey Ġorganize Ġthem Ġin Ġchapters , Ġsections Ġetc .. ĠScientific Ġpapers Ġare Ġan Ġexample Ġof Ġlonger Ġdocuments Ġand Ġthey Ġfollow Ġa Ġstandard Ġdiscourse Ġstructure Ġdescribing Ġthe Ġproblem , Ġmethodology , Ġexperiments / results , Ġand Ġfinally Ġconclusions ĠB IB REF 4 Ġ. </s> To Ġthe Ġbest Ġof Ġour Ġknowledge Ġonly Ġone Ġprevious Ġwork Ġin Ġextract ive Ġsummar ization Ġhas Ġexplicitly Ġlever aged Ġsection Ġinformation Ġto Ġguide Ġthe Ġgeneration Ġof Ġsumm aries ĠB IB REF 5 Ġ. ĠHowever , Ġthe Ġonly Ġinformation Ġabout Ġsections Ġfed Ġinto Ġtheir Ġsentence Ġclass ifier Ġis Ġa Ġcateg orical Ġfeature Ġwith Ġvalues Ġlike ĠHigh light , ĠAbstract , ĠIntroduction , Ġetc ., Ġdepending Ġon Ġwhich Ġsection Ġthe Ġsentence Ġappears Ġin . </s> In Ġcontrast , Ġin Ġorder Ġto Ġexploit Ġsection Ġinformation , Ġin Ġthis Ġpaper Ġwe Ġpropose Ġto Ġcapture Ġa Ġdistributed Ġrepresentation Ġof Ġboth Ġthe Ġglobal Ġ( the Ġwhole Ġdocument ) Ġand Ġthe Ġlocal Ġcontext Ġ( e . g ., Ġthe Ġsection / topic ) Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary </s> Our Ġmain Ġcontributions Ġare Ġas Ġfollows : Ġ( i ) ĠIn Ġorder Ġto Ġcapture Ġthe Ġlocal Ġcontext , Ġwe Ġare Ġthe Ġfirst Ġto Ġapply ĠL ST M - minus Ġto Ġtext Ġsummar ization . ĠL ST M - minus Ġis Ġa Ġmethod Ġfor Ġlearning Ġembed d ings Ġof Ġtext Ġspans , Ġwhich Ġhas Ġachieved Ġgood Ġperformance Ġin Ġdependency Ġparsing ĠB IB REF 6 Ġ, Ġin Ġconstituency Ġparsing ĠB IB REF 7 Ġ, Ġas Ġwell Ġas Ġin Ġdiscourse Ġparsing ĠB IB REF 8 Ġ. ĠWith Ġrespect Ġto Ġmore Ġtraditional Ġmethods Ġfor Ġcapturing Ġlocal Ġcontext , Ġwhich Ġrely Ġon Ġhierarchical Ġstructures , ĠL ST M - minus Ġproduces Ġsimpler Ġmodels Ġi . e . Ġwith Ġless Ġparameters , Ġand Ġtherefore Ġfaster Ġto Ġtrain Ġand Ġless Ġprone Ġto Ġover fitting . Ġ( ii ) ĠWe Ġtest Ġour Ġmethod Ġon Ġthe ĠPub med Ġand Ġar X iv Ġdatasets Ġand Ġresults Ġappear Ġto Ġsupport Ġour Ġgoal Ġof Ġeffectively Ġsummar izing Ġlong Ġdocuments . ĠIn Ġparticular , Ġwhile Ġoverall Ġwe Ġoutper form Ġthe Ġbaseline Ġand Ġprevious Ġapproaches Ġonly Ġby Ġa Ġnarrow Ġmargin Ġon Ġboth Ġdatasets , Ġthe Ġbenefit Ġof Ġour Ġmethod Ġbecome Ġmuch Ġstronger Ġas Ġwe Ġapply Ġit Ġto Ġlonger Ġdocuments . Ġpurple Furthermore , Ġin Ġan Ġab lation Ġstudy Ġto Ġassess Ġthe Ġrelative Ġcontributions Ġof Ġthe Ġglobal Ġand Ġthe Ġlocal Ġmodel Ġwe Ġfound Ġthat , Ġrather Ġsurprisingly , Ġthe Ġbenefits Ġof Ġour Ġmodel Ġseem Ġto Ġcome Ġexclusively Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext , Ġeven Ġfor Ġthe Ġlongest Ġdocuments .[ 6 ] Ġ( iii ) ĠIn Ġorder Ġto Ġevaluate Ġour Ġapproach , Ġwe Ġhave Ġcreated Ġor acle Ġlabels Ġfor Ġboth ĠPub med Ġand Ġar X iv ĠB IB REF 9 Ġ, Ġby Ġapplying Ġa Ġgreedy Ġor acle Ġlabeling Ġalgorithm . ĠThe Ġtwo Ġdatasets Ġannot ated Ġwith Ġextract ive Ġlabels Ġwill Ġbe Ġmade Ġpublic . </s> Ext ractive Ġsummar ization </s> Traditional Ġextract ive Ġsummar ization Ġmethods Ġare Ġmostly Ġbased Ġon Ġexplicit Ġsurface Ġfeatures ĠB IB REF 10 Ġ, Ġrelying Ġon Ġgraph - based Ġmethods ĠB IB REF 11 Ġ, Ġor Ġon Ġsub mod ular Ġmaxim ization ĠB IB REF 12 Ġ. ĠBenef iting Ġfrom Ġthe Ġsuccess Ġof Ġneural Ġsequence Ġmodels Ġin Ġother ĠN LP Ġtasks , Ġche ng lap ata Ġpropose Ġa Ġnovel Ġapproach Ġto Ġextract ive Ġsummar ization Ġbased Ġon Ġneural Ġnetworks Ġand Ġcontinuous Ġsentence Ġfeatures , Ġwhich Ġoutper forms Ġtraditional Ġmethods Ġon Ġthe ĠDaily Mail Ġdataset . ĠIn Ġparticular , Ġthey Ġdevelop Ġa Ġgeneral Ġenc oder - dec oder Ġarchitecture , Ġwhere Ġa ĠCNN Ġis Ġused Ġas Ġsentence Ġenc oder , Ġa Ġun i - direction al ĠL ST M Ġas Ġdocument Ġenc oder , Ġwith Ġanother Ġun i - direction al ĠL ST M Ġas Ġdec oder . ĠTo Ġdecrease Ġthe Ġnumber Ġof Ġparameters Ġwhile Ġmaintaining Ġthe Ġaccuracy , Ġsummar un ner Ġpresent ĠSum ma Ru NN er , Ġa Ġsimple ĠR NN - based Ġsequence Ġclass ifier Ġwithout Ġdec oder , Ġoutper forming Ġor Ġmatching Ġthe Ġmodel Ġof ĠB IB REF 2 Ġ. ĠThey Ġtake Ġcontent , Ġsal ience , Ġnovelty , Ġand Ġposition Ġof Ġeach Ġsentence Ġinto Ġconsideration Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġextract ive Ġsummary . ĠYet , Ġthey Ġdo Ġnot Ġcapture Ġany Ġaspect Ġof Ġthe Ġtopical Ġstructure , Ġas Ġwe Ġdo Ġin Ġthis Ġpaper . ĠSo Ġtheir Ġapproach Ġwould Ġarguably Ġsuffer Ġwhen Ġapplied Ġto Ġlong Ġdocuments , Ġlikely Ġcontaining Ġmultiple Ġand Ġdiverse Ġtopics . </s> While ĠSum ma Ru NN er Ġwas Ġtested Ġonly Ġon Ġnews , ĠEM N LP 2018 Ġcarry Ġout Ġa Ġcomprehensive Ġset Ġof Ġexperiments Ġwith Ġdeep Ġlearning Ġmodels Ġof Ġextract ive Ġsummar ization Ġacross Ġdifferent Ġdomains , Ġi . e . Ġnews , Ġpersonal Ġstories , Ġmeetings , Ġand Ġmedical Ġarticles , Ġas Ġwell Ġas Ġacross Ġdifferent Ġneural Ġarchitectures , Ġin Ġorder Ġto Ġbetter Ġunderstand Ġthe Ġgeneral Ġpros Ġand Ġcons Ġof Ġdifferent Ġdesign Ġchoices . ĠThey Ġfind Ġthat Ġnon Ġauto - reg ressive Ġsentence Ġextraction Ġperforms Ġas Ġwell Ġor Ġbetter Ġthan Ġauto - reg ressive Ġextraction Ġin Ġall Ġdomains , Ġwhere Ġby Ġauto - reg ressive Ġsentence Ġextraction Ġthey Ġmean Ġusing Ġprevious Ġpredictions Ġto Ġinform Ġfuture Ġpredictions . ĠFurthermore , Ġthey Ġfind Ġthat Ġthe ĠAverage ĠWord ĠEmb edd ing Ġsentence Ġenc oder Ġworks Ġat Ġleast Ġas Ġwell Ġas Ġenc od ers Ġbased Ġon ĠCNN Ġand ĠR NN . ĠIn Ġlight Ġof Ġthese Ġfindings , Ġour Ġmodel Ġis Ġnot Ġauto - reg ressive Ġand Ġuses Ġthe ĠAverage ĠWord ĠEmb edd ing Ġenc oder . </s> Ext ractive Ġsummar ization Ġon ĠScientific Ġpapers </s> Research Ġon Ġsummar izing Ġscientific Ġarticles Ġhas Ġa Ġlong Ġhistory ĠB IB REF 13 Ġ. ĠEarlier Ġon , Ġit Ġwas Ġrealized Ġthat Ġsummar izing Ġscientific Ġpapers Ġrequires Ġdifferent Ġapproaches Ġthan Ġwhat Ġwas Ġused Ġfor Ġsummar izing Ġnews Ġarticles , Ġdue Ġto Ġdifferences Ġin Ġdocument Ġlength , Ġwriting Ġstyle Ġand Ġrhetorical Ġstructure . ĠFor Ġinstance , ĠB IB REF 14 Ġpresented Ġa Ġsupervised ĠNa ive ĠBay es Ġclass ifier Ġto Ġselect Ġcontent Ġfrom Ġa Ġscientific Ġpaper Ġbased Ġon Ġthe Ġrhetorical Ġstatus Ġof Ġeach Ġsentence Ġ( e . g ., Ġwhether Ġit Ġspecified Ġa Ġresearch Ġgoal , Ġor Ġsome Ġgenerally Ġaccepted Ġscientific Ġbackground Ġknowledge , Ġetc .). ĠMore Ġrecently , Ġresearchers Ġhave Ġextended Ġthis Ġwork Ġby Ġapplying Ġmore Ġsophisticated Ġclass ifiers Ġto Ġidentify Ġmore Ġfine - grain Ġrhetorical Ġcategories , Ġas Ġwell Ġas Ġby Ġexploiting Ġcitation Ġcontexts . Ġ2013 - disc ourse Ġpropose Ġthe ĠCore SC Ġdiscourse - driven Ġcontent , Ġwhich Ġrelies Ġon ĠCR Fs Ġand ĠSV Ms , Ġto Ġclassify Ġthe Ġdiscourse Ġcategories Ġ( e . g . ĠBackground , ĠHyp ot hesis , ĠMot ivation , Ġetc .) Ġat Ġthe Ġsentence Ġlevel . ĠThe Ġrecent Ġwork Ġmost Ġsimilar Ġto Ġours Ġis ĠB IB REF 5 Ġwhere , Ġin Ġorder Ġto Ġdetermine Ġwhether Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary , Ġthey Ġdirectly Ġuse Ġthe Ġsection Ġeach Ġsentence Ġappears Ġin Ġas Ġa Ġcateg orical Ġfeature Ġwith Ġvalues Ġlike ĠHigh light , ĠAbstract , ĠIntroduction , Ġetc .. ĠIn Ġthis Ġpaper , Ġinstead Ġof Ġusing Ġsections Ġas Ġcateg orical Ġfeatures , Ġwe Ġrely Ġon Ġa Ġdistributed Ġrepresentation Ġof Ġthe Ġsemantic Ġinformation Ġwithin Ġeach Ġsection , Ġas Ġthe Ġlocal Ġcontext Ġof Ġeach Ġsentence . ĠIn Ġa Ġvery Ġdifferent Ġline Ġof Ġwork , Ġco han - 2015 - scientific Ġform Ġthe Ġsummary Ġby Ġalso Ġexploiting Ġinformation Ġon Ġhow Ġthe Ġtarget Ġpaper Ġis Ġcited Ġin Ġother Ġpapers . ĠCurrently , Ġwe Ġdo Ġnot Ġuse Ġany Ġinformation Ġfrom Ġcitation Ġcontexts . </s> Dat as ets Ġfor Ġlong Ġdocuments </s> summary dat as et Ġprovide Ġa Ġcomprehensive Ġoverview Ġof Ġthe Ġcurrent Ġdatasets Ġfor Ġsummar ization . ĠNotice ably , Ġmost Ġof Ġthe Ġlarger - scale Ġsummar ization Ġdatasets Ġconsists Ġof Ġrelatively Ġshort Ġdocuments , Ġlike ĠCNN / Daily Mail ĠB IB REF 1 Ġand ĠNew ĠYork ĠTimes ĠB IB REF 15 Ġ. ĠOne Ġexception Ġis ĠB IB REF 9 Ġthat Ġrecently Ġintroduce Ġtwo Ġlarge - scale Ġdatasets Ġof Ġlong Ġand Ġstructured Ġscientific Ġpapers Ġobtained Ġfrom Ġar X iv Ġand ĠPubMed . ĠThese Ġtwo Ġnew Ġdatasets Ġcontain Ġmuch Ġlonger Ġdocuments Ġthan Ġall Ġthe Ġnews Ġdatasets Ġ( See ĠTable ĠT AB REF 6 Ġ) Ġand Ġare Ġtherefore Ġideal Ġtest - bed s Ġfor Ġthe Ġmethod Ġwe Ġpresent Ġin Ġthis Ġpaper . </s> Ne ural ĠAbstract ive Ġsummar ization Ġon Ġlong Ġdocuments </s> While Ġmost Ġcurrent Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġhave Ġfocused Ġon Ġsummar izing Ġrelatively Ġshort Ġnews Ġarticles Ġ( e . g ., ĠB IB REF 16 Ġ), Ġfew Ġresearchers Ġhave Ġstarted Ġto Ġinvestigate Ġthe Ġsummar ization Ġof Ġlonger Ġdocuments Ġby Ġexploiting Ġtheir Ġnatural Ġstructure . Ġagents Ġpresent Ġan Ġenc oder - dec oder Ġarchitecture Ġto Ġaddress Ġthe Ġchallenges Ġof Ġrepresenting Ġa Ġlong Ġdocument Ġfor Ġabstract ive Ġsummar ization . ĠThe Ġencoding Ġtask Ġis Ġdivided Ġacross Ġseveral Ġcollaborating Ġagents , Ġeach Ġis Ġresponsible Ġfor Ġa Ġsubsection Ġof Ġtext Ġthrough Ġa Ġmulti - layer ĠL ST M Ġwith Ġword Ġattention . ĠTheir Ġmodel Ġseems Ġhowever Ġoverly Ġcomplicated Ġwhen Ġit Ġcomes Ġto Ġthe Ġextract ive Ġsummar ization Ġtask , Ġwhere Ġword Ġattention Ġis Ġarguably Ġmuch Ġless Ġcritical . ĠSo , Ġwe Ġdo Ġnot Ġconsider Ġthis Ġmodel Ġfurther Ġin Ġthis Ġpaper . </s> disc our sel ong document Ġalso Ġpropose Ġa Ġmodel Ġfor Ġabstract ive Ġsummar ization Ġtaking Ġthe Ġstructure Ġof Ġdocuments Ġinto Ġconsideration Ġwith Ġa Ġhierarchical Ġapproach , Ġand Ġtest Ġit Ġon Ġlonger Ġdocuments Ġwith Ġsection Ġinformation , Ġi . e . Ġscientific Ġpapers . ĠIn Ġparticular , Ġthey Ġapply Ġa Ġhierarchical Ġenc oder Ġat Ġthe Ġword Ġand Ġsection Ġlevels . ĠThen , Ġin Ġthe Ġdecoding Ġstep , Ġthey Ġcombine Ġthe Ġword Ġattention Ġand Ġsection Ġattention Ġto Ġobtain Ġa Ġcontext Ġvector . </s> This Ġapproach Ġto Ġcapture Ġdiscourse Ġstructure Ġis Ġhowever Ġquite Ġlimited Ġboth Ġin Ġgeneral Ġand Ġespecially Ġwhen Ġyou Ġconsider Ġits Ġapplication Ġto Ġextract ive Ġsummar ization . ĠFirst , Ġtheir Ġhierarchical Ġmethod Ġhas Ġa Ġlarge Ġnumber Ġof Ġparameters Ġand Ġit Ġis Ġtherefore Ġslow Ġto Ġtrain Ġand Ġlikely Ġprone Ġto Ġover fitting . ĠSecondly , Ġit Ġdoes Ġnot Ġtake Ġthe Ġglobal Ġcontext Ġof Ġthe Ġwhole Ġdocument Ġinto Ġaccount , Ġwhich Ġmay Ġarguably Ġbe Ġcritical Ġin Ġextract ive Ġmethods , Ġwhen Ġdeciding Ġon Ġthe Ġsal ience Ġof Ġa Ġsentence Ġ( or Ġeven Ġa Ġword ). ĠThe Ġextract ive Ġsummar izer Ġwe Ġpresent Ġin Ġthis Ġpaper Ġtries Ġto Ġaddress Ġthese Ġlimitations Ġby Ġadopting Ġthe Ġparameter Ġlean ĠL ST M - minus Ġmethod , Ġand Ġby Ġexplicitly Ġmodeling Ġthe Ġglobal Ġcontext . </s> L ST M - Min us </s> The ĠL ST M - Min us Ġmethod Ġis Ġfirst Ġproposed Ġin ĠB IB REF 6 Ġas Ġa Ġnovel Ġway Ġto Ġlearn Ġsentence Ġsegment Ġembed d ings Ġfor Ġgraph - based Ġdependency Ġparsing , Ġi . e . Ġestimating Ġthe Ġmost Ġlikely Ġdependency Ġtree Ġgiven Ġan Ġinput Ġsentence . ĠFor Ġeach Ġdependency Ġpair , Ġthey Ġdivide Ġa Ġsentence Ġinto Ġthree Ġsegments Ġ( prefix , Ġinf ix Ġand Ġsuffix ), Ġand ĠL ST M - Min us Ġis Ġused Ġto Ġrepresent Ġeach Ġsegment . ĠThey Ġapply Ġa Ġsingle ĠL ST M Ġto Ġthe Ġwhole Ġsentence Ġand Ġuse Ġthe Ġdifference Ġbetween Ġtwo Ġhidden Ġstates ĠIN LINE FORM 0 Ġto Ġrepresent Ġthe Ġsegment Ġfrom Ġword ĠIN LINE FORM 1 Ġto Ġword ĠIN LINE FORM 2 Ġ. ĠThis Ġenables Ġtheir Ġmodel Ġto Ġlearn Ġsegment Ġembed d ings Ġfrom Ġinformation Ġboth Ġoutside Ġand Ġinside Ġthe Ġsegments Ġand Ġthus Ġenhancing Ġtheir Ġmodel Ġability Ġto Ġaccess Ġto Ġsentence - level Ġinformation . ĠThe Ġintuition Ġbehind Ġthe Ġmethod Ġis Ġthat Ġeach Ġhidden Ġvector ĠIN LINE FORM 3 Ġcan Ġcapture Ġuseful Ġinformation Ġbefore Ġand Ġincluding Ġthe Ġword ĠIN LINE FORM 4 Ġ. </s> Shortly Ġafter , Ġl st m - minus const itu ency Ġuse Ġthe Ġsame Ġmethod Ġon Ġthe Ġtask Ġof Ġconstituency Ġparsing , Ġas Ġthe Ġrepresentation Ġof Ġa Ġsentence Ġspan , Ġextending Ġthe Ġoriginal Ġun i - direction al ĠL ST M - Min us Ġto Ġthe Ġbi - direction al Ġcase . ĠMore Ġrecently , Ġinspired Ġby Ġthe Ġsuccess Ġof ĠL ST M - Min us Ġin Ġboth Ġdependency Ġand Ġconstituency Ġparsing , Ġl st m - minus disc ourse Ġextend Ġthe Ġtechnique Ġto Ġdiscourse Ġparsing . ĠThey Ġpropose Ġa Ġtwo - stage Ġmodel Ġconsisting Ġof Ġan Ġintra - sent ential Ġparser Ġand Ġa Ġmulti - sent ential Ġparser , Ġlearning Ġcontext ually Ġinformed Ġrepresentations Ġof Ġconstituents Ġwith ĠL ST M - Min us , Ġat Ġthe Ġsentence Ġand Ġdocument Ġlevel , Ġrespectively . </s> Similarly , Ġin Ġthis Ġpaper , Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary , Ġthe Ġlocal Ġcontext Ġof Ġthat Ġsentence Ġis Ġcaptured Ġby Ġapplying ĠL ST M - Min us Ġat Ġthe Ġdocument Ġlevel , Ġto Ġrepresent Ġthe Ġsub - sequence Ġof Ġsentences Ġof Ġthe Ġdocument Ġ( i . e ., Ġthe Ġtopic / section ) Ġthe Ġtarget Ġsentence Ġbelongs Ġto . </s> Our ĠModel </s> In Ġthis Ġwork , Ġwe Ġpropose Ġan Ġextract ive Ġmodel Ġfor Ġlong Ġdocuments , Ġincorporating Ġlocal Ġand Ġglobal Ġcontext Ġinformation , Ġmotivated Ġby Ġnatural Ġtopic - oriented Ġstructure Ġof Ġhuman - written Ġlong Ġdocuments . ĠThe Ġarchitecture Ġof Ġour Ġmodel Ġis Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, Ġeach Ġsentence Ġis Ġvisited Ġsequ entially Ġin Ġthe Ġoriginal Ġdocument Ġorder , Ġand Ġa Ġcorresponding Ġconfidence Ġscore Ġis Ġcomputed Ġexpressing Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġextract ive Ġsummary . ĠOur Ġmodel Ġcomprises Ġthree Ġcomponents : Ġthe Ġsentence Ġenc oder , Ġthe Ġdocument Ġenc oder Ġand Ġthe Ġsentence Ġclass ifier . </s> Sent ence ĠEnc oder </s> The Ġgoal Ġof Ġthe Ġsentence Ġenc oder Ġis Ġmapping Ġsequences Ġof Ġword Ġembed d ings Ġto Ġa Ġfixed Ġlength Ġvector Ġ( See Ġbottom Ġcenter Ġof ĠFigure ĠFIG REF 10 Ġ). ĠThere Ġare Ġseveral Ġcommon Ġmethods Ġto Ġembed Ġsentences . ĠFor Ġextract ive Ġsummar ization , ĠR NN Ġwere Ġused Ġin ĠB IB REF 17 Ġ, ĠCNN Ġin ĠB IB REF 2 Ġ, Ġand ĠAverage ĠWord ĠEmb edd ing Ġin ĠB IB REF 18 Ġ. ĠEM N LP 2018 Ġexperiment Ġwith Ġall Ġthe Ġthree Ġmethods Ġand Ġconclude Ġthat ĠWord ĠEmb edd ing ĠA ver aging Ġis Ġas Ġgood Ġor Ġbetter Ġthan Ġeither ĠR NN s Ġor ĠCNN s Ġfor Ġsentence Ġembed ding Ġacross Ġdifferent Ġdomains Ġand Ġsummar izer Ġarchitectures . ĠThus , Ġwe Ġuse Ġthe ĠAverage ĠWord ĠEmb edd ing Ġas Ġour Ġsentence Ġenc oder , Ġby Ġwhich Ġa Ġsentence Ġembed ding Ġis Ġsimply Ġthe Ġaverage Ġof Ġits Ġword Ġembed d ings , Ġi . e . ĠIN LINE FORM 0 </s> Besides , Ġwe Ġalso Ġtried Ġthe Ġpopular Ġpre - trained ĠB ERT Ġsentence Ġembed ding ĠB IB REF 19 Ġ, Ġbut Ġinitial Ġresults Ġwere Ġrather Ġpoor . ĠSo Ġwe Ġdo Ġnot Ġpursue Ġthis Ġpossibility Ġany Ġfurther . </s> Document ĠEnc oder </s> At Ġthe Ġdocument Ġlevel , Ġa Ġbi - direction al Ġrecurrent Ġneural Ġnetwork ĠB IB REF 20 Ġis Ġoften Ġused Ġto Ġencode Ġall Ġthe Ġsentences Ġsequ entially Ġforward Ġand Ġbackward , Ġwith Ġsuch Ġmodel Ġachieving Ġremarkable Ġsuccess Ġin Ġmachine Ġtranslation ĠB IB REF 21 Ġ. ĠAs Ġunits , Ġwe Ġselected Ġg ated Ġrecurrent Ġunits Ġ( GR U ) ĠB IB REF 22 Ġ, Ġin Ġlight Ġof Ġfavorable Ġresults Ġshown Ġin ĠB IB REF 23 Ġ. ĠThe ĠGR U Ġis Ġrepresented Ġwith Ġthe Ġstandard Ġreset , Ġupdate , Ġand Ġnew Ġgates . </s> The Ġoutput Ġof Ġthe Ġbi - direction al ĠGR U Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġcomprises Ġtwo Ġhidden Ġstates , ĠIN LINE FORM 1 Ġas Ġforward Ġand Ġbackward Ġhidden Ġstate , Ġrespectively . </s> A . ĠSent ence Ġrepresentation ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ( A ), Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġsentence Ġrepresentation Ġis Ġthe Ġconc aten ation Ġof Ġboth Ġbackward Ġand Ġforward Ġhidden Ġstate Ġof Ġthat Ġsentence . ĠIN LINE FORM 1 </s> In Ġthis Ġway , Ġthe Ġsentence Ġrepresentation Ġnot Ġonly Ġrepresents Ġthe Ġcurrent Ġsentence , Ġbut Ġalso Ġpartially Ġcovers Ġcontextual Ġinformation Ġboth Ġbefore Ġand Ġafter Ġthis Ġsentence . </s> B . ĠDocument Ġrepresentation ĠThe Ġdocument Ġrepresentation Ġprovides Ġglobal Ġinformation Ġon Ġthe Ġwhole Ġdocument . ĠIt Ġis Ġcomputed Ġas Ġthe Ġconc aten ation Ġof Ġthe Ġfinal Ġstate Ġof Ġthe Ġforward Ġand Ġbackward ĠGR U , Ġlabeled Ġas ĠB Ġin ĠFigure ĠFIG REF 10 Ġ. ĠB IB REF 24 ĠIN LINE FORM 0 </s> C . ĠTopic Ġsegment Ġrepresentation ĠTo Ġcapture Ġthe Ġlocal Ġcontext Ġof Ġeach Ġsentence , Ġnamely Ġthe Ġinformation Ġof Ġthe Ġtopic Ġsegment Ġthat Ġsentence Ġfalls Ġinto , Ġwe Ġapply Ġthe ĠL ST M - Min us Ġmethod , Ġa Ġmethod Ġfor Ġlearning Ġembed d ings Ġof Ġtext Ġspans . ĠL ST M - Min us Ġis Ġshown Ġin Ġdetail Ġin ĠFigure Ġ1 Ġ( left Ġpanel ĠC ), Ġeach Ġtopic Ġsegment Ġis Ġrepresented Ġas Ġthe Ġsubt raction Ġbetween Ġthe Ġhidden Ġstates Ġof Ġthe Ġstart Ġand Ġthe Ġend Ġof Ġthat Ġtopic . ĠAs Ġillustrated Ġin ĠFigure ĠFIG REF 10 Ġ, Ġthe Ġrepresentation Ġfor Ġsection Ġ2 Ġof Ġthe Ġsample Ġdocument Ġ( containing Ġthree Ġsections Ġand Ġeight Ġsentences Ġoverall ) Ġcan Ġbe Ġcomputed Ġas ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġare Ġthe Ġforward Ġhidden Ġstates Ġof Ġsentence Ġ5 Ġand Ġ2 , Ġrespectively , Ġwhile ĠIN LINE FORM 2 Ġare Ġthe Ġbackward Ġhidden Ġstates Ġof Ġsentence Ġ3 Ġand Ġ6 , Ġrespectively . ĠIn Ġgeneral , Ġthe Ġtopic Ġsegment Ġrepresentation ĠIN LINE FORM 3 Ġfor Ġsegment ĠIN LINE FORM 4 Ġis Ġcomputed Ġas : ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġindex Ġof Ġthe Ġbeginning Ġand Ġthe Ġend Ġof Ġtopic ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġdenote Ġthe Ġtopic Ġsegment Ġrepresentation Ġof Ġforward Ġand Ġbackward , Ġrespectively . ĠThe Ġfinal Ġrepresentation Ġof Ġtopic ĠIN LINE FORM 4 Ġis Ġthe Ġconc aten ation Ġof Ġforward Ġand Ġbackward Ġrepresentation ĠIN LINE FORM 5 Ġ. ĠTo Ġobtain ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ, Ġwe Ġutilize Ġsubt raction Ġbetween ĠGR U Ġhidden Ġvectors Ġof ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġand Ġwe Ġpad Ġthe Ġhidden Ġstates Ġwith Ġzero Ġvectors Ġboth Ġin Ġthe Ġbeginning Ġand Ġthe Ġend , Ġto Ġensure Ġthe Ġindex Ġcan Ġnot Ġbe Ġout Ġof Ġbound . ĠThe Ġintuition Ġbehind Ġthis Ġprocess Ġis Ġthat Ġthe ĠGR Us Ġcan Ġkeep Ġprevious Ġuseful Ġinformation Ġin Ġtheir Ġmemory Ġcell Ġby Ġexploiting Ġreset , Ġupdate , Ġand Ġnew Ġgates Ġto Ġdecide Ġhow Ġto Ġutilize Ġand Ġupdate Ġthe Ġmemory Ġof Ġprevious Ġinformation . ĠIn Ġthis Ġway , Ġwe Ġcan Ġrepresent Ġthe Ġcontextual Ġinformation Ġwithin Ġeach Ġtopic Ġsegment Ġfor Ġall Ġthe Ġsentences Ġin Ġthat Ġsegment . </s> Dec oder </s> Once Ġwe Ġhave Ġobtained Ġa Ġrepresentation Ġfor Ġthe Ġsentence , Ġfor Ġits Ġtopic Ġsegment Ġ( i . e ., Ġlocal Ġcontext ) Ġand Ġfor Ġthe Ġdocument Ġ( i . e ., Ġglobal Ġcontext ), Ġthese Ġthree Ġfactors Ġare Ġcombined Ġto Ġmake Ġa Ġfinal Ġprediction ĠIN LINE FORM 0 Ġon Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠWe Ġconsider Ġtwo Ġways Ġin Ġwhich Ġthese Ġthree Ġfactors Ġcan Ġbe Ġcombined . </s> Con cat en ation ĠWe Ġcan Ġsimply Ġconc aten ate Ġthe Ġvectors Ġof Ġthese Ġthree Ġfactors Ġas , ĠIN LINE FORM 0 </s> where Ġsentence ĠIN LINE FORM 0 Ġis Ġpart Ġof Ġthe Ġtopic ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġrepresentation Ġof Ġsentence ĠIN LINE FORM 3 Ġwith Ġtopic Ġsegment Ġinformation Ġand Ġglobal Ġcontext Ġinformation . </s> Att ent ive Ġcontext ĠAs Ġlocal Ġcontext Ġand Ġglobal Ġcontext Ġare Ġall Ġcontextual Ġinformation Ġof Ġthe Ġgiven Ġsentence , Ġwe Ġuse Ġan Ġattention Ġmechanism Ġto Ġdecide Ġthe Ġweight Ġof Ġeach Ġcontext Ġvector , Ġrepresented Ġas ĠIN LINE FORM 0 </s> where Ġthe ĠIN LINE FORM 0 Ġis Ġthe Ġweighted Ġcontext Ġvector Ġof Ġeach Ġsentence ĠIN LINE FORM 1 Ġ, Ġand Ġassume Ġsentence ĠIN LINE FORM 2 Ġis Ġin Ġtopic ĠIN LINE FORM 3 Ġ. </s> Then Ġthere Ġis Ġa Ġfinal Ġmulti - layer Ġpercept ron ( ML P ) Ġfollowed Ġwith Ġa Ġs igm oid Ġactivation Ġfunction Ġindicating Ġthe Ġconfidence Ġscore Ġfor Ġselecting Ġeach Ġsentence : ĠIN LINE FORM 0 </s> Exper iments </s> To Ġvalidate Ġour Ġmethod , Ġwe Ġset Ġup Ġexperiments Ġon Ġthe Ġtwo Ġscientific Ġpaper Ġdatasets Ġ( ar X iv Ġand ĠPubMed ). ĠWith ĠR OU GE Ġand ĠMET E OR Ġscores Ġas Ġautomatic Ġevaluation Ġmetrics , Ġwe Ġcompare Ġwith Ġprevious Ġworks , Ġboth Ġabstract ive Ġand Ġextract ive . </s> Training </s> The Ġweighted Ġnegative Ġlog - like lihood Ġis Ġminimized , Ġwhere Ġthe Ġweight Ġis Ġcomputed Ġas ĠIN LINE FORM 0 Ġ, Ġto Ġsolve Ġthe Ġproblem Ġof Ġhighly Ġim balanced Ġdata Ġ( typ ical Ġin Ġextract ive Ġsummar ization ). ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġrepresent Ġthe Ġground - truth Ġlabel Ġof Ġsentence ĠIN LINE FORM 1 Ġ, Ġwith ĠIN LINE FORM 2 Ġmeaning Ġsentence ĠIN LINE FORM 3 Ġis Ġin Ġthe Ġgold - standard Ġextract Ġsummary . </s> Ext ractive ĠLabel ĠGeneration </s> In Ġthe ĠPub med Ġand Ġar X iv Ġdatasets , Ġthe Ġextract ive Ġsumm aries Ġare Ġmissing . ĠSo Ġwe Ġfollow Ġthe Ġwork Ġof ĠB IB REF 18 Ġon Ġextract ive Ġsummary Ġlabeling , Ġconstructing Ġgold Ġlabel Ġsequences Ġby Ġgreed ily Ġoptimizing ĠR OU GE - 1 Ġon Ġthe Ġgold - standard Ġabstract s , Ġwhich Ġare Ġavailable Ġfor Ġeach Ġarticle . ĠThe Ġalgorithm Ġis Ġshown Ġin ĠAppendix ĠA . </s> Im plementation ĠDetails </s> We Ġtrain Ġour Ġmodel Ġusing Ġthe ĠAdam Ġoptim izer ĠB IB REF 25 Ġwith Ġlearning Ġrate ĠIN LINE FORM 0 Ġand Ġa Ġdrop Ġout Ġrate Ġof Ġ0 . 3 . ĠWe Ġuse Ġa Ġmini - batch Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġdocuments , Ġand Ġthe Ġsize Ġof Ġthe ĠGR U Ġhidden Ġstates Ġis Ġ300 . ĠFor Ġword Ġembed d ings , Ġwe Ġuse ĠGl o Ve ĠB IB REF 26 Ġwith Ġdimension Ġ300 , Ġpre - trained Ġon Ġthe ĠWikipedia Ġand ĠGig aw ord . ĠThe Ġvocabulary Ġsize Ġof Ġour Ġmodel Ġis Ġ5 0000 . ĠAll Ġthe Ġabove Ġparameters Ġwere Ġset Ġbased Ġon ĠB IB REF 18 Ġwithout Ġany Ġfine - tun ing . ĠAgain Ġfollowing ĠB IB REF 18 Ġ, Ġwe Ġtrain Ġeach Ġmodel Ġfor Ġ50 Ġepoch s , Ġand Ġthe Ġbest Ġmodel Ġis Ġselected Ġwith Ġearly Ġstopping Ġon Ġthe Ġvalidation Ġset Ġaccording Ġto ĠRouge - 2 ĠF - score . </s> Mod els Ġfor ĠComparison </s> We Ġperform Ġa Ġsystematic Ġcomparison Ġwith Ġprevious Ġwork Ġin Ġextract ive Ġsummar ization . ĠFor Ġcomple teness , Ġwe Ġalso Ġcompare Ġwith Ġrecent Ġneural Ġabstract ive Ġapproaches . ĠIn Ġall Ġthe Ġexperiments , Ġwe Ġuse Ġthe Ġsame Ġtrain / val / test Ġsplitting . </s> Traditional Ġextract ive Ġsummar ization Ġmodels : ĠSum Basic ĠB IB REF 27 Ġ, ĠL SA ĠB IB REF 28 Ġ, Ġand ĠLex Rank ĠB IB REF 29 </s> Ne ural Ġabstract ive Ġsummar ization Ġmodels : ĠAtt n - Se q 2 Se q ĠB IB REF 1 Ġ, ĠP nt r - Gen - Se q 2 Se q ĠB IB REF 16 Ġand ĠDisc ourse - aware ĠB IB REF 9 </s> Ne ural Ġextract ive Ġsummar ization Ġmodels : ĠCheng & L ap ata ĠB IB REF 2 Ġand ĠSum ma Ru NN er ĠB IB REF 17 Ġ. ĠBased Ġon ĠB IB REF 18 Ġ, Ġwe Ġuse Ġthe ĠAverage ĠWord ĠEnc oder Ġas Ġsentence Ġenc oder Ġfor Ġboth Ġmodels , Ġinstead Ġof Ġthe ĠCNN Ġand ĠR NN Ġsentence Ġenc od ers Ġthat Ġwere Ġoriginally Ġused Ġin Ġthe Ġtwo Ġsystems , Ġrespectively . </s> Bas eline : ĠSimilar Ġto Ġour Ġmodel , Ġbut Ġwithout Ġlocal Ġcontext Ġand Ġglobal Ġcontext , Ġi . e . Ġthe Ġinput Ġto ĠML P Ġis Ġthe Ġsentence Ġrepresentation Ġonly . </s> Lead : ĠGiven Ġa Ġlength Ġlimit Ġof ĠIN LINE FORM 0 Ġwords Ġfor Ġthe Ġsummary , ĠLead Ġwill Ġreturn Ġthe Ġfirst ĠIN LINE FORM 1 Ġwords Ġof Ġthe Ġsource Ġdocument . </s> Oracle : Ġuses Ġthe ĠGold ĠStandard Ġextract ive Ġlabels , Ġgenerated Ġbased Ġon ĠR OU GE Ġ( Sec . Ġ4 . 2 ). </s> Results Ġand ĠAnalysis </s> For Ġevaluation , Ġwe Ġfollow Ġthe Ġsame Ġprocedure Ġas Ġin ĠB IB REF 18 Ġ. ĠSum m aries Ġare Ġgenerated Ġby Ġselecting Ġthe Ġtop Ġranked Ġsentences Ġby Ġmodel Ġprobability ĠIN LINE FORM 0 Ġ, Ġuntil Ġthe Ġlength Ġlimit Ġis Ġmet Ġor Ġexceeded . ĠBased Ġon Ġthe Ġaverage Ġlength Ġof Ġabstract s Ġin Ġthese Ġtwo Ġdatasets , Ġwe Ġset Ġthe Ġlength Ġlimit Ġto Ġ200 Ġwords . ĠWe Ġuse ĠR OU GE Ġscores ĠB IB REF 30 Ġand ĠMET E OR Ġscores ĠB IB REF 31 Ġbetween Ġthe Ġmodel Ġresults Ġand Ġground - truth Ġabstract ive Ġsumm aries Ġas Ġevaluation Ġmetric . ĠThe Ġun ig ram Ġand Ġbig ram Ġoverlap Ġ( R OU GE - 1 , 2 ) Ġare Ġintended Ġto Ġmeasure Ġthe Ġinform at iveness , Ġwhile Ġlongest Ġcommon Ġsubsequ ence Ġ( R OU GE - L ) Ġcaptures Ġflu ency Ġto Ġsome Ġextent ĠB IB REF 2 Ġ. ĠMET E OR Ġwas Ġoriginally Ġproposed Ġto Ġevaluate Ġtranslation Ġsystems Ġby Ġmeasuring Ġthe Ġalignment Ġbetween Ġthe Ġsystem Ġoutput Ġand Ġreference Ġtranslations . ĠAs Ġsuch , Ġit Ġcan Ġalso Ġbe Ġused Ġas Ġan Ġautomatic Ġevaluation Ġmetric Ġfor Ġsummar ization ĠB IB REF 18 Ġ. </s> The Ġperformance Ġof Ġall Ġmodels Ġon Ġar X iv Ġand ĠPub med Ġis Ġshown Ġin ĠTable ĠT AB REF 28 Ġand ĠTable ĠT AB REF 29 Ġ, Ġrespectively . ĠFollow Ġthe Ġwork ĠB IB REF 18 Ġ, Ġwe Ġuse Ġthe Ġapproximate Ġrandom ization Ġas Ġthe Ġstatistical Ġsignificance Ġtest Ġmethod ĠB IB REF 32 Ġwith Ġa ĠBon fer ron i Ġcorrection Ġfor Ġmultiple Ġcomparisons , Ġat Ġthe Ġconfidence Ġlevel Ġ0 . 01 Ġ( ĠIN LINE FORM 0 Ġ). ĠAs Ġwe Ġcan Ġsee Ġin Ġthese Ġtables , Ġon Ġboth Ġdatasets , Ġthe Ġneural Ġextract ive Ġmodels Ġoutper forms Ġthe Ġtraditional Ġextract ive Ġmodels Ġon Ġinform at iveness Ġ( R OU GE - 1 , 2 ) Ġby Ġa Ġwide Ġmargin , Ġbut Ġresults Ġare Ġmixed Ġon ĠR OU GE - L . ĠPresumably , Ġthis Ġis Ġdue Ġto Ġthe Ġneural Ġtraining Ġprocess , Ġwhich Ġrelies Ġon Ġa Ġgoal Ġstandard Ġbased Ġon ĠR OU GE - 1 . ĠExpl oring Ġother Ġtraining Ġschemes Ġand / or Ġa Ġcombination Ġof Ġtraditional Ġand Ġneural Ġapproaches Ġis Ġleft Ġas Ġfuture Ġwork . ĠSimilarly , Ġthe Ġneural Ġextract ive Ġmodels Ġalso Ġdominate Ġthe Ġneural Ġabstract ive Ġmodels Ġon ĠR OU GE - 1 , 2 , Ġbut Ġthese Ġabstract ive Ġmodels Ġtend Ġto Ġhave Ġthe Ġhighest ĠR OU GE - L Ġscores , Ġpossibly Ġbecause Ġthey Ġare Ġtrained Ġdirectly Ġon Ġgold Ġstandard Ġabstract Ġsumm aries . </s> Compared Ġwith Ġother Ġneural Ġextract ive Ġmodels , Ġour Ġmodels Ġ( both Ġwith Ġattentive Ġcontext Ġand Ġconc aten ation Ġdec oder ) Ġhave Ġbetter Ġperformances Ġon Ġall Ġthree ĠR OU GE Ġscores , Ġas Ġwell Ġas ĠMET E OR . ĠIn Ġparticular , Ġthe Ġimprovements Ġover Ġthe ĠBas eline Ġmodel Ġshow Ġthat Ġa Ġcombination Ġof Ġlocal Ġand Ġglobal Ġcontextual Ġinformation Ġdoes Ġhelp Ġto Ġidentify Ġthe Ġmost Ġimportant Ġsentences Ġ( more Ġon Ġthis Ġin Ġthe Ġnext Ġsection ). ĠInterestingly , Ġjust Ġthe ĠBas eline Ġmodel Ġalready Ġachieves Ġa Ġslightly Ġbetter Ġperformance Ġthan Ġprevious Ġworks ; Ġpossibly Ġbecause Ġthe Ġauto - reg ressive Ġapproach Ġused Ġin Ġthose Ġmodels Ġis Ġeven Ġmore Ġdetrimental Ġfor Ġlong Ġdocuments . </s> Figure ĠFIG REF 32 Ġshows Ġthe Ġmost Ġimportant Ġresult Ġof Ġour Ġanalysis : Ġthe Ġbenefits Ġof Ġour Ġmethod , Ġexplicitly Ġdesigned Ġto Ġdeal Ġwith Ġlonger Ġdocuments , Ġdo Ġactually Ġbecome Ġstronger Ġas Ġwe Ġapply Ġit Ġto Ġlonger Ġdocuments . ĠAs Ġit Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 32 Ġ, Ġthe Ġperformance Ġgain Ġof Ġour Ġmodel Ġwith Ġrespect Ġto Ġcurrent Ġstate - of - the - art Ġextract ive Ġsummar izer Ġis Ġmore Ġpronounced Ġfor Ġdocuments Ġwith ĠIN LINE FORM 0 Ġwords Ġin Ġboth Ġdatasets . </s> Finally , Ġthe Ġresult Ġof ĠLead Ġ( Table ĠT AB REF 28 Ġ, ĠT AB REF 29 Ġ) Ġshows Ġthat Ġscientific Ġpapers Ġhave Ġless Ġposition Ġbias Ġthan Ġnews ; Ġi . e ., Ġthe Ġfirst Ġsentences Ġof Ġthese Ġpapers Ġare Ġnot Ġa Ġgood Ġchoice Ġto Ġform Ġan Ġextract ive Ġsummary . </s> As Ġa Ġteaser Ġfor Ġthe Ġpotential Ġand Ġchallenges Ġthat Ġstill Ġface Ġour Ġapproach , Ġits Ġoutput Ġ( i . e ., Ġthe Ġextracted Ġsentences ) Ġwhen Ġapplied Ġto Ġthis Ġpaper Ġis Ġcolored Ġin Ġred Ġand Ġthe Ġorder Ġin Ġwhich Ġthe Ġsentences Ġare Ġextracted Ġis Ġmarked Ġwith Ġthe ĠRoman Ġnumbering . ĠIf Ġwe Ġset Ġthe Ġsummary Ġlength Ġlimit Ġto Ġthe Ġlength Ġof Ġour Ġabstract , Ġthe Ġfirst Ġfive Ġsentences Ġin Ġthe Ġconclusions Ġsection Ġare Ġextracted . ĠIf Ġwe Ġincrease Ġthe Ġlength Ġto Ġ200 Ġwords , Ġtwo Ġmore Ġsentences Ġare Ġextracted , Ġwhich Ġdo Ġseem Ġto Ġprovide Ġuseful Ġcomplementary Ġinformation . ĠNot Ġsurprisingly , Ġsome Ġredundancy Ġis Ġpresent , Ġas Ġdealing Ġexplicitly Ġwith Ġredundancy Ġis Ġnot Ġa Ġgoal Ġof Ġour Ġcurrent Ġproposal Ġand Ġleft Ġas Ġfuture Ġwork . </s> A bl ation ĠStudy </s> In Ġorder Ġto Ġassess Ġthe Ġrelative Ġcontributions Ġof Ġthe Ġglobal Ġand Ġlocal Ġmodels Ġto Ġthe Ġperformance Ġof Ġour Ġapproach , Ġwe Ġran Ġan Ġab lation Ġstudy . ĠThis Ġwas Ġdone Ġfor Ġeach Ġdataset Ġboth Ġwith Ġthe Ġwhole Ġtest Ġset , Ġas Ġwell Ġas Ġwith Ġa Ġsubset Ġof Ġlong Ġdocuments . ĠThe Ġresults Ġfor ĠPub med Ġand Ġar X iv Ġare Ġshown Ġin ĠTable ĠT AB REF 34 Ġand ĠTable ĠT AB REF 35 Ġ, Ġrespectively . ĠFor Ġstatistical Ġsignificance , Ġas Ġit Ġwas Ġdone Ġfor Ġthe Ġgeneral Ġresults Ġin ĠSection Ġ4 . 5 , Ġwe Ġuse Ġthe Ġapproximate Ġrandom ization Ġmethod ĠB IB REF 32 Ġwith Ġthe ĠBon fer ron i Ġcorrection Ġat Ġ( ĠIN LINE FORM 0 Ġ). </s> From Ġthese Ġtables , Ġwe Ġcan Ġsee Ġthat Ġon Ġboth Ġdatasets Ġthe Ġperformance Ġsignificantly Ġimproves Ġwhen Ġlocal Ġtopic Ġinformation Ġ( i . e . Ġlocal Ġcontext ) Ġis Ġadded . ĠAnd Ġthe Ġimprovement Ġis Ġeven Ġgreater Ġwhen Ġwe Ġonly Ġconsider Ġlong Ġdocuments . ĠRather Ġsurprisingly , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe Ġglobal Ġcontext . ĠAdding Ġa Ġrepresentation Ġof Ġthe Ġwhole Ġdocument Ġ( i . e . Ġglobal Ġcontext ) Ġnever Ġsignificantly Ġimproves Ġperformance . ĠIn Ġessence , Ġit Ġseems Ġthat Ġall Ġthe Ġbenefits Ġof Ġour Ġmodel Ġcome Ġexclusively Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext , Ġeven Ġfor Ġthe Ġlongest Ġdocuments . ĠFurther Ġinvestigation Ġof Ġthis Ġfinding Ġis Ġleft Ġas Ġfuture Ġwork . </s> Con clusions Ġand ĠFuture ĠWork </s> pur ple In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġextract ive Ġsummar ization Ġmodel Ġespecially Ġdesigned Ġfor Ġlong Ġdocuments , Ġby Ġincorporating Ġthe Ġlocal Ġcontext Ġwithin Ġeach Ġtopic , Ġalong Ġwith Ġthe Ġglobal Ġcontext Ġof Ġthe Ġwhole Ġdocument .[ 2 ] Ġpurple Our Ġapproach Ġintegrates Ġrecent Ġfindings Ġon Ġneural Ġextract ive Ġsummar ization Ġin Ġa Ġparameter Ġlean Ġand Ġmodular Ġarchitecture .[ 3 ] Ġpurple We Ġevaluate Ġour Ġmodel Ġand Ġcompare Ġwith Ġprevious Ġworks Ġin Ġboth Ġextract ive Ġand Ġabstract ive Ġsummar ization Ġon Ġtwo Ġlarge Ġscientific Ġpaper Ġdatasets , Ġwhich Ġcontain Ġdocuments Ġthat Ġare Ġmuch Ġlonger Ġthan Ġin Ġpreviously Ġused Ġcorpor a .[ 4 ] Ġpurple Our Ġmodel Ġnot Ġonly Ġachieves Ġstate - of - the - art Ġon Ġthese Ġtwo Ġdatasets , Ġbut Ġin Ġan Ġadditional Ġexperiment , Ġin Ġwhich Ġwe Ġconsider Ġdocuments Ġwith Ġincreasing Ġlength , Ġit Ġbecomes Ġmore Ġcompetitive Ġfor Ġlonger Ġdocuments .[ 5 ] Ġpurple We Ġalso Ġran Ġan Ġab lation Ġstudy Ġto Ġassess Ġthe Ġrelative Ġcontribution Ġof Ġthe Ġglobal Ġand Ġlocal Ġcomponents Ġof Ġour Ġapproach . Ġ[ 1 ] ĠRather Ġsurprisingly , Ġit Ġappears Ġthat Ġthe Ġbenefits Ġof Ġour Ġmodel Ġcome Ġonly Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext . </s> For Ġfuture Ġwork , Ġwe Ġinitially Ġintend Ġto Ġinvestigate Ġneural Ġmethods Ġto Ġdeal Ġwith Ġredundancy . ĠThen , Ġit Ġcould Ġbe Ġbeneficial Ġto Ġintegrate Ġexplicit Ġfeatures , Ġlike Ġsentence Ġposition Ġand Ġsal ience , Ġinto Ġour Ġneural Ġapproach . ĠMore Ġgenerally , Ġwe Ġplan Ġto Ġcombine Ġof Ġtraditional Ġand Ġneural Ġmodels , Ġas Ġsuggested Ġby Ġour Ġresults . ĠFurthermore , Ġwe Ġwould Ġlike Ġto Ġexplore Ġmore Ġsoph istical Ġstructure Ġof Ġdocuments , Ġlike Ġdiscourse Ġtree , Ġinstead Ġof Ġrough Ġtopic Ġsegments . ĠAs Ġfor Ġevaluation , Ġwe Ġwould Ġlike Ġto Ġelicit Ġhuman Ġjudgments , Ġfor Ġinstance Ġby Ġinviting Ġauthors Ġto Ġrate Ġthe Ġoutputs Ġfrom Ġdifferent Ġsystems , Ġwhen Ġapplied Ġto Ġtheir Ġown Ġpapers . ĠMore Ġlong Ġterm , Ġwe Ġwill Ġstudy Ġhow Ġextract ive / ab stract ive Ġtechniques Ġcan Ġbe Ġintegrated ; Ġfor Ġinstance , Ġthe Ġoutput Ġof Ġan Ġextract ive Ġsystem Ġcould Ġbe Ġfed Ġinto Ġan Ġabstract ive Ġone , Ġtraining Ġthe Ġtwo Ġjointly . </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġby Ġthe ĠLanguage Ġ& ĠSpeech ĠInnovation ĠLab Ġof ĠCloud ĠBU , ĠHuawei ĠTechnologies ĠCo ., ĠLtd . ĠExtract ive ĠLabel ĠGeneration ĠThe Ġalgorithm ĠSEC REF 6 Ġis Ġused Ġto Ġgenerate Ġthe Ġextract ive Ġlabels Ġbased Ġon Ġthe Ġhuman - made Ġabstract ive Ġsumm aries , Ġi . e . Ġabstract s Ġof Ġscientific Ġpapers . ĠExtract ive Ġlabel Ġgeneration ĠLabel Gener ation Reference , sent ences , length Limit ĠIN LINE FORM 0 Ġ= ĠâĢ Ŀ ĠIN LINE FORM 1 Ġ= Ġ0 ĠIN LINE FORM 2 Ġ= Ġ[] ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġin Ġrange ( len ( ĠIN LINE FORM 7 Ġ)) ĠIN LINE FORM 8 ĠIN LINE FORM 9 ĠIN LINE FORM 10 ĠIN LINE FORM 11 ĠIN LINE FORM 12 Ġ!= ĠIN LINE FORM 13 ĠIN LINE FORM 14 Ġ. append ( ĠIN LINE FORM 15 Ġ) ĠIN LINE FORM 16 Ġ= ĠIN LINE FORM 17 Ġ+ ĠIN LINE FORM 18 Ġ[ ĠIN LINE FORM 19 Ġ] ĠIN LINE FORM 20 Ġ+= ĠNumber Of Words ( ĠIN LINE FORM 21 Ġ[ ĠIN LINE FORM 22 Ġ]) Ġbreak ĠIN LINE FORM 23
Introduction Single-document summarization is the task of generating a short summary for a given document. Ideally, the generated summaries should be fluent and coherent, and should faithfully maintain the most important information in the source document. purpleThis is a very challenging task, because it arguably requires an in-depth understanding of the source document, and current automatic solutions are still far from human performance BIBREF0 . Single-document summarization can be either extractive or abstractive. Extractive methods typically pick sentences directly from the original document based on their importance, and form the summary as an aggregate of these sentences. Usually, summaries generated in this way have a better performance on fluency and grammar, but they may contain much redundancy and lack in coherence across sentences. In contrast, abstractive methods attempt to mimic what humans do by first extracting content from the source document and then produce new sentences that aggregate and organize the extracted information. Since the sentences are generated from scratch they tend to have a relatively worse performance on fluency and grammar. Furthermore, while abstractive summaries are typically less redundant, they may end up including misleading or even utterly false statements, because the methods to extract and aggregate information form the source document are still rather noisy. In this work, we focus on extracting informative sentences from a given document (without dealing with redundancy), especially when the document is relatively long (e.g., scientific articles). Most recent works on neural extractive summarization have been rather successful in generating summaries of short news documents (around 650 words/document) BIBREF1 by applying neural Seq2Seq models BIBREF2 . However when it comes to long documents, these models tend to struggle with longer sequences because at each decoding step, the decoder needs to learn to construct a context vector capturing relevant information from all the tokens in the source sequence BIBREF3 . Long documents typically cover multiple topics. In general, the longer a document is, the more topics are discussed. As a matter of fact, when humans write long documents they organize them in chapters, sections etc.. Scientific papers are an example of longer documents and they follow a standard discourse structure describing the problem, methodology, experiments/results, and finally conclusions BIBREF4 . To the best of our knowledge only one previous work in extractive summarization has explicitly leveraged section information to guide the generation of summaries BIBREF5 . However, the only information about sections fed into their sentence classifier is a categorical feature with values like Highlight, Abstract, Introduction, etc., depending on which section the sentence appears in. In contrast, in order to exploit section information, in this paper we propose to capture a distributed representation of both the global (the whole document) and the local context (e.g., the section/topic) when deciding if a sentence should be included in the summary Our main contributions are as follows: (i) In order to capture the local context, we are the first to apply LSTM-minus to text summarization. LSTM-minus is a method for learning embeddings of text spans, which has achieved good performance in dependency parsing BIBREF6 , in constituency parsing BIBREF7 , as well as in discourse parsing BIBREF8 . With respect to more traditional methods for capturing local context, which rely on hierarchical structures, LSTM-minus produces simpler models i.e. with less parameters, and therefore faster to train and less prone to overfitting. (ii) We test our method on the Pubmed and arXiv datasets and results appear to support our goal of effectively summarizing long documents. In particular, while overall we outperform the baseline and previous approaches only by a narrow margin on both datasets, the benefit of our method become much stronger as we apply it to longer documents. purpleFurthermore, in an ablation study to assess the relative contributions of the global and the local model we found that, rather surprisingly, the benefits of our model seem to come exclusively from modeling the local context, even for the longest documents.[6] (iii) In order to evaluate our approach, we have created oracle labels for both Pubmed and arXiv BIBREF9 , by applying a greedy oracle labeling algorithm. The two datasets annotated with extractive labels will be made public. Extractive summarization Traditional extractive summarization methods are mostly based on explicit surface features BIBREF10 , relying on graph-based methods BIBREF11 , or on submodular maximization BIBREF12 . Benefiting from the success of neural sequence models in other NLP tasks, chenglapata propose a novel approach to extractive summarization based on neural networks and continuous sentence features, which outperforms traditional methods on the DailyMail dataset. In particular, they develop a general encoder-decoder architecture, where a CNN is used as sentence encoder, a uni-directional LSTM as document encoder, with another uni-directional LSTM as decoder. To decrease the number of parameters while maintaining the accuracy, summarunner present SummaRuNNer, a simple RNN-based sequence classifier without decoder, outperforming or matching the model of BIBREF2 . They take content, salience, novelty, and position of each sentence into consideration when deciding if a sentence should be included in the extractive summary. Yet, they do not capture any aspect of the topical structure, as we do in this paper. So their approach would arguably suffer when applied to long documents, likely containing multiple and diverse topics. While SummaRuNNer was tested only on news, EMNLP2018 carry out a comprehensive set of experiments with deep learning models of extractive summarization across different domains, i.e. news, personal stories, meetings, and medical articles, as well as across different neural architectures, in order to better understand the general pros and cons of different design choices. They find that non auto-regressive sentence extraction performs as well or better than auto-regressive extraction in all domains, where by auto-regressive sentence extraction they mean using previous predictions to inform future predictions. Furthermore, they find that the Average Word Embedding sentence encoder works at least as well as encoders based on CNN and RNN. In light of these findings, our model is not auto-regressive and uses the Average Word Embedding encoder. Extractive summarization on Scientific papers Research on summarizing scientific articles has a long history BIBREF13 . Earlier on, it was realized that summarizing scientific papers requires different approaches than what was used for summarizing news articles, due to differences in document length, writing style and rhetorical structure. For instance, BIBREF14 presented a supervised Naive Bayes classifier to select content from a scientific paper based on the rhetorical status of each sentence (e.g., whether it specified a research goal, or some generally accepted scientific background knowledge, etc.). More recently, researchers have extended this work by applying more sophisticated classifiers to identify more fine-grain rhetorical categories, as well as by exploiting citation contexts. 2013-discourse propose the CoreSC discourse-driven content, which relies on CRFs and SVMs, to classify the discourse categories (e.g. Background, Hypothesis, Motivation, etc.) at the sentence level. The recent work most similar to ours is BIBREF5 where, in order to determine whether a sentence should be included in the summary, they directly use the section each sentence appears in as a categorical feature with values like Highlight, Abstract, Introduction, etc.. In this paper, instead of using sections as categorical features, we rely on a distributed representation of the semantic information within each section, as the local context of each sentence. In a very different line of work, cohan-2015-scientific form the summary by also exploiting information on how the target paper is cited in other papers. Currently, we do not use any information from citation contexts. Datasets for long documents summarydataset provide a comprehensive overview of the current datasets for summarization. Noticeably, most of the larger-scale summarization datasets consists of relatively short documents, like CNN/DailyMail BIBREF1 and New York Times BIBREF15 . One exception is BIBREF9 that recently introduce two large-scale datasets of long and structured scientific papers obtained from arXiv and PubMed. These two new datasets contain much longer documents than all the news datasets (See Table TABREF6 ) and are therefore ideal test-beds for the method we present in this paper. Neural Abstractive summarization on long documents While most current neural abstractive summarization models have focused on summarizing relatively short news articles (e.g., BIBREF16 ), few researchers have started to investigate the summarization of longer documents by exploiting their natural structure. agents present an encoder-decoder architecture to address the challenges of representing a long document for abstractive summarization. The encoding task is divided across several collaborating agents, each is responsible for a subsection of text through a multi-layer LSTM with word attention. Their model seems however overly complicated when it comes to the extractive summarization task, where word attention is arguably much less critical. So, we do not consider this model further in this paper. discourselongdocument also propose a model for abstractive summarization taking the structure of documents into consideration with a hierarchical approach, and test it on longer documents with section information, i.e. scientific papers. In particular, they apply a hierarchical encoder at the word and section levels. Then, in the decoding step, they combine the word attention and section attention to obtain a context vector. This approach to capture discourse structure is however quite limited both in general and especially when you consider its application to extractive summarization. First, their hierarchical method has a large number of parameters and it is therefore slow to train and likely prone to overfitting. Secondly, it does not take the global context of the whole document into account, which may arguably be critical in extractive methods, when deciding on the salience of a sentence (or even a word). The extractive summarizer we present in this paper tries to address these limitations by adopting the parameter lean LSTM-minus method, and by explicitly modeling the global context. LSTM-Minus The LSTM-Minus method is first proposed in BIBREF6 as a novel way to learn sentence segment embeddings for graph-based dependency parsing, i.e. estimating the most likely dependency tree given an input sentence. For each dependency pair, they divide a sentence into three segments (prefix, infix and suffix), and LSTM-Minus is used to represent each segment. They apply a single LSTM to the whole sentence and use the difference between two hidden states INLINEFORM0 to represent the segment from word INLINEFORM1 to word INLINEFORM2 . This enables their model to learn segment embeddings from information both outside and inside the segments and thus enhancing their model ability to access to sentence-level information. The intuition behind the method is that each hidden vector INLINEFORM3 can capture useful information before and including the word INLINEFORM4 . Shortly after, lstm-minusconstituency use the same method on the task of constituency parsing, as the representation of a sentence span, extending the original uni-directional LSTM-Minus to the bi-directional case. More recently, inspired by the success of LSTM-Minus in both dependency and constituency parsing, lstm-minusdiscourse extend the technique to discourse parsing. They propose a two-stage model consisting of an intra-sentential parser and a multi-sentential parser, learning contextually informed representations of constituents with LSTM-Minus, at the sentence and document level, respectively. Similarly, in this paper, when deciding if a sentence should be included in the summary, the local context of that sentence is captured by applying LSTM-Minus at the document level, to represent the sub-sequence of sentences of the document (i.e., the topic/section) the target sentence belongs to. Our Model In this work, we propose an extractive model for long documents, incorporating local and global context information, motivated by natural topic-oriented structure of human-written long documents. The architecture of our model is shown in Figure FIGREF10 , each sentence is visited sequentially in the original document order, and a corresponding confidence score is computed expressing whether the sentence should be included in the extractive summary. Our model comprises three components: the sentence encoder, the document encoder and the sentence classifier. Sentence Encoder The goal of the sentence encoder is mapping sequences of word embeddings to a fixed length vector (See bottom center of Figure FIGREF10 ). There are several common methods to embed sentences. For extractive summarization, RNN were used in BIBREF17 , CNN in BIBREF2 , and Average Word Embedding in BIBREF18 . EMNLP2018 experiment with all the three methods and conclude that Word Embedding Averaging is as good or better than either RNNs or CNNs for sentence embedding across different domains and summarizer architectures. Thus, we use the Average Word Embedding as our sentence encoder, by which a sentence embedding is simply the average of its word embeddings, i.e. INLINEFORM0 Besides, we also tried the popular pre-trained BERT sentence embedding BIBREF19 , but initial results were rather poor. So we do not pursue this possibility any further. Document Encoder At the document level, a bi-directional recurrent neural network BIBREF20 is often used to encode all the sentences sequentially forward and backward, with such model achieving remarkable success in machine translation BIBREF21 . As units, we selected gated recurrent units (GRU) BIBREF22 , in light of favorable results shown in BIBREF23 . The GRU is represented with the standard reset, update, and new gates. The output of the bi-directional GRU for each sentence INLINEFORM0 comprises two hidden states, INLINEFORM1 as forward and backward hidden state, respectively. A. Sentence representation As shown in Figure FIGREF10 (A), for each sentence INLINEFORM0 , the sentence representation is the concatenation of both backward and forward hidden state of that sentence. INLINEFORM1 In this way, the sentence representation not only represents the current sentence, but also partially covers contextual information both before and after this sentence. B. Document representation The document representation provides global information on the whole document. It is computed as the concatenation of the final state of the forward and backward GRU, labeled as B in Figure FIGREF10 . BIBREF24 INLINEFORM0 C. Topic segment representation To capture the local context of each sentence, namely the information of the topic segment that sentence falls into, we apply the LSTM-Minus method, a method for learning embeddings of text spans. LSTM-Minus is shown in detail in Figure 1 (left panel C), each topic segment is represented as the subtraction between the hidden states of the start and the end of that topic. As illustrated in Figure FIGREF10 , the representation for section 2 of the sample document (containing three sections and eight sentences overall) can be computed as INLINEFORM0 , where INLINEFORM1 are the forward hidden states of sentence 5 and 2, respectively, while INLINEFORM2 are the backward hidden states of sentence 3 and 6, respectively. In general, the topic segment representation INLINEFORM3 for segment INLINEFORM4 is computed as: INLINEFORM5 where INLINEFORM0 is the index of the beginning and the end of topic INLINEFORM1 , INLINEFORM2 and INLINEFORM3 denote the topic segment representation of forward and backward, respectively. The final representation of topic INLINEFORM4 is the concatenation of forward and backward representation INLINEFORM5 . To obtain INLINEFORM6 and INLINEFORM7 , we utilize subtraction between GRU hidden vectors of INLINEFORM8 and INLINEFORM9 , and we pad the hidden states with zero vectors both in the beginning and the end, to ensure the index can not be out of bound. The intuition behind this process is that the GRUs can keep previous useful information in their memory cell by exploiting reset, update, and new gates to decide how to utilize and update the memory of previous information. In this way, we can represent the contextual information within each topic segment for all the sentences in that segment. Decoder Once we have obtained a representation for the sentence, for its topic segment (i.e., local context) and for the document (i.e., global context), these three factors are combined to make a final prediction INLINEFORM0 on whether the sentence should be included in the summary. We consider two ways in which these three factors can be combined. Concatenation We can simply concatenate the vectors of these three factors as, INLINEFORM0 where sentence INLINEFORM0 is part of the topic INLINEFORM1 , and INLINEFORM2 is the representation of sentence INLINEFORM3 with topic segment information and global context information. Attentive context As local context and global context are all contextual information of the given sentence, we use an attention mechanism to decide the weight of each context vector, represented as INLINEFORM0 where the INLINEFORM0 is the weighted context vector of each sentence INLINEFORM1 , and assume sentence INLINEFORM2 is in topic INLINEFORM3 . Then there is a final multi-layer perceptron(MLP) followed with a sigmoid activation function indicating the confidence score for selecting each sentence: INLINEFORM0 Experiments To validate our method, we set up experiments on the two scientific paper datasets (arXiv and PubMed). With ROUGE and METEOR scores as automatic evaluation metrics, we compare with previous works, both abstractive and extractive. Training The weighted negative log-likelihood is minimized, where the weight is computed as INLINEFORM0 , to solve the problem of highly imbalanced data (typical in extractive summarization). INLINEFORM1 where INLINEFORM0 represent the ground-truth label of sentence INLINEFORM1 , with INLINEFORM2 meaning sentence INLINEFORM3 is in the gold-standard extract summary. Extractive Label Generation In the Pubmed and arXiv datasets, the extractive summaries are missing. So we follow the work of BIBREF18 on extractive summary labeling, constructing gold label sequences by greedily optimizing ROUGE-1 on the gold-standard abstracts, which are available for each article. The algorithm is shown in Appendix A. Implementation Details We train our model using the Adam optimizer BIBREF25 with learning rate INLINEFORM0 and a drop out rate of 0.3. We use a mini-batch with a batch size of 32 documents, and the size of the GRU hidden states is 300. For word embeddings, we use GloVe BIBREF26 with dimension 300, pre-trained on the Wikipedia and Gigaword. The vocabulary size of our model is 50000. All the above parameters were set based on BIBREF18 without any fine-tuning. Again following BIBREF18 , we train each model for 50 epochs, and the best model is selected with early stopping on the validation set according to Rouge-2 F-score. Models for Comparison We perform a systematic comparison with previous work in extractive summarization. For completeness, we also compare with recent neural abstractive approaches. In all the experiments, we use the same train/val/test splitting. Traditional extractive summarization models: SumBasic BIBREF27 , LSA BIBREF28 , and LexRank BIBREF29 Neural abstractive summarization models: Attn-Seq2Seq BIBREF1 , Pntr-Gen-Seq2Seq BIBREF16 and Discourse-aware BIBREF9 Neural extractive summarization models: Cheng&Lapata BIBREF2 and SummaRuNNer BIBREF17 . Based on BIBREF18 , we use the Average Word Encoder as sentence encoder for both models, instead of the CNN and RNN sentence encoders that were originally used in the two systems, respectively. Baseline: Similar to our model, but without local context and global context, i.e. the input to MLP is the sentence representation only. Lead: Given a length limit of INLINEFORM0 words for the summary, Lead will return the first INLINEFORM1 words of the source document. Oracle: uses the Gold Standard extractive labels, generated based on ROUGE (Sec. 4.2). Results and Analysis For evaluation, we follow the same procedure as in BIBREF18 . Summaries are generated by selecting the top ranked sentences by model probability INLINEFORM0 , until the length limit is met or exceeded. Based on the average length of abstracts in these two datasets, we set the length limit to 200 words. We use ROUGE scores BIBREF30 and METEOR scores BIBREF31 between the model results and ground-truth abstractive summaries as evaluation metric. The unigram and bigram overlap (ROUGE-1,2) are intended to measure the informativeness, while longest common subsequence (ROUGE-L) captures fluency to some extent BIBREF2 . METEOR was originally proposed to evaluate translation systems by measuring the alignment between the system output and reference translations. As such, it can also be used as an automatic evaluation metric for summarization BIBREF18 . The performance of all models on arXiv and Pubmed is shown in Table TABREF28 and Table TABREF29 , respectively. Follow the work BIBREF18 , we use the approximate randomization as the statistical significance test method BIBREF32 with a Bonferroni correction for multiple comparisons, at the confidence level 0.01 ( INLINEFORM0 ). As we can see in these tables, on both datasets, the neural extractive models outperforms the traditional extractive models on informativeness (ROUGE-1,2) by a wide margin, but results are mixed on ROUGE-L. Presumably, this is due to the neural training process, which relies on a goal standard based on ROUGE-1. Exploring other training schemes and/or a combination of traditional and neural approaches is left as future work. Similarly, the neural extractive models also dominate the neural abstractive models on ROUGE-1,2, but these abstractive models tend to have the highest ROUGE-L scores, possibly because they are trained directly on gold standard abstract summaries. Compared with other neural extractive models, our models (both with attentive context and concatenation decoder) have better performances on all three ROUGE scores, as well as METEOR. In particular, the improvements over the Baseline model show that a combination of local and global contextual information does help to identify the most important sentences (more on this in the next section). Interestingly, just the Baseline model already achieves a slightly better performance than previous works; possibly because the auto-regressive approach used in those models is even more detrimental for long documents. Figure FIGREF32 shows the most important result of our analysis: the benefits of our method, explicitly designed to deal with longer documents, do actually become stronger as we apply it to longer documents. As it can be seen in Figure FIGREF32 , the performance gain of our model with respect to current state-of-the-art extractive summarizer is more pronounced for documents with INLINEFORM0 words in both datasets. Finally, the result of Lead (Table TABREF28 , TABREF29 ) shows that scientific papers have less position bias than news; i.e., the first sentences of these papers are not a good choice to form an extractive summary. As a teaser for the potential and challenges that still face our approach, its output (i.e., the extracted sentences) when applied to this paper is colored in red and the order in which the sentences are extracted is marked with the Roman numbering. If we set the summary length limit to the length of our abstract, the first five sentences in the conclusions section are extracted. If we increase the length to 200 words, two more sentences are extracted, which do seem to provide useful complementary information. Not surprisingly, some redundancy is present, as dealing explicitly with redundancy is not a goal of our current proposal and left as future work. Ablation Study In order to assess the relative contributions of the global and local models to the performance of our approach, we ran an ablation study. This was done for each dataset both with the whole test set, as well as with a subset of long documents. The results for Pubmed and arXiv are shown in Table TABREF34 and Table TABREF35 , respectively. For statistical significance, as it was done for the general results in Section 4.5, we use the approximate randomization method BIBREF32 with the Bonferroni correction at ( INLINEFORM0 ). From these tables, we can see that on both datasets the performance significantly improves when local topic information (i.e. local context) is added. And the improvement is even greater when we only consider long documents. Rather surprisingly, this is not the case for the global context. Adding a representation of the whole document (i.e. global context) never significantly improves performance. In essence, it seems that all the benefits of our model come exclusively from modeling the local context, even for the longest documents. Further investigation of this finding is left as future work. Conclusions and Future Work purpleIn this paper, we propose a novel extractive summarization model especially designed for long documents, by incorporating the local context within each topic, along with the global context of the whole document.[2] purpleOur approach integrates recent findings on neural extractive summarization in a parameter lean and modular architecture.[3] purpleWe evaluate our model and compare with previous works in both extractive and abstractive summarization on two large scientific paper datasets, which contain documents that are much longer than in previously used corpora.[4] purpleOur model not only achieves state-of-the-art on these two datasets, but in an additional experiment, in which we consider documents with increasing length, it becomes more competitive for longer documents.[5] purpleWe also ran an ablation study to assess the relative contribution of the global and local components of our approach. [1] Rather surprisingly, it appears that the benefits of our model come only from modeling the local context. For future work, we initially intend to investigate neural methods to deal with redundancy. Then, it could be beneficial to integrate explicit features, like sentence position and salience, into our neural approach. More generally, we plan to combine of traditional and neural models, as suggested by our results. Furthermore, we would like to explore more sophistical structure of documents, like discourse tree, instead of rough topic segments. As for evaluation, we would like to elicit human judgments, for instance by inviting authors to rate the outputs from different systems, when applied to their own papers. More long term, we will study how extractive/abstractive techniques can be integrated; for instance, the output of an extractive system could be fed into an abstractive one, training the two jointly. Acknowledgments This research was supported by the Language & Speech Innovation Lab of Cloud BU, Huawei Technologies Co., Ltd. Extractive Label Generation The algorithm SECREF6 is used to generate the extractive labels based on the human-made abstractive summaries, i.e. abstracts of scientific papers. Extractive label generation LabelGenerationReference,sentences,lengthLimit INLINEFORM0 = ” INLINEFORM1 = 0 INLINEFORM2 = [] INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 in range(len( INLINEFORM7 )) INLINEFORM8 INLINEFORM9 INLINEFORM10 INLINEFORM11 INLINEFORM12 != INLINEFORM13 INLINEFORM14 .append( INLINEFORM15 ) INLINEFORM16 = INLINEFORM17 + INLINEFORM18 [ INLINEFORM19 ] INLINEFORM20 += NumberOfWords( INLINEFORM21 [ INLINEFORM22 ]) break INLINEFORM23
What do they mean by global and local context?
b66c9a4021b6c8529cac1a2b54dacd8ec79afa5f
What Ġdo Ġthey Ġmean Ġby Ġglobal Ġand Ġlocal Ġcontext ?
global Ġ( the Ġwhole Ġdocument ), Ġlocal Ġcontext Ġ( e . g ., Ġthe Ġsection / topic )
<s> What Ġdo Ġthey Ġmean Ġby Ġglobal Ġand Ġlocal Ġcontext ? </s> Introduction </s> Single - document Ġsummar ization Ġis Ġthe Ġtask Ġof Ġgenerating Ġa Ġshort Ġsummary Ġfor Ġa Ġgiven Ġdocument . ĠIdeally , Ġthe Ġgenerated Ġsumm aries Ġshould Ġbe Ġfluent Ġand Ġcoherent , Ġand Ġshould Ġfaithfully Ġmaintain Ġthe Ġmost Ġimportant Ġinformation Ġin Ġthe Ġsource Ġdocument . Ġpurple This Ġis Ġa Ġvery Ġchallenging Ġtask , Ġbecause Ġit Ġarguably Ġrequires Ġan Ġin - depth Ġunderstanding Ġof Ġthe Ġsource Ġdocument , Ġand Ġcurrent Ġautomatic Ġsolutions Ġare Ġstill Ġfar Ġfrom Ġhuman Ġperformance ĠB IB REF 0 Ġ. </s> Single - document Ġsummar ization Ġcan Ġbe Ġeither Ġextract ive Ġor Ġabstract ive . ĠExtract ive Ġmethods Ġtypically Ġpick Ġsentences Ġdirectly Ġfrom Ġthe Ġoriginal Ġdocument Ġbased Ġon Ġtheir Ġimportance , Ġand Ġform Ġthe Ġsummary Ġas Ġan Ġaggregate Ġof Ġthese Ġsentences . ĠUsually , Ġsumm aries Ġgenerated Ġin Ġthis Ġway Ġhave Ġa Ġbetter Ġperformance Ġon Ġflu ency Ġand Ġgrammar , Ġbut Ġthey Ġmay Ġcontain Ġmuch Ġredundancy Ġand Ġlack Ġin Ġco herence Ġacross Ġsentences . ĠIn Ġcontrast , Ġabstract ive Ġmethods Ġattempt Ġto Ġmimic Ġwhat Ġhumans Ġdo Ġby Ġfirst Ġextracting Ġcontent Ġfrom Ġthe Ġsource Ġdocument Ġand Ġthen Ġproduce Ġnew Ġsentences Ġthat Ġaggregate Ġand Ġorganize Ġthe Ġextracted Ġinformation . ĠSince Ġthe Ġsentences Ġare Ġgenerated Ġfrom Ġscratch Ġthey Ġtend Ġto Ġhave Ġa Ġrelatively Ġworse Ġperformance Ġon Ġflu ency Ġand Ġgrammar . ĠFurthermore , Ġwhile Ġabstract ive Ġsumm aries Ġare Ġtypically Ġless Ġredundant , Ġthey Ġmay Ġend Ġup Ġincluding Ġmisleading Ġor Ġeven Ġutterly Ġfalse Ġstatements , Ġbecause Ġthe Ġmethods Ġto Ġextract Ġand Ġaggregate Ġinformation Ġform Ġthe Ġsource Ġdocument Ġare Ġstill Ġrather Ġnoisy . </s> In Ġthis Ġwork , Ġwe Ġfocus Ġon Ġextracting Ġinformative Ġsentences Ġfrom Ġa Ġgiven Ġdocument Ġ( without Ġdealing Ġwith Ġredundancy ), Ġespecially Ġwhen Ġthe Ġdocument Ġis Ġrelatively Ġlong Ġ( e . g ., Ġscientific Ġarticles ). </s> Most Ġrecent Ġworks Ġon Ġneural Ġextract ive Ġsummar ization Ġhave Ġbeen Ġrather Ġsuccessful Ġin Ġgenerating Ġsumm aries Ġof Ġshort Ġnews Ġdocuments Ġ( around Ġ650 Ġwords / document ) ĠB IB REF 1 Ġby Ġapplying Ġneural ĠSe q 2 Se q Ġmodels ĠB IB REF 2 Ġ. ĠHowever Ġwhen Ġit Ġcomes Ġto Ġlong Ġdocuments , Ġthese Ġmodels Ġtend Ġto Ġstruggle Ġwith Ġlonger Ġsequences Ġbecause Ġat Ġeach Ġdecoding Ġstep , Ġthe Ġdec oder Ġneeds Ġto Ġlearn Ġto Ġconstruct Ġa Ġcontext Ġvector Ġcapturing Ġrelevant Ġinformation Ġfrom Ġall Ġthe Ġtokens Ġin Ġthe Ġsource Ġsequence ĠB IB REF 3 Ġ. </s> Long Ġdocuments Ġtypically Ġcover Ġmultiple Ġtopics . ĠIn Ġgeneral , Ġthe Ġlonger Ġa Ġdocument Ġis , Ġthe Ġmore Ġtopics Ġare Ġdiscussed . ĠAs Ġa Ġmatter Ġof Ġfact , Ġwhen Ġhumans Ġwrite Ġlong Ġdocuments Ġthey Ġorganize Ġthem Ġin Ġchapters , Ġsections Ġetc .. ĠScientific Ġpapers Ġare Ġan Ġexample Ġof Ġlonger Ġdocuments Ġand Ġthey Ġfollow Ġa Ġstandard Ġdiscourse Ġstructure Ġdescribing Ġthe Ġproblem , Ġmethodology , Ġexperiments / results , Ġand Ġfinally Ġconclusions ĠB IB REF 4 Ġ. </s> To Ġthe Ġbest Ġof Ġour Ġknowledge Ġonly Ġone Ġprevious Ġwork Ġin Ġextract ive Ġsummar ization Ġhas Ġexplicitly Ġlever aged Ġsection Ġinformation Ġto Ġguide Ġthe Ġgeneration Ġof Ġsumm aries ĠB IB REF 5 Ġ. ĠHowever , Ġthe Ġonly Ġinformation Ġabout Ġsections Ġfed Ġinto Ġtheir Ġsentence Ġclass ifier Ġis Ġa Ġcateg orical Ġfeature Ġwith Ġvalues Ġlike ĠHigh light , ĠAbstract , ĠIntroduction , Ġetc ., Ġdepending Ġon Ġwhich Ġsection Ġthe Ġsentence Ġappears Ġin . </s> In Ġcontrast , Ġin Ġorder Ġto Ġexploit Ġsection Ġinformation , Ġin Ġthis Ġpaper Ġwe Ġpropose Ġto Ġcapture Ġa Ġdistributed Ġrepresentation Ġof Ġboth Ġthe Ġglobal Ġ( the Ġwhole Ġdocument ) Ġand Ġthe Ġlocal Ġcontext Ġ( e . g ., Ġthe Ġsection / topic ) Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary </s> Our Ġmain Ġcontributions Ġare Ġas Ġfollows : Ġ( i ) ĠIn Ġorder Ġto Ġcapture Ġthe Ġlocal Ġcontext , Ġwe Ġare Ġthe Ġfirst Ġto Ġapply ĠL ST M - minus Ġto Ġtext Ġsummar ization . ĠL ST M - minus Ġis Ġa Ġmethod Ġfor Ġlearning Ġembed d ings Ġof Ġtext Ġspans , Ġwhich Ġhas Ġachieved Ġgood Ġperformance Ġin Ġdependency Ġparsing ĠB IB REF 6 Ġ, Ġin Ġconstituency Ġparsing ĠB IB REF 7 Ġ, Ġas Ġwell Ġas Ġin Ġdiscourse Ġparsing ĠB IB REF 8 Ġ. ĠWith Ġrespect Ġto Ġmore Ġtraditional Ġmethods Ġfor Ġcapturing Ġlocal Ġcontext , Ġwhich Ġrely Ġon Ġhierarchical Ġstructures , ĠL ST M - minus Ġproduces Ġsimpler Ġmodels Ġi . e . Ġwith Ġless Ġparameters , Ġand Ġtherefore Ġfaster Ġto Ġtrain Ġand Ġless Ġprone Ġto Ġover fitting . Ġ( ii ) ĠWe Ġtest Ġour Ġmethod Ġon Ġthe ĠPub med Ġand Ġar X iv Ġdatasets Ġand Ġresults Ġappear Ġto Ġsupport Ġour Ġgoal Ġof Ġeffectively Ġsummar izing Ġlong Ġdocuments . ĠIn Ġparticular , Ġwhile Ġoverall Ġwe Ġoutper form Ġthe Ġbaseline Ġand Ġprevious Ġapproaches Ġonly Ġby Ġa Ġnarrow Ġmargin Ġon Ġboth Ġdatasets , Ġthe Ġbenefit Ġof Ġour Ġmethod Ġbecome Ġmuch Ġstronger Ġas Ġwe Ġapply Ġit Ġto Ġlonger Ġdocuments . Ġpurple Furthermore , Ġin Ġan Ġab lation Ġstudy Ġto Ġassess Ġthe Ġrelative Ġcontributions Ġof Ġthe Ġglobal Ġand Ġthe Ġlocal Ġmodel Ġwe Ġfound Ġthat , Ġrather Ġsurprisingly , Ġthe Ġbenefits Ġof Ġour Ġmodel Ġseem Ġto Ġcome Ġexclusively Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext , Ġeven Ġfor Ġthe Ġlongest Ġdocuments .[ 6 ] Ġ( iii ) ĠIn Ġorder Ġto Ġevaluate Ġour Ġapproach , Ġwe Ġhave Ġcreated Ġor acle Ġlabels Ġfor Ġboth ĠPub med Ġand Ġar X iv ĠB IB REF 9 Ġ, Ġby Ġapplying Ġa Ġgreedy Ġor acle Ġlabeling Ġalgorithm . ĠThe Ġtwo Ġdatasets Ġannot ated Ġwith Ġextract ive Ġlabels Ġwill Ġbe Ġmade Ġpublic . </s> Ext ractive Ġsummar ization </s> Traditional Ġextract ive Ġsummar ization Ġmethods Ġare Ġmostly Ġbased Ġon Ġexplicit Ġsurface Ġfeatures ĠB IB REF 10 Ġ, Ġrelying Ġon Ġgraph - based Ġmethods ĠB IB REF 11 Ġ, Ġor Ġon Ġsub mod ular Ġmaxim ization ĠB IB REF 12 Ġ. ĠBenef iting Ġfrom Ġthe Ġsuccess Ġof Ġneural Ġsequence Ġmodels Ġin Ġother ĠN LP Ġtasks , Ġche ng lap ata Ġpropose Ġa Ġnovel Ġapproach Ġto Ġextract ive Ġsummar ization Ġbased Ġon Ġneural Ġnetworks Ġand Ġcontinuous Ġsentence Ġfeatures , Ġwhich Ġoutper forms Ġtraditional Ġmethods Ġon Ġthe ĠDaily Mail Ġdataset . ĠIn Ġparticular , Ġthey Ġdevelop Ġa Ġgeneral Ġenc oder - dec oder Ġarchitecture , Ġwhere Ġa ĠCNN Ġis Ġused Ġas Ġsentence Ġenc oder , Ġa Ġun i - direction al ĠL ST M Ġas Ġdocument Ġenc oder , Ġwith Ġanother Ġun i - direction al ĠL ST M Ġas Ġdec oder . ĠTo Ġdecrease Ġthe Ġnumber Ġof Ġparameters Ġwhile Ġmaintaining Ġthe Ġaccuracy , Ġsummar un ner Ġpresent ĠSum ma Ru NN er , Ġa Ġsimple ĠR NN - based Ġsequence Ġclass ifier Ġwithout Ġdec oder , Ġoutper forming Ġor Ġmatching Ġthe Ġmodel Ġof ĠB IB REF 2 Ġ. ĠThey Ġtake Ġcontent , Ġsal ience , Ġnovelty , Ġand Ġposition Ġof Ġeach Ġsentence Ġinto Ġconsideration Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġextract ive Ġsummary . ĠYet , Ġthey Ġdo Ġnot Ġcapture Ġany Ġaspect Ġof Ġthe Ġtopical Ġstructure , Ġas Ġwe Ġdo Ġin Ġthis Ġpaper . ĠSo Ġtheir Ġapproach Ġwould Ġarguably Ġsuffer Ġwhen Ġapplied Ġto Ġlong Ġdocuments , Ġlikely Ġcontaining Ġmultiple Ġand Ġdiverse Ġtopics . </s> While ĠSum ma Ru NN er Ġwas Ġtested Ġonly Ġon Ġnews , ĠEM N LP 2018 Ġcarry Ġout Ġa Ġcomprehensive Ġset Ġof Ġexperiments Ġwith Ġdeep Ġlearning Ġmodels Ġof Ġextract ive Ġsummar ization Ġacross Ġdifferent Ġdomains , Ġi . e . Ġnews , Ġpersonal Ġstories , Ġmeetings , Ġand Ġmedical Ġarticles , Ġas Ġwell Ġas Ġacross Ġdifferent Ġneural Ġarchitectures , Ġin Ġorder Ġto Ġbetter Ġunderstand Ġthe Ġgeneral Ġpros Ġand Ġcons Ġof Ġdifferent Ġdesign Ġchoices . ĠThey Ġfind Ġthat Ġnon Ġauto - reg ressive Ġsentence Ġextraction Ġperforms Ġas Ġwell Ġor Ġbetter Ġthan Ġauto - reg ressive Ġextraction Ġin Ġall Ġdomains , Ġwhere Ġby Ġauto - reg ressive Ġsentence Ġextraction Ġthey Ġmean Ġusing Ġprevious Ġpredictions Ġto Ġinform Ġfuture Ġpredictions . ĠFurthermore , Ġthey Ġfind Ġthat Ġthe ĠAverage ĠWord ĠEmb edd ing Ġsentence Ġenc oder Ġworks Ġat Ġleast Ġas Ġwell Ġas Ġenc od ers Ġbased Ġon ĠCNN Ġand ĠR NN . ĠIn Ġlight Ġof Ġthese Ġfindings , Ġour Ġmodel Ġis Ġnot Ġauto - reg ressive Ġand Ġuses Ġthe ĠAverage ĠWord ĠEmb edd ing Ġenc oder . </s> Ext ractive Ġsummar ization Ġon ĠScientific Ġpapers </s> Research Ġon Ġsummar izing Ġscientific Ġarticles Ġhas Ġa Ġlong Ġhistory ĠB IB REF 13 Ġ. ĠEarlier Ġon , Ġit Ġwas Ġrealized Ġthat Ġsummar izing Ġscientific Ġpapers Ġrequires Ġdifferent Ġapproaches Ġthan Ġwhat Ġwas Ġused Ġfor Ġsummar izing Ġnews Ġarticles , Ġdue Ġto Ġdifferences Ġin Ġdocument Ġlength , Ġwriting Ġstyle Ġand Ġrhetorical Ġstructure . ĠFor Ġinstance , ĠB IB REF 14 Ġpresented Ġa Ġsupervised ĠNa ive ĠBay es Ġclass ifier Ġto Ġselect Ġcontent Ġfrom Ġa Ġscientific Ġpaper Ġbased Ġon Ġthe Ġrhetorical Ġstatus Ġof Ġeach Ġsentence Ġ( e . g ., Ġwhether Ġit Ġspecified Ġa Ġresearch Ġgoal , Ġor Ġsome Ġgenerally Ġaccepted Ġscientific Ġbackground Ġknowledge , Ġetc .). ĠMore Ġrecently , Ġresearchers Ġhave Ġextended Ġthis Ġwork Ġby Ġapplying Ġmore Ġsophisticated Ġclass ifiers Ġto Ġidentify Ġmore Ġfine - grain Ġrhetorical Ġcategories , Ġas Ġwell Ġas Ġby Ġexploiting Ġcitation Ġcontexts . Ġ2013 - disc ourse Ġpropose Ġthe ĠCore SC Ġdiscourse - driven Ġcontent , Ġwhich Ġrelies Ġon ĠCR Fs Ġand ĠSV Ms , Ġto Ġclassify Ġthe Ġdiscourse Ġcategories Ġ( e . g . ĠBackground , ĠHyp ot hesis , ĠMot ivation , Ġetc .) Ġat Ġthe Ġsentence Ġlevel . ĠThe Ġrecent Ġwork Ġmost Ġsimilar Ġto Ġours Ġis ĠB IB REF 5 Ġwhere , Ġin Ġorder Ġto Ġdetermine Ġwhether Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary , Ġthey Ġdirectly Ġuse Ġthe Ġsection Ġeach Ġsentence Ġappears Ġin Ġas Ġa Ġcateg orical Ġfeature Ġwith Ġvalues Ġlike ĠHigh light , ĠAbstract , ĠIntroduction , Ġetc .. ĠIn Ġthis Ġpaper , Ġinstead Ġof Ġusing Ġsections Ġas Ġcateg orical Ġfeatures , Ġwe Ġrely Ġon Ġa Ġdistributed Ġrepresentation Ġof Ġthe Ġsemantic Ġinformation Ġwithin Ġeach Ġsection , Ġas Ġthe Ġlocal Ġcontext Ġof Ġeach Ġsentence . ĠIn Ġa Ġvery Ġdifferent Ġline Ġof Ġwork , Ġco han - 2015 - scientific Ġform Ġthe Ġsummary Ġby Ġalso Ġexploiting Ġinformation Ġon Ġhow Ġthe Ġtarget Ġpaper Ġis Ġcited Ġin Ġother Ġpapers . ĠCurrently , Ġwe Ġdo Ġnot Ġuse Ġany Ġinformation Ġfrom Ġcitation Ġcontexts . </s> Dat as ets Ġfor Ġlong Ġdocuments </s> summary dat as et Ġprovide Ġa Ġcomprehensive Ġoverview Ġof Ġthe Ġcurrent Ġdatasets Ġfor Ġsummar ization . ĠNotice ably , Ġmost Ġof Ġthe Ġlarger - scale Ġsummar ization Ġdatasets Ġconsists Ġof Ġrelatively Ġshort Ġdocuments , Ġlike ĠCNN / Daily Mail ĠB IB REF 1 Ġand ĠNew ĠYork ĠTimes ĠB IB REF 15 Ġ. ĠOne Ġexception Ġis ĠB IB REF 9 Ġthat Ġrecently Ġintroduce Ġtwo Ġlarge - scale Ġdatasets Ġof Ġlong Ġand Ġstructured Ġscientific Ġpapers Ġobtained Ġfrom Ġar X iv Ġand ĠPubMed . ĠThese Ġtwo Ġnew Ġdatasets Ġcontain Ġmuch Ġlonger Ġdocuments Ġthan Ġall Ġthe Ġnews Ġdatasets Ġ( See ĠTable ĠT AB REF 6 Ġ) Ġand Ġare Ġtherefore Ġideal Ġtest - bed s Ġfor Ġthe Ġmethod Ġwe Ġpresent Ġin Ġthis Ġpaper . </s> Ne ural ĠAbstract ive Ġsummar ization Ġon Ġlong Ġdocuments </s> While Ġmost Ġcurrent Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġhave Ġfocused Ġon Ġsummar izing Ġrelatively Ġshort Ġnews Ġarticles Ġ( e . g ., ĠB IB REF 16 Ġ), Ġfew Ġresearchers Ġhave Ġstarted Ġto Ġinvestigate Ġthe Ġsummar ization Ġof Ġlonger Ġdocuments Ġby Ġexploiting Ġtheir Ġnatural Ġstructure . Ġagents Ġpresent Ġan Ġenc oder - dec oder Ġarchitecture Ġto Ġaddress Ġthe Ġchallenges Ġof Ġrepresenting Ġa Ġlong Ġdocument Ġfor Ġabstract ive Ġsummar ization . ĠThe Ġencoding Ġtask Ġis Ġdivided Ġacross Ġseveral Ġcollaborating Ġagents , Ġeach Ġis Ġresponsible Ġfor Ġa Ġsubsection Ġof Ġtext Ġthrough Ġa Ġmulti - layer ĠL ST M Ġwith Ġword Ġattention . ĠTheir Ġmodel Ġseems Ġhowever Ġoverly Ġcomplicated Ġwhen Ġit Ġcomes Ġto Ġthe Ġextract ive Ġsummar ization Ġtask , Ġwhere Ġword Ġattention Ġis Ġarguably Ġmuch Ġless Ġcritical . ĠSo , Ġwe Ġdo Ġnot Ġconsider Ġthis Ġmodel Ġfurther Ġin Ġthis Ġpaper . </s> disc our sel ong document Ġalso Ġpropose Ġa Ġmodel Ġfor Ġabstract ive Ġsummar ization Ġtaking Ġthe Ġstructure Ġof Ġdocuments Ġinto Ġconsideration Ġwith Ġa Ġhierarchical Ġapproach , Ġand Ġtest Ġit Ġon Ġlonger Ġdocuments Ġwith Ġsection Ġinformation , Ġi . e . Ġscientific Ġpapers . ĠIn Ġparticular , Ġthey Ġapply Ġa Ġhierarchical Ġenc oder Ġat Ġthe Ġword Ġand Ġsection Ġlevels . ĠThen , Ġin Ġthe Ġdecoding Ġstep , Ġthey Ġcombine Ġthe Ġword Ġattention Ġand Ġsection Ġattention Ġto Ġobtain Ġa Ġcontext Ġvector . </s> This Ġapproach Ġto Ġcapture Ġdiscourse Ġstructure Ġis Ġhowever Ġquite Ġlimited Ġboth Ġin Ġgeneral Ġand Ġespecially Ġwhen Ġyou Ġconsider Ġits Ġapplication Ġto Ġextract ive Ġsummar ization . ĠFirst , Ġtheir Ġhierarchical Ġmethod Ġhas Ġa Ġlarge Ġnumber Ġof Ġparameters Ġand Ġit Ġis Ġtherefore Ġslow Ġto Ġtrain Ġand Ġlikely Ġprone Ġto Ġover fitting . ĠSecondly , Ġit Ġdoes Ġnot Ġtake Ġthe Ġglobal Ġcontext Ġof Ġthe Ġwhole Ġdocument Ġinto Ġaccount , Ġwhich Ġmay Ġarguably Ġbe Ġcritical Ġin Ġextract ive Ġmethods , Ġwhen Ġdeciding Ġon Ġthe Ġsal ience Ġof Ġa Ġsentence Ġ( or Ġeven Ġa Ġword ). ĠThe Ġextract ive Ġsummar izer Ġwe Ġpresent Ġin Ġthis Ġpaper Ġtries Ġto Ġaddress Ġthese Ġlimitations Ġby Ġadopting Ġthe Ġparameter Ġlean ĠL ST M - minus Ġmethod , Ġand Ġby Ġexplicitly Ġmodeling Ġthe Ġglobal Ġcontext . </s> L ST M - Min us </s> The ĠL ST M - Min us Ġmethod Ġis Ġfirst Ġproposed Ġin ĠB IB REF 6 Ġas Ġa Ġnovel Ġway Ġto Ġlearn Ġsentence Ġsegment Ġembed d ings Ġfor Ġgraph - based Ġdependency Ġparsing , Ġi . e . Ġestimating Ġthe Ġmost Ġlikely Ġdependency Ġtree Ġgiven Ġan Ġinput Ġsentence . ĠFor Ġeach Ġdependency Ġpair , Ġthey Ġdivide Ġa Ġsentence Ġinto Ġthree Ġsegments Ġ( prefix , Ġinf ix Ġand Ġsuffix ), Ġand ĠL ST M - Min us Ġis Ġused Ġto Ġrepresent Ġeach Ġsegment . ĠThey Ġapply Ġa Ġsingle ĠL ST M Ġto Ġthe Ġwhole Ġsentence Ġand Ġuse Ġthe Ġdifference Ġbetween Ġtwo Ġhidden Ġstates ĠIN LINE FORM 0 Ġto Ġrepresent Ġthe Ġsegment Ġfrom Ġword ĠIN LINE FORM 1 Ġto Ġword ĠIN LINE FORM 2 Ġ. ĠThis Ġenables Ġtheir Ġmodel Ġto Ġlearn Ġsegment Ġembed d ings Ġfrom Ġinformation Ġboth Ġoutside Ġand Ġinside Ġthe Ġsegments Ġand Ġthus Ġenhancing Ġtheir Ġmodel Ġability Ġto Ġaccess Ġto Ġsentence - level Ġinformation . ĠThe Ġintuition Ġbehind Ġthe Ġmethod Ġis Ġthat Ġeach Ġhidden Ġvector ĠIN LINE FORM 3 Ġcan Ġcapture Ġuseful Ġinformation Ġbefore Ġand Ġincluding Ġthe Ġword ĠIN LINE FORM 4 Ġ. </s> Shortly Ġafter , Ġl st m - minus const itu ency Ġuse Ġthe Ġsame Ġmethod Ġon Ġthe Ġtask Ġof Ġconstituency Ġparsing , Ġas Ġthe Ġrepresentation Ġof Ġa Ġsentence Ġspan , Ġextending Ġthe Ġoriginal Ġun i - direction al ĠL ST M - Min us Ġto Ġthe Ġbi - direction al Ġcase . ĠMore Ġrecently , Ġinspired Ġby Ġthe Ġsuccess Ġof ĠL ST M - Min us Ġin Ġboth Ġdependency Ġand Ġconstituency Ġparsing , Ġl st m - minus disc ourse Ġextend Ġthe Ġtechnique Ġto Ġdiscourse Ġparsing . ĠThey Ġpropose Ġa Ġtwo - stage Ġmodel Ġconsisting Ġof Ġan Ġintra - sent ential Ġparser Ġand Ġa Ġmulti - sent ential Ġparser , Ġlearning Ġcontext ually Ġinformed Ġrepresentations Ġof Ġconstituents Ġwith ĠL ST M - Min us , Ġat Ġthe Ġsentence Ġand Ġdocument Ġlevel , Ġrespectively . </s> Similarly , Ġin Ġthis Ġpaper , Ġwhen Ġdeciding Ġif Ġa Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary , Ġthe Ġlocal Ġcontext Ġof Ġthat Ġsentence Ġis Ġcaptured Ġby Ġapplying ĠL ST M - Min us Ġat Ġthe Ġdocument Ġlevel , Ġto Ġrepresent Ġthe Ġsub - sequence Ġof Ġsentences Ġof Ġthe Ġdocument Ġ( i . e ., Ġthe Ġtopic / section ) Ġthe Ġtarget Ġsentence Ġbelongs Ġto . </s> Our ĠModel </s> In Ġthis Ġwork , Ġwe Ġpropose Ġan Ġextract ive Ġmodel Ġfor Ġlong Ġdocuments , Ġincorporating Ġlocal Ġand Ġglobal Ġcontext Ġinformation , Ġmotivated Ġby Ġnatural Ġtopic - oriented Ġstructure Ġof Ġhuman - written Ġlong Ġdocuments . ĠThe Ġarchitecture Ġof Ġour Ġmodel Ġis Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, Ġeach Ġsentence Ġis Ġvisited Ġsequ entially Ġin Ġthe Ġoriginal Ġdocument Ġorder , Ġand Ġa Ġcorresponding Ġconfidence Ġscore Ġis Ġcomputed Ġexpressing Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġextract ive Ġsummary . ĠOur Ġmodel Ġcomprises Ġthree Ġcomponents : Ġthe Ġsentence Ġenc oder , Ġthe Ġdocument Ġenc oder Ġand Ġthe Ġsentence Ġclass ifier . </s> Sent ence ĠEnc oder </s> The Ġgoal Ġof Ġthe Ġsentence Ġenc oder Ġis Ġmapping Ġsequences Ġof Ġword Ġembed d ings Ġto Ġa Ġfixed Ġlength Ġvector Ġ( See Ġbottom Ġcenter Ġof ĠFigure ĠFIG REF 10 Ġ). ĠThere Ġare Ġseveral Ġcommon Ġmethods Ġto Ġembed Ġsentences . ĠFor Ġextract ive Ġsummar ization , ĠR NN Ġwere Ġused Ġin ĠB IB REF 17 Ġ, ĠCNN Ġin ĠB IB REF 2 Ġ, Ġand ĠAverage ĠWord ĠEmb edd ing Ġin ĠB IB REF 18 Ġ. ĠEM N LP 2018 Ġexperiment Ġwith Ġall Ġthe Ġthree Ġmethods Ġand Ġconclude Ġthat ĠWord ĠEmb edd ing ĠA ver aging Ġis Ġas Ġgood Ġor Ġbetter Ġthan Ġeither ĠR NN s Ġor ĠCNN s Ġfor Ġsentence Ġembed ding Ġacross Ġdifferent Ġdomains Ġand Ġsummar izer Ġarchitectures . ĠThus , Ġwe Ġuse Ġthe ĠAverage ĠWord ĠEmb edd ing Ġas Ġour Ġsentence Ġenc oder , Ġby Ġwhich Ġa Ġsentence Ġembed ding Ġis Ġsimply Ġthe Ġaverage Ġof Ġits Ġword Ġembed d ings , Ġi . e . ĠIN LINE FORM 0 </s> Besides , Ġwe Ġalso Ġtried Ġthe Ġpopular Ġpre - trained ĠB ERT Ġsentence Ġembed ding ĠB IB REF 19 Ġ, Ġbut Ġinitial Ġresults Ġwere Ġrather Ġpoor . ĠSo Ġwe Ġdo Ġnot Ġpursue Ġthis Ġpossibility Ġany Ġfurther . </s> Document ĠEnc oder </s> At Ġthe Ġdocument Ġlevel , Ġa Ġbi - direction al Ġrecurrent Ġneural Ġnetwork ĠB IB REF 20 Ġis Ġoften Ġused Ġto Ġencode Ġall Ġthe Ġsentences Ġsequ entially Ġforward Ġand Ġbackward , Ġwith Ġsuch Ġmodel Ġachieving Ġremarkable Ġsuccess Ġin Ġmachine Ġtranslation ĠB IB REF 21 Ġ. ĠAs Ġunits , Ġwe Ġselected Ġg ated Ġrecurrent Ġunits Ġ( GR U ) ĠB IB REF 22 Ġ, Ġin Ġlight Ġof Ġfavorable Ġresults Ġshown Ġin ĠB IB REF 23 Ġ. ĠThe ĠGR U Ġis Ġrepresented Ġwith Ġthe Ġstandard Ġreset , Ġupdate , Ġand Ġnew Ġgates . </s> The Ġoutput Ġof Ġthe Ġbi - direction al ĠGR U Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġcomprises Ġtwo Ġhidden Ġstates , ĠIN LINE FORM 1 Ġas Ġforward Ġand Ġbackward Ġhidden Ġstate , Ġrespectively . </s> A . ĠSent ence Ġrepresentation ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ( A ), Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġsentence Ġrepresentation Ġis Ġthe Ġconc aten ation Ġof Ġboth Ġbackward Ġand Ġforward Ġhidden Ġstate Ġof Ġthat Ġsentence . ĠIN LINE FORM 1 </s> In Ġthis Ġway , Ġthe Ġsentence Ġrepresentation Ġnot Ġonly Ġrepresents Ġthe Ġcurrent Ġsentence , Ġbut Ġalso Ġpartially Ġcovers Ġcontextual Ġinformation Ġboth Ġbefore Ġand Ġafter Ġthis Ġsentence . </s> B . ĠDocument Ġrepresentation ĠThe Ġdocument Ġrepresentation Ġprovides Ġglobal Ġinformation Ġon Ġthe Ġwhole Ġdocument . ĠIt Ġis Ġcomputed Ġas Ġthe Ġconc aten ation Ġof Ġthe Ġfinal Ġstate Ġof Ġthe Ġforward Ġand Ġbackward ĠGR U , Ġlabeled Ġas ĠB Ġin ĠFigure ĠFIG REF 10 Ġ. ĠB IB REF 24 ĠIN LINE FORM 0 </s> C . ĠTopic Ġsegment Ġrepresentation ĠTo Ġcapture Ġthe Ġlocal Ġcontext Ġof Ġeach Ġsentence , Ġnamely Ġthe Ġinformation Ġof Ġthe Ġtopic Ġsegment Ġthat Ġsentence Ġfalls Ġinto , Ġwe Ġapply Ġthe ĠL ST M - Min us Ġmethod , Ġa Ġmethod Ġfor Ġlearning Ġembed d ings Ġof Ġtext Ġspans . ĠL ST M - Min us Ġis Ġshown Ġin Ġdetail Ġin ĠFigure Ġ1 Ġ( left Ġpanel ĠC ), Ġeach Ġtopic Ġsegment Ġis Ġrepresented Ġas Ġthe Ġsubt raction Ġbetween Ġthe Ġhidden Ġstates Ġof Ġthe Ġstart Ġand Ġthe Ġend Ġof Ġthat Ġtopic . ĠAs Ġillustrated Ġin ĠFigure ĠFIG REF 10 Ġ, Ġthe Ġrepresentation Ġfor Ġsection Ġ2 Ġof Ġthe Ġsample Ġdocument Ġ( containing Ġthree Ġsections Ġand Ġeight Ġsentences Ġoverall ) Ġcan Ġbe Ġcomputed Ġas ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġare Ġthe Ġforward Ġhidden Ġstates Ġof Ġsentence Ġ5 Ġand Ġ2 , Ġrespectively , Ġwhile ĠIN LINE FORM 2 Ġare Ġthe Ġbackward Ġhidden Ġstates Ġof Ġsentence Ġ3 Ġand Ġ6 , Ġrespectively . ĠIn Ġgeneral , Ġthe Ġtopic Ġsegment Ġrepresentation ĠIN LINE FORM 3 Ġfor Ġsegment ĠIN LINE FORM 4 Ġis Ġcomputed Ġas : ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġindex Ġof Ġthe Ġbeginning Ġand Ġthe Ġend Ġof Ġtopic ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġdenote Ġthe Ġtopic Ġsegment Ġrepresentation Ġof Ġforward Ġand Ġbackward , Ġrespectively . ĠThe Ġfinal Ġrepresentation Ġof Ġtopic ĠIN LINE FORM 4 Ġis Ġthe Ġconc aten ation Ġof Ġforward Ġand Ġbackward Ġrepresentation ĠIN LINE FORM 5 Ġ. ĠTo Ġobtain ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ, Ġwe Ġutilize Ġsubt raction Ġbetween ĠGR U Ġhidden Ġvectors Ġof ĠIN LINE FORM 8 Ġand ĠIN LINE FORM 9 Ġ, Ġand Ġwe Ġpad Ġthe Ġhidden Ġstates Ġwith Ġzero Ġvectors Ġboth Ġin Ġthe Ġbeginning Ġand Ġthe Ġend , Ġto Ġensure Ġthe Ġindex Ġcan Ġnot Ġbe Ġout Ġof Ġbound . ĠThe Ġintuition Ġbehind Ġthis Ġprocess Ġis Ġthat Ġthe ĠGR Us Ġcan Ġkeep Ġprevious Ġuseful Ġinformation Ġin Ġtheir Ġmemory Ġcell Ġby Ġexploiting Ġreset , Ġupdate , Ġand Ġnew Ġgates Ġto Ġdecide Ġhow Ġto Ġutilize Ġand Ġupdate Ġthe Ġmemory Ġof Ġprevious Ġinformation . ĠIn Ġthis Ġway , Ġwe Ġcan Ġrepresent Ġthe Ġcontextual Ġinformation Ġwithin Ġeach Ġtopic Ġsegment Ġfor Ġall Ġthe Ġsentences Ġin Ġthat Ġsegment . </s> Dec oder </s> Once Ġwe Ġhave Ġobtained Ġa Ġrepresentation Ġfor Ġthe Ġsentence , Ġfor Ġits Ġtopic Ġsegment Ġ( i . e ., Ġlocal Ġcontext ) Ġand Ġfor Ġthe Ġdocument Ġ( i . e ., Ġglobal Ġcontext ), Ġthese Ġthree Ġfactors Ġare Ġcombined Ġto Ġmake Ġa Ġfinal Ġprediction ĠIN LINE FORM 0 Ġon Ġwhether Ġthe Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠWe Ġconsider Ġtwo Ġways Ġin Ġwhich Ġthese Ġthree Ġfactors Ġcan Ġbe Ġcombined . </s> Con cat en ation ĠWe Ġcan Ġsimply Ġconc aten ate Ġthe Ġvectors Ġof Ġthese Ġthree Ġfactors Ġas , ĠIN LINE FORM 0 </s> where Ġsentence ĠIN LINE FORM 0 Ġis Ġpart Ġof Ġthe Ġtopic ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġrepresentation Ġof Ġsentence ĠIN LINE FORM 3 Ġwith Ġtopic Ġsegment Ġinformation Ġand Ġglobal Ġcontext Ġinformation . </s> Att ent ive Ġcontext ĠAs Ġlocal Ġcontext Ġand Ġglobal Ġcontext Ġare Ġall Ġcontextual Ġinformation Ġof Ġthe Ġgiven Ġsentence , Ġwe Ġuse Ġan Ġattention Ġmechanism Ġto Ġdecide Ġthe Ġweight Ġof Ġeach Ġcontext Ġvector , Ġrepresented Ġas ĠIN LINE FORM 0 </s> where Ġthe ĠIN LINE FORM 0 Ġis Ġthe Ġweighted Ġcontext Ġvector Ġof Ġeach Ġsentence ĠIN LINE FORM 1 Ġ, Ġand Ġassume Ġsentence ĠIN LINE FORM 2 Ġis Ġin Ġtopic ĠIN LINE FORM 3 Ġ. </s> Then Ġthere Ġis Ġa Ġfinal Ġmulti - layer Ġpercept ron ( ML P ) Ġfollowed Ġwith Ġa Ġs igm oid Ġactivation Ġfunction Ġindicating Ġthe Ġconfidence Ġscore Ġfor Ġselecting Ġeach Ġsentence : ĠIN LINE FORM 0 </s> Exper iments </s> To Ġvalidate Ġour Ġmethod , Ġwe Ġset Ġup Ġexperiments Ġon Ġthe Ġtwo Ġscientific Ġpaper Ġdatasets Ġ( ar X iv Ġand ĠPubMed ). ĠWith ĠR OU GE Ġand ĠMET E OR Ġscores Ġas Ġautomatic Ġevaluation Ġmetrics , Ġwe Ġcompare Ġwith Ġprevious Ġworks , Ġboth Ġabstract ive Ġand Ġextract ive . </s> Training </s> The Ġweighted Ġnegative Ġlog - like lihood Ġis Ġminimized , Ġwhere Ġthe Ġweight Ġis Ġcomputed Ġas ĠIN LINE FORM 0 Ġ, Ġto Ġsolve Ġthe Ġproblem Ġof Ġhighly Ġim balanced Ġdata Ġ( typ ical Ġin Ġextract ive Ġsummar ization ). ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġrepresent Ġthe Ġground - truth Ġlabel Ġof Ġsentence ĠIN LINE FORM 1 Ġ, Ġwith ĠIN LINE FORM 2 Ġmeaning Ġsentence ĠIN LINE FORM 3 Ġis Ġin Ġthe Ġgold - standard Ġextract Ġsummary . </s> Ext ractive ĠLabel ĠGeneration </s> In Ġthe ĠPub med Ġand Ġar X iv Ġdatasets , Ġthe Ġextract ive Ġsumm aries Ġare Ġmissing . ĠSo Ġwe Ġfollow Ġthe Ġwork Ġof ĠB IB REF 18 Ġon Ġextract ive Ġsummary Ġlabeling , Ġconstructing Ġgold Ġlabel Ġsequences Ġby Ġgreed ily Ġoptimizing ĠR OU GE - 1 Ġon Ġthe Ġgold - standard Ġabstract s , Ġwhich Ġare Ġavailable Ġfor Ġeach Ġarticle . ĠThe Ġalgorithm Ġis Ġshown Ġin ĠAppendix ĠA . </s> Im plementation ĠDetails </s> We Ġtrain Ġour Ġmodel Ġusing Ġthe ĠAdam Ġoptim izer ĠB IB REF 25 Ġwith Ġlearning Ġrate ĠIN LINE FORM 0 Ġand Ġa Ġdrop Ġout Ġrate Ġof Ġ0 . 3 . ĠWe Ġuse Ġa Ġmini - batch Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġdocuments , Ġand Ġthe Ġsize Ġof Ġthe ĠGR U Ġhidden Ġstates Ġis Ġ300 . ĠFor Ġword Ġembed d ings , Ġwe Ġuse ĠGl o Ve ĠB IB REF 26 Ġwith Ġdimension Ġ300 , Ġpre - trained Ġon Ġthe ĠWikipedia Ġand ĠGig aw ord . ĠThe Ġvocabulary Ġsize Ġof Ġour Ġmodel Ġis Ġ5 0000 . ĠAll Ġthe Ġabove Ġparameters Ġwere Ġset Ġbased Ġon ĠB IB REF 18 Ġwithout Ġany Ġfine - tun ing . ĠAgain Ġfollowing ĠB IB REF 18 Ġ, Ġwe Ġtrain Ġeach Ġmodel Ġfor Ġ50 Ġepoch s , Ġand Ġthe Ġbest Ġmodel Ġis Ġselected Ġwith Ġearly Ġstopping Ġon Ġthe Ġvalidation Ġset Ġaccording Ġto ĠRouge - 2 ĠF - score . </s> Mod els Ġfor ĠComparison </s> We Ġperform Ġa Ġsystematic Ġcomparison Ġwith Ġprevious Ġwork Ġin Ġextract ive Ġsummar ization . ĠFor Ġcomple teness , Ġwe Ġalso Ġcompare Ġwith Ġrecent Ġneural Ġabstract ive Ġapproaches . ĠIn Ġall Ġthe Ġexperiments , Ġwe Ġuse Ġthe Ġsame Ġtrain / val / test Ġsplitting . </s> Traditional Ġextract ive Ġsummar ization Ġmodels : ĠSum Basic ĠB IB REF 27 Ġ, ĠL SA ĠB IB REF 28 Ġ, Ġand ĠLex Rank ĠB IB REF 29 </s> Ne ural Ġabstract ive Ġsummar ization Ġmodels : ĠAtt n - Se q 2 Se q ĠB IB REF 1 Ġ, ĠP nt r - Gen - Se q 2 Se q ĠB IB REF 16 Ġand ĠDisc ourse - aware ĠB IB REF 9 </s> Ne ural Ġextract ive Ġsummar ization Ġmodels : ĠCheng & L ap ata ĠB IB REF 2 Ġand ĠSum ma Ru NN er ĠB IB REF 17 Ġ. ĠBased Ġon ĠB IB REF 18 Ġ, Ġwe Ġuse Ġthe ĠAverage ĠWord ĠEnc oder Ġas Ġsentence Ġenc oder Ġfor Ġboth Ġmodels , Ġinstead Ġof Ġthe ĠCNN Ġand ĠR NN Ġsentence Ġenc od ers Ġthat Ġwere Ġoriginally Ġused Ġin Ġthe Ġtwo Ġsystems , Ġrespectively . </s> Bas eline : ĠSimilar Ġto Ġour Ġmodel , Ġbut Ġwithout Ġlocal Ġcontext Ġand Ġglobal Ġcontext , Ġi . e . Ġthe Ġinput Ġto ĠML P Ġis Ġthe Ġsentence Ġrepresentation Ġonly . </s> Lead : ĠGiven Ġa Ġlength Ġlimit Ġof ĠIN LINE FORM 0 Ġwords Ġfor Ġthe Ġsummary , ĠLead Ġwill Ġreturn Ġthe Ġfirst ĠIN LINE FORM 1 Ġwords Ġof Ġthe Ġsource Ġdocument . </s> Oracle : Ġuses Ġthe ĠGold ĠStandard Ġextract ive Ġlabels , Ġgenerated Ġbased Ġon ĠR OU GE Ġ( Sec . Ġ4 . 2 ). </s> Results Ġand ĠAnalysis </s> For Ġevaluation , Ġwe Ġfollow Ġthe Ġsame Ġprocedure Ġas Ġin ĠB IB REF 18 Ġ. ĠSum m aries Ġare Ġgenerated Ġby Ġselecting Ġthe Ġtop Ġranked Ġsentences Ġby Ġmodel Ġprobability ĠIN LINE FORM 0 Ġ, Ġuntil Ġthe Ġlength Ġlimit Ġis Ġmet Ġor Ġexceeded . ĠBased Ġon Ġthe Ġaverage Ġlength Ġof Ġabstract s Ġin Ġthese Ġtwo Ġdatasets , Ġwe Ġset Ġthe Ġlength Ġlimit Ġto Ġ200 Ġwords . ĠWe Ġuse ĠR OU GE Ġscores ĠB IB REF 30 Ġand ĠMET E OR Ġscores ĠB IB REF 31 Ġbetween Ġthe Ġmodel Ġresults Ġand Ġground - truth Ġabstract ive Ġsumm aries Ġas Ġevaluation Ġmetric . ĠThe Ġun ig ram Ġand Ġbig ram Ġoverlap Ġ( R OU GE - 1 , 2 ) Ġare Ġintended Ġto Ġmeasure Ġthe Ġinform at iveness , Ġwhile Ġlongest Ġcommon Ġsubsequ ence Ġ( R OU GE - L ) Ġcaptures Ġflu ency Ġto Ġsome Ġextent ĠB IB REF 2 Ġ. ĠMET E OR Ġwas Ġoriginally Ġproposed Ġto Ġevaluate Ġtranslation Ġsystems Ġby Ġmeasuring Ġthe Ġalignment Ġbetween Ġthe Ġsystem Ġoutput Ġand Ġreference Ġtranslations . ĠAs Ġsuch , Ġit Ġcan Ġalso Ġbe Ġused Ġas Ġan Ġautomatic Ġevaluation Ġmetric Ġfor Ġsummar ization ĠB IB REF 18 Ġ. </s> The Ġperformance Ġof Ġall Ġmodels Ġon Ġar X iv Ġand ĠPub med Ġis Ġshown Ġin ĠTable ĠT AB REF 28 Ġand ĠTable ĠT AB REF 29 Ġ, Ġrespectively . ĠFollow Ġthe Ġwork ĠB IB REF 18 Ġ, Ġwe Ġuse Ġthe Ġapproximate Ġrandom ization Ġas Ġthe Ġstatistical Ġsignificance Ġtest Ġmethod ĠB IB REF 32 Ġwith Ġa ĠBon fer ron i Ġcorrection Ġfor Ġmultiple Ġcomparisons , Ġat Ġthe Ġconfidence Ġlevel Ġ0 . 01 Ġ( ĠIN LINE FORM 0 Ġ). ĠAs Ġwe Ġcan Ġsee Ġin Ġthese Ġtables , Ġon Ġboth Ġdatasets , Ġthe Ġneural Ġextract ive Ġmodels Ġoutper forms Ġthe Ġtraditional Ġextract ive Ġmodels Ġon Ġinform at iveness Ġ( R OU GE - 1 , 2 ) Ġby Ġa Ġwide Ġmargin , Ġbut Ġresults Ġare Ġmixed Ġon ĠR OU GE - L . ĠPresumably , Ġthis Ġis Ġdue Ġto Ġthe Ġneural Ġtraining Ġprocess , Ġwhich Ġrelies Ġon Ġa Ġgoal Ġstandard Ġbased Ġon ĠR OU GE - 1 . ĠExpl oring Ġother Ġtraining Ġschemes Ġand / or Ġa Ġcombination Ġof Ġtraditional Ġand Ġneural Ġapproaches Ġis Ġleft Ġas Ġfuture Ġwork . ĠSimilarly , Ġthe Ġneural Ġextract ive Ġmodels Ġalso Ġdominate Ġthe Ġneural Ġabstract ive Ġmodels Ġon ĠR OU GE - 1 , 2 , Ġbut Ġthese Ġabstract ive Ġmodels Ġtend Ġto Ġhave Ġthe Ġhighest ĠR OU GE - L Ġscores , Ġpossibly Ġbecause Ġthey Ġare Ġtrained Ġdirectly Ġon Ġgold Ġstandard Ġabstract Ġsumm aries . </s> Compared Ġwith Ġother Ġneural Ġextract ive Ġmodels , Ġour Ġmodels Ġ( both Ġwith Ġattentive Ġcontext Ġand Ġconc aten ation Ġdec oder ) Ġhave Ġbetter Ġperformances Ġon Ġall Ġthree ĠR OU GE Ġscores , Ġas Ġwell Ġas ĠMET E OR . ĠIn Ġparticular , Ġthe Ġimprovements Ġover Ġthe ĠBas eline Ġmodel Ġshow Ġthat Ġa Ġcombination Ġof Ġlocal Ġand Ġglobal Ġcontextual Ġinformation Ġdoes Ġhelp Ġto Ġidentify Ġthe Ġmost Ġimportant Ġsentences Ġ( more Ġon Ġthis Ġin Ġthe Ġnext Ġsection ). ĠInterestingly , Ġjust Ġthe ĠBas eline Ġmodel Ġalready Ġachieves Ġa Ġslightly Ġbetter Ġperformance Ġthan Ġprevious Ġworks ; Ġpossibly Ġbecause Ġthe Ġauto - reg ressive Ġapproach Ġused Ġin Ġthose Ġmodels Ġis Ġeven Ġmore Ġdetrimental Ġfor Ġlong Ġdocuments . </s> Figure ĠFIG REF 32 Ġshows Ġthe Ġmost Ġimportant Ġresult Ġof Ġour Ġanalysis : Ġthe Ġbenefits Ġof Ġour Ġmethod , Ġexplicitly Ġdesigned Ġto Ġdeal Ġwith Ġlonger Ġdocuments , Ġdo Ġactually Ġbecome Ġstronger Ġas Ġwe Ġapply Ġit Ġto Ġlonger Ġdocuments . ĠAs Ġit Ġcan Ġbe Ġseen Ġin ĠFigure ĠFIG REF 32 Ġ, Ġthe Ġperformance Ġgain Ġof Ġour Ġmodel Ġwith Ġrespect Ġto Ġcurrent Ġstate - of - the - art Ġextract ive Ġsummar izer Ġis Ġmore Ġpronounced Ġfor Ġdocuments Ġwith ĠIN LINE FORM 0 Ġwords Ġin Ġboth Ġdatasets . </s> Finally , Ġthe Ġresult Ġof ĠLead Ġ( Table ĠT AB REF 28 Ġ, ĠT AB REF 29 Ġ) Ġshows Ġthat Ġscientific Ġpapers Ġhave Ġless Ġposition Ġbias Ġthan Ġnews ; Ġi . e ., Ġthe Ġfirst Ġsentences Ġof Ġthese Ġpapers Ġare Ġnot Ġa Ġgood Ġchoice Ġto Ġform Ġan Ġextract ive Ġsummary . </s> As Ġa Ġteaser Ġfor Ġthe Ġpotential Ġand Ġchallenges Ġthat Ġstill Ġface Ġour Ġapproach , Ġits Ġoutput Ġ( i . e ., Ġthe Ġextracted Ġsentences ) Ġwhen Ġapplied Ġto Ġthis Ġpaper Ġis Ġcolored Ġin Ġred Ġand Ġthe Ġorder Ġin Ġwhich Ġthe Ġsentences Ġare Ġextracted Ġis Ġmarked Ġwith Ġthe ĠRoman Ġnumbering . ĠIf Ġwe Ġset Ġthe Ġsummary Ġlength Ġlimit Ġto Ġthe Ġlength Ġof Ġour Ġabstract , Ġthe Ġfirst Ġfive Ġsentences Ġin Ġthe Ġconclusions Ġsection Ġare Ġextracted . ĠIf Ġwe Ġincrease Ġthe Ġlength Ġto Ġ200 Ġwords , Ġtwo Ġmore Ġsentences Ġare Ġextracted , Ġwhich Ġdo Ġseem Ġto Ġprovide Ġuseful Ġcomplementary Ġinformation . ĠNot Ġsurprisingly , Ġsome Ġredundancy Ġis Ġpresent , Ġas Ġdealing Ġexplicitly Ġwith Ġredundancy Ġis Ġnot Ġa Ġgoal Ġof Ġour Ġcurrent Ġproposal Ġand Ġleft Ġas Ġfuture Ġwork . </s> A bl ation ĠStudy </s> In Ġorder Ġto Ġassess Ġthe Ġrelative Ġcontributions Ġof Ġthe Ġglobal Ġand Ġlocal Ġmodels Ġto Ġthe Ġperformance Ġof Ġour Ġapproach , Ġwe Ġran Ġan Ġab lation Ġstudy . ĠThis Ġwas Ġdone Ġfor Ġeach Ġdataset Ġboth Ġwith Ġthe Ġwhole Ġtest Ġset , Ġas Ġwell Ġas Ġwith Ġa Ġsubset Ġof Ġlong Ġdocuments . ĠThe Ġresults Ġfor ĠPub med Ġand Ġar X iv Ġare Ġshown Ġin ĠTable ĠT AB REF 34 Ġand ĠTable ĠT AB REF 35 Ġ, Ġrespectively . ĠFor Ġstatistical Ġsignificance , Ġas Ġit Ġwas Ġdone Ġfor Ġthe Ġgeneral Ġresults Ġin ĠSection Ġ4 . 5 , Ġwe Ġuse Ġthe Ġapproximate Ġrandom ization Ġmethod ĠB IB REF 32 Ġwith Ġthe ĠBon fer ron i Ġcorrection Ġat Ġ( ĠIN LINE FORM 0 Ġ). </s> From Ġthese Ġtables , Ġwe Ġcan Ġsee Ġthat Ġon Ġboth Ġdatasets Ġthe Ġperformance Ġsignificantly Ġimproves Ġwhen Ġlocal Ġtopic Ġinformation Ġ( i . e . Ġlocal Ġcontext ) Ġis Ġadded . ĠAnd Ġthe Ġimprovement Ġis Ġeven Ġgreater Ġwhen Ġwe Ġonly Ġconsider Ġlong Ġdocuments . ĠRather Ġsurprisingly , Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe Ġglobal Ġcontext . ĠAdding Ġa Ġrepresentation Ġof Ġthe Ġwhole Ġdocument Ġ( i . e . Ġglobal Ġcontext ) Ġnever Ġsignificantly Ġimproves Ġperformance . ĠIn Ġessence , Ġit Ġseems Ġthat Ġall Ġthe Ġbenefits Ġof Ġour Ġmodel Ġcome Ġexclusively Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext , Ġeven Ġfor Ġthe Ġlongest Ġdocuments . ĠFurther Ġinvestigation Ġof Ġthis Ġfinding Ġis Ġleft Ġas Ġfuture Ġwork . </s> Con clusions Ġand ĠFuture ĠWork </s> pur ple In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel Ġextract ive Ġsummar ization Ġmodel Ġespecially Ġdesigned Ġfor Ġlong Ġdocuments , Ġby Ġincorporating Ġthe Ġlocal Ġcontext Ġwithin Ġeach Ġtopic , Ġalong Ġwith Ġthe Ġglobal Ġcontext Ġof Ġthe Ġwhole Ġdocument .[ 2 ] Ġpurple Our Ġapproach Ġintegrates Ġrecent Ġfindings Ġon Ġneural Ġextract ive Ġsummar ization Ġin Ġa Ġparameter Ġlean Ġand Ġmodular Ġarchitecture .[ 3 ] Ġpurple We Ġevaluate Ġour Ġmodel Ġand Ġcompare Ġwith Ġprevious Ġworks Ġin Ġboth Ġextract ive Ġand Ġabstract ive Ġsummar ization Ġon Ġtwo Ġlarge Ġscientific Ġpaper Ġdatasets , Ġwhich Ġcontain Ġdocuments Ġthat Ġare Ġmuch Ġlonger Ġthan Ġin Ġpreviously Ġused Ġcorpor a .[ 4 ] Ġpurple Our Ġmodel Ġnot Ġonly Ġachieves Ġstate - of - the - art Ġon Ġthese Ġtwo Ġdatasets , Ġbut Ġin Ġan Ġadditional Ġexperiment , Ġin Ġwhich Ġwe Ġconsider Ġdocuments Ġwith Ġincreasing Ġlength , Ġit Ġbecomes Ġmore Ġcompetitive Ġfor Ġlonger Ġdocuments .[ 5 ] Ġpurple We Ġalso Ġran Ġan Ġab lation Ġstudy Ġto Ġassess Ġthe Ġrelative Ġcontribution Ġof Ġthe Ġglobal Ġand Ġlocal Ġcomponents Ġof Ġour Ġapproach . Ġ[ 1 ] ĠRather Ġsurprisingly , Ġit Ġappears Ġthat Ġthe Ġbenefits Ġof Ġour Ġmodel Ġcome Ġonly Ġfrom Ġmodeling Ġthe Ġlocal Ġcontext . </s> For Ġfuture Ġwork , Ġwe Ġinitially Ġintend Ġto Ġinvestigate Ġneural Ġmethods Ġto Ġdeal Ġwith Ġredundancy . ĠThen , Ġit Ġcould Ġbe Ġbeneficial Ġto Ġintegrate Ġexplicit Ġfeatures , Ġlike Ġsentence Ġposition Ġand Ġsal ience , Ġinto Ġour Ġneural Ġapproach . ĠMore Ġgenerally , Ġwe Ġplan Ġto Ġcombine Ġof Ġtraditional Ġand Ġneural Ġmodels , Ġas Ġsuggested Ġby Ġour Ġresults . ĠFurthermore , Ġwe Ġwould Ġlike Ġto Ġexplore Ġmore Ġsoph istical Ġstructure Ġof Ġdocuments , Ġlike Ġdiscourse Ġtree , Ġinstead Ġof Ġrough Ġtopic Ġsegments . ĠAs Ġfor Ġevaluation , Ġwe Ġwould Ġlike Ġto Ġelicit Ġhuman Ġjudgments , Ġfor Ġinstance Ġby Ġinviting Ġauthors Ġto Ġrate Ġthe Ġoutputs Ġfrom Ġdifferent Ġsystems , Ġwhen Ġapplied Ġto Ġtheir Ġown Ġpapers . ĠMore Ġlong Ġterm , Ġwe Ġwill Ġstudy Ġhow Ġextract ive / ab stract ive Ġtechniques Ġcan Ġbe Ġintegrated ; Ġfor Ġinstance , Ġthe Ġoutput Ġof Ġan Ġextract ive Ġsystem Ġcould Ġbe Ġfed Ġinto Ġan Ġabstract ive Ġone , Ġtraining Ġthe Ġtwo Ġjointly . </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġby Ġthe ĠLanguage Ġ& ĠSpeech ĠInnovation ĠLab Ġof ĠCloud ĠBU , ĠHuawei ĠTechnologies ĠCo ., ĠLtd . ĠExtract ive ĠLabel ĠGeneration ĠThe Ġalgorithm ĠSEC REF 6 Ġis Ġused Ġto Ġgenerate Ġthe Ġextract ive Ġlabels Ġbased Ġon Ġthe Ġhuman - made Ġabstract ive Ġsumm aries , Ġi . e . Ġabstract s Ġof Ġscientific Ġpapers . ĠExtract ive Ġlabel Ġgeneration ĠLabel Gener ation Reference , sent ences , length Limit ĠIN LINE FORM 0 Ġ= ĠâĢ Ŀ ĠIN LINE FORM 1 Ġ= Ġ0 ĠIN LINE FORM 2 Ġ= Ġ[] ĠIN LINE FORM 3 ĠIN LINE FORM 4 ĠIN LINE FORM 5 ĠIN LINE FORM 6 Ġin Ġrange ( len ( ĠIN LINE FORM 7 Ġ)) ĠIN LINE FORM 8 ĠIN LINE FORM 9 ĠIN LINE FORM 10 ĠIN LINE FORM 11 ĠIN LINE FORM 12 Ġ!= ĠIN LINE FORM 13 ĠIN LINE FORM 14 Ġ. append ( ĠIN LINE FORM 15 Ġ) ĠIN LINE FORM 16 Ġ= ĠIN LINE FORM 17 Ġ+ ĠIN LINE FORM 18 Ġ[ ĠIN LINE FORM 19 Ġ] ĠIN LINE FORM 20 Ġ+= ĠNumber Of Words ( ĠIN LINE FORM 21 Ġ[ ĠIN LINE FORM 22 Ġ]) Ġbreak ĠIN LINE FORM 23