Unanswerable
NONE
1802.08636
Introduction </s> Aut omatic Ġsummar ization Ġhas Ġenjoyed Ġwide Ġpopularity Ġin Ġnatural Ġlanguage Ġprocessing Ġdue Ġto Ġits Ġpotential Ġfor Ġvarious Ġinformation Ġaccess Ġapplications . ĠExamples Ġinclude Ġtools Ġwhich Ġaid Ġusers Ġnavigate Ġand Ġdigest Ġweb Ġcontent Ġ( e . g ., Ġnews , Ġsocial Ġmedia , Ġproduct Ġreviews ), Ġquestion Ġanswering , Ġand Ġpersonalized Ġrecommendation Ġengines . ĠSingle Ġdocument Ġsummar ization ĠâĢĶ Ġthe Ġtask Ġof Ġproducing Ġa Ġshorter Ġversion Ġof Ġa Ġdocument Ġwhile Ġpreserving Ġits Ġinformation Ġcontent ĠâĢĶ Ġis Ġperhaps Ġthe Ġmost Ġbasic Ġof Ġsummar ization Ġtasks Ġthat Ġhave Ġbeen Ġidentified Ġover Ġthe Ġyears Ġ( see ĠB IB REF 0 Ġ, ĠB IB REF 0 Ġfor Ġa Ġcomprehensive Ġoverview ). </s> Modern Ġapproaches Ġto Ġsingle Ġdocument Ġsummar ization Ġare Ġdata - driven , Ġtaking Ġadvantage Ġof Ġthe Ġsuccess Ġof Ġneural Ġnetwork Ġarchitectures Ġand Ġtheir Ġability Ġto Ġlearn Ġcontinuous Ġfeatures Ġwithout Ġrecourse Ġto Ġpre processing Ġtools Ġor Ġlinguistic Ġannotations . ĠAbstract ive Ġsummar ization Ġinvolves Ġvarious Ġtext Ġrewriting Ġoperations Ġ( e . g ., Ġsubstitution , Ġdeletion , Ġre ordering ) Ġand Ġhas Ġbeen Ġrecently Ġframed Ġas Ġa Ġsequence - to - sequence Ġproblem ĠB IB REF 1 Ġ. ĠCentral Ġin Ġmost Ġapproaches ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġis Ġan Ġenc oder - dec oder Ġarchitecture Ġmodeled Ġby Ġrecurrent Ġneural Ġnetworks . ĠThe Ġenc oder Ġreads Ġthe Ġsource Ġsequence Ġinto Ġa Ġlist Ġof Ġcontinuous - space Ġrepresentations Ġfrom Ġwhich Ġthe Ġdec oder Ġgenerates Ġthe Ġtarget Ġsequence . ĠAn Ġattention Ġmechanism ĠB IB REF 8 Ġis Ġoften Ġused Ġto Ġlocate Ġthe Ġregion Ġof Ġfocus Ġduring Ġdecoding . </s> Ext ractive Ġsystems Ġcreate Ġa Ġsummary Ġby Ġidentifying Ġ( and Ġsubsequently Ġconc aten ating ) Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument . ĠA Ġfew Ġrecent Ġapproaches ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġconceptual ize Ġextract ive Ġsummar ization Ġas Ġa Ġsequence Ġlabeling Ġtask Ġin Ġwhich Ġeach Ġlabel Ġspecifies Ġwhether Ġeach Ġdocument Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠEx isting Ġmodels Ġrely Ġon Ġrecurrent Ġneural Ġnetworks Ġto Ġderive Ġa Ġmeaning Ġrepresentation Ġof Ġthe Ġdocument Ġwhich Ġis Ġthen Ġused Ġto Ġlabel Ġeach Ġsentence , Ġtaking Ġthe Ġpreviously Ġlabeled Ġsentences Ġinto Ġaccount . ĠThese Ġmodels Ġare Ġtypically Ġtrained Ġusing Ġcross - ent ropy Ġloss Ġin Ġorder Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġand Ġdo Ġnot Ġnecessarily Ġlearn Ġto Ġrank Ġsentences Ġbased Ġon Ġtheir Ġimportance Ġdue Ġto Ġthe Ġabsence Ġof Ġa Ġranking - based Ġobjective . ĠAnother Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġmismatch Ġbetween Ġthe Ġlearning Ġobjective Ġand Ġthe Ġevaluation Ġcriterion , Ġnamely ĠR OU GE ĠB IB REF 13 Ġ, Ġwhich Ġtakes Ġthe Ġentire Ġsummary Ġinto Ġaccount . </s> In Ġthis Ġpaper Ġwe Ġargue Ġthat Ġcross - ent ropy Ġtraining Ġis Ġnot Ġoptimal Ġfor Ġextract ive Ġsummar ization . ĠModels Ġtrained Ġthis Ġway Ġare Ġprone Ġto Ġgenerating Ġverb ose Ġsumm aries Ġwith Ġunnecessarily Ġlong Ġsentences Ġand Ġredundant Ġinformation . ĠWe Ġpropose Ġto Ġovercome Ġthese Ġdifficulties Ġby Ġglobally Ġoptimizing Ġthe ĠR OU GE Ġevaluation Ġmetric Ġand Ġlearning Ġto Ġrank Ġsentences Ġfor Ġsummary Ġgeneration Ġthrough Ġa Ġreinforcement Ġlearning Ġobjective . ĠSimilar Ġto Ġprevious Ġwork ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 10 Ġ, Ġour Ġneural Ġsummar ization Ġmodel Ġconsists Ġof Ġa Ġhierarchical Ġdocument Ġenc oder Ġand Ġa Ġhierarchical Ġsentence Ġextract or . ĠDuring Ġtraining , Ġit Ġcombines Ġthe Ġmaximum - like lihood Ġcross - ent ropy Ġloss Ġwith Ġrewards Ġfrom Ġpolicy Ġgradient Ġreinforcement Ġlearning Ġto Ġdirectly Ġoptimize Ġthe Ġevaluation Ġmetric Ġrelevant Ġfor Ġthe Ġsummar ization Ġtask . ĠWe Ġshow Ġthat Ġthis Ġglobal Ġoptimization Ġframework Ġrenders Ġextract ive Ġmodels Ġbetter Ġat Ġdiscriminating Ġamong Ġsentences Ġfor Ġthe Ġfinal Ġsummary ; Ġa Ġsentence Ġis Ġranked Ġhigh Ġfor Ġselection Ġif Ġit Ġoften Ġoccurs Ġin Ġhigh Ġscoring Ġsumm aries . </s> We Ġreport Ġresults Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġnews Ġhighlights Ġdatasets ĠB IB REF 14 Ġwhich Ġhave Ġbeen Ġrecently Ġused Ġas Ġtest bed s Ġfor Ġthe Ġevaluation Ġof Ġneural Ġsummar ization Ġsystems . ĠExperimental Ġresults Ġshow Ġthat Ġwhen Ġevaluated Ġautomatically Ġ( in Ġterms Ġof ĠR OU GE ), Ġour Ġmodel Ġoutper forms Ġstate - of - the - art Ġextract ive Ġand Ġabstract ive Ġsystems . ĠWe Ġalso Ġconduct Ġtwo Ġhuman Ġevaluations Ġin Ġorder Ġto Ġassess Ġ( a ) Ġwhich Ġtype Ġof Ġsummary Ġparticipants Ġprefer Ġ( we Ġcompare Ġextract ive Ġand Ġabstract ive Ġsystems ) Ġand Ġ( b ) Ġhow Ġmuch Ġkey Ġinformation Ġfrom Ġthe Ġdocument Ġis Ġpreserved Ġin Ġthe Ġsummary Ġ( we Ġask Ġparticipants Ġto Ġanswer Ġquestions Ġpertaining Ġto Ġthe Ġcontent Ġin Ġthe Ġdocument Ġby Ġreading Ġsystem Ġsumm aries ). ĠBoth Ġevaluations Ġoverwhelmingly Ġshow Ġthat Ġhuman Ġsubjects Ġfind Ġour Ġsumm aries Ġmore Ġinformative Ġand Ġcomplete . </s> Our Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree - fold : Ġa Ġnovel Ġapplication Ġof Ġreinforcement Ġlearning Ġto Ġsentence Ġranking Ġfor Ġextract ive Ġsummar ization ; Ġcorrobor ated Ġby Ġanalysis Ġand Ġempirical Ġresults Ġshowing Ġthat Ġcross - ent ropy Ġtraining Ġis Ġnot Ġwell - su ited Ġto Ġthe Ġsummar ization Ġtask ; Ġand Ġlarge Ġscale Ġuser Ġstudies Ġfollowing Ġtwo Ġevaluation Ġparad ig ms Ġwhich Ġdemonstrate Ġthat Ġstate - of - the - art Ġabstract ive Ġsystems Ġlag Ġbehind Ġextract ive Ġones Ġwhen Ġthe Ġlatter Ġare Ġglobally Ġtrained . </s> Sum mar ization Ġas ĠSent ence ĠRanking </s> Given Ġa Ġdocument ĠD Ġconsisting Ġof Ġa Ġsequence Ġof Ġsentences ĠIN LINE FORM 0 Ġ, Ġan Ġextract ive Ġsummar izer Ġaims Ġto Ġproduce Ġa Ġsummary ĠIN LINE FORM 1 Ġby Ġselecting ĠIN LINE FORM 2 Ġsentences Ġfrom ĠD Ġ( where ĠIN LINE FORM 3 Ġ). ĠFor Ġeach Ġsentence ĠIN LINE FORM 4 Ġ, Ġwe Ġpredict Ġa Ġlabel ĠIN LINE FORM 5 Ġ( where Ġ1 Ġmeans Ġthat ĠIN LINE FORM 6 Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary ) Ġand Ġassign Ġa Ġscore ĠIN LINE FORM 7 Ġquant ifying ĠIN LINE FORM 8 Ġ' s Ġrelevance Ġto Ġthe Ġsummary . ĠThe Ġmodel Ġlearns Ġto Ġassign ĠIN LINE FORM 9 Ġwhen Ġsentence ĠIN LINE FORM 10 Ġis Ġmore Ġrelevant Ġthan ĠIN LINE FORM 11 Ġ. ĠModel Ġparameters Ġare Ġden oted Ġby ĠIN LINE FORM 12 Ġ. ĠWe Ġestimate ĠIN LINE FORM 13 Ġusing Ġa Ġneural Ġnetwork Ġmodel Ġand Ġassemble Ġa Ġsummary ĠIN LINE FORM 14 Ġby Ġselecting ĠIN LINE FORM 15 Ġsentences Ġwith Ġtop ĠIN LINE FORM 16 Ġscores . </s> Our Ġarchitecture Ġresembles Ġthose Ġpreviously Ġproposed Ġin Ġthe Ġliterature ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠThe Ġmain Ġcomponents Ġinclude Ġa Ġsentence Ġenc oder , Ġa Ġdocument Ġenc oder , Ġand Ġa Ġsentence Ġextract or Ġ( see Ġthe Ġleft Ġblock Ġof ĠFigure ĠFIG REF 2 Ġ) Ġwhich Ġwe Ġdescribe Ġin Ġmore Ġdetail Ġbelow . </s> The ĠPit falls Ġof ĠCross - Ent ropy ĠLoss </s> Previous Ġwork Ġoptim izes Ġsummar ization Ġmodels Ġby Ġmaximizing ĠIN LINE FORM 0 Ġ, Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġy Ġ= ĠIN LINE FORM 1 Ġfor Ġsentences ĠIN LINE FORM 2 Ġ, Ġgiven Ġdocument ĠD Ġand Ġmodel Ġparameters ĠIN LINE FORM 3 Ġ. ĠThis Ġobjective Ġcan Ġbe Ġachieved Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġloss Ġat Ġeach Ġdecoding Ġstep : ĠDIS PLAY FORM 0 </s> Cross - ent ropy Ġtraining Ġleads Ġto Ġtwo Ġkinds Ġof Ġdiscrepancies Ġin Ġthe Ġmodel . ĠThe Ġfirst Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġdisconnect Ġbetween Ġthe Ġtask Ġdefinition Ġand Ġthe Ġtraining Ġobjective . ĠWhile ĠM LE Ġin ĠEqu ation Ġ( ĠEQ REF 7 Ġ) Ġaims Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels , Ġthe Ġmodel Ġis Ġ( a ) Ġexpected Ġto Ġrank Ġsentences Ġto Ġgenerate Ġa Ġsummary Ġand Ġ( b ) Ġevaluated Ġusing ĠIN LINE FORM 0 Ġat Ġtest Ġtime . ĠThe Ġsecond Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġreliance Ġon Ġground - truth Ġlabels . ĠDocument Ġcollections Ġfor Ġtraining Ġsummar ization Ġsystems Ġdo Ġnot Ġnaturally Ġcontain Ġlabels Ġindicating Ġwhich Ġsentences Ġshould Ġbe Ġextracted . ĠInstead , Ġthey Ġare Ġtypically Ġaccompanied Ġby Ġabstract ive Ġsumm aries Ġfrom Ġwhich Ġsentence - level Ġlabels Ġare Ġextrap olated . Ġj p - acl 16 Ġfollow Ġwoods end - acl 10 Ġin Ġadopting Ġa Ġrule - based Ġmethod Ġwhich Ġassigns Ġlabels Ġto Ġeach Ġsentence Ġin Ġthe Ġdocument Ġindividually Ġbased Ġon Ġtheir Ġsemantic Ġcorrespondence Ġwith Ġthe Ġgold Ġsummary Ġ( see Ġthe Ġfourth Ġcolumn Ġin ĠTable ĠT AB REF 6 Ġ). ĠAn Ġalternative Ġmethod ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ, ĠB IB REF 10 Ġidentifies Ġthe Ġset Ġof Ġsentences Ġwhich Ġcollectively Ġgives Ġthe Ġhighest ĠR OU GE Ġwith Ġrespect Ġto Ġthe Ġgold Ġsummary . ĠSent ences Ġin Ġthis Ġset Ġare Ġlabeled Ġwith Ġ1 Ġand Ġ0 Ġotherwise Ġ( see Ġthe Ġcolumn Ġ5 Ġin ĠTable ĠT AB REF 6 Ġ). </s> Label ing Ġsentences Ġindividually Ġoften Ġgenerates Ġtoo Ġmany Ġpositive Ġlabels Ġcausing Ġthe Ġmodel Ġto Ġover fit Ġthe Ġdata . ĠFor Ġexample , Ġthe Ġdocument Ġin ĠTable ĠT AB REF 6 Ġhas Ġ12 Ġpositively Ġlabeled Ġsentences Ġout Ġof Ġ31 Ġin Ġtotal Ġ( only Ġfirst Ġ10 Ġare Ġshown ). ĠCollective Ġlabels Ġpresent Ġa Ġbetter Ġalternative Ġsince Ġthey Ġonly Ġper tain Ġto Ġthe Ġfew Ġsentences Ġdeemed Ġmost Ġsuitable Ġto Ġform Ġthe Ġsummary . ĠHowever , Ġa Ġmodel Ġtrained Ġwith Ġcross - ent ropy Ġloss Ġon Ġcollective Ġlabels Ġwill Ġunder fit Ġthe Ġdata Ġas Ġit Ġwill Ġonly Ġmaximize Ġprobabilities ĠIN LINE FORM 0 Ġfor Ġsentences Ġin Ġthis Ġset Ġ( e . g ., Ġsentences ĠIN LINE FORM 1 Ġin ĠTable ĠT AB REF 6 Ġ) Ġand Ġignore Ġall Ġother Ġsentences . ĠWe Ġfound Ġthat Ġthere Ġare Ġmany Ġcandidate Ġsumm aries Ġwith Ġhigh ĠR OU GE Ġscores Ġwhich Ġcould Ġbe Ġconsidered Ġduring Ġtraining . </s> Table ĠT AB REF 6 Ġ( last Ġcolumn ) Ġshows Ġcandidate Ġsumm aries Ġranked Ġaccording Ġto Ġthe Ġmean Ġof ĠR OU GE - 1 , ĠR OU GE - 2 , Ġand ĠR OU GE - L ĠF ĠIN LINE FORM 0 Ġscores . ĠInterestingly , Ġmultiple Ġtop Ġranked Ġsumm aries Ġhave Ġreasonably Ġhigh ĠR OU GE Ġscores . ĠFor Ġexample , Ġthe Ġaverage ĠR OU GE Ġfor Ġthe Ġsumm aries Ġranked Ġsecond Ġ( 0 , 13 ), Ġthird Ġ( 11 , 13 ), Ġand Ġfourth Ġ( 0 , 1 , 13 ) Ġis Ġ57 . 5 %, Ġ57 . 2 %, Ġand Ġ57 . 1 %, Ġand Ġall Ġtop Ġ16 Ġsumm aries Ġhave ĠR OU GE Ġscores Ġmore Ġor Ġequal Ġto Ġ50 %. ĠA Ġfew Ġsentences Ġare Ġindicative Ġof Ġimportant Ġcontent Ġand Ġappear Ġfrequently Ġin Ġthe Ġsumm aries : Ġsentence Ġ13 Ġoccurs Ġin Ġall Ġsumm aries Ġexcept Ġone , Ġwhile Ġsentence Ġ0 Ġappears Ġin Ġseveral Ġsumm aries Ġtoo . ĠAlso Ġnote Ġthat Ġsumm aries Ġ( 11 , 13 ) Ġand Ġ( 1 , 13 ) Ġyield Ġbetter ĠR OU GE Ġscores Ġcompared Ġto Ġlonger Ġsumm aries , Ġand Ġmay Ġbe Ġas Ġinformative , Ġyet Ġmore Ġconcise , Ġalternatives . </s> These Ġdiscrepancies Ġrender Ġthe Ġmodel Ġless Ġefficient Ġat Ġranking Ġsentences Ġfor Ġthe Ġsummar ization Ġtask . ĠInstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels , Ġwe Ġcould Ġtrain Ġthe Ġmodel Ġto Ġpredict Ġthe Ġindividual ĠR OU GE Ġscore Ġfor Ġeach Ġsentence Ġin Ġthe Ġdocument Ġand Ġthen Ġselect Ġthe Ġtop ĠIN LINE FORM 0 Ġsentences Ġwith Ġhighest Ġscores . ĠBut Ġsentences Ġwith Ġindividual ĠR OU GE Ġscores Ġdo Ġnot Ġnecessarily Ġlead Ġto Ġa Ġhigh Ġscoring Ġsummary , Ġe . g ., Ġthey Ġmay Ġconvey Ġoverlapping Ġcontent Ġand Ġform Ġverb ose Ġand Ġredundant Ġsumm aries . ĠFor Ġexample , Ġsentence Ġ3 , Ġdespite Ġhaving Ġa Ġhigh Ġindividual ĠR OU GE Ġscore Ġ( 35 . 6 %), Ġdoes Ġnot Ġoccur Ġin Ġany Ġof Ġthe Ġtop Ġ5 Ġsumm aries . ĠWe Ġnext Ġexplain Ġhow Ġwe Ġaddress Ġthese Ġissues Ġusing Ġreinforcement Ġlearning . </s> Sent ence ĠRanking Ġwith ĠRein forcement ĠLearning </s> Re in forcement Ġlearning ĠB IB REF 28 Ġhas Ġbeen Ġproposed Ġas Ġa Ġway Ġof Ġtraining Ġsequence - to - sequence Ġgeneration Ġmodels Ġin Ġorder Ġto Ġdirectly Ġoptimize Ġthe Ġmetric Ġused Ġat Ġtest Ġtime , Ġe . g ., ĠB LE U Ġor ĠR OU GE ĠB IB REF 29 Ġ. ĠWe Ġadapt Ġreinforcement Ġlearning Ġto Ġour Ġformulation Ġof Ġextract ive Ġsummar ization Ġto Ġrank Ġsentences Ġfor Ġsummary Ġgeneration . ĠWe Ġpropose Ġan Ġobjective Ġfunction Ġthat Ġcombines Ġthe Ġmaximum - like lihood Ġcross - ent ropy Ġloss Ġwith Ġrewards Ġfrom Ġpolicy Ġgradient Ġreinforcement Ġlearning Ġto Ġglobally Ġoptimize ĠR OU GE . ĠOur Ġtraining Ġalgorithm Ġallows Ġto Ġexplore Ġthe Ġspace Ġof Ġpossible Ġsumm aries , Ġmaking Ġour Ġmodel Ġmore Ġrobust Ġto Ġunseen Ġdata . ĠAs Ġa Ġresult , Ġreinforcement Ġlearning Ġhelps Ġextract ive Ġsummar ization Ġin Ġtwo Ġways : Ġ( a ) Ġit Ġdirectly Ġoptim izes Ġthe Ġevaluation Ġmetric Ġinstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġand Ġ( b ) Ġit Ġmakes Ġour Ġmodel Ġbetter Ġat Ġdiscriminating Ġamong Ġsentences ; Ġa Ġsentence Ġis Ġranked Ġhigh Ġfor Ġselection Ġif Ġit Ġoften Ġoccurs Ġin Ġhigh Ġscoring Ġsumm aries . </s> Policy ĠLearning </s> We Ġcast Ġthe Ġneural Ġsummar ization Ġmodel Ġintroduced Ġin ĠFigure ĠFIG REF 2 Ġin Ġthe ĠRein forcement ĠLearning Ġparadigm ĠB IB REF 28 Ġ. ĠAccordingly , Ġthe Ġmodel Ġcan Ġbe Ġviewed Ġas Ġan ĠâĢ ľ agent âĢ Ŀ Ġwhich Ġinteracts Ġwith Ġan ĠâĢ ľ environment âĢ Ŀ Ġconsisting Ġof Ġdocuments . ĠAt Ġfirst , Ġthe Ġagent Ġis Ġinitialized Ġrandomly , Ġit Ġreads Ġdocument ĠD Ġand Ġpredicts Ġa Ġrelevance Ġscore Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġusing ĠâĢ ľ policy âĢ Ŀ ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġare Ġmodel Ġparameters . ĠOnce Ġthe Ġagent Ġis Ġdone Ġreading Ġthe Ġdocument , Ġa Ġsummary Ġwith Ġlabels ĠIN LINE FORM 3 Ġis Ġsampled Ġout Ġof Ġthe Ġranked Ġsentences . ĠThe Ġagent Ġis Ġthen Ġgiven Ġa ĠâĢ ľ re ward âĢ Ŀ ĠIN LINE FORM 4 Ġcomm ens urate Ġwith Ġhow Ġwell Ġthe Ġextract Ġresembles Ġthe Ġgold - standard Ġsummary . ĠSpecifically , Ġas Ġreward Ġfunction Ġwe Ġuse Ġmean ĠF ĠIN LINE FORM 5 Ġof ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, Ġand ĠIN LINE FORM 8 Ġ. ĠUn ig ram Ġand Ġbig ram Ġoverlap Ġ( ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ) Ġare Ġmeant Ġto Ġassess Ġinform at iveness , Ġwhereas Ġthe Ġlongest Ġcommon Ġsubsequ ence Ġ( ĠIN LINE FORM 11 Ġ) Ġis Ġmeant Ġto Ġassess Ġflu ency . ĠWe Ġupdate Ġthe Ġagent Ġusing Ġthe ĠRE IN FOR CE Ġalgorithm ĠB IB REF 15 Ġwhich Ġaims Ġto Ġminimize Ġthe Ġnegative Ġexpected Ġreward : ĠDIS PLAY FORM 0 </s> where , ĠIN LINE FORM 0 Ġstands Ġfor ĠIN LINE FORM 1 Ġ. ĠRE IN FOR CE Ġis Ġbased Ġon Ġthe Ġobservation Ġthat Ġthe Ġexpected Ġgradient Ġof Ġa Ġnon - different iable Ġreward Ġfunction Ġ( R OU GE , Ġin Ġour Ġcase ) Ġcan Ġbe Ġcomputed Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> While ĠM LE Ġin ĠEqu ation Ġ( ĠEQ REF 7 Ġ) Ġaims Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġtraining Ġdata , Ġthe Ġobjective Ġin ĠEqu ation Ġ( ĠEQ REF 9 Ġ) Ġlearns Ġto Ġdiscriminate Ġamong Ġsentences Ġwith Ġrespect Ġto Ġhow Ġoften Ġthey Ġoccur Ġin Ġhigh Ġscoring Ġsumm aries . </s> Training Ġwith ĠHigh ĠProb ability ĠSam ples </s> Com puting Ġthe Ġexpectation Ġterm Ġin ĠEqu ation Ġ( ĠEQ REF 10 Ġ) Ġis Ġprohib itive , Ġsince Ġthere Ġis Ġa Ġlarge Ġnumber Ġof Ġpossible Ġextracts . ĠIn Ġpractice , Ġwe Ġapproximate Ġthe Ġexpected Ġgradient Ġusing Ġa Ġsingle Ġsample ĠIN LINE FORM 0 Ġfrom ĠIN LINE FORM 1 Ġfor Ġeach Ġtraining Ġexample Ġin Ġa Ġbatch : ĠDIS PLAY FORM 0 </s> Present ed Ġin Ġits Ġoriginal Ġform , Ġthe ĠRE IN FOR CE Ġalgorithm Ġstarts Ġlearning Ġwith Ġa Ġrandom Ġpolicy Ġwhich Ġcan Ġmake Ġmodel Ġtraining Ġchallenging Ġfor Ġcomplex Ġtasks Ġlike Ġours Ġwhere Ġa Ġsingle Ġdocument Ġcan Ġgive Ġrise Ġto Ġa Ġvery Ġlarge Ġnumber Ġof Ġcandidate Ġsumm aries . ĠWe Ġtherefore Ġlimit Ġthe Ġsearch Ġspace Ġof ĠIN LINE FORM 0 Ġin ĠEqu ation Ġ( ĠEQ REF 12 Ġ) Ġto Ġthe Ġset Ġof Ġlargest Ġprobability Ġsamples ĠIN LINE FORM 1 Ġ. ĠWe Ġapproximate ĠIN LINE FORM 2 Ġby Ġthe ĠIN LINE FORM 3 Ġextracts Ġwhich Ġreceive Ġhighest ĠR OU GE Ġscores . ĠMore Ġconcrete ly , Ġwe Ġassemble Ġcandidate Ġsumm aries Ġefficiently Ġby Ġfirst Ġselecting ĠIN LINE FORM 4 Ġsentences Ġfrom Ġthe Ġdocument Ġwhich Ġon Ġtheir Ġown Ġhave Ġhigh ĠR OU GE Ġscores . ĠWe Ġthen Ġgenerate Ġall Ġpossible Ġcombinations Ġof ĠIN LINE FORM 5 Ġsentences Ġsubject Ġto Ġmaximum Ġlength ĠIN LINE FORM 6 Ġand Ġevaluate Ġthem Ġagainst Ġthe Ġgold Ġsummary . ĠSum m aries Ġare Ġranked Ġaccording Ġto ĠF ĠIN LINE FORM 7 Ġby Ġtaking Ġthe Ġmean Ġof ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, Ġand ĠIN LINE FORM 10 Ġ. ĠIN LINE FORM 11 Ġcontains Ġthese Ġtop ĠIN LINE FORM 12 Ġcandidate Ġsumm aries . ĠDuring Ġtraining , Ġwe Ġsample ĠIN LINE FORM 13 Ġfrom ĠIN LINE FORM 14 Ġinstead Ġof ĠIN LINE FORM 15 Ġ. </s> ran z ato - ar x iv 15 - b ias Ġproposed Ġan Ġalternative Ġto ĠRE IN FOR CE Ġcalled ĠM IX ER Ġ( M ixed ĠIncre mental ĠCross - Ent ropy ĠRein force ) Ġwhich Ġfirst Ġpret r ains Ġthe Ġmodel Ġwith Ġthe Ġcross - ent ropy Ġloss Ġusing Ġground Ġtruth Ġlabels Ġand Ġthen Ġfollows Ġa Ġcurriculum Ġlearning Ġstrategy ĠB IB REF 30 Ġto Ġgradually Ġteach Ġthe Ġmodel Ġto Ġproduce Ġstable Ġpredictions Ġon Ġits Ġown . ĠIn Ġour Ġexperiments ĠM IX ER Ġperformed Ġworse Ġthan Ġthe Ġmodel Ġof Ġn all ap ati 17 Ġjust Ġtrained Ġon Ġcollective Ġlabels . ĠWe Ġconjecture Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġunb ounded Ġnature Ġof Ġour Ġranking Ġproblem . ĠRecall Ġthat Ġour Ġmodel Ġassigns Ġrelevance Ġscores Ġto Ġsentences Ġrather Ġthan Ġwords . ĠThe Ġspace Ġof Ġsent ential Ġrepresentations Ġis Ġvast Ġand Ġfairly Ġuncon str ained Ġcompared Ġto Ġother Ġprediction Ġtasks Ġoperating Ġwith Ġfixed Ġvoc ab ular ies ĠB IB REF 31 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 32 Ġ. ĠMoreover , Ġour Ġapproximation Ġof Ġthe Ġgradient Ġallows Ġthe Ġmodel Ġto Ġconverge Ġmuch Ġfaster Ġto Ġan Ġoptimal Ġpolicy . ĠAdvantage ously , Ġwe Ġdo Ġnot Ġrequire Ġan Ġonline Ġreward Ġestim ator , Ġwe Ġpre - comp ute ĠIN LINE FORM 0 Ġ, Ġwhich Ġleads Ġto Ġa Ġsignificant Ġspeed up Ġduring Ġtraining Ġcompared Ġto ĠM IX ER ĠB IB REF 29 Ġand Ġrelated Ġtraining Ġschemes ĠB IB REF 33 Ġ. </s> Exper imental ĠSetup </s> In Ġthis Ġsection Ġwe Ġpresent Ġour Ġexperimental Ġsetup Ġfor Ġassessing Ġthe Ġperformance Ġof Ġour Ġmodel Ġwhich Ġwe Ġcall ĠRefresh Ġas Ġa Ġshorthand Ġfor ĠRE in F o R ce ment ĠLearning - based ĠExtract ive ĠSum mar ization . ĠWe Ġdescribe Ġour Ġdatasets , Ġdiscuss Ġimplementation Ġdetails , Ġour Ġevaluation Ġprotocol , Ġand Ġthe Ġsystems Ġused Ġfor Ġcomparison . </s> Results </s> We Ġreport Ġresults Ġusing Ġautomatic Ġmetrics Ġin ĠTable ĠT AB REF 20 Ġ. ĠThe Ġtop Ġpart Ġof Ġthe Ġtable Ġcompares ĠRefresh Ġagainst Ġrelated Ġextract ive Ġsystems . ĠThe Ġbottom Ġpart Ġreports Ġthe Ġperformance Ġof Ġabstract ive Ġsystems . ĠWe Ġpresent Ġthree Ġvariants Ġof ĠLead , Ġone Ġis Ġcomputed Ġby Ġourselves Ġand Ġthe Ġother Ġtwo Ġare Ġreported Ġin Ġn all ap ati 17 Ġand Ġsee - acl 17 . ĠNote Ġthat Ġthey Ġvary Ġslightly Ġdue Ġto Ġdifferences Ġin Ġthe Ġpre processing Ġof Ġthe Ġdata . ĠWe Ġreport Ġresults Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġdatasets Ġand Ġtheir Ġcombination Ġ( CNN ĠIN LINE FORM 0 ĠDaily Mail ). </s> Related ĠWork </s> Traditional Ġsummar ization Ġmethods Ġmanually Ġdefine Ġfeatures Ġto Ġrank Ġsentences Ġfor Ġtheir Ġsal ience Ġin Ġorder Ġto Ġidentify Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument Ġor Ġset Ġof Ġdocuments ĠB IB REF 42 Ġ, ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ, ĠB IB REF 45 Ġ, ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠA Ġvast Ġmajority Ġof Ġthese Ġmethods Ġlearn Ġto Ġscore Ġeach Ġsentence Ġindependently ĠB IB REF 48 Ġ, ĠB IB REF 49 Ġ, ĠB IB REF 50 Ġ, ĠB IB REF 51 Ġ, ĠB IB REF 52 Ġ, ĠB IB REF 53 Ġ, ĠB IB REF 54 Ġand Ġa Ġsummary Ġis Ġgenerated Ġby Ġselecting Ġtop - sc ored Ġsentences Ġin Ġa Ġway Ġthat Ġis Ġnot Ġincorporated Ġinto Ġthe Ġlearning Ġprocess . ĠSummary Ġquality Ġcan Ġbe Ġimproved Ġhe ur istically , ĠB IB REF 55 Ġ, Ġvia Ġmax - margin Ġmethods ĠB IB REF 56 Ġ, ĠB IB REF 57 Ġ, Ġor Ġinteger - linear Ġprogramming ĠB IB REF 58 Ġ, ĠB IB REF 59 Ġ, ĠB IB REF 60 Ġ, ĠB IB REF 61 Ġ, ĠB IB REF 62 Ġ. </s> Recent Ġdeep Ġlearning Ġmethods ĠB IB REF 63 Ġ, ĠB IB REF 64 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġlearn Ġcontinuous Ġfeatures Ġwithout Ġany Ġlinguistic Ġpre processing Ġ( e . g ., Ġnamed Ġentities ). ĠLike Ġtraditional Ġmethods , Ġthese Ġapproaches Ġalso Ġsuffer Ġfrom Ġthe Ġmismatch Ġbetween Ġthe Ġlearning Ġobjective Ġand Ġthe Ġevaluation Ġcriterion Ġ( e . g ., ĠR OU GE ) Ġused Ġat Ġthe Ġtest Ġtime . ĠIn Ġcomparison , Ġour Ġneural Ġmodel Ġglobally Ġoptim izes Ġthe ĠR OU GE Ġevaluation Ġmetric Ġthrough Ġa Ġreinforcement Ġlearning Ġobjective : Ġsentences Ġare Ġhighly Ġranked Ġif Ġthey Ġoccur Ġin Ġhighly Ġscoring Ġsumm aries . </s> Re in forcement Ġlearning Ġhas Ġbeen Ġpreviously Ġused Ġin Ġthe Ġcontext Ġof Ġtraditional Ġmulti - document Ġsummar ization Ġas Ġa Ġmeans Ġof Ġselecting Ġa Ġsentence Ġor Ġa Ġsubset Ġof Ġsentences Ġfrom Ġa Ġdocument Ġcluster . ĠRy ang : 2012 Ġcast Ġthe Ġsentence Ġselection Ġtask Ġas Ġa Ġsearch Ġproblem . ĠTheir Ġagent Ġobserves Ġa Ġstate Ġ( e . g ., Ġa Ġcandidate Ġsummary ), Ġexecutes Ġan Ġaction Ġ( a Ġtransition Ġoperation Ġthat Ġproduces Ġa Ġnew Ġstate Ġselecting Ġa Ġnot - yet - selected Ġsentence ), Ġand Ġthen Ġreceives Ġa Ġdelayed Ġreward Ġbased Ġon ĠIN LINE FORM 0 Ġ. ĠFollow - on Ġwork ĠB IB REF 65 Ġextends Ġthis Ġapproach Ġby Ġemploying ĠR OU GE Ġas Ġpart Ġof Ġthe Ġreward Ġfunction , Ġwhile Ġhe ns - gs cl 15 Ġfurther Ġexperiment Ġwith ĠIN LINE FORM 1 Ġ- learning . ĠM olla Al iod : 2017 : AL TA 2017 Ġhas Ġadapt Ġthis Ġapproach Ġto Ġquery - focused Ġsummar ization . ĠOur Ġmodel Ġdiffers Ġfrom Ġthese Ġapproaches Ġboth Ġin Ġapplication Ġand Ġformulation . ĠWe Ġfocus Ġsolely Ġon Ġextract ive Ġsummar ization , Ġin Ġour Ġcase Ġstates Ġare Ġdocuments Ġ( not Ġsumm aries ) Ġand Ġactions Ġare Ġrelevance Ġscores Ġwhich Ġlead Ġto Ġsentence Ġranking Ġ( not Ġsentence - to - sent ence Ġtransitions ). ĠRather Ġthan Ġemploying Ġreinforcement Ġlearning Ġfor Ġsentence Ġselection , Ġour Ġalgorithm Ġperforms Ġsentence Ġranking Ġusing ĠR OU GE Ġas Ġthe Ġreward Ġfunction . </s> The ĠRE IN FOR CE Ġalgorithm ĠB IB REF 15 Ġhas Ġbeen Ġshown Ġto Ġimprove Ġenc oder - dec oder Ġtext - rew rit ing Ġsystems Ġby Ġallowing Ġto Ġdirectly Ġoptimize Ġa Ġnon - different iable Ġobjective ĠB IB REF 29 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 7 Ġor Ġto Ġinject Ġtask - specific Ġconstraints ĠB IB REF 32 Ġ, ĠB IB REF 66 Ġ. ĠHowever , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġattempts Ġto Ġuse Ġreinforcement Ġlearning Ġfor Ġtraining Ġa Ġsentence Ġrank er Ġin Ġthe Ġcontext Ġof Ġextract ive Ġsummar ization . </s> Con clusions </s> In Ġthis Ġwork Ġwe Ġdeveloped Ġan Ġextract ive Ġsummar ization Ġmodel Ġwhich Ġis Ġglobally Ġtrained Ġby Ġoptimizing Ġthe ĠR OU GE Ġevaluation Ġmetric . ĠOur Ġtraining Ġalgorithm Ġexplores Ġthe Ġspace Ġof Ġcandidate Ġsumm aries Ġwhile Ġlearning Ġto Ġoptimize Ġa Ġreward Ġfunction Ġwhich Ġis Ġrelevant Ġfor Ġthe Ġtask Ġat Ġhand . ĠExperimental Ġresults Ġshow Ġthat Ġreinforcement Ġlearning Ġoffers Ġa Ġgreat Ġmeans Ġto Ġsteer Ġour Ġmodel Ġtowards Ġgenerating Ġinformative , Ġfluent , Ġand Ġconcise Ġsumm aries Ġoutper forming Ġstate - of - the - art Ġextract ive Ġand Ġabstract ive Ġsystems Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġdatasets . ĠIn Ġthe Ġfuture Ġwe Ġwould Ġlike Ġto Ġfocus Ġon Ġsmaller Ġdiscourse Ġunits ĠB IB REF 67 Ġrather Ġthan Ġindividual Ġsentences , Ġmodeling Ġcompression Ġand Ġextraction Ġjointly .
Introduction Automatic summarization has enjoyed wide popularity in natural language processing due to its potential for various information access applications. Examples include tools which aid users navigate and digest web content (e.g., news, social media, product reviews), question answering, and personalized recommendation engines. Single document summarization — the task of producing a shorter version of a document while preserving its information content — is perhaps the most basic of summarization tasks that have been identified over the years (see BIBREF0 , BIBREF0 for a comprehensive overview). Modern approaches to single document summarization are data-driven, taking advantage of the success of neural network architectures and their ability to learn continuous features without recourse to preprocessing tools or linguistic annotations. Abstractive summarization involves various text rewriting operations (e.g., substitution, deletion, reordering) and has been recently framed as a sequence-to-sequence problem BIBREF1 . Central in most approaches BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 is an encoder-decoder architecture modeled by recurrent neural networks. The encoder reads the source sequence into a list of continuous-space representations from which the decoder generates the target sequence. An attention mechanism BIBREF8 is often used to locate the region of focus during decoding. Extractive systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document. A few recent approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 conceptualize extractive summarization as a sequence labeling task in which each label specifies whether each document sentence should be included in the summary. Existing models rely on recurrent neural networks to derive a meaning representation of the document which is then used to label each sentence, taking the previously labeled sentences into account. These models are typically trained using cross-entropy loss in order to maximize the likelihood of the ground-truth labels and do not necessarily learn to rank sentences based on their importance due to the absence of a ranking-based objective. Another discrepancy comes from the mismatch between the learning objective and the evaluation criterion, namely ROUGE BIBREF13 , which takes the entire summary into account. In this paper we argue that cross-entropy training is not optimal for extractive summarization. Models trained this way are prone to generating verbose summaries with unnecessarily long sentences and redundant information. We propose to overcome these difficulties by globally optimizing the ROUGE evaluation metric and learning to rank sentences for summary generation through a reinforcement learning objective. Similar to previous work BIBREF9 , BIBREF11 , BIBREF10 , our neural summarization model consists of a hierarchical document encoder and a hierarchical sentence extractor. During training, it combines the maximum-likelihood cross-entropy loss with rewards from policy gradient reinforcement learning to directly optimize the evaluation metric relevant for the summarization task. We show that this global optimization framework renders extractive models better at discriminating among sentences for the final summary; a sentence is ranked high for selection if it often occurs in high scoring summaries. We report results on the CNN and DailyMail news highlights datasets BIBREF14 which have been recently used as testbeds for the evaluation of neural summarization systems. Experimental results show that when evaluated automatically (in terms of ROUGE), our model outperforms state-of-the-art extractive and abstractive systems. We also conduct two human evaluations in order to assess (a) which type of summary participants prefer (we compare extractive and abstractive systems) and (b) how much key information from the document is preserved in the summary (we ask participants to answer questions pertaining to the content in the document by reading system summaries). Both evaluations overwhelmingly show that human subjects find our summaries more informative and complete. Our contributions in this work are three-fold: a novel application of reinforcement learning to sentence ranking for extractive summarization; corroborated by analysis and empirical results showing that cross-entropy training is not well-suited to the summarization task; and large scale user studies following two evaluation paradigms which demonstrate that state-of-the-art abstractive systems lag behind extractive ones when the latter are globally trained. Summarization as Sentence Ranking Given a document D consisting of a sequence of sentences INLINEFORM0 , an extractive summarizer aims to produce a summary INLINEFORM1 by selecting INLINEFORM2 sentences from D (where INLINEFORM3 ). For each sentence INLINEFORM4 , we predict a label INLINEFORM5 (where 1 means that INLINEFORM6 should be included in the summary) and assign a score INLINEFORM7 quantifying INLINEFORM8 's relevance to the summary. The model learns to assign INLINEFORM9 when sentence INLINEFORM10 is more relevant than INLINEFORM11 . Model parameters are denoted by INLINEFORM12 . We estimate INLINEFORM13 using a neural network model and assemble a summary INLINEFORM14 by selecting INLINEFORM15 sentences with top INLINEFORM16 scores. Our architecture resembles those previously proposed in the literature BIBREF9 , BIBREF10 , BIBREF11 . The main components include a sentence encoder, a document encoder, and a sentence extractor (see the left block of Figure FIGREF2 ) which we describe in more detail below. The Pitfalls of Cross-Entropy Loss Previous work optimizes summarization models by maximizing INLINEFORM0 , the likelihood of the ground-truth labels y = INLINEFORM1 for sentences INLINEFORM2 , given document D and model parameters INLINEFORM3 . This objective can be achieved by minimizing the cross-entropy loss at each decoding step: DISPLAYFORM0 Cross-entropy training leads to two kinds of discrepancies in the model. The first discrepancy comes from the disconnect between the task definition and the training objective. While MLE in Equation ( EQREF7 ) aims to maximize the likelihood of the ground-truth labels, the model is (a) expected to rank sentences to generate a summary and (b) evaluated using INLINEFORM0 at test time. The second discrepancy comes from the reliance on ground-truth labels. Document collections for training summarization systems do not naturally contain labels indicating which sentences should be extracted. Instead, they are typically accompanied by abstractive summaries from which sentence-level labels are extrapolated. jp-acl16 follow woodsend-acl10 in adopting a rule-based method which assigns labels to each sentence in the document individually based on their semantic correspondence with the gold summary (see the fourth column in Table TABREF6 ). An alternative method BIBREF26 , BIBREF27 , BIBREF10 identifies the set of sentences which collectively gives the highest ROUGE with respect to the gold summary. Sentences in this set are labeled with 1 and 0 otherwise (see the column 5 in Table TABREF6 ). Labeling sentences individually often generates too many positive labels causing the model to overfit the data. For example, the document in Table TABREF6 has 12 positively labeled sentences out of 31 in total (only first 10 are shown). Collective labels present a better alternative since they only pertain to the few sentences deemed most suitable to form the summary. However, a model trained with cross-entropy loss on collective labels will underfit the data as it will only maximize probabilities INLINEFORM0 for sentences in this set (e.g., sentences INLINEFORM1 in Table TABREF6 ) and ignore all other sentences. We found that there are many candidate summaries with high ROUGE scores which could be considered during training. Table TABREF6 (last column) shows candidate summaries ranked according to the mean of ROUGE-1, ROUGE-2, and ROUGE-L F INLINEFORM0 scores. Interestingly, multiple top ranked summaries have reasonably high ROUGE scores. For example, the average ROUGE for the summaries ranked second (0,13), third (11,13), and fourth (0,1,13) is 57.5%, 57.2%, and 57.1%, and all top 16 summaries have ROUGE scores more or equal to 50%. A few sentences are indicative of important content and appear frequently in the summaries: sentence 13 occurs in all summaries except one, while sentence 0 appears in several summaries too. Also note that summaries (11,13) and (1,13) yield better ROUGE scores compared to longer summaries, and may be as informative, yet more concise, alternatives. These discrepancies render the model less efficient at ranking sentences for the summarization task. Instead of maximizing the likelihood of the ground-truth labels, we could train the model to predict the individual ROUGE score for each sentence in the document and then select the top INLINEFORM0 sentences with highest scores. But sentences with individual ROUGE scores do not necessarily lead to a high scoring summary, e.g., they may convey overlapping content and form verbose and redundant summaries. For example, sentence 3, despite having a high individual ROUGE score (35.6%), does not occur in any of the top 5 summaries. We next explain how we address these issues using reinforcement learning. Sentence Ranking with Reinforcement Learning Reinforcement learning BIBREF28 has been proposed as a way of training sequence-to-sequence generation models in order to directly optimize the metric used at test time, e.g., BLEU or ROUGE BIBREF29 . We adapt reinforcement learning to our formulation of extractive summarization to rank sentences for summary generation. We propose an objective function that combines the maximum-likelihood cross-entropy loss with rewards from policy gradient reinforcement learning to globally optimize ROUGE. Our training algorithm allows to explore the space of possible summaries, making our model more robust to unseen data. As a result, reinforcement learning helps extractive summarization in two ways: (a) it directly optimizes the evaluation metric instead of maximizing the likelihood of the ground-truth labels and (b) it makes our model better at discriminating among sentences; a sentence is ranked high for selection if it often occurs in high scoring summaries. Policy Learning We cast the neural summarization model introduced in Figure FIGREF2 in the Reinforcement Learning paradigm BIBREF28 . Accordingly, the model can be viewed as an “agent” which interacts with an “environment” consisting of documents. At first, the agent is initialized randomly, it reads document D and predicts a relevance score for each sentence INLINEFORM0 using “policy” INLINEFORM1 , where INLINEFORM2 are model parameters. Once the agent is done reading the document, a summary with labels INLINEFORM3 is sampled out of the ranked sentences. The agent is then given a “reward” INLINEFORM4 commensurate with how well the extract resembles the gold-standard summary. Specifically, as reward function we use mean F INLINEFORM5 of INLINEFORM6 , INLINEFORM7 , and INLINEFORM8 . Unigram and bigram overlap ( INLINEFORM9 and INLINEFORM10 ) are meant to assess informativeness, whereas the longest common subsequence ( INLINEFORM11 ) is meant to assess fluency. We update the agent using the REINFORCE algorithm BIBREF15 which aims to minimize the negative expected reward: DISPLAYFORM0 where, INLINEFORM0 stands for INLINEFORM1 . REINFORCE is based on the observation that the expected gradient of a non-differentiable reward function (ROUGE, in our case) can be computed as follows: DISPLAYFORM0 While MLE in Equation ( EQREF7 ) aims to maximize the likelihood of the training data, the objective in Equation ( EQREF9 ) learns to discriminate among sentences with respect to how often they occur in high scoring summaries. Training with High Probability Samples Computing the expectation term in Equation ( EQREF10 ) is prohibitive, since there is a large number of possible extracts. In practice, we approximate the expected gradient using a single sample INLINEFORM0 from INLINEFORM1 for each training example in a batch: DISPLAYFORM0 Presented in its original form, the REINFORCE algorithm starts learning with a random policy which can make model training challenging for complex tasks like ours where a single document can give rise to a very large number of candidate summaries. We therefore limit the search space of INLINEFORM0 in Equation ( EQREF12 ) to the set of largest probability samples INLINEFORM1 . We approximate INLINEFORM2 by the INLINEFORM3 extracts which receive highest ROUGE scores. More concretely, we assemble candidate summaries efficiently by first selecting INLINEFORM4 sentences from the document which on their own have high ROUGE scores. We then generate all possible combinations of INLINEFORM5 sentences subject to maximum length INLINEFORM6 and evaluate them against the gold summary. Summaries are ranked according to F INLINEFORM7 by taking the mean of INLINEFORM8 , INLINEFORM9 , and INLINEFORM10 . INLINEFORM11 contains these top INLINEFORM12 candidate summaries. During training, we sample INLINEFORM13 from INLINEFORM14 instead of INLINEFORM15 . ranzato-arxiv15-bias proposed an alternative to REINFORCE called MIXER (Mixed Incremental Cross-Entropy Reinforce) which first pretrains the model with the cross-entropy loss using ground truth labels and then follows a curriculum learning strategy BIBREF30 to gradually teach the model to produce stable predictions on its own. In our experiments MIXER performed worse than the model of nallapati17 just trained on collective labels. We conjecture that this is due to the unbounded nature of our ranking problem. Recall that our model assigns relevance scores to sentences rather than words. The space of sentential representations is vast and fairly unconstrained compared to other prediction tasks operating with fixed vocabularies BIBREF31 , BIBREF7 , BIBREF32 . Moreover, our approximation of the gradient allows the model to converge much faster to an optimal policy. Advantageously, we do not require an online reward estimator, we pre-compute INLINEFORM0 , which leads to a significant speedup during training compared to MIXER BIBREF29 and related training schemes BIBREF33 . Experimental Setup In this section we present our experimental setup for assessing the performance of our model which we call Refresh as a shorthand for REinFoRcement Learning-based Extractive Summarization. We describe our datasets, discuss implementation details, our evaluation protocol, and the systems used for comparison. Results We report results using automatic metrics in Table TABREF20 . The top part of the table compares Refresh against related extractive systems. The bottom part reports the performance of abstractive systems. We present three variants of Lead, one is computed by ourselves and the other two are reported in nallapati17 and see-acl17. Note that they vary slightly due to differences in the preprocessing of the data. We report results on the CNN and DailyMail datasets and their combination (CNN INLINEFORM0 DailyMail). Related Work Traditional summarization methods manually define features to rank sentences for their salience in order to identify the most important sentences in a document or set of documents BIBREF42 , BIBREF43 , BIBREF44 , BIBREF45 , BIBREF46 , BIBREF47 . A vast majority of these methods learn to score each sentence independently BIBREF48 , BIBREF49 , BIBREF50 , BIBREF51 , BIBREF52 , BIBREF53 , BIBREF54 and a summary is generated by selecting top-scored sentences in a way that is not incorporated into the learning process. Summary quality can be improved heuristically, BIBREF55 , via max-margin methods BIBREF56 , BIBREF57 , or integer-linear programming BIBREF58 , BIBREF59 , BIBREF60 , BIBREF61 , BIBREF62 . Recent deep learning methods BIBREF63 , BIBREF64 , BIBREF9 , BIBREF10 learn continuous features without any linguistic preprocessing (e.g., named entities). Like traditional methods, these approaches also suffer from the mismatch between the learning objective and the evaluation criterion (e.g., ROUGE) used at the test time. In comparison, our neural model globally optimizes the ROUGE evaluation metric through a reinforcement learning objective: sentences are highly ranked if they occur in highly scoring summaries. Reinforcement learning has been previously used in the context of traditional multi-document summarization as a means of selecting a sentence or a subset of sentences from a document cluster. Ryang:2012 cast the sentence selection task as a search problem. Their agent observes a state (e.g., a candidate summary), executes an action (a transition operation that produces a new state selecting a not-yet-selected sentence), and then receives a delayed reward based on INLINEFORM0 . Follow-on work BIBREF65 extends this approach by employing ROUGE as part of the reward function, while hens-gscl15 further experiment with INLINEFORM1 -learning. MollaAliod:2017:ALTA2017 has adapt this approach to query-focused summarization. Our model differs from these approaches both in application and formulation. We focus solely on extractive summarization, in our case states are documents (not summaries) and actions are relevance scores which lead to sentence ranking (not sentence-to-sentence transitions). Rather than employing reinforcement learning for sentence selection, our algorithm performs sentence ranking using ROUGE as the reward function. The REINFORCE algorithm BIBREF15 has been shown to improve encoder-decoder text-rewriting systems by allowing to directly optimize a non-differentiable objective BIBREF29 , BIBREF31 , BIBREF7 or to inject task-specific constraints BIBREF32 , BIBREF66 . However, we are not aware of any attempts to use reinforcement learning for training a sentence ranker in the context of extractive summarization. Conclusions In this work we developed an extractive summarization model which is globally trained by optimizing the ROUGE evaluation metric. Our training algorithm explores the space of candidate summaries while learning to optimize a reward function which is relevant for the task at hand. Experimental results show that reinforcement learning offers a great means to steer our model towards generating informative, fluent, and concise summaries outperforming state-of-the-art extractive and abstractive systems on the CNN and DailyMail datasets. In the future we would like to focus on smaller discourse units BIBREF67 rather than individual sentences, modeling compression and extraction jointly.
What is their ROUGE score?
5152b78f5dfee26f1b13f221c1405ffa9b9ba3a4
What Ġis Ġtheir ĠR OU GE Ġscore ?
Un answer able
<s> What Ġis Ġtheir ĠR OU GE Ġscore ? </s> Introduction </s> Aut omatic Ġsummar ization Ġhas Ġenjoyed Ġwide Ġpopularity Ġin Ġnatural Ġlanguage Ġprocessing Ġdue Ġto Ġits Ġpotential Ġfor Ġvarious Ġinformation Ġaccess Ġapplications . ĠExamples Ġinclude Ġtools Ġwhich Ġaid Ġusers Ġnavigate Ġand Ġdigest Ġweb Ġcontent Ġ( e . g ., Ġnews , Ġsocial Ġmedia , Ġproduct Ġreviews ), Ġquestion Ġanswering , Ġand Ġpersonalized Ġrecommendation Ġengines . ĠSingle Ġdocument Ġsummar ization ĠâĢĶ Ġthe Ġtask Ġof Ġproducing Ġa Ġshorter Ġversion Ġof Ġa Ġdocument Ġwhile Ġpreserving Ġits Ġinformation Ġcontent ĠâĢĶ Ġis Ġperhaps Ġthe Ġmost Ġbasic Ġof Ġsummar ization Ġtasks Ġthat Ġhave Ġbeen Ġidentified Ġover Ġthe Ġyears Ġ( see ĠB IB REF 0 Ġ, ĠB IB REF 0 Ġfor Ġa Ġcomprehensive Ġoverview ). </s> Modern Ġapproaches Ġto Ġsingle Ġdocument Ġsummar ization Ġare Ġdata - driven , Ġtaking Ġadvantage Ġof Ġthe Ġsuccess Ġof Ġneural Ġnetwork Ġarchitectures Ġand Ġtheir Ġability Ġto Ġlearn Ġcontinuous Ġfeatures Ġwithout Ġrecourse Ġto Ġpre processing Ġtools Ġor Ġlinguistic Ġannotations . ĠAbstract ive Ġsummar ization Ġinvolves Ġvarious Ġtext Ġrewriting Ġoperations Ġ( e . g ., Ġsubstitution , Ġdeletion , Ġre ordering ) Ġand Ġhas Ġbeen Ġrecently Ġframed Ġas Ġa Ġsequence - to - sequence Ġproblem ĠB IB REF 1 Ġ. ĠCentral Ġin Ġmost Ġapproaches ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġis Ġan Ġenc oder - dec oder Ġarchitecture Ġmodeled Ġby Ġrecurrent Ġneural Ġnetworks . ĠThe Ġenc oder Ġreads Ġthe Ġsource Ġsequence Ġinto Ġa Ġlist Ġof Ġcontinuous - space Ġrepresentations Ġfrom Ġwhich Ġthe Ġdec oder Ġgenerates Ġthe Ġtarget Ġsequence . ĠAn Ġattention Ġmechanism ĠB IB REF 8 Ġis Ġoften Ġused Ġto Ġlocate Ġthe Ġregion Ġof Ġfocus Ġduring Ġdecoding . </s> Ext ractive Ġsystems Ġcreate Ġa Ġsummary Ġby Ġidentifying Ġ( and Ġsubsequently Ġconc aten ating ) Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument . ĠA Ġfew Ġrecent Ġapproaches ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġconceptual ize Ġextract ive Ġsummar ization Ġas Ġa Ġsequence Ġlabeling Ġtask Ġin Ġwhich Ġeach Ġlabel Ġspecifies Ġwhether Ġeach Ġdocument Ġsentence Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary . ĠEx isting Ġmodels Ġrely Ġon Ġrecurrent Ġneural Ġnetworks Ġto Ġderive Ġa Ġmeaning Ġrepresentation Ġof Ġthe Ġdocument Ġwhich Ġis Ġthen Ġused Ġto Ġlabel Ġeach Ġsentence , Ġtaking Ġthe Ġpreviously Ġlabeled Ġsentences Ġinto Ġaccount . ĠThese Ġmodels Ġare Ġtypically Ġtrained Ġusing Ġcross - ent ropy Ġloss Ġin Ġorder Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġand Ġdo Ġnot Ġnecessarily Ġlearn Ġto Ġrank Ġsentences Ġbased Ġon Ġtheir Ġimportance Ġdue Ġto Ġthe Ġabsence Ġof Ġa Ġranking - based Ġobjective . ĠAnother Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġmismatch Ġbetween Ġthe Ġlearning Ġobjective Ġand Ġthe Ġevaluation Ġcriterion , Ġnamely ĠR OU GE ĠB IB REF 13 Ġ, Ġwhich Ġtakes Ġthe Ġentire Ġsummary Ġinto Ġaccount . </s> In Ġthis Ġpaper Ġwe Ġargue Ġthat Ġcross - ent ropy Ġtraining Ġis Ġnot Ġoptimal Ġfor Ġextract ive Ġsummar ization . ĠModels Ġtrained Ġthis Ġway Ġare Ġprone Ġto Ġgenerating Ġverb ose Ġsumm aries Ġwith Ġunnecessarily Ġlong Ġsentences Ġand Ġredundant Ġinformation . ĠWe Ġpropose Ġto Ġovercome Ġthese Ġdifficulties Ġby Ġglobally Ġoptimizing Ġthe ĠR OU GE Ġevaluation Ġmetric Ġand Ġlearning Ġto Ġrank Ġsentences Ġfor Ġsummary Ġgeneration Ġthrough Ġa Ġreinforcement Ġlearning Ġobjective . ĠSimilar Ġto Ġprevious Ġwork ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 10 Ġ, Ġour Ġneural Ġsummar ization Ġmodel Ġconsists Ġof Ġa Ġhierarchical Ġdocument Ġenc oder Ġand Ġa Ġhierarchical Ġsentence Ġextract or . ĠDuring Ġtraining , Ġit Ġcombines Ġthe Ġmaximum - like lihood Ġcross - ent ropy Ġloss Ġwith Ġrewards Ġfrom Ġpolicy Ġgradient Ġreinforcement Ġlearning Ġto Ġdirectly Ġoptimize Ġthe Ġevaluation Ġmetric Ġrelevant Ġfor Ġthe Ġsummar ization Ġtask . ĠWe Ġshow Ġthat Ġthis Ġglobal Ġoptimization Ġframework Ġrenders Ġextract ive Ġmodels Ġbetter Ġat Ġdiscriminating Ġamong Ġsentences Ġfor Ġthe Ġfinal Ġsummary ; Ġa Ġsentence Ġis Ġranked Ġhigh Ġfor Ġselection Ġif Ġit Ġoften Ġoccurs Ġin Ġhigh Ġscoring Ġsumm aries . </s> We Ġreport Ġresults Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġnews Ġhighlights Ġdatasets ĠB IB REF 14 Ġwhich Ġhave Ġbeen Ġrecently Ġused Ġas Ġtest bed s Ġfor Ġthe Ġevaluation Ġof Ġneural Ġsummar ization Ġsystems . ĠExperimental Ġresults Ġshow Ġthat Ġwhen Ġevaluated Ġautomatically Ġ( in Ġterms Ġof ĠR OU GE ), Ġour Ġmodel Ġoutper forms Ġstate - of - the - art Ġextract ive Ġand Ġabstract ive Ġsystems . ĠWe Ġalso Ġconduct Ġtwo Ġhuman Ġevaluations Ġin Ġorder Ġto Ġassess Ġ( a ) Ġwhich Ġtype Ġof Ġsummary Ġparticipants Ġprefer Ġ( we Ġcompare Ġextract ive Ġand Ġabstract ive Ġsystems ) Ġand Ġ( b ) Ġhow Ġmuch Ġkey Ġinformation Ġfrom Ġthe Ġdocument Ġis Ġpreserved Ġin Ġthe Ġsummary Ġ( we Ġask Ġparticipants Ġto Ġanswer Ġquestions Ġpertaining Ġto Ġthe Ġcontent Ġin Ġthe Ġdocument Ġby Ġreading Ġsystem Ġsumm aries ). ĠBoth Ġevaluations Ġoverwhelmingly Ġshow Ġthat Ġhuman Ġsubjects Ġfind Ġour Ġsumm aries Ġmore Ġinformative Ġand Ġcomplete . </s> Our Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree - fold : Ġa Ġnovel Ġapplication Ġof Ġreinforcement Ġlearning Ġto Ġsentence Ġranking Ġfor Ġextract ive Ġsummar ization ; Ġcorrobor ated Ġby Ġanalysis Ġand Ġempirical Ġresults Ġshowing Ġthat Ġcross - ent ropy Ġtraining Ġis Ġnot Ġwell - su ited Ġto Ġthe Ġsummar ization Ġtask ; Ġand Ġlarge Ġscale Ġuser Ġstudies Ġfollowing Ġtwo Ġevaluation Ġparad ig ms Ġwhich Ġdemonstrate Ġthat Ġstate - of - the - art Ġabstract ive Ġsystems Ġlag Ġbehind Ġextract ive Ġones Ġwhen Ġthe Ġlatter Ġare Ġglobally Ġtrained . </s> Sum mar ization Ġas ĠSent ence ĠRanking </s> Given Ġa Ġdocument ĠD Ġconsisting Ġof Ġa Ġsequence Ġof Ġsentences ĠIN LINE FORM 0 Ġ, Ġan Ġextract ive Ġsummar izer Ġaims Ġto Ġproduce Ġa Ġsummary ĠIN LINE FORM 1 Ġby Ġselecting ĠIN LINE FORM 2 Ġsentences Ġfrom ĠD Ġ( where ĠIN LINE FORM 3 Ġ). ĠFor Ġeach Ġsentence ĠIN LINE FORM 4 Ġ, Ġwe Ġpredict Ġa Ġlabel ĠIN LINE FORM 5 Ġ( where Ġ1 Ġmeans Ġthat ĠIN LINE FORM 6 Ġshould Ġbe Ġincluded Ġin Ġthe Ġsummary ) Ġand Ġassign Ġa Ġscore ĠIN LINE FORM 7 Ġquant ifying ĠIN LINE FORM 8 Ġ' s Ġrelevance Ġto Ġthe Ġsummary . ĠThe Ġmodel Ġlearns Ġto Ġassign ĠIN LINE FORM 9 Ġwhen Ġsentence ĠIN LINE FORM 10 Ġis Ġmore Ġrelevant Ġthan ĠIN LINE FORM 11 Ġ. ĠModel Ġparameters Ġare Ġden oted Ġby ĠIN LINE FORM 12 Ġ. ĠWe Ġestimate ĠIN LINE FORM 13 Ġusing Ġa Ġneural Ġnetwork Ġmodel Ġand Ġassemble Ġa Ġsummary ĠIN LINE FORM 14 Ġby Ġselecting ĠIN LINE FORM 15 Ġsentences Ġwith Ġtop ĠIN LINE FORM 16 Ġscores . </s> Our Ġarchitecture Ġresembles Ġthose Ġpreviously Ġproposed Ġin Ġthe Ġliterature ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠThe Ġmain Ġcomponents Ġinclude Ġa Ġsentence Ġenc oder , Ġa Ġdocument Ġenc oder , Ġand Ġa Ġsentence Ġextract or Ġ( see Ġthe Ġleft Ġblock Ġof ĠFigure ĠFIG REF 2 Ġ) Ġwhich Ġwe Ġdescribe Ġin Ġmore Ġdetail Ġbelow . </s> The ĠPit falls Ġof ĠCross - Ent ropy ĠLoss </s> Previous Ġwork Ġoptim izes Ġsummar ization Ġmodels Ġby Ġmaximizing ĠIN LINE FORM 0 Ġ, Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġy Ġ= ĠIN LINE FORM 1 Ġfor Ġsentences ĠIN LINE FORM 2 Ġ, Ġgiven Ġdocument ĠD Ġand Ġmodel Ġparameters ĠIN LINE FORM 3 Ġ. ĠThis Ġobjective Ġcan Ġbe Ġachieved Ġby Ġminimizing Ġthe Ġcross - ent ropy Ġloss Ġat Ġeach Ġdecoding Ġstep : ĠDIS PLAY FORM 0 </s> Cross - ent ropy Ġtraining Ġleads Ġto Ġtwo Ġkinds Ġof Ġdiscrepancies Ġin Ġthe Ġmodel . ĠThe Ġfirst Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġdisconnect Ġbetween Ġthe Ġtask Ġdefinition Ġand Ġthe Ġtraining Ġobjective . ĠWhile ĠM LE Ġin ĠEqu ation Ġ( ĠEQ REF 7 Ġ) Ġaims Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels , Ġthe Ġmodel Ġis Ġ( a ) Ġexpected Ġto Ġrank Ġsentences Ġto Ġgenerate Ġa Ġsummary Ġand Ġ( b ) Ġevaluated Ġusing ĠIN LINE FORM 0 Ġat Ġtest Ġtime . ĠThe Ġsecond Ġdiscrepancy Ġcomes Ġfrom Ġthe Ġreliance Ġon Ġground - truth Ġlabels . ĠDocument Ġcollections Ġfor Ġtraining Ġsummar ization Ġsystems Ġdo Ġnot Ġnaturally Ġcontain Ġlabels Ġindicating Ġwhich Ġsentences Ġshould Ġbe Ġextracted . ĠInstead , Ġthey Ġare Ġtypically Ġaccompanied Ġby Ġabstract ive Ġsumm aries Ġfrom Ġwhich Ġsentence - level Ġlabels Ġare Ġextrap olated . Ġj p - acl 16 Ġfollow Ġwoods end - acl 10 Ġin Ġadopting Ġa Ġrule - based Ġmethod Ġwhich Ġassigns Ġlabels Ġto Ġeach Ġsentence Ġin Ġthe Ġdocument Ġindividually Ġbased Ġon Ġtheir Ġsemantic Ġcorrespondence Ġwith Ġthe Ġgold Ġsummary Ġ( see Ġthe Ġfourth Ġcolumn Ġin ĠTable ĠT AB REF 6 Ġ). ĠAn Ġalternative Ġmethod ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ, ĠB IB REF 10 Ġidentifies Ġthe Ġset Ġof Ġsentences Ġwhich Ġcollectively Ġgives Ġthe Ġhighest ĠR OU GE Ġwith Ġrespect Ġto Ġthe Ġgold Ġsummary . ĠSent ences Ġin Ġthis Ġset Ġare Ġlabeled Ġwith Ġ1 Ġand Ġ0 Ġotherwise Ġ( see Ġthe Ġcolumn Ġ5 Ġin ĠTable ĠT AB REF 6 Ġ). </s> Label ing Ġsentences Ġindividually Ġoften Ġgenerates Ġtoo Ġmany Ġpositive Ġlabels Ġcausing Ġthe Ġmodel Ġto Ġover fit Ġthe Ġdata . ĠFor Ġexample , Ġthe Ġdocument Ġin ĠTable ĠT AB REF 6 Ġhas Ġ12 Ġpositively Ġlabeled Ġsentences Ġout Ġof Ġ31 Ġin Ġtotal Ġ( only Ġfirst Ġ10 Ġare Ġshown ). ĠCollective Ġlabels Ġpresent Ġa Ġbetter Ġalternative Ġsince Ġthey Ġonly Ġper tain Ġto Ġthe Ġfew Ġsentences Ġdeemed Ġmost Ġsuitable Ġto Ġform Ġthe Ġsummary . ĠHowever , Ġa Ġmodel Ġtrained Ġwith Ġcross - ent ropy Ġloss Ġon Ġcollective Ġlabels Ġwill Ġunder fit Ġthe Ġdata Ġas Ġit Ġwill Ġonly Ġmaximize Ġprobabilities ĠIN LINE FORM 0 Ġfor Ġsentences Ġin Ġthis Ġset Ġ( e . g ., Ġsentences ĠIN LINE FORM 1 Ġin ĠTable ĠT AB REF 6 Ġ) Ġand Ġignore Ġall Ġother Ġsentences . ĠWe Ġfound Ġthat Ġthere Ġare Ġmany Ġcandidate Ġsumm aries Ġwith Ġhigh ĠR OU GE Ġscores Ġwhich Ġcould Ġbe Ġconsidered Ġduring Ġtraining . </s> Table ĠT AB REF 6 Ġ( last Ġcolumn ) Ġshows Ġcandidate Ġsumm aries Ġranked Ġaccording Ġto Ġthe Ġmean Ġof ĠR OU GE - 1 , ĠR OU GE - 2 , Ġand ĠR OU GE - L ĠF ĠIN LINE FORM 0 Ġscores . ĠInterestingly , Ġmultiple Ġtop Ġranked Ġsumm aries Ġhave Ġreasonably Ġhigh ĠR OU GE Ġscores . ĠFor Ġexample , Ġthe Ġaverage ĠR OU GE Ġfor Ġthe Ġsumm aries Ġranked Ġsecond Ġ( 0 , 13 ), Ġthird Ġ( 11 , 13 ), Ġand Ġfourth Ġ( 0 , 1 , 13 ) Ġis Ġ57 . 5 %, Ġ57 . 2 %, Ġand Ġ57 . 1 %, Ġand Ġall Ġtop Ġ16 Ġsumm aries Ġhave ĠR OU GE Ġscores Ġmore Ġor Ġequal Ġto Ġ50 %. ĠA Ġfew Ġsentences Ġare Ġindicative Ġof Ġimportant Ġcontent Ġand Ġappear Ġfrequently Ġin Ġthe Ġsumm aries : Ġsentence Ġ13 Ġoccurs Ġin Ġall Ġsumm aries Ġexcept Ġone , Ġwhile Ġsentence Ġ0 Ġappears Ġin Ġseveral Ġsumm aries Ġtoo . ĠAlso Ġnote Ġthat Ġsumm aries Ġ( 11 , 13 ) Ġand Ġ( 1 , 13 ) Ġyield Ġbetter ĠR OU GE Ġscores Ġcompared Ġto Ġlonger Ġsumm aries , Ġand Ġmay Ġbe Ġas Ġinformative , Ġyet Ġmore Ġconcise , Ġalternatives . </s> These Ġdiscrepancies Ġrender Ġthe Ġmodel Ġless Ġefficient Ġat Ġranking Ġsentences Ġfor Ġthe Ġsummar ization Ġtask . ĠInstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels , Ġwe Ġcould Ġtrain Ġthe Ġmodel Ġto Ġpredict Ġthe Ġindividual ĠR OU GE Ġscore Ġfor Ġeach Ġsentence Ġin Ġthe Ġdocument Ġand Ġthen Ġselect Ġthe Ġtop ĠIN LINE FORM 0 Ġsentences Ġwith Ġhighest Ġscores . ĠBut Ġsentences Ġwith Ġindividual ĠR OU GE Ġscores Ġdo Ġnot Ġnecessarily Ġlead Ġto Ġa Ġhigh Ġscoring Ġsummary , Ġe . g ., Ġthey Ġmay Ġconvey Ġoverlapping Ġcontent Ġand Ġform Ġverb ose Ġand Ġredundant Ġsumm aries . ĠFor Ġexample , Ġsentence Ġ3 , Ġdespite Ġhaving Ġa Ġhigh Ġindividual ĠR OU GE Ġscore Ġ( 35 . 6 %), Ġdoes Ġnot Ġoccur Ġin Ġany Ġof Ġthe Ġtop Ġ5 Ġsumm aries . ĠWe Ġnext Ġexplain Ġhow Ġwe Ġaddress Ġthese Ġissues Ġusing Ġreinforcement Ġlearning . </s> Sent ence ĠRanking Ġwith ĠRein forcement ĠLearning </s> Re in forcement Ġlearning ĠB IB REF 28 Ġhas Ġbeen Ġproposed Ġas Ġa Ġway Ġof Ġtraining Ġsequence - to - sequence Ġgeneration Ġmodels Ġin Ġorder Ġto Ġdirectly Ġoptimize Ġthe Ġmetric Ġused Ġat Ġtest Ġtime , Ġe . g ., ĠB LE U Ġor ĠR OU GE ĠB IB REF 29 Ġ. ĠWe Ġadapt Ġreinforcement Ġlearning Ġto Ġour Ġformulation Ġof Ġextract ive Ġsummar ization Ġto Ġrank Ġsentences Ġfor Ġsummary Ġgeneration . ĠWe Ġpropose Ġan Ġobjective Ġfunction Ġthat Ġcombines Ġthe Ġmaximum - like lihood Ġcross - ent ropy Ġloss Ġwith Ġrewards Ġfrom Ġpolicy Ġgradient Ġreinforcement Ġlearning Ġto Ġglobally Ġoptimize ĠR OU GE . ĠOur Ġtraining Ġalgorithm Ġallows Ġto Ġexplore Ġthe Ġspace Ġof Ġpossible Ġsumm aries , Ġmaking Ġour Ġmodel Ġmore Ġrobust Ġto Ġunseen Ġdata . ĠAs Ġa Ġresult , Ġreinforcement Ġlearning Ġhelps Ġextract ive Ġsummar ization Ġin Ġtwo Ġways : Ġ( a ) Ġit Ġdirectly Ġoptim izes Ġthe Ġevaluation Ġmetric Ġinstead Ġof Ġmaximizing Ġthe Ġlikelihood Ġof Ġthe Ġground - truth Ġlabels Ġand Ġ( b ) Ġit Ġmakes Ġour Ġmodel Ġbetter Ġat Ġdiscriminating Ġamong Ġsentences ; Ġa Ġsentence Ġis Ġranked Ġhigh Ġfor Ġselection Ġif Ġit Ġoften Ġoccurs Ġin Ġhigh Ġscoring Ġsumm aries . </s> Policy ĠLearning </s> We Ġcast Ġthe Ġneural Ġsummar ization Ġmodel Ġintroduced Ġin ĠFigure ĠFIG REF 2 Ġin Ġthe ĠRein forcement ĠLearning Ġparadigm ĠB IB REF 28 Ġ. ĠAccordingly , Ġthe Ġmodel Ġcan Ġbe Ġviewed Ġas Ġan ĠâĢ ľ agent âĢ Ŀ Ġwhich Ġinteracts Ġwith Ġan ĠâĢ ľ environment âĢ Ŀ Ġconsisting Ġof Ġdocuments . ĠAt Ġfirst , Ġthe Ġagent Ġis Ġinitialized Ġrandomly , Ġit Ġreads Ġdocument ĠD Ġand Ġpredicts Ġa Ġrelevance Ġscore Ġfor Ġeach Ġsentence ĠIN LINE FORM 0 Ġusing ĠâĢ ľ policy âĢ Ŀ ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġare Ġmodel Ġparameters . ĠOnce Ġthe Ġagent Ġis Ġdone Ġreading Ġthe Ġdocument , Ġa Ġsummary Ġwith Ġlabels ĠIN LINE FORM 3 Ġis Ġsampled Ġout Ġof Ġthe Ġranked Ġsentences . ĠThe Ġagent Ġis Ġthen Ġgiven Ġa ĠâĢ ľ re ward âĢ Ŀ ĠIN LINE FORM 4 Ġcomm ens urate Ġwith Ġhow Ġwell Ġthe Ġextract Ġresembles Ġthe Ġgold - standard Ġsummary . ĠSpecifically , Ġas Ġreward Ġfunction Ġwe Ġuse Ġmean ĠF ĠIN LINE FORM 5 Ġof ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġ, Ġand ĠIN LINE FORM 8 Ġ. ĠUn ig ram Ġand Ġbig ram Ġoverlap Ġ( ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ) Ġare Ġmeant Ġto Ġassess Ġinform at iveness , Ġwhereas Ġthe Ġlongest Ġcommon Ġsubsequ ence Ġ( ĠIN LINE FORM 11 Ġ) Ġis Ġmeant Ġto Ġassess Ġflu ency . ĠWe Ġupdate Ġthe Ġagent Ġusing Ġthe ĠRE IN FOR CE Ġalgorithm ĠB IB REF 15 Ġwhich Ġaims Ġto Ġminimize Ġthe Ġnegative Ġexpected Ġreward : ĠDIS PLAY FORM 0 </s> where , ĠIN LINE FORM 0 Ġstands Ġfor ĠIN LINE FORM 1 Ġ. ĠRE IN FOR CE Ġis Ġbased Ġon Ġthe Ġobservation Ġthat Ġthe Ġexpected Ġgradient Ġof Ġa Ġnon - different iable Ġreward Ġfunction Ġ( R OU GE , Ġin Ġour Ġcase ) Ġcan Ġbe Ġcomputed Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> While ĠM LE Ġin ĠEqu ation Ġ( ĠEQ REF 7 Ġ) Ġaims Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġtraining Ġdata , Ġthe Ġobjective Ġin ĠEqu ation Ġ( ĠEQ REF 9 Ġ) Ġlearns Ġto Ġdiscriminate Ġamong Ġsentences Ġwith Ġrespect Ġto Ġhow Ġoften Ġthey Ġoccur Ġin Ġhigh Ġscoring Ġsumm aries . </s> Training Ġwith ĠHigh ĠProb ability ĠSam ples </s> Com puting Ġthe Ġexpectation Ġterm Ġin ĠEqu ation Ġ( ĠEQ REF 10 Ġ) Ġis Ġprohib itive , Ġsince Ġthere Ġis Ġa Ġlarge Ġnumber Ġof Ġpossible Ġextracts . ĠIn Ġpractice , Ġwe Ġapproximate Ġthe Ġexpected Ġgradient Ġusing Ġa Ġsingle Ġsample ĠIN LINE FORM 0 Ġfrom ĠIN LINE FORM 1 Ġfor Ġeach Ġtraining Ġexample Ġin Ġa Ġbatch : ĠDIS PLAY FORM 0 </s> Present ed Ġin Ġits Ġoriginal Ġform , Ġthe ĠRE IN FOR CE Ġalgorithm Ġstarts Ġlearning Ġwith Ġa Ġrandom Ġpolicy Ġwhich Ġcan Ġmake Ġmodel Ġtraining Ġchallenging Ġfor Ġcomplex Ġtasks Ġlike Ġours Ġwhere Ġa Ġsingle Ġdocument Ġcan Ġgive Ġrise Ġto Ġa Ġvery Ġlarge Ġnumber Ġof Ġcandidate Ġsumm aries . ĠWe Ġtherefore Ġlimit Ġthe Ġsearch Ġspace Ġof ĠIN LINE FORM 0 Ġin ĠEqu ation Ġ( ĠEQ REF 12 Ġ) Ġto Ġthe Ġset Ġof Ġlargest Ġprobability Ġsamples ĠIN LINE FORM 1 Ġ. ĠWe Ġapproximate ĠIN LINE FORM 2 Ġby Ġthe ĠIN LINE FORM 3 Ġextracts Ġwhich Ġreceive Ġhighest ĠR OU GE Ġscores . ĠMore Ġconcrete ly , Ġwe Ġassemble Ġcandidate Ġsumm aries Ġefficiently Ġby Ġfirst Ġselecting ĠIN LINE FORM 4 Ġsentences Ġfrom Ġthe Ġdocument Ġwhich Ġon Ġtheir Ġown Ġhave Ġhigh ĠR OU GE Ġscores . ĠWe Ġthen Ġgenerate Ġall Ġpossible Ġcombinations Ġof ĠIN LINE FORM 5 Ġsentences Ġsubject Ġto Ġmaximum Ġlength ĠIN LINE FORM 6 Ġand Ġevaluate Ġthem Ġagainst Ġthe Ġgold Ġsummary . ĠSum m aries Ġare Ġranked Ġaccording Ġto ĠF ĠIN LINE FORM 7 Ġby Ġtaking Ġthe Ġmean Ġof ĠIN LINE FORM 8 Ġ, ĠIN LINE FORM 9 Ġ, Ġand ĠIN LINE FORM 10 Ġ. ĠIN LINE FORM 11 Ġcontains Ġthese Ġtop ĠIN LINE FORM 12 Ġcandidate Ġsumm aries . ĠDuring Ġtraining , Ġwe Ġsample ĠIN LINE FORM 13 Ġfrom ĠIN LINE FORM 14 Ġinstead Ġof ĠIN LINE FORM 15 Ġ. </s> ran z ato - ar x iv 15 - b ias Ġproposed Ġan Ġalternative Ġto ĠRE IN FOR CE Ġcalled ĠM IX ER Ġ( M ixed ĠIncre mental ĠCross - Ent ropy ĠRein force ) Ġwhich Ġfirst Ġpret r ains Ġthe Ġmodel Ġwith Ġthe Ġcross - ent ropy Ġloss Ġusing Ġground Ġtruth Ġlabels Ġand Ġthen Ġfollows Ġa Ġcurriculum Ġlearning Ġstrategy ĠB IB REF 30 Ġto Ġgradually Ġteach Ġthe Ġmodel Ġto Ġproduce Ġstable Ġpredictions Ġon Ġits Ġown . ĠIn Ġour Ġexperiments ĠM IX ER Ġperformed Ġworse Ġthan Ġthe Ġmodel Ġof Ġn all ap ati 17 Ġjust Ġtrained Ġon Ġcollective Ġlabels . ĠWe Ġconjecture Ġthat Ġthis Ġis Ġdue Ġto Ġthe Ġunb ounded Ġnature Ġof Ġour Ġranking Ġproblem . ĠRecall Ġthat Ġour Ġmodel Ġassigns Ġrelevance Ġscores Ġto Ġsentences Ġrather Ġthan Ġwords . ĠThe Ġspace Ġof Ġsent ential Ġrepresentations Ġis Ġvast Ġand Ġfairly Ġuncon str ained Ġcompared Ġto Ġother Ġprediction Ġtasks Ġoperating Ġwith Ġfixed Ġvoc ab ular ies ĠB IB REF 31 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 32 Ġ. ĠMoreover , Ġour Ġapproximation Ġof Ġthe Ġgradient Ġallows Ġthe Ġmodel Ġto Ġconverge Ġmuch Ġfaster Ġto Ġan Ġoptimal Ġpolicy . ĠAdvantage ously , Ġwe Ġdo Ġnot Ġrequire Ġan Ġonline Ġreward Ġestim ator , Ġwe Ġpre - comp ute ĠIN LINE FORM 0 Ġ, Ġwhich Ġleads Ġto Ġa Ġsignificant Ġspeed up Ġduring Ġtraining Ġcompared Ġto ĠM IX ER ĠB IB REF 29 Ġand Ġrelated Ġtraining Ġschemes ĠB IB REF 33 Ġ. </s> Exper imental ĠSetup </s> In Ġthis Ġsection Ġwe Ġpresent Ġour Ġexperimental Ġsetup Ġfor Ġassessing Ġthe Ġperformance Ġof Ġour Ġmodel Ġwhich Ġwe Ġcall ĠRefresh Ġas Ġa Ġshorthand Ġfor ĠRE in F o R ce ment ĠLearning - based ĠExtract ive ĠSum mar ization . ĠWe Ġdescribe Ġour Ġdatasets , Ġdiscuss Ġimplementation Ġdetails , Ġour Ġevaluation Ġprotocol , Ġand Ġthe Ġsystems Ġused Ġfor Ġcomparison . </s> Results </s> We Ġreport Ġresults Ġusing Ġautomatic Ġmetrics Ġin ĠTable ĠT AB REF 20 Ġ. ĠThe Ġtop Ġpart Ġof Ġthe Ġtable Ġcompares ĠRefresh Ġagainst Ġrelated Ġextract ive Ġsystems . ĠThe Ġbottom Ġpart Ġreports Ġthe Ġperformance Ġof Ġabstract ive Ġsystems . ĠWe Ġpresent Ġthree Ġvariants Ġof ĠLead , Ġone Ġis Ġcomputed Ġby Ġourselves Ġand Ġthe Ġother Ġtwo Ġare Ġreported Ġin Ġn all ap ati 17 Ġand Ġsee - acl 17 . ĠNote Ġthat Ġthey Ġvary Ġslightly Ġdue Ġto Ġdifferences Ġin Ġthe Ġpre processing Ġof Ġthe Ġdata . ĠWe Ġreport Ġresults Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġdatasets Ġand Ġtheir Ġcombination Ġ( CNN ĠIN LINE FORM 0 ĠDaily Mail ). </s> Related ĠWork </s> Traditional Ġsummar ization Ġmethods Ġmanually Ġdefine Ġfeatures Ġto Ġrank Ġsentences Ġfor Ġtheir Ġsal ience Ġin Ġorder Ġto Ġidentify Ġthe Ġmost Ġimportant Ġsentences Ġin Ġa Ġdocument Ġor Ġset Ġof Ġdocuments ĠB IB REF 42 Ġ, ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ, ĠB IB REF 45 Ġ, ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠA Ġvast Ġmajority Ġof Ġthese Ġmethods Ġlearn Ġto Ġscore Ġeach Ġsentence Ġindependently ĠB IB REF 48 Ġ, ĠB IB REF 49 Ġ, ĠB IB REF 50 Ġ, ĠB IB REF 51 Ġ, ĠB IB REF 52 Ġ, ĠB IB REF 53 Ġ, ĠB IB REF 54 Ġand Ġa Ġsummary Ġis Ġgenerated Ġby Ġselecting Ġtop - sc ored Ġsentences Ġin Ġa Ġway Ġthat Ġis Ġnot Ġincorporated Ġinto Ġthe Ġlearning Ġprocess . ĠSummary Ġquality Ġcan Ġbe Ġimproved Ġhe ur istically , ĠB IB REF 55 Ġ, Ġvia Ġmax - margin Ġmethods ĠB IB REF 56 Ġ, ĠB IB REF 57 Ġ, Ġor Ġinteger - linear Ġprogramming ĠB IB REF 58 Ġ, ĠB IB REF 59 Ġ, ĠB IB REF 60 Ġ, ĠB IB REF 61 Ġ, ĠB IB REF 62 Ġ. </s> Recent Ġdeep Ġlearning Ġmethods ĠB IB REF 63 Ġ, ĠB IB REF 64 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġlearn Ġcontinuous Ġfeatures Ġwithout Ġany Ġlinguistic Ġpre processing Ġ( e . g ., Ġnamed Ġentities ). ĠLike Ġtraditional Ġmethods , Ġthese Ġapproaches Ġalso Ġsuffer Ġfrom Ġthe Ġmismatch Ġbetween Ġthe Ġlearning Ġobjective Ġand Ġthe Ġevaluation Ġcriterion Ġ( e . g ., ĠR OU GE ) Ġused Ġat Ġthe Ġtest Ġtime . ĠIn Ġcomparison , Ġour Ġneural Ġmodel Ġglobally Ġoptim izes Ġthe ĠR OU GE Ġevaluation Ġmetric Ġthrough Ġa Ġreinforcement Ġlearning Ġobjective : Ġsentences Ġare Ġhighly Ġranked Ġif Ġthey Ġoccur Ġin Ġhighly Ġscoring Ġsumm aries . </s> Re in forcement Ġlearning Ġhas Ġbeen Ġpreviously Ġused Ġin Ġthe Ġcontext Ġof Ġtraditional Ġmulti - document Ġsummar ization Ġas Ġa Ġmeans Ġof Ġselecting Ġa Ġsentence Ġor Ġa Ġsubset Ġof Ġsentences Ġfrom Ġa Ġdocument Ġcluster . ĠRy ang : 2012 Ġcast Ġthe Ġsentence Ġselection Ġtask Ġas Ġa Ġsearch Ġproblem . ĠTheir Ġagent Ġobserves Ġa Ġstate Ġ( e . g ., Ġa Ġcandidate Ġsummary ), Ġexecutes Ġan Ġaction Ġ( a Ġtransition Ġoperation Ġthat Ġproduces Ġa Ġnew Ġstate Ġselecting Ġa Ġnot - yet - selected Ġsentence ), Ġand Ġthen Ġreceives Ġa Ġdelayed Ġreward Ġbased Ġon ĠIN LINE FORM 0 Ġ. ĠFollow - on Ġwork ĠB IB REF 65 Ġextends Ġthis Ġapproach Ġby Ġemploying ĠR OU GE Ġas Ġpart Ġof Ġthe Ġreward Ġfunction , Ġwhile Ġhe ns - gs cl 15 Ġfurther Ġexperiment Ġwith ĠIN LINE FORM 1 Ġ- learning . ĠM olla Al iod : 2017 : AL TA 2017 Ġhas Ġadapt Ġthis Ġapproach Ġto Ġquery - focused Ġsummar ization . ĠOur Ġmodel Ġdiffers Ġfrom Ġthese Ġapproaches Ġboth Ġin Ġapplication Ġand Ġformulation . ĠWe Ġfocus Ġsolely Ġon Ġextract ive Ġsummar ization , Ġin Ġour Ġcase Ġstates Ġare Ġdocuments Ġ( not Ġsumm aries ) Ġand Ġactions Ġare Ġrelevance Ġscores Ġwhich Ġlead Ġto Ġsentence Ġranking Ġ( not Ġsentence - to - sent ence Ġtransitions ). ĠRather Ġthan Ġemploying Ġreinforcement Ġlearning Ġfor Ġsentence Ġselection , Ġour Ġalgorithm Ġperforms Ġsentence Ġranking Ġusing ĠR OU GE Ġas Ġthe Ġreward Ġfunction . </s> The ĠRE IN FOR CE Ġalgorithm ĠB IB REF 15 Ġhas Ġbeen Ġshown Ġto Ġimprove Ġenc oder - dec oder Ġtext - rew rit ing Ġsystems Ġby Ġallowing Ġto Ġdirectly Ġoptimize Ġa Ġnon - different iable Ġobjective ĠB IB REF 29 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 7 Ġor Ġto Ġinject Ġtask - specific Ġconstraints ĠB IB REF 32 Ġ, ĠB IB REF 66 Ġ. ĠHowever , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġattempts Ġto Ġuse Ġreinforcement Ġlearning Ġfor Ġtraining Ġa Ġsentence Ġrank er Ġin Ġthe Ġcontext Ġof Ġextract ive Ġsummar ization . </s> Con clusions </s> In Ġthis Ġwork Ġwe Ġdeveloped Ġan Ġextract ive Ġsummar ization Ġmodel Ġwhich Ġis Ġglobally Ġtrained Ġby Ġoptimizing Ġthe ĠR OU GE Ġevaluation Ġmetric . ĠOur Ġtraining Ġalgorithm Ġexplores Ġthe Ġspace Ġof Ġcandidate Ġsumm aries Ġwhile Ġlearning Ġto Ġoptimize Ġa Ġreward Ġfunction Ġwhich Ġis Ġrelevant Ġfor Ġthe Ġtask Ġat Ġhand . ĠExperimental Ġresults Ġshow Ġthat Ġreinforcement Ġlearning Ġoffers Ġa Ġgreat Ġmeans Ġto Ġsteer Ġour Ġmodel Ġtowards Ġgenerating Ġinformative , Ġfluent , Ġand Ġconcise Ġsumm aries Ġoutper forming Ġstate - of - the - art Ġextract ive Ġand Ġabstract ive Ġsystems Ġon Ġthe ĠCNN Ġand ĠDaily Mail Ġdatasets . ĠIn Ġthe Ġfuture Ġwe Ġwould Ġlike Ġto Ġfocus Ġon Ġsmaller Ġdiscourse Ġunits ĠB IB REF 67 Ġrather Ġthan Ġindividual Ġsentences , Ġmodeling Ġcompression Ġand Ġextraction Ġjointly .