 finding the important sentences from the story, extracting the key information from those sentences using their AMR graphs
EXTRACTIVE
 Two methods: first is to simply pick initial few sentences,  second is to capture the relation between the two most important entities  (select the first sentence which contains both these entities).
ABSTRACTIVE
After parsing (Step 1) we have the AMR graphs for the story sentences. In this step we extract the AMR graphs of the summary sentences using story sentence AMRs. We divide this task in two parts. First is finding the important sentences from the story and then extracting the key information from those sentences using their AMR graphs.
Using this idea of picking important sentences from the beginning, we propose two methods, first is to simply pick initial few sentences, we call this first-n method where n stands for the number of sentences. We pick initial 3 sentences for the CNN-Dailymail corpus i.e. first-3 and only the first sentence for the proxy report section (AMR Bank) i.e. first-1 as they produce the best scores on the ROGUE metric compared to any other first-n. Second, we try to capture the relation between the two most important entities (we define importance by the number of occurrences of the entity in the story) of the document. For this we simply find the first sentence which contains both these entities. We call this the first co-occurrence based sentence selection. We also select the first sentence along with first co-occurrence based sentence selection as the important sentences. We call this the first co-occurrence+first based sentence selection.
1706.01678
Introduction </s> Sum mar ization Ġof Ġlarge Ġtexts Ġis Ġstill Ġan Ġopen Ġproblem Ġin Ġlanguage Ġprocessing . ĠPeople Ġnowadays Ġhave Ġlesser Ġtime Ġand Ġpatience Ġto Ġgo Ġthrough Ġlarge Ġpieces Ġof Ġtext Ġwhich Ġmake Ġautomatic Ġsummar ization Ġimportant . ĠAutomatic Ġsummar ization Ġhas Ġsignificant Ġapplications Ġin Ġsummar izing Ġlarge Ġtexts Ġlike Ġstories , Ġjournal Ġpapers , Ġnews Ġarticles Ġand Ġeven Ġlarger Ġtexts Ġlike Ġbooks . </s> Ex isting Ġmethods Ġfor Ġsummar ization Ġcan Ġbe Ġbroadly Ġcategorized Ġinto Ġtwo Ġcategories ĠExtract ive Ġand ĠAbstract ive . ĠExtract ive Ġmethods Ġpicks Ġup Ġwords Ġand Ġsometimes Ġdirectly Ġsentences Ġfrom Ġthe Ġtext . ĠThese Ġmethods Ġare Ġinherently Ġlimited Ġin Ġthe Ġsense Ġthat Ġthey Ġcan Ġnever Ġgenerate Ġhuman Ġlevel Ġsumm aries Ġfor Ġlarge Ġand Ġcomplicated Ġdocuments Ġwhich Ġrequire Ġre ph r asing Ġsentences Ġand Ġincorporating Ġinformation Ġfrom Ġfull Ġtext Ġto Ġgenerate Ġsumm aries . ĠMost Ġof Ġthe Ġwork Ġdone Ġon Ġsummar ization Ġin Ġpast Ġhas Ġbeen Ġextract ive . </s> On Ġthe Ġother Ġhand Ġmost ĠAbstract ive Ġmethods Ġtake Ġadvantages Ġof Ġthe Ġrecent Ġdevelopments Ġin Ġdeep Ġlearning . ĠSpecifically Ġthe Ġrecent Ġsuccess Ġof Ġthe Ġsequence Ġto Ġsequence Ġlearning Ġmodels Ġwhere Ġrecurrent Ġnetworks Ġread Ġthe Ġtext , Ġenc odes Ġit Ġand Ġthen Ġgenerate Ġtarget Ġtext . ĠThough Ġthese Ġmethods Ġhave Ġrecently Ġshown Ġto Ġbe Ġcompetitive Ġwith Ġthe Ġextract ive Ġmethods Ġthey Ġare Ġstill Ġfar Ġaway Ġfrom Ġreaching Ġhuman Ġlevel Ġquality Ġin Ġsummary Ġgeneration . </s> The Ġwork Ġon Ġsummar ization Ġusing ĠAM R Ġwas Ġstarted Ġby ĠB IB REF 0 Ġ. ĠAbstract ĠMeaning ĠRepresent ation Ġ( AM R ) Ġwas Ġas Ġintroduced Ġby ĠB IB REF 1 Ġ. ĠAM R Ġfocuses Ġon Ġcapturing Ġthe Ġmeaning Ġof Ġthe Ġtext , Ġby Ġgiving Ġa Ġspecific Ġmeaning Ġrepresentation Ġto Ġthe Ġtext . ĠAM R Ġtries Ġto Ġcapture Ġthe Ġ" who Ġis Ġdoing Ġwhat Ġto Ġwhom " Ġin Ġa Ġsentence . ĠThe Ġformal ism Ġaims Ġto Ġgive Ġsame Ġrepresentation Ġto Ġsentences Ġwhich Ġhave Ġthe Ġsame Ġunderlying Ġmeaning . ĠFor Ġexample Ġ" He Ġlikes Ġapple " Ġand Ġ" App les Ġare Ġliked Ġby Ġhim " Ġshould Ġbe Ġassigned Ġthe Ġsame ĠAM R . </s> BI B REF 0 Ġ' s Ġapproach Ġaimed Ġto Ġproduce Ġa Ġsummary Ġfor Ġa Ġstory Ġby Ġextracting Ġa Ġsummary Ġsub graph Ġfrom Ġthe Ġstory Ġgraph Ġand Ġfinally Ġgenerate Ġa Ġsummary Ġfrom Ġthis Ġextracted Ġgraph . ĠBut , Ġbecause Ġof Ġthe Ġun availability Ġof ĠAM R Ġto Ġtext Ġgenerator Ġat Ġthat Ġtime Ġtheir Ġwork Ġwas Ġlimited Ġtill Ġextracting Ġthe Ġsummary Ġgraph . ĠThis Ġmethod Ġextracts Ġa Ġsingle Ġsummary Ġgraph Ġfrom Ġthe Ġstory Ġgraph . ĠExtract ing Ġa Ġsingle Ġsummary Ġgraph Ġassumes Ġthat Ġall Ġof Ġthe Ġimportant Ġinformation Ġfrom Ġthe Ġgraph Ġcan Ġbe Ġextracted Ġfrom Ġa Ġsingle Ġsub graph . ĠBut , Ġit Ġcan Ġbe Ġdifficult Ġin Ġcases Ġwhere Ġthe Ġinformation Ġis Ġspread Ġout Ġin Ġthe Ġgraph . ĠThus , Ġthe Ġmethod Ġcompromises Ġbetween Ġsize Ġof Ġthe Ġsummary Ġsub - graph Ġand Ġthe Ġamount Ġof Ġinformation Ġit Ġcan Ġextract . ĠThis Ġcan Ġbe Ġeasily Ġsolved Ġif Ġinstead Ġof Ġa Ġsingle Ġsub - graph , Ġwe Ġextract Ġmultiple Ġsub graph s Ġeach Ġfocusing Ġon Ġinformation Ġin Ġa Ġdifferent Ġpart Ġof Ġthe Ġstory . </s> We Ġpropose Ġa Ġtwo Ġstep Ġprocess Ġfor Ġextracting Ġmultiple Ġsummary Ġgraphs . ĠFirst Ġstep Ġis Ġto Ġselect Ġfew Ġsentences Ġfrom Ġthe Ġstory . ĠWe Ġuse Ġthe Ġidea Ġthat Ġthere Ġare Ġonly Ġfew Ġsentences Ġthat Ġare Ġimportant Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġsummary , Ġi . e . Ġmost Ġof Ġthe Ġinformation Ġcontained Ġin Ġthe Ġsummary Ġis Ġpresent Ġin Ġvery Ġfew Ġsentences Ġand Ġthey Ġcan Ġbe Ġused Ġto Ġgenerate Ġthe Ġsummary . ĠSecond Ġstep Ġis Ġto Ġextract Ġimportant Ġinformation Ġfrom Ġthe Ġselected Ġsentences Ġby Ġextracting Ġa Ġsub - graph Ġfrom Ġthe Ġselected Ġsentences . </s> Our Ġmain Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree Ġfolds , </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠSEC REF 2 Ġcontains Ġintroduction Ġto ĠAM R , Ġsection ĠSEC REF 3 Ġand ĠSEC REF 4 Ġcontains Ġthe Ġdatasets Ġand Ġthe Ġalgorithm Ġused Ġfor Ġsummary Ġgeneration Ġrespectively . ĠSection ĠSEC REF 5 Ġhas Ġa Ġdetailed Ġstep - by - step Ġevaluation Ġof Ġthe Ġpipeline Ġand Ġin Ġsection ĠSEC REF 6 Ġwe Ġdiscuss Ġthe Ġproblems Ġwith Ġthe Ġcurrent Ġdataset Ġand Ġevaluation Ġmetric . </s> Background : ĠAM R ĠPars ing Ġand ĠGeneration </s> AM R Ġwas Ġintroduced Ġby ĠB IB REF 1 Ġwith Ġthe Ġaim Ġto Ġinduce Ġwork Ġon Ġstatistical ĠNatural ĠLanguage ĠUnderstanding Ġand ĠGeneration . ĠAM R Ġrepresents Ġmeaning Ġusing Ġgraphs . ĠAM R Ġgraphs Ġare Ġrooted , Ġdirected , Ġedge Ġand Ġvertex Ġlabeled Ġgraphs . ĠFigure ĠFIG REF 4 Ġshows Ġthe Ġgraphical Ġrepresentation Ġof Ġthe ĠAM R Ġgraph Ġof Ġthe Ġsentence Ġ" I Ġlooked Ġcarefully Ġall Ġaround Ġme " Ġgenerated Ġby ĠJ AM R Ġparser Ġ( ĠB IB REF 2 Ġ). ĠThe Ġgraphical Ġrepresentation Ġwas Ġproduced Ġusing ĠAM R ICA ĠB IB REF 3 Ġ. ĠThe Ġnodes Ġin Ġthe ĠAM R Ġare Ġlabeled Ġwith Ġconcepts Ġas Ġin ĠFigure ĠFIG REF 4 Ġaround Ġrepresents Ġa Ġconcept . ĠEd ges Ġcontains Ġthe Ġinformation Ġregarding Ġthe Ġrelations Ġbetween Ġthe Ġconcepts . ĠIn ĠFigure ĠFIG REF 4 Ġdirection Ġis Ġthe Ġrelation Ġbetween Ġthe Ġconcepts Ġlook - 01 Ġand Ġaround . ĠAM R Ġrelies Ġon ĠProp bank Ġfor Ġsemantic Ġrelations Ġ( edge Ġlabels ). ĠConcepts Ġcan Ġalso Ġbe Ġof Ġthe Ġform Ġrun - 01 Ġwhere Ġthe Ġindex Ġ01 Ġrepresents Ġthe Ġfirst Ġsense Ġof Ġthe Ġword Ġrun . ĠFurther Ġdetails Ġabout Ġthe ĠAM R Ġcan Ġbe Ġfound Ġin Ġthe ĠAM R Ġguidelines ĠB IB REF 4 Ġ. </s> A Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġon Ġparsing Ġsentences Ġto Ġtheir ĠAM R Ġgraphs . ĠThere Ġare Ġthree Ġmain Ġapproaches Ġto Ġparsing . ĠThere Ġis Ġalignment Ġbased Ġparsing ĠB IB REF 2 Ġ( J AM R - Parser ), ĠB IB REF 5 Ġwhich Ġuses Ġgraph Ġbased Ġalgorithms Ġfor Ġconcept Ġand Ġrelation Ġidentification . ĠSecond , Ġgrammar Ġbased Ġpars ers Ġlike ĠB IB REF 6 Ġ( C AM R ) Ġgenerate Ġoutput Ġby Ġperforming Ġshift Ġreduce Ġtransformations Ġon Ġoutput Ġof Ġa Ġdependency Ġparser . ĠNeural Ġparsing ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġis Ġbased Ġon Ġusing Ġseq 2 seq Ġmodels Ġfor Ġparsing , Ġthe Ġmain Ġproblem Ġfor Ġneural Ġmethods Ġis Ġthe Ġabsence Ġof Ġa Ġhuge Ġcorpus Ġof Ġhuman Ġgenerated ĠAM Rs . ĠB IB REF 8 Ġreduced Ġthe Ġvocabulary Ġsize Ġto Ġtackle Ġthis Ġproblem Ġwhile ĠB IB REF 7 Ġused Ġlarger Ġexternal Ġcorpus Ġof Ġexternal Ġsentences . </s> Recently , Ġsome Ġwork Ġhas Ġbeen Ġdone Ġon Ġproducing Ġmeaningful Ġsentences Ġform ĠAM R Ġgraphs . ĠB IB REF 2 Ġused Ġa Ġnumber Ġof Ġtree Ġto Ġstring Ġconversion Ġrules Ġfor Ġgenerating Ġsentences . ĠB IB REF 9 Ġreformed Ġthe Ġproblem Ġas Ġa Ġtraveling Ġsalesman Ġproblem . ĠB IB REF 7 Ġused Ġseq 2 seq Ġlearning Ġmethods . </s> Dat as ets </s> We Ġused Ġtwo Ġdatasets Ġfor Ġthe Ġtask Ġ- ĠAM R ĠBank ĠB IB REF 10 Ġand ĠCNN - Daily mail Ġ( ĠB IB REF 11 ĠB IB REF 12 Ġ). ĠWe Ġuse Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R ĠBank , Ġas Ġit Ġis Ġthe Ġonly Ġone Ġthat Ġis Ġrelevant Ġfor Ġthe Ġtask Ġbecause Ġit Ġcontains Ġthe Ġgold - standard Ġ( human Ġgenerated ) ĠAM R Ġgraphs Ġfor Ġnews Ġarticles , Ġand Ġthe Ġsumm aries . ĠIn Ġthe Ġtraining Ġset Ġthe Ġstories Ġand Ġsumm aries Ġcontain Ġ17 . 5 Ġsentences Ġand Ġ1 . 5 Ġsentences Ġon Ġan Ġaverage Ġrespectively . ĠThe Ġtraining Ġand Ġtest Ġsets Ġcontain Ġ298 Ġand Ġ33 Ġsummary Ġdocument Ġpairs Ġrespectively . </s> CNN - Daily mail Ġcorpus Ġis Ġbetter Ġsuited Ġfor Ġsummar ization Ġas Ġthe Ġaverage Ġsummary Ġsize Ġis Ġaround Ġ3 Ġor Ġ4 Ġsentences . ĠThis Ġdataset Ġhas Ġaround Ġ300 k Ġdocument Ġsummary Ġpairs Ġwith Ġstories Ġhaving Ġ39 Ġsentences Ġon Ġaverage . ĠThe Ġdataset Ġcomes Ġin Ġ2 Ġversions , Ġone Ġis Ġthe Ġanonym ized Ġversion , Ġwhich Ġhas Ġbeen Ġpre process ed Ġto Ġreplace Ġnamed Ġentities , Ġe . g ., ĠThe ĠTimes Ġof ĠIndia , Ġwith Ġa Ġunique Ġidentifier Ġfor Ġexample Ġ@ entity 1 . ĠSecond Ġis Ġthe Ġnon - an onym ized Ġwhich Ġhas Ġthe Ġoriginal Ġtext . ĠWe Ġuse Ġthe Ġnon - an onym ized Ġversion Ġof Ġthe Ġdataset Ġas Ġit Ġis Ġmore Ġsuitable Ġfor ĠAM R Ġparsing Ġas Ġmost Ġof Ġthe Ġpars ers Ġhave Ġbeen Ġtrained Ġon Ġnon - an onym ized Ġtext . ĠThe Ġdataset Ġdoes Ġnot Ġhave Ġgold - standard ĠAM R Ġgraphs . ĠWe Ġuse Ġautomatic Ġpars ers Ġto Ġget Ġthe ĠAM R Ġgraphs Ġbut Ġthey Ġare Ġnot Ġgold - standard Ġand Ġwill Ġeffect Ġthe Ġquality Ġof Ġfinal Ġsummary . ĠTo Ġget Ġan Ġidea Ġof Ġthe Ġerror Ġintroduced Ġby Ġusing Ġautomatic Ġpars ers , Ġwe Ġcompare Ġthe Ġresults Ġafter Ġusing Ġgold - standard Ġand Ġautomatically Ġgenerated ĠAM R Ġgraphs Ġon Ġthe Ġgold - standard Ġdataset . </s> P ip eline Ġfor ĠSummary ĠGeneration </s> The Ġpipeline Ġconsists Ġof Ġthree Ġsteps , Ġfirst Ġconvert Ġall Ġthe Ġgiven Ġstory Ġsentences Ġto Ġthere ĠAM R Ġgraphs Ġfollowed Ġby Ġextracting Ġsummary Ġgraphs Ġfrom Ġthe Ġstory Ġsentence Ġgraphs Ġand Ġfinally Ġgenerating Ġsentences Ġfrom Ġthese Ġextracted Ġsummary Ġgraphs . ĠIn Ġthe Ġfollowing Ġsubsections Ġwe Ġexplain Ġeach Ġof Ġthe Ġmethods Ġin Ġgreater Ġdetail . </s> Step Ġ1 : ĠStory Ġto ĠAM R </s> As Ġthe Ġfirst Ġstep Ġwe Ġconvert Ġthe Ġstory Ġsentences Ġto Ġtheir ĠAbstract ĠMeaning ĠRepresent ations . ĠWe Ġuse ĠJ AM R - Parser Ġversion Ġ2 ĠB IB REF 2 Ġas Ġit âĢ Ļ s Ġopenly Ġavailable Ġand Ġhas Ġa Ġperformance Ġclose Ġto Ġthe Ġstate Ġof Ġthe Ġart Ġpars ers Ġfor Ġparsing Ġthe ĠCNN - Daily mail Ġcorpus . ĠFor Ġthe ĠAM R - bank Ġwe Ġhave Ġthe Ġgold - standard ĠAM R Ġpars es Ġbut Ġwe Ġstill Ġparse Ġthe Ġinput Ġstories Ġwith ĠJ AM R - Parser Ġto Ġstudy Ġthe Ġeffect Ġof Ġusing Ġgraphs Ġproduced Ġby ĠJ AM R - Parser Ġinstead Ġof Ġthe Ġgold - standard ĠAM R Ġgraphs . </s> Step Ġ2 : ĠStory ĠAM R Ġto ĠSummary ĠAM R </s> After Ġparsing Ġ( Step Ġ1 ) Ġwe Ġhave Ġthe ĠAM R Ġgraphs Ġfor Ġthe Ġstory Ġsentences . ĠIn Ġthis Ġstep Ġwe Ġextract Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġsummary Ġsentences Ġusing Ġstory Ġsentence ĠAM Rs . ĠWe Ġdivide Ġthis Ġtask Ġin Ġtwo Ġparts . ĠFirst Ġis Ġfinding Ġthe Ġimportant Ġsentences Ġfrom Ġthe Ġstory Ġand Ġthen Ġextracting Ġthe Ġkey Ġinformation Ġfrom Ġthose Ġsentences Ġusing Ġtheir ĠAM R Ġgraphs . </s> Our Ġalgorithm Ġis Ġbased Ġon Ġthe Ġidea Ġthat Ġonly Ġfew Ġsentences Ġare Ġimportant Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġsummary Ġi . e . Ġthere Ġare Ġonly Ġa Ġfew Ġsentences Ġwhich Ġcontain Ġmost Ġof Ġthe Ġimportant Ġinformation Ġand Ġfrom Ġthese Ġsentences Ġwe Ġcan Ġgenerate Ġthe Ġsummary . </s> Hyp ot hesis : ĠMost Ġof Ġthe Ġinformation Ġcorresponding Ġto Ġa Ġsummary Ġsentence Ġcan Ġbe Ġfound Ġin Ġonly Ġone Ġsentence Ġfrom Ġthe Ġstory . </s> To Ġtest Ġthis Ġhypothesis , Ġfor Ġeach Ġsummary Ġsentence Ġwe Ġfind Ġthe Ġsentence Ġfrom Ġthe Ġstory Ġthat Ġcontains Ġmaximum Ġinformation Ġof Ġthis Ġsummary Ġsentence . ĠWe Ġuse ĠR OG UE - 1 ĠB IB REF 13 ĠRecall Ġscores Ġ( measures Ġthe Ġratio Ġof Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsummary Ġthat Ġare Ġcontained Ġin Ġthe Ġpredicted Ġsummary Ġto Ġthe Ġtotal Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsummary ) Ġas Ġthe Ġmetric Ġfor Ġthe Ġinformation Ġcontained Ġin Ġthe Ġstory Ġsentence . ĠWe Ġconsider Ġthe Ġstory Ġsentence Ġas Ġthe Ġpredicted Ġsummary Ġand Ġthe Ġsummary Ġsentence Ġas Ġthe Ġtarget Ġsummary . ĠThe Ġresults Ġthat Ġwe Ġobtained Ġfor Ġ5000 Ġrandomly Ġchosen Ġdocument Ġsummary Ġpairs Ġfrom Ġthe ĠCNN - Daily mail Ġcorpus Ġare Ġgiven Ġin Ġfigure ĠFIG REF 8 Ġ. ĠThe Ġaverage Ġrecall Ġscore Ġthat Ġwe Ġobtained Ġis Ġ79 %. ĠThe Ġscore Ġwill Ġbe Ġperfectly Ġ1 Ġwhen Ġthe Ġsummary Ġsentence Ġis Ġdirectly Ġpicked Ġup Ġfrom Ġa Ġstory Ġsentence . ĠUpon Ġmanual Ġinspection Ġof Ġthe Ġsummary Ġsentence Ġand Ġthe Ġcorresponding Ġbest Ġsentence Ġfrom Ġthe Ġstory Ġwe Ġrealized , Ġwhen Ġthis Ġscore Ġis Ġmore Ġthan Ġ0 . 5 Ġor Ġ0 . 6 , Ġalmost Ġalways Ġthe Ġinformation Ġin Ġthe Ġsummary Ġsentence Ġis Ġcontained Ġin Ġthis Ġchosen Ġstory Ġsentence . ĠThe Ġscore Ġfor Ġin Ġthese Ġcases Ġis Ġnot Ġperfectly Ġ1 Ġbecause Ġof Ġstop Ġwords Ġand Ġdifferent Ġverb Ġforms Ġused Ġin Ġstory Ġand Ġsummary Ġsentence . ĠAround Ġ80 % Ġof Ġsummary Ġsentences Ġhave Ġscore Ġabove Ġ0 . 5 . ĠSo , Ġour Ġhypothesis Ġseems Ġto Ġbe Ġcorrect Ġfor Ġmost Ġof Ġthe Ġsummary Ġsentences . ĠThis Ġalso Ġsuggests Ġthe Ġhighly Ġextract ive Ġnature Ġof Ġthe Ġsummary Ġin Ġthe Ġcorpus . </s> Now Ġthe Ġtask Ġin Ġhand Ġis Ġto Ġselect Ġfew Ġimportant Ġsentences . ĠMethods Ġthat Ġuse Ġsentence Ġextraction Ġfor Ġsummary Ġgeneration Ġcan Ġbe Ġused Ġfor Ġthe Ġtask . ĠIt Ġis Ġvery Ġcommon Ġin Ġsummar ization Ġtasks Ġspecifically Ġin Ġnews Ġarticles Ġthat Ġa Ġlot Ġof Ġinformation Ġis Ġcontained Ġin Ġthe Ġinitial Ġfew Ġsentences . ĠCho osing Ġinitial Ġfew Ġsentences Ġas Ġthe Ġsummary Ġproduces Ġvery Ġstrong Ġbas elines Ġwhich Ġthe Ġstate - of - the - art Ġmethods Ġbeat Ġonly Ġmarginally . ĠEx . ĠOn Ġthe ĠCNN - Daily mail Ġcorpus Ġthe Ġstate - of - the - art Ġextract ive Ġmethod Ġbeats Ġinitial Ġ3 Ġsentences Ġonly Ġby Ġ0 . 4 % Ġas Ġreported Ġby ĠB IB REF 14 Ġ. </s> Using Ġthis Ġidea Ġof Ġpicking Ġimportant Ġsentences Ġfrom Ġthe Ġbeginning , Ġwe Ġpropose Ġtwo Ġmethods , Ġfirst Ġis Ġto Ġsimply Ġpick Ġinitial Ġfew Ġsentences , Ġwe Ġcall Ġthis Ġfirst - n Ġmethod Ġwhere Ġn Ġstands Ġfor Ġthe Ġnumber Ġof Ġsentences . ĠWe Ġpick Ġinitial Ġ3 Ġsentences Ġfor Ġthe ĠCNN - Daily mail Ġcorpus Ġi . e . Ġfirst - 3 Ġand Ġonly Ġthe Ġfirst Ġsentence Ġfor Ġthe Ġproxy Ġreport Ġsection Ġ( AM R ĠBank ) Ġi . e . Ġfirst - 1 Ġas Ġthey Ġproduce Ġthe Ġbest Ġscores Ġon Ġthe ĠR OG UE Ġmetric Ġcompared Ġto Ġany Ġother Ġfirst - n . ĠSecond , Ġwe Ġtry Ġto Ġcapture Ġthe Ġrelation Ġbetween Ġthe Ġtwo Ġmost Ġimportant Ġentities Ġ( we Ġdefine Ġimportance Ġby Ġthe Ġnumber Ġof Ġoccurrences Ġof Ġthe Ġentity Ġin Ġthe Ġstory ) Ġof Ġthe Ġdocument . ĠFor Ġthis Ġwe Ġsimply Ġfind Ġthe Ġfirst Ġsentence Ġwhich Ġcontains Ġboth Ġthese Ġentities . ĠWe Ġcall Ġthis Ġthe Ġfirst Ġco - occ urrence Ġbased Ġsentence Ġselection . ĠWe Ġalso Ġselect Ġthe Ġfirst Ġsentence Ġalong Ġwith Ġfirst Ġco - occ urrence Ġbased Ġsentence Ġselection Ġas Ġthe Ġimportant Ġsentences . ĠWe Ġcall Ġthis Ġthe Ġfirst Ġco - occ urrence + first Ġbased Ġsentence Ġselection . </s> As Ġthe Ġdatasets Ġunder Ġconsideration Ġare Ġnews Ġarticles . ĠThe Ġmost Ġimportant Ġinformation Ġin Ġthem Ġis Ġabout Ġan Ġentity Ġand Ġa Ġverb Ġassociated Ġwith Ġit . ĠSo , Ġto Ġextract Ġimportant Ġinformation Ġfrom Ġthe Ġsentence . ĠWe Ġtry Ġto Ġfind Ġthe Ġentity Ġbeing Ġtalked Ġabout Ġin Ġthe Ġsentence , Ġwe Ġconsider Ġthe Ġmost Ġreferred Ġentity Ġ( one Ġthat Ġoccurs Ġmost Ġfrequently Ġin Ġthe Ġtext ), Ġnow Ġfor Ġthe Ġmain Ġverb Ġassociated Ġwith Ġthe Ġentity Ġin Ġthe Ġsentence , Ġwe Ġfind Ġthe Ġverb Ġclosest Ġto Ġthis Ġentity Ġin Ġthe ĠAM R Ġgraph . ĠWe Ġdefine Ġthe Ġclosest Ġverb Ġas Ġthe Ġone Ġwhich Ġlies Ġfirst Ġin Ġthe Ġpath Ġfrom Ġthe Ġentity Ġto Ġthe Ġroot . </s> We Ġstart Ġby Ġfinding Ġthe Ġposition Ġof Ġthe Ġmost Ġreferred Ġentity Ġin Ġthe Ġgraph , Ġthen Ġwe Ġfind Ġthe Ġclosest Ġverb Ġto Ġthe Ġentity . Ġand Ġfinally Ġselect Ġthe Ġsubt ree Ġhanging Ġfrom Ġthat Ġverb Ġas Ġthe Ġsummary ĠAM R . </s> Step Ġ3 : ĠSummary ĠGeneration </s> To Ġgenerate Ġsentences Ġfrom Ġthe Ġextracted ĠAM R Ġgraphs Ġwe Ġcan Ġuse Ġalready Ġavailable Ġgenerators . ĠWe Ġuse ĠNeural ĠAM R Ġ( ĠB IB REF 7 Ġ) Ġas Ġit Ġprovides Ġstate Ġof Ġthe Ġart Ġresults Ġin Ġsentence Ġgeneration . ĠWe Ġalso Ġuse ĠB IB REF 15 Ġ( J AM R - Gener ator ) Ġin Ġone Ġof Ġthe Ġexperiments Ġin Ġthe Ġnext Ġsection . ĠGener ators Ġsignificantly Ġeffect Ġthe Ġresults , Ġwe Ġwill Ġanalyze Ġthe Ġeffectiveness Ġof Ġgenerator Ġin Ġthe Ġnext Ġsection . </s> Bas elines </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġbaseline Ġmodels Ġand Ġanalysis Ġmethod Ġused Ġfor Ġeach Ġstep Ġof Ġour Ġpipeline . </s> For Ġthe ĠCNN - Daily mail Ġdataset , Ġthe ĠLead - 3 Ġmodel Ġis Ġconsidered Ġa Ġstrong Ġbaseline ; Ġboth Ġthe Ġabstract ive ĠB IB REF 16 Ġand Ġextract ive ĠB IB REF 14 Ġstate - of - the Ġart Ġmethods Ġon Ġthis Ġdataset Ġbeat Ġthis Ġbaseline Ġonly Ġmarginally . ĠThe ĠLead - 3 Ġmodel Ġsimply Ġproduces Ġthe Ġleading Ġthree Ġsentences Ġof Ġthe Ġdocument Ġas Ġits Ġsummary . </s> The Ġkey Ġstep Ġin Ġour Ġpipeline Ġis Ġstep - 2 Ġi . e . Ġsummary Ġgraph Ġextraction . </s> Direct ly Ġcomparing Ġthe ĠLead - 3 Ġbaseline , Ġwith ĠAM R Ġbased Ġpipeline Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġstep - 2 Ġis Ġan Ġunfair Ġcomparison Ġbecause Ġof Ġthe Ġerrors Ġintroduced Ġby Ġimperfect Ġparser Ġand Ġgenerator Ġin Ġthe ĠAM R Ġpipeline . ĠThus Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġstep - 2 Ġagainst ĠLead - 3 Ġbaseline , Ġwe Ġneed Ġto Ġnull ify Ġthe Ġeffect Ġof Ġerrors Ġintroduce Ġby ĠAM R Ġparser Ġand Ġgenerator . ĠWe Ġachieve Ġthis Ġby Ġtrying Ġto Ġintroduce Ġsimilar Ġerrors Ġin Ġthe Ġleading Ġth re Ġsentences Ġof Ġeach Ġdocument . ĠWe Ġgenerate Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġleading Ġthree Ġsentences Ġand Ġthen Ġgenerate Ġthe Ġsentences Ġusing Ġthese ĠAM R Ġgraph . ĠWe Ġuse Ġparser Ġand Ġgenerator Ġthat Ġwere Ġused Ġin Ġour Ġpipeline . ĠWe Ġconsider Ġthese Ġgenerated Ġsentences Ġas Ġthe Ġnew Ġbaseline Ġsummary , Ġwe Ġshall Ġnow Ġrefer Ġto Ġit ĠLead - 3 - AM R Ġbaseline Ġin Ġthe Ġremaining Ġof Ġthe Ġpaper . </s> For Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R Ġbank , Ġwe Ġconsider Ġthe ĠLead - 1 - AM R Ġmodel Ġas Ġthe Ġbaseline . ĠFor Ġthis Ġdataset Ġwe Ġalready Ġhave Ġthe Ġgold - standard ĠAM R Ġgraphs Ġof Ġthe Ġsentences . ĠTherefore , Ġwe Ġonly Ġneed Ġto Ġnull ify Ġthe Ġerror Ġintroduced Ġby Ġthe Ġgenerator . </s> Pro ced ure Ġto ĠAnaly ze Ġand ĠEval uate Ġeach Ġstep </s> For Ġthe Ġevaluation Ġof Ġsumm aries Ġwe Ġuse Ġthe Ġstandard ĠR OG UE Ġmetric . ĠFor Ġcomparison Ġwith Ġprevious ĠAM R Ġbased Ġsummar ization Ġmethods , Ġwe Ġreport Ġthe ĠRecall , ĠPrecision Ġand ĠIN LINE FORM 0 Ġscores Ġfor ĠR OG UE - 1 . ĠSince Ġmost Ġof Ġthe Ġliterature Ġon Ġsummar ization Ġuses ĠIN LINE FORM 1 Ġscores Ġfor ĠR OG UE - 2 Ġand ĠR OG UE - L Ġfor Ġcomparison , Ġwe Ġalso Ġreport ĠIN LINE FORM 2 Ġscores Ġfor ĠR OG UE - 2 Ġand ĠR OG UE - L Ġfor Ġour Ġmethod . ĠR OG UE - 1 ĠRecall Ġand ĠPrecision Ġare Ġmeasured Ġfor Ġun i - gram Ġoverlap Ġbetween Ġthe Ġreference Ġand Ġthe Ġpredicted Ġsummary . ĠOn Ġthe Ġother Ġhand , ĠR OG UE - 2 Ġuses Ġbi - gram Ġoverlap Ġwhile ĠR OG UE - L Ġuses Ġthe Ġlongest Ġcommon Ġsequence Ġbetween Ġthe Ġtarget Ġand Ġthe Ġpredicted Ġsumm aries Ġfor Ġevaluation . ĠIn Ġrest Ġof Ġthis Ġsection , Ġwe Ġprovide Ġmethods Ġto Ġanalyze Ġand Ġevaluate Ġour Ġpipeline Ġat Ġeach Ġstep . </s> Step - 1 : ĠAM R Ġparsing ĠTo Ġunderstand Ġthe Ġeffects Ġof Ġusing Ġan ĠAM R Ġparser Ġon Ġthe Ġresults , Ġwe Ġcompare Ġthe Ġfinal Ġscores Ġafter Ġthe Ġfollowing Ġtwo Ġcases - Ġfirst , Ġwhen Ġwe Ġuse Ġthe Ġgold - standard ĠAM R Ġgraphs Ġand Ġsecond Ġwhen Ġwe Ġused Ġthe ĠAM R Ġgraphs Ġgenerated Ġby ĠJ AM R - Parser Ġin Ġthe Ġpipeline . ĠSection ĠSEC REF 19 Ġcontains Ġa Ġcomparison Ġbetween Ġthe Ġtwo . </s> Step - 2 : ĠSummary Ġgraph Ġextraction ĠFor Ġevaluating Ġthe Ġeffectiveness Ġof Ġthe Ġsummary Ġgraph Ġextraction Ġstep Ġwe Ġcompare Ġthe Ġfinal Ġscores Ġwith Ġthe ĠLead - n - AM R Ġbas elines Ġdescribed Ġin Ġsection ĠSEC REF 12 Ġ. </s> In Ġorder Ġto Ġcompare Ġour Ġsummary Ġgraph Ġextraction Ġstep Ġwith Ġthe Ġprevious Ġwork Ġ( ĠB IB REF 0 Ġ), Ġwe Ġgenerate Ġthe Ġfinal Ġsummary Ġusing Ġthe Ġsame Ġgeneration Ġmethod Ġas Ġused Ġby Ġthem . ĠTheir Ġmethod Ġuses Ġa Ġsimple Ġmodule Ġbased Ġon Ġalign ments Ġfor Ġgenerating Ġsummary Ġafter Ġstep - 2 . ĠThe Ġalign ments Ġsimply Ġmap Ġthe Ġwords Ġin Ġthe Ġoriginal Ġsentence Ġwith Ġthe Ġnode Ġor Ġedge Ġin Ġthe ĠAM R Ġgraph . ĠTo Ġgenerate Ġthe Ġsummary Ġwe Ġfind Ġthe Ġwords Ġaligned Ġwith Ġthe Ġsentence Ġin Ġthe Ġselected Ġgraph Ġand Ġoutput Ġthem Ġin Ġno Ġparticular Ġorder Ġas Ġthe Ġpredicted Ġsummary . ĠThough Ġthis Ġdoes Ġnot Ġgenerate Ġgram matically Ġcorrect Ġsentences , Ġwe Ġcan Ġstill Ġuse Ġthe ĠR OG UE - 1 Ġmetric Ġsimilar Ġto ĠB IB REF 0 Ġ, Ġas Ġit Ġis Ġbased Ġon Ġcomparing Ġun i - gram s Ġbetween Ġthe Ġtarget Ġand Ġpredicted Ġsumm aries . </s> Step - 3 : ĠGeneration ĠFor Ġevaluating Ġthe Ġquality Ġof Ġthe Ġsentences Ġgenerated Ġby Ġour Ġmethod , Ġwe Ġcompare Ġthe Ġsumm aries Ġgenerated Ġby Ġthe Ġfirst - 1 Ġmodel Ġand ĠLead - 1 - AM R Ġmodel Ġon Ġthe Ġgold - standard Ġdataset . ĠHowever , Ġwhen Ġwe Ġlooked Ġat Ġthe Ġscores Ġgiven Ġby ĠR OG UE , Ġwe Ġdecided Ġto Ġdo Ġget Ġthe Ġabove Ġsumm aries Ġevaluated Ġby Ġhumans . ĠThis Ġproduced Ġinteresting Ġresults Ġwhich Ġare Ġgiven Ġin Ġmore Ġdetail Ġin Ġsection ĠSEC REF 20 Ġ. </s> Results Ġon Ġthe ĠProxy Ġreport Ġsection </s> In Ġtable ĠT AB REF 13 Ġwe Ġreport Ġthe Ġresults Ġof Ġusing Ġthe Ġpipeline Ġwith Ġgeneration Ġusing Ġthe Ġalignment Ġbased Ġgeneration Ġmodule Ġdefined Ġin Ġsection ĠSEC REF 12 Ġ, Ġon Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R ĠBank . ĠAll Ġof Ġour Ġmethods Ġout - per form ĠB IB REF 0 Ġ' s Ġmethod . ĠWe Ġobtain Ġbest ĠR OG UE - 1 ĠIN LINE FORM 0 Ġscores Ġusing Ġthe Ġfirst Ġco - occ urrence + first Ġmodel Ġfor Ġimportant Ġsentences . ĠThis Ġalso Ġout - per form Ġour ĠLead - 1 - AM R Ġbaseline Ġby Ġ0 . 3 ĠR OG UE - 1 ĠIN LINE FORM 1 Ġpoints . </s> Effects Ġof Ġusing ĠJ AM R ĠPars er </s> In Ġthis Ġsubsection Ġwe Ġanalyze Ġthe Ġeffect Ġof Ġusing ĠJ AM R Ġparser Ġfor Ġstep - 1 Ġinstead Ġof Ġthe Ġgold - standard ĠAM R Ġgraphs . ĠFirst Ġpart Ġof Ġtable ĠT AB REF 14 Ġhas Ġscores Ġafter Ġusing Ġthe Ġgold - standard ĠAM R Ġgraphs . ĠIn Ġthe Ġsecond Ġpart Ġof Ġtable ĠT AB REF 14 Ġwe Ġhave Ġincluded Ġthe Ġscores Ġof Ġusing Ġthe ĠJ AM R Ġparser Ġfor ĠAM R Ġgraph Ġgeneration . ĠWe Ġhave Ġused Ġthe Ġsame ĠNeural ĠAM R Ġfor Ġsentence Ġgeneration Ġin Ġall Ġmethods . ĠScores Ġof Ġall Ġmethods Ġincluding Ġthe ĠLead - 1 - AM R Ġbaseline Ġhave Ġdropped Ġsignificantly . </s> The Ġusage Ġof ĠJ AM R ĠPars er Ġhas Ġaffected Ġthe Ġscores Ġof Ġfirst Ġco - occ urrence + first Ġand Ġfirst - 1 Ġmore Ġthan Ġthat Ġfor Ġthe ĠLead - 1 - AM R . ĠThe Ġdrop Ġin ĠR OG UE ĠIN LINE FORM 0 Ġscore Ġwhen Ġwe Ġuse Ġfirst Ġco - occ urrence + first Ġis Ġaround Ġtwo ĠR OG UE ĠIN LINE FORM 1 Ġpoints Ġmore Ġthan Ġwhen ĠLead - 1 - AM R . ĠThis Ġis Ġa Ġsurprising Ġresult , Ġand Ġwe Ġbelieve Ġthat Ġit Ġis Ġworthy Ġof Ġfurther Ġresearch . </s> Effect iveness Ġof Ġthe ĠGenerator </s> In Ġthis Ġsubsection Ġwe Ġevaluate Ġthe Ġeffectiveness Ġof Ġthe Ġsentence Ġgeneration Ġstep . ĠFor Ġfair Ġcomparison Ġat Ġthe Ġgeneration Ġstep Ġwe Ġuse Ġthe Ġgold - standard ĠAM Rs Ġand Ġdon 't Ġperform Ġany Ġextraction Ġin Ġstep - 2 Ġinstead Ġwe Ġuse Ġfull ĠAM Rs , Ġthis Ġallows Ġto Ġremove Ġany Ġerrors Ġthat Ġmight Ġhave Ġbeen Ġgenerated Ġin Ġstep - 1 Ġand Ġstep - 2 . ĠIn Ġorder Ġto Ġcompare Ġthe Ġquality Ġof Ġsentences Ġgenerated Ġby Ġthe ĠAM R , Ġwe Ġneed Ġa Ġgold - standard Ġfor Ġsentence Ġgeneration Ġstep . ĠFor Ġthis , Ġwe Ġsimply Ġuse Ġthe Ġoriginal Ġsentence Ġas Ġgold - standard Ġfor Ġsentence Ġgeneration . ĠThus , Ġwe Ġcompare Ġthe Ġquality Ġof Ġsummary Ġgenerated Ġby ĠLead - 1 Ġand ĠLead - 1 - AM R . ĠThe Ġscores Ġusing Ġthe ĠR OG UE Ġmetric Ġare Ġgiven Ġin Ġbottom Ġtwo Ġrows Ġof Ġtable ĠT AB REF 17 Ġ. ĠThe Ġresults Ġshow Ġthat Ġthere Ġis Ġsignificant Ġdrop Ġin ĠLead - 1 - AM R Ġwhen Ġcompared Ġto ĠLead - 1 . </s> We Ġperform Ġhuman Ġevaluation Ġto Ġcheck Ġwhether Ġthe Ġdrop Ġin ĠR OG UE Ġscores Ġis Ġbecause Ġof Ġdrop Ġin Ġinformation Ġcontained , Ġand Ġhuman Ġread ability Ġor Ġis Ġit Ġbecause Ġof Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġmetric Ġto Ġjudge . ĠTo Ġperform Ġthis Ġevaluation Ġwe Ġrandomly Ġselect Ġten Ġtest Ġexamples Ġfrom Ġthe Ġthirty - Ġthree Ġtest Ġcases Ġof Ġthe Ġproxy Ġreport Ġsection . ĠFor Ġeach Ġexample , Ġwe Ġshow Ġthe Ġsumm aries Ġgenerated Ġby Ġfour Ġdifferent Ġmodels Ġside Ġby Ġside Ġto Ġthe Ġhuman Ġeval u ators . ĠThe Ġhuman Ġeval u ator Ġdoes Ġnot Ġknow Ġwhich Ġsumm aries Ġcome Ġfrom Ġwhich Ġmodel . ĠA Ġscore Ġfrom Ġ1 Ġto Ġ10 Ġis Ġthen Ġassigned Ġto Ġeach Ġsummary Ġon Ġthe Ġbasis Ġof Ġread ability , Ġand Ġinformation Ġcontained Ġof Ġsummary , Ġwhere Ġ1 Ġcorresponds Ġto Ġthe Ġlower Ġlevel Ġand Ġ10 Ġto Ġthe Ġhighest . ĠIn Ġtable ĠT AB REF 17 Ġwe Ġcompare Ġthe Ġscores Ġof Ġthese Ġfour Ġcases Ġas Ġgiven Ġby ĠR OG UE Ġalong Ġwith Ġhuman Ġevaluation . ĠThe Ġparser - gener ator Ġpairs Ġfor Ġthe Ġfour Ġcases Ġare Ġgold - J AM R ( gener ator ), ĠJ AM R ( parser )- ne ural , Ġgold - ne ural , Ġand Ġthe Ġoriginal Ġsentence Ġrespectively . ĠHere Ġgold Ġparser Ġmeans Ġthat Ġwe Ġhave Ġused Ġthe Ġgold - standard ĠAM R Ġgraphs . </s> The Ġscores Ġgiven Ġby Ġthe Ġhumans Ġdo Ġnot Ġcorrelate Ġwith ĠR OG UE . ĠHuman Ġeval u ators Ġgives Ġalmost Ġsimilarly Ġscores Ġto Ġsummary Ġgenerated Ġby Ġthe ĠLead - 1 Ġand ĠLead - 1 - AM R Ġwith ĠLead - 1 - AM R Ġactually Ġperforming Ġbetter Ġon Ġread ability Ġthough Ġit Ġdropped Ġsome Ġinformation Ġas Ġclear Ġfrom Ġthe Ġscores Ġon Ġinformation Ġcontained . ĠOn Ġthe Ġother Ġhand , ĠR OG UE Ġgives Ġvery Ġhigh Ġscore Ġto ĠLead - 1 Ġwhile Ġmodels Ġ1 , 2 Ġand Ġ4 Ġget Ġalmost Ġsame Ġscores . ĠThe Ġsimilar Ġscores Ġof Ġmodel Ġ2 Ġand Ġ3 Ġshows Ġthat Ġgenerators Ġare Ġactually Ġproducing Ġmeaningful Ġsentences . ĠThus Ġthe Ġdrop Ġin ĠR OG UE Ġscores Ġis Ġmainly Ġdue Ġto Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġto Ġevaluate Ġabstract ive Ġsumm aries . ĠMoreover , Ġthe ĠR OG UE Ġgives Ġmodel Ġ4 Ġhigher Ġscore Ġcompared Ġto Ġmodel Ġ1 Ġwhile Ġhuman Ġeval u ators Ġgive Ġthe Ġopposite Ġscores Ġon Ġinformation Ġcontained Ġin Ġthe Ġsentence . </s> A Ġpossible Ġreason Ġfor Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġmetric Ġto Ġproperly Ġevaluate Ġthe Ġsumm aries Ġgenerated Ġby Ġour Ġmethod Ġmight Ġbe Ġdue Ġto Ġits Ġinability Ġto Ġevaluate Ġrestruct ured Ġsentences . ĠAM R Ġformal ism Ġtries Ġto Ġassign Ġthe Ġsame ĠAM R Ġgraphs Ġto Ġthe Ġsentences Ġthat Ġhave Ġsame Ġmeaning Ġso Ġthere Ġexists Ġa Ġone - to - many Ġmapping Ġfrom ĠAM R Ġgraphs Ġto Ġsentences . ĠThis Ġmeans Ġthat Ġthe Ġautomatic Ġgenerators Ġthat Ġwe Ġare Ġusing Ġmight Ġnot Ġbe Ġtrying Ġto Ġgenerate Ġthe Ġoriginal Ġsentence ; Ġinstead Ġit Ġis Ġtrying Ġto Ġgenerate Ġsome Ġother Ġsentence Ġthat Ġhas Ġthe Ġsame Ġunderlying Ġmeaning . ĠThis Ġalso Ġhelps Ġin Ġexplaining Ġthe Ġlow ĠR OG UE - 2 Ġand ĠR OG UE - L Ġscores . ĠIf Ġthe Ġsentences Ġmight Ġbe Ġgetting Ġre ph r ased , Ġthey Ġwould Ġloose Ġmost Ġof Ġthe Ġbi - Ġand Ġtri - gram s Ġfrom Ġthe Ġoriginal Ġsentence Ġresulting Ġin Ġlow ĠR OG UE - 2 Ġand ĠR OG UE - L Ġscores . </s> Analy zing Ġthe Ġeffectiveness Ġof ĠAM R Ġextraction </s> The Ġaim Ġof Ġextracting Ġsummary Ġgraphs Ġfrom Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġsentence Ġis Ġto Ġdrop Ġthe Ġnot Ġso Ġimportant Ġinformation Ġfrom Ġthe Ġsentences . ĠIf Ġwe Ġare Ġable Ġto Ġachieve Ġthis Ġperfectly , Ġthe ĠR OG UE - 1 ĠRecall Ġscores Ġthat Ġwe Ġare Ġgetting Ġshould Ġremain Ġalmost Ġthe Ġsame Ġ( since Ġwe Ġare Ġnot Ġadd Ġany Ġnew Ġinformation ) Ġand Ġthe ĠR OG UE - 1 Ġprecision Ġshould Ġgo Ġup Ġ( as Ġwe Ġhave Ġthrown Ġout Ġsome Ġuseless Ġinformation ); Ġthus Ġeffectively Ġimproving Ġthe Ġoverall ĠR OG UE - 1 ĠIN LINE FORM 0 Ġscore . ĠIn Ġthe Ġfirst Ġtwo Ġrows Ġof Ġtable ĠT AB REF 14 Ġwe Ġhave Ġthe Ġscores Ġafter Ġusing Ġthe Ġfull - AM R Ġand Ġextracted ĠAM R Ġfor Ġgeneration Ġrespectively . ĠIt Ġis Ġsafe Ġto Ġsay Ġthat Ġextracting Ġthe ĠAM R Ġresults Ġin Ġimproved ĠR OG UE - 1 Ġprecision Ġwhereas ĠR OG UE - 1 ĠRecall Ġreduces Ġonly Ġslightly , Ġresulting Ġin Ġan Ġoverall Ġimproved ĠR OG UE - 1 ĠIN LINE FORM 1 Ġ. </s> Results Ġon Ġthe ĠCNN - Daily mail Ġcorpus </s> In Ġtable ĠT AB REF 18 Ġwe Ġreport Ġthe Ġresults Ġon Ġthe ĠCNN - Daily mail Ġcorpus . ĠWe Ġpresent Ġscores Ġby Ġusing Ġthe Ġfirst - 3 Ġmodel . ĠThe Ġfirst Ġrow Ġcontains Ġthe ĠLead - 3 - AM R Ġbaseline . ĠThe Ġresults Ġwe Ġachieve Ġare Ġcompetitive Ġwith Ġthe ĠLead - 3 - AM R Ġbaseline . ĠThe Ġrest Ġof Ġthe Ġtable Ġcontains Ġscores Ġof ĠLead - 3 Ġbaseline Ġfollowed Ġby Ġthe Ġstate - of - the - art Ġmethod Ġon Ġthe Ġanonym ized Ġand Ġnon - an onym ized Ġversions Ġof Ġthe Ġdataset . ĠThe Ġdrop Ġin Ġthe Ġscores Ġfrom Ġthe ĠLead - 3 ( non - an onym ized ) Ġto ĠLead - 3 - AM R Ġis Ġsignificant Ġand Ġis Ġlargely Ġbecause Ġof Ġthe Ġerror Ġintroduced Ġby Ġparser Ġand Ġgenerator . </s> Related ĠWork </s> BI B REF 18 Ġshowed Ġthat Ġmost Ġof Ġthe Ġwork Ġin Ġtext Ġsummar ization Ġhas Ġbeen Ġextract ive , Ġwhere Ġsentences Ġare Ġselected Ġfrom Ġthe Ġtext Ġwhich Ġare Ġthen Ġconc aten ated Ġto Ġform Ġa Ġsummary . ĠB IB REF 19 Ġtransformed Ġthe Ġinput Ġto Ġnodes , Ġthen Ġused Ġthe ĠP ager ank Ġalgorithm Ġto Ġscore Ġnodes , Ġand Ġfinally Ġgrow Ġthe Ġnodes Ġfrom Ġhigh - value Ġto Ġlow - value Ġusing Ġsome Ġhe ur istics . ĠSome Ġof Ġthe Ġapproaches Ġcombine Ġthis Ġwith Ġsentence Ġcompression , Ġso Ġmore Ġsentences Ġcan Ġbe Ġpacked Ġin Ġthe Ġsummary . ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, Ġand ĠB IB REF 23 Ġamong Ġothers Ġused ĠIL Ps Ġand Ġapprox im ations Ġfor Ġencoding Ġcompression Ġand Ġextraction . </s> Recently Ġsome Ġabstract ive Ġapproaches Ġhave Ġalso Ġbeen Ġproposed Ġmost Ġof Ġwhich Ġused Ġsequence Ġto Ġsequence Ġlearning Ġmodels Ġfor Ġthe Ġtask . ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 17 Ġused Ġstandard Ġenc oder - dec oder Ġmodels Ġalong Ġwith Ġtheir Ġvariants Ġto Ġgenerate Ġsumm aries . ĠB IB REF 26 Ġincorporated Ġthe ĠAM R Ġinformation Ġin Ġthe Ġstandard Ġenc oder - dec oder Ġmodels Ġto Ġimprove Ġresults . ĠOur Ġwork Ġin Ġsimilar Ġto Ġother Ġgraph Ġbased Ġabstract ive Ġsummar ization Ġmethods ĠB IB REF 27 Ġand ĠB IB REF 28 Ġ. ĠB IB REF 27 Ġused Ġdependency Ġparse Ġtrees Ġto Ġproduce Ġsumm aries . ĠOn Ġthe Ġother Ġhand Ġour Ġwork Ġtakes Ġadvantage Ġof Ġsemantic Ġgraphs . </s> Need Ġof Ġan Ġnew ĠDat as et Ġand ĠEvaluation ĠMet ric </s> R OG UE Ġmetric , Ġby Ġit Ġis Ġdesign Ġhas Ġlots Ġof Ġproperties Ġthat Ġmake Ġit Ġunsu itable Ġfor Ġevaluating Ġabstract ive Ġsumm aries . ĠFor Ġexample , ĠR OG UE Ġmatches Ġexact Ġwords Ġand Ġnot Ġthe Ġstems Ġof Ġthe Ġwords , Ġit Ġalso Ġconsiders Ġstop Ġwords Ġfor Ġevaluation . ĠOne Ġof Ġthe Ġreasons Ġwhy ĠR OG UE Ġlike Ġmetrics Ġmight Ġnever Ġbecome Ġsuitable Ġfor Ġevaluating Ġabstract ive Ġsumm aries Ġis Ġits Ġincap abilities Ġof Ġknowing Ġif Ġthe Ġsentences Ġhave Ġbeen Ġrestruct ured . ĠA Ġgood Ġevaluation Ġmetric Ġshould Ġbe Ġone Ġwhere Ġwe Ġcompare Ġthe Ġmeaning Ġof Ġthe Ġsentence Ġand Ġnot Ġthe Ġexact Ġwords . ĠAs Ġwe Ġshowed Ġsection ĠSEC REF 20 ĠR OG UE Ġis Ġnot Ġsuitable Ġfor Ġevaluating Ġsumm aries Ġgenerated Ġby Ġthe ĠAM R Ġpipeline . </s> We Ġnow Ġshow Ġwhy Ġthe ĠCNN - Daily mail Ġcorpus Ġis Ġnot Ġsuitable Ġfor ĠAbstract ive Ġsummar ization . ĠThe Ġnature Ġof Ġsummary Ġpoints Ġin Ġthe Ġcorpus Ġis Ġhighly Ġextract ive Ġ( Section ĠSEC REF 7 Ġfor Ġdetails ) Ġwhere Ġmost Ġof Ġthe Ġsummary Ġpoints Ġare Ġsimply Ġpicked Ġup Ġfrom Ġsome Ġsentences Ġin Ġthe Ġstory . ĠTough , Ġthis Ġis Ġa Ġgood Ġenough Ġreason Ġto Ġstart Ġsearching Ġfor Ġbetter Ġdataset , Ġit Ġis Ġnot Ġthe Ġbiggest Ġproblem Ġwith Ġthe Ġdataset . ĠThe Ġdataset Ġhas Ġthe Ġproperty Ġthat Ġa Ġlot Ġof Ġimportant Ġinformation Ġis Ġin Ġthe Ġfirst Ġfew Ġsentences Ġand Ġmost Ġof Ġthe Ġsummary Ġpoints Ġare Ġdirectly Ġpick Ġfrom Ġthese Ġsentences . ĠThe Ġextract ive Ġmethods Ġbased Ġon Ġsentence Ġselection Ġlike ĠSum ma Run N er Ġare Ġnot Ġactually Ġperforming Ġwell , Ġthe Ġresults Ġthey Ġhave Ġgot Ġare Ġonly Ġslightly Ġbetter Ġthan Ġthe ĠLead - 3 Ġbaseline . ĠThe Ġwork Ġdoesn 't Ġshow Ġhow Ġmuch Ġof Ġthe Ġselected Ġsentences Ġare Ġamong Ġthe Ġfirst Ġfew Ġand Ġit Ġmight Ġbe Ġthe Ġcase Ġthat Ġthe Ġsentences Ġselected Ġby Ġthe Ġextract ive Ġmethods Ġare Ġmostly Ġamong Ġthe Ġfirst Ġfew Ġsentences , Ġthe Ġsame Ġcan Ġbe Ġthe Ġproblem Ġwith Ġthe Ġabstract ive Ġmethods , Ġwhere Ġmost Ġthe Ġoutput Ġmight Ġbe Ġgetting Ġcopied Ġfrom Ġthe Ġinitial Ġfew Ġsentences . </s> These Ġproblems Ġwith Ġthis Ġcorpus Ġevoke Ġthe Ġneed Ġto Ġhave Ġanother Ġcorpus Ġwhere Ġwe Ġdon 't Ġhave Ġso Ġmuch Ġconcentration Ġof Ġimportant Ġinformation Ġat Ġany Ġlocation Ġbut Ġrather Ġthe Ġinformation Ġis Ġmore Ġspread Ġout Ġand Ġthe Ġsumm aries Ġare Ġmore Ġabstract ive Ġin Ġnature . </s> P ossible ĠFuture ĠDirections </s> As Ġthis Ġproposed Ġalgorithm Ġis Ġa Ġstep Ġby Ġstep Ġprocess Ġwe Ġcan Ġfocus Ġon Ġimproving Ġeach Ġstep Ġto Ġproduce Ġbetter Ġresults . ĠThe Ġmost Ġexciting Ġimprovements Ġcan Ġbe Ġdone Ġin Ġthe Ġsummary Ġgraph Ġextraction Ġmethod . ĠNot Ġa Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġto Ġextract ĠAM R Ġgraphs Ġfor Ġsumm aries . ĠIn Ġorder Ġto Ġmake Ġthis Ġpipeline Ġgeneral izable Ġfor Ġany Ġsort Ġof Ġtext , Ġwe Ġneed Ġto Ġget Ġrid Ġof Ġthe Ġhypothesis Ġthat Ġthe Ġsummary Ġis Ġbeing Ġextracted Ġexactly Ġfrom Ġone Ġsentence . ĠSo , Ġthe Ġnatural Ġdirection Ġseems Ġto Ġbe Ġjoining ĠAM R Ġgraphs Ġof Ġmultiple Ġsentences Ġthat Ġare Ġsimilar Ġand Ġthen Ġextracting Ġthe Ġsummary ĠAM R Ġfrom Ġthat Ġlarge Ġgraph . ĠIt Ġwill Ġbe Ġlike Ġclust ering Ġsimilar Ġsentences Ġand Ġthen Ġextracting Ġa Ġsummary Ġgraph Ġfrom Ġeach Ġof Ġthese Ġcluster . ĠAnother Ġidea Ġis Ġto Ġuse ĠAM R Ġgraphs Ġfor Ġimportant Ġsentence Ġselection . </s> Conclusion </s> In Ġthis Ġwork Ġwe Ġhave Ġexplored Ġa Ġfull - fledged Ġpipeline Ġusing ĠAM R Ġfor Ġsummar ization Ġfor Ġthe Ġfirst Ġtime . ĠWe Ġpropose Ġa Ġnew Ġmethod Ġfor Ġextracting Ġsummary Ġgraph , Ġwhich Ġoutper formed Ġprevious Ġmethods . ĠOverall Ġwe Ġprovide Ġstrong Ġbaseline Ġfor Ġtext Ġsummar ization Ġusing ĠAM R Ġfor Ġpossible Ġfuture Ġworks . ĠWe Ġalso Ġshowed Ġthat ĠR OG UE Ġcan 't Ġbe Ġused Ġfor Ġevaluating Ġthe Ġabstract ive Ġsumm aries Ġgenerated Ġby Ġour ĠAM R Ġpipeline .
Introduction Summarization of large texts is still an open problem in language processing. People nowadays have lesser time and patience to go through large pieces of text which make automatic summarization important. Automatic summarization has significant applications in summarizing large texts like stories, journal papers, news articles and even larger texts like books. Existing methods for summarization can be broadly categorized into two categories Extractive and Abstractive. Extractive methods picks up words and sometimes directly sentences from the text. These methods are inherently limited in the sense that they can never generate human level summaries for large and complicated documents which require rephrasing sentences and incorporating information from full text to generate summaries. Most of the work done on summarization in past has been extractive. On the other hand most Abstractive methods take advantages of the recent developments in deep learning. Specifically the recent success of the sequence to sequence learning models where recurrent networks read the text, encodes it and then generate target text. Though these methods have recently shown to be competitive with the extractive methods they are still far away from reaching human level quality in summary generation. The work on summarization using AMR was started by BIBREF0 . Abstract Meaning Representation (AMR) was as introduced by BIBREF1 . AMR focuses on capturing the meaning of the text, by giving a specific meaning representation to the text. AMR tries to capture the "who is doing what to whom" in a sentence. The formalism aims to give same representation to sentences which have the same underlying meaning. For example "He likes apple" and "Apples are liked by him" should be assigned the same AMR. BIBREF0 's approach aimed to produce a summary for a story by extracting a summary subgraph from the story graph and finally generate a summary from this extracted graph. But, because of the unavailability of AMR to text generator at that time their work was limited till extracting the summary graph. This method extracts a single summary graph from the story graph. Extracting a single summary graph assumes that all of the important information from the graph can be extracted from a single subgraph. But, it can be difficult in cases where the information is spread out in the graph. Thus, the method compromises between size of the summary sub-graph and the amount of information it can extract. This can be easily solved if instead of a single sub-graph, we extract multiple subgraphs each focusing on information in a different part of the story. We propose a two step process for extracting multiple summary graphs. First step is to select few sentences from the story. We use the idea that there are only few sentences that are important from the point of view of summary, i.e. most of the information contained in the summary is present in very few sentences and they can be used to generate the summary. Second step is to extract important information from the selected sentences by extracting a sub-graph from the selected sentences. Our main contributions in this work are three folds, The rest of the paper is organized as follows. Section SECREF2 contains introduction to AMR, section SECREF3 and SECREF4 contains the datasets and the algorithm used for summary generation respectively. Section SECREF5 has a detailed step-by-step evaluation of the pipeline and in section SECREF6 we discuss the problems with the current dataset and evaluation metric. Background: AMR Parsing and Generation AMR was introduced by BIBREF1 with the aim to induce work on statistical Natural Language Understanding and Generation. AMR represents meaning using graphs. AMR graphs are rooted, directed, edge and vertex labeled graphs. Figure FIGREF4 shows the graphical representation of the AMR graph of the sentence "I looked carefully all around me" generated by JAMR parser ( BIBREF2 ). The graphical representation was produced using AMRICA BIBREF3 . The nodes in the AMR are labeled with concepts as in Figure FIGREF4 around represents a concept. Edges contains the information regarding the relations between the concepts. In Figure FIGREF4 direction is the relation between the concepts look-01 and around. AMR relies on Propbank for semantic relations (edge labels). Concepts can also be of the form run-01 where the index 01 represents the first sense of the word run. Further details about the AMR can be found in the AMR guidelines BIBREF4 . A lot of work has been done on parsing sentences to their AMR graphs. There are three main approaches to parsing. There is alignment based parsing BIBREF2 (JAMR-Parser), BIBREF5 which uses graph based algorithms for concept and relation identification. Second, grammar based parsers like BIBREF6 (CAMR) generate output by performing shift reduce transformations on output of a dependency parser. Neural parsing BIBREF7 , BIBREF8 is based on using seq2seq models for parsing, the main problem for neural methods is the absence of a huge corpus of human generated AMRs. BIBREF8 reduced the vocabulary size to tackle this problem while BIBREF7 used larger external corpus of external sentences. Recently, some work has been done on producing meaningful sentences form AMR graphs. BIBREF2 used a number of tree to string conversion rules for generating sentences. BIBREF9 reformed the problem as a traveling salesman problem. BIBREF7 used seq2seq learning methods. Datasets We used two datasets for the task - AMR Bank BIBREF10 and CNN-Dailymail ( BIBREF11 BIBREF12 ). We use the proxy report section of the AMR Bank, as it is the only one that is relevant for the task because it contains the gold-standard (human generated) AMR graphs for news articles, and the summaries. In the training set the stories and summaries contain 17.5 sentences and 1.5 sentences on an average respectively. The training and test sets contain 298 and 33 summary document pairs respectively. CNN-Dailymail corpus is better suited for summarization as the average summary size is around 3 or 4 sentences. This dataset has around 300k document summary pairs with stories having 39 sentences on average. The dataset comes in 2 versions, one is the anonymized version, which has been preprocessed to replace named entities, e.g., The Times of India, with a unique identifier for example @entity1. Second is the non-anonymized which has the original text. We use the non-anonymized version of the dataset as it is more suitable for AMR parsing as most of the parsers have been trained on non-anonymized text. The dataset does not have gold-standard AMR graphs. We use automatic parsers to get the AMR graphs but they are not gold-standard and will effect the quality of final summary. To get an idea of the error introduced by using automatic parsers, we compare the results after using gold-standard and automatically generated AMR graphs on the gold-standard dataset. Pipeline for Summary Generation The pipeline consists of three steps, first convert all the given story sentences to there AMR graphs followed by extracting summary graphs from the story sentence graphs and finally generating sentences from these extracted summary graphs. In the following subsections we explain each of the methods in greater detail. Step 1: Story to AMR As the first step we convert the story sentences to their Abstract Meaning Representations. We use JAMR-Parser version 2 BIBREF2 as it’s openly available and has a performance close to the state of the art parsers for parsing the CNN-Dailymail corpus. For the AMR-bank we have the gold-standard AMR parses but we still parse the input stories with JAMR-Parser to study the effect of using graphs produced by JAMR-Parser instead of the gold-standard AMR graphs. Step 2: Story AMR to Summary AMR After parsing (Step 1) we have the AMR graphs for the story sentences. In this step we extract the AMR graphs of the summary sentences using story sentence AMRs. We divide this task in two parts. First is finding the important sentences from the story and then extracting the key information from those sentences using their AMR graphs. Our algorithm is based on the idea that only few sentences are important from the point of view of summary i.e. there are only a few sentences which contain most of the important information and from these sentences we can generate the summary. Hypothesis: Most of the information corresponding to a summary sentence can be found in only one sentence from the story. To test this hypothesis, for each summary sentence we find the sentence from the story that contains maximum information of this summary sentence. We use ROGUE-1 BIBREF13 Recall scores (measures the ratio of number of words in the target summary that are contained in the predicted summary to the total number of words in the target summary) as the metric for the information contained in the story sentence. We consider the story sentence as the predicted summary and the summary sentence as the target summary. The results that we obtained for 5000 randomly chosen document summary pairs from the CNN-Dailymail corpus are given in figure FIGREF8 . The average recall score that we obtained is 79%. The score will be perfectly 1 when the summary sentence is directly picked up from a story sentence. Upon manual inspection of the summary sentence and the corresponding best sentence from the story we realized, when this score is more than 0.5 or 0.6, almost always the information in the summary sentence is contained in this chosen story sentence. The score for in these cases is not perfectly 1 because of stop words and different verb forms used in story and summary sentence. Around 80% of summary sentences have score above 0.5. So, our hypothesis seems to be correct for most of the summary sentences. This also suggests the highly extractive nature of the summary in the corpus. Now the task in hand is to select few important sentences. Methods that use sentence extraction for summary generation can be used for the task. It is very common in summarization tasks specifically in news articles that a lot of information is contained in the initial few sentences. Choosing initial few sentences as the summary produces very strong baselines which the state-of-the-art methods beat only marginally. Ex. On the CNN-Dailymail corpus the state-of-the-art extractive method beats initial 3 sentences only by 0.4% as reported by BIBREF14 . Using this idea of picking important sentences from the beginning, we propose two methods, first is to simply pick initial few sentences, we call this first-n method where n stands for the number of sentences. We pick initial 3 sentences for the CNN-Dailymail corpus i.e. first-3 and only the first sentence for the proxy report section (AMR Bank) i.e. first-1 as they produce the best scores on the ROGUE metric compared to any other first-n. Second, we try to capture the relation between the two most important entities (we define importance by the number of occurrences of the entity in the story) of the document. For this we simply find the first sentence which contains both these entities. We call this the first co-occurrence based sentence selection. We also select the first sentence along with first co-occurrence based sentence selection as the important sentences. We call this the first co-occurrence+first based sentence selection. As the datasets under consideration are news articles. The most important information in them is about an entity and a verb associated with it. So, to extract important information from the sentence. We try to find the entity being talked about in the sentence, we consider the most referred entity (one that occurs most frequently in the text), now for the main verb associated with the entity in the sentence, we find the verb closest to this entity in the AMR graph. We define the closest verb as the one which lies first in the path from the entity to the root. We start by finding the position of the most referred entity in the graph, then we find the closest verb to the entity. and finally select the subtree hanging from that verb as the summary AMR. Step 3: Summary Generation To generate sentences from the extracted AMR graphs we can use already available generators. We use Neural AMR ( BIBREF7 ) as it provides state of the art results in sentence generation. We also use BIBREF15 (JAMR-Generator) in one of the experiments in the next section. Generators significantly effect the results, we will analyze the effectiveness of generator in the next section. Baselines In this section, we present the baseline models and analysis method used for each step of our pipeline. For the CNN-Dailymail dataset, the Lead-3 model is considered a strong baseline; both the abstractive BIBREF16 and extractive BIBREF14 state-of-the art methods on this dataset beat this baseline only marginally. The Lead-3 model simply produces the leading three sentences of the document as its summary. The key step in our pipeline is step-2 i.e. summary graph extraction. Directly comparing the Lead-3 baseline, with AMR based pipeline to evaluate the effectiveness of step-2 is an unfair comparison because of the errors introduced by imperfect parser and generator in the AMR pipeline. Thus to evaluate the effectiveness of step-2 against Lead-3 baseline, we need to nullify the effect of errors introduce by AMR parser and generator. We achieve this by trying to introduce similar errors in the leading thre sentences of each document. We generate the AMR graphs of the leading three sentences and then generate the sentences using these AMR graph. We use parser and generator that were used in our pipeline. We consider these generated sentences as the new baseline summary, we shall now refer to it Lead-3-AMR baseline in the remaining of the paper. For the proxy report section of the AMR bank, we consider the Lead-1-AMR model as the baseline. For this dataset we already have the gold-standard AMR graphs of the sentences. Therefore, we only need to nullify the error introduced by the generator. Procedure to Analyze and Evaluate each step For the evaluation of summaries we use the standard ROGUE metric. For comparison with previous AMR based summarization methods, we report the Recall, Precision and INLINEFORM0 scores for ROGUE-1. Since most of the literature on summarization uses INLINEFORM1 scores for ROGUE-2 and ROGUE-L for comparison, we also report INLINEFORM2 scores for ROGUE-2 and ROGUE-L for our method. ROGUE-1 Recall and Precision are measured for uni-gram overlap between the reference and the predicted summary. On the other hand, ROGUE-2 uses bi-gram overlap while ROGUE-L uses the longest common sequence between the target and the predicted summaries for evaluation. In rest of this section, we provide methods to analyze and evaluate our pipeline at each step. Step-1: AMR parsing To understand the effects of using an AMR parser on the results, we compare the final scores after the following two cases- first, when we use the gold-standard AMR graphs and second when we used the AMR graphs generated by JAMR-Parser in the pipeline. Section SECREF19 contains a comparison between the two. Step-2: Summary graph extraction For evaluating the effectiveness of the summary graph extraction step we compare the final scores with the Lead-n-AMR baselines described in section SECREF12 . In order to compare our summary graph extraction step with the previous work ( BIBREF0 ), we generate the final summary using the same generation method as used by them. Their method uses a simple module based on alignments for generating summary after step-2. The alignments simply map the words in the original sentence with the node or edge in the AMR graph. To generate the summary we find the words aligned with the sentence in the selected graph and output them in no particular order as the predicted summary. Though this does not generate grammatically correct sentences, we can still use the ROGUE-1 metric similar to BIBREF0 , as it is based on comparing uni-grams between the target and predicted summaries. Step-3: Generation For evaluating the quality of the sentences generated by our method, we compare the summaries generated by the first-1 model and Lead-1-AMR model on the gold-standard dataset. However, when we looked at the scores given by ROGUE, we decided to do get the above summaries evaluated by humans. This produced interesting results which are given in more detail in section SECREF20 . Results on the Proxy report section In table TABREF13 we report the results of using the pipeline with generation using the alignment based generation module defined in section SECREF12 , on the proxy report section of the AMR Bank. All of our methods out-perform BIBREF0 's method. We obtain best ROGUE-1 INLINEFORM0 scores using the first co-occurrence+first model for important sentences. This also out-perform our Lead-1-AMR baseline by 0.3 ROGUE-1 INLINEFORM1 points. Effects of using JAMR Parser In this subsection we analyze the effect of using JAMR parser for step-1 instead of the gold-standard AMR graphs. First part of table TABREF14 has scores after using the gold-standard AMR graphs. In the second part of table TABREF14 we have included the scores of using the JAMR parser for AMR graph generation. We have used the same Neural AMR for sentence generation in all methods. Scores of all methods including the Lead-1-AMR baseline have dropped significantly. The usage of JAMR Parser has affected the scores of first co-occurrence+first and first-1 more than that for the Lead-1-AMR. The drop in ROGUE INLINEFORM0 score when we use first co-occurrence+first is around two ROGUE INLINEFORM1 points more than when Lead-1-AMR. This is a surprising result, and we believe that it is worthy of further research. Effectiveness of the Generator In this subsection we evaluate the effectiveness of the sentence generation step. For fair comparison at the generation step we use the gold-standard AMRs and don't perform any extraction in step-2 instead we use full AMRs, this allows to remove any errors that might have been generated in step-1 and step-2. In order to compare the quality of sentences generated by the AMR, we need a gold-standard for sentence generation step. For this, we simply use the original sentence as gold-standard for sentence generation. Thus, we compare the quality of summary generated by Lead-1 and Lead-1-AMR. The scores using the ROGUE metric are given in bottom two rows of table TABREF17 . The results show that there is significant drop in Lead-1-AMR when compared to Lead-1. We perform human evaluation to check whether the drop in ROGUE scores is because of drop in information contained, and human readability or is it because of the inability of the ROGUE metric to judge. To perform this evaluation we randomly select ten test examples from the thirty- three test cases of the proxy report section. For each example, we show the summaries generated by four different models side by side to the human evaluators. The human evaluator does not know which summaries come from which model. A score from 1 to 10 is then assigned to each summary on the basis of readability, and information contained of summary, where 1 corresponds to the lower level and 10 to the highest. In table TABREF17 we compare the scores of these four cases as given by ROGUE along with human evaluation. The parser-generator pairs for the four cases are gold-JAMR(generator), JAMR(parser)-neural, gold-neural, and the original sentence respectively. Here gold parser means that we have used the gold-standard AMR graphs. The scores given by the humans do not correlate with ROGUE. Human evaluators gives almost similarly scores to summary generated by the Lead-1 and Lead-1-AMR with Lead-1-AMR actually performing better on readability though it dropped some information as clear from the scores on information contained. On the other hand, ROGUE gives very high score to Lead-1 while models 1,2 and 4 get almost same scores. The similar scores of model 2 and 3 shows that generators are actually producing meaningful sentences. Thus the drop in ROGUE scores is mainly due to the inability of the ROGUE to evaluate abstractive summaries. Moreover, the ROGUE gives model 4 higher score compared to model 1 while human evaluators give the opposite scores on information contained in the sentence. A possible reason for the inability of the ROGUE metric to properly evaluate the summaries generated by our method might be due to its inability to evaluate restructured sentences. AMR formalism tries to assign the same AMR graphs to the sentences that have same meaning so there exists a one-to-many mapping from AMR graphs to sentences. This means that the automatic generators that we are using might not be trying to generate the original sentence; instead it is trying to generate some other sentence that has the same underlying meaning. This also helps in explaining the low ROGUE-2 and ROGUE-L scores. If the sentences might be getting rephrased, they would loose most of the bi- and tri-grams from the original sentence resulting in low ROGUE-2 and ROGUE-L scores. Analyzing the effectiveness of AMR extraction The aim of extracting summary graphs from the AMR graphs of the sentence is to drop the not so important information from the sentences. If we are able to achieve this perfectly, the ROGUE-1 Recall scores that we are getting should remain almost the same (since we are not add any new information) and the ROGUE-1 precision should go up (as we have thrown out some useless information); thus effectively improving the overall ROGUE-1 INLINEFORM0 score. In the first two rows of table TABREF14 we have the scores after using the full-AMR and extracted AMR for generation respectively. It is safe to say that extracting the AMR results in improved ROGUE-1 precision whereas ROGUE-1 Recall reduces only slightly, resulting in an overall improved ROGUE-1 INLINEFORM1 . Results on the CNN-Dailymail corpus In table TABREF18 we report the results on the CNN-Dailymail corpus. We present scores by using the first-3 model. The first row contains the Lead-3-AMR baseline. The results we achieve are competitive with the Lead-3-AMR baseline. The rest of the table contains scores of Lead-3 baseline followed by the state-of-the-art method on the anonymized and non-anonymized versions of the dataset. The drop in the scores from the Lead-3(non-anonymized) to Lead-3-AMR is significant and is largely because of the error introduced by parser and generator. Related Work BIBREF18 showed that most of the work in text summarization has been extractive, where sentences are selected from the text which are then concatenated to form a summary. BIBREF19 transformed the input to nodes, then used the Pagerank algorithm to score nodes, and finally grow the nodes from high-value to low-value using some heuristics. Some of the approaches combine this with sentence compression, so more sentences can be packed in the summary. BIBREF20 , BIBREF21 , BIBREF22 , and BIBREF23 among others used ILPs and approximations for encoding compression and extraction. Recently some abstractive approaches have also been proposed most of which used sequence to sequence learning models for the task. BIBREF24 , BIBREF25 , BIBREF12 , BIBREF17 used standard encoder-decoder models along with their variants to generate summaries. BIBREF26 incorporated the AMR information in the standard encoder-decoder models to improve results. Our work in similar to other graph based abstractive summarization methods BIBREF27 and BIBREF28 . BIBREF27 used dependency parse trees to produce summaries. On the other hand our work takes advantage of semantic graphs. Need of an new Dataset and Evaluation Metric ROGUE metric, by it is design has lots of properties that make it unsuitable for evaluating abstractive summaries. For example, ROGUE matches exact words and not the stems of the words, it also considers stop words for evaluation. One of the reasons why ROGUE like metrics might never become suitable for evaluating abstractive summaries is its incapabilities of knowing if the sentences have been restructured. A good evaluation metric should be one where we compare the meaning of the sentence and not the exact words. As we showed section SECREF20 ROGUE is not suitable for evaluating summaries generated by the AMR pipeline. We now show why the CNN-Dailymail corpus is not suitable for Abstractive summarization. The nature of summary points in the corpus is highly extractive (Section SECREF7 for details) where most of the summary points are simply picked up from some sentences in the story. Tough, this is a good enough reason to start searching for better dataset, it is not the biggest problem with the dataset. The dataset has the property that a lot of important information is in the first few sentences and most of the summary points are directly pick from these sentences. The extractive methods based on sentence selection like SummaRunNer are not actually performing well, the results they have got are only slightly better than the Lead-3 baseline. The work doesn't show how much of the selected sentences are among the first few and it might be the case that the sentences selected by the extractive methods are mostly among the first few sentences, the same can be the problem with the abstractive methods, where most the output might be getting copied from the initial few sentences. These problems with this corpus evoke the need to have another corpus where we don't have so much concentration of important information at any location but rather the information is more spread out and the summaries are more abstractive in nature. Possible Future Directions As this proposed algorithm is a step by step process we can focus on improving each step to produce better results. The most exciting improvements can be done in the summary graph extraction method. Not a lot of work has been done to extract AMR graphs for summaries. In order to make this pipeline generalizable for any sort of text, we need to get rid of the hypothesis that the summary is being extracted exactly from one sentence. So, the natural direction seems to be joining AMR graphs of multiple sentences that are similar and then extracting the summary AMR from that large graph. It will be like clustering similar sentences and then extracting a summary graph from each of these cluster. Another idea is to use AMR graphs for important sentence selection. Conclusion In this work we have explored a full-fledged pipeline using AMR for summarization for the first time. We propose a new method for extracting summary graph, which outperformed previous methods. Overall we provide strong baseline for text summarization using AMR for possible future works. We also showed that ROGUE can't be used for evaluating the abstractive summaries generated by our AMR pipeline.
How are sentences selected from the summary graph?
b65a83a24fc66728451bb063cf6ec50134c8bfb0
How Ġare Ġsentences Ġselected Ġfrom Ġthe Ġsummary Ġgraph ?
Ġfinding Ġthe Ġimportant Ġsentences Ġfrom Ġthe Ġstory , Ġextracting Ġthe Ġkey Ġinformation Ġfrom Ġthose Ġsentences Ġusing Ġtheir ĠAM R Ġgraphs
<s> How Ġare Ġsentences Ġselected Ġfrom Ġthe Ġsummary Ġgraph ? </s> Introduction </s> Sum mar ization Ġof Ġlarge Ġtexts Ġis Ġstill Ġan Ġopen Ġproblem Ġin Ġlanguage Ġprocessing . ĠPeople Ġnowadays Ġhave Ġlesser Ġtime Ġand Ġpatience Ġto Ġgo Ġthrough Ġlarge Ġpieces Ġof Ġtext Ġwhich Ġmake Ġautomatic Ġsummar ization Ġimportant . ĠAutomatic Ġsummar ization Ġhas Ġsignificant Ġapplications Ġin Ġsummar izing Ġlarge Ġtexts Ġlike Ġstories , Ġjournal Ġpapers , Ġnews Ġarticles Ġand Ġeven Ġlarger Ġtexts Ġlike Ġbooks . </s> Ex isting Ġmethods Ġfor Ġsummar ization Ġcan Ġbe Ġbroadly Ġcategorized Ġinto Ġtwo Ġcategories ĠExtract ive Ġand ĠAbstract ive . ĠExtract ive Ġmethods Ġpicks Ġup Ġwords Ġand Ġsometimes Ġdirectly Ġsentences Ġfrom Ġthe Ġtext . ĠThese Ġmethods Ġare Ġinherently Ġlimited Ġin Ġthe Ġsense Ġthat Ġthey Ġcan Ġnever Ġgenerate Ġhuman Ġlevel Ġsumm aries Ġfor Ġlarge Ġand Ġcomplicated Ġdocuments Ġwhich Ġrequire Ġre ph r asing Ġsentences Ġand Ġincorporating Ġinformation Ġfrom Ġfull Ġtext Ġto Ġgenerate Ġsumm aries . ĠMost Ġof Ġthe Ġwork Ġdone Ġon Ġsummar ization Ġin Ġpast Ġhas Ġbeen Ġextract ive . </s> On Ġthe Ġother Ġhand Ġmost ĠAbstract ive Ġmethods Ġtake Ġadvantages Ġof Ġthe Ġrecent Ġdevelopments Ġin Ġdeep Ġlearning . ĠSpecifically Ġthe Ġrecent Ġsuccess Ġof Ġthe Ġsequence Ġto Ġsequence Ġlearning Ġmodels Ġwhere Ġrecurrent Ġnetworks Ġread Ġthe Ġtext , Ġenc odes Ġit Ġand Ġthen Ġgenerate Ġtarget Ġtext . ĠThough Ġthese Ġmethods Ġhave Ġrecently Ġshown Ġto Ġbe Ġcompetitive Ġwith Ġthe Ġextract ive Ġmethods Ġthey Ġare Ġstill Ġfar Ġaway Ġfrom Ġreaching Ġhuman Ġlevel Ġquality Ġin Ġsummary Ġgeneration . </s> The Ġwork Ġon Ġsummar ization Ġusing ĠAM R Ġwas Ġstarted Ġby ĠB IB REF 0 Ġ. ĠAbstract ĠMeaning ĠRepresent ation Ġ( AM R ) Ġwas Ġas Ġintroduced Ġby ĠB IB REF 1 Ġ. ĠAM R Ġfocuses Ġon Ġcapturing Ġthe Ġmeaning Ġof Ġthe Ġtext , Ġby Ġgiving Ġa Ġspecific Ġmeaning Ġrepresentation Ġto Ġthe Ġtext . ĠAM R Ġtries Ġto Ġcapture Ġthe Ġ" who Ġis Ġdoing Ġwhat Ġto Ġwhom " Ġin Ġa Ġsentence . ĠThe Ġformal ism Ġaims Ġto Ġgive Ġsame Ġrepresentation Ġto Ġsentences Ġwhich Ġhave Ġthe Ġsame Ġunderlying Ġmeaning . ĠFor Ġexample Ġ" He Ġlikes Ġapple " Ġand Ġ" App les Ġare Ġliked Ġby Ġhim " Ġshould Ġbe Ġassigned Ġthe Ġsame ĠAM R . </s> BI B REF 0 Ġ' s Ġapproach Ġaimed Ġto Ġproduce Ġa Ġsummary Ġfor Ġa Ġstory Ġby Ġextracting Ġa Ġsummary Ġsub graph Ġfrom Ġthe Ġstory Ġgraph Ġand Ġfinally Ġgenerate Ġa Ġsummary Ġfrom Ġthis Ġextracted Ġgraph . ĠBut , Ġbecause Ġof Ġthe Ġun availability Ġof ĠAM R Ġto Ġtext Ġgenerator Ġat Ġthat Ġtime Ġtheir Ġwork Ġwas Ġlimited Ġtill Ġextracting Ġthe Ġsummary Ġgraph . ĠThis Ġmethod Ġextracts Ġa Ġsingle Ġsummary Ġgraph Ġfrom Ġthe Ġstory Ġgraph . ĠExtract ing Ġa Ġsingle Ġsummary Ġgraph Ġassumes Ġthat Ġall Ġof Ġthe Ġimportant Ġinformation Ġfrom Ġthe Ġgraph Ġcan Ġbe Ġextracted Ġfrom Ġa Ġsingle Ġsub graph . ĠBut , Ġit Ġcan Ġbe Ġdifficult Ġin Ġcases Ġwhere Ġthe Ġinformation Ġis Ġspread Ġout Ġin Ġthe Ġgraph . ĠThus , Ġthe Ġmethod Ġcompromises Ġbetween Ġsize Ġof Ġthe Ġsummary Ġsub - graph Ġand Ġthe Ġamount Ġof Ġinformation Ġit Ġcan Ġextract . ĠThis Ġcan Ġbe Ġeasily Ġsolved Ġif Ġinstead Ġof Ġa Ġsingle Ġsub - graph , Ġwe Ġextract Ġmultiple Ġsub graph s Ġeach Ġfocusing Ġon Ġinformation Ġin Ġa Ġdifferent Ġpart Ġof Ġthe Ġstory . </s> We Ġpropose Ġa Ġtwo Ġstep Ġprocess Ġfor Ġextracting Ġmultiple Ġsummary Ġgraphs . ĠFirst Ġstep Ġis Ġto Ġselect Ġfew Ġsentences Ġfrom Ġthe Ġstory . ĠWe Ġuse Ġthe Ġidea Ġthat Ġthere Ġare Ġonly Ġfew Ġsentences Ġthat Ġare Ġimportant Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġsummary , Ġi . e . Ġmost Ġof Ġthe Ġinformation Ġcontained Ġin Ġthe Ġsummary Ġis Ġpresent Ġin Ġvery Ġfew Ġsentences Ġand Ġthey Ġcan Ġbe Ġused Ġto Ġgenerate Ġthe Ġsummary . ĠSecond Ġstep Ġis Ġto Ġextract Ġimportant Ġinformation Ġfrom Ġthe Ġselected Ġsentences Ġby Ġextracting Ġa Ġsub - graph Ġfrom Ġthe Ġselected Ġsentences . </s> Our Ġmain Ġcontributions Ġin Ġthis Ġwork Ġare Ġthree Ġfolds , </s> The Ġrest Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠSEC REF 2 Ġcontains Ġintroduction Ġto ĠAM R , Ġsection ĠSEC REF 3 Ġand ĠSEC REF 4 Ġcontains Ġthe Ġdatasets Ġand Ġthe Ġalgorithm Ġused Ġfor Ġsummary Ġgeneration Ġrespectively . ĠSection ĠSEC REF 5 Ġhas Ġa Ġdetailed Ġstep - by - step Ġevaluation Ġof Ġthe Ġpipeline Ġand Ġin Ġsection ĠSEC REF 6 Ġwe Ġdiscuss Ġthe Ġproblems Ġwith Ġthe Ġcurrent Ġdataset Ġand Ġevaluation Ġmetric . </s> Background : ĠAM R ĠPars ing Ġand ĠGeneration </s> AM R Ġwas Ġintroduced Ġby ĠB IB REF 1 Ġwith Ġthe Ġaim Ġto Ġinduce Ġwork Ġon Ġstatistical ĠNatural ĠLanguage ĠUnderstanding Ġand ĠGeneration . ĠAM R Ġrepresents Ġmeaning Ġusing Ġgraphs . ĠAM R Ġgraphs Ġare Ġrooted , Ġdirected , Ġedge Ġand Ġvertex Ġlabeled Ġgraphs . ĠFigure ĠFIG REF 4 Ġshows Ġthe Ġgraphical Ġrepresentation Ġof Ġthe ĠAM R Ġgraph Ġof Ġthe Ġsentence Ġ" I Ġlooked Ġcarefully Ġall Ġaround Ġme " Ġgenerated Ġby ĠJ AM R Ġparser Ġ( ĠB IB REF 2 Ġ). ĠThe Ġgraphical Ġrepresentation Ġwas Ġproduced Ġusing ĠAM R ICA ĠB IB REF 3 Ġ. ĠThe Ġnodes Ġin Ġthe ĠAM R Ġare Ġlabeled Ġwith Ġconcepts Ġas Ġin ĠFigure ĠFIG REF 4 Ġaround Ġrepresents Ġa Ġconcept . ĠEd ges Ġcontains Ġthe Ġinformation Ġregarding Ġthe Ġrelations Ġbetween Ġthe Ġconcepts . ĠIn ĠFigure ĠFIG REF 4 Ġdirection Ġis Ġthe Ġrelation Ġbetween Ġthe Ġconcepts Ġlook - 01 Ġand Ġaround . ĠAM R Ġrelies Ġon ĠProp bank Ġfor Ġsemantic Ġrelations Ġ( edge Ġlabels ). ĠConcepts Ġcan Ġalso Ġbe Ġof Ġthe Ġform Ġrun - 01 Ġwhere Ġthe Ġindex Ġ01 Ġrepresents Ġthe Ġfirst Ġsense Ġof Ġthe Ġword Ġrun . ĠFurther Ġdetails Ġabout Ġthe ĠAM R Ġcan Ġbe Ġfound Ġin Ġthe ĠAM R Ġguidelines ĠB IB REF 4 Ġ. </s> A Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġon Ġparsing Ġsentences Ġto Ġtheir ĠAM R Ġgraphs . ĠThere Ġare Ġthree Ġmain Ġapproaches Ġto Ġparsing . ĠThere Ġis Ġalignment Ġbased Ġparsing ĠB IB REF 2 Ġ( J AM R - Parser ), ĠB IB REF 5 Ġwhich Ġuses Ġgraph Ġbased Ġalgorithms Ġfor Ġconcept Ġand Ġrelation Ġidentification . ĠSecond , Ġgrammar Ġbased Ġpars ers Ġlike ĠB IB REF 6 Ġ( C AM R ) Ġgenerate Ġoutput Ġby Ġperforming Ġshift Ġreduce Ġtransformations Ġon Ġoutput Ġof Ġa Ġdependency Ġparser . ĠNeural Ġparsing ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġis Ġbased Ġon Ġusing Ġseq 2 seq Ġmodels Ġfor Ġparsing , Ġthe Ġmain Ġproblem Ġfor Ġneural Ġmethods Ġis Ġthe Ġabsence Ġof Ġa Ġhuge Ġcorpus Ġof Ġhuman Ġgenerated ĠAM Rs . ĠB IB REF 8 Ġreduced Ġthe Ġvocabulary Ġsize Ġto Ġtackle Ġthis Ġproblem Ġwhile ĠB IB REF 7 Ġused Ġlarger Ġexternal Ġcorpus Ġof Ġexternal Ġsentences . </s> Recently , Ġsome Ġwork Ġhas Ġbeen Ġdone Ġon Ġproducing Ġmeaningful Ġsentences Ġform ĠAM R Ġgraphs . ĠB IB REF 2 Ġused Ġa Ġnumber Ġof Ġtree Ġto Ġstring Ġconversion Ġrules Ġfor Ġgenerating Ġsentences . ĠB IB REF 9 Ġreformed Ġthe Ġproblem Ġas Ġa Ġtraveling Ġsalesman Ġproblem . ĠB IB REF 7 Ġused Ġseq 2 seq Ġlearning Ġmethods . </s> Dat as ets </s> We Ġused Ġtwo Ġdatasets Ġfor Ġthe Ġtask Ġ- ĠAM R ĠBank ĠB IB REF 10 Ġand ĠCNN - Daily mail Ġ( ĠB IB REF 11 ĠB IB REF 12 Ġ). ĠWe Ġuse Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R ĠBank , Ġas Ġit Ġis Ġthe Ġonly Ġone Ġthat Ġis Ġrelevant Ġfor Ġthe Ġtask Ġbecause Ġit Ġcontains Ġthe Ġgold - standard Ġ( human Ġgenerated ) ĠAM R Ġgraphs Ġfor Ġnews Ġarticles , Ġand Ġthe Ġsumm aries . ĠIn Ġthe Ġtraining Ġset Ġthe Ġstories Ġand Ġsumm aries Ġcontain Ġ17 . 5 Ġsentences Ġand Ġ1 . 5 Ġsentences Ġon Ġan Ġaverage Ġrespectively . ĠThe Ġtraining Ġand Ġtest Ġsets Ġcontain Ġ298 Ġand Ġ33 Ġsummary Ġdocument Ġpairs Ġrespectively . </s> CNN - Daily mail Ġcorpus Ġis Ġbetter Ġsuited Ġfor Ġsummar ization Ġas Ġthe Ġaverage Ġsummary Ġsize Ġis Ġaround Ġ3 Ġor Ġ4 Ġsentences . ĠThis Ġdataset Ġhas Ġaround Ġ300 k Ġdocument Ġsummary Ġpairs Ġwith Ġstories Ġhaving Ġ39 Ġsentences Ġon Ġaverage . ĠThe Ġdataset Ġcomes Ġin Ġ2 Ġversions , Ġone Ġis Ġthe Ġanonym ized Ġversion , Ġwhich Ġhas Ġbeen Ġpre process ed Ġto Ġreplace Ġnamed Ġentities , Ġe . g ., ĠThe ĠTimes Ġof ĠIndia , Ġwith Ġa Ġunique Ġidentifier Ġfor Ġexample Ġ@ entity 1 . ĠSecond Ġis Ġthe Ġnon - an onym ized Ġwhich Ġhas Ġthe Ġoriginal Ġtext . ĠWe Ġuse Ġthe Ġnon - an onym ized Ġversion Ġof Ġthe Ġdataset Ġas Ġit Ġis Ġmore Ġsuitable Ġfor ĠAM R Ġparsing Ġas Ġmost Ġof Ġthe Ġpars ers Ġhave Ġbeen Ġtrained Ġon Ġnon - an onym ized Ġtext . ĠThe Ġdataset Ġdoes Ġnot Ġhave Ġgold - standard ĠAM R Ġgraphs . ĠWe Ġuse Ġautomatic Ġpars ers Ġto Ġget Ġthe ĠAM R Ġgraphs Ġbut Ġthey Ġare Ġnot Ġgold - standard Ġand Ġwill Ġeffect Ġthe Ġquality Ġof Ġfinal Ġsummary . ĠTo Ġget Ġan Ġidea Ġof Ġthe Ġerror Ġintroduced Ġby Ġusing Ġautomatic Ġpars ers , Ġwe Ġcompare Ġthe Ġresults Ġafter Ġusing Ġgold - standard Ġand Ġautomatically Ġgenerated ĠAM R Ġgraphs Ġon Ġthe Ġgold - standard Ġdataset . </s> P ip eline Ġfor ĠSummary ĠGeneration </s> The Ġpipeline Ġconsists Ġof Ġthree Ġsteps , Ġfirst Ġconvert Ġall Ġthe Ġgiven Ġstory Ġsentences Ġto Ġthere ĠAM R Ġgraphs Ġfollowed Ġby Ġextracting Ġsummary Ġgraphs Ġfrom Ġthe Ġstory Ġsentence Ġgraphs Ġand Ġfinally Ġgenerating Ġsentences Ġfrom Ġthese Ġextracted Ġsummary Ġgraphs . ĠIn Ġthe Ġfollowing Ġsubsections Ġwe Ġexplain Ġeach Ġof Ġthe Ġmethods Ġin Ġgreater Ġdetail . </s> Step Ġ1 : ĠStory Ġto ĠAM R </s> As Ġthe Ġfirst Ġstep Ġwe Ġconvert Ġthe Ġstory Ġsentences Ġto Ġtheir ĠAbstract ĠMeaning ĠRepresent ations . ĠWe Ġuse ĠJ AM R - Parser Ġversion Ġ2 ĠB IB REF 2 Ġas Ġit âĢ Ļ s Ġopenly Ġavailable Ġand Ġhas Ġa Ġperformance Ġclose Ġto Ġthe Ġstate Ġof Ġthe Ġart Ġpars ers Ġfor Ġparsing Ġthe ĠCNN - Daily mail Ġcorpus . ĠFor Ġthe ĠAM R - bank Ġwe Ġhave Ġthe Ġgold - standard ĠAM R Ġpars es Ġbut Ġwe Ġstill Ġparse Ġthe Ġinput Ġstories Ġwith ĠJ AM R - Parser Ġto Ġstudy Ġthe Ġeffect Ġof Ġusing Ġgraphs Ġproduced Ġby ĠJ AM R - Parser Ġinstead Ġof Ġthe Ġgold - standard ĠAM R Ġgraphs . </s> Step Ġ2 : ĠStory ĠAM R Ġto ĠSummary ĠAM R </s> After Ġparsing Ġ( Step Ġ1 ) Ġwe Ġhave Ġthe ĠAM R Ġgraphs Ġfor Ġthe Ġstory Ġsentences . ĠIn Ġthis Ġstep Ġwe Ġextract Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġsummary Ġsentences Ġusing Ġstory Ġsentence ĠAM Rs . ĠWe Ġdivide Ġthis Ġtask Ġin Ġtwo Ġparts . ĠFirst Ġis Ġfinding Ġthe Ġimportant Ġsentences Ġfrom Ġthe Ġstory Ġand Ġthen Ġextracting Ġthe Ġkey Ġinformation Ġfrom Ġthose Ġsentences Ġusing Ġtheir ĠAM R Ġgraphs . </s> Our Ġalgorithm Ġis Ġbased Ġon Ġthe Ġidea Ġthat Ġonly Ġfew Ġsentences Ġare Ġimportant Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġsummary Ġi . e . Ġthere Ġare Ġonly Ġa Ġfew Ġsentences Ġwhich Ġcontain Ġmost Ġof Ġthe Ġimportant Ġinformation Ġand Ġfrom Ġthese Ġsentences Ġwe Ġcan Ġgenerate Ġthe Ġsummary . </s> Hyp ot hesis : ĠMost Ġof Ġthe Ġinformation Ġcorresponding Ġto Ġa Ġsummary Ġsentence Ġcan Ġbe Ġfound Ġin Ġonly Ġone Ġsentence Ġfrom Ġthe Ġstory . </s> To Ġtest Ġthis Ġhypothesis , Ġfor Ġeach Ġsummary Ġsentence Ġwe Ġfind Ġthe Ġsentence Ġfrom Ġthe Ġstory Ġthat Ġcontains Ġmaximum Ġinformation Ġof Ġthis Ġsummary Ġsentence . ĠWe Ġuse ĠR OG UE - 1 ĠB IB REF 13 ĠRecall Ġscores Ġ( measures Ġthe Ġratio Ġof Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsummary Ġthat Ġare Ġcontained Ġin Ġthe Ġpredicted Ġsummary Ġto Ġthe Ġtotal Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsummary ) Ġas Ġthe Ġmetric Ġfor Ġthe Ġinformation Ġcontained Ġin Ġthe Ġstory Ġsentence . ĠWe Ġconsider Ġthe Ġstory Ġsentence Ġas Ġthe Ġpredicted Ġsummary Ġand Ġthe Ġsummary Ġsentence Ġas Ġthe Ġtarget Ġsummary . ĠThe Ġresults Ġthat Ġwe Ġobtained Ġfor Ġ5000 Ġrandomly Ġchosen Ġdocument Ġsummary Ġpairs Ġfrom Ġthe ĠCNN - Daily mail Ġcorpus Ġare Ġgiven Ġin Ġfigure ĠFIG REF 8 Ġ. ĠThe Ġaverage Ġrecall Ġscore Ġthat Ġwe Ġobtained Ġis Ġ79 %. ĠThe Ġscore Ġwill Ġbe Ġperfectly Ġ1 Ġwhen Ġthe Ġsummary Ġsentence Ġis Ġdirectly Ġpicked Ġup Ġfrom Ġa Ġstory Ġsentence . ĠUpon Ġmanual Ġinspection Ġof Ġthe Ġsummary Ġsentence Ġand Ġthe Ġcorresponding Ġbest Ġsentence Ġfrom Ġthe Ġstory Ġwe Ġrealized , Ġwhen Ġthis Ġscore Ġis Ġmore Ġthan Ġ0 . 5 Ġor Ġ0 . 6 , Ġalmost Ġalways Ġthe Ġinformation Ġin Ġthe Ġsummary Ġsentence Ġis Ġcontained Ġin Ġthis Ġchosen Ġstory Ġsentence . ĠThe Ġscore Ġfor Ġin Ġthese Ġcases Ġis Ġnot Ġperfectly Ġ1 Ġbecause Ġof Ġstop Ġwords Ġand Ġdifferent Ġverb Ġforms Ġused Ġin Ġstory Ġand Ġsummary Ġsentence . ĠAround Ġ80 % Ġof Ġsummary Ġsentences Ġhave Ġscore Ġabove Ġ0 . 5 . ĠSo , Ġour Ġhypothesis Ġseems Ġto Ġbe Ġcorrect Ġfor Ġmost Ġof Ġthe Ġsummary Ġsentences . ĠThis Ġalso Ġsuggests Ġthe Ġhighly Ġextract ive Ġnature Ġof Ġthe Ġsummary Ġin Ġthe Ġcorpus . </s> Now Ġthe Ġtask Ġin Ġhand Ġis Ġto Ġselect Ġfew Ġimportant Ġsentences . ĠMethods Ġthat Ġuse Ġsentence Ġextraction Ġfor Ġsummary Ġgeneration Ġcan Ġbe Ġused Ġfor Ġthe Ġtask . ĠIt Ġis Ġvery Ġcommon Ġin Ġsummar ization Ġtasks Ġspecifically Ġin Ġnews Ġarticles Ġthat Ġa Ġlot Ġof Ġinformation Ġis Ġcontained Ġin Ġthe Ġinitial Ġfew Ġsentences . ĠCho osing Ġinitial Ġfew Ġsentences Ġas Ġthe Ġsummary Ġproduces Ġvery Ġstrong Ġbas elines Ġwhich Ġthe Ġstate - of - the - art Ġmethods Ġbeat Ġonly Ġmarginally . ĠEx . ĠOn Ġthe ĠCNN - Daily mail Ġcorpus Ġthe Ġstate - of - the - art Ġextract ive Ġmethod Ġbeats Ġinitial Ġ3 Ġsentences Ġonly Ġby Ġ0 . 4 % Ġas Ġreported Ġby ĠB IB REF 14 Ġ. </s> Using Ġthis Ġidea Ġof Ġpicking Ġimportant Ġsentences Ġfrom Ġthe Ġbeginning , Ġwe Ġpropose Ġtwo Ġmethods , Ġfirst Ġis Ġto Ġsimply Ġpick Ġinitial Ġfew Ġsentences , Ġwe Ġcall Ġthis Ġfirst - n Ġmethod Ġwhere Ġn Ġstands Ġfor Ġthe Ġnumber Ġof Ġsentences . ĠWe Ġpick Ġinitial Ġ3 Ġsentences Ġfor Ġthe ĠCNN - Daily mail Ġcorpus Ġi . e . Ġfirst - 3 Ġand Ġonly Ġthe Ġfirst Ġsentence Ġfor Ġthe Ġproxy Ġreport Ġsection Ġ( AM R ĠBank ) Ġi . e . Ġfirst - 1 Ġas Ġthey Ġproduce Ġthe Ġbest Ġscores Ġon Ġthe ĠR OG UE Ġmetric Ġcompared Ġto Ġany Ġother Ġfirst - n . ĠSecond , Ġwe Ġtry Ġto Ġcapture Ġthe Ġrelation Ġbetween Ġthe Ġtwo Ġmost Ġimportant Ġentities Ġ( we Ġdefine Ġimportance Ġby Ġthe Ġnumber Ġof Ġoccurrences Ġof Ġthe Ġentity Ġin Ġthe Ġstory ) Ġof Ġthe Ġdocument . ĠFor Ġthis Ġwe Ġsimply Ġfind Ġthe Ġfirst Ġsentence Ġwhich Ġcontains Ġboth Ġthese Ġentities . ĠWe Ġcall Ġthis Ġthe Ġfirst Ġco - occ urrence Ġbased Ġsentence Ġselection . ĠWe Ġalso Ġselect Ġthe Ġfirst Ġsentence Ġalong Ġwith Ġfirst Ġco - occ urrence Ġbased Ġsentence Ġselection Ġas Ġthe Ġimportant Ġsentences . ĠWe Ġcall Ġthis Ġthe Ġfirst Ġco - occ urrence + first Ġbased Ġsentence Ġselection . </s> As Ġthe Ġdatasets Ġunder Ġconsideration Ġare Ġnews Ġarticles . ĠThe Ġmost Ġimportant Ġinformation Ġin Ġthem Ġis Ġabout Ġan Ġentity Ġand Ġa Ġverb Ġassociated Ġwith Ġit . ĠSo , Ġto Ġextract Ġimportant Ġinformation Ġfrom Ġthe Ġsentence . ĠWe Ġtry Ġto Ġfind Ġthe Ġentity Ġbeing Ġtalked Ġabout Ġin Ġthe Ġsentence , Ġwe Ġconsider Ġthe Ġmost Ġreferred Ġentity Ġ( one Ġthat Ġoccurs Ġmost Ġfrequently Ġin Ġthe Ġtext ), Ġnow Ġfor Ġthe Ġmain Ġverb Ġassociated Ġwith Ġthe Ġentity Ġin Ġthe Ġsentence , Ġwe Ġfind Ġthe Ġverb Ġclosest Ġto Ġthis Ġentity Ġin Ġthe ĠAM R Ġgraph . ĠWe Ġdefine Ġthe Ġclosest Ġverb Ġas Ġthe Ġone Ġwhich Ġlies Ġfirst Ġin Ġthe Ġpath Ġfrom Ġthe Ġentity Ġto Ġthe Ġroot . </s> We Ġstart Ġby Ġfinding Ġthe Ġposition Ġof Ġthe Ġmost Ġreferred Ġentity Ġin Ġthe Ġgraph , Ġthen Ġwe Ġfind Ġthe Ġclosest Ġverb Ġto Ġthe Ġentity . Ġand Ġfinally Ġselect Ġthe Ġsubt ree Ġhanging Ġfrom Ġthat Ġverb Ġas Ġthe Ġsummary ĠAM R . </s> Step Ġ3 : ĠSummary ĠGeneration </s> To Ġgenerate Ġsentences Ġfrom Ġthe Ġextracted ĠAM R Ġgraphs Ġwe Ġcan Ġuse Ġalready Ġavailable Ġgenerators . ĠWe Ġuse ĠNeural ĠAM R Ġ( ĠB IB REF 7 Ġ) Ġas Ġit Ġprovides Ġstate Ġof Ġthe Ġart Ġresults Ġin Ġsentence Ġgeneration . ĠWe Ġalso Ġuse ĠB IB REF 15 Ġ( J AM R - Gener ator ) Ġin Ġone Ġof Ġthe Ġexperiments Ġin Ġthe Ġnext Ġsection . ĠGener ators Ġsignificantly Ġeffect Ġthe Ġresults , Ġwe Ġwill Ġanalyze Ġthe Ġeffectiveness Ġof Ġgenerator Ġin Ġthe Ġnext Ġsection . </s> Bas elines </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġbaseline Ġmodels Ġand Ġanalysis Ġmethod Ġused Ġfor Ġeach Ġstep Ġof Ġour Ġpipeline . </s> For Ġthe ĠCNN - Daily mail Ġdataset , Ġthe ĠLead - 3 Ġmodel Ġis Ġconsidered Ġa Ġstrong Ġbaseline ; Ġboth Ġthe Ġabstract ive ĠB IB REF 16 Ġand Ġextract ive ĠB IB REF 14 Ġstate - of - the Ġart Ġmethods Ġon Ġthis Ġdataset Ġbeat Ġthis Ġbaseline Ġonly Ġmarginally . ĠThe ĠLead - 3 Ġmodel Ġsimply Ġproduces Ġthe Ġleading Ġthree Ġsentences Ġof Ġthe Ġdocument Ġas Ġits Ġsummary . </s> The Ġkey Ġstep Ġin Ġour Ġpipeline Ġis Ġstep - 2 Ġi . e . Ġsummary Ġgraph Ġextraction . </s> Direct ly Ġcomparing Ġthe ĠLead - 3 Ġbaseline , Ġwith ĠAM R Ġbased Ġpipeline Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġstep - 2 Ġis Ġan Ġunfair Ġcomparison Ġbecause Ġof Ġthe Ġerrors Ġintroduced Ġby Ġimperfect Ġparser Ġand Ġgenerator Ġin Ġthe ĠAM R Ġpipeline . ĠThus Ġto Ġevaluate Ġthe Ġeffectiveness Ġof Ġstep - 2 Ġagainst ĠLead - 3 Ġbaseline , Ġwe Ġneed Ġto Ġnull ify Ġthe Ġeffect Ġof Ġerrors Ġintroduce Ġby ĠAM R Ġparser Ġand Ġgenerator . ĠWe Ġachieve Ġthis Ġby Ġtrying Ġto Ġintroduce Ġsimilar Ġerrors Ġin Ġthe Ġleading Ġth re Ġsentences Ġof Ġeach Ġdocument . ĠWe Ġgenerate Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġleading Ġthree Ġsentences Ġand Ġthen Ġgenerate Ġthe Ġsentences Ġusing Ġthese ĠAM R Ġgraph . ĠWe Ġuse Ġparser Ġand Ġgenerator Ġthat Ġwere Ġused Ġin Ġour Ġpipeline . ĠWe Ġconsider Ġthese Ġgenerated Ġsentences Ġas Ġthe Ġnew Ġbaseline Ġsummary , Ġwe Ġshall Ġnow Ġrefer Ġto Ġit ĠLead - 3 - AM R Ġbaseline Ġin Ġthe Ġremaining Ġof Ġthe Ġpaper . </s> For Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R Ġbank , Ġwe Ġconsider Ġthe ĠLead - 1 - AM R Ġmodel Ġas Ġthe Ġbaseline . ĠFor Ġthis Ġdataset Ġwe Ġalready Ġhave Ġthe Ġgold - standard ĠAM R Ġgraphs Ġof Ġthe Ġsentences . ĠTherefore , Ġwe Ġonly Ġneed Ġto Ġnull ify Ġthe Ġerror Ġintroduced Ġby Ġthe Ġgenerator . </s> Pro ced ure Ġto ĠAnaly ze Ġand ĠEval uate Ġeach Ġstep </s> For Ġthe Ġevaluation Ġof Ġsumm aries Ġwe Ġuse Ġthe Ġstandard ĠR OG UE Ġmetric . ĠFor Ġcomparison Ġwith Ġprevious ĠAM R Ġbased Ġsummar ization Ġmethods , Ġwe Ġreport Ġthe ĠRecall , ĠPrecision Ġand ĠIN LINE FORM 0 Ġscores Ġfor ĠR OG UE - 1 . ĠSince Ġmost Ġof Ġthe Ġliterature Ġon Ġsummar ization Ġuses ĠIN LINE FORM 1 Ġscores Ġfor ĠR OG UE - 2 Ġand ĠR OG UE - L Ġfor Ġcomparison , Ġwe Ġalso Ġreport ĠIN LINE FORM 2 Ġscores Ġfor ĠR OG UE - 2 Ġand ĠR OG UE - L Ġfor Ġour Ġmethod . ĠR OG UE - 1 ĠRecall Ġand ĠPrecision Ġare Ġmeasured Ġfor Ġun i - gram Ġoverlap Ġbetween Ġthe Ġreference Ġand Ġthe Ġpredicted Ġsummary . ĠOn Ġthe Ġother Ġhand , ĠR OG UE - 2 Ġuses Ġbi - gram Ġoverlap Ġwhile ĠR OG UE - L Ġuses Ġthe Ġlongest Ġcommon Ġsequence Ġbetween Ġthe Ġtarget Ġand Ġthe Ġpredicted Ġsumm aries Ġfor Ġevaluation . ĠIn Ġrest Ġof Ġthis Ġsection , Ġwe Ġprovide Ġmethods Ġto Ġanalyze Ġand Ġevaluate Ġour Ġpipeline Ġat Ġeach Ġstep . </s> Step - 1 : ĠAM R Ġparsing ĠTo Ġunderstand Ġthe Ġeffects Ġof Ġusing Ġan ĠAM R Ġparser Ġon Ġthe Ġresults , Ġwe Ġcompare Ġthe Ġfinal Ġscores Ġafter Ġthe Ġfollowing Ġtwo Ġcases - Ġfirst , Ġwhen Ġwe Ġuse Ġthe Ġgold - standard ĠAM R Ġgraphs Ġand Ġsecond Ġwhen Ġwe Ġused Ġthe ĠAM R Ġgraphs Ġgenerated Ġby ĠJ AM R - Parser Ġin Ġthe Ġpipeline . ĠSection ĠSEC REF 19 Ġcontains Ġa Ġcomparison Ġbetween Ġthe Ġtwo . </s> Step - 2 : ĠSummary Ġgraph Ġextraction ĠFor Ġevaluating Ġthe Ġeffectiveness Ġof Ġthe Ġsummary Ġgraph Ġextraction Ġstep Ġwe Ġcompare Ġthe Ġfinal Ġscores Ġwith Ġthe ĠLead - n - AM R Ġbas elines Ġdescribed Ġin Ġsection ĠSEC REF 12 Ġ. </s> In Ġorder Ġto Ġcompare Ġour Ġsummary Ġgraph Ġextraction Ġstep Ġwith Ġthe Ġprevious Ġwork Ġ( ĠB IB REF 0 Ġ), Ġwe Ġgenerate Ġthe Ġfinal Ġsummary Ġusing Ġthe Ġsame Ġgeneration Ġmethod Ġas Ġused Ġby Ġthem . ĠTheir Ġmethod Ġuses Ġa Ġsimple Ġmodule Ġbased Ġon Ġalign ments Ġfor Ġgenerating Ġsummary Ġafter Ġstep - 2 . ĠThe Ġalign ments Ġsimply Ġmap Ġthe Ġwords Ġin Ġthe Ġoriginal Ġsentence Ġwith Ġthe Ġnode Ġor Ġedge Ġin Ġthe ĠAM R Ġgraph . ĠTo Ġgenerate Ġthe Ġsummary Ġwe Ġfind Ġthe Ġwords Ġaligned Ġwith Ġthe Ġsentence Ġin Ġthe Ġselected Ġgraph Ġand Ġoutput Ġthem Ġin Ġno Ġparticular Ġorder Ġas Ġthe Ġpredicted Ġsummary . ĠThough Ġthis Ġdoes Ġnot Ġgenerate Ġgram matically Ġcorrect Ġsentences , Ġwe Ġcan Ġstill Ġuse Ġthe ĠR OG UE - 1 Ġmetric Ġsimilar Ġto ĠB IB REF 0 Ġ, Ġas Ġit Ġis Ġbased Ġon Ġcomparing Ġun i - gram s Ġbetween Ġthe Ġtarget Ġand Ġpredicted Ġsumm aries . </s> Step - 3 : ĠGeneration ĠFor Ġevaluating Ġthe Ġquality Ġof Ġthe Ġsentences Ġgenerated Ġby Ġour Ġmethod , Ġwe Ġcompare Ġthe Ġsumm aries Ġgenerated Ġby Ġthe Ġfirst - 1 Ġmodel Ġand ĠLead - 1 - AM R Ġmodel Ġon Ġthe Ġgold - standard Ġdataset . ĠHowever , Ġwhen Ġwe Ġlooked Ġat Ġthe Ġscores Ġgiven Ġby ĠR OG UE , Ġwe Ġdecided Ġto Ġdo Ġget Ġthe Ġabove Ġsumm aries Ġevaluated Ġby Ġhumans . ĠThis Ġproduced Ġinteresting Ġresults Ġwhich Ġare Ġgiven Ġin Ġmore Ġdetail Ġin Ġsection ĠSEC REF 20 Ġ. </s> Results Ġon Ġthe ĠProxy Ġreport Ġsection </s> In Ġtable ĠT AB REF 13 Ġwe Ġreport Ġthe Ġresults Ġof Ġusing Ġthe Ġpipeline Ġwith Ġgeneration Ġusing Ġthe Ġalignment Ġbased Ġgeneration Ġmodule Ġdefined Ġin Ġsection ĠSEC REF 12 Ġ, Ġon Ġthe Ġproxy Ġreport Ġsection Ġof Ġthe ĠAM R ĠBank . ĠAll Ġof Ġour Ġmethods Ġout - per form ĠB IB REF 0 Ġ' s Ġmethod . ĠWe Ġobtain Ġbest ĠR OG UE - 1 ĠIN LINE FORM 0 Ġscores Ġusing Ġthe Ġfirst Ġco - occ urrence + first Ġmodel Ġfor Ġimportant Ġsentences . ĠThis Ġalso Ġout - per form Ġour ĠLead - 1 - AM R Ġbaseline Ġby Ġ0 . 3 ĠR OG UE - 1 ĠIN LINE FORM 1 Ġpoints . </s> Effects Ġof Ġusing ĠJ AM R ĠPars er </s> In Ġthis Ġsubsection Ġwe Ġanalyze Ġthe Ġeffect Ġof Ġusing ĠJ AM R Ġparser Ġfor Ġstep - 1 Ġinstead Ġof Ġthe Ġgold - standard ĠAM R Ġgraphs . ĠFirst Ġpart Ġof Ġtable ĠT AB REF 14 Ġhas Ġscores Ġafter Ġusing Ġthe Ġgold - standard ĠAM R Ġgraphs . ĠIn Ġthe Ġsecond Ġpart Ġof Ġtable ĠT AB REF 14 Ġwe Ġhave Ġincluded Ġthe Ġscores Ġof Ġusing Ġthe ĠJ AM R Ġparser Ġfor ĠAM R Ġgraph Ġgeneration . ĠWe Ġhave Ġused Ġthe Ġsame ĠNeural ĠAM R Ġfor Ġsentence Ġgeneration Ġin Ġall Ġmethods . ĠScores Ġof Ġall Ġmethods Ġincluding Ġthe ĠLead - 1 - AM R Ġbaseline Ġhave Ġdropped Ġsignificantly . </s> The Ġusage Ġof ĠJ AM R ĠPars er Ġhas Ġaffected Ġthe Ġscores Ġof Ġfirst Ġco - occ urrence + first Ġand Ġfirst - 1 Ġmore Ġthan Ġthat Ġfor Ġthe ĠLead - 1 - AM R . ĠThe Ġdrop Ġin ĠR OG UE ĠIN LINE FORM 0 Ġscore Ġwhen Ġwe Ġuse Ġfirst Ġco - occ urrence + first Ġis Ġaround Ġtwo ĠR OG UE ĠIN LINE FORM 1 Ġpoints Ġmore Ġthan Ġwhen ĠLead - 1 - AM R . ĠThis Ġis Ġa Ġsurprising Ġresult , Ġand Ġwe Ġbelieve Ġthat Ġit Ġis Ġworthy Ġof Ġfurther Ġresearch . </s> Effect iveness Ġof Ġthe ĠGenerator </s> In Ġthis Ġsubsection Ġwe Ġevaluate Ġthe Ġeffectiveness Ġof Ġthe Ġsentence Ġgeneration Ġstep . ĠFor Ġfair Ġcomparison Ġat Ġthe Ġgeneration Ġstep Ġwe Ġuse Ġthe Ġgold - standard ĠAM Rs Ġand Ġdon 't Ġperform Ġany Ġextraction Ġin Ġstep - 2 Ġinstead Ġwe Ġuse Ġfull ĠAM Rs , Ġthis Ġallows Ġto Ġremove Ġany Ġerrors Ġthat Ġmight Ġhave Ġbeen Ġgenerated Ġin Ġstep - 1 Ġand Ġstep - 2 . ĠIn Ġorder Ġto Ġcompare Ġthe Ġquality Ġof Ġsentences Ġgenerated Ġby Ġthe ĠAM R , Ġwe Ġneed Ġa Ġgold - standard Ġfor Ġsentence Ġgeneration Ġstep . ĠFor Ġthis , Ġwe Ġsimply Ġuse Ġthe Ġoriginal Ġsentence Ġas Ġgold - standard Ġfor Ġsentence Ġgeneration . ĠThus , Ġwe Ġcompare Ġthe Ġquality Ġof Ġsummary Ġgenerated Ġby ĠLead - 1 Ġand ĠLead - 1 - AM R . ĠThe Ġscores Ġusing Ġthe ĠR OG UE Ġmetric Ġare Ġgiven Ġin Ġbottom Ġtwo Ġrows Ġof Ġtable ĠT AB REF 17 Ġ. ĠThe Ġresults Ġshow Ġthat Ġthere Ġis Ġsignificant Ġdrop Ġin ĠLead - 1 - AM R Ġwhen Ġcompared Ġto ĠLead - 1 . </s> We Ġperform Ġhuman Ġevaluation Ġto Ġcheck Ġwhether Ġthe Ġdrop Ġin ĠR OG UE Ġscores Ġis Ġbecause Ġof Ġdrop Ġin Ġinformation Ġcontained , Ġand Ġhuman Ġread ability Ġor Ġis Ġit Ġbecause Ġof Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġmetric Ġto Ġjudge . ĠTo Ġperform Ġthis Ġevaluation Ġwe Ġrandomly Ġselect Ġten Ġtest Ġexamples Ġfrom Ġthe Ġthirty - Ġthree Ġtest Ġcases Ġof Ġthe Ġproxy Ġreport Ġsection . ĠFor Ġeach Ġexample , Ġwe Ġshow Ġthe Ġsumm aries Ġgenerated Ġby Ġfour Ġdifferent Ġmodels Ġside Ġby Ġside Ġto Ġthe Ġhuman Ġeval u ators . ĠThe Ġhuman Ġeval u ator Ġdoes Ġnot Ġknow Ġwhich Ġsumm aries Ġcome Ġfrom Ġwhich Ġmodel . ĠA Ġscore Ġfrom Ġ1 Ġto Ġ10 Ġis Ġthen Ġassigned Ġto Ġeach Ġsummary Ġon Ġthe Ġbasis Ġof Ġread ability , Ġand Ġinformation Ġcontained Ġof Ġsummary , Ġwhere Ġ1 Ġcorresponds Ġto Ġthe Ġlower Ġlevel Ġand Ġ10 Ġto Ġthe Ġhighest . ĠIn Ġtable ĠT AB REF 17 Ġwe Ġcompare Ġthe Ġscores Ġof Ġthese Ġfour Ġcases Ġas Ġgiven Ġby ĠR OG UE Ġalong Ġwith Ġhuman Ġevaluation . ĠThe Ġparser - gener ator Ġpairs Ġfor Ġthe Ġfour Ġcases Ġare Ġgold - J AM R ( gener ator ), ĠJ AM R ( parser )- ne ural , Ġgold - ne ural , Ġand Ġthe Ġoriginal Ġsentence Ġrespectively . ĠHere Ġgold Ġparser Ġmeans Ġthat Ġwe Ġhave Ġused Ġthe Ġgold - standard ĠAM R Ġgraphs . </s> The Ġscores Ġgiven Ġby Ġthe Ġhumans Ġdo Ġnot Ġcorrelate Ġwith ĠR OG UE . ĠHuman Ġeval u ators Ġgives Ġalmost Ġsimilarly Ġscores Ġto Ġsummary Ġgenerated Ġby Ġthe ĠLead - 1 Ġand ĠLead - 1 - AM R Ġwith ĠLead - 1 - AM R Ġactually Ġperforming Ġbetter Ġon Ġread ability Ġthough Ġit Ġdropped Ġsome Ġinformation Ġas Ġclear Ġfrom Ġthe Ġscores Ġon Ġinformation Ġcontained . ĠOn Ġthe Ġother Ġhand , ĠR OG UE Ġgives Ġvery Ġhigh Ġscore Ġto ĠLead - 1 Ġwhile Ġmodels Ġ1 , 2 Ġand Ġ4 Ġget Ġalmost Ġsame Ġscores . ĠThe Ġsimilar Ġscores Ġof Ġmodel Ġ2 Ġand Ġ3 Ġshows Ġthat Ġgenerators Ġare Ġactually Ġproducing Ġmeaningful Ġsentences . ĠThus Ġthe Ġdrop Ġin ĠR OG UE Ġscores Ġis Ġmainly Ġdue Ġto Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġto Ġevaluate Ġabstract ive Ġsumm aries . ĠMoreover , Ġthe ĠR OG UE Ġgives Ġmodel Ġ4 Ġhigher Ġscore Ġcompared Ġto Ġmodel Ġ1 Ġwhile Ġhuman Ġeval u ators Ġgive Ġthe Ġopposite Ġscores Ġon Ġinformation Ġcontained Ġin Ġthe Ġsentence . </s> A Ġpossible Ġreason Ġfor Ġthe Ġinability Ġof Ġthe ĠR OG UE Ġmetric Ġto Ġproperly Ġevaluate Ġthe Ġsumm aries Ġgenerated Ġby Ġour Ġmethod Ġmight Ġbe Ġdue Ġto Ġits Ġinability Ġto Ġevaluate Ġrestruct ured Ġsentences . ĠAM R Ġformal ism Ġtries Ġto Ġassign Ġthe Ġsame ĠAM R Ġgraphs Ġto Ġthe Ġsentences Ġthat Ġhave Ġsame Ġmeaning Ġso Ġthere Ġexists Ġa Ġone - to - many Ġmapping Ġfrom ĠAM R Ġgraphs Ġto Ġsentences . ĠThis Ġmeans Ġthat Ġthe Ġautomatic Ġgenerators Ġthat Ġwe Ġare Ġusing Ġmight Ġnot Ġbe Ġtrying Ġto Ġgenerate Ġthe Ġoriginal Ġsentence ; Ġinstead Ġit Ġis Ġtrying Ġto Ġgenerate Ġsome Ġother Ġsentence Ġthat Ġhas Ġthe Ġsame Ġunderlying Ġmeaning . ĠThis Ġalso Ġhelps Ġin Ġexplaining Ġthe Ġlow ĠR OG UE - 2 Ġand ĠR OG UE - L Ġscores . ĠIf Ġthe Ġsentences Ġmight Ġbe Ġgetting Ġre ph r ased , Ġthey Ġwould Ġloose Ġmost Ġof Ġthe Ġbi - Ġand Ġtri - gram s Ġfrom Ġthe Ġoriginal Ġsentence Ġresulting Ġin Ġlow ĠR OG UE - 2 Ġand ĠR OG UE - L Ġscores . </s> Analy zing Ġthe Ġeffectiveness Ġof ĠAM R Ġextraction </s> The Ġaim Ġof Ġextracting Ġsummary Ġgraphs Ġfrom Ġthe ĠAM R Ġgraphs Ġof Ġthe Ġsentence Ġis Ġto Ġdrop Ġthe Ġnot Ġso Ġimportant Ġinformation Ġfrom Ġthe Ġsentences . ĠIf Ġwe Ġare Ġable Ġto Ġachieve Ġthis Ġperfectly , Ġthe ĠR OG UE - 1 ĠRecall Ġscores Ġthat Ġwe Ġare Ġgetting Ġshould Ġremain Ġalmost Ġthe Ġsame Ġ( since Ġwe Ġare Ġnot Ġadd Ġany Ġnew Ġinformation ) Ġand Ġthe ĠR OG UE - 1 Ġprecision Ġshould Ġgo Ġup Ġ( as Ġwe Ġhave Ġthrown Ġout Ġsome Ġuseless Ġinformation ); Ġthus Ġeffectively Ġimproving Ġthe Ġoverall ĠR OG UE - 1 ĠIN LINE FORM 0 Ġscore . ĠIn Ġthe Ġfirst Ġtwo Ġrows Ġof Ġtable ĠT AB REF 14 Ġwe Ġhave Ġthe Ġscores Ġafter Ġusing Ġthe Ġfull - AM R Ġand Ġextracted ĠAM R Ġfor Ġgeneration Ġrespectively . ĠIt Ġis Ġsafe Ġto Ġsay Ġthat Ġextracting Ġthe ĠAM R Ġresults Ġin Ġimproved ĠR OG UE - 1 Ġprecision Ġwhereas ĠR OG UE - 1 ĠRecall Ġreduces Ġonly Ġslightly , Ġresulting Ġin Ġan Ġoverall Ġimproved ĠR OG UE - 1 ĠIN LINE FORM 1 Ġ. </s> Results Ġon Ġthe ĠCNN - Daily mail Ġcorpus </s> In Ġtable ĠT AB REF 18 Ġwe Ġreport Ġthe Ġresults Ġon Ġthe ĠCNN - Daily mail Ġcorpus . ĠWe Ġpresent Ġscores Ġby Ġusing Ġthe Ġfirst - 3 Ġmodel . ĠThe Ġfirst Ġrow Ġcontains Ġthe ĠLead - 3 - AM R Ġbaseline . ĠThe Ġresults Ġwe Ġachieve Ġare Ġcompetitive Ġwith Ġthe ĠLead - 3 - AM R Ġbaseline . ĠThe Ġrest Ġof Ġthe Ġtable Ġcontains Ġscores Ġof ĠLead - 3 Ġbaseline Ġfollowed Ġby Ġthe Ġstate - of - the - art Ġmethod Ġon Ġthe Ġanonym ized Ġand Ġnon - an onym ized Ġversions Ġof Ġthe Ġdataset . ĠThe Ġdrop Ġin Ġthe Ġscores Ġfrom Ġthe ĠLead - 3 ( non - an onym ized ) Ġto ĠLead - 3 - AM R Ġis Ġsignificant Ġand Ġis Ġlargely Ġbecause Ġof Ġthe Ġerror Ġintroduced Ġby Ġparser Ġand Ġgenerator . </s> Related ĠWork </s> BI B REF 18 Ġshowed Ġthat Ġmost Ġof Ġthe Ġwork Ġin Ġtext Ġsummar ization Ġhas Ġbeen Ġextract ive , Ġwhere Ġsentences Ġare Ġselected Ġfrom Ġthe Ġtext Ġwhich Ġare Ġthen Ġconc aten ated Ġto Ġform Ġa Ġsummary . ĠB IB REF 19 Ġtransformed Ġthe Ġinput Ġto Ġnodes , Ġthen Ġused Ġthe ĠP ager ank Ġalgorithm Ġto Ġscore Ġnodes , Ġand Ġfinally Ġgrow Ġthe Ġnodes Ġfrom Ġhigh - value Ġto Ġlow - value Ġusing Ġsome Ġhe ur istics . ĠSome Ġof Ġthe Ġapproaches Ġcombine Ġthis Ġwith Ġsentence Ġcompression , Ġso Ġmore Ġsentences Ġcan Ġbe Ġpacked Ġin Ġthe Ġsummary . ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ, Ġand ĠB IB REF 23 Ġamong Ġothers Ġused ĠIL Ps Ġand Ġapprox im ations Ġfor Ġencoding Ġcompression Ġand Ġextraction . </s> Recently Ġsome Ġabstract ive Ġapproaches Ġhave Ġalso Ġbeen Ġproposed Ġmost Ġof Ġwhich Ġused Ġsequence Ġto Ġsequence Ġlearning Ġmodels Ġfor Ġthe Ġtask . ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 17 Ġused Ġstandard Ġenc oder - dec oder Ġmodels Ġalong Ġwith Ġtheir Ġvariants Ġto Ġgenerate Ġsumm aries . ĠB IB REF 26 Ġincorporated Ġthe ĠAM R Ġinformation Ġin Ġthe Ġstandard Ġenc oder - dec oder Ġmodels Ġto Ġimprove Ġresults . ĠOur Ġwork Ġin Ġsimilar Ġto Ġother Ġgraph Ġbased Ġabstract ive Ġsummar ization Ġmethods ĠB IB REF 27 Ġand ĠB IB REF 28 Ġ. ĠB IB REF 27 Ġused Ġdependency Ġparse Ġtrees Ġto Ġproduce Ġsumm aries . ĠOn Ġthe Ġother Ġhand Ġour Ġwork Ġtakes Ġadvantage Ġof Ġsemantic Ġgraphs . </s> Need Ġof Ġan Ġnew ĠDat as et Ġand ĠEvaluation ĠMet ric </s> R OG UE Ġmetric , Ġby Ġit Ġis Ġdesign Ġhas Ġlots Ġof Ġproperties Ġthat Ġmake Ġit Ġunsu itable Ġfor Ġevaluating Ġabstract ive Ġsumm aries . ĠFor Ġexample , ĠR OG UE Ġmatches Ġexact Ġwords Ġand Ġnot Ġthe Ġstems Ġof Ġthe Ġwords , Ġit Ġalso Ġconsiders Ġstop Ġwords Ġfor Ġevaluation . ĠOne Ġof Ġthe Ġreasons Ġwhy ĠR OG UE Ġlike Ġmetrics Ġmight Ġnever Ġbecome Ġsuitable Ġfor Ġevaluating Ġabstract ive Ġsumm aries Ġis Ġits Ġincap abilities Ġof Ġknowing Ġif Ġthe Ġsentences Ġhave Ġbeen Ġrestruct ured . ĠA Ġgood Ġevaluation Ġmetric Ġshould Ġbe Ġone Ġwhere Ġwe Ġcompare Ġthe Ġmeaning Ġof Ġthe Ġsentence Ġand Ġnot Ġthe Ġexact Ġwords . ĠAs Ġwe Ġshowed Ġsection ĠSEC REF 20 ĠR OG UE Ġis Ġnot Ġsuitable Ġfor Ġevaluating Ġsumm aries Ġgenerated Ġby Ġthe ĠAM R Ġpipeline . </s> We Ġnow Ġshow Ġwhy Ġthe ĠCNN - Daily mail Ġcorpus Ġis Ġnot Ġsuitable Ġfor ĠAbstract ive Ġsummar ization . ĠThe Ġnature Ġof Ġsummary Ġpoints Ġin Ġthe Ġcorpus Ġis Ġhighly Ġextract ive Ġ( Section ĠSEC REF 7 Ġfor Ġdetails ) Ġwhere Ġmost Ġof Ġthe Ġsummary Ġpoints Ġare Ġsimply Ġpicked Ġup Ġfrom Ġsome Ġsentences Ġin Ġthe Ġstory . ĠTough , Ġthis Ġis Ġa Ġgood Ġenough Ġreason Ġto Ġstart Ġsearching Ġfor Ġbetter Ġdataset , Ġit Ġis Ġnot Ġthe Ġbiggest Ġproblem Ġwith Ġthe Ġdataset . ĠThe Ġdataset Ġhas Ġthe Ġproperty Ġthat Ġa Ġlot Ġof Ġimportant Ġinformation Ġis Ġin Ġthe Ġfirst Ġfew Ġsentences Ġand Ġmost Ġof Ġthe Ġsummary Ġpoints Ġare Ġdirectly Ġpick Ġfrom Ġthese Ġsentences . ĠThe Ġextract ive Ġmethods Ġbased Ġon Ġsentence Ġselection Ġlike ĠSum ma Run N er Ġare Ġnot Ġactually Ġperforming Ġwell , Ġthe Ġresults Ġthey Ġhave Ġgot Ġare Ġonly Ġslightly Ġbetter Ġthan Ġthe ĠLead - 3 Ġbaseline . ĠThe Ġwork Ġdoesn 't Ġshow Ġhow Ġmuch Ġof Ġthe Ġselected Ġsentences Ġare Ġamong Ġthe Ġfirst Ġfew Ġand Ġit Ġmight Ġbe Ġthe Ġcase Ġthat Ġthe Ġsentences Ġselected Ġby Ġthe Ġextract ive Ġmethods Ġare Ġmostly Ġamong Ġthe Ġfirst Ġfew Ġsentences , Ġthe Ġsame Ġcan Ġbe Ġthe Ġproblem Ġwith Ġthe Ġabstract ive Ġmethods , Ġwhere Ġmost Ġthe Ġoutput Ġmight Ġbe Ġgetting Ġcopied Ġfrom Ġthe Ġinitial Ġfew Ġsentences . </s> These Ġproblems Ġwith Ġthis Ġcorpus Ġevoke Ġthe Ġneed Ġto Ġhave Ġanother Ġcorpus Ġwhere Ġwe Ġdon 't Ġhave Ġso Ġmuch Ġconcentration Ġof Ġimportant Ġinformation Ġat Ġany Ġlocation Ġbut Ġrather Ġthe Ġinformation Ġis Ġmore Ġspread Ġout Ġand Ġthe Ġsumm aries Ġare Ġmore Ġabstract ive Ġin Ġnature . </s> P ossible ĠFuture ĠDirections </s> As Ġthis Ġproposed Ġalgorithm Ġis Ġa Ġstep Ġby Ġstep Ġprocess Ġwe Ġcan Ġfocus Ġon Ġimproving Ġeach Ġstep Ġto Ġproduce Ġbetter Ġresults . ĠThe Ġmost Ġexciting Ġimprovements Ġcan Ġbe Ġdone Ġin Ġthe Ġsummary Ġgraph Ġextraction Ġmethod . ĠNot Ġa Ġlot Ġof Ġwork Ġhas Ġbeen Ġdone Ġto Ġextract ĠAM R Ġgraphs Ġfor Ġsumm aries . ĠIn Ġorder Ġto Ġmake Ġthis Ġpipeline Ġgeneral izable Ġfor Ġany Ġsort Ġof Ġtext , Ġwe Ġneed Ġto Ġget Ġrid Ġof Ġthe Ġhypothesis Ġthat Ġthe Ġsummary Ġis Ġbeing Ġextracted Ġexactly Ġfrom Ġone Ġsentence . ĠSo , Ġthe Ġnatural Ġdirection Ġseems Ġto Ġbe Ġjoining ĠAM R Ġgraphs Ġof Ġmultiple Ġsentences Ġthat Ġare Ġsimilar Ġand Ġthen Ġextracting Ġthe Ġsummary ĠAM R Ġfrom Ġthat Ġlarge Ġgraph . ĠIt Ġwill Ġbe Ġlike Ġclust ering Ġsimilar Ġsentences Ġand Ġthen Ġextracting Ġa Ġsummary Ġgraph Ġfrom Ġeach Ġof Ġthese Ġcluster . ĠAnother Ġidea Ġis Ġto Ġuse ĠAM R Ġgraphs Ġfor Ġimportant Ġsentence Ġselection . </s> Conclusion </s> In Ġthis Ġwork Ġwe Ġhave Ġexplored Ġa Ġfull - fledged Ġpipeline Ġusing ĠAM R Ġfor Ġsummar ization Ġfor Ġthe Ġfirst Ġtime . ĠWe Ġpropose Ġa Ġnew Ġmethod Ġfor Ġextracting Ġsummary Ġgraph , Ġwhich Ġoutper formed Ġprevious Ġmethods . ĠOverall Ġwe Ġprovide Ġstrong Ġbaseline Ġfor Ġtext Ġsummar ization Ġusing ĠAM R Ġfor Ġpossible Ġfuture Ġworks . ĠWe Ġalso Ġshowed Ġthat ĠR OG UE Ġcan 't Ġbe Ġused Ġfor Ġevaluating Ġthe Ġabstract ive Ġsumm aries Ġgenerated Ġby Ġour ĠAM R Ġpipeline .