correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization
EXTRACTIVE
Scientific summarization has attracted more attention recently (examples include works by abu2011coherent, qazvinian2013generating, and cohan2015scientific). Thus, it is important to study the validity of existing methodologies applied to the evaluation of news article summarization for this task. In particular, we raise the important question of how effective is Rouge, as an evaluation metric for scientific summarization? We answer this question by comparing Rouge scores with semi-manual evaluation score (Pyramid) in TAC 2014 scientific summarization dataset[1]. Results reveal that, contrary to the common belief, correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization. Furthermore, we show a large variance of correlations between different Rouge variants and the manual evaluations which further makes the reliability of Rouge for evaluating scientific summaries less clear. We then propose an evaluation metric based on relevance analysis of summaries which aims to overcome the limitation of high lexical dependence in Rouge. We call our metric Sera (Summarization Evaluation by Relevance Analysis). Results show that the proposed metric achieves higher and more consistent correlations with semi-manual assessment scores.
1604.00400
Introduction </s> Aut omatic Ġtext Ġsummar ization Ġhas Ġbeen Ġan Ġactive Ġresearch Ġarea Ġin Ġnatural Ġlanguage Ġprocessing Ġfor Ġseveral Ġdecades . ĠTo Ġcompare Ġand Ġevaluate Ġthe Ġperformance Ġof Ġdifferent Ġsummar ization Ġsystems , Ġthe Ġmost Ġintuitive Ġapproach Ġis Ġassessing Ġthe Ġquality Ġof Ġthe Ġsumm aries Ġby Ġhuman Ġeval u ators . ĠHowever , Ġmanual Ġevaluation Ġis Ġexpensive Ġand Ġthe Ġobtained Ġresults Ġare Ġsubjective Ġand Ġdifficult Ġto Ġreproduce ĠB IB REF 0 Ġ. ĠTo Ġaddress Ġthese Ġproblems , Ġautomatic Ġevaluation Ġmeasures Ġfor Ġsummar ization Ġhave Ġbeen Ġproposed . ĠRouge ĠB IB REF 1 Ġis Ġone Ġof Ġthe Ġfirst Ġand Ġmost Ġwidely Ġused Ġmetrics Ġin Ġsummar ization Ġevaluation . ĠIt Ġfacilitates Ġevaluation Ġof Ġsystem Ġgenerated Ġsumm aries Ġby Ġcomparing Ġthem Ġto Ġa Ġset Ġof Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠIt Ġis Ġinspired Ġby Ġthe Ġsuccess Ġof Ġa Ġsimilar Ġmetric ĠBle u ĠB IB REF 2 Ġwhich Ġis Ġbeing Ġused Ġin ĠMachine ĠTranslation Ġ( MT ) Ġevaluation . ĠThe Ġmain Ġsuccess Ġof ĠRouge Ġis Ġdue Ġto Ġits Ġhigh Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores Ġon Ġstandard Ġbenchmarks ĠB IB REF 1 Ġ. ĠRouge Ġhas Ġbeen Ġused Ġas Ġone Ġof Ġthe Ġmain Ġevaluation Ġmetrics Ġin Ġlater Ġsummar ization Ġbenchmarks Ġsuch Ġas ĠT AC [ 1 ] ĠB IB REF 3 Ġ. </s> [ 1 ] Text ĠAnalysis ĠConference Ġ( T AC ) Ġis Ġa Ġseries Ġof Ġworkshops Ġfor Ġevaluating Ġresearch Ġin ĠNatural ĠLanguage ĠProcessing </s> Since Ġthe Ġestablishment Ġof ĠRouge , Ġalmost Ġall Ġresearch Ġin Ġtext Ġsummar ization Ġhave Ġused Ġthis Ġmetric Ġas Ġthe Ġmain Ġmeans Ġfor Ġevaluating Ġthe Ġquality Ġof Ġthe Ġproposed Ġapproaches . ĠThe Ġpublic Ġavailability Ġof ĠRouge Ġas Ġa Ġtool kit Ġfor Ġsummar ization Ġevaluation Ġhas Ġcontributed Ġto Ġits Ġwide Ġusage . ĠWhile ĠRouge Ġhas Ġoriginally Ġshown Ġgood Ġcorrelations Ġwith Ġhuman Ġassessments , Ġthe Ġstudy Ġof Ġits Ġeffectiveness Ġwas Ġonly Ġlimited Ġto Ġa Ġfew Ġbenchmarks Ġon Ġnews Ġsummar ization Ġdata Ġ( D UC [ 2 ] Ġ2001 - 2003 Ġbenchmarks ). ĠSince Ġ2003 , Ġsummar ization Ġhas Ġgrown Ġto Ġmuch Ġfurther Ġdomains Ġand Ġgenres Ġsuch Ġas Ġscientific Ġdocuments , Ġsocial Ġmedia Ġand Ġquestion Ġanswering . ĠWhile Ġthere Ġis Ġnot Ġenough Ġcompelling Ġevidence Ġabout Ġthe Ġeffectiveness Ġof ĠRouge Ġon Ġthese Ġother Ġsummar ization Ġtasks , Ġpublished Ġresearch Ġis Ġalmost Ġalways Ġevaluated Ġby ĠRouge . ĠIn Ġaddition , ĠRouge Ġhas Ġa Ġlarge Ġnumber Ġof Ġpossible Ġvariants Ġand Ġthe Ġpublished Ġresearch Ġoften Ġ( ar bit rarily ) Ġreports Ġonly Ġa Ġfew Ġof Ġthese Ġvariants . </s> [ 2 ] Document ĠUnderstanding ĠConference Ġ( D UC ) Ġwas Ġone Ġof ĠN IST Ġworkshops Ġthat Ġprovided Ġinfrastructure Ġfor Ġevaluation Ġof Ġtext Ġsummar ization Ġmethod ologies Ġ( http :// duc . n ist . gov / ). </s> By Ġdefinition , ĠRouge Ġsolely Ġrelies Ġon Ġlex ical Ġoverl aps Ġ( such Ġas Ġn - gram Ġand Ġsequence Ġoverl aps ) Ġbetween Ġthe Ġsystem Ġgenerated Ġand Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠHigher Ġlex ical Ġoverl aps Ġbetween Ġthe Ġtwo Ġshow Ġthat Ġthe Ġsystem Ġgenerated Ġsummary Ġis Ġof Ġhigher Ġquality . ĠTherefore , Ġin Ġcases Ġof Ġterminology Ġnuances Ġand Ġparaph r asing , ĠRouge Ġis Ġnot Ġaccurate Ġin Ġestimating Ġthe Ġquality Ġof Ġthe Ġsummary . </s> We Ġstudy Ġthe Ġeffectiveness Ġof ĠRouge Ġfor Ġevaluating Ġscientific Ġsummar ization . ĠScientific Ġsummar ization Ġtargets Ġmuch Ġmore Ġtechnical Ġand Ġfocused Ġdomains Ġin Ġwhich Ġthe Ġgoal Ġis Ġproviding Ġsumm aries Ġfor Ġscientific Ġarticles . ĠScientific Ġarticles Ġare Ġmuch Ġdifferent Ġthan Ġnews Ġarticles Ġin Ġelements Ġsuch Ġas Ġlength , Ġcomplexity Ġand Ġstructure . ĠThus , Ġeffective Ġsummar ization Ġapproaches Ġusually Ġhave Ġmuch Ġhigher Ġcompression Ġrate , Ġterminology Ġvariations Ġand Ġparaph r asing ĠB IB REF 4 Ġ. </s> Scient ific Ġsummar ization Ġhas Ġattracted Ġmore Ġattention Ġrecently Ġ( ex amples Ġinclude Ġworks Ġby Ġab u 2011 co herent , Ġq az vin ian 2013 gener ating , Ġand Ġco han 2015 scientific ). ĠThus , Ġit Ġis Ġimportant Ġto Ġstudy Ġthe Ġvalidity Ġof Ġexisting Ġmethod ologies Ġapplied Ġto Ġthe Ġevaluation Ġof Ġnews Ġarticle Ġsummar ization Ġfor Ġthis Ġtask . ĠIn Ġparticular , Ġwe Ġraise Ġthe Ġimportant Ġquestion Ġof Ġhow Ġeffective Ġis ĠRouge , Ġas Ġan Ġevaluation Ġmetric Ġfor Ġscientific Ġsummar ization ? ĠWe Ġanswer Ġthis Ġquestion Ġby Ġcomparing ĠRouge Ġscores Ġwith Ġsemi - man ual Ġevaluation Ġscore Ġ( Py ramid ) Ġin ĠT AC Ġ2014 Ġscientific Ġsummar ization Ġdataset [ 1 ]. ĠResults Ġreveal Ġthat , Ġcontrary Ġto Ġthe Ġcommon Ġbelief , Ġcorrelations Ġbetween ĠRouge Ġand Ġthe ĠPyramid Ġscores Ġare Ġweak , Ġwhich Ġchallenges Ġits Ġeffectiveness Ġfor Ġscientific Ġsummar ization . ĠFurthermore , Ġwe Ġshow Ġa Ġlarge Ġvariance Ġof Ġcorrelations Ġbetween Ġdifferent ĠRouge Ġvariants Ġand Ġthe Ġmanual Ġevaluations Ġwhich Ġfurther Ġmakes Ġthe Ġreliability Ġof ĠRouge Ġfor Ġevaluating Ġscientific Ġsumm aries Ġless Ġclear . ĠWe Ġthen Ġpropose Ġan Ġevaluation Ġmetric Ġbased Ġon Ġrelevance Ġanalysis Ġof Ġsumm aries Ġwhich Ġaims Ġto Ġovercome Ġthe Ġlimitation Ġof Ġhigh Ġlex ical Ġdependence Ġin ĠRouge . ĠWe Ġcall Ġour Ġmetric ĠSer a Ġ( Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis ). ĠResults Ġshow Ġthat Ġthe Ġproposed Ġmetric Ġachieves Ġhigher Ġand Ġmore Ġconsistent Ġcorrelations Ġwith Ġsemi - man ual Ġassessment Ġscores . </s> [ 1 ] http :// www . n ist . gov / t ac / 2014 / Bi omed Sum m / </s> Our Ġcontributions Ġare Ġas Ġfollows : </s> [ 2 ] The Ġannotations Ġcan Ġbe Ġaccessed Ġvia Ġthe Ġfollowing Ġrepository : Ġhttps :// github . com / ac ohan / T AC - py ramid - An notations / </s> Sum mar ization Ġevaluation Ġby ĠRouge </s> R ou ge Ġhas Ġbeen Ġthe Ġmost Ġwidely Ġused Ġfamily Ġof Ġmetrics Ġin Ġsummar ization Ġevaluation . ĠIn Ġthe Ġfollowing , Ġwe Ġbriefly Ġdescribe Ġthe Ġdifferent Ġvariants Ġof ĠRouge : </s> R ou ge - L , ĠRouge - W , ĠRouge - S Ġand ĠRouge - SU Ġwere Ġlater Ġextended Ġto Ġconsider Ġboth Ġthe Ġrecall Ġand Ġprecision . ĠIn Ġcalculating ĠRouge , Ġstop word Ġremoval Ġor Ġstemming Ġcan Ġalso Ġbe Ġconsidered , Ġresulting Ġin Ġmore Ġvariants . </s> In Ġthe Ġsummar ization Ġliterature , Ġdespite Ġthe Ġlarge Ġnumber Ġof Ġvariants Ġof ĠRouge , Ġonly Ġone Ġor Ġvery Ġfew Ġof Ġthese Ġvariants Ġare Ġoften Ġchosen Ġ( ar bit rarily ) Ġfor Ġevaluation Ġof Ġthe Ġquality Ġof Ġthe Ġsummar ization Ġapproaches . ĠWhen ĠRouge Ġwas Ġproposed , Ġthe Ġoriginal Ġvariants Ġwere Ġonly Ġrecall - oriented Ġand Ġhence Ġthe Ġreported Ġcorrelation Ġresults ĠB IB REF 1 Ġ. ĠThe Ġlater Ġextension Ġof ĠRouge Ġfamily Ġby Ġprecision Ġwere Ġonly Ġreflected Ġin Ġthe Ġlater Ġversions Ġof Ġthe ĠRouge Ġtool kit Ġand Ġadditional Ġevaluation Ġof Ġits Ġeffectiveness Ġwas Ġnot Ġreported . ĠNevertheless , Ġlater Ġpublished Ġwork Ġin Ġsummar ization Ġadopted Ġthis Ġtool kit Ġfor Ġits Ġready Ġimplementation Ġand Ġrelatively Ġefficient Ġperformance . </s> The Ġoriginal ĠRouge Ġmetrics Ġshow Ġhigh Ġcorrelations Ġwith Ġhuman Ġjudgments Ġof Ġthe Ġquality Ġof Ġsumm aries Ġon Ġthe ĠD UC Ġ2001 - 2003 Ġbenchmarks . ĠHowever , Ġthese Ġbenchmarks Ġconsist Ġof Ġnews wire Ġdata Ġand Ġare Ġintrinsically Ġvery Ġdifferent Ġthan Ġother Ġsummar ization Ġtasks Ġsuch Ġas Ġsummar ization Ġof Ġscientific Ġpapers . ĠWe Ġargue Ġthat ĠRouge Ġis Ġnot Ġthe Ġbest Ġmetric Ġfor Ġall Ġsummar ization Ġtasks Ġand Ġwe Ġpropose Ġan Ġalternative Ġmetric Ġfor Ġevaluation Ġof Ġscientific Ġsummar ization . ĠThe Ġproposed Ġalternative Ġmetric Ġshows Ġmuch Ġhigher Ġand Ġmore Ġconsistent Ġcorrelations Ġwith Ġmanual Ġjudgments Ġin Ġcomparison Ġwith Ġthe Ġwell - established ĠRouge . </s> Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis Ġ( Ser a ) </s> R ou ge Ġfunctions Ġbased Ġon Ġthe Ġassumption Ġthat Ġin Ġorder Ġfor Ġa Ġsummary Ġto Ġbe Ġof Ġhigh Ġquality , Ġit Ġhas Ġto Ġshare Ġmany Ġwords Ġor Ġphrases Ġwith Ġa Ġhuman Ġgold Ġsummary . ĠHowever , Ġdifferent Ġterminology Ġmay Ġbe Ġused Ġto Ġrefer Ġto Ġthe Ġsame Ġconcepts Ġand Ġthus Ġrelying Ġonly Ġon Ġlex ical Ġoverl aps Ġmay Ġunder rate Ġcontent Ġquality Ġscores . ĠTo Ġovercome Ġthis Ġproblem , Ġwe Ġpropose Ġan Ġapproach Ġbased Ġon Ġthe Ġpremise Ġthat Ġconcepts Ġtake Ġmeanings Ġfrom Ġthe Ġcontext Ġthey Ġare Ġin , Ġand Ġthat Ġrelated Ġconcepts Ġco - occ ur Ġfrequently . </s> Our Ġproposed Ġmetric Ġis Ġbased Ġon Ġanalysis Ġof Ġthe Ġcontent Ġrelevance Ġbetween Ġa Ġsystem Ġgenerated Ġsummary Ġand Ġthe Ġcorresponding Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠOn Ġhigh Ġlevel , Ġwe Ġindirectly Ġevaluate Ġthe Ġcontent Ġrelevance Ġbetween Ġthe Ġcandidate Ġsummary Ġand Ġthe Ġhuman Ġsummary Ġusing Ġinformation Ġretrieval . ĠTo Ġaccomplish Ġthis , Ġwe Ġuse Ġthe Ġsumm aries Ġas Ġsearch Ġqueries Ġand Ġcompare Ġthe Ġoverl aps Ġof Ġthe Ġretrieved Ġresults . ĠL arger Ġnumber Ġof Ġoverl aps , Ġsuggest Ġthat Ġthe Ġcandidate Ġsummary Ġhas Ġhigher Ġcontent Ġquality Ġwith Ġrespect Ġto Ġthe Ġgold - standard . ĠThis Ġmethod , Ġenables Ġus Ġto Ġalso Ġreward Ġfor Ġterms Ġthat Ġare Ġnot Ġlex ically Ġequivalent Ġbut Ġsem antically Ġrelated . ĠOur Ġmethod Ġis Ġbased Ġon Ġthe Ġwell Ġestablished Ġlinguistic Ġpremise Ġthat Ġsem antically Ġrelated Ġwords Ġoccur Ġin Ġsimilar Ġcontexts ĠB IB REF 5 Ġ. ĠThe Ġcontext Ġof Ġthe Ġwords Ġcan Ġbe Ġconsidered Ġas Ġsurrounding Ġwords , Ġsentences Ġin Ġwhich Ġthey Ġappear Ġor Ġthe Ġdocuments . ĠFor Ġscientific Ġsummar ization , Ġwe Ġconsider Ġthe Ġcontext Ġof Ġthe Ġwords Ġas Ġthe Ġscientific Ġarticles Ġin Ġwhich Ġthey Ġappear . ĠThus , Ġif Ġtwo Ġconcepts Ġappear Ġin Ġidentical Ġset Ġof Ġarticles , Ġthey Ġare Ġsem antically Ġrelated . ĠWe Ġconsider Ġthe Ġtwo Ġsumm aries Ġas Ġsimilar Ġif Ġthey Ġrefer Ġto Ġsame Ġset Ġof Ġarticles Ġeven Ġif Ġthe Ġtwo Ġsumm aries Ġdo Ġnot Ġhave Ġhigh Ġlex ical Ġoverl aps . ĠTo Ġcapture Ġif Ġa Ġsummary Ġrelates Ġto Ġa Ġarticle , Ġwe Ġuse Ġinformation Ġretrieval Ġby Ġconsidering Ġthe Ġsumm aries Ġas Ġqueries Ġand Ġthe Ġarticles Ġas Ġdocuments Ġand Ġwe Ġrank Ġthe Ġarticles Ġbased Ġon Ġtheir Ġrelated ness Ġto Ġa Ġgiven Ġsummary . ĠFor Ġa Ġgiven Ġpair Ġof Ġsystem Ġsummary Ġand Ġthe Ġgold Ġsummary , Ġsimilar Ġrankings Ġof Ġthe Ġretrieved Ġarticles Ġsuggest Ġthat Ġthe Ġsumm aries Ġare Ġsem antically Ġrelated , Ġand Ġthus Ġthe Ġsystem Ġsummary Ġis Ġof Ġhigher Ġquality . </s> Based Ġon Ġthe Ġdomain Ġof Ġinterest , Ġwe Ġfirst Ġconstruct Ġan Ġindex Ġfrom Ġa Ġset Ġof Ġarticles Ġin Ġthe Ġsame Ġdomain . ĠSince ĠT AC Ġ2014 Ġwas Ġfocused Ġon Ġsummar ization Ġin Ġthe Ġbiomedical Ġdomain , Ġour Ġindex Ġalso Ġcomprises Ġof Ġbiomedical Ġarticles . ĠGiven Ġa Ġcandidate Ġsummary ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġgold Ġsumm aries ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġ; ĠIN LINE FORM 3 Ġis Ġthe Ġtotal Ġnumber Ġof Ġhuman Ġsumm aries ), Ġwe Ġsubmit Ġthe Ġcandidate Ġsummary Ġand Ġgold Ġsumm aries Ġto Ġthe Ġsearch Ġengine Ġas Ġqueries Ġand Ġcompare Ġtheir Ġranked Ġresults . ĠLet ĠIN LINE FORM 4 Ġbe Ġthe Ġentire Ġindex Ġwhich Ġcomprises Ġof ĠIN LINE FORM 5 Ġtotal Ġdocuments . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġranked Ġlist Ġof Ġretrieved Ġdocuments Ġfor Ġcandidate Ġsummary ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġthe Ġranked Ġlist Ġof Ġresults Ġfor Ġthe Ġgold Ġsummary ĠIN LINE FORM 3 Ġ. ĠThese Ġlists Ġof Ġresults Ġare Ġbased Ġon Ġa Ġrank Ġcut - off Ġpoint ĠIN LINE FORM 4 Ġthat Ġis Ġa Ġparameter Ġof Ġthe Ġsystem . ĠWe Ġprovide Ġevaluation Ġresults Ġon Ġdifferent Ġchoices Ġof Ġcut - off Ġpoint ĠIN LINE FORM 5 Ġin Ġthe ĠSection ĠSEC REF 5 ĠWe Ġconsider Ġthe Ġfollowing Ġtwo Ġscores : Ġ( i ) Ġsimple Ġintersection Ġand Ġ( ii ) Ġdiscounted Ġintersection Ġby Ġrankings . ĠThe Ġsimple Ġintersection Ġjust Ġconsiders Ġthe Ġoverl aps Ġof Ġthe Ġresults Ġin Ġthe Ġtwo Ġranked Ġlists Ġand Ġignores Ġthe Ġrankings . ĠThe Ġdiscounted Ġranked Ġscores , Ġon Ġthe Ġother Ġhand , Ġpenal izes Ġranking Ġdifferences Ġbetween Ġthe Ġtwo Ġresult Ġsets . ĠAs Ġan Ġexample Ġconsider Ġthe Ġfollowing Ġlist Ġof Ġretrieved Ġdocuments Ġ( den oted Ġby ĠIN LINE FORM 6 Ġs ) Ġfor Ġa Ġcandidate Ġand Ġa Ġgold Ġsummary Ġas Ġqueries : </s> Results Ġfor Ġcandidate Ġsummary : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 </s> Results Ġfor Ġgold Ġsummary : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 </s> These Ġtwo Ġsets Ġof Ġresults Ġconsist Ġof Ġidentical Ġdocuments Ġbut Ġthe Ġranking Ġof Ġthe Ġretrieved Ġdocuments Ġdiffer . ĠTherefore , Ġthe Ġsimple Ġintersection Ġmethod Ġassigns Ġa Ġscore Ġof Ġ1 . 0 Ġwhile Ġin Ġthe Ġdiscounted Ġranked Ġscore , Ġthe Ġscore Ġwill Ġbe Ġless Ġthan Ġ1 . 0 Ġ( due Ġto Ġranking Ġdifferences Ġbetween Ġthe Ġresult Ġlists ). </s> We Ġnow Ġdefine Ġthe Ġmetrics Ġmore Ġprecisely . ĠUsing Ġthe Ġabove Ġnot ations , Ġwithout Ġloss Ġof Ġgener ality , Ġwe Ġassume Ġthat ĠIN LINE FORM 0 Ġ. ĠSer a Ġis Ġdefined Ġas Ġfollows : ĠIN LINE FORM 1 </s> To Ġalso Ġaccount Ġfor Ġthe Ġranked Ġposition Ġdifferences , Ġwe Ġmodify Ġthis Ġscore Ġto Ġdiscount Ġrewards Ġbased Ġon Ġrank Ġdifferences . ĠThat Ġis , Ġin Ġideal Ġscore , Ġwe Ġwant Ġsearch Ġresults Ġfrom Ġcandidate Ġsummary Ġ( ĠIN LINE FORM 0 Ġ) Ġto Ġbe Ġthe Ġsame Ġas Ġresults Ġfor Ġgold - standard Ġsumm aries Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġthe Ġrankings Ġof Ġthe Ġresults Ġalso Ġbe Ġthe Ġsame . ĠIf Ġthe Ġrankings Ġdiffer , Ġwe Ġdiscount Ġthe Ġreward Ġby Ġlog Ġof Ġthe Ġdifferences Ġof Ġthe Ġranks . ĠMore Ġspecifically , Ġthe Ġdiscounted Ġscore Ġ( Ser a - Dis ) Ġis Ġdefined Ġas : ĠIN LINE FORM 2 </s> where , Ġas Ġpreviously Ġdefined , ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtotal Ġnumber Ġof Ġhuman Ġgold Ġsumm aries , Ġresult Ġlist Ġfor Ġthe Ġcandidate Ġsummary Ġand Ġresult Ġlist Ġfor Ġthe Ġhuman Ġgold Ġsummary , Ġrespectively . ĠIn Ġaddition , ĠIN LINE FORM 3 Ġshows Ġthe ĠIN LINE FORM 4 Ġth Ġresults Ġin Ġthe Ġranked Ġlist ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġmaximum Ġattain able Ġscore Ġused Ġas Ġthe Ġnormal izing Ġfactor . </s> We Ġuse Ġelastic search [ 1 ], Ġan Ġopen - source Ġsearch Ġengine , Ġfor Ġindex ing Ġand Ġquer ying Ġthe Ġarticles . ĠFor Ġretrieval Ġmodel , Ġwe Ġuse Ġthe ĠLanguage ĠModel ing Ġretrieval Ġmodel Ġwith ĠDir ich let Ġsmoot hing ĠB IB REF 6 Ġ. ĠSince ĠT AC Ġ2014 Ġbenchmark Ġis Ġon Ġsummar ization Ġof Ġbiomedical Ġarticles , Ġthe Ġappropriate Ġindex Ġwould Ġbe Ġthe Ġone Ġconstructed Ġfrom Ġarticles Ġin Ġthe Ġsame Ġdomain . ĠTherefore , Ġwe Ġuse Ġthe Ġopen Ġaccess Ġsubset Ġof ĠPub med [ 2 ] Ġwhich Ġconsists Ġof Ġpublished Ġarticles Ġin Ġbiomedical Ġliterature . </s> [ 1 ] https :// github . com / el astic / el astic search Ġ[ 2 ] Pub Med Ġis Ġa Ġcomprehensive Ġresource Ġof Ġarticles Ġand Ġabstract s Ġpublished Ġin Ġlife Ġsciences Ġand Ġbiomedical Ġliterature Ġhttp :// www . nc bi . nl m . nih . gov / pm c / </s> We Ġalso Ġexperiment Ġwith Ġdifferent Ġquery Ġ( re ) form ulation Ġapproaches . ĠQuery Ġreform ulation Ġis Ġa Ġmethod Ġin ĠInformation ĠRet ri eval Ġthat Ġaims Ġto Ġrefine Ġthe Ġquery Ġfor Ġbetter Ġretrieval Ġof Ġresults . ĠQuery Ġreform ulation Ġmethods Ġoften Ġconsist Ġof Ġremoving Ġineffective Ġterms Ġand Ġexpressions Ġfrom Ġthe Ġquery Ġ( query Ġreduction ) Ġor Ġadding Ġterms Ġto Ġthe Ġquery Ġthat Ġhelp Ġthe Ġretrieval Ġ( query Ġexpansion ). ĠQuery Ġreduction Ġis Ġspecially Ġimportant Ġwhen Ġqueries Ġare Ġverb ose . ĠSince Ġwe Ġuse Ġthe Ġsumm aries Ġas Ġqueries , Ġthe Ġqueries Ġare Ġusually Ġlong Ġand Ġtherefore Ġwe Ġconsider Ġquery Ġreductions . </s> In Ġour Ġexperiments , Ġthe Ġquery Ġreform ulation Ġis Ġdone Ġby Ġ3 Ġdifferent Ġways : Ġ( i ) ĠPlain : ĠThe Ġentire Ġsummary Ġwithout Ġstop words Ġand Ġnumeric Ġvalues ; Ġ( ii ) ĠN oun ĠPh r ases Ġ( NP ): ĠWe Ġonly Ġkeep Ġthe Ġnoun Ġphrases Ġas Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġand Ġeliminate Ġall Ġother Ġterms ; Ġand Ġ( iii ) ĠKey words Ġ( K W ): ĠWe Ġonly Ġkeep Ġthe Ġkeywords Ġand Ġkey Ġphrases Ġin Ġthe Ġsummary . ĠFor Ġextracting Ġthe Ġkeywords Ġand Ġkey ph r ases Ġ( with Ġlength Ġof Ġup Ġto Ġ3 Ġterms ), Ġwe Ġextract Ġexpressions Ġwhose Ġid f [ 1 ] Ġvalues Ġis Ġhigher Ġthan Ġa Ġpred efined Ġthreshold Ġthat Ġis Ġset Ġas Ġa Ġparameter . ĠWe Ġset Ġthis Ġthreshold Ġto Ġthe Ġaverage Ġid f Ġvalues Ġof Ġall Ġterms Ġexcept Ġstop words . Ġid f Ġvalues Ġare Ġcalculated Ġon Ġthe Ġsame Ġindex Ġthat Ġis Ġused Ġfor Ġthe Ġretrieval . </s> [ 1 ] In verted ĠDocument ĠFrequency </s> We Ġhypothes ize Ġthat Ġusing Ġonly Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġprevents Ġquery Ġdrift Ġand Ġleads Ġto Ġretrieval Ġof Ġmore Ġrelevant Ġdocuments . ĠN oun Ġphrases Ġand Ġkeywords Ġare Ġtwo Ġhe ur istics Ġfor Ġidentifying Ġthe Ġinformative Ġconcepts . </s> Data </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġonly Ġscientific Ġsummar ization Ġbenchmark Ġis Ġfrom ĠT AC Ġ2014 Ġsummar ization Ġtrack . ĠFor Ġevaluating Ġthe Ġeffectiveness Ġof ĠRouge Ġvariants Ġand Ġour Ġmetric Ġ( Ser a ), Ġwe Ġuse Ġthis Ġbenchmark , Ġwhich Ġconsists Ġof Ġ20 Ġtopics Ġeach Ġwith Ġa Ġbiomedical Ġjournal Ġarticle Ġand Ġ4 Ġgold Ġhuman Ġwritten Ġsumm aries . </s> An notations </s> In Ġthe ĠT AC Ġ2014 Ġsummar ization Ġtrack , ĠRouge Ġwas Ġsuggested Ġas Ġthe Ġevaluation Ġmetric Ġfor Ġsummar ization Ġand Ġno Ġhuman Ġassessment Ġwas Ġprovided Ġfor Ġthe Ġtopics . ĠTherefore , Ġto Ġstudy Ġthe Ġeffectiveness Ġof Ġthe Ġevaluation Ġmetrics , Ġwe Ġuse Ġthe Ġsemi - man ual ĠPyramid Ġevaluation Ġframework ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIn Ġthe Ġpyramid Ġscoring , Ġthe Ġcontent Ġunits Ġin Ġthe Ġgold Ġhuman Ġwritten Ġsumm aries Ġare Ġorganized Ġin Ġa Ġpyramid . ĠIn Ġthis Ġpyramid , Ġthe Ġcontent Ġunits Ġare Ġorganized Ġin Ġtiers Ġand Ġhigher Ġtiers Ġof Ġthe Ġpyramid Ġindicate Ġhigher Ġimportance . ĠThe Ġcontent Ġquality Ġof Ġa Ġgiven Ġcandidate Ġsummary Ġis Ġevaluated Ġwith Ġrespect Ġto Ġthis Ġpyramid . </s> To Ġanalyze Ġthe Ġquality Ġof Ġthe Ġevaluation Ġmetrics , Ġfollowing Ġthe Ġpyramid Ġframework , Ġwe Ġdesign Ġan Ġannotation Ġscheme Ġthat Ġis Ġbased Ġon Ġidentification Ġof Ġimportant Ġcontent Ġunits . ĠConsider Ġthe Ġfollowing Ġexample : </s> End ogeneous Ġsmall ĠRN As Ġ( mi RNA ) Ġwere Ġgenetically Ġscreened Ġand Ġstudied Ġto Ġfind Ġthe Ġmi RN As Ġwhich Ġare Ġrelated Ġto Ġtumor igen esis . </s> In Ġthe Ġabove Ġexample , Ġthe Ġunder lined Ġexpressions Ġare Ġthe Ġcontent Ġunits Ġthat Ġconvey Ġthe Ġmain Ġmeaning Ġof Ġthe Ġtext . ĠWe Ġcall Ġthese Ġsmall Ġunits , Ġn uggets Ġwhich Ġare Ġphrases Ġor Ġconcepts Ġthat Ġare Ġthe Ġmain Ġcontributors Ġto Ġthe Ġcontent Ġquality Ġof Ġthe Ġsummary . </s> We Ġasked Ġtwo Ġhuman Ġannot ators Ġto Ġreview Ġthe Ġgold Ġsumm aries Ġand Ġextract Ġcontent Ġunits Ġin Ġthese Ġsumm aries . ĠThe Ġpyramid Ġtiers Ġrepresent Ġthe Ġoccurrences Ġof Ġn uggets Ġacross Ġall Ġthe Ġhuman Ġwritten Ġgold - standard Ġsumm aries , Ġand Ġtherefore Ġthe Ġn uggets Ġare Ġweighted Ġbased Ġon Ġthese Ġtiers . ĠThe Ġintuition Ġis Ġthat , Ġif Ġa Ġn ug get Ġoccurs Ġmore Ġfrequently Ġin Ġthe Ġhuman Ġsumm aries , Ġit Ġis Ġa Ġmore Ġimportant Ġcontributor Ġ( thus Ġbelongs Ġto Ġhigher Ġtier Ġin Ġthe Ġpyramid ). ĠThus , Ġif Ġa Ġcandidate Ġsummary Ġcontains Ġthis Ġn ug get , Ġit Ġshould Ġbe Ġrewarded Ġmore . ĠAn Ġexample Ġof Ġthe Ġn uggets Ġannotations Ġin Ġpyramid Ġframework Ġis Ġshown Ġin ĠTable ĠT AB REF 12 Ġ. ĠIn Ġthis Ġexample , Ġthe Ġn ug get ĠâĢ ľ cell Ġmutation âĢ Ŀ Ġbelongs Ġto Ġthe Ġ4 th Ġtier Ġand Ġit Ġsuggests Ġthat Ġthe ĠâĢ ľ cell Ġmutation âĢ Ŀ Ġn ug get Ġis Ġa Ġvery Ġimportant Ġrepresentative Ġof Ġthe Ġcontent Ġof Ġthe Ġcorresponding Ġdocument . </s> Let ĠIN LINE FORM 0 Ġdefine Ġthe Ġtiers Ġof Ġthe Ġpyramid Ġwith ĠIN LINE FORM 1 Ġbeing Ġthe Ġbottom Ġtier Ġand ĠIN LINE FORM 2 Ġthe Ġtop Ġtier . ĠLet ĠIN LINE FORM 3 Ġbe Ġthe Ġnumber Ġof Ġthe Ġn uggets Ġin Ġthe Ġcandidate Ġsummary Ġthat Ġappear Ġin Ġthe Ġtier ĠIN LINE FORM 4 Ġ. ĠThen Ġthe Ġpyramid Ġscore ĠIN LINE FORM 5 Ġof Ġthe Ġcandidate Ġsummary Ġwill Ġbe : ĠIN LINE FORM 6 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmaximum Ġattain able Ġscore Ġused Ġfor Ġnormal izing Ġthe Ġscores : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġtotal Ġnumber Ġof Ġn uggets Ġin Ġthe Ġsummary Ġand ĠIN LINE FORM 1 Ġ. </s> We Ġrelease Ġthe Ġpyramid Ġannotations Ġof Ġthe ĠT AC Ġ2014 Ġdataset Ġthrough Ġa Ġpublic Ġrepository [ 2 ]. </s> [ 2 ] https :// github . com / ac ohan / T AC - py ramid - An notations </s> 3 . 1 pt </s> Sum mar ization Ġapproaches </s> We Ġstudy Ġthe Ġeffectiveness Ġof ĠRouge Ġand Ġour Ġproposed Ġmethod Ġ( Ser a ) Ġby Ġanalyzing Ġthe Ġcorrelations Ġwith Ġsemi - man ual Ġhuman Ġjudgments . ĠVery Ġfew Ġteams Ġparticipated Ġin ĠT AC Ġ2014 Ġsummar ization Ġtrack Ġand Ġthe Ġofficial Ġresults Ġand Ġthe Ġreview Ġpaper Ġof ĠT AC Ġ2014 Ġsystems Ġwere Ġnever Ġpublished . ĠTherefore , Ġto Ġevaluate Ġthe Ġeffectiveness Ġof ĠRouge , Ġwe Ġapplied Ġ9 Ġwell - known Ġsummar ization Ġapproaches Ġon Ġthe ĠT AC Ġ2014 Ġscientific Ġsummar ization Ġdataset . ĠOb tained ĠRouge Ġand ĠSer a Ġresults Ġof Ġeach Ġof Ġthese Ġapproaches Ġare Ġthen Ġcorrelated Ġwith Ġsemi - man ual Ġhuman Ġjudgments . ĠIn Ġthe Ġfollowing , Ġwe Ġbriefly Ġdescribe Ġeach Ġof Ġthese Ġsummar ization Ġapproaches . </s> Lex Rank ĠB IB REF 9 Ġ: ĠLex Rank Ġfinds Ġthe Ġmost Ġimportant Ġ( central ) Ġsentences Ġin Ġa Ġdocument Ġby Ġusing Ġrandom Ġwalks Ġin Ġa Ġgraph Ġconstructed Ġfrom Ġthe Ġdocument Ġsentences . ĠIn Ġthis Ġgraph , Ġthe Ġsentences Ġare Ġnodes Ġand Ġthe Ġsimilarity Ġbetween Ġthe Ġsentences Ġdetermines Ġthe Ġedges . ĠSent ences Ġare Ġranked Ġaccording Ġto Ġtheir Ġimportance . ĠImport ance Ġis Ġmeasured Ġin Ġterms Ġof Ġcentral ity Ġof Ġthe Ġsentence ĠâĢĶ Ġthe Ġtotal Ġnumber Ġof Ġedges Ġincident Ġon Ġthe Ġnode Ġ( sent ence ) Ġin Ġthe Ġgraph . ĠThe Ġintuition Ġbehind ĠLex Rank Ġis Ġthat Ġa Ġdocument Ġcan Ġbe Ġsummarized Ġusing Ġthe Ġmost Ġcentral Ġsentences Ġin Ġthe Ġdocument Ġthat Ġcapture Ġits Ġmain Ġaspects . </s> Lat ent ĠSem antic ĠAnalysis Ġ( L SA ) Ġbased Ġsummar ization ĠB IB REF 10 Ġ: ĠIn Ġthis Ġsummar ization Ġmethod , ĠSing ular ĠValue ĠDec om position Ġ( S VD ) ĠB IB REF 11 Ġis Ġused Ġfor Ġder iving Ġlatent Ġsemantic Ġstructure Ġof Ġthe Ġdocument . ĠThe Ġdocument Ġis Ġdivided Ġinto Ġsentences Ġand Ġa Ġterm - sent ence Ġmatrix ĠIN LINE FORM 0 Ġis Ġconstructed . ĠThe Ġmatrix ĠIN LINE FORM 1 Ġis Ġthen Ġdecomp osed Ġinto Ġa Ġnumber Ġof Ġlin early - independent Ġsingular Ġvectors Ġwhich Ġrepresent Ġthe Ġlatent Ġconcepts Ġin Ġthe Ġdocument . ĠThis Ġmethod , Ġintuitive ly , Ġdecom poses Ġthe Ġdocument Ġinto Ġseveral Ġlatent Ġtopics Ġand Ġthen Ġselects Ġthe Ġmost Ġrepresentative Ġsentences Ġfor Ġeach Ġof Ġthese Ġtopics Ġas Ġthe Ġsummary Ġof Ġthe Ġdocument . </s> Max imal ĠMarg inal ĠRe lev ance Ġ( MM R ) ĠB IB REF 12 Ġ: ĠMax imal ĠMarg inal ĠRe lev ance Ġ( MM R ) Ġis Ġa Ġgreedy Ġstrategy Ġfor Ġselecting Ġsentences Ġfor Ġthe Ġsummary . ĠSent ences Ġare Ġadded Ġiter atively Ġto Ġthe Ġsummary Ġbased Ġon Ġtheir Ġrelated ness Ġto Ġthe Ġdocument Ġas Ġwell Ġas Ġtheir Ġnovelty Ġwith Ġrespect Ġto Ġthe Ġcurrent Ġsummary . </s> C itation Ġbased Ġsummar ization ĠB IB REF 13 Ġ: ĠIn Ġthis Ġmethod , Ġcitations Ġare Ġused Ġfor Ġsummar izing Ġan Ġarticle . ĠUsing Ġthe ĠLex Rank Ġalgorithm Ġon Ġthe Ġcitation Ġnetwork Ġof Ġthe Ġarticle , Ġtop Ġsentences Ġare Ġselected Ġfor Ġthe Ġfinal Ġsummary . </s> Using Ġfrequency Ġof Ġthe Ġwords ĠB IB REF 14 Ġ: ĠIn Ġthis Ġmethod , Ġwhich Ġis Ġone Ġthe Ġearliest Ġworks Ġin Ġtext Ġsummar ization , Ġraw Ġword Ġfrequencies Ġare Ġused Ġto Ġestimate Ġthe Ġsal iency Ġof Ġsentences Ġin Ġthe Ġdocument . ĠThe Ġmost Ġsalient Ġsentences Ġare Ġchosen Ġfor Ġthe Ġfinal Ġsummary . </s> Sum Basic ĠB IB REF 15 Ġ: ĠSum Basic Ġis Ġan Ġapproach Ġthat Ġweights Ġsentences Ġbased Ġon Ġthe Ġdistribution Ġof Ġwords Ġthat Ġis Ġderived Ġfrom Ġthe Ġdocument . ĠSent ence Ġselection Ġis Ġapplied Ġiter atively Ġby Ġselecting Ġwords Ġwith Ġhighest Ġprobability Ġand Ġthen Ġfinding Ġthe Ġhighest Ġscoring Ġsentence Ġthat Ġcontains Ġthat Ġword . ĠThe Ġword Ġweights Ġare Ġupdated Ġafter Ġeach Ġiteration Ġto Ġprevent Ġselection Ġof Ġsimilar Ġsentences . </s> Sum mar ization Ġusing Ġcitation - context Ġand Ġdiscourse Ġstructure ĠB IB REF 16 Ġ: ĠIn Ġthis Ġmethod , Ġthe Ġset Ġof Ġcitations Ġto Ġthe Ġarticle Ġare Ġused Ġto Ġfind Ġthe Ġarticle Ġsentences Ġthat Ġdirectly Ġreflect Ġthose Ġcitations Ġ( c itation - context s ). ĠIn Ġaddition , Ġthe Ġscientific Ġdiscourse Ġof Ġthe Ġarticle Ġis Ġutilized Ġto Ġcapture Ġdifferent Ġaspects Ġof Ġthe Ġarticle . ĠThe Ġscientific Ġdiscourse Ġusually Ġfollows Ġa Ġstructure Ġin Ġwhich Ġthe Ġauthors Ġfirst Ġdescribe Ġtheir Ġhypothesis , Ġthen Ġthe Ġmethods , Ġexperiment , Ġresults Ġand Ġimplications . ĠSent ence Ġselection Ġis Ġbased Ġon Ġfinding Ġthe Ġmost Ġimportant Ġsentences Ġin Ġeach Ġof Ġthe Ġdiscourse Ġfacets Ġof Ġthe Ġdocument Ġusing Ġthe ĠMMR Ġhe uristic . </s> K L ĠDiver gence ĠB IB REF 17 ĠIn Ġthis Ġmethod , Ġthe Ġdocument Ġun ig ram Ġdistribution ĠIN LINE FORM 0 Ġand Ġthe Ġsummary Ġun ig ram Ġdistribut ation ĠIN LINE FORM 1 Ġare Ġconsidered ; Ġthe Ġgoal Ġis Ġto Ġfind Ġa Ġsummary Ġwhose Ġdistribution Ġis Ġvery Ġclose Ġto Ġthe Ġdocument Ġdistribution . ĠThe Ġdifference Ġof Ġthe Ġdistributions Ġis Ġcaptured Ġby Ġthe ĠK ull back - Lie ber Ġ( K L ) Ġdivergence , Ġden oted Ġby ĠIN LINE FORM 2 Ġ. </s> Sum mar ization Ġbased Ġon ĠTopic ĠModels ĠB IB REF 17 Ġ: ĠInstead Ġof Ġusing Ġun ig ram Ġdistributions Ġfor Ġmodeling Ġthe Ġcontent Ġdistribution Ġof Ġthe Ġdocument Ġand Ġthe Ġsummary , Ġthis Ġmethod Ġmodels Ġthe Ġdocument Ġcontent Ġusing Ġan ĠL DA Ġbased Ġtopic Ġmodel ĠB IB REF 18 Ġ. ĠIt Ġthen Ġuses Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġdocument Ġand Ġthe Ġsummary Ġcontent Ġmodels Ġfor Ġselecting Ġsentences Ġfor Ġthe Ġsummary . </s> Results Ġand ĠDiscussion </s> We Ġcalculated Ġall Ġvariants Ġof ĠRouge Ġscores , Ġour Ġproposed Ġmetric , ĠSer a , Ġand Ġthe ĠPyramid Ġscore Ġon Ġthe Ġgenerated Ġsumm aries Ġfrom Ġthe Ġsummar izers Ġdescribed Ġin ĠSection ĠSEC REF 13 Ġ. ĠWe Ġdo Ġnot Ġreport Ġthe ĠRouge , ĠSer a Ġor Ġpyramid Ġscores Ġof Ġindividual Ġsystems Ġas Ġit Ġis Ġnot Ġthe Ġfocus Ġof Ġthis Ġstudy . ĠOur Ġaim Ġis Ġto Ġanalyze Ġthe Ġeffectiveness Ġof Ġthe Ġevaluation Ġmetrics , Ġnot Ġthe Ġsummar ization Ġapproaches . ĠTherefore , Ġwe Ġconsider Ġthe Ġcorrelations Ġof Ġthe Ġautomatic Ġevaluation Ġmetrics Ġwith Ġthe Ġmanual ĠPyramid Ġscores Ġto Ġevaluate Ġtheir Ġeffectiveness ; Ġthe Ġmetrics Ġthat Ġshow Ġhigher Ġcorrelations Ġwith Ġmanual Ġjudgments Ġare Ġmore Ġeffective . </s> Table ĠT AB REF 23 Ġshows Ġthe ĠPearson , ĠSpear man Ġand ĠKendall Ġcorrelation Ġof ĠRouge Ġand ĠSer a , Ġwith Ġpyramid Ġscores . ĠBoth ĠRouge Ġand ĠSer a Ġare Ġcalculated Ġwith Ġstop words Ġremoved Ġand Ġwith Ġstemming . ĠOur Ġexperiments Ġwith Ġinclusion Ġof Ġstop words Ġand Ġwithout Ġstemming Ġshowed Ġsimilar Ġresults Ġand Ġthus , Ġwe Ġdo Ġnot Ġinclude Ġthose Ġto Ġavoid Ġredundancy . </s> Ser a </s> The Ġresults Ġof Ġour Ġproposed Ġmethod Ġ( Ser a ) Ġare Ġshown Ġin Ġthe Ġbottom Ġpart Ġof ĠTable ĠT AB REF 23 Ġ. ĠIn Ġgeneral , ĠSer a Ġshows Ġbetter Ġcorrelation Ġwith Ġpyramid Ġscores Ġin Ġcomparison Ġwith ĠRouge . ĠWe Ġobserve Ġthat Ġthe ĠPearson Ġcorrelation Ġof ĠSer a Ġwith Ġcut - off Ġpoint Ġof Ġ5 Ġ( shown Ġby ĠSer a - 5 ) Ġis Ġ0 . 8 23 Ġwhich Ġis Ġhigher Ġthan Ġmost Ġof Ġthe ĠRouge Ġvariants . ĠSimilarly , Ġthe ĠSpear man Ġand ĠKendall Ġcorrelations Ġof Ġthe ĠSer a Ġevaluation Ġscore Ġis Ġ0 . 9 41 Ġand Ġ0 . 8 57 Ġrespectively , Ġwhich Ġare Ġhigher Ġthan Ġall ĠRouge Ġcorrelation Ġvalues . ĠThis Ġshows Ġthe Ġeffectiveness Ġof Ġthe Ġsimple Ġvariant Ġof Ġour Ġproposed Ġsummar ization Ġevaluation Ġmetric . </s> Table ĠT AB REF 23 Ġalso Ġshows Ġthe Ġresults Ġof Ġother ĠSer a Ġvariants Ġincluding Ġdiscount ing Ġand Ġquery Ġreform ulation Ġmethods . ĠSome Ġof Ġthese Ġvariants Ġare Ġthe Ġresult Ġof Ġapplying Ġquery Ġreform ulation Ġin Ġthe Ġprocess Ġof Ġdocument Ġretrieval Ġwhich Ġare Ġdescribed Ġin Ġsection ĠSEC REF 3 ĠAs Ġillustrated , Ġthe ĠN oun ĠPh r ases Ġ( NP ) Ġquery Ġreform ulation Ġat Ġcut - off Ġpoint Ġof Ġ5 Ġ( shown Ġas ĠSer a - np - 5 ) Ġachieves Ġthe Ġhighest Ġcorrelations Ġamong Ġall Ġthe ĠSer a Ġvariants Ġ( ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ= ĠIN LINE FORM 3 Ġ= ĠIN LINE FORM 4 Ġ). ĠIn Ġthe Ġcase Ġof ĠKey words Ġ( K W ) Ġquery Ġreform ulation , Ġwithout Ġusing Ġdiscount ing , Ġwe Ġcan Ġsee Ġthat Ġthere Ġis Ġno Ġpositive Ġgain Ġin Ġcorrelation . ĠHowever , Ġkeywords Ġwhen Ġapplied Ġon Ġthe Ġdiscounted Ġvariant Ġof ĠSer a , Ġresult Ġin Ġhigher Ġcorrelations . </s> Disc ount ing Ġhas Ġmore Ġpositive Ġeffect Ġwhen Ġapplied Ġon Ġquery Ġreform ulation - based ĠSer a Ġthan Ġon Ġthe Ġsimple Ġvariant Ġof ĠSer a . ĠIn Ġthe Ġcase Ġof Ġdiscount ing Ġand ĠNP Ġquery Ġreform ulation Ġ( Ser a - dis - np ), Ġwe Ġobserve Ġhigher Ġcorrelations Ġin Ġcomparison Ġwith Ġsimple ĠSer a . ĠSimilarly , Ġin Ġthe Ġcase Ġof ĠKey words Ġ( K W ), Ġpositive Ġcorrelation Ġgain Ġis Ġobtained Ġin Ġmost Ġof Ġcorrelation Ġcoefficients . ĠNP Ġwithout Ġdiscount ing Ġand Ġat Ġcut - off Ġpoint Ġof Ġ5 Ġ( Ser a - np - 5 ) Ġshows Ġthe Ġhighest Ġnon - param etric Ġcorrelation . ĠIn Ġaddition , Ġthe Ġdiscounted ĠNP Ġat Ġcut - off Ġpoint Ġof Ġ10 Ġ( Ser a - np - dis - 10 ) Ġshows Ġthe Ġhighest Ġparam etric Ġcorrelations . </s> In Ġgeneral , Ġusing ĠNP Ġand ĠK W Ġas Ġhe ur istics Ġfor Ġfinding Ġthe Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġeffectively Ġincreases Ġthe Ġcorrelations Ġwith Ġthe Ġmanual Ġscores . ĠSelect ing Ġinformative Ġterms Ġfrom Ġlong Ġqueries Ġresults Ġin Ġmore Ġrelevant Ġdocuments Ġand Ġprevents Ġquery Ġdrift . ĠTherefore , Ġthe Ġoverall Ġsimilarity Ġbetween Ġthe Ġtwo Ġsumm aries Ġ( cand idate Ġand Ġthe Ġhuman Ġwritten Ġgold Ġsummary ) Ġis Ġbetter Ġcaptured . </s> R ou ge </s> Another Ġimportant Ġobservation Ġis Ġregarding Ġthe Ġeffectiveness Ġof ĠRouge Ġscores Ġ( top Ġpart Ġof ĠTable ĠT AB REF 23 Ġ). ĠInterestingly , Ġwe Ġobserve Ġthat Ġmany Ġvariants Ġof ĠRouge Ġscores Ġdo Ġnot Ġhave Ġhigh Ġcorrelations Ġwith Ġhuman Ġpyramid Ġscores . ĠThe Ġlowest ĠF - score Ġcorrelations Ġare Ġfor ĠRouge - 1 Ġand ĠRouge - L Ġ( with ĠIN LINE FORM 0 Ġ= 0 . 454 ). ĠWeak Ġcorrelation Ġof ĠRouge - 1 Ġshows Ġthat Ġmatching Ġun ig rams Ġbetween Ġthe Ġcandidate Ġsummary Ġand Ġgold Ġsumm aries Ġis Ġnot Ġaccurate Ġin Ġquant ifying Ġthe Ġquality Ġof Ġthe Ġsummary . ĠOn Ġhigher Ġorder Ġn - gram s , Ġhowever , Ġwe Ġcan Ġsee Ġthat ĠRouge Ġcorrelates Ġbetter Ġwith Ġpyramid . ĠIn Ġfact , Ġthe Ġhighest Ġoverall ĠIN LINE FORM 1 Ġis Ġobtained Ġby ĠRouge - 3 . ĠRouge - L Ġand Ġits Ġweighted Ġversion ĠRouge - W , Ġboth Ġhave Ġweak Ġcorrelations Ġwith Ġpyramid . ĠSkip - big rams Ġ( R ou ge - S ) Ġand Ġits Ġcombination Ġwith Ġun ig rams Ġ( R ou ge - SU ) Ġalso Ġshow Ġsub - opt imal Ġcorrelations . ĠNote Ġthat ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġcorrelations Ġare Ġmore Ġreliable Ġin Ġour Ġsetup Ġdue Ġto Ġthe Ġsmall Ġsample Ġsize . </s> These Ġresults Ġconfirm Ġour Ġinitial Ġhypothesis Ġthat ĠRouge Ġis Ġnot Ġaccurate Ġestim ator Ġof Ġthe Ġquality Ġof Ġthe Ġsummary Ġin Ġscientific Ġsummar ization . ĠWe Ġattribute Ġthis Ġto Ġthe Ġdifferences Ġof Ġscientific Ġsummar ization Ġwith Ġgeneral Ġdomain Ġsumm aries . ĠWhen Ġhumans Ġsummarize Ġa Ġrelatively Ġlong Ġresearch Ġpaper , Ġthey Ġmight Ġuse Ġdifferent Ġterminology Ġand Ġparaph r asing . ĠTherefore , ĠRouge Ġwhich Ġonly Ġrelies Ġon Ġterm Ġmatching Ġbetween Ġa Ġcandidate Ġand Ġa Ġgold Ġsummary , Ġis Ġnot Ġaccurate Ġin Ġquant ifying Ġthe Ġquality Ġof Ġthe Ġcandidate Ġsummary . </s> Cor relation Ġof ĠSer a Ġwith ĠRouge </s> Table ĠT AB REF 25 Ġshows Ġcorrelations Ġof Ġour Ġmetric ĠSer a Ġwith ĠRouge - 2 Ġand ĠRouge - 3 , Ġwhich Ġare Ġthe Ġhighest Ġcorrelated ĠRouge Ġvariants Ġwith Ġpyramid . ĠWe Ġcan Ġsee Ġthat Ġin Ġgeneral , Ġthe Ġcorrelation Ġis Ġnot Ġstrong . ĠKey word Ġbased Ġreduction Ġvariants Ġare Ġthe Ġonly Ġvariants Ġfor Ġwhich Ġthe Ġcorrelation Ġwith ĠRouge Ġis Ġhigh . ĠLooking Ġat Ġthe Ġcorrelations Ġof ĠK W Ġvariants Ġof ĠSer a Ġwith Ġpyramid Ġ( Table ĠT AB REF 23 Ġ, Ġbottom Ġpart ), Ġwe Ġobserve Ġthat Ġthese Ġvariants Ġare Ġalso Ġhighly Ġcorrelated Ġwith Ġmanual Ġevaluation . </s> Effect Ġof Ġthe Ġrank Ġcut - off Ġpoint </s> Finally , ĠFigure ĠFIG REF 28 Ġshows ĠIN LINE FORM 0 Ġcorrelation Ġof Ġdifferent Ġvariants Ġof ĠSer a Ġwith Ġpyramid Ġbased Ġon Ġselection Ġof Ġdifferent Ġcut - off Ġpoints Ġ( ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġcorrelations Ġresult Ġin Ġvery Ġsimilar Ġgraphs ). ĠWhen Ġthe Ġcut - off Ġpoint Ġincreases , Ġmore Ġdocuments Ġare Ġretrieved Ġfor Ġthe Ġcandidate Ġand Ġthe Ġgold Ġsumm aries , Ġand Ġtherefore Ġthe Ġfinal ĠSer a Ġscore Ġis Ġmore Ġfine - gr ained . ĠA Ġgeneral Ġobservation Ġis Ġthat Ġas Ġthe Ġsearch Ġcut - off Ġpoint Ġincreases , Ġthe Ġcorrelation Ġwith Ġpyramid Ġscores Ġdecreases . ĠThis Ġis Ġbecause Ġwhen Ġthe Ġretrieved Ġresult Ġlist Ġbecomes Ġlarger , Ġthe Ġprobability Ġof Ġincluding Ġless Ġrelated Ġdocuments Ġincreases Ġwhich Ġnegatively Ġaffects Ġcorrect Ġestimation Ġof Ġthe Ġsimilarity Ġof Ġthe Ġcandidate Ġand Ġgold Ġsumm aries . ĠThe Ġmost Ġaccurate Ġestim ations Ġare Ġfor Ġmetrics Ġwith Ġcut - off Ġpoints Ġof Ġ5 Ġand Ġ10 Ġwhich Ġare Ġincluded Ġin Ġthe Ġreported Ġresults Ġof Ġall Ġvariants Ġin ĠTable ĠT AB REF 23 Ġ. </s> Related Ġwork </s> R ou ge ĠB IB REF 1 Ġassess es Ġthe Ġcontent Ġquality Ġof Ġa Ġcandidate Ġsummary Ġwith Ġrespect Ġto Ġa Ġset Ġof Ġhuman Ġgold Ġsumm aries Ġbased Ġon Ġtheir Ġlex ical Ġoverl aps . ĠRouge Ġconsists Ġof Ġseveral Ġvariants . ĠSince Ġits Ġintroduction , ĠRouge Ġhas Ġbeen Ġone Ġof Ġthe Ġmost Ġwidely Ġreported Ġmetrics Ġin Ġthe Ġsummar ization Ġliterature , Ġand Ġits Ġhigh Ġadoption Ġhas Ġbeen Ġdue Ġto Ġits Ġhigh Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores Ġin ĠD UC Ġdatasets ĠB IB REF 1 Ġ. ĠHowever , Ġlater Ġresearch Ġhas Ġcast ed Ġdoubts Ġabout Ġthe Ġaccuracy Ġof ĠRouge Ġagainst Ġmanual Ġevaluations . Ġcon roy 2008 mind Ġanalyzed ĠD UC Ġ2005 Ġto Ġ2007 Ġdata Ġand Ġshowed Ġthat Ġwhile Ġsome Ġsystems Ġachieve Ġhigh ĠRouge Ġscores Ġwith Ġrespect Ġto Ġhuman Ġsumm aries , Ġthe Ġlinguistic Ġand Ġresponsiveness Ġscores Ġof Ġthose Ġsystems Ġdo Ġnot Ġcorrespond Ġto Ġthe Ġhigh ĠRouge Ġscores . </s> We Ġstudied Ġthe Ġeffectiveness Ġof ĠRouge Ġthrough Ġcorrelation Ġanalysis Ġwith Ġmanual Ġscores . ĠBesides Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores , Ġother Ġapproaches Ġhave Ġbeen Ġexplored Ġfor Ġanalyzing Ġthe Ġeffectiveness Ġof Ġsummar ization Ġevaluation . ĠRan kel : 2011 Ġstudied Ġthe Ġextent Ġto Ġwhich Ġa Ġmetric Ġcan Ġdistinguish Ġbetween Ġthe Ġhuman Ġand Ġsystem Ġgenerated Ġsumm aries . ĠThey Ġalso Ġproposed Ġthe Ġuse Ġof Ġpaired Ġtwo - sample Ġt - tests Ġand Ġthe ĠWil co xon Ġsigned - rank Ġtest Ġas Ġan Ġalternative Ġto ĠRouge Ġin Ġevaluating Ġseveral Ġsummar izers . ĠSimilarly , Ġow cz ar z ak 2012 ass essment Ġproposed Ġthe Ġuse Ġof Ġmultiple Ġbinary Ġsignificance Ġtests Ġbetween Ġthe Ġsystem Ġsumm aries Ġfor Ġranking Ġthe Ġbest Ġsummar izers . </s> Since Ġintroduction Ġof ĠRouge , Ġthere Ġhave Ġbeen Ġother Ġefforts Ġfor Ġimproving Ġautomatic Ġsummar ization Ġevaluation . Ġh ovy 2006 aut om ated Ġproposed Ġan Ġapproach Ġbased Ġon Ġcomparison Ġof Ġso Ġcalled ĠBasic ĠElements Ġ( BE ) Ġbetween Ġthe Ġcandidate Ġand Ġreference Ġsumm aries . ĠBE s Ġwere Ġextracted Ġbased Ġon Ġsynt actic Ġstructure Ġof Ġthe Ġsentence . ĠThe Ġwork Ġby Ġcon roy 2011 n ou ve au Ġwas Ġanother Ġattempt Ġfor Ġimproving ĠRouge Ġfor Ġupdate Ġsummar ization Ġwhich Ġcombined Ġtwo Ġdifferent ĠRouge Ġvariants Ġand Ġshowed Ġhigher Ġcorrelations Ġwith Ġmanual Ġjudgments Ġfor ĠT AC Ġ2008 Ġupdate Ġsumm aries . </s> Apart Ġfrom Ġthe Ġcontent , Ġother Ġaspects Ġof Ġsummar ization Ġsuch Ġas Ġlinguistic Ġquality Ġhave Ġbeen Ġalso Ġstudied . Ġpit ler 2010 automatic Ġevaluated Ġa Ġset Ġof Ġmodels Ġbased Ġon Ġsynt actic Ġfeatures , Ġlanguage Ġmodels Ġand Ġentity Ġco he rences Ġfor Ġassessing Ġthe Ġlinguistic Ġquality Ġof Ġthe Ġsumm aries . ĠMachine Ġtranslation Ġevaluation Ġmetrics Ġsuch Ġas Ġblue Ġhave Ġalso Ġbeen Ġcompared Ġand Ġcontrasted Ġagainst ĠRouge ĠB IB REF 19 Ġ. ĠDespite Ġthese Ġworks , Ġwhen Ġgold - standard Ġsumm aries Ġare Ġavailable , ĠRouge Ġis Ġstill Ġthe Ġmost Ġcommon Ġevaluation Ġmetric Ġthat Ġis Ġused Ġin Ġthe Ġsummar ization Ġpublished Ġresearch . ĠApart Ġfrom ĠRouge 's Ġinitial Ġgood Ġresults Ġon Ġthe Ġnews wire Ġdata , Ġthe Ġavailability Ġof Ġthe Ġsoftware Ġand Ġits Ġefficient Ġperformance Ġhave Ġfurther Ġcontributed Ġto Ġits Ġpopularity . </s> Con clusions </s> We Ġprovided Ġan Ġanalysis Ġof Ġexisting Ġevaluation Ġmetrics Ġfor Ġscientific Ġsummar ization Ġwith Ġevaluation Ġof Ġall Ġvariants Ġof ĠRouge . ĠWe Ġshowed Ġthat ĠRouge Ġmay Ġnot Ġbe Ġthe Ġbest Ġmetric Ġfor Ġsummar ization Ġevaluation ; Ġespecially Ġin Ġsumm aries Ġwith Ġhigh Ġterminology Ġvariations Ġand Ġparaph r asing Ġ( e . g . Ġscientific Ġsumm aries ). ĠFurthermore , Ġwe Ġshowed Ġthat Ġdifferent Ġvariants Ġof ĠRouge Ġresult Ġin Ġdifferent Ġcorrelation Ġvalues Ġwith Ġhuman Ġjudgments , Ġindicating Ġthat Ġnot Ġall ĠRouge Ġscores Ġare Ġequally Ġeffective . ĠAmong Ġall Ġvariants Ġof ĠRouge , ĠRouge - 2 Ġand ĠRouge - 3 Ġare Ġbetter Ġcorrelated Ġwith Ġmanual Ġjudgments Ġin Ġthe Ġcontext Ġof Ġscientific Ġsummar ization . ĠWe Ġfurthermore Ġproposed Ġan Ġalternative Ġand Ġmore Ġeffective Ġapproach Ġfor Ġscientific Ġsummar ization Ġevaluation Ġ( Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis Ġ- ĠSer a ). ĠResults Ġrevealed Ġthat Ġin Ġgeneral , Ġthe Ġproposed Ġevaluation Ġmetric Ġachieves Ġhigher Ġcorrelations Ġwith Ġsemi - man ual Ġpyramid Ġevaluation Ġscores Ġin Ġcomparison Ġwith ĠRouge . </s> Our Ġanalysis Ġon Ġthe Ġeffectiveness Ġof Ġevaluation Ġmeasures Ġfor Ġscientific Ġsumm aries Ġwas Ġperformed Ġusing Ġcorrelations Ġwith Ġmanual Ġjudgments . ĠAn Ġalternative Ġapproach Ġto Ġfollow Ġwould Ġbe Ġto Ġuse Ġstatistical Ġsignificance Ġtesting Ġon Ġthe Ġability Ġof Ġthe Ġmetrics Ġto Ġdistinguish Ġbetween Ġthe Ġsummar izers Ġ( similar Ġto ĠRan kel : 2011 ). ĠWe Ġstudied Ġthe Ġeffectiveness Ġof Ġexisting Ġsummar ization Ġevaluation Ġmetrics Ġin Ġthe Ġscientific Ġtext Ġgenre Ġand Ġproposed Ġan Ġalternative Ġsuperior Ġmetric . ĠAnother Ġextension Ġof Ġthis Ġwork Ġwould Ġbe Ġto Ġevaluate Ġautomatic Ġsummar ization Ġevaluation Ġin Ġother Ġgenres Ġof Ġtext Ġ( such Ġas Ġsocial Ġmedia ). ĠOur Ġproposed Ġmethod Ġonly Ġevaluates Ġthe Ġcontent Ġquality Ġof Ġthe Ġsummary . ĠSimilar Ġto Ġmost Ġof Ġexisting Ġsummar ization Ġevaluation Ġmetrics , Ġother Ġqualities Ġsuch Ġas Ġlinguistic Ġcohesion , Ġco herence Ġand Ġread ability Ġare Ġnot Ġcaptured Ġby Ġthis Ġmethod . ĠDevelop ing Ġmetrics Ġthat Ġalso Ġincorporate Ġthese Ġqualities Ġis Ġyet Ġanother Ġfuture Ġdirection Ġto Ġfollow . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġall Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġfeedback Ġand Ġcomments , Ġand ĠMary am ĠIran man esh Ġfor Ġhelping Ġin Ġannotation . ĠThis Ġwork Ġwas Ġpartially Ġsupported Ġby ĠNational ĠScience ĠFoundation Ġ( NS F ) Ġthrough Ġgrant ĠCNS - 12 04 347 .
Introduction Automatic text summarization has been an active research area in natural language processing for several decades. To compare and evaluate the performance of different summarization systems, the most intuitive approach is assessing the quality of the summaries by human evaluators. However, manual evaluation is expensive and the obtained results are subjective and difficult to reproduce BIBREF0 . To address these problems, automatic evaluation measures for summarization have been proposed. Rouge BIBREF1 is one of the first and most widely used metrics in summarization evaluation. It facilitates evaluation of system generated summaries by comparing them to a set of human written gold-standard summaries. It is inspired by the success of a similar metric Bleu BIBREF2 which is being used in Machine Translation (MT) evaluation. The main success of Rouge is due to its high correlation with human assessment scores on standard benchmarks BIBREF1 . Rouge has been used as one of the main evaluation metrics in later summarization benchmarks such as TAC[1] BIBREF3 . [1]Text Analysis Conference (TAC) is a series of workshops for evaluating research in Natural Language Processing Since the establishment of Rouge, almost all research in text summarization have used this metric as the main means for evaluating the quality of the proposed approaches. The public availability of Rouge as a toolkit for summarization evaluation has contributed to its wide usage. While Rouge has originally shown good correlations with human assessments, the study of its effectiveness was only limited to a few benchmarks on news summarization data (DUC[2] 2001-2003 benchmarks). Since 2003, summarization has grown to much further domains and genres such as scientific documents, social media and question answering. While there is not enough compelling evidence about the effectiveness of Rouge on these other summarization tasks, published research is almost always evaluated by Rouge. In addition, Rouge has a large number of possible variants and the published research often (arbitrarily) reports only a few of these variants. [2]Document Understanding Conference (DUC) was one of NIST workshops that provided infrastructure for evaluation of text summarization methodologies (http://duc.nist.gov/). By definition, Rouge solely relies on lexical overlaps (such as n-gram and sequence overlaps) between the system generated and human written gold-standard summaries. Higher lexical overlaps between the two show that the system generated summary is of higher quality. Therefore, in cases of terminology nuances and paraphrasing, Rouge is not accurate in estimating the quality of the summary. We study the effectiveness of Rouge for evaluating scientific summarization. Scientific summarization targets much more technical and focused domains in which the goal is providing summaries for scientific articles. Scientific articles are much different than news articles in elements such as length, complexity and structure. Thus, effective summarization approaches usually have much higher compression rate, terminology variations and paraphrasing BIBREF4 . Scientific summarization has attracted more attention recently (examples include works by abu2011coherent, qazvinian2013generating, and cohan2015scientific). Thus, it is important to study the validity of existing methodologies applied to the evaluation of news article summarization for this task. In particular, we raise the important question of how effective is Rouge, as an evaluation metric for scientific summarization? We answer this question by comparing Rouge scores with semi-manual evaluation score (Pyramid) in TAC 2014 scientific summarization dataset[1]. Results reveal that, contrary to the common belief, correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization. Furthermore, we show a large variance of correlations between different Rouge variants and the manual evaluations which further makes the reliability of Rouge for evaluating scientific summaries less clear. We then propose an evaluation metric based on relevance analysis of summaries which aims to overcome the limitation of high lexical dependence in Rouge. We call our metric Sera (Summarization Evaluation by Relevance Analysis). Results show that the proposed metric achieves higher and more consistent correlations with semi-manual assessment scores. [1]http://www.nist.gov/tac/2014/BiomedSumm/ Our contributions are as follows: [2]The annotations can be accessed via the following repository: https://github.com/acohan/TAC-pyramid-Annotations/ Summarization evaluation by Rouge Rouge has been the most widely used family of metrics in summarization evaluation. In the following, we briefly describe the different variants of Rouge: Rouge-L, Rouge-W, Rouge-S and Rouge-SU were later extended to consider both the recall and precision. In calculating Rouge, stopword removal or stemming can also be considered, resulting in more variants. In the summarization literature, despite the large number of variants of Rouge, only one or very few of these variants are often chosen (arbitrarily) for evaluation of the quality of the summarization approaches. When Rouge was proposed, the original variants were only recall-oriented and hence the reported correlation results BIBREF1 . The later extension of Rouge family by precision were only reflected in the later versions of the Rouge toolkit and additional evaluation of its effectiveness was not reported. Nevertheless, later published work in summarization adopted this toolkit for its ready implementation and relatively efficient performance. The original Rouge metrics show high correlations with human judgments of the quality of summaries on the DUC 2001-2003 benchmarks. However, these benchmarks consist of newswire data and are intrinsically very different than other summarization tasks such as summarization of scientific papers. We argue that Rouge is not the best metric for all summarization tasks and we propose an alternative metric for evaluation of scientific summarization. The proposed alternative metric shows much higher and more consistent correlations with manual judgments in comparison with the well-established Rouge. Summarization Evaluation by Relevance Analysis (Sera) Rouge functions based on the assumption that in order for a summary to be of high quality, it has to share many words or phrases with a human gold summary. However, different terminology may be used to refer to the same concepts and thus relying only on lexical overlaps may underrate content quality scores. To overcome this problem, we propose an approach based on the premise that concepts take meanings from the context they are in, and that related concepts co-occur frequently. Our proposed metric is based on analysis of the content relevance between a system generated summary and the corresponding human written gold-standard summaries. On high level, we indirectly evaluate the content relevance between the candidate summary and the human summary using information retrieval. To accomplish this, we use the summaries as search queries and compare the overlaps of the retrieved results. Larger number of overlaps, suggest that the candidate summary has higher content quality with respect to the gold-standard. This method, enables us to also reward for terms that are not lexically equivalent but semantically related. Our method is based on the well established linguistic premise that semantically related words occur in similar contexts BIBREF5 . The context of the words can be considered as surrounding words, sentences in which they appear or the documents. For scientific summarization, we consider the context of the words as the scientific articles in which they appear. Thus, if two concepts appear in identical set of articles, they are semantically related. We consider the two summaries as similar if they refer to same set of articles even if the two summaries do not have high lexical overlaps. To capture if a summary relates to a article, we use information retrieval by considering the summaries as queries and the articles as documents and we rank the articles based on their relatedness to a given summary. For a given pair of system summary and the gold summary, similar rankings of the retrieved articles suggest that the summaries are semantically related, and thus the system summary is of higher quality. Based on the domain of interest, we first construct an index from a set of articles in the same domain. Since TAC 2014 was focused on summarization in the biomedical domain, our index also comprises of biomedical articles. Given a candidate summary INLINEFORM0 and a set of gold summaries INLINEFORM1 ( INLINEFORM2 ; INLINEFORM3 is the total number of human summaries), we submit the candidate summary and gold summaries to the search engine as queries and compare their ranked results. Let INLINEFORM4 be the entire index which comprises of INLINEFORM5 total documents. Let INLINEFORM0 be the ranked list of retrieved documents for candidate summary INLINEFORM1 , and INLINEFORM2 the ranked list of results for the gold summary INLINEFORM3 . These lists of results are based on a rank cut-off point INLINEFORM4 that is a parameter of the system. We provide evaluation results on different choices of cut-off point INLINEFORM5 in the Section SECREF5 We consider the following two scores: (i) simple intersection and (ii) discounted intersection by rankings. The simple intersection just considers the overlaps of the results in the two ranked lists and ignores the rankings. The discounted ranked scores, on the other hand, penalizes ranking differences between the two result sets. As an example consider the following list of retrieved documents (denoted by INLINEFORM6 s) for a candidate and a gold summary as queries: Results for candidate summary: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 Results for gold summary: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 These two sets of results consist of identical documents but the ranking of the retrieved documents differ. Therefore, the simple intersection method assigns a score of 1.0 while in the discounted ranked score, the score will be less than 1.0 (due to ranking differences between the result lists). We now define the metrics more precisely. Using the above notations, without loss of generality, we assume that INLINEFORM0 . Sera is defined as follows: INLINEFORM1 To also account for the ranked position differences, we modify this score to discount rewards based on rank differences. That is, in ideal score, we want search results from candidate summary ( INLINEFORM0 ) to be the same as results for gold-standard summaries ( INLINEFORM1 ) and the rankings of the results also be the same. If the rankings differ, we discount the reward by log of the differences of the ranks. More specifically, the discounted score (Sera-Dis) is defined as: INLINEFORM2 where, as previously defined, INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are total number of human gold summaries, result list for the candidate summary and result list for the human gold summary, respectively. In addition, INLINEFORM3 shows the INLINEFORM4 th results in the ranked list INLINEFORM5 and INLINEFORM6 is the maximum attainable score used as the normalizing factor. We use elasticsearch[1], an open-source search engine, for indexing and querying the articles. For retrieval model, we use the Language Modeling retrieval model with Dirichlet smoothing BIBREF6 . Since TAC 2014 benchmark is on summarization of biomedical articles, the appropriate index would be the one constructed from articles in the same domain. Therefore, we use the open access subset of Pubmed[2] which consists of published articles in biomedical literature. [1]https://github.com/elastic/elasticsearch [2]PubMed is a comprehensive resource of articles and abstracts published in life sciences and biomedical literature http://www.ncbi.nlm.nih.gov/pmc/ We also experiment with different query (re)formulation approaches. Query reformulation is a method in Information Retrieval that aims to refine the query for better retrieval of results. Query reformulation methods often consist of removing ineffective terms and expressions from the query (query reduction) or adding terms to the query that help the retrieval (query expansion). Query reduction is specially important when queries are verbose. Since we use the summaries as queries, the queries are usually long and therefore we consider query reductions. In our experiments, the query reformulation is done by 3 different ways: (i) Plain: The entire summary without stopwords and numeric values; (ii) Noun Phrases (NP): We only keep the noun phrases as informative concepts in the summary and eliminate all other terms; and (iii) Keywords (KW): We only keep the keywords and key phrases in the summary. For extracting the keywords and keyphrases (with length of up to 3 terms), we extract expressions whose idf[1] values is higher than a predefined threshold that is set as a parameter. We set this threshold to the average idf values of all terms except stopwords. idf values are calculated on the same index that is used for the retrieval. [1]Inverted Document Frequency We hypothesize that using only informative concepts in the summary prevents query drift and leads to retrieval of more relevant documents. Noun phrases and keywords are two heuristics for identifying the informative concepts. Data To the best of our knowledge, the only scientific summarization benchmark is from TAC 2014 summarization track. For evaluating the effectiveness of Rouge variants and our metric (Sera), we use this benchmark, which consists of 20 topics each with a biomedical journal article and 4 gold human written summaries. Annotations In the TAC 2014 summarization track, Rouge was suggested as the evaluation metric for summarization and no human assessment was provided for the topics. Therefore, to study the effectiveness of the evaluation metrics, we use the semi-manual Pyramid evaluation framework BIBREF7 , BIBREF8 . In the pyramid scoring, the content units in the gold human written summaries are organized in a pyramid. In this pyramid, the content units are organized in tiers and higher tiers of the pyramid indicate higher importance. The content quality of a given candidate summary is evaluated with respect to this pyramid. To analyze the quality of the evaluation metrics, following the pyramid framework, we design an annotation scheme that is based on identification of important content units. Consider the following example: Endogeneous small RNAs (miRNA) were genetically screened and studied to find the miRNAs which are related to tumorigenesis. In the above example, the underlined expressions are the content units that convey the main meaning of the text. We call these small units, nuggets which are phrases or concepts that are the main contributors to the content quality of the summary. We asked two human annotators to review the gold summaries and extract content units in these summaries. The pyramid tiers represent the occurrences of nuggets across all the human written gold-standard summaries, and therefore the nuggets are weighted based on these tiers. The intuition is that, if a nugget occurs more frequently in the human summaries, it is a more important contributor (thus belongs to higher tier in the pyramid). Thus, if a candidate summary contains this nugget, it should be rewarded more. An example of the nuggets annotations in pyramid framework is shown in Table TABREF12 . In this example, the nugget “cell mutation” belongs to the 4th tier and it suggests that the “cell mutation” nugget is a very important representative of the content of the corresponding document. Let INLINEFORM0 define the tiers of the pyramid with INLINEFORM1 being the bottom tier and INLINEFORM2 the top tier. Let INLINEFORM3 be the number of the nuggets in the candidate summary that appear in the tier INLINEFORM4 . Then the pyramid score INLINEFORM5 of the candidate summary will be: INLINEFORM6 where INLINEFORM0 is the maximum attainable score used for normalizing the scores: INLINEFORM1 where INLINEFORM0 is the total number of nuggets in the summary and INLINEFORM1 . We release the pyramid annotations of the TAC 2014 dataset through a public repository[2]. [2]https://github.com/acohan/TAC-pyramid-Annotations 3.1pt Summarization approaches We study the effectiveness of Rouge and our proposed method (Sera) by analyzing the correlations with semi-manual human judgments. Very few teams participated in TAC 2014 summarization track and the official results and the review paper of TAC 2014 systems were never published. Therefore, to evaluate the effectiveness of Rouge, we applied 9 well-known summarization approaches on the TAC 2014 scientific summarization dataset. Obtained Rouge and Sera results of each of these approaches are then correlated with semi-manual human judgments. In the following, we briefly describe each of these summarization approaches. LexRank BIBREF9 : LexRank finds the most important (central) sentences in a document by using random walks in a graph constructed from the document sentences. In this graph, the sentences are nodes and the similarity between the sentences determines the edges. Sentences are ranked according to their importance. Importance is measured in terms of centrality of the sentence — the total number of edges incident on the node (sentence) in the graph. The intuition behind LexRank is that a document can be summarized using the most central sentences in the document that capture its main aspects. Latent Semantic Analysis (LSA) based summarization BIBREF10 : In this summarization method, Singular Value Decomposition (SVD) BIBREF11 is used for deriving latent semantic structure of the document. The document is divided into sentences and a term-sentence matrix INLINEFORM0 is constructed. The matrix INLINEFORM1 is then decomposed into a number of linearly-independent singular vectors which represent the latent concepts in the document. This method, intuitively, decomposes the document into several latent topics and then selects the most representative sentences for each of these topics as the summary of the document. Maximal Marginal Relevance (MMR) BIBREF12 : Maximal Marginal Relevance (MMR) is a greedy strategy for selecting sentences for the summary. Sentences are added iteratively to the summary based on their relatedness to the document as well as their novelty with respect to the current summary. Citation based summarization BIBREF13 : In this method, citations are used for summarizing an article. Using the LexRank algorithm on the citation network of the article, top sentences are selected for the final summary. Using frequency of the words BIBREF14 : In this method, which is one the earliest works in text summarization, raw word frequencies are used to estimate the saliency of sentences in the document. The most salient sentences are chosen for the final summary. SumBasic BIBREF15 : SumBasic is an approach that weights sentences based on the distribution of words that is derived from the document. Sentence selection is applied iteratively by selecting words with highest probability and then finding the highest scoring sentence that contains that word. The word weights are updated after each iteration to prevent selection of similar sentences. Summarization using citation-context and discourse structure BIBREF16 : In this method, the set of citations to the article are used to find the article sentences that directly reflect those citations (citation-contexts). In addition, the scientific discourse of the article is utilized to capture different aspects of the article. The scientific discourse usually follows a structure in which the authors first describe their hypothesis, then the methods, experiment, results and implications. Sentence selection is based on finding the most important sentences in each of the discourse facets of the document using the MMR heuristic. KL Divergence BIBREF17 In this method, the document unigram distribution INLINEFORM0 and the summary unigram distributation INLINEFORM1 are considered; the goal is to find a summary whose distribution is very close to the document distribution. The difference of the distributions is captured by the Kullback-Lieber (KL) divergence, denoted by INLINEFORM2 . Summarization based on Topic Models BIBREF17 : Instead of using unigram distributions for modeling the content distribution of the document and the summary, this method models the document content using an LDA based topic model BIBREF18 . It then uses the KL divergence between the document and the summary content models for selecting sentences for the summary. Results and Discussion We calculated all variants of Rouge scores, our proposed metric, Sera, and the Pyramid score on the generated summaries from the summarizers described in Section SECREF13 . We do not report the Rouge, Sera or pyramid scores of individual systems as it is not the focus of this study. Our aim is to analyze the effectiveness of the evaluation metrics, not the summarization approaches. Therefore, we consider the correlations of the automatic evaluation metrics with the manual Pyramid scores to evaluate their effectiveness; the metrics that show higher correlations with manual judgments are more effective. Table TABREF23 shows the Pearson, Spearman and Kendall correlation of Rouge and Sera, with pyramid scores. Both Rouge and Sera are calculated with stopwords removed and with stemming. Our experiments with inclusion of stopwords and without stemming showed similar results and thus, we do not include those to avoid redundancy. Sera The results of our proposed method (Sera) are shown in the bottom part of Table TABREF23 . In general, Sera shows better correlation with pyramid scores in comparison with Rouge. We observe that the Pearson correlation of Sera with cut-off point of 5 (shown by Sera-5) is 0.823 which is higher than most of the Rouge variants. Similarly, the Spearman and Kendall correlations of the Sera evaluation score is 0.941 and 0.857 respectively, which are higher than all Rouge correlation values. This shows the effectiveness of the simple variant of our proposed summarization evaluation metric. Table TABREF23 also shows the results of other Sera variants including discounting and query reformulation methods. Some of these variants are the result of applying query reformulation in the process of document retrieval which are described in section SECREF3 As illustrated, the Noun Phrases (NP) query reformulation at cut-off point of 5 (shown as Sera-np-5) achieves the highest correlations among all the Sera variants ( INLINEFORM0 = INLINEFORM1 , INLINEFORM2 = INLINEFORM3 = INLINEFORM4 ). In the case of Keywords (KW) query reformulation, without using discounting, we can see that there is no positive gain in correlation. However, keywords when applied on the discounted variant of Sera, result in higher correlations. Discounting has more positive effect when applied on query reformulation-based Sera than on the simple variant of Sera. In the case of discounting and NP query reformulation (Sera-dis-np), we observe higher correlations in comparison with simple Sera. Similarly, in the case of Keywords (KW), positive correlation gain is obtained in most of correlation coefficients. NP without discounting and at cut-off point of 5 (Sera-np-5) shows the highest non-parametric correlation. In addition, the discounted NP at cut-off point of 10 (Sera-np-dis-10) shows the highest parametric correlations. In general, using NP and KW as heuristics for finding the informative concepts in the summary effectively increases the correlations with the manual scores. Selecting informative terms from long queries results in more relevant documents and prevents query drift. Therefore, the overall similarity between the two summaries (candidate and the human written gold summary) is better captured. Rouge Another important observation is regarding the effectiveness of Rouge scores (top part of Table TABREF23 ). Interestingly, we observe that many variants of Rouge scores do not have high correlations with human pyramid scores. The lowest F-score correlations are for Rouge-1 and Rouge-L (with INLINEFORM0 =0.454). Weak correlation of Rouge-1 shows that matching unigrams between the candidate summary and gold summaries is not accurate in quantifying the quality of the summary. On higher order n-grams, however, we can see that Rouge correlates better with pyramid. In fact, the highest overall INLINEFORM1 is obtained by Rouge-3. Rouge-L and its weighted version Rouge-W, both have weak correlations with pyramid. Skip-bigrams (Rouge-S) and its combination with unigrams (Rouge-SU) also show sub-optimal correlations. Note that INLINEFORM2 and INLINEFORM3 correlations are more reliable in our setup due to the small sample size. These results confirm our initial hypothesis that Rouge is not accurate estimator of the quality of the summary in scientific summarization. We attribute this to the differences of scientific summarization with general domain summaries. When humans summarize a relatively long research paper, they might use different terminology and paraphrasing. Therefore, Rouge which only relies on term matching between a candidate and a gold summary, is not accurate in quantifying the quality of the candidate summary. Correlation of Sera with Rouge Table TABREF25 shows correlations of our metric Sera with Rouge-2 and Rouge-3, which are the highest correlated Rouge variants with pyramid. We can see that in general, the correlation is not strong. Keyword based reduction variants are the only variants for which the correlation with Rouge is high. Looking at the correlations of KW variants of Sera with pyramid (Table TABREF23 , bottom part), we observe that these variants are also highly correlated with manual evaluation. Effect of the rank cut-off point Finally, Figure FIGREF28 shows INLINEFORM0 correlation of different variants of Sera with pyramid based on selection of different cut-off points ( INLINEFORM1 and INLINEFORM2 correlations result in very similar graphs). When the cut-off point increases, more documents are retrieved for the candidate and the gold summaries, and therefore the final Sera score is more fine-grained. A general observation is that as the search cut-off point increases, the correlation with pyramid scores decreases. This is because when the retrieved result list becomes larger, the probability of including less related documents increases which negatively affects correct estimation of the similarity of the candidate and gold summaries. The most accurate estimations are for metrics with cut-off points of 5 and 10 which are included in the reported results of all variants in Table TABREF23 . Related work Rouge BIBREF1 assesses the content quality of a candidate summary with respect to a set of human gold summaries based on their lexical overlaps. Rouge consists of several variants. Since its introduction, Rouge has been one of the most widely reported metrics in the summarization literature, and its high adoption has been due to its high correlation with human assessment scores in DUC datasets BIBREF1 . However, later research has casted doubts about the accuracy of Rouge against manual evaluations. conroy2008mind analyzed DUC 2005 to 2007 data and showed that while some systems achieve high Rouge scores with respect to human summaries, the linguistic and responsiveness scores of those systems do not correspond to the high Rouge scores. We studied the effectiveness of Rouge through correlation analysis with manual scores. Besides correlation with human assessment scores, other approaches have been explored for analyzing the effectiveness of summarization evaluation. Rankel:2011 studied the extent to which a metric can distinguish between the human and system generated summaries. They also proposed the use of paired two-sample t-tests and the Wilcoxon signed-rank test as an alternative to Rouge in evaluating several summarizers. Similarly, owczarzak2012assessment proposed the use of multiple binary significance tests between the system summaries for ranking the best summarizers. Since introduction of Rouge, there have been other efforts for improving automatic summarization evaluation. hovy2006automated proposed an approach based on comparison of so called Basic Elements (BE) between the candidate and reference summaries. BEs were extracted based on syntactic structure of the sentence. The work by conroy2011nouveau was another attempt for improving Rouge for update summarization which combined two different Rouge variants and showed higher correlations with manual judgments for TAC 2008 update summaries. Apart from the content, other aspects of summarization such as linguistic quality have been also studied. pitler2010automatic evaluated a set of models based on syntactic features, language models and entity coherences for assessing the linguistic quality of the summaries. Machine translation evaluation metrics such as blue have also been compared and contrasted against Rouge BIBREF19 . Despite these works, when gold-standard summaries are available, Rouge is still the most common evaluation metric that is used in the summarization published research. Apart from Rouge's initial good results on the newswire data, the availability of the software and its efficient performance have further contributed to its popularity. Conclusions We provided an analysis of existing evaluation metrics for scientific summarization with evaluation of all variants of Rouge. We showed that Rouge may not be the best metric for summarization evaluation; especially in summaries with high terminology variations and paraphrasing (e.g. scientific summaries). Furthermore, we showed that different variants of Rouge result in different correlation values with human judgments, indicating that not all Rouge scores are equally effective. Among all variants of Rouge, Rouge-2 and Rouge-3 are better correlated with manual judgments in the context of scientific summarization. We furthermore proposed an alternative and more effective approach for scientific summarization evaluation (Summarization Evaluation by Relevance Analysis - Sera). Results revealed that in general, the proposed evaluation metric achieves higher correlations with semi-manual pyramid evaluation scores in comparison with Rouge. Our analysis on the effectiveness of evaluation measures for scientific summaries was performed using correlations with manual judgments. An alternative approach to follow would be to use statistical significance testing on the ability of the metrics to distinguish between the summarizers (similar to Rankel:2011). We studied the effectiveness of existing summarization evaluation metrics in the scientific text genre and proposed an alternative superior metric. Another extension of this work would be to evaluate automatic summarization evaluation in other genres of text (such as social media). Our proposed method only evaluates the content quality of the summary. Similar to most of existing summarization evaluation metrics, other qualities such as linguistic cohesion, coherence and readability are not captured by this method. Developing metrics that also incorporate these qualities is yet another future direction to follow. Acknowledgments We would like to thank all three anonymous reviewers for their feedback and comments, and Maryam Iranmanesh for helping in annotation. This work was partially supported by National Science Foundation (NSF) through grant CNS-1204347.
What is the common belief that this paper refutes? (c.f. 'contrary to the common belief, ROUGE is not much [sic] reliable'
184b0082e10ce191940c1d24785b631828a9f714
What Ġis Ġthe Ġcommon Ġbelief Ġthat Ġthis Ġpaper Ġref utes ? Ġ( c . f . Ġ' cont rary Ġto Ġthe Ġcommon Ġbelief , ĠR OU GE Ġis Ġnot Ġmuch Ġ[ sic ] Ġreliable '
cor relations Ġbetween ĠRouge Ġand Ġthe ĠPyramid Ġscores Ġare Ġweak , Ġwhich Ġchallenges Ġits Ġeffectiveness Ġfor Ġscientific Ġsummar ization
<s> What Ġis Ġthe Ġcommon Ġbelief Ġthat Ġthis Ġpaper Ġref utes ? Ġ( c . f . Ġ' cont rary Ġto Ġthe Ġcommon Ġbelief , ĠR OU GE Ġis Ġnot Ġmuch Ġ[ sic ] Ġreliable ' </s> Introduction </s> Aut omatic Ġtext Ġsummar ization Ġhas Ġbeen Ġan Ġactive Ġresearch Ġarea Ġin Ġnatural Ġlanguage Ġprocessing Ġfor Ġseveral Ġdecades . ĠTo Ġcompare Ġand Ġevaluate Ġthe Ġperformance Ġof Ġdifferent Ġsummar ization Ġsystems , Ġthe Ġmost Ġintuitive Ġapproach Ġis Ġassessing Ġthe Ġquality Ġof Ġthe Ġsumm aries Ġby Ġhuman Ġeval u ators . ĠHowever , Ġmanual Ġevaluation Ġis Ġexpensive Ġand Ġthe Ġobtained Ġresults Ġare Ġsubjective Ġand Ġdifficult Ġto Ġreproduce ĠB IB REF 0 Ġ. ĠTo Ġaddress Ġthese Ġproblems , Ġautomatic Ġevaluation Ġmeasures Ġfor Ġsummar ization Ġhave Ġbeen Ġproposed . ĠRouge ĠB IB REF 1 Ġis Ġone Ġof Ġthe Ġfirst Ġand Ġmost Ġwidely Ġused Ġmetrics Ġin Ġsummar ization Ġevaluation . ĠIt Ġfacilitates Ġevaluation Ġof Ġsystem Ġgenerated Ġsumm aries Ġby Ġcomparing Ġthem Ġto Ġa Ġset Ġof Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠIt Ġis Ġinspired Ġby Ġthe Ġsuccess Ġof Ġa Ġsimilar Ġmetric ĠBle u ĠB IB REF 2 Ġwhich Ġis Ġbeing Ġused Ġin ĠMachine ĠTranslation Ġ( MT ) Ġevaluation . ĠThe Ġmain Ġsuccess Ġof ĠRouge Ġis Ġdue Ġto Ġits Ġhigh Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores Ġon Ġstandard Ġbenchmarks ĠB IB REF 1 Ġ. ĠRouge Ġhas Ġbeen Ġused Ġas Ġone Ġof Ġthe Ġmain Ġevaluation Ġmetrics Ġin Ġlater Ġsummar ization Ġbenchmarks Ġsuch Ġas ĠT AC [ 1 ] ĠB IB REF 3 Ġ. </s> [ 1 ] Text ĠAnalysis ĠConference Ġ( T AC ) Ġis Ġa Ġseries Ġof Ġworkshops Ġfor Ġevaluating Ġresearch Ġin ĠNatural ĠLanguage ĠProcessing </s> Since Ġthe Ġestablishment Ġof ĠRouge , Ġalmost Ġall Ġresearch Ġin Ġtext Ġsummar ization Ġhave Ġused Ġthis Ġmetric Ġas Ġthe Ġmain Ġmeans Ġfor Ġevaluating Ġthe Ġquality Ġof Ġthe Ġproposed Ġapproaches . ĠThe Ġpublic Ġavailability Ġof ĠRouge Ġas Ġa Ġtool kit Ġfor Ġsummar ization Ġevaluation Ġhas Ġcontributed Ġto Ġits Ġwide Ġusage . ĠWhile ĠRouge Ġhas Ġoriginally Ġshown Ġgood Ġcorrelations Ġwith Ġhuman Ġassessments , Ġthe Ġstudy Ġof Ġits Ġeffectiveness Ġwas Ġonly Ġlimited Ġto Ġa Ġfew Ġbenchmarks Ġon Ġnews Ġsummar ization Ġdata Ġ( D UC [ 2 ] Ġ2001 - 2003 Ġbenchmarks ). ĠSince Ġ2003 , Ġsummar ization Ġhas Ġgrown Ġto Ġmuch Ġfurther Ġdomains Ġand Ġgenres Ġsuch Ġas Ġscientific Ġdocuments , Ġsocial Ġmedia Ġand Ġquestion Ġanswering . ĠWhile Ġthere Ġis Ġnot Ġenough Ġcompelling Ġevidence Ġabout Ġthe Ġeffectiveness Ġof ĠRouge Ġon Ġthese Ġother Ġsummar ization Ġtasks , Ġpublished Ġresearch Ġis Ġalmost Ġalways Ġevaluated Ġby ĠRouge . ĠIn Ġaddition , ĠRouge Ġhas Ġa Ġlarge Ġnumber Ġof Ġpossible Ġvariants Ġand Ġthe Ġpublished Ġresearch Ġoften Ġ( ar bit rarily ) Ġreports Ġonly Ġa Ġfew Ġof Ġthese Ġvariants . </s> [ 2 ] Document ĠUnderstanding ĠConference Ġ( D UC ) Ġwas Ġone Ġof ĠN IST Ġworkshops Ġthat Ġprovided Ġinfrastructure Ġfor Ġevaluation Ġof Ġtext Ġsummar ization Ġmethod ologies Ġ( http :// duc . n ist . gov / ). </s> By Ġdefinition , ĠRouge Ġsolely Ġrelies Ġon Ġlex ical Ġoverl aps Ġ( such Ġas Ġn - gram Ġand Ġsequence Ġoverl aps ) Ġbetween Ġthe Ġsystem Ġgenerated Ġand Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠHigher Ġlex ical Ġoverl aps Ġbetween Ġthe Ġtwo Ġshow Ġthat Ġthe Ġsystem Ġgenerated Ġsummary Ġis Ġof Ġhigher Ġquality . ĠTherefore , Ġin Ġcases Ġof Ġterminology Ġnuances Ġand Ġparaph r asing , ĠRouge Ġis Ġnot Ġaccurate Ġin Ġestimating Ġthe Ġquality Ġof Ġthe Ġsummary . </s> We Ġstudy Ġthe Ġeffectiveness Ġof ĠRouge Ġfor Ġevaluating Ġscientific Ġsummar ization . ĠScientific Ġsummar ization Ġtargets Ġmuch Ġmore Ġtechnical Ġand Ġfocused Ġdomains Ġin Ġwhich Ġthe Ġgoal Ġis Ġproviding Ġsumm aries Ġfor Ġscientific Ġarticles . ĠScientific Ġarticles Ġare Ġmuch Ġdifferent Ġthan Ġnews Ġarticles Ġin Ġelements Ġsuch Ġas Ġlength , Ġcomplexity Ġand Ġstructure . ĠThus , Ġeffective Ġsummar ization Ġapproaches Ġusually Ġhave Ġmuch Ġhigher Ġcompression Ġrate , Ġterminology Ġvariations Ġand Ġparaph r asing ĠB IB REF 4 Ġ. </s> Scient ific Ġsummar ization Ġhas Ġattracted Ġmore Ġattention Ġrecently Ġ( ex amples Ġinclude Ġworks Ġby Ġab u 2011 co herent , Ġq az vin ian 2013 gener ating , Ġand Ġco han 2015 scientific ). ĠThus , Ġit Ġis Ġimportant Ġto Ġstudy Ġthe Ġvalidity Ġof Ġexisting Ġmethod ologies Ġapplied Ġto Ġthe Ġevaluation Ġof Ġnews Ġarticle Ġsummar ization Ġfor Ġthis Ġtask . ĠIn Ġparticular , Ġwe Ġraise Ġthe Ġimportant Ġquestion Ġof Ġhow Ġeffective Ġis ĠRouge , Ġas Ġan Ġevaluation Ġmetric Ġfor Ġscientific Ġsummar ization ? ĠWe Ġanswer Ġthis Ġquestion Ġby Ġcomparing ĠRouge Ġscores Ġwith Ġsemi - man ual Ġevaluation Ġscore Ġ( Py ramid ) Ġin ĠT AC Ġ2014 Ġscientific Ġsummar ization Ġdataset [ 1 ]. ĠResults Ġreveal Ġthat , Ġcontrary Ġto Ġthe Ġcommon Ġbelief , Ġcorrelations Ġbetween ĠRouge Ġand Ġthe ĠPyramid Ġscores Ġare Ġweak , Ġwhich Ġchallenges Ġits Ġeffectiveness Ġfor Ġscientific Ġsummar ization . ĠFurthermore , Ġwe Ġshow Ġa Ġlarge Ġvariance Ġof Ġcorrelations Ġbetween Ġdifferent ĠRouge Ġvariants Ġand Ġthe Ġmanual Ġevaluations Ġwhich Ġfurther Ġmakes Ġthe Ġreliability Ġof ĠRouge Ġfor Ġevaluating Ġscientific Ġsumm aries Ġless Ġclear . ĠWe Ġthen Ġpropose Ġan Ġevaluation Ġmetric Ġbased Ġon Ġrelevance Ġanalysis Ġof Ġsumm aries Ġwhich Ġaims Ġto Ġovercome Ġthe Ġlimitation Ġof Ġhigh Ġlex ical Ġdependence Ġin ĠRouge . ĠWe Ġcall Ġour Ġmetric ĠSer a Ġ( Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis ). ĠResults Ġshow Ġthat Ġthe Ġproposed Ġmetric Ġachieves Ġhigher Ġand Ġmore Ġconsistent Ġcorrelations Ġwith Ġsemi - man ual Ġassessment Ġscores . </s> [ 1 ] http :// www . n ist . gov / t ac / 2014 / Bi omed Sum m / </s> Our Ġcontributions Ġare Ġas Ġfollows : </s> [ 2 ] The Ġannotations Ġcan Ġbe Ġaccessed Ġvia Ġthe Ġfollowing Ġrepository : Ġhttps :// github . com / ac ohan / T AC - py ramid - An notations / </s> Sum mar ization Ġevaluation Ġby ĠRouge </s> R ou ge Ġhas Ġbeen Ġthe Ġmost Ġwidely Ġused Ġfamily Ġof Ġmetrics Ġin Ġsummar ization Ġevaluation . ĠIn Ġthe Ġfollowing , Ġwe Ġbriefly Ġdescribe Ġthe Ġdifferent Ġvariants Ġof ĠRouge : </s> R ou ge - L , ĠRouge - W , ĠRouge - S Ġand ĠRouge - SU Ġwere Ġlater Ġextended Ġto Ġconsider Ġboth Ġthe Ġrecall Ġand Ġprecision . ĠIn Ġcalculating ĠRouge , Ġstop word Ġremoval Ġor Ġstemming Ġcan Ġalso Ġbe Ġconsidered , Ġresulting Ġin Ġmore Ġvariants . </s> In Ġthe Ġsummar ization Ġliterature , Ġdespite Ġthe Ġlarge Ġnumber Ġof Ġvariants Ġof ĠRouge , Ġonly Ġone Ġor Ġvery Ġfew Ġof Ġthese Ġvariants Ġare Ġoften Ġchosen Ġ( ar bit rarily ) Ġfor Ġevaluation Ġof Ġthe Ġquality Ġof Ġthe Ġsummar ization Ġapproaches . ĠWhen ĠRouge Ġwas Ġproposed , Ġthe Ġoriginal Ġvariants Ġwere Ġonly Ġrecall - oriented Ġand Ġhence Ġthe Ġreported Ġcorrelation Ġresults ĠB IB REF 1 Ġ. ĠThe Ġlater Ġextension Ġof ĠRouge Ġfamily Ġby Ġprecision Ġwere Ġonly Ġreflected Ġin Ġthe Ġlater Ġversions Ġof Ġthe ĠRouge Ġtool kit Ġand Ġadditional Ġevaluation Ġof Ġits Ġeffectiveness Ġwas Ġnot Ġreported . ĠNevertheless , Ġlater Ġpublished Ġwork Ġin Ġsummar ization Ġadopted Ġthis Ġtool kit Ġfor Ġits Ġready Ġimplementation Ġand Ġrelatively Ġefficient Ġperformance . </s> The Ġoriginal ĠRouge Ġmetrics Ġshow Ġhigh Ġcorrelations Ġwith Ġhuman Ġjudgments Ġof Ġthe Ġquality Ġof Ġsumm aries Ġon Ġthe ĠD UC Ġ2001 - 2003 Ġbenchmarks . ĠHowever , Ġthese Ġbenchmarks Ġconsist Ġof Ġnews wire Ġdata Ġand Ġare Ġintrinsically Ġvery Ġdifferent Ġthan Ġother Ġsummar ization Ġtasks Ġsuch Ġas Ġsummar ization Ġof Ġscientific Ġpapers . ĠWe Ġargue Ġthat ĠRouge Ġis Ġnot Ġthe Ġbest Ġmetric Ġfor Ġall Ġsummar ization Ġtasks Ġand Ġwe Ġpropose Ġan Ġalternative Ġmetric Ġfor Ġevaluation Ġof Ġscientific Ġsummar ization . ĠThe Ġproposed Ġalternative Ġmetric Ġshows Ġmuch Ġhigher Ġand Ġmore Ġconsistent Ġcorrelations Ġwith Ġmanual Ġjudgments Ġin Ġcomparison Ġwith Ġthe Ġwell - established ĠRouge . </s> Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis Ġ( Ser a ) </s> R ou ge Ġfunctions Ġbased Ġon Ġthe Ġassumption Ġthat Ġin Ġorder Ġfor Ġa Ġsummary Ġto Ġbe Ġof Ġhigh Ġquality , Ġit Ġhas Ġto Ġshare Ġmany Ġwords Ġor Ġphrases Ġwith Ġa Ġhuman Ġgold Ġsummary . ĠHowever , Ġdifferent Ġterminology Ġmay Ġbe Ġused Ġto Ġrefer Ġto Ġthe Ġsame Ġconcepts Ġand Ġthus Ġrelying Ġonly Ġon Ġlex ical Ġoverl aps Ġmay Ġunder rate Ġcontent Ġquality Ġscores . ĠTo Ġovercome Ġthis Ġproblem , Ġwe Ġpropose Ġan Ġapproach Ġbased Ġon Ġthe Ġpremise Ġthat Ġconcepts Ġtake Ġmeanings Ġfrom Ġthe Ġcontext Ġthey Ġare Ġin , Ġand Ġthat Ġrelated Ġconcepts Ġco - occ ur Ġfrequently . </s> Our Ġproposed Ġmetric Ġis Ġbased Ġon Ġanalysis Ġof Ġthe Ġcontent Ġrelevance Ġbetween Ġa Ġsystem Ġgenerated Ġsummary Ġand Ġthe Ġcorresponding Ġhuman Ġwritten Ġgold - standard Ġsumm aries . ĠOn Ġhigh Ġlevel , Ġwe Ġindirectly Ġevaluate Ġthe Ġcontent Ġrelevance Ġbetween Ġthe Ġcandidate Ġsummary Ġand Ġthe Ġhuman Ġsummary Ġusing Ġinformation Ġretrieval . ĠTo Ġaccomplish Ġthis , Ġwe Ġuse Ġthe Ġsumm aries Ġas Ġsearch Ġqueries Ġand Ġcompare Ġthe Ġoverl aps Ġof Ġthe Ġretrieved Ġresults . ĠL arger Ġnumber Ġof Ġoverl aps , Ġsuggest Ġthat Ġthe Ġcandidate Ġsummary Ġhas Ġhigher Ġcontent Ġquality Ġwith Ġrespect Ġto Ġthe Ġgold - standard . ĠThis Ġmethod , Ġenables Ġus Ġto Ġalso Ġreward Ġfor Ġterms Ġthat Ġare Ġnot Ġlex ically Ġequivalent Ġbut Ġsem antically Ġrelated . ĠOur Ġmethod Ġis Ġbased Ġon Ġthe Ġwell Ġestablished Ġlinguistic Ġpremise Ġthat Ġsem antically Ġrelated Ġwords Ġoccur Ġin Ġsimilar Ġcontexts ĠB IB REF 5 Ġ. ĠThe Ġcontext Ġof Ġthe Ġwords Ġcan Ġbe Ġconsidered Ġas Ġsurrounding Ġwords , Ġsentences Ġin Ġwhich Ġthey Ġappear Ġor Ġthe Ġdocuments . ĠFor Ġscientific Ġsummar ization , Ġwe Ġconsider Ġthe Ġcontext Ġof Ġthe Ġwords Ġas Ġthe Ġscientific Ġarticles Ġin Ġwhich Ġthey Ġappear . ĠThus , Ġif Ġtwo Ġconcepts Ġappear Ġin Ġidentical Ġset Ġof Ġarticles , Ġthey Ġare Ġsem antically Ġrelated . ĠWe Ġconsider Ġthe Ġtwo Ġsumm aries Ġas Ġsimilar Ġif Ġthey Ġrefer Ġto Ġsame Ġset Ġof Ġarticles Ġeven Ġif Ġthe Ġtwo Ġsumm aries Ġdo Ġnot Ġhave Ġhigh Ġlex ical Ġoverl aps . ĠTo Ġcapture Ġif Ġa Ġsummary Ġrelates Ġto Ġa Ġarticle , Ġwe Ġuse Ġinformation Ġretrieval Ġby Ġconsidering Ġthe Ġsumm aries Ġas Ġqueries Ġand Ġthe Ġarticles Ġas Ġdocuments Ġand Ġwe Ġrank Ġthe Ġarticles Ġbased Ġon Ġtheir Ġrelated ness Ġto Ġa Ġgiven Ġsummary . ĠFor Ġa Ġgiven Ġpair Ġof Ġsystem Ġsummary Ġand Ġthe Ġgold Ġsummary , Ġsimilar Ġrankings Ġof Ġthe Ġretrieved Ġarticles Ġsuggest Ġthat Ġthe Ġsumm aries Ġare Ġsem antically Ġrelated , Ġand Ġthus Ġthe Ġsystem Ġsummary Ġis Ġof Ġhigher Ġquality . </s> Based Ġon Ġthe Ġdomain Ġof Ġinterest , Ġwe Ġfirst Ġconstruct Ġan Ġindex Ġfrom Ġa Ġset Ġof Ġarticles Ġin Ġthe Ġsame Ġdomain . ĠSince ĠT AC Ġ2014 Ġwas Ġfocused Ġon Ġsummar ization Ġin Ġthe Ġbiomedical Ġdomain , Ġour Ġindex Ġalso Ġcomprises Ġof Ġbiomedical Ġarticles . ĠGiven Ġa Ġcandidate Ġsummary ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġgold Ġsumm aries ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġ; ĠIN LINE FORM 3 Ġis Ġthe Ġtotal Ġnumber Ġof Ġhuman Ġsumm aries ), Ġwe Ġsubmit Ġthe Ġcandidate Ġsummary Ġand Ġgold Ġsumm aries Ġto Ġthe Ġsearch Ġengine Ġas Ġqueries Ġand Ġcompare Ġtheir Ġranked Ġresults . ĠLet ĠIN LINE FORM 4 Ġbe Ġthe Ġentire Ġindex Ġwhich Ġcomprises Ġof ĠIN LINE FORM 5 Ġtotal Ġdocuments . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġranked Ġlist Ġof Ġretrieved Ġdocuments Ġfor Ġcandidate Ġsummary ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġthe Ġranked Ġlist Ġof Ġresults Ġfor Ġthe Ġgold Ġsummary ĠIN LINE FORM 3 Ġ. ĠThese Ġlists Ġof Ġresults Ġare Ġbased Ġon Ġa Ġrank Ġcut - off Ġpoint ĠIN LINE FORM 4 Ġthat Ġis Ġa Ġparameter Ġof Ġthe Ġsystem . ĠWe Ġprovide Ġevaluation Ġresults Ġon Ġdifferent Ġchoices Ġof Ġcut - off Ġpoint ĠIN LINE FORM 5 Ġin Ġthe ĠSection ĠSEC REF 5 ĠWe Ġconsider Ġthe Ġfollowing Ġtwo Ġscores : Ġ( i ) Ġsimple Ġintersection Ġand Ġ( ii ) Ġdiscounted Ġintersection Ġby Ġrankings . ĠThe Ġsimple Ġintersection Ġjust Ġconsiders Ġthe Ġoverl aps Ġof Ġthe Ġresults Ġin Ġthe Ġtwo Ġranked Ġlists Ġand Ġignores Ġthe Ġrankings . ĠThe Ġdiscounted Ġranked Ġscores , Ġon Ġthe Ġother Ġhand , Ġpenal izes Ġranking Ġdifferences Ġbetween Ġthe Ġtwo Ġresult Ġsets . ĠAs Ġan Ġexample Ġconsider Ġthe Ġfollowing Ġlist Ġof Ġretrieved Ġdocuments Ġ( den oted Ġby ĠIN LINE FORM 6 Ġs ) Ġfor Ġa Ġcandidate Ġand Ġa Ġgold Ġsummary Ġas Ġqueries : </s> Results Ġfor Ġcandidate Ġsummary : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 </s> Results Ġfor Ġgold Ġsummary : ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 </s> These Ġtwo Ġsets Ġof Ġresults Ġconsist Ġof Ġidentical Ġdocuments Ġbut Ġthe Ġranking Ġof Ġthe Ġretrieved Ġdocuments Ġdiffer . ĠTherefore , Ġthe Ġsimple Ġintersection Ġmethod Ġassigns Ġa Ġscore Ġof Ġ1 . 0 Ġwhile Ġin Ġthe Ġdiscounted Ġranked Ġscore , Ġthe Ġscore Ġwill Ġbe Ġless Ġthan Ġ1 . 0 Ġ( due Ġto Ġranking Ġdifferences Ġbetween Ġthe Ġresult Ġlists ). </s> We Ġnow Ġdefine Ġthe Ġmetrics Ġmore Ġprecisely . ĠUsing Ġthe Ġabove Ġnot ations , Ġwithout Ġloss Ġof Ġgener ality , Ġwe Ġassume Ġthat ĠIN LINE FORM 0 Ġ. ĠSer a Ġis Ġdefined Ġas Ġfollows : ĠIN LINE FORM 1 </s> To Ġalso Ġaccount Ġfor Ġthe Ġranked Ġposition Ġdifferences , Ġwe Ġmodify Ġthis Ġscore Ġto Ġdiscount Ġrewards Ġbased Ġon Ġrank Ġdifferences . ĠThat Ġis , Ġin Ġideal Ġscore , Ġwe Ġwant Ġsearch Ġresults Ġfrom Ġcandidate Ġsummary Ġ( ĠIN LINE FORM 0 Ġ) Ġto Ġbe Ġthe Ġsame Ġas Ġresults Ġfor Ġgold - standard Ġsumm aries Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġthe Ġrankings Ġof Ġthe Ġresults Ġalso Ġbe Ġthe Ġsame . ĠIf Ġthe Ġrankings Ġdiffer , Ġwe Ġdiscount Ġthe Ġreward Ġby Ġlog Ġof Ġthe Ġdifferences Ġof Ġthe Ġranks . ĠMore Ġspecifically , Ġthe Ġdiscounted Ġscore Ġ( Ser a - Dis ) Ġis Ġdefined Ġas : ĠIN LINE FORM 2 </s> where , Ġas Ġpreviously Ġdefined , ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtotal Ġnumber Ġof Ġhuman Ġgold Ġsumm aries , Ġresult Ġlist Ġfor Ġthe Ġcandidate Ġsummary Ġand Ġresult Ġlist Ġfor Ġthe Ġhuman Ġgold Ġsummary , Ġrespectively . ĠIn Ġaddition , ĠIN LINE FORM 3 Ġshows Ġthe ĠIN LINE FORM 4 Ġth Ġresults Ġin Ġthe Ġranked Ġlist ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġis Ġthe Ġmaximum Ġattain able Ġscore Ġused Ġas Ġthe Ġnormal izing Ġfactor . </s> We Ġuse Ġelastic search [ 1 ], Ġan Ġopen - source Ġsearch Ġengine , Ġfor Ġindex ing Ġand Ġquer ying Ġthe Ġarticles . ĠFor Ġretrieval Ġmodel , Ġwe Ġuse Ġthe ĠLanguage ĠModel ing Ġretrieval Ġmodel Ġwith ĠDir ich let Ġsmoot hing ĠB IB REF 6 Ġ. ĠSince ĠT AC Ġ2014 Ġbenchmark Ġis Ġon Ġsummar ization Ġof Ġbiomedical Ġarticles , Ġthe Ġappropriate Ġindex Ġwould Ġbe Ġthe Ġone Ġconstructed Ġfrom Ġarticles Ġin Ġthe Ġsame Ġdomain . ĠTherefore , Ġwe Ġuse Ġthe Ġopen Ġaccess Ġsubset Ġof ĠPub med [ 2 ] Ġwhich Ġconsists Ġof Ġpublished Ġarticles Ġin Ġbiomedical Ġliterature . </s> [ 1 ] https :// github . com / el astic / el astic search Ġ[ 2 ] Pub Med Ġis Ġa Ġcomprehensive Ġresource Ġof Ġarticles Ġand Ġabstract s Ġpublished Ġin Ġlife Ġsciences Ġand Ġbiomedical Ġliterature Ġhttp :// www . nc bi . nl m . nih . gov / pm c / </s> We Ġalso Ġexperiment Ġwith Ġdifferent Ġquery Ġ( re ) form ulation Ġapproaches . ĠQuery Ġreform ulation Ġis Ġa Ġmethod Ġin ĠInformation ĠRet ri eval Ġthat Ġaims Ġto Ġrefine Ġthe Ġquery Ġfor Ġbetter Ġretrieval Ġof Ġresults . ĠQuery Ġreform ulation Ġmethods Ġoften Ġconsist Ġof Ġremoving Ġineffective Ġterms Ġand Ġexpressions Ġfrom Ġthe Ġquery Ġ( query Ġreduction ) Ġor Ġadding Ġterms Ġto Ġthe Ġquery Ġthat Ġhelp Ġthe Ġretrieval Ġ( query Ġexpansion ). ĠQuery Ġreduction Ġis Ġspecially Ġimportant Ġwhen Ġqueries Ġare Ġverb ose . ĠSince Ġwe Ġuse Ġthe Ġsumm aries Ġas Ġqueries , Ġthe Ġqueries Ġare Ġusually Ġlong Ġand Ġtherefore Ġwe Ġconsider Ġquery Ġreductions . </s> In Ġour Ġexperiments , Ġthe Ġquery Ġreform ulation Ġis Ġdone Ġby Ġ3 Ġdifferent Ġways : Ġ( i ) ĠPlain : ĠThe Ġentire Ġsummary Ġwithout Ġstop words Ġand Ġnumeric Ġvalues ; Ġ( ii ) ĠN oun ĠPh r ases Ġ( NP ): ĠWe Ġonly Ġkeep Ġthe Ġnoun Ġphrases Ġas Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġand Ġeliminate Ġall Ġother Ġterms ; Ġand Ġ( iii ) ĠKey words Ġ( K W ): ĠWe Ġonly Ġkeep Ġthe Ġkeywords Ġand Ġkey Ġphrases Ġin Ġthe Ġsummary . ĠFor Ġextracting Ġthe Ġkeywords Ġand Ġkey ph r ases Ġ( with Ġlength Ġof Ġup Ġto Ġ3 Ġterms ), Ġwe Ġextract Ġexpressions Ġwhose Ġid f [ 1 ] Ġvalues Ġis Ġhigher Ġthan Ġa Ġpred efined Ġthreshold Ġthat Ġis Ġset Ġas Ġa Ġparameter . ĠWe Ġset Ġthis Ġthreshold Ġto Ġthe Ġaverage Ġid f Ġvalues Ġof Ġall Ġterms Ġexcept Ġstop words . Ġid f Ġvalues Ġare Ġcalculated Ġon Ġthe Ġsame Ġindex Ġthat Ġis Ġused Ġfor Ġthe Ġretrieval . </s> [ 1 ] In verted ĠDocument ĠFrequency </s> We Ġhypothes ize Ġthat Ġusing Ġonly Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġprevents Ġquery Ġdrift Ġand Ġleads Ġto Ġretrieval Ġof Ġmore Ġrelevant Ġdocuments . ĠN oun Ġphrases Ġand Ġkeywords Ġare Ġtwo Ġhe ur istics Ġfor Ġidentifying Ġthe Ġinformative Ġconcepts . </s> Data </s> To Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġonly Ġscientific Ġsummar ization Ġbenchmark Ġis Ġfrom ĠT AC Ġ2014 Ġsummar ization Ġtrack . ĠFor Ġevaluating Ġthe Ġeffectiveness Ġof ĠRouge Ġvariants Ġand Ġour Ġmetric Ġ( Ser a ), Ġwe Ġuse Ġthis Ġbenchmark , Ġwhich Ġconsists Ġof Ġ20 Ġtopics Ġeach Ġwith Ġa Ġbiomedical Ġjournal Ġarticle Ġand Ġ4 Ġgold Ġhuman Ġwritten Ġsumm aries . </s> An notations </s> In Ġthe ĠT AC Ġ2014 Ġsummar ization Ġtrack , ĠRouge Ġwas Ġsuggested Ġas Ġthe Ġevaluation Ġmetric Ġfor Ġsummar ization Ġand Ġno Ġhuman Ġassessment Ġwas Ġprovided Ġfor Ġthe Ġtopics . ĠTherefore , Ġto Ġstudy Ġthe Ġeffectiveness Ġof Ġthe Ġevaluation Ġmetrics , Ġwe Ġuse Ġthe Ġsemi - man ual ĠPyramid Ġevaluation Ġframework ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIn Ġthe Ġpyramid Ġscoring , Ġthe Ġcontent Ġunits Ġin Ġthe Ġgold Ġhuman Ġwritten Ġsumm aries Ġare Ġorganized Ġin Ġa Ġpyramid . ĠIn Ġthis Ġpyramid , Ġthe Ġcontent Ġunits Ġare Ġorganized Ġin Ġtiers Ġand Ġhigher Ġtiers Ġof Ġthe Ġpyramid Ġindicate Ġhigher Ġimportance . ĠThe Ġcontent Ġquality Ġof Ġa Ġgiven Ġcandidate Ġsummary Ġis Ġevaluated Ġwith Ġrespect Ġto Ġthis Ġpyramid . </s> To Ġanalyze Ġthe Ġquality Ġof Ġthe Ġevaluation Ġmetrics , Ġfollowing Ġthe Ġpyramid Ġframework , Ġwe Ġdesign Ġan Ġannotation Ġscheme Ġthat Ġis Ġbased Ġon Ġidentification Ġof Ġimportant Ġcontent Ġunits . ĠConsider Ġthe Ġfollowing Ġexample : </s> End ogeneous Ġsmall ĠRN As Ġ( mi RNA ) Ġwere Ġgenetically Ġscreened Ġand Ġstudied Ġto Ġfind Ġthe Ġmi RN As Ġwhich Ġare Ġrelated Ġto Ġtumor igen esis . </s> In Ġthe Ġabove Ġexample , Ġthe Ġunder lined Ġexpressions Ġare Ġthe Ġcontent Ġunits Ġthat Ġconvey Ġthe Ġmain Ġmeaning Ġof Ġthe Ġtext . ĠWe Ġcall Ġthese Ġsmall Ġunits , Ġn uggets Ġwhich Ġare Ġphrases Ġor Ġconcepts Ġthat Ġare Ġthe Ġmain Ġcontributors Ġto Ġthe Ġcontent Ġquality Ġof Ġthe Ġsummary . </s> We Ġasked Ġtwo Ġhuman Ġannot ators Ġto Ġreview Ġthe Ġgold Ġsumm aries Ġand Ġextract Ġcontent Ġunits Ġin Ġthese Ġsumm aries . ĠThe Ġpyramid Ġtiers Ġrepresent Ġthe Ġoccurrences Ġof Ġn uggets Ġacross Ġall Ġthe Ġhuman Ġwritten Ġgold - standard Ġsumm aries , Ġand Ġtherefore Ġthe Ġn uggets Ġare Ġweighted Ġbased Ġon Ġthese Ġtiers . ĠThe Ġintuition Ġis Ġthat , Ġif Ġa Ġn ug get Ġoccurs Ġmore Ġfrequently Ġin Ġthe Ġhuman Ġsumm aries , Ġit Ġis Ġa Ġmore Ġimportant Ġcontributor Ġ( thus Ġbelongs Ġto Ġhigher Ġtier Ġin Ġthe Ġpyramid ). ĠThus , Ġif Ġa Ġcandidate Ġsummary Ġcontains Ġthis Ġn ug get , Ġit Ġshould Ġbe Ġrewarded Ġmore . ĠAn Ġexample Ġof Ġthe Ġn uggets Ġannotations Ġin Ġpyramid Ġframework Ġis Ġshown Ġin ĠTable ĠT AB REF 12 Ġ. ĠIn Ġthis Ġexample , Ġthe Ġn ug get ĠâĢ ľ cell Ġmutation âĢ Ŀ Ġbelongs Ġto Ġthe Ġ4 th Ġtier Ġand Ġit Ġsuggests Ġthat Ġthe ĠâĢ ľ cell Ġmutation âĢ Ŀ Ġn ug get Ġis Ġa Ġvery Ġimportant Ġrepresentative Ġof Ġthe Ġcontent Ġof Ġthe Ġcorresponding Ġdocument . </s> Let ĠIN LINE FORM 0 Ġdefine Ġthe Ġtiers Ġof Ġthe Ġpyramid Ġwith ĠIN LINE FORM 1 Ġbeing Ġthe Ġbottom Ġtier Ġand ĠIN LINE FORM 2 Ġthe Ġtop Ġtier . ĠLet ĠIN LINE FORM 3 Ġbe Ġthe Ġnumber Ġof Ġthe Ġn uggets Ġin Ġthe Ġcandidate Ġsummary Ġthat Ġappear Ġin Ġthe Ġtier ĠIN LINE FORM 4 Ġ. ĠThen Ġthe Ġpyramid Ġscore ĠIN LINE FORM 5 Ġof Ġthe Ġcandidate Ġsummary Ġwill Ġbe : ĠIN LINE FORM 6 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmaximum Ġattain able Ġscore Ġused Ġfor Ġnormal izing Ġthe Ġscores : ĠIN LINE FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġtotal Ġnumber Ġof Ġn uggets Ġin Ġthe Ġsummary Ġand ĠIN LINE FORM 1 Ġ. </s> We Ġrelease Ġthe Ġpyramid Ġannotations Ġof Ġthe ĠT AC Ġ2014 Ġdataset Ġthrough Ġa Ġpublic Ġrepository [ 2 ]. </s> [ 2 ] https :// github . com / ac ohan / T AC - py ramid - An notations </s> 3 . 1 pt </s> Sum mar ization Ġapproaches </s> We Ġstudy Ġthe Ġeffectiveness Ġof ĠRouge Ġand Ġour Ġproposed Ġmethod Ġ( Ser a ) Ġby Ġanalyzing Ġthe Ġcorrelations Ġwith Ġsemi - man ual Ġhuman Ġjudgments . ĠVery Ġfew Ġteams Ġparticipated Ġin ĠT AC Ġ2014 Ġsummar ization Ġtrack Ġand Ġthe Ġofficial Ġresults Ġand Ġthe Ġreview Ġpaper Ġof ĠT AC Ġ2014 Ġsystems Ġwere Ġnever Ġpublished . ĠTherefore , Ġto Ġevaluate Ġthe Ġeffectiveness Ġof ĠRouge , Ġwe Ġapplied Ġ9 Ġwell - known Ġsummar ization Ġapproaches Ġon Ġthe ĠT AC Ġ2014 Ġscientific Ġsummar ization Ġdataset . ĠOb tained ĠRouge Ġand ĠSer a Ġresults Ġof Ġeach Ġof Ġthese Ġapproaches Ġare Ġthen Ġcorrelated Ġwith Ġsemi - man ual Ġhuman Ġjudgments . ĠIn Ġthe Ġfollowing , Ġwe Ġbriefly Ġdescribe Ġeach Ġof Ġthese Ġsummar ization Ġapproaches . </s> Lex Rank ĠB IB REF 9 Ġ: ĠLex Rank Ġfinds Ġthe Ġmost Ġimportant Ġ( central ) Ġsentences Ġin Ġa Ġdocument Ġby Ġusing Ġrandom Ġwalks Ġin Ġa Ġgraph Ġconstructed Ġfrom Ġthe Ġdocument Ġsentences . ĠIn Ġthis Ġgraph , Ġthe Ġsentences Ġare Ġnodes Ġand Ġthe Ġsimilarity Ġbetween Ġthe Ġsentences Ġdetermines Ġthe Ġedges . ĠSent ences Ġare Ġranked Ġaccording Ġto Ġtheir Ġimportance . ĠImport ance Ġis Ġmeasured Ġin Ġterms Ġof Ġcentral ity Ġof Ġthe Ġsentence ĠâĢĶ Ġthe Ġtotal Ġnumber Ġof Ġedges Ġincident Ġon Ġthe Ġnode Ġ( sent ence ) Ġin Ġthe Ġgraph . ĠThe Ġintuition Ġbehind ĠLex Rank Ġis Ġthat Ġa Ġdocument Ġcan Ġbe Ġsummarized Ġusing Ġthe Ġmost Ġcentral Ġsentences Ġin Ġthe Ġdocument Ġthat Ġcapture Ġits Ġmain Ġaspects . </s> Lat ent ĠSem antic ĠAnalysis Ġ( L SA ) Ġbased Ġsummar ization ĠB IB REF 10 Ġ: ĠIn Ġthis Ġsummar ization Ġmethod , ĠSing ular ĠValue ĠDec om position Ġ( S VD ) ĠB IB REF 11 Ġis Ġused Ġfor Ġder iving Ġlatent Ġsemantic Ġstructure Ġof Ġthe Ġdocument . ĠThe Ġdocument Ġis Ġdivided Ġinto Ġsentences Ġand Ġa Ġterm - sent ence Ġmatrix ĠIN LINE FORM 0 Ġis Ġconstructed . ĠThe Ġmatrix ĠIN LINE FORM 1 Ġis Ġthen Ġdecomp osed Ġinto Ġa Ġnumber Ġof Ġlin early - independent Ġsingular Ġvectors Ġwhich Ġrepresent Ġthe Ġlatent Ġconcepts Ġin Ġthe Ġdocument . ĠThis Ġmethod , Ġintuitive ly , Ġdecom poses Ġthe Ġdocument Ġinto Ġseveral Ġlatent Ġtopics Ġand Ġthen Ġselects Ġthe Ġmost Ġrepresentative Ġsentences Ġfor Ġeach Ġof Ġthese Ġtopics Ġas Ġthe Ġsummary Ġof Ġthe Ġdocument . </s> Max imal ĠMarg inal ĠRe lev ance Ġ( MM R ) ĠB IB REF 12 Ġ: ĠMax imal ĠMarg inal ĠRe lev ance Ġ( MM R ) Ġis Ġa Ġgreedy Ġstrategy Ġfor Ġselecting Ġsentences Ġfor Ġthe Ġsummary . ĠSent ences Ġare Ġadded Ġiter atively Ġto Ġthe Ġsummary Ġbased Ġon Ġtheir Ġrelated ness Ġto Ġthe Ġdocument Ġas Ġwell Ġas Ġtheir Ġnovelty Ġwith Ġrespect Ġto Ġthe Ġcurrent Ġsummary . </s> C itation Ġbased Ġsummar ization ĠB IB REF 13 Ġ: ĠIn Ġthis Ġmethod , Ġcitations Ġare Ġused Ġfor Ġsummar izing Ġan Ġarticle . ĠUsing Ġthe ĠLex Rank Ġalgorithm Ġon Ġthe Ġcitation Ġnetwork Ġof Ġthe Ġarticle , Ġtop Ġsentences Ġare Ġselected Ġfor Ġthe Ġfinal Ġsummary . </s> Using Ġfrequency Ġof Ġthe Ġwords ĠB IB REF 14 Ġ: ĠIn Ġthis Ġmethod , Ġwhich Ġis Ġone Ġthe Ġearliest Ġworks Ġin Ġtext Ġsummar ization , Ġraw Ġword Ġfrequencies Ġare Ġused Ġto Ġestimate Ġthe Ġsal iency Ġof Ġsentences Ġin Ġthe Ġdocument . ĠThe Ġmost Ġsalient Ġsentences Ġare Ġchosen Ġfor Ġthe Ġfinal Ġsummary . </s> Sum Basic ĠB IB REF 15 Ġ: ĠSum Basic Ġis Ġan Ġapproach Ġthat Ġweights Ġsentences Ġbased Ġon Ġthe Ġdistribution Ġof Ġwords Ġthat Ġis Ġderived Ġfrom Ġthe Ġdocument . ĠSent ence Ġselection Ġis Ġapplied Ġiter atively Ġby Ġselecting Ġwords Ġwith Ġhighest Ġprobability Ġand Ġthen Ġfinding Ġthe Ġhighest Ġscoring Ġsentence Ġthat Ġcontains Ġthat Ġword . ĠThe Ġword Ġweights Ġare Ġupdated Ġafter Ġeach Ġiteration Ġto Ġprevent Ġselection Ġof Ġsimilar Ġsentences . </s> Sum mar ization Ġusing Ġcitation - context Ġand Ġdiscourse Ġstructure ĠB IB REF 16 Ġ: ĠIn Ġthis Ġmethod , Ġthe Ġset Ġof Ġcitations Ġto Ġthe Ġarticle Ġare Ġused Ġto Ġfind Ġthe Ġarticle Ġsentences Ġthat Ġdirectly Ġreflect Ġthose Ġcitations Ġ( c itation - context s ). ĠIn Ġaddition , Ġthe Ġscientific Ġdiscourse Ġof Ġthe Ġarticle Ġis Ġutilized Ġto Ġcapture Ġdifferent Ġaspects Ġof Ġthe Ġarticle . ĠThe Ġscientific Ġdiscourse Ġusually Ġfollows Ġa Ġstructure Ġin Ġwhich Ġthe Ġauthors Ġfirst Ġdescribe Ġtheir Ġhypothesis , Ġthen Ġthe Ġmethods , Ġexperiment , Ġresults Ġand Ġimplications . ĠSent ence Ġselection Ġis Ġbased Ġon Ġfinding Ġthe Ġmost Ġimportant Ġsentences Ġin Ġeach Ġof Ġthe Ġdiscourse Ġfacets Ġof Ġthe Ġdocument Ġusing Ġthe ĠMMR Ġhe uristic . </s> K L ĠDiver gence ĠB IB REF 17 ĠIn Ġthis Ġmethod , Ġthe Ġdocument Ġun ig ram Ġdistribution ĠIN LINE FORM 0 Ġand Ġthe Ġsummary Ġun ig ram Ġdistribut ation ĠIN LINE FORM 1 Ġare Ġconsidered ; Ġthe Ġgoal Ġis Ġto Ġfind Ġa Ġsummary Ġwhose Ġdistribution Ġis Ġvery Ġclose Ġto Ġthe Ġdocument Ġdistribution . ĠThe Ġdifference Ġof Ġthe Ġdistributions Ġis Ġcaptured Ġby Ġthe ĠK ull back - Lie ber Ġ( K L ) Ġdivergence , Ġden oted Ġby ĠIN LINE FORM 2 Ġ. </s> Sum mar ization Ġbased Ġon ĠTopic ĠModels ĠB IB REF 17 Ġ: ĠInstead Ġof Ġusing Ġun ig ram Ġdistributions Ġfor Ġmodeling Ġthe Ġcontent Ġdistribution Ġof Ġthe Ġdocument Ġand Ġthe Ġsummary , Ġthis Ġmethod Ġmodels Ġthe Ġdocument Ġcontent Ġusing Ġan ĠL DA Ġbased Ġtopic Ġmodel ĠB IB REF 18 Ġ. ĠIt Ġthen Ġuses Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġdocument Ġand Ġthe Ġsummary Ġcontent Ġmodels Ġfor Ġselecting Ġsentences Ġfor Ġthe Ġsummary . </s> Results Ġand ĠDiscussion </s> We Ġcalculated Ġall Ġvariants Ġof ĠRouge Ġscores , Ġour Ġproposed Ġmetric , ĠSer a , Ġand Ġthe ĠPyramid Ġscore Ġon Ġthe Ġgenerated Ġsumm aries Ġfrom Ġthe Ġsummar izers Ġdescribed Ġin ĠSection ĠSEC REF 13 Ġ. ĠWe Ġdo Ġnot Ġreport Ġthe ĠRouge , ĠSer a Ġor Ġpyramid Ġscores Ġof Ġindividual Ġsystems Ġas Ġit Ġis Ġnot Ġthe Ġfocus Ġof Ġthis Ġstudy . ĠOur Ġaim Ġis Ġto Ġanalyze Ġthe Ġeffectiveness Ġof Ġthe Ġevaluation Ġmetrics , Ġnot Ġthe Ġsummar ization Ġapproaches . ĠTherefore , Ġwe Ġconsider Ġthe Ġcorrelations Ġof Ġthe Ġautomatic Ġevaluation Ġmetrics Ġwith Ġthe Ġmanual ĠPyramid Ġscores Ġto Ġevaluate Ġtheir Ġeffectiveness ; Ġthe Ġmetrics Ġthat Ġshow Ġhigher Ġcorrelations Ġwith Ġmanual Ġjudgments Ġare Ġmore Ġeffective . </s> Table ĠT AB REF 23 Ġshows Ġthe ĠPearson , ĠSpear man Ġand ĠKendall Ġcorrelation Ġof ĠRouge Ġand ĠSer a , Ġwith Ġpyramid Ġscores . ĠBoth ĠRouge Ġand ĠSer a Ġare Ġcalculated Ġwith Ġstop words Ġremoved Ġand Ġwith Ġstemming . ĠOur Ġexperiments Ġwith Ġinclusion Ġof Ġstop words Ġand Ġwithout Ġstemming Ġshowed Ġsimilar Ġresults Ġand Ġthus , Ġwe Ġdo Ġnot Ġinclude Ġthose Ġto Ġavoid Ġredundancy . </s> Ser a </s> The Ġresults Ġof Ġour Ġproposed Ġmethod Ġ( Ser a ) Ġare Ġshown Ġin Ġthe Ġbottom Ġpart Ġof ĠTable ĠT AB REF 23 Ġ. ĠIn Ġgeneral , ĠSer a Ġshows Ġbetter Ġcorrelation Ġwith Ġpyramid Ġscores Ġin Ġcomparison Ġwith ĠRouge . ĠWe Ġobserve Ġthat Ġthe ĠPearson Ġcorrelation Ġof ĠSer a Ġwith Ġcut - off Ġpoint Ġof Ġ5 Ġ( shown Ġby ĠSer a - 5 ) Ġis Ġ0 . 8 23 Ġwhich Ġis Ġhigher Ġthan Ġmost Ġof Ġthe ĠRouge Ġvariants . ĠSimilarly , Ġthe ĠSpear man Ġand ĠKendall Ġcorrelations Ġof Ġthe ĠSer a Ġevaluation Ġscore Ġis Ġ0 . 9 41 Ġand Ġ0 . 8 57 Ġrespectively , Ġwhich Ġare Ġhigher Ġthan Ġall ĠRouge Ġcorrelation Ġvalues . ĠThis Ġshows Ġthe Ġeffectiveness Ġof Ġthe Ġsimple Ġvariant Ġof Ġour Ġproposed Ġsummar ization Ġevaluation Ġmetric . </s> Table ĠT AB REF 23 Ġalso Ġshows Ġthe Ġresults Ġof Ġother ĠSer a Ġvariants Ġincluding Ġdiscount ing Ġand Ġquery Ġreform ulation Ġmethods . ĠSome Ġof Ġthese Ġvariants Ġare Ġthe Ġresult Ġof Ġapplying Ġquery Ġreform ulation Ġin Ġthe Ġprocess Ġof Ġdocument Ġretrieval Ġwhich Ġare Ġdescribed Ġin Ġsection ĠSEC REF 3 ĠAs Ġillustrated , Ġthe ĠN oun ĠPh r ases Ġ( NP ) Ġquery Ġreform ulation Ġat Ġcut - off Ġpoint Ġof Ġ5 Ġ( shown Ġas ĠSer a - np - 5 ) Ġachieves Ġthe Ġhighest Ġcorrelations Ġamong Ġall Ġthe ĠSer a Ġvariants Ġ( ĠIN LINE FORM 0 Ġ= ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ= ĠIN LINE FORM 3 Ġ= ĠIN LINE FORM 4 Ġ). ĠIn Ġthe Ġcase Ġof ĠKey words Ġ( K W ) Ġquery Ġreform ulation , Ġwithout Ġusing Ġdiscount ing , Ġwe Ġcan Ġsee Ġthat Ġthere Ġis Ġno Ġpositive Ġgain Ġin Ġcorrelation . ĠHowever , Ġkeywords Ġwhen Ġapplied Ġon Ġthe Ġdiscounted Ġvariant Ġof ĠSer a , Ġresult Ġin Ġhigher Ġcorrelations . </s> Disc ount ing Ġhas Ġmore Ġpositive Ġeffect Ġwhen Ġapplied Ġon Ġquery Ġreform ulation - based ĠSer a Ġthan Ġon Ġthe Ġsimple Ġvariant Ġof ĠSer a . ĠIn Ġthe Ġcase Ġof Ġdiscount ing Ġand ĠNP Ġquery Ġreform ulation Ġ( Ser a - dis - np ), Ġwe Ġobserve Ġhigher Ġcorrelations Ġin Ġcomparison Ġwith Ġsimple ĠSer a . ĠSimilarly , Ġin Ġthe Ġcase Ġof ĠKey words Ġ( K W ), Ġpositive Ġcorrelation Ġgain Ġis Ġobtained Ġin Ġmost Ġof Ġcorrelation Ġcoefficients . ĠNP Ġwithout Ġdiscount ing Ġand Ġat Ġcut - off Ġpoint Ġof Ġ5 Ġ( Ser a - np - 5 ) Ġshows Ġthe Ġhighest Ġnon - param etric Ġcorrelation . ĠIn Ġaddition , Ġthe Ġdiscounted ĠNP Ġat Ġcut - off Ġpoint Ġof Ġ10 Ġ( Ser a - np - dis - 10 ) Ġshows Ġthe Ġhighest Ġparam etric Ġcorrelations . </s> In Ġgeneral , Ġusing ĠNP Ġand ĠK W Ġas Ġhe ur istics Ġfor Ġfinding Ġthe Ġinformative Ġconcepts Ġin Ġthe Ġsummary Ġeffectively Ġincreases Ġthe Ġcorrelations Ġwith Ġthe Ġmanual Ġscores . ĠSelect ing Ġinformative Ġterms Ġfrom Ġlong Ġqueries Ġresults Ġin Ġmore Ġrelevant Ġdocuments Ġand Ġprevents Ġquery Ġdrift . ĠTherefore , Ġthe Ġoverall Ġsimilarity Ġbetween Ġthe Ġtwo Ġsumm aries Ġ( cand idate Ġand Ġthe Ġhuman Ġwritten Ġgold Ġsummary ) Ġis Ġbetter Ġcaptured . </s> R ou ge </s> Another Ġimportant Ġobservation Ġis Ġregarding Ġthe Ġeffectiveness Ġof ĠRouge Ġscores Ġ( top Ġpart Ġof ĠTable ĠT AB REF 23 Ġ). ĠInterestingly , Ġwe Ġobserve Ġthat Ġmany Ġvariants Ġof ĠRouge Ġscores Ġdo Ġnot Ġhave Ġhigh Ġcorrelations Ġwith Ġhuman Ġpyramid Ġscores . ĠThe Ġlowest ĠF - score Ġcorrelations Ġare Ġfor ĠRouge - 1 Ġand ĠRouge - L Ġ( with ĠIN LINE FORM 0 Ġ= 0 . 454 ). ĠWeak Ġcorrelation Ġof ĠRouge - 1 Ġshows Ġthat Ġmatching Ġun ig rams Ġbetween Ġthe Ġcandidate Ġsummary Ġand Ġgold Ġsumm aries Ġis Ġnot Ġaccurate Ġin Ġquant ifying Ġthe Ġquality Ġof Ġthe Ġsummary . ĠOn Ġhigher Ġorder Ġn - gram s , Ġhowever , Ġwe Ġcan Ġsee Ġthat ĠRouge Ġcorrelates Ġbetter Ġwith Ġpyramid . ĠIn Ġfact , Ġthe Ġhighest Ġoverall ĠIN LINE FORM 1 Ġis Ġobtained Ġby ĠRouge - 3 . ĠRouge - L Ġand Ġits Ġweighted Ġversion ĠRouge - W , Ġboth Ġhave Ġweak Ġcorrelations Ġwith Ġpyramid . ĠSkip - big rams Ġ( R ou ge - S ) Ġand Ġits Ġcombination Ġwith Ġun ig rams Ġ( R ou ge - SU ) Ġalso Ġshow Ġsub - opt imal Ġcorrelations . ĠNote Ġthat ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġcorrelations Ġare Ġmore Ġreliable Ġin Ġour Ġsetup Ġdue Ġto Ġthe Ġsmall Ġsample Ġsize . </s> These Ġresults Ġconfirm Ġour Ġinitial Ġhypothesis Ġthat ĠRouge Ġis Ġnot Ġaccurate Ġestim ator Ġof Ġthe Ġquality Ġof Ġthe Ġsummary Ġin Ġscientific Ġsummar ization . ĠWe Ġattribute Ġthis Ġto Ġthe Ġdifferences Ġof Ġscientific Ġsummar ization Ġwith Ġgeneral Ġdomain Ġsumm aries . ĠWhen Ġhumans Ġsummarize Ġa Ġrelatively Ġlong Ġresearch Ġpaper , Ġthey Ġmight Ġuse Ġdifferent Ġterminology Ġand Ġparaph r asing . ĠTherefore , ĠRouge Ġwhich Ġonly Ġrelies Ġon Ġterm Ġmatching Ġbetween Ġa Ġcandidate Ġand Ġa Ġgold Ġsummary , Ġis Ġnot Ġaccurate Ġin Ġquant ifying Ġthe Ġquality Ġof Ġthe Ġcandidate Ġsummary . </s> Cor relation Ġof ĠSer a Ġwith ĠRouge </s> Table ĠT AB REF 25 Ġshows Ġcorrelations Ġof Ġour Ġmetric ĠSer a Ġwith ĠRouge - 2 Ġand ĠRouge - 3 , Ġwhich Ġare Ġthe Ġhighest Ġcorrelated ĠRouge Ġvariants Ġwith Ġpyramid . ĠWe Ġcan Ġsee Ġthat Ġin Ġgeneral , Ġthe Ġcorrelation Ġis Ġnot Ġstrong . ĠKey word Ġbased Ġreduction Ġvariants Ġare Ġthe Ġonly Ġvariants Ġfor Ġwhich Ġthe Ġcorrelation Ġwith ĠRouge Ġis Ġhigh . ĠLooking Ġat Ġthe Ġcorrelations Ġof ĠK W Ġvariants Ġof ĠSer a Ġwith Ġpyramid Ġ( Table ĠT AB REF 23 Ġ, Ġbottom Ġpart ), Ġwe Ġobserve Ġthat Ġthese Ġvariants Ġare Ġalso Ġhighly Ġcorrelated Ġwith Ġmanual Ġevaluation . </s> Effect Ġof Ġthe Ġrank Ġcut - off Ġpoint </s> Finally , ĠFigure ĠFIG REF 28 Ġshows ĠIN LINE FORM 0 Ġcorrelation Ġof Ġdifferent Ġvariants Ġof ĠSer a Ġwith Ġpyramid Ġbased Ġon Ġselection Ġof Ġdifferent Ġcut - off Ġpoints Ġ( ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġcorrelations Ġresult Ġin Ġvery Ġsimilar Ġgraphs ). ĠWhen Ġthe Ġcut - off Ġpoint Ġincreases , Ġmore Ġdocuments Ġare Ġretrieved Ġfor Ġthe Ġcandidate Ġand Ġthe Ġgold Ġsumm aries , Ġand Ġtherefore Ġthe Ġfinal ĠSer a Ġscore Ġis Ġmore Ġfine - gr ained . ĠA Ġgeneral Ġobservation Ġis Ġthat Ġas Ġthe Ġsearch Ġcut - off Ġpoint Ġincreases , Ġthe Ġcorrelation Ġwith Ġpyramid Ġscores Ġdecreases . ĠThis Ġis Ġbecause Ġwhen Ġthe Ġretrieved Ġresult Ġlist Ġbecomes Ġlarger , Ġthe Ġprobability Ġof Ġincluding Ġless Ġrelated Ġdocuments Ġincreases Ġwhich Ġnegatively Ġaffects Ġcorrect Ġestimation Ġof Ġthe Ġsimilarity Ġof Ġthe Ġcandidate Ġand Ġgold Ġsumm aries . ĠThe Ġmost Ġaccurate Ġestim ations Ġare Ġfor Ġmetrics Ġwith Ġcut - off Ġpoints Ġof Ġ5 Ġand Ġ10 Ġwhich Ġare Ġincluded Ġin Ġthe Ġreported Ġresults Ġof Ġall Ġvariants Ġin ĠTable ĠT AB REF 23 Ġ. </s> Related Ġwork </s> R ou ge ĠB IB REF 1 Ġassess es Ġthe Ġcontent Ġquality Ġof Ġa Ġcandidate Ġsummary Ġwith Ġrespect Ġto Ġa Ġset Ġof Ġhuman Ġgold Ġsumm aries Ġbased Ġon Ġtheir Ġlex ical Ġoverl aps . ĠRouge Ġconsists Ġof Ġseveral Ġvariants . ĠSince Ġits Ġintroduction , ĠRouge Ġhas Ġbeen Ġone Ġof Ġthe Ġmost Ġwidely Ġreported Ġmetrics Ġin Ġthe Ġsummar ization Ġliterature , Ġand Ġits Ġhigh Ġadoption Ġhas Ġbeen Ġdue Ġto Ġits Ġhigh Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores Ġin ĠD UC Ġdatasets ĠB IB REF 1 Ġ. ĠHowever , Ġlater Ġresearch Ġhas Ġcast ed Ġdoubts Ġabout Ġthe Ġaccuracy Ġof ĠRouge Ġagainst Ġmanual Ġevaluations . Ġcon roy 2008 mind Ġanalyzed ĠD UC Ġ2005 Ġto Ġ2007 Ġdata Ġand Ġshowed Ġthat Ġwhile Ġsome Ġsystems Ġachieve Ġhigh ĠRouge Ġscores Ġwith Ġrespect Ġto Ġhuman Ġsumm aries , Ġthe Ġlinguistic Ġand Ġresponsiveness Ġscores Ġof Ġthose Ġsystems Ġdo Ġnot Ġcorrespond Ġto Ġthe Ġhigh ĠRouge Ġscores . </s> We Ġstudied Ġthe Ġeffectiveness Ġof ĠRouge Ġthrough Ġcorrelation Ġanalysis Ġwith Ġmanual Ġscores . ĠBesides Ġcorrelation Ġwith Ġhuman Ġassessment Ġscores , Ġother Ġapproaches Ġhave Ġbeen Ġexplored Ġfor Ġanalyzing Ġthe Ġeffectiveness Ġof Ġsummar ization Ġevaluation . ĠRan kel : 2011 Ġstudied Ġthe Ġextent Ġto Ġwhich Ġa Ġmetric Ġcan Ġdistinguish Ġbetween Ġthe Ġhuman Ġand Ġsystem Ġgenerated Ġsumm aries . ĠThey Ġalso Ġproposed Ġthe Ġuse Ġof Ġpaired Ġtwo - sample Ġt - tests Ġand Ġthe ĠWil co xon Ġsigned - rank Ġtest Ġas Ġan Ġalternative Ġto ĠRouge Ġin Ġevaluating Ġseveral Ġsummar izers . ĠSimilarly , Ġow cz ar z ak 2012 ass essment Ġproposed Ġthe Ġuse Ġof Ġmultiple Ġbinary Ġsignificance Ġtests Ġbetween Ġthe Ġsystem Ġsumm aries Ġfor Ġranking Ġthe Ġbest Ġsummar izers . </s> Since Ġintroduction Ġof ĠRouge , Ġthere Ġhave Ġbeen Ġother Ġefforts Ġfor Ġimproving Ġautomatic Ġsummar ization Ġevaluation . Ġh ovy 2006 aut om ated Ġproposed Ġan Ġapproach Ġbased Ġon Ġcomparison Ġof Ġso Ġcalled ĠBasic ĠElements Ġ( BE ) Ġbetween Ġthe Ġcandidate Ġand Ġreference Ġsumm aries . ĠBE s Ġwere Ġextracted Ġbased Ġon Ġsynt actic Ġstructure Ġof Ġthe Ġsentence . ĠThe Ġwork Ġby Ġcon roy 2011 n ou ve au Ġwas Ġanother Ġattempt Ġfor Ġimproving ĠRouge Ġfor Ġupdate Ġsummar ization Ġwhich Ġcombined Ġtwo Ġdifferent ĠRouge Ġvariants Ġand Ġshowed Ġhigher Ġcorrelations Ġwith Ġmanual Ġjudgments Ġfor ĠT AC Ġ2008 Ġupdate Ġsumm aries . </s> Apart Ġfrom Ġthe Ġcontent , Ġother Ġaspects Ġof Ġsummar ization Ġsuch Ġas Ġlinguistic Ġquality Ġhave Ġbeen Ġalso Ġstudied . Ġpit ler 2010 automatic Ġevaluated Ġa Ġset Ġof Ġmodels Ġbased Ġon Ġsynt actic Ġfeatures , Ġlanguage Ġmodels Ġand Ġentity Ġco he rences Ġfor Ġassessing Ġthe Ġlinguistic Ġquality Ġof Ġthe Ġsumm aries . ĠMachine Ġtranslation Ġevaluation Ġmetrics Ġsuch Ġas Ġblue Ġhave Ġalso Ġbeen Ġcompared Ġand Ġcontrasted Ġagainst ĠRouge ĠB IB REF 19 Ġ. ĠDespite Ġthese Ġworks , Ġwhen Ġgold - standard Ġsumm aries Ġare Ġavailable , ĠRouge Ġis Ġstill Ġthe Ġmost Ġcommon Ġevaluation Ġmetric Ġthat Ġis Ġused Ġin Ġthe Ġsummar ization Ġpublished Ġresearch . ĠApart Ġfrom ĠRouge 's Ġinitial Ġgood Ġresults Ġon Ġthe Ġnews wire Ġdata , Ġthe Ġavailability Ġof Ġthe Ġsoftware Ġand Ġits Ġefficient Ġperformance Ġhave Ġfurther Ġcontributed Ġto Ġits Ġpopularity . </s> Con clusions </s> We Ġprovided Ġan Ġanalysis Ġof Ġexisting Ġevaluation Ġmetrics Ġfor Ġscientific Ġsummar ization Ġwith Ġevaluation Ġof Ġall Ġvariants Ġof ĠRouge . ĠWe Ġshowed Ġthat ĠRouge Ġmay Ġnot Ġbe Ġthe Ġbest Ġmetric Ġfor Ġsummar ization Ġevaluation ; Ġespecially Ġin Ġsumm aries Ġwith Ġhigh Ġterminology Ġvariations Ġand Ġparaph r asing Ġ( e . g . Ġscientific Ġsumm aries ). ĠFurthermore , Ġwe Ġshowed Ġthat Ġdifferent Ġvariants Ġof ĠRouge Ġresult Ġin Ġdifferent Ġcorrelation Ġvalues Ġwith Ġhuman Ġjudgments , Ġindicating Ġthat Ġnot Ġall ĠRouge Ġscores Ġare Ġequally Ġeffective . ĠAmong Ġall Ġvariants Ġof ĠRouge , ĠRouge - 2 Ġand ĠRouge - 3 Ġare Ġbetter Ġcorrelated Ġwith Ġmanual Ġjudgments Ġin Ġthe Ġcontext Ġof Ġscientific Ġsummar ization . ĠWe Ġfurthermore Ġproposed Ġan Ġalternative Ġand Ġmore Ġeffective Ġapproach Ġfor Ġscientific Ġsummar ization Ġevaluation Ġ( Sum mar ization ĠEvaluation Ġby ĠRe lev ance ĠAnalysis Ġ- ĠSer a ). ĠResults Ġrevealed Ġthat Ġin Ġgeneral , Ġthe Ġproposed Ġevaluation Ġmetric Ġachieves Ġhigher Ġcorrelations Ġwith Ġsemi - man ual Ġpyramid Ġevaluation Ġscores Ġin Ġcomparison Ġwith ĠRouge . </s> Our Ġanalysis Ġon Ġthe Ġeffectiveness Ġof Ġevaluation Ġmeasures Ġfor Ġscientific Ġsumm aries Ġwas Ġperformed Ġusing Ġcorrelations Ġwith Ġmanual Ġjudgments . ĠAn Ġalternative Ġapproach Ġto Ġfollow Ġwould Ġbe Ġto Ġuse Ġstatistical Ġsignificance Ġtesting Ġon Ġthe Ġability Ġof Ġthe Ġmetrics Ġto Ġdistinguish Ġbetween Ġthe Ġsummar izers Ġ( similar Ġto ĠRan kel : 2011 ). ĠWe Ġstudied Ġthe Ġeffectiveness Ġof Ġexisting Ġsummar ization Ġevaluation Ġmetrics Ġin Ġthe Ġscientific Ġtext Ġgenre Ġand Ġproposed Ġan Ġalternative Ġsuperior Ġmetric . ĠAnother Ġextension Ġof Ġthis Ġwork Ġwould Ġbe Ġto Ġevaluate Ġautomatic Ġsummar ization Ġevaluation Ġin Ġother Ġgenres Ġof Ġtext Ġ( such Ġas Ġsocial Ġmedia ). ĠOur Ġproposed Ġmethod Ġonly Ġevaluates Ġthe Ġcontent Ġquality Ġof Ġthe Ġsummary . ĠSimilar Ġto Ġmost Ġof Ġexisting Ġsummar ization Ġevaluation Ġmetrics , Ġother Ġqualities Ġsuch Ġas Ġlinguistic Ġcohesion , Ġco herence Ġand Ġread ability Ġare Ġnot Ġcaptured Ġby Ġthis Ġmethod . ĠDevelop ing Ġmetrics Ġthat Ġalso Ġincorporate Ġthese Ġqualities Ġis Ġyet Ġanother Ġfuture Ġdirection Ġto Ġfollow . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġall Ġthree Ġanonymous Ġreviewers Ġfor Ġtheir Ġfeedback Ġand Ġcomments , Ġand ĠMary am ĠIran man esh Ġfor Ġhelping Ġin Ġannotation . ĠThis Ġwork Ġwas Ġpartially Ġsupported Ġby ĠNational ĠScience ĠFoundation Ġ( NS F ) Ġthrough Ġgrant ĠCNS - 12 04 347 .