DIP corpus BIBREF37
EXTRACTIVE
As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e.g. bullying, homeschooling, drugs) with a short description of each topic. It was created from a large web crawl using state-of-the-art information retrieval. We selected 30 of the topics for which we created the necessary concept map annotations.
1704.04452
Introduction </s> Multi - document Ġsummar ization Ġ( M DS ), Ġthe Ġtransformation Ġof Ġa Ġset Ġof Ġdocuments Ġinto Ġa Ġshort Ġtext Ġcontaining Ġtheir Ġmost Ġimportant Ġaspects , Ġis Ġa Ġlong - stud ied Ġproblem Ġin ĠN LP . ĠGener ated Ġsumm aries Ġhave Ġbeen Ġshown Ġto Ġsupport Ġhumans Ġdealing Ġwith Ġlarge Ġdocument Ġcollections Ġin Ġinformation Ġseeking Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠHowever , Ġwhen Ġexploring Ġa Ġset Ġof Ġdocuments Ġmanually , Ġhumans Ġrarely Ġwrite Ġa Ġfully - form ulated Ġsummary Ġfor Ġthemselves . ĠInstead , Ġuser Ġstudies ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġshow Ġthat Ġthey Ġnote Ġdown Ġimportant Ġkeywords Ġand Ġphrases , Ġtry Ġto Ġidentify Ġrelationships Ġbetween Ġthem Ġand Ġorganize Ġthem Ġaccordingly . ĠTherefore , Ġwe Ġbelieve Ġthat Ġthe Ġstudy Ġof Ġsummar ization Ġwith Ġsimilarly Ġstructured Ġoutputs Ġis Ġan Ġimportant Ġextension Ġof Ġthe Ġtraditional Ġtask . </s> A Ġrepresentation Ġthat Ġis Ġmore Ġin Ġline Ġwith Ġobserved Ġuser Ġbehavior Ġis Ġa Ġconcept Ġmap ĠB IB REF 5 Ġ, Ġa Ġlabeled Ġgraph Ġshowing Ġconcepts Ġas Ġnodes Ġand Ġrelationships Ġbetween Ġthem Ġas Ġedges Ġ( Figure ĠFIG REF 2 Ġ). ĠIntroduced Ġin Ġ1972 Ġas Ġa Ġteaching Ġtool ĠB IB REF 6 Ġ, Ġconcept Ġmaps Ġhave Ġfound Ġmany Ġapplications Ġin Ġeducation ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġfor Ġwriting Ġassistance ĠB IB REF 9 Ġor Ġto Ġstructure Ġinformation Ġrepositories ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠFor Ġsummar ization , Ġconcept Ġmaps Ġmake Ġit Ġpossible Ġto Ġrepresent Ġa Ġsummary Ġconcise ly Ġand Ġclearly Ġreveal Ġrelationships . ĠMoreover , Ġwe Ġsee Ġa Ġsecond Ġinteresting Ġuse Ġcase Ġthat Ġgoes Ġbeyond Ġthe Ġcapabilities Ġof Ġtextual Ġsumm aries : ĠWhen Ġconcepts Ġand Ġrelations Ġare Ġlinked Ġto Ġcorresponding Ġlocations Ġin Ġthe Ġdocuments Ġthey Ġhave Ġbeen Ġextracted Ġfrom , Ġthe Ġgraph Ġcan Ġbe Ġused Ġto Ġnavigate Ġin Ġa Ġdocument Ġcollection , Ġsimilar Ġto Ġa Ġtable Ġof Ġcontents . ĠAn Ġimplementation Ġof Ġthis Ġidea Ġhas Ġbeen Ġrecently Ġdescribed Ġby ĠB IB REF 12 Ġ. </s> The Ġcorresponding Ġtask Ġthat Ġwe Ġpropose Ġis Ġconcept - map - based ĠM DS , Ġthe Ġsummar ization Ġof Ġa Ġdocument Ġcluster Ġin Ġthe Ġform Ġof Ġa Ġconcept Ġmap . ĠIn Ġorder Ġto Ġdevelop Ġand Ġevaluate Ġmethods Ġfor Ġthe Ġtask , Ġgold - standard Ġcorpor a Ġare Ġnecessary , Ġbut Ġno Ġsuitable Ġcorpus Ġis Ġavailable . ĠThe Ġmanual Ġcreation Ġof Ġsuch Ġa Ġdataset Ġis Ġvery Ġtime - consuming , Ġas Ġthe Ġannotation Ġincludes Ġmany Ġsubt asks . ĠIn Ġparticular , Ġan Ġannot ator Ġwould Ġneed Ġto Ġmanually Ġidentify Ġall Ġconcepts Ġin Ġthe Ġdocuments , Ġwhile Ġonly Ġa Ġfew Ġof Ġthem Ġwill Ġeventually Ġend Ġup Ġin Ġthe Ġsummary . </s> To Ġovercome Ġthese Ġissues , Ġwe Ġpresent Ġa Ġcorpus Ġcreation Ġmethod Ġthat Ġeffectively Ġcombines Ġautomatic Ġpre processing , Ġscalable Ġcrowds ourcing Ġand Ġhigh - quality Ġexpert Ġannotations . ĠUsing Ġit , Ġwe Ġcan Ġavoid Ġthe Ġhigh Ġeffort Ġfor Ġsingle Ġannot ators , Ġallowing Ġus Ġto Ġscale Ġto Ġdocument Ġclusters Ġthat Ġare Ġ15 Ġtimes Ġlarger Ġthan Ġin Ġtraditional Ġsummar ization Ġcorpor a . ĠWe Ġcreated Ġa Ġnew Ġcorpus Ġof Ġ30 Ġtopics , Ġeach Ġwith Ġaround Ġ40 Ġsource Ġdocuments Ġon Ġeducational Ġtopics Ġand Ġa Ġsummar izing Ġconcept Ġmap Ġthat Ġis Ġthe Ġconsensus Ġof Ġmany Ġcrowd workers Ġ( see ĠFigure ĠFIG REF 3 Ġ). </s> As Ġa Ġcrucial Ġstep Ġof Ġthe Ġcorpus Ġcreation , Ġwe Ġdeveloped Ġa Ġnew Ġcrowds ourcing Ġscheme Ġcalled Ġlow - context Ġimportance Ġannotation . ĠIn Ġcontrast Ġto Ġtraditional Ġapproaches , Ġit Ġallows Ġus Ġto Ġdetermine Ġimportant Ġelements Ġin Ġa Ġdocument Ġcluster Ġwithout Ġrequiring Ġannot ators Ġto Ġread Ġall Ġdocuments , Ġmaking Ġit Ġfeasible Ġto Ġcrowd source Ġthe Ġtask Ġand Ġovercome Ġquality Ġissues Ġobserved Ġin Ġprevious Ġwork ĠB IB REF 13 Ġ. ĠWe Ġshow Ġthat Ġthe Ġapproach Ġcreates Ġreliable Ġdata Ġfor Ġour Ġfocused Ġsummar ization Ġscenario Ġand , Ġwhen Ġtested Ġon Ġtraditional Ġsummar ization Ġcorpor a , Ġcreates Ġannotations Ġthat Ġare Ġsimilar Ġto Ġthose Ġobtained Ġby Ġearlier Ġefforts . </s> To Ġsummarize , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions : Ġ( 1 ) ĠWe Ġpropose Ġa Ġnovel Ġtask , Ġconcept - map - based ĠM DS Ġ( Â§ ĠSEC REF 2 Ġ), Ġ( 2 ) Ġpresent Ġa Ġnew Ġcrowds ourcing Ġscheme Ġto Ġcreate Ġreference Ġsumm aries Ġ( Â§ ĠSEC REF 4 Ġ), Ġ( 3 ) Ġpublish Ġa Ġnew Ġdataset Ġfor Ġthe Ġproposed Ġtask Ġ( Â§ ĠSEC REF 5 Ġ) Ġand Ġ( 4 ) Ġprovide Ġan Ġevaluation Ġprotocol Ġand Ġbaseline Ġ( Â§ ĠSEC REF 7 Ġ). ĠWe Ġmake Ġthese Ġresources Ġpublicly Ġavailable Ġunder Ġa Ġper missive Ġlicense . </s> Task </s> Con cept - map - based ĠM DS Ġis Ġdefined Ġas Ġfollows : ĠGiven Ġa Ġset Ġof Ġrelated Ġdocuments , Ġcreate Ġa Ġconcept Ġmap Ġthat Ġrepresents Ġits Ġmost Ġimportant Ġcontent , Ġsatisfies Ġa Ġspecified Ġsize Ġlimit Ġand Ġis Ġconnected . </s> We Ġdefine Ġa Ġconcept Ġmap Ġas Ġa Ġlabeled Ġgraph Ġshowing Ġconcepts Ġas Ġnodes Ġand Ġrelationships Ġbetween Ġthem Ġas Ġedges . ĠLab els Ġare Ġarbitrary Ġsequences Ġof Ġtokens Ġtaken Ġfrom Ġthe Ġdocuments , Ġmaking Ġthe Ġsummar ization Ġtask Ġextract ive . ĠA Ġconcept Ġcan Ġbe Ġan Ġentity , Ġabstract Ġidea , Ġevent Ġor Ġactivity , Ġdesignated Ġby Ġits Ġunique Ġlabel . ĠGood Ġmaps Ġshould Ġbe Ġpropos itionally Ġcoherent , Ġmeaning Ġthat Ġevery Ġrelation Ġtogether Ġwith Ġthe Ġtwo Ġconnected Ġconcepts Ġform Ġa Ġmeaningful Ġproposition . </s> The Ġtask Ġis Ġcomplex , Ġconsisting Ġof Ġseveral Ġinter dependent Ġsubt asks . ĠOne Ġhas Ġto Ġextract Ġappropriate Ġlabels Ġfor Ġconcepts Ġand Ġrelations Ġand Ġrecognize Ġdifferent Ġexpressions Ġthat Ġrefer Ġto Ġthe Ġsame Ġconcept Ġacross Ġmultiple Ġdocuments . ĠFurther , Ġone Ġhas Ġto Ġselect Ġthe Ġmost Ġimportant Ġconcepts Ġand Ġrelations Ġfor Ġthe Ġsummary Ġand Ġfinally Ġorganize Ġthem Ġin Ġa Ġgraph Ġsatisfying Ġthe Ġconnected ness Ġand Ġsize Ġconstraints . </s> Related ĠWork </s> Some Ġattempts Ġhave Ġbeen Ġmade Ġto Ġautomatically Ġconstruct Ġconcept Ġmaps Ġfrom Ġtext , Ġworking Ġwith Ġeither Ġsingle Ġdocuments ĠB IB REF 14 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġor Ġdocument Ġclusters ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThese Ġapproaches Ġextract Ġconcept Ġand Ġrelation Ġlabels Ġfrom Ġsynt actic Ġstructures Ġand Ġconnect Ġthem Ġto Ġbuild Ġa Ġconcept Ġmap . ĠHowever , Ġcommon Ġtask Ġdefinitions Ġand Ġcomparable Ġevaluations Ġare Ġmissing . ĠIn Ġaddition , Ġonly Ġa Ġfew Ġof Ġthem , Ġnamely ĠVill alon . 2012 Ġand ĠVal er io . 2006 , Ġdefine Ġsummar ization Ġas Ġtheir Ġgoal Ġand Ġtry Ġto Ġcompress Ġthe Ġinput Ġto Ġa Ġsubstantially Ġsmaller Ġsize . ĠOur Ġnewly Ġproposed Ġtask Ġand Ġthe Ġcreated Ġlarge - cl uster Ġdataset Ġfill Ġthese Ġgaps Ġas Ġthey Ġemphasize Ġthe Ġsummar ization Ġaspect Ġof Ġthe Ġtask . </s> For Ġthe Ġsubt ask Ġof Ġselecting Ġsummary - worthy Ġconcepts Ġand Ġrelations , Ġtechniques Ġdeveloped Ġfor Ġtraditional Ġsummar ization ĠB IB REF 20 Ġand Ġkey phrase Ġextraction ĠB IB REF 21 Ġare Ġrelated Ġand Ġapplicable . ĠAppro aches Ġthat Ġbuild Ġgraphs Ġof Ġpropositions Ġto Ġcreate Ġa Ġsummary ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġseem Ġto Ġbe Ġparticularly Ġrelated , Ġhowever , Ġthere Ġis Ġone Ġimportant Ġdifference : ĠWhile Ġthey Ġuse Ġgraphs Ġas Ġan Ġintermediate Ġrepresentation Ġfrom Ġwhich Ġa Ġtextual Ġsummary Ġis Ġthen Ġgenerated , Ġthe Ġgoal Ġof Ġthe Ġproposed Ġtask Ġis Ġto Ġcreate Ġa Ġgraph Ġthat Ġis Ġdirectly Ġinterpret able Ġand Ġuseful Ġfor Ġa Ġuser . ĠIn Ġcontrast , Ġthese Ġintermediate Ġgraphs , Ġe . g . ĠAM R , Ġare Ġhardly Ġuseful Ġfor Ġa Ġtypical , Ġnon - ling u ist Ġuser . </s> For Ġtraditional Ġsummar ization , Ġthe Ġmost Ġwell - known Ġdatasets Ġemerged Ġout Ġof Ġthe ĠD UC Ġand ĠT AC Ġcompetitions . ĠThey Ġprovide Ġclusters Ġof Ġnews Ġarticles Ġwith Ġgold - standard Ġsumm aries . ĠExt ending Ġthese Ġefforts , Ġseveral Ġmore Ġspecialized Ġcorpor a Ġhave Ġbeen Ġcreated : ĠWith Ġregard Ġto Ġsize , ĠNak ano . 2010 Ġpresent Ġa Ġcorpus Ġof Ġsumm aries Ġfor Ġlarge - scale Ġcollections Ġof Ġweb Ġpages . ĠRecently , Ġcorpor a Ġwith Ġmore Ġheter ogeneous Ġdocuments Ġhave Ġbeen Ġsuggested , Ġe . g . ĠB IB REF 26 Ġand ĠB IB REF 27 Ġ. ĠThe Ġcorpus Ġwe Ġpresent Ġcombines Ġthese Ġaspects , Ġas Ġit Ġhas Ġlarge Ġclusters Ġof Ġheter ogeneous Ġdocuments , Ġand Ġprovides Ġa Ġnecessary Ġbenchmark Ġto Ġevaluate Ġthe Ġproposed Ġtask . </s> For Ġconcept Ġmap Ġgeneration , Ġone Ġcorpus Ġwith Ġhuman - created Ġsummary Ġconcept Ġmaps Ġfor Ġstudent Ġessays Ġhas Ġbeen Ġcreated ĠB IB REF 28 Ġ. ĠIn Ġcontrast Ġto Ġour Ġcorpus , Ġit Ġonly Ġdeals Ġwith Ġsingle Ġdocuments , Ġrequires Ġa Ġtwo Ġorders Ġof Ġmagnitude Ġsmaller Ġamount Ġof Ġcompression Ġof Ġthe Ġinput Ġand Ġis Ġnot Ġpublicly Ġavailable Ġ. </s> Other Ġtypes Ġof Ġinformation Ġrepresentation Ġthat Ġalso Ġmodel Ġconcepts Ġand Ġtheir Ġrelationships Ġare Ġknowledge Ġbases , Ġsuch Ġas ĠFree base ĠB IB REF 29 Ġ, Ġand Ġont ologies . ĠHowever , Ġthey Ġboth Ġdiffer Ġin Ġimportant Ġaspects : ĠWhereas Ġconcept Ġmaps Ġfollow Ġan Ġopen Ġlabel Ġparadigm Ġand Ġare Ġmeant Ġto Ġbe Ġinterpret able Ġby Ġhumans , Ġknowledge Ġbases Ġand Ġont ologies Ġare Ġusually Ġmore Ġstrictly Ġtyped Ġand Ġmade Ġto Ġbe Ġmachine - readable . ĠMoreover , Ġapproaches Ġto Ġautomatically Ġconstruct Ġthem Ġfrom Ġtext Ġtypically Ġtry Ġto Ġextract Ġas Ġmuch Ġinformation Ġas Ġpossible , Ġwhile Ġwe Ġwant Ġto Ġsummarize Ġa Ġdocument . </s> Low - Context ĠImport ance ĠAn notation </s> L l oret . 2013 Ġdescribe Ġseveral Ġexperiments Ġto Ġcrowd source Ġreference Ġsumm aries . ĠWorkers Ġare Ġasked Ġto Ġread Ġ10 Ġdocuments Ġand Ġthen Ġselect Ġ10 Ġsummary Ġsentences Ġfrom Ġthem Ġfor Ġa Ġreward Ġof Ġ$ 0 . 05 . ĠThey Ġdiscovered Ġseveral Ġchallenges , Ġincluding Ġpoor Ġwork Ġquality Ġand Ġthe Ġsubject iveness Ġof Ġthe Ġannotation Ġtask , Ġindicating Ġthat Ġcrowds ourcing Ġis Ġnot Ġuseful Ġfor Ġthis Ġpurpose . </s> To Ġovercome Ġthese Ġissues , Ġwe Ġintroduce Ġa Ġnew Ġtask Ġdesign , Ġlow - context Ġimportance Ġannotation , Ġto Ġdetermine Ġsummary - worthy Ġparts Ġof Ġdocuments . ĠCompared Ġto ĠLl oret Ġet Ġal .' s Ġapproach , Ġit Ġis Ġmore Ġin Ġline Ġwith Ġcrowds ourcing Ġbest Ġpractices , Ġas Ġthe Ġtasks Ġare Ġsimple , Ġintuitive Ġand Ġsmall ĠB IB REF 30 Ġand Ġworkers Ġreceive Ġreasonable Ġpayment ĠB IB REF 31 Ġ. ĠMost Ġimportantly , Ġit Ġis Ġalso Ġmuch Ġmore Ġefficient Ġand Ġscalable , Ġas Ġit Ġdoes Ġnot Ġrequire Ġworkers Ġto Ġread Ġall Ġdocuments Ġin Ġa Ġcluster . </s> Task ĠDesign </s> We Ġbreak Ġdown Ġthe Ġtask Ġof Ġimportance Ġannotation Ġto Ġthe Ġlevel Ġof Ġsingle Ġpropositions . ĠThe Ġgoal Ġof Ġour Ġcrowds ourcing Ġscheme Ġis Ġto Ġobtain Ġa Ġscore Ġfor Ġeach Ġproposition Ġindicating Ġits Ġimportance Ġin Ġa Ġdocument Ġcluster , Ġsuch Ġthat Ġa Ġranking Ġaccording Ġto Ġthe Ġscore Ġwould Ġreveal Ġwhat Ġis Ġmost Ġimportant Ġand Ġshould Ġbe Ġincluded Ġin Ġa Ġsummary . ĠIn Ġcontrast Ġto Ġother Ġwork , Ġwe Ġdo Ġnot Ġshow Ġthe Ġdocuments Ġto Ġthe Ġworkers Ġat Ġall , Ġbut Ġprovide Ġonly Ġa Ġdescription Ġof Ġthe Ġdocument Ġcluster 's Ġtopic Ġalong Ġwith Ġthe Ġpropositions . ĠThis Ġensures Ġthat Ġtasks Ġare Ġsmall , Ġsimple Ġand Ġcan Ġbe Ġdone Ġquickly Ġ( see ĠFigure ĠFIG REF 4 Ġ). </s> In Ġpreliminary Ġtests , Ġwe Ġfound Ġthat Ġthis Ġdesign , Ġdespite Ġthe Ġminimal Ġcontext , Ġworks Ġreasonably Ġon Ġour Ġfocused Ġclusters Ġon Ġcommon Ġeducational Ġtopics . ĠFor Ġinstance , Ġconsider ĠFigure ĠFIG REF 4 Ġ: ĠOne Ġcan Ġeasily Ġsay Ġthat ĠP 1 Ġis Ġmore Ġimportant Ġthan ĠP 2 Ġwithout Ġreading Ġthe Ġdocuments . </s> We Ġdistinguish Ġtwo Ġtask Ġvariants : </s> Instead Ġof Ġenforcing Ġbinary Ġimportance Ġdecisions , Ġwe Ġuse Ġa Ġ5 - point ĠL ik ert - scale Ġto Ġallow Ġmore Ġfine - gr ained Ġannotations . ĠThe Ġobtained Ġlabels Ġare Ġtranslated Ġinto Ġscores Ġ( 5 .. 1 ) Ġand Ġthe Ġaverage Ġof Ġall Ġscores Ġfor Ġa Ġproposition Ġis Ġused Ġas Ġan Ġestimate Ġfor Ġits Ġimportance . ĠThis Ġfollows Ġthe Ġidea Ġthat Ġwhile Ġsingle Ġworkers Ġmight Ġfind Ġthe Ġtask Ġsubjective , Ġthe Ġconsensus Ġof Ġmultiple Ġworkers , Ġrepresented Ġin Ġthe Ġaverage Ġscore , Ġtends Ġto Ġbe Ġless Ġsubjective Ġdue Ġto Ġthe ĠâĢ ľ w isdom Ġof Ġthe Ġcrowd âĢ Ŀ . ĠWe Ġrandomly Ġgroup Ġfive Ġpropositions Ġinto Ġa Ġtask . </s> As Ġan Ġalternative , Ġwe Ġuse Ġa Ġsecond Ġtask Ġdesign Ġbased Ġon Ġpair wise Ġcomparisons . ĠCompar isons Ġare Ġknown Ġto Ġbe Ġeasier Ġto Ġmake Ġand Ġmore Ġconsistent ĠB IB REF 32 Ġ, Ġbut Ġalso Ġmore Ġexpensive , Ġas Ġthe Ġnumber Ġof Ġpairs Ġgrows Ġquad r atically Ġwith Ġthe Ġnumber Ġof Ġobjects . ĠTo Ġreduce Ġthe Ġcost , Ġwe Ġgroup Ġfive Ġpropositions Ġinto Ġa Ġtask Ġand Ġask Ġworkers Ġto Ġorder Ġthem Ġby Ġimportance Ġper Ġdrag - and - drop . ĠFrom Ġthe Ġresults , Ġwe Ġderive Ġpair wise Ġcomparisons Ġand Ġuse ĠTrue Skill ĠB IB REF 35 Ġ, Ġa Ġpowerful ĠBay esian Ġrank Ġinduction Ġmodel ĠB IB REF 34 Ġ, Ġto Ġobtain Ġimportance Ġestimates Ġfor Ġeach Ġproposition . </s> P ilot ĠStudy </s> To Ġverify Ġthe Ġproposed Ġapproach , Ġwe Ġconducted Ġa Ġpilot Ġstudy Ġon ĠAmazon ĠMechanical ĠTurk Ġusing Ġdata Ġfrom ĠT AC 2008 ĠB IB REF 36 Ġ. ĠWe Ġcollected Ġimportance Ġestimates Ġfor Ġ4 74 Ġpropositions Ġextracted Ġfrom Ġthe Ġfirst Ġthree Ġclusters Ġusing Ġboth Ġtask Ġdesigns . ĠEach ĠL ik ert - scale Ġtask Ġwas Ġassigned Ġto Ġ5 Ġdifferent Ġworkers Ġand Ġawarded Ġ$ 0 . 06 . ĠFor Ġcomparison Ġtasks , Ġwe Ġalso Ġcollected Ġ5 Ġlabels Ġeach , Ġpaid Ġ$ 0 . 05 Ġand Ġsampled Ġaround Ġ7 % Ġof Ġall Ġpossible Ġpairs . ĠWe Ġsubmitted Ġthem Ġin Ġbatches Ġof Ġ100 Ġpairs Ġand Ġselected Ġpairs Ġfor Ġsubsequent Ġbatches Ġbased Ġon Ġthe Ġconfidence Ġof Ġthe ĠTrue Skill Ġmodel . </s> Following Ġthe Ġobservations Ġof ĠLl oret . 2013 , Ġwe Ġestablished Ġseveral Ġmeasures Ġfor Ġquality Ġcontrol . ĠFirst , Ġwe Ġrestricted Ġour Ġtasks Ġto Ġworkers Ġfrom Ġthe ĠUS Ġwith Ġan Ġapproval Ġrate Ġof Ġat Ġleast Ġ95 %. ĠSecond , Ġwe Ġidentified Ġlow Ġquality Ġworkers Ġby Ġmeasuring Ġthe Ġcorrelation Ġof Ġeach Ġworker 's ĠL ik ert - sc ores Ġwith Ġthe Ġaverage Ġof Ġthe Ġother Ġfour Ġscores . ĠThe Ġworst Ġworkers Ġ( at Ġmost Ġ5 % Ġof Ġall Ġlabels ) Ġwere Ġremoved . </s> In Ġaddition , Ġwe Ġincluded Ġtrap Ġsentences , Ġsimilar Ġas Ġin ĠB IB REF 13 Ġ, Ġin Ġaround Ġ80 Ġof Ġthe Ġtasks . ĠIn Ġcontrast Ġto ĠLl oret Ġet Ġal .' s Ġfindings , Ġboth Ġan Ġobvious Ġtrap Ġsentence Ġ( This Ġsentence Ġis Ġnot Ġimportant ) Ġand Ġa Ġless Ġobvious Ġbut Ġun important Ġone Ġ( Bar ack ĠObama Ġgraduated Ġfrom ĠHarvard ĠLaw ) Ġwere Ġconsistently Ġlabeled Ġas Ġun important Ġ( 1 . 08 Ġand Ġ1 . 14 ), Ġindicating Ġthat Ġthe Ġworkers Ġdid Ġthe Ġtask Ġproperly . </s> For ĠL ik ert - scale Ġtasks , Ġwe Ġfollow ĠSnow . 2008 Ġand Ġcalculate Ġagreement Ġas Ġthe Ġaverage ĠPearson Ġcorrelation Ġof Ġa Ġworker 's ĠL ik ert - score Ġwith Ġthe Ġaverage Ġscore Ġof Ġthe Ġremaining Ġworkers . ĠThis Ġmeasure Ġis Ġless Ġstrict Ġthan Ġexact Ġlabel Ġagreement Ġand Ġcan Ġaccount Ġfor Ġclose Ġlabels Ġand Ġhigh - Ġor Ġlow - scoring Ġworkers . ĠWe Ġobserve Ġa Ġcorrelation Ġof Ġ0 . 81 , Ġindicating Ġsubstantial Ġagreement . ĠFor Ġcomparisons , Ġthe Ġmajority Ġagreement Ġis Ġ0 . 73 . ĠTo Ġfurther Ġexamine Ġthe Ġreliability Ġof Ġthe Ġcollected Ġdata , Ġwe Ġfollowed Ġthe Ġapproach Ġof ĠK irit chenko . 2016 Ġand Ġsimply Ġrepeated Ġthe Ġcrowds ourcing Ġfor Ġone Ġof Ġthe Ġthree Ġtopics . ĠBetween Ġthe Ġimportance Ġestimates Ġcalculated Ġfrom Ġthe Ġfirst Ġand Ġsecond Ġrun , Ġwe Ġfound Ġa ĠPearson Ġcorrelation Ġof Ġ0 . 82 Ġ( Spe ar man Ġ0 . 78 ) Ġfor ĠL ik ert - scale Ġtasks Ġand Ġ0 . 69 Ġ( Spe ar man Ġ0 . 66 ) Ġfor Ġcomparison Ġtasks . ĠThis Ġshows Ġthat Ġthe Ġapproach , Ġdespite Ġthe Ġsubject iveness Ġof Ġthe Ġtask , Ġallows Ġus Ġto Ġcollect Ġreliable Ġannotations . </s> In Ġaddition Ġto Ġthe Ġreliability Ġstudies , Ġwe Ġextr ins ically Ġevaluated Ġthe Ġannotations Ġin Ġthe Ġtask Ġof Ġsummary Ġevaluation . ĠFor Ġeach Ġof Ġthe Ġ58 Ġpeer Ġsumm aries Ġin ĠT AC 2008 , Ġwe Ġcalculated Ġa Ġscore Ġas Ġthe Ġsum Ġof Ġthe Ġimportance Ġestimates Ġof Ġthe Ġpropositions Ġit Ġcontains . ĠTable ĠT AB REF 13 Ġshows Ġhow Ġthese Ġpeer Ġscores , Ġaveraged Ġover Ġthe Ġthree Ġtopics , Ġcorrelate Ġwith Ġthe Ġmanual Ġresponsiveness Ġscores Ġassigned Ġduring ĠT AC Ġin Ġcomparison Ġto ĠR OU GE - 2 Ġand ĠPyramid Ġscores . ĠThe Ġresults Ġdemonstrate Ġthat Ġwith Ġboth Ġtask Ġdesigns , Ġwe Ġobtain Ġimportance Ġannotations Ġthat Ġare Ġsimilarly Ġuseful Ġfor Ġsummary Ġevaluation Ġas Ġpyramid Ġannotations Ġor Ġgold - standard Ġsumm aries Ġ( used Ġfor ĠR OU GE ). </s> Based Ġon Ġthe Ġpilot Ġstudy , Ġwe Ġconclude Ġthat Ġthe Ġproposed Ġcrowds ourcing Ġscheme Ġallows Ġus Ġto Ġobtain Ġproper Ġimportance Ġannotations Ġfor Ġpropositions . ĠAs Ġworkers Ġare Ġnot Ġrequired Ġto Ġread Ġall Ġdocuments , Ġthe Ġannotation Ġis Ġmuch Ġmore Ġefficient Ġand Ġscalable Ġas Ġwith Ġtraditional Ġmethods . </s> Corp us ĠCreation </s> This Ġsection Ġpresents Ġthe Ġcorpus Ġconstruction Ġprocess , Ġas Ġoutlined Ġin ĠFigure ĠFIG REF 16 Ġ, Ġcombining Ġautomatic Ġpre processing , Ġscalable Ġcrowds ourcing Ġand Ġhigh - quality Ġexpert Ġannotations Ġto Ġbe Ġable Ġto Ġscale Ġto Ġthe Ġsize Ġof Ġour Ġdocument Ġclusters . ĠFor Ġevery Ġtopic , Ġwe Ġspent Ġabout Ġ$ 150 Ġon Ġcrowds ourcing Ġand Ġ1 . 5 h Ġof Ġexpert Ġannotations , Ġwhile Ġjust Ġa Ġsingle Ġannot ator Ġwould Ġalready Ġneed Ġover Ġ8 Ġhours Ġ( at Ġ200 Ġwords Ġper Ġminute ) Ġto Ġread Ġall Ġdocuments Ġof Ġa Ġtopic . </s> Source ĠData </s> As Ġa Ġstarting Ġpoint , Ġwe Ġused Ġthe ĠD IP Ġcorpus ĠB IB REF 37 Ġ, Ġa Ġcollection Ġof Ġ49 Ġclusters Ġof Ġ100 Ġweb Ġpages Ġon Ġeducational Ġtopics Ġ( e . g . Ġbullying , Ġhomes chool ing , Ġdrugs ) Ġwith Ġa Ġshort Ġdescription Ġof Ġeach Ġtopic . ĠIt Ġwas Ġcreated Ġfrom Ġa Ġlarge Ġweb Ġcrawl Ġusing Ġstate - of - the - art Ġinformation Ġretrieval . ĠWe Ġselected Ġ30 Ġof Ġthe Ġtopics Ġfor Ġwhich Ġwe Ġcreated Ġthe Ġnecessary Ġconcept Ġmap Ġannotations . </s> Prop osition ĠExt raction </s> As Ġconcept Ġmaps Ġconsist Ġof Ġpropositions Ġexpressing Ġthe Ġrelation Ġbetween Ġconcepts Ġ( see ĠFigure ĠFIG REF 2 Ġ), Ġwe Ġneed Ġto Ġimpose Ġsuch Ġa Ġstructure Ġupon Ġthe Ġplain Ġtext Ġin Ġthe Ġdocument Ġclusters . ĠThis Ġcould Ġbe Ġdone Ġby Ġmanually Ġannot ating Ġspans Ġrepresenting Ġconcepts Ġand Ġrelations , Ġhowever , Ġthe Ġsize Ġof Ġour Ġclusters Ġmakes Ġthis Ġa Ġhuge Ġeffort : Ġ2 288 Ġsentences Ġper Ġtopic Ġ( 69 k Ġin Ġtotal ) Ġneed Ġto Ġbe Ġprocessed . ĠTherefore , Ġwe Ġresort Ġto Ġan Ġautomatic Ġapproach . </s> The ĠOpen ĠInformation ĠExt raction Ġparadigm ĠB IB REF 38 Ġoffers Ġa Ġrepresentation Ġvery Ġsimilar Ġto Ġthe Ġdesired Ġone . ĠFor Ġinstance , Ġfrom </s> Students Ġwith Ġbad Ġcredit Ġhistory Ġshould Ġnot Ġlose Ġhope Ġand Ġapply Ġfor Ġfederal Ġloans Ġwith Ġthe ĠF AF SA . </s> Open ĠIE Ġsystems Ġextract Ġtu ples Ġof Ġtwo Ġarguments Ġand Ġa Ġrelation Ġphrase Ġrepresenting Ġpropositions : </s> ( s . Ġwith Ġbad Ġcredit Ġhistory , Ġshould Ġnot Ġlose , Ġhope ) </s> ( s . Ġwith Ġbad Ġcredit Ġhistory , Ġapply Ġfor , Ġfederal Ġloans Ġwith Ġthe ĠF AF SA ) </s> While Ġthe Ġrelation Ġphrase Ġis Ġsimilar Ġto Ġa Ġrelation Ġin Ġa Ġconcept Ġmap , Ġmany Ġarguments Ġin Ġthese Ġtu ples Ġrepresent Ġuseful Ġconcepts . ĠWe Ġused ĠOpen ĠIE Ġ4 , Ġa Ġstate - of - the - art Ġsystem ĠB IB REF 39 Ġto Ġprocess Ġall Ġsentences . ĠAfter Ġremoving Ġdupl icates , Ġwe Ġobtained Ġ4 137 Ġtu ples Ġper Ġtopic . </s> Since Ġwe Ġwant Ġto Ġcreate Ġa Ġgold - standard Ġcorpus , Ġwe Ġhave Ġto Ġensure Ġthat Ġwe Ġproduce Ġhigh - quality Ġdata . ĠWe Ġtherefore Ġmade Ġuse Ġof Ġthe Ġconfidence Ġassigned Ġto Ġevery Ġextracted Ġtuple Ġto Ġfilter Ġout Ġlow Ġquality Ġones . ĠTo Ġensure Ġthat Ġwe Ġdo Ġnot Ġfilter Ġtoo Ġaggressively Ġ( and Ġmiss Ġimportant Ġaspects Ġin Ġthe Ġfinal Ġsummary ), Ġwe Ġmanually Ġannot ated Ġ500 Ġtu ples Ġsampled Ġfrom Ġall Ġtopics Ġfor Ġcorrectness . ĠOn Ġthe Ġfirst Ġ250 Ġof Ġthem , Ġwe Ġtuned Ġthe Ġfilter Ġthreshold Ġto Ġ0 . 5 , Ġwhich Ġkeeps Ġ98 . 7 % Ġof Ġthe Ġcorrect Ġextract ions Ġin Ġthe Ġunseen Ġsecond Ġhalf . ĠAfter Ġfiltering , Ġa Ġtopic Ġhad Ġon Ġaverage Ġ28 50 Ġpropositions Ġ( 85 k Ġin Ġtotal ). </s> Prop osition ĠFil tering </s> Despite Ġthe Ġsimilarity Ġof Ġthe ĠOpen ĠIE Ġparadigm , Ġnot Ġevery Ġextracted Ġtuple Ġis Ġa Ġsuitable Ġproposition Ġfor Ġa Ġconcept Ġmap . ĠTo Ġreduce Ġthe Ġeffort Ġin Ġthe Ġsubsequent Ġsteps , Ġwe Ġtherefore Ġwant Ġto Ġfilter Ġout Ġunsu itable Ġones . ĠA Ġtuple Ġis Ġsuitable Ġif Ġit Ġ( 1 ) Ġis Ġa Ġcorrect Ġextraction , Ġ( 2 ) Ġis Ġmeaningful Ġwithout Ġany Ġcontext Ġand Ġ( 3 ) Ġhas Ġarguments Ġthat Ġrepresent Ġproper Ġconcepts . ĠWe Ġcreated Ġa Ġguideline Ġexplaining Ġwhen Ġto Ġlabel Ġa Ġtuple Ġas Ġsuitable Ġfor Ġa Ġconcept Ġmap Ġand Ġperformed Ġa Ġsmall Ġannotation Ġstudy . ĠThree Ġannot ators Ġindependently Ġlabeled Ġ500 Ġrandomly Ġsampled Ġtu ples . ĠThe Ġagreement Ġwas Ġ82 % Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġfound Ġtu ples Ġto Ġbe Ġunsu itable Ġmostly Ġbecause Ġthey Ġhad Ġunres ol vable Ġpronouns , Ġconflicting Ġwith Ġ( 2 ), Ġor Ġarguments Ġthat Ġwere Ġfull Ġclauses Ġor Ġpropositions , Ġconflicting Ġwith Ġ( 3 ), Ġwhile Ġ( 1 ) Ġwas Ġmostly Ġtaken Ġcare Ġof Ġby Ġthe Ġconfidence Ġfiltering Ġin ĠÂ§ ĠSEC REF 21 Ġ. </s> Due Ġto Ġthe Ġhigh Ġnumber Ġof Ġtu ples Ġwe Ġdecided Ġto Ġautomate Ġthe Ġfiltering Ġstep . ĠWe Ġtrained Ġa Ġlinear ĠS VM Ġon Ġthe Ġmajority Ġvoted Ġannotations . ĠAs Ġfeatures , Ġwe Ġused Ġthe Ġextraction Ġconfidence , Ġlength Ġof Ġarguments Ġand Ġrelations Ġas Ġwell Ġas Ġpart - of - speech Ġtags , Ġamong Ġothers . ĠTo Ġensure Ġthat Ġthe Ġautomatic Ġclassification Ġdoes Ġnot Ġremove Ġsuitable Ġpropositions , Ġwe Ġtuned Ġthe Ġclass ifier Ġto Ġavoid Ġfalse Ġnegatives . ĠIn Ġparticular , Ġwe Ġintroduced Ġclass Ġweights , Ġimproving Ġprecision Ġon Ġthe Ġnegative Ġclass Ġat Ġthe Ġcost Ġof Ġa Ġhigher Ġfraction Ġof Ġpositive Ġclass ifications . ĠAdditionally , Ġwe Ġmanually Ġverified Ġa Ġcertain Ġnumber Ġof Ġthe Ġmost Ġuncertain Ġnegative Ġclass ifications Ġto Ġfurther Ġimprove Ġperformance . ĠWhen Ġ20 % Ġof Ġthe Ġclass ifications Ġare Ġmanually Ġverified Ġand Ġcorrected , Ġwe Ġfound Ġthat Ġour Ġmodel Ġtrained Ġon Ġ350 Ġlabeled Ġinstances Ġachieves Ġ93 % Ġprecision Ġon Ġnegative Ġclass ifications Ġon Ġthe Ġunseen Ġ150 Ġinstances . ĠWe Ġfound Ġthis Ġto Ġbe Ġa Ġreasonable Ġtrade - off Ġof Ġautomation Ġand Ġdata Ġquality Ġand Ġapplied Ġthe Ġmodel Ġto Ġthe Ġfull Ġdataset . </s> The Ġclass ifier Ġfiltered Ġout Ġ43 % Ġof Ġthe Ġpropositions , Ġleaving Ġ16 22 Ġper Ġtopic . ĠWe Ġmanually Ġexamined Ġthe Ġ17 k Ġleast Ġconfident Ġnegative Ġclass ifications Ġand Ġcorrected Ġ9 55 Ġof Ġthem . ĠWe Ġalso Ġcorrected Ġpositive Ġclass ifications Ġfor Ġcertain Ġtypes Ġof Ġtu ples Ġfor Ġwhich Ġwe Ġknew Ġthe Ġclass ifier Ġto Ġbe Ġimp rec ise . ĠFinally , Ġeach Ġtopic Ġwas Ġleft Ġwith Ġan Ġaverage Ġof Ġ15 54 Ġpropositions Ġ( 47 k Ġin Ġtotal ). </s> Import ance ĠAn notation </s> Given Ġthe Ġpropositions Ġidentified Ġin Ġthe Ġprevious Ġstep , Ġwe Ġnow Ġapplied Ġour Ġcrowds ourcing Ġscheme Ġas Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġto Ġdetermine Ġtheir Ġimportance . ĠTo Ġcope Ġwith Ġthe Ġlarge Ġnumber Ġof Ġpropositions , Ġwe Ġcombine Ġthe Ġtwo Ġtask Ġdesigns : ĠFirst , Ġwe Ġcollect ĠL ik ert - sc ores Ġfrom Ġ5 Ġworkers Ġfor Ġeach Ġproposition , Ġclean Ġthe Ġdata Ġand Ġcalculate Ġaverage Ġscores . ĠThen , Ġusing Ġonly Ġthe Ġtop Ġ100 Ġpropositions Ġaccording Ġto Ġthese Ġscores , Ġwe Ġcrowd source Ġ10 % Ġof Ġall Ġpossible Ġpair wise Ġcomparisons Ġamong Ġthem . ĠUsing ĠTrue Skill , Ġwe Ġobtain Ġa Ġfine - gr ained Ġranking Ġof Ġthe Ġ100 Ġmost Ġimportant Ġpropositions . </s> For ĠL ik ert - sc ores , Ġthe Ġaverage Ġagreement Ġover Ġall Ġtopics Ġis Ġ0 . 80 , Ġwhile Ġthe Ġmajority Ġagreement Ġfor Ġcomparisons Ġis Ġ0 . 78 . ĠWe Ġrepeated Ġthe Ġdata Ġcollection Ġfor Ġthree Ġrandomly Ġselected Ġtopics Ġand Ġfound Ġthe ĠPearson Ġcorrelation Ġbetween Ġboth Ġruns Ġto Ġbe Ġ0 . 73 Ġ( Spe ar man Ġ0 . 73 ) Ġfor ĠL ik ert - sc ores Ġand Ġ0 . 72 Ġ( Spe ar man Ġ0 . 71 ) Ġfor Ġcomparisons . ĠThese Ġfigures Ġshow Ġthat Ġthe Ġcrowds ourcing Ġapproach Ġworks Ġon Ġthis Ġdataset Ġas Ġreliably Ġas Ġon Ġthe ĠT AC Ġdocuments . </s> In Ġtotal , Ġwe Ġuploaded Ġ53 k Ġscoring Ġand Ġ12 k Ġcomparison Ġtasks Ġto ĠMechanical ĠTurk , Ġspending Ġ$ 44 25 . 45 Ġincluding Ġfees . ĠFrom Ġthe Ġfine - gr ained Ġranking Ġof Ġthe Ġ100 Ġmost Ġimportant Ġpropositions , Ġwe Ġselect Ġthe Ġtop Ġ50 Ġper Ġtopic Ġto Ġconstruct Ġa Ġsummary Ġconcept Ġmap Ġin Ġthe Ġsubsequent Ġsteps . </s> Prop osition ĠRevision </s> Having Ġa Ġmanageable Ġnumber Ġof Ġpropositions , Ġan Ġannot ator Ġthen Ġapplied Ġa Ġfew Ġstraightforward Ġtransformations Ġthat Ġcorrect Ġcommon Ġerrors Ġof Ġthe ĠOpen ĠIE Ġsystem . ĠFirst , Ġwe Ġbreak Ġdown Ġpropositions Ġwith Ġconj unctions Ġin Ġeither Ġof Ġthe Ġarguments Ġinto Ġseparate Ġpropositions Ġper Ġconj unct , Ġwhich Ġthe ĠOpen ĠIE Ġsystem Ġsometimes Ġfails Ġto Ġdo . ĠAnd Ġsecond , Ġwe Ġcorrect Ġspan Ġerrors Ġthat Ġmight Ġoccur Ġin Ġthe Ġargument Ġor Ġrelation Ġphrases , Ġespecially Ġwhen Ġsentences Ġwere Ġnot Ġproperly Ġsegment ed . ĠAs Ġa Ġresult , Ġwe Ġhave Ġa Ġset Ġof Ġhigh Ġquality Ġpropositions Ġfor Ġour Ġconcept Ġmap , Ġconsisting Ġof , Ġdue Ġto Ġthe Ġfirst Ġtransformation , Ġ56 . 1 Ġpropositions Ġper Ġtopic Ġon Ġaverage . </s> Con cept ĠMap ĠConstruction </s> In Ġthis Ġfinal Ġstep , Ġwe Ġconnect Ġthe Ġset Ġof Ġimportant Ġpropositions Ġto Ġform Ġa Ġgraph . ĠFor Ġinstance , Ġgiven Ġthe Ġfollowing Ġtwo Ġpropositions </s> ( student , Ġmay Ġborrow , ĠStafford ĠLoan ) </s> ( the Ġstudent , Ġdoes Ġnot Ġhave , Ġa Ġcredit Ġhistory ) </s> one Ġcan Ġeasily Ġsee , Ġalthough Ġthe Ġfirst Ġarguments Ġdiffer Ġslightly , Ġthat Ġboth Ġlabels Ġdescribe Ġthe Ġconcept Ġstudent , Ġallowing Ġus Ġto Ġbuild Ġa Ġconcept Ġmap Ġwith Ġthe Ġconcepts Ġstudent , ĠStafford ĠLoan Ġand Ġcredit Ġhistory . ĠThe Ġannotation Ġtask Ġthus Ġinvolves Ġdeciding Ġwhich Ġof Ġthe Ġavailable Ġpropositions Ġto Ġinclude Ġin Ġthe Ġmap , Ġwhich Ġof Ġtheir Ġconcepts Ġto Ġmerge Ġand , Ġwhen Ġmerging , Ġwhich Ġof Ġthe Ġavailable Ġlabels Ġto Ġuse . ĠAs Ġthese Ġdecisions Ġhighly Ġdepend Ġupon Ġeach Ġother Ġand Ġrequire Ġcontext , Ġwe Ġdecided Ġto Ġuse Ġexpert Ġannot ators Ġrather Ġthan Ġcrowd source Ġthe Ġsubt asks . </s> An not ators Ġwere Ġgiven Ġthe Ġtopic Ġdescription Ġand Ġthe Ġmost Ġimportant , Ġranked Ġpropositions . ĠUsing Ġa Ġsimple Ġannotation Ġtool Ġproviding Ġa Ġvisualization Ġof Ġthe Ġgraph , Ġthey Ġcould Ġconnect Ġthe Ġpropositions Ġstep Ġby Ġstep . ĠThey Ġwere Ġinstructed Ġto Ġreach Ġa Ġsize Ġof Ġ25 Ġconcepts , Ġthe Ġrecommended Ġmaximum Ġsize Ġfor Ġa Ġconcept Ġmap ĠB IB REF 6 Ġ. ĠFurther , Ġthey Ġshould Ġprefer Ġmore Ġimportant Ġpropositions Ġand Ġensure Ġconnected ness . ĠWhen Ġconnecting Ġtwo Ġpropositions , Ġthey Ġwere Ġasked Ġto Ġkeep Ġthe Ġconcept Ġlabel Ġthat Ġwas Ġappropriate Ġfor Ġboth Ġpropositions . ĠTo Ġsupport Ġthe Ġannot ators , Ġthe Ġtool Ġused ĠAD W ĠB IB REF 40 Ġ, Ġa Ġstate - of - the - art Ġapproach Ġfor Ġsemantic Ġsimilarity , Ġto Ġsuggest Ġpossible Ġconnections . ĠThe Ġannotation Ġwas Ġcarried Ġout Ġby Ġgraduate Ġstudents Ġwith Ġa Ġbackground Ġin ĠN LP Ġafter Ġreceiving Ġan Ġintroduction Ġinto Ġthe Ġguidelines Ġand Ġtool Ġand Ġannot ating Ġa Ġfirst Ġexample . </s> If Ġan Ġannot ator Ġwas Ġnot Ġable Ġto Ġconnect Ġ25 Ġconcepts , Ġshe Ġwas Ġallowed Ġto Ġcreate Ġup Ġto Ġthree Ġsynthetic Ġrelations Ġwith Ġfreely Ġdefined Ġlabels , Ġmaking Ġthe Ġmaps Ġslightly Ġabstract ive . ĠOn Ġaverage , Ġthe Ġconstructed Ġmaps Ġhave Ġ0 . 77 Ġsynthetic Ġrelations , Ġmostly Ġconnecting Ġconcepts Ġwhose Ġrelation Ġis Ġtoo Ġobvious Ġto Ġbe Ġexplicitly Ġstated Ġin Ġtext Ġ( e . g . Ġbetween ĠMont essor i Ġteacher Ġand ĠMont essor i Ġeducation ). </s> To Ġassess Ġthe Ġreliability Ġof Ġthis Ġannotation Ġstep , Ġwe Ġhad Ġthe Ġfirst Ġthree Ġmaps Ġcreated Ġby Ġtwo Ġannot ators . ĠWe Ġcast ed Ġthe Ġtask Ġof Ġselecting Ġpropositions Ġto Ġbe Ġincluded Ġin Ġthe Ġmap Ġas Ġa Ġbinary Ġdecision Ġtask Ġand Ġobserved Ġan Ġagreement Ġof Ġ84 % Ġ( ĠIN LINE FORM 0 Ġ). ĠSecond , Ġwe Ġmodeled Ġthe Ġdecision Ġwhich Ġconcepts Ġto Ġjoin Ġas Ġa Ġbinary Ġdecision Ġon Ġall Ġpairs Ġof Ġcommon Ġconcepts , Ġobserving Ġan Ġagreement Ġof Ġ95 % Ġ( ĠIN LINE FORM 1 Ġ). ĠAnd Ġfinally , Ġwe Ġcompared Ġwhich Ġconcept Ġlabels Ġthe Ġannot ators Ġdecided Ġto Ġinclude Ġin Ġthe Ġfinal Ġmap , Ġobserving Ġ85 % Ġ( ĠIN LINE FORM 2 Ġ) Ġagreement . ĠHence , Ġthe Ġannotation Ġshows Ġsubstantial Ġagreement ĠB IB REF 41 Ġ. </s> Corp us ĠAnalysis </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġnewly Ġcreated Ġcorpus , Ġwhich , Ġin Ġaddition Ġto Ġhaving Ġsumm aries Ġin Ġthe Ġform Ġof Ġconcept Ġmaps , Ġdiffers Ġfrom Ġtraditional Ġsummar ization Ġcorpor a Ġin Ġseveral Ġaspects . </s> Document ĠCl usters </s> The Ġcorpus Ġconsists Ġof Ġdocument Ġclusters Ġfor Ġ30 Ġdifferent Ġtopics . ĠEach Ġof Ġthem Ġcontains Ġaround Ġ40 Ġdocuments Ġwith Ġon Ġaverage Ġ24 13 Ġtokens , Ġwhich Ġleads Ġto Ġan Ġaverage Ġcluster Ġsize Ġof Ġ97 , 880 Ġtoken . ĠWith Ġthese Ġcharacteristics , Ġthe Ġdocument Ġclusters Ġare Ġ15 Ġtimes Ġlarger Ġthan Ġtypical ĠD UC Ġclusters Ġof Ġten Ġdocuments Ġand Ġfive Ġtimes Ġlarger Ġthan Ġthe Ġ25 - document - cl usters Ġ( Table ĠT AB REF 26 Ġ). ĠIn Ġaddition , Ġthe Ġdocuments Ġare Ġalso Ġmore Ġvariable Ġin Ġterms Ġof Ġlength , Ġas Ġthe Ġ( length - adjusted ) Ġstandard Ġdeviation Ġis Ġtwice Ġas Ġhigh Ġas Ġin Ġthe Ġother Ġcorpor a . ĠWith Ġthese Ġproperties , Ġthe Ġcorpus Ġrepresents Ġan Ġinteresting Ġchallenge Ġtowards Ġreal - world Ġapplication Ġscenarios , Ġin Ġwhich Ġusers Ġtypically Ġhave Ġto Ġdeal Ġwith Ġmuch Ġmore Ġthan Ġten Ġdocuments . </s> Because Ġwe Ġused Ġa Ġlarge Ġweb Ġcrawl Ġas Ġthe Ġsource Ġfor Ġour Ġcorpus , Ġit Ġcontains Ġdocuments Ġfrom Ġa Ġvariety Ġof Ġgenres . ĠTo Ġfurther Ġanalyze Ġthis Ġproperty , Ġwe Ġcategorized Ġa Ġsample Ġof Ġ50 Ġdocuments Ġfrom Ġthe Ġcorpus . ĠAmong Ġthem , Ġwe Ġfound Ġprofessionally Ġwritten Ġarticles Ġand Ġblog Ġposts Ġ( 28 %), Ġeducational Ġmaterial Ġfor Ġparents Ġand Ġkids Ġ( 26 %), Ġpersonal Ġblog Ġposts Ġ( 16 %), Ġforum Ġdiscussions Ġand Ġcomments Ġ( 12 %), Ġcommented Ġlink Ġcollections Ġ( 12 %) Ġand Ġscientific Ġarticles Ġ( 6 %). </s> In Ġaddition Ġto Ġthe Ġvariety Ġof Ġgenres , Ġthe Ġdocuments Ġalso Ġdiffer Ġin Ġterms Ġof Ġlanguage Ġuse . ĠTo Ġcapture Ġthis Ġproperty , Ġwe Ġfollow ĠZ op f . 2016 Ġand Ġcompute , Ġfor Ġevery Ġtopic , Ġthe Ġaverage ĠJensen - Sh annon Ġdivergence Ġbetween Ġthe Ġword Ġdistribution Ġof Ġone Ġdocument Ġand Ġthe Ġword Ġdistribution Ġin Ġthe Ġremaining Ġdocuments . ĠThe Ġhigher Ġthis Ġvalue Ġis , Ġthe Ġmore Ġthe Ġlanguage Ġdiffers Ġbetween Ġdocuments . ĠWe Ġfound Ġthe Ġaverage Ġdivergence Ġover Ġall Ġtopics Ġto Ġbe Ġ0 . 34 90 , Ġwhereas Ġit Ġis Ġ0 . 301 9 Ġin ĠD UC Ġ2004 Ġand Ġ0 . 3 188 Ġin ĠT AC Ġ2008 A . </s> Con cept ĠMaps </s> As ĠTable ĠT AB REF 33 Ġshows , Ġeach Ġof Ġthe Ġ30 Ġreference Ġconcept Ġmaps Ġhas Ġexactly Ġ25 Ġconcepts Ġand Ġbetween Ġ24 Ġand Ġ28 Ġrelations . ĠLab els Ġfor Ġboth Ġconcepts Ġand Ġrelations Ġconsist Ġon Ġaverage Ġof Ġ3 . 2 Ġtokens , Ġwhereas Ġthe Ġlatter Ġare Ġa Ġbit Ġshorter Ġin Ġcharacters . </s> To Ġobtain Ġa Ġbetter Ġpicture Ġof Ġwhat Ġkind Ġof Ġtext Ġspans Ġhave Ġbeen Ġused Ġas Ġlabels , Ġwe Ġautomatically Ġtagged Ġthem Ġwith Ġtheir Ġpart - of - speech Ġand Ġdetermined Ġtheir Ġhead Ġwith Ġa Ġdependency Ġparser . ĠConcept Ġlabels Ġtend Ġto Ġbe Ġheaded Ġby Ġnoun s Ġ( 82 %) Ġor Ġverbs Ġ( 15 %), Ġwhile Ġthey Ġalso Ġcontain Ġadject ives , Ġpre pos itions Ġand Ġdetermin ers . ĠRel ation Ġlabels , Ġon Ġthe Ġother Ġhand , Ġare Ġalmost Ġalways Ġheaded Ġby Ġa Ġverb Ġ( 94 %) Ġand Ġcontain Ġpre pos itions , Ġnoun s Ġand Ġparticles Ġin Ġaddition . ĠThese Ġdistributions Ġare Ġvery Ġsimilar Ġto Ġthose Ġreported Ġby ĠVill alon . 2010 Ġfor Ġtheir Ġ( single - document ) Ġconcept Ġmap Ġcorpus . </s> Analy zing Ġthe Ġgraph Ġstructure Ġof Ġthe Ġmaps , Ġwe Ġfound Ġthat Ġall Ġof Ġthem Ġare Ġconnected . ĠThey Ġhave Ġon Ġaverage Ġ7 . 2 Ġcentral Ġconcepts Ġwith Ġmore Ġthan Ġone Ġrelation , Ġwhile Ġthe Ġremaining Ġones Ġoccur Ġin Ġonly Ġone Ġproposition . ĠWe Ġfound Ġthat Ġachieving Ġa Ġhigher Ġnumber Ġof Ġconnections Ġwould Ġmean Ġcompromising Ġimportance , Ġi . e . Ġincluding Ġless Ġimportant Ġpropositions , Ġand Ġdecided Ġagainst Ġit . </s> Bas eline ĠExper iments </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġdescribe Ġa Ġbaseline Ġand Ġevaluation Ġscripts Ġthat Ġwe Ġrelease , Ġwith Ġa Ġdetailed Ġdocumentation , Ġalong Ġwith Ġthe Ġcorpus . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpresented Ġlow - context Ġimportance Ġannotation , Ġa Ġnovel Ġcrowds ourcing Ġscheme Ġthat Ġwe Ġused Ġto Ġcreate Ġa Ġnew Ġbenchmark Ġcorpus Ġfor Ġconcept - map - based ĠM DS . ĠThe Ġcorpus Ġhas Ġlarge - scale Ġdocument Ġclusters Ġof Ġheter ogeneous Ġweb Ġdocuments , Ġposing Ġa Ġchallenging Ġsummar ization Ġtask . ĠTogether Ġwith Ġthe Ġcorpus , Ġwe Ġprovide Ġimplementations Ġof Ġa Ġbaseline Ġmethod Ġand Ġevaluation Ġscripts Ġand Ġhope Ġthat Ġour Ġefforts Ġfacilitate Ġfuture Ġresearch Ġon Ġthis Ġvariant Ġof Ġsummar ization . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank ĠTeresa ĠBot sc hen , ĠAndreas ĠHan sel owski Ġand ĠMarkus ĠZ op f Ġfor Ġtheir Ġhelp Ġwith Ġthe Ġannotation Ġwork Ġand ĠChristian ĠMeyer Ġfor Ġhis Ġvaluable Ġfeedback . ĠThis Ġwork Ġhas Ġbeen Ġsupported Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġas Ġpart Ġof Ġthe ĠResearch ĠTraining ĠGroup ĠâĢ ľ Adapt ive ĠPrepar ation Ġof ĠInformation Ġfrom ĠH eter ogeneous ĠSources âĢ Ŀ Ġ( A IP H ES ) Ġunder Ġgrant ĠNo . ĠGR K Ġ1994 / 1 .
Introduction Multi-document summarization (MDS), the transformation of a set of documents into a short text containing their most important aspects, is a long-studied problem in NLP. Generated summaries have been shown to support humans dealing with large document collections in information seeking tasks BIBREF0 , BIBREF1 , BIBREF2 . However, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves. Instead, user studies BIBREF3 , BIBREF4 show that they note down important keywords and phrases, try to identify relationships between them and organize them accordingly. Therefore, we believe that the study of summarization with similarly structured outputs is an important extension of the traditional task. A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 ). Introduced in 1972 as a teaching tool BIBREF6 , concept maps have found many applications in education BIBREF7 , BIBREF8 , for writing assistance BIBREF9 or to structure information repositories BIBREF10 , BIBREF11 . For summarization, concept maps make it possible to represent a summary concisely and clearly reveal relationships. Moreover, we see a second interesting use case that goes beyond the capabilities of textual summaries: When concepts and relations are linked to corresponding locations in the documents they have been extracted from, the graph can be used to navigate in a document collection, similar to a table of contents. An implementation of this idea has been recently described by BIBREF12 . The corresponding task that we propose is concept-map-based MDS, the summarization of a document cluster in the form of a concept map. In order to develop and evaluate methods for the task, gold-standard corpora are necessary, but no suitable corpus is available. The manual creation of such a dataset is very time-consuming, as the annotation includes many subtasks. In particular, an annotator would need to manually identify all concepts in the documents, while only a few of them will eventually end up in the summary. To overcome these issues, we present a corpus creation method that effectively combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations. Using it, we can avoid the high effort for single annotators, allowing us to scale to document clusters that are 15 times larger than in traditional summarization corpora. We created a new corpus of 30 topics, each with around 40 source documents on educational topics and a summarizing concept map that is the consensus of many crowdworkers (see Figure FIGREF3 ). As a crucial step of the corpus creation, we developed a new crowdsourcing scheme called low-context importance annotation. In contrast to traditional approaches, it allows us to determine important elements in a document cluster without requiring annotators to read all documents, making it feasible to crowdsource the task and overcome quality issues observed in previous work BIBREF13 . We show that the approach creates reliable data for our focused summarization scenario and, when tested on traditional summarization corpora, creates annotations that are similar to those obtained by earlier efforts. To summarize, we make the following contributions: (1) We propose a novel task, concept-map-based MDS (§ SECREF2 ), (2) present a new crowdsourcing scheme to create reference summaries (§ SECREF4 ), (3) publish a new dataset for the proposed task (§ SECREF5 ) and (4) provide an evaluation protocol and baseline (§ SECREF7 ). We make these resources publicly available under a permissive license. Task Concept-map-based MDS is defined as follows: Given a set of related documents, create a concept map that represents its most important content, satisfies a specified size limit and is connected. We define a concept map as a labeled graph showing concepts as nodes and relationships between them as edges. Labels are arbitrary sequences of tokens taken from the documents, making the summarization task extractive. A concept can be an entity, abstract idea, event or activity, designated by its unique label. Good maps should be propositionally coherent, meaning that every relation together with the two connected concepts form a meaningful proposition. The task is complex, consisting of several interdependent subtasks. One has to extract appropriate labels for concepts and relations and recognize different expressions that refer to the same concept across multiple documents. Further, one has to select the most important concepts and relations for the summary and finally organize them in a graph satisfying the connectedness and size constraints. Related Work Some attempts have been made to automatically construct concept maps from text, working with either single documents BIBREF14 , BIBREF9 , BIBREF15 , BIBREF16 or document clusters BIBREF17 , BIBREF18 , BIBREF19 . These approaches extract concept and relation labels from syntactic structures and connect them to build a concept map. However, common task definitions and comparable evaluations are missing. In addition, only a few of them, namely Villalon.2012 and Valerio.2006, define summarization as their goal and try to compress the input to a substantially smaller size. Our newly proposed task and the created large-cluster dataset fill these gaps as they emphasize the summarization aspect of the task. For the subtask of selecting summary-worthy concepts and relations, techniques developed for traditional summarization BIBREF20 and keyphrase extraction BIBREF21 are related and applicable. Approaches that build graphs of propositions to create a summary BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 seem to be particularly related, however, there is one important difference: While they use graphs as an intermediate representation from which a textual summary is then generated, the goal of the proposed task is to create a graph that is directly interpretable and useful for a user. In contrast, these intermediate graphs, e.g. AMR, are hardly useful for a typical, non-linguist user. For traditional summarization, the most well-known datasets emerged out of the DUC and TAC competitions. They provide clusters of news articles with gold-standard summaries. Extending these efforts, several more specialized corpora have been created: With regard to size, Nakano.2010 present a corpus of summaries for large-scale collections of web pages. Recently, corpora with more heterogeneous documents have been suggested, e.g. BIBREF26 and BIBREF27 . The corpus we present combines these aspects, as it has large clusters of heterogeneous documents, and provides a necessary benchmark to evaluate the proposed task. For concept map generation, one corpus with human-created summary concept maps for student essays has been created BIBREF28 . In contrast to our corpus, it only deals with single documents, requires a two orders of magnitude smaller amount of compression of the input and is not publicly available . Other types of information representation that also model concepts and their relationships are knowledge bases, such as Freebase BIBREF29 , and ontologies. However, they both differ in important aspects: Whereas concept maps follow an open label paradigm and are meant to be interpretable by humans, knowledge bases and ontologies are usually more strictly typed and made to be machine-readable. Moreover, approaches to automatically construct them from text typically try to extract as much information as possible, while we want to summarize a document. Low-Context Importance Annotation Lloret.2013 describe several experiments to crowdsource reference summaries. Workers are asked to read 10 documents and then select 10 summary sentences from them for a reward of $0.05. They discovered several challenges, including poor work quality and the subjectiveness of the annotation task, indicating that crowdsourcing is not useful for this purpose. To overcome these issues, we introduce a new task design, low-context importance annotation, to determine summary-worthy parts of documents. Compared to Lloret et al.'s approach, it is more in line with crowdsourcing best practices, as the tasks are simple, intuitive and small BIBREF30 and workers receive reasonable payment BIBREF31 . Most importantly, it is also much more efficient and scalable, as it does not require workers to read all documents in a cluster. Task Design We break down the task of importance annotation to the level of single propositions. The goal of our crowdsourcing scheme is to obtain a score for each proposition indicating its importance in a document cluster, such that a ranking according to the score would reveal what is most important and should be included in a summary. In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster's topic along with the propositions. This ensures that tasks are small, simple and can be done quickly (see Figure FIGREF4 ). In preliminary tests, we found that this design, despite the minimal context, works reasonably on our focused clusters on common educational topics. For instance, consider Figure FIGREF4 : One can easily say that P1 is more important than P2 without reading the documents. We distinguish two task variants: Instead of enforcing binary importance decisions, we use a 5-point Likert-scale to allow more fine-grained annotations. The obtained labels are translated into scores (5..1) and the average of all scores for a proposition is used as an estimate for its importance. This follows the idea that while single workers might find the task subjective, the consensus of multiple workers, represented in the average score, tends to be less subjective due to the “wisdom of the crowd”. We randomly group five propositions into a task. As an alternative, we use a second task design based on pairwise comparisons. Comparisons are known to be easier to make and more consistent BIBREF32 , but also more expensive, as the number of pairs grows quadratically with the number of objects. To reduce the cost, we group five propositions into a task and ask workers to order them by importance per drag-and-drop. From the results, we derive pairwise comparisons and use TrueSkill BIBREF35 , a powerful Bayesian rank induction model BIBREF34 , to obtain importance estimates for each proposition. Pilot Study To verify the proposed approach, we conducted a pilot study on Amazon Mechanical Turk using data from TAC2008 BIBREF36 . We collected importance estimates for 474 propositions extracted from the first three clusters using both task designs. Each Likert-scale task was assigned to 5 different workers and awarded $0.06. For comparison tasks, we also collected 5 labels each, paid $0.05 and sampled around 7% of all possible pairs. We submitted them in batches of 100 pairs and selected pairs for subsequent batches based on the confidence of the TrueSkill model. Following the observations of Lloret.2013, we established several measures for quality control. First, we restricted our tasks to workers from the US with an approval rate of at least 95%. Second, we identified low quality workers by measuring the correlation of each worker's Likert-scores with the average of the other four scores. The worst workers (at most 5% of all labels) were removed. In addition, we included trap sentences, similar as in BIBREF13 , in around 80 of the tasks. In contrast to Lloret et al.'s findings, both an obvious trap sentence (This sentence is not important) and a less obvious but unimportant one (Barack Obama graduated from Harvard Law) were consistently labeled as unimportant (1.08 and 1.14), indicating that the workers did the task properly. For Likert-scale tasks, we follow Snow.2008 and calculate agreement as the average Pearson correlation of a worker's Likert-score with the average score of the remaining workers. This measure is less strict than exact label agreement and can account for close labels and high- or low-scoring workers. We observe a correlation of 0.81, indicating substantial agreement. For comparisons, the majority agreement is 0.73. To further examine the reliability of the collected data, we followed the approach of Kiritchenko.2016 and simply repeated the crowdsourcing for one of the three topics. Between the importance estimates calculated from the first and second run, we found a Pearson correlation of 0.82 (Spearman 0.78) for Likert-scale tasks and 0.69 (Spearman 0.66) for comparison tasks. This shows that the approach, despite the subjectiveness of the task, allows us to collect reliable annotations. In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation. For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains. Table TABREF13 shows how these peer scores, averaged over the three topics, correlate with the manual responsiveness scores assigned during TAC in comparison to ROUGE-2 and Pyramid scores. The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE). Based on the pilot study, we conclude that the proposed crowdsourcing scheme allows us to obtain proper importance annotations for propositions. As workers are not required to read all documents, the annotation is much more efficient and scalable as with traditional methods. Corpus Creation This section presents the corpus construction process, as outlined in Figure FIGREF16 , combining automatic preprocessing, scalable crowdsourcing and high-quality expert annotations to be able to scale to the size of our document clusters. For every topic, we spent about $150 on crowdsourcing and 1.5h of expert annotations, while just a single annotator would already need over 8 hours (at 200 words per minute) to read all documents of a topic. Source Data As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e.g. bullying, homeschooling, drugs) with a short description of each topic. It was created from a large web crawl using state-of-the-art information retrieval. We selected 30 of the topics for which we created the necessary concept map annotations. Proposition Extraction As concept maps consist of propositions expressing the relation between concepts (see Figure FIGREF2 ), we need to impose such a structure upon the plain text in the document clusters. This could be done by manually annotating spans representing concepts and relations, however, the size of our clusters makes this a huge effort: 2288 sentences per topic (69k in total) need to be processed. Therefore, we resort to an automatic approach. The Open Information Extraction paradigm BIBREF38 offers a representation very similar to the desired one. For instance, from Students with bad credit history should not lose hope and apply for federal loans with the FAFSA. Open IE systems extract tuples of two arguments and a relation phrase representing propositions: (s. with bad credit history, should not lose, hope) (s. with bad credit history, apply for, federal loans with the FAFSA) While the relation phrase is similar to a relation in a concept map, many arguments in these tuples represent useful concepts. We used Open IE 4, a state-of-the-art system BIBREF39 to process all sentences. After removing duplicates, we obtained 4137 tuples per topic. Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data. We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones. To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness. On the first 250 of them, we tuned the filter threshold to 0.5, which keeps 98.7% of the correct extractions in the unseen second half. After filtering, a topic had on average 2850 propositions (85k in total). Proposition Filtering Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map. To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones. A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts. We created a guideline explaining when to label a tuple as suitable for a concept map and performed a small annotation study. Three annotators independently labeled 500 randomly sampled tuples. The agreement was 82% ( INLINEFORM0 ). We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in § SECREF21 . Due to the high number of tuples we decided to automate the filtering step. We trained a linear SVM on the majority voted annotations. As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech tags, among others. To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives. In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications. Additionally, we manually verified a certain number of the most uncertain negative classifications to further improve performance. When 20% of the classifications are manually verified and corrected, we found that our model trained on 350 labeled instances achieves 93% precision on negative classifications on the unseen 150 instances. We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset. The classifier filtered out 43% of the propositions, leaving 1622 per topic. We manually examined the 17k least confident negative classifications and corrected 955 of them. We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise. Finally, each topic was left with an average of 1554 propositions (47k in total). Importance Annotation Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in § SECREF4 to determine their importance. To cope with the large number of propositions, we combine the two task designs: First, we collect Likert-scores from 5 workers for each proposition, clean the data and calculate average scores. Then, using only the top 100 propositions according to these scores, we crowdsource 10% of all possible pairwise comparisons among them. Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions. For Likert-scores, the average agreement over all topics is 0.80, while the majority agreement for comparisons is 0.78. We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0.73 (Spearman 0.73) for Likert-scores and 0.72 (Spearman 0.71) for comparisons. These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents. In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425.45 including fees. From the fine-grained ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps. Proposition Revision Having a manageable number of propositions, an annotator then applied a few straightforward transformations that correct common errors of the Open IE system. First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do. And second, we correct span errors that might occur in the argument or relation phrases, especially when sentences were not properly segmented. As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56.1 propositions per topic on average. Concept Map Construction In this final step, we connect the set of important propositions to form a graph. For instance, given the following two propositions (student, may borrow, Stafford Loan) (the student, does not have, a credit history) one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with the concepts student, Stafford Loan and credit history. The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to merge and, when merging, which of the available labels to use. As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks. Annotators were given the topic description and the most important, ranked propositions. Using a simple annotation tool providing a visualization of the graph, they could connect the propositions step by step. They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map BIBREF6 . Further, they should prefer more important propositions and ensure connectedness. When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions. To support the annotators, the tool used ADW BIBREF40 , a state-of-the-art approach for semantic similarity, to suggest possible connections. The annotation was carried out by graduate students with a background in NLP after receiving an introduction into the guidelines and tool and annotating a first example. If an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive. On average, the constructed maps have 0.77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e.g. between Montessori teacher and Montessori education). To assess the reliability of this annotation step, we had the first three maps created by two annotators. We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% ( INLINEFORM0 ). Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% ( INLINEFORM1 ). And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% ( INLINEFORM2 ) agreement. Hence, the annotation shows substantial agreement BIBREF41 . Corpus Analysis In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects. Document Clusters The corpus consists of document clusters for 30 different topics. Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token. With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table TABREF26 ). In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora. With these properties, the corpus represents an interesting challenge towards real-world application scenarios, in which users typically have to deal with much more than ten documents. Because we used a large web crawl as the source for our corpus, it contains documents from a variety of genres. To further analyze this property, we categorized a sample of 50 documents from the corpus. Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%). In addition to the variety of genres, the documents also differ in terms of language use. To capture this property, we follow Zopf.2016 and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents. The higher this value is, the more the language differs between documents. We found the average divergence over all topics to be 0.3490, whereas it is 0.3019 in DUC 2004 and 0.3188 in TAC 2008A. Concept Maps As Table TABREF33 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations. Labels for both concepts and relations consist on average of 3.2 tokens, whereas the latter are a bit shorter in characters. To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser. Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners. Relation labels, on the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition. These distributions are very similar to those reported by Villalon.2010 for their (single-document) concept map corpus. Analyzing the graph structure of the maps, we found that all of them are connected. They have on average 7.2 central concepts with more than one relation, while the remaining ones occur in only one proposition. We found that achieving a higher number of connections would mean compromising importance, i.e. including less important propositions, and decided against it. Baseline Experiments In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus. Conclusion In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS. The corpus has large-scale document clusters of heterogeneous web documents, posing a challenging summarization task. Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization. Acknowledgments We would like to thank Teresa Botschen, Andreas Hanselowski and Markus Zopf for their help with the annotation work and Christian Meyer for his valuable feedback. This work has been supported by the German Research Foundation as part of the Research Training Group “Adaptive Preparation of Information from Heterogeneous Sources” (AIPHES) under grant No. GRK 1994/1.
Which collections of web documents are included in the corpus?
d6191c4643201262a770947fc95a613f57bedb6b
Which Ġcollections Ġof Ġweb Ġdocuments Ġare Ġincluded Ġin Ġthe Ġcorpus ?
D IP Ġcorpus ĠB IB REF 37
<s> Which Ġcollections Ġof Ġweb Ġdocuments Ġare Ġincluded Ġin Ġthe Ġcorpus ? </s> Introduction </s> Multi - document Ġsummar ization Ġ( M DS ), Ġthe Ġtransformation Ġof Ġa Ġset Ġof Ġdocuments Ġinto Ġa Ġshort Ġtext Ġcontaining Ġtheir Ġmost Ġimportant Ġaspects , Ġis Ġa Ġlong - stud ied Ġproblem Ġin ĠN LP . ĠGener ated Ġsumm aries Ġhave Ġbeen Ġshown Ġto Ġsupport Ġhumans Ġdealing Ġwith Ġlarge Ġdocument Ġcollections Ġin Ġinformation Ġseeking Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠHowever , Ġwhen Ġexploring Ġa Ġset Ġof Ġdocuments Ġmanually , Ġhumans Ġrarely Ġwrite Ġa Ġfully - form ulated Ġsummary Ġfor Ġthemselves . ĠInstead , Ġuser Ġstudies ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġshow Ġthat Ġthey Ġnote Ġdown Ġimportant Ġkeywords Ġand Ġphrases , Ġtry Ġto Ġidentify Ġrelationships Ġbetween Ġthem Ġand Ġorganize Ġthem Ġaccordingly . ĠTherefore , Ġwe Ġbelieve Ġthat Ġthe Ġstudy Ġof Ġsummar ization Ġwith Ġsimilarly Ġstructured Ġoutputs Ġis Ġan Ġimportant Ġextension Ġof Ġthe Ġtraditional Ġtask . </s> A Ġrepresentation Ġthat Ġis Ġmore Ġin Ġline Ġwith Ġobserved Ġuser Ġbehavior Ġis Ġa Ġconcept Ġmap ĠB IB REF 5 Ġ, Ġa Ġlabeled Ġgraph Ġshowing Ġconcepts Ġas Ġnodes Ġand Ġrelationships Ġbetween Ġthem Ġas Ġedges Ġ( Figure ĠFIG REF 2 Ġ). ĠIntroduced Ġin Ġ1972 Ġas Ġa Ġteaching Ġtool ĠB IB REF 6 Ġ, Ġconcept Ġmaps Ġhave Ġfound Ġmany Ġapplications Ġin Ġeducation ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġfor Ġwriting Ġassistance ĠB IB REF 9 Ġor Ġto Ġstructure Ġinformation Ġrepositories ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠFor Ġsummar ization , Ġconcept Ġmaps Ġmake Ġit Ġpossible Ġto Ġrepresent Ġa Ġsummary Ġconcise ly Ġand Ġclearly Ġreveal Ġrelationships . ĠMoreover , Ġwe Ġsee Ġa Ġsecond Ġinteresting Ġuse Ġcase Ġthat Ġgoes Ġbeyond Ġthe Ġcapabilities Ġof Ġtextual Ġsumm aries : ĠWhen Ġconcepts Ġand Ġrelations Ġare Ġlinked Ġto Ġcorresponding Ġlocations Ġin Ġthe Ġdocuments Ġthey Ġhave Ġbeen Ġextracted Ġfrom , Ġthe Ġgraph Ġcan Ġbe Ġused Ġto Ġnavigate Ġin Ġa Ġdocument Ġcollection , Ġsimilar Ġto Ġa Ġtable Ġof Ġcontents . ĠAn Ġimplementation Ġof Ġthis Ġidea Ġhas Ġbeen Ġrecently Ġdescribed Ġby ĠB IB REF 12 Ġ. </s> The Ġcorresponding Ġtask Ġthat Ġwe Ġpropose Ġis Ġconcept - map - based ĠM DS , Ġthe Ġsummar ization Ġof Ġa Ġdocument Ġcluster Ġin Ġthe Ġform Ġof Ġa Ġconcept Ġmap . ĠIn Ġorder Ġto Ġdevelop Ġand Ġevaluate Ġmethods Ġfor Ġthe Ġtask , Ġgold - standard Ġcorpor a Ġare Ġnecessary , Ġbut Ġno Ġsuitable Ġcorpus Ġis Ġavailable . ĠThe Ġmanual Ġcreation Ġof Ġsuch Ġa Ġdataset Ġis Ġvery Ġtime - consuming , Ġas Ġthe Ġannotation Ġincludes Ġmany Ġsubt asks . ĠIn Ġparticular , Ġan Ġannot ator Ġwould Ġneed Ġto Ġmanually Ġidentify Ġall Ġconcepts Ġin Ġthe Ġdocuments , Ġwhile Ġonly Ġa Ġfew Ġof Ġthem Ġwill Ġeventually Ġend Ġup Ġin Ġthe Ġsummary . </s> To Ġovercome Ġthese Ġissues , Ġwe Ġpresent Ġa Ġcorpus Ġcreation Ġmethod Ġthat Ġeffectively Ġcombines Ġautomatic Ġpre processing , Ġscalable Ġcrowds ourcing Ġand Ġhigh - quality Ġexpert Ġannotations . ĠUsing Ġit , Ġwe Ġcan Ġavoid Ġthe Ġhigh Ġeffort Ġfor Ġsingle Ġannot ators , Ġallowing Ġus Ġto Ġscale Ġto Ġdocument Ġclusters Ġthat Ġare Ġ15 Ġtimes Ġlarger Ġthan Ġin Ġtraditional Ġsummar ization Ġcorpor a . ĠWe Ġcreated Ġa Ġnew Ġcorpus Ġof Ġ30 Ġtopics , Ġeach Ġwith Ġaround Ġ40 Ġsource Ġdocuments Ġon Ġeducational Ġtopics Ġand Ġa Ġsummar izing Ġconcept Ġmap Ġthat Ġis Ġthe Ġconsensus Ġof Ġmany Ġcrowd workers Ġ( see ĠFigure ĠFIG REF 3 Ġ). </s> As Ġa Ġcrucial Ġstep Ġof Ġthe Ġcorpus Ġcreation , Ġwe Ġdeveloped Ġa Ġnew Ġcrowds ourcing Ġscheme Ġcalled Ġlow - context Ġimportance Ġannotation . ĠIn Ġcontrast Ġto Ġtraditional Ġapproaches , Ġit Ġallows Ġus Ġto Ġdetermine Ġimportant Ġelements Ġin Ġa Ġdocument Ġcluster Ġwithout Ġrequiring Ġannot ators Ġto Ġread Ġall Ġdocuments , Ġmaking Ġit Ġfeasible Ġto Ġcrowd source Ġthe Ġtask Ġand Ġovercome Ġquality Ġissues Ġobserved Ġin Ġprevious Ġwork ĠB IB REF 13 Ġ. ĠWe Ġshow Ġthat Ġthe Ġapproach Ġcreates Ġreliable Ġdata Ġfor Ġour Ġfocused Ġsummar ization Ġscenario Ġand , Ġwhen Ġtested Ġon Ġtraditional Ġsummar ization Ġcorpor a , Ġcreates Ġannotations Ġthat Ġare Ġsimilar Ġto Ġthose Ġobtained Ġby Ġearlier Ġefforts . </s> To Ġsummarize , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions : Ġ( 1 ) ĠWe Ġpropose Ġa Ġnovel Ġtask , Ġconcept - map - based ĠM DS Ġ( Â§ ĠSEC REF 2 Ġ), Ġ( 2 ) Ġpresent Ġa Ġnew Ġcrowds ourcing Ġscheme Ġto Ġcreate Ġreference Ġsumm aries Ġ( Â§ ĠSEC REF 4 Ġ), Ġ( 3 ) Ġpublish Ġa Ġnew Ġdataset Ġfor Ġthe Ġproposed Ġtask Ġ( Â§ ĠSEC REF 5 Ġ) Ġand Ġ( 4 ) Ġprovide Ġan Ġevaluation Ġprotocol Ġand Ġbaseline Ġ( Â§ ĠSEC REF 7 Ġ). ĠWe Ġmake Ġthese Ġresources Ġpublicly Ġavailable Ġunder Ġa Ġper missive Ġlicense . </s> Task </s> Con cept - map - based ĠM DS Ġis Ġdefined Ġas Ġfollows : ĠGiven Ġa Ġset Ġof Ġrelated Ġdocuments , Ġcreate Ġa Ġconcept Ġmap Ġthat Ġrepresents Ġits Ġmost Ġimportant Ġcontent , Ġsatisfies Ġa Ġspecified Ġsize Ġlimit Ġand Ġis Ġconnected . </s> We Ġdefine Ġa Ġconcept Ġmap Ġas Ġa Ġlabeled Ġgraph Ġshowing Ġconcepts Ġas Ġnodes Ġand Ġrelationships Ġbetween Ġthem Ġas Ġedges . ĠLab els Ġare Ġarbitrary Ġsequences Ġof Ġtokens Ġtaken Ġfrom Ġthe Ġdocuments , Ġmaking Ġthe Ġsummar ization Ġtask Ġextract ive . ĠA Ġconcept Ġcan Ġbe Ġan Ġentity , Ġabstract Ġidea , Ġevent Ġor Ġactivity , Ġdesignated Ġby Ġits Ġunique Ġlabel . ĠGood Ġmaps Ġshould Ġbe Ġpropos itionally Ġcoherent , Ġmeaning Ġthat Ġevery Ġrelation Ġtogether Ġwith Ġthe Ġtwo Ġconnected Ġconcepts Ġform Ġa Ġmeaningful Ġproposition . </s> The Ġtask Ġis Ġcomplex , Ġconsisting Ġof Ġseveral Ġinter dependent Ġsubt asks . ĠOne Ġhas Ġto Ġextract Ġappropriate Ġlabels Ġfor Ġconcepts Ġand Ġrelations Ġand Ġrecognize Ġdifferent Ġexpressions Ġthat Ġrefer Ġto Ġthe Ġsame Ġconcept Ġacross Ġmultiple Ġdocuments . ĠFurther , Ġone Ġhas Ġto Ġselect Ġthe Ġmost Ġimportant Ġconcepts Ġand Ġrelations Ġfor Ġthe Ġsummary Ġand Ġfinally Ġorganize Ġthem Ġin Ġa Ġgraph Ġsatisfying Ġthe Ġconnected ness Ġand Ġsize Ġconstraints . </s> Related ĠWork </s> Some Ġattempts Ġhave Ġbeen Ġmade Ġto Ġautomatically Ġconstruct Ġconcept Ġmaps Ġfrom Ġtext , Ġworking Ġwith Ġeither Ġsingle Ġdocuments ĠB IB REF 14 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġor Ġdocument Ġclusters ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ. ĠThese Ġapproaches Ġextract Ġconcept Ġand Ġrelation Ġlabels Ġfrom Ġsynt actic Ġstructures Ġand Ġconnect Ġthem Ġto Ġbuild Ġa Ġconcept Ġmap . ĠHowever , Ġcommon Ġtask Ġdefinitions Ġand Ġcomparable Ġevaluations Ġare Ġmissing . ĠIn Ġaddition , Ġonly Ġa Ġfew Ġof Ġthem , Ġnamely ĠVill alon . 2012 Ġand ĠVal er io . 2006 , Ġdefine Ġsummar ization Ġas Ġtheir Ġgoal Ġand Ġtry Ġto Ġcompress Ġthe Ġinput Ġto Ġa Ġsubstantially Ġsmaller Ġsize . ĠOur Ġnewly Ġproposed Ġtask Ġand Ġthe Ġcreated Ġlarge - cl uster Ġdataset Ġfill Ġthese Ġgaps Ġas Ġthey Ġemphasize Ġthe Ġsummar ization Ġaspect Ġof Ġthe Ġtask . </s> For Ġthe Ġsubt ask Ġof Ġselecting Ġsummary - worthy Ġconcepts Ġand Ġrelations , Ġtechniques Ġdeveloped Ġfor Ġtraditional Ġsummar ization ĠB IB REF 20 Ġand Ġkey phrase Ġextraction ĠB IB REF 21 Ġare Ġrelated Ġand Ġapplicable . ĠAppro aches Ġthat Ġbuild Ġgraphs Ġof Ġpropositions Ġto Ġcreate Ġa Ġsummary ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġseem Ġto Ġbe Ġparticularly Ġrelated , Ġhowever , Ġthere Ġis Ġone Ġimportant Ġdifference : ĠWhile Ġthey Ġuse Ġgraphs Ġas Ġan Ġintermediate Ġrepresentation Ġfrom Ġwhich Ġa Ġtextual Ġsummary Ġis Ġthen Ġgenerated , Ġthe Ġgoal Ġof Ġthe Ġproposed Ġtask Ġis Ġto Ġcreate Ġa Ġgraph Ġthat Ġis Ġdirectly Ġinterpret able Ġand Ġuseful Ġfor Ġa Ġuser . ĠIn Ġcontrast , Ġthese Ġintermediate Ġgraphs , Ġe . g . ĠAM R , Ġare Ġhardly Ġuseful Ġfor Ġa Ġtypical , Ġnon - ling u ist Ġuser . </s> For Ġtraditional Ġsummar ization , Ġthe Ġmost Ġwell - known Ġdatasets Ġemerged Ġout Ġof Ġthe ĠD UC Ġand ĠT AC Ġcompetitions . ĠThey Ġprovide Ġclusters Ġof Ġnews Ġarticles Ġwith Ġgold - standard Ġsumm aries . ĠExt ending Ġthese Ġefforts , Ġseveral Ġmore Ġspecialized Ġcorpor a Ġhave Ġbeen Ġcreated : ĠWith Ġregard Ġto Ġsize , ĠNak ano . 2010 Ġpresent Ġa Ġcorpus Ġof Ġsumm aries Ġfor Ġlarge - scale Ġcollections Ġof Ġweb Ġpages . ĠRecently , Ġcorpor a Ġwith Ġmore Ġheter ogeneous Ġdocuments Ġhave Ġbeen Ġsuggested , Ġe . g . ĠB IB REF 26 Ġand ĠB IB REF 27 Ġ. ĠThe Ġcorpus Ġwe Ġpresent Ġcombines Ġthese Ġaspects , Ġas Ġit Ġhas Ġlarge Ġclusters Ġof Ġheter ogeneous Ġdocuments , Ġand Ġprovides Ġa Ġnecessary Ġbenchmark Ġto Ġevaluate Ġthe Ġproposed Ġtask . </s> For Ġconcept Ġmap Ġgeneration , Ġone Ġcorpus Ġwith Ġhuman - created Ġsummary Ġconcept Ġmaps Ġfor Ġstudent Ġessays Ġhas Ġbeen Ġcreated ĠB IB REF 28 Ġ. ĠIn Ġcontrast Ġto Ġour Ġcorpus , Ġit Ġonly Ġdeals Ġwith Ġsingle Ġdocuments , Ġrequires Ġa Ġtwo Ġorders Ġof Ġmagnitude Ġsmaller Ġamount Ġof Ġcompression Ġof Ġthe Ġinput Ġand Ġis Ġnot Ġpublicly Ġavailable Ġ. </s> Other Ġtypes Ġof Ġinformation Ġrepresentation Ġthat Ġalso Ġmodel Ġconcepts Ġand Ġtheir Ġrelationships Ġare Ġknowledge Ġbases , Ġsuch Ġas ĠFree base ĠB IB REF 29 Ġ, Ġand Ġont ologies . ĠHowever , Ġthey Ġboth Ġdiffer Ġin Ġimportant Ġaspects : ĠWhereas Ġconcept Ġmaps Ġfollow Ġan Ġopen Ġlabel Ġparadigm Ġand Ġare Ġmeant Ġto Ġbe Ġinterpret able Ġby Ġhumans , Ġknowledge Ġbases Ġand Ġont ologies Ġare Ġusually Ġmore Ġstrictly Ġtyped Ġand Ġmade Ġto Ġbe Ġmachine - readable . ĠMoreover , Ġapproaches Ġto Ġautomatically Ġconstruct Ġthem Ġfrom Ġtext Ġtypically Ġtry Ġto Ġextract Ġas Ġmuch Ġinformation Ġas Ġpossible , Ġwhile Ġwe Ġwant Ġto Ġsummarize Ġa Ġdocument . </s> Low - Context ĠImport ance ĠAn notation </s> L l oret . 2013 Ġdescribe Ġseveral Ġexperiments Ġto Ġcrowd source Ġreference Ġsumm aries . ĠWorkers Ġare Ġasked Ġto Ġread Ġ10 Ġdocuments Ġand Ġthen Ġselect Ġ10 Ġsummary Ġsentences Ġfrom Ġthem Ġfor Ġa Ġreward Ġof Ġ$ 0 . 05 . ĠThey Ġdiscovered Ġseveral Ġchallenges , Ġincluding Ġpoor Ġwork Ġquality Ġand Ġthe Ġsubject iveness Ġof Ġthe Ġannotation Ġtask , Ġindicating Ġthat Ġcrowds ourcing Ġis Ġnot Ġuseful Ġfor Ġthis Ġpurpose . </s> To Ġovercome Ġthese Ġissues , Ġwe Ġintroduce Ġa Ġnew Ġtask Ġdesign , Ġlow - context Ġimportance Ġannotation , Ġto Ġdetermine Ġsummary - worthy Ġparts Ġof Ġdocuments . ĠCompared Ġto ĠLl oret Ġet Ġal .' s Ġapproach , Ġit Ġis Ġmore Ġin Ġline Ġwith Ġcrowds ourcing Ġbest Ġpractices , Ġas Ġthe Ġtasks Ġare Ġsimple , Ġintuitive Ġand Ġsmall ĠB IB REF 30 Ġand Ġworkers Ġreceive Ġreasonable Ġpayment ĠB IB REF 31 Ġ. ĠMost Ġimportantly , Ġit Ġis Ġalso Ġmuch Ġmore Ġefficient Ġand Ġscalable , Ġas Ġit Ġdoes Ġnot Ġrequire Ġworkers Ġto Ġread Ġall Ġdocuments Ġin Ġa Ġcluster . </s> Task ĠDesign </s> We Ġbreak Ġdown Ġthe Ġtask Ġof Ġimportance Ġannotation Ġto Ġthe Ġlevel Ġof Ġsingle Ġpropositions . ĠThe Ġgoal Ġof Ġour Ġcrowds ourcing Ġscheme Ġis Ġto Ġobtain Ġa Ġscore Ġfor Ġeach Ġproposition Ġindicating Ġits Ġimportance Ġin Ġa Ġdocument Ġcluster , Ġsuch Ġthat Ġa Ġranking Ġaccording Ġto Ġthe Ġscore Ġwould Ġreveal Ġwhat Ġis Ġmost Ġimportant Ġand Ġshould Ġbe Ġincluded Ġin Ġa Ġsummary . ĠIn Ġcontrast Ġto Ġother Ġwork , Ġwe Ġdo Ġnot Ġshow Ġthe Ġdocuments Ġto Ġthe Ġworkers Ġat Ġall , Ġbut Ġprovide Ġonly Ġa Ġdescription Ġof Ġthe Ġdocument Ġcluster 's Ġtopic Ġalong Ġwith Ġthe Ġpropositions . ĠThis Ġensures Ġthat Ġtasks Ġare Ġsmall , Ġsimple Ġand Ġcan Ġbe Ġdone Ġquickly Ġ( see ĠFigure ĠFIG REF 4 Ġ). </s> In Ġpreliminary Ġtests , Ġwe Ġfound Ġthat Ġthis Ġdesign , Ġdespite Ġthe Ġminimal Ġcontext , Ġworks Ġreasonably Ġon Ġour Ġfocused Ġclusters Ġon Ġcommon Ġeducational Ġtopics . ĠFor Ġinstance , Ġconsider ĠFigure ĠFIG REF 4 Ġ: ĠOne Ġcan Ġeasily Ġsay Ġthat ĠP 1 Ġis Ġmore Ġimportant Ġthan ĠP 2 Ġwithout Ġreading Ġthe Ġdocuments . </s> We Ġdistinguish Ġtwo Ġtask Ġvariants : </s> Instead Ġof Ġenforcing Ġbinary Ġimportance Ġdecisions , Ġwe Ġuse Ġa Ġ5 - point ĠL ik ert - scale Ġto Ġallow Ġmore Ġfine - gr ained Ġannotations . ĠThe Ġobtained Ġlabels Ġare Ġtranslated Ġinto Ġscores Ġ( 5 .. 1 ) Ġand Ġthe Ġaverage Ġof Ġall Ġscores Ġfor Ġa Ġproposition Ġis Ġused Ġas Ġan Ġestimate Ġfor Ġits Ġimportance . ĠThis Ġfollows Ġthe Ġidea Ġthat Ġwhile Ġsingle Ġworkers Ġmight Ġfind Ġthe Ġtask Ġsubjective , Ġthe Ġconsensus Ġof Ġmultiple Ġworkers , Ġrepresented Ġin Ġthe Ġaverage Ġscore , Ġtends Ġto Ġbe Ġless Ġsubjective Ġdue Ġto Ġthe ĠâĢ ľ w isdom Ġof Ġthe Ġcrowd âĢ Ŀ . ĠWe Ġrandomly Ġgroup Ġfive Ġpropositions Ġinto Ġa Ġtask . </s> As Ġan Ġalternative , Ġwe Ġuse Ġa Ġsecond Ġtask Ġdesign Ġbased Ġon Ġpair wise Ġcomparisons . ĠCompar isons Ġare Ġknown Ġto Ġbe Ġeasier Ġto Ġmake Ġand Ġmore Ġconsistent ĠB IB REF 32 Ġ, Ġbut Ġalso Ġmore Ġexpensive , Ġas Ġthe Ġnumber Ġof Ġpairs Ġgrows Ġquad r atically Ġwith Ġthe Ġnumber Ġof Ġobjects . ĠTo Ġreduce Ġthe Ġcost , Ġwe Ġgroup Ġfive Ġpropositions Ġinto Ġa Ġtask Ġand Ġask Ġworkers Ġto Ġorder Ġthem Ġby Ġimportance Ġper Ġdrag - and - drop . ĠFrom Ġthe Ġresults , Ġwe Ġderive Ġpair wise Ġcomparisons Ġand Ġuse ĠTrue Skill ĠB IB REF 35 Ġ, Ġa Ġpowerful ĠBay esian Ġrank Ġinduction Ġmodel ĠB IB REF 34 Ġ, Ġto Ġobtain Ġimportance Ġestimates Ġfor Ġeach Ġproposition . </s> P ilot ĠStudy </s> To Ġverify Ġthe Ġproposed Ġapproach , Ġwe Ġconducted Ġa Ġpilot Ġstudy Ġon ĠAmazon ĠMechanical ĠTurk Ġusing Ġdata Ġfrom ĠT AC 2008 ĠB IB REF 36 Ġ. ĠWe Ġcollected Ġimportance Ġestimates Ġfor Ġ4 74 Ġpropositions Ġextracted Ġfrom Ġthe Ġfirst Ġthree Ġclusters Ġusing Ġboth Ġtask Ġdesigns . ĠEach ĠL ik ert - scale Ġtask Ġwas Ġassigned Ġto Ġ5 Ġdifferent Ġworkers Ġand Ġawarded Ġ$ 0 . 06 . ĠFor Ġcomparison Ġtasks , Ġwe Ġalso Ġcollected Ġ5 Ġlabels Ġeach , Ġpaid Ġ$ 0 . 05 Ġand Ġsampled Ġaround Ġ7 % Ġof Ġall Ġpossible Ġpairs . ĠWe Ġsubmitted Ġthem Ġin Ġbatches Ġof Ġ100 Ġpairs Ġand Ġselected Ġpairs Ġfor Ġsubsequent Ġbatches Ġbased Ġon Ġthe Ġconfidence Ġof Ġthe ĠTrue Skill Ġmodel . </s> Following Ġthe Ġobservations Ġof ĠLl oret . 2013 , Ġwe Ġestablished Ġseveral Ġmeasures Ġfor Ġquality Ġcontrol . ĠFirst , Ġwe Ġrestricted Ġour Ġtasks Ġto Ġworkers Ġfrom Ġthe ĠUS Ġwith Ġan Ġapproval Ġrate Ġof Ġat Ġleast Ġ95 %. ĠSecond , Ġwe Ġidentified Ġlow Ġquality Ġworkers Ġby Ġmeasuring Ġthe Ġcorrelation Ġof Ġeach Ġworker 's ĠL ik ert - sc ores Ġwith Ġthe Ġaverage Ġof Ġthe Ġother Ġfour Ġscores . ĠThe Ġworst Ġworkers Ġ( at Ġmost Ġ5 % Ġof Ġall Ġlabels ) Ġwere Ġremoved . </s> In Ġaddition , Ġwe Ġincluded Ġtrap Ġsentences , Ġsimilar Ġas Ġin ĠB IB REF 13 Ġ, Ġin Ġaround Ġ80 Ġof Ġthe Ġtasks . ĠIn Ġcontrast Ġto ĠLl oret Ġet Ġal .' s Ġfindings , Ġboth Ġan Ġobvious Ġtrap Ġsentence Ġ( This Ġsentence Ġis Ġnot Ġimportant ) Ġand Ġa Ġless Ġobvious Ġbut Ġun important Ġone Ġ( Bar ack ĠObama Ġgraduated Ġfrom ĠHarvard ĠLaw ) Ġwere Ġconsistently Ġlabeled Ġas Ġun important Ġ( 1 . 08 Ġand Ġ1 . 14 ), Ġindicating Ġthat Ġthe Ġworkers Ġdid Ġthe Ġtask Ġproperly . </s> For ĠL ik ert - scale Ġtasks , Ġwe Ġfollow ĠSnow . 2008 Ġand Ġcalculate Ġagreement Ġas Ġthe Ġaverage ĠPearson Ġcorrelation Ġof Ġa Ġworker 's ĠL ik ert - score Ġwith Ġthe Ġaverage Ġscore Ġof Ġthe Ġremaining Ġworkers . ĠThis Ġmeasure Ġis Ġless Ġstrict Ġthan Ġexact Ġlabel Ġagreement Ġand Ġcan Ġaccount Ġfor Ġclose Ġlabels Ġand Ġhigh - Ġor Ġlow - scoring Ġworkers . ĠWe Ġobserve Ġa Ġcorrelation Ġof Ġ0 . 81 , Ġindicating Ġsubstantial Ġagreement . ĠFor Ġcomparisons , Ġthe Ġmajority Ġagreement Ġis Ġ0 . 73 . ĠTo Ġfurther Ġexamine Ġthe Ġreliability Ġof Ġthe Ġcollected Ġdata , Ġwe Ġfollowed Ġthe Ġapproach Ġof ĠK irit chenko . 2016 Ġand Ġsimply Ġrepeated Ġthe Ġcrowds ourcing Ġfor Ġone Ġof Ġthe Ġthree Ġtopics . ĠBetween Ġthe Ġimportance Ġestimates Ġcalculated Ġfrom Ġthe Ġfirst Ġand Ġsecond Ġrun , Ġwe Ġfound Ġa ĠPearson Ġcorrelation Ġof Ġ0 . 82 Ġ( Spe ar man Ġ0 . 78 ) Ġfor ĠL ik ert - scale Ġtasks Ġand Ġ0 . 69 Ġ( Spe ar man Ġ0 . 66 ) Ġfor Ġcomparison Ġtasks . ĠThis Ġshows Ġthat Ġthe Ġapproach , Ġdespite Ġthe Ġsubject iveness Ġof Ġthe Ġtask , Ġallows Ġus Ġto Ġcollect Ġreliable Ġannotations . </s> In Ġaddition Ġto Ġthe Ġreliability Ġstudies , Ġwe Ġextr ins ically Ġevaluated Ġthe Ġannotations Ġin Ġthe Ġtask Ġof Ġsummary Ġevaluation . ĠFor Ġeach Ġof Ġthe Ġ58 Ġpeer Ġsumm aries Ġin ĠT AC 2008 , Ġwe Ġcalculated Ġa Ġscore Ġas Ġthe Ġsum Ġof Ġthe Ġimportance Ġestimates Ġof Ġthe Ġpropositions Ġit Ġcontains . ĠTable ĠT AB REF 13 Ġshows Ġhow Ġthese Ġpeer Ġscores , Ġaveraged Ġover Ġthe Ġthree Ġtopics , Ġcorrelate Ġwith Ġthe Ġmanual Ġresponsiveness Ġscores Ġassigned Ġduring ĠT AC Ġin Ġcomparison Ġto ĠR OU GE - 2 Ġand ĠPyramid Ġscores . ĠThe Ġresults Ġdemonstrate Ġthat Ġwith Ġboth Ġtask Ġdesigns , Ġwe Ġobtain Ġimportance Ġannotations Ġthat Ġare Ġsimilarly Ġuseful Ġfor Ġsummary Ġevaluation Ġas Ġpyramid Ġannotations Ġor Ġgold - standard Ġsumm aries Ġ( used Ġfor ĠR OU GE ). </s> Based Ġon Ġthe Ġpilot Ġstudy , Ġwe Ġconclude Ġthat Ġthe Ġproposed Ġcrowds ourcing Ġscheme Ġallows Ġus Ġto Ġobtain Ġproper Ġimportance Ġannotations Ġfor Ġpropositions . ĠAs Ġworkers Ġare Ġnot Ġrequired Ġto Ġread Ġall Ġdocuments , Ġthe Ġannotation Ġis Ġmuch Ġmore Ġefficient Ġand Ġscalable Ġas Ġwith Ġtraditional Ġmethods . </s> Corp us ĠCreation </s> This Ġsection Ġpresents Ġthe Ġcorpus Ġconstruction Ġprocess , Ġas Ġoutlined Ġin ĠFigure ĠFIG REF 16 Ġ, Ġcombining Ġautomatic Ġpre processing , Ġscalable Ġcrowds ourcing Ġand Ġhigh - quality Ġexpert Ġannotations Ġto Ġbe Ġable Ġto Ġscale Ġto Ġthe Ġsize Ġof Ġour Ġdocument Ġclusters . ĠFor Ġevery Ġtopic , Ġwe Ġspent Ġabout Ġ$ 150 Ġon Ġcrowds ourcing Ġand Ġ1 . 5 h Ġof Ġexpert Ġannotations , Ġwhile Ġjust Ġa Ġsingle Ġannot ator Ġwould Ġalready Ġneed Ġover Ġ8 Ġhours Ġ( at Ġ200 Ġwords Ġper Ġminute ) Ġto Ġread Ġall Ġdocuments Ġof Ġa Ġtopic . </s> Source ĠData </s> As Ġa Ġstarting Ġpoint , Ġwe Ġused Ġthe ĠD IP Ġcorpus ĠB IB REF 37 Ġ, Ġa Ġcollection Ġof Ġ49 Ġclusters Ġof Ġ100 Ġweb Ġpages Ġon Ġeducational Ġtopics Ġ( e . g . Ġbullying , Ġhomes chool ing , Ġdrugs ) Ġwith Ġa Ġshort Ġdescription Ġof Ġeach Ġtopic . ĠIt Ġwas Ġcreated Ġfrom Ġa Ġlarge Ġweb Ġcrawl Ġusing Ġstate - of - the - art Ġinformation Ġretrieval . ĠWe Ġselected Ġ30 Ġof Ġthe Ġtopics Ġfor Ġwhich Ġwe Ġcreated Ġthe Ġnecessary Ġconcept Ġmap Ġannotations . </s> Prop osition ĠExt raction </s> As Ġconcept Ġmaps Ġconsist Ġof Ġpropositions Ġexpressing Ġthe Ġrelation Ġbetween Ġconcepts Ġ( see ĠFigure ĠFIG REF 2 Ġ), Ġwe Ġneed Ġto Ġimpose Ġsuch Ġa Ġstructure Ġupon Ġthe Ġplain Ġtext Ġin Ġthe Ġdocument Ġclusters . ĠThis Ġcould Ġbe Ġdone Ġby Ġmanually Ġannot ating Ġspans Ġrepresenting Ġconcepts Ġand Ġrelations , Ġhowever , Ġthe Ġsize Ġof Ġour Ġclusters Ġmakes Ġthis Ġa Ġhuge Ġeffort : Ġ2 288 Ġsentences Ġper Ġtopic Ġ( 69 k Ġin Ġtotal ) Ġneed Ġto Ġbe Ġprocessed . ĠTherefore , Ġwe Ġresort Ġto Ġan Ġautomatic Ġapproach . </s> The ĠOpen ĠInformation ĠExt raction Ġparadigm ĠB IB REF 38 Ġoffers Ġa Ġrepresentation Ġvery Ġsimilar Ġto Ġthe Ġdesired Ġone . ĠFor Ġinstance , Ġfrom </s> Students Ġwith Ġbad Ġcredit Ġhistory Ġshould Ġnot Ġlose Ġhope Ġand Ġapply Ġfor Ġfederal Ġloans Ġwith Ġthe ĠF AF SA . </s> Open ĠIE Ġsystems Ġextract Ġtu ples Ġof Ġtwo Ġarguments Ġand Ġa Ġrelation Ġphrase Ġrepresenting Ġpropositions : </s> ( s . Ġwith Ġbad Ġcredit Ġhistory , Ġshould Ġnot Ġlose , Ġhope ) </s> ( s . Ġwith Ġbad Ġcredit Ġhistory , Ġapply Ġfor , Ġfederal Ġloans Ġwith Ġthe ĠF AF SA ) </s> While Ġthe Ġrelation Ġphrase Ġis Ġsimilar Ġto Ġa Ġrelation Ġin Ġa Ġconcept Ġmap , Ġmany Ġarguments Ġin Ġthese Ġtu ples Ġrepresent Ġuseful Ġconcepts . ĠWe Ġused ĠOpen ĠIE Ġ4 , Ġa Ġstate - of - the - art Ġsystem ĠB IB REF 39 Ġto Ġprocess Ġall Ġsentences . ĠAfter Ġremoving Ġdupl icates , Ġwe Ġobtained Ġ4 137 Ġtu ples Ġper Ġtopic . </s> Since Ġwe Ġwant Ġto Ġcreate Ġa Ġgold - standard Ġcorpus , Ġwe Ġhave Ġto Ġensure Ġthat Ġwe Ġproduce Ġhigh - quality Ġdata . ĠWe Ġtherefore Ġmade Ġuse Ġof Ġthe Ġconfidence Ġassigned Ġto Ġevery Ġextracted Ġtuple Ġto Ġfilter Ġout Ġlow Ġquality Ġones . ĠTo Ġensure Ġthat Ġwe Ġdo Ġnot Ġfilter Ġtoo Ġaggressively Ġ( and Ġmiss Ġimportant Ġaspects Ġin Ġthe Ġfinal Ġsummary ), Ġwe Ġmanually Ġannot ated Ġ500 Ġtu ples Ġsampled Ġfrom Ġall Ġtopics Ġfor Ġcorrectness . ĠOn Ġthe Ġfirst Ġ250 Ġof Ġthem , Ġwe Ġtuned Ġthe Ġfilter Ġthreshold Ġto Ġ0 . 5 , Ġwhich Ġkeeps Ġ98 . 7 % Ġof Ġthe Ġcorrect Ġextract ions Ġin Ġthe Ġunseen Ġsecond Ġhalf . ĠAfter Ġfiltering , Ġa Ġtopic Ġhad Ġon Ġaverage Ġ28 50 Ġpropositions Ġ( 85 k Ġin Ġtotal ). </s> Prop osition ĠFil tering </s> Despite Ġthe Ġsimilarity Ġof Ġthe ĠOpen ĠIE Ġparadigm , Ġnot Ġevery Ġextracted Ġtuple Ġis Ġa Ġsuitable Ġproposition Ġfor Ġa Ġconcept Ġmap . ĠTo Ġreduce Ġthe Ġeffort Ġin Ġthe Ġsubsequent Ġsteps , Ġwe Ġtherefore Ġwant Ġto Ġfilter Ġout Ġunsu itable Ġones . ĠA Ġtuple Ġis Ġsuitable Ġif Ġit Ġ( 1 ) Ġis Ġa Ġcorrect Ġextraction , Ġ( 2 ) Ġis Ġmeaningful Ġwithout Ġany Ġcontext Ġand Ġ( 3 ) Ġhas Ġarguments Ġthat Ġrepresent Ġproper Ġconcepts . ĠWe Ġcreated Ġa Ġguideline Ġexplaining Ġwhen Ġto Ġlabel Ġa Ġtuple Ġas Ġsuitable Ġfor Ġa Ġconcept Ġmap Ġand Ġperformed Ġa Ġsmall Ġannotation Ġstudy . ĠThree Ġannot ators Ġindependently Ġlabeled Ġ500 Ġrandomly Ġsampled Ġtu ples . ĠThe Ġagreement Ġwas Ġ82 % Ġ( ĠIN LINE FORM 0 Ġ). ĠWe Ġfound Ġtu ples Ġto Ġbe Ġunsu itable Ġmostly Ġbecause Ġthey Ġhad Ġunres ol vable Ġpronouns , Ġconflicting Ġwith Ġ( 2 ), Ġor Ġarguments Ġthat Ġwere Ġfull Ġclauses Ġor Ġpropositions , Ġconflicting Ġwith Ġ( 3 ), Ġwhile Ġ( 1 ) Ġwas Ġmostly Ġtaken Ġcare Ġof Ġby Ġthe Ġconfidence Ġfiltering Ġin ĠÂ§ ĠSEC REF 21 Ġ. </s> Due Ġto Ġthe Ġhigh Ġnumber Ġof Ġtu ples Ġwe Ġdecided Ġto Ġautomate Ġthe Ġfiltering Ġstep . ĠWe Ġtrained Ġa Ġlinear ĠS VM Ġon Ġthe Ġmajority Ġvoted Ġannotations . ĠAs Ġfeatures , Ġwe Ġused Ġthe Ġextraction Ġconfidence , Ġlength Ġof Ġarguments Ġand Ġrelations Ġas Ġwell Ġas Ġpart - of - speech Ġtags , Ġamong Ġothers . ĠTo Ġensure Ġthat Ġthe Ġautomatic Ġclassification Ġdoes Ġnot Ġremove Ġsuitable Ġpropositions , Ġwe Ġtuned Ġthe Ġclass ifier Ġto Ġavoid Ġfalse Ġnegatives . ĠIn Ġparticular , Ġwe Ġintroduced Ġclass Ġweights , Ġimproving Ġprecision Ġon Ġthe Ġnegative Ġclass Ġat Ġthe Ġcost Ġof Ġa Ġhigher Ġfraction Ġof Ġpositive Ġclass ifications . ĠAdditionally , Ġwe Ġmanually Ġverified Ġa Ġcertain Ġnumber Ġof Ġthe Ġmost Ġuncertain Ġnegative Ġclass ifications Ġto Ġfurther Ġimprove Ġperformance . ĠWhen Ġ20 % Ġof Ġthe Ġclass ifications Ġare Ġmanually Ġverified Ġand Ġcorrected , Ġwe Ġfound Ġthat Ġour Ġmodel Ġtrained Ġon Ġ350 Ġlabeled Ġinstances Ġachieves Ġ93 % Ġprecision Ġon Ġnegative Ġclass ifications Ġon Ġthe Ġunseen Ġ150 Ġinstances . ĠWe Ġfound Ġthis Ġto Ġbe Ġa Ġreasonable Ġtrade - off Ġof Ġautomation Ġand Ġdata Ġquality Ġand Ġapplied Ġthe Ġmodel Ġto Ġthe Ġfull Ġdataset . </s> The Ġclass ifier Ġfiltered Ġout Ġ43 % Ġof Ġthe Ġpropositions , Ġleaving Ġ16 22 Ġper Ġtopic . ĠWe Ġmanually Ġexamined Ġthe Ġ17 k Ġleast Ġconfident Ġnegative Ġclass ifications Ġand Ġcorrected Ġ9 55 Ġof Ġthem . ĠWe Ġalso Ġcorrected Ġpositive Ġclass ifications Ġfor Ġcertain Ġtypes Ġof Ġtu ples Ġfor Ġwhich Ġwe Ġknew Ġthe Ġclass ifier Ġto Ġbe Ġimp rec ise . ĠFinally , Ġeach Ġtopic Ġwas Ġleft Ġwith Ġan Ġaverage Ġof Ġ15 54 Ġpropositions Ġ( 47 k Ġin Ġtotal ). </s> Import ance ĠAn notation </s> Given Ġthe Ġpropositions Ġidentified Ġin Ġthe Ġprevious Ġstep , Ġwe Ġnow Ġapplied Ġour Ġcrowds ourcing Ġscheme Ġas Ġdescribed Ġin ĠÂ§ ĠSEC REF 4 Ġto Ġdetermine Ġtheir Ġimportance . ĠTo Ġcope Ġwith Ġthe Ġlarge Ġnumber Ġof Ġpropositions , Ġwe Ġcombine Ġthe Ġtwo Ġtask Ġdesigns : ĠFirst , Ġwe Ġcollect ĠL ik ert - sc ores Ġfrom Ġ5 Ġworkers Ġfor Ġeach Ġproposition , Ġclean Ġthe Ġdata Ġand Ġcalculate Ġaverage Ġscores . ĠThen , Ġusing Ġonly Ġthe Ġtop Ġ100 Ġpropositions Ġaccording Ġto Ġthese Ġscores , Ġwe Ġcrowd source Ġ10 % Ġof Ġall Ġpossible Ġpair wise Ġcomparisons Ġamong Ġthem . ĠUsing ĠTrue Skill , Ġwe Ġobtain Ġa Ġfine - gr ained Ġranking Ġof Ġthe Ġ100 Ġmost Ġimportant Ġpropositions . </s> For ĠL ik ert - sc ores , Ġthe Ġaverage Ġagreement Ġover Ġall Ġtopics Ġis Ġ0 . 80 , Ġwhile Ġthe Ġmajority Ġagreement Ġfor Ġcomparisons Ġis Ġ0 . 78 . ĠWe Ġrepeated Ġthe Ġdata Ġcollection Ġfor Ġthree Ġrandomly Ġselected Ġtopics Ġand Ġfound Ġthe ĠPearson Ġcorrelation Ġbetween Ġboth Ġruns Ġto Ġbe Ġ0 . 73 Ġ( Spe ar man Ġ0 . 73 ) Ġfor ĠL ik ert - sc ores Ġand Ġ0 . 72 Ġ( Spe ar man Ġ0 . 71 ) Ġfor Ġcomparisons . ĠThese Ġfigures Ġshow Ġthat Ġthe Ġcrowds ourcing Ġapproach Ġworks Ġon Ġthis Ġdataset Ġas Ġreliably Ġas Ġon Ġthe ĠT AC Ġdocuments . </s> In Ġtotal , Ġwe Ġuploaded Ġ53 k Ġscoring Ġand Ġ12 k Ġcomparison Ġtasks Ġto ĠMechanical ĠTurk , Ġspending Ġ$ 44 25 . 45 Ġincluding Ġfees . ĠFrom Ġthe Ġfine - gr ained Ġranking Ġof Ġthe Ġ100 Ġmost Ġimportant Ġpropositions , Ġwe Ġselect Ġthe Ġtop Ġ50 Ġper Ġtopic Ġto Ġconstruct Ġa Ġsummary Ġconcept Ġmap Ġin Ġthe Ġsubsequent Ġsteps . </s> Prop osition ĠRevision </s> Having Ġa Ġmanageable Ġnumber Ġof Ġpropositions , Ġan Ġannot ator Ġthen Ġapplied Ġa Ġfew Ġstraightforward Ġtransformations Ġthat Ġcorrect Ġcommon Ġerrors Ġof Ġthe ĠOpen ĠIE Ġsystem . ĠFirst , Ġwe Ġbreak Ġdown Ġpropositions Ġwith Ġconj unctions Ġin Ġeither Ġof Ġthe Ġarguments Ġinto Ġseparate Ġpropositions Ġper Ġconj unct , Ġwhich Ġthe ĠOpen ĠIE Ġsystem Ġsometimes Ġfails Ġto Ġdo . ĠAnd Ġsecond , Ġwe Ġcorrect Ġspan Ġerrors Ġthat Ġmight Ġoccur Ġin Ġthe Ġargument Ġor Ġrelation Ġphrases , Ġespecially Ġwhen Ġsentences Ġwere Ġnot Ġproperly Ġsegment ed . ĠAs Ġa Ġresult , Ġwe Ġhave Ġa Ġset Ġof Ġhigh Ġquality Ġpropositions Ġfor Ġour Ġconcept Ġmap , Ġconsisting Ġof , Ġdue Ġto Ġthe Ġfirst Ġtransformation , Ġ56 . 1 Ġpropositions Ġper Ġtopic Ġon Ġaverage . </s> Con cept ĠMap ĠConstruction </s> In Ġthis Ġfinal Ġstep , Ġwe Ġconnect Ġthe Ġset Ġof Ġimportant Ġpropositions Ġto Ġform Ġa Ġgraph . ĠFor Ġinstance , Ġgiven Ġthe Ġfollowing Ġtwo Ġpropositions </s> ( student , Ġmay Ġborrow , ĠStafford ĠLoan ) </s> ( the Ġstudent , Ġdoes Ġnot Ġhave , Ġa Ġcredit Ġhistory ) </s> one Ġcan Ġeasily Ġsee , Ġalthough Ġthe Ġfirst Ġarguments Ġdiffer Ġslightly , Ġthat Ġboth Ġlabels Ġdescribe Ġthe Ġconcept Ġstudent , Ġallowing Ġus Ġto Ġbuild Ġa Ġconcept Ġmap Ġwith Ġthe Ġconcepts Ġstudent , ĠStafford ĠLoan Ġand Ġcredit Ġhistory . ĠThe Ġannotation Ġtask Ġthus Ġinvolves Ġdeciding Ġwhich Ġof Ġthe Ġavailable Ġpropositions Ġto Ġinclude Ġin Ġthe Ġmap , Ġwhich Ġof Ġtheir Ġconcepts Ġto Ġmerge Ġand , Ġwhen Ġmerging , Ġwhich Ġof Ġthe Ġavailable Ġlabels Ġto Ġuse . ĠAs Ġthese Ġdecisions Ġhighly Ġdepend Ġupon Ġeach Ġother Ġand Ġrequire Ġcontext , Ġwe Ġdecided Ġto Ġuse Ġexpert Ġannot ators Ġrather Ġthan Ġcrowd source Ġthe Ġsubt asks . </s> An not ators Ġwere Ġgiven Ġthe Ġtopic Ġdescription Ġand Ġthe Ġmost Ġimportant , Ġranked Ġpropositions . ĠUsing Ġa Ġsimple Ġannotation Ġtool Ġproviding Ġa Ġvisualization Ġof Ġthe Ġgraph , Ġthey Ġcould Ġconnect Ġthe Ġpropositions Ġstep Ġby Ġstep . ĠThey Ġwere Ġinstructed Ġto Ġreach Ġa Ġsize Ġof Ġ25 Ġconcepts , Ġthe Ġrecommended Ġmaximum Ġsize Ġfor Ġa Ġconcept Ġmap ĠB IB REF 6 Ġ. ĠFurther , Ġthey Ġshould Ġprefer Ġmore Ġimportant Ġpropositions Ġand Ġensure Ġconnected ness . ĠWhen Ġconnecting Ġtwo Ġpropositions , Ġthey Ġwere Ġasked Ġto Ġkeep Ġthe Ġconcept Ġlabel Ġthat Ġwas Ġappropriate Ġfor Ġboth Ġpropositions . ĠTo Ġsupport Ġthe Ġannot ators , Ġthe Ġtool Ġused ĠAD W ĠB IB REF 40 Ġ, Ġa Ġstate - of - the - art Ġapproach Ġfor Ġsemantic Ġsimilarity , Ġto Ġsuggest Ġpossible Ġconnections . ĠThe Ġannotation Ġwas Ġcarried Ġout Ġby Ġgraduate Ġstudents Ġwith Ġa Ġbackground Ġin ĠN LP Ġafter Ġreceiving Ġan Ġintroduction Ġinto Ġthe Ġguidelines Ġand Ġtool Ġand Ġannot ating Ġa Ġfirst Ġexample . </s> If Ġan Ġannot ator Ġwas Ġnot Ġable Ġto Ġconnect Ġ25 Ġconcepts , Ġshe Ġwas Ġallowed Ġto Ġcreate Ġup Ġto Ġthree Ġsynthetic Ġrelations Ġwith Ġfreely Ġdefined Ġlabels , Ġmaking Ġthe Ġmaps Ġslightly Ġabstract ive . ĠOn Ġaverage , Ġthe Ġconstructed Ġmaps Ġhave Ġ0 . 77 Ġsynthetic Ġrelations , Ġmostly Ġconnecting Ġconcepts Ġwhose Ġrelation Ġis Ġtoo Ġobvious Ġto Ġbe Ġexplicitly Ġstated Ġin Ġtext Ġ( e . g . Ġbetween ĠMont essor i Ġteacher Ġand ĠMont essor i Ġeducation ). </s> To Ġassess Ġthe Ġreliability Ġof Ġthis Ġannotation Ġstep , Ġwe Ġhad Ġthe Ġfirst Ġthree Ġmaps Ġcreated Ġby Ġtwo Ġannot ators . ĠWe Ġcast ed Ġthe Ġtask Ġof Ġselecting Ġpropositions Ġto Ġbe Ġincluded Ġin Ġthe Ġmap Ġas Ġa Ġbinary Ġdecision Ġtask Ġand Ġobserved Ġan Ġagreement Ġof Ġ84 % Ġ( ĠIN LINE FORM 0 Ġ). ĠSecond , Ġwe Ġmodeled Ġthe Ġdecision Ġwhich Ġconcepts Ġto Ġjoin Ġas Ġa Ġbinary Ġdecision Ġon Ġall Ġpairs Ġof Ġcommon Ġconcepts , Ġobserving Ġan Ġagreement Ġof Ġ95 % Ġ( ĠIN LINE FORM 1 Ġ). ĠAnd Ġfinally , Ġwe Ġcompared Ġwhich Ġconcept Ġlabels Ġthe Ġannot ators Ġdecided Ġto Ġinclude Ġin Ġthe Ġfinal Ġmap , Ġobserving Ġ85 % Ġ( ĠIN LINE FORM 2 Ġ) Ġagreement . ĠHence , Ġthe Ġannotation Ġshows Ġsubstantial Ġagreement ĠB IB REF 41 Ġ. </s> Corp us ĠAnalysis </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġnewly Ġcreated Ġcorpus , Ġwhich , Ġin Ġaddition Ġto Ġhaving Ġsumm aries Ġin Ġthe Ġform Ġof Ġconcept Ġmaps , Ġdiffers Ġfrom Ġtraditional Ġsummar ization Ġcorpor a Ġin Ġseveral Ġaspects . </s> Document ĠCl usters </s> The Ġcorpus Ġconsists Ġof Ġdocument Ġclusters Ġfor Ġ30 Ġdifferent Ġtopics . ĠEach Ġof Ġthem Ġcontains Ġaround Ġ40 Ġdocuments Ġwith Ġon Ġaverage Ġ24 13 Ġtokens , Ġwhich Ġleads Ġto Ġan Ġaverage Ġcluster Ġsize Ġof Ġ97 , 880 Ġtoken . ĠWith Ġthese Ġcharacteristics , Ġthe Ġdocument Ġclusters Ġare Ġ15 Ġtimes Ġlarger Ġthan Ġtypical ĠD UC Ġclusters Ġof Ġten Ġdocuments Ġand Ġfive Ġtimes Ġlarger Ġthan Ġthe Ġ25 - document - cl usters Ġ( Table ĠT AB REF 26 Ġ). ĠIn Ġaddition , Ġthe Ġdocuments Ġare Ġalso Ġmore Ġvariable Ġin Ġterms Ġof Ġlength , Ġas Ġthe Ġ( length - adjusted ) Ġstandard Ġdeviation Ġis Ġtwice Ġas Ġhigh Ġas Ġin Ġthe Ġother Ġcorpor a . ĠWith Ġthese Ġproperties , Ġthe Ġcorpus Ġrepresents Ġan Ġinteresting Ġchallenge Ġtowards Ġreal - world Ġapplication Ġscenarios , Ġin Ġwhich Ġusers Ġtypically Ġhave Ġto Ġdeal Ġwith Ġmuch Ġmore Ġthan Ġten Ġdocuments . </s> Because Ġwe Ġused Ġa Ġlarge Ġweb Ġcrawl Ġas Ġthe Ġsource Ġfor Ġour Ġcorpus , Ġit Ġcontains Ġdocuments Ġfrom Ġa Ġvariety Ġof Ġgenres . ĠTo Ġfurther Ġanalyze Ġthis Ġproperty , Ġwe Ġcategorized Ġa Ġsample Ġof Ġ50 Ġdocuments Ġfrom Ġthe Ġcorpus . ĠAmong Ġthem , Ġwe Ġfound Ġprofessionally Ġwritten Ġarticles Ġand Ġblog Ġposts Ġ( 28 %), Ġeducational Ġmaterial Ġfor Ġparents Ġand Ġkids Ġ( 26 %), Ġpersonal Ġblog Ġposts Ġ( 16 %), Ġforum Ġdiscussions Ġand Ġcomments Ġ( 12 %), Ġcommented Ġlink Ġcollections Ġ( 12 %) Ġand Ġscientific Ġarticles Ġ( 6 %). </s> In Ġaddition Ġto Ġthe Ġvariety Ġof Ġgenres , Ġthe Ġdocuments Ġalso Ġdiffer Ġin Ġterms Ġof Ġlanguage Ġuse . ĠTo Ġcapture Ġthis Ġproperty , Ġwe Ġfollow ĠZ op f . 2016 Ġand Ġcompute , Ġfor Ġevery Ġtopic , Ġthe Ġaverage ĠJensen - Sh annon Ġdivergence Ġbetween Ġthe Ġword Ġdistribution Ġof Ġone Ġdocument Ġand Ġthe Ġword Ġdistribution Ġin Ġthe Ġremaining Ġdocuments . ĠThe Ġhigher Ġthis Ġvalue Ġis , Ġthe Ġmore Ġthe Ġlanguage Ġdiffers Ġbetween Ġdocuments . ĠWe Ġfound Ġthe Ġaverage Ġdivergence Ġover Ġall Ġtopics Ġto Ġbe Ġ0 . 34 90 , Ġwhereas Ġit Ġis Ġ0 . 301 9 Ġin ĠD UC Ġ2004 Ġand Ġ0 . 3 188 Ġin ĠT AC Ġ2008 A . </s> Con cept ĠMaps </s> As ĠTable ĠT AB REF 33 Ġshows , Ġeach Ġof Ġthe Ġ30 Ġreference Ġconcept Ġmaps Ġhas Ġexactly Ġ25 Ġconcepts Ġand Ġbetween Ġ24 Ġand Ġ28 Ġrelations . ĠLab els Ġfor Ġboth Ġconcepts Ġand Ġrelations Ġconsist Ġon Ġaverage Ġof Ġ3 . 2 Ġtokens , Ġwhereas Ġthe Ġlatter Ġare Ġa Ġbit Ġshorter Ġin Ġcharacters . </s> To Ġobtain Ġa Ġbetter Ġpicture Ġof Ġwhat Ġkind Ġof Ġtext Ġspans Ġhave Ġbeen Ġused Ġas Ġlabels , Ġwe Ġautomatically Ġtagged Ġthem Ġwith Ġtheir Ġpart - of - speech Ġand Ġdetermined Ġtheir Ġhead Ġwith Ġa Ġdependency Ġparser . ĠConcept Ġlabels Ġtend Ġto Ġbe Ġheaded Ġby Ġnoun s Ġ( 82 %) Ġor Ġverbs Ġ( 15 %), Ġwhile Ġthey Ġalso Ġcontain Ġadject ives , Ġpre pos itions Ġand Ġdetermin ers . ĠRel ation Ġlabels , Ġon Ġthe Ġother Ġhand , Ġare Ġalmost Ġalways Ġheaded Ġby Ġa Ġverb Ġ( 94 %) Ġand Ġcontain Ġpre pos itions , Ġnoun s Ġand Ġparticles Ġin Ġaddition . ĠThese Ġdistributions Ġare Ġvery Ġsimilar Ġto Ġthose Ġreported Ġby ĠVill alon . 2010 Ġfor Ġtheir Ġ( single - document ) Ġconcept Ġmap Ġcorpus . </s> Analy zing Ġthe Ġgraph Ġstructure Ġof Ġthe Ġmaps , Ġwe Ġfound Ġthat Ġall Ġof Ġthem Ġare Ġconnected . ĠThey Ġhave Ġon Ġaverage Ġ7 . 2 Ġcentral Ġconcepts Ġwith Ġmore Ġthan Ġone Ġrelation , Ġwhile Ġthe Ġremaining Ġones Ġoccur Ġin Ġonly Ġone Ġproposition . ĠWe Ġfound Ġthat Ġachieving Ġa Ġhigher Ġnumber Ġof Ġconnections Ġwould Ġmean Ġcompromising Ġimportance , Ġi . e . Ġincluding Ġless Ġimportant Ġpropositions , Ġand Ġdecided Ġagainst Ġit . </s> Bas eline ĠExper iments </s> In Ġthis Ġsection , Ġwe Ġbriefly Ġdescribe Ġa Ġbaseline Ġand Ġevaluation Ġscripts Ġthat Ġwe Ġrelease , Ġwith Ġa Ġdetailed Ġdocumentation , Ġalong Ġwith Ġthe Ġcorpus . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpresented Ġlow - context Ġimportance Ġannotation , Ġa Ġnovel Ġcrowds ourcing Ġscheme Ġthat Ġwe Ġused Ġto Ġcreate Ġa Ġnew Ġbenchmark Ġcorpus Ġfor Ġconcept - map - based ĠM DS . ĠThe Ġcorpus Ġhas Ġlarge - scale Ġdocument Ġclusters Ġof Ġheter ogeneous Ġweb Ġdocuments , Ġposing Ġa Ġchallenging Ġsummar ization Ġtask . ĠTogether Ġwith Ġthe Ġcorpus , Ġwe Ġprovide Ġimplementations Ġof Ġa Ġbaseline Ġmethod Ġand Ġevaluation Ġscripts Ġand Ġhope Ġthat Ġour Ġefforts Ġfacilitate Ġfuture Ġresearch Ġon Ġthis Ġvariant Ġof Ġsummar ization . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank ĠTeresa ĠBot sc hen , ĠAndreas ĠHan sel owski Ġand ĠMarkus ĠZ op f Ġfor Ġtheir Ġhelp Ġwith Ġthe Ġannotation Ġwork Ġand ĠChristian ĠMeyer Ġfor Ġhis Ġvaluable Ġfeedback . ĠThis Ġwork Ġhas Ġbeen Ġsupported Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġas Ġpart Ġof Ġthe ĠResearch ĠTraining ĠGroup ĠâĢ ľ Adapt ive ĠPrepar ation Ġof ĠInformation Ġfrom ĠH eter ogeneous ĠSources âĢ Ŀ Ġ( A IP H ES ) Ġunder Ġgrant ĠNo . ĠGR K Ġ1994 / 1 .