Answer with content missing: (LVL1, LVL2, LVL3) 
- Stanford CoreNLP
- Optical Character Recognition (OCR) system, ParsCIT 
- further abridge the input text from level 2 preprocessed documents to the following: title, headers, abstract, introduction, related work, background and conclusion.
ABSTRACTIVE
In this study, we concentrate our effort on re-assessing keyphrase extraction performance on three increasingly sophisticated levels of document preprocessing described below.
1610.07809
Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLic ence . ĠLic ence Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Recent Ġyears Ġhave Ġseen Ġa Ġsurge Ġof Ġinterest Ġin Ġautomatic Ġkey phrase Ġextraction , Ġthanks Ġto Ġthe Ġavailability Ġof Ġthe ĠSem E val - 2010 Ġbenchmark Ġdataset ĠB IB REF 0 Ġ. ĠThis Ġdataset Ġis Ġcomposed Ġof Ġdocuments Ġ( scientific Ġarticles ) Ġthat Ġwere Ġautomatically Ġconverted Ġfrom ĠPDF Ġformat Ġto Ġplain Ġtext . ĠAs Ġa Ġresult , Ġmost Ġdocuments Ġcontain Ġirrelevant Ġpieces Ġof Ġtext Ġ( e . g . Ġm uddled Ġsentences , Ġtables , Ġequations , Ġfoot notes ) Ġthat Ġrequire Ġspecial Ġhandling , Ġso Ġas Ġto Ġnot Ġhinder Ġthe Ġperformance Ġof Ġkey phrase Ġextraction Ġsystems . ĠIn Ġprevious Ġwork , Ġthese Ġare Ġusually Ġremoved Ġat Ġthe Ġpre processing Ġstep , Ġbut Ġusing Ġa Ġvariety Ġof Ġtechniques Ġranging Ġfrom Ġsimple Ġhe ur istics ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġto Ġsophisticated Ġdocument Ġlogical Ġstructure Ġdetection Ġon Ġrich ly - form atted Ġdocuments Ġrecovered Ġfrom ĠGoogle ĠScholar ĠB IB REF 4 Ġ. ĠUnder Ġsuch Ġconditions , Ġit Ġmay Ġprove Ġdifficult Ġto Ġdraw Ġfirm Ġconclusions Ġabout Ġwhich Ġkey phrase Ġextraction Ġmodel Ġperforms Ġbest , Ġas Ġthe Ġimpact Ġof Ġpre processing Ġon Ġoverall Ġperformance Ġcannot Ġbe Ġproperly Ġquant ified . </s> While Ġprevious Ġwork Ġclearly Ġstates Ġthat Ġefficient Ġdocument Ġpre processing Ġis Ġa Ġprerequisite Ġfor Ġthe Ġextraction Ġof Ġhigh Ġquality Ġkey ph r ases , Ġthere Ġis , Ġto Ġour Ġbest Ġknowledge , Ġno Ġempirical Ġevidence Ġof Ġhow Ġpre processing Ġaffects Ġkey phrase Ġextraction Ġperformance . ĠIn Ġthis Ġpaper , Ġwe Ġre - ass ess Ġthe Ġperformance Ġof Ġseveral Ġstate - of - the - art Ġkey phrase Ġextraction Ġmodels Ġat Ġincreasingly Ġsophisticated Ġlevels Ġof Ġpre processing . ĠThree Ġincremental Ġlevels Ġof Ġdocument Ġpre processing Ġare Ġexperimented Ġwith : Ġraw Ġtext , Ġtext Ġcleaning Ġthrough Ġdocument Ġlogical Ġstructure Ġdetection , Ġand Ġremoval Ġof Ġkey phrase Ġsparse Ġsections Ġof Ġthe Ġdocument . ĠIn Ġdoing Ġso , Ġwe Ġpresent Ġthe Ġfirst Ġconsistent Ġcomparison Ġof Ġdifferent Ġkey phrase Ġextraction Ġmodels Ġand Ġstudy Ġtheir Ġrobust ness Ġover Ġnoisy Ġtext . ĠMore Ġprecisely , Ġour Ġcontributions Ġare : </s> Dat as et Ġand ĠPre processing </s> The ĠSem E val - 2010 Ġbenchmark Ġdataset ĠB IB REF 0 Ġis Ġcomposed Ġof Ġ244 Ġscientific Ġarticles Ġcollected Ġfrom Ġthe ĠAC M ĠDigital ĠLibrary Ġ( conference Ġand Ġworkshop Ġpapers ). ĠThe Ġinput Ġpapers Ġranged Ġfrom Ġ6 Ġto Ġ8 Ġpages Ġand Ġwere Ġconverted Ġfrom ĠPDF Ġformat Ġto Ġplain Ġtext Ġusing Ġan Ġoff - the - she lf Ġtool . ĠThe Ġonly Ġpre processing Ġapplied Ġis Ġa Ġsystematic Ġde hyp hen ation Ġat Ġline Ġbreaks Ġand Ġremoval Ġof Ġauthor - ass igned Ġkey ph r ases . ĠScientific Ġarticles Ġwere Ġselected Ġfrom Ġfour Ġdifferent Ġresearch Ġareas Ġas Ġdefined Ġin Ġthe ĠAC M Ġclassification , Ġand Ġwere Ġequally Ġdistributed Ġinto Ġtraining Ġ( 144 Ġarticles ) Ġand Ġtest Ġ( 100 Ġarticles ) Ġsets . ĠGold Ġstandard Ġkey ph r ases Ġare Ġcomposed Ġof Ġboth Ġauthor - ass igned Ġkey ph r ases Ġcollected Ġfrom Ġthe Ġoriginal ĠPDF Ġfiles Ġand Ġreader - ass igned Ġkey ph r ases Ġprovided Ġby Ġstudent Ġannot ators . </s> Long Ġdocuments Ġsuch Ġas Ġthose Ġin Ġthe ĠSem E val - 2010 Ġbenchmark Ġdataset Ġare Ġnotoriously Ġdifficult Ġto Ġhandle Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġkey phrase Ġcandidates Ġ( i . e . Ġphrases Ġthat Ġare Ġeligible Ġto Ġbe Ġkey ph r ases ) Ġthat Ġthe Ġsystems Ġhave Ġto Ġcope Ġwith ĠB IB REF 6 Ġ. ĠFurthermore , Ġnoisy Ġtextual Ġcontent , Ġwhether Ġdue Ġto Ġformat Ġconversion Ġerrors Ġor Ġto Ġunus able Ġelements Ġ( e . g . Ġequations ), Ġyield Ġmany Ġspurious Ġkey phrase Ġcandidates Ġthat Ġnegatively Ġaffect Ġkey phrase Ġextraction Ġperformance . ĠThis Ġis Ġparticularly Ġtrue Ġfor Ġsystems Ġthat Ġmake Ġuse Ġof Ġcore ĠN LP Ġtools Ġto Ġselect Ġcandidates , Ġthat Ġin Ġturn Ġexhibit Ġpoor Ġperformance Ġon Ġdegraded Ġtext . ĠFil tering Ġout Ġirrelevant Ġtext Ġis Ġtherefore Ġneeded Ġfor Ġaddressing Ġthese Ġissues . </s> In Ġthis Ġstudy , Ġwe Ġconcentrate Ġour Ġeffort Ġon Ġre - ass essing Ġkey phrase Ġextraction Ġperformance Ġon Ġthree Ġincreasingly Ġsophisticated Ġlevels Ġof Ġdocument Ġpre processing Ġdescribed Ġbelow . </s> Table Ġshows Ġthe Ġaverage Ġnumber Ġof Ġsentences Ġand Ġwords Ġalong Ġwith Ġthe Ġmaximum Ġpossible Ġrecall Ġfor Ġeach Ġlevel Ġof Ġpre processing . ĠThe Ġmaximum Ġrecall Ġis Ġobtained Ġby Ġcomputing Ġthe Ġfraction Ġof Ġthe Ġreference Ġkey ph r ases Ġthat Ġoccur Ġin Ġthe Ġdocuments . ĠWe Ġobserve Ġthat Ġthe Ġlevel Ġ2 Ġpre processing Ġsucceeds Ġin Ġeliminating Ġirrelevant Ġtext Ġby Ġsignificantly Ġreducing Ġthe Ġnumber Ġof Ġwords Ġ(- 19 %) Ġwhile Ġmaintaining Ġa Ġhigh Ġmaximum Ġrecall Ġ(- 2 %). ĠLevel Ġ3 Ġpre processing Ġdrastically Ġreduce Ġthe Ġnumber Ġof Ġwords Ġto Ġless Ġthan Ġa Ġquarter Ġof Ġthe Ġoriginal Ġamount Ġwhile Ġinterestingly Ġstill Ġpreserving Ġhigh Ġrecall . </s> Key phrase ĠExt raction ĠModels </s> We Ġre - im ple mented Ġfive Ġkey phrase Ġextraction Ġmodels Ġ: Ġthe Ġfirst Ġtwo Ġare Ġcommonly Ġused Ġas Ġbas elines , Ġthe Ġthird Ġis Ġa Ġresource - lean Ġun super vised Ġgraph - based Ġranking Ġapproach , Ġand Ġthe Ġlast Ġtwo Ġwere Ġamong Ġthe Ġtop Ġperforming Ġsystems Ġin Ġthe ĠSem E val - 2010 Ġkey phrase Ġextraction Ġtask ĠB IB REF 0 Ġ. ĠWe Ġnote Ġthat Ġtwo Ġof Ġthe Ġsystems Ġare Ġsupervised Ġand Ġrely Ġon Ġthe Ġtraining Ġset Ġto Ġbuild Ġtheir Ġclassification Ġmodels . ĠDocument Ġfrequency Ġcounts Ġare Ġalso Ġcomputed Ġon Ġthe Ġtraining Ġset . ĠSt em ming Ġis Ġapplied Ġto Ġallow Ġmore Ġrobust Ġmatching . ĠThe Ġdifferent Ġkey phrase Ġextraction Ġmodels Ġare Ġbriefly Ġdescribed Ġbelow : </s> Each Ġmodel Ġuses Ġa Ġdistinct Ġkey phrase Ġcandidate Ġselection Ġmethod Ġthat Ġprovides Ġa Ġtrade - off Ġbetween Ġthe Ġhighest Ġattain able Ġrecall Ġand Ġthe Ġsize Ġof Ġset Ġof Ġcandidates . ĠTable Ġsummarizes Ġthese Ġnumbers Ġfor Ġeach Ġmodel . ĠSynt ax - based Ġselection Ġhe ur istics , Ġas Ġused Ġby ĠTopic Rank Ġand ĠW ING N US , Ġare Ġbetter Ġsuited Ġto Ġpr une Ġcandidates Ġthat Ġare Ġunlikely Ġto Ġbe Ġkey ph r ases . ĠAs Ġfor ĠKP - min er , Ġremoving Ġinf requent Ġcandidates Ġmay Ġseem Ġrather Ġblunt , Ġbut Ġit Ġturns Ġout Ġto Ġbe Ġa Ġsimple Ġyet Ġeffective Ġpr uning Ġmethod Ġwhen Ġdealing Ġwith Ġlong Ġdocuments . ĠFor Ġdetails Ġon Ġhow Ġcandidate Ġselection Ġmethods Ġaffect Ġkey phrase Ġextraction , Ġplease Ġrefer Ġto ĠB IB REF 16 Ġ. </s> Apart Ġfrom ĠTopic Rank Ġthat Ġgroups Ġsimilar Ġcandidates Ġinto Ġtopics , Ġthe Ġother Ġmodels Ġdo Ġnot Ġhave Ġany Ġredundancy Ġcontrol Ġmechanism . ĠYet , Ġrecent Ġwork Ġhas Ġshown Ġthat Ġup Ġto Ġ12 % Ġof Ġthe Ġoverall Ġerror Ġmade Ġby Ġstate - of - the - art Ġkey phrase Ġextraction Ġsystems Ġwere Ġdue Ġto Ġredundancy ĠB IB REF 6 Ġ, ĠB IB REF 17 Ġ. ĠTherefore Ġas Ġa Ġpost - ranking Ġstep , Ġwe Ġremove Ġredundant Ġkey ph r ases Ġfrom Ġthe Ġranked Ġlists Ġgenerated Ġby Ġall Ġmodels . ĠA Ġkey phrase Ġis Ġconsidered Ġredundant Ġif Ġit Ġis Ġincluded Ġin Ġanother Ġkey phrase Ġthat Ġis Ġranked Ġhigher Ġin Ġthe Ġlist . </s> Exper imental Ġsettings </s> We Ġfollow Ġthe Ġevaluation Ġprocedure Ġused Ġin Ġthe ĠSem E val - 2010 Ġcompetition Ġand Ġevaluate Ġthe Ġperformance Ġof Ġeach Ġmodel Ġin Ġterms Ġof Ġf - me asure Ġ( F ) Ġat Ġthe Ġtop ĠIN LINE FORM 0 Ġkey ph r ases . ĠWe Ġuse Ġthe Ġset Ġof Ġcombined Ġauthor - Ġand Ġreader - ass igned Ġkey ph r ases Ġas Ġreference Ġkey ph r ases . ĠExt racted Ġand Ġreference Ġkey ph r ases Ġare Ġstemmed Ġto Ġreduce Ġthe Ġnumber Ġof Ġmism atches . </s> Results </s> The Ġperformances Ġof Ġthe Ġkey phrase Ġextraction Ġmodels Ġat Ġeach Ġpre processing Ġlevel Ġare Ġshown Ġin ĠTable Ġ. ĠOverall , Ġwe Ġobserve Ġa Ġsignificant Ġincrease Ġof Ġperformance Ġfor Ġall Ġmodels Ġat Ġlevels Ġ3 , Ġconfirming Ġthat Ġdocument Ġpre processing Ġplays Ġan Ġimportant Ġrole Ġin Ġkey phrase Ġextraction Ġperformance . ĠAlso , Ġthe Ġdifference Ġof Ġf - score Ġbetween Ġthe Ġmodels , Ġas Ġmeasured Ġby Ġthe Ġstandard Ġdeviation ĠIN LINE FORM 0 Ġ, Ġgradually Ġdecreases Ġwith Ġthe Ġincreasing Ġlevel Ġof Ġpre processing . ĠThis Ġresult Ġstrengthens Ġthe Ġassumption Ġmade Ġin Ġthis Ġpaper , Ġthat Ġperformance Ġvariation Ġacross Ġmodels Ġis Ġpartly Ġa Ġfunction Ġof Ġthe Ġeffectiveness Ġof Ġdocument Ġpre processing . </s> S omew hat Ġsurprisingly , Ġthe Ġordering Ġof Ġthe Ġtwo Ġbest Ġmodels Ġrevers es Ġat Ġlevel Ġ3 . ĠThis Ġshowcases Ġthat Ġrankings Ġare Ġheavily Ġinfluenced Ġby Ġthe Ġpre processing Ġstage , Ġdespite Ġthe Ġcommon Ġlack Ġof Ġdetails Ġand Ġanalysis Ġit Ġsuffers Ġfrom Ġin Ġexplanatory Ġpapers . ĠWe Ġalso Ġremark Ġthat Ġthe Ġtop Ġperforming Ġmodel , Ġnamely ĠKP - M iner , Ġis Ġun super vised Ġwhich Ġsupports Ġthe Ġfindings Ġof ĠB IB REF 6 Ġindicating Ġthat Ġrecent Ġun super vised Ġapproaches Ġhave Ġrival ed Ġtheir Ġsupervised Ġcounterparts Ġin Ġperformance . </s> In Ġan Ġattempt Ġto Ġquantify Ġthe Ġperformance Ġvariation Ġacross Ġpre processing Ġlevels , Ġwe Ġcompute Ġthe Ġstandard Ġdeviation ĠIN LINE FORM 0 Ġfor Ġeach Ġmodel . ĠHere Ġwe Ġsee Ġthat Ġun super vised Ġmodels Ġare Ġmore Ġsensitive Ġto Ġnoisy Ġinput , Ġas Ġrevealed Ġby Ġhigher Ġstandard Ġdeviations . ĠWe Ġfound Ġtwo Ġmain Ġreasons Ġfor Ġthis . ĠFirst , Ġusing Ġmultiple Ġdiscrim in ative Ġfeatures Ġto Ġrank Ġkey phrase Ġcandidates Ġadds Ġinherent Ġrobust ness Ġto Ġthe Ġmodels . ĠSecond , Ġthe Ġsupervision Ġsignal Ġhelps Ġmodels Ġto Ġdisregard Ġnoise . </s> In ĠTable Ġ, Ġwe Ġcompare Ġthe Ġoutputs Ġof Ġthe Ġfive Ġmodels Ġby Ġmeasuring Ġthe Ġpercentage Ġof Ġvalid Ġkey ph r ases Ġthat Ġare Ġretrieved Ġby Ġall Ġmodels Ġat Ġonce Ġfor Ġeach Ġpre processing Ġlevel . ĠBy Ġthese Ġadditional Ġresults , Ġwe Ġaim Ġto Ġassess Ġwhether Ġdocument Ġpre processing Ġsmoot hes Ġdifferences Ġbetween Ġmodels . ĠWe Ġobserve Ġthat Ġthe Ġoverlap Ġbetween Ġthe Ġoutputs Ġof Ġthe Ġdifferent Ġmodels Ġincreases Ġalong Ġwith Ġthe Ġlevel Ġof Ġpre processing . ĠThis Ġsuggests Ġthat Ġdocument Ġpre processing Ġreduces Ġthe Ġeffect Ġthat Ġthe Ġkey phrase Ġextraction Ġmodel Ġin Ġitself Ġhas Ġon Ġoverall Ġperformance . ĠIn Ġother Ġwords , Ġthe Ġsingular ity Ġof Ġeach Ġmodel Ġfades Ġaway Ġgradually Ġwith Ġincrease Ġin Ġpre processing Ġeffort . </s> Rep rodu c ibility </s> Being Ġable Ġto Ġreproduce Ġexperimental Ġresults Ġis Ġa Ġcentral Ġaspect Ġof Ġthe Ġscientific Ġmethod . ĠWhile Ġassessing Ġthe Ġimportance Ġof Ġthe Ġpre processing Ġstage Ġfor Ġfive Ġapproaches , Ġwe Ġfound Ġthat Ġseveral Ġresults Ġwere Ġnot Ġreprodu cible , Ġas Ġshown Ġin ĠTable Ġ. </s> We Ġnote Ġthat Ġthe Ġtrends Ġfor Ġbas elines Ġand Ġhigh Ġranking Ġsystems Ġare Ġopposite : Ġcompared Ġto Ġthe Ġpublished Ġresults , Ġour Ġreproduction Ġof Ġtop Ġsystems Ġunder - per forms Ġand Ġour Ġreproduction Ġof Ġbas elines Ġover - per forms . ĠWe Ġhypothes ise Ġthat Ġthis Ġstems Ġfrom Ġa Ġdifference Ġin Ġhyper param eter Ġtuning , Ġincluding Ġthe Ġones Ġthat Ġthe Ġpre processing Ġstage Ġmakes Ġimplicit . ĠCompet itors Ġhave Ġstrong Ġincentives Ġto Ġcorrectly Ġoptimize Ġhyper param eters , Ġto Ġachieve Ġa Ġhigh Ġranking Ġand Ġget Ġmore Ġpublicity Ġfor Ġtheir Ġwork Ġwhile Ġcompetition Ġorganizers Ġmight Ġhave Ġthe Ġopposite Ġincentive : Ġtoo Ġstrong Ġa Ġbaseline Ġmight Ġnot Ġbe Ġconsidered Ġa Ġbaseline Ġanymore . </s> We Ġalso Ġobserve Ġthat Ġwith Ġthis Ġleveled Ġpre processing , Ġthe Ġgap Ġbetween Ġbas elines Ġand Ġtop Ġsystems Ġis Ġmuch Ġsmaller , Ġless ening Ġagain Ġthe Ġimportance Ġof Ġraw Ġscores Ġand Ġrankings Ġto Ġinterpret Ġthe Ġshared Ġtask Ġresults Ġand Ġemphasizing Ġthe Ġimportance Ġof Ġunderstanding Ġcorrectly Ġthe Ġpre processing Ġstage . </s> Additional Ġexperiments </s> In Ġthe Ġprevious Ġsections , Ġwe Ġprovided Ġempirical Ġevidence Ġthat Ġdocument Ġpre processing Ġweighs Ġheavily Ġon Ġthe Ġoutcome Ġof Ġkey phrase Ġextraction Ġmodels . ĠThis Ġraises Ġthe Ġquestion Ġof Ġwhether Ġfurther Ġimprovement Ġmight Ġbe Ġgained Ġfrom Ġa Ġmore Ġaggressive Ġpre processing . ĠTo Ġanswer Ġthis Ġquestion , Ġwe Ġtake Ġanother Ġstep Ġbeyond Ġcontent Ġfiltering Ġand Ġfurther Ġab ridge Ġthe Ġinput Ġtext Ġfrom Ġlevel Ġ3 Ġpre process ed Ġdocuments Ġusing Ġan Ġun super vised Ġsummar ization Ġtechnique . ĠMore Ġspecifically , Ġwe Ġkeep Ġthe Ġtitle Ġand Ġabstract Ġintact , Ġas Ġthey Ġare Ġthe Ġtwo Ġmost Ġkey phrase Ġdense Ġparts Ġwithin Ġscientific Ġarticles ĠB IB REF 4 Ġ, Ġand Ġselect Ġonly Ġthe Ġmost Ġcontent Ġbearing Ġsentences Ġfrom Ġthe Ġremaining Ġcontents . ĠTo Ġdo Ġthis , Ġsentences Ġare Ġordered Ġusing ĠText Rank ĠB IB REF 14 Ġand Ġthe Ġless Ġinformative Ġones , Ġas Ġdetermined Ġby Ġtheir ĠText Rank Ġscores Ġnormalized Ġby Ġtheir Ġlengths Ġin Ġwords , Ġare Ġfiltered Ġout . </s> Finding Ġthe Ġoptimal Ġsubset Ġof Ġsentences Ġfrom Ġalready Ġshortened Ġdocuments Ġis Ġhowever Ġno Ġtrivial Ġtask Ġas Ġmaximum Ġrecall Ġlin early Ġdecreases Ġwith Ġthe Ġnumber Ġof Ġsentences Ġdiscarded . ĠHere , Ġwe Ġsimply Ġset Ġthe Ġreduction Ġratio Ġto Ġ0 . 8 65 Ġso Ġthat Ġthe Ġaverage Ġmaximum Ġrecall Ġon Ġthe Ġtraining Ġset Ġdoes Ġnot Ġlose Ġmore Ġthan Ġ5 %. ĠTable Ġshows Ġthe Ġreduction Ġin Ġthe Ġaverage Ġnumber Ġof Ġsentences Ġand Ġwords Ġcompared Ġto Ġlevel Ġ3 Ġpre processing . </s> The Ġperformances Ġof Ġthe Ġkey phrase Ġextraction Ġmodels Ġat Ġlevel Ġ4 Ġpre processing Ġare Ġshown Ġin ĠTable Ġ. ĠWe Ġnote Ġthat Ġtwo Ġmodels , Ġnamely ĠTopic Rank Ġand ĠTF ĠIN LINE FORM 0 ĠIDF , Ġlose Ġon Ġperformance . ĠThese Ġtwo Ġmodels Ġmainly Ġrely Ġon Ġfrequency Ġcounts Ġto Ġrank Ġkey phrase Ġcandidates , Ġwhich Ġin Ġturn Ġbecome Ġless Ġreliable Ġat Ġlevel Ġ4 Ġbecause Ġof Ġthe Ġvery Ġshort Ġlength Ġof Ġthe Ġdocuments . ĠOther Ġmodels Ġhowever Ġhave Ġtheir Ġf - sc ores Ġonce Ġagain Ġincreased , Ġthus Ġindicating Ġthat Ġfurther Ġimprovement Ġis Ġpossible Ġfrom Ġmore Ġredu ctive Ġdocument Ġpre processing Ġstrategies . </s> Conclusion </s> In Ġthis Ġstudy , Ġwe Ġre - ass essed Ġthe Ġperformance Ġof Ġseveral Ġkey phrase Ġextraction Ġmodels Ġand Ġshowed Ġthat Ġperformance Ġvariation Ġacross Ġmodels Ġis Ġpartly Ġa Ġfunction Ġof Ġthe Ġeffectiveness Ġof Ġthe Ġdocument Ġpre processing . ĠOur Ġresults Ġalso Ġsuggest Ġthat Ġsupervised Ġkey phrase Ġextraction Ġmodels Ġare Ġmore Ġrobust Ġto Ġnoisy Ġinput . </s> Given Ġour Ġfindings , Ġwe Ġrecommend Ġthat Ġfuture Ġworks Ġuse Ġa Ġcommon Ġpre processing Ġto Ġevaluate Ġthe Ġinterest Ġof Ġkey phrase Ġextraction Ġapproaches . ĠFor Ġthis Ġreason Ġwe Ġmake Ġthe Ġfour Ġlevels Ġof Ġpre processing Ġused Ġin Ġthis Ġstudy Ġavailable Ġfor Ġthe Ġcommunity .
Introduction This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/ Recent years have seen a surge of interest in automatic keyphrase extraction, thanks to the availability of the SemEval-2010 benchmark dataset BIBREF0 . This dataset is composed of documents (scientific articles) that were automatically converted from PDF format to plain text. As a result, most documents contain irrelevant pieces of text (e.g. muddled sentences, tables, equations, footnotes) that require special handling, so as to not hinder the performance of keyphrase extraction systems. In previous work, these are usually removed at the preprocessing step, but using a variety of techniques ranging from simple heuristics BIBREF1 , BIBREF2 , BIBREF3 to sophisticated document logical structure detection on richly-formatted documents recovered from Google Scholar BIBREF4 . Under such conditions, it may prove difficult to draw firm conclusions about which keyphrase extraction model performs best, as the impact of preprocessing on overall performance cannot be properly quantified. While previous work clearly states that efficient document preprocessing is a prerequisite for the extraction of high quality keyphrases, there is, to our best knowledge, no empirical evidence of how preprocessing affects keyphrase extraction performance. In this paper, we re-assess the performance of several state-of-the-art keyphrase extraction models at increasingly sophisticated levels of preprocessing. Three incremental levels of document preprocessing are experimented with: raw text, text cleaning through document logical structure detection, and removal of keyphrase sparse sections of the document. In doing so, we present the first consistent comparison of different keyphrase extraction models and study their robustness over noisy text. More precisely, our contributions are: Dataset and Preprocessing The SemEval-2010 benchmark dataset BIBREF0 is composed of 244 scientific articles collected from the ACM Digital Library (conference and workshop papers). The input papers ranged from 6 to 8 pages and were converted from PDF format to plain text using an off-the-shelf tool. The only preprocessing applied is a systematic dehyphenation at line breaks and removal of author-assigned keyphrases. Scientific articles were selected from four different research areas as defined in the ACM classification, and were equally distributed into training (144 articles) and test (100 articles) sets. Gold standard keyphrases are composed of both author-assigned keyphrases collected from the original PDF files and reader-assigned keyphrases provided by student annotators. Long documents such as those in the SemEval-2010 benchmark dataset are notoriously difficult to handle due to the large number of keyphrase candidates (i.e. phrases that are eligible to be keyphrases) that the systems have to cope with BIBREF6 . Furthermore, noisy textual content, whether due to format conversion errors or to unusable elements (e.g. equations), yield many spurious keyphrase candidates that negatively affect keyphrase extraction performance. This is particularly true for systems that make use of core NLP tools to select candidates, that in turn exhibit poor performance on degraded text. Filtering out irrelevant text is therefore needed for addressing these issues. In this study, we concentrate our effort on re-assessing keyphrase extraction performance on three increasingly sophisticated levels of document preprocessing described below. Table shows the average number of sentences and words along with the maximum possible recall for each level of preprocessing. The maximum recall is obtained by computing the fraction of the reference keyphrases that occur in the documents. We observe that the level 2 preprocessing succeeds in eliminating irrelevant text by significantly reducing the number of words (-19%) while maintaining a high maximum recall (-2%). Level 3 preprocessing drastically reduce the number of words to less than a quarter of the original amount while interestingly still preserving high recall. Keyphrase Extraction Models We re-implemented five keyphrase extraction models : the first two are commonly used as baselines, the third is a resource-lean unsupervised graph-based ranking approach, and the last two were among the top performing systems in the SemEval-2010 keyphrase extraction task BIBREF0 . We note that two of the systems are supervised and rely on the training set to build their classification models. Document frequency counts are also computed on the training set. Stemming is applied to allow more robust matching. The different keyphrase extraction models are briefly described below: Each model uses a distinct keyphrase candidate selection method that provides a trade-off between the highest attainable recall and the size of set of candidates. Table summarizes these numbers for each model. Syntax-based selection heuristics, as used by TopicRank and WINGNUS, are better suited to prune candidates that are unlikely to be keyphrases. As for KP-miner, removing infrequent candidates may seem rather blunt, but it turns out to be a simple yet effective pruning method when dealing with long documents. For details on how candidate selection methods affect keyphrase extraction, please refer to BIBREF16 . Apart from TopicRank that groups similar candidates into topics, the other models do not have any redundancy control mechanism. Yet, recent work has shown that up to 12% of the overall error made by state-of-the-art keyphrase extraction systems were due to redundancy BIBREF6 , BIBREF17 . Therefore as a post-ranking step, we remove redundant keyphrases from the ranked lists generated by all models. A keyphrase is considered redundant if it is included in another keyphrase that is ranked higher in the list. Experimental settings We follow the evaluation procedure used in the SemEval-2010 competition and evaluate the performance of each model in terms of f-measure (F) at the top INLINEFORM0 keyphrases. We use the set of combined author- and reader-assigned keyphrases as reference keyphrases. Extracted and reference keyphrases are stemmed to reduce the number of mismatches. Results The performances of the keyphrase extraction models at each preprocessing level are shown in Table . Overall, we observe a significant increase of performance for all models at levels 3, confirming that document preprocessing plays an important role in keyphrase extraction performance. Also, the difference of f-score between the models, as measured by the standard deviation INLINEFORM0 , gradually decreases with the increasing level of preprocessing. This result strengthens the assumption made in this paper, that performance variation across models is partly a function of the effectiveness of document preprocessing. Somewhat surprisingly, the ordering of the two best models reverses at level 3. This showcases that rankings are heavily influenced by the preprocessing stage, despite the common lack of details and analysis it suffers from in explanatory papers. We also remark that the top performing model, namely KP-Miner, is unsupervised which supports the findings of BIBREF6 indicating that recent unsupervised approaches have rivaled their supervised counterparts in performance. In an attempt to quantify the performance variation across preprocessing levels, we compute the standard deviation INLINEFORM0 for each model. Here we see that unsupervised models are more sensitive to noisy input, as revealed by higher standard deviations. We found two main reasons for this. First, using multiple discriminative features to rank keyphrase candidates adds inherent robustness to the models. Second, the supervision signal helps models to disregard noise. In Table , we compare the outputs of the five models by measuring the percentage of valid keyphrases that are retrieved by all models at once for each preprocessing level. By these additional results, we aim to assess whether document preprocessing smoothes differences between models. We observe that the overlap between the outputs of the different models increases along with the level of preprocessing. This suggests that document preprocessing reduces the effect that the keyphrase extraction model in itself has on overall performance. In other words, the singularity of each model fades away gradually with increase in preprocessing effort. Reproducibility Being able to reproduce experimental results is a central aspect of the scientific method. While assessing the importance of the preprocessing stage for five approaches, we found that several results were not reproducible, as shown in Table . We note that the trends for baselines and high ranking systems are opposite: compared to the published results, our reproduction of top systems under-performs and our reproduction of baselines over-performs. We hypothesise that this stems from a difference in hyperparameter tuning, including the ones that the preprocessing stage makes implicit. Competitors have strong incentives to correctly optimize hyperparameters, to achieve a high ranking and get more publicity for their work while competition organizers might have the opposite incentive: too strong a baseline might not be considered a baseline anymore. We also observe that with this leveled preprocessing, the gap between baselines and top systems is much smaller, lessening again the importance of raw scores and rankings to interpret the shared task results and emphasizing the importance of understanding correctly the preprocessing stage. Additional experiments In the previous sections, we provided empirical evidence that document preprocessing weighs heavily on the outcome of keyphrase extraction models. This raises the question of whether further improvement might be gained from a more aggressive preprocessing. To answer this question, we take another step beyond content filtering and further abridge the input text from level 3 preprocessed documents using an unsupervised summarization technique. More specifically, we keep the title and abstract intact, as they are the two most keyphrase dense parts within scientific articles BIBREF4 , and select only the most content bearing sentences from the remaining contents. To do this, sentences are ordered using TextRank BIBREF14 and the less informative ones, as determined by their TextRank scores normalized by their lengths in words, are filtered out. Finding the optimal subset of sentences from already shortened documents is however no trivial task as maximum recall linearly decreases with the number of sentences discarded. Here, we simply set the reduction ratio to 0.865 so that the average maximum recall on the training set does not lose more than 5%. Table shows the reduction in the average number of sentences and words compared to level 3 preprocessing. The performances of the keyphrase extraction models at level 4 preprocessing are shown in Table . We note that two models, namely TopicRank and TF INLINEFORM0 IDF, lose on performance. These two models mainly rely on frequency counts to rank keyphrase candidates, which in turn become less reliable at level 4 because of the very short length of the documents. Other models however have their f-scores once again increased, thus indicating that further improvement is possible from more reductive document preprocessing strategies. Conclusion In this study, we re-assessed the performance of several keyphrase extraction models and showed that performance variation across models is partly a function of the effectiveness of the document preprocessing. Our results also suggest that supervised keyphrase extraction models are more robust to noisy input. Given our findings, we recommend that future works use a common preprocessing to evaluate the interest of keyphrase extraction approaches. For this reason we make the four levels of preprocessing used in this study available for the community.
what keyphrase extraction models were reassessed?
e54257585cc75564341eb02bdc63ff8111992f82
what Ġkey phrase Ġextraction Ġmodels Ġwere Ġreass essed ?
Answer Ġwith Ġcontent Ġmissing : Ġ( LV L 1 , ĠLV L 2 , ĠLV L 3 ) Ġ Ċ - ĠStanford ĠCore N LP Ċ - ĠOptical ĠCharacter ĠRecogn ition Ġ( OC R ) Ġsystem , ĠPars C IT Ġ Ċ - Ġfurther Ġab ridge Ġthe Ġinput Ġtext Ġfrom Ġlevel Ġ2 Ġpre process ed Ġdocuments Ġto Ġthe Ġfollowing : Ġtitle , Ġheaders , Ġabstract , Ġintroduction , Ġrelated Ġwork , Ġbackground Ġand Ġconclusion .
<s> what Ġkey phrase Ġextraction Ġmodels Ġwere Ġreass essed ? </s> Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLic ence . ĠLic ence Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Recent Ġyears Ġhave Ġseen Ġa Ġsurge Ġof Ġinterest Ġin Ġautomatic Ġkey phrase Ġextraction , Ġthanks Ġto Ġthe Ġavailability Ġof Ġthe ĠSem E val - 2010 Ġbenchmark Ġdataset ĠB IB REF 0 Ġ. ĠThis Ġdataset Ġis Ġcomposed Ġof Ġdocuments Ġ( scientific Ġarticles ) Ġthat Ġwere Ġautomatically Ġconverted Ġfrom ĠPDF Ġformat Ġto Ġplain Ġtext . ĠAs Ġa Ġresult , Ġmost Ġdocuments Ġcontain Ġirrelevant Ġpieces Ġof Ġtext Ġ( e . g . Ġm uddled Ġsentences , Ġtables , Ġequations , Ġfoot notes ) Ġthat Ġrequire Ġspecial Ġhandling , Ġso Ġas Ġto Ġnot Ġhinder Ġthe Ġperformance Ġof Ġkey phrase Ġextraction Ġsystems . ĠIn Ġprevious Ġwork , Ġthese Ġare Ġusually Ġremoved Ġat Ġthe Ġpre processing Ġstep , Ġbut Ġusing Ġa Ġvariety Ġof Ġtechniques Ġranging Ġfrom Ġsimple Ġhe ur istics ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġto Ġsophisticated Ġdocument Ġlogical Ġstructure Ġdetection Ġon Ġrich ly - form atted Ġdocuments Ġrecovered Ġfrom ĠGoogle ĠScholar ĠB IB REF 4 Ġ. ĠUnder Ġsuch Ġconditions , Ġit Ġmay Ġprove Ġdifficult Ġto Ġdraw Ġfirm Ġconclusions Ġabout Ġwhich Ġkey phrase Ġextraction Ġmodel Ġperforms Ġbest , Ġas Ġthe Ġimpact Ġof Ġpre processing Ġon Ġoverall Ġperformance Ġcannot Ġbe Ġproperly Ġquant ified . </s> While Ġprevious Ġwork Ġclearly Ġstates Ġthat Ġefficient Ġdocument Ġpre processing Ġis Ġa Ġprerequisite Ġfor Ġthe Ġextraction Ġof Ġhigh Ġquality Ġkey ph r ases , Ġthere Ġis , Ġto Ġour Ġbest Ġknowledge , Ġno Ġempirical Ġevidence Ġof Ġhow Ġpre processing Ġaffects Ġkey phrase Ġextraction Ġperformance . ĠIn Ġthis Ġpaper , Ġwe Ġre - ass ess Ġthe Ġperformance Ġof Ġseveral Ġstate - of - the - art Ġkey phrase Ġextraction Ġmodels Ġat Ġincreasingly Ġsophisticated Ġlevels Ġof Ġpre processing . ĠThree Ġincremental Ġlevels Ġof Ġdocument Ġpre processing Ġare Ġexperimented Ġwith : Ġraw Ġtext , Ġtext Ġcleaning Ġthrough Ġdocument Ġlogical Ġstructure Ġdetection , Ġand Ġremoval Ġof Ġkey phrase Ġsparse Ġsections Ġof Ġthe Ġdocument . ĠIn Ġdoing Ġso , Ġwe Ġpresent Ġthe Ġfirst Ġconsistent Ġcomparison Ġof Ġdifferent Ġkey phrase Ġextraction Ġmodels Ġand Ġstudy Ġtheir Ġrobust ness Ġover Ġnoisy Ġtext . ĠMore Ġprecisely , Ġour Ġcontributions Ġare : </s> Dat as et Ġand ĠPre processing </s> The ĠSem E val - 2010 Ġbenchmark Ġdataset ĠB IB REF 0 Ġis Ġcomposed Ġof Ġ244 Ġscientific Ġarticles Ġcollected Ġfrom Ġthe ĠAC M ĠDigital ĠLibrary Ġ( conference Ġand Ġworkshop Ġpapers ). ĠThe Ġinput Ġpapers Ġranged Ġfrom Ġ6 Ġto Ġ8 Ġpages Ġand Ġwere Ġconverted Ġfrom ĠPDF Ġformat Ġto Ġplain Ġtext Ġusing Ġan Ġoff - the - she lf Ġtool . ĠThe Ġonly Ġpre processing Ġapplied Ġis Ġa Ġsystematic Ġde hyp hen ation Ġat Ġline Ġbreaks Ġand Ġremoval Ġof Ġauthor - ass igned Ġkey ph r ases . ĠScientific Ġarticles Ġwere Ġselected Ġfrom Ġfour Ġdifferent Ġresearch Ġareas Ġas Ġdefined Ġin Ġthe ĠAC M Ġclassification , Ġand Ġwere Ġequally Ġdistributed Ġinto Ġtraining Ġ( 144 Ġarticles ) Ġand Ġtest Ġ( 100 Ġarticles ) Ġsets . ĠGold Ġstandard Ġkey ph r ases Ġare Ġcomposed Ġof Ġboth Ġauthor - ass igned Ġkey ph r ases Ġcollected Ġfrom Ġthe Ġoriginal ĠPDF Ġfiles Ġand Ġreader - ass igned Ġkey ph r ases Ġprovided Ġby Ġstudent Ġannot ators . </s> Long Ġdocuments Ġsuch Ġas Ġthose Ġin Ġthe ĠSem E val - 2010 Ġbenchmark Ġdataset Ġare Ġnotoriously Ġdifficult Ġto Ġhandle Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġkey phrase Ġcandidates Ġ( i . e . Ġphrases Ġthat Ġare Ġeligible Ġto Ġbe Ġkey ph r ases ) Ġthat Ġthe Ġsystems Ġhave Ġto Ġcope Ġwith ĠB IB REF 6 Ġ. ĠFurthermore , Ġnoisy Ġtextual Ġcontent , Ġwhether Ġdue Ġto Ġformat Ġconversion Ġerrors Ġor Ġto Ġunus able Ġelements Ġ( e . g . Ġequations ), Ġyield Ġmany Ġspurious Ġkey phrase Ġcandidates Ġthat Ġnegatively Ġaffect Ġkey phrase Ġextraction Ġperformance . ĠThis Ġis Ġparticularly Ġtrue Ġfor Ġsystems Ġthat Ġmake Ġuse Ġof Ġcore ĠN LP Ġtools Ġto Ġselect Ġcandidates , Ġthat Ġin Ġturn Ġexhibit Ġpoor Ġperformance Ġon Ġdegraded Ġtext . ĠFil tering Ġout Ġirrelevant Ġtext Ġis Ġtherefore Ġneeded Ġfor Ġaddressing Ġthese Ġissues . </s> In Ġthis Ġstudy , Ġwe Ġconcentrate Ġour Ġeffort Ġon Ġre - ass essing Ġkey phrase Ġextraction Ġperformance Ġon Ġthree Ġincreasingly Ġsophisticated Ġlevels Ġof Ġdocument Ġpre processing Ġdescribed Ġbelow . </s> Table Ġshows Ġthe Ġaverage Ġnumber Ġof Ġsentences Ġand Ġwords Ġalong Ġwith Ġthe Ġmaximum Ġpossible Ġrecall Ġfor Ġeach Ġlevel Ġof Ġpre processing . ĠThe Ġmaximum Ġrecall Ġis Ġobtained Ġby Ġcomputing Ġthe Ġfraction Ġof Ġthe Ġreference Ġkey ph r ases Ġthat Ġoccur Ġin Ġthe Ġdocuments . ĠWe Ġobserve Ġthat Ġthe Ġlevel Ġ2 Ġpre processing Ġsucceeds Ġin Ġeliminating Ġirrelevant Ġtext Ġby Ġsignificantly Ġreducing Ġthe Ġnumber Ġof Ġwords Ġ(- 19 %) Ġwhile Ġmaintaining Ġa Ġhigh Ġmaximum Ġrecall Ġ(- 2 %). ĠLevel Ġ3 Ġpre processing Ġdrastically Ġreduce Ġthe Ġnumber Ġof Ġwords Ġto Ġless Ġthan Ġa Ġquarter Ġof Ġthe Ġoriginal Ġamount Ġwhile Ġinterestingly Ġstill Ġpreserving Ġhigh Ġrecall . </s> Key phrase ĠExt raction ĠModels </s> We Ġre - im ple mented Ġfive Ġkey phrase Ġextraction Ġmodels Ġ: Ġthe Ġfirst Ġtwo Ġare Ġcommonly Ġused Ġas Ġbas elines , Ġthe Ġthird Ġis Ġa Ġresource - lean Ġun super vised Ġgraph - based Ġranking Ġapproach , Ġand Ġthe Ġlast Ġtwo Ġwere Ġamong Ġthe Ġtop Ġperforming Ġsystems Ġin Ġthe ĠSem E val - 2010 Ġkey phrase Ġextraction Ġtask ĠB IB REF 0 Ġ. ĠWe Ġnote Ġthat Ġtwo Ġof Ġthe Ġsystems Ġare Ġsupervised Ġand Ġrely Ġon Ġthe Ġtraining Ġset Ġto Ġbuild Ġtheir Ġclassification Ġmodels . ĠDocument Ġfrequency Ġcounts Ġare Ġalso Ġcomputed Ġon Ġthe Ġtraining Ġset . ĠSt em ming Ġis Ġapplied Ġto Ġallow Ġmore Ġrobust Ġmatching . ĠThe Ġdifferent Ġkey phrase Ġextraction Ġmodels Ġare Ġbriefly Ġdescribed Ġbelow : </s> Each Ġmodel Ġuses Ġa Ġdistinct Ġkey phrase Ġcandidate Ġselection Ġmethod Ġthat Ġprovides Ġa Ġtrade - off Ġbetween Ġthe Ġhighest Ġattain able Ġrecall Ġand Ġthe Ġsize Ġof Ġset Ġof Ġcandidates . ĠTable Ġsummarizes Ġthese Ġnumbers Ġfor Ġeach Ġmodel . ĠSynt ax - based Ġselection Ġhe ur istics , Ġas Ġused Ġby ĠTopic Rank Ġand ĠW ING N US , Ġare Ġbetter Ġsuited Ġto Ġpr une Ġcandidates Ġthat Ġare Ġunlikely Ġto Ġbe Ġkey ph r ases . ĠAs Ġfor ĠKP - min er , Ġremoving Ġinf requent Ġcandidates Ġmay Ġseem Ġrather Ġblunt , Ġbut Ġit Ġturns Ġout Ġto Ġbe Ġa Ġsimple Ġyet Ġeffective Ġpr uning Ġmethod Ġwhen Ġdealing Ġwith Ġlong Ġdocuments . ĠFor Ġdetails Ġon Ġhow Ġcandidate Ġselection Ġmethods Ġaffect Ġkey phrase Ġextraction , Ġplease Ġrefer Ġto ĠB IB REF 16 Ġ. </s> Apart Ġfrom ĠTopic Rank Ġthat Ġgroups Ġsimilar Ġcandidates Ġinto Ġtopics , Ġthe Ġother Ġmodels Ġdo Ġnot Ġhave Ġany Ġredundancy Ġcontrol Ġmechanism . ĠYet , Ġrecent Ġwork Ġhas Ġshown Ġthat Ġup Ġto Ġ12 % Ġof Ġthe Ġoverall Ġerror Ġmade Ġby Ġstate - of - the - art Ġkey phrase Ġextraction Ġsystems Ġwere Ġdue Ġto Ġredundancy ĠB IB REF 6 Ġ, ĠB IB REF 17 Ġ. ĠTherefore Ġas Ġa Ġpost - ranking Ġstep , Ġwe Ġremove Ġredundant Ġkey ph r ases Ġfrom Ġthe Ġranked Ġlists Ġgenerated Ġby Ġall Ġmodels . ĠA Ġkey phrase Ġis Ġconsidered Ġredundant Ġif Ġit Ġis Ġincluded Ġin Ġanother Ġkey phrase Ġthat Ġis Ġranked Ġhigher Ġin Ġthe Ġlist . </s> Exper imental Ġsettings </s> We Ġfollow Ġthe Ġevaluation Ġprocedure Ġused Ġin Ġthe ĠSem E val - 2010 Ġcompetition Ġand Ġevaluate Ġthe Ġperformance Ġof Ġeach Ġmodel Ġin Ġterms Ġof Ġf - me asure Ġ( F ) Ġat Ġthe Ġtop ĠIN LINE FORM 0 Ġkey ph r ases . ĠWe Ġuse Ġthe Ġset Ġof Ġcombined Ġauthor - Ġand Ġreader - ass igned Ġkey ph r ases Ġas Ġreference Ġkey ph r ases . ĠExt racted Ġand Ġreference Ġkey ph r ases Ġare Ġstemmed Ġto Ġreduce Ġthe Ġnumber Ġof Ġmism atches . </s> Results </s> The Ġperformances Ġof Ġthe Ġkey phrase Ġextraction Ġmodels Ġat Ġeach Ġpre processing Ġlevel Ġare Ġshown Ġin ĠTable Ġ. ĠOverall , Ġwe Ġobserve Ġa Ġsignificant Ġincrease Ġof Ġperformance Ġfor Ġall Ġmodels Ġat Ġlevels Ġ3 , Ġconfirming Ġthat Ġdocument Ġpre processing Ġplays Ġan Ġimportant Ġrole Ġin Ġkey phrase Ġextraction Ġperformance . ĠAlso , Ġthe Ġdifference Ġof Ġf - score Ġbetween Ġthe Ġmodels , Ġas Ġmeasured Ġby Ġthe Ġstandard Ġdeviation ĠIN LINE FORM 0 Ġ, Ġgradually Ġdecreases Ġwith Ġthe Ġincreasing Ġlevel Ġof Ġpre processing . ĠThis Ġresult Ġstrengthens Ġthe Ġassumption Ġmade Ġin Ġthis Ġpaper , Ġthat Ġperformance Ġvariation Ġacross Ġmodels Ġis Ġpartly Ġa Ġfunction Ġof Ġthe Ġeffectiveness Ġof Ġdocument Ġpre processing . </s> S omew hat Ġsurprisingly , Ġthe Ġordering Ġof Ġthe Ġtwo Ġbest Ġmodels Ġrevers es Ġat Ġlevel Ġ3 . ĠThis Ġshowcases Ġthat Ġrankings Ġare Ġheavily Ġinfluenced Ġby Ġthe Ġpre processing Ġstage , Ġdespite Ġthe Ġcommon Ġlack Ġof Ġdetails Ġand Ġanalysis Ġit Ġsuffers Ġfrom Ġin Ġexplanatory Ġpapers . ĠWe Ġalso Ġremark Ġthat Ġthe Ġtop Ġperforming Ġmodel , Ġnamely ĠKP - M iner , Ġis Ġun super vised Ġwhich Ġsupports Ġthe Ġfindings Ġof ĠB IB REF 6 Ġindicating Ġthat Ġrecent Ġun super vised Ġapproaches Ġhave Ġrival ed Ġtheir Ġsupervised Ġcounterparts Ġin Ġperformance . </s> In Ġan Ġattempt Ġto Ġquantify Ġthe Ġperformance Ġvariation Ġacross Ġpre processing Ġlevels , Ġwe Ġcompute Ġthe Ġstandard Ġdeviation ĠIN LINE FORM 0 Ġfor Ġeach Ġmodel . ĠHere Ġwe Ġsee Ġthat Ġun super vised Ġmodels Ġare Ġmore Ġsensitive Ġto Ġnoisy Ġinput , Ġas Ġrevealed Ġby Ġhigher Ġstandard Ġdeviations . ĠWe Ġfound Ġtwo Ġmain Ġreasons Ġfor Ġthis . ĠFirst , Ġusing Ġmultiple Ġdiscrim in ative Ġfeatures Ġto Ġrank Ġkey phrase Ġcandidates Ġadds Ġinherent Ġrobust ness Ġto Ġthe Ġmodels . ĠSecond , Ġthe Ġsupervision Ġsignal Ġhelps Ġmodels Ġto Ġdisregard Ġnoise . </s> In ĠTable Ġ, Ġwe Ġcompare Ġthe Ġoutputs Ġof Ġthe Ġfive Ġmodels Ġby Ġmeasuring Ġthe Ġpercentage Ġof Ġvalid Ġkey ph r ases Ġthat Ġare Ġretrieved Ġby Ġall Ġmodels Ġat Ġonce Ġfor Ġeach Ġpre processing Ġlevel . ĠBy Ġthese Ġadditional Ġresults , Ġwe Ġaim Ġto Ġassess Ġwhether Ġdocument Ġpre processing Ġsmoot hes Ġdifferences Ġbetween Ġmodels . ĠWe Ġobserve Ġthat Ġthe Ġoverlap Ġbetween Ġthe Ġoutputs Ġof Ġthe Ġdifferent Ġmodels Ġincreases Ġalong Ġwith Ġthe Ġlevel Ġof Ġpre processing . ĠThis Ġsuggests Ġthat Ġdocument Ġpre processing Ġreduces Ġthe Ġeffect Ġthat Ġthe Ġkey phrase Ġextraction Ġmodel Ġin Ġitself Ġhas Ġon Ġoverall Ġperformance . ĠIn Ġother Ġwords , Ġthe Ġsingular ity Ġof Ġeach Ġmodel Ġfades Ġaway Ġgradually Ġwith Ġincrease Ġin Ġpre processing Ġeffort . </s> Rep rodu c ibility </s> Being Ġable Ġto Ġreproduce Ġexperimental Ġresults Ġis Ġa Ġcentral Ġaspect Ġof Ġthe Ġscientific Ġmethod . ĠWhile Ġassessing Ġthe Ġimportance Ġof Ġthe Ġpre processing Ġstage Ġfor Ġfive Ġapproaches , Ġwe Ġfound Ġthat Ġseveral Ġresults Ġwere Ġnot Ġreprodu cible , Ġas Ġshown Ġin ĠTable Ġ. </s> We Ġnote Ġthat Ġthe Ġtrends Ġfor Ġbas elines Ġand Ġhigh Ġranking Ġsystems Ġare Ġopposite : Ġcompared Ġto Ġthe Ġpublished Ġresults , Ġour Ġreproduction Ġof Ġtop Ġsystems Ġunder - per forms Ġand Ġour Ġreproduction Ġof Ġbas elines Ġover - per forms . ĠWe Ġhypothes ise Ġthat Ġthis Ġstems Ġfrom Ġa Ġdifference Ġin Ġhyper param eter Ġtuning , Ġincluding Ġthe Ġones Ġthat Ġthe Ġpre processing Ġstage Ġmakes Ġimplicit . ĠCompet itors Ġhave Ġstrong Ġincentives Ġto Ġcorrectly Ġoptimize Ġhyper param eters , Ġto Ġachieve Ġa Ġhigh Ġranking Ġand Ġget Ġmore Ġpublicity Ġfor Ġtheir Ġwork Ġwhile Ġcompetition Ġorganizers Ġmight Ġhave Ġthe Ġopposite Ġincentive : Ġtoo Ġstrong Ġa Ġbaseline Ġmight Ġnot Ġbe Ġconsidered Ġa Ġbaseline Ġanymore . </s> We Ġalso Ġobserve Ġthat Ġwith Ġthis Ġleveled Ġpre processing , Ġthe Ġgap Ġbetween Ġbas elines Ġand Ġtop Ġsystems Ġis Ġmuch Ġsmaller , Ġless ening Ġagain Ġthe Ġimportance Ġof Ġraw Ġscores Ġand Ġrankings Ġto Ġinterpret Ġthe Ġshared Ġtask Ġresults Ġand Ġemphasizing Ġthe Ġimportance Ġof Ġunderstanding Ġcorrectly Ġthe Ġpre processing Ġstage . </s> Additional Ġexperiments </s> In Ġthe Ġprevious Ġsections , Ġwe Ġprovided Ġempirical Ġevidence Ġthat Ġdocument Ġpre processing Ġweighs Ġheavily Ġon Ġthe Ġoutcome Ġof Ġkey phrase Ġextraction Ġmodels . ĠThis Ġraises Ġthe Ġquestion Ġof Ġwhether Ġfurther Ġimprovement Ġmight Ġbe Ġgained Ġfrom Ġa Ġmore Ġaggressive Ġpre processing . ĠTo Ġanswer Ġthis Ġquestion , Ġwe Ġtake Ġanother Ġstep Ġbeyond Ġcontent Ġfiltering Ġand Ġfurther Ġab ridge Ġthe Ġinput Ġtext Ġfrom Ġlevel Ġ3 Ġpre process ed Ġdocuments Ġusing Ġan Ġun super vised Ġsummar ization Ġtechnique . ĠMore Ġspecifically , Ġwe Ġkeep Ġthe Ġtitle Ġand Ġabstract Ġintact , Ġas Ġthey Ġare Ġthe Ġtwo Ġmost Ġkey phrase Ġdense Ġparts Ġwithin Ġscientific Ġarticles ĠB IB REF 4 Ġ, Ġand Ġselect Ġonly Ġthe Ġmost Ġcontent Ġbearing Ġsentences Ġfrom Ġthe Ġremaining Ġcontents . ĠTo Ġdo Ġthis , Ġsentences Ġare Ġordered Ġusing ĠText Rank ĠB IB REF 14 Ġand Ġthe Ġless Ġinformative Ġones , Ġas Ġdetermined Ġby Ġtheir ĠText Rank Ġscores Ġnormalized Ġby Ġtheir Ġlengths Ġin Ġwords , Ġare Ġfiltered Ġout . </s> Finding Ġthe Ġoptimal Ġsubset Ġof Ġsentences Ġfrom Ġalready Ġshortened Ġdocuments Ġis Ġhowever Ġno Ġtrivial Ġtask Ġas Ġmaximum Ġrecall Ġlin early Ġdecreases Ġwith Ġthe Ġnumber Ġof Ġsentences Ġdiscarded . ĠHere , Ġwe Ġsimply Ġset Ġthe Ġreduction Ġratio Ġto Ġ0 . 8 65 Ġso Ġthat Ġthe Ġaverage Ġmaximum Ġrecall Ġon Ġthe Ġtraining Ġset Ġdoes Ġnot Ġlose Ġmore Ġthan Ġ5 %. ĠTable Ġshows Ġthe Ġreduction Ġin Ġthe Ġaverage Ġnumber Ġof Ġsentences Ġand Ġwords Ġcompared Ġto Ġlevel Ġ3 Ġpre processing . </s> The Ġperformances Ġof Ġthe Ġkey phrase Ġextraction Ġmodels Ġat Ġlevel Ġ4 Ġpre processing Ġare Ġshown Ġin ĠTable Ġ. ĠWe Ġnote Ġthat Ġtwo Ġmodels , Ġnamely ĠTopic Rank Ġand ĠTF ĠIN LINE FORM 0 ĠIDF , Ġlose Ġon Ġperformance . ĠThese Ġtwo Ġmodels Ġmainly Ġrely Ġon Ġfrequency Ġcounts Ġto Ġrank Ġkey phrase Ġcandidates , Ġwhich Ġin Ġturn Ġbecome Ġless Ġreliable Ġat Ġlevel Ġ4 Ġbecause Ġof Ġthe Ġvery Ġshort Ġlength Ġof Ġthe Ġdocuments . ĠOther Ġmodels Ġhowever Ġhave Ġtheir Ġf - sc ores Ġonce Ġagain Ġincreased , Ġthus Ġindicating Ġthat Ġfurther Ġimprovement Ġis Ġpossible Ġfrom Ġmore Ġredu ctive Ġdocument Ġpre processing Ġstrategies . </s> Conclusion </s> In Ġthis Ġstudy , Ġwe Ġre - ass essed Ġthe Ġperformance Ġof Ġseveral Ġkey phrase Ġextraction Ġmodels Ġand Ġshowed Ġthat Ġperformance Ġvariation Ġacross Ġmodels Ġis Ġpartly Ġa Ġfunction Ġof Ġthe Ġeffectiveness Ġof Ġthe Ġdocument Ġpre processing . ĠOur Ġresults Ġalso Ġsuggest Ġthat Ġsupervised Ġkey phrase Ġextraction Ġmodels Ġare Ġmore Ġrobust Ġto Ġnoisy Ġinput . </s> Given Ġour Ġfindings , Ġwe Ġrecommend Ġthat Ġfuture Ġworks Ġuse Ġa Ġcommon Ġpre processing Ġto Ġevaluate Ġthe Ġinterest Ġof Ġkey phrase Ġextraction Ġapproaches . ĠFor Ġthis Ġreason Ġwe Ġmake Ġthe Ġfour Ġlevels Ġof Ġpre processing Ġused Ġin Ġthis Ġstudy Ġavailable Ġfor Ġthe Ġcommunity .