Yes
BOOLEAN
Yes
BOOLEAN
FLOAT SELECTED: Figure 2: News suggestion approach overview. However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay. Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively. While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf. Figure FIGREF2 ) is not mentioned in the page for Odisha. Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets. This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources. In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7 . FLOAT SELECTED: Figure 1: Comparing how cyclones are reported in Wikipedia entity pages.
We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages. Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years. The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach. In summary, we make the following contributions. The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history. The generated data represents one of the contributions of our paper. The datasets are the following: Entity Classes. We focus on a manually predetermined set of entity classes for which we expect to have news coverage. The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference. The entity classes were selected from the DBpedia class ontology. Figure FIGREF42 shows the number of entities per class for the years (2009-2014). News Articles. We extract all news references from the collected Wikipedia entity pages. The extracted news references are associated with the sections in which they appear. In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles. The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44 .
1703.10344
Introduction </s> Wikipedia Ġis Ġthe Ġlargest Ġsource Ġof Ġopen Ġand Ġcollabor atively Ġcurated Ġknowledge Ġin Ġthe Ġworld . ĠIntroduced Ġin Ġ2001 , Ġit Ġhas Ġevolved Ġinto Ġa Ġreference Ġwork Ġwith Ġaround Ġ5 m Ġpages Ġfor Ġthe ĠEnglish ĠWikipedia Ġalone . ĠIn Ġaddition , Ġentities Ġand Ġevent Ġpages Ġare Ġupdated Ġquickly Ġvia Ġcollaborative Ġediting Ġand Ġall Ġedits Ġare Ġencouraged Ġto Ġinclude Ġsource Ġcitations , Ġcreating Ġa Ġknowledge Ġbase Ġwhich Ġaims Ġat Ġbeing Ġboth Ġtimely Ġas Ġwell Ġas Ġauthoritative . ĠAs Ġa Ġresult , Ġit Ġhas Ġbecome Ġthe Ġpreferred Ġsource Ġof Ġinformation Ġconsumption Ġabout Ġentities Ġand Ġevents . ĠMore so , Ġthis Ġknowledge Ġis Ġharvested Ġand Ġutilized Ġin Ġbuilding Ġknowledge Ġbases Ġlike ĠY AG O ĠB IB REF 0 Ġand ĠDB pedia ĠB IB REF 1 Ġ, Ġand Ġused Ġin Ġapplications Ġlike Ġtext Ġcategor ization ĠB IB REF 2 Ġ, Ġentity Ġdis amb ig uation ĠB IB REF 3 Ġ, Ġentity Ġranking ĠB IB REF 4 Ġand Ġdistant Ġsupervision ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> However , Ġnot Ġall ĠWikipedia Ġpages Ġreferring Ġto Ġentities Ġ( entity Ġpages ) Ġare Ġcomprehensive : Ġrelevant Ġinformation Ġcan Ġeither Ġbe Ġmissing Ġor Ġadded Ġwith Ġa Ġdelay . ĠConsider Ġthe Ġcity Ġof ĠNew ĠOrleans Ġand Ġthe Ġstate Ġof ĠOd isha Ġwhich Ġwere Ġseverely Ġaffected Ġby Ġcycl ones ĠHurricane ĠKatrina Ġand ĠOd isha ĠCy clone , Ġrespectively . ĠWhile ĠKatrina Ġfinds Ġextensive Ġmention Ġin Ġthe Ġentity Ġpage Ġfor ĠNew ĠOrleans , ĠOd isha ĠCy clone Ġwhich Ġhas Ġ5 Ġtimes Ġmore Ġhuman Ġcasualties Ġ( cf . ĠFigure ĠFIG REF 2 Ġ) Ġis Ġnot Ġmentioned Ġin Ġthe Ġpage Ġfor ĠOd isha . ĠAr gu ably ĠKatrina Ġand ĠNew ĠOrleans Ġare Ġmore Ġpopular Ġentities , Ġbut ĠOd isha ĠCy clone Ġwas Ġalso Ġreported Ġextensively Ġin Ġnational Ġand Ġinternational Ġnews Ġoutlets . ĠThis Ġhighlights Ġthe Ġlack Ġof Ġimportant Ġfacts Ġin Ġtrunk Ġand Ġlong - tail Ġentity Ġpages , Ġeven Ġin Ġthe Ġpresence Ġof Ġrelevant Ġsources . ĠIn Ġaddition , Ġprevious Ġstudies Ġhave Ġshown Ġthat Ġthere Ġis Ġan Ġinherent Ġdelay Ġor Ġlag Ġwhen Ġfacts Ġare Ġadded Ġto Ġentity Ġpages ĠB IB REF 7 Ġ. </s> To Ġremedy Ġthese Ġproblems , Ġit Ġis Ġimportant Ġto Ġidentify Ġinformation Ġsources Ġthat Ġcontain Ġnovel Ġand Ġsalient Ġfacts Ġto Ġa Ġgiven Ġentity Ġpage . ĠHowever , Ġnot Ġall Ġinformation Ġsources Ġare Ġequal . ĠThe Ġonline Ġpresence Ġof Ġmajor Ġnews Ġoutlets Ġis Ġan Ġauthoritative Ġsource Ġdue Ġto Ġactive Ġeditorial Ġcontrol Ġand Ġtheir Ġarticles Ġare Ġalso Ġa Ġtimely Ġcontainer Ġof Ġfacts . ĠIn Ġaddition , Ġtheir Ġuse Ġis Ġin Ġline Ġwith Ġcurrent ĠWikipedia Ġediting Ġpractice , Ġas Ġis Ġshown Ġin ĠB IB REF 7 Ġthat Ġalmost Ġ20 % Ġof Ġcurrent Ġcitations Ġin Ġall Ġentity Ġpages Ġare Ġnews Ġarticles . ĠWe Ġtherefore Ġpropose Ġnews Ġsuggestion Ġas Ġa Ġnovel Ġtask Ġthat Ġenhances Ġentity Ġpages Ġand Ġreduces Ġdelay Ġwhile Ġkeeping Ġits Ġpages Ġauthoritative . </s> Ex isting Ġefforts Ġto Ġpopulate ĠWikipedia ĠB IB REF 8 Ġstart Ġfrom Ġan Ġentity Ġpage Ġand Ġthen Ġgenerate Ġcandidate Ġdocuments Ġabout Ġthis Ġentity Ġusing Ġan Ġexternal Ġsearch Ġengine Ġ( and Ġthen Ġpost - process Ġthem ). ĠHowever , Ġsuch Ġan Ġapproach Ġlacks Ġin Ġ( a ) Ġreprodu c ibility Ġsince Ġrankings Ġvary Ġwith Ġtime Ġwith Ġobvious Ġbias Ġto Ġrecent Ġnews Ġ( b ) Ġmaintain ability Ġsince Ġdocument Ġacquisition Ġfor Ġeach Ġentity Ġhas Ġto Ġbe Ġperiodically Ġperformed . ĠTo Ġthis Ġeffect , Ġour Ġnews Ġsuggestion Ġconsiders Ġa Ġnews Ġarticle Ġas Ġinput , Ġand Ġdetermines Ġif Ġit Ġis Ġvaluable Ġfor ĠWikipedia . ĠSpecifically , Ġgiven Ġan Ġinput Ġnews Ġarticle ĠIN LINE FORM 0 Ġand Ġa Ġstate Ġof ĠWikipedia , Ġthe Ġnews Ġsuggestion Ġproblem Ġidentifies Ġthe Ġentities Ġmentioned Ġin ĠIN LINE FORM 1 Ġwhose Ġentity Ġpages Ġcan Ġimprove Ġupon Ġsuggesting ĠIN LINE FORM 2 Ġ. ĠMost Ġof Ġthe Ġworks Ġon Ġknowledge Ġbase Ġacceleration ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġor ĠWikipedia Ġpage Ġgeneration ĠB IB REF 8 Ġrely Ġon Ġhigh Ġquality Ġinput Ġsources Ġwhich Ġare Ġthen Ġutilized Ġto Ġextract Ġtextual Ġfacts Ġfor ĠWikipedia Ġpage Ġpopulation . ĠIn Ġthis Ġwork , Ġwe Ġdo Ġnot Ġsuggest Ġsnippets Ġor Ġparaph r ases Ġbut Ġrather Ġentire Ġarticles Ġwhich Ġhave Ġa Ġhigh Ġpotential Ġimportance Ġfor Ġentity Ġpages . ĠThese Ġsuggested Ġnews Ġarticles Ġcould Ġbe Ġconsequently Ġused Ġfor Ġextraction , Ġsummar ization Ġor Ġpopulation Ġeither Ġmanually Ġor Ġautomatically ĠâĢĵ Ġall Ġof Ġwhich Ġrely Ġon Ġhigh Ġquality Ġand Ġrelevant Ġinput Ġsources . </s> We Ġidentify Ġfour Ġproperties Ġof Ġgood Ġnews Ġrecommendations : Ġsal ience , Ġrelative Ġauthority , Ġnovelty Ġand Ġplacement . ĠFirst , Ġwe Ġneed Ġto Ġidentify Ġthe Ġmost Ġsalient Ġentities Ġin Ġa Ġnews Ġarticle . ĠThis Ġis Ġdone Ġto Ġavoid Ġpollution Ġof Ġentity Ġpages Ġwith Ġonly Ġmarginally Ġrelated Ġnews . ĠSecond , Ġwe Ġneed Ġto Ġdetermine Ġwhether Ġthe Ġnews Ġis Ġimportant Ġto Ġthe Ġentity Ġas Ġonly Ġthe Ġmost Ġrelevant Ġnews Ġshould Ġbe Ġadded Ġto Ġa Ġprecise Ġreference Ġwork . ĠTo Ġdo Ġthis , Ġwe Ġcompute Ġthe Ġrelative Ġauthority Ġof Ġall Ġentities Ġin Ġthe Ġnews Ġarticle : Ġwe Ġcall Ġan Ġentity Ġmore Ġauthoritative Ġthan Ġanother Ġif Ġit Ġis Ġmore Ġpopular Ġor Ġnoteworthy Ġin Ġthe Ġreal Ġworld . ĠEnt ities Ġwith Ġvery Ġhigh Ġauthority Ġhave Ġmany Ġnews Ġitems Ġassociated Ġwith Ġthem Ġand Ġonly Ġthe Ġmost Ġrelevant Ġof Ġthese Ġshould Ġbe Ġincluded Ġin ĠWikipedia Ġwhereas Ġfor Ġentities Ġof Ġlower Ġauthority Ġthe Ġthreshold Ġfor Ġinclusion Ġof Ġa Ġnews Ġarticle Ġwill Ġbe Ġlower . ĠThird , Ġa Ġgood Ġrecommendation Ġshould Ġbe Ġable Ġto Ġidentify Ġnovel Ġnews Ġby Ġminimizing Ġredundancy Ġcoming Ġfrom Ġmultiple Ġnews Ġarticles . ĠFinally , Ġaddition Ġof Ġfacts Ġis Ġfacilitated Ġif Ġthe Ġrecommendations Ġare Ġfine - gr ained , Ġi . e ., Ġrecommendations Ġare Ġmade Ġon Ġthe Ġsection Ġlevel Ġrather Ġthan Ġthe Ġpage Ġlevel Ġ( pl acement ). </s> App roach Ġand ĠContributions . ĠWe Ġpropose Ġa Ġtwo - stage Ġnews Ġsuggestion Ġapproach Ġto Ġentity Ġpages . ĠIn Ġthe Ġfirst Ġstage , Ġwe Ġdetermine Ġwhether Ġa Ġnews Ġarticle Ġshould Ġbe Ġsuggested Ġfor Ġan Ġentity , Ġbased Ġon Ġthe Ġentity 's Ġsal ience Ġin Ġthe Ġnews Ġarticle , Ġits Ġrelative Ġauthority Ġand Ġthe Ġnovelty Ġof Ġthe Ġarticle Ġto Ġthe Ġentity Ġpage . ĠThe Ġsecond Ġstage Ġtakes Ġinto Ġaccount Ġthe Ġclass Ġof Ġthe Ġentity Ġfor Ġwhich Ġthe Ġnews Ġis Ġsuggested Ġand Ġconstructs Ġsection Ġtemplates Ġfrom Ġentities Ġof Ġthe Ġsame Ġclass . ĠThe Ġgeneration Ġof Ġsuch Ġtemplates Ġhas Ġthe Ġadvantage Ġof Ġsuggesting Ġand Ġexpanding Ġentity Ġpages Ġthat Ġdo Ġnot Ġhave Ġa Ġcomplete Ġsection Ġstructure Ġin ĠWikipedia , Ġexplicitly Ġaddressing Ġlong - tail Ġand Ġtrunk Ġentities . ĠAfterwards , Ġbased Ġon Ġthe Ġconstructed Ġtemplate Ġour Ġmethod Ġdetermines Ġthe Ġbest Ġfit Ġfor Ġthe Ġnews Ġarticle Ġwith Ġone Ġof Ġthe Ġsections . </s> We Ġevaluate Ġthe Ġproposed Ġapproach Ġon Ġa Ġnews Ġcorpus Ġconsisting Ġof Ġ351 , 98 2 Ġarticles Ġcrawled Ġfrom Ġthe Ġnews Ġexternal Ġreferences Ġin ĠWikipedia Ġfrom Ġ73 , 7 34 Ġentity Ġpages . ĠGiven Ġthe ĠWikipedia Ġsnapshot Ġat Ġa Ġgiven Ġyear Ġ( in Ġour Ġcase Ġ[ 2009 - 2014 ]), Ġwe Ġsuggest Ġnews Ġarticles Ġthat Ġmight Ġbe Ġcited Ġin Ġthe Ġcoming Ġyears . ĠThe Ġexisting Ġnews Ġreferences Ġin Ġthe Ġentity Ġpages Ġalong Ġwith Ġtheir Ġreference Ġdate Ġact Ġas Ġour Ġground - truth Ġto Ġevaluate Ġour Ġapproach . ĠIn Ġsummary , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions . </s> Related ĠWork </s> As Ġwe Ġsuggest Ġa Ġnew Ġproblem Ġthere Ġis Ġno Ġcurrent Ġwork Ġaddressing Ġexactly Ġthe Ġsame Ġtask . ĠHowever , Ġour Ġtask Ġhas Ġsimilarities Ġto ĠWikipedia Ġpage Ġgeneration Ġand Ġknowledge Ġbase Ġacceleration . ĠIn Ġaddition , Ġwe Ġtake Ġinspiration Ġfrom ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġmethods Ġfor Ġsal ience Ġdetection . </s> Wikipedia ĠPage ĠGeneration Ġis Ġthe Ġproblem Ġof Ġpop ulating ĠWikipedia Ġpages Ġwith Ġcontent Ġcoming Ġfrom Ġexternal Ġsources . ĠSau per Ġand ĠBar z il ay ĠB IB REF 8 Ġpropose Ġan Ġapproach Ġfor Ġautomatically Ġgenerating Ġwhole Ġentity Ġpages Ġfor Ġspecific Ġentity Ġclasses . ĠThe Ġapproach Ġis Ġtrained Ġon Ġalready - pop ulated Ġentity Ġpages Ġof Ġa Ġgiven Ġclass Ġ( e . g . Ġ` D ise ases ') Ġby Ġlearning Ġtemplates Ġabout Ġthe Ġentity Ġpage Ġstructure Ġ( e . g . Ġdiseases Ġhave Ġa Ġtreatment Ġsection ). ĠFor Ġa Ġnew Ġentity Ġpage , Ġfirst , Ġthey Ġextract Ġdocuments Ġvia ĠWeb Ġsearch Ġusing Ġthe Ġentity Ġtitle Ġand Ġthe Ġsection Ġtitle Ġas Ġa Ġquery , Ġfor Ġexample Ġ` L ung ĠCancer ' + ` T reatment '. ĠAs Ġalready Ġdiscussed Ġin Ġthe Ġintroduction , Ġthis Ġhas Ġproblems Ġwith Ġreprodu c ibility Ġand Ġmaintain ability . ĠHowever , Ġtheir Ġmain Ġfocus Ġis Ġon Ġidentifying Ġthe Ġbest Ġparagraphs Ġextracted Ġfrom Ġthe Ġcollected Ġdocuments . ĠThey Ġrank Ġthe Ġparagraphs Ġvia Ġan Ġoptimized Ġsupervised Ġpercept ron Ġmodel Ġfor Ġfinding Ġthe Ġmost Ġrepresentative Ġparagraph Ġthat Ġis Ġthe Ġleast Ġsimilar Ġto Ġparagraphs Ġin Ġother Ġsections . ĠThis Ġparagraph Ġis Ġthen Ġincluded Ġin Ġthe Ġnewly Ġgenerated Ġentity Ġpage . ĠT ane va Ġand ĠWe ik um ĠB IB REF 12 Ġpropose Ġan Ġapproach Ġthat Ġconstructs Ġshort Ġsumm aries Ġfor Ġthe Ġlong Ġtail . ĠThe Ġsumm aries Ġare Ġcalled Ġ` g ems ' Ġand Ġthe Ġsize Ġof Ġa Ġ` gem ' Ġcan Ġbe Ġuser Ġdefined . ĠThey Ġfocus Ġon Ġgenerating Ġsumm aries Ġthat Ġare Ġnovel Ġand Ġdiverse . ĠHowever , Ġthey Ġdo Ġnot Ġconsider Ġany Ġstructure Ġof Ġentities , Ġwhich Ġis Ġpresent Ġin ĠWikipedia . </s> In Ġcontrast Ġto ĠB IB REF 8 Ġand ĠB IB REF 12 Ġ, Ġwe Ġactually Ġfocus Ġon Ġsuggesting Ġentire Ġdocuments Ġto ĠWikipedia Ġentity Ġpages . ĠThese Ġare Ġauthoritative Ġdocuments Ġ( news ), Ġwhich Ġare Ġhighly Ġrelevant Ġfor Ġthe Ġentity , Ġnovel Ġfor Ġthe Ġentity Ġand Ġin Ġwhich Ġthe Ġentity Ġis Ġsalient . ĠWhereas Ġrelevance Ġin ĠSau per Ġand ĠBar z il ay Ġis Ġimplicitly Ġcomputed Ġby Ġweb Ġpage Ġranking Ġwe Ġsolve Ġthat Ġproblem Ġby Ġlooking Ġat Ġrelative Ġauthority Ġand Ġsal ience Ġof Ġan Ġentity , Ġusing Ġthe Ġnews Ġarticle Ġand Ġentity Ġpage Ġonly . ĠAs ĠSau per Ġand ĠBar z il ay Ġconcentrate Ġon Ġempty Ġentity Ġpages , Ġthe Ġproblem Ġof Ġnovelty Ġof Ġtheir Ġcontent Ġis Ġnot Ġan Ġissue Ġin Ġtheir Ġwork Ġwhereas Ġit Ġis Ġin Ġour Ġcase Ġwhich Ġfocuses Ġmore Ġon Ġupdating Ġentities . ĠUp dating Ġentities Ġwill Ġbe Ġmore Ġand Ġmore Ġimportant Ġthe Ġbigger Ġan Ġexisting Ġreference Ġwork Ġis . ĠBoth Ġthe Ġapproaches Ġin ĠB IB REF 8 Ġand ĠB IB REF 12 Ġ( finding Ġparagraphs Ġand Ġsummar ization ) Ġcould Ġthen Ġbe Ġused Ġto Ġprocess Ġthe Ġdocuments Ġwe Ġsuggest Ġfurther . ĠOur Ġconcentration Ġon Ġnews Ġis Ġalso Ġnovel . </s> Know ledge ĠBase ĠAcceler ation . ĠIn Ġthis Ġtask , Ġgiven Ġspecific Ġinformation Ġextraction Ġtemplates , Ġa Ġgiven Ġcorpus Ġis Ġanalyzed Ġin Ġorder Ġto Ġfind Ġworthwhile Ġmentions Ġof Ġan Ġentity Ġor Ġsnippets Ġthat Ġmatch Ġthe Ġtemplates . ĠBal og ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġrecommend Ġnews Ġcitations Ġfor Ġan Ġentity . ĠPrior Ġto Ġthat , Ġthe Ġnews Ġarticles Ġare Ġclassified Ġfor Ġtheir Ġappropri at eness Ġfor Ġan Ġentity , Ġwhere Ġas Ġfeatures Ġfor Ġthe Ġclassification Ġtask Ġthey Ġuse Ġentity , Ġdocument , Ġentity - document Ġand Ġtemporal Ġfeatures . ĠThe Ġbest Ġperforming Ġfeatures Ġare Ġthose Ġthat Ġmeasure Ġsimilarity Ġbetween Ġan Ġentity Ġand Ġthe Ġnews Ġdocument . ĠWest Ġet Ġal . ĠB IB REF 13 Ġconsider Ġthe Ġproblem Ġof Ġknowledge Ġbase Ġcompletion , Ġthrough Ġquestion Ġanswering Ġand Ġcomplete Ġmissing Ġfacts Ġin ĠFree base Ġbased Ġon Ġtemplates , Ġi . e . ĠFrank _ Z appa Ġborn In ĠB alty more , ĠMaryland . </s> In Ġcontrast , Ġwe Ġdo Ġnot Ġextract Ġfacts Ġfor Ġpre - defined Ġtemplates Ġbut Ġrather Ġsuggest Ġnews Ġarticles Ġbased Ġon Ġtheir Ġrelevance Ġto Ġan Ġentity . ĠIn Ġcases Ġof Ġlong - tail Ġentities , Ġwe Ġcan Ġsuggest Ġto Ġadd Ġa Ġnovel Ġsection Ġthrough Ġour Ġabstraction Ġand Ġgeneration Ġof Ġsection Ġtemplates Ġat Ġentity Ġclass Ġlevel . </s> Entity ĠSal ience . ĠD etermin ing Ġwhich Ġentities Ġare Ġprominent Ġor Ġsalient Ġin Ġa Ġgiven Ġtext Ġhas Ġa Ġlong Ġhistory Ġin ĠN LP , Ġsparked Ġby Ġthe Ġlinguistic Ġtheory Ġof ĠCent ering ĠB IB REF 14 Ġ. ĠSal ience Ġhas Ġbeen Ġused Ġin Ġpronoun Ġand Ġco - reference Ġresolution ĠB IB REF 15 Ġ, Ġor Ġto Ġpredict Ġwhich Ġentities Ġwill Ġbe Ġincluded Ġin Ġan Ġabstract Ġof Ġan Ġarticle ĠB IB REF 11 Ġ. ĠFrequ ent Ġfeatures Ġto Ġmeasure Ġsal ience Ġinclude Ġthe Ġfrequency Ġof Ġan Ġentity Ġin Ġa Ġdocument , Ġpositioning Ġof Ġan Ġentity , Ġgram matical Ġfunction Ġor Ġinternal Ġentity Ġstructure Ġ( POS Ġtags , Ġhead Ġnoun s Ġetc .). ĠThese Ġapproaches Ġare Ġnot Ġcurrently Ġaimed Ġat Ġknowledge Ġbase Ġgeneration Ġor ĠWikipedia Ġcoverage Ġextension Ġbut Ġwe Ġpost ulate Ġthat Ġan Ġentity 's Ġsal ience Ġin Ġa Ġnews Ġarticle Ġis Ġa Ġprerequisite Ġto Ġthe Ġnews Ġarticle Ġbeing Ġrelevant Ġenough Ġto Ġbe Ġincluded Ġin Ġan Ġentity Ġpage . ĠWe Ġtherefore Ġuse Ġthe Ġsal ience Ġfeatures Ġin ĠB IB REF 11 Ġas Ġpart Ġof Ġour Ġmodel . ĠHowever , Ġthese Ġfeatures Ġare Ġdocument - internal ĠâĢĶ Ġwe Ġwill Ġshow Ġthat Ġthey Ġare Ġnot Ġsufficient Ġto Ġpredict Ġnews Ġinclusion Ġinto Ġan Ġentity Ġpage Ġand Ġadd Ġfeatures Ġof Ġentity Ġauthority , Ġnews Ġauthority Ġand Ġnovelty Ġthat Ġmeasure Ġthe Ġrelations Ġbetween Ġseveral Ġentities , Ġbetween Ġentity Ġand Ġnews Ġarticle Ġas Ġwell Ġas Ġbetween Ġseveral Ġcompeting Ġnews Ġarticles . </s> Termin ology Ġand ĠProblem ĠDefinition </s> We Ġare Ġinterested Ġin Ġnamed Ġentities Ġmentioned Ġin Ġdocuments . ĠAn Ġentity ĠIN LINE FORM 0 Ġcan Ġbe Ġidentified Ġby Ġa Ġcanonical Ġname , Ġand Ġcan Ġbe Ġmentioned Ġdifferently Ġin Ġtext Ġvia Ġdifferent Ġsurface Ġforms . ĠWe Ġcanonical ize Ġthese Ġmentions Ġto Ġentity Ġpages Ġin ĠWikipedia , Ġa Ġmethod Ġtypically Ġknown Ġas Ġentity Ġlinking . ĠWe Ġdenote Ġthe Ġset Ġof Ġcanonical ized Ġentities Ġextracted Ġand Ġlinked Ġfrom Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġas ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , Ġin ĠFigure ĠFIG REF 7 Ġ, Ġentities Ġare Ġcanonical ized Ġinto ĠWikipedia Ġentity Ġpages Ġ( e . g . ĠOd isha Ġis Ġcanonical ized Ġto Ġthe Ġcorresponding Ġarticle ). ĠFor Ġa Ġcollection Ġof Ġnews Ġarticles ĠIN LINE FORM 3 Ġ, Ġwe Ġfurther Ġdenote Ġthe Ġresulting Ġset Ġof Ġentities Ġby ĠIN LINE FORM 4 Ġ. </s> Information Ġin Ġan Ġentity Ġpage Ġis Ġorganized Ġinto Ġsections Ġand Ġevolves Ġwith Ġtime Ġas Ġmore Ġcontent Ġis Ġadded . ĠWe Ġrefer Ġto Ġthe Ġstate Ġof ĠWikipedia Ġat Ġa Ġtime ĠIN LINE FORM 0 Ġas ĠIN LINE FORM 1 Ġand Ġthe Ġset Ġof Ġsections Ġfor Ġan Ġentity Ġpage ĠIN LINE FORM 2 Ġas Ġits Ġentity Ġprofile ĠIN LINE FORM 3 Ġ. ĠUnlike Ġnews Ġarticles , Ġtext Ġin ĠWikipedia Ġcould Ġbe Ġexplicitly Ġlinked Ġto Ġentity Ġpages Ġthrough Ġanchors . ĠThe Ġset Ġof Ġentities Ġexplicitly Ġreferred Ġin Ġtext Ġfrom Ġsection ĠIN LINE FORM 4 Ġis Ġdefined Ġas ĠIN LINE FORM 5 Ġ. ĠFurthermore , ĠWikipedia Ġinduces Ġa Ġcategory Ġstructure Ġover Ġits Ġentities , Ġwhich Ġis Ġexploited Ġby Ġknowledge Ġbases Ġlike ĠY AG O Ġ( e . g . ĠBarack _ Obama Ġis A ĠPerson ). ĠConsequently , Ġeach Ġentity Ġpage Ġbelongs Ġto Ġone Ġor Ġmore Ġentity Ġcategories Ġor Ġclasses ĠIN LINE FORM 6 Ġ. ĠNow Ġwe Ġcan Ġdefine Ġour Ġnews Ġsuggestion Ġproblem Ġbelow : </s> Definition Ġ1 Ġ( News ĠSuggest ion ĠProblem ) ĠGiven Ġa Ġset Ġof Ġnews Ġarticles ĠIN LINE FORM 0 Ġand Ġset Ġof ĠWikipedia Ġentity Ġpages ĠIN LINE FORM 1 Ġ( from ĠIN LINE FORM 2 Ġ) Ġwe Ġintend Ġto Ġsuggest Ġa Ġnews Ġarticle ĠIN LINE FORM 3 Ġpublished Ġat Ġtime ĠIN LINE FORM 4 Ġto Ġentity Ġpage ĠIN LINE FORM 5 Ġand Ġadditionally Ġto Ġthe Ġmost Ġrelevant Ġsection Ġfor Ġthe Ġentity Ġpage ĠIN LINE FORM 6 Ġ. </s> App roach ĠOverview </s> We Ġapproach Ġthe Ġnews Ġsuggestion Ġproblem Ġby Ġdecom posing Ġit Ġinto Ġtwo Ġtasks : </s> A EP : ĠArticle âĢĵ Entity Ġplacement </s> AS P : ĠArticle âĢĵ Section Ġplacement </s> In Ġthis Ġfirst Ġstep , Ġfor Ġa Ġgiven Ġentity - news Ġpair ĠIN LINE FORM 0 Ġ, Ġwe Ġdetermine Ġwhether Ġthe Ġgiven Ġnews Ġarticle ĠIN LINE FORM 1 Ġshould Ġbe Ġsuggested Ġ( we Ġwill Ġrefer Ġto Ġthis Ġas Ġ` relevant ') Ġto Ġentity ĠIN LINE FORM 2 Ġ. ĠTo Ġgenerate Ġsuch ĠIN LINE FORM 3 Ġpairs , Ġwe Ġperform Ġthe Ġentity Ġlinking Ġprocess , ĠIN LINE FORM 4 Ġ, Ġfor ĠIN LINE FORM 5 Ġ. </s> The Ġarticle âĢĵ entity Ġplacement Ġtask Ġ( described Ġin Ġdetail Ġin ĠSection ĠSEC REF 16 Ġ) Ġfor Ġa Ġpair ĠIN LINE FORM 0 Ġoutputs Ġa Ġbinary Ġlabel Ġ( either Ġ` non - relevant ' Ġor Ġ` relevant ') Ġand Ġis Ġformal ized Ġin ĠEqu ation ĠEQ REF 14 Ġ. ĠDIS PLAY FORM 0 </s> In Ġthe Ġsecond Ġstep , Ġwe Ġtake Ġinto Ġaccount Ġall Ġ` relevant ' Ġpairs ĠIN LINE FORM 0 Ġand Ġfind Ġthe Ġcorrect Ġsection Ġfor Ġarticle ĠIN LINE FORM 1 Ġin Ġentity ĠIN LINE FORM 2 Ġ, Ġrespectively Ġits Ġprofile ĠIN LINE FORM 3 Ġ( see ĠSection ĠSEC REF 30 Ġ). ĠThe Ġarticle âĢĵ section Ġplacement Ġtask , Ġdetermines Ġthe Ġcorrect Ġsection Ġfor Ġthe Ġtriple ĠIN LINE FORM 4 Ġ, Ġand Ġis Ġformal ized Ġin ĠEqu ation ĠEQ REF 15 Ġ. ĠDIS PLAY FORM 0 </s> In Ġthe Ġsubsequent Ġsections Ġwe Ġdescribe Ġin Ġdetails Ġhow Ġwe Ġapproach Ġthe Ġtwo Ġtasks Ġfor Ġsuggesting Ġnews Ġarticles Ġto Ġentity Ġpages . </s> News ĠArticle ĠSuggest ion </s> In Ġthis Ġsection , Ġwe Ġprovide Ġan Ġoverview Ġof Ġthe Ġnews Ġsuggestion Ġapproach Ġto ĠWikipedia Ġentity Ġpages Ġ( see ĠFigure ĠFIG REF 7 Ġ). ĠThe Ġapproach Ġis Ġsplit Ġinto Ġtwo Ġtasks : Ġ( i ) Ġarticle - entity Ġ( A EP ) Ġand Ġ( ii ) Ġarticle - section Ġ( AS P ) Ġplacement . ĠFor Ġa ĠWikipedia Ġsnapshot ĠIN LINE FORM 0 Ġand Ġa Ġnews Ġcorpus ĠIN LINE FORM 1 Ġ, Ġwe Ġfirst Ġdetermine Ġwhich Ġnews Ġarticles Ġshould Ġbe Ġsuggested Ġto Ġan Ġentity ĠIN LINE FORM 2 Ġ. ĠWe Ġwill Ġdenote Ġour Ġapproach Ġfor ĠA EP Ġby ĠIN LINE FORM 3 Ġ. ĠFinally , Ġwe Ġdetermine Ġthe Ġmost Ġappropriate Ġsection Ġfor Ġthe ĠASP Ġtask Ġand Ġwe Ġdenote Ġour Ġapproach Ġwith ĠIN LINE FORM 4 Ġ. </s> In Ġthe Ġfollowing , Ġwe Ġdescribe Ġthe Ġprocess Ġof Ġlearning Ġthe Ġfunctions ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠWe Ġintroduce Ġfeatures Ġfor Ġthe Ġlearning Ġprocess , Ġwhich Ġencode Ġinformation Ġregarding Ġthe Ġentity Ġsal ience , Ġrelative Ġauthority Ġand Ġnovelty Ġin Ġthe Ġcase Ġof ĠA EP Ġtask . ĠFor Ġthe ĠASP Ġtask , Ġwe Ġmeasure Ġthe Ġoverall Ġfit Ġof Ġan Ġarticle Ġto Ġthe Ġentity Ġsections , Ġwith Ġthe Ġentity Ġbeing Ġan Ġinput Ġfrom ĠA EP Ġtask . ĠAdditionally , Ġconsidering Ġthat Ġthe Ġentity Ġprofiles ĠIN LINE FORM 2 Ġare Ġincomplete , Ġin Ġthe Ġcase Ġof Ġa Ġmissing Ġsection Ġwe Ġsuggest Ġand Ġexpand Ġthe Ġentity Ġprofiles Ġbased Ġon Ġsection Ġtemplates Ġgenerated Ġfrom Ġentities Ġof Ġthe Ġsame Ġclass ĠIN LINE FORM 3 Ġ( see ĠSection ĠUID 34 Ġ). </s> Article âĢĵ Entity ĠPl acement </s> In Ġthis Ġstep Ġwe Ġlearn Ġthe Ġfunction ĠIN LINE FORM 0 Ġto Ġcorrectly Ġdetermine Ġwhether ĠIN LINE FORM 1 Ġshould Ġbe Ġsuggested Ġfor ĠIN LINE FORM 2 Ġ, Ġbasically Ġa Ġbinary Ġclassification Ġmodel Ġ( 0 = ` non - relevant ' Ġand Ġ1 = ` relevant '). ĠNote Ġthat Ġwe Ġare Ġmainly Ġinterested Ġin Ġfinding Ġthe Ġrelevant Ġpairs Ġin Ġthis Ġtask . ĠFor Ġevery Ġnews Ġarticle , Ġthe Ġnumber Ġof Ġdis amb ig uated Ġentities Ġis Ġaround Ġ30 Ġ( but ĠIN LINE FORM 3 Ġis Ġsuggested Ġfor Ġonly Ġtwo Ġof Ġthem Ġon Ġaverage ). ĠTherefore , Ġthe Ġdistribution Ġof Ġ` non - relevant ' Ġand Ġ` relevant ' Ġpairs Ġis Ġskewed Ġtowards Ġthe Ġearlier , Ġand Ġby Ġsimply Ġchoosing Ġthe Ġ` non - relevant ' Ġlabel Ġwe Ġcan Ġachieve Ġa Ġhigh Ġaccuracy Ġfor ĠIN LINE FORM 4 Ġ. ĠFinding Ġthe Ġrelevant Ġpairs Ġis Ġtherefore Ġa Ġconsiderable Ġchallenge . </s> An Ġarticle ĠIN LINE FORM 0 Ġis Ġsuggested Ġto ĠIN LINE FORM 1 Ġby Ġour Ġfunction ĠIN LINE FORM 2 Ġif Ġit Ġfulf ills Ġthe Ġfollowing Ġproperties . ĠThe Ġentity ĠIN LINE FORM 3 Ġis Ġsalient Ġin ĠIN LINE FORM 4 Ġ( a Ġcentral Ġconcept ), Ġtherefore Ġensuring Ġthat ĠIN LINE FORM 5 Ġis Ġabout ĠIN LINE FORM 6 Ġand Ġthat ĠIN LINE FORM 7 Ġis Ġimportant Ġfor ĠIN LINE FORM 8 Ġ. ĠNext , Ġgiven Ġthe Ġfact Ġthere Ġmight Ġbe Ġmany Ġarticles Ġin Ġwhich ĠIN LINE FORM 9 Ġis Ġsalient , Ġwe Ġalso Ġlook Ġat Ġthe Ġreverse Ġproperty , Ġnamely Ġwhether ĠIN LINE FORM 10 Ġis Ġimportant Ġfor ĠIN LINE FORM 11 Ġ. ĠWe Ġdo Ġthis Ġby Ġcomparing Ġthe Ġauthority Ġof ĠIN LINE FORM 12 Ġ( which Ġis Ġa Ġmeasure Ġof Ġpopularity Ġof Ġan Ġentity , Ġsuch Ġas Ġits Ġfrequency Ġof Ġmention Ġin Ġa Ġwhole Ġcorpus ) Ġwith Ġthe Ġauthority Ġof Ġits Ġco - occ urring Ġentities Ġin ĠIN LINE FORM 13 Ġ, Ġleading Ġto Ġa Ġfeature Ġwe Ġcall Ġrelative Ġauthority . ĠThe Ġintuition Ġis Ġthat Ġfor Ġan Ġentity Ġthat Ġhas Ġoverall Ġlower Ġauthority Ġthan Ġits Ġco - occ urring Ġentities , Ġa Ġnews Ġarticle Ġis Ġmore Ġeasily Ġof Ġimportance . ĠFinally , Ġif Ġthe Ġarticle Ġwe Ġare Ġabout Ġto Ġsuggest Ġis Ġalready Ġcovered Ġin Ġthe Ġentity Ġprofile ĠIN LINE FORM 14 Ġ, Ġwe Ġdo Ġnot Ġwish Ġto Ġsuggest Ġredundant Ġinformation , Ġhence Ġthe Ġnovelty . ĠTherefore , Ġthe Ġlearning Ġobjective Ġof ĠIN LINE FORM 15 Ġshould Ġfulfill Ġthe Ġfollowing Ġproperties . ĠTable ĠT AB REF 21 Ġshows Ġa Ġsummary Ġof Ġthe Ġcomputed Ġfeatures Ġfor ĠIN LINE FORM 16 Ġ. </s> Sal ience : Ġentity ĠIN LINE FORM 0 Ġshould Ġbe Ġa Ġsalient Ġentity Ġin Ġnews Ġarticle ĠIN LINE FORM 1 </s> Rel ative ĠAuthority : Ġthe Ġset Ġof Ġentities ĠIN LINE FORM 0 Ġwith Ġwhich ĠIN LINE FORM 1 Ġco - occ urs Ġshould Ġhave Ġhigher Ġauthority Ġthan ĠIN LINE FORM 2 Ġ, Ġmaking ĠIN LINE FORM 3 Ġimportant Ġfor ĠIN LINE FORM 4 </s> No vel ty : Ġnews Ġarticle ĠIN LINE FORM 0 Ġshould Ġprovide Ġnovel Ġinformation Ġfor Ġentity ĠIN LINE FORM 1 Ġtaking Ġinto Ġaccount Ġits Ġprofile ĠIN LINE FORM 2 </s> Bas eline ĠFeatures . ĠAs Ġdiscussed Ġin ĠSection ĠSEC REF 2 Ġ, Ġa Ġvariety Ġof Ġfeatures Ġthat Ġmeasure Ġsal ience Ġof Ġan Ġentity Ġin Ġtext Ġare Ġavailable Ġfrom Ġthe ĠN LP Ġcommunity . ĠWe Ġreim ple mented Ġthe Ġones Ġin ĠDun iet z Ġand ĠGill ick ĠB IB REF 11 Ġ. ĠThis Ġincludes Ġa Ġvariety Ġof Ġfeatures , Ġe . g . Ġpositional Ġfeatures , Ġoccurrence Ġfrequency Ġand Ġthe Ġinternal ĠPOS Ġstructure Ġof Ġthe Ġentity Ġand Ġthe Ġsentence Ġit Ġoccurs Ġin . ĠTable Ġ2 Ġin ĠB IB REF 11 Ġgives Ġdetails . </s> Rel ative ĠEntity ĠFrequency . ĠAlthough Ġfrequency Ġof Ġmention Ġand Ġpositional Ġfeatures Ġplay Ġsome Ġrole Ġin Ġbaseline Ġfeatures , Ġtheir Ġinteraction Ġis Ġnot Ġmodeled Ġby Ġa Ġsingle Ġfeature Ġnor Ġdo Ġthe Ġpositional Ġfeatures Ġencode Ġmore Ġthan Ġsentence Ġposition . ĠWe Ġtherefore Ġsuggest Ġa Ġnovel Ġfeature Ġcalled Ġrelative Ġentity Ġfrequency , ĠIN LINE FORM 0 Ġ, Ġthat Ġhas Ġthree Ġproperties .: Ġ( i ) ĠIt Ġrewards Ġentities Ġfor Ġoccurring Ġthroughout Ġthe Ġtext Ġinstead Ġof Ġonly Ġin Ġsome Ġparts Ġof Ġthe Ġtext , Ġmeasured Ġby Ġthe Ġnumber Ġof Ġparagraphs Ġit Ġoccurs Ġin Ġ( ii ) Ġit Ġrewards Ġentities Ġthat Ġoccur Ġmore Ġfrequently Ġin Ġthe Ġopening Ġparagraphs Ġof Ġan Ġarticle Ġas Ġwe Ġmodel ĠIN LINE FORM 1 Ġas Ġan Ġexponential Ġdecay Ġfunction . ĠThe Ġdecay Ġcorresponds Ġto Ġthe Ġpositional Ġindex Ġof Ġthe Ġnews Ġparagraph . ĠThis Ġis Ġinspired Ġby Ġthe Ġnews - specific Ġdiscourse Ġstructure Ġthat Ġtends Ġto Ġgive Ġshort Ġsumm aries Ġof Ġthe Ġmost Ġimportant Ġfacts Ġand Ġentities Ġin Ġthe Ġopening Ġparagraphs . Ġ( iii ) Ġit Ġcompares Ġentity Ġfrequency Ġto Ġthe Ġfrequency Ġof Ġits Ġco - occ urring Ġmentions Ġas Ġthe Ġweight Ġof Ġan Ġentity Ġappearing Ġin Ġa Ġspecific Ġparagraph , Ġnormalized Ġby Ġthe Ġsum Ġof Ġthe Ġfrequencies Ġof Ġother Ġentities Ġin ĠIN LINE FORM 2 Ġ. ĠDIS PLAY FORM 0 </s> where , ĠIN LINE FORM 0 Ġrepresents Ġa Ġnews Ġparagraph Ġfrom ĠIN LINE FORM 1 Ġ, Ġand Ġwith ĠIN LINE FORM 2 Ġwe Ġindicate Ġthe Ġset Ġof Ġall Ġparagraphs Ġin ĠIN LINE FORM 3 Ġ. ĠThe Ġfrequency Ġof ĠIN LINE FORM 4 Ġin Ġa Ġparagraph ĠIN LINE FORM 5 Ġis Ġden oted Ġby ĠIN LINE FORM 6 Ġ. ĠWith ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġwe Ġindicate Ġthe Ġnumber Ġof Ġparagraphs Ġin Ġwhich Ġentity ĠIN LINE FORM 9 Ġoccurs , Ġand Ġthe Ġtotal Ġnumber Ġof Ġparagraphs , Ġrespectively . </s> Rel ative ĠAuthority . ĠIn Ġthis Ġcase , Ġwe Ġconsider Ġthe Ġcomparative Ġrelevance Ġof Ġthe Ġnews Ġarticle Ġto Ġthe Ġdifferent Ġentities Ġoccurring Ġin Ġit . ĠAs Ġan Ġexample , Ġlet Ġus Ġconsider Ġthe Ġmeeting Ġof Ġthe ĠSudan ese Ġbishop ĠElias ĠTab an Ġwith ĠHillary ĠClinton . ĠBoth Ġentities Ġare Ġsalient Ġfor Ġthe Ġmeeting . ĠHowever , Ġin ĠTab an 's ĠWikipedia Ġpage , Ġthis Ġmeeting Ġis Ġdiscussed Ġprominently Ġwith Ġa Ġcorresponding Ġnews Ġreference , Ġwhereas Ġin ĠHillary ĠClinton 's ĠWikipedia Ġpage Ġit Ġis Ġnot Ġreported Ġat Ġall . ĠWe Ġbelieve Ġthis Ġis Ġnot Ġjust Ġan Ġomission Ġin ĠClinton 's Ġpage Ġbut Ġmirrors Ġthe Ġfact Ġthat Ġfor Ġthe Ġlesser Ġknown ĠTab an Ġthe Ġmeeting Ġis Ġbig Ġnews Ġwhereas Ġfor Ġthe Ġmore Ġfamous ĠClinton Ġthese Ġkind Ġof Ġmeetings Ġare Ġa Ġregular Ġoccurrence , Ġnot Ġall Ġof Ġwhich Ġcan Ġbe Ġreported Ġin Ġwhat Ġis Ġsupposed Ġto Ġbe Ġa Ġselection Ġof Ġthe Ġmost Ġimportant Ġevents Ġfor Ġher . ĠTherefore , Ġif Ġtwo Ġentities Ġco - occ ur , Ġthe Ġnews Ġis Ġmore Ġrelevant Ġfor Ġthe Ġentity Ġwith Ġthe Ġlower Ġa Ġprior i Ġauthority . </s> The Ġa Ġprior i Ġauthority Ġof Ġan Ġentity Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ) Ġcan Ġbe Ġmeasured Ġin Ġseveral Ġways . ĠWe Ġopt Ġfor Ġtwo Ġapproaches : Ġ( i ) Ġprobability Ġof Ġentity ĠIN LINE FORM 1 Ġoccurring Ġin Ġthe Ġcorpus ĠIN LINE FORM 2 Ġ, Ġand Ġ( ii ) Ġauthority Ġassessed Ġthrough Ġcentral ity Ġmeasures Ġlike ĠPage Rank ĠB IB REF 16 Ġ. ĠFor Ġthe Ġsecond Ġcase Ġwe Ġconstruct Ġthe Ġgraph ĠIN LINE FORM 3 Ġconsisting Ġof Ġentities Ġin ĠIN LINE FORM 4 Ġand Ġnews Ġarticles Ġin ĠIN LINE FORM 5 Ġas Ġvert ices . ĠThe Ġedges Ġare Ġestablished Ġbetween ĠIN LINE FORM 6 Ġand Ġentities Ġin ĠIN LINE FORM 7 Ġ, Ġthat Ġis ĠIN LINE FORM 8 Ġ, Ġand Ġthe Ġout - links Ġfrom ĠIN LINE FORM 9 Ġ, Ġthat Ġis ĠIN LINE FORM 10 Ġ( arrow s Ġpresent Ġthe Ġedge Ġdirection ). </s> Starting Ġfrom Ġa Ġprior i Ġauthority , Ġwe Ġproceed Ġto Ġrelative Ġauthority Ġby Ġcomparing Ġthe Ġa Ġprior i Ġauthority Ġof Ġco - occ urring Ġentities Ġin ĠIN LINE FORM 0 Ġ. ĠWe Ġdefine Ġthe Ġrelative Ġauthority Ġof ĠIN LINE FORM 1 Ġas Ġthe Ġproportion Ġof Ġco - occ urring Ġentities ĠIN LINE FORM 2 Ġthat Ġhave Ġa Ġhigher Ġa Ġprior i Ġauthority Ġthan ĠIN LINE FORM 3 Ġ( see ĠEqu ation ĠEQ REF 28 Ġ. ĠDIS PLAY FORM 0 </s> As Ġwe Ġmight Ġrun Ġthe Ġdanger Ġof Ġnot Ġsuggesting Ġany Ġnews Ġarticles Ġfor Ġentities Ġwith Ġvery Ġhigh Ġa Ġprior i Ġauthority Ġ( such Ġas ĠClinton ) Ġdue Ġto Ġthe Ġstrict Ġinequality Ġconstraint , Ġwe Ġcan Ġrelax Ġthe Ġconstraint Ġsuch Ġthat Ġthe Ġauthority Ġof Ġco - occ urring Ġentities Ġis Ġabove Ġa Ġcertain Ġthreshold . </s> News ĠDomain ĠAuthority . ĠThe Ġnews Ġdomain Ġauthority Ġaddresses Ġtwo Ġmain Ġaspects . ĠFirstly , Ġif Ġbundled Ġtogether Ġwith Ġthe Ġrelative Ġauthority Ġfeature , Ġwe Ġcan Ġensure Ġthat Ġdependent Ġon Ġthe Ġentity Ġauthority , Ġwe Ġsuggest Ġnews Ġfrom Ġauthoritative Ġsources , Ġhence Ġensuring Ġthe Ġquality Ġof Ġsuggested Ġarticles . ĠThe Ġsecond Ġaspect Ġis Ġin Ġa Ġnews Ġstreaming Ġscenario Ġwhere Ġmultiple Ġnews Ġdomains Ġreport Ġthe Ġsame Ġevent ĠâĢĶ Ġideally Ġonly Ġarticles Ġcoming Ġfrom Ġauthoritative Ġsources Ġwould Ġfulfill Ġthe Ġconditions Ġfor Ġthe Ġnews Ġsuggestion Ġtask . </s> The Ġnews Ġdomain Ġauthority Ġis Ġcomputed Ġbased Ġon Ġthe Ġnumber Ġof Ġnews Ġreferences Ġin ĠWikipedia Ġcoming Ġfrom Ġa Ġparticular Ġnews Ġdomain ĠIN LINE FORM 0 Ġ. ĠThis Ġrepresents Ġa Ġsimple Ġprior Ġthat Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġis Ġfrom Ġdomain ĠIN LINE FORM 2 Ġin Ġcorpus ĠIN LINE FORM 3 Ġ. ĠWe Ġextract Ġthe Ġdomains Ġby Ġtaking Ġthe Ġbase ĠURLs Ġfrom Ġthe Ġnews Ġarticle ĠURLs . </s> An Ġimportant Ġfeature Ġwhen Ġsuggesting Ġan Ġarticle ĠIN LINE FORM 0 Ġto Ġan Ġentity ĠIN LINE FORM 1 Ġis Ġthe Ġnovelty Ġof ĠIN LINE FORM 2 Ġw . r . t Ġthe Ġalready Ġexisting Ġentity Ġprofile ĠIN LINE FORM 3 Ġ. ĠStudies ĠB IB REF 17 Ġhave Ġshown Ġthat Ġon Ġcomparable Ġcollections Ġto Ġours Ġ( T REC ĠG OV 2 ) Ġthe Ġnumber Ġof Ġdupl icates Ġcan Ġgo Ġup Ġto ĠIN LINE FORM 4 Ġ. ĠThis Ġfigure Ġis Ġlikely Ġhigher Ġfor Ġmajor Ġevents Ġconcerning Ġhighly Ġauthoritative Ġentities Ġon Ġwhich Ġall Ġnews Ġmedia Ġwill Ġreport . </s> Given Ġan Ġentity ĠIN LINE FORM 0 Ġand Ġthe Ġalready Ġadded Ġnews Ġreferences ĠIN LINE FORM 1 Ġup Ġto Ġyear ĠIN LINE FORM 2 Ġ, Ġthe Ġnovelty Ġof ĠIN LINE FORM 3 Ġat Ġyear ĠIN LINE FORM 4 Ġis Ġmeasured Ġby Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġlanguage Ġmodel Ġof ĠIN LINE FORM 5 Ġand Ġarticles Ġin ĠIN LINE FORM 6 Ġ. ĠWe Ġcombine Ġthis Ġmeasure Ġwith Ġthe Ġentity Ġoverlap Ġof ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġ. ĠThe Ġnovelty Ġvalue Ġof ĠIN LINE FORM 9 Ġis Ġgiven Ġby Ġthe Ġminimal Ġdivergence Ġvalue . ĠLow Ġscores Ġindicate Ġlow Ġnovelty Ġfor Ġthe Ġentity Ġprofile ĠIN LINE FORM 10 Ġ. </s> N ( n | e ) Ġ= Ġn ' N t - 1 { DK L (( n ') Ġ|| Ġ( n )) Ġ+ ĠDK L (( N ) Ġ|| Ġ( n )). </s> DK L (( n ') Ġ|| Ġ( n )). Ġ( 1 -) Ġj acc ard (( n '), ( n )) } Ġwhere ĠIN LINE FORM 0 Ġis Ġthe ĠKL Ġdivergence Ġof Ġthe Ġlanguage Ġmodels Ġ( ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ), Ġwhereas ĠIN LINE FORM 3 Ġis Ġthe Ġmixing Ġweight Ġ( ĠIN LINE FORM 4 Ġ) Ġbetween Ġthe Ġlanguage Ġmodels ĠIN LINE FORM 5 Ġand Ġthe Ġentity Ġoverlap Ġin ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. </s> Here Ġwe Ġintroduce Ġthe Ġevaluation Ġsetup Ġand Ġanalyze Ġthe Ġresults Ġfor Ġthe Ġarticle âĢĵ entity Ġ( A EP ) Ġplacement Ġtask . ĠWe Ġonly Ġreport Ġthe Ġevaluation Ġmetrics Ġfor Ġthe Ġ` relevant ' Ġnews - entity Ġpairs . ĠA Ġdetailed Ġexplanation Ġon Ġwhy Ġwe Ġfocus Ġon Ġthe Ġ` relevant ' Ġpairs Ġis Ġprovided Ġin ĠSection ĠSEC REF 16 Ġ. </s> Bas elines . ĠWe Ġconsider Ġthe Ġfollowing Ġbas elines Ġfor Ġthis Ġtask . </s> B 1 . ĠThe Ġfirst Ġbaseline Ġuses Ġonly Ġthe Ġsal ience - based Ġfeatures Ġby ĠDun iet z Ġand ĠGill ick ĠB IB REF 11 Ġ. </s> B 2 . ĠThe Ġsecond Ġbaseline Ġassigns Ġthe Ġvalue Ġrelevant Ġto Ġa Ġpair ĠIN LINE FORM 0 Ġ, Ġif Ġand Ġonly Ġif ĠIN LINE FORM 1 Ġappears Ġin Ġthe Ġtitle Ġof ĠIN LINE FORM 2 Ġ. </s> Learning ĠModels . ĠWe Ġuse ĠRandom ĠFore sts Ġ( RF ) ĠB IB REF 23 Ġ. ĠWe Ġlearn Ġthe ĠRF Ġon Ġall Ġcomputed Ġfeatures Ġin ĠTable ĠT AB REF 21 Ġ. ĠThe Ġoptimization Ġon ĠRF Ġis Ġdone Ġby Ġsplitting Ġthe Ġfeature Ġspace Ġinto Ġmultiple Ġtrees Ġthat Ġare Ġconsidered Ġas Ġensemble Ġclass ifiers . ĠConsequently , Ġfor Ġeach Ġclass ifier Ġit Ġcomp utes Ġthe Ġmargin Ġfunction Ġas Ġa Ġmeasure Ġof Ġthe Ġaverage Ġcount Ġof Ġpredicting Ġthe Ġcorrect Ġclass Ġin Ġcontrast Ġto Ġany Ġother Ġclass . ĠThe Ġhigher Ġthe Ġmargin Ġscore Ġthe Ġmore Ġrobust Ġthe Ġmodel . </s> Met rics . ĠWe Ġcompute Ġprecision ĠP , Ġrecall ĠR Ġand ĠF 1 Ġscore Ġfor Ġthe Ġrelevant Ġclass . ĠFor Ġexample , Ġprecision Ġis Ġthe Ġnumber Ġof Ġnews - entity Ġpairs Ġwe Ġcorrectly Ġlabeled Ġas Ġrelevant Ġcompared Ġto Ġour Ġground Ġtruth Ġdivided Ġby Ġthe Ġnumber Ġof Ġall Ġnews - entity Ġpairs Ġwe Ġlabeled Ġas Ġrelevant . </s> The Ġfollowing Ġresults Ġmeasure Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġin Ġthree Ġmain Ġaspects : Ġ( i ) Ġoverall Ġperformance Ġof ĠIN LINE FORM 0 Ġand Ġcomparison Ġto Ġbas elines , Ġ( ii ) Ġrobust ness Ġacross Ġthe Ġyears , Ġand Ġ( iii ) Ġoptimal Ġmodel Ġfor Ġthe ĠA EP Ġplacement Ġtask . </s> Performance . ĠFigure ĠFIG REF 55 Ġshows Ġthe Ġresults Ġfor Ġthe Ġyears Ġ2009 Ġand Ġ2013 , Ġwhere Ġwe Ġoptimized Ġthe Ġlearning Ġobjective Ġwith Ġinstances Ġfrom Ġyear ĠIN LINE FORM 0 Ġand Ġevaluate Ġon Ġthe Ġyears ĠIN LINE FORM 1 Ġ( see ĠSection ĠSEC REF 46 Ġ). ĠThe Ġresults Ġshow Ġthe Ġprecision âĢĵ rec all Ġcurve . ĠThe Ġred Ġcurve Ġshows Ġbaseline ĠB 1 ĠB IB REF 11 Ġ, Ġand Ġthe Ġblue Ġone Ġshows Ġthe Ġperformance Ġof ĠIN LINE FORM 2 Ġ. ĠThe Ġcurve Ġshows Ġfor Ġvarying Ġconfidence Ġscores Ġ( high Ġto Ġlow ) Ġthe Ġprecision Ġon Ġlabeling Ġthe Ġpair ĠIN LINE FORM 3 Ġas Ġ` relevant '. ĠIn Ġaddition , Ġat Ġeach Ġconfidence Ġscore Ġwe Ġcan Ġcompute Ġthe Ġcorresponding Ġrecall Ġfor Ġthe Ġ` relevant ' Ġlabel . ĠFor Ġhigh Ġconfidence Ġscores Ġon Ġlabeling Ġthe Ġnews - entity Ġpairs , Ġthe Ġbaseline ĠB 1 Ġachieves Ġon Ġaverage Ġa Ġprecision Ġscore Ġof ĠP = 0 . 50 , Ġwhile ĠIN LINE FORM 4 Ġhas ĠP = 0 . 93 . ĠWe Ġnote Ġthat Ġwith Ġthe Ġdrop Ġin Ġthe Ġconfidence Ġscore Ġthe Ġcorresponding Ġprecision Ġand Ġrecall Ġvalues Ġdrop Ġtoo , Ġand Ġthe Ġoverall ĠF 1 Ġscore Ġfor ĠB 1 Ġis Ġaround ĠF 1 = 0 . 2 , Ġin Ġcontrast Ġwe Ġachieve Ġan Ġaverage Ġscore Ġof ĠF 1 = 0 . 67 . </s> It Ġis Ġevident Ġfrom ĠFigure ĠFIG REF 55 Ġthat Ġfor Ġthe Ġyears Ġ2009 Ġand Ġ2013 , ĠIN LINE FORM 0 Ġsignificantly Ġoutper forms Ġthe Ġbaseline ĠB 1 . ĠWe Ġmeasure Ġthe Ġsignificance Ġthrough Ġthe Ġt - test Ġstatistic Ġand Ġget Ġa Ġp - value Ġof ĠIN LINE FORM 1 Ġ. ĠThe Ġimprovement Ġwe Ġachieve Ġover ĠB 1 Ġin Ġabsolute Ġnumbers , ĠIN LINE FORM 2 ĠP = + 0 . 5 Ġin Ġterms Ġof Ġprecision Ġfor Ġthe Ġyears Ġbetween Ġ2009 Ġand Ġ2014 , Ġand Ġa Ġsimilar Ġimprovement Ġin Ġterms Ġof ĠF 1 Ġscore . ĠThe Ġimprovement Ġfor Ġrecall Ġis ĠIN LINE FORM 3 ĠR = + 0 . 4 . ĠThe Ġrelative Ġimprovement Ġover ĠB 1 Ġfor ĠP Ġand ĠF 1 Ġis Ġalmost Ġ1 . 8 Ġtimes Ġbetter , Ġwhile Ġfor Ġrecall Ġwe Ġare Ġ3 . 5 Ġtimes Ġbetter . ĠIn ĠTable ĠT AB REF 58 Ġwe Ġshow Ġthe Ġoverall Ġscores Ġfor Ġthe Ġevaluation Ġmetrics Ġfor ĠB 1 Ġand ĠIN LINE FORM 4 Ġ. ĠFinally , Ġfor ĠB 2 Ġwe Ġachieve Ġmuch Ġpoorer Ġperformance , Ġwith Ġaverage Ġscores Ġof ĠP = 0 . 21 , ĠR = 0 . 20 Ġand ĠF 1 = 0 . 21 . </s> Rob ust ness . ĠIn ĠTable ĠT AB REF 58 Ġ, Ġwe Ġshow Ġthe Ġoverall Ġperformance Ġfor Ġthe Ġyears Ġbetween Ġ2009 Ġand Ġ2013 . ĠAn Ġinteresting Ġobservation Ġwe Ġmake Ġis Ġthat Ġwe Ġhave Ġa Ġvery Ġrobust Ġperformance Ġand Ġthe Ġresults Ġare Ġstable Ġacross Ġthe Ġyears . ĠIf Ġwe Ġconsider Ġthe Ġexperimental Ġsetup , Ġwhere Ġfor Ġyear ĠIN LINE FORM 0 Ġwe Ġoptimize Ġthe Ġlearning Ġobjective Ġwith Ġonly Ġ74 k Ġtraining Ġinstances Ġand Ġevaluate Ġon Ġthe Ġrest Ġof Ġthe Ġinstances , Ġit Ġachieves Ġa Ġvery Ġgood Ġperformance . ĠWe Ġpredict Ġwith ĠF 1 = 0 . 68 Ġthe Ġremaining Ġ4 69 k Ġinstances Ġfor Ġthe Ġyears ĠIN LINE FORM 1 Ġ. </s> The Ġresults Ġare Ġparticularly Ġpromising Ġconsidering Ġthe Ġfact Ġthat Ġthe Ġdistribution Ġbetween Ġour Ġtwo Ġclasses Ġis Ġhighly Ġskewed . ĠOn Ġaverage Ġthe Ġnumber Ġof Ġ` relevant ' Ġpairs Ġaccount Ġfor Ġonly Ġaround ĠIN LINE FORM 0 Ġof Ġall Ġpairs . ĠA Ġgood Ġindicator Ġto Ġsupport Ġsuch Ġa Ġstatement Ġis Ġthe Ġk appa Ġ( den oted Ġby ĠIN LINE FORM 1 Ġ) Ġstatistic . ĠIN LINE FORM 2 Ġmeasures Ġagreement Ġbetween Ġthe Ġalgorithm Ġand Ġthe Ġgold Ġstandard Ġon Ġboth Ġlabels Ġwhile Ġcorrecting Ġfor Ġchance Ġagreement Ġ( often Ġexpected Ġdue Ġto Ġextreme Ġdistributions ). ĠThe ĠIN LINE FORM 3 Ġscores Ġfor ĠB 1 Ġacross Ġthe Ġyears Ġis Ġon Ġaverage ĠIN LINE FORM 4 Ġ, Ġwhile Ġfor ĠIN LINE FORM 5 Ġwe Ġachieve Ġa Ġscore Ġof ĠIN LINE FORM 6 Ġ( the Ġmaximum Ġscore Ġfor ĠIN LINE FORM 7 Ġis Ġ1 ). </s> In ĠFigure ĠFIG REF 60 Ġwe Ġshow Ġthe Ġimpact Ġof Ġthe Ġindividual Ġfeature Ġgroups Ġthat Ġcontribute Ġto Ġthe Ġsuperior Ġperformance Ġin Ġcomparison Ġto Ġthe Ġbas elines . ĠRelative Ġentity Ġfrequency Ġfrom Ġthe Ġsal ience Ġfeature , Ġmodels Ġthe Ġentity Ġsal ience Ġas Ġan Ġexponentially Ġdecaying Ġfunction Ġbased Ġon Ġthe Ġpositional Ġindex Ġof Ġthe Ġparagraph Ġwhere Ġthe Ġentity Ġappears . ĠThe Ġperformance Ġof ĠIN LINE FORM 0 Ġwith Ġrelative Ġentity Ġfrequency Ġfrom Ġthe Ġsal ience Ġfeature Ġgroup Ġis Ġclose Ġto Ġthat Ġof Ġall Ġthe Ġfeatures Ġcombined . ĠThe Ġauthority Ġand Ġnovelty Ġfeatures Ġaccount Ġto Ġa Ġfurther Ġimprovement Ġin Ġterms Ġof Ġprecision , Ġby Ġadding Ġroughly Ġa Ġ7 %- 10 % Ġincrease . ĠHowever , Ġif Ġboth Ġfeature Ġgroups Ġare Ġconsidered Ġseparately , Ġthey Ġsignificantly Ġoutper form Ġthe Ġbaseline ĠB 1 . </s> Article âĢĵ Section ĠPl acement </s> We Ġmodel Ġthe ĠASP Ġplacement Ġtask Ġas Ġa Ġsuccessor Ġof Ġthe ĠA EP Ġtask . ĠFor Ġall Ġthe Ġ` relevant ' Ġnews Ġentity Ġpairs , Ġthe Ġtask Ġis Ġto Ġdetermine Ġthe Ġcorrect Ġentity Ġsection . ĠEach Ġsection Ġin Ġa ĠWikipedia Ġentity Ġpage Ġrepresents Ġa Ġdifferent Ġtopic . ĠFor Ġexample , ĠBarack ĠObama Ġhas Ġthe Ġsections Ġ` Early ĠLife ', Ġ` Pres idency ', Ġ` Family Ġand ĠPersonal ĠLife ' Ġetc . ĠHowever , Ġmany Ġentity Ġpages Ġhave Ġan Ġincomplete Ġsection Ġstructure . ĠIn complete Ġor Ġmissing Ġsections Ġare Ġdue Ġto Ġtwo ĠWikipedia Ġproperties . ĠFirst , Ġlong - tail Ġentities Ġmiss Ġinformation Ġand Ġsections Ġdue Ġto Ġtheir Ġlack Ġof Ġpopularity . ĠSecond , Ġfor Ġall Ġentities Ġwhether Ġpopular Ġor Ġnot , Ġcertain Ġsections Ġmight Ġoccur Ġfor Ġthe Ġfirst Ġtime Ġdue Ġto Ġreal Ġworld Ġdevelopments . ĠAs Ġan Ġexample , Ġthe Ġentity ĠGerman wings Ġdid Ġnot Ġhave Ġan Ġ` Acc idents ' Ġsection Ġbefore Ġthis Ġyear 's Ġdisaster , Ġwhich Ġwas Ġthe Ġfirst Ġin Ġthe Ġhistory Ġof Ġthe Ġairline . </s> Even Ġif Ġsections Ġare Ġmissing Ġfor Ġcertain Ġentities , Ġsimilar Ġsections Ġusually Ġoccur Ġin Ġother Ġentities Ġof Ġthe Ġsame Ġclass Ġ( e . g . Ġother Ġairlines Ġhad Ġdisasters Ġand Ġtherefore Ġtheir Ġpages Ġhave Ġan Ġaccidents Ġsection ). ĠWe Ġexploit Ġsuch Ġhom ogeneity Ġof Ġsection Ġstructure Ġand Ġconstruct Ġtemplates Ġthat Ġwe Ġuse Ġto Ġexpand Ġentity Ġprofiles . ĠThe Ġlearning Ġobjective Ġfor ĠIN LINE FORM 0 Ġtakes Ġinto Ġaccount Ġthe Ġfollowing Ġproperties : </s> Section - tem plates : Ġaccount Ġfor Ġincomplete Ġsection Ġstructure Ġfor Ġan Ġentity Ġprofile ĠIN LINE FORM 0 Ġby Ġconstructing Ġsection Ġtemplates ĠIN LINE FORM 1 Ġfrom Ġan Ġentity Ġclass ĠIN LINE FORM 2 </s> Overall Ġfit : Ġmeasures Ġthe Ġoverall Ġfit Ġof Ġa Ġnews Ġarticle Ġto Ġsections Ġin Ġthe Ġsection Ġtemplates ĠIN LINE FORM 0 </s> Given Ġthe Ġfact Ġthat Ġentity Ġprofiles Ġare Ġoften Ġincomplete , Ġwe Ġconstruct Ġsection Ġtemplates Ġfor Ġevery Ġentity Ġclass . ĠWe Ġgroup Ġentities Ġbased Ġon Ġtheir Ġclass ĠIN LINE FORM 0 Ġand Ġconstruct Ġsection Ġtemplates ĠIN LINE FORM 1 Ġ. ĠFor Ġdifferent Ġentity Ġclasses , Ġe . g . ĠPerson Ġand ĠLocation , Ġthe Ġsection Ġstructure Ġand Ġthe Ġinformation Ġrepresented Ġin Ġthose Ġsection Ġvaries Ġheavily . ĠTherefore , Ġthe Ġsection Ġtemplates Ġare Ġwith Ġrespect Ġto Ġthe Ġindividual Ġclasses Ġin Ġour Ġexperimental Ġsetup Ġ( see ĠFigure ĠFIG REF 42 Ġ). ĠDIS PLAY FORM 0 </s> Gener ating Ġsection Ġtemplates Ġhas Ġtwo Ġmain Ġadvantages . ĠFirstly , Ġby Ġconsidering Ġclass - based Ġprofiles , Ġwe Ġcan Ġovercome Ġthe Ġproblem Ġof Ġincomplete Ġindividual Ġentity Ġprofiles Ġand Ġthereby Ġare Ġable Ġto Ġsuggest Ġnews Ġarticles Ġto Ġsections Ġthat Ġdo Ġnot Ġyet Ġexist Ġin Ġa Ġspecific Ġentity ĠIN LINE FORM 0 Ġ. ĠThe Ġsecond Ġadvantage Ġis Ġthat Ġwe Ġare Ġable Ġto Ġcanonical ize Ġthe Ġsections , Ġi . e . Ġ` Early ĠLife ' Ġand Ġ` Early ĠLife Ġand ĠChildhood ' Ġwould Ġbe Ġtreated Ġsimilarly . </s> To Ġgenerate Ġthe Ġsection Ġtemplate ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġall Ġsections Ġfrom Ġentities Ġof Ġa Ġgiven Ġtype ĠIN LINE FORM 1 Ġat Ġyear ĠIN LINE FORM 2 Ġ. ĠNext , Ġwe Ġcluster Ġthe Ġentity Ġsections , Ġbased Ġon Ġan Ġextended Ġversion Ġof Ġk âĢĵ me ans Ġclust ering ĠB IB REF 18 Ġ, Ġnamely Ġx âĢĵ me ans Ġclust ering Ġintroduced Ġin ĠPel leg Ġet Ġal . Ġwhich Ġestimates Ġthe Ġnumber Ġof Ġclusters Ġefficiently ĠB IB REF 19 Ġ. ĠAs Ġa Ġsimilarity Ġmetric Ġwe Ġuse Ġthe Ġcos ine Ġsimilarity Ġcomputed Ġbased Ġon Ġthe Ġtf âĢĵ id f Ġmodels Ġof Ġthe Ġsections . ĠUsing Ġthe Ġx âĢĵ me ans Ġalgorithm Ġwe Ġovercome Ġthe Ġrequirement Ġto Ġprovide Ġthe Ġnumber Ġof Ġclusters Ġk Ġbeforehand . Ġx âĢĵ me ans Ġextends Ġthe Ġk âĢĵ me ans Ġalgorithm , Ġsuch Ġthat Ġa Ġuser Ġonly Ġspecifies Ġa Ġrange Ġ[ ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ] Ġthat Ġthe Ġnumber Ġof Ġclusters Ġmay Ġreasonably Ġlie Ġin . </s> The Ġlearning Ġobjective Ġof ĠIN LINE FORM 0 Ġis Ġto Ġdetermine Ġthe Ġoverall Ġfit Ġof Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġto Ġone Ġof Ġthe Ġsections Ġin Ġa Ġgiven Ġsection Ġtemplate ĠIN LINE FORM 2 Ġ. ĠThe Ġtemplate Ġis Ġpre - d etermined Ġby Ġthe Ġclass Ġof Ġthe Ġentity Ġfor Ġwhich Ġthe Ġnews Ġis Ġsuggested Ġas Ġrelevant Ġby ĠIN LINE FORM 3 Ġ. ĠIn Ġall Ġcases , Ġwe Ġmeasure Ġhow Ġwell ĠIN LINE FORM 4 Ġfits Ġeach Ġof Ġthe Ġsections ĠIN LINE FORM 5 Ġas Ġwell Ġas Ġthe Ġspecific Ġentity Ġsection ĠIN LINE FORM 6 Ġ. ĠThe Ġsection Ġprofiles Ġin ĠIN LINE FORM 7 Ġrepresent Ġthe Ġaggreg ated Ġentity Ġprofiles Ġfrom Ġall Ġentities Ġof Ġclass ĠIN LINE FORM 8 Ġat Ġyear ĠIN LINE FORM 9 Ġ. </s> To Ġlearn ĠIN LINE FORM 0 Ġwe Ġrely Ġon Ġa Ġvariety Ġof Ġfeatures Ġthat Ġconsider Ġseveral Ġsimilarity Ġaspects Ġas Ġshown Ġin ĠTable ĠT AB REF 31 Ġ. ĠFor Ġthe Ġsake Ġof Ġsimplicity Ġwe Ġdo Ġnot Ġmake Ġthe Ġdistinction Ġin ĠTable ĠT AB REF 31 Ġbetween Ġthe Ġindividual Ġentity Ġsection Ġand Ġclass - based Ġsection Ġsimilarities , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġrespectively . ĠBear Ġin Ġmind Ġthat Ġan Ġentity Ġsection ĠIN LINE FORM 3 Ġmight Ġbe Ġpresent Ġat Ġyear ĠIN LINE FORM 4 Ġbut Ġnot Ġat Ġyear ĠIN LINE FORM 5 Ġ( see Ġfor Ġmore Ġdetails Ġthe Ġdiscussion Ġon Ġentity Ġprofile Ġexpansion Ġin ĠSection ĠUID 69 Ġ). </s> Topic . ĠWe Ġuse Ġtopic Ġsimilarities Ġto Ġensure Ġ( i ) Ġthat Ġthe Ġcontent Ġof ĠIN LINE FORM 0 Ġfits Ġtopic - wise Ġwith Ġa Ġspecific Ġsection Ġtext Ġand Ġ( ii ) Ġthat Ġit Ġhas Ġa Ġsimilar Ġtopic Ġto Ġpreviously Ġreferred Ġnews Ġarticles Ġin Ġthat Ġsection . ĠIn Ġa Ġpre - processing Ġstage Ġwe Ġcompute Ġthe Ġtopic Ġmodels Ġfor Ġthe Ġnews Ġarticles , Ġentity Ġsections ĠIN LINE FORM 1 Ġand Ġthe Ġaggreg ated Ġclass - based Ġsections Ġin ĠIN LINE FORM 2 Ġ. ĠThe Ġtopic Ġmodels Ġare Ġcomputed Ġusing ĠL DA ĠB IB REF 20 Ġ. ĠWe Ġonly Ġcomputed Ġa Ġsingle Ġtopic Ġper Ġarticle / section Ġas Ġwe Ġare Ġonly Ġinterested Ġin Ġtopic Ġterm Ġoverl aps Ġbetween Ġarticle Ġand Ġsections . ĠWe Ġdistinguish Ġtwo Ġmain Ġfeatures : Ġthe Ġfirst Ġfeature Ġmeasures Ġthe Ġoverlap Ġof Ġtopic Ġterms Ġbetween ĠIN LINE FORM 3 Ġand Ġthe Ġentity Ġsection ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ, Ġand Ġthe Ġsecond Ġfeature Ġmeasures Ġthe Ġoverlap Ġof Ġthe Ġtopic Ġmodel Ġof ĠIN LINE FORM 6 Ġagainst Ġreferred Ġnews Ġarticles Ġin ĠIN LINE FORM 7 Ġat Ġtime ĠIN LINE FORM 8 Ġ. </s> Sy nt actic . ĠThese Ġfeatures Ġrepresent Ġa Ġmechanism Ġfor Ġconve ying Ġthe Ġimportance Ġof Ġa Ġspecific Ġtext Ġsnippet , Ġsolely Ġbased Ġon Ġthe Ġfrequency Ġof Ġspecific ĠPOS Ġtags Ġ( i . e . ĠN NP , ĠCD Ġetc .), Ġas Ġcommonly Ġused Ġin Ġtext Ġsummar ization Ġtasks . ĠFollowing Ġthe Ġsame Ġintuition Ġas Ġin ĠB IB REF 8 Ġ, Ġwe Ġweigh Ġthe Ġimportance Ġof Ġarticles Ġby Ġthe Ġcount Ġof Ġspecific ĠPOS Ġtags . ĠWe Ġexpect Ġthat Ġfor Ġdifferent Ġsections , Ġthe Ġimportance Ġof ĠPOS Ġtags Ġwill Ġvary . ĠWe Ġmeasure Ġthe Ġsimilarity Ġof ĠPOS Ġtags Ġin Ġa Ġnews Ġarticle Ġagainst Ġthe Ġsection Ġtext . ĠAdditionally , Ġwe Ġconsider Ġbi - gram Ġand Ġtri - gram ĠPOS Ġtag Ġoverlap . ĠThis Ġexploits Ġsimilarity Ġin Ġsynt act ical Ġpatterns Ġbetween Ġthe Ġnews Ġand Ġsection Ġtext . </s> Lex ical . ĠAs Ġlex ical Ġfeatures , Ġwe Ġmeasure Ġthe Ġsimilarity Ġof ĠIN LINE FORM 0 Ġagainst Ġthe Ġentity Ġsection Ġtext ĠIN LINE FORM 1 Ġand Ġthe Ġaggregate Ġsection Ġtext ĠIN LINE FORM 2 Ġ. ĠFurther , Ġwe Ġdistinguish Ġbetween Ġthe Ġoverall Ġsimilarity Ġof ĠIN LINE FORM 3 Ġand Ġthat Ġof Ġthe Ġdifferent Ġnews Ġparagraphs Ġ( ĠIN LINE FORM 4 Ġwhich Ġdenotes Ġthe Ġparagraphs Ġof ĠIN LINE FORM 5 Ġup Ġto Ġthe Ġ5 th Ġparagraph ). ĠA Ġhigher Ġsimilarity Ġon Ġthe Ġfirst Ġparagraphs Ġrepresents Ġa Ġmore Ġconfident Ġindicator Ġthat ĠIN LINE FORM 6 Ġshould Ġbe Ġsuggested Ġto Ġa Ġspecific Ġsection ĠIN LINE FORM 7 Ġ. ĠWe Ġmeasure Ġthe Ġsimilarity Ġbased Ġon Ġtwo Ġmetrics : Ġ( i ) Ġthe ĠKL - d iver gence Ġbetween Ġthe Ġcomputed Ġlanguage Ġmodels Ġand Ġ( ii ) Ġcos ine Ġsimilarity Ġof Ġthe Ġcorresponding Ġparagraph Ġtext ĠIN LINE FORM 8 Ġand Ġsection Ġtext . </s> Entity - based . ĠAnother Ġfeature Ġset Ġwe Ġconsider Ġis Ġthe Ġoverlap Ġof Ġnamed Ġentities Ġand Ġtheir Ġcorresponding Ġentity Ġclasses . ĠFor Ġdifferent Ġentity Ġsections , Ġwe Ġexpect Ġto Ġfind Ġa Ġparticular Ġset Ġof Ġentity Ġclasses Ġthat Ġwill Ġcorrelate Ġwith Ġthe Ġsection , Ġe . g . Ġ` Early ĠLife ' Ġcontains Ġmostly Ġentities Ġrelated Ġto Ġfamily , Ġschool , Ġuniversities Ġetc . </s> F requency . ĠFinally , Ġwe Ġgather Ġstatistics Ġabout Ġthe Ġnumber Ġof Ġentities , Ġparagraphs , Ġnews Ġarticle Ġlength , Ġtop âĢĵ ĠIN LINE FORM 0 Ġentities Ġand Ġentity Ġclasses , Ġand Ġthe Ġfrequency Ġof Ġdifferent ĠPOS Ġtags . ĠHere Ġwe Ġtry Ġto Ġcapture Ġpatterns Ġof Ġarticles Ġthat Ġare Ġusually Ġcited Ġin Ġspecific Ġsections . </s> E val uation ĠPlan </s> In Ġthis Ġsection Ġwe Ġoutline Ġthe Ġevaluation Ġplan Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġour Ġlearning Ġapproaches . ĠTo Ġevaluate Ġthe Ġnews Ġsuggestion Ġproblem Ġwe Ġare Ġfaced Ġwith Ġtwo Ġchallenges . </s> What Ġcomprises Ġthe Ġground Ġtruth Ġfor Ġsuch Ġa Ġtask Ġ? </s> How Ġdo Ġwe Ġconstruct Ġtraining Ġand Ġtest Ġsplits Ġgiven Ġthat Ġentity Ġpages Ġconsists Ġof Ġtext Ġadded Ġat Ġdifferent Ġpoints Ġin Ġtime Ġ? </s> Consider Ġthe Ġground Ġtruth Ġchallenge . ĠEval uating Ġif Ġan Ġarbitrary Ġnews Ġarticle Ġshould Ġbe Ġincluded Ġin ĠWikipedia Ġis Ġboth Ġsubjective Ġand Ġdifficult Ġfor Ġa Ġhuman Ġif Ġshe Ġis Ġnot Ġan Ġexpert . ĠAn Ġinvasive Ġapproach , Ġwhich Ġwas Ġproposed Ġby ĠBar z il ay Ġand ĠSau per ĠB IB REF 8 Ġ, Ġadds Ġcontent Ġdirectly Ġto ĠWikipedia Ġand Ġexpects Ġthe Ġeditors Ġor Ġother Ġusers Ġto Ġred act Ġirrelevant Ġcontent Ġover Ġa Ġperiod Ġof Ġtime . ĠThe Ġlimitations Ġof Ġsuch Ġan Ġevaluation Ġtechnique Ġis Ġthat Ġcontent Ġadded Ġto Ġlong - tail Ġentities Ġmight Ġnot Ġbe Ġevaluated Ġby Ġinformed Ġusers Ġor Ġeditors Ġin Ġthe Ġexperiment Ġtime Ġframe . ĠIt Ġis Ġhard Ġto Ġestimate Ġhow Ġmuch Ġtime Ġthe Ġadded Ġcontent Ġshould Ġbe Ġleft Ġon Ġthe Ġentity Ġpage . ĠA Ġmore Ġnon - in vasive Ġapproach Ġcould Ġinvolve Ġcrowds ourcing Ġof Ġentity Ġand Ġnews Ġarticle Ġpairs Ġin Ġan ĠIR Ġstyle Ġrelevance Ġassessment Ġsetup . ĠThe Ġproblem Ġof Ġsuch Ġan Ġapproach Ġis Ġagain Ġfinding Ġknowledgeable Ġusers Ġor Ġexperts Ġfor Ġlong - tail Ġentities . ĠThus Ġthe Ġnotion Ġof Ġrelevance Ġof Ġa Ġnews Ġrecommendation Ġis Ġchallenging Ġto Ġevaluate Ġin Ġa Ġcrowd Ġsetup . </s> We Ġtake Ġa Ġslightly Ġdifferent Ġapproach Ġby Ġmaking Ġan Ġassumption Ġthat Ġthe Ġnews Ġarticles Ġalready Ġpresent Ġin ĠWikipedia Ġentity Ġpages Ġare Ġrelevant . ĠTo Ġthis Ġextent , Ġwe Ġextract Ġa Ġdataset Ġcomprising Ġof Ġall Ġnews Ġarticles Ġreferenced Ġin Ġentity Ġpages Ġ( details Ġin ĠSection ĠSEC REF 40 Ġ). ĠAt Ġthe Ġexpense Ġof Ġnot Ġevaluating Ġthe Ġspace Ġcomprising Ġof Ġnews Ġarticles Ġabsent Ġin ĠWikipedia , Ġwe Ġsucceed Ġin Ġ( i ) Ġavoiding Ġrestrictive Ġassumptions Ġabout Ġthe Ġquality Ġof Ġhuman Ġjudgments , Ġ( ii ) Ġbeing Ġinvasive Ġand Ġpoll uting ĠWikipedia , Ġand Ġ( iii ) Ġder iving Ġa Ġreusable Ġtest Ġbed Ġfor Ġquicker Ġexperimentation . </s> The Ġsecond Ġchallenge Ġof Ġconstruction Ġof Ġtraining Ġand Ġtest Ġset Ġseparation Ġis Ġslightly Ġeasier Ġand Ġis Ġaddressed Ġin ĠSection ĠSEC REF 46 Ġ. </s> Dat as ets </s> The Ġdatasets Ġwe Ġuse Ġfor Ġour Ġexperimental Ġevaluation Ġare Ġdirectly Ġextracted Ġfrom Ġthe ĠWikipedia Ġentity Ġpages Ġand Ġtheir Ġrevision Ġhistory . ĠThe Ġgenerated Ġdata Ġrepresents Ġone Ġof Ġthe Ġcontributions Ġof Ġour Ġpaper . ĠThe Ġdatasets Ġare Ġthe Ġfollowing : </s> Entity ĠClasses . ĠWe Ġfocus Ġon Ġa Ġmanually Ġpredetermined Ġset Ġof Ġentity Ġclasses Ġfor Ġwhich Ġwe Ġexpect Ġto Ġhave Ġnews Ġcoverage . ĠThe Ġnumber Ġof Ġanalyzed Ġentity Ġclasses Ġis Ġ27 , Ġincluding ĠIN LINE FORM 0 Ġentities Ġwith Ġat Ġleast Ġone Ġnews Ġreference . ĠThe Ġentity Ġclasses Ġwere Ġselected Ġfrom Ġthe ĠDB pedia Ġclass Ġont ology . ĠFigure ĠFIG REF 42 Ġshows Ġthe Ġnumber Ġof Ġentities Ġper Ġclass Ġfor Ġthe Ġyears Ġ( 2009 - 2014 ). </s> News ĠArticles . ĠWe Ġextract Ġall Ġnews Ġreferences Ġfrom Ġthe Ġcollected ĠWikipedia Ġentity Ġpages . ĠThe Ġextracted Ġnews Ġreferences Ġare Ġassociated Ġwith Ġthe Ġsections Ġin Ġwhich Ġthey Ġappear . ĠIn Ġtotal Ġthere Ġwere ĠIN LINE FORM 0 Ġnews Ġreferences , Ġand Ġafter Ġcrawling Ġwe Ġend Ġup Ġwith ĠIN LINE FORM 1 Ġsuccessfully Ġcrawled Ġnews Ġarticles . ĠThe Ġdetails Ġof Ġthe Ġnews Ġarticle Ġdistribution , Ġand Ġthe Ġnumber Ġof Ġentities Ġand Ġsections Ġfrom Ġwhich Ġthey Ġare Ġreferred Ġare Ġshown Ġin ĠTable ĠT AB REF 44 Ġ. </s> Article - Entity ĠGround - truth . ĠThe Ġdataset Ġcomprises Ġof Ġthe Ġnews Ġand Ġentity Ġpairs ĠIN LINE FORM 0 Ġ. ĠNews - entity Ġpairs Ġare Ġrelevant Ġif Ġthe Ġnews Ġarticle Ġis Ġreferenced Ġin Ġthe Ġentity Ġpage . ĠNon - relevant Ġpairs Ġ( i . e . Ġnegative Ġtraining Ġexamples ) Ġconsist Ġof Ġnews Ġarticles Ġthat Ġcontain Ġan Ġentity Ġbut Ġare Ġnot Ġreferenced Ġin Ġthat Ġentity 's Ġpage . ĠIf Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġis Ġreferred Ġfrom ĠIN LINE FORM 2 Ġat Ġyear ĠIN LINE FORM 3 Ġ, Ġthe Ġfeatures Ġare Ġcomputed Ġtaking Ġinto Ġaccount Ġthe Ġentity Ġprofiles Ġat Ġyear ĠIN LINE FORM 4 Ġ. </s> Article - Section ĠGround - truth . ĠThe Ġdataset Ġconsists Ġof Ġthe Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ, Ġwhere Ġwe Ġassume Ġthat ĠIN LINE FORM 2 Ġhas Ġalready Ġbeen Ġdetermined Ġas Ġrelevant . ĠWe Ġtherefore Ġhave Ġa Ġmulti - class Ġclassification Ġproblem Ġwhere Ġwe Ġneed Ġto Ġdetermine Ġthe Ġsection Ġof ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġis Ġcited . ĠSimilar Ġto Ġthe Ġarticle - entity Ġground Ġtruth , Ġhere Ġtoo Ġthe Ġfeatures Ġcompute Ġthe Ġsimilarity Ġbetween ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. </s> Data ĠPre - Process ing </s> We ĠPOS - tag Ġthe Ġnews Ġarticles Ġand Ġentity Ġprofiles ĠIN LINE FORM 0 Ġwith Ġthe ĠStanford Ġtag ger ĠB IB REF 21 Ġ. ĠFor Ġentity Ġlinking Ġthe Ġnews Ġarticles , Ġwe Ġuse ĠTag Me ! ĠB IB REF 22 Ġwith Ġa Ġconfidence Ġscore Ġof Ġ0 . 3 . ĠOn Ġa Ġmanual Ġinspection Ġof Ġa Ġrandom Ġsample Ġof Ġ1000 Ġdis amb ig uated Ġentities , Ġthe Ġaccuracy Ġis Ġabove Ġ0 . 9 . ĠOn Ġaverage , Ġthe Ġnumber Ġof Ġentities Ġper Ġnews Ġarticle Ġis Ġapproximately Ġ30 . ĠFor Ġentity Ġlinking Ġthe Ġentity Ġprofiles , Ġwe Ġsimply Ġfollow Ġthe Ġanchor Ġtext Ġthat Ġrefers Ġto ĠWikipedia Ġentities . </s> Train Ġand ĠTesting ĠEvaluation ĠSetup </s> We Ġevaluate Ġthe Ġgenerated Ġsupervised Ġmodels Ġfor Ġthe Ġtwo Ġtasks , ĠA EP Ġand ĠASP , Ġby Ġsplitting Ġthe Ġtrain Ġand Ġtesting Ġinstances . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġfor Ġthe Ġpairs ĠIN LINE FORM 0 Ġand Ġthe Ġtriple ĠIN LINE FORM 1 Ġ, Ġthe Ġnews Ġarticle ĠIN LINE FORM 2 Ġis Ġreferenced Ġat Ġtime ĠIN LINE FORM 3 Ġby Ġentity ĠIN LINE FORM 4 Ġ, Ġwhile Ġthe Ġfeatures Ġtake Ġinto Ġaccount Ġthe Ġentity Ġprofile Ġat Ġtime ĠIN LINE FORM 5 Ġ. ĠThis Ġavoids Ġany Ġ` over l apping ' Ġcontent Ġbetween Ġthe Ġnews Ġarticle Ġand Ġthe Ġentity Ġpage , Ġwhich Ġcould Ġaffect Ġthe Ġlearning Ġtask Ġof Ġthe Ġfunctions ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. ĠTable ĠT AB REF 47 Ġshows Ġthe Ġstatistics Ġof Ġtrain Ġand Ġtest Ġinstances . ĠWe Ġlearn Ġthe Ġfunctions Ġat Ġyear ĠIN LINE FORM 8 Ġand Ġtest Ġon Ġinstances Ġfor Ġthe Ġyears Ġgreater Ġthan ĠIN LINE FORM 9 Ġ. ĠPlease Ġnote Ġthat Ġwe Ġdo Ġnot Ġshow Ġthe Ġperformance Ġfor Ġyear Ġ2014 Ġas Ġwe Ġdo Ġnot Ġhave Ġdata Ġfor Ġ2015 Ġfor Ġevaluation . </s> Article - Section ĠPl acement </s> Here Ġwe Ġshow Ġthe Ġevaluation Ġsetup Ġfor ĠASP Ġtask Ġand Ġdiscuss Ġthe Ġresults Ġwith Ġa Ġfocus Ġon Ġthree Ġmain Ġaspects , Ġ( i ) Ġthe Ġoverall Ġperformance Ġacross Ġthe Ġyears , Ġ( ii ) Ġthe Ġentity Ġclass Ġspecific Ġperformance , Ġand Ġ( iii ) Ġthe Ġimpact Ġon Ġentity Ġprofile Ġexpansion Ġby Ġsuggesting Ġmissing Ġsections Ġto Ġentities Ġbased Ġon Ġthe Ġpre - com puted Ġtemplates . </s> Bas elines . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġcomparable Ġapproach Ġfor Ġthis Ġtask . ĠTherefore , Ġthe Ġbas elines Ġwe Ġconsider Ġare Ġthe Ġfollowing : </s> S 1 : ĠPick Ġthe Ġsection Ġfrom Ġtemplate ĠIN LINE FORM 0 Ġwith Ġthe Ġhighest Ġlex ical Ġsimilarity Ġto ĠIN LINE FORM 1 Ġ: ĠS 1 ĠIN LINE FORM 2 </s> S 2 : ĠPlace Ġthe Ġnews Ġinto Ġthe Ġmost Ġfrequent Ġsection Ġin ĠIN LINE FORM 0 </s> Learning ĠModels . ĠWe Ġuse ĠRandom ĠFore sts Ġ( RF ) ĠB IB REF 23 Ġand ĠSupport ĠVector ĠMachines Ġ( S VM ) ĠB IB REF 24 Ġ. ĠThe Ġmodels Ġare Ġoptimized Ġtaking Ġinto Ġaccount Ġthe Ġfeatures Ġin ĠTable ĠT AB REF 31 Ġ. ĠIn Ġcontrast Ġto Ġthe ĠA EP Ġtask , Ġhere Ġthe Ġscale Ġof Ġthe Ġnumber Ġof Ġinstances Ġallows Ġus Ġto Ġlearn Ġthe ĠS VM Ġmodels . ĠThe ĠS VM Ġmodel Ġis Ġoptimized Ġusing Ġthe ĠIN LINE FORM 0 Ġloss Ġfunction Ġand Ġuses Ġthe ĠGa ussian Ġkernels . </s> Met rics . ĠWe Ġcompute Ġprecision ĠP Ġas Ġthe Ġratio Ġof Ġnews Ġfor Ġwhich Ġwe Ġpick Ġa Ġsection ĠIN LINE FORM 0 Ġfrom ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġconform s Ġto Ġthe Ġone Ġin Ġour Ġground - truth Ġ( see ĠSection ĠSEC REF 40 Ġ). ĠThe Ġdefinition Ġof Ġrecall ĠR Ġand ĠF 1 Ġscore Ġfollows Ġfrom Ġthat Ġof Ġprecision . </s> Figure ĠFIG REF 66 Ġshows Ġthe Ġoverall Ġperformance Ġand Ġa Ġcomparison Ġof Ġour Ġapproach Ġ( when ĠIN LINE FORM 0 Ġis Ġoptimized Ġusing ĠS VM ) Ġagainst Ġthe Ġbest Ġperforming Ġbaseline ĠS 2 . ĠWith Ġthe Ġincrease Ġin Ġthe Ġnumber Ġof Ġtraining Ġinstances Ġfor Ġthe ĠASP Ġtask Ġthe Ġperformance Ġis Ġa Ġmon oton ically Ġnon - dec re asing Ġfunction . ĠFor Ġthe Ġyear Ġ2009 , Ġwe Ġoptimize Ġthe Ġlearning Ġobjective Ġof ĠIN LINE FORM 1 Ġwith Ġaround Ġ8 % Ġof Ġthe Ġtotal Ġinstances , Ġand Ġevaluate Ġon Ġthe Ġrest . ĠThe Ġperformance Ġon Ġaverage Ġis Ġaround ĠP = 0 . 66 Ġacross Ġall Ġclasses . ĠEven Ġthough Ġfor Ġmany Ġclasses Ġthe Ġperformance Ġis Ġalready Ġstable Ġ( as Ġwe Ġwill Ġsee Ġin Ġthe Ġnext Ġsection ), Ġfor Ġsome Ġclasses Ġwe Ġimprove Ġfurther . ĠIf Ġwe Ġtake Ġinto Ġaccount Ġthe Ġyears Ġbetween Ġ2010 Ġand Ġ2012 , Ġwe Ġhave Ġan Ġincrease Ġof ĠIN LINE FORM 2 ĠP = 0 . 17 , Ġwith Ġaround Ġ70 % Ġof Ġinstances Ġused Ġfor Ġtraining Ġand Ġthe Ġremainder Ġfor Ġevaluation . ĠFor Ġthe Ġremaining Ġyears Ġthe Ġtotal Ġimprovement Ġis ĠIN LINE FORM 3 ĠP = 0 . 18 Ġin Ġcontrast Ġto Ġthe Ġperformance Ġat Ġyear Ġ2009 . </s> On Ġthe Ġother Ġhand , Ġthe Ġbaseline ĠS 1 Ġhas Ġan Ġaverage Ġprecision Ġof ĠP = 0 . 12 . ĠThe Ġperformance Ġacross Ġthe Ġyears Ġvaries Ġslightly , Ġwith Ġthe Ġyear Ġ2011 Ġhaving Ġthe Ġhighest Ġaverage Ġprecision Ġof ĠP = 0 . 13 . ĠAlways Ġpicking Ġthe Ġmost Ġfrequent Ġsection Ġas Ġin ĠS 2 , Ġas Ġshown Ġin ĠFigure ĠFIG REF 66 Ġ, Ġresults Ġin Ġan Ġaverage Ġprecision Ġof ĠP = 0 . 17 , Ġwith Ġa Ġuniform Ġdistribution Ġacross Ġthe Ġyears . </s> Here Ġwe Ġshow Ġthe Ġperformance Ġof ĠIN LINE FORM 0 Ġdecomp osed Ġfor Ġthe Ġdifferent Ġentity Ġclasses . ĠSpecifically Ġwe Ġanalyze Ġthe Ġ27 Ġclasses Ġin ĠFigure ĠFIG REF 42 Ġ. ĠIn ĠTable ĠT AB REF 68 Ġ, Ġwe Ġshow Ġthe Ġresults Ġfor Ġa Ġrange Ġof Ġyears Ġ( we Ġomit Ġshowing Ġall Ġyears Ġdue Ġto Ġspace Ġconstraints ). ĠFor Ġillustration Ġpurposes Ġonly , Ġwe Ġgroup Ġthem Ġinto Ġfour Ġmain Ġclasses Ġ( ĠIN LINE FORM 1 ĠPerson , ĠOrganization , ĠLocation , ĠEvent ĠIN LINE FORM 2 Ġ) Ġand Ġinto Ġthe Ġspecific Ġsub - classes Ġshown Ġin Ġthe Ġsecond Ġcolumn Ġin ĠTable ĠT AB REF 68 Ġ. ĠFor Ġinstance , Ġthe Ġentity Ġclasses ĠOffice H older Ġand ĠPolit ician Ġare Ġaggreg ated Ġinto ĠPerson âĢĵ Politics . </s> It Ġis Ġevident Ġthat Ġin Ġthe Ġfirst Ġyear Ġthe Ġperformance Ġis Ġlower Ġin Ġcontrast Ġto Ġthe Ġlater Ġyears . ĠThis Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġas Ġwe Ġproceed , Ġwe Ġcan Ġbetter Ġgeneral ize Ġand Ġaccurately Ġdetermine Ġthe Ġcorrect Ġfit Ġof Ġan Ġarticle ĠIN LINE FORM 0 Ġinto Ġone Ġof Ġthe Ġsections Ġfrom Ġthe Ġpre - com puted Ġtemplates ĠIN LINE FORM 1 Ġ. ĠThe Ġresults Ġare Ġalready Ġstable Ġfor Ġthe Ġyear Ġrange ĠIN LINE FORM 2 Ġ. ĠFor Ġa Ġfew ĠPerson Ġsub - classes , Ġe . g . ĠPolitics , ĠEntertainment , Ġwe Ġachieve Ġan ĠF 1 Ġscore Ġabove Ġ0 . 9 . ĠThese Ġadditionally Ġrepresent Ġclasses Ġwith Ġa Ġsufficient Ġnumber Ġof Ġtraining Ġinstances Ġfor Ġthe Ġyears ĠIN LINE FORM 3 Ġ. ĠThe Ġlowest ĠF 1 Ġscore Ġis Ġfor Ġthe ĠCriminal Ġand ĠTelevision Ġclasses . ĠHowever , Ġthis Ġis Ġdirectly Ġcorrelated Ġwith Ġthe Ġinsufficient Ġnumber Ġof Ġinstances . </s> The Ġbaseline Ġapproaches Ġfor Ġthe ĠASP Ġtask Ġperform Ġpoorly . ĠS 1 , Ġbased Ġon Ġlex ical Ġsimilarity , Ġhas Ġa Ġvarying Ġperformance Ġfor Ġdifferent Ġentity Ġclasses . ĠThe Ġbest Ġperformance Ġis Ġachieved Ġfor Ġthe Ġclass ĠPerson ĠâĢĵ ĠPolitics , Ġwith ĠP = 0 . 43 . ĠThis Ġhighlights Ġthe Ġimportance Ġof Ġour Ġfeature Ġchoice Ġand Ġthat Ġthe ĠASP Ġcannot Ġbe Ġconsidered Ġas Ġa Ġlinear Ġfunction , Ġwhere Ġthe Ġmaximum Ġsimilarity Ġyields Ġthe Ġbest Ġresults . ĠFor Ġdifferent Ġentity Ġclasses Ġdifferent Ġfeatures Ġand Ġcombination Ġof Ġfeatures Ġis Ġnecessary . ĠConsidering Ġthat ĠS 2 Ġis Ġthe Ġoverall Ġbest Ġperforming Ġbaseline , Ġthrough Ġour Ġapproach ĠIN LINE FORM 0 Ġwe Ġhave Ġa Ġsignificant Ġimprovement Ġof Ġover ĠIN LINE FORM 1 ĠP = + 0 . 64 . </s> The Ġmodels Ġwe Ġlearn Ġare Ġvery Ġrobust Ġand Ġobtain Ġhigh Ġaccuracy , Ġfulfilling Ġour Ġpre - condition Ġfor Ġaccurate Ġnews Ġsuggestions Ġinto Ġthe Ġentity Ġsections . ĠWe Ġmeasure Ġthe Ġrobust ness Ġof ĠIN LINE FORM 0 Ġthrough Ġthe ĠIN LINE FORM 1 Ġstatistic . ĠIn Ġthis Ġcase , Ġwe Ġhave Ġa Ġmodel Ġwith Ġroughly Ġ10 Ġlabels Ġ( cor respond ing Ġto Ġthe Ġnumber Ġof Ġsections Ġin Ġa Ġtemplate ĠIN LINE FORM 2 Ġ). ĠThe Ġscore Ġwe Ġachieve Ġshows Ġthat Ġour Ġmodel Ġpredicts Ġwith Ġhigh Ġconfidence Ġwith ĠIN LINE FORM 3 Ġ. </s> The Ġlast Ġanalysis Ġis Ġthe Ġimpact Ġwe Ġhave Ġon Ġexpanding Ġentity Ġprofiles ĠIN LINE FORM 0 Ġwith Ġnew Ġsections . ĠFigure ĠFIG REF 70 Ġshows Ġthe Ġratio Ġof Ġsections Ġfor Ġwhich Ġwe Ġcorrectly Ġsuggest Ġan Ġarticle ĠIN LINE FORM 1 Ġto Ġthe Ġright Ġsection Ġin Ġthe Ġsection Ġtemplate ĠIN LINE FORM 2 Ġ. ĠThe Ġratio Ġhere Ġcorresponds Ġto Ġsections Ġthat Ġare Ġnot Ġpresent Ġin Ġthe Ġentity Ġprofile Ġat Ġyear ĠIN LINE FORM 3 Ġ, Ġthat Ġis ĠIN LINE FORM 4 Ġ. ĠHowever , Ġgiven Ġthe Ġgenerated Ġtemplates ĠIN LINE FORM 5 Ġ, Ġwe Ġcan Ġexpand Ġthe Ġentity Ġprofile ĠIN LINE FORM 6 Ġwith Ġa Ġnew Ġsection Ġat Ġtime ĠIN LINE FORM 7 Ġ. ĠIn Ġdetails , Ġin Ġthe Ġabsence Ġof Ġa Ġsection Ġat Ġtime ĠIN LINE FORM 8 Ġ, Ġour Ġmodel Ġtrains Ġwell Ġon Ġsimilar Ġsections Ġfrom Ġthe Ġsection Ġtemplate ĠIN LINE FORM 9 Ġ, Ġhence Ġwe Ġcan Ġpredict Ġaccurately Ġthe Ġsection Ġand Ġin Ġthis Ġcase Ġsuggest Ġits Ġaddition Ġto Ġthe Ġentity Ġprofile . ĠWith Ġtime , Ġit Ġis Ġobvious Ġthat Ġthe Ġexpansion Ġrate Ġdecreases Ġat Ġlater Ġyears Ġas Ġthe Ġentity Ġprofiles Ġbecome Ġmore Ġ` complete '. </s> This Ġis Ġparticularly Ġinteresting Ġfor Ġexpanding Ġthe Ġentity Ġprofiles Ġof Ġlong - tail Ġentities Ġas Ġwell Ġas Ġupdating Ġentities Ġwith Ġreal - world Ġemerging Ġevents Ġthat Ġare Ġadded Ġconstantly . ĠIn Ġmany Ġcases Ġsuch Ġmissing Ġsections Ġare Ġpresent Ġat Ġone Ġof Ġthe Ġentities Ġof Ġthe Ġrespective Ġentity Ġclass ĠIN LINE FORM 0 Ġ. ĠAn Ġobvious Ġcase Ġis Ġthe Ġexample Ġtaken Ġin ĠSection ĠSEC REF 16 Ġ, Ġwhere Ġthe Ġ` Acc idents ' Ġis Ġrather Ġcommon Ġfor Ġentities Ġof Ġtype ĠAir line . ĠHowever , Ġit Ġis Ġnon - existent Ġfor Ġsome Ġspecific Ġentity Ġinstances , Ġi . e ĠGerman wings Ġairline . </s> Through Ġour ĠASP Ġapproach ĠIN LINE FORM 0 Ġ, Ġwe Ġare Ġable Ġto Ġexpand Ġboth Ġlong - tail Ġand Ġtrunk Ġentities . ĠWe Ġdistinguish Ġbetween Ġthe Ġtwo Ġtypes Ġof Ġentities Ġby Ġsimply Ġmeasuring Ġtheir Ġsection Ġtext Ġlength . ĠThe Ġreal Ġdistribution Ġin Ġthe Ġground Ġtruth Ġ( see ĠSection ĠSEC REF 40 Ġ) Ġis Ġ27 % Ġand Ġ73 % Ġare Ġlong - tail Ġand Ġtrunk Ġentities , Ġrespectively . ĠWe Ġare Ġable Ġto Ġexpand Ġthe Ġentity Ġprofiles Ġfor Ġboth Ġcases Ġand Ġall Ġentity Ġclasses Ġwithout Ġa Ġsignificant Ġdifference , Ġwith Ġthe Ġonly Ġexception Ġbeing Ġthe Ġclass ĠCreative ĠWork , Ġwhere Ġwe Ġexpand Ġsignificantly Ġmore Ġtrunk Ġentities . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġhave Ġproposed Ġan Ġautomated Ġapproach Ġfor Ġthe Ġnovel Ġtask Ġof Ġsuggesting Ġnews Ġarticles Ġto ĠWikipedia Ġentity Ġpages Ġto Ġfacilitate ĠWikipedia Ġupdating . ĠThe Ġprocess Ġconsists Ġof Ġtwo Ġstages . ĠIn Ġthe Ġfirst Ġstage , Ġarticle âĢĵ entity Ġplacement , Ġwe Ġsuggest Ġnews Ġarticles Ġto Ġentity Ġpages Ġby Ġconsidering Ġthree Ġmain Ġfactors , Ġsuch Ġas Ġentity Ġsal ience Ġin Ġa Ġnews Ġarticle , Ġrelative Ġauthority Ġand Ġnovelty Ġof Ġnews Ġarticles Ġfor Ġan Ġentity Ġpage . ĠIn Ġthe Ġsecond Ġstage , Ġarticle âĢĵ section Ġplacement , Ġwe Ġdetermine Ġthe Ġbest Ġfitting Ġsection Ġin Ġan Ġentity Ġpage . ĠHere , Ġwe Ġremedy Ġthe Ġproblem Ġof Ġincomplete Ġentity Ġsection Ġprofiles Ġby Ġconstructing Ġsection Ġtemplates Ġfor Ġspecific Ġentity Ġclasses . ĠThis Ġallows Ġus Ġto Ġadd Ġmissing Ġsections Ġto Ġentity Ġpages . ĠWe Ġcarry Ġout Ġan Ġextensive Ġexperimental Ġevaluation Ġon Ġ351 , 98 3 Ġnews Ġarticles Ġand Ġ73 , 7 34 Ġentities Ġcoming Ġfrom Ġ27 Ġdistinct Ġentity Ġclasses . ĠFor Ġthe Ġfirst Ġstage , Ġwe Ġachieve Ġan Ġoverall Ġperformance Ġwith ĠP = 0 . 93 , ĠR = 0 . 514 Ġand ĠF 1 = 0 . 676 , Ġoutper forming Ġour Ġbaseline Ġcompetitors Ġsignificantly . ĠFor Ġthe Ġsecond Ġstage , Ġwe Ġshow Ġthat Ġwe Ġcan Ġlearn Ġincrement ally Ġto Ġdetermine Ġthe Ġcorrect Ġsection Ġfor Ġa Ġnews Ġarticle Ġbased Ġon Ġsection Ġtemplates . ĠThe Ġoverall Ġperformance Ġacross Ġdifferent Ġclasses Ġis ĠP = 0 . 8 44 , ĠR = 0 . 885 Ġand ĠF 1 = 0 . 860 . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġenhance Ġour Ġwork Ġby Ġextracting Ġfacts Ġfrom Ġthe Ġsuggested Ġnews Ġarticles . ĠResults Ġsuggest Ġthat Ġthe Ġnews Ġcontent Ġcited Ġin Ġentity Ġpages Ġcomes Ġfrom Ġthe Ġfirst Ġparagraphs . ĠHowever , Ġchallenging Ġtask Ġsuch Ġas Ġthe Ġcanonical ization Ġand Ġchronological Ġordering Ġof Ġfacts , Ġstill Ġremain .
Introduction Wikipedia is the largest source of open and collaboratively curated knowledge in the world. Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone. In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative. As a result, it has become the preferred source of information consumption about entities and events. Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6 . However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay. Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively. While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf. Figure FIGREF2 ) is not mentioned in the page for Odisha. Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets. This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources. In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7 . To remedy these problems, it is important to identify information sources that contain novel and salient facts to a given entity page. However, not all information sources are equal. The online presence of major news outlets is an authoritative source due to active editorial control and their articles are also a timely container of facts. In addition, their use is in line with current Wikipedia editing practice, as is shown in BIBREF7 that almost 20% of current citations in all entity pages are news articles. We therefore propose news suggestion as a novel task that enhances entity pages and reduces delay while keeping its pages authoritative. Existing efforts to populate Wikipedia BIBREF8 start from an entity page and then generate candidate documents about this entity using an external search engine (and then post-process them). However, such an approach lacks in (a) reproducibility since rankings vary with time with obvious bias to recent news (b) maintainability since document acquisition for each entity has to be periodically performed. To this effect, our news suggestion considers a news article as input, and determines if it is valuable for Wikipedia. Specifically, given an input news article INLINEFORM0 and a state of Wikipedia, the news suggestion problem identifies the entities mentioned in INLINEFORM1 whose entity pages can improve upon suggesting INLINEFORM2 . Most of the works on knowledge base acceleration BIBREF9 , BIBREF10 , BIBREF11 , or Wikipedia page generation BIBREF8 rely on high quality input sources which are then utilized to extract textual facts for Wikipedia page population. In this work, we do not suggest snippets or paraphrases but rather entire articles which have a high potential importance for entity pages. These suggested news articles could be consequently used for extraction, summarization or population either manually or automatically – all of which rely on high quality and relevant input sources. We identify four properties of good news recommendations: salience, relative authority, novelty and placement. First, we need to identify the most salient entities in a news article. This is done to avoid pollution of entity pages with only marginally related news. Second, we need to determine whether the news is important to the entity as only the most relevant news should be added to a precise reference work. To do this, we compute the relative authority of all entities in the news article: we call an entity more authoritative than another if it is more popular or noteworthy in the real world. Entities with very high authority have many news items associated with them and only the most relevant of these should be included in Wikipedia whereas for entities of lower authority the threshold for inclusion of a news article will be lower. Third, a good recommendation should be able to identify novel news by minimizing redundancy coming from multiple news articles. Finally, addition of facts is facilitated if the recommendations are fine-grained, i.e., recommendations are made on the section level rather than the page level (placement). Approach and Contributions. We propose a two-stage news suggestion approach to entity pages. In the first stage, we determine whether a news article should be suggested for an entity, based on the entity's salience in the news article, its relative authority and the novelty of the article to the entity page. The second stage takes into account the class of the entity for which the news is suggested and constructs section templates from entities of the same class. The generation of such templates has the advantage of suggesting and expanding entity pages that do not have a complete section structure in Wikipedia, explicitly addressing long-tail and trunk entities. Afterwards, based on the constructed template our method determines the best fit for the news article with one of the sections. We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages. Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years. The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach. In summary, we make the following contributions. Related Work As we suggest a new problem there is no current work addressing exactly the same task. However, our task has similarities to Wikipedia page generation and knowledge base acceleration. In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection. Wikipedia Page Generation is the problem of populating Wikipedia pages with content coming from external sources. Sauper and Barzilay BIBREF8 propose an approach for automatically generating whole entity pages for specific entity classes. The approach is trained on already-populated entity pages of a given class (e.g. `Diseases') by learning templates about the entity page structure (e.g. diseases have a treatment section). For a new entity page, first, they extract documents via Web search using the entity title and the section title as a query, for example `Lung Cancer'+`Treatment'. As already discussed in the introduction, this has problems with reproducibility and maintainability. However, their main focus is on identifying the best paragraphs extracted from the collected documents. They rank the paragraphs via an optimized supervised perceptron model for finding the most representative paragraph that is the least similar to paragraphs in other sections. This paragraph is then included in the newly generated entity page. Taneva and Weikum BIBREF12 propose an approach that constructs short summaries for the long tail. The summaries are called `gems' and the size of a `gem' can be user defined. They focus on generating summaries that are novel and diverse. However, they do not consider any structure of entities, which is present in Wikipedia. In contrast to BIBREF8 and BIBREF12 , we actually focus on suggesting entire documents to Wikipedia entity pages. These are authoritative documents (news), which are highly relevant for the entity, novel for the entity and in which the entity is salient. Whereas relevance in Sauper and Barzilay is implicitly computed by web page ranking we solve that problem by looking at relative authority and salience of an entity, using the news article and entity page only. As Sauper and Barzilay concentrate on empty entity pages, the problem of novelty of their content is not an issue in their work whereas it is in our case which focuses more on updating entities. Updating entities will be more and more important the bigger an existing reference work is. Both the approaches in BIBREF8 and BIBREF12 (finding paragraphs and summarization) could then be used to process the documents we suggest further. Our concentration on news is also novel. Knowledge Base Acceleration. In this task, given specific information extraction templates, a given corpus is analyzed in order to find worthwhile mentions of an entity or snippets that match the templates. Balog BIBREF9 , BIBREF10 recommend news citations for an entity. Prior to that, the news articles are classified for their appropriateness for an entity, where as features for the classification task they use entity, document, entity-document and temporal features. The best performing features are those that measure similarity between an entity and the news document. West et al. BIBREF13 consider the problem of knowledge base completion, through question answering and complete missing facts in Freebase based on templates, i.e. Frank_Zappa bornIn Baltymore, Maryland. In contrast, we do not extract facts for pre-defined templates but rather suggest news articles based on their relevance to an entity. In cases of long-tail entities, we can suggest to add a novel section through our abstraction and generation of section templates at entity class level. Entity Salience. Determining which entities are prominent or salient in a given text has a long history in NLP, sparked by the linguistic theory of Centering BIBREF14 . Salience has been used in pronoun and co-reference resolution BIBREF15 , or to predict which entities will be included in an abstract of an article BIBREF11 . Frequent features to measure salience include the frequency of an entity in a document, positioning of an entity, grammatical function or internal entity structure (POS tags, head nouns etc.). These approaches are not currently aimed at knowledge base generation or Wikipedia coverage extension but we postulate that an entity's salience in a news article is a prerequisite to the news article being relevant enough to be included in an entity page. We therefore use the salience features in BIBREF11 as part of our model. However, these features are document-internal — we will show that they are not sufficient to predict news inclusion into an entity page and add features of entity authority, news authority and novelty that measure the relations between several entities, between entity and news article as well as between several competing news articles. Terminology and Problem Definition We are interested in named entities mentioned in documents. An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms. We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking. We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2 . For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e.g. Odisha is canonicalized to the corresponding article). For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4 . Information in an entity page is organized into sections and evolves with time as more content is added. We refer to the state of Wikipedia at a time INLINEFORM0 as INLINEFORM1 and the set of sections for an entity page INLINEFORM2 as its entity profile INLINEFORM3 . Unlike news articles, text in Wikipedia could be explicitly linked to entity pages through anchors. The set of entities explicitly referred in text from section INLINEFORM4 is defined as INLINEFORM5 . Furthermore, Wikipedia induces a category structure over its entities, which is exploited by knowledge bases like YAGO (e.g. Barack_Obama isA Person). Consequently, each entity page belongs to one or more entity categories or classes INLINEFORM6 . Now we can define our news suggestion problem below: Definition 1 (News Suggestion Problem) Given a set of news articles INLINEFORM0 and set of Wikipedia entity pages INLINEFORM1 (from INLINEFORM2 ) we intend to suggest a news article INLINEFORM3 published at time INLINEFORM4 to entity page INLINEFORM5 and additionally to the most relevant section for the entity page INLINEFORM6 . Approach Overview We approach the news suggestion problem by decomposing it into two tasks: AEP: Article–Entity placement ASP: Article–Section placement In this first step, for a given entity-news pair INLINEFORM0 , we determine whether the given news article INLINEFORM1 should be suggested (we will refer to this as `relevant') to entity INLINEFORM2 . To generate such INLINEFORM3 pairs, we perform the entity linking process, INLINEFORM4 , for INLINEFORM5 . The article–entity placement task (described in detail in Section SECREF16 ) for a pair INLINEFORM0 outputs a binary label (either `non-relevant' or `relevant') and is formalized in Equation EQREF14 . DISPLAYFORM0 In the second step, we take into account all `relevant' pairs INLINEFORM0 and find the correct section for article INLINEFORM1 in entity INLINEFORM2 , respectively its profile INLINEFORM3 (see Section SECREF30 ). The article–section placement task, determines the correct section for the triple INLINEFORM4 , and is formalized in Equation EQREF15 . DISPLAYFORM0 In the subsequent sections we describe in details how we approach the two tasks for suggesting news articles to entity pages. News Article Suggestion In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 ). The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement. For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2 . We will denote our approach for AEP by INLINEFORM3 . Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4 . In the following, we describe the process of learning the functions INLINEFORM0 and INLINEFORM1 . We introduce features for the learning process, which encode information regarding the entity salience, relative authority and novelty in the case of AEP task. For the ASP task, we measure the overall fit of an article to the entity sections, with the entity being an input from AEP task. Additionally, considering that the entity profiles INLINEFORM2 are incomplete, in the case of a missing section we suggest and expand the entity profiles based on section templates generated from entities of the same class INLINEFORM3 (see Section UID34 ). Article–Entity Placement In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant'). Note that we are mainly interested in finding the relevant pairs in this task. For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average). Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4 . Finding the relevant pairs is therefore a considerable challenge. An article INLINEFORM0 is suggested to INLINEFORM1 by our function INLINEFORM2 if it fulfills the following properties. The entity INLINEFORM3 is salient in INLINEFORM4 (a central concept), therefore ensuring that INLINEFORM5 is about INLINEFORM6 and that INLINEFORM7 is important for INLINEFORM8 . Next, given the fact there might be many articles in which INLINEFORM9 is salient, we also look at the reverse property, namely whether INLINEFORM10 is important for INLINEFORM11 . We do this by comparing the authority of INLINEFORM12 (which is a measure of popularity of an entity, such as its frequency of mention in a whole corpus) with the authority of its co-occurring entities in INLINEFORM13 , leading to a feature we call relative authority. The intuition is that for an entity that has overall lower authority than its co-occurring entities, a news article is more easily of importance. Finally, if the article we are about to suggest is already covered in the entity profile INLINEFORM14 , we do not wish to suggest redundant information, hence the novelty. Therefore, the learning objective of INLINEFORM15 should fulfill the following properties. Table TABREF21 shows a summary of the computed features for INLINEFORM16 . Salience: entity INLINEFORM0 should be a salient entity in news article INLINEFORM1 Relative Authority: the set of entities INLINEFORM0 with which INLINEFORM1 co-occurs should have higher authority than INLINEFORM2 , making INLINEFORM3 important for INLINEFORM4 Novelty: news article INLINEFORM0 should provide novel information for entity INLINEFORM1 taking into account its profile INLINEFORM2 Baseline Features. As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community. We reimplemented the ones in Dunietz and Gillick BIBREF11 . This includes a variety of features, e.g. positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in. Table 2 in BIBREF11 gives details. Relative Entity Frequency. Although frequency of mention and positional features play some role in baseline features, their interaction is not modeled by a single feature nor do the positional features encode more than sentence position. We therefore suggest a novel feature called relative entity frequency, INLINEFORM0 , that has three properties.: (i) It rewards entities for occurring throughout the text instead of only in some parts of the text, measured by the number of paragraphs it occurs in (ii) it rewards entities that occur more frequently in the opening paragraphs of an article as we model INLINEFORM1 as an exponential decay function. The decay corresponds to the positional index of the news paragraph. This is inspired by the news-specific discourse structure that tends to give short summaries of the most important facts and entities in the opening paragraphs. (iii) it compares entity frequency to the frequency of its co-occurring mentions as the weight of an entity appearing in a specific paragraph, normalized by the sum of the frequencies of other entities in INLINEFORM2 . DISPLAYFORM0 where, INLINEFORM0 represents a news paragraph from INLINEFORM1 , and with INLINEFORM2 we indicate the set of all paragraphs in INLINEFORM3 . The frequency of INLINEFORM4 in a paragraph INLINEFORM5 is denoted by INLINEFORM6 . With INLINEFORM7 and INLINEFORM8 we indicate the number of paragraphs in which entity INLINEFORM9 occurs, and the total number of paragraphs, respectively. Relative Authority. In this case, we consider the comparative relevance of the news article to the different entities occurring in it. As an example, let us consider the meeting of the Sudanese bishop Elias Taban with Hillary Clinton. Both entities are salient for the meeting. However, in Taban's Wikipedia page, this meeting is discussed prominently with a corresponding news reference, whereas in Hillary Clinton's Wikipedia page it is not reported at all. We believe this is not just an omission in Clinton's page but mirrors the fact that for the lesser known Taban the meeting is big news whereas for the more famous Clinton these kind of meetings are a regular occurrence, not all of which can be reported in what is supposed to be a selection of the most important events for her. Therefore, if two entities co-occur, the news is more relevant for the entity with the lower a priori authority. The a priori authority of an entity (denoted by INLINEFORM0 ) can be measured in several ways. We opt for two approaches: (i) probability of entity INLINEFORM1 occurring in the corpus INLINEFORM2 , and (ii) authority assessed through centrality measures like PageRank BIBREF16 . For the second case we construct the graph INLINEFORM3 consisting of entities in INLINEFORM4 and news articles in INLINEFORM5 as vertices. The edges are established between INLINEFORM6 and entities in INLINEFORM7 , that is INLINEFORM8 , and the out-links from INLINEFORM9 , that is INLINEFORM10 (arrows present the edge direction). Starting from a priori authority, we proceed to relative authority by comparing the a priori authority of co-occurring entities in INLINEFORM0 . We define the relative authority of INLINEFORM1 as the proportion of co-occurring entities INLINEFORM2 that have a higher a priori authority than INLINEFORM3 (see Equation EQREF28 . DISPLAYFORM0 As we might run the danger of not suggesting any news articles for entities with very high a priori authority (such as Clinton) due to the strict inequality constraint, we can relax the constraint such that the authority of co-occurring entities is above a certain threshold. News Domain Authority. The news domain authority addresses two main aspects. Firstly, if bundled together with the relative authority feature, we can ensure that dependent on the entity authority, we suggest news from authoritative sources, hence ensuring the quality of suggested articles. The second aspect is in a news streaming scenario where multiple news domains report the same event — ideally only articles coming from authoritative sources would fulfill the conditions for the news suggestion task. The news domain authority is computed based on the number of news references in Wikipedia coming from a particular news domain INLINEFORM0 . This represents a simple prior that a news article INLINEFORM1 is from domain INLINEFORM2 in corpus INLINEFORM3 . We extract the domains by taking the base URLs from the news article URLs. An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w.r.t the already existing entity profile INLINEFORM3 . Studies BIBREF17 have shown that on comparable collections to ours (TREC GOV2) the number of duplicates can go up to INLINEFORM4 . This figure is likely higher for major events concerning highly authoritative entities on which all news media will report. Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6 . We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8 . The novelty value of INLINEFORM9 is given by the minimal divergence value. Low scores indicate low novelty for the entity profile INLINEFORM10 . N(n|e) = n'Nt-1{DKL((n') || (n)) + DKL((N) || (n)). DKL((n') || (n)). (1-) jaccard((n'),(n))} where INLINEFORM0 is the KL divergence of the language models ( INLINEFORM1 and INLINEFORM2 ), whereas INLINEFORM3 is the mixing weight ( INLINEFORM4 ) between the language models INLINEFORM5 and the entity overlap in INLINEFORM6 and INLINEFORM7 . Here we introduce the evaluation setup and analyze the results for the article–entity (AEP) placement task. We only report the evaluation metrics for the `relevant' news-entity pairs. A detailed explanation on why we focus on the `relevant' pairs is provided in Section SECREF16 . Baselines. We consider the following baselines for this task. B1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 . B2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 . Learning Models. We use Random Forests (RF) BIBREF23 . We learn the RF on all computed features in Table TABREF21 . The optimization on RF is done by splitting the feature space into multiple trees that are considered as ensemble classifiers. Consequently, for each classifier it computes the margin function as a measure of the average count of predicting the correct class in contrast to any other class. The higher the margin score the more robust the model. Metrics. We compute precision P, recall R and F1 score for the relevant class. For example, precision is the number of news-entity pairs we correctly labeled as relevant compared to our ground truth divided by the number of all news-entity pairs we labeled as relevant. The following results measure the effectiveness of our approach in three main aspects: (i) overall performance of INLINEFORM0 and comparison to baselines, (ii) robustness across the years, and (iii) optimal model for the AEP placement task. Performance. Figure FIGREF55 shows the results for the years 2009 and 2013, where we optimized the learning objective with instances from year INLINEFORM0 and evaluate on the years INLINEFORM1 (see Section SECREF46 ). The results show the precision–recall curve. The red curve shows baseline B1 BIBREF11 , and the blue one shows the performance of INLINEFORM2 . The curve shows for varying confidence scores (high to low) the precision on labeling the pair INLINEFORM3 as `relevant'. In addition, at each confidence score we can compute the corresponding recall for the `relevant' label. For high confidence scores on labeling the news-entity pairs, the baseline B1 achieves on average a precision score of P=0.50, while INLINEFORM4 has P=0.93. We note that with the drop in the confidence score the corresponding precision and recall values drop too, and the overall F1 score for B1 is around F1=0.2, in contrast we achieve an average score of F1=0.67. It is evident from Figure FIGREF55 that for the years 2009 and 2013, INLINEFORM0 significantly outperforms the baseline B1. We measure the significance through the t-test statistic and get a p-value of INLINEFORM1 . The improvement we achieve over B1 in absolute numbers, INLINEFORM2 P=+0.5 in terms of precision for the years between 2009 and 2014, and a similar improvement in terms of F1 score. The improvement for recall is INLINEFORM3 R=+0.4. The relative improvement over B1 for P and F1 is almost 1.8 times better, while for recall we are 3.5 times better. In Table TABREF58 we show the overall scores for the evaluation metrics for B1 and INLINEFORM4 . Finally, for B2 we achieve much poorer performance, with average scores of P=0.21, R=0.20 and F1=0.21. Robustness. In Table TABREF58 , we show the overall performance for the years between 2009 and 2013. An interesting observation we make is that we have a very robust performance and the results are stable across the years. If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance. We predict with F1=0.68 the remaining 469k instances for the years INLINEFORM1 . The results are particularly promising considering the fact that the distribution between our two classes is highly skewed. On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs. A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic. INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions). The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1). In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines. Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears. The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined. The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase. However, if both feature groups are considered separately, they significantly outperform the baseline B1. Article–Section Placement We model the ASP placement task as a successor of the AEP task. For all the `relevant' news entity pairs, the task is to determine the correct entity section. Each section in a Wikipedia entity page represents a different topic. For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc. However, many entity pages have an incomplete section structure. Incomplete or missing sections are due to two Wikipedia properties. First, long-tail entities miss information and sections due to their lack of popularity. Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments. As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline. Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e.g. other airlines had disasters and therefore their pages have an accidents section). We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles. The learning objective for INLINEFORM0 takes into account the following properties: Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2 Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0 Given the fact that entity profiles are often incomplete, we construct section templates for every entity class. We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1 . For different entity classes, e.g. Person and Location, the section structure and the information represented in those section varies heavily. Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 ). DISPLAYFORM0 Generating section templates has two main advantages. Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0 . The second advantage is that we are able to canonicalize the sections, i.e. `Early Life' and `Early Life and Childhood' would be treated similarly. To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2 . Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al. which estimates the number of clusters efficiently BIBREF19 . As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections. Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand. x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in. The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2 . The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3 . In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6 . The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9 . To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31 . For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively. Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ). Topic. We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section. In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2 . The topic models are computed using LDA BIBREF20 . We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections. We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8 . Syntactic. These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i.e. NNP, CD etc.), as commonly used in text summarization tasks. Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags. We expect that for different sections, the importance of POS tags will vary. We measure the similarity of POS tags in a news article against the section text. Additionally, we consider bi-gram and tri-gram POS tag overlap. This exploits similarity in syntactical patterns between the news and section text. Lexical. As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2 . Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph). A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7 . We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text. Entity-based. Another feature set we consider is the overlap of named entities and their corresponding entity classes. For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e.g. `Early Life' contains mostly entities related to family, school, universities etc. Frequency. Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags. Here we try to capture patterns of articles that are usually cited in specific sections. Evaluation Plan In this section we outline the evaluation plan to verify the effectiveness of our learning approaches. To evaluate the news suggestion problem we are faced with two challenges. What comprises the ground truth for such a task ? How do we construct training and test splits given that entity pages consists of text added at different points in time ? Consider the ground truth challenge. Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert. An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time. The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame. It is hard to estimate how much time the added content should be left on the entity page. A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup. The problem of such an approach is again finding knowledgeable users or experts for long-tail entities. Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup. We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant. To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 ). At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation. The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46 . Datasets The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history. The generated data represents one of the contributions of our paper. The datasets are the following: Entity Classes. We focus on a manually predetermined set of entity classes for which we expect to have news coverage. The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference. The entity classes were selected from the DBpedia class ontology. Figure FIGREF42 shows the number of entities per class for the years (2009-2014). News Articles. We extract all news references from the collected Wikipedia entity pages. The extracted news references are associated with the sections in which they appear. In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles. The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44 . Article-Entity Ground-truth. The dataset comprises of the news and entity pairs INLINEFORM0 . News-entity pairs are relevant if the news article is referenced in the entity page. Non-relevant pairs (i.e. negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page. If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4 . Article-Section Ground-truth. The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant. We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited. Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7 . Data Pre-Processing We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21 . For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0.3. On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0.9. On average, the number of entities per news article is approximately 30. For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities. Train and Testing Evaluation Setup We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances. It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5 . This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7 . Table TABREF47 shows the statistics of train and test instances. We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9 . Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation. Article-Section Placement Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates. Baselines. To the best of our knowledge, we are not aware of any comparable approach for this task. Therefore, the baselines we consider are the following: S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2 S2: Place the news into the most frequent section in INLINEFORM0 Learning Models. We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24 . The models are optimized taking into account the features in Table TABREF31 . In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models. The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels. Metrics. We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 ). The definition of recall R and F1 score follows from that of precision. Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2. With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function. For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest. The performance on average is around P=0.66 across all classes. Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further. If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0.17, with around 70% of instances used for training and the remainder for evaluation. For the remaining years the total improvement is INLINEFORM3 P=0.18 in contrast to the performance at year 2009. On the other hand, the baseline S1 has an average precision of P=0.12. The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0.13. Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0.17, with a uniform distribution across the years. Here we show the performance of INLINEFORM0 decomposed for the different entity classes. Specifically we analyze the 27 classes in Figure FIGREF42 . In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints). For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68 . For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics. It is evident that in the first year the performance is lower in contrast to the later years. This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1 . The results are already stable for the year range INLINEFORM2 . For a few Person sub-classes, e.g. Politics, Entertainment, we achieve an F1 score above 0.9. These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3 . The lowest F1 score is for the Criminal and Television classes. However, this is directly correlated with the insufficient number of instances. The baseline approaches for the ASP task perform poorly. S1, based on lexical similarity, has a varying performance for different entity classes. The best performance is achieved for the class Person – Politics, with P=0.43. This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results. For different entity classes different features and combination of features is necessary. Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0.64. The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections. We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic. In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 ). The score we achieve shows that our model predicts with high confidence with INLINEFORM3 . The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections. Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2 . The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4 . However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7 . In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile. With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete'. This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly. In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0 . An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline. However, it is non-existent for some specific entity instances, i.e Germanwings airline. Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities. We distinguish between the two types of entities by simply measuring their section text length. The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively. We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities. Conclusion and Future Work In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating. The process consists of two stages. In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page. In the second stage, article–section placement, we determine the best fitting section in an entity page. Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes. This allows us to add missing sections to entity pages. We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes. For the first stage, we achieve an overall performance with P=0.93, R=0.514 and F1=0.676, outperforming our baseline competitors significantly. For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates. The overall performance across different classes is P=0.844, R=0.885 and F1=0.860. In the future, we will enhance our work by extracting facts from the suggested news articles. Results suggest that the news content cited in entity pages comes from the first paragraphs. However, challenging task such as the canonicalization and chronological ordering of facts, still remain.
Do they report results only on English data?
77c34f1033702278f7f044806c1eba0c6ecb8b04
Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish Ġdata ?
Yes
<s> Do Ġthey Ġreport Ġresults Ġonly Ġon ĠEnglish Ġdata ? </s> Introduction </s> Wikipedia Ġis Ġthe Ġlargest Ġsource Ġof Ġopen Ġand Ġcollabor atively Ġcurated Ġknowledge Ġin Ġthe Ġworld . ĠIntroduced Ġin Ġ2001 , Ġit Ġhas Ġevolved Ġinto Ġa Ġreference Ġwork Ġwith Ġaround Ġ5 m Ġpages Ġfor Ġthe ĠEnglish ĠWikipedia Ġalone . ĠIn Ġaddition , Ġentities Ġand Ġevent Ġpages Ġare Ġupdated Ġquickly Ġvia Ġcollaborative Ġediting Ġand Ġall Ġedits Ġare Ġencouraged Ġto Ġinclude Ġsource Ġcitations , Ġcreating Ġa Ġknowledge Ġbase Ġwhich Ġaims Ġat Ġbeing Ġboth Ġtimely Ġas Ġwell Ġas Ġauthoritative . ĠAs Ġa Ġresult , Ġit Ġhas Ġbecome Ġthe Ġpreferred Ġsource Ġof Ġinformation Ġconsumption Ġabout Ġentities Ġand Ġevents . ĠMore so , Ġthis Ġknowledge Ġis Ġharvested Ġand Ġutilized Ġin Ġbuilding Ġknowledge Ġbases Ġlike ĠY AG O ĠB IB REF 0 Ġand ĠDB pedia ĠB IB REF 1 Ġ, Ġand Ġused Ġin Ġapplications Ġlike Ġtext Ġcategor ization ĠB IB REF 2 Ġ, Ġentity Ġdis amb ig uation ĠB IB REF 3 Ġ, Ġentity Ġranking ĠB IB REF 4 Ġand Ġdistant Ġsupervision ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> However , Ġnot Ġall ĠWikipedia Ġpages Ġreferring Ġto Ġentities Ġ( entity Ġpages ) Ġare Ġcomprehensive : Ġrelevant Ġinformation Ġcan Ġeither Ġbe Ġmissing Ġor Ġadded Ġwith Ġa Ġdelay . ĠConsider Ġthe Ġcity Ġof ĠNew ĠOrleans Ġand Ġthe Ġstate Ġof ĠOd isha Ġwhich Ġwere Ġseverely Ġaffected Ġby Ġcycl ones ĠHurricane ĠKatrina Ġand ĠOd isha ĠCy clone , Ġrespectively . ĠWhile ĠKatrina Ġfinds Ġextensive Ġmention Ġin Ġthe Ġentity Ġpage Ġfor ĠNew ĠOrleans , ĠOd isha ĠCy clone Ġwhich Ġhas Ġ5 Ġtimes Ġmore Ġhuman Ġcasualties Ġ( cf . ĠFigure ĠFIG REF 2 Ġ) Ġis Ġnot Ġmentioned Ġin Ġthe Ġpage Ġfor ĠOd isha . ĠAr gu ably ĠKatrina Ġand ĠNew ĠOrleans Ġare Ġmore Ġpopular Ġentities , Ġbut ĠOd isha ĠCy clone Ġwas Ġalso Ġreported Ġextensively Ġin Ġnational Ġand Ġinternational Ġnews Ġoutlets . ĠThis Ġhighlights Ġthe Ġlack Ġof Ġimportant Ġfacts Ġin Ġtrunk Ġand Ġlong - tail Ġentity Ġpages , Ġeven Ġin Ġthe Ġpresence Ġof Ġrelevant Ġsources . ĠIn Ġaddition , Ġprevious Ġstudies Ġhave Ġshown Ġthat Ġthere Ġis Ġan Ġinherent Ġdelay Ġor Ġlag Ġwhen Ġfacts Ġare Ġadded Ġto Ġentity Ġpages ĠB IB REF 7 Ġ. </s> To Ġremedy Ġthese Ġproblems , Ġit Ġis Ġimportant Ġto Ġidentify Ġinformation Ġsources Ġthat Ġcontain Ġnovel Ġand Ġsalient Ġfacts Ġto Ġa Ġgiven Ġentity Ġpage . ĠHowever , Ġnot Ġall Ġinformation Ġsources Ġare Ġequal . ĠThe Ġonline Ġpresence Ġof Ġmajor Ġnews Ġoutlets Ġis Ġan Ġauthoritative Ġsource Ġdue Ġto Ġactive Ġeditorial Ġcontrol Ġand Ġtheir Ġarticles Ġare Ġalso Ġa Ġtimely Ġcontainer Ġof Ġfacts . ĠIn Ġaddition , Ġtheir Ġuse Ġis Ġin Ġline Ġwith Ġcurrent ĠWikipedia Ġediting Ġpractice , Ġas Ġis Ġshown Ġin ĠB IB REF 7 Ġthat Ġalmost Ġ20 % Ġof Ġcurrent Ġcitations Ġin Ġall Ġentity Ġpages Ġare Ġnews Ġarticles . ĠWe Ġtherefore Ġpropose Ġnews Ġsuggestion Ġas Ġa Ġnovel Ġtask Ġthat Ġenhances Ġentity Ġpages Ġand Ġreduces Ġdelay Ġwhile Ġkeeping Ġits Ġpages Ġauthoritative . </s> Ex isting Ġefforts Ġto Ġpopulate ĠWikipedia ĠB IB REF 8 Ġstart Ġfrom Ġan Ġentity Ġpage Ġand Ġthen Ġgenerate Ġcandidate Ġdocuments Ġabout Ġthis Ġentity Ġusing Ġan Ġexternal Ġsearch Ġengine Ġ( and Ġthen Ġpost - process Ġthem ). ĠHowever , Ġsuch Ġan Ġapproach Ġlacks Ġin Ġ( a ) Ġreprodu c ibility Ġsince Ġrankings Ġvary Ġwith Ġtime Ġwith Ġobvious Ġbias Ġto Ġrecent Ġnews Ġ( b ) Ġmaintain ability Ġsince Ġdocument Ġacquisition Ġfor Ġeach Ġentity Ġhas Ġto Ġbe Ġperiodically Ġperformed . ĠTo Ġthis Ġeffect , Ġour Ġnews Ġsuggestion Ġconsiders Ġa Ġnews Ġarticle Ġas Ġinput , Ġand Ġdetermines Ġif Ġit Ġis Ġvaluable Ġfor ĠWikipedia . ĠSpecifically , Ġgiven Ġan Ġinput Ġnews Ġarticle ĠIN LINE FORM 0 Ġand Ġa Ġstate Ġof ĠWikipedia , Ġthe Ġnews Ġsuggestion Ġproblem Ġidentifies Ġthe Ġentities Ġmentioned Ġin ĠIN LINE FORM 1 Ġwhose Ġentity Ġpages Ġcan Ġimprove Ġupon Ġsuggesting ĠIN LINE FORM 2 Ġ. ĠMost Ġof Ġthe Ġworks Ġon Ġknowledge Ġbase Ġacceleration ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġor ĠWikipedia Ġpage Ġgeneration ĠB IB REF 8 Ġrely Ġon Ġhigh Ġquality Ġinput Ġsources Ġwhich Ġare Ġthen Ġutilized Ġto Ġextract Ġtextual Ġfacts Ġfor ĠWikipedia Ġpage Ġpopulation . ĠIn Ġthis Ġwork , Ġwe Ġdo Ġnot Ġsuggest Ġsnippets Ġor Ġparaph r ases Ġbut Ġrather Ġentire Ġarticles Ġwhich Ġhave Ġa Ġhigh Ġpotential Ġimportance Ġfor Ġentity Ġpages . ĠThese Ġsuggested Ġnews Ġarticles Ġcould Ġbe Ġconsequently Ġused Ġfor Ġextraction , Ġsummar ization Ġor Ġpopulation Ġeither Ġmanually Ġor Ġautomatically ĠâĢĵ Ġall Ġof Ġwhich Ġrely Ġon Ġhigh Ġquality Ġand Ġrelevant Ġinput Ġsources . </s> We Ġidentify Ġfour Ġproperties Ġof Ġgood Ġnews Ġrecommendations : Ġsal ience , Ġrelative Ġauthority , Ġnovelty Ġand Ġplacement . ĠFirst , Ġwe Ġneed Ġto Ġidentify Ġthe Ġmost Ġsalient Ġentities Ġin Ġa Ġnews Ġarticle . ĠThis Ġis Ġdone Ġto Ġavoid Ġpollution Ġof Ġentity Ġpages Ġwith Ġonly Ġmarginally Ġrelated Ġnews . ĠSecond , Ġwe Ġneed Ġto Ġdetermine Ġwhether Ġthe Ġnews Ġis Ġimportant Ġto Ġthe Ġentity Ġas Ġonly Ġthe Ġmost Ġrelevant Ġnews Ġshould Ġbe Ġadded Ġto Ġa Ġprecise Ġreference Ġwork . ĠTo Ġdo Ġthis , Ġwe Ġcompute Ġthe Ġrelative Ġauthority Ġof Ġall Ġentities Ġin Ġthe Ġnews Ġarticle : Ġwe Ġcall Ġan Ġentity Ġmore Ġauthoritative Ġthan Ġanother Ġif Ġit Ġis Ġmore Ġpopular Ġor Ġnoteworthy Ġin Ġthe Ġreal Ġworld . ĠEnt ities Ġwith Ġvery Ġhigh Ġauthority Ġhave Ġmany Ġnews Ġitems Ġassociated Ġwith Ġthem Ġand Ġonly Ġthe Ġmost Ġrelevant Ġof Ġthese Ġshould Ġbe Ġincluded Ġin ĠWikipedia Ġwhereas Ġfor Ġentities Ġof Ġlower Ġauthority Ġthe Ġthreshold Ġfor Ġinclusion Ġof Ġa Ġnews Ġarticle Ġwill Ġbe Ġlower . ĠThird , Ġa Ġgood Ġrecommendation Ġshould Ġbe Ġable Ġto Ġidentify Ġnovel Ġnews Ġby Ġminimizing Ġredundancy Ġcoming Ġfrom Ġmultiple Ġnews Ġarticles . ĠFinally , Ġaddition Ġof Ġfacts Ġis Ġfacilitated Ġif Ġthe Ġrecommendations Ġare Ġfine - gr ained , Ġi . e ., Ġrecommendations Ġare Ġmade Ġon Ġthe Ġsection Ġlevel Ġrather Ġthan Ġthe Ġpage Ġlevel Ġ( pl acement ). </s> App roach Ġand ĠContributions . ĠWe Ġpropose Ġa Ġtwo - stage Ġnews Ġsuggestion Ġapproach Ġto Ġentity Ġpages . ĠIn Ġthe Ġfirst Ġstage , Ġwe Ġdetermine Ġwhether Ġa Ġnews Ġarticle Ġshould Ġbe Ġsuggested Ġfor Ġan Ġentity , Ġbased Ġon Ġthe Ġentity 's Ġsal ience Ġin Ġthe Ġnews Ġarticle , Ġits Ġrelative Ġauthority Ġand Ġthe Ġnovelty Ġof Ġthe Ġarticle Ġto Ġthe Ġentity Ġpage . ĠThe Ġsecond Ġstage Ġtakes Ġinto Ġaccount Ġthe Ġclass Ġof Ġthe Ġentity Ġfor Ġwhich Ġthe Ġnews Ġis Ġsuggested Ġand Ġconstructs Ġsection Ġtemplates Ġfrom Ġentities Ġof Ġthe Ġsame Ġclass . ĠThe Ġgeneration Ġof Ġsuch Ġtemplates Ġhas Ġthe Ġadvantage Ġof Ġsuggesting Ġand Ġexpanding Ġentity Ġpages Ġthat Ġdo Ġnot Ġhave Ġa Ġcomplete Ġsection Ġstructure Ġin ĠWikipedia , Ġexplicitly Ġaddressing Ġlong - tail Ġand Ġtrunk Ġentities . ĠAfterwards , Ġbased Ġon Ġthe Ġconstructed Ġtemplate Ġour Ġmethod Ġdetermines Ġthe Ġbest Ġfit Ġfor Ġthe Ġnews Ġarticle Ġwith Ġone Ġof Ġthe Ġsections . </s> We Ġevaluate Ġthe Ġproposed Ġapproach Ġon Ġa Ġnews Ġcorpus Ġconsisting Ġof Ġ351 , 98 2 Ġarticles Ġcrawled Ġfrom Ġthe Ġnews Ġexternal Ġreferences Ġin ĠWikipedia Ġfrom Ġ73 , 7 34 Ġentity Ġpages . ĠGiven Ġthe ĠWikipedia Ġsnapshot Ġat Ġa Ġgiven Ġyear Ġ( in Ġour Ġcase Ġ[ 2009 - 2014 ]), Ġwe Ġsuggest Ġnews Ġarticles Ġthat Ġmight Ġbe Ġcited Ġin Ġthe Ġcoming Ġyears . ĠThe Ġexisting Ġnews Ġreferences Ġin Ġthe Ġentity Ġpages Ġalong Ġwith Ġtheir Ġreference Ġdate Ġact Ġas Ġour Ġground - truth Ġto Ġevaluate Ġour Ġapproach . ĠIn Ġsummary , Ġwe Ġmake Ġthe Ġfollowing Ġcontributions . </s> Related ĠWork </s> As Ġwe Ġsuggest Ġa Ġnew Ġproblem Ġthere Ġis Ġno Ġcurrent Ġwork Ġaddressing Ġexactly Ġthe Ġsame Ġtask . ĠHowever , Ġour Ġtask Ġhas Ġsimilarities Ġto ĠWikipedia Ġpage Ġgeneration Ġand Ġknowledge Ġbase Ġacceleration . ĠIn Ġaddition , Ġwe Ġtake Ġinspiration Ġfrom ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġmethods Ġfor Ġsal ience Ġdetection . </s> Wikipedia ĠPage ĠGeneration Ġis Ġthe Ġproblem Ġof Ġpop ulating ĠWikipedia Ġpages Ġwith Ġcontent Ġcoming Ġfrom Ġexternal Ġsources . ĠSau per Ġand ĠBar z il ay ĠB IB REF 8 Ġpropose Ġan Ġapproach Ġfor Ġautomatically Ġgenerating Ġwhole Ġentity Ġpages Ġfor Ġspecific Ġentity Ġclasses . ĠThe Ġapproach Ġis Ġtrained Ġon Ġalready - pop ulated Ġentity Ġpages Ġof Ġa Ġgiven Ġclass Ġ( e . g . Ġ` D ise ases ') Ġby Ġlearning Ġtemplates Ġabout Ġthe Ġentity Ġpage Ġstructure Ġ( e . g . Ġdiseases Ġhave Ġa Ġtreatment Ġsection ). ĠFor Ġa Ġnew Ġentity Ġpage , Ġfirst , Ġthey Ġextract Ġdocuments Ġvia ĠWeb Ġsearch Ġusing Ġthe Ġentity Ġtitle Ġand Ġthe Ġsection Ġtitle Ġas Ġa Ġquery , Ġfor Ġexample Ġ` L ung ĠCancer ' + ` T reatment '. ĠAs Ġalready Ġdiscussed Ġin Ġthe Ġintroduction , Ġthis Ġhas Ġproblems Ġwith Ġreprodu c ibility Ġand Ġmaintain ability . ĠHowever , Ġtheir Ġmain Ġfocus Ġis Ġon Ġidentifying Ġthe Ġbest Ġparagraphs Ġextracted Ġfrom Ġthe Ġcollected Ġdocuments . ĠThey Ġrank Ġthe Ġparagraphs Ġvia Ġan Ġoptimized Ġsupervised Ġpercept ron Ġmodel Ġfor Ġfinding Ġthe Ġmost Ġrepresentative Ġparagraph Ġthat Ġis Ġthe Ġleast Ġsimilar Ġto Ġparagraphs Ġin Ġother Ġsections . ĠThis Ġparagraph Ġis Ġthen Ġincluded Ġin Ġthe Ġnewly Ġgenerated Ġentity Ġpage . ĠT ane va Ġand ĠWe ik um ĠB IB REF 12 Ġpropose Ġan Ġapproach Ġthat Ġconstructs Ġshort Ġsumm aries Ġfor Ġthe Ġlong Ġtail . ĠThe Ġsumm aries Ġare Ġcalled Ġ` g ems ' Ġand Ġthe Ġsize Ġof Ġa Ġ` gem ' Ġcan Ġbe Ġuser Ġdefined . ĠThey Ġfocus Ġon Ġgenerating Ġsumm aries Ġthat Ġare Ġnovel Ġand Ġdiverse . ĠHowever , Ġthey Ġdo Ġnot Ġconsider Ġany Ġstructure Ġof Ġentities , Ġwhich Ġis Ġpresent Ġin ĠWikipedia . </s> In Ġcontrast Ġto ĠB IB REF 8 Ġand ĠB IB REF 12 Ġ, Ġwe Ġactually Ġfocus Ġon Ġsuggesting Ġentire Ġdocuments Ġto ĠWikipedia Ġentity Ġpages . ĠThese Ġare Ġauthoritative Ġdocuments Ġ( news ), Ġwhich Ġare Ġhighly Ġrelevant Ġfor Ġthe Ġentity , Ġnovel Ġfor Ġthe Ġentity Ġand Ġin Ġwhich Ġthe Ġentity Ġis Ġsalient . ĠWhereas Ġrelevance Ġin ĠSau per Ġand ĠBar z il ay Ġis Ġimplicitly Ġcomputed Ġby Ġweb Ġpage Ġranking Ġwe Ġsolve Ġthat Ġproblem Ġby Ġlooking Ġat Ġrelative Ġauthority Ġand Ġsal ience Ġof Ġan Ġentity , Ġusing Ġthe Ġnews Ġarticle Ġand Ġentity Ġpage Ġonly . ĠAs ĠSau per Ġand ĠBar z il ay Ġconcentrate Ġon Ġempty Ġentity Ġpages , Ġthe Ġproblem Ġof Ġnovelty Ġof Ġtheir Ġcontent Ġis Ġnot Ġan Ġissue Ġin Ġtheir Ġwork Ġwhereas Ġit Ġis Ġin Ġour Ġcase Ġwhich Ġfocuses Ġmore Ġon Ġupdating Ġentities . ĠUp dating Ġentities Ġwill Ġbe Ġmore Ġand Ġmore Ġimportant Ġthe Ġbigger Ġan Ġexisting Ġreference Ġwork Ġis . ĠBoth Ġthe Ġapproaches Ġin ĠB IB REF 8 Ġand ĠB IB REF 12 Ġ( finding Ġparagraphs Ġand Ġsummar ization ) Ġcould Ġthen Ġbe Ġused Ġto Ġprocess Ġthe Ġdocuments Ġwe Ġsuggest Ġfurther . ĠOur Ġconcentration Ġon Ġnews Ġis Ġalso Ġnovel . </s> Know ledge ĠBase ĠAcceler ation . ĠIn Ġthis Ġtask , Ġgiven Ġspecific Ġinformation Ġextraction Ġtemplates , Ġa Ġgiven Ġcorpus Ġis Ġanalyzed Ġin Ġorder Ġto Ġfind Ġworthwhile Ġmentions Ġof Ġan Ġentity Ġor Ġsnippets Ġthat Ġmatch Ġthe Ġtemplates . ĠBal og ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġrecommend Ġnews Ġcitations Ġfor Ġan Ġentity . ĠPrior Ġto Ġthat , Ġthe Ġnews Ġarticles Ġare Ġclassified Ġfor Ġtheir Ġappropri at eness Ġfor Ġan Ġentity , Ġwhere Ġas Ġfeatures Ġfor Ġthe Ġclassification Ġtask Ġthey Ġuse Ġentity , Ġdocument , Ġentity - document Ġand Ġtemporal Ġfeatures . ĠThe Ġbest Ġperforming Ġfeatures Ġare Ġthose Ġthat Ġmeasure Ġsimilarity Ġbetween Ġan Ġentity Ġand Ġthe Ġnews Ġdocument . ĠWest Ġet Ġal . ĠB IB REF 13 Ġconsider Ġthe Ġproblem Ġof Ġknowledge Ġbase Ġcompletion , Ġthrough Ġquestion Ġanswering Ġand Ġcomplete Ġmissing Ġfacts Ġin ĠFree base Ġbased Ġon Ġtemplates , Ġi . e . ĠFrank _ Z appa Ġborn In ĠB alty more , ĠMaryland . </s> In Ġcontrast , Ġwe Ġdo Ġnot Ġextract Ġfacts Ġfor Ġpre - defined Ġtemplates Ġbut Ġrather Ġsuggest Ġnews Ġarticles Ġbased Ġon Ġtheir Ġrelevance Ġto Ġan Ġentity . ĠIn Ġcases Ġof Ġlong - tail Ġentities , Ġwe Ġcan Ġsuggest Ġto Ġadd Ġa Ġnovel Ġsection Ġthrough Ġour Ġabstraction Ġand Ġgeneration Ġof Ġsection Ġtemplates Ġat Ġentity Ġclass Ġlevel . </s> Entity ĠSal ience . ĠD etermin ing Ġwhich Ġentities Ġare Ġprominent Ġor Ġsalient Ġin Ġa Ġgiven Ġtext Ġhas Ġa Ġlong Ġhistory Ġin ĠN LP , Ġsparked Ġby Ġthe Ġlinguistic Ġtheory Ġof ĠCent ering ĠB IB REF 14 Ġ. ĠSal ience Ġhas Ġbeen Ġused Ġin Ġpronoun Ġand Ġco - reference Ġresolution ĠB IB REF 15 Ġ, Ġor Ġto Ġpredict Ġwhich Ġentities Ġwill Ġbe Ġincluded Ġin Ġan Ġabstract Ġof Ġan Ġarticle ĠB IB REF 11 Ġ. ĠFrequ ent Ġfeatures Ġto Ġmeasure Ġsal ience Ġinclude Ġthe Ġfrequency Ġof Ġan Ġentity Ġin Ġa Ġdocument , Ġpositioning Ġof Ġan Ġentity , Ġgram matical Ġfunction Ġor Ġinternal Ġentity Ġstructure Ġ( POS Ġtags , Ġhead Ġnoun s Ġetc .). ĠThese Ġapproaches Ġare Ġnot Ġcurrently Ġaimed Ġat Ġknowledge Ġbase Ġgeneration Ġor ĠWikipedia Ġcoverage Ġextension Ġbut Ġwe Ġpost ulate Ġthat Ġan Ġentity 's Ġsal ience Ġin Ġa Ġnews Ġarticle Ġis Ġa Ġprerequisite Ġto Ġthe Ġnews Ġarticle Ġbeing Ġrelevant Ġenough Ġto Ġbe Ġincluded Ġin Ġan Ġentity Ġpage . ĠWe Ġtherefore Ġuse Ġthe Ġsal ience Ġfeatures Ġin ĠB IB REF 11 Ġas Ġpart Ġof Ġour Ġmodel . ĠHowever , Ġthese Ġfeatures Ġare Ġdocument - internal ĠâĢĶ Ġwe Ġwill Ġshow Ġthat Ġthey Ġare Ġnot Ġsufficient Ġto Ġpredict Ġnews Ġinclusion Ġinto Ġan Ġentity Ġpage Ġand Ġadd Ġfeatures Ġof Ġentity Ġauthority , Ġnews Ġauthority Ġand Ġnovelty Ġthat Ġmeasure Ġthe Ġrelations Ġbetween Ġseveral Ġentities , Ġbetween Ġentity Ġand Ġnews Ġarticle Ġas Ġwell Ġas Ġbetween Ġseveral Ġcompeting Ġnews Ġarticles . </s> Termin ology Ġand ĠProblem ĠDefinition </s> We Ġare Ġinterested Ġin Ġnamed Ġentities Ġmentioned Ġin Ġdocuments . ĠAn Ġentity ĠIN LINE FORM 0 Ġcan Ġbe Ġidentified Ġby Ġa Ġcanonical Ġname , Ġand Ġcan Ġbe Ġmentioned Ġdifferently Ġin Ġtext Ġvia Ġdifferent Ġsurface Ġforms . ĠWe Ġcanonical ize Ġthese Ġmentions Ġto Ġentity Ġpages Ġin ĠWikipedia , Ġa Ġmethod Ġtypically Ġknown Ġas Ġentity Ġlinking . ĠWe Ġdenote Ġthe Ġset Ġof Ġcanonical ized Ġentities Ġextracted Ġand Ġlinked Ġfrom Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġas ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , Ġin ĠFigure ĠFIG REF 7 Ġ, Ġentities Ġare Ġcanonical ized Ġinto ĠWikipedia Ġentity Ġpages Ġ( e . g . ĠOd isha Ġis Ġcanonical ized Ġto Ġthe Ġcorresponding Ġarticle ). ĠFor Ġa Ġcollection Ġof Ġnews Ġarticles ĠIN LINE FORM 3 Ġ, Ġwe Ġfurther Ġdenote Ġthe Ġresulting Ġset Ġof Ġentities Ġby ĠIN LINE FORM 4 Ġ. </s> Information Ġin Ġan Ġentity Ġpage Ġis Ġorganized Ġinto Ġsections Ġand Ġevolves Ġwith Ġtime Ġas Ġmore Ġcontent Ġis Ġadded . ĠWe Ġrefer Ġto Ġthe Ġstate Ġof ĠWikipedia Ġat Ġa Ġtime ĠIN LINE FORM 0 Ġas ĠIN LINE FORM 1 Ġand Ġthe Ġset Ġof Ġsections Ġfor Ġan Ġentity Ġpage ĠIN LINE FORM 2 Ġas Ġits Ġentity Ġprofile ĠIN LINE FORM 3 Ġ. ĠUnlike Ġnews Ġarticles , Ġtext Ġin ĠWikipedia Ġcould Ġbe Ġexplicitly Ġlinked Ġto Ġentity Ġpages Ġthrough Ġanchors . ĠThe Ġset Ġof Ġentities Ġexplicitly Ġreferred Ġin Ġtext Ġfrom Ġsection ĠIN LINE FORM 4 Ġis Ġdefined Ġas ĠIN LINE FORM 5 Ġ. ĠFurthermore , ĠWikipedia Ġinduces Ġa Ġcategory Ġstructure Ġover Ġits Ġentities , Ġwhich Ġis Ġexploited Ġby Ġknowledge Ġbases Ġlike ĠY AG O Ġ( e . g . ĠBarack _ Obama Ġis A ĠPerson ). ĠConsequently , Ġeach Ġentity Ġpage Ġbelongs Ġto Ġone Ġor Ġmore Ġentity Ġcategories Ġor Ġclasses ĠIN LINE FORM 6 Ġ. ĠNow Ġwe Ġcan Ġdefine Ġour Ġnews Ġsuggestion Ġproblem Ġbelow : </s> Definition Ġ1 Ġ( News ĠSuggest ion ĠProblem ) ĠGiven Ġa Ġset Ġof Ġnews Ġarticles ĠIN LINE FORM 0 Ġand Ġset Ġof ĠWikipedia Ġentity Ġpages ĠIN LINE FORM 1 Ġ( from ĠIN LINE FORM 2 Ġ) Ġwe Ġintend Ġto Ġsuggest Ġa Ġnews Ġarticle ĠIN LINE FORM 3 Ġpublished Ġat Ġtime ĠIN LINE FORM 4 Ġto Ġentity Ġpage ĠIN LINE FORM 5 Ġand Ġadditionally Ġto Ġthe Ġmost Ġrelevant Ġsection Ġfor Ġthe Ġentity Ġpage ĠIN LINE FORM 6 Ġ. </s> App roach ĠOverview </s> We Ġapproach Ġthe Ġnews Ġsuggestion Ġproblem Ġby Ġdecom posing Ġit Ġinto Ġtwo Ġtasks : </s> A EP : ĠArticle âĢĵ Entity Ġplacement </s> AS P : ĠArticle âĢĵ Section Ġplacement </s> In Ġthis Ġfirst Ġstep , Ġfor Ġa Ġgiven Ġentity - news Ġpair ĠIN LINE FORM 0 Ġ, Ġwe Ġdetermine Ġwhether Ġthe Ġgiven Ġnews Ġarticle ĠIN LINE FORM 1 Ġshould Ġbe Ġsuggested Ġ( we Ġwill Ġrefer Ġto Ġthis Ġas Ġ` relevant ') Ġto Ġentity ĠIN LINE FORM 2 Ġ. ĠTo Ġgenerate Ġsuch ĠIN LINE FORM 3 Ġpairs , Ġwe Ġperform Ġthe Ġentity Ġlinking Ġprocess , ĠIN LINE FORM 4 Ġ, Ġfor ĠIN LINE FORM 5 Ġ. </s> The Ġarticle âĢĵ entity Ġplacement Ġtask Ġ( described Ġin Ġdetail Ġin ĠSection ĠSEC REF 16 Ġ) Ġfor Ġa Ġpair ĠIN LINE FORM 0 Ġoutputs Ġa Ġbinary Ġlabel Ġ( either Ġ` non - relevant ' Ġor Ġ` relevant ') Ġand Ġis Ġformal ized Ġin ĠEqu ation ĠEQ REF 14 Ġ. ĠDIS PLAY FORM 0 </s> In Ġthe Ġsecond Ġstep , Ġwe Ġtake Ġinto Ġaccount Ġall Ġ` relevant ' Ġpairs ĠIN LINE FORM 0 Ġand Ġfind Ġthe Ġcorrect Ġsection Ġfor Ġarticle ĠIN LINE FORM 1 Ġin Ġentity ĠIN LINE FORM 2 Ġ, Ġrespectively Ġits Ġprofile ĠIN LINE FORM 3 Ġ( see ĠSection ĠSEC REF 30 Ġ). ĠThe Ġarticle âĢĵ section Ġplacement Ġtask , Ġdetermines Ġthe Ġcorrect Ġsection Ġfor Ġthe Ġtriple ĠIN LINE FORM 4 Ġ, Ġand Ġis Ġformal ized Ġin ĠEqu ation ĠEQ REF 15 Ġ. ĠDIS PLAY FORM 0 </s> In Ġthe Ġsubsequent Ġsections Ġwe Ġdescribe Ġin Ġdetails Ġhow Ġwe Ġapproach Ġthe Ġtwo Ġtasks Ġfor Ġsuggesting Ġnews Ġarticles Ġto Ġentity Ġpages . </s> News ĠArticle ĠSuggest ion </s> In Ġthis Ġsection , Ġwe Ġprovide Ġan Ġoverview Ġof Ġthe Ġnews Ġsuggestion Ġapproach Ġto ĠWikipedia Ġentity Ġpages Ġ( see ĠFigure ĠFIG REF 7 Ġ). ĠThe Ġapproach Ġis Ġsplit Ġinto Ġtwo Ġtasks : Ġ( i ) Ġarticle - entity Ġ( A EP ) Ġand Ġ( ii ) Ġarticle - section Ġ( AS P ) Ġplacement . ĠFor Ġa ĠWikipedia Ġsnapshot ĠIN LINE FORM 0 Ġand Ġa Ġnews Ġcorpus ĠIN LINE FORM 1 Ġ, Ġwe Ġfirst Ġdetermine Ġwhich Ġnews Ġarticles Ġshould Ġbe Ġsuggested Ġto Ġan Ġentity ĠIN LINE FORM 2 Ġ. ĠWe Ġwill Ġdenote Ġour Ġapproach Ġfor ĠA EP Ġby ĠIN LINE FORM 3 Ġ. ĠFinally , Ġwe Ġdetermine Ġthe Ġmost Ġappropriate Ġsection Ġfor Ġthe ĠASP Ġtask Ġand Ġwe Ġdenote Ġour Ġapproach Ġwith ĠIN LINE FORM 4 Ġ. </s> In Ġthe Ġfollowing , Ġwe Ġdescribe Ġthe Ġprocess Ġof Ġlearning Ġthe Ġfunctions ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠWe Ġintroduce Ġfeatures Ġfor Ġthe Ġlearning Ġprocess , Ġwhich Ġencode Ġinformation Ġregarding Ġthe Ġentity Ġsal ience , Ġrelative Ġauthority Ġand Ġnovelty Ġin Ġthe Ġcase Ġof ĠA EP Ġtask . ĠFor Ġthe ĠASP Ġtask , Ġwe Ġmeasure Ġthe Ġoverall Ġfit Ġof Ġan Ġarticle Ġto Ġthe Ġentity Ġsections , Ġwith Ġthe Ġentity Ġbeing Ġan Ġinput Ġfrom ĠA EP Ġtask . ĠAdditionally , Ġconsidering Ġthat Ġthe Ġentity Ġprofiles ĠIN LINE FORM 2 Ġare Ġincomplete , Ġin Ġthe Ġcase Ġof Ġa Ġmissing Ġsection Ġwe Ġsuggest Ġand Ġexpand Ġthe Ġentity Ġprofiles Ġbased Ġon Ġsection Ġtemplates Ġgenerated Ġfrom Ġentities Ġof Ġthe Ġsame Ġclass ĠIN LINE FORM 3 Ġ( see ĠSection ĠUID 34 Ġ). </s> Article âĢĵ Entity ĠPl acement </s> In Ġthis Ġstep Ġwe Ġlearn Ġthe Ġfunction ĠIN LINE FORM 0 Ġto Ġcorrectly Ġdetermine Ġwhether ĠIN LINE FORM 1 Ġshould Ġbe Ġsuggested Ġfor ĠIN LINE FORM 2 Ġ, Ġbasically Ġa Ġbinary Ġclassification Ġmodel Ġ( 0 = ` non - relevant ' Ġand Ġ1 = ` relevant '). ĠNote Ġthat Ġwe Ġare Ġmainly Ġinterested Ġin Ġfinding Ġthe Ġrelevant Ġpairs Ġin Ġthis Ġtask . ĠFor Ġevery Ġnews Ġarticle , Ġthe Ġnumber Ġof Ġdis amb ig uated Ġentities Ġis Ġaround Ġ30 Ġ( but ĠIN LINE FORM 3 Ġis Ġsuggested Ġfor Ġonly Ġtwo Ġof Ġthem Ġon Ġaverage ). ĠTherefore , Ġthe Ġdistribution Ġof Ġ` non - relevant ' Ġand Ġ` relevant ' Ġpairs Ġis Ġskewed Ġtowards Ġthe Ġearlier , Ġand Ġby Ġsimply Ġchoosing Ġthe Ġ` non - relevant ' Ġlabel Ġwe Ġcan Ġachieve Ġa Ġhigh Ġaccuracy Ġfor ĠIN LINE FORM 4 Ġ. ĠFinding Ġthe Ġrelevant Ġpairs Ġis Ġtherefore Ġa Ġconsiderable Ġchallenge . </s> An Ġarticle ĠIN LINE FORM 0 Ġis Ġsuggested Ġto ĠIN LINE FORM 1 Ġby Ġour Ġfunction ĠIN LINE FORM 2 Ġif Ġit Ġfulf ills Ġthe Ġfollowing Ġproperties . ĠThe Ġentity ĠIN LINE FORM 3 Ġis Ġsalient Ġin ĠIN LINE FORM 4 Ġ( a Ġcentral Ġconcept ), Ġtherefore Ġensuring Ġthat ĠIN LINE FORM 5 Ġis Ġabout ĠIN LINE FORM 6 Ġand Ġthat ĠIN LINE FORM 7 Ġis Ġimportant Ġfor ĠIN LINE FORM 8 Ġ. ĠNext , Ġgiven Ġthe Ġfact Ġthere Ġmight Ġbe Ġmany Ġarticles Ġin Ġwhich ĠIN LINE FORM 9 Ġis Ġsalient , Ġwe Ġalso Ġlook Ġat Ġthe Ġreverse Ġproperty , Ġnamely Ġwhether ĠIN LINE FORM 10 Ġis Ġimportant Ġfor ĠIN LINE FORM 11 Ġ. ĠWe Ġdo Ġthis Ġby Ġcomparing Ġthe Ġauthority Ġof ĠIN LINE FORM 12 Ġ( which Ġis Ġa Ġmeasure Ġof Ġpopularity Ġof Ġan Ġentity , Ġsuch Ġas Ġits Ġfrequency Ġof Ġmention Ġin Ġa Ġwhole Ġcorpus ) Ġwith Ġthe Ġauthority Ġof Ġits Ġco - occ urring Ġentities Ġin ĠIN LINE FORM 13 Ġ, Ġleading Ġto Ġa Ġfeature Ġwe Ġcall Ġrelative Ġauthority . ĠThe Ġintuition Ġis Ġthat Ġfor Ġan Ġentity Ġthat Ġhas Ġoverall Ġlower Ġauthority Ġthan Ġits Ġco - occ urring Ġentities , Ġa Ġnews Ġarticle Ġis Ġmore Ġeasily Ġof Ġimportance . ĠFinally , Ġif Ġthe Ġarticle Ġwe Ġare Ġabout Ġto Ġsuggest Ġis Ġalready Ġcovered Ġin Ġthe Ġentity Ġprofile ĠIN LINE FORM 14 Ġ, Ġwe Ġdo Ġnot Ġwish Ġto Ġsuggest Ġredundant Ġinformation , Ġhence Ġthe Ġnovelty . ĠTherefore , Ġthe Ġlearning Ġobjective Ġof ĠIN LINE FORM 15 Ġshould Ġfulfill Ġthe Ġfollowing Ġproperties . ĠTable ĠT AB REF 21 Ġshows Ġa Ġsummary Ġof Ġthe Ġcomputed Ġfeatures Ġfor ĠIN LINE FORM 16 Ġ. </s> Sal ience : Ġentity ĠIN LINE FORM 0 Ġshould Ġbe Ġa Ġsalient Ġentity Ġin Ġnews Ġarticle ĠIN LINE FORM 1 </s> Rel ative ĠAuthority : Ġthe Ġset Ġof Ġentities ĠIN LINE FORM 0 Ġwith Ġwhich ĠIN LINE FORM 1 Ġco - occ urs Ġshould Ġhave Ġhigher Ġauthority Ġthan ĠIN LINE FORM 2 Ġ, Ġmaking ĠIN LINE FORM 3 Ġimportant Ġfor ĠIN LINE FORM 4 </s> No vel ty : Ġnews Ġarticle ĠIN LINE FORM 0 Ġshould Ġprovide Ġnovel Ġinformation Ġfor Ġentity ĠIN LINE FORM 1 Ġtaking Ġinto Ġaccount Ġits Ġprofile ĠIN LINE FORM 2 </s> Bas eline ĠFeatures . ĠAs Ġdiscussed Ġin ĠSection ĠSEC REF 2 Ġ, Ġa Ġvariety Ġof Ġfeatures Ġthat Ġmeasure Ġsal ience Ġof Ġan Ġentity Ġin Ġtext Ġare Ġavailable Ġfrom Ġthe ĠN LP Ġcommunity . ĠWe Ġreim ple mented Ġthe Ġones Ġin ĠDun iet z Ġand ĠGill ick ĠB IB REF 11 Ġ. ĠThis Ġincludes Ġa Ġvariety Ġof Ġfeatures , Ġe . g . Ġpositional Ġfeatures , Ġoccurrence Ġfrequency Ġand Ġthe Ġinternal ĠPOS Ġstructure Ġof Ġthe Ġentity Ġand Ġthe Ġsentence Ġit Ġoccurs Ġin . ĠTable Ġ2 Ġin ĠB IB REF 11 Ġgives Ġdetails . </s> Rel ative ĠEntity ĠFrequency . ĠAlthough Ġfrequency Ġof Ġmention Ġand Ġpositional Ġfeatures Ġplay Ġsome Ġrole Ġin Ġbaseline Ġfeatures , Ġtheir Ġinteraction Ġis Ġnot Ġmodeled Ġby Ġa Ġsingle Ġfeature Ġnor Ġdo Ġthe Ġpositional Ġfeatures Ġencode Ġmore Ġthan Ġsentence Ġposition . ĠWe Ġtherefore Ġsuggest Ġa Ġnovel Ġfeature Ġcalled Ġrelative Ġentity Ġfrequency , ĠIN LINE FORM 0 Ġ, Ġthat Ġhas Ġthree Ġproperties .: Ġ( i ) ĠIt Ġrewards Ġentities Ġfor Ġoccurring Ġthroughout Ġthe Ġtext Ġinstead Ġof Ġonly Ġin Ġsome Ġparts Ġof Ġthe Ġtext , Ġmeasured Ġby Ġthe Ġnumber Ġof Ġparagraphs Ġit Ġoccurs Ġin Ġ( ii ) Ġit Ġrewards Ġentities Ġthat Ġoccur Ġmore Ġfrequently Ġin Ġthe Ġopening Ġparagraphs Ġof Ġan Ġarticle Ġas Ġwe Ġmodel ĠIN LINE FORM 1 Ġas Ġan Ġexponential Ġdecay Ġfunction . ĠThe Ġdecay Ġcorresponds Ġto Ġthe Ġpositional Ġindex Ġof Ġthe Ġnews Ġparagraph . ĠThis Ġis Ġinspired Ġby Ġthe Ġnews - specific Ġdiscourse Ġstructure Ġthat Ġtends Ġto Ġgive Ġshort Ġsumm aries Ġof Ġthe Ġmost Ġimportant Ġfacts Ġand Ġentities Ġin Ġthe Ġopening Ġparagraphs . Ġ( iii ) Ġit Ġcompares Ġentity Ġfrequency Ġto Ġthe Ġfrequency Ġof Ġits Ġco - occ urring Ġmentions Ġas Ġthe Ġweight Ġof Ġan Ġentity Ġappearing Ġin Ġa Ġspecific Ġparagraph , Ġnormalized Ġby Ġthe Ġsum Ġof Ġthe Ġfrequencies Ġof Ġother Ġentities Ġin ĠIN LINE FORM 2 Ġ. ĠDIS PLAY FORM 0 </s> where , ĠIN LINE FORM 0 Ġrepresents Ġa Ġnews Ġparagraph Ġfrom ĠIN LINE FORM 1 Ġ, Ġand Ġwith ĠIN LINE FORM 2 Ġwe Ġindicate Ġthe Ġset Ġof Ġall Ġparagraphs Ġin ĠIN LINE FORM 3 Ġ. ĠThe Ġfrequency Ġof ĠIN LINE FORM 4 Ġin Ġa Ġparagraph ĠIN LINE FORM 5 Ġis Ġden oted Ġby ĠIN LINE FORM 6 Ġ. ĠWith ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġwe Ġindicate Ġthe Ġnumber Ġof Ġparagraphs Ġin Ġwhich Ġentity ĠIN LINE FORM 9 Ġoccurs , Ġand Ġthe Ġtotal Ġnumber Ġof Ġparagraphs , Ġrespectively . </s> Rel ative ĠAuthority . ĠIn Ġthis Ġcase , Ġwe Ġconsider Ġthe Ġcomparative Ġrelevance Ġof Ġthe Ġnews Ġarticle Ġto Ġthe Ġdifferent Ġentities Ġoccurring Ġin Ġit . ĠAs Ġan Ġexample , Ġlet Ġus Ġconsider Ġthe Ġmeeting Ġof Ġthe ĠSudan ese Ġbishop ĠElias ĠTab an Ġwith ĠHillary ĠClinton . ĠBoth Ġentities Ġare Ġsalient Ġfor Ġthe Ġmeeting . ĠHowever , Ġin ĠTab an 's ĠWikipedia Ġpage , Ġthis Ġmeeting Ġis Ġdiscussed Ġprominently Ġwith Ġa Ġcorresponding Ġnews Ġreference , Ġwhereas Ġin ĠHillary ĠClinton 's ĠWikipedia Ġpage Ġit Ġis Ġnot Ġreported Ġat Ġall . ĠWe Ġbelieve Ġthis Ġis Ġnot Ġjust Ġan Ġomission Ġin ĠClinton 's Ġpage Ġbut Ġmirrors Ġthe Ġfact Ġthat Ġfor Ġthe Ġlesser Ġknown ĠTab an Ġthe Ġmeeting Ġis Ġbig Ġnews Ġwhereas Ġfor Ġthe Ġmore Ġfamous ĠClinton Ġthese Ġkind Ġof Ġmeetings Ġare Ġa Ġregular Ġoccurrence , Ġnot Ġall Ġof Ġwhich Ġcan Ġbe Ġreported Ġin Ġwhat Ġis Ġsupposed Ġto Ġbe Ġa Ġselection Ġof Ġthe Ġmost Ġimportant Ġevents Ġfor Ġher . ĠTherefore , Ġif Ġtwo Ġentities Ġco - occ ur , Ġthe Ġnews Ġis Ġmore Ġrelevant Ġfor Ġthe Ġentity Ġwith Ġthe Ġlower Ġa Ġprior i Ġauthority . </s> The Ġa Ġprior i Ġauthority Ġof Ġan Ġentity Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ) Ġcan Ġbe Ġmeasured Ġin Ġseveral Ġways . ĠWe Ġopt Ġfor Ġtwo Ġapproaches : Ġ( i ) Ġprobability Ġof Ġentity ĠIN LINE FORM 1 Ġoccurring Ġin Ġthe Ġcorpus ĠIN LINE FORM 2 Ġ, Ġand Ġ( ii ) Ġauthority Ġassessed Ġthrough Ġcentral ity Ġmeasures Ġlike ĠPage Rank ĠB IB REF 16 Ġ. ĠFor Ġthe Ġsecond Ġcase Ġwe Ġconstruct Ġthe Ġgraph ĠIN LINE FORM 3 Ġconsisting Ġof Ġentities Ġin ĠIN LINE FORM 4 Ġand Ġnews Ġarticles Ġin ĠIN LINE FORM 5 Ġas Ġvert ices . ĠThe Ġedges Ġare Ġestablished Ġbetween ĠIN LINE FORM 6 Ġand Ġentities Ġin ĠIN LINE FORM 7 Ġ, Ġthat Ġis ĠIN LINE FORM 8 Ġ, Ġand Ġthe Ġout - links Ġfrom ĠIN LINE FORM 9 Ġ, Ġthat Ġis ĠIN LINE FORM 10 Ġ( arrow s Ġpresent Ġthe Ġedge Ġdirection ). </s> Starting Ġfrom Ġa Ġprior i Ġauthority , Ġwe Ġproceed Ġto Ġrelative Ġauthority Ġby Ġcomparing Ġthe Ġa Ġprior i Ġauthority Ġof Ġco - occ urring Ġentities Ġin ĠIN LINE FORM 0 Ġ. ĠWe Ġdefine Ġthe Ġrelative Ġauthority Ġof ĠIN LINE FORM 1 Ġas Ġthe Ġproportion Ġof Ġco - occ urring Ġentities ĠIN LINE FORM 2 Ġthat Ġhave Ġa Ġhigher Ġa Ġprior i Ġauthority Ġthan ĠIN LINE FORM 3 Ġ( see ĠEqu ation ĠEQ REF 28 Ġ. ĠDIS PLAY FORM 0 </s> As Ġwe Ġmight Ġrun Ġthe Ġdanger Ġof Ġnot Ġsuggesting Ġany Ġnews Ġarticles Ġfor Ġentities Ġwith Ġvery Ġhigh Ġa Ġprior i Ġauthority Ġ( such Ġas ĠClinton ) Ġdue Ġto Ġthe Ġstrict Ġinequality Ġconstraint , Ġwe Ġcan Ġrelax Ġthe Ġconstraint Ġsuch Ġthat Ġthe Ġauthority Ġof Ġco - occ urring Ġentities Ġis Ġabove Ġa Ġcertain Ġthreshold . </s> News ĠDomain ĠAuthority . ĠThe Ġnews Ġdomain Ġauthority Ġaddresses Ġtwo Ġmain Ġaspects . ĠFirstly , Ġif Ġbundled Ġtogether Ġwith Ġthe Ġrelative Ġauthority Ġfeature , Ġwe Ġcan Ġensure Ġthat Ġdependent Ġon Ġthe Ġentity Ġauthority , Ġwe Ġsuggest Ġnews Ġfrom Ġauthoritative Ġsources , Ġhence Ġensuring Ġthe Ġquality Ġof Ġsuggested Ġarticles . ĠThe Ġsecond Ġaspect Ġis Ġin Ġa Ġnews Ġstreaming Ġscenario Ġwhere Ġmultiple Ġnews Ġdomains Ġreport Ġthe Ġsame Ġevent ĠâĢĶ Ġideally Ġonly Ġarticles Ġcoming Ġfrom Ġauthoritative Ġsources Ġwould Ġfulfill Ġthe Ġconditions Ġfor Ġthe Ġnews Ġsuggestion Ġtask . </s> The Ġnews Ġdomain Ġauthority Ġis Ġcomputed Ġbased Ġon Ġthe Ġnumber Ġof Ġnews Ġreferences Ġin ĠWikipedia Ġcoming Ġfrom Ġa Ġparticular Ġnews Ġdomain ĠIN LINE FORM 0 Ġ. ĠThis Ġrepresents Ġa Ġsimple Ġprior Ġthat Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġis Ġfrom Ġdomain ĠIN LINE FORM 2 Ġin Ġcorpus ĠIN LINE FORM 3 Ġ. ĠWe Ġextract Ġthe Ġdomains Ġby Ġtaking Ġthe Ġbase ĠURLs Ġfrom Ġthe Ġnews Ġarticle ĠURLs . </s> An Ġimportant Ġfeature Ġwhen Ġsuggesting Ġan Ġarticle ĠIN LINE FORM 0 Ġto Ġan Ġentity ĠIN LINE FORM 1 Ġis Ġthe Ġnovelty Ġof ĠIN LINE FORM 2 Ġw . r . t Ġthe Ġalready Ġexisting Ġentity Ġprofile ĠIN LINE FORM 3 Ġ. ĠStudies ĠB IB REF 17 Ġhave Ġshown Ġthat Ġon Ġcomparable Ġcollections Ġto Ġours Ġ( T REC ĠG OV 2 ) Ġthe Ġnumber Ġof Ġdupl icates Ġcan Ġgo Ġup Ġto ĠIN LINE FORM 4 Ġ. ĠThis Ġfigure Ġis Ġlikely Ġhigher Ġfor Ġmajor Ġevents Ġconcerning Ġhighly Ġauthoritative Ġentities Ġon Ġwhich Ġall Ġnews Ġmedia Ġwill Ġreport . </s> Given Ġan Ġentity ĠIN LINE FORM 0 Ġand Ġthe Ġalready Ġadded Ġnews Ġreferences ĠIN LINE FORM 1 Ġup Ġto Ġyear ĠIN LINE FORM 2 Ġ, Ġthe Ġnovelty Ġof ĠIN LINE FORM 3 Ġat Ġyear ĠIN LINE FORM 4 Ġis Ġmeasured Ġby Ġthe ĠKL Ġdivergence Ġbetween Ġthe Ġlanguage Ġmodel Ġof ĠIN LINE FORM 5 Ġand Ġarticles Ġin ĠIN LINE FORM 6 Ġ. ĠWe Ġcombine Ġthis Ġmeasure Ġwith Ġthe Ġentity Ġoverlap Ġof ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġ. ĠThe Ġnovelty Ġvalue Ġof ĠIN LINE FORM 9 Ġis Ġgiven Ġby Ġthe Ġminimal Ġdivergence Ġvalue . ĠLow Ġscores Ġindicate Ġlow Ġnovelty Ġfor Ġthe Ġentity Ġprofile ĠIN LINE FORM 10 Ġ. </s> N ( n | e ) Ġ= Ġn ' N t - 1 { DK L (( n ') Ġ|| Ġ( n )) Ġ+ ĠDK L (( N ) Ġ|| Ġ( n )). </s> DK L (( n ') Ġ|| Ġ( n )). Ġ( 1 -) Ġj acc ard (( n '), ( n )) } Ġwhere ĠIN LINE FORM 0 Ġis Ġthe ĠKL Ġdivergence Ġof Ġthe Ġlanguage Ġmodels Ġ( ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ), Ġwhereas ĠIN LINE FORM 3 Ġis Ġthe Ġmixing Ġweight Ġ( ĠIN LINE FORM 4 Ġ) Ġbetween Ġthe Ġlanguage Ġmodels ĠIN LINE FORM 5 Ġand Ġthe Ġentity Ġoverlap Ġin ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. </s> Here Ġwe Ġintroduce Ġthe Ġevaluation Ġsetup Ġand Ġanalyze Ġthe Ġresults Ġfor Ġthe Ġarticle âĢĵ entity Ġ( A EP ) Ġplacement Ġtask . ĠWe Ġonly Ġreport Ġthe Ġevaluation Ġmetrics Ġfor Ġthe Ġ` relevant ' Ġnews - entity Ġpairs . ĠA Ġdetailed Ġexplanation Ġon Ġwhy Ġwe Ġfocus Ġon Ġthe Ġ` relevant ' Ġpairs Ġis Ġprovided Ġin ĠSection ĠSEC REF 16 Ġ. </s> Bas elines . ĠWe Ġconsider Ġthe Ġfollowing Ġbas elines Ġfor Ġthis Ġtask . </s> B 1 . ĠThe Ġfirst Ġbaseline Ġuses Ġonly Ġthe Ġsal ience - based Ġfeatures Ġby ĠDun iet z Ġand ĠGill ick ĠB IB REF 11 Ġ. </s> B 2 . ĠThe Ġsecond Ġbaseline Ġassigns Ġthe Ġvalue Ġrelevant Ġto Ġa Ġpair ĠIN LINE FORM 0 Ġ, Ġif Ġand Ġonly Ġif ĠIN LINE FORM 1 Ġappears Ġin Ġthe Ġtitle Ġof ĠIN LINE FORM 2 Ġ. </s> Learning ĠModels . ĠWe Ġuse ĠRandom ĠFore sts Ġ( RF ) ĠB IB REF 23 Ġ. ĠWe Ġlearn Ġthe ĠRF Ġon Ġall Ġcomputed Ġfeatures Ġin ĠTable ĠT AB REF 21 Ġ. ĠThe Ġoptimization Ġon ĠRF Ġis Ġdone Ġby Ġsplitting Ġthe Ġfeature Ġspace Ġinto Ġmultiple Ġtrees Ġthat Ġare Ġconsidered Ġas Ġensemble Ġclass ifiers . ĠConsequently , Ġfor Ġeach Ġclass ifier Ġit Ġcomp utes Ġthe Ġmargin Ġfunction Ġas Ġa Ġmeasure Ġof Ġthe Ġaverage Ġcount Ġof Ġpredicting Ġthe Ġcorrect Ġclass Ġin Ġcontrast Ġto Ġany Ġother Ġclass . ĠThe Ġhigher Ġthe Ġmargin Ġscore Ġthe Ġmore Ġrobust Ġthe Ġmodel . </s> Met rics . ĠWe Ġcompute Ġprecision ĠP , Ġrecall ĠR Ġand ĠF 1 Ġscore Ġfor Ġthe Ġrelevant Ġclass . ĠFor Ġexample , Ġprecision Ġis Ġthe Ġnumber Ġof Ġnews - entity Ġpairs Ġwe Ġcorrectly Ġlabeled Ġas Ġrelevant Ġcompared Ġto Ġour Ġground Ġtruth Ġdivided Ġby Ġthe Ġnumber Ġof Ġall Ġnews - entity Ġpairs Ġwe Ġlabeled Ġas Ġrelevant . </s> The Ġfollowing Ġresults Ġmeasure Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġin Ġthree Ġmain Ġaspects : Ġ( i ) Ġoverall Ġperformance Ġof ĠIN LINE FORM 0 Ġand Ġcomparison Ġto Ġbas elines , Ġ( ii ) Ġrobust ness Ġacross Ġthe Ġyears , Ġand Ġ( iii ) Ġoptimal Ġmodel Ġfor Ġthe ĠA EP Ġplacement Ġtask . </s> Performance . ĠFigure ĠFIG REF 55 Ġshows Ġthe Ġresults Ġfor Ġthe Ġyears Ġ2009 Ġand Ġ2013 , Ġwhere Ġwe Ġoptimized Ġthe Ġlearning Ġobjective Ġwith Ġinstances Ġfrom Ġyear ĠIN LINE FORM 0 Ġand Ġevaluate Ġon Ġthe Ġyears ĠIN LINE FORM 1 Ġ( see ĠSection ĠSEC REF 46 Ġ). ĠThe Ġresults Ġshow Ġthe Ġprecision âĢĵ rec all Ġcurve . ĠThe Ġred Ġcurve Ġshows Ġbaseline ĠB 1 ĠB IB REF 11 Ġ, Ġand Ġthe Ġblue Ġone Ġshows Ġthe Ġperformance Ġof ĠIN LINE FORM 2 Ġ. ĠThe Ġcurve Ġshows Ġfor Ġvarying Ġconfidence Ġscores Ġ( high Ġto Ġlow ) Ġthe Ġprecision Ġon Ġlabeling Ġthe Ġpair ĠIN LINE FORM 3 Ġas Ġ` relevant '. ĠIn Ġaddition , Ġat Ġeach Ġconfidence Ġscore Ġwe Ġcan Ġcompute Ġthe Ġcorresponding Ġrecall Ġfor Ġthe Ġ` relevant ' Ġlabel . ĠFor Ġhigh Ġconfidence Ġscores Ġon Ġlabeling Ġthe Ġnews - entity Ġpairs , Ġthe Ġbaseline ĠB 1 Ġachieves Ġon Ġaverage Ġa Ġprecision Ġscore Ġof ĠP = 0 . 50 , Ġwhile ĠIN LINE FORM 4 Ġhas ĠP = 0 . 93 . ĠWe Ġnote Ġthat Ġwith Ġthe Ġdrop Ġin Ġthe Ġconfidence Ġscore Ġthe Ġcorresponding Ġprecision Ġand Ġrecall Ġvalues Ġdrop Ġtoo , Ġand Ġthe Ġoverall ĠF 1 Ġscore Ġfor ĠB 1 Ġis Ġaround ĠF 1 = 0 . 2 , Ġin Ġcontrast Ġwe Ġachieve Ġan Ġaverage Ġscore Ġof ĠF 1 = 0 . 67 . </s> It Ġis Ġevident Ġfrom ĠFigure ĠFIG REF 55 Ġthat Ġfor Ġthe Ġyears Ġ2009 Ġand Ġ2013 , ĠIN LINE FORM 0 Ġsignificantly Ġoutper forms Ġthe Ġbaseline ĠB 1 . ĠWe Ġmeasure Ġthe Ġsignificance Ġthrough Ġthe Ġt - test Ġstatistic Ġand Ġget Ġa Ġp - value Ġof ĠIN LINE FORM 1 Ġ. ĠThe Ġimprovement Ġwe Ġachieve Ġover ĠB 1 Ġin Ġabsolute Ġnumbers , ĠIN LINE FORM 2 ĠP = + 0 . 5 Ġin Ġterms Ġof Ġprecision Ġfor Ġthe Ġyears Ġbetween Ġ2009 Ġand Ġ2014 , Ġand Ġa Ġsimilar Ġimprovement Ġin Ġterms Ġof ĠF 1 Ġscore . ĠThe Ġimprovement Ġfor Ġrecall Ġis ĠIN LINE FORM 3 ĠR = + 0 . 4 . ĠThe Ġrelative Ġimprovement Ġover ĠB 1 Ġfor ĠP Ġand ĠF 1 Ġis Ġalmost Ġ1 . 8 Ġtimes Ġbetter , Ġwhile Ġfor Ġrecall Ġwe Ġare Ġ3 . 5 Ġtimes Ġbetter . ĠIn ĠTable ĠT AB REF 58 Ġwe Ġshow Ġthe Ġoverall Ġscores Ġfor Ġthe Ġevaluation Ġmetrics Ġfor ĠB 1 Ġand ĠIN LINE FORM 4 Ġ. ĠFinally , Ġfor ĠB 2 Ġwe Ġachieve Ġmuch Ġpoorer Ġperformance , Ġwith Ġaverage Ġscores Ġof ĠP = 0 . 21 , ĠR = 0 . 20 Ġand ĠF 1 = 0 . 21 . </s> Rob ust ness . ĠIn ĠTable ĠT AB REF 58 Ġ, Ġwe Ġshow Ġthe Ġoverall Ġperformance Ġfor Ġthe Ġyears Ġbetween Ġ2009 Ġand Ġ2013 . ĠAn Ġinteresting Ġobservation Ġwe Ġmake Ġis Ġthat Ġwe Ġhave Ġa Ġvery Ġrobust Ġperformance Ġand Ġthe Ġresults Ġare Ġstable Ġacross Ġthe Ġyears . ĠIf Ġwe Ġconsider Ġthe Ġexperimental Ġsetup , Ġwhere Ġfor Ġyear ĠIN LINE FORM 0 Ġwe Ġoptimize Ġthe Ġlearning Ġobjective Ġwith Ġonly Ġ74 k Ġtraining Ġinstances Ġand Ġevaluate Ġon Ġthe Ġrest Ġof Ġthe Ġinstances , Ġit Ġachieves Ġa Ġvery Ġgood Ġperformance . ĠWe Ġpredict Ġwith ĠF 1 = 0 . 68 Ġthe Ġremaining Ġ4 69 k Ġinstances Ġfor Ġthe Ġyears ĠIN LINE FORM 1 Ġ. </s> The Ġresults Ġare Ġparticularly Ġpromising Ġconsidering Ġthe Ġfact Ġthat Ġthe Ġdistribution Ġbetween Ġour Ġtwo Ġclasses Ġis Ġhighly Ġskewed . ĠOn Ġaverage Ġthe Ġnumber Ġof Ġ` relevant ' Ġpairs Ġaccount Ġfor Ġonly Ġaround ĠIN LINE FORM 0 Ġof Ġall Ġpairs . ĠA Ġgood Ġindicator Ġto Ġsupport Ġsuch Ġa Ġstatement Ġis Ġthe Ġk appa Ġ( den oted Ġby ĠIN LINE FORM 1 Ġ) Ġstatistic . ĠIN LINE FORM 2 Ġmeasures Ġagreement Ġbetween Ġthe Ġalgorithm Ġand Ġthe Ġgold Ġstandard Ġon Ġboth Ġlabels Ġwhile Ġcorrecting Ġfor Ġchance Ġagreement Ġ( often Ġexpected Ġdue Ġto Ġextreme Ġdistributions ). ĠThe ĠIN LINE FORM 3 Ġscores Ġfor ĠB 1 Ġacross Ġthe Ġyears Ġis Ġon Ġaverage ĠIN LINE FORM 4 Ġ, Ġwhile Ġfor ĠIN LINE FORM 5 Ġwe Ġachieve Ġa Ġscore Ġof ĠIN LINE FORM 6 Ġ( the Ġmaximum Ġscore Ġfor ĠIN LINE FORM 7 Ġis Ġ1 ). </s> In ĠFigure ĠFIG REF 60 Ġwe Ġshow Ġthe Ġimpact Ġof Ġthe Ġindividual Ġfeature Ġgroups Ġthat Ġcontribute Ġto Ġthe Ġsuperior Ġperformance Ġin Ġcomparison Ġto Ġthe Ġbas elines . ĠRelative Ġentity Ġfrequency Ġfrom Ġthe Ġsal ience Ġfeature , Ġmodels Ġthe Ġentity Ġsal ience Ġas Ġan Ġexponentially Ġdecaying Ġfunction Ġbased Ġon Ġthe Ġpositional Ġindex Ġof Ġthe Ġparagraph Ġwhere Ġthe Ġentity Ġappears . ĠThe Ġperformance Ġof ĠIN LINE FORM 0 Ġwith Ġrelative Ġentity Ġfrequency Ġfrom Ġthe Ġsal ience Ġfeature Ġgroup Ġis Ġclose Ġto Ġthat Ġof Ġall Ġthe Ġfeatures Ġcombined . ĠThe Ġauthority Ġand Ġnovelty Ġfeatures Ġaccount Ġto Ġa Ġfurther Ġimprovement Ġin Ġterms Ġof Ġprecision , Ġby Ġadding Ġroughly Ġa Ġ7 %- 10 % Ġincrease . ĠHowever , Ġif Ġboth Ġfeature Ġgroups Ġare Ġconsidered Ġseparately , Ġthey Ġsignificantly Ġoutper form Ġthe Ġbaseline ĠB 1 . </s> Article âĢĵ Section ĠPl acement </s> We Ġmodel Ġthe ĠASP Ġplacement Ġtask Ġas Ġa Ġsuccessor Ġof Ġthe ĠA EP Ġtask . ĠFor Ġall Ġthe Ġ` relevant ' Ġnews Ġentity Ġpairs , Ġthe Ġtask Ġis Ġto Ġdetermine Ġthe Ġcorrect Ġentity Ġsection . ĠEach Ġsection Ġin Ġa ĠWikipedia Ġentity Ġpage Ġrepresents Ġa Ġdifferent Ġtopic . ĠFor Ġexample , ĠBarack ĠObama Ġhas Ġthe Ġsections Ġ` Early ĠLife ', Ġ` Pres idency ', Ġ` Family Ġand ĠPersonal ĠLife ' Ġetc . ĠHowever , Ġmany Ġentity Ġpages Ġhave Ġan Ġincomplete Ġsection Ġstructure . ĠIn complete Ġor Ġmissing Ġsections Ġare Ġdue Ġto Ġtwo ĠWikipedia Ġproperties . ĠFirst , Ġlong - tail Ġentities Ġmiss Ġinformation Ġand Ġsections Ġdue Ġto Ġtheir Ġlack Ġof Ġpopularity . ĠSecond , Ġfor Ġall Ġentities Ġwhether Ġpopular Ġor Ġnot , Ġcertain Ġsections Ġmight Ġoccur Ġfor Ġthe Ġfirst Ġtime Ġdue Ġto Ġreal Ġworld Ġdevelopments . ĠAs Ġan Ġexample , Ġthe Ġentity ĠGerman wings Ġdid Ġnot Ġhave Ġan Ġ` Acc idents ' Ġsection Ġbefore Ġthis Ġyear 's Ġdisaster , Ġwhich Ġwas Ġthe Ġfirst Ġin Ġthe Ġhistory Ġof Ġthe Ġairline . </s> Even Ġif Ġsections Ġare Ġmissing Ġfor Ġcertain Ġentities , Ġsimilar Ġsections Ġusually Ġoccur Ġin Ġother Ġentities Ġof Ġthe Ġsame Ġclass Ġ( e . g . Ġother Ġairlines Ġhad Ġdisasters Ġand Ġtherefore Ġtheir Ġpages Ġhave Ġan Ġaccidents Ġsection ). ĠWe Ġexploit Ġsuch Ġhom ogeneity Ġof Ġsection Ġstructure Ġand Ġconstruct Ġtemplates Ġthat Ġwe Ġuse Ġto Ġexpand Ġentity Ġprofiles . ĠThe Ġlearning Ġobjective Ġfor ĠIN LINE FORM 0 Ġtakes Ġinto Ġaccount Ġthe Ġfollowing Ġproperties : </s> Section - tem plates : Ġaccount Ġfor Ġincomplete Ġsection Ġstructure Ġfor Ġan Ġentity Ġprofile ĠIN LINE FORM 0 Ġby Ġconstructing Ġsection Ġtemplates ĠIN LINE FORM 1 Ġfrom Ġan Ġentity Ġclass ĠIN LINE FORM 2 </s> Overall Ġfit : Ġmeasures Ġthe Ġoverall Ġfit Ġof Ġa Ġnews Ġarticle Ġto Ġsections Ġin Ġthe Ġsection Ġtemplates ĠIN LINE FORM 0 </s> Given Ġthe Ġfact Ġthat Ġentity Ġprofiles Ġare Ġoften Ġincomplete , Ġwe Ġconstruct Ġsection Ġtemplates Ġfor Ġevery Ġentity Ġclass . ĠWe Ġgroup Ġentities Ġbased Ġon Ġtheir Ġclass ĠIN LINE FORM 0 Ġand Ġconstruct Ġsection Ġtemplates ĠIN LINE FORM 1 Ġ. ĠFor Ġdifferent Ġentity Ġclasses , Ġe . g . ĠPerson Ġand ĠLocation , Ġthe Ġsection Ġstructure Ġand Ġthe Ġinformation Ġrepresented Ġin Ġthose Ġsection Ġvaries Ġheavily . ĠTherefore , Ġthe Ġsection Ġtemplates Ġare Ġwith Ġrespect Ġto Ġthe Ġindividual Ġclasses Ġin Ġour Ġexperimental Ġsetup Ġ( see ĠFigure ĠFIG REF 42 Ġ). ĠDIS PLAY FORM 0 </s> Gener ating Ġsection Ġtemplates Ġhas Ġtwo Ġmain Ġadvantages . ĠFirstly , Ġby Ġconsidering Ġclass - based Ġprofiles , Ġwe Ġcan Ġovercome Ġthe Ġproblem Ġof Ġincomplete Ġindividual Ġentity Ġprofiles Ġand Ġthereby Ġare Ġable Ġto Ġsuggest Ġnews Ġarticles Ġto Ġsections Ġthat Ġdo Ġnot Ġyet Ġexist Ġin Ġa Ġspecific Ġentity ĠIN LINE FORM 0 Ġ. ĠThe Ġsecond Ġadvantage Ġis Ġthat Ġwe Ġare Ġable Ġto Ġcanonical ize Ġthe Ġsections , Ġi . e . Ġ` Early ĠLife ' Ġand Ġ` Early ĠLife Ġand ĠChildhood ' Ġwould Ġbe Ġtreated Ġsimilarly . </s> To Ġgenerate Ġthe Ġsection Ġtemplate ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġall Ġsections Ġfrom Ġentities Ġof Ġa Ġgiven Ġtype ĠIN LINE FORM 1 Ġat Ġyear ĠIN LINE FORM 2 Ġ. ĠNext , Ġwe Ġcluster Ġthe Ġentity Ġsections , Ġbased Ġon Ġan Ġextended Ġversion Ġof Ġk âĢĵ me ans Ġclust ering ĠB IB REF 18 Ġ, Ġnamely Ġx âĢĵ me ans Ġclust ering Ġintroduced Ġin ĠPel leg Ġet Ġal . Ġwhich Ġestimates Ġthe Ġnumber Ġof Ġclusters Ġefficiently ĠB IB REF 19 Ġ. ĠAs Ġa Ġsimilarity Ġmetric Ġwe Ġuse Ġthe Ġcos ine Ġsimilarity Ġcomputed Ġbased Ġon Ġthe Ġtf âĢĵ id f Ġmodels Ġof Ġthe Ġsections . ĠUsing Ġthe Ġx âĢĵ me ans Ġalgorithm Ġwe Ġovercome Ġthe Ġrequirement Ġto Ġprovide Ġthe Ġnumber Ġof Ġclusters Ġk Ġbeforehand . Ġx âĢĵ me ans Ġextends Ġthe Ġk âĢĵ me ans Ġalgorithm , Ġsuch Ġthat Ġa Ġuser Ġonly Ġspecifies Ġa Ġrange Ġ[ ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġ] Ġthat Ġthe Ġnumber Ġof Ġclusters Ġmay Ġreasonably Ġlie Ġin . </s> The Ġlearning Ġobjective Ġof ĠIN LINE FORM 0 Ġis Ġto Ġdetermine Ġthe Ġoverall Ġfit Ġof Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġto Ġone Ġof Ġthe Ġsections Ġin Ġa Ġgiven Ġsection Ġtemplate ĠIN LINE FORM 2 Ġ. ĠThe Ġtemplate Ġis Ġpre - d etermined Ġby Ġthe Ġclass Ġof Ġthe Ġentity Ġfor Ġwhich Ġthe Ġnews Ġis Ġsuggested Ġas Ġrelevant Ġby ĠIN LINE FORM 3 Ġ. ĠIn Ġall Ġcases , Ġwe Ġmeasure Ġhow Ġwell ĠIN LINE FORM 4 Ġfits Ġeach Ġof Ġthe Ġsections ĠIN LINE FORM 5 Ġas Ġwell Ġas Ġthe Ġspecific Ġentity Ġsection ĠIN LINE FORM 6 Ġ. ĠThe Ġsection Ġprofiles Ġin ĠIN LINE FORM 7 Ġrepresent Ġthe Ġaggreg ated Ġentity Ġprofiles Ġfrom Ġall Ġentities Ġof Ġclass ĠIN LINE FORM 8 Ġat Ġyear ĠIN LINE FORM 9 Ġ. </s> To Ġlearn ĠIN LINE FORM 0 Ġwe Ġrely Ġon Ġa Ġvariety Ġof Ġfeatures Ġthat Ġconsider Ġseveral Ġsimilarity Ġaspects Ġas Ġshown Ġin ĠTable ĠT AB REF 31 Ġ. ĠFor Ġthe Ġsake Ġof Ġsimplicity Ġwe Ġdo Ġnot Ġmake Ġthe Ġdistinction Ġin ĠTable ĠT AB REF 31 Ġbetween Ġthe Ġindividual Ġentity Ġsection Ġand Ġclass - based Ġsection Ġsimilarities , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ, Ġrespectively . ĠBear Ġin Ġmind Ġthat Ġan Ġentity Ġsection ĠIN LINE FORM 3 Ġmight Ġbe Ġpresent Ġat Ġyear ĠIN LINE FORM 4 Ġbut Ġnot Ġat Ġyear ĠIN LINE FORM 5 Ġ( see Ġfor Ġmore Ġdetails Ġthe Ġdiscussion Ġon Ġentity Ġprofile Ġexpansion Ġin ĠSection ĠUID 69 Ġ). </s> Topic . ĠWe Ġuse Ġtopic Ġsimilarities Ġto Ġensure Ġ( i ) Ġthat Ġthe Ġcontent Ġof ĠIN LINE FORM 0 Ġfits Ġtopic - wise Ġwith Ġa Ġspecific Ġsection Ġtext Ġand Ġ( ii ) Ġthat Ġit Ġhas Ġa Ġsimilar Ġtopic Ġto Ġpreviously Ġreferred Ġnews Ġarticles Ġin Ġthat Ġsection . ĠIn Ġa Ġpre - processing Ġstage Ġwe Ġcompute Ġthe Ġtopic Ġmodels Ġfor Ġthe Ġnews Ġarticles , Ġentity Ġsections ĠIN LINE FORM 1 Ġand Ġthe Ġaggreg ated Ġclass - based Ġsections Ġin ĠIN LINE FORM 2 Ġ. ĠThe Ġtopic Ġmodels Ġare Ġcomputed Ġusing ĠL DA ĠB IB REF 20 Ġ. ĠWe Ġonly Ġcomputed Ġa Ġsingle Ġtopic Ġper Ġarticle / section Ġas Ġwe Ġare Ġonly Ġinterested Ġin Ġtopic Ġterm Ġoverl aps Ġbetween Ġarticle Ġand Ġsections . ĠWe Ġdistinguish Ġtwo Ġmain Ġfeatures : Ġthe Ġfirst Ġfeature Ġmeasures Ġthe Ġoverlap Ġof Ġtopic Ġterms Ġbetween ĠIN LINE FORM 3 Ġand Ġthe Ġentity Ġsection ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ, Ġand Ġthe Ġsecond Ġfeature Ġmeasures Ġthe Ġoverlap Ġof Ġthe Ġtopic Ġmodel Ġof ĠIN LINE FORM 6 Ġagainst Ġreferred Ġnews Ġarticles Ġin ĠIN LINE FORM 7 Ġat Ġtime ĠIN LINE FORM 8 Ġ. </s> Sy nt actic . ĠThese Ġfeatures Ġrepresent Ġa Ġmechanism Ġfor Ġconve ying Ġthe Ġimportance Ġof Ġa Ġspecific Ġtext Ġsnippet , Ġsolely Ġbased Ġon Ġthe Ġfrequency Ġof Ġspecific ĠPOS Ġtags Ġ( i . e . ĠN NP , ĠCD Ġetc .), Ġas Ġcommonly Ġused Ġin Ġtext Ġsummar ization Ġtasks . ĠFollowing Ġthe Ġsame Ġintuition Ġas Ġin ĠB IB REF 8 Ġ, Ġwe Ġweigh Ġthe Ġimportance Ġof Ġarticles Ġby Ġthe Ġcount Ġof Ġspecific ĠPOS Ġtags . ĠWe Ġexpect Ġthat Ġfor Ġdifferent Ġsections , Ġthe Ġimportance Ġof ĠPOS Ġtags Ġwill Ġvary . ĠWe Ġmeasure Ġthe Ġsimilarity Ġof ĠPOS Ġtags Ġin Ġa Ġnews Ġarticle Ġagainst Ġthe Ġsection Ġtext . ĠAdditionally , Ġwe Ġconsider Ġbi - gram Ġand Ġtri - gram ĠPOS Ġtag Ġoverlap . ĠThis Ġexploits Ġsimilarity Ġin Ġsynt act ical Ġpatterns Ġbetween Ġthe Ġnews Ġand Ġsection Ġtext . </s> Lex ical . ĠAs Ġlex ical Ġfeatures , Ġwe Ġmeasure Ġthe Ġsimilarity Ġof ĠIN LINE FORM 0 Ġagainst Ġthe Ġentity Ġsection Ġtext ĠIN LINE FORM 1 Ġand Ġthe Ġaggregate Ġsection Ġtext ĠIN LINE FORM 2 Ġ. ĠFurther , Ġwe Ġdistinguish Ġbetween Ġthe Ġoverall Ġsimilarity Ġof ĠIN LINE FORM 3 Ġand Ġthat Ġof Ġthe Ġdifferent Ġnews Ġparagraphs Ġ( ĠIN LINE FORM 4 Ġwhich Ġdenotes Ġthe Ġparagraphs Ġof ĠIN LINE FORM 5 Ġup Ġto Ġthe Ġ5 th Ġparagraph ). ĠA Ġhigher Ġsimilarity Ġon Ġthe Ġfirst Ġparagraphs Ġrepresents Ġa Ġmore Ġconfident Ġindicator Ġthat ĠIN LINE FORM 6 Ġshould Ġbe Ġsuggested Ġto Ġa Ġspecific Ġsection ĠIN LINE FORM 7 Ġ. ĠWe Ġmeasure Ġthe Ġsimilarity Ġbased Ġon Ġtwo Ġmetrics : Ġ( i ) Ġthe ĠKL - d iver gence Ġbetween Ġthe Ġcomputed Ġlanguage Ġmodels Ġand Ġ( ii ) Ġcos ine Ġsimilarity Ġof Ġthe Ġcorresponding Ġparagraph Ġtext ĠIN LINE FORM 8 Ġand Ġsection Ġtext . </s> Entity - based . ĠAnother Ġfeature Ġset Ġwe Ġconsider Ġis Ġthe Ġoverlap Ġof Ġnamed Ġentities Ġand Ġtheir Ġcorresponding Ġentity Ġclasses . ĠFor Ġdifferent Ġentity Ġsections , Ġwe Ġexpect Ġto Ġfind Ġa Ġparticular Ġset Ġof Ġentity Ġclasses Ġthat Ġwill Ġcorrelate Ġwith Ġthe Ġsection , Ġe . g . Ġ` Early ĠLife ' Ġcontains Ġmostly Ġentities Ġrelated Ġto Ġfamily , Ġschool , Ġuniversities Ġetc . </s> F requency . ĠFinally , Ġwe Ġgather Ġstatistics Ġabout Ġthe Ġnumber Ġof Ġentities , Ġparagraphs , Ġnews Ġarticle Ġlength , Ġtop âĢĵ ĠIN LINE FORM 0 Ġentities Ġand Ġentity Ġclasses , Ġand Ġthe Ġfrequency Ġof Ġdifferent ĠPOS Ġtags . ĠHere Ġwe Ġtry Ġto Ġcapture Ġpatterns Ġof Ġarticles Ġthat Ġare Ġusually Ġcited Ġin Ġspecific Ġsections . </s> E val uation ĠPlan </s> In Ġthis Ġsection Ġwe Ġoutline Ġthe Ġevaluation Ġplan Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġour Ġlearning Ġapproaches . ĠTo Ġevaluate Ġthe Ġnews Ġsuggestion Ġproblem Ġwe Ġare Ġfaced Ġwith Ġtwo Ġchallenges . </s> What Ġcomprises Ġthe Ġground Ġtruth Ġfor Ġsuch Ġa Ġtask Ġ? </s> How Ġdo Ġwe Ġconstruct Ġtraining Ġand Ġtest Ġsplits Ġgiven Ġthat Ġentity Ġpages Ġconsists Ġof Ġtext Ġadded Ġat Ġdifferent Ġpoints Ġin Ġtime Ġ? </s> Consider Ġthe Ġground Ġtruth Ġchallenge . ĠEval uating Ġif Ġan Ġarbitrary Ġnews Ġarticle Ġshould Ġbe Ġincluded Ġin ĠWikipedia Ġis Ġboth Ġsubjective Ġand Ġdifficult Ġfor Ġa Ġhuman Ġif Ġshe Ġis Ġnot Ġan Ġexpert . ĠAn Ġinvasive Ġapproach , Ġwhich Ġwas Ġproposed Ġby ĠBar z il ay Ġand ĠSau per ĠB IB REF 8 Ġ, Ġadds Ġcontent Ġdirectly Ġto ĠWikipedia Ġand Ġexpects Ġthe Ġeditors Ġor Ġother Ġusers Ġto Ġred act Ġirrelevant Ġcontent Ġover Ġa Ġperiod Ġof Ġtime . ĠThe Ġlimitations Ġof Ġsuch Ġan Ġevaluation Ġtechnique Ġis Ġthat Ġcontent Ġadded Ġto Ġlong - tail Ġentities Ġmight Ġnot Ġbe Ġevaluated Ġby Ġinformed Ġusers Ġor Ġeditors Ġin Ġthe Ġexperiment Ġtime Ġframe . ĠIt Ġis Ġhard Ġto Ġestimate Ġhow Ġmuch Ġtime Ġthe Ġadded Ġcontent Ġshould Ġbe Ġleft Ġon Ġthe Ġentity Ġpage . ĠA Ġmore Ġnon - in vasive Ġapproach Ġcould Ġinvolve Ġcrowds ourcing Ġof Ġentity Ġand Ġnews Ġarticle Ġpairs Ġin Ġan ĠIR Ġstyle Ġrelevance Ġassessment Ġsetup . ĠThe Ġproblem Ġof Ġsuch Ġan Ġapproach Ġis Ġagain Ġfinding Ġknowledgeable Ġusers Ġor Ġexperts Ġfor Ġlong - tail Ġentities . ĠThus Ġthe Ġnotion Ġof Ġrelevance Ġof Ġa Ġnews Ġrecommendation Ġis Ġchallenging Ġto Ġevaluate Ġin Ġa Ġcrowd Ġsetup . </s> We Ġtake Ġa Ġslightly Ġdifferent Ġapproach Ġby Ġmaking Ġan Ġassumption Ġthat Ġthe Ġnews Ġarticles Ġalready Ġpresent Ġin ĠWikipedia Ġentity Ġpages Ġare Ġrelevant . ĠTo Ġthis Ġextent , Ġwe Ġextract Ġa Ġdataset Ġcomprising Ġof Ġall Ġnews Ġarticles Ġreferenced Ġin Ġentity Ġpages Ġ( details Ġin ĠSection ĠSEC REF 40 Ġ). ĠAt Ġthe Ġexpense Ġof Ġnot Ġevaluating Ġthe Ġspace Ġcomprising Ġof Ġnews Ġarticles Ġabsent Ġin ĠWikipedia , Ġwe Ġsucceed Ġin Ġ( i ) Ġavoiding Ġrestrictive Ġassumptions Ġabout Ġthe Ġquality Ġof Ġhuman Ġjudgments , Ġ( ii ) Ġbeing Ġinvasive Ġand Ġpoll uting ĠWikipedia , Ġand Ġ( iii ) Ġder iving Ġa Ġreusable Ġtest Ġbed Ġfor Ġquicker Ġexperimentation . </s> The Ġsecond Ġchallenge Ġof Ġconstruction Ġof Ġtraining Ġand Ġtest Ġset Ġseparation Ġis Ġslightly Ġeasier Ġand Ġis Ġaddressed Ġin ĠSection ĠSEC REF 46 Ġ. </s> Dat as ets </s> The Ġdatasets Ġwe Ġuse Ġfor Ġour Ġexperimental Ġevaluation Ġare Ġdirectly Ġextracted Ġfrom Ġthe ĠWikipedia Ġentity Ġpages Ġand Ġtheir Ġrevision Ġhistory . ĠThe Ġgenerated Ġdata Ġrepresents Ġone Ġof Ġthe Ġcontributions Ġof Ġour Ġpaper . ĠThe Ġdatasets Ġare Ġthe Ġfollowing : </s> Entity ĠClasses . ĠWe Ġfocus Ġon Ġa Ġmanually Ġpredetermined Ġset Ġof Ġentity Ġclasses Ġfor Ġwhich Ġwe Ġexpect Ġto Ġhave Ġnews Ġcoverage . ĠThe Ġnumber Ġof Ġanalyzed Ġentity Ġclasses Ġis Ġ27 , Ġincluding ĠIN LINE FORM 0 Ġentities Ġwith Ġat Ġleast Ġone Ġnews Ġreference . ĠThe Ġentity Ġclasses Ġwere Ġselected Ġfrom Ġthe ĠDB pedia Ġclass Ġont ology . ĠFigure ĠFIG REF 42 Ġshows Ġthe Ġnumber Ġof Ġentities Ġper Ġclass Ġfor Ġthe Ġyears Ġ( 2009 - 2014 ). </s> News ĠArticles . ĠWe Ġextract Ġall Ġnews Ġreferences Ġfrom Ġthe Ġcollected ĠWikipedia Ġentity Ġpages . ĠThe Ġextracted Ġnews Ġreferences Ġare Ġassociated Ġwith Ġthe Ġsections Ġin Ġwhich Ġthey Ġappear . ĠIn Ġtotal Ġthere Ġwere ĠIN LINE FORM 0 Ġnews Ġreferences , Ġand Ġafter Ġcrawling Ġwe Ġend Ġup Ġwith ĠIN LINE FORM 1 Ġsuccessfully Ġcrawled Ġnews Ġarticles . ĠThe Ġdetails Ġof Ġthe Ġnews Ġarticle Ġdistribution , Ġand Ġthe Ġnumber Ġof Ġentities Ġand Ġsections Ġfrom Ġwhich Ġthey Ġare Ġreferred Ġare Ġshown Ġin ĠTable ĠT AB REF 44 Ġ. </s> Article - Entity ĠGround - truth . ĠThe Ġdataset Ġcomprises Ġof Ġthe Ġnews Ġand Ġentity Ġpairs ĠIN LINE FORM 0 Ġ. ĠNews - entity Ġpairs Ġare Ġrelevant Ġif Ġthe Ġnews Ġarticle Ġis Ġreferenced Ġin Ġthe Ġentity Ġpage . ĠNon - relevant Ġpairs Ġ( i . e . Ġnegative Ġtraining Ġexamples ) Ġconsist Ġof Ġnews Ġarticles Ġthat Ġcontain Ġan Ġentity Ġbut Ġare Ġnot Ġreferenced Ġin Ġthat Ġentity 's Ġpage . ĠIf Ġa Ġnews Ġarticle ĠIN LINE FORM 1 Ġis Ġreferred Ġfrom ĠIN LINE FORM 2 Ġat Ġyear ĠIN LINE FORM 3 Ġ, Ġthe Ġfeatures Ġare Ġcomputed Ġtaking Ġinto Ġaccount Ġthe Ġentity Ġprofiles Ġat Ġyear ĠIN LINE FORM 4 Ġ. </s> Article - Section ĠGround - truth . ĠThe Ġdataset Ġconsists Ġof Ġthe Ġtriple ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ, Ġwhere Ġwe Ġassume Ġthat ĠIN LINE FORM 2 Ġhas Ġalready Ġbeen Ġdetermined Ġas Ġrelevant . ĠWe Ġtherefore Ġhave Ġa Ġmulti - class Ġclassification Ġproblem Ġwhere Ġwe Ġneed Ġto Ġdetermine Ġthe Ġsection Ġof ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġis Ġcited . ĠSimilar Ġto Ġthe Ġarticle - entity Ġground Ġtruth , Ġhere Ġtoo Ġthe Ġfeatures Ġcompute Ġthe Ġsimilarity Ġbetween ĠIN LINE FORM 5 Ġ, ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. </s> Data ĠPre - Process ing </s> We ĠPOS - tag Ġthe Ġnews Ġarticles Ġand Ġentity Ġprofiles ĠIN LINE FORM 0 Ġwith Ġthe ĠStanford Ġtag ger ĠB IB REF 21 Ġ. ĠFor Ġentity Ġlinking Ġthe Ġnews Ġarticles , Ġwe Ġuse ĠTag Me ! ĠB IB REF 22 Ġwith Ġa Ġconfidence Ġscore Ġof Ġ0 . 3 . ĠOn Ġa Ġmanual Ġinspection Ġof Ġa Ġrandom Ġsample Ġof Ġ1000 Ġdis amb ig uated Ġentities , Ġthe Ġaccuracy Ġis Ġabove Ġ0 . 9 . ĠOn Ġaverage , Ġthe Ġnumber Ġof Ġentities Ġper Ġnews Ġarticle Ġis Ġapproximately Ġ30 . ĠFor Ġentity Ġlinking Ġthe Ġentity Ġprofiles , Ġwe Ġsimply Ġfollow Ġthe Ġanchor Ġtext Ġthat Ġrefers Ġto ĠWikipedia Ġentities . </s> Train Ġand ĠTesting ĠEvaluation ĠSetup </s> We Ġevaluate Ġthe Ġgenerated Ġsupervised Ġmodels Ġfor Ġthe Ġtwo Ġtasks , ĠA EP Ġand ĠASP , Ġby Ġsplitting Ġthe Ġtrain Ġand Ġtesting Ġinstances . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat Ġfor Ġthe Ġpairs ĠIN LINE FORM 0 Ġand Ġthe Ġtriple ĠIN LINE FORM 1 Ġ, Ġthe Ġnews Ġarticle ĠIN LINE FORM 2 Ġis Ġreferenced Ġat Ġtime ĠIN LINE FORM 3 Ġby Ġentity ĠIN LINE FORM 4 Ġ, Ġwhile Ġthe Ġfeatures Ġtake Ġinto Ġaccount Ġthe Ġentity Ġprofile Ġat Ġtime ĠIN LINE FORM 5 Ġ. ĠThis Ġavoids Ġany Ġ` over l apping ' Ġcontent Ġbetween Ġthe Ġnews Ġarticle Ġand Ġthe Ġentity Ġpage , Ġwhich Ġcould Ġaffect Ġthe Ġlearning Ġtask Ġof Ġthe Ġfunctions ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ. ĠTable ĠT AB REF 47 Ġshows Ġthe Ġstatistics Ġof Ġtrain Ġand Ġtest Ġinstances . ĠWe Ġlearn Ġthe Ġfunctions Ġat Ġyear ĠIN LINE FORM 8 Ġand Ġtest Ġon Ġinstances Ġfor Ġthe Ġyears Ġgreater Ġthan ĠIN LINE FORM 9 Ġ. ĠPlease Ġnote Ġthat Ġwe Ġdo Ġnot Ġshow Ġthe Ġperformance Ġfor Ġyear Ġ2014 Ġas Ġwe Ġdo Ġnot Ġhave Ġdata Ġfor Ġ2015 Ġfor Ġevaluation . </s> Article - Section ĠPl acement </s> Here Ġwe Ġshow Ġthe Ġevaluation Ġsetup Ġfor ĠASP Ġtask Ġand Ġdiscuss Ġthe Ġresults Ġwith Ġa Ġfocus Ġon Ġthree Ġmain Ġaspects , Ġ( i ) Ġthe Ġoverall Ġperformance Ġacross Ġthe Ġyears , Ġ( ii ) Ġthe Ġentity Ġclass Ġspecific Ġperformance , Ġand Ġ( iii ) Ġthe Ġimpact Ġon Ġentity Ġprofile Ġexpansion Ġby Ġsuggesting Ġmissing Ġsections Ġto Ġentities Ġbased Ġon Ġthe Ġpre - com puted Ġtemplates . </s> Bas elines . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġwe Ġare Ġnot Ġaware Ġof Ġany Ġcomparable Ġapproach Ġfor Ġthis Ġtask . ĠTherefore , Ġthe Ġbas elines Ġwe Ġconsider Ġare Ġthe Ġfollowing : </s> S 1 : ĠPick Ġthe Ġsection Ġfrom Ġtemplate ĠIN LINE FORM 0 Ġwith Ġthe Ġhighest Ġlex ical Ġsimilarity Ġto ĠIN LINE FORM 1 Ġ: ĠS 1 ĠIN LINE FORM 2 </s> S 2 : ĠPlace Ġthe Ġnews Ġinto Ġthe Ġmost Ġfrequent Ġsection Ġin ĠIN LINE FORM 0 </s> Learning ĠModels . ĠWe Ġuse ĠRandom ĠFore sts Ġ( RF ) ĠB IB REF 23 Ġand ĠSupport ĠVector ĠMachines Ġ( S VM ) ĠB IB REF 24 Ġ. ĠThe Ġmodels Ġare Ġoptimized Ġtaking Ġinto Ġaccount Ġthe Ġfeatures Ġin ĠTable ĠT AB REF 31 Ġ. ĠIn Ġcontrast Ġto Ġthe ĠA EP Ġtask , Ġhere Ġthe Ġscale Ġof Ġthe Ġnumber Ġof Ġinstances Ġallows Ġus Ġto Ġlearn Ġthe ĠS VM Ġmodels . ĠThe ĠS VM Ġmodel Ġis Ġoptimized Ġusing Ġthe ĠIN LINE FORM 0 Ġloss Ġfunction Ġand Ġuses Ġthe ĠGa ussian Ġkernels . </s> Met rics . ĠWe Ġcompute Ġprecision ĠP Ġas Ġthe Ġratio Ġof Ġnews Ġfor Ġwhich Ġwe Ġpick Ġa Ġsection ĠIN LINE FORM 0 Ġfrom ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġconform s Ġto Ġthe Ġone Ġin Ġour Ġground - truth Ġ( see ĠSection ĠSEC REF 40 Ġ). ĠThe Ġdefinition Ġof Ġrecall ĠR Ġand ĠF 1 Ġscore Ġfollows Ġfrom Ġthat Ġof Ġprecision . </s> Figure ĠFIG REF 66 Ġshows Ġthe Ġoverall Ġperformance Ġand Ġa Ġcomparison Ġof Ġour Ġapproach Ġ( when ĠIN LINE FORM 0 Ġis Ġoptimized Ġusing ĠS VM ) Ġagainst Ġthe Ġbest Ġperforming Ġbaseline ĠS 2 . ĠWith Ġthe Ġincrease Ġin Ġthe Ġnumber Ġof Ġtraining Ġinstances Ġfor Ġthe ĠASP Ġtask Ġthe Ġperformance Ġis Ġa Ġmon oton ically Ġnon - dec re asing Ġfunction . ĠFor Ġthe Ġyear Ġ2009 , Ġwe Ġoptimize Ġthe Ġlearning Ġobjective Ġof ĠIN LINE FORM 1 Ġwith Ġaround Ġ8 % Ġof Ġthe Ġtotal Ġinstances , Ġand Ġevaluate Ġon Ġthe Ġrest . ĠThe Ġperformance Ġon Ġaverage Ġis Ġaround ĠP = 0 . 66 Ġacross Ġall Ġclasses . ĠEven Ġthough Ġfor Ġmany Ġclasses Ġthe Ġperformance Ġis Ġalready Ġstable Ġ( as Ġwe Ġwill Ġsee Ġin Ġthe Ġnext Ġsection ), Ġfor Ġsome Ġclasses Ġwe Ġimprove Ġfurther . ĠIf Ġwe Ġtake Ġinto Ġaccount Ġthe Ġyears Ġbetween Ġ2010 Ġand Ġ2012 , Ġwe Ġhave Ġan Ġincrease Ġof ĠIN LINE FORM 2 ĠP = 0 . 17 , Ġwith Ġaround Ġ70 % Ġof Ġinstances Ġused Ġfor Ġtraining Ġand Ġthe Ġremainder Ġfor Ġevaluation . ĠFor Ġthe Ġremaining Ġyears Ġthe Ġtotal Ġimprovement Ġis ĠIN LINE FORM 3 ĠP = 0 . 18 Ġin Ġcontrast Ġto Ġthe Ġperformance Ġat Ġyear Ġ2009 . </s> On Ġthe Ġother Ġhand , Ġthe Ġbaseline ĠS 1 Ġhas Ġan Ġaverage Ġprecision Ġof ĠP = 0 . 12 . ĠThe Ġperformance Ġacross Ġthe Ġyears Ġvaries Ġslightly , Ġwith Ġthe Ġyear Ġ2011 Ġhaving Ġthe Ġhighest Ġaverage Ġprecision Ġof ĠP = 0 . 13 . ĠAlways Ġpicking Ġthe Ġmost Ġfrequent Ġsection Ġas Ġin ĠS 2 , Ġas Ġshown Ġin ĠFigure ĠFIG REF 66 Ġ, Ġresults Ġin Ġan Ġaverage Ġprecision Ġof ĠP = 0 . 17 , Ġwith Ġa Ġuniform Ġdistribution Ġacross Ġthe Ġyears . </s> Here Ġwe Ġshow Ġthe Ġperformance Ġof ĠIN LINE FORM 0 Ġdecomp osed Ġfor Ġthe Ġdifferent Ġentity Ġclasses . ĠSpecifically Ġwe Ġanalyze Ġthe Ġ27 Ġclasses Ġin ĠFigure ĠFIG REF 42 Ġ. ĠIn ĠTable ĠT AB REF 68 Ġ, Ġwe Ġshow Ġthe Ġresults Ġfor Ġa Ġrange Ġof Ġyears Ġ( we Ġomit Ġshowing Ġall Ġyears Ġdue Ġto Ġspace Ġconstraints ). ĠFor Ġillustration Ġpurposes Ġonly , Ġwe Ġgroup Ġthem Ġinto Ġfour Ġmain Ġclasses Ġ( ĠIN LINE FORM 1 ĠPerson , ĠOrganization , ĠLocation , ĠEvent ĠIN LINE FORM 2 Ġ) Ġand Ġinto Ġthe Ġspecific Ġsub - classes Ġshown Ġin Ġthe Ġsecond Ġcolumn Ġin ĠTable ĠT AB REF 68 Ġ. ĠFor Ġinstance , Ġthe Ġentity Ġclasses ĠOffice H older Ġand ĠPolit ician Ġare Ġaggreg ated Ġinto ĠPerson âĢĵ Politics . </s> It Ġis Ġevident Ġthat Ġin Ġthe Ġfirst Ġyear Ġthe Ġperformance Ġis Ġlower Ġin Ġcontrast Ġto Ġthe Ġlater Ġyears . ĠThis Ġis Ġdue Ġto Ġthe Ġfact Ġthat Ġas Ġwe Ġproceed , Ġwe Ġcan Ġbetter Ġgeneral ize Ġand Ġaccurately Ġdetermine Ġthe Ġcorrect Ġfit Ġof Ġan Ġarticle ĠIN LINE FORM 0 Ġinto Ġone Ġof Ġthe Ġsections Ġfrom Ġthe Ġpre - com puted Ġtemplates ĠIN LINE FORM 1 Ġ. ĠThe Ġresults Ġare Ġalready Ġstable Ġfor Ġthe Ġyear Ġrange ĠIN LINE FORM 2 Ġ. ĠFor Ġa Ġfew ĠPerson Ġsub - classes , Ġe . g . ĠPolitics , ĠEntertainment , Ġwe Ġachieve Ġan ĠF 1 Ġscore Ġabove Ġ0 . 9 . ĠThese Ġadditionally Ġrepresent Ġclasses Ġwith Ġa Ġsufficient Ġnumber Ġof Ġtraining Ġinstances Ġfor Ġthe Ġyears ĠIN LINE FORM 3 Ġ. ĠThe Ġlowest ĠF 1 Ġscore Ġis Ġfor Ġthe ĠCriminal Ġand ĠTelevision Ġclasses . ĠHowever , Ġthis Ġis Ġdirectly Ġcorrelated Ġwith Ġthe Ġinsufficient Ġnumber Ġof Ġinstances . </s> The Ġbaseline Ġapproaches Ġfor Ġthe ĠASP Ġtask Ġperform Ġpoorly . ĠS 1 , Ġbased Ġon Ġlex ical Ġsimilarity , Ġhas Ġa Ġvarying Ġperformance Ġfor Ġdifferent Ġentity Ġclasses . ĠThe Ġbest Ġperformance Ġis Ġachieved Ġfor Ġthe Ġclass ĠPerson ĠâĢĵ ĠPolitics , Ġwith ĠP = 0 . 43 . ĠThis Ġhighlights Ġthe Ġimportance Ġof Ġour Ġfeature Ġchoice Ġand Ġthat Ġthe ĠASP Ġcannot Ġbe Ġconsidered Ġas Ġa Ġlinear Ġfunction , Ġwhere Ġthe Ġmaximum Ġsimilarity Ġyields Ġthe Ġbest Ġresults . ĠFor Ġdifferent Ġentity Ġclasses Ġdifferent Ġfeatures Ġand Ġcombination Ġof Ġfeatures Ġis Ġnecessary . ĠConsidering Ġthat ĠS 2 Ġis Ġthe Ġoverall Ġbest Ġperforming Ġbaseline , Ġthrough Ġour Ġapproach ĠIN LINE FORM 0 Ġwe Ġhave Ġa Ġsignificant Ġimprovement Ġof Ġover ĠIN LINE FORM 1 ĠP = + 0 . 64 . </s> The Ġmodels Ġwe Ġlearn Ġare Ġvery Ġrobust Ġand Ġobtain Ġhigh Ġaccuracy , Ġfulfilling Ġour Ġpre - condition Ġfor Ġaccurate Ġnews Ġsuggestions Ġinto Ġthe Ġentity Ġsections . ĠWe Ġmeasure Ġthe Ġrobust ness Ġof ĠIN LINE FORM 0 Ġthrough Ġthe ĠIN LINE FORM 1 Ġstatistic . ĠIn Ġthis Ġcase , Ġwe Ġhave Ġa Ġmodel Ġwith Ġroughly Ġ10 Ġlabels Ġ( cor respond ing Ġto Ġthe Ġnumber Ġof Ġsections Ġin Ġa Ġtemplate ĠIN LINE FORM 2 Ġ). ĠThe Ġscore Ġwe Ġachieve Ġshows Ġthat Ġour Ġmodel Ġpredicts Ġwith Ġhigh Ġconfidence Ġwith ĠIN LINE FORM 3 Ġ. </s> The Ġlast Ġanalysis Ġis Ġthe Ġimpact Ġwe Ġhave Ġon Ġexpanding Ġentity Ġprofiles ĠIN LINE FORM 0 Ġwith Ġnew Ġsections . ĠFigure ĠFIG REF 70 Ġshows Ġthe Ġratio Ġof Ġsections Ġfor Ġwhich Ġwe Ġcorrectly Ġsuggest Ġan Ġarticle ĠIN LINE FORM 1 Ġto Ġthe Ġright Ġsection Ġin Ġthe Ġsection Ġtemplate ĠIN LINE FORM 2 Ġ. ĠThe Ġratio Ġhere Ġcorresponds Ġto Ġsections Ġthat Ġare Ġnot Ġpresent Ġin Ġthe Ġentity Ġprofile Ġat Ġyear ĠIN LINE FORM 3 Ġ, Ġthat Ġis ĠIN LINE FORM 4 Ġ. ĠHowever , Ġgiven Ġthe Ġgenerated Ġtemplates ĠIN LINE FORM 5 Ġ, Ġwe Ġcan Ġexpand Ġthe Ġentity Ġprofile ĠIN LINE FORM 6 Ġwith Ġa Ġnew Ġsection Ġat Ġtime ĠIN LINE FORM 7 Ġ. ĠIn Ġdetails , Ġin Ġthe Ġabsence Ġof Ġa Ġsection Ġat Ġtime ĠIN LINE FORM 8 Ġ, Ġour Ġmodel Ġtrains Ġwell Ġon Ġsimilar Ġsections Ġfrom Ġthe Ġsection Ġtemplate ĠIN LINE FORM 9 Ġ, Ġhence Ġwe Ġcan Ġpredict Ġaccurately Ġthe Ġsection Ġand Ġin Ġthis Ġcase Ġsuggest Ġits Ġaddition Ġto Ġthe Ġentity Ġprofile . ĠWith Ġtime , Ġit Ġis Ġobvious Ġthat Ġthe Ġexpansion Ġrate Ġdecreases Ġat Ġlater Ġyears Ġas Ġthe Ġentity Ġprofiles Ġbecome Ġmore Ġ` complete '. </s> This Ġis Ġparticularly Ġinteresting Ġfor Ġexpanding Ġthe Ġentity Ġprofiles Ġof Ġlong - tail Ġentities Ġas Ġwell Ġas Ġupdating Ġentities Ġwith Ġreal - world Ġemerging Ġevents Ġthat Ġare Ġadded Ġconstantly . ĠIn Ġmany Ġcases Ġsuch Ġmissing Ġsections Ġare Ġpresent Ġat Ġone Ġof Ġthe Ġentities Ġof Ġthe Ġrespective Ġentity Ġclass ĠIN LINE FORM 0 Ġ. ĠAn Ġobvious Ġcase Ġis Ġthe Ġexample Ġtaken Ġin ĠSection ĠSEC REF 16 Ġ, Ġwhere Ġthe Ġ` Acc idents ' Ġis Ġrather Ġcommon Ġfor Ġentities Ġof Ġtype ĠAir line . ĠHowever , Ġit Ġis Ġnon - existent Ġfor Ġsome Ġspecific Ġentity Ġinstances , Ġi . e ĠGerman wings Ġairline . </s> Through Ġour ĠASP Ġapproach ĠIN LINE FORM 0 Ġ, Ġwe Ġare Ġable Ġto Ġexpand Ġboth Ġlong - tail Ġand Ġtrunk Ġentities . ĠWe Ġdistinguish Ġbetween Ġthe Ġtwo Ġtypes Ġof Ġentities Ġby Ġsimply Ġmeasuring Ġtheir Ġsection Ġtext Ġlength . ĠThe Ġreal Ġdistribution Ġin Ġthe Ġground Ġtruth Ġ( see ĠSection ĠSEC REF 40 Ġ) Ġis Ġ27 % Ġand Ġ73 % Ġare Ġlong - tail Ġand Ġtrunk Ġentities , Ġrespectively . ĠWe Ġare Ġable Ġto Ġexpand Ġthe Ġentity Ġprofiles Ġfor Ġboth Ġcases Ġand Ġall Ġentity Ġclasses Ġwithout Ġa Ġsignificant Ġdifference , Ġwith Ġthe Ġonly Ġexception Ġbeing Ġthe Ġclass ĠCreative ĠWork , Ġwhere Ġwe Ġexpand Ġsignificantly Ġmore Ġtrunk Ġentities . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġwork , Ġwe Ġhave Ġproposed Ġan Ġautomated Ġapproach Ġfor Ġthe Ġnovel Ġtask Ġof Ġsuggesting Ġnews Ġarticles Ġto ĠWikipedia Ġentity Ġpages Ġto Ġfacilitate ĠWikipedia Ġupdating . ĠThe Ġprocess Ġconsists Ġof Ġtwo Ġstages . ĠIn Ġthe Ġfirst Ġstage , Ġarticle âĢĵ entity Ġplacement , Ġwe Ġsuggest Ġnews Ġarticles Ġto Ġentity Ġpages Ġby Ġconsidering Ġthree Ġmain Ġfactors , Ġsuch Ġas Ġentity Ġsal ience Ġin Ġa Ġnews Ġarticle , Ġrelative Ġauthority Ġand Ġnovelty Ġof Ġnews Ġarticles Ġfor Ġan Ġentity Ġpage . ĠIn Ġthe Ġsecond Ġstage , Ġarticle âĢĵ section Ġplacement , Ġwe Ġdetermine Ġthe Ġbest Ġfitting Ġsection Ġin Ġan Ġentity Ġpage . ĠHere , Ġwe Ġremedy Ġthe Ġproblem Ġof Ġincomplete Ġentity Ġsection Ġprofiles Ġby Ġconstructing Ġsection Ġtemplates Ġfor Ġspecific Ġentity Ġclasses . ĠThis Ġallows Ġus Ġto Ġadd Ġmissing Ġsections Ġto Ġentity Ġpages . ĠWe Ġcarry Ġout Ġan Ġextensive Ġexperimental Ġevaluation Ġon Ġ351 , 98 3 Ġnews Ġarticles Ġand Ġ73 , 7 34 Ġentities Ġcoming Ġfrom Ġ27 Ġdistinct Ġentity Ġclasses . ĠFor Ġthe Ġfirst Ġstage , Ġwe Ġachieve Ġan Ġoverall Ġperformance Ġwith ĠP = 0 . 93 , ĠR = 0 . 514 Ġand ĠF 1 = 0 . 676 , Ġoutper forming Ġour Ġbaseline Ġcompetitors Ġsignificantly . ĠFor Ġthe Ġsecond Ġstage , Ġwe Ġshow Ġthat Ġwe Ġcan Ġlearn Ġincrement ally Ġto Ġdetermine Ġthe Ġcorrect Ġsection Ġfor Ġa Ġnews Ġarticle Ġbased Ġon Ġsection Ġtemplates . ĠThe Ġoverall Ġperformance Ġacross Ġdifferent Ġclasses Ġis ĠP = 0 . 8 44 , ĠR = 0 . 885 Ġand ĠF 1 = 0 . 860 . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġenhance Ġour Ġwork Ġby Ġextracting Ġfacts Ġfrom Ġthe Ġsuggested Ġnews Ġarticles . ĠResults Ġsuggest Ġthat Ġthe Ġnews Ġcontent Ġcited Ġin Ġentity Ġpages Ġcomes Ġfrom Ġthe Ġfirst Ġparagraphs . ĠHowever , Ġchallenging Ġtask Ġsuch Ġas Ġthe Ġcanonical ization Ġand Ġchronological Ġordering Ġof Ġfacts , Ġstill Ġremain .