By calculating Macro F1 metric at the document level.
ABSTRACTIVE
by evaluating their model on five different benchmarks
ABSTRACTIVE
Training We collect 50,000 Wikipedia articles according to the number of its hyperlinks as our training data. For efficiency, we trim the articles to the first three paragraphs leading to 1,035,665 mentions in total. Using CoNLL-Test A as the development set, we evaluate the trained NCEL on the above benchmarks. We set context window to 20, neighbor mention window to 6, and top INLINEFORM0 candidates for each mention. We use two layers with 2000 and 1 hidden units in MLP encoder, and 3 layers in sub-GCN. We use early stop and fine tune the embeddings. With a batch size of 16, nearly 3 epochs cost less than 15 minutes on the server with 20 core CPU and the GeForce GTX 1080Ti GPU with 12Gb memory. We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements. The results are shown in Table FIGREF28 and Table FIGREF28 . We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases. In the “easy" case, local models have similar performance with global models since only little global information is available (2 mentions per document). Besides, NN-based models, NTEE and NCEL-local, perform significantly better than others including most global models, demonstrating that the effectiveness of neural models deals with the first limitation in the introduction.
To avoid overfitting with some dataset, we train NCEL using collected Wikipedia hyperlinks instead of specific annotated data. We then evaluate the trained model on five different benchmarks to verify the linking precision as well as the generalization ability. Furthermore, we investigate the effectiveness of key modules in NCEL and give qualitative results for comprehensive analysis.
1811.08603
Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Entity Ġlinking Ġ( EL ), Ġmapping Ġentity Ġmentions Ġin Ġtexts Ġto Ġa Ġgiven Ġknowledge Ġbase Ġ( KB ), Ġserves Ġas Ġa Ġfundamental Ġrole Ġin Ġmany Ġfields , Ġsuch Ġas Ġquestion Ġanswering ĠB IB REF 0 Ġ, Ġsemantic Ġsearch ĠB IB REF 1 Ġ, Ġand Ġinformation Ġextraction ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġthis Ġtask Ġis Ġnon - t riv ial Ġbecause Ġentity Ġmentions Ġare Ġusually Ġambiguous . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ, Ġthe Ġmention ĠEngland Ġrefers Ġto Ġthree Ġentities Ġin ĠKB , Ġand Ġan Ġentity Ġlinking Ġsystem Ġshould Ġbe Ġcapable Ġof Ġidentifying Ġthe Ġcorrect Ġentity Ġas ĠEngland Ġcricket Ġteam Ġrather Ġthan ĠEngland Ġand ĠEngland Ġnational Ġfootball Ġteam . </s> Entity Ġlinking Ġis Ġtypically Ġbroken Ġdown Ġinto Ġtwo Ġmain Ġphases : Ġ( i ) Ġcandidate Ġgeneration Ġob tains Ġa Ġset Ġof Ġrefere nt Ġentities Ġin ĠKB Ġfor Ġeach Ġmention , Ġand Ġ( ii ) Ġnamed Ġentity Ġdis amb ig uation Ġselects Ġthe Ġpossible Ġcandidate Ġentity Ġby Ġsolving Ġa Ġranking Ġproblem . ĠThe Ġkey Ġchallenge Ġlies Ġin Ġthe Ġranking Ġmodel Ġthat Ġcomp utes Ġthe Ġrelevance Ġbetween Ġcandidates Ġand Ġthe Ġcorresponding Ġmentions Ġbased Ġon Ġthe Ġinformation Ġboth Ġin Ġtexts Ġand ĠKB s ĠB IB REF 4 Ġ. ĠIn Ġterms Ġof Ġthe Ġfeatures Ġused Ġfor Ġranking , Ġwe Ġclassify Ġexisting ĠEL Ġmodels Ġinto Ġtwo Ġgroups : Ġlocal Ġmodels Ġto Ġresolve Ġmentions Ġindependently Ġrelying Ġon Ġtextual Ġcontext Ġinformation Ġfrom Ġthe Ġsurrounding Ġwords ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġglobal Ġ( collect ive ) Ġmodels , Ġwhich Ġare Ġthe Ġmain Ġfocus Ġof Ġthis Ġpaper , Ġthat Ġencourage Ġthe Ġtarget Ġentities Ġof Ġall Ġmentions Ġin Ġa Ġdocument Ġto Ġbe Ġtop ically Ġcoherent ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> Global Ġmodels Ġusually Ġbuild Ġan Ġentity Ġgraph Ġbased Ġon ĠKB s Ġto Ġcapture Ġcoherent Ġentities Ġfor Ġall Ġidentified Ġmentions Ġin Ġa Ġdocument , Ġwhere Ġthe Ġnodes Ġare Ġentities , Ġand Ġedges Ġdenote Ġtheir Ġrelations . ĠThe Ġgraph Ġprovides Ġhighly Ġdiscrim in ative Ġsemantic Ġsignals Ġ( e . g ., Ġentity Ġrelated ness ) Ġthat Ġare Ġunavailable Ġto Ġlocal Ġmodel ĠB IB REF 15 Ġ. ĠFor Ġexample Ġ( Figure ĠFIG REF 1 Ġ), Ġan ĠEL Ġmodel Ġseem ly Ġcannot Ġfind Ġsufficient Ġdis amb ig uation Ġclues Ġfor Ġthe Ġmention ĠEngland Ġfrom Ġits Ġsurrounding Ġwords , Ġunless Ġit Ġutilizes Ġthe Ġco herence Ġinformation Ġof Ġconsistent Ġtopic ĠâĢ ľ c rick et " Ġamong Ġadjacent Ġmentions ĠEngland , ĠHussain , Ġand ĠEssex . ĠAlthough Ġthe Ġglobal Ġmodel Ġhas Ġachieved Ġsignificant Ġimprovements , Ġits Ġlimitation Ġis Ġthree fold : </s> To Ġmitigate Ġthe Ġfirst Ġlimitation , Ġrecent ĠEL Ġstudies Ġintroduce Ġneural Ġnetwork Ġ( NN ) Ġmodels Ġdue Ġto Ġits Ġamazing Ġfeature Ġabstraction Ġand Ġgeneral ization Ġability . ĠIn Ġsuch Ġmodels , Ġwords / ent ities Ġare Ġrepresented Ġby Ġlow Ġdimensional Ġvectors Ġin Ġa Ġcontinuous Ġspace , Ġand Ġfeatures Ġfor Ġmention Ġas Ġwell Ġas Ġcandidate Ġentities Ġare Ġautomatically Ġlearned Ġfrom Ġdata ĠB IB REF 4 Ġ. ĠHowever , Ġexisting ĠN N - based Ġmethods Ġfor ĠEL Ġare Ġeither Ġlocal Ġmodels ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġor Ġmerely Ġuse Ġword / entity Ġembed d ings Ġfor Ġfeature Ġextraction Ġand Ġrely Ġon Ġanother Ġmodules Ġfor Ġcollective Ġdis amb ig uation , Ġwhich Ġthus Ġcannot Ġfully Ġutilize Ġthe Ġpower Ġof ĠN N Ġmodels Ġfor Ġcollective ĠEL ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. </s> The Ġsecond Ġdrawback Ġof Ġthe Ġglobal Ġapproach Ġhas Ġbeen Ġallev iated Ġthrough Ġapproximate Ġoptimization Ġtechniques , Ġsuch Ġas ĠPage Rank / random Ġwalks ĠB IB REF 21 Ġ, Ġgraph Ġpr uning ĠB IB REF 22 Ġ, Ġranking ĠSV Ms ĠB IB REF 23 Ġ, Ġor Ġloop y Ġbelief Ġpropagation Ġ( L BP ) ĠB IB REF 18 Ġ, ĠB IB REF 24 Ġ. ĠHowever , Ġthese Ġmethods Ġare Ġnot Ġdifferent iable Ġand Ġthus Ġdifficult Ġto Ġbe Ġintegrated Ġinto Ġneural Ġnetwork Ġmodels Ġ( the Ġsolution Ġfor Ġthe Ġfirst Ġlimitation ). </s> To Ġovercome Ġthe Ġthird Ġissue Ġof Ġinadequate Ġtraining Ġdata , ĠB IB REF 17 Ġhas Ġexplored Ġa Ġmassive Ġamount Ġof Ġhyper links Ġin ĠWikipedia , Ġbut Ġthese Ġpotential Ġannotations Ġfor ĠEL Ġcontain Ġmuch Ġnoise , Ġwhich Ġmay Ġdistract Ġa Ġnaive Ġdis amb ig uation Ġmodel ĠB IB REF 6 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel ĠNeural ĠCollective ĠEntity ĠLink ing Ġmodel Ġ( NC EL ), Ġwhich Ġperforms Ġglobal ĠEL Ġcombining Ġdeep Ġneural Ġnetworks Ġwith ĠGraph ĠConv olution al ĠNetwork Ġ( GC N ) ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġthat Ġallows Ġflexible Ġencoding Ġof Ġentity Ġgraphs . ĠIt Ġintegrates Ġboth Ġlocal Ġcontextual Ġinformation Ġand Ġglobal Ġinter d ependence Ġof Ġmentions Ġin Ġa Ġdocument , Ġand Ġis Ġefficiently Ġtrain able Ġin Ġan Ġend - to - end Ġfashion . ĠParticularly , Ġwe Ġintroduce Ġattention Ġmechanism Ġto Ġrobust ly Ġmodel Ġlocal Ġcontextual Ġinformation Ġby Ġselecting Ġinformative Ġwords Ġand Ġfiltering Ġout Ġthe Ġnoise . ĠOn Ġthe Ġother Ġhand , Ġwe Ġapply ĠGC Ns Ġto Ġimprove Ġdiscrim in ative Ġsignals Ġof Ġcandidate Ġentities Ġby Ġexploiting Ġthe Ġrich Ġstructure Ġunderlying Ġthe Ġcorrect Ġentities . ĠTo Ġalleviate Ġthe Ġglobal Ġcomput ations , Ġwe Ġpropose Ġto Ġconv olute Ġon Ġthe Ġsub graph Ġof Ġadjacent Ġmentions . ĠThus , Ġthe Ġoverall Ġco herence Ġshall Ġbe Ġachieved Ġin Ġa Ġchain - like Ġway Ġvia Ġa Ġsliding Ġwindow Ġover Ġthe Ġdocument . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġeffort Ġto Ġdevelop Ġa Ġunified Ġmodel Ġfor Ġneural Ġcollective Ġentity Ġlinking . </s> In Ġexperiments , Ġwe Ġfirst Ġverify Ġthe Ġefficiency Ġof ĠNC EL Ġvia Ġtheoretically Ġcomparing Ġits Ġtime Ġcomplexity Ġwith Ġother Ġcollective Ġalternatives . ĠAfterwards , Ġwe Ġtrain Ġour Ġneural Ġmodel Ġusing Ġcollected ĠWikipedia Ġhyper links Ġinstead Ġof Ġdataset - specific Ġannotations , Ġand Ġperform Ġevaluations Ġon Ġfive Ġpublic Ġavailable Ġbenchmarks . ĠThe Ġresults Ġshow Ġthat ĠNC EL Ġconsistently Ġoutper forms Ġvarious Ġbas elines Ġwith Ġa Ġfavorable Ġgeneral ization Ġability . ĠFinally , Ġwe Ġfurther Ġpresent Ġthe Ġperformance Ġon Ġa Ġchallenging Ġdataset ĠWW ĠB IB REF 19 Ġas Ġwell Ġas Ġqualitative Ġresults , Ġinvestigating Ġthe Ġeffectiveness Ġof Ġeach Ġkey Ġmodule . </s> P rel im in aries Ġand ĠFramework </s> We Ġdenote ĠIN LINE FORM 0 Ġas Ġa Ġset Ġof Ġentity Ġmentions Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġis Ġeither Ġa Ġword ĠIN LINE FORM 3 Ġor Ġa Ġmention ĠIN LINE FORM 4 Ġ. ĠIN LINE FORM 5 Ġis Ġthe Ġentity Ġgraph Ġfor Ġdocument ĠIN LINE FORM 6 Ġderived Ġfrom Ġthe Ġgiven Ġknowledge Ġbase , Ġwhere ĠIN LINE FORM 7 Ġis Ġa Ġset Ġof Ġentities , ĠIN LINE FORM 8 Ġdenotes Ġthe Ġrelated ness Ġbetween ĠIN LINE FORM 9 Ġand Ġhigher Ġvalues Ġindicate Ġstronger Ġrelations . ĠBased Ġon ĠIN LINE FORM 10 Ġ, Ġwe Ġextract Ġa Ġsub graph ĠIN LINE FORM 11 Ġfor ĠIN LINE FORM 12 Ġ, Ġwhere ĠIN LINE FORM 13 Ġdenotes Ġthe Ġset Ġof Ġcandidate Ġentities Ġfor ĠIN LINE FORM 14 Ġ. ĠNote Ġthat Ġwe Ġdon 't Ġinclude Ġthe Ġrelations Ġamong Ġcandidates Ġof Ġthe Ġsame Ġmention Ġin ĠIN LINE FORM 15 Ġbecause Ġthese Ġcandidates Ġare Ġmutually Ġexclusive Ġin Ġdis amb ig uation . </s> Form ally , Ġwe Ġdefine Ġthe Ġentity Ġlinking Ġproblem Ġas Ġfollows : ĠGiven Ġa Ġset Ġof Ġmentions ĠIN LINE FORM 0 Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġ, Ġand Ġan Ġentity Ġgraph ĠIN LINE FORM 2 Ġ, Ġthe Ġgoal Ġis Ġto Ġfind Ġan Ġassignment ĠIN LINE FORM 3 Ġ. </s> To Ġcollectively Ġfind Ġthe Ġbest Ġassignment , ĠNC EL Ġaims Ġto Ġimprove Ġthe Ġdiscrim in ability Ġof Ġcandidates ' Ġlocal Ġfeatures Ġby Ġusing Ġentity Ġrelated ness Ġwithin Ġa Ġdocument Ġvia ĠGC N , Ġwhich Ġis Ġcapable Ġof Ġlearning Ġa Ġfunction Ġof Ġfeatures Ġon Ġthe Ġgraph Ġthrough Ġshared Ġparameters Ġover Ġall Ġnodes . ĠFigure ĠFIG REF 10 Ġshows Ġthe Ġframework Ġof ĠNC EL Ġincluding Ġthree Ġmain Ġcomponents : </s> Example ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, Ġfor Ġthe Ġcurrent Ġmention ĠEngland , Ġwe Ġutilize Ġits Ġsurrounding Ġwords Ġas Ġlocal Ġcontexts Ġ( e . g ., Ġsurplus ), Ġand Ġadjacent Ġmentions Ġ( e . g ., ĠH ussian ) Ġas Ġglobal Ġinformation . ĠCollect ively , Ġwe Ġutilize Ġthe Ġcandidates Ġof ĠEngland ĠIN LINE FORM 0 Ġas Ġwell Ġas Ġthose Ġentities Ġof Ġits Ġadj ac encies ĠIN LINE FORM 1 Ġto Ġconstruct Ġfeature Ġvectors Ġfor ĠIN LINE FORM 2 Ġand Ġthe Ġsub graph Ġof Ġrelated ness Ġas Ġinputs Ġof Ġour Ġneural Ġmodel . ĠLet Ġdarker Ġblue Ġindicate Ġhigher Ġprobability Ġof Ġbeing Ġpredicted , Ġthe Ġcorrect Ġcandidate ĠIN LINE FORM 3 Ġbecomes Ġbl uer Ġdue Ġto Ġits Ġbl uer Ġneighbor Ġnodes Ġof Ġother Ġmentions ĠIN LINE FORM 4 Ġ. ĠThe Ġdashed Ġlines Ġdenote Ġentity Ġrelations Ġthat Ġhave Ġindirect Ġimpacts Ġthrough Ġthe Ġsliding Ġadjacent Ġwindow Ġ, Ġand Ġthe Ġoverall Ġstructure Ġshall Ġbe Ġachieved Ġvia Ġmultiple Ġsub - graph s Ġby Ġtravers ing Ġall Ġmentions . </s> Before Ġintroducing Ġour Ġmodel , Ġwe Ġfirst Ġdescribe Ġthe Ġcomponent Ġof Ġcandidate Ġgeneration . </s> Cand idate ĠGeneration </s> Similar Ġto Ġprevious Ġwork ĠB IB REF 24 Ġ, Ġwe Ġuse Ġthe Ġprior Ġprobability ĠIN LINE FORM 0 Ġof Ġentity ĠIN LINE FORM 1 Ġconditioned Ġon Ġmention ĠIN LINE FORM 2 Ġboth Ġas Ġa Ġlocal Ġfeature Ġand Ġto Ġgenerate Ġcandidate Ġentities : ĠIN LINE FORM 3 Ġ. ĠWe Ġcompute ĠIN LINE FORM 4 Ġbased Ġon Ġstatistics Ġof Ġmention - entity Ġpairs Ġfrom : Ġ( i ) ĠWikipedia Ġpage Ġtitles , Ġredirect Ġtitles Ġand Ġhyper links , Ġ( ii ) Ġthe Ġdictionary Ġderived Ġfrom Ġa Ġlarge ĠWeb ĠCorpus ĠB IB REF 27 Ġ, Ġand Ġ( iii ) Ġthe ĠY AG O Ġdictionary Ġwith Ġa Ġuniform Ġdistribution ĠB IB REF 22 Ġ. ĠWe Ġpick Ġup Ġthe Ġmaximal Ġprior Ġif Ġa Ġmention - entity Ġpair Ġoccurs Ġin Ġdifferent Ġresources . ĠIn Ġexperiments , Ġto Ġoptimize Ġfor Ġmemory Ġand Ġrun Ġtime , Ġwe Ġkeep Ġonly Ġtop ĠIN LINE FORM 5 Ġentities Ġbased Ġon ĠIN LINE FORM 6 Ġ. ĠIn Ġthe Ġfollowing Ġtwo Ġsections , Ġwe Ġwill Ġpresent Ġthe Ġkey Ġcomponents Ġof ĠNEC L , Ġnamely Ġfeature Ġextraction Ġand Ġneural Ġnetwork Ġfor Ġcollective Ġentity Ġlinking . </s> Feature ĠExt raction </s> The Ġmain Ġgoal Ġof ĠNC EL Ġis Ġto Ġfind Ġa Ġsolution Ġfor Ġcollective Ġentity Ġlinking Ġusing Ġan Ġend - to - end Ġneural Ġmodel , Ġrather Ġthan Ġto Ġimprove Ġthe Ġmeasurements Ġof Ġlocal Ġtextual Ġsimilarity Ġor Ġglobal Ġmention / entity Ġrelated ness . ĠTherefore , Ġwe Ġuse Ġjoint Ġembed d ings Ġof Ġwords Ġand Ġentities Ġat Ġsense Ġlevel ĠB IB REF 28 Ġto Ġrepresent Ġmentions Ġand Ġits Ġcontexts Ġfor Ġfeature Ġextraction . ĠIn Ġthis Ġsection , Ġwe Ġgive Ġa Ġbrief Ġdescription Ġof Ġour Ġembed d ings Ġfollowed Ġby Ġour Ġfeatures Ġused Ġin Ġthe Ġneural Ġmodel . </s> Learning ĠJoint ĠEmb edd ings Ġof ĠWord Ġand ĠEntity </s> Following ĠB IB REF 28 Ġ, Ġwe Ġuse ĠWikipedia Ġarticles , Ġhyper links , Ġand Ġentity Ġout links Ġto Ġjointly Ġlearn Ġword / ment ion Ġand Ġentity Ġembed d ings Ġin Ġa Ġunified Ġvector Ġspace , Ġso Ġthat Ġsimilar Ġwords / ment ions Ġand Ġentities Ġhave Ġsimilar Ġvectors . ĠTo Ġaddress Ġthe Ġambiguity Ġof Ġwords / ment ions , ĠB IB REF 28 Ġrepresents Ġeach Ġword / ment ion Ġwith Ġmultiple Ġvectors , Ġand Ġeach Ġvector Ġdenotes Ġa Ġsense Ġreferring Ġto Ġan Ġentity Ġin ĠKB . ĠThe Ġquality Ġof Ġthe Ġembed d ings Ġis Ġverified Ġon Ġboth Ġtextual Ġsimilarity Ġand Ġentity Ġrelated ness Ġtasks . </s> Form ally , Ġeach Ġword / ment ion Ġhas Ġa Ġglobal Ġembed ding ĠIN LINE FORM 0 Ġ, Ġand Ġmultiple Ġsense Ġembed d ings ĠIN LINE FORM 1 Ġ. ĠEach Ġsense Ġembed ding ĠIN LINE FORM 2 Ġrefers Ġto Ġan Ġentity Ġembed ding ĠIN LINE FORM 3 Ġ, Ġwhile Ġthe Ġdifference Ġbetween ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġis Ġthat ĠIN LINE FORM 6 Ġmodels Ġthe Ġco - occ urrence Ġinformation Ġof Ġan Ġentity Ġin Ġtexts Ġ( via Ġhyper links ) Ġand ĠIN LINE FORM 7 Ġenc odes Ġthe Ġstructured Ġentity Ġrelations Ġin ĠKB s . ĠMore Ġdetails Ġcan Ġbe Ġfound Ġin Ġthe Ġoriginal Ġpaper . </s> Local ĠFeatures </s> Local Ġfeatures Ġfocus Ġon Ġhow Ġcompatible Ġthe Ġentity Ġis Ġmentioned Ġin Ġa Ġpiece Ġof Ġtext Ġ( i . e ., Ġthe Ġmention Ġand Ġthe Ġcontext Ġwords ). ĠExcept Ġfor Ġthe Ġprior Ġprobability Ġ( Section ĠSEC REF 9 Ġ), Ġwe Ġdefine Ġtwo Ġtypes Ġof Ġlocal Ġfeatures Ġfor Ġeach Ġcandidate Ġentity ĠIN LINE FORM 0 Ġ: </s> String ĠSimilar ity ĠSimilar Ġto ĠB IB REF 16 Ġ, Ġwe Ġdefine Ġstring Ġbased Ġfeatures Ġas Ġfollows : Ġthe Ġedit Ġdistance Ġbetween Ġmention 's Ġsurface Ġform Ġand Ġentity Ġtitle , Ġand Ġboolean Ġfeatures Ġindicating Ġwhether Ġthey Ġare Ġequivalent , Ġwhether Ġthe Ġmention Ġis Ġinside , Ġstarts Ġwith Ġor Ġends Ġwith Ġentity Ġtitle Ġand Ġvice Ġversa . </s> Comp atibility ĠWe Ġalso Ġmeasure Ġthe Ġcompatibility Ġof ĠIN LINE FORM 0 Ġwith Ġthe Ġmention 's Ġcontext Ġwords ĠIN LINE FORM 1 Ġby Ġcomputing Ġtheir Ġsimilarities Ġbased Ġon Ġjoint Ġembed d ings : ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġcontext Ġembed ding Ġof ĠIN LINE FORM 5 Ġconditioned Ġon Ġcandidate ĠIN LINE FORM 6 Ġand Ġis Ġdefined Ġas Ġthe Ġaverage Ġsum Ġof Ġword Ġglobal Ġvectors Ġweighted Ġby Ġattent ions : ĠIN LINE FORM 7 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠIN LINE FORM 1 Ġ- th Ġword 's Ġattention Ġfrom ĠIN LINE FORM 2 Ġ. ĠIn Ġthis Ġway , Ġwe Ġautomatically Ġselect Ġinformative Ġwords Ġby Ġassigning Ġhigher Ġattention Ġweights , Ġand Ġfilter Ġout Ġirrelevant Ġnoise Ġthrough Ġsmall Ġweights . ĠThe Ġattention ĠIN LINE FORM 3 Ġis Ġcomputed Ġas Ġfollows : ĠIN LINE FORM 4 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġsimilarity Ġmeasurement , Ġand Ġwe Ġuse Ġcos ine Ġsimilarity Ġin Ġthe Ġpresented Ġwork . ĠWe Ġconc aten ate Ġthe Ġprior Ġprobability , Ġstring Ġbased Ġsimilarities , Ġcompatibility Ġsimilarities Ġand Ġthe Ġembed d ings Ġof Ġcontexts Ġas Ġwell Ġas Ġthe Ġentity Ġas Ġthe Ġlocal Ġfeature Ġvectors . </s> Global ĠFeatures </s> The Ġkey Ġidea Ġof Ġcollective ĠEL Ġis Ġto Ġutilize Ġthe Ġtopical Ġco herence Ġthroughout Ġthe Ġentire Ġdocument . ĠThe Ġconsistency Ġassumption Ġbehind Ġit Ġis Ġthat : Ġall Ġmentions Ġin Ġa Ġdocument Ġshall Ġbe Ġon Ġthe Ġsame Ġtopic . ĠHowever , Ġthis Ġleads Ġto Ġexhaustive Ġcomput ations Ġif Ġthe Ġnumber Ġof Ġmentions Ġis Ġlarge . ĠBased Ġon Ġthe Ġobservation Ġthat Ġthe Ġconsistency Ġatten uates Ġalong Ġwith Ġthe Ġdistance Ġbetween Ġtwo Ġmentions , Ġwe Ġargue Ġthat Ġthe Ġadjacent Ġmentions Ġmight Ġbe Ġsufficient Ġfor Ġsupporting Ġthe Ġassumption Ġefficiently . </s> Form ally , Ġwe Ġdefine Ġneighbor Ġmentions Ġas ĠIN LINE FORM 0 Ġadjacent Ġmentions Ġbefore Ġand Ġafter Ġcurrent Ġmention ĠIN LINE FORM 1 Ġ: ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġis Ġthe Ġpre - defined Ġwindow Ġsize . ĠThus , Ġthe Ġtopical Ġco herence Ġat Ġdocument Ġlevel Ġshall Ġbe Ġachieved Ġin Ġa Ġchain - like Ġway . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ( ĠIN LINE FORM 4 Ġ), Ġmentions ĠHussain Ġand ĠEssex , Ġa Ġcricket Ġplayer Ġand Ġthe Ġcricket Ġclub , Ġprovide Ġadequate Ġdis amb ig uation Ġclues Ġto Ġinduce Ġthe Ġunderlying Ġtopic ĠâĢ ľ c rick et " Ġfor Ġthe Ġcurrent Ġmention ĠEngland , Ġwhich Ġimpacts Ġpositively Ġon Ġidentifying Ġthe Ġmention Ġsur rey Ġas Ġanother Ġcricket Ġclub Ġvia Ġthe Ġcommon Ġneighbor Ġmention ĠEssex . </s> A Ġdegraded Ġcase Ġhappens Ġif ĠIN LINE FORM 0 Ġis Ġlarge Ġenough Ġto Ġcover Ġthe Ġentire Ġdocument , Ġand Ġthe Ġmentions Ġused Ġfor Ġglobal Ġfeatures Ġbecome Ġthe Ġsame Ġas Ġthe Ġprevious Ġwork , Ġsuch Ġas ĠB IB REF 21 Ġ. ĠIn Ġexperiments , Ġwe Ġhe ur istically Ġfound Ġa Ġsuitable ĠIN LINE FORM 1 Ġwhich Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġtotal Ġnumber Ġof Ġmentions . ĠThe Ġbenefits Ġof Ġefficiency Ġare Ġin Ġtwo Ġways : Ġ( i ) Ġto Ġdecrease Ġtime Ġcomplexity , Ġand Ġ( ii ) Ġto Ġtrim Ġthe Ġentity Ġgraph Ġinto Ġa Ġfixed Ġsize Ġof Ġsub graph Ġthat Ġfacilitates Ġcomputation Ġacceleration Ġthrough ĠGPUs Ġand Ġbatch Ġtechniques , Ġwhich Ġwill Ġbe Ġdiscussed Ġin ĠSection ĠSEC REF 24 Ġ. </s> Given Ġneighbor Ġmentions ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġtwo Ġtypes Ġof Ġvector ial Ġglobal Ġfeatures Ġand Ġstructured Ġglobal Ġfeatures Ġfor Ġeach Ġcandidate ĠIN LINE FORM 1 Ġ: </s> Neigh bor ĠM ention ĠCompatibility ĠSuppose Ġneighbor Ġmentions Ġare Ġtopical Ġcoherent , Ġa Ġcandidate Ġentity Ġshall Ġalso Ġbe Ġcompatible Ġwith Ġneighbor Ġmentions Ġif Ġit Ġhas Ġa Ġhigh Ġcompatibility Ġscore Ġwith Ġthe Ġcurrent Ġmention , Ġotherwise Ġnot . ĠThat Ġis , Ġwe Ġextract Ġthe Ġvector ial Ġglobal Ġfeatures Ġby Ġcomputing Ġthe Ġsimilarities Ġbetween ĠIN LINE FORM 0 Ġand Ġall Ġneighbor Ġmentions : ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġis Ġthe Ġmention Ġembed ding Ġby Ġaveraging Ġthe Ġglobal Ġvectors Ġof Ġwords Ġin Ġits Ġsurface Ġform : ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġare Ġtoken ized Ġwords Ġof Ġmention ĠIN LINE FORM 5 Ġ. </s> Sub graph ĠStructure ĠThe Ġabove Ġfeatures Ġreflect Ġthe Ġconsistent Ġsemantics Ġin Ġtexts Ġ( i . e ., Ġmentions ). ĠWe Ġnow Ġextract Ġstructured Ġglobal Ġfeatures Ġusing Ġthe Ġrelations Ġin ĠKB , Ġwhich Ġfacilitates Ġthe Ġinference Ġamong Ġcandidates Ġto Ġfind Ġthe Ġmost Ġtopical Ġcoherent Ġsubset . ĠFor Ġeach Ġdocument , Ġwe Ġobtain Ġthe Ġentity Ġgraph ĠIN LINE FORM 0 Ġby Ġtaking Ġcandidate Ġentities Ġof Ġall Ġmentions ĠIN LINE FORM 1 Ġas Ġnodes , Ġand Ġusing Ġentity Ġembed d ings Ġto Ġcompute Ġtheir Ġsimilarities Ġas Ġedges ĠIN LINE FORM 2 Ġ. ĠThen , Ġwe Ġextract Ġthe Ġsub graph Ġstructured Ġfeatures ĠIN LINE FORM 3 Ġfor Ġeach Ġentity ĠIN LINE FORM 4 Ġfor Ġefficiency . </s> Form ally , Ġwe Ġdefine Ġthe Ġsub graph Ġas : ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ. ĠFor Ġexample Ġ( Figure ĠFIG REF 1 Ġ), Ġfor Ġentity ĠEngland Ġcricket Ġteam , Ġthe Ġsub graph Ġcontains Ġthe Ġrelation Ġfrom Ġit Ġto Ġall Ġcandidates Ġof Ġneighbor Ġmentions : ĠEngland Ġcricket Ġteam , ĠN asser ĠHussain Ġ( rug by Ġunion ), ĠN asser ĠHussain , ĠEssex , ĠEssex ĠCounty ĠCricket ĠClub Ġand ĠEssex , ĠNew ĠYork . ĠTo Ġsupport Ġbatch - wise Ġacceleration , Ġwe Ġrepresent ĠIN LINE FORM 2 Ġin Ġthe Ġform Ġof Ġadj ac ency Ġtable Ġbased Ġvectors : ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġnumber Ġof Ġcandidates Ġper Ġmention . </s> Finally , Ġfor Ġeach Ġcandidate ĠIN LINE FORM 0 Ġ, Ġwe Ġconc aten ate Ġlocal Ġfeatures Ġand Ġneighbor Ġmention Ġcompatibility Ġscores Ġas Ġthe Ġfeature Ġvector ĠIN LINE FORM 1 Ġ, Ġand Ġconstruct Ġthe Ġsub graph Ġstructure Ġrepresentation ĠIN LINE FORM 2 Ġas Ġthe Ġinputs Ġof ĠNC EL . </s> Ne ural ĠCollective ĠEntity ĠLink ing </s> NC EL Ġincorporates ĠGC N Ġinto Ġa Ġdeep Ġneural Ġnetwork Ġto Ġutilize Ġstructured Ġgraph Ġinformation Ġfor Ġcollectively Ġfeature Ġabstraction , Ġwhile Ġdiffers Ġfrom Ġconventional ĠGC N Ġin Ġthe Ġway Ġof Ġapplying Ġthe Ġgraph . ĠInstead Ġof Ġthe Ġentire Ġgraph , Ġonly Ġa Ġsubset Ġof Ġnodes Ġis ĠâĢ ľ visible " Ġto Ġeach Ġnode Ġin Ġour Ġproposed Ġmethod , Ġand Ġthen Ġthe Ġoverall Ġstructured Ġinformation Ġshall Ġbe Ġreached Ġin Ġa Ġchain - like Ġway . ĠFix ing Ġthe Ġsize Ġof Ġthe Ġsubset , ĠNC EL Ġis Ġfurther Ġspeed ed Ġup Ġby Ġbatch Ġtechniques Ġand ĠGPUs , Ġand Ġis Ġefficient Ġto Ġlarge - scale Ġdata . </s> Graph ĠConv olution al ĠNetwork </s> GC Ns Ġare Ġa Ġtype Ġof Ġneural Ġnetwork Ġmodel Ġthat Ġdeals Ġwith Ġstructured Ġdata . ĠIt Ġtakes Ġa Ġgraph Ġas Ġan Ġinput Ġand Ġoutput Ġlabels Ġfor Ġeach Ġnode . ĠAs Ġa Ġsimpl ification Ġof Ġspectral Ġgraph Ġconv olutions , Ġthe Ġmain Ġidea Ġof ĠB IB REF 26 Ġis Ġsimilar Ġto Ġa Ġpropagation Ġmodel : Ġto Ġenhance Ġthe Ġfeatures Ġof Ġa Ġnode Ġaccording Ġto Ġits Ġneighbor Ġnodes . ĠThe Ġformulation Ġis Ġas Ġfollows : ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġadjacent Ġmatrix Ġof Ġthe Ġinput Ġgraph Ġwith Ġself - connection , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġhidden Ġstates Ġand Ġweights Ġin Ġthe ĠIN LINE FORM 3 Ġ- th Ġlayer , Ġand ĠIN LINE FORM 4 Ġis Ġa Ġnon - linear Ġactivation , Ġsuch Ġas ĠRe Lu . </s> Model ĠArchitecture </s> As Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, ĠNC EL Ġidentifies Ġthe Ġcorrect Ġcandidate ĠIN LINE FORM 0 Ġfor Ġthe Ġmention ĠIN LINE FORM 1 Ġby Ġusing Ġvector ial Ġfeatures Ġas Ġwell Ġas Ġstructured Ġrelated ness Ġwith Ġcandidates Ġof Ġneighbor Ġmentions ĠIN LINE FORM 2 Ġ. ĠGiven Ġfeature Ġvector ĠIN LINE FORM 3 Ġand Ġsub graph Ġrepresentation ĠIN LINE FORM 4 Ġof Ġeach Ġcandidate ĠIN LINE FORM 5 Ġ, Ġwe Ġstack Ġthem Ġas Ġinputs Ġfor Ġmention ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 Ġ, Ġand Ġthe Ġadjacent Ġmatrix ĠIN LINE FORM 8 Ġ, Ġwhere ĠIN LINE FORM 9 Ġdenotes Ġthe Ġsub graph Ġwith Ġself - connection . ĠWe Ġnormal ize ĠIN LINE FORM 10 Ġsuch Ġthat Ġall Ġrows Ġsum Ġto Ġone , Ġden oted Ġas ĠIN LINE FORM 11 Ġ, Ġavoiding Ġthe Ġchange Ġin Ġthe Ġscale Ġof Ġthe Ġfeature Ġvectors . </s> Given ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġthe Ġgoal Ġof ĠNC EL Ġis Ġto Ġfind Ġthe Ġbest Ġassignment : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġoutput Ġvariable Ġof Ġcandidates , Ġand ĠIN LINE FORM 1 Ġis Ġa Ġprobability Ġfunction Ġas Ġfollows : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġscore Ġfunction Ġparameters Ġby ĠIN LINE FORM 1 Ġ. ĠNC EL Ġlearns Ġthe Ġmapping ĠIN LINE FORM 2 Ġthrough Ġa Ġneural Ġnetwork Ġincluding Ġthree Ġmain Ġmodules : Ġenc oder , Ġsub - graph Ġconv olution Ġnetwork Ġ( sub - GC N ) Ġand Ġdec oder . ĠNext , Ġwe Ġintroduce Ġthem Ġin Ġturn . </s> Enc oder ĠThe Ġfunction Ġof Ġthis Ġmodule Ġis Ġto Ġintegrate Ġdifferent Ġfeatures Ġby Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ): ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstates Ġof Ġthe Ġcurrent Ġmention , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters Ġand Ġbias . ĠWe Ġuse ĠRe Lu Ġas Ġthe Ġnon - linear Ġactivation ĠIN LINE FORM 3 Ġ. </s> Sub - Graph ĠConv olution ĠNetwork ĠSimilar Ġto ĠGC N , Ġthis Ġmodule Ġlearns Ġto Ġabstract Ġfeatures Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġmention Ġitself Ġas Ġwell Ġas Ġits Ġneighbors . ĠSuppose ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstates Ġof Ġthe Ġneighbor ĠIN LINE FORM 1 Ġ, Ġwe Ġstack Ġthem Ġto Ġexpand Ġthe Ġcurrent Ġhidden Ġstates Ġof ĠIN LINE FORM 2 Ġas ĠIN LINE FORM 3 Ġ, Ġsuch Ġthat Ġeach Ġrow Ġcorresponds Ġto Ġthat Ġin Ġthe Ġsub graph Ġadjacent Ġmatrix ĠIN LINE FORM 4 Ġ. ĠWe Ġdefine Ġsub - graph Ġconv olution Ġas : ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġtrain able Ġparameter . </s> Dec oder ĠAfter ĠIN LINE FORM 0 Ġiterations Ġof Ġsub - graph Ġconv olution , Ġthe Ġhidden Ġstates Ġintegrate Ġboth Ġfeatures Ġof ĠIN LINE FORM 1 Ġand Ġits Ġneighbors . ĠA Ġfully Ġconnected Ġdec oder Ġmaps ĠIN LINE FORM 2 Ġto Ġthe Ġnumber Ġof Ġcandidates Ġas Ġfollows : ĠIN LINE FORM 3 </s> where ĠIN LINE FORM 0 Ġ. </s> Training </s> The Ġparameters Ġof Ġnetwork Ġare Ġtrained Ġto Ġminimize Ġcross - ent ropy Ġof Ġthe Ġpredicted Ġand Ġground Ġtruth ĠIN LINE FORM 0 Ġ: ĠIN LINE FORM 1 </s> Supp ose Ġthere Ġare ĠIN LINE FORM 0 Ġdocuments Ġin Ġtraining Ġcorpus , Ġeach Ġdocument Ġhas Ġa Ġset Ġof Ġmentions ĠIN LINE FORM 1 Ġ, Ġleading Ġto Ġtotally ĠIN LINE FORM 2 Ġmention Ġsets . ĠThe Ġoverall Ġobjective Ġfunction Ġis Ġas Ġfollows : ĠIN LINE FORM 3 </s> Exper iments </s> To Ġavoid Ġover fitting Ġwith Ġsome Ġdataset , Ġwe Ġtrain ĠNC EL Ġusing Ġcollected ĠWikipedia Ġhyper links Ġinstead Ġof Ġspecific Ġannot ated Ġdata . ĠWe Ġthen Ġevaluate Ġthe Ġtrained Ġmodel Ġon Ġfive Ġdifferent Ġbenchmarks Ġto Ġverify Ġthe Ġlinking Ġprecision Ġas Ġwell Ġas Ġthe Ġgeneral ization Ġability . ĠFurthermore , Ġwe Ġinvestigate Ġthe Ġeffectiveness Ġof Ġkey Ġmodules Ġin ĠNC EL Ġand Ġgive Ġqualitative Ġresults Ġfor Ġcomprehensive Ġanalysis . </s> Bas elines Ġand ĠDat as ets </s> We Ġcompare ĠNC EL Ġwith Ġthe Ġfollowing Ġstate - of - the - art ĠEL Ġmethods Ġincluding Ġthree Ġlocal Ġmodels Ġand Ġthree Ġtypes Ġof Ġglobal Ġmodels : </s> Local Ġmodels : ĠHe ĠB IB REF 29 Ġand ĠCh ish ol m ĠB IB REF 6 Ġbeat Ġmany Ġglobal Ġmodels Ġby Ġusing Ġauto - enc od ers Ġand Ġweb Ġlinks , Ġrespectively , Ġand ĠNT EE ĠB IB REF 16 Ġachieves Ġthe Ġbest Ġperformance Ġbased Ġon Ġjoint Ġembed d ings Ġof Ġwords Ġand Ġentities . </s> Iter ative Ġmodel : ĠA IDA ĠB IB REF 22 Ġlinks Ġentities Ġby Ġiter atively Ġfinding Ġa Ġdense Ġsub graph . </s> Lo opy ĠBelief ĠProp ag ation : ĠGlob erson ĠB IB REF 18 Ġand ĠP Bo H ĠB IB REF 30 Ġintroduce ĠL BP ĠB IB REF 31 Ġtechniques Ġfor Ġcollective Ġinference , Ġand ĠG ane a ĠB IB REF 24 Ġsolves Ġthe Ġglobal Ġtraining Ġproblem Ġvia Ġtrunc ated Ġfitting ĠL BP . </s> Page Rank / Random ĠWalk : ĠBoost ing ĠB IB REF 32 Ġ, ĠAG D IST IS G ĠB IB REF 33 Ġ, ĠBab elf y ĠB IB REF 34 Ġ, ĠW AT ĠB IB REF 35 Ġ, Ġx Lisa ĠB IB REF 36 Ġand ĠW N ED ĠB IB REF 19 Ġperforms ĠPage Rank ĠB IB REF 37 Ġor Ġrandom Ġwalk ĠB IB REF 38 Ġon Ġthe Ġmention - entity Ġgraph Ġand Ġuse Ġthe Ġconvergence Ġscore Ġfor Ġdis amb ig uation . </s> For Ġfairly Ġcomparison , Ġwe Ġreport Ġthe Ġoriginal Ġscores Ġof Ġthe Ġbas elines Ġin Ġthe Ġpapers . ĠFollowing Ġthese Ġmethods , Ġwe Ġevaluate ĠNC EL Ġon Ġthe Ġfollowing Ġfive Ġdatasets : Ġ( 1 ) ĠCo N LL - Y AG O ĠB IB REF 22 Ġ: Ġthe ĠCo N LL Ġ2003 Ġshared Ġtask Ġincluding Ġtest a Ġof Ġ4 79 1 Ġmentions Ġin Ġ216 Ġdocuments , Ġand Ġtest b Ġof Ġ4 485 Ġmentions Ġin Ġ213 Ġdocuments . Ġ( 2 ) ĠT AC 2010 ĠB IB REF 39 Ġ: Ġconstructed Ġfor Ġthe ĠText ĠAnalysis ĠConference Ġthat Ġcomprises Ġ6 76 Ġmentions Ġin Ġ352 Ġdocuments Ġfor Ġtesting . Ġ( 3 ) ĠACE 2004 ĠB IB REF 23 Ġ: Ġa Ġsubset Ġof ĠACE 2004 Ġco - reference Ġdocuments Ġincluding Ġ248 Ġmentions Ġin Ġ35 Ġdocuments , Ġwhich Ġis Ġannot ated Ġby ĠAmazon ĠMechanical ĠTurk . Ġ( 4 ) ĠA QU A INT ĠB IB REF 40 Ġ: Ġ50 Ġnews Ġarticles Ġincluding Ġ6 99 Ġmentions Ġfrom Ġthree Ġdifferent Ġnews Ġagencies . Ġ( 5 ) ĠWW ĠB IB REF 19 Ġ: Ġa Ġnew Ġbenchmark Ġwith Ġbalanced Ġprior Ġdistributions Ġof Ġmentions , Ġleading Ġto Ġa Ġhard Ġcase Ġof Ġdis amb ig uation . ĠIt Ġhas Ġ6 374 Ġmentions Ġin Ġ310 Ġdocuments Ġautomatically Ġextracted Ġfrom ĠWikipedia . </s> Training ĠDetails Ġand ĠRunning ĠTime ĠAnalysis </s> Training ĠWe Ġcollect Ġ50 , 000 ĠWikipedia Ġarticles Ġaccording Ġto Ġthe Ġnumber Ġof Ġits Ġhyper links Ġas Ġour Ġtraining Ġdata . ĠFor Ġefficiency , Ġwe Ġtrim Ġthe Ġarticles Ġto Ġthe Ġfirst Ġthree Ġparagraphs Ġleading Ġto Ġ1 , 035 , 665 Ġmentions Ġin Ġtotal . ĠUsing ĠCo N LL - Test ĠA Ġas Ġthe Ġdevelopment Ġset , Ġwe Ġevaluate Ġthe Ġtrained ĠNC EL Ġon Ġthe Ġabove Ġbenchmarks . ĠWe Ġset Ġcontext Ġwindow Ġto Ġ20 , Ġneighbor Ġmention Ġwindow Ġto Ġ6 , Ġand Ġtop ĠIN LINE FORM 0 Ġcandidates Ġfor Ġeach Ġmention . ĠWe Ġuse Ġtwo Ġlayers Ġwith Ġ2000 Ġand Ġ1 Ġhidden Ġunits Ġin ĠML P Ġenc oder , Ġand Ġ3 Ġlayers Ġin Ġsub - GC N . ĠWe Ġuse Ġearly Ġstop Ġand Ġfine Ġtune Ġthe Ġembed d ings . ĠWith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġnearly Ġ3 Ġepoch s Ġcost Ġless Ġthan Ġ15 Ġminutes Ġon Ġthe Ġserver Ġwith Ġ20 Ġcore ĠCPU Ġand Ġthe ĠGeForce ĠGTX Ġ1080 Ti ĠGPU Ġwith Ġ12 Gb Ġmemory . ĠWe Ġuse Ġstandard ĠPrecision , ĠRecall Ġand ĠF 1 Ġat Ġmention Ġlevel Ġ( Micro ) Ġand Ġat Ġthe Ġdocument Ġlevel Ġ( Mac ro ) Ġas Ġmeasurements . </s> Com plex ity ĠAnalysis ĠCompared Ġwith Ġlocal Ġmethods , Ġthe Ġmain Ġdisadvantage Ġof Ġcollective Ġmethods Ġis Ġhigh Ġcomplexity Ġand Ġexpensive Ġcosts . ĠSuppose Ġthere Ġare ĠIN LINE FORM 0 Ġmentions Ġin Ġdocuments Ġon Ġaverage , Ġamong Ġthese Ġglobal Ġmodels , ĠNC EL Ġnot Ġsurprisingly Ġhas Ġthe Ġlowest Ġtime Ġcomplexity ĠIN LINE FORM 1 Ġsince Ġit Ġonly Ġconsiders Ġadjacent Ġmentions , Ġwhere ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġsub - GC N Ġlayers Ġindicating Ġthe Ġiterations Ġuntil Ġconvergence . ĠA IDA Ġhas Ġthe Ġhighest Ġtime Ġcomplexity ĠIN LINE FORM 3 Ġin Ġworst Ġcase Ġdue Ġto Ġexhaustive Ġiter atively Ġfinding Ġand Ġsorting Ġthe Ġgraph . ĠThe ĠL BP Ġand ĠPage Rank / random Ġwalk Ġbased Ġmethods Ġachieve Ġsimilar Ġhigh Ġtime Ġcomplexity Ġof ĠIN LINE FORM 4 Ġmainly Ġbecause Ġof Ġthe Ġinference Ġon Ġthe Ġentire Ġgraph . </s> Results Ġon ĠGER BIL </s> GER BIL ĠB IB REF 41 Ġis Ġa Ġbenchmark Ġentity Ġannotation Ġframework Ġthat Ġaims Ġto Ġprovide Ġa Ġunified Ġcomparison Ġamong Ġdifferent ĠEL Ġmethods Ġacross Ġdatasets Ġincluding ĠACE 2004 , ĠA QU A INT Ġand ĠCo N LL . ĠWe Ġcompare ĠNC EL Ġwith Ġthe Ġglobal Ġmodels Ġthat Ġreport Ġthe Ġperformance Ġon ĠGER BIL . </s> As Ġshown Ġin ĠTable ĠT AB REF 26 Ġ, ĠNC EL Ġachieves Ġthe Ġbest Ġperformance Ġin Ġmost Ġcases Ġwith Ġan Ġaverage Ġgain Ġof Ġ2 % Ġon ĠMicro ĠF 1 Ġand Ġ3 % ĠMacro ĠF 1 . ĠThe Ġbaseline Ġmethods Ġalso Ġachieve Ġcompetitive Ġresults Ġon Ġsome Ġdatasets Ġbut Ġfail Ġto Ġadapt Ġto Ġthe Ġothers . ĠFor Ġexample , ĠA IDA Ġand Ġx Lisa Ġperform Ġquite Ġwell Ġon ĠACE 2004 Ġbut Ġpoorly Ġon Ġother Ġdatasets , Ġor ĠW AT , ĠP Bo H , Ġand ĠW N ED Ġhave Ġa Ġfavorable Ġperformance Ġon ĠCo N LL Ġbut Ġlower Ġvalues Ġon ĠACE 2004 Ġand ĠA QU A INT . ĠOur Ġproposed Ġmethod Ġperforms Ġconsistently Ġwell Ġon Ġall Ġdatasets Ġthat Ġdemonstrates Ġthe Ġgood Ġgeneral ization Ġability . </s> Results Ġon ĠT AC 2010 Ġand ĠWW </s> In Ġthis Ġsection , Ġwe Ġinvestigate Ġthe Ġeffectiveness Ġof ĠNC EL Ġin Ġthe ĠâĢ ľ easy " Ġand ĠâĢ ľ hard " Ġdatasets , Ġrespectively . ĠParticularly , ĠT AC 2010 , Ġwhich Ġhas Ġtwo Ġmentions Ġper Ġdocument Ġon Ġaverage Ġ( Section ĠSEC REF 19 Ġ) Ġand Ġhigh Ġprior Ġprobabilities Ġof Ġcorrect Ġcandidates Ġ( Figure ĠFIG REF 28 Ġ), Ġis Ġregarded Ġas Ġthe ĠâĢ ľ easy " Ġcase Ġfor ĠEL , Ġand ĠWW Ġis Ġthe ĠâĢ ľ hard " Ġcase Ġsince Ġit Ġhas Ġthe Ġmost Ġmentions Ġwith Ġbalanced Ġprior Ġprobabilities ĠB IB REF 19 Ġ. ĠBesides , Ġwe Ġfurther Ġcompare Ġthe Ġimpact Ġof Ġkey Ġmodules Ġby Ġremoving Ġthe Ġfollowing Ġpart Ġfrom ĠNC EL : Ġglobal Ġfeatures Ġ( NC EL - local ), Ġattention Ġ( NC EL - no att ), Ġembed ding Ġfeatures Ġ( NC EL - no emb ), Ġand Ġthe Ġimpact Ġof Ġthe Ġprior Ġprobability Ġ( pri or ). </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠFIG REF 28 Ġand ĠTable ĠFIG REF 28 Ġ. ĠWe Ġcan Ġsee Ġthe Ġaverage Ġlinking Ġprecision Ġ( Micro ) Ġof ĠWW Ġis Ġlower Ġthan Ġthat Ġof ĠT AC 2010 , Ġand ĠNC EL Ġoutper forms Ġall Ġbaseline Ġmethods Ġin Ġboth Ġeasy Ġand Ġhard Ġcases . ĠIn Ġthe ĠâĢ ľ easy " Ġcase , Ġlocal Ġmodels Ġhave Ġsimilar Ġperformance Ġwith Ġglobal Ġmodels Ġsince Ġonly Ġlittle Ġglobal Ġinformation Ġis Ġavailable Ġ( 2 Ġmentions Ġper Ġdocument ). ĠBesides , ĠN N - based Ġmodels , ĠNT EE Ġand ĠNC EL - local , Ġperform Ġsignificantly Ġbetter Ġthan Ġothers Ġincluding Ġmost Ġglobal Ġmodels , Ġdemonstrating Ġthat Ġthe Ġeffectiveness Ġof Ġneural Ġmodels Ġdeals Ġwith Ġthe Ġfirst Ġlimitation Ġin Ġthe Ġintroduction . </s> Imp act Ġof ĠNC EL ĠMod ules </s> As Ġshown Ġin ĠFigure ĠFIG REF 28 Ġ, Ġthe Ġprior Ġprobability Ġperforms Ġquite Ġwell Ġin ĠT AC 2010 Ġbut Ġpoorly Ġin ĠWW . ĠCompared Ġwith ĠNC EL - local , Ġthe Ġglobal Ġmodule Ġin ĠNC EL Ġbrings Ġmore Ġimprovements Ġin Ġthe ĠâĢ ľ hard " Ġcase Ġthan Ġthat Ġfor ĠâĢ ľ easy " Ġdataset , Ġbecause Ġlocal Ġfeatures Ġare Ġdiscrim in ative Ġenough Ġin Ġmost Ġcases Ġof ĠT AC 2010 , Ġand Ġglobal Ġinformation Ġbecomes Ġquite Ġhelpful Ġwhen Ġlocal Ġfeatures Ġcannot Ġhandle . ĠThat Ġis , Ġour Ġpropose Ġcollective Ġmodel Ġis Ġrobust Ġand Ġshows Ġa Ġgood Ġgeneral ization Ġability Ġto Ġdifficult ĠEL . ĠThe Ġimprovements Ġby Ġeach Ġmain Ġmodule Ġare Ġrelatively Ġsmall Ġin ĠT AC 2010 , Ġwhile Ġthe Ġmodules Ġof Ġattention Ġand Ġembed ding Ġfeatures Ġshow Ġnon - neg lig ible Ġimpacts Ġin ĠWW Ġ( even Ġworse Ġthan Ġlocal Ġmodel ), Ġmainly Ġbecause ĠWW Ġcontains Ġmuch Ġnoise , Ġand Ġthese Ġtwo Ġmodules Ġare Ġeffective Ġin Ġimproving Ġthe Ġrobust ness Ġto Ġnoise Ġand Ġthe Ġability Ġof Ġgeneral ization Ġby Ġselecting Ġinformative Ġwords Ġand Ġproviding Ġmore Ġaccurate Ġsemantics , Ġrespectively . </s> Qual itative ĠAnalysis </s> The Ġresults Ġof Ġexample Ġin ĠFigure ĠFIG REF 1 Ġare Ġshown Ġin ĠTable ĠT AB REF 30 Ġ, Ġwhich Ġis Ġfrom ĠCo N LL Ġtest a Ġdataset . ĠFor Ġmention ĠEssex , Ġalthough Ġboth ĠNC EL Ġand ĠNC EL - local Ġcorrectly Ġidentify Ġentity ĠEssex ĠCounty ĠCricket ĠClub , ĠNC EL Ġoutputs Ġhigher Ġprobability Ġdue Ġto Ġthe Ġenhancement Ġof Ġneighbor Ġmentions . ĠMoreover , Ġfor Ġmention ĠEngland , ĠNC EL - local Ġcannot Ġfind Ġenough Ġdis amb ig uation Ġclues Ġfrom Ġits Ġcontext Ġwords , Ġsuch Ġas Ġsurplus Ġand Ġrequirements , Ġand Ġthus Ġassigns Ġa Ġhigher Ġprobability Ġof Ġ0 . 42 Ġto Ġthe Ġcountry ĠEngland Ġaccording Ġto Ġthe Ġprior Ġprobability . ĠCollect ively , ĠNC EL Ġcorrectly Ġidentifies ĠEngland Ġcricket Ġteam Ġwith Ġa Ġprobability Ġof Ġ0 . 72 Ġas Ġcompared Ġwith Ġ0 . 20 Ġin ĠNC EL - local Ġwith Ġthe Ġhelp Ġof Ġits Ġneighbor Ġmention ĠEssex . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġneural Ġmodel Ġfor Ġcollective Ġentity Ġlinking Ġthat Ġis Ġend - to - end Ġtrain able . ĠIt Ġapplies ĠGC N Ġon Ġsub graph s Ġinstead Ġof Ġthe Ġentire Ġentity Ġgraph Ġto Ġefficiently Ġlearn Ġfeatures Ġfrom Ġboth Ġlocal Ġand Ġglobal Ġinformation . ĠWe Ġdesign Ġan Ġattention Ġmechanism Ġthat Ġend ows ĠNC EL Ġrobust Ġto Ġnoisy Ġdata . ĠTr ained Ġon Ġcollected ĠWikipedia Ġhyper links , ĠNC EL Ġoutper forms Ġthe Ġstate - of - the - art Ġcollective Ġmethods Ġacross Ġfive Ġdifferent Ġdatasets . ĠBesides , Ġfurther Ġanalysis Ġof Ġthe Ġimpacts Ġof Ġmain Ġmodules Ġas Ġwell Ġas Ġqualitative Ġresults Ġdemonstrates Ġits Ġeffectiveness . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġextend Ġour Ġmethod Ġinto Ġcross - ling ual Ġsettings Ġto Ġhelp Ġlink Ġentities Ġin Ġlow - res ourced Ġlanguages Ġby Ġexploiting Ġrich Ġknowledge Ġfrom Ġhigh - res ourced Ġlanguages , Ġand Ġdeal Ġwith ĠN IL Ġentities Ġto Ġfacilitate Ġspecific Ġapplications . </s> Acknowled gments </s> The Ġwork Ġis Ġsupported Ġby ĠNational ĠKey ĠResearch Ġand ĠDevelopment ĠProgram Ġof ĠChina Ġ( 2017 Y FB 100 21 01 ), ĠNS FC Ġkey Ġproject Ġ( U 17 36 204 , Ġ6 166 11 46 007 ), Ġand ĠTH UN US ĠN Ex T ĠCo - Lab .
Introduction This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ Entity linking (EL), mapping entity mentions in texts to a given knowledge base (KB), serves as a fundamental role in many fields, such as question answering BIBREF0 , semantic search BIBREF1 , and information extraction BIBREF2 , BIBREF3 . However, this task is non-trivial because entity mentions are usually ambiguous. As shown in Figure FIGREF1 , the mention England refers to three entities in KB, and an entity linking system should be capable of identifying the correct entity as England cricket team rather than England and England national football team. Entity linking is typically broken down into two main phases: (i) candidate generation obtains a set of referent entities in KB for each mention, and (ii) named entity disambiguation selects the possible candidate entity by solving a ranking problem. The key challenge lies in the ranking model that computes the relevance between candidates and the corresponding mentions based on the information both in texts and KBs BIBREF4 . In terms of the features used for ranking, we classify existing EL models into two groups: local models to resolve mentions independently relying on textual context information from the surrounding words BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , and global (collective) models, which are the main focus of this paper, that encourage the target entities of all mentions in a document to be topically coherent BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Global models usually build an entity graph based on KBs to capture coherent entities for all identified mentions in a document, where the nodes are entities, and edges denote their relations. The graph provides highly discriminative semantic signals (e.g., entity relatedness) that are unavailable to local model BIBREF15 . For example (Figure FIGREF1 ), an EL model seemly cannot find sufficient disambiguation clues for the mention England from its surrounding words, unless it utilizes the coherence information of consistent topic “cricket" among adjacent mentions England, Hussain, and Essex. Although the global model has achieved significant improvements, its limitation is threefold: To mitigate the first limitation, recent EL studies introduce neural network (NN) models due to its amazing feature abstraction and generalization ability. In such models, words/entities are represented by low dimensional vectors in a continuous space, and features for mention as well as candidate entities are automatically learned from data BIBREF4 . However, existing NN-based methods for EL are either local models BIBREF16 , BIBREF17 or merely use word/entity embeddings for feature extraction and rely on another modules for collective disambiguation, which thus cannot fully utilize the power of NN models for collective EL BIBREF18 , BIBREF19 , BIBREF20 . The second drawback of the global approach has been alleviated through approximate optimization techniques, such as PageRank/random walks BIBREF21 , graph pruning BIBREF22 , ranking SVMs BIBREF23 , or loopy belief propagation (LBP) BIBREF18 , BIBREF24 . However, these methods are not differentiable and thus difficult to be integrated into neural network models (the solution for the first limitation). To overcome the third issue of inadequate training data, BIBREF17 has explored a massive amount of hyperlinks in Wikipedia, but these potential annotations for EL contain much noise, which may distract a naive disambiguation model BIBREF6 . In this paper, we propose a novel Neural Collective Entity Linking model (NCEL), which performs global EL combining deep neural networks with Graph Convolutional Network (GCN) BIBREF25 , BIBREF26 that allows flexible encoding of entity graphs. It integrates both local contextual information and global interdependence of mentions in a document, and is efficiently trainable in an end-to-end fashion. Particularly, we introduce attention mechanism to robustly model local contextual information by selecting informative words and filtering out the noise. On the other hand, we apply GCNs to improve discriminative signals of candidate entities by exploiting the rich structure underlying the correct entities. To alleviate the global computations, we propose to convolute on the subgraph of adjacent mentions. Thus, the overall coherence shall be achieved in a chain-like way via a sliding window over the document. To the best of our knowledge, this is the first effort to develop a unified model for neural collective entity linking. In experiments, we first verify the efficiency of NCEL via theoretically comparing its time complexity with other collective alternatives. Afterwards, we train our neural model using collected Wikipedia hyperlinks instead of dataset-specific annotations, and perform evaluations on five public available benchmarks. The results show that NCEL consistently outperforms various baselines with a favorable generalization ability. Finally, we further present the performance on a challenging dataset WW BIBREF19 as well as qualitative results, investigating the effectiveness of each key module. Preliminaries and Framework We denote INLINEFORM0 as a set of entity mentions in a document INLINEFORM1 , where INLINEFORM2 is either a word INLINEFORM3 or a mention INLINEFORM4 . INLINEFORM5 is the entity graph for document INLINEFORM6 derived from the given knowledge base, where INLINEFORM7 is a set of entities, INLINEFORM8 denotes the relatedness between INLINEFORM9 and higher values indicate stronger relations. Based on INLINEFORM10 , we extract a subgraph INLINEFORM11 for INLINEFORM12 , where INLINEFORM13 denotes the set of candidate entities for INLINEFORM14 . Note that we don't include the relations among candidates of the same mention in INLINEFORM15 because these candidates are mutually exclusive in disambiguation. Formally, we define the entity linking problem as follows: Given a set of mentions INLINEFORM0 in a document INLINEFORM1 , and an entity graph INLINEFORM2 , the goal is to find an assignment INLINEFORM3 . To collectively find the best assignment, NCEL aims to improve the discriminability of candidates' local features by using entity relatedness within a document via GCN, which is capable of learning a function of features on the graph through shared parameters over all nodes. Figure FIGREF10 shows the framework of NCEL including three main components: Example As shown in Figure FIGREF10 , for the current mention England, we utilize its surrounding words as local contexts (e.g., surplus), and adjacent mentions (e.g., Hussian) as global information. Collectively, we utilize the candidates of England INLINEFORM0 as well as those entities of its adjacencies INLINEFORM1 to construct feature vectors for INLINEFORM2 and the subgraph of relatedness as inputs of our neural model. Let darker blue indicate higher probability of being predicted, the correct candidate INLINEFORM3 becomes bluer due to its bluer neighbor nodes of other mentions INLINEFORM4 . The dashed lines denote entity relations that have indirect impacts through the sliding adjacent window , and the overall structure shall be achieved via multiple sub-graphs by traversing all mentions. Before introducing our model, we first describe the component of candidate generation. Candidate Generation Similar to previous work BIBREF24 , we use the prior probability INLINEFORM0 of entity INLINEFORM1 conditioned on mention INLINEFORM2 both as a local feature and to generate candidate entities: INLINEFORM3 . We compute INLINEFORM4 based on statistics of mention-entity pairs from: (i) Wikipedia page titles, redirect titles and hyperlinks, (ii) the dictionary derived from a large Web Corpus BIBREF27 , and (iii) the YAGO dictionary with a uniform distribution BIBREF22 . We pick up the maximal prior if a mention-entity pair occurs in different resources. In experiments, to optimize for memory and run time, we keep only top INLINEFORM5 entities based on INLINEFORM6 . In the following two sections, we will present the key components of NECL, namely feature extraction and neural network for collective entity linking. Feature Extraction The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness. Therefore, we use joint embeddings of words and entities at sense level BIBREF28 to represent mentions and its contexts for feature extraction. In this section, we give a brief description of our embeddings followed by our features used in the neural model. Learning Joint Embeddings of Word and Entity Following BIBREF28 , we use Wikipedia articles, hyperlinks, and entity outlinks to jointly learn word/mention and entity embeddings in a unified vector space, so that similar words/mentions and entities have similar vectors. To address the ambiguity of words/mentions, BIBREF28 represents each word/mention with multiple vectors, and each vector denotes a sense referring to an entity in KB. The quality of the embeddings is verified on both textual similarity and entity relatedness tasks. Formally, each word/mention has a global embedding INLINEFORM0 , and multiple sense embeddings INLINEFORM1 . Each sense embedding INLINEFORM2 refers to an entity embedding INLINEFORM3 , while the difference between INLINEFORM4 and INLINEFORM5 is that INLINEFORM6 models the co-occurrence information of an entity in texts (via hyperlinks) and INLINEFORM7 encodes the structured entity relations in KBs. More details can be found in the original paper. Local Features Local features focus on how compatible the entity is mentioned in a piece of text (i.e., the mention and the context words). Except for the prior probability (Section SECREF9 ), we define two types of local features for each candidate entity INLINEFORM0 : String Similarity Similar to BIBREF16 , we define string based features as follows: the edit distance between mention's surface form and entity title, and boolean features indicating whether they are equivalent, whether the mention is inside, starts with or ends with entity title and vice versa. Compatibility We also measure the compatibility of INLINEFORM0 with the mention's context words INLINEFORM1 by computing their similarities based on joint embeddings: INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 is the context embedding of INLINEFORM5 conditioned on candidate INLINEFORM6 and is defined as the average sum of word global vectors weighted by attentions: INLINEFORM7 where INLINEFORM0 is the INLINEFORM1 -th word's attention from INLINEFORM2 . In this way, we automatically select informative words by assigning higher attention weights, and filter out irrelevant noise through small weights. The attention INLINEFORM3 is computed as follows: INLINEFORM4 where INLINEFORM0 is the similarity measurement, and we use cosine similarity in the presented work. We concatenate the prior probability, string based similarities, compatibility similarities and the embeddings of contexts as well as the entity as the local feature vectors. Global Features The key idea of collective EL is to utilize the topical coherence throughout the entire document. The consistency assumption behind it is that: all mentions in a document shall be on the same topic. However, this leads to exhaustive computations if the number of mentions is large. Based on the observation that the consistency attenuates along with the distance between two mentions, we argue that the adjacent mentions might be sufficient for supporting the assumption efficiently. Formally, we define neighbor mentions as INLINEFORM0 adjacent mentions before and after current mention INLINEFORM1 : INLINEFORM2 , where INLINEFORM3 is the pre-defined window size. Thus, the topical coherence at document level shall be achieved in a chain-like way. As shown in Figure FIGREF10 ( INLINEFORM4 ), mentions Hussain and Essex, a cricket player and the cricket club, provide adequate disambiguation clues to induce the underlying topic “cricket" for the current mention England, which impacts positively on identifying the mention surrey as another cricket club via the common neighbor mention Essex. A degraded case happens if INLINEFORM0 is large enough to cover the entire document, and the mentions used for global features become the same as the previous work, such as BIBREF21 . In experiments, we heuristically found a suitable INLINEFORM1 which is much smaller than the total number of mentions. The benefits of efficiency are in two ways: (i) to decrease time complexity, and (ii) to trim the entity graph into a fixed size of subgraph that facilitates computation acceleration through GPUs and batch techniques, which will be discussed in Section SECREF24 . Given neighbor mentions INLINEFORM0 , we extract two types of vectorial global features and structured global features for each candidate INLINEFORM1 : Neighbor Mention Compatibility Suppose neighbor mentions are topical coherent, a candidate entity shall also be compatible with neighbor mentions if it has a high compatibility score with the current mention, otherwise not. That is, we extract the vectorial global features by computing the similarities between INLINEFORM0 and all neighbor mentions: INLINEFORM1 , where INLINEFORM2 is the mention embedding by averaging the global vectors of words in its surface form: INLINEFORM3 , where INLINEFORM4 are tokenized words of mention INLINEFORM5 . Subgraph Structure The above features reflect the consistent semantics in texts (i.e., mentions). We now extract structured global features using the relations in KB, which facilitates the inference among candidates to find the most topical coherent subset. For each document, we obtain the entity graph INLINEFORM0 by taking candidate entities of all mentions INLINEFORM1 as nodes, and using entity embeddings to compute their similarities as edges INLINEFORM2 . Then, we extract the subgraph structured features INLINEFORM3 for each entity INLINEFORM4 for efficiency. Formally, we define the subgraph as: INLINEFORM0 , where INLINEFORM1 . For example (Figure FIGREF1 ), for entity England cricket team, the subgraph contains the relation from it to all candidates of neighbor mentions: England cricket team, Nasser Hussain (rugby union), Nasser Hussain, Essex, Essex County Cricket Club and Essex, New York. To support batch-wise acceleration, we represent INLINEFORM2 in the form of adjacency table based vectors: INLINEFORM3 , where INLINEFORM4 is the number of candidates per mention. Finally, for each candidate INLINEFORM0 , we concatenate local features and neighbor mention compatibility scores as the feature vector INLINEFORM1 , and construct the subgraph structure representation INLINEFORM2 as the inputs of NCEL. Neural Collective Entity Linking NCEL incorporates GCN into a deep neural network to utilize structured graph information for collectively feature abstraction, while differs from conventional GCN in the way of applying the graph. Instead of the entire graph, only a subset of nodes is “visible" to each node in our proposed method, and then the overall structured information shall be reached in a chain-like way. Fixing the size of the subset, NCEL is further speeded up by batch techniques and GPUs, and is efficient to large-scale data. Graph Convolutional Network GCNs are a type of neural network model that deals with structured data. It takes a graph as an input and output labels for each node. As a simplification of spectral graph convolutions, the main idea of BIBREF26 is similar to a propagation model: to enhance the features of a node according to its neighbor nodes. The formulation is as follows: INLINEFORM0 where INLINEFORM0 is a normalized adjacent matrix of the input graph with self-connection, INLINEFORM1 and INLINEFORM2 are the hidden states and weights in the INLINEFORM3 -th layer, and INLINEFORM4 is a non-linear activation, such as ReLu. Model Architecture As shown in Figure FIGREF10 , NCEL identifies the correct candidate INLINEFORM0 for the mention INLINEFORM1 by using vectorial features as well as structured relatedness with candidates of neighbor mentions INLINEFORM2 . Given feature vector INLINEFORM3 and subgraph representation INLINEFORM4 of each candidate INLINEFORM5 , we stack them as inputs for mention INLINEFORM6 : INLINEFORM7 , and the adjacent matrix INLINEFORM8 , where INLINEFORM9 denotes the subgraph with self-connection. We normalize INLINEFORM10 such that all rows sum to one, denoted as INLINEFORM11 , avoiding the change in the scale of the feature vectors. Given INLINEFORM0 and INLINEFORM1 , the goal of NCEL is to find the best assignment: INLINEFORM2 where INLINEFORM0 is the output variable of candidates, and INLINEFORM1 is a probability function as follows: INLINEFORM2 where INLINEFORM0 is the score function parameters by INLINEFORM1 . NCEL learns the mapping INLINEFORM2 through a neural network including three main modules: encoder, sub-graph convolution network (sub-GCN) and decoder. Next, we introduce them in turn. Encoder The function of this module is to integrate different features by a multi-layer perceptron (MLP): INLINEFORM0 where INLINEFORM0 is the hidden states of the current mention, INLINEFORM1 and INLINEFORM2 are trainable parameters and bias. We use ReLu as the non-linear activation INLINEFORM3 . Sub-Graph Convolution Network Similar to GCN, this module learns to abstract features from the hidden state of the mention itself as well as its neighbors. Suppose INLINEFORM0 is the hidden states of the neighbor INLINEFORM1 , we stack them to expand the current hidden states of INLINEFORM2 as INLINEFORM3 , such that each row corresponds to that in the subgraph adjacent matrix INLINEFORM4 . We define sub-graph convolution as: INLINEFORM5 where INLINEFORM0 is a trainable parameter. Decoder After INLINEFORM0 iterations of sub-graph convolution, the hidden states integrate both features of INLINEFORM1 and its neighbors. A fully connected decoder maps INLINEFORM2 to the number of candidates as follows: INLINEFORM3 where INLINEFORM0 . Training The parameters of network are trained to minimize cross-entropy of the predicted and ground truth INLINEFORM0 : INLINEFORM1 Suppose there are INLINEFORM0 documents in training corpus, each document has a set of mentions INLINEFORM1 , leading to totally INLINEFORM2 mention sets. The overall objective function is as follows: INLINEFORM3 Experiments To avoid overfitting with some dataset, we train NCEL using collected Wikipedia hyperlinks instead of specific annotated data. We then evaluate the trained model on five different benchmarks to verify the linking precision as well as the generalization ability. Furthermore, we investigate the effectiveness of key modules in NCEL and give qualitative results for comprehensive analysis. Baselines and Datasets We compare NCEL with the following state-of-the-art EL methods including three local models and three types of global models: Local models: He BIBREF29 and Chisholm BIBREF6 beat many global models by using auto-encoders and web links, respectively, and NTEE BIBREF16 achieves the best performance based on joint embeddings of words and entities. Iterative model: AIDA BIBREF22 links entities by iteratively finding a dense subgraph. Loopy Belief Propagation: Globerson BIBREF18 and PBoH BIBREF30 introduce LBP BIBREF31 techniques for collective inference, and Ganea BIBREF24 solves the global training problem via truncated fitting LBP. PageRank/Random Walk: Boosting BIBREF32 , AGDISTISG BIBREF33 , Babelfy BIBREF34 , WAT BIBREF35 , xLisa BIBREF36 and WNED BIBREF19 performs PageRank BIBREF37 or random walk BIBREF38 on the mention-entity graph and use the convergence score for disambiguation. For fairly comparison, we report the original scores of the baselines in the papers. Following these methods, we evaluate NCEL on the following five datasets: (1) CoNLL-YAGO BIBREF22 : the CoNLL 2003 shared task including testa of 4791 mentions in 216 documents, and testb of 4485 mentions in 213 documents. (2) TAC2010 BIBREF39 : constructed for the Text Analysis Conference that comprises 676 mentions in 352 documents for testing. (3) ACE2004 BIBREF23 : a subset of ACE2004 co-reference documents including 248 mentions in 35 documents, which is annotated by Amazon Mechanical Turk. (4) AQUAINT BIBREF40 : 50 news articles including 699 mentions from three different news agencies. (5) WW BIBREF19 : a new benchmark with balanced prior distributions of mentions, leading to a hard case of disambiguation. It has 6374 mentions in 310 documents automatically extracted from Wikipedia. Training Details and Running Time Analysis Training We collect 50,000 Wikipedia articles according to the number of its hyperlinks as our training data. For efficiency, we trim the articles to the first three paragraphs leading to 1,035,665 mentions in total. Using CoNLL-Test A as the development set, we evaluate the trained NCEL on the above benchmarks. We set context window to 20, neighbor mention window to 6, and top INLINEFORM0 candidates for each mention. We use two layers with 2000 and 1 hidden units in MLP encoder, and 3 layers in sub-GCN. We use early stop and fine tune the embeddings. With a batch size of 16, nearly 3 epochs cost less than 15 minutes on the server with 20 core CPU and the GeForce GTX 1080Ti GPU with 12Gb memory. We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements. Complexity Analysis Compared with local methods, the main disadvantage of collective methods is high complexity and expensive costs. Suppose there are INLINEFORM0 mentions in documents on average, among these global models, NCEL not surprisingly has the lowest time complexity INLINEFORM1 since it only considers adjacent mentions, where INLINEFORM2 is the number of sub-GCN layers indicating the iterations until convergence. AIDA has the highest time complexity INLINEFORM3 in worst case due to exhaustive iteratively finding and sorting the graph. The LBP and PageRank/random walk based methods achieve similar high time complexity of INLINEFORM4 mainly because of the inference on the entire graph. Results on GERBIL GERBIL BIBREF41 is a benchmark entity annotation framework that aims to provide a unified comparison among different EL methods across datasets including ACE2004, AQUAINT and CoNLL. We compare NCEL with the global models that report the performance on GERBIL. As shown in Table TABREF26 , NCEL achieves the best performance in most cases with an average gain of 2% on Micro F1 and 3% Macro F1. The baseline methods also achieve competitive results on some datasets but fail to adapt to the others. For example, AIDA and xLisa perform quite well on ACE2004 but poorly on other datasets, or WAT, PBoH, and WNED have a favorable performance on CoNLL but lower values on ACE2004 and AQUAINT. Our proposed method performs consistently well on all datasets that demonstrates the good generalization ability. Results on TAC2010 and WW In this section, we investigate the effectiveness of NCEL in the “easy" and “hard" datasets, respectively. Particularly, TAC2010, which has two mentions per document on average (Section SECREF19 ) and high prior probabilities of correct candidates (Figure FIGREF28 ), is regarded as the “easy" case for EL, and WW is the “hard" case since it has the most mentions with balanced prior probabilities BIBREF19 . Besides, we further compare the impact of key modules by removing the following part from NCEL: global features (NCEL-local), attention (NCEL-noatt), embedding features (NCEL-noemb), and the impact of the prior probability (prior). The results are shown in Table FIGREF28 and Table FIGREF28 . We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases. In the “easy" case, local models have similar performance with global models since only little global information is available (2 mentions per document). Besides, NN-based models, NTEE and NCEL-local, perform significantly better than others including most global models, demonstrating that the effectiveness of neural models deals with the first limitation in the introduction. Impact of NCEL Modules As shown in Figure FIGREF28 , the prior probability performs quite well in TAC2010 but poorly in WW. Compared with NCEL-local, the global module in NCEL brings more improvements in the “hard" case than that for “easy" dataset, because local features are discriminative enough in most cases of TAC2010, and global information becomes quite helpful when local features cannot handle. That is, our propose collective model is robust and shows a good generalization ability to difficult EL. The improvements by each main module are relatively small in TAC2010, while the modules of attention and embedding features show non-negligible impacts in WW (even worse than local model), mainly because WW contains much noise, and these two modules are effective in improving the robustness to noise and the ability of generalization by selecting informative words and providing more accurate semantics, respectively. Qualitative Analysis The results of example in Figure FIGREF1 are shown in Table TABREF30 , which is from CoNLL testa dataset. For mention Essex, although both NCEL and NCEL-local correctly identify entity Essex County Cricket Club, NCEL outputs higher probability due to the enhancement of neighbor mentions. Moreover, for mention England, NCEL-local cannot find enough disambiguation clues from its context words, such as surplus and requirements, and thus assigns a higher probability of 0.42 to the country England according to the prior probability. Collectively, NCEL correctly identifies England cricket team with a probability of 0.72 as compared with 0.20 in NCEL-local with the help of its neighbor mention Essex. Conclusion In this paper, we propose a neural model for collective entity linking that is end-to-end trainable. It applies GCN on subgraphs instead of the entire entity graph to efficiently learn features from both local and global information. We design an attention mechanism that endows NCEL robust to noisy data. Trained on collected Wikipedia hyperlinks, NCEL outperforms the state-of-the-art collective methods across five different datasets. Besides, further analysis of the impacts of main modules as well as qualitative results demonstrates its effectiveness. In the future, we will extend our method into cross-lingual settings to help link entities in low-resourced languages by exploiting rich knowledge from high-resourced languages, and deal with NIL entities to facilitate specific applications. Acknowledgments The work is supported by National Key Research and Development Program of China (2017YFB1002101), NSFC key project (U1736204, 61661146007), and THUNUS NExT Co-Lab.
How do they verify generalization ability?
2439b6b92d73f660fe6af8d24b7bbecf2b3a3d72
How Ġdo Ġthey Ġverify Ġgeneral ization Ġability ?
By Ġcalculating ĠMacro ĠF 1 Ġmetric Ġat Ġthe Ġdocument Ġlevel .
<s> How Ġdo Ġthey Ġverify Ġgeneral ization Ġability ? </s> Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 / </s> Entity Ġlinking Ġ( EL ), Ġmapping Ġentity Ġmentions Ġin Ġtexts Ġto Ġa Ġgiven Ġknowledge Ġbase Ġ( KB ), Ġserves Ġas Ġa Ġfundamental Ġrole Ġin Ġmany Ġfields , Ġsuch Ġas Ġquestion Ġanswering ĠB IB REF 0 Ġ, Ġsemantic Ġsearch ĠB IB REF 1 Ġ, Ġand Ġinformation Ġextraction ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġthis Ġtask Ġis Ġnon - t riv ial Ġbecause Ġentity Ġmentions Ġare Ġusually Ġambiguous . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ, Ġthe Ġmention ĠEngland Ġrefers Ġto Ġthree Ġentities Ġin ĠKB , Ġand Ġan Ġentity Ġlinking Ġsystem Ġshould Ġbe Ġcapable Ġof Ġidentifying Ġthe Ġcorrect Ġentity Ġas ĠEngland Ġcricket Ġteam Ġrather Ġthan ĠEngland Ġand ĠEngland Ġnational Ġfootball Ġteam . </s> Entity Ġlinking Ġis Ġtypically Ġbroken Ġdown Ġinto Ġtwo Ġmain Ġphases : Ġ( i ) Ġcandidate Ġgeneration Ġob tains Ġa Ġset Ġof Ġrefere nt Ġentities Ġin ĠKB Ġfor Ġeach Ġmention , Ġand Ġ( ii ) Ġnamed Ġentity Ġdis amb ig uation Ġselects Ġthe Ġpossible Ġcandidate Ġentity Ġby Ġsolving Ġa Ġranking Ġproblem . ĠThe Ġkey Ġchallenge Ġlies Ġin Ġthe Ġranking Ġmodel Ġthat Ġcomp utes Ġthe Ġrelevance Ġbetween Ġcandidates Ġand Ġthe Ġcorresponding Ġmentions Ġbased Ġon Ġthe Ġinformation Ġboth Ġin Ġtexts Ġand ĠKB s ĠB IB REF 4 Ġ. ĠIn Ġterms Ġof Ġthe Ġfeatures Ġused Ġfor Ġranking , Ġwe Ġclassify Ġexisting ĠEL Ġmodels Ġinto Ġtwo Ġgroups : Ġlocal Ġmodels Ġto Ġresolve Ġmentions Ġindependently Ġrelying Ġon Ġtextual Ġcontext Ġinformation Ġfrom Ġthe Ġsurrounding Ġwords ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġglobal Ġ( collect ive ) Ġmodels , Ġwhich Ġare Ġthe Ġmain Ġfocus Ġof Ġthis Ġpaper , Ġthat Ġencourage Ġthe Ġtarget Ġentities Ġof Ġall Ġmentions Ġin Ġa Ġdocument Ġto Ġbe Ġtop ically Ġcoherent ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ. </s> Global Ġmodels Ġusually Ġbuild Ġan Ġentity Ġgraph Ġbased Ġon ĠKB s Ġto Ġcapture Ġcoherent Ġentities Ġfor Ġall Ġidentified Ġmentions Ġin Ġa Ġdocument , Ġwhere Ġthe Ġnodes Ġare Ġentities , Ġand Ġedges Ġdenote Ġtheir Ġrelations . ĠThe Ġgraph Ġprovides Ġhighly Ġdiscrim in ative Ġsemantic Ġsignals Ġ( e . g ., Ġentity Ġrelated ness ) Ġthat Ġare Ġunavailable Ġto Ġlocal Ġmodel ĠB IB REF 15 Ġ. ĠFor Ġexample Ġ( Figure ĠFIG REF 1 Ġ), Ġan ĠEL Ġmodel Ġseem ly Ġcannot Ġfind Ġsufficient Ġdis amb ig uation Ġclues Ġfor Ġthe Ġmention ĠEngland Ġfrom Ġits Ġsurrounding Ġwords , Ġunless Ġit Ġutilizes Ġthe Ġco herence Ġinformation Ġof Ġconsistent Ġtopic ĠâĢ ľ c rick et " Ġamong Ġadjacent Ġmentions ĠEngland , ĠHussain , Ġand ĠEssex . ĠAlthough Ġthe Ġglobal Ġmodel Ġhas Ġachieved Ġsignificant Ġimprovements , Ġits Ġlimitation Ġis Ġthree fold : </s> To Ġmitigate Ġthe Ġfirst Ġlimitation , Ġrecent ĠEL Ġstudies Ġintroduce Ġneural Ġnetwork Ġ( NN ) Ġmodels Ġdue Ġto Ġits Ġamazing Ġfeature Ġabstraction Ġand Ġgeneral ization Ġability . ĠIn Ġsuch Ġmodels , Ġwords / ent ities Ġare Ġrepresented Ġby Ġlow Ġdimensional Ġvectors Ġin Ġa Ġcontinuous Ġspace , Ġand Ġfeatures Ġfor Ġmention Ġas Ġwell Ġas Ġcandidate Ġentities Ġare Ġautomatically Ġlearned Ġfrom Ġdata ĠB IB REF 4 Ġ. ĠHowever , Ġexisting ĠN N - based Ġmethods Ġfor ĠEL Ġare Ġeither Ġlocal Ġmodels ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġor Ġmerely Ġuse Ġword / entity Ġembed d ings Ġfor Ġfeature Ġextraction Ġand Ġrely Ġon Ġanother Ġmodules Ġfor Ġcollective Ġdis amb ig uation , Ġwhich Ġthus Ġcannot Ġfully Ġutilize Ġthe Ġpower Ġof ĠN N Ġmodels Ġfor Ġcollective ĠEL ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. </s> The Ġsecond Ġdrawback Ġof Ġthe Ġglobal Ġapproach Ġhas Ġbeen Ġallev iated Ġthrough Ġapproximate Ġoptimization Ġtechniques , Ġsuch Ġas ĠPage Rank / random Ġwalks ĠB IB REF 21 Ġ, Ġgraph Ġpr uning ĠB IB REF 22 Ġ, Ġranking ĠSV Ms ĠB IB REF 23 Ġ, Ġor Ġloop y Ġbelief Ġpropagation Ġ( L BP ) ĠB IB REF 18 Ġ, ĠB IB REF 24 Ġ. ĠHowever , Ġthese Ġmethods Ġare Ġnot Ġdifferent iable Ġand Ġthus Ġdifficult Ġto Ġbe Ġintegrated Ġinto Ġneural Ġnetwork Ġmodels Ġ( the Ġsolution Ġfor Ġthe Ġfirst Ġlimitation ). </s> To Ġovercome Ġthe Ġthird Ġissue Ġof Ġinadequate Ġtraining Ġdata , ĠB IB REF 17 Ġhas Ġexplored Ġa Ġmassive Ġamount Ġof Ġhyper links Ġin ĠWikipedia , Ġbut Ġthese Ġpotential Ġannotations Ġfor ĠEL Ġcontain Ġmuch Ġnoise , Ġwhich Ġmay Ġdistract Ġa Ġnaive Ġdis amb ig uation Ġmodel ĠB IB REF 6 Ġ. </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġnovel ĠNeural ĠCollective ĠEntity ĠLink ing Ġmodel Ġ( NC EL ), Ġwhich Ġperforms Ġglobal ĠEL Ġcombining Ġdeep Ġneural Ġnetworks Ġwith ĠGraph ĠConv olution al ĠNetwork Ġ( GC N ) ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġthat Ġallows Ġflexible Ġencoding Ġof Ġentity Ġgraphs . ĠIt Ġintegrates Ġboth Ġlocal Ġcontextual Ġinformation Ġand Ġglobal Ġinter d ependence Ġof Ġmentions Ġin Ġa Ġdocument , Ġand Ġis Ġefficiently Ġtrain able Ġin Ġan Ġend - to - end Ġfashion . ĠParticularly , Ġwe Ġintroduce Ġattention Ġmechanism Ġto Ġrobust ly Ġmodel Ġlocal Ġcontextual Ġinformation Ġby Ġselecting Ġinformative Ġwords Ġand Ġfiltering Ġout Ġthe Ġnoise . ĠOn Ġthe Ġother Ġhand , Ġwe Ġapply ĠGC Ns Ġto Ġimprove Ġdiscrim in ative Ġsignals Ġof Ġcandidate Ġentities Ġby Ġexploiting Ġthe Ġrich Ġstructure Ġunderlying Ġthe Ġcorrect Ġentities . ĠTo Ġalleviate Ġthe Ġglobal Ġcomput ations , Ġwe Ġpropose Ġto Ġconv olute Ġon Ġthe Ġsub graph Ġof Ġadjacent Ġmentions . ĠThus , Ġthe Ġoverall Ġco herence Ġshall Ġbe Ġachieved Ġin Ġa Ġchain - like Ġway Ġvia Ġa Ġsliding Ġwindow Ġover Ġthe Ġdocument . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġis Ġthe Ġfirst Ġeffort Ġto Ġdevelop Ġa Ġunified Ġmodel Ġfor Ġneural Ġcollective Ġentity Ġlinking . </s> In Ġexperiments , Ġwe Ġfirst Ġverify Ġthe Ġefficiency Ġof ĠNC EL Ġvia Ġtheoretically Ġcomparing Ġits Ġtime Ġcomplexity Ġwith Ġother Ġcollective Ġalternatives . ĠAfterwards , Ġwe Ġtrain Ġour Ġneural Ġmodel Ġusing Ġcollected ĠWikipedia Ġhyper links Ġinstead Ġof Ġdataset - specific Ġannotations , Ġand Ġperform Ġevaluations Ġon Ġfive Ġpublic Ġavailable Ġbenchmarks . ĠThe Ġresults Ġshow Ġthat ĠNC EL Ġconsistently Ġoutper forms Ġvarious Ġbas elines Ġwith Ġa Ġfavorable Ġgeneral ization Ġability . ĠFinally , Ġwe Ġfurther Ġpresent Ġthe Ġperformance Ġon Ġa Ġchallenging Ġdataset ĠWW ĠB IB REF 19 Ġas Ġwell Ġas Ġqualitative Ġresults , Ġinvestigating Ġthe Ġeffectiveness Ġof Ġeach Ġkey Ġmodule . </s> P rel im in aries Ġand ĠFramework </s> We Ġdenote ĠIN LINE FORM 0 Ġas Ġa Ġset Ġof Ġentity Ġmentions Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġis Ġeither Ġa Ġword ĠIN LINE FORM 3 Ġor Ġa Ġmention ĠIN LINE FORM 4 Ġ. ĠIN LINE FORM 5 Ġis Ġthe Ġentity Ġgraph Ġfor Ġdocument ĠIN LINE FORM 6 Ġderived Ġfrom Ġthe Ġgiven Ġknowledge Ġbase , Ġwhere ĠIN LINE FORM 7 Ġis Ġa Ġset Ġof Ġentities , ĠIN LINE FORM 8 Ġdenotes Ġthe Ġrelated ness Ġbetween ĠIN LINE FORM 9 Ġand Ġhigher Ġvalues Ġindicate Ġstronger Ġrelations . ĠBased Ġon ĠIN LINE FORM 10 Ġ, Ġwe Ġextract Ġa Ġsub graph ĠIN LINE FORM 11 Ġfor ĠIN LINE FORM 12 Ġ, Ġwhere ĠIN LINE FORM 13 Ġdenotes Ġthe Ġset Ġof Ġcandidate Ġentities Ġfor ĠIN LINE FORM 14 Ġ. ĠNote Ġthat Ġwe Ġdon 't Ġinclude Ġthe Ġrelations Ġamong Ġcandidates Ġof Ġthe Ġsame Ġmention Ġin ĠIN LINE FORM 15 Ġbecause Ġthese Ġcandidates Ġare Ġmutually Ġexclusive Ġin Ġdis amb ig uation . </s> Form ally , Ġwe Ġdefine Ġthe Ġentity Ġlinking Ġproblem Ġas Ġfollows : ĠGiven Ġa Ġset Ġof Ġmentions ĠIN LINE FORM 0 Ġin Ġa Ġdocument ĠIN LINE FORM 1 Ġ, Ġand Ġan Ġentity Ġgraph ĠIN LINE FORM 2 Ġ, Ġthe Ġgoal Ġis Ġto Ġfind Ġan Ġassignment ĠIN LINE FORM 3 Ġ. </s> To Ġcollectively Ġfind Ġthe Ġbest Ġassignment , ĠNC EL Ġaims Ġto Ġimprove Ġthe Ġdiscrim in ability Ġof Ġcandidates ' Ġlocal Ġfeatures Ġby Ġusing Ġentity Ġrelated ness Ġwithin Ġa Ġdocument Ġvia ĠGC N , Ġwhich Ġis Ġcapable Ġof Ġlearning Ġa Ġfunction Ġof Ġfeatures Ġon Ġthe Ġgraph Ġthrough Ġshared Ġparameters Ġover Ġall Ġnodes . ĠFigure ĠFIG REF 10 Ġshows Ġthe Ġframework Ġof ĠNC EL Ġincluding Ġthree Ġmain Ġcomponents : </s> Example ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, Ġfor Ġthe Ġcurrent Ġmention ĠEngland , Ġwe Ġutilize Ġits Ġsurrounding Ġwords Ġas Ġlocal Ġcontexts Ġ( e . g ., Ġsurplus ), Ġand Ġadjacent Ġmentions Ġ( e . g ., ĠH ussian ) Ġas Ġglobal Ġinformation . ĠCollect ively , Ġwe Ġutilize Ġthe Ġcandidates Ġof ĠEngland ĠIN LINE FORM 0 Ġas Ġwell Ġas Ġthose Ġentities Ġof Ġits Ġadj ac encies ĠIN LINE FORM 1 Ġto Ġconstruct Ġfeature Ġvectors Ġfor ĠIN LINE FORM 2 Ġand Ġthe Ġsub graph Ġof Ġrelated ness Ġas Ġinputs Ġof Ġour Ġneural Ġmodel . ĠLet Ġdarker Ġblue Ġindicate Ġhigher Ġprobability Ġof Ġbeing Ġpredicted , Ġthe Ġcorrect Ġcandidate ĠIN LINE FORM 3 Ġbecomes Ġbl uer Ġdue Ġto Ġits Ġbl uer Ġneighbor Ġnodes Ġof Ġother Ġmentions ĠIN LINE FORM 4 Ġ. ĠThe Ġdashed Ġlines Ġdenote Ġentity Ġrelations Ġthat Ġhave Ġindirect Ġimpacts Ġthrough Ġthe Ġsliding Ġadjacent Ġwindow Ġ, Ġand Ġthe Ġoverall Ġstructure Ġshall Ġbe Ġachieved Ġvia Ġmultiple Ġsub - graph s Ġby Ġtravers ing Ġall Ġmentions . </s> Before Ġintroducing Ġour Ġmodel , Ġwe Ġfirst Ġdescribe Ġthe Ġcomponent Ġof Ġcandidate Ġgeneration . </s> Cand idate ĠGeneration </s> Similar Ġto Ġprevious Ġwork ĠB IB REF 24 Ġ, Ġwe Ġuse Ġthe Ġprior Ġprobability ĠIN LINE FORM 0 Ġof Ġentity ĠIN LINE FORM 1 Ġconditioned Ġon Ġmention ĠIN LINE FORM 2 Ġboth Ġas Ġa Ġlocal Ġfeature Ġand Ġto Ġgenerate Ġcandidate Ġentities : ĠIN LINE FORM 3 Ġ. ĠWe Ġcompute ĠIN LINE FORM 4 Ġbased Ġon Ġstatistics Ġof Ġmention - entity Ġpairs Ġfrom : Ġ( i ) ĠWikipedia Ġpage Ġtitles , Ġredirect Ġtitles Ġand Ġhyper links , Ġ( ii ) Ġthe Ġdictionary Ġderived Ġfrom Ġa Ġlarge ĠWeb ĠCorpus ĠB IB REF 27 Ġ, Ġand Ġ( iii ) Ġthe ĠY AG O Ġdictionary Ġwith Ġa Ġuniform Ġdistribution ĠB IB REF 22 Ġ. ĠWe Ġpick Ġup Ġthe Ġmaximal Ġprior Ġif Ġa Ġmention - entity Ġpair Ġoccurs Ġin Ġdifferent Ġresources . ĠIn Ġexperiments , Ġto Ġoptimize Ġfor Ġmemory Ġand Ġrun Ġtime , Ġwe Ġkeep Ġonly Ġtop ĠIN LINE FORM 5 Ġentities Ġbased Ġon ĠIN LINE FORM 6 Ġ. ĠIn Ġthe Ġfollowing Ġtwo Ġsections , Ġwe Ġwill Ġpresent Ġthe Ġkey Ġcomponents Ġof ĠNEC L , Ġnamely Ġfeature Ġextraction Ġand Ġneural Ġnetwork Ġfor Ġcollective Ġentity Ġlinking . </s> Feature ĠExt raction </s> The Ġmain Ġgoal Ġof ĠNC EL Ġis Ġto Ġfind Ġa Ġsolution Ġfor Ġcollective Ġentity Ġlinking Ġusing Ġan Ġend - to - end Ġneural Ġmodel , Ġrather Ġthan Ġto Ġimprove Ġthe Ġmeasurements Ġof Ġlocal Ġtextual Ġsimilarity Ġor Ġglobal Ġmention / entity Ġrelated ness . ĠTherefore , Ġwe Ġuse Ġjoint Ġembed d ings Ġof Ġwords Ġand Ġentities Ġat Ġsense Ġlevel ĠB IB REF 28 Ġto Ġrepresent Ġmentions Ġand Ġits Ġcontexts Ġfor Ġfeature Ġextraction . ĠIn Ġthis Ġsection , Ġwe Ġgive Ġa Ġbrief Ġdescription Ġof Ġour Ġembed d ings Ġfollowed Ġby Ġour Ġfeatures Ġused Ġin Ġthe Ġneural Ġmodel . </s> Learning ĠJoint ĠEmb edd ings Ġof ĠWord Ġand ĠEntity </s> Following ĠB IB REF 28 Ġ, Ġwe Ġuse ĠWikipedia Ġarticles , Ġhyper links , Ġand Ġentity Ġout links Ġto Ġjointly Ġlearn Ġword / ment ion Ġand Ġentity Ġembed d ings Ġin Ġa Ġunified Ġvector Ġspace , Ġso Ġthat Ġsimilar Ġwords / ment ions Ġand Ġentities Ġhave Ġsimilar Ġvectors . ĠTo Ġaddress Ġthe Ġambiguity Ġof Ġwords / ment ions , ĠB IB REF 28 Ġrepresents Ġeach Ġword / ment ion Ġwith Ġmultiple Ġvectors , Ġand Ġeach Ġvector Ġdenotes Ġa Ġsense Ġreferring Ġto Ġan Ġentity Ġin ĠKB . ĠThe Ġquality Ġof Ġthe Ġembed d ings Ġis Ġverified Ġon Ġboth Ġtextual Ġsimilarity Ġand Ġentity Ġrelated ness Ġtasks . </s> Form ally , Ġeach Ġword / ment ion Ġhas Ġa Ġglobal Ġembed ding ĠIN LINE FORM 0 Ġ, Ġand Ġmultiple Ġsense Ġembed d ings ĠIN LINE FORM 1 Ġ. ĠEach Ġsense Ġembed ding ĠIN LINE FORM 2 Ġrefers Ġto Ġan Ġentity Ġembed ding ĠIN LINE FORM 3 Ġ, Ġwhile Ġthe Ġdifference Ġbetween ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġis Ġthat ĠIN LINE FORM 6 Ġmodels Ġthe Ġco - occ urrence Ġinformation Ġof Ġan Ġentity Ġin Ġtexts Ġ( via Ġhyper links ) Ġand ĠIN LINE FORM 7 Ġenc odes Ġthe Ġstructured Ġentity Ġrelations Ġin ĠKB s . ĠMore Ġdetails Ġcan Ġbe Ġfound Ġin Ġthe Ġoriginal Ġpaper . </s> Local ĠFeatures </s> Local Ġfeatures Ġfocus Ġon Ġhow Ġcompatible Ġthe Ġentity Ġis Ġmentioned Ġin Ġa Ġpiece Ġof Ġtext Ġ( i . e ., Ġthe Ġmention Ġand Ġthe Ġcontext Ġwords ). ĠExcept Ġfor Ġthe Ġprior Ġprobability Ġ( Section ĠSEC REF 9 Ġ), Ġwe Ġdefine Ġtwo Ġtypes Ġof Ġlocal Ġfeatures Ġfor Ġeach Ġcandidate Ġentity ĠIN LINE FORM 0 Ġ: </s> String ĠSimilar ity ĠSimilar Ġto ĠB IB REF 16 Ġ, Ġwe Ġdefine Ġstring Ġbased Ġfeatures Ġas Ġfollows : Ġthe Ġedit Ġdistance Ġbetween Ġmention 's Ġsurface Ġform Ġand Ġentity Ġtitle , Ġand Ġboolean Ġfeatures Ġindicating Ġwhether Ġthey Ġare Ġequivalent , Ġwhether Ġthe Ġmention Ġis Ġinside , Ġstarts Ġwith Ġor Ġends Ġwith Ġentity Ġtitle Ġand Ġvice Ġversa . </s> Comp atibility ĠWe Ġalso Ġmeasure Ġthe Ġcompatibility Ġof ĠIN LINE FORM 0 Ġwith Ġthe Ġmention 's Ġcontext Ġwords ĠIN LINE FORM 1 Ġby Ġcomputing Ġtheir Ġsimilarities Ġbased Ġon Ġjoint Ġembed d ings : ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġcontext Ġembed ding Ġof ĠIN LINE FORM 5 Ġconditioned Ġon Ġcandidate ĠIN LINE FORM 6 Ġand Ġis Ġdefined Ġas Ġthe Ġaverage Ġsum Ġof Ġword Ġglobal Ġvectors Ġweighted Ġby Ġattent ions : ĠIN LINE FORM 7 </s> where ĠIN LINE FORM 0 Ġis Ġthe ĠIN LINE FORM 1 Ġ- th Ġword 's Ġattention Ġfrom ĠIN LINE FORM 2 Ġ. ĠIn Ġthis Ġway , Ġwe Ġautomatically Ġselect Ġinformative Ġwords Ġby Ġassigning Ġhigher Ġattention Ġweights , Ġand Ġfilter Ġout Ġirrelevant Ġnoise Ġthrough Ġsmall Ġweights . ĠThe Ġattention ĠIN LINE FORM 3 Ġis Ġcomputed Ġas Ġfollows : ĠIN LINE FORM 4 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġsimilarity Ġmeasurement , Ġand Ġwe Ġuse Ġcos ine Ġsimilarity Ġin Ġthe Ġpresented Ġwork . ĠWe Ġconc aten ate Ġthe Ġprior Ġprobability , Ġstring Ġbased Ġsimilarities , Ġcompatibility Ġsimilarities Ġand Ġthe Ġembed d ings Ġof Ġcontexts Ġas Ġwell Ġas Ġthe Ġentity Ġas Ġthe Ġlocal Ġfeature Ġvectors . </s> Global ĠFeatures </s> The Ġkey Ġidea Ġof Ġcollective ĠEL Ġis Ġto Ġutilize Ġthe Ġtopical Ġco herence Ġthroughout Ġthe Ġentire Ġdocument . ĠThe Ġconsistency Ġassumption Ġbehind Ġit Ġis Ġthat : Ġall Ġmentions Ġin Ġa Ġdocument Ġshall Ġbe Ġon Ġthe Ġsame Ġtopic . ĠHowever , Ġthis Ġleads Ġto Ġexhaustive Ġcomput ations Ġif Ġthe Ġnumber Ġof Ġmentions Ġis Ġlarge . ĠBased Ġon Ġthe Ġobservation Ġthat Ġthe Ġconsistency Ġatten uates Ġalong Ġwith Ġthe Ġdistance Ġbetween Ġtwo Ġmentions , Ġwe Ġargue Ġthat Ġthe Ġadjacent Ġmentions Ġmight Ġbe Ġsufficient Ġfor Ġsupporting Ġthe Ġassumption Ġefficiently . </s> Form ally , Ġwe Ġdefine Ġneighbor Ġmentions Ġas ĠIN LINE FORM 0 Ġadjacent Ġmentions Ġbefore Ġand Ġafter Ġcurrent Ġmention ĠIN LINE FORM 1 Ġ: ĠIN LINE FORM 2 Ġ, Ġwhere ĠIN LINE FORM 3 Ġis Ġthe Ġpre - defined Ġwindow Ġsize . ĠThus , Ġthe Ġtopical Ġco herence Ġat Ġdocument Ġlevel Ġshall Ġbe Ġachieved Ġin Ġa Ġchain - like Ġway . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ( ĠIN LINE FORM 4 Ġ), Ġmentions ĠHussain Ġand ĠEssex , Ġa Ġcricket Ġplayer Ġand Ġthe Ġcricket Ġclub , Ġprovide Ġadequate Ġdis amb ig uation Ġclues Ġto Ġinduce Ġthe Ġunderlying Ġtopic ĠâĢ ľ c rick et " Ġfor Ġthe Ġcurrent Ġmention ĠEngland , Ġwhich Ġimpacts Ġpositively Ġon Ġidentifying Ġthe Ġmention Ġsur rey Ġas Ġanother Ġcricket Ġclub Ġvia Ġthe Ġcommon Ġneighbor Ġmention ĠEssex . </s> A Ġdegraded Ġcase Ġhappens Ġif ĠIN LINE FORM 0 Ġis Ġlarge Ġenough Ġto Ġcover Ġthe Ġentire Ġdocument , Ġand Ġthe Ġmentions Ġused Ġfor Ġglobal Ġfeatures Ġbecome Ġthe Ġsame Ġas Ġthe Ġprevious Ġwork , Ġsuch Ġas ĠB IB REF 21 Ġ. ĠIn Ġexperiments , Ġwe Ġhe ur istically Ġfound Ġa Ġsuitable ĠIN LINE FORM 1 Ġwhich Ġis Ġmuch Ġsmaller Ġthan Ġthe Ġtotal Ġnumber Ġof Ġmentions . ĠThe Ġbenefits Ġof Ġefficiency Ġare Ġin Ġtwo Ġways : Ġ( i ) Ġto Ġdecrease Ġtime Ġcomplexity , Ġand Ġ( ii ) Ġto Ġtrim Ġthe Ġentity Ġgraph Ġinto Ġa Ġfixed Ġsize Ġof Ġsub graph Ġthat Ġfacilitates Ġcomputation Ġacceleration Ġthrough ĠGPUs Ġand Ġbatch Ġtechniques , Ġwhich Ġwill Ġbe Ġdiscussed Ġin ĠSection ĠSEC REF 24 Ġ. </s> Given Ġneighbor Ġmentions ĠIN LINE FORM 0 Ġ, Ġwe Ġextract Ġtwo Ġtypes Ġof Ġvector ial Ġglobal Ġfeatures Ġand Ġstructured Ġglobal Ġfeatures Ġfor Ġeach Ġcandidate ĠIN LINE FORM 1 Ġ: </s> Neigh bor ĠM ention ĠCompatibility ĠSuppose Ġneighbor Ġmentions Ġare Ġtopical Ġcoherent , Ġa Ġcandidate Ġentity Ġshall Ġalso Ġbe Ġcompatible Ġwith Ġneighbor Ġmentions Ġif Ġit Ġhas Ġa Ġhigh Ġcompatibility Ġscore Ġwith Ġthe Ġcurrent Ġmention , Ġotherwise Ġnot . ĠThat Ġis , Ġwe Ġextract Ġthe Ġvector ial Ġglobal Ġfeatures Ġby Ġcomputing Ġthe Ġsimilarities Ġbetween ĠIN LINE FORM 0 Ġand Ġall Ġneighbor Ġmentions : ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġis Ġthe Ġmention Ġembed ding Ġby Ġaveraging Ġthe Ġglobal Ġvectors Ġof Ġwords Ġin Ġits Ġsurface Ġform : ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġare Ġtoken ized Ġwords Ġof Ġmention ĠIN LINE FORM 5 Ġ. </s> Sub graph ĠStructure ĠThe Ġabove Ġfeatures Ġreflect Ġthe Ġconsistent Ġsemantics Ġin Ġtexts Ġ( i . e ., Ġmentions ). ĠWe Ġnow Ġextract Ġstructured Ġglobal Ġfeatures Ġusing Ġthe Ġrelations Ġin ĠKB , Ġwhich Ġfacilitates Ġthe Ġinference Ġamong Ġcandidates Ġto Ġfind Ġthe Ġmost Ġtopical Ġcoherent Ġsubset . ĠFor Ġeach Ġdocument , Ġwe Ġobtain Ġthe Ġentity Ġgraph ĠIN LINE FORM 0 Ġby Ġtaking Ġcandidate Ġentities Ġof Ġall Ġmentions ĠIN LINE FORM 1 Ġas Ġnodes , Ġand Ġusing Ġentity Ġembed d ings Ġto Ġcompute Ġtheir Ġsimilarities Ġas Ġedges ĠIN LINE FORM 2 Ġ. ĠThen , Ġwe Ġextract Ġthe Ġsub graph Ġstructured Ġfeatures ĠIN LINE FORM 3 Ġfor Ġeach Ġentity ĠIN LINE FORM 4 Ġfor Ġefficiency . </s> Form ally , Ġwe Ġdefine Ġthe Ġsub graph Ġas : ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġ. ĠFor Ġexample Ġ( Figure ĠFIG REF 1 Ġ), Ġfor Ġentity ĠEngland Ġcricket Ġteam , Ġthe Ġsub graph Ġcontains Ġthe Ġrelation Ġfrom Ġit Ġto Ġall Ġcandidates Ġof Ġneighbor Ġmentions : ĠEngland Ġcricket Ġteam , ĠN asser ĠHussain Ġ( rug by Ġunion ), ĠN asser ĠHussain , ĠEssex , ĠEssex ĠCounty ĠCricket ĠClub Ġand ĠEssex , ĠNew ĠYork . ĠTo Ġsupport Ġbatch - wise Ġacceleration , Ġwe Ġrepresent ĠIN LINE FORM 2 Ġin Ġthe Ġform Ġof Ġadj ac ency Ġtable Ġbased Ġvectors : ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġnumber Ġof Ġcandidates Ġper Ġmention . </s> Finally , Ġfor Ġeach Ġcandidate ĠIN LINE FORM 0 Ġ, Ġwe Ġconc aten ate Ġlocal Ġfeatures Ġand Ġneighbor Ġmention Ġcompatibility Ġscores Ġas Ġthe Ġfeature Ġvector ĠIN LINE FORM 1 Ġ, Ġand Ġconstruct Ġthe Ġsub graph Ġstructure Ġrepresentation ĠIN LINE FORM 2 Ġas Ġthe Ġinputs Ġof ĠNC EL . </s> Ne ural ĠCollective ĠEntity ĠLink ing </s> NC EL Ġincorporates ĠGC N Ġinto Ġa Ġdeep Ġneural Ġnetwork Ġto Ġutilize Ġstructured Ġgraph Ġinformation Ġfor Ġcollectively Ġfeature Ġabstraction , Ġwhile Ġdiffers Ġfrom Ġconventional ĠGC N Ġin Ġthe Ġway Ġof Ġapplying Ġthe Ġgraph . ĠInstead Ġof Ġthe Ġentire Ġgraph , Ġonly Ġa Ġsubset Ġof Ġnodes Ġis ĠâĢ ľ visible " Ġto Ġeach Ġnode Ġin Ġour Ġproposed Ġmethod , Ġand Ġthen Ġthe Ġoverall Ġstructured Ġinformation Ġshall Ġbe Ġreached Ġin Ġa Ġchain - like Ġway . ĠFix ing Ġthe Ġsize Ġof Ġthe Ġsubset , ĠNC EL Ġis Ġfurther Ġspeed ed Ġup Ġby Ġbatch Ġtechniques Ġand ĠGPUs , Ġand Ġis Ġefficient Ġto Ġlarge - scale Ġdata . </s> Graph ĠConv olution al ĠNetwork </s> GC Ns Ġare Ġa Ġtype Ġof Ġneural Ġnetwork Ġmodel Ġthat Ġdeals Ġwith Ġstructured Ġdata . ĠIt Ġtakes Ġa Ġgraph Ġas Ġan Ġinput Ġand Ġoutput Ġlabels Ġfor Ġeach Ġnode . ĠAs Ġa Ġsimpl ification Ġof Ġspectral Ġgraph Ġconv olutions , Ġthe Ġmain Ġidea Ġof ĠB IB REF 26 Ġis Ġsimilar Ġto Ġa Ġpropagation Ġmodel : Ġto Ġenhance Ġthe Ġfeatures Ġof Ġa Ġnode Ġaccording Ġto Ġits Ġneighbor Ġnodes . ĠThe Ġformulation Ġis Ġas Ġfollows : ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnormalized Ġadjacent Ġmatrix Ġof Ġthe Ġinput Ġgraph Ġwith Ġself - connection , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġhidden Ġstates Ġand Ġweights Ġin Ġthe ĠIN LINE FORM 3 Ġ- th Ġlayer , Ġand ĠIN LINE FORM 4 Ġis Ġa Ġnon - linear Ġactivation , Ġsuch Ġas ĠRe Lu . </s> Model ĠArchitecture </s> As Ġshown Ġin ĠFigure ĠFIG REF 10 Ġ, ĠNC EL Ġidentifies Ġthe Ġcorrect Ġcandidate ĠIN LINE FORM 0 Ġfor Ġthe Ġmention ĠIN LINE FORM 1 Ġby Ġusing Ġvector ial Ġfeatures Ġas Ġwell Ġas Ġstructured Ġrelated ness Ġwith Ġcandidates Ġof Ġneighbor Ġmentions ĠIN LINE FORM 2 Ġ. ĠGiven Ġfeature Ġvector ĠIN LINE FORM 3 Ġand Ġsub graph Ġrepresentation ĠIN LINE FORM 4 Ġof Ġeach Ġcandidate ĠIN LINE FORM 5 Ġ, Ġwe Ġstack Ġthem Ġas Ġinputs Ġfor Ġmention ĠIN LINE FORM 6 Ġ: ĠIN LINE FORM 7 Ġ, Ġand Ġthe Ġadjacent Ġmatrix ĠIN LINE FORM 8 Ġ, Ġwhere ĠIN LINE FORM 9 Ġdenotes Ġthe Ġsub graph Ġwith Ġself - connection . ĠWe Ġnormal ize ĠIN LINE FORM 10 Ġsuch Ġthat Ġall Ġrows Ġsum Ġto Ġone , Ġden oted Ġas ĠIN LINE FORM 11 Ġ, Ġavoiding Ġthe Ġchange Ġin Ġthe Ġscale Ġof Ġthe Ġfeature Ġvectors . </s> Given ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ, Ġthe Ġgoal Ġof ĠNC EL Ġis Ġto Ġfind Ġthe Ġbest Ġassignment : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġoutput Ġvariable Ġof Ġcandidates , Ġand ĠIN LINE FORM 1 Ġis Ġa Ġprobability Ġfunction Ġas Ġfollows : ĠIN LINE FORM 2 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġscore Ġfunction Ġparameters Ġby ĠIN LINE FORM 1 Ġ. ĠNC EL Ġlearns Ġthe Ġmapping ĠIN LINE FORM 2 Ġthrough Ġa Ġneural Ġnetwork Ġincluding Ġthree Ġmain Ġmodules : Ġenc oder , Ġsub - graph Ġconv olution Ġnetwork Ġ( sub - GC N ) Ġand Ġdec oder . ĠNext , Ġwe Ġintroduce Ġthem Ġin Ġturn . </s> Enc oder ĠThe Ġfunction Ġof Ġthis Ġmodule Ġis Ġto Ġintegrate Ġdifferent Ġfeatures Ġby Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ): ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstates Ġof Ġthe Ġcurrent Ġmention , ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġtrain able Ġparameters Ġand Ġbias . ĠWe Ġuse ĠRe Lu Ġas Ġthe Ġnon - linear Ġactivation ĠIN LINE FORM 3 Ġ. </s> Sub - Graph ĠConv olution ĠNetwork ĠSimilar Ġto ĠGC N , Ġthis Ġmodule Ġlearns Ġto Ġabstract Ġfeatures Ġfrom Ġthe Ġhidden Ġstate Ġof Ġthe Ġmention Ġitself Ġas Ġwell Ġas Ġits Ġneighbors . ĠSuppose ĠIN LINE FORM 0 Ġis Ġthe Ġhidden Ġstates Ġof Ġthe Ġneighbor ĠIN LINE FORM 1 Ġ, Ġwe Ġstack Ġthem Ġto Ġexpand Ġthe Ġcurrent Ġhidden Ġstates Ġof ĠIN LINE FORM 2 Ġas ĠIN LINE FORM 3 Ġ, Ġsuch Ġthat Ġeach Ġrow Ġcorresponds Ġto Ġthat Ġin Ġthe Ġsub graph Ġadjacent Ġmatrix ĠIN LINE FORM 4 Ġ. ĠWe Ġdefine Ġsub - graph Ġconv olution Ġas : ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġtrain able Ġparameter . </s> Dec oder ĠAfter ĠIN LINE FORM 0 Ġiterations Ġof Ġsub - graph Ġconv olution , Ġthe Ġhidden Ġstates Ġintegrate Ġboth Ġfeatures Ġof ĠIN LINE FORM 1 Ġand Ġits Ġneighbors . ĠA Ġfully Ġconnected Ġdec oder Ġmaps ĠIN LINE FORM 2 Ġto Ġthe Ġnumber Ġof Ġcandidates Ġas Ġfollows : ĠIN LINE FORM 3 </s> where ĠIN LINE FORM 0 Ġ. </s> Training </s> The Ġparameters Ġof Ġnetwork Ġare Ġtrained Ġto Ġminimize Ġcross - ent ropy Ġof Ġthe Ġpredicted Ġand Ġground Ġtruth ĠIN LINE FORM 0 Ġ: ĠIN LINE FORM 1 </s> Supp ose Ġthere Ġare ĠIN LINE FORM 0 Ġdocuments Ġin Ġtraining Ġcorpus , Ġeach Ġdocument Ġhas Ġa Ġset Ġof Ġmentions ĠIN LINE FORM 1 Ġ, Ġleading Ġto Ġtotally ĠIN LINE FORM 2 Ġmention Ġsets . ĠThe Ġoverall Ġobjective Ġfunction Ġis Ġas Ġfollows : ĠIN LINE FORM 3 </s> Exper iments </s> To Ġavoid Ġover fitting Ġwith Ġsome Ġdataset , Ġwe Ġtrain ĠNC EL Ġusing Ġcollected ĠWikipedia Ġhyper links Ġinstead Ġof Ġspecific Ġannot ated Ġdata . ĠWe Ġthen Ġevaluate Ġthe Ġtrained Ġmodel Ġon Ġfive Ġdifferent Ġbenchmarks Ġto Ġverify Ġthe Ġlinking Ġprecision Ġas Ġwell Ġas Ġthe Ġgeneral ization Ġability . ĠFurthermore , Ġwe Ġinvestigate Ġthe Ġeffectiveness Ġof Ġkey Ġmodules Ġin ĠNC EL Ġand Ġgive Ġqualitative Ġresults Ġfor Ġcomprehensive Ġanalysis . </s> Bas elines Ġand ĠDat as ets </s> We Ġcompare ĠNC EL Ġwith Ġthe Ġfollowing Ġstate - of - the - art ĠEL Ġmethods Ġincluding Ġthree Ġlocal Ġmodels Ġand Ġthree Ġtypes Ġof Ġglobal Ġmodels : </s> Local Ġmodels : ĠHe ĠB IB REF 29 Ġand ĠCh ish ol m ĠB IB REF 6 Ġbeat Ġmany Ġglobal Ġmodels Ġby Ġusing Ġauto - enc od ers Ġand Ġweb Ġlinks , Ġrespectively , Ġand ĠNT EE ĠB IB REF 16 Ġachieves Ġthe Ġbest Ġperformance Ġbased Ġon Ġjoint Ġembed d ings Ġof Ġwords Ġand Ġentities . </s> Iter ative Ġmodel : ĠA IDA ĠB IB REF 22 Ġlinks Ġentities Ġby Ġiter atively Ġfinding Ġa Ġdense Ġsub graph . </s> Lo opy ĠBelief ĠProp ag ation : ĠGlob erson ĠB IB REF 18 Ġand ĠP Bo H ĠB IB REF 30 Ġintroduce ĠL BP ĠB IB REF 31 Ġtechniques Ġfor Ġcollective Ġinference , Ġand ĠG ane a ĠB IB REF 24 Ġsolves Ġthe Ġglobal Ġtraining Ġproblem Ġvia Ġtrunc ated Ġfitting ĠL BP . </s> Page Rank / Random ĠWalk : ĠBoost ing ĠB IB REF 32 Ġ, ĠAG D IST IS G ĠB IB REF 33 Ġ, ĠBab elf y ĠB IB REF 34 Ġ, ĠW AT ĠB IB REF 35 Ġ, Ġx Lisa ĠB IB REF 36 Ġand ĠW N ED ĠB IB REF 19 Ġperforms ĠPage Rank ĠB IB REF 37 Ġor Ġrandom Ġwalk ĠB IB REF 38 Ġon Ġthe Ġmention - entity Ġgraph Ġand Ġuse Ġthe Ġconvergence Ġscore Ġfor Ġdis amb ig uation . </s> For Ġfairly Ġcomparison , Ġwe Ġreport Ġthe Ġoriginal Ġscores Ġof Ġthe Ġbas elines Ġin Ġthe Ġpapers . ĠFollowing Ġthese Ġmethods , Ġwe Ġevaluate ĠNC EL Ġon Ġthe Ġfollowing Ġfive Ġdatasets : Ġ( 1 ) ĠCo N LL - Y AG O ĠB IB REF 22 Ġ: Ġthe ĠCo N LL Ġ2003 Ġshared Ġtask Ġincluding Ġtest a Ġof Ġ4 79 1 Ġmentions Ġin Ġ216 Ġdocuments , Ġand Ġtest b Ġof Ġ4 485 Ġmentions Ġin Ġ213 Ġdocuments . Ġ( 2 ) ĠT AC 2010 ĠB IB REF 39 Ġ: Ġconstructed Ġfor Ġthe ĠText ĠAnalysis ĠConference Ġthat Ġcomprises Ġ6 76 Ġmentions Ġin Ġ352 Ġdocuments Ġfor Ġtesting . Ġ( 3 ) ĠACE 2004 ĠB IB REF 23 Ġ: Ġa Ġsubset Ġof ĠACE 2004 Ġco - reference Ġdocuments Ġincluding Ġ248 Ġmentions Ġin Ġ35 Ġdocuments , Ġwhich Ġis Ġannot ated Ġby ĠAmazon ĠMechanical ĠTurk . Ġ( 4 ) ĠA QU A INT ĠB IB REF 40 Ġ: Ġ50 Ġnews Ġarticles Ġincluding Ġ6 99 Ġmentions Ġfrom Ġthree Ġdifferent Ġnews Ġagencies . Ġ( 5 ) ĠWW ĠB IB REF 19 Ġ: Ġa Ġnew Ġbenchmark Ġwith Ġbalanced Ġprior Ġdistributions Ġof Ġmentions , Ġleading Ġto Ġa Ġhard Ġcase Ġof Ġdis amb ig uation . ĠIt Ġhas Ġ6 374 Ġmentions Ġin Ġ310 Ġdocuments Ġautomatically Ġextracted Ġfrom ĠWikipedia . </s> Training ĠDetails Ġand ĠRunning ĠTime ĠAnalysis </s> Training ĠWe Ġcollect Ġ50 , 000 ĠWikipedia Ġarticles Ġaccording Ġto Ġthe Ġnumber Ġof Ġits Ġhyper links Ġas Ġour Ġtraining Ġdata . ĠFor Ġefficiency , Ġwe Ġtrim Ġthe Ġarticles Ġto Ġthe Ġfirst Ġthree Ġparagraphs Ġleading Ġto Ġ1 , 035 , 665 Ġmentions Ġin Ġtotal . ĠUsing ĠCo N LL - Test ĠA Ġas Ġthe Ġdevelopment Ġset , Ġwe Ġevaluate Ġthe Ġtrained ĠNC EL Ġon Ġthe Ġabove Ġbenchmarks . ĠWe Ġset Ġcontext Ġwindow Ġto Ġ20 , Ġneighbor Ġmention Ġwindow Ġto Ġ6 , Ġand Ġtop ĠIN LINE FORM 0 Ġcandidates Ġfor Ġeach Ġmention . ĠWe Ġuse Ġtwo Ġlayers Ġwith Ġ2000 Ġand Ġ1 Ġhidden Ġunits Ġin ĠML P Ġenc oder , Ġand Ġ3 Ġlayers Ġin Ġsub - GC N . ĠWe Ġuse Ġearly Ġstop Ġand Ġfine Ġtune Ġthe Ġembed d ings . ĠWith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġnearly Ġ3 Ġepoch s Ġcost Ġless Ġthan Ġ15 Ġminutes Ġon Ġthe Ġserver Ġwith Ġ20 Ġcore ĠCPU Ġand Ġthe ĠGeForce ĠGTX Ġ1080 Ti ĠGPU Ġwith Ġ12 Gb Ġmemory . ĠWe Ġuse Ġstandard ĠPrecision , ĠRecall Ġand ĠF 1 Ġat Ġmention Ġlevel Ġ( Micro ) Ġand Ġat Ġthe Ġdocument Ġlevel Ġ( Mac ro ) Ġas Ġmeasurements . </s> Com plex ity ĠAnalysis ĠCompared Ġwith Ġlocal Ġmethods , Ġthe Ġmain Ġdisadvantage Ġof Ġcollective Ġmethods Ġis Ġhigh Ġcomplexity Ġand Ġexpensive Ġcosts . ĠSuppose Ġthere Ġare ĠIN LINE FORM 0 Ġmentions Ġin Ġdocuments Ġon Ġaverage , Ġamong Ġthese Ġglobal Ġmodels , ĠNC EL Ġnot Ġsurprisingly Ġhas Ġthe Ġlowest Ġtime Ġcomplexity ĠIN LINE FORM 1 Ġsince Ġit Ġonly Ġconsiders Ġadjacent Ġmentions , Ġwhere ĠIN LINE FORM 2 Ġis Ġthe Ġnumber Ġof Ġsub - GC N Ġlayers Ġindicating Ġthe Ġiterations Ġuntil Ġconvergence . ĠA IDA Ġhas Ġthe Ġhighest Ġtime Ġcomplexity ĠIN LINE FORM 3 Ġin Ġworst Ġcase Ġdue Ġto Ġexhaustive Ġiter atively Ġfinding Ġand Ġsorting Ġthe Ġgraph . ĠThe ĠL BP Ġand ĠPage Rank / random Ġwalk Ġbased Ġmethods Ġachieve Ġsimilar Ġhigh Ġtime Ġcomplexity Ġof ĠIN LINE FORM 4 Ġmainly Ġbecause Ġof Ġthe Ġinference Ġon Ġthe Ġentire Ġgraph . </s> Results Ġon ĠGER BIL </s> GER BIL ĠB IB REF 41 Ġis Ġa Ġbenchmark Ġentity Ġannotation Ġframework Ġthat Ġaims Ġto Ġprovide Ġa Ġunified Ġcomparison Ġamong Ġdifferent ĠEL Ġmethods Ġacross Ġdatasets Ġincluding ĠACE 2004 , ĠA QU A INT Ġand ĠCo N LL . ĠWe Ġcompare ĠNC EL Ġwith Ġthe Ġglobal Ġmodels Ġthat Ġreport Ġthe Ġperformance Ġon ĠGER BIL . </s> As Ġshown Ġin ĠTable ĠT AB REF 26 Ġ, ĠNC EL Ġachieves Ġthe Ġbest Ġperformance Ġin Ġmost Ġcases Ġwith Ġan Ġaverage Ġgain Ġof Ġ2 % Ġon ĠMicro ĠF 1 Ġand Ġ3 % ĠMacro ĠF 1 . ĠThe Ġbaseline Ġmethods Ġalso Ġachieve Ġcompetitive Ġresults Ġon Ġsome Ġdatasets Ġbut Ġfail Ġto Ġadapt Ġto Ġthe Ġothers . ĠFor Ġexample , ĠA IDA Ġand Ġx Lisa Ġperform Ġquite Ġwell Ġon ĠACE 2004 Ġbut Ġpoorly Ġon Ġother Ġdatasets , Ġor ĠW AT , ĠP Bo H , Ġand ĠW N ED Ġhave Ġa Ġfavorable Ġperformance Ġon ĠCo N LL Ġbut Ġlower Ġvalues Ġon ĠACE 2004 Ġand ĠA QU A INT . ĠOur Ġproposed Ġmethod Ġperforms Ġconsistently Ġwell Ġon Ġall Ġdatasets Ġthat Ġdemonstrates Ġthe Ġgood Ġgeneral ization Ġability . </s> Results Ġon ĠT AC 2010 Ġand ĠWW </s> In Ġthis Ġsection , Ġwe Ġinvestigate Ġthe Ġeffectiveness Ġof ĠNC EL Ġin Ġthe ĠâĢ ľ easy " Ġand ĠâĢ ľ hard " Ġdatasets , Ġrespectively . ĠParticularly , ĠT AC 2010 , Ġwhich Ġhas Ġtwo Ġmentions Ġper Ġdocument Ġon Ġaverage Ġ( Section ĠSEC REF 19 Ġ) Ġand Ġhigh Ġprior Ġprobabilities Ġof Ġcorrect Ġcandidates Ġ( Figure ĠFIG REF 28 Ġ), Ġis Ġregarded Ġas Ġthe ĠâĢ ľ easy " Ġcase Ġfor ĠEL , Ġand ĠWW Ġis Ġthe ĠâĢ ľ hard " Ġcase Ġsince Ġit Ġhas Ġthe Ġmost Ġmentions Ġwith Ġbalanced Ġprior Ġprobabilities ĠB IB REF 19 Ġ. ĠBesides , Ġwe Ġfurther Ġcompare Ġthe Ġimpact Ġof Ġkey Ġmodules Ġby Ġremoving Ġthe Ġfollowing Ġpart Ġfrom ĠNC EL : Ġglobal Ġfeatures Ġ( NC EL - local ), Ġattention Ġ( NC EL - no att ), Ġembed ding Ġfeatures Ġ( NC EL - no emb ), Ġand Ġthe Ġimpact Ġof Ġthe Ġprior Ġprobability Ġ( pri or ). </s> The Ġresults Ġare Ġshown Ġin ĠTable ĠFIG REF 28 Ġand ĠTable ĠFIG REF 28 Ġ. ĠWe Ġcan Ġsee Ġthe Ġaverage Ġlinking Ġprecision Ġ( Micro ) Ġof ĠWW Ġis Ġlower Ġthan Ġthat Ġof ĠT AC 2010 , Ġand ĠNC EL Ġoutper forms Ġall Ġbaseline Ġmethods Ġin Ġboth Ġeasy Ġand Ġhard Ġcases . ĠIn Ġthe ĠâĢ ľ easy " Ġcase , Ġlocal Ġmodels Ġhave Ġsimilar Ġperformance Ġwith Ġglobal Ġmodels Ġsince Ġonly Ġlittle Ġglobal Ġinformation Ġis Ġavailable Ġ( 2 Ġmentions Ġper Ġdocument ). ĠBesides , ĠN N - based Ġmodels , ĠNT EE Ġand ĠNC EL - local , Ġperform Ġsignificantly Ġbetter Ġthan Ġothers Ġincluding Ġmost Ġglobal Ġmodels , Ġdemonstrating Ġthat Ġthe Ġeffectiveness Ġof Ġneural Ġmodels Ġdeals Ġwith Ġthe Ġfirst Ġlimitation Ġin Ġthe Ġintroduction . </s> Imp act Ġof ĠNC EL ĠMod ules </s> As Ġshown Ġin ĠFigure ĠFIG REF 28 Ġ, Ġthe Ġprior Ġprobability Ġperforms Ġquite Ġwell Ġin ĠT AC 2010 Ġbut Ġpoorly Ġin ĠWW . ĠCompared Ġwith ĠNC EL - local , Ġthe Ġglobal Ġmodule Ġin ĠNC EL Ġbrings Ġmore Ġimprovements Ġin Ġthe ĠâĢ ľ hard " Ġcase Ġthan Ġthat Ġfor ĠâĢ ľ easy " Ġdataset , Ġbecause Ġlocal Ġfeatures Ġare Ġdiscrim in ative Ġenough Ġin Ġmost Ġcases Ġof ĠT AC 2010 , Ġand Ġglobal Ġinformation Ġbecomes Ġquite Ġhelpful Ġwhen Ġlocal Ġfeatures Ġcannot Ġhandle . ĠThat Ġis , Ġour Ġpropose Ġcollective Ġmodel Ġis Ġrobust Ġand Ġshows Ġa Ġgood Ġgeneral ization Ġability Ġto Ġdifficult ĠEL . ĠThe Ġimprovements Ġby Ġeach Ġmain Ġmodule Ġare Ġrelatively Ġsmall Ġin ĠT AC 2010 , Ġwhile Ġthe Ġmodules Ġof Ġattention Ġand Ġembed ding Ġfeatures Ġshow Ġnon - neg lig ible Ġimpacts Ġin ĠWW Ġ( even Ġworse Ġthan Ġlocal Ġmodel ), Ġmainly Ġbecause ĠWW Ġcontains Ġmuch Ġnoise , Ġand Ġthese Ġtwo Ġmodules Ġare Ġeffective Ġin Ġimproving Ġthe Ġrobust ness Ġto Ġnoise Ġand Ġthe Ġability Ġof Ġgeneral ization Ġby Ġselecting Ġinformative Ġwords Ġand Ġproviding Ġmore Ġaccurate Ġsemantics , Ġrespectively . </s> Qual itative ĠAnalysis </s> The Ġresults Ġof Ġexample Ġin ĠFigure ĠFIG REF 1 Ġare Ġshown Ġin ĠTable ĠT AB REF 30 Ġ, Ġwhich Ġis Ġfrom ĠCo N LL Ġtest a Ġdataset . ĠFor Ġmention ĠEssex , Ġalthough Ġboth ĠNC EL Ġand ĠNC EL - local Ġcorrectly Ġidentify Ġentity ĠEssex ĠCounty ĠCricket ĠClub , ĠNC EL Ġoutputs Ġhigher Ġprobability Ġdue Ġto Ġthe Ġenhancement Ġof Ġneighbor Ġmentions . ĠMoreover , Ġfor Ġmention ĠEngland , ĠNC EL - local Ġcannot Ġfind Ġenough Ġdis amb ig uation Ġclues Ġfrom Ġits Ġcontext Ġwords , Ġsuch Ġas Ġsurplus Ġand Ġrequirements , Ġand Ġthus Ġassigns Ġa Ġhigher Ġprobability Ġof Ġ0 . 42 Ġto Ġthe Ġcountry ĠEngland Ġaccording Ġto Ġthe Ġprior Ġprobability . ĠCollect ively , ĠNC EL Ġcorrectly Ġidentifies ĠEngland Ġcricket Ġteam Ġwith Ġa Ġprobability Ġof Ġ0 . 72 Ġas Ġcompared Ġwith Ġ0 . 20 Ġin ĠNC EL - local Ġwith Ġthe Ġhelp Ġof Ġits Ġneighbor Ġmention ĠEssex . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġneural Ġmodel Ġfor Ġcollective Ġentity Ġlinking Ġthat Ġis Ġend - to - end Ġtrain able . ĠIt Ġapplies ĠGC N Ġon Ġsub graph s Ġinstead Ġof Ġthe Ġentire Ġentity Ġgraph Ġto Ġefficiently Ġlearn Ġfeatures Ġfrom Ġboth Ġlocal Ġand Ġglobal Ġinformation . ĠWe Ġdesign Ġan Ġattention Ġmechanism Ġthat Ġend ows ĠNC EL Ġrobust Ġto Ġnoisy Ġdata . ĠTr ained Ġon Ġcollected ĠWikipedia Ġhyper links , ĠNC EL Ġoutper forms Ġthe Ġstate - of - the - art Ġcollective Ġmethods Ġacross Ġfive Ġdifferent Ġdatasets . ĠBesides , Ġfurther Ġanalysis Ġof Ġthe Ġimpacts Ġof Ġmain Ġmodules Ġas Ġwell Ġas Ġqualitative Ġresults Ġdemonstrates Ġits Ġeffectiveness . </s> In Ġthe Ġfuture , Ġwe Ġwill Ġextend Ġour Ġmethod Ġinto Ġcross - ling ual Ġsettings Ġto Ġhelp Ġlink Ġentities Ġin Ġlow - res ourced Ġlanguages Ġby Ġexploiting Ġrich Ġknowledge Ġfrom Ġhigh - res ourced Ġlanguages , Ġand Ġdeal Ġwith ĠN IL Ġentities Ġto Ġfacilitate Ġspecific Ġapplications . </s> Acknowled gments </s> The Ġwork Ġis Ġsupported Ġby ĠNational ĠKey ĠResearch Ġand ĠDevelopment ĠProgram Ġof ĠChina Ġ( 2017 Y FB 100 21 01 ), ĠNS FC Ġkey Ġproject Ġ( U 17 36 204 , Ġ6 166 11 46 007 ), Ġand ĠTH UN US ĠN Ex T ĠCo - Lab .