Entities from a deep learning model are linked to the related entities from a knowledge base by a lookup.
ABSTRACTIVE
ELMo embeddings are then used with a residual LSTM to learn informative morphological representations from the character sequence of each token
EXTRACTIVE
While these knowledge bases provide semantically rich and fine-granular classes and relationship types, the task of entity classification often requires associating coarse-grained classes with discovered surface forms of entities. Most existing studies consider NER and entity linking as two separate tasks, whereas we try to combine the two. It has been shown that one can significantly increase the semantic information carried by a NER system when we successfully linking entities from a deep learning method to the related entities from a knowledge base BIBREF26 , BIBREF27 . Redirection: For the Wikidata linking element, we recognize that the lookup will be constrained by the most common lookup name for each entity. Consider the utterance (referring to the NBA basketball player) from Figure FIGREF12 “Michael Jeffrey Jordan in San Jose” as an example. The lookup for this entity in Wikidata is “Michael Jordan” and consequently will not be picked up if we were to use an exact string match. A simple method to circumvent such a problem is the usage of a redirection list. Such a list is provided on an entity by entity basis in the “Also known as” section in Wikidata. Using this redirection list, when we do not find an exact string match improves the recall of our model by 5-10%. Moreover, with the example of Michael Jordan (person), using our current framework, we will always refer to the retired basketball player (Q41421). We will never, for instance, pick up Michael Jordan (Q27069141) the American football cornerback. Or in fact any other Michael Jordan, famous or otherwise. One possible method to overcome this is to add a disambiguation layer, which seeks to use context from earlier parts of the text. This is, however, work for future improvement and we only consider the most common version of that entity.
The architecture of our proposed model is shown in Figure FIGREF12 . The input is a list of tokens and the output are the predicted entity types. The ELMo embeddings are then used with a residual LSTM to learn informative morphological representations from the character sequence of each token. We then pass this to a softmax layer as a tag decoder to predict the entity types.
1904.10503
Introduction </s> N amed Ġentity Ġrecognition Ġ( NER ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġis Ġthe Ġprocess Ġby Ġwhich Ġwe Ġidentify Ġtext Ġspans Ġwhich Ġmention Ġnamed Ġentities , Ġand Ġto Ġclassify Ġthem Ġinto Ġpred efined Ġcategories Ġsuch Ġas Ġperson , Ġlocation , Ġorganization Ġetc . ĠN ER Ġserves Ġas Ġthe Ġbasis Ġfor Ġa Ġvariety Ġof Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġapplications Ġsuch Ġas Ġrelation Ġextraction ĠB IB REF 4 Ġ, Ġmachine Ġtranslation ĠB IB REF 5 Ġ, Ġquestion Ġanswering ĠB IB REF 6 Ġand Ġknowledge Ġbase Ġconstruction ĠB IB REF 7 Ġ. ĠAlthough Ġearly ĠN ER Ġsystems Ġhave Ġbeen Ġsuccessful Ġin Ġproducing Ġadequate Ġrecognition Ġaccuracy , Ġthey Ġoften Ġrequire Ġsignificant Ġhuman Ġeffort Ġin Ġcarefully Ġdesigning Ġrules Ġor Ġfeatures . </s> In Ġrecent Ġyears , Ġdeep Ġlearning Ġmethods Ġbeen Ġemployed Ġin ĠN ER Ġsystems , Ġyielding Ġstate - of - the - art Ġperformance . ĠHowever , Ġthe Ġnumber Ġof Ġtypes Ġdetected Ġare Ġstill Ġnot Ġsufficient Ġfor Ġcertain Ġdomain - specific Ġapplications . ĠFor Ġrelation Ġextraction , Ġidentifying Ġfine - gr ained Ġtypes Ġhas Ġbeen Ġshown Ġto Ġsignificantly Ġincrease Ġthe Ġperformance Ġof Ġthe Ġextract or ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġsince Ġthis Ġhelps Ġin Ġfiltering Ġout Ġcandidate Ġrelation Ġtypes Ġwhich Ġdo Ġnot Ġfollow Ġthis Ġtype Ġconstraint . ĠFurthermore , Ġfor Ġquestion Ġanswering Ġfine - gr ained ĠNamed ĠEntity ĠRecogn ition Ġ( F g NER ) Ġcan Ġprovide Ġadditional Ġinformation Ġhelping Ġto Ġmatch Ġquestions Ġto Ġits Ġpotential Ġanswers Ġthus Ġimproving Ġperformance ĠB IB REF 10 Ġ. ĠFor Ġexample , ĠLi Ġand ĠRoth ĠB IB REF 11 Ġrank Ġquestions Ġbased Ġon Ġtheir Ġexpected Ġanswer Ġtypes Ġ( i . e . Ġwill Ġthe Ġanswer Ġbe Ġfood , Ġvehicle Ġor Ġdisease ). </s> Typically , ĠF g NER Ġsystems Ġuse Ġover Ġa Ġhundred Ġlabels , Ġarranged Ġin Ġa Ġhierarchical Ġstructure . ĠWe Ġfind Ġthat Ġavailable Ġtraining Ġdata Ġfor ĠF g NER Ġtypically Ġcontain Ġnoisy Ġlabels , Ġand Ġcreating Ġmanually Ġannot ated Ġtraining Ġdata Ġfor ĠF g NER Ġis Ġa Ġtime - consuming Ġprocess . ĠFurthermore , Ġhuman Ġannot ators Ġwill Ġhave Ġto Ġassign Ġa Ġsubset Ġof Ġcorrect Ġlabels Ġfrom Ġhundreds Ġof Ġpossible Ġlabels Ġmaking Ġthis Ġa Ġsomewhat Ġardu ous Ġtask . ĠCurrently , ĠF g NER Ġsystems Ġuse Ġdistant Ġsupervision ĠB IB REF 12 Ġto Ġautomatically Ġgenerate Ġtraining Ġdata . ĠDist ant Ġsupervision Ġis Ġa Ġtechnique Ġwhich Ġmaps Ġeach Ġentity Ġin Ġthe Ġcorpus Ġto Ġknowledge Ġbases Ġsuch Ġas ĠFree base ĠB IB REF 13 Ġ, ĠDB pedia ĠB IB REF 14 Ġ, ĠY AG O ĠB IB REF 15 Ġand Ġhelps Ġwith Ġthe Ġgeneration Ġof Ġlabeled Ġdata . ĠThis Ġmethod Ġwill Ġassign Ġthe Ġsame Ġset Ġof Ġlabels Ġto Ġall Ġmentions Ġof Ġa Ġparticular Ġentity Ġin Ġthe Ġcorpus . ĠFor Ġexample , ĠâĢ ľ Bar ack ĠObama âĢ Ŀ Ġis Ġa Ġperson , Ġpolitician , Ġlawyer , Ġand Ġauthor . ĠIf Ġa Ġknowledge Ġbase Ġhas Ġthese Ġfour Ġmatching Ġlabels , Ġthe Ġdistant Ġsupervision Ġtechnique Ġwill Ġassign Ġall Ġof Ġthem Ġto Ġevery Ġmention Ġof ĠâĢ ľ Bar ack ĠObama âĢ Ŀ . ĠTherefore , Ġthe Ġtraining Ġdata Ġwill Ġalso Ġfail Ġto Ġdistinguish Ġbetween Ġmentions Ġof ĠâĢ ľ Bar ack ĠObama âĢ Ŀ Ġin Ġall Ġsubsequent Ġutter ances . </s> L ing Ġet Ġal . Ġling 2012 fine Ġproposed Ġthe Ġfirst Ġsystem Ġfor ĠF g NER , Ġwhere Ġthey Ġused Ġ112 Ġoverlapping Ġlabels Ġwith Ġa Ġlinear Ġclass ifier Ġpercept ron Ġfor Ġmulti - label Ġclassification . ĠYose f Ġet Ġal . Ġspan iol 2012 hy ena Ġused Ġmultiple Ġbinary ĠS VM Ġclass ifiers Ġto Ġassign Ġentities Ġto Ġa Ġset Ġof Ġ505 Ġtypes . ĠGill ick Ġet Ġal . Ġg ill ick 2014 context Ġintroduced Ġcontext Ġdependent ĠF g NER Ġand Ġproposed Ġa Ġset Ġof Ġhe ur istics Ġfor Ġpr uning Ġlabels Ġthat Ġmight Ġnot Ġbe Ġrelevant Ġgiven Ġthe Ġlocal Ġcontext Ġof Ġthe Ġentity . ĠYog at ama Ġet Ġal . Ġyog at ama 2015 embed ding Ġproposed Ġan Ġembed ding Ġbased Ġmodel Ġwhere Ġuser - defined Ġfeatures Ġand Ġlabels Ġwere Ġembedded Ġinto Ġa Ġlow Ġdimensional Ġfeature Ġspace Ġto Ġfacilitate Ġinformation Ġsharing Ġamong Ġlabels . </s> Sh ima oka Ġet Ġal . Ġsh ima oka 2016 att ent ive Ġproposed Ġan Ġattentive Ġneural Ġnetwork Ġmodel Ġwhich Ġused Ġlong Ġshort - term Ġmemory Ġ( L ST Ms ) Ġto Ġencode Ġthe Ġcontext Ġof Ġthe Ġentity , Ġthen Ġused Ġan Ġattention Ġmechanism Ġto Ġallow Ġthe Ġmodel Ġto Ġfocus Ġon Ġrelevant Ġexpressions Ġin Ġthe Ġentity Ġmention 's Ġcontext . ĠTo Ġlearn Ġentity Ġrepresentations , Ġwe Ġpropose Ġa Ġscheme Ġwhich Ġis Ġpotentially Ġmore Ġgeneral izable . </s> Dat as ets </s> We Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġpublicly Ġavailable Ġdatasets . ĠThe Ġstatistics Ġfor Ġboth Ġare Ġshown Ġin ĠTable ĠT AB REF 3 Ġ. ĠThe Ġdetails Ġof Ġthese Ġdatasets Ġare Ġas Ġfollows : </s> Ont o Notes : ĠOnt o Notes Ġ5 . 0 ĠB IB REF 16 Ġincludes Ġtexts Ġfrom Ġfive Ġdifferent Ġtext Ġgenres : Ġbroadcast Ġconversation Ġ( 200 k ), Ġbroadcast Ġnews Ġ( 200 k ), Ġmagazine Ġ( 120 k ), Ġnews wire Ġ( 625 k ), Ġand Ġweb Ġdata Ġ( 300 k ). ĠThis Ġdataset Ġis Ġannot ated Ġwith Ġ18 Ġcategories . </s> Wiki ( gold ): ĠThe Ġtraining Ġdata Ġconsists Ġof ĠWikipedia Ġsentences Ġand Ġwas Ġautomatically Ġgenerated Ġusing Ġa Ġdistant Ġsupervision Ġmethod , Ġmapping Ġhyper links Ġin ĠWikipedia Ġarticles Ġto ĠFree base , Ġwhich Ġwe Ġdo Ġnot Ġuse Ġin Ġthis Ġstudy . ĠThe Ġtest Ġdata , Ġmainly Ġconsisting Ġof Ġsentences Ġfrom Ġnews Ġreports , Ġwas Ġmanually Ġannot ated Ġas Ġdescribed Ġin ĠB IB REF 8 Ġ. ĠThe Ġclass Ġhierarchy Ġis Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThis Ġdataset Ġis Ġannot ated Ġwith Ġ7 Ġmain Ġcategories Ġ( bold Ġtext Ġin ĠFigure ĠFIG REF 2 Ġ), Ġwhich Ġmaps Ġdirectly Ġto ĠOnt o Notes . ĠThe Ġmis cellaneous Ġcategory Ġin ĠFigure ĠFIG REF 2 Ġdoes Ġnot Ġhave Ġdirect Ġm appings , Ġso Ġfuture Ġwork Ġmay Ġinclude Ġredef ining Ġthese Ġcategories Ġso Ġthe Ġm appings Ġare Ġmore Ġmeaningful . </s> E val uation ĠMet rics </s> NER Ġinvolves Ġidentifying Ġboth Ġentity Ġboundaries Ġand Ġentity Ġtypes . ĠWith ĠâĢ ľ ex act - match Ġevaluation âĢ Ŀ , Ġa Ġnamed Ġentity Ġis Ġconsidered Ġcorrectly Ġrecognized Ġonly Ġif Ġboth Ġthe Ġboundaries Ġand Ġtype Ġmatch Ġthe Ġground Ġtruth ĠB IB REF 8 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠPrecision , ĠRecall , Ġand ĠF - 1 Ġscores Ġare Ġcomputed Ġon Ġthe Ġnumber Ġof Ġtrue Ġpositives Ġ( TP ), Ġfalse Ġpositives Ġ( FP ), Ġand Ġfalse Ġnegatives Ġ( FN ). ĠTheir Ġformal Ġdefinitions Ġare Ġas Ġfollows : </s> True ĠPositive Ġ( TP ): Ġentities Ġthat Ġare Ġrecognized Ġby ĠN ER Ġand Ġmatch Ġthe Ġground Ġtruth . </s> False ĠPositive Ġ( FP ): Ġentities Ġthat Ġare Ġrecognized Ġby ĠN ER Ġbut Ġdo Ġnot Ġmatch Ġthe Ġground Ġtruth . </s> False ĠNegative Ġ( FN ): Ġentities Ġannot ated Ġin Ġthe Ġground Ġwhich Ġthat Ġare Ġnot Ġrecognized Ġby ĠN ER . </s> Pre cision Ġmeasures Ġthe Ġability Ġof Ġa ĠN ER Ġsystem Ġto Ġpresent Ġonly Ġcorrect Ġentities , Ġand ĠRecall Ġmeasures Ġthe Ġability Ġof Ġa ĠN ER Ġsystem Ġto Ġrecognize Ġall Ġentities Ġin Ġa Ġcorpus . ĠDIS PLAY FORM 0 </s> The ĠF - 1 Ġscore Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall , Ġand Ġthe Ġbalanced ĠF - 1 Ġscore Ġis Ġthe Ġvariant Ġwhich Ġis Ġmost Ġcommonly Ġused . ĠThis Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> Since Ġmost ĠN ER Ġsystems Ġinvolve Ġmultiple Ġentity Ġtypes , Ġit Ġis Ġoften Ġrequired Ġto Ġassess Ġthe Ġperformance Ġacross Ġall Ġentity Ġclasses . ĠTwo Ġmeasures Ġare Ġcommonly Ġused Ġfor Ġthis Ġpurpose : Ġthe Ġmacro - aver aged ĠF - 1 Ġscore Ġand Ġthe Ġmicro - aver aged ĠF - 1 Ġscore . ĠThe Ġmacro - aver aged ĠF - 1 Ġscore Ġcomp utes Ġthe ĠF - 1 Ġscore Ġindependently Ġfor Ġeach Ġentity Ġtype , Ġthen Ġtakes Ġthe Ġaverage Ġ( hen ce Ġtreating Ġall Ġentity Ġtypes Ġequally ). ĠThe Ġmicro - aver aged ĠF - 1 Ġscore Ġaggreg ates Ġthe Ġcontributions Ġof Ġentities Ġfrom Ġall Ġclasses Ġto Ġcompute Ġthe Ġaverage Ġ( t reating Ġall Ġentities Ġequally ). ĠWe Ġuse Ġthe Ġmicro - aver aged ĠF - 1 Ġin Ġour Ġstudy Ġsince Ġthis Ġaccounts Ġfor Ġlabel Ġim bal ances Ġin Ġthe Ġevaluation Ġdata Ġand Ġtherefore Ġa Ġmore Ġmeaningful Ġstatistic . </s> Method </s> Over Ġthe Ġfew Ġpast Ġyears , Ġthe Ġemergence Ġof Ġdeep Ġneural Ġnetworks Ġhas Ġfundamentally Ġchanged Ġthe Ġdesign Ġof Ġentity Ġdetection Ġsystems . ĠConsequently , Ġrecurrent Ġneural Ġnetworks Ġ( R NN ) Ġhave Ġfound Ġpopularity Ġin Ġthe Ġfield Ġsince Ġthey Ġare Ġable Ġto Ġlearn Ġlong Ġterm Ġdependencies Ġof Ġsequential Ġdata . ĠThe Ġrecent Ġsuccess Ġof Ġneural Ġnetwork Ġbased Ġarchitectures Ġprincipally Ġcomes Ġfrom Ġits Ġdeep Ġstructure . ĠTraining Ġa Ġdeep Ġneural Ġnetwork , Ġhowever , Ġis Ġa Ġdifficult Ġproblem Ġdue Ġto Ġvanishing Ġor Ġexploding Ġgrad ients . ĠIn Ġorder Ġto Ġsolve Ġthis , ĠL ST Ms Ġwere Ġproposed . ĠAn ĠL ST M Ġis Ġan Ġinternal Ġmemory Ġcell Ġcontrolled Ġby Ġforget Ġgate Ġand Ġinput Ġgate Ġnetworks . ĠA Ġforget Ġgate Ġin Ġan ĠL ST M Ġlayer Ġwhich Ġdetermines Ġhow Ġmuch Ġprior Ġmemory Ġshould Ġbe Ġpassed Ġinto Ġthe Ġnext Ġtime Ġincrement . ĠSimilarly , Ġan Ġinput Ġgate Ġscales Ġnew Ġinput Ġto Ġmemory Ġcells . ĠDepending Ġon Ġthe Ġstates Ġof Ġboth Ġgates , ĠL ST M Ġis Ġable Ġto Ġcapture Ġlong - term Ġor Ġshort - term Ġdependencies Ġfor Ġsequential Ġdata . ĠThis Ġis Ġan Ġideal Ġproperty Ġfor Ġmany ĠN LP Ġtasks . </s> NER Ġusing ĠEL Mo </s> Recently , ĠPeters Ġet Ġal . ĠB IB REF 19 Ġproposed ĠEL Mo Ġword Ġrepresentations . ĠEL Mo Ġextends Ġa Ġtraditional Ġword Ġembed ding Ġmodel Ġwith Ġfeatures Ġproduced Ġbid irection ally Ġwith Ġcharacter Ġconv olutions . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġthe Ġutilization Ġof ĠEL Mo Ġfor Ġdifferent ĠN LP Ġtasks Ġresult Ġin Ġimproved Ġperformance Ġcompared Ġto Ġother Ġtypes Ġof Ġword Ġembed ding Ġmodels Ġsuch Ġas ĠWord 2 V ec ĠB IB REF 20 Ġ, ĠGl o Ve ĠB IB REF 21 Ġ, Ġand Ġfast Text ĠB IB REF 22 Ġ. </s> The Ġarchitecture Ġof Ġour Ġproposed Ġmodel Ġis Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. ĠThe Ġinput Ġis Ġa Ġlist Ġof Ġtokens Ġand Ġthe Ġoutput Ġare Ġthe Ġpredicted Ġentity Ġtypes . ĠThe ĠEL Mo Ġembed d ings Ġare Ġthen Ġused Ġwith Ġa Ġresidual ĠL ST M Ġto Ġlearn Ġinformative Ġmorph ological Ġrepresentations Ġfrom Ġthe Ġcharacter Ġsequence Ġof Ġeach Ġtoken . ĠWe Ġthen Ġpass Ġthis Ġto Ġa Ġsoft max Ġlayer Ġas Ġa Ġtag Ġdec oder Ġto Ġpredict Ġthe Ġentity Ġtypes . </s> Hyper param eter Ġsettings : ĠThe Ġhidden - layer Ġsize Ġof Ġeach ĠL ST M Ġwithin Ġthe Ġmodel Ġis Ġset Ġ512 . ĠWe Ġuse Ġa Ġdrop out Ġwith Ġthe Ġprobability Ġof Ġ0 . 2 Ġon Ġthe Ġoutput Ġof Ġthe ĠL ST M Ġenc od ers . ĠThe Ġembed ding Ġdimension Ġfrom ĠEL Mo Ġis Ġ1024 . ĠThe Ġoptimization Ġmethod Ġwe Ġuse Ġis ĠAdam ĠB IB REF 23 Ġ. ĠWe Ġtrain Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġfor Ġ30 Ġepoch s . ĠThe Ġmodel Ġwas Ġimplemented Ġusing Ġthe ĠT ensor Flow Ġframework . </s> Entity ĠLink ing Ġusing ĠWik id ata </s> Entity Ġlinking Ġ( EL ) ĠB IB REF 24 Ġ, Ġalso Ġknown Ġas Ġnamed Ġentity Ġdis amb ig uation Ġor Ġnormal ization , Ġis Ġthe Ġtask Ġto Ġdetermine Ġthe Ġidentity Ġof Ġentities Ġmentioned Ġin Ġa Ġpiece Ġof Ġtext Ġwith Ġreference Ġto Ġa Ġknowledge Ġbase . ĠThere Ġare Ġa Ġnumber Ġof Ġknowledge Ġbases Ġthat Ġprovide Ġa Ġbackground Ġrepository Ġfor Ġentity Ġclassification Ġof Ġthis Ġtype . ĠFor Ġthis Ġstudy , Ġwe Ġuse ĠWik id ata , Ġwhich Ġcan Ġbe Ġseen Ġdiagram matically Ġin ĠFigure ĠFIG REF 12 Ġ. ĠSystems Ġsuch Ġas ĠDeep Type ĠB IB REF 25 Ġintegrate Ġsymbolic Ġinformation Ġinto Ġthe Ġreasoning Ġprocess Ġof Ġa Ġneural Ġnetwork Ġwith Ġa Ġtype Ġsystem Ġand Ġshow Ġstate - of - the - art Ġperformances Ġfor ĠEL . ĠThey Ġdo Ġnot , Ġhowever , Ġquote Ġresults Ġon ĠWiki ( gold ) Ġso Ġa Ġdirect Ġcomparison Ġis Ġdifficult . </s> While Ġthese Ġknowledge Ġbases Ġprovide Ġsem antically Ġrich Ġand Ġfine - gran ular Ġclasses Ġand Ġrelationship Ġtypes , Ġthe Ġtask Ġof Ġentity Ġclassification Ġoften Ġrequires Ġassoci ating Ġcoarse - gr ained Ġclasses Ġwith Ġdiscovered Ġsurface Ġforms Ġof Ġentities . ĠMost Ġexisting Ġstudies Ġconsider ĠN ER Ġand Ġentity Ġlinking Ġas Ġtwo Ġseparate Ġtasks , Ġwhereas Ġwe Ġtry Ġto Ġcombine Ġthe Ġtwo . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġone Ġcan Ġsignificantly Ġincrease Ġthe Ġsemantic Ġinformation Ġcarried Ġby Ġa ĠN ER Ġsystem Ġwhen Ġwe Ġsuccessfully Ġlinking Ġentities Ġfrom Ġa Ġdeep Ġlearning Ġmethod Ġto Ġthe Ġrelated Ġentities Ġfrom Ġa Ġknowledge Ġbase ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> Red irection : ĠFor Ġthe ĠWik id ata Ġlinking Ġelement , Ġwe Ġrecognize Ġthat Ġthe Ġlookup Ġwill Ġbe Ġconstrained Ġby Ġthe Ġmost Ġcommon Ġlookup Ġname Ġfor Ġeach Ġentity . ĠConsider Ġthe Ġutter ance Ġ( re fer ring Ġto Ġthe ĠNBA Ġbasketball Ġplayer ) Ġfrom ĠFigure ĠFIG REF 12 ĠâĢ ľ Michael ĠJeffrey ĠJordan Ġin ĠSan ĠJose âĢ Ŀ Ġas Ġan Ġexample . ĠThe Ġlookup Ġfor Ġthis Ġentity Ġin ĠWik id ata Ġis ĠâĢ ľ Michael ĠJordan âĢ Ŀ Ġand Ġconsequently Ġwill Ġnot Ġbe Ġpicked Ġup Ġif Ġwe Ġwere Ġto Ġuse Ġan Ġexact Ġstring Ġmatch . ĠA Ġsimple Ġmethod Ġto Ġcircumvent Ġsuch Ġa Ġproblem Ġis Ġthe Ġusage Ġof Ġa Ġred irection Ġlist . ĠSuch Ġa Ġlist Ġis Ġprovided Ġon Ġan Ġentity Ġby Ġentity Ġbasis Ġin Ġthe ĠâĢ ľ Also Ġknown Ġas âĢ Ŀ Ġsection Ġin ĠWik id ata . ĠUsing Ġthis Ġred irection Ġlist , Ġwhen Ġwe Ġdo Ġnot Ġfind Ġan Ġexact Ġstring Ġmatch Ġimproves Ġthe Ġrecall Ġof Ġour Ġmodel Ġby Ġ5 - 10 %. ĠMoreover , Ġwith Ġthe Ġexample Ġof ĠMichael ĠJordan Ġ( person ), Ġusing Ġour Ġcurrent Ġframework , Ġwe Ġwill Ġalways Ġrefer Ġto Ġthe Ġretired Ġbasketball Ġplayer Ġ( Q 414 21 ). ĠWe Ġwill Ġnever , Ġfor Ġinstance , Ġpick Ġup ĠMichael ĠJordan Ġ( Q 27 06 9 141 ) Ġthe ĠAmerican Ġfootball Ġcornerback . ĠOr Ġin Ġfact Ġany Ġother ĠMichael ĠJordan , Ġfamous Ġor Ġotherwise . ĠOne Ġpossible Ġmethod Ġto Ġovercome Ġthis Ġis Ġto Ġadd Ġa Ġdis amb ig uation Ġlayer , Ġwhich Ġseeks Ġto Ġuse Ġcontext Ġfrom Ġearlier Ġparts Ġof Ġthe Ġtext . ĠThis Ġis , Ġhowever , Ġwork Ġfor Ġfuture Ġimprovement Ġand Ġwe Ġonly Ġconsider Ġthe Ġmost Ġcommon Ġversion Ġof Ġthat Ġentity . </s> Cl ust ering : ĠThe ĠWik id ata Ġtax onomy Ġprovides Ġthousands Ġof Ġpossible Ġinstance Ġof , Ġand Ġsubclass Ġof Ġtypes Ġfor Ġour Ġentities . ĠConsequently , Ġin Ġorder Ġto Ġperform Ġa Ġmeaningful Ġvalidation Ġof Ġour Ġmodel , Ġwe Ġmust Ġfind Ġa Ġway Ġto Ġcluster Ġthese Ġonto Ġthe Ġ112 Ġtypes Ġprovided Ġby ĠWiki ( gold ). ĠOur Ġclust ering Ġis Ġperformed Ġas Ġfollows : </s> If Ġthe Ġentity Ġtype Ġis Ġeither Ġperson , Ġlocation , Ġorganization Ġwe Ġuse Ġthe ĠN ECK Ar ĠB IB REF 28 Ġtool Ġto Ġnarrow Ġdown Ġour Ġlist Ġof Ġsearch able Ġentities . </s> We Ġthen Ġlook Ġat Ġeither Ġthe Ġoccupation Ġfor Ġperson , Ġor Ġinstance Ġof Ġfor Ġlocation / organ ization Ġcategories Ġto Ġmap Ġto Ġthe Ġavailable Ġsub types . </s> If Ġthe Ġentity Ġtype Ġis Ġnot Ġperson , Ġlocation , Ġor Ġorganization Ġwe Ġsearch Ġall Ġof ĠWik id ata . </s> The Ġclust ering Ġwe Ġperform Ġin Ġpart Ġ1 Ġor Ġ2 Ġis Ġfrom Ġa Ġcos ine Ġsimilarity Ġof Ġthe Ġentity Ġdescription Ġto Ġthe Ġlist Ġof Ġpossible Ġsub types Ġfor Ġthat Ġentity . ĠFor Ġthis Ġwe Ġuse ĠWord 2 V ec Ġword Ġembed d ings Ġtrained Ġon ĠWikipedia . ĠWe Ġset Ġthe Ġminimum Ġthreshold Ġof Ġthe Ġaverage Ġcos ine Ġsimilarity Ġto Ġbe Ġ0 . 1 . </s> As Ġan Ġexample , Ġconsider Ġthe Ġtest Ġsentence : ĠâĢ ľ The Ġdevice Ġwill Ġbe Ġavailable Ġon Ġsale Ġon Ġ20 th ĠApril Ġ2011 Ġon Ġam azon Ġu k ĠApple 's ĠiPad âĢ Ŀ Ġfrom ĠFigure ĠFIG REF 18 Ġ. ĠFirst , Ġwe Ġtag ĠiPad Ġas Ġproduct Ġusing Ġthe Ġcontext Ġenc oder Ġdescribed Ġin ĠSection Ġ2 . 1 . ĠWe Ġthen Ġsearch ĠWik id ata Ġand Ġreturn Ġthe Ġmost Ġcommon Ġvariant Ġfor Ġthat Ġentity Ġin Ġthis Ġcase ĠQ 27 96 Ġ( the Ġmost Ġreferenced Ġvariant Ġis Ġthe Ġone Ġwith Ġthe Ġlowest ĠQ - id ). ĠWe Ġthen Ġcalculate Ġa Ġcos ine Ġsimilarity Ġof Ġthe Ġdescription , Ġin Ġthis Ġcase ĠâĢ ľ line Ġof Ġtablet Ġcomputers âĢ Ŀ , Ġwith Ġthe Ġpossible Ġsub types Ġof Ġproduct . ĠThe Ġpossible Ġsub types , Ġin Ġthis Ġcase , Ġare Ġengine , Ġairplane , Ġcar , Ġship , Ġspacecraft , Ġtrain , Ġcamera , Ġmobile Ġphone , Ġcomputer , Ġsoftware , Ġgame , Ġinstrument , Ġship , Ġweapon . ĠWe Ġreturn Ġthe Ġhighest Ġresult Ġabove Ġ0 . 1 , Ġwhich Ġin Ġthis Ġcase Ġis Ġcomputer Ġ( 0 . 54 ). </s> Results </s> The Ġresults Ġfor Ġeach Ġclass Ġtype Ġare Ġshown Ġin ĠTable ĠT AB REF 19 Ġ, Ġwith Ġsome Ġspecific Ġexamples Ġshown Ġin ĠFigure ĠFIG REF 18 Ġ. ĠFor Ġthe ĠWiki ( gold ) Ġwe Ġquote Ġthe Ġmicro - aver aged ĠF - 1 Ġscores Ġfor Ġthe Ġentire Ġtop Ġlevel Ġentity Ġcategory . ĠThe Ġtotal ĠF - 1 Ġscore Ġon Ġthe ĠOnt o Notes Ġdataset Ġis Ġ88 %, Ġand Ġthe Ġtotal ĠF - 1 Ġcross - valid ation Ġscore Ġon Ġthe Ġ112 Ġclass ĠWiki ( gold ) Ġdataset Ġis Ġ53 %. ĠIt Ġis Ġworth Ġnoting Ġthat Ġone Ġcould Ġimprove ĠWiki ( gold ) Ġresults Ġby Ġtraining Ġdirectly Ġusing Ġthis Ġdataset . ĠHowever , Ġthe Ġaim Ġis Ġnot Ġto Ġtune Ġour Ġmodel Ġspecifically Ġon Ġthis Ġclass Ġhierarchy . ĠWe Ġinstead Ġaim Ġto Ġpresent Ġa Ġframework Ġwhich Ġcan Ġbe Ġmodified Ġeasily Ġto Ġany Ġdomain Ġhierarchy Ġand Ġhas Ġacceptable Ġout - of - the - box Ġperformances Ġto Ġany Ġfine - gr ained Ġdataset . ĠThe Ġresults Ġin ĠTable ĠT AB REF 19 Ġ( Ont o Notes ) Ġonly Ġshow Ġthe Ġmain Ġ7 Ġcategories Ġin ĠOnt o Notes Ġwhich Ġmap Ġto ĠWiki ( gold ) Ġfor Ġclarity . ĠThe Ġother Ġcategories Ġ( date , Ġtime , Ġnor p , Ġlanguage , Ġord inal , Ġcardinal , Ġquantity , Ġpercent , Ġmoney , Ġlaw ) Ġhave ĠF - 1 Ġscores Ġbetween Ġ80 - 90 %, Ġwith Ġthe Ġexception Ġof Ġtime Ġ( 65 %) </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġdeep Ġneural Ġnetwork Ġmodel Ġfor Ġthe Ġtask Ġof Ġfine - gr ained Ġnamed Ġentity Ġclassification Ġusing ĠEL Mo Ġembed d ings Ġand ĠWik id ata . ĠThe Ġproposed Ġmodel Ġlearns Ġrepresentations Ġfor Ġentity Ġmentions Ġbased Ġon Ġits Ġcontext Ġand Ġincorporates Ġthe Ġrich Ġstructure Ġof ĠWik id ata Ġto Ġaugment Ġthese Ġlabels Ġinto Ġfiner - gr ained Ġsub types . ĠWe Ġcan Ġsee Ġcomparisons Ġof Ġour Ġmodel Ġmade Ġon ĠWiki ( gold ) Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġnote Ġthat Ġthe Ġmodel Ġperforms Ġsimilarly Ġto Ġexisting Ġsystems Ġwithout Ġbeing Ġtrained Ġor Ġtuned Ġon Ġthat Ġparticular Ġdataset . ĠFuture Ġwork Ġmay Ġinclude Ġrefining Ġthe Ġclust ering Ġmethod Ġdescribed Ġin ĠSection Ġ2 . 2 Ġto Ġextend Ġto Ġtypes Ġother Ġthan Ġperson , Ġlocation , Ġorganization , Ġand Ġalso Ġto Ġinclude Ġdis amb ig uation Ġof Ġentity Ġtypes .
Introduction Named entity recognition (NER) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 is the process by which we identify text spans which mention named entities, and to classify them into predefined categories such as person, location, organization etc. NER serves as the basis for a variety of natural language processing (NLP) applications such as relation extraction BIBREF4 , machine translation BIBREF5 , question answering BIBREF6 and knowledge base construction BIBREF7 . Although early NER systems have been successful in producing adequate recognition accuracy, they often require significant human effort in carefully designing rules or features. In recent years, deep learning methods been employed in NER systems, yielding state-of-the-art performance. However, the number of types detected are still not sufficient for certain domain-specific applications. For relation extraction, identifying fine-grained types has been shown to significantly increase the performance of the extractor BIBREF8 , BIBREF9 since this helps in filtering out candidate relation types which do not follow this type constraint. Furthermore, for question answering fine-grained Named Entity Recognition (FgNER) can provide additional information helping to match questions to its potential answers thus improving performance BIBREF10 . For example, Li and Roth BIBREF11 rank questions based on their expected answer types (i.e. will the answer be food, vehicle or disease). Typically, FgNER systems use over a hundred labels, arranged in a hierarchical structure. We find that available training data for FgNER typically contain noisy labels, and creating manually annotated training data for FgNER is a time-consuming process. Furthermore, human annotators will have to assign a subset of correct labels from hundreds of possible labels making this a somewhat arduous task. Currently, FgNER systems use distant supervision BIBREF12 to automatically generate training data. Distant supervision is a technique which maps each entity in the corpus to knowledge bases such as Freebase BIBREF13 , DBpedia BIBREF14 , YAGO BIBREF15 and helps with the generation of labeled data. This method will assign the same set of labels to all mentions of a particular entity in the corpus. For example, “Barack Obama” is a person, politician, lawyer, and author. If a knowledge base has these four matching labels, the distant supervision technique will assign all of them to every mention of “Barack Obama”. Therefore, the training data will also fail to distinguish between mentions of “Barack Obama” in all subsequent utterances. Ling et al. ling2012fine proposed the first system for FgNER, where they used 112 overlapping labels with a linear classifier perceptron for multi-label classification. Yosef et al. spaniol2012hyena used multiple binary SVM classifiers to assign entities to a set of 505 types. Gillick et al. gillick2014context introduced context dependent FgNER and proposed a set of heuristics for pruning labels that might not be relevant given the local context of the entity. Yogatama et al. yogatama2015embedding proposed an embedding based model where user-defined features and labels were embedded into a low dimensional feature space to facilitate information sharing among labels. Shimaoka et al. shimaoka2016attentive proposed an attentive neural network model which used long short-term memory (LSTMs) to encode the context of the entity, then used an attention mechanism to allow the model to focus on relevant expressions in the entity mention's context. To learn entity representations, we propose a scheme which is potentially more generalizable. Datasets We evaluate our model on two publicly available datasets. The statistics for both are shown in Table TABREF3 . The details of these datasets are as follows: OntoNotes: OntoNotes 5.0 BIBREF16 includes texts from five different text genres: broadcast conversation (200k), broadcast news (200k), magazine (120k), newswire (625k), and web data (300k). This dataset is annotated with 18 categories. Wiki(gold): The training data consists of Wikipedia sentences and was automatically generated using a distant supervision method, mapping hyperlinks in Wikipedia articles to Freebase, which we do not use in this study. The test data, mainly consisting of sentences from news reports, was manually annotated as described in BIBREF8 . The class hierarchy is shown in Figure FIGREF2 . This dataset is annotated with 7 main categories (bold text in Figure FIGREF2 ), which maps directly to OntoNotes. The miscellaneous category in Figure FIGREF2 does not have direct mappings, so future work may include redefining these categories so the mappings are more meaningful. Evaluation Metrics NER involves identifying both entity boundaries and entity types. With “exact-match evaluation”, a named entity is considered correctly recognized only if both the boundaries and type match the ground truth BIBREF8 , BIBREF17 , BIBREF18 . Precision, Recall, and F-1 scores are computed on the number of true positives (TP), false positives (FP), and false negatives (FN). Their formal definitions are as follows: True Positive (TP): entities that are recognized by NER and match the ground truth. False Positive (FP): entities that are recognized by NER but do not match the ground truth. False Negative (FN): entities annotated in the ground which that are not recognized by NER. Precision measures the ability of a NER system to present only correct entities, and Recall measures the ability of a NER system to recognize all entities in a corpus. DISPLAYFORM0 The F-1 score is the harmonic mean of precision and recall, and the balanced F-1 score is the variant which is most commonly used. This is defined as: DISPLAYFORM0 Since most NER systems involve multiple entity types, it is often required to assess the performance across all entity classes. Two measures are commonly used for this purpose: the macro-averaged F-1 score and the micro-averaged F-1 score. The macro-averaged F-1 score computes the F-1 score independently for each entity type, then takes the average (hence treating all entity types equally). The micro-averaged F-1 score aggregates the contributions of entities from all classes to compute the average (treating all entities equally). We use the micro-averaged F-1 in our study since this accounts for label imbalances in the evaluation data and therefore a more meaningful statistic. Method Over the few past years, the emergence of deep neural networks has fundamentally changed the design of entity detection systems. Consequently, recurrent neural networks (RNN) have found popularity in the field since they are able to learn long term dependencies of sequential data. The recent success of neural network based architectures principally comes from its deep structure. Training a deep neural network, however, is a difficult problem due to vanishing or exploding gradients. In order to solve this, LSTMs were proposed. An LSTM is an internal memory cell controlled by forget gate and input gate networks. A forget gate in an LSTM layer which determines how much prior memory should be passed into the next time increment. Similarly, an input gate scales new input to memory cells. Depending on the states of both gates, LSTM is able to capture long-term or short-term dependencies for sequential data. This is an ideal property for many NLP tasks. NER using ELMo Recently, Peters et al. BIBREF19 proposed ELMo word representations. ELMo extends a traditional word embedding model with features produced bidirectionally with character convolutions. It has been shown that the utilization of ELMo for different NLP tasks result in improved performance compared to other types of word embedding models such as Word2Vec BIBREF20 , GloVe BIBREF21 , and fastText BIBREF22 . The architecture of our proposed model is shown in Figure FIGREF12 . The input is a list of tokens and the output are the predicted entity types. The ELMo embeddings are then used with a residual LSTM to learn informative morphological representations from the character sequence of each token. We then pass this to a softmax layer as a tag decoder to predict the entity types. Hyperparameter settings: The hidden-layer size of each LSTM within the model is set 512. We use a dropout with the probability of 0.2 on the output of the LSTM encoders. The embedding dimension from ELMo is 1024. The optimization method we use is Adam BIBREF23 . We train with a batch size of 32 for 30 epochs. The model was implemented using the TensorFlow framework. Entity Linking using Wikidata Entity linking (EL) BIBREF24 , also known as named entity disambiguation or normalization, is the task to determine the identity of entities mentioned in a piece of text with reference to a knowledge base. There are a number of knowledge bases that provide a background repository for entity classification of this type. For this study, we use Wikidata, which can be seen diagrammatically in Figure FIGREF12 . Systems such as DeepType BIBREF25 integrate symbolic information into the reasoning process of a neural network with a type system and show state-of-the-art performances for EL. They do not, however, quote results on Wiki(gold) so a direct comparison is difficult. While these knowledge bases provide semantically rich and fine-granular classes and relationship types, the task of entity classification often requires associating coarse-grained classes with discovered surface forms of entities. Most existing studies consider NER and entity linking as two separate tasks, whereas we try to combine the two. It has been shown that one can significantly increase the semantic information carried by a NER system when we successfully linking entities from a deep learning method to the related entities from a knowledge base BIBREF26 , BIBREF27 . Redirection: For the Wikidata linking element, we recognize that the lookup will be constrained by the most common lookup name for each entity. Consider the utterance (referring to the NBA basketball player) from Figure FIGREF12 “Michael Jeffrey Jordan in San Jose” as an example. The lookup for this entity in Wikidata is “Michael Jordan” and consequently will not be picked up if we were to use an exact string match. A simple method to circumvent such a problem is the usage of a redirection list. Such a list is provided on an entity by entity basis in the “Also known as” section in Wikidata. Using this redirection list, when we do not find an exact string match improves the recall of our model by 5-10%. Moreover, with the example of Michael Jordan (person), using our current framework, we will always refer to the retired basketball player (Q41421). We will never, for instance, pick up Michael Jordan (Q27069141) the American football cornerback. Or in fact any other Michael Jordan, famous or otherwise. One possible method to overcome this is to add a disambiguation layer, which seeks to use context from earlier parts of the text. This is, however, work for future improvement and we only consider the most common version of that entity. Clustering: The Wikidata taxonomy provides thousands of possible instance of, and subclass of types for our entities. Consequently, in order to perform a meaningful validation of our model, we must find a way to cluster these onto the 112 types provided by Wiki(gold). Our clustering is performed as follows: If the entity type is either person, location, organization we use the NECKAr BIBREF28 tool to narrow down our list of searchable entities. We then look at either the occupation for person, or instance of for location/organization categories to map to the available subtypes. If the entity type is not person, location, or organization we search all of Wikidata. The clustering we perform in part 1 or 2 is from a cosine similarity of the entity description to the list of possible subtypes for that entity. For this we use Word2Vec word embeddings trained on Wikipedia. We set the minimum threshold of the average cosine similarity to be 0.1. As an example, consider the test sentence: “The device will be available on sale on 20th April 2011 on amazon uk Apple's iPad” from Figure FIGREF18 . First, we tag iPad as product using the context encoder described in Section 2.1. We then search Wikidata and return the most common variant for that entity in this case Q2796 (the most referenced variant is the one with the lowest Q-id). We then calculate a cosine similarity of the description, in this case “line of tablet computers”, with the possible subtypes of product. The possible subtypes, in this case, are engine, airplane, car, ship, spacecraft, train, camera, mobile phone, computer, software, game, instrument, ship, weapon. We return the highest result above 0.1, which in this case is computer (0.54). Results The results for each class type are shown in Table TABREF19 , with some specific examples shown in Figure FIGREF18 . For the Wiki(gold) we quote the micro-averaged F-1 scores for the entire top level entity category. The total F-1 score on the OntoNotes dataset is 88%, and the total F-1 cross-validation score on the 112 class Wiki(gold) dataset is 53%. It is worth noting that one could improve Wiki(gold) results by training directly using this dataset. However, the aim is not to tune our model specifically on this class hierarchy. We instead aim to present a framework which can be modified easily to any domain hierarchy and has acceptable out-of-the-box performances to any fine-grained dataset. The results in Table TABREF19 (OntoNotes) only show the main 7 categories in OntoNotes which map to Wiki(gold) for clarity. The other categories (date, time, norp, language, ordinal, cardinal, quantity, percent, money, law) have F-1 scores between 80-90%, with the exception of time (65%) Conclusion and Future Work In this paper, we present a deep neural network model for the task of fine-grained named entity classification using ELMo embeddings and Wikidata. The proposed model learns representations for entity mentions based on its context and incorporates the rich structure of Wikidata to augment these labels into finer-grained subtypes. We can see comparisons of our model made on Wiki(gold) in Table TABREF20 . We note that the model performs similarly to existing systems without being trained or tuned on that particular dataset. Future work may include refining the clustering method described in Section 2.2 to extend to types other than person, location, organization, and also to include disambiguation of entity types.
How do they combine a deep learning model with a knowledge base?
1c997c268c68149ae6fb43d83ffcd53f0e7fe57e
How Ġdo Ġthey Ġcombine Ġa Ġdeep Ġlearning Ġmodel Ġwith Ġa Ġknowledge Ġbase ?
Ent ities Ġfrom Ġa Ġdeep Ġlearning Ġmodel Ġare Ġlinked Ġto Ġthe Ġrelated Ġentities Ġfrom Ġa Ġknowledge Ġbase Ġby Ġa Ġlookup .
<s> How Ġdo Ġthey Ġcombine Ġa Ġdeep Ġlearning Ġmodel Ġwith Ġa Ġknowledge Ġbase ? </s> Introduction </s> N amed Ġentity Ġrecognition Ġ( NER ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġis Ġthe Ġprocess Ġby Ġwhich Ġwe Ġidentify Ġtext Ġspans Ġwhich Ġmention Ġnamed Ġentities , Ġand Ġto Ġclassify Ġthem Ġinto Ġpred efined Ġcategories Ġsuch Ġas Ġperson , Ġlocation , Ġorganization Ġetc . ĠN ER Ġserves Ġas Ġthe Ġbasis Ġfor Ġa Ġvariety Ġof Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġapplications Ġsuch Ġas Ġrelation Ġextraction ĠB IB REF 4 Ġ, Ġmachine Ġtranslation ĠB IB REF 5 Ġ, Ġquestion Ġanswering ĠB IB REF 6 Ġand Ġknowledge Ġbase Ġconstruction ĠB IB REF 7 Ġ. ĠAlthough Ġearly ĠN ER Ġsystems Ġhave Ġbeen Ġsuccessful Ġin Ġproducing Ġadequate Ġrecognition Ġaccuracy , Ġthey Ġoften Ġrequire Ġsignificant Ġhuman Ġeffort Ġin Ġcarefully Ġdesigning Ġrules Ġor Ġfeatures . </s> In Ġrecent Ġyears , Ġdeep Ġlearning Ġmethods Ġbeen Ġemployed Ġin ĠN ER Ġsystems , Ġyielding Ġstate - of - the - art Ġperformance . ĠHowever , Ġthe Ġnumber Ġof Ġtypes Ġdetected Ġare Ġstill Ġnot Ġsufficient Ġfor Ġcertain Ġdomain - specific Ġapplications . ĠFor Ġrelation Ġextraction , Ġidentifying Ġfine - gr ained Ġtypes Ġhas Ġbeen Ġshown Ġto Ġsignificantly Ġincrease Ġthe Ġperformance Ġof Ġthe Ġextract or ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġsince Ġthis Ġhelps Ġin Ġfiltering Ġout Ġcandidate Ġrelation Ġtypes Ġwhich Ġdo Ġnot Ġfollow Ġthis Ġtype Ġconstraint . ĠFurthermore , Ġfor Ġquestion Ġanswering Ġfine - gr ained ĠNamed ĠEntity ĠRecogn ition Ġ( F g NER ) Ġcan Ġprovide Ġadditional Ġinformation Ġhelping Ġto Ġmatch Ġquestions Ġto Ġits Ġpotential Ġanswers Ġthus Ġimproving Ġperformance ĠB IB REF 10 Ġ. ĠFor Ġexample , ĠLi Ġand ĠRoth ĠB IB REF 11 Ġrank Ġquestions Ġbased Ġon Ġtheir Ġexpected Ġanswer Ġtypes Ġ( i . e . Ġwill Ġthe Ġanswer Ġbe Ġfood , Ġvehicle Ġor Ġdisease ). </s> Typically , ĠF g NER Ġsystems Ġuse Ġover Ġa Ġhundred Ġlabels , Ġarranged Ġin Ġa Ġhierarchical Ġstructure . ĠWe Ġfind Ġthat Ġavailable Ġtraining Ġdata Ġfor ĠF g NER Ġtypically Ġcontain Ġnoisy Ġlabels , Ġand Ġcreating Ġmanually Ġannot ated Ġtraining Ġdata Ġfor ĠF g NER Ġis Ġa Ġtime - consuming Ġprocess . ĠFurthermore , Ġhuman Ġannot ators Ġwill Ġhave Ġto Ġassign Ġa Ġsubset Ġof Ġcorrect Ġlabels Ġfrom Ġhundreds Ġof Ġpossible Ġlabels Ġmaking Ġthis Ġa Ġsomewhat Ġardu ous Ġtask . ĠCurrently , ĠF g NER Ġsystems Ġuse Ġdistant Ġsupervision ĠB IB REF 12 Ġto Ġautomatically Ġgenerate Ġtraining Ġdata . ĠDist ant Ġsupervision Ġis Ġa Ġtechnique Ġwhich Ġmaps Ġeach Ġentity Ġin Ġthe Ġcorpus Ġto Ġknowledge Ġbases Ġsuch Ġas ĠFree base ĠB IB REF 13 Ġ, ĠDB pedia ĠB IB REF 14 Ġ, ĠY AG O ĠB IB REF 15 Ġand Ġhelps Ġwith Ġthe Ġgeneration Ġof Ġlabeled Ġdata . ĠThis Ġmethod Ġwill Ġassign Ġthe Ġsame Ġset Ġof Ġlabels Ġto Ġall Ġmentions Ġof Ġa Ġparticular Ġentity Ġin Ġthe Ġcorpus . ĠFor Ġexample , ĠâĢ ľ Bar ack ĠObama âĢ Ŀ Ġis Ġa Ġperson , Ġpolitician , Ġlawyer , Ġand Ġauthor . ĠIf Ġa Ġknowledge Ġbase Ġhas Ġthese Ġfour Ġmatching Ġlabels , Ġthe Ġdistant Ġsupervision Ġtechnique Ġwill Ġassign Ġall Ġof Ġthem Ġto Ġevery Ġmention Ġof ĠâĢ ľ Bar ack ĠObama âĢ Ŀ . ĠTherefore , Ġthe Ġtraining Ġdata Ġwill Ġalso Ġfail Ġto Ġdistinguish Ġbetween Ġmentions Ġof ĠâĢ ľ Bar ack ĠObama âĢ Ŀ Ġin Ġall Ġsubsequent Ġutter ances . </s> L ing Ġet Ġal . Ġling 2012 fine Ġproposed Ġthe Ġfirst Ġsystem Ġfor ĠF g NER , Ġwhere Ġthey Ġused Ġ112 Ġoverlapping Ġlabels Ġwith Ġa Ġlinear Ġclass ifier Ġpercept ron Ġfor Ġmulti - label Ġclassification . ĠYose f Ġet Ġal . Ġspan iol 2012 hy ena Ġused Ġmultiple Ġbinary ĠS VM Ġclass ifiers Ġto Ġassign Ġentities Ġto Ġa Ġset Ġof Ġ505 Ġtypes . ĠGill ick Ġet Ġal . Ġg ill ick 2014 context Ġintroduced Ġcontext Ġdependent ĠF g NER Ġand Ġproposed Ġa Ġset Ġof Ġhe ur istics Ġfor Ġpr uning Ġlabels Ġthat Ġmight Ġnot Ġbe Ġrelevant Ġgiven Ġthe Ġlocal Ġcontext Ġof Ġthe Ġentity . ĠYog at ama Ġet Ġal . Ġyog at ama 2015 embed ding Ġproposed Ġan Ġembed ding Ġbased Ġmodel Ġwhere Ġuser - defined Ġfeatures Ġand Ġlabels Ġwere Ġembedded Ġinto Ġa Ġlow Ġdimensional Ġfeature Ġspace Ġto Ġfacilitate Ġinformation Ġsharing Ġamong Ġlabels . </s> Sh ima oka Ġet Ġal . Ġsh ima oka 2016 att ent ive Ġproposed Ġan Ġattentive Ġneural Ġnetwork Ġmodel Ġwhich Ġused Ġlong Ġshort - term Ġmemory Ġ( L ST Ms ) Ġto Ġencode Ġthe Ġcontext Ġof Ġthe Ġentity , Ġthen Ġused Ġan Ġattention Ġmechanism Ġto Ġallow Ġthe Ġmodel Ġto Ġfocus Ġon Ġrelevant Ġexpressions Ġin Ġthe Ġentity Ġmention 's Ġcontext . ĠTo Ġlearn Ġentity Ġrepresentations , Ġwe Ġpropose Ġa Ġscheme Ġwhich Ġis Ġpotentially Ġmore Ġgeneral izable . </s> Dat as ets </s> We Ġevaluate Ġour Ġmodel Ġon Ġtwo Ġpublicly Ġavailable Ġdatasets . ĠThe Ġstatistics Ġfor Ġboth Ġare Ġshown Ġin ĠTable ĠT AB REF 3 Ġ. ĠThe Ġdetails Ġof Ġthese Ġdatasets Ġare Ġas Ġfollows : </s> Ont o Notes : ĠOnt o Notes Ġ5 . 0 ĠB IB REF 16 Ġincludes Ġtexts Ġfrom Ġfive Ġdifferent Ġtext Ġgenres : Ġbroadcast Ġconversation Ġ( 200 k ), Ġbroadcast Ġnews Ġ( 200 k ), Ġmagazine Ġ( 120 k ), Ġnews wire Ġ( 625 k ), Ġand Ġweb Ġdata Ġ( 300 k ). ĠThis Ġdataset Ġis Ġannot ated Ġwith Ġ18 Ġcategories . </s> Wiki ( gold ): ĠThe Ġtraining Ġdata Ġconsists Ġof ĠWikipedia Ġsentences Ġand Ġwas Ġautomatically Ġgenerated Ġusing Ġa Ġdistant Ġsupervision Ġmethod , Ġmapping Ġhyper links Ġin ĠWikipedia Ġarticles Ġto ĠFree base , Ġwhich Ġwe Ġdo Ġnot Ġuse Ġin Ġthis Ġstudy . ĠThe Ġtest Ġdata , Ġmainly Ġconsisting Ġof Ġsentences Ġfrom Ġnews Ġreports , Ġwas Ġmanually Ġannot ated Ġas Ġdescribed Ġin ĠB IB REF 8 Ġ. ĠThe Ġclass Ġhierarchy Ġis Ġshown Ġin ĠFigure ĠFIG REF 2 Ġ. ĠThis Ġdataset Ġis Ġannot ated Ġwith Ġ7 Ġmain Ġcategories Ġ( bold Ġtext Ġin ĠFigure ĠFIG REF 2 Ġ), Ġwhich Ġmaps Ġdirectly Ġto ĠOnt o Notes . ĠThe Ġmis cellaneous Ġcategory Ġin ĠFigure ĠFIG REF 2 Ġdoes Ġnot Ġhave Ġdirect Ġm appings , Ġso Ġfuture Ġwork Ġmay Ġinclude Ġredef ining Ġthese Ġcategories Ġso Ġthe Ġm appings Ġare Ġmore Ġmeaningful . </s> E val uation ĠMet rics </s> NER Ġinvolves Ġidentifying Ġboth Ġentity Ġboundaries Ġand Ġentity Ġtypes . ĠWith ĠâĢ ľ ex act - match Ġevaluation âĢ Ŀ , Ġa Ġnamed Ġentity Ġis Ġconsidered Ġcorrectly Ġrecognized Ġonly Ġif Ġboth Ġthe Ġboundaries Ġand Ġtype Ġmatch Ġthe Ġground Ġtruth ĠB IB REF 8 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠPrecision , ĠRecall , Ġand ĠF - 1 Ġscores Ġare Ġcomputed Ġon Ġthe Ġnumber Ġof Ġtrue Ġpositives Ġ( TP ), Ġfalse Ġpositives Ġ( FP ), Ġand Ġfalse Ġnegatives Ġ( FN ). ĠTheir Ġformal Ġdefinitions Ġare Ġas Ġfollows : </s> True ĠPositive Ġ( TP ): Ġentities Ġthat Ġare Ġrecognized Ġby ĠN ER Ġand Ġmatch Ġthe Ġground Ġtruth . </s> False ĠPositive Ġ( FP ): Ġentities Ġthat Ġare Ġrecognized Ġby ĠN ER Ġbut Ġdo Ġnot Ġmatch Ġthe Ġground Ġtruth . </s> False ĠNegative Ġ( FN ): Ġentities Ġannot ated Ġin Ġthe Ġground Ġwhich Ġthat Ġare Ġnot Ġrecognized Ġby ĠN ER . </s> Pre cision Ġmeasures Ġthe Ġability Ġof Ġa ĠN ER Ġsystem Ġto Ġpresent Ġonly Ġcorrect Ġentities , Ġand ĠRecall Ġmeasures Ġthe Ġability Ġof Ġa ĠN ER Ġsystem Ġto Ġrecognize Ġall Ġentities Ġin Ġa Ġcorpus . ĠDIS PLAY FORM 0 </s> The ĠF - 1 Ġscore Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall , Ġand Ġthe Ġbalanced ĠF - 1 Ġscore Ġis Ġthe Ġvariant Ġwhich Ġis Ġmost Ġcommonly Ġused . ĠThis Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> Since Ġmost ĠN ER Ġsystems Ġinvolve Ġmultiple Ġentity Ġtypes , Ġit Ġis Ġoften Ġrequired Ġto Ġassess Ġthe Ġperformance Ġacross Ġall Ġentity Ġclasses . ĠTwo Ġmeasures Ġare Ġcommonly Ġused Ġfor Ġthis Ġpurpose : Ġthe Ġmacro - aver aged ĠF - 1 Ġscore Ġand Ġthe Ġmicro - aver aged ĠF - 1 Ġscore . ĠThe Ġmacro - aver aged ĠF - 1 Ġscore Ġcomp utes Ġthe ĠF - 1 Ġscore Ġindependently Ġfor Ġeach Ġentity Ġtype , Ġthen Ġtakes Ġthe Ġaverage Ġ( hen ce Ġtreating Ġall Ġentity Ġtypes Ġequally ). ĠThe Ġmicro - aver aged ĠF - 1 Ġscore Ġaggreg ates Ġthe Ġcontributions Ġof Ġentities Ġfrom Ġall Ġclasses Ġto Ġcompute Ġthe Ġaverage Ġ( t reating Ġall Ġentities Ġequally ). ĠWe Ġuse Ġthe Ġmicro - aver aged ĠF - 1 Ġin Ġour Ġstudy Ġsince Ġthis Ġaccounts Ġfor Ġlabel Ġim bal ances Ġin Ġthe Ġevaluation Ġdata Ġand Ġtherefore Ġa Ġmore Ġmeaningful Ġstatistic . </s> Method </s> Over Ġthe Ġfew Ġpast Ġyears , Ġthe Ġemergence Ġof Ġdeep Ġneural Ġnetworks Ġhas Ġfundamentally Ġchanged Ġthe Ġdesign Ġof Ġentity Ġdetection Ġsystems . ĠConsequently , Ġrecurrent Ġneural Ġnetworks Ġ( R NN ) Ġhave Ġfound Ġpopularity Ġin Ġthe Ġfield Ġsince Ġthey Ġare Ġable Ġto Ġlearn Ġlong Ġterm Ġdependencies Ġof Ġsequential Ġdata . ĠThe Ġrecent Ġsuccess Ġof Ġneural Ġnetwork Ġbased Ġarchitectures Ġprincipally Ġcomes Ġfrom Ġits Ġdeep Ġstructure . ĠTraining Ġa Ġdeep Ġneural Ġnetwork , Ġhowever , Ġis Ġa Ġdifficult Ġproblem Ġdue Ġto Ġvanishing Ġor Ġexploding Ġgrad ients . ĠIn Ġorder Ġto Ġsolve Ġthis , ĠL ST Ms Ġwere Ġproposed . ĠAn ĠL ST M Ġis Ġan Ġinternal Ġmemory Ġcell Ġcontrolled Ġby Ġforget Ġgate Ġand Ġinput Ġgate Ġnetworks . ĠA Ġforget Ġgate Ġin Ġan ĠL ST M Ġlayer Ġwhich Ġdetermines Ġhow Ġmuch Ġprior Ġmemory Ġshould Ġbe Ġpassed Ġinto Ġthe Ġnext Ġtime Ġincrement . ĠSimilarly , Ġan Ġinput Ġgate Ġscales Ġnew Ġinput Ġto Ġmemory Ġcells . ĠDepending Ġon Ġthe Ġstates Ġof Ġboth Ġgates , ĠL ST M Ġis Ġable Ġto Ġcapture Ġlong - term Ġor Ġshort - term Ġdependencies Ġfor Ġsequential Ġdata . ĠThis Ġis Ġan Ġideal Ġproperty Ġfor Ġmany ĠN LP Ġtasks . </s> NER Ġusing ĠEL Mo </s> Recently , ĠPeters Ġet Ġal . ĠB IB REF 19 Ġproposed ĠEL Mo Ġword Ġrepresentations . ĠEL Mo Ġextends Ġa Ġtraditional Ġword Ġembed ding Ġmodel Ġwith Ġfeatures Ġproduced Ġbid irection ally Ġwith Ġcharacter Ġconv olutions . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġthe Ġutilization Ġof ĠEL Mo Ġfor Ġdifferent ĠN LP Ġtasks Ġresult Ġin Ġimproved Ġperformance Ġcompared Ġto Ġother Ġtypes Ġof Ġword Ġembed ding Ġmodels Ġsuch Ġas ĠWord 2 V ec ĠB IB REF 20 Ġ, ĠGl o Ve ĠB IB REF 21 Ġ, Ġand Ġfast Text ĠB IB REF 22 Ġ. </s> The Ġarchitecture Ġof Ġour Ġproposed Ġmodel Ġis Ġshown Ġin ĠFigure ĠFIG REF 12 Ġ. ĠThe Ġinput Ġis Ġa Ġlist Ġof Ġtokens Ġand Ġthe Ġoutput Ġare Ġthe Ġpredicted Ġentity Ġtypes . ĠThe ĠEL Mo Ġembed d ings Ġare Ġthen Ġused Ġwith Ġa Ġresidual ĠL ST M Ġto Ġlearn Ġinformative Ġmorph ological Ġrepresentations Ġfrom Ġthe Ġcharacter Ġsequence Ġof Ġeach Ġtoken . ĠWe Ġthen Ġpass Ġthis Ġto Ġa Ġsoft max Ġlayer Ġas Ġa Ġtag Ġdec oder Ġto Ġpredict Ġthe Ġentity Ġtypes . </s> Hyper param eter Ġsettings : ĠThe Ġhidden - layer Ġsize Ġof Ġeach ĠL ST M Ġwithin Ġthe Ġmodel Ġis Ġset Ġ512 . ĠWe Ġuse Ġa Ġdrop out Ġwith Ġthe Ġprobability Ġof Ġ0 . 2 Ġon Ġthe Ġoutput Ġof Ġthe ĠL ST M Ġenc od ers . ĠThe Ġembed ding Ġdimension Ġfrom ĠEL Mo Ġis Ġ1024 . ĠThe Ġoptimization Ġmethod Ġwe Ġuse Ġis ĠAdam ĠB IB REF 23 Ġ. ĠWe Ġtrain Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġfor Ġ30 Ġepoch s . ĠThe Ġmodel Ġwas Ġimplemented Ġusing Ġthe ĠT ensor Flow Ġframework . </s> Entity ĠLink ing Ġusing ĠWik id ata </s> Entity Ġlinking Ġ( EL ) ĠB IB REF 24 Ġ, Ġalso Ġknown Ġas Ġnamed Ġentity Ġdis amb ig uation Ġor Ġnormal ization , Ġis Ġthe Ġtask Ġto Ġdetermine Ġthe Ġidentity Ġof Ġentities Ġmentioned Ġin Ġa Ġpiece Ġof Ġtext Ġwith Ġreference Ġto Ġa Ġknowledge Ġbase . ĠThere Ġare Ġa Ġnumber Ġof Ġknowledge Ġbases Ġthat Ġprovide Ġa Ġbackground Ġrepository Ġfor Ġentity Ġclassification Ġof Ġthis Ġtype . ĠFor Ġthis Ġstudy , Ġwe Ġuse ĠWik id ata , Ġwhich Ġcan Ġbe Ġseen Ġdiagram matically Ġin ĠFigure ĠFIG REF 12 Ġ. ĠSystems Ġsuch Ġas ĠDeep Type ĠB IB REF 25 Ġintegrate Ġsymbolic Ġinformation Ġinto Ġthe Ġreasoning Ġprocess Ġof Ġa Ġneural Ġnetwork Ġwith Ġa Ġtype Ġsystem Ġand Ġshow Ġstate - of - the - art Ġperformances Ġfor ĠEL . ĠThey Ġdo Ġnot , Ġhowever , Ġquote Ġresults Ġon ĠWiki ( gold ) Ġso Ġa Ġdirect Ġcomparison Ġis Ġdifficult . </s> While Ġthese Ġknowledge Ġbases Ġprovide Ġsem antically Ġrich Ġand Ġfine - gran ular Ġclasses Ġand Ġrelationship Ġtypes , Ġthe Ġtask Ġof Ġentity Ġclassification Ġoften Ġrequires Ġassoci ating Ġcoarse - gr ained Ġclasses Ġwith Ġdiscovered Ġsurface Ġforms Ġof Ġentities . ĠMost Ġexisting Ġstudies Ġconsider ĠN ER Ġand Ġentity Ġlinking Ġas Ġtwo Ġseparate Ġtasks , Ġwhereas Ġwe Ġtry Ġto Ġcombine Ġthe Ġtwo . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġone Ġcan Ġsignificantly Ġincrease Ġthe Ġsemantic Ġinformation Ġcarried Ġby Ġa ĠN ER Ġsystem Ġwhen Ġwe Ġsuccessfully Ġlinking Ġentities Ġfrom Ġa Ġdeep Ġlearning Ġmethod Ġto Ġthe Ġrelated Ġentities Ġfrom Ġa Ġknowledge Ġbase ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> Red irection : ĠFor Ġthe ĠWik id ata Ġlinking Ġelement , Ġwe Ġrecognize Ġthat Ġthe Ġlookup Ġwill Ġbe Ġconstrained Ġby Ġthe Ġmost Ġcommon Ġlookup Ġname Ġfor Ġeach Ġentity . ĠConsider Ġthe Ġutter ance Ġ( re fer ring Ġto Ġthe ĠNBA Ġbasketball Ġplayer ) Ġfrom ĠFigure ĠFIG REF 12 ĠâĢ ľ Michael ĠJeffrey ĠJordan Ġin ĠSan ĠJose âĢ Ŀ Ġas Ġan Ġexample . ĠThe Ġlookup Ġfor Ġthis Ġentity Ġin ĠWik id ata Ġis ĠâĢ ľ Michael ĠJordan âĢ Ŀ Ġand Ġconsequently Ġwill Ġnot Ġbe Ġpicked Ġup Ġif Ġwe Ġwere Ġto Ġuse Ġan Ġexact Ġstring Ġmatch . ĠA Ġsimple Ġmethod Ġto Ġcircumvent Ġsuch Ġa Ġproblem Ġis Ġthe Ġusage Ġof Ġa Ġred irection Ġlist . ĠSuch Ġa Ġlist Ġis Ġprovided Ġon Ġan Ġentity Ġby Ġentity Ġbasis Ġin Ġthe ĠâĢ ľ Also Ġknown Ġas âĢ Ŀ Ġsection Ġin ĠWik id ata . ĠUsing Ġthis Ġred irection Ġlist , Ġwhen Ġwe Ġdo Ġnot Ġfind Ġan Ġexact Ġstring Ġmatch Ġimproves Ġthe Ġrecall Ġof Ġour Ġmodel Ġby Ġ5 - 10 %. ĠMoreover , Ġwith Ġthe Ġexample Ġof ĠMichael ĠJordan Ġ( person ), Ġusing Ġour Ġcurrent Ġframework , Ġwe Ġwill Ġalways Ġrefer Ġto Ġthe Ġretired Ġbasketball Ġplayer Ġ( Q 414 21 ). ĠWe Ġwill Ġnever , Ġfor Ġinstance , Ġpick Ġup ĠMichael ĠJordan Ġ( Q 27 06 9 141 ) Ġthe ĠAmerican Ġfootball Ġcornerback . ĠOr Ġin Ġfact Ġany Ġother ĠMichael ĠJordan , Ġfamous Ġor Ġotherwise . ĠOne Ġpossible Ġmethod Ġto Ġovercome Ġthis Ġis Ġto Ġadd Ġa Ġdis amb ig uation Ġlayer , Ġwhich Ġseeks Ġto Ġuse Ġcontext Ġfrom Ġearlier Ġparts Ġof Ġthe Ġtext . ĠThis Ġis , Ġhowever , Ġwork Ġfor Ġfuture Ġimprovement Ġand Ġwe Ġonly Ġconsider Ġthe Ġmost Ġcommon Ġversion Ġof Ġthat Ġentity . </s> Cl ust ering : ĠThe ĠWik id ata Ġtax onomy Ġprovides Ġthousands Ġof Ġpossible Ġinstance Ġof , Ġand Ġsubclass Ġof Ġtypes Ġfor Ġour Ġentities . ĠConsequently , Ġin Ġorder Ġto Ġperform Ġa Ġmeaningful Ġvalidation Ġof Ġour Ġmodel , Ġwe Ġmust Ġfind Ġa Ġway Ġto Ġcluster Ġthese Ġonto Ġthe Ġ112 Ġtypes Ġprovided Ġby ĠWiki ( gold ). ĠOur Ġclust ering Ġis Ġperformed Ġas Ġfollows : </s> If Ġthe Ġentity Ġtype Ġis Ġeither Ġperson , Ġlocation , Ġorganization Ġwe Ġuse Ġthe ĠN ECK Ar ĠB IB REF 28 Ġtool Ġto Ġnarrow Ġdown Ġour Ġlist Ġof Ġsearch able Ġentities . </s> We Ġthen Ġlook Ġat Ġeither Ġthe Ġoccupation Ġfor Ġperson , Ġor Ġinstance Ġof Ġfor Ġlocation / organ ization Ġcategories Ġto Ġmap Ġto Ġthe Ġavailable Ġsub types . </s> If Ġthe Ġentity Ġtype Ġis Ġnot Ġperson , Ġlocation , Ġor Ġorganization Ġwe Ġsearch Ġall Ġof ĠWik id ata . </s> The Ġclust ering Ġwe Ġperform Ġin Ġpart Ġ1 Ġor Ġ2 Ġis Ġfrom Ġa Ġcos ine Ġsimilarity Ġof Ġthe Ġentity Ġdescription Ġto Ġthe Ġlist Ġof Ġpossible Ġsub types Ġfor Ġthat Ġentity . ĠFor Ġthis Ġwe Ġuse ĠWord 2 V ec Ġword Ġembed d ings Ġtrained Ġon ĠWikipedia . ĠWe Ġset Ġthe Ġminimum Ġthreshold Ġof Ġthe Ġaverage Ġcos ine Ġsimilarity Ġto Ġbe Ġ0 . 1 . </s> As Ġan Ġexample , Ġconsider Ġthe Ġtest Ġsentence : ĠâĢ ľ The Ġdevice Ġwill Ġbe Ġavailable Ġon Ġsale Ġon Ġ20 th ĠApril Ġ2011 Ġon Ġam azon Ġu k ĠApple 's ĠiPad âĢ Ŀ Ġfrom ĠFigure ĠFIG REF 18 Ġ. ĠFirst , Ġwe Ġtag ĠiPad Ġas Ġproduct Ġusing Ġthe Ġcontext Ġenc oder Ġdescribed Ġin ĠSection Ġ2 . 1 . ĠWe Ġthen Ġsearch ĠWik id ata Ġand Ġreturn Ġthe Ġmost Ġcommon Ġvariant Ġfor Ġthat Ġentity Ġin Ġthis Ġcase ĠQ 27 96 Ġ( the Ġmost Ġreferenced Ġvariant Ġis Ġthe Ġone Ġwith Ġthe Ġlowest ĠQ - id ). ĠWe Ġthen Ġcalculate Ġa Ġcos ine Ġsimilarity Ġof Ġthe Ġdescription , Ġin Ġthis Ġcase ĠâĢ ľ line Ġof Ġtablet Ġcomputers âĢ Ŀ , Ġwith Ġthe Ġpossible Ġsub types Ġof Ġproduct . ĠThe Ġpossible Ġsub types , Ġin Ġthis Ġcase , Ġare Ġengine , Ġairplane , Ġcar , Ġship , Ġspacecraft , Ġtrain , Ġcamera , Ġmobile Ġphone , Ġcomputer , Ġsoftware , Ġgame , Ġinstrument , Ġship , Ġweapon . ĠWe Ġreturn Ġthe Ġhighest Ġresult Ġabove Ġ0 . 1 , Ġwhich Ġin Ġthis Ġcase Ġis Ġcomputer Ġ( 0 . 54 ). </s> Results </s> The Ġresults Ġfor Ġeach Ġclass Ġtype Ġare Ġshown Ġin ĠTable ĠT AB REF 19 Ġ, Ġwith Ġsome Ġspecific Ġexamples Ġshown Ġin ĠFigure ĠFIG REF 18 Ġ. ĠFor Ġthe ĠWiki ( gold ) Ġwe Ġquote Ġthe Ġmicro - aver aged ĠF - 1 Ġscores Ġfor Ġthe Ġentire Ġtop Ġlevel Ġentity Ġcategory . ĠThe Ġtotal ĠF - 1 Ġscore Ġon Ġthe ĠOnt o Notes Ġdataset Ġis Ġ88 %, Ġand Ġthe Ġtotal ĠF - 1 Ġcross - valid ation Ġscore Ġon Ġthe Ġ112 Ġclass ĠWiki ( gold ) Ġdataset Ġis Ġ53 %. ĠIt Ġis Ġworth Ġnoting Ġthat Ġone Ġcould Ġimprove ĠWiki ( gold ) Ġresults Ġby Ġtraining Ġdirectly Ġusing Ġthis Ġdataset . ĠHowever , Ġthe Ġaim Ġis Ġnot Ġto Ġtune Ġour Ġmodel Ġspecifically Ġon Ġthis Ġclass Ġhierarchy . ĠWe Ġinstead Ġaim Ġto Ġpresent Ġa Ġframework Ġwhich Ġcan Ġbe Ġmodified Ġeasily Ġto Ġany Ġdomain Ġhierarchy Ġand Ġhas Ġacceptable Ġout - of - the - box Ġperformances Ġto Ġany Ġfine - gr ained Ġdataset . ĠThe Ġresults Ġin ĠTable ĠT AB REF 19 Ġ( Ont o Notes ) Ġonly Ġshow Ġthe Ġmain Ġ7 Ġcategories Ġin ĠOnt o Notes Ġwhich Ġmap Ġto ĠWiki ( gold ) Ġfor Ġclarity . ĠThe Ġother Ġcategories Ġ( date , Ġtime , Ġnor p , Ġlanguage , Ġord inal , Ġcardinal , Ġquantity , Ġpercent , Ġmoney , Ġlaw ) Ġhave ĠF - 1 Ġscores Ġbetween Ġ80 - 90 %, Ġwith Ġthe Ġexception Ġof Ġtime Ġ( 65 %) </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġdeep Ġneural Ġnetwork Ġmodel Ġfor Ġthe Ġtask Ġof Ġfine - gr ained Ġnamed Ġentity Ġclassification Ġusing ĠEL Mo Ġembed d ings Ġand ĠWik id ata . ĠThe Ġproposed Ġmodel Ġlearns Ġrepresentations Ġfor Ġentity Ġmentions Ġbased Ġon Ġits Ġcontext Ġand Ġincorporates Ġthe Ġrich Ġstructure Ġof ĠWik id ata Ġto Ġaugment Ġthese Ġlabels Ġinto Ġfiner - gr ained Ġsub types . ĠWe Ġcan Ġsee Ġcomparisons Ġof Ġour Ġmodel Ġmade Ġon ĠWiki ( gold ) Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġnote Ġthat Ġthe Ġmodel Ġperforms Ġsimilarly Ġto Ġexisting Ġsystems Ġwithout Ġbeing Ġtrained Ġor Ġtuned Ġon Ġthat Ġparticular Ġdataset . ĠFuture Ġwork Ġmay Ġinclude Ġrefining Ġthe Ġclust ering Ġmethod Ġdescribed Ġin ĠSection Ġ2 . 2 Ġto Ġextend Ġto Ġtypes Ġother Ġthan Ġperson , Ġlocation , Ġorganization , Ġand Ġalso Ġto Ġinclude Ġdis amb ig uation Ġof Ġentity Ġtypes .