The GENIA Corpus , CoNLL2003
EXTRACTIVE
GENIA Corpus BIBREF3, CoNLL2003 BIBREF14, KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC
EXTRACTIVE
CoNLL2003-testA, GENIA
EXTRACTIVE
Table TABREF33 gives an overview of the standard data sets we use for training. The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc. For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 .
Table TABREF33 gives an overview of the standard data sets we use for training. The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc. For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 .
For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 .
1608.06757
Introduction </s> Information Ġextraction Ġtasks Ġhave Ġbecome Ġvery Ġimportant Ġnot Ġonly Ġin Ġthe ĠWeb , Ġbut Ġalso Ġfor Ġin - house Ġenterprise Ġsettings . ĠOne Ġof Ġthe Ġcrucial Ġsteps Ġtowards Ġunderstanding Ġnatural Ġlanguage Ġis Ġnamed Ġentity Ġrecognition Ġ( NER ), Ġwhich Ġaims Ġto Ġextract Ġmentions Ġof Ġentity Ġnames Ġin Ġtext . ĠN ER Ġis Ġnecessary Ġfor Ġmany Ġhigher - level Ġtasks Ġsuch Ġas Ġentity Ġlinking , Ġrelation Ġextraction , Ġbuilding Ġknowledge Ġgraphs , Ġquestion Ġanswering Ġand Ġintent Ġbased Ġsearch . ĠIn Ġthese Ġscenarios , ĠN ER Ġrecall Ġis Ġcritical , Ġas Ġcandidates Ġthat Ġare Ġnever Ġgenerated Ġcan Ġnot Ġbe Ġrecovered Ġlater ĠB IB REF 0 Ġ. </s> Rob ust ĠContext ual ĠWord ĠLab eling </s> We Ġabstract Ġthe Ġtask Ġof ĠN ER Ġas Ġsequential Ġword Ġlabeling Ġproblem . ĠFigure ĠFIG REF 15 Ġillustrates Ġan Ġexample Ġfor Ġsequential Ġtransformation Ġof Ġa Ġsentence Ġinto Ġword Ġlabels . ĠWe Ġexpress Ġeach Ġsentence Ġin Ġa Ġdocument Ġas Ġa Ġsequence Ġof Ġwords : ĠIN LINE FORM 0 Ġ, Ġe . g . ĠIN LINE FORM 1 ĠAs pir in . ĠWe Ġdefine Ġa Ġmention Ġas Ġthe Ġlongest Ġpossible Ġspan Ġof Ġadjacent Ġtokens Ġthat Ġrefer Ġto Ġa Ġan Ġentity Ġor Ġrelevant Ġconcept Ġof Ġa Ġreal - world Ġobject , Ġsuch Ġas ĠAs pir in Ġ( AS A ). ĠWe Ġfurther Ġassume Ġthat Ġmentions Ġare Ġnon - rec ursive Ġand Ġnon - over l apping . ĠTo Ġencode Ġboundaries Ġof Ġthe Ġmention Ġspan , Ġwe Ġadapt Ġthe Ġidea Ġof Ġram shaw 1995 text , Ġwhich Ġhas Ġbeen Ġadapted Ġas ĠB IO 2 Ġstandard Ġin Ġthe ĠCo N LL 2003 Ġshared Ġtask ĠB IB REF 15 Ġ. ĠWe Ġassign Ġlabels ĠIN LINE FORM 2 Ġto Ġeach Ġtoken Ġto Ġmark Ġbegin , Ġinside Ġand Ġoutside Ġof Ġa Ġmention Ġfrom Ġleft Ġto Ġright . ĠWe Ġuse Ġthe Ġinput Ġsequence ĠIN LINE FORM 3 Ġtogether Ġwith Ġa Ġtarget Ġsequence ĠIN LINE FORM 4 Ġof Ġthe Ġsame Ġlength Ġthat Ġcontains Ġa ĠB IO 2 Ġlabel Ġfor Ġeach Ġword : ĠIN LINE FORM 5 Ġ, Ġe . g . ĠIN LINE FORM 6 ĠB . ĠTo Ġpredict Ġthe Ġmost Ġlikely Ġlabel ĠIN LINE FORM 7 Ġof Ġa Ġtoken Ġregarding Ġits Ġcontext , Ġwe Ġutilize Ġrecurrent Ġneural Ġnetworks . </s> Rob ust ĠWord ĠEnc oding ĠMethods </s> We Ġhave Ġshown Ġthat Ġmost Ġcommon Ġerrors Ġfor Ġrecall Ġloss Ġare Ġmiss pell ings , ĠPOS Ġerrors , Ġcapital ization , Ġunseen Ġwords Ġand Ġirregular Ġcontext . ĠTherefore Ġwe Ġgeneral ize Ġour Ġmodel Ġthroughout Ġthree Ġlayers : Ġrobust Ġword Ġencoding , Ġin - sent ence Ġword Ġcontext Ġand Ġcontextual Ġsequence Ġlabeling . </s> D ictionary - based Ġword Ġvector ization Ġmethods Ġsuffer Ġfrom Ġsparse Ġtraining Ġsets , Ġespecially Ġin Ġthe Ġcase Ġof Ġnon - ver bat im Ġmentions , Ġrare Ġwords , Ġtyping Ġand Ġcapital ization Ġerrors . ĠFor Ġexample , Ġthe Ġword 2 vec Ġmodel Ġof Ġm ik ol ov 2013 efficient Ġgeneral izes Ġinsufficient ly Ġfor Ġrare Ġwords Ġin Ġidiosyncr atic Ġdomains Ġor Ġfor Ġmiss pelled Ġwords , Ġsince Ġfor Ġthese Ġwords Ġno Ġvector Ġrepresentation Ġis Ġlearned Ġat Ġtraining Ġtime . ĠIn Ġthe ĠGEN IA Ġdata Ġset , Ġwe Ġnotice Ġ27 % Ġunseen Ġwords Ġ( d ictionary Ġmisses ) Ġin Ġthe Ġpret rained Ġword 2 vec Ġmodel . ĠAs Ġtraining Ġdata Ġgeneration Ġis Ġexpensive , Ġwe Ġinvestigate Ġa Ġgeneric Ġapproach Ġfor Ġthe Ġgeneration Ġof Ġword Ġvectors . ĠWe Ġuse Ġletter - tr ig ram Ġword Ġhashing Ġas Ġintroduced Ġby Ġh u ang 2013 learning . ĠThis Ġtechnique Ġgoes Ġbeyond Ġwords Ġand Ġgenerates Ġword Ġvectors Ġas Ġa Ġcomposite Ġof Ġdiscrim in ative Ġthree - letter ĠâĢ ľ sy ll ables âĢ Ŀ , Ġthat Ġmight Ġalso Ġinclude Ġmiss pell ings . ĠTherefore , Ġit Ġis Ġrobust Ġagainst Ġdictionary Ġmisses Ġand Ġhas Ġthe Ġadvantage Ġ( despite Ġits Ġname ) Ġto Ġgroup Ġsynt act ically Ġsimilar Ġwords Ġin Ġsimilar Ġvector Ġspaces . ĠWe Ġcompare Ġthis Ġapproach Ġto Ġword Ġembed ding Ġmodels Ġsuch Ġas Ġword 2 vec . </s> The Ġmost Ġimportant Ġfeatures Ġfor ĠN ER Ġare Ġword Ġshape Ġproperties , Ġsuch Ġas Ġlength , Ġinitial Ġcapital ization , Ġall - word Ġu pperc ase , Ġin - word Ġcapital ization Ġand Ġuse Ġof Ġnumbers Ġor Ġpunct uation ĠB IB REF 16 Ġ. ĠMixed - case Ġword Ġenc od ings Ġimplicitly Ġinclude Ġcapital ization Ġfeatures . ĠHowever , Ġthis Ġapproach Ġimped es Ġgeneral ization , Ġas Ġwords Ġappear Ġin Ġvarious Ġsurface Ġforms , Ġe . g . Ġcapital ized Ġat Ġthe Ġbeginning Ġof Ġsentences , Ġu pperc ase Ġin Ġheadlines , Ġlower case Ġin Ġsocial Ġmedia Ġtext . ĠThe Ġstrong Ġco herence Ġbetween Ġu pperc ase Ġand Ġlower case Ġcharacters ĠâĢĵ Ġthey Ġmight Ġhave Ġidentical Ġsemantics ĠâĢĵ Ġis Ġnot Ġencoded Ġin Ġthe Ġembed ding . ĠTherefore , Ġwe Ġencode Ġthe Ġwords Ġusing Ġlower case Ġletter - tr ig rams . ĠTo Ġkeep Ġthe Ġsurface Ġinformation , Ġwe Ġadd Ġflag Ġbits Ġto Ġthe Ġvector Ġthat Ġindicate Ġinitial Ġcapital ization , Ġu pperc ase , Ġlower Ġcase Ġor Ġmixed Ġcase . </s> Deep ĠContext ual ĠSequence ĠLearning </s> With Ġsparse Ġtraining Ġdata Ġin Ġthe Ġidiosyncr atic Ġdomain , Ġwe Ġexpect Ġinput Ġdata Ġwith Ġhigh Ġvariance . ĠTherefore , Ġwe Ġrequire Ġa Ġstrong Ġgeneral ization Ġfor Ġthe Ġsynt actic Ġand Ġsemantic Ġrepresentation Ġof Ġlanguage . ĠTo Ġreach Ġinto Ġthe Ġhigh Ġ80 âĢĵ 90 % ĠN ER ĠF 1 Ġperformance , Ġlong - range Ġcontext - sensitive Ġinformation Ġis Ġindispensable . ĠWe Ġapply Ġthe Ġcomputational Ġmodel Ġof Ġrecurrent Ġneural Ġnetworks , Ġin Ġparticular Ġlong Ġshort - term Ġmemory Ġnetworks Ġ( L ST Ms ) ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġto Ġthe Ġproblem Ġof Ġsequence Ġlabeling . ĠLike Ġneural Ġfeed - forward Ġnetworks , ĠL ST Ms Ġare Ġable Ġto Ġlearn Ġcomplex Ġparameters Ġusing Ġgradient Ġdescent , Ġbut Ġinclude Ġadditional Ġrecurrent Ġconnections Ġbetween Ġcells Ġto Ġinfluence Ġweight Ġupdates Ġover Ġadjacent Ġtime Ġsteps . ĠWith Ġtheir Ġability Ġto Ġmemor ize Ġand Ġforget Ġover Ġtime , ĠL ST Ms Ġhave Ġproven Ġto Ġgeneral ize Ġcontext - sensitive Ġsequential Ġdata Ġwell ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. </s> Figure ĠFIG REF 15 Ġshows Ġan Ġunfolded Ġrepresentation Ġof Ġthe Ġsteps Ġthrough Ġa Ġsentence . ĠWe Ġfeed Ġthe ĠL ST M Ġwith Ġletter - tr ig ram Ġvectors ĠIN LINE FORM 0 Ġas Ġinput Ġdata , Ġone Ġword Ġat Ġa Ġtime . ĠThe Ġhidden Ġlayer Ġof Ġthe ĠL ST M Ġrepresents Ġcontext Ġfrom Ġlong Ġrange Ġdependencies Ġover Ġthe Ġentire Ġsentence Ġfrom Ġleft Ġto Ġright . ĠHowever , Ġto Ġachieve Ġdeeper Ġcontextual Ġunderstanding Ġover Ġthe Ġboundaries Ġof Ġmulti - word Ġannotations Ġand Ġat Ġthe Ġbeginning Ġof Ġsentences , Ġwe Ġrequire Ġa Ġbackwards Ġpass Ġthrough Ġthe Ġsentence . ĠWe Ġtherefore Ġimplement Ġa Ġbid irection al ĠL ST M Ġand Ġfeed Ġthe Ġoutput Ġof Ġboth Ġdirections Ġinto Ġa Ġsecond ĠL ST M Ġlayer Ġfor Ġcombined Ġlabel Ġprediction . </s> For Ġthe Ġuse Ġin Ġthe Ġneural Ġnetwork , Ġword Ġenc od ings ĠIN LINE FORM 0 Ġand Ġlabels ĠIN LINE FORM 1 Ġare Ġreal - valued Ġvectors . ĠTo Ġpredict Ġthe Ġmost Ġlikely Ġlabel ĠIN LINE FORM 2 Ġof Ġa Ġtoken , Ġwe Ġutilize Ġa ĠL ST M Ġwith Ġinput Ġnodes ĠIN LINE FORM 3 Ġ, Ġinput Ġgates ĠIN LINE FORM 4 Ġ, Ġforget Ġgate ĠIN LINE FORM 5 Ġ, Ġoutput Ġgate ĠIN LINE FORM 6 Ġand Ġinternal Ġstate ĠIN LINE FORM 7 Ġ. ĠFor Ġthe Ġbid irection al Ġcase , Ġall Ġgates Ġare Ġdupl icated Ġand Ġcombined Ġinto Ġforward Ġstate ĠIN LINE FORM 8 Ġand Ġbackward Ġstate ĠIN LINE FORM 9 Ġ. ĠThe Ġnetwork Ġis Ġtrained Ġusing Ġback prop ag ation Ġthrough Ġtime Ġ( B PT T ) Ġby Ġadapting Ġweights ĠIN LINE FORM 10 Ġand Ġbias Ġparameters ĠIN LINE FORM 11 Ġto Ġfit Ġthe Ġtraining Ġexamples . ĠDIS PLAY FORM 0 </s> We Ġiter ate Ġover Ġlabeled Ġsentences Ġin Ġmini - bat ches Ġand Ġupdate Ġthe Ġweights Ġaccordingly . ĠThe Ġnetwork Ġis Ġthen Ġused Ġto Ġpredict Ġlabel Ġprobabilities ĠIN LINE FORM 0 Ġfor Ġunseen Ġword Ġsequences ĠIN LINE FORM 1 Ġ. </s> Im plementation Ġof ĠN ER ĠComponents Ġ </s> To Ġshow Ġthe Ġimpact Ġof Ġour Ġbid irection al ĠL ST M Ġmodel , Ġwe Ġmeasure Ġannotation Ġperformance Ġon Ġthree Ġdifferent Ġneural Ġnetwork Ġconfigurations . ĠWe Ġimplement Ġall Ġcomponents Ġusing Ġthe ĠDeep learning 4 j Ġframework . ĠFor Ġpre processing Ġ( sent ence Ġand Ġword Ġtoken ization ), Ġwe Ġuse ĠStanford ĠCore N LP ĠB IB REF 11 Ġ. ĠWe Ġtest Ġthe Ġsequence Ġlabel er Ġusing Ġthree Ġinput Ġenc od ings : </s> [ no item se p ] </s> D ICT : ĠWe Ġbuild Ġa Ġdictionary Ġover Ġall Ġwords Ġin Ġthe Ġcorpus Ġand Ġgenerate Ġthe Ġinput Ġvector Ġusing Ġ1 - hot Ġencoding Ġfor Ġeach Ġword </s> EM B : ĠWe Ġuse Ġthe ĠGoogle News Ġword 2 vec Ġembed d ings , Ġwhich Ġenc odes Ġeach Ġword Ġas Ġvector Ġof Ġsize Ġ300 </s> TR I : Ġwe Ġimplement Ġletter - tr ig ram Ġword Ġhashing Ġas Ġdescribed Ġin ĠSection ĠSEC REF 14 Ġ. </s> During Ġtraining Ġand Ġtest , Ġwe Ġgroup Ġall Ġtokens Ġof Ġa Ġsentence Ġas Ġmini - batch . ĠWe Ġevaluate Ġthree Ġdifferent Ġneural Ġnetwork Ġtypes Ġto Ġshow Ġthe Ġimpact Ġof Ġthe Ġbid irection al Ġsequence Ġlear ner . </s> [ no item se p ] </s> FF : ĠAs Ġbaseline , Ġwe Ġtrain Ġa Ġnon - sequ ential Ġfeed - forward Ġmodel Ġbased Ġon Ġa Ġfully Ġconnected Ġmult il ayer Ġpercept ron Ġnetwork Ġwith Ġ3 Ġhidden Ġlayers Ġof Ġsize Ġ150 Ġwith Ġrel u Ġactivation , Ġfeeding Ġinto Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġback prop ag ation Ġwith Ġsto ch astic Ġgradient Ġdescent Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 005 . </s> L ST M : ĠWe Ġuse Ġa Ġconfiguration Ġof Ġa Ġsingle Ġfeed - forward Ġlayer Ġof Ġsize Ġ150 Ġwith Ġtwo Ġadditional Ġlayers Ġof Ġsingle - direction ĠL ST M Ġwith Ġ20 Ġcells Ġand Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġback prop ag ation - through - time Ġ( B PT T ) Ġwith Ġsto ch astic Ġgradient Ġdescent Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 005 . </s> BL ST M : ĠOur Ġfinal Ġconfiguration Ġconsists Ġof Ġa Ġsingle Ġfeed - forward Ġlayer Ġof Ġsize Ġ150 Ġwith Ġone Ġbid irection al ĠL ST M Ġlayer Ġwith Ġ20 Ġcells Ġand Ġan Ġadditional Ġsingle - direction ĠL ST M Ġwith Ġ20 Ġcells Ġinto Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠThe ĠBL ST M Ġmodel Ġis Ġtrained Ġthe Ġsame Ġway Ġas Ġthe Ġsingle - direction ĠL ST M . </s> E val uation </s> We Ġevaluate Ġnine Ġconfigurations Ġof Ġour Ġmodel Ġon Ġfive Ġgold Ġstandard Ġevaluation Ġdata Ġsets . ĠWe Ġshow Ġthat Ġthe Ġcombination Ġof Ġletter - tr ig ram Ġword Ġhashing Ġwith Ġbid irection al ĠL ST M Ġyields Ġthe Ġbest Ġresults Ġand Ġoutper forms Ġsequence Ġlearners Ġbased Ġon Ġdiction aries Ġor Ġword 2 vec . ĠTo Ġhighlight Ġthe Ġgeneral ization Ġof Ġour Ġmodel Ġto Ġidiosyncr atic Ġdomains , Ġwe Ġrun Ġtests Ġon Ġcommon - ty ped Ġdata Ġsets Ġas Ġwell Ġas Ġon Ġspecialized Ġmedical Ġdocuments . ĠWe Ġcompare Ġour Ġsystem Ġon Ġthese Ġdata Ġsets Ġwith Ġspecialized Ġstate - of - the - art Ġsystems . </s> E val uation ĠSet ĠUp </s> We Ġtrain Ġtwo Ġmodels Ġwith Ġidentical Ġparameter ization , Ġeach Ġwith Ġ2000 Ġrandomly Ġchosen Ġlabeled Ġsentences Ġfrom Ġa Ġstandard Ġdata Ġset . ĠTo Ġshow Ġthe Ġeffectiveness Ġof Ġthe Ġcomponents , Ġwe Ġevaluate Ġdifferent Ġconfigurations Ġof Ġthis Ġsetting Ġwith Ġ2000 Ġrandom Ġsentences Ġfrom Ġthe Ġremaining Ġset . ĠThe Ġmodel Ġwas Ġtrained Ġusing ĠDeep learning 4 j Ġwith Ġn d 4 j - x 86 Ġbackend . ĠTraining Ġthe ĠTRI + BL ST M Ġconfiguration Ġon Ġa Ġcommodity ĠIntel Ġi 7 Ġnotebook Ġwith Ġ4 Ġcores Ġat Ġ2 . 8 GHz Ġtakes Ġapproximately Ġ50 Ġminutes . </s> Table ĠT AB REF 33 Ġgives Ġan Ġoverview Ġof Ġthe Ġstandard Ġdata Ġsets Ġwe Ġuse Ġfor Ġtraining . ĠThe ĠGEN IA ĠCorpus ĠB IB REF 3 Ġcontains Ġbiomedical Ġabstract s Ġfrom Ġthe ĠPubMed Ġdatabase . ĠWe Ġuse ĠGEN IA Ġtechnical Ġterm Ġannotations Ġ3 . 02 , Ġwhich Ġcover Ġlinguistic Ġexpressions Ġto Ġentities Ġof Ġinterest Ġin Ġmolecular Ġbiology , Ġe . g . Ġproteins , Ġgenes Ġand Ġcells . ĠCo N LL 2003 ĠB IB REF 14 Ġis Ġa Ġstandard ĠN ER Ġdataset Ġbased Ġon Ġthe ĠReuters ĠRC V - 1 Ġnews Ġcorpus . ĠIt Ġcovers Ġnamed Ġentities Ġof Ġtype Ġperson , Ġlocation , Ġorganization Ġand Ġmisc . </s> For Ġtesting Ġthe Ġoverall Ġannotation Ġperformance , Ġwe Ġutilize ĠCo N LL 2003 - test A Ġand Ġa Ġ50 Ġdocument Ġsplit Ġfrom ĠGEN IA . ĠAdditionally , Ġwe Ġtest Ġon Ġthe Ġcomplete ĠK ORE 50 ĠB IB REF 21 Ġ, ĠACE 2004 ĠB IB REF 22 Ġand ĠMSNBC Ġdata Ġsets Ġusing Ġthe ĠGER BIL Ġevaluation Ġframework ĠB IB REF 23 Ġ. </s> Measure ments </s> We Ġmeasure Ġprecision , Ġrecall Ġand ĠF 1 Ġscore Ġof Ġour ĠD ATE X IS - NER Ġsystem Ġand Ġstate - of - the - art Ġannot ators Ġintroduced Ġin ĠSection ĠSEC REF 2 Ġ. ĠFor Ġthe Ġcomparison Ġwith Ġblack Ġbox Ġsystems , Ġwe Ġevaluate Ġannotation Ġresults Ġusing Ġweak Ġannotation Ġmatch . ĠFor Ġa Ġmore Ġdetailed Ġin - system Ġerror Ġanalysis , Ġwe Ġmeasure ĠB IO 2 Ġlabeling Ġperformance Ġbased Ġon Ġeach Ġtoken . </s> We Ġmeasure Ġthe Ġoverall Ġperformance Ġof Ġmention Ġannotation Ġusing Ġthe Ġevaluation Ġmeasures Ġdefined Ġby Ġcorn olt i 2013 framework , Ġwhich Ġare Ġalso Ġused Ġby Ġling 2015 design . ĠLet ĠIN LINE FORM 0 Ġbe Ġa Ġset Ġof Ġdocuments Ġwith Ġgold Ġstandard Ġmention Ġannotations ĠIN LINE FORM 1 Ġwith Ġa Ġtotal Ġof ĠIN LINE FORM 2 Ġexamples . ĠEach Ġmention ĠIN LINE FORM 3 Ġis Ġdefined Ġby Ġstart Ġposition ĠIN LINE FORM 4 Ġand Ġend Ġposition ĠIN LINE FORM 5 Ġin Ġthe Ġsource Ġdocument ĠIN LINE FORM 6 Ġ. ĠTo Ġquantify Ġthe Ġperformance Ġof Ġthe Ġsystem , Ġwe Ġcompare ĠIN LINE FORM 7 Ġto Ġthe Ġset Ġof Ġpredicted Ġannotations ĠIN LINE FORM 8 Ġwith Ġmentions ĠIN LINE FORM 9 Ġ: ĠDIS PLAY FORM 0 </s> We Ġcompare Ġusing Ġa Ġweak Ġannotation Ġmatch : ĠDIS PLAY FORM 0 </s> We Ġmeasure Ġmicro - aver aged Ġprecision Ġ( ĠIN LINE FORM 0 Ġ), Ġrecall Ġ( ĠIN LINE FORM 1 Ġ) Ġand ĠN ER - style Ġ( ĠIN LINE FORM 2 Ġ) Ġscore : ĠDIS PLAY FORM 0 </s> T uning Ġthe Ġmodel Ġconfiguration Ġwith Ġannotation Ġmatch Ġmeasurement Ġis Ġnot Ġalways Ġfeasible . ĠWe Ġtherefore Ġmeasure ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġseparately Ġfor Ġeach Ġlabel Ġclass ĠIN LINE FORM 4 Ġin Ġour Ġclassification Ġmodel Ġand Ġcalculate Ġbinary Ġclassification Ġprecision ĠIN LINE FORM 5 Ġ, Ġrecall ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġscores . ĠTo Ġavoid Ġskewed Ġresults Ġfrom Ġthe Ġexpected ly Ġlarge ĠIN LINE FORM 8 Ġclass , Ġwe Ġuse Ġmacro - aver aging Ġover Ġthe Ġthree Ġclasses : ĠDIS PLAY FORM 0 </s> E val uation ĠResults </s> We Ġnow Ġdiscuss Ġthe Ġevaluation Ġof Ġour ĠD ATE X IS - NER Ġsystem Ġon Ġcommon Ġand Ġidiosyncr atic Ġdata . </s> Table ĠT AB REF 35 Ġshows Ġthe Ġcomparison Ġof ĠD ATE X IS - NER Ġwith Ġeight Ġstate - of - the - art Ġannot ators Ġon Ġfour Ġcommon Ġnews Ġdata Ġsets . ĠBoth Ġcommon Ġand Ġmedical Ġmodels Ġare Ġconfigured Ġident ically Ġand Ġtrained Ġon Ġonly Ġ2000 Ġlabeled Ġsentences , Ġwithout Ġany Ġexternal Ġprior Ġknowledge . ĠWe Ġobserve Ġthat ĠD ATE X IS - NER Ġachieves Ġthe Ġhighest Ġrecall Ġscores Ġof Ġall Ġtested Ġannot ators , Ġwith Ġ95 % âĢĵ 98 % Ġon Ġall Ġmeasured Ġdata Ġsets . ĠMoreover , ĠD ATE X IS - NER Ġprecision Ġscores Ġare Ġequal Ġor Ġbetter Ġthan Ġmedian . ĠOverall , Ġwe Ġachieve Ġhigh Ġmicro - F 1 Ġscores Ġof Ġ84 % âĢĵ 94 % Ġon Ġnews Ġentity Ġrecognition , Ġwhich Ġis Ġslightly Ġbetter Ġthan Ġthe Ġont ology - based ĠEntity class ifier . eu ĠN ER Ġand Ġreveals Ġa Ġbetter Ġgeneral ization Ġthan Ġthe Ġ3 - type ĠStanford ĠN ER Ġwith Ġdistribution al Ġsemantics . ĠWe Ġnotice Ġthat Ġsystems Ġspecialized Ġon Ġword - sense Ġdis amb ig uation Ġ( B ab elf y , ĠDB pedia ĠSpotlight ) Ġdon 't Ġperform Ġwell Ġon ĠâĢ ľ raw âĢ Ŀ Ġunt yp ed Ġentity Ġrecognition Ġtasks . ĠThe Ġhighest Ġprecision Ġscores Ġare Ġreached Ġby ĠStanford ĠN ER . ĠWe Ġalso Ġnotice Ġa Ġlow Ġprecision Ġof Ġall Ġannot ators Ġon Ġthe ĠACE 2004 Ġdataset Ġand Ġhigh Ġvariance Ġin ĠMSNBC Ġperformance , Ġwhich Ġare Ġprobably Ġcaused Ġby Ġdiffering Ġannotation Ġstandards . </s> Table ĠT AB REF 45 Ġshows Ġthe Ġresults Ġof Ġbiomedical Ġentity Ġrecognition Ġcompared Ġto Ġthe Ġparticipants Ġof Ġthe ĠJ N LP BA Ġ2004 Ġbio - entity Ġrecognition Ġtask ĠB IB REF 14 Ġ. ĠWe Ġnotice Ġthat Ġfor Ġthese Ġwell - written ĠMedline Ġabstract s , Ġthere Ġis Ġnot Ġsuch Ġa Ġstrong Ġskew Ġbetween Ġprecision Ġand Ġrecall . ĠOur ĠD ATE X IS - NER Ġsystem Ġoutper forms Ġthe ĠH MM , ĠMEM M , ĠCR F Ġand ĠCD N Ġbased Ġmodels Ġwith Ġa Ġmicro - F 1 Ġscore Ġof Ġ84 %. ĠHowever , Ġthe Ġhighly Ġspecialized ĠGEN IA Ġchunk er Ġfor ĠLing P ipe Ġachieves Ġhigher Ġscores . ĠThis Ġchunk er Ġis Ġa Ġvery Ġsimple Ġgener ative Ġmodel Ġpredictor Ġthat Ġis Ġbased Ġon Ġa Ġsliding Ġwindow Ġof Ġtwo Ġtokens , Ġword Ġshape Ġand Ġdiction aries . ĠWe Ġinterpret Ġthis Ġscore Ġas Ġstrong Ġover fitting Ġusing Ġa Ġdictionary Ġof Ġthe Ġwell - defined ĠGEN IA Ġterms . ĠTherefore , Ġthis Ġmodel Ġwill Ġgeneral ize Ġhardly Ġconsidering Ġthe Ġsimple Ġmodel . ĠWe Ġcan Ġconfirm Ġthis Ġpresumption Ġin Ġthe Ġcommon Ġdata Ġsets , Ġwhere Ġthe ĠM UC - 6 Ġtrained ĠH MM ĠLing P ipe Ġchunk er Ġperforms Ġon Ġaverage Ġon Ġunseen Ġdata . </s> We Ġevaluate Ġdifferent Ġconfigurations Ġof Ġthe Ġcomponents Ġthat Ġwe Ġdescribe Ġin ĠSection ĠSEC REF 22 Ġ. ĠTable ĠT AB REF 47 Ġshows Ġthe Ġresults Ġof Ġexperiments Ġon Ġboth ĠCo N LL 2003 Ġand ĠGEN IA Ġdata Ġsets . ĠWe Ġreport Ġthe Ġhighest Ġmacro - F 1 Ġscores Ġfor ĠB IO 2 Ġlabeling Ġfor Ġthe Ġconfiguration Ġof Ġletter - tr ig ram Ġword Ġvectors Ġand Ġbid irection al ĠL ST M . ĠWe Ġnotice Ġthat Ġdictionary - based Ġword Ġenc od ings Ġ( D ICT ) Ġwork Ġwell Ġfor Ġidiosyncr atic Ġmedical Ġdomains , Ġwhereas Ġthey Ġsuffer Ġfrom Ġhigh Ġword Ġambiguity Ġin Ġthe Ġnews Ġtexts . ĠPret rained Ġword 2 vec Ġembed d ings Ġ( EM B ) Ġperform Ġwell Ġon Ġnews Ġdata , Ġbut Ġcannot Ġadapt Ġto Ġthe Ġmedical Ġdomain Ġwithout Ġret raining , Ġbecause Ġof Ġa Ġlarge Ġnumber Ġof Ġunseen Ġwords . ĠTherefore , Ġword 2 vec Ġgenerally Ġachieves Ġa Ġhigh Ġprecision Ġon Ġnews Ġtexts , Ġbut Ġlow Ġrecall Ġon Ġmedical Ġtext . ĠThe Ġletter - tr ig ram Ġapproach Ġ( TR I ) Ġcombines Ġboth Ġword Ġvector Ġgeneral ization Ġand Ġrobust ness Ġtowards Ġidiosyncr atic Ġlanguage . </s> We Ġobserve Ġthat Ġthe Ġcontextual ĠL ST M Ġmodel Ġachieves Ġscores Ġthroughout Ġin Ġthe Ġ85 % âĢĵ 94 % Ġrange Ġand Ġsignificantly Ġoutper forms Ġthe Ġfeed - forward Ġ( FF ) Ġbaseline Ġthat Ġshows Ġa Ġmaximum Ġof Ġ75 %. ĠBid irection al ĠL ST Ms Ġcan Ġfurther Ġimprove Ġlabel Ġclassification Ġin Ġboth Ġprecision Ġand Ġrecall . </s> Discussion Ġand ĠError ĠAnalysis </s> We Ġinvestigate Ġdifferent Ġaspects Ġof Ġthe ĠD ATE X IS - NER Ġcomponents Ġby Ġmanual Ġinspection Ġof Ġclassification Ġerrors Ġin Ġthe Ġcontext Ġof Ġthe Ġdocument . ĠFor Ġthe Ġerror Ġclasses Ġdescribed Ġin Ġthe Ġintroduction Ġ( false Ġnegative Ġdetect ions , Ġfalse Ġpositives Ġand Ġinvalid Ġboundaries ), Ġwe Ġobserve Ġfollowing Ġcauses : </s> In Ġdictionary Ġbased Ġconfigurations Ġ( e . g . Ġ1 - hot Ġword Ġvector Ġencoding ĠD ICT ), Ġwe Ġobserve Ġfalse Ġnegative Ġpredictions Ġcaused Ġby Ġdictionary Ġmisses Ġfor Ġwords Ġthat Ġdo Ġnot Ġexist Ġin Ġthe Ġtraining Ġdata . ĠThe Ġcause Ġcan Ġbe Ġrare Ġunseen Ġor Ġnovel Ġwords Ġ( e . g . ĠT - pro ly mph ocy tic Ġcells ) Ġor Ġmiss pell ings Ġ( e . g . Ġstrength n end ). ĠThese Ġwords Ġyield Ġa Ġnull Ġvector Ġresult Ġfrom Ġthe Ġenc oder Ġand Ġcan Ġtherefore Ġnot Ġbe Ġdistinguished Ġby Ġthe ĠL ST M . ĠThe Ġerror Ġincreases Ġwhen Ġusing Ġword 2 vec , Ġbecause Ġthese Ġmodels Ġare Ġtrained Ġwith Ġstop Ġwords Ġfiltered Ġout . ĠThis Ġimpl icates Ġthat Ġe . g . Ġmentions Ġsurrounded Ġby Ġor Ġcontaining Ġa Ġdetermin er Ġ( e . g . ĠThe ĠSunday ĠTelegraph Ġquoted ĠMajor ie ĠOr r ) Ġare Ġhighly Ġerror Ġprone Ġtowards Ġthe Ġdetection Ġof Ġtheir Ġboundaries . ĠWe Ġresolve Ġthis Ġerror Ġby Ġthe Ġletter - tr ig ram Ġapproach . ĠUn seen Ġtrig rams Ġ( e . g . Ġth h ) Ġmay Ġstill Ġbe Ġmissing Ġin Ġthe Ġword Ġvector , Ġbut Ġonly Ġaffect Ġsingle Ġdimensions Ġas Ġopposed Ġto Ġthe Ġvector Ġas Ġa Ġwhole . </s> Sur face Ġforms Ġencode Ġimportant Ġfeatures Ġfor ĠN ER Ġ( e . g . Ġcapital ization Ġof ĠâĢ ľ new âĢ Ŀ Ġin ĠAlan ĠShe arer Ġwas Ġnamed Ġas Ġthe Ġnew ĠEngland Ġcaptain Ġ/ Ġas ĠNew ĠYork Ġbeat Ġthe ĠAngels ). ĠHowever , Ġcase - sensitive Ġword Ġvector ization Ġmethods Ġyield Ġa Ġlarge Ġamount Ġof Ġfalse Ġpositive Ġpredictions Ġcaused Ġby Ġincorrect Ġcapital ization Ġin Ġthe Ġinput Ġdata . ĠAn Ġu pperc ase Ġheadline Ġ( e . g . ĠT ENN IS Ġ- ĠU . S . ĠTEAM ĠON ĠTHE ĠRO AD ĠFOR Ġ1997 ĠF ED ĠC UP ) Ġis Ġencoded Ġcompletely Ġdifferent Ġthan Ġa Ġlower case Ġone Ġ( e . g . ĠU . S . Ġteam Ġon Ġthe Ġroad Ġfor ĠFed ĠCup ). ĠBecause Ġof Ġthat , Ġwe Ġachieve Ġbest Ġresults Ġwith Ġlower case Ġword Ġvectors Ġand Ġadditional Ġsurface Ġform Ġfeature Ġflags , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 14 Ġ. </s> We Ġobserve Ġmentions Ġthat Ġare Ġcomposed Ġof Ġco - occ urring Ġwords Ġwith Ġhigh Ġambiguity Ġ( e . g . Ġdegradation Ġof ĠIk B Ġalpha Ġin ĠT Ġcell Ġlines ). ĠThese Ġgroups Ġencode Ġstrong Ġsynt ag matic Ġword Ġrelations ĠB IB REF 24 Ġthat Ġcan Ġbe Ġlever aged Ġto Ġresolve Ġword Ġsense Ġand Ġhom onyms Ġfrom Ġsentence Ġcontext . ĠTherefore , Ġcorrect Ġboundaries Ġin Ġthese Ġgroups Ġcan Ġeffectively Ġbe Ġidentified Ġonly Ġwith Ġcontextual Ġmodels Ġsuch Ġas ĠL ST Ms . </s> Or th og onal Ġto Ġthe Ġprevious Ġproblem , Ġdifferent Ġwords Ġin Ġa Ġparad igmatic Ġrelation ĠB IB REF 24 Ġcan Ġoccur Ġin Ġthe Ġsame Ġcontext Ġ( e . g . Ġcycl os por in ĠA - treated Ġcells Ġ/ ĠH U Ġtreated Ġcells ). ĠThese Ġgroups Ġare Ġefficiently Ġrepresented Ġin Ġword 2 vec . ĠHowever , Ġletter - tr ig ram Ġvectors Ġcannot Ġencode Ġparad igmatic Ġgroups Ġand Ġtherefore Ġrequire Ġa Ġlarger Ġtraining Ġsample Ġto Ġcapture Ġthese Ġrelations . </s> Often , Ġsyn onyms Ġcan Ġonly Ġbe Ġresolved Ġregarding Ġa Ġlarger Ġdocument Ġcontext Ġthan Ġthe Ġlocal Ġsentence Ġcontext Ġknown Ġby Ġthe ĠL ST M . ĠIn Ġthese Ġcases , Ġword Ġsense Ġis Ġred efined Ġby Ġa Ġtopic Ġmodel Ġlocal Ġto Ġthe Ġparagraph Ġ( e . g . Ġsports : ĠTiger Ġwas Ġlost Ġin Ġthe Ġwoods Ġafter Ġdivorce .). ĠThis Ġproblem Ġdoes Ġnot Ġheavily Ġaffect ĠN ER Ġrecall , Ġbut Ġis Ġcrucial Ġfor Ġnamed Ġentity Ġdis amb ig uation Ġand Ġcore ference Ġresolution . </s> The Ġproposed ĠD ATE X IS - NER Ġmodel Ġis Ġrestricted Ġto Ġrecognize Ġboundaries Ġof Ġgeneric Ġmentions Ġin Ġtext . ĠWe Ġevaluate Ġthe Ġmodel Ġon Ġannotations Ġof Ġisolated Ġtypes Ġ( e . g . Ġpersons , Ġorganizations , Ġlocations ) Ġfor Ġcomparison Ġpurposes Ġonly , Ġbut Ġwe Ġdo Ġnot Ġapproach ĠN ER - style Ġtyping . ĠContrary , Ġwe Ġapproach Ġto Ġdetect Ġmentions Ġwithout Ġtype Ġinformation . ĠThe Ġdetection Ġof Ġspecific Ġtypes Ġcan Ġbe Ġrealized Ġby Ġtraining Ġmultiple Ġindependent Ġmodels Ġon Ġa Ġselection Ġof Ġlabels Ġper Ġtype Ġand Ġnesting Ġthe Ġresulting Ġannotations Ġusing Ġa Ġlongest - span Ġsemantic Ġtype Ġhe uristic ĠB IB REF 25 Ġ. </s> Summary </s> ling 2015 design Ġshow Ġthat Ġthe Ġtask Ġof ĠN ER Ġis Ġnot Ġclearly Ġdefined Ġand Ġrather Ġdepends Ġon Ġa Ġspecific Ġproblem Ġcontext . ĠContrary , Ġmost ĠN ER Ġapproaches Ġare Ġspecifically Ġtrained Ġon Ġfixed Ġdatasets Ġin Ġa Ġbatch Ġmode . ĠWorse , Ġthey Ġoften Ġsuffer Ġfrom Ġpoor Ġrecall ĠB IB REF 26 Ġ. ĠIdeally , Ġone Ġcould Ġpersonal ize Ġthe Ġtask Ġof Ġrecognizing Ġnamed Ġentities , Ġconcepts Ġor Ġphrases Ġaccording Ġto Ġthe Ġspecific Ġproblem . ĠâĢ ľ Personal izing âĢ Ŀ Ġand Ġadapting Ġsuch Ġannot ators Ġshould Ġhappen Ġwith Ġvery Ġlimited Ġhuman Ġlabeling Ġeffort , Ġin Ġparticular Ġfor Ġidiosyncr atic Ġdomains Ġwith Ġsparse Ġtraining Ġdata . </s> Our Ġwork Ġfollows Ġthis Ġline . ĠFrom Ġour Ġresults Ġwe Ġreport ĠF 1 Ġscores Ġbetween Ġ84 âĢĵ 94 % Ġwhen Ġusing Ġbid irection al Ġmulti - lay ered ĠL ST Ms , Ġletter - tr ig ram Ġword Ġhashing Ġand Ġsurface Ġform Ġfeatures Ġon Ġonly Ġfew Ġhundred Ġtraining Ġexamples . </s> This Ġwork Ġis Ġonly Ġa Ġpreliminary Ġstep Ġtowards Ġthe Ġvision Ġof Ġpersonal izing Ġannotation Ġguidelines Ġfor ĠN ER ĠB IB REF 2 Ġ. ĠIn Ġour Ġfuture Ġwork , Ġwe Ġwill Ġfocus Ġon Ġadditional Ġimportant Ġidiosyncr atic Ġdomains , Ġsuch Ġas Ġhealth , Ġlife Ġscience , Ġfashion , Ġengineering Ġor Ġautomotive . ĠFor Ġthese Ġdomains , Ġwe Ġwill Ġconsider Ġthe Ġprocess Ġof Ġdetecting Ġmentions Ġand Ġlinking Ġthem Ġto Ġan Ġont ology Ġas Ġa Ġjoint Ġtask Ġand Ġwe Ġwill Ġinvestigate Ġsimple Ġand Ġinteractive Ġwork flows Ġfor Ġcreating Ġrobust Ġpersonalized Ġnamed Ġentity Ġlinking Ġsystems .
Introduction Information extraction tasks have become very important not only in the Web, but also for in-house enterprise settings. One of the crucial steps towards understanding natural language is named entity recognition (NER), which aims to extract mentions of entity names in text. NER is necessary for many higher-level tasks such as entity linking, relation extraction, building knowledge graphs, question answering and intent based search. In these scenarios, NER recall is critical, as candidates that are never generated can not be recovered later BIBREF0 . Robust Contextual Word Labeling We abstract the task of NER as sequential word labeling problem. Figure FIGREF15 illustrates an example for sequential transformation of a sentence into word labels. We express each sentence in a document as a sequence of words: INLINEFORM0 , e.g. INLINEFORM1 Aspirin. We define a mention as the longest possible span of adjacent tokens that refer to a an entity or relevant concept of a real-world object, such as Aspirin (ASA). We further assume that mentions are non-recursive and non-overlapping. To encode boundaries of the mention span, we adapt the idea of ramshaw1995text, which has been adapted as BIO2 standard in the CoNLL2003 shared task BIBREF15 . We assign labels INLINEFORM2 to each token to mark begin, inside and outside of a mention from left to right. We use the input sequence INLINEFORM3 together with a target sequence INLINEFORM4 of the same length that contains a BIO2 label for each word: INLINEFORM5 , e.g. INLINEFORM6 B. To predict the most likely label INLINEFORM7 of a token regarding its context, we utilize recurrent neural networks. Robust Word Encoding Methods We have shown that most common errors for recall loss are misspellings, POS errors, capitalization, unseen words and irregular context. Therefore we generalize our model throughout three layers: robust word encoding, in-sentence word context and contextual sequence labeling. Dictionary-based word vectorization methods suffer from sparse training sets, especially in the case of non-verbatim mentions, rare words, typing and capitalization errors. For example, the word2vec model of mikolov2013efficient generalizes insufficiently for rare words in idiosyncratic domains or for misspelled words, since for these words no vector representation is learned at training time. In the GENIA data set, we notice 27% unseen words (dictionary misses) in the pretrained word2vec model. As training data generation is expensive, we investigate a generic approach for the generation of word vectors. We use letter-trigram word hashing as introduced by huang2013learning. This technique goes beyond words and generates word vectors as a composite of discriminative three-letter “syllables”, that might also include misspellings. Therefore, it is robust against dictionary misses and has the advantage (despite its name) to group syntactically similar words in similar vector spaces. We compare this approach to word embedding models such as word2vec. The most important features for NER are word shape properties, such as length, initial capitalization, all-word uppercase, in-word capitalization and use of numbers or punctuation BIBREF16 . Mixed-case word encodings implicitly include capitalization features. However, this approach impedes generalization, as words appear in various surface forms, e.g. capitalized at the beginning of sentences, uppercase in headlines, lowercase in social media text. The strong coherence between uppercase and lowercase characters – they might have identical semantics – is not encoded in the embedding. Therefore, we encode the words using lowercase letter-trigrams. To keep the surface information, we add flag bits to the vector that indicate initial capitalization, uppercase, lower case or mixed case. Deep Contextual Sequence Learning With sparse training data in the idiosyncratic domain, we expect input data with high variance. Therefore, we require a strong generalization for the syntactic and semantic representation of language. To reach into the high 80–90% NER F1 performance, long-range context-sensitive information is indispensable. We apply the computational model of recurrent neural networks, in particular long short-term memory networks (LSTMs) BIBREF17 , BIBREF18 to the problem of sequence labeling. Like neural feed-forward networks, LSTMs are able to learn complex parameters using gradient descent, but include additional recurrent connections between cells to influence weight updates over adjacent time steps. With their ability to memorize and forget over time, LSTMs have proven to generalize context-sensitive sequential data well BIBREF19 , BIBREF20 . Figure FIGREF15 shows an unfolded representation of the steps through a sentence. We feed the LSTM with letter-trigram vectors INLINEFORM0 as input data, one word at a time. The hidden layer of the LSTM represents context from long range dependencies over the entire sentence from left to right. However, to achieve deeper contextual understanding over the boundaries of multi-word annotations and at the beginning of sentences, we require a backwards pass through the sentence. We therefore implement a bidirectional LSTM and feed the output of both directions into a second LSTM layer for combined label prediction. For the use in the neural network, word encodings INLINEFORM0 and labels INLINEFORM1 are real-valued vectors. To predict the most likely label INLINEFORM2 of a token, we utilize a LSTM with input nodes INLINEFORM3 , input gates INLINEFORM4 , forget gate INLINEFORM5 , output gate INLINEFORM6 and internal state INLINEFORM7 . For the bidirectional case, all gates are duplicated and combined into forward state INLINEFORM8 and backward state INLINEFORM9 . The network is trained using backpropagation through time (BPTT) by adapting weights INLINEFORM10 and bias parameters INLINEFORM11 to fit the training examples. DISPLAYFORM0 We iterate over labeled sentences in mini-batches and update the weights accordingly. The network is then used to predict label probabilities INLINEFORM0 for unseen word sequences INLINEFORM1 . Implementation of NER Components  To show the impact of our bidirectional LSTM model, we measure annotation performance on three different neural network configurations. We implement all components using the Deeplearning4j framework. For preprocessing (sentence and word tokenization), we use Stanford CoreNLP BIBREF11 . We test the sequence labeler using three input encodings: [noitemsep] DICT: We build a dictionary over all words in the corpus and generate the input vector using 1-hot encoding for each word EMB: We use the GoogleNews word2vec embeddings, which encodes each word as vector of size 300 TRI: we implement letter-trigram word hashing as described in Section SECREF14 . During training and test, we group all tokens of a sentence as mini-batch. We evaluate three different neural network types to show the impact of the bidirectional sequence learner. [noitemsep] FF: As baseline, we train a non-sequential feed-forward model based on a fully connected multilayer perceptron network with 3 hidden layers of size 150 with relu activation, feeding into a 3-class softmax classifier. We train the model using backpropagation with stochastic gradient descent and a learning rate of 0.005. LSTM: We use a configuration of a single feed-forward layer of size 150 with two additional layers of single-direction LSTM with 20 cells and a 3-class softmax classifier. We train the model using backpropagation-through-time (BPTT) with stochastic gradient descent and a learning rate of 0.005. BLSTM: Our final configuration consists of a single feed-forward layer of size 150 with one bidirectional LSTM layer with 20 cells and an additional single-direction LSTM with 20 cells into a 3-class softmax classifier. The BLSTM model is trained the same way as the single-direction LSTM. Evaluation We evaluate nine configurations of our model on five gold standard evaluation data sets. We show that the combination of letter-trigram word hashing with bidirectional LSTM yields the best results and outperforms sequence learners based on dictionaries or word2vec. To highlight the generalization of our model to idiosyncratic domains, we run tests on common-typed data sets as well as on specialized medical documents. We compare our system on these data sets with specialized state-of-the-art systems. Evaluation Set Up We train two models with identical parameterization, each with 2000 randomly chosen labeled sentences from a standard data set. To show the effectiveness of the components, we evaluate different configurations of this setting with 2000 random sentences from the remaining set. The model was trained using Deeplearning4j with nd4j-x86 backend. Training the TRI+BLSTM configuration on a commodity Intel i7 notebook with 4 cores at 2.8GHz takes approximately 50 minutes. Table TABREF33 gives an overview of the standard data sets we use for training. The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc. For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 . Measurements We measure precision, recall and F1 score of our DATEXIS-NER system and state-of-the-art annotators introduced in Section SECREF2 . For the comparison with black box systems, we evaluate annotation results using weak annotation match. For a more detailed in-system error analysis, we measure BIO2 labeling performance based on each token. We measure the overall performance of mention annotation using the evaluation measures defined by cornolti2013framework, which are also used by ling2015design. Let INLINEFORM0 be a set of documents with gold standard mention annotations INLINEFORM1 with a total of INLINEFORM2 examples. Each mention INLINEFORM3 is defined by start position INLINEFORM4 and end position INLINEFORM5 in the source document INLINEFORM6 . To quantify the performance of the system, we compare INLINEFORM7 to the set of predicted annotations INLINEFORM8 with mentions INLINEFORM9 : DISPLAYFORM0 We compare using a weak annotation match: DISPLAYFORM0 We measure micro-averaged precision ( INLINEFORM0 ), recall ( INLINEFORM1 ) and NER-style ( INLINEFORM2 ) score: DISPLAYFORM0 Tuning the model configuration with annotation match measurement is not always feasible. We therefore measure INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 separately for each label class INLINEFORM4 in our classification model and calculate binary classification precision INLINEFORM5 , recall INLINEFORM6 and INLINEFORM7 scores. To avoid skewed results from the expectedly large INLINEFORM8 class, we use macro-averaging over the three classes: DISPLAYFORM0 Evaluation Results We now discuss the evaluation of our DATEXIS-NER system on common and idiosyncratic data. Table TABREF35 shows the comparison of DATEXIS-NER with eight state-of-the-art annotators on four common news data sets. Both common and medical models are configured identically and trained on only 2000 labeled sentences, without any external prior knowledge. We observe that DATEXIS-NER achieves the highest recall scores of all tested annotators, with 95%–98% on all measured data sets. Moreover, DATEXIS-NER precision scores are equal or better than median. Overall, we achieve high micro-F1 scores of 84%–94% on news entity recognition, which is slightly better than the ontology-based Entityclassifier.eu NER and reveals a better generalization than the 3-type Stanford NER with distributional semantics. We notice that systems specialized on word-sense disambiguation (Babelfy, DBpedia Spotlight) don't perform well on “raw” untyped entity recognition tasks. The highest precision scores are reached by Stanford NER. We also notice a low precision of all annotators on the ACE2004 dataset and high variance in MSNBC performance, which are probably caused by differing annotation standards. Table TABREF45 shows the results of biomedical entity recognition compared to the participants of the JNLPBA 2004 bio-entity recognition task BIBREF14 . We notice that for these well-written Medline abstracts, there is not such a strong skew between precision and recall. Our DATEXIS-NER system outperforms the HMM, MEMM, CRF and CDN based models with a micro-F1 score of 84%. However, the highly specialized GENIA chunker for LingPipe achieves higher scores. This chunker is a very simple generative model predictor that is based on a sliding window of two tokens, word shape and dictionaries. We interpret this score as strong overfitting using a dictionary of the well-defined GENIA terms. Therefore, this model will generalize hardly considering the simple model. We can confirm this presumption in the common data sets, where the MUC-6 trained HMM LingPipe chunker performs on average on unseen data. We evaluate different configurations of the components that we describe in Section SECREF22 . Table TABREF47 shows the results of experiments on both CoNLL2003 and GENIA data sets. We report the highest macro-F1 scores for BIO2 labeling for the configuration of letter-trigram word vectors and bidirectional LSTM. We notice that dictionary-based word encodings (DICT) work well for idiosyncratic medical domains, whereas they suffer from high word ambiguity in the news texts. Pretrained word2vec embeddings (EMB) perform well on news data, but cannot adapt to the medical domain without retraining, because of a large number of unseen words. Therefore, word2vec generally achieves a high precision on news texts, but low recall on medical text. The letter-trigram approach (TRI) combines both word vector generalization and robustness towards idiosyncratic language. We observe that the contextual LSTM model achieves scores throughout in the 85%–94% range and significantly outperforms the feed-forward (FF) baseline that shows a maximum of 75%. Bidirectional LSTMs can further improve label classification in both precision and recall. Discussion and Error Analysis We investigate different aspects of the DATEXIS-NER components by manual inspection of classification errors in the context of the document. For the error classes described in the introduction (false negative detections, false positives and invalid boundaries), we observe following causes: In dictionary based configurations (e.g. 1-hot word vector encoding DICT), we observe false negative predictions caused by dictionary misses for words that do not exist in the training data. The cause can be rare unseen or novel words (e.g. T-prolymphocytic cells) or misspellings (e.g. strengthnend). These words yield a null vector result from the encoder and can therefore not be distinguished by the LSTM. The error increases when using word2vec, because these models are trained with stop words filtered out. This implicates that e.g. mentions surrounded by or containing a determiner (e.g. The Sunday Telegraph quoted Majorie Orr) are highly error prone towards the detection of their boundaries. We resolve this error by the letter-trigram approach. Unseen trigrams (e.g. thh) may still be missing in the word vector, but only affect single dimensions as opposed to the vector as a whole. Surface forms encode important features for NER (e.g. capitalization of “new” in Alan Shearer was named as the new England captain / as New York beat the Angels). However, case-sensitive word vectorization methods yield a large amount of false positive predictions caused by incorrect capitalization in the input data. An uppercase headline (e.g. TENNIS - U.S. TEAM ON THE ROAD FOR 1997 FED CUP) is encoded completely different than a lowercase one (e.g. U.S. team on the road for Fed Cup). Because of that, we achieve best results with lowercase word vectors and additional surface form feature flags, as described in Section SECREF14 . We observe mentions that are composed of co-occurring words with high ambiguity (e.g. degradation of IkB alpha in T cell lines). These groups encode strong syntagmatic word relations BIBREF24 that can be leveraged to resolve word sense and homonyms from sentence context. Therefore, correct boundaries in these groups can effectively be identified only with contextual models such as LSTMs. Orthogonal to the previous problem, different words in a paradigmatic relation BIBREF24 can occur in the same context (e.g. cyclosporin A-treated cells / HU treated cells). These groups are efficiently represented in word2vec. However, letter-trigram vectors cannot encode paradigmatic groups and therefore require a larger training sample to capture these relations. Often, synonyms can only be resolved regarding a larger document context than the local sentence context known by the LSTM. In these cases, word sense is redefined by a topic model local to the paragraph (e.g. sports: Tiger was lost in the woods after divorce.). This problem does not heavily affect NER recall, but is crucial for named entity disambiguation and coreference resolution. The proposed DATEXIS-NER model is restricted to recognize boundaries of generic mentions in text. We evaluate the model on annotations of isolated types (e.g. persons, organizations, locations) for comparison purposes only, but we do not approach NER-style typing. Contrary, we approach to detect mentions without type information. The detection of specific types can be realized by training multiple independent models on a selection of labels per type and nesting the resulting annotations using a longest-span semantic type heuristic BIBREF25 . Summary ling2015design show that the task of NER is not clearly defined and rather depends on a specific problem context. Contrary, most NER approaches are specifically trained on fixed datasets in a batch mode. Worse, they often suffer from poor recall BIBREF26 . Ideally, one could personalize the task of recognizing named entities, concepts or phrases according to the specific problem. “Personalizing” and adapting such annotators should happen with very limited human labeling effort, in particular for idiosyncratic domains with sparse training data. Our work follows this line. From our results we report F1 scores between 84–94% when using bidirectional multi-layered LSTMs, letter-trigram word hashing and surface form features on only few hundred training examples. This work is only a preliminary step towards the vision of personalizing annotation guidelines for NER BIBREF2 . In our future work, we will focus on additional important idiosyncratic domains, such as health, life science, fashion, engineering or automotive. For these domains, we will consider the process of detecting mentions and linking them to an ontology as a joint task and we will investigate simple and interactive workflows for creating robust personalized named entity linking systems.
what standard dataset were used?
8eefa116e3c3d3db751423cc4095d1c4153d3a5f
what Ġstandard Ġdataset Ġwere Ġused ?
The ĠGEN IA ĠCorpus Ġ, ĠCo N LL 2003
<s> what Ġstandard Ġdataset Ġwere Ġused ? </s> Introduction </s> Information Ġextraction Ġtasks Ġhave Ġbecome Ġvery Ġimportant Ġnot Ġonly Ġin Ġthe ĠWeb , Ġbut Ġalso Ġfor Ġin - house Ġenterprise Ġsettings . ĠOne Ġof Ġthe Ġcrucial Ġsteps Ġtowards Ġunderstanding Ġnatural Ġlanguage Ġis Ġnamed Ġentity Ġrecognition Ġ( NER ), Ġwhich Ġaims Ġto Ġextract Ġmentions Ġof Ġentity Ġnames Ġin Ġtext . ĠN ER Ġis Ġnecessary Ġfor Ġmany Ġhigher - level Ġtasks Ġsuch Ġas Ġentity Ġlinking , Ġrelation Ġextraction , Ġbuilding Ġknowledge Ġgraphs , Ġquestion Ġanswering Ġand Ġintent Ġbased Ġsearch . ĠIn Ġthese Ġscenarios , ĠN ER Ġrecall Ġis Ġcritical , Ġas Ġcandidates Ġthat Ġare Ġnever Ġgenerated Ġcan Ġnot Ġbe Ġrecovered Ġlater ĠB IB REF 0 Ġ. </s> Rob ust ĠContext ual ĠWord ĠLab eling </s> We Ġabstract Ġthe Ġtask Ġof ĠN ER Ġas Ġsequential Ġword Ġlabeling Ġproblem . ĠFigure ĠFIG REF 15 Ġillustrates Ġan Ġexample Ġfor Ġsequential Ġtransformation Ġof Ġa Ġsentence Ġinto Ġword Ġlabels . ĠWe Ġexpress Ġeach Ġsentence Ġin Ġa Ġdocument Ġas Ġa Ġsequence Ġof Ġwords : ĠIN LINE FORM 0 Ġ, Ġe . g . ĠIN LINE FORM 1 ĠAs pir in . ĠWe Ġdefine Ġa Ġmention Ġas Ġthe Ġlongest Ġpossible Ġspan Ġof Ġadjacent Ġtokens Ġthat Ġrefer Ġto Ġa Ġan Ġentity Ġor Ġrelevant Ġconcept Ġof Ġa Ġreal - world Ġobject , Ġsuch Ġas ĠAs pir in Ġ( AS A ). ĠWe Ġfurther Ġassume Ġthat Ġmentions Ġare Ġnon - rec ursive Ġand Ġnon - over l apping . ĠTo Ġencode Ġboundaries Ġof Ġthe Ġmention Ġspan , Ġwe Ġadapt Ġthe Ġidea Ġof Ġram shaw 1995 text , Ġwhich Ġhas Ġbeen Ġadapted Ġas ĠB IO 2 Ġstandard Ġin Ġthe ĠCo N LL 2003 Ġshared Ġtask ĠB IB REF 15 Ġ. ĠWe Ġassign Ġlabels ĠIN LINE FORM 2 Ġto Ġeach Ġtoken Ġto Ġmark Ġbegin , Ġinside Ġand Ġoutside Ġof Ġa Ġmention Ġfrom Ġleft Ġto Ġright . ĠWe Ġuse Ġthe Ġinput Ġsequence ĠIN LINE FORM 3 Ġtogether Ġwith Ġa Ġtarget Ġsequence ĠIN LINE FORM 4 Ġof Ġthe Ġsame Ġlength Ġthat Ġcontains Ġa ĠB IO 2 Ġlabel Ġfor Ġeach Ġword : ĠIN LINE FORM 5 Ġ, Ġe . g . ĠIN LINE FORM 6 ĠB . ĠTo Ġpredict Ġthe Ġmost Ġlikely Ġlabel ĠIN LINE FORM 7 Ġof Ġa Ġtoken Ġregarding Ġits Ġcontext , Ġwe Ġutilize Ġrecurrent Ġneural Ġnetworks . </s> Rob ust ĠWord ĠEnc oding ĠMethods </s> We Ġhave Ġshown Ġthat Ġmost Ġcommon Ġerrors Ġfor Ġrecall Ġloss Ġare Ġmiss pell ings , ĠPOS Ġerrors , Ġcapital ization , Ġunseen Ġwords Ġand Ġirregular Ġcontext . ĠTherefore Ġwe Ġgeneral ize Ġour Ġmodel Ġthroughout Ġthree Ġlayers : Ġrobust Ġword Ġencoding , Ġin - sent ence Ġword Ġcontext Ġand Ġcontextual Ġsequence Ġlabeling . </s> D ictionary - based Ġword Ġvector ization Ġmethods Ġsuffer Ġfrom Ġsparse Ġtraining Ġsets , Ġespecially Ġin Ġthe Ġcase Ġof Ġnon - ver bat im Ġmentions , Ġrare Ġwords , Ġtyping Ġand Ġcapital ization Ġerrors . ĠFor Ġexample , Ġthe Ġword 2 vec Ġmodel Ġof Ġm ik ol ov 2013 efficient Ġgeneral izes Ġinsufficient ly Ġfor Ġrare Ġwords Ġin Ġidiosyncr atic Ġdomains Ġor Ġfor Ġmiss pelled Ġwords , Ġsince Ġfor Ġthese Ġwords Ġno Ġvector Ġrepresentation Ġis Ġlearned Ġat Ġtraining Ġtime . ĠIn Ġthe ĠGEN IA Ġdata Ġset , Ġwe Ġnotice Ġ27 % Ġunseen Ġwords Ġ( d ictionary Ġmisses ) Ġin Ġthe Ġpret rained Ġword 2 vec Ġmodel . ĠAs Ġtraining Ġdata Ġgeneration Ġis Ġexpensive , Ġwe Ġinvestigate Ġa Ġgeneric Ġapproach Ġfor Ġthe Ġgeneration Ġof Ġword Ġvectors . ĠWe Ġuse Ġletter - tr ig ram Ġword Ġhashing Ġas Ġintroduced Ġby Ġh u ang 2013 learning . ĠThis Ġtechnique Ġgoes Ġbeyond Ġwords Ġand Ġgenerates Ġword Ġvectors Ġas Ġa Ġcomposite Ġof Ġdiscrim in ative Ġthree - letter ĠâĢ ľ sy ll ables âĢ Ŀ , Ġthat Ġmight Ġalso Ġinclude Ġmiss pell ings . ĠTherefore , Ġit Ġis Ġrobust Ġagainst Ġdictionary Ġmisses Ġand Ġhas Ġthe Ġadvantage Ġ( despite Ġits Ġname ) Ġto Ġgroup Ġsynt act ically Ġsimilar Ġwords Ġin Ġsimilar Ġvector Ġspaces . ĠWe Ġcompare Ġthis Ġapproach Ġto Ġword Ġembed ding Ġmodels Ġsuch Ġas Ġword 2 vec . </s> The Ġmost Ġimportant Ġfeatures Ġfor ĠN ER Ġare Ġword Ġshape Ġproperties , Ġsuch Ġas Ġlength , Ġinitial Ġcapital ization , Ġall - word Ġu pperc ase , Ġin - word Ġcapital ization Ġand Ġuse Ġof Ġnumbers Ġor Ġpunct uation ĠB IB REF 16 Ġ. ĠMixed - case Ġword Ġenc od ings Ġimplicitly Ġinclude Ġcapital ization Ġfeatures . ĠHowever , Ġthis Ġapproach Ġimped es Ġgeneral ization , Ġas Ġwords Ġappear Ġin Ġvarious Ġsurface Ġforms , Ġe . g . Ġcapital ized Ġat Ġthe Ġbeginning Ġof Ġsentences , Ġu pperc ase Ġin Ġheadlines , Ġlower case Ġin Ġsocial Ġmedia Ġtext . ĠThe Ġstrong Ġco herence Ġbetween Ġu pperc ase Ġand Ġlower case Ġcharacters ĠâĢĵ Ġthey Ġmight Ġhave Ġidentical Ġsemantics ĠâĢĵ Ġis Ġnot Ġencoded Ġin Ġthe Ġembed ding . ĠTherefore , Ġwe Ġencode Ġthe Ġwords Ġusing Ġlower case Ġletter - tr ig rams . ĠTo Ġkeep Ġthe Ġsurface Ġinformation , Ġwe Ġadd Ġflag Ġbits Ġto Ġthe Ġvector Ġthat Ġindicate Ġinitial Ġcapital ization , Ġu pperc ase , Ġlower Ġcase Ġor Ġmixed Ġcase . </s> Deep ĠContext ual ĠSequence ĠLearning </s> With Ġsparse Ġtraining Ġdata Ġin Ġthe Ġidiosyncr atic Ġdomain , Ġwe Ġexpect Ġinput Ġdata Ġwith Ġhigh Ġvariance . ĠTherefore , Ġwe Ġrequire Ġa Ġstrong Ġgeneral ization Ġfor Ġthe Ġsynt actic Ġand Ġsemantic Ġrepresentation Ġof Ġlanguage . ĠTo Ġreach Ġinto Ġthe Ġhigh Ġ80 âĢĵ 90 % ĠN ER ĠF 1 Ġperformance , Ġlong - range Ġcontext - sensitive Ġinformation Ġis Ġindispensable . ĠWe Ġapply Ġthe Ġcomputational Ġmodel Ġof Ġrecurrent Ġneural Ġnetworks , Ġin Ġparticular Ġlong Ġshort - term Ġmemory Ġnetworks Ġ( L ST Ms ) ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġto Ġthe Ġproblem Ġof Ġsequence Ġlabeling . ĠLike Ġneural Ġfeed - forward Ġnetworks , ĠL ST Ms Ġare Ġable Ġto Ġlearn Ġcomplex Ġparameters Ġusing Ġgradient Ġdescent , Ġbut Ġinclude Ġadditional Ġrecurrent Ġconnections Ġbetween Ġcells Ġto Ġinfluence Ġweight Ġupdates Ġover Ġadjacent Ġtime Ġsteps . ĠWith Ġtheir Ġability Ġto Ġmemor ize Ġand Ġforget Ġover Ġtime , ĠL ST Ms Ġhave Ġproven Ġto Ġgeneral ize Ġcontext - sensitive Ġsequential Ġdata Ġwell ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. </s> Figure ĠFIG REF 15 Ġshows Ġan Ġunfolded Ġrepresentation Ġof Ġthe Ġsteps Ġthrough Ġa Ġsentence . ĠWe Ġfeed Ġthe ĠL ST M Ġwith Ġletter - tr ig ram Ġvectors ĠIN LINE FORM 0 Ġas Ġinput Ġdata , Ġone Ġword Ġat Ġa Ġtime . ĠThe Ġhidden Ġlayer Ġof Ġthe ĠL ST M Ġrepresents Ġcontext Ġfrom Ġlong Ġrange Ġdependencies Ġover Ġthe Ġentire Ġsentence Ġfrom Ġleft Ġto Ġright . ĠHowever , Ġto Ġachieve Ġdeeper Ġcontextual Ġunderstanding Ġover Ġthe Ġboundaries Ġof Ġmulti - word Ġannotations Ġand Ġat Ġthe Ġbeginning Ġof Ġsentences , Ġwe Ġrequire Ġa Ġbackwards Ġpass Ġthrough Ġthe Ġsentence . ĠWe Ġtherefore Ġimplement Ġa Ġbid irection al ĠL ST M Ġand Ġfeed Ġthe Ġoutput Ġof Ġboth Ġdirections Ġinto Ġa Ġsecond ĠL ST M Ġlayer Ġfor Ġcombined Ġlabel Ġprediction . </s> For Ġthe Ġuse Ġin Ġthe Ġneural Ġnetwork , Ġword Ġenc od ings ĠIN LINE FORM 0 Ġand Ġlabels ĠIN LINE FORM 1 Ġare Ġreal - valued Ġvectors . ĠTo Ġpredict Ġthe Ġmost Ġlikely Ġlabel ĠIN LINE FORM 2 Ġof Ġa Ġtoken , Ġwe Ġutilize Ġa ĠL ST M Ġwith Ġinput Ġnodes ĠIN LINE FORM 3 Ġ, Ġinput Ġgates ĠIN LINE FORM 4 Ġ, Ġforget Ġgate ĠIN LINE FORM 5 Ġ, Ġoutput Ġgate ĠIN LINE FORM 6 Ġand Ġinternal Ġstate ĠIN LINE FORM 7 Ġ. ĠFor Ġthe Ġbid irection al Ġcase , Ġall Ġgates Ġare Ġdupl icated Ġand Ġcombined Ġinto Ġforward Ġstate ĠIN LINE FORM 8 Ġand Ġbackward Ġstate ĠIN LINE FORM 9 Ġ. ĠThe Ġnetwork Ġis Ġtrained Ġusing Ġback prop ag ation Ġthrough Ġtime Ġ( B PT T ) Ġby Ġadapting Ġweights ĠIN LINE FORM 10 Ġand Ġbias Ġparameters ĠIN LINE FORM 11 Ġto Ġfit Ġthe Ġtraining Ġexamples . ĠDIS PLAY FORM 0 </s> We Ġiter ate Ġover Ġlabeled Ġsentences Ġin Ġmini - bat ches Ġand Ġupdate Ġthe Ġweights Ġaccordingly . ĠThe Ġnetwork Ġis Ġthen Ġused Ġto Ġpredict Ġlabel Ġprobabilities ĠIN LINE FORM 0 Ġfor Ġunseen Ġword Ġsequences ĠIN LINE FORM 1 Ġ. </s> Im plementation Ġof ĠN ER ĠComponents Ġ </s> To Ġshow Ġthe Ġimpact Ġof Ġour Ġbid irection al ĠL ST M Ġmodel , Ġwe Ġmeasure Ġannotation Ġperformance Ġon Ġthree Ġdifferent Ġneural Ġnetwork Ġconfigurations . ĠWe Ġimplement Ġall Ġcomponents Ġusing Ġthe ĠDeep learning 4 j Ġframework . ĠFor Ġpre processing Ġ( sent ence Ġand Ġword Ġtoken ization ), Ġwe Ġuse ĠStanford ĠCore N LP ĠB IB REF 11 Ġ. ĠWe Ġtest Ġthe Ġsequence Ġlabel er Ġusing Ġthree Ġinput Ġenc od ings : </s> [ no item se p ] </s> D ICT : ĠWe Ġbuild Ġa Ġdictionary Ġover Ġall Ġwords Ġin Ġthe Ġcorpus Ġand Ġgenerate Ġthe Ġinput Ġvector Ġusing Ġ1 - hot Ġencoding Ġfor Ġeach Ġword </s> EM B : ĠWe Ġuse Ġthe ĠGoogle News Ġword 2 vec Ġembed d ings , Ġwhich Ġenc odes Ġeach Ġword Ġas Ġvector Ġof Ġsize Ġ300 </s> TR I : Ġwe Ġimplement Ġletter - tr ig ram Ġword Ġhashing Ġas Ġdescribed Ġin ĠSection ĠSEC REF 14 Ġ. </s> During Ġtraining Ġand Ġtest , Ġwe Ġgroup Ġall Ġtokens Ġof Ġa Ġsentence Ġas Ġmini - batch . ĠWe Ġevaluate Ġthree Ġdifferent Ġneural Ġnetwork Ġtypes Ġto Ġshow Ġthe Ġimpact Ġof Ġthe Ġbid irection al Ġsequence Ġlear ner . </s> [ no item se p ] </s> FF : ĠAs Ġbaseline , Ġwe Ġtrain Ġa Ġnon - sequ ential Ġfeed - forward Ġmodel Ġbased Ġon Ġa Ġfully Ġconnected Ġmult il ayer Ġpercept ron Ġnetwork Ġwith Ġ3 Ġhidden Ġlayers Ġof Ġsize Ġ150 Ġwith Ġrel u Ġactivation , Ġfeeding Ġinto Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġback prop ag ation Ġwith Ġsto ch astic Ġgradient Ġdescent Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 005 . </s> L ST M : ĠWe Ġuse Ġa Ġconfiguration Ġof Ġa Ġsingle Ġfeed - forward Ġlayer Ġof Ġsize Ġ150 Ġwith Ġtwo Ġadditional Ġlayers Ġof Ġsingle - direction ĠL ST M Ġwith Ġ20 Ġcells Ġand Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠWe Ġtrain Ġthe Ġmodel Ġusing Ġback prop ag ation - through - time Ġ( B PT T ) Ġwith Ġsto ch astic Ġgradient Ġdescent Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 005 . </s> BL ST M : ĠOur Ġfinal Ġconfiguration Ġconsists Ġof Ġa Ġsingle Ġfeed - forward Ġlayer Ġof Ġsize Ġ150 Ġwith Ġone Ġbid irection al ĠL ST M Ġlayer Ġwith Ġ20 Ġcells Ġand Ġan Ġadditional Ġsingle - direction ĠL ST M Ġwith Ġ20 Ġcells Ġinto Ġa Ġ3 - class Ġsoft max Ġclass ifier . ĠThe ĠBL ST M Ġmodel Ġis Ġtrained Ġthe Ġsame Ġway Ġas Ġthe Ġsingle - direction ĠL ST M . </s> E val uation </s> We Ġevaluate Ġnine Ġconfigurations Ġof Ġour Ġmodel Ġon Ġfive Ġgold Ġstandard Ġevaluation Ġdata Ġsets . ĠWe Ġshow Ġthat Ġthe Ġcombination Ġof Ġletter - tr ig ram Ġword Ġhashing Ġwith Ġbid irection al ĠL ST M Ġyields Ġthe Ġbest Ġresults Ġand Ġoutper forms Ġsequence Ġlearners Ġbased Ġon Ġdiction aries Ġor Ġword 2 vec . ĠTo Ġhighlight Ġthe Ġgeneral ization Ġof Ġour Ġmodel Ġto Ġidiosyncr atic Ġdomains , Ġwe Ġrun Ġtests Ġon Ġcommon - ty ped Ġdata Ġsets Ġas Ġwell Ġas Ġon Ġspecialized Ġmedical Ġdocuments . ĠWe Ġcompare Ġour Ġsystem Ġon Ġthese Ġdata Ġsets Ġwith Ġspecialized Ġstate - of - the - art Ġsystems . </s> E val uation ĠSet ĠUp </s> We Ġtrain Ġtwo Ġmodels Ġwith Ġidentical Ġparameter ization , Ġeach Ġwith Ġ2000 Ġrandomly Ġchosen Ġlabeled Ġsentences Ġfrom Ġa Ġstandard Ġdata Ġset . ĠTo Ġshow Ġthe Ġeffectiveness Ġof Ġthe Ġcomponents , Ġwe Ġevaluate Ġdifferent Ġconfigurations Ġof Ġthis Ġsetting Ġwith Ġ2000 Ġrandom Ġsentences Ġfrom Ġthe Ġremaining Ġset . ĠThe Ġmodel Ġwas Ġtrained Ġusing ĠDeep learning 4 j Ġwith Ġn d 4 j - x 86 Ġbackend . ĠTraining Ġthe ĠTRI + BL ST M Ġconfiguration Ġon Ġa Ġcommodity ĠIntel Ġi 7 Ġnotebook Ġwith Ġ4 Ġcores Ġat Ġ2 . 8 GHz Ġtakes Ġapproximately Ġ50 Ġminutes . </s> Table ĠT AB REF 33 Ġgives Ġan Ġoverview Ġof Ġthe Ġstandard Ġdata Ġsets Ġwe Ġuse Ġfor Ġtraining . ĠThe ĠGEN IA ĠCorpus ĠB IB REF 3 Ġcontains Ġbiomedical Ġabstract s Ġfrom Ġthe ĠPubMed Ġdatabase . ĠWe Ġuse ĠGEN IA Ġtechnical Ġterm Ġannotations Ġ3 . 02 , Ġwhich Ġcover Ġlinguistic Ġexpressions Ġto Ġentities Ġof Ġinterest Ġin Ġmolecular Ġbiology , Ġe . g . Ġproteins , Ġgenes Ġand Ġcells . ĠCo N LL 2003 ĠB IB REF 14 Ġis Ġa Ġstandard ĠN ER Ġdataset Ġbased Ġon Ġthe ĠReuters ĠRC V - 1 Ġnews Ġcorpus . ĠIt Ġcovers Ġnamed Ġentities Ġof Ġtype Ġperson , Ġlocation , Ġorganization Ġand Ġmisc . </s> For Ġtesting Ġthe Ġoverall Ġannotation Ġperformance , Ġwe Ġutilize ĠCo N LL 2003 - test A Ġand Ġa Ġ50 Ġdocument Ġsplit Ġfrom ĠGEN IA . ĠAdditionally , Ġwe Ġtest Ġon Ġthe Ġcomplete ĠK ORE 50 ĠB IB REF 21 Ġ, ĠACE 2004 ĠB IB REF 22 Ġand ĠMSNBC Ġdata Ġsets Ġusing Ġthe ĠGER BIL Ġevaluation Ġframework ĠB IB REF 23 Ġ. </s> Measure ments </s> We Ġmeasure Ġprecision , Ġrecall Ġand ĠF 1 Ġscore Ġof Ġour ĠD ATE X IS - NER Ġsystem Ġand Ġstate - of - the - art Ġannot ators Ġintroduced Ġin ĠSection ĠSEC REF 2 Ġ. ĠFor Ġthe Ġcomparison Ġwith Ġblack Ġbox Ġsystems , Ġwe Ġevaluate Ġannotation Ġresults Ġusing Ġweak Ġannotation Ġmatch . ĠFor Ġa Ġmore Ġdetailed Ġin - system Ġerror Ġanalysis , Ġwe Ġmeasure ĠB IO 2 Ġlabeling Ġperformance Ġbased Ġon Ġeach Ġtoken . </s> We Ġmeasure Ġthe Ġoverall Ġperformance Ġof Ġmention Ġannotation Ġusing Ġthe Ġevaluation Ġmeasures Ġdefined Ġby Ġcorn olt i 2013 framework , Ġwhich Ġare Ġalso Ġused Ġby Ġling 2015 design . ĠLet ĠIN LINE FORM 0 Ġbe Ġa Ġset Ġof Ġdocuments Ġwith Ġgold Ġstandard Ġmention Ġannotations ĠIN LINE FORM 1 Ġwith Ġa Ġtotal Ġof ĠIN LINE FORM 2 Ġexamples . ĠEach Ġmention ĠIN LINE FORM 3 Ġis Ġdefined Ġby Ġstart Ġposition ĠIN LINE FORM 4 Ġand Ġend Ġposition ĠIN LINE FORM 5 Ġin Ġthe Ġsource Ġdocument ĠIN LINE FORM 6 Ġ. ĠTo Ġquantify Ġthe Ġperformance Ġof Ġthe Ġsystem , Ġwe Ġcompare ĠIN LINE FORM 7 Ġto Ġthe Ġset Ġof Ġpredicted Ġannotations ĠIN LINE FORM 8 Ġwith Ġmentions ĠIN LINE FORM 9 Ġ: ĠDIS PLAY FORM 0 </s> We Ġcompare Ġusing Ġa Ġweak Ġannotation Ġmatch : ĠDIS PLAY FORM 0 </s> We Ġmeasure Ġmicro - aver aged Ġprecision Ġ( ĠIN LINE FORM 0 Ġ), Ġrecall Ġ( ĠIN LINE FORM 1 Ġ) Ġand ĠN ER - style Ġ( ĠIN LINE FORM 2 Ġ) Ġscore : ĠDIS PLAY FORM 0 </s> T uning Ġthe Ġmodel Ġconfiguration Ġwith Ġannotation Ġmatch Ġmeasurement Ġis Ġnot Ġalways Ġfeasible . ĠWe Ġtherefore Ġmeasure ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġseparately Ġfor Ġeach Ġlabel Ġclass ĠIN LINE FORM 4 Ġin Ġour Ġclassification Ġmodel Ġand Ġcalculate Ġbinary Ġclassification Ġprecision ĠIN LINE FORM 5 Ġ, Ġrecall ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġscores . ĠTo Ġavoid Ġskewed Ġresults Ġfrom Ġthe Ġexpected ly Ġlarge ĠIN LINE FORM 8 Ġclass , Ġwe Ġuse Ġmacro - aver aging Ġover Ġthe Ġthree Ġclasses : ĠDIS PLAY FORM 0 </s> E val uation ĠResults </s> We Ġnow Ġdiscuss Ġthe Ġevaluation Ġof Ġour ĠD ATE X IS - NER Ġsystem Ġon Ġcommon Ġand Ġidiosyncr atic Ġdata . </s> Table ĠT AB REF 35 Ġshows Ġthe Ġcomparison Ġof ĠD ATE X IS - NER Ġwith Ġeight Ġstate - of - the - art Ġannot ators Ġon Ġfour Ġcommon Ġnews Ġdata Ġsets . ĠBoth Ġcommon Ġand Ġmedical Ġmodels Ġare Ġconfigured Ġident ically Ġand Ġtrained Ġon Ġonly Ġ2000 Ġlabeled Ġsentences , Ġwithout Ġany Ġexternal Ġprior Ġknowledge . ĠWe Ġobserve Ġthat ĠD ATE X IS - NER Ġachieves Ġthe Ġhighest Ġrecall Ġscores Ġof Ġall Ġtested Ġannot ators , Ġwith Ġ95 % âĢĵ 98 % Ġon Ġall Ġmeasured Ġdata Ġsets . ĠMoreover , ĠD ATE X IS - NER Ġprecision Ġscores Ġare Ġequal Ġor Ġbetter Ġthan Ġmedian . ĠOverall , Ġwe Ġachieve Ġhigh Ġmicro - F 1 Ġscores Ġof Ġ84 % âĢĵ 94 % Ġon Ġnews Ġentity Ġrecognition , Ġwhich Ġis Ġslightly Ġbetter Ġthan Ġthe Ġont ology - based ĠEntity class ifier . eu ĠN ER Ġand Ġreveals Ġa Ġbetter Ġgeneral ization Ġthan Ġthe Ġ3 - type ĠStanford ĠN ER Ġwith Ġdistribution al Ġsemantics . ĠWe Ġnotice Ġthat Ġsystems Ġspecialized Ġon Ġword - sense Ġdis amb ig uation Ġ( B ab elf y , ĠDB pedia ĠSpotlight ) Ġdon 't Ġperform Ġwell Ġon ĠâĢ ľ raw âĢ Ŀ Ġunt yp ed Ġentity Ġrecognition Ġtasks . ĠThe Ġhighest Ġprecision Ġscores Ġare Ġreached Ġby ĠStanford ĠN ER . ĠWe Ġalso Ġnotice Ġa Ġlow Ġprecision Ġof Ġall Ġannot ators Ġon Ġthe ĠACE 2004 Ġdataset Ġand Ġhigh Ġvariance Ġin ĠMSNBC Ġperformance , Ġwhich Ġare Ġprobably Ġcaused Ġby Ġdiffering Ġannotation Ġstandards . </s> Table ĠT AB REF 45 Ġshows Ġthe Ġresults Ġof Ġbiomedical Ġentity Ġrecognition Ġcompared Ġto Ġthe Ġparticipants Ġof Ġthe ĠJ N LP BA Ġ2004 Ġbio - entity Ġrecognition Ġtask ĠB IB REF 14 Ġ. ĠWe Ġnotice Ġthat Ġfor Ġthese Ġwell - written ĠMedline Ġabstract s , Ġthere Ġis Ġnot Ġsuch Ġa Ġstrong Ġskew Ġbetween Ġprecision Ġand Ġrecall . ĠOur ĠD ATE X IS - NER Ġsystem Ġoutper forms Ġthe ĠH MM , ĠMEM M , ĠCR F Ġand ĠCD N Ġbased Ġmodels Ġwith Ġa Ġmicro - F 1 Ġscore Ġof Ġ84 %. ĠHowever , Ġthe Ġhighly Ġspecialized ĠGEN IA Ġchunk er Ġfor ĠLing P ipe Ġachieves Ġhigher Ġscores . ĠThis Ġchunk er Ġis Ġa Ġvery Ġsimple Ġgener ative Ġmodel Ġpredictor Ġthat Ġis Ġbased Ġon Ġa Ġsliding Ġwindow Ġof Ġtwo Ġtokens , Ġword Ġshape Ġand Ġdiction aries . ĠWe Ġinterpret Ġthis Ġscore Ġas Ġstrong Ġover fitting Ġusing Ġa Ġdictionary Ġof Ġthe Ġwell - defined ĠGEN IA Ġterms . ĠTherefore , Ġthis Ġmodel Ġwill Ġgeneral ize Ġhardly Ġconsidering Ġthe Ġsimple Ġmodel . ĠWe Ġcan Ġconfirm Ġthis Ġpresumption Ġin Ġthe Ġcommon Ġdata Ġsets , Ġwhere Ġthe ĠM UC - 6 Ġtrained ĠH MM ĠLing P ipe Ġchunk er Ġperforms Ġon Ġaverage Ġon Ġunseen Ġdata . </s> We Ġevaluate Ġdifferent Ġconfigurations Ġof Ġthe Ġcomponents Ġthat Ġwe Ġdescribe Ġin ĠSection ĠSEC REF 22 Ġ. ĠTable ĠT AB REF 47 Ġshows Ġthe Ġresults Ġof Ġexperiments Ġon Ġboth ĠCo N LL 2003 Ġand ĠGEN IA Ġdata Ġsets . ĠWe Ġreport Ġthe Ġhighest Ġmacro - F 1 Ġscores Ġfor ĠB IO 2 Ġlabeling Ġfor Ġthe Ġconfiguration Ġof Ġletter - tr ig ram Ġword Ġvectors Ġand Ġbid irection al ĠL ST M . ĠWe Ġnotice Ġthat Ġdictionary - based Ġword Ġenc od ings Ġ( D ICT ) Ġwork Ġwell Ġfor Ġidiosyncr atic Ġmedical Ġdomains , Ġwhereas Ġthey Ġsuffer Ġfrom Ġhigh Ġword Ġambiguity Ġin Ġthe Ġnews Ġtexts . ĠPret rained Ġword 2 vec Ġembed d ings Ġ( EM B ) Ġperform Ġwell Ġon Ġnews Ġdata , Ġbut Ġcannot Ġadapt Ġto Ġthe Ġmedical Ġdomain Ġwithout Ġret raining , Ġbecause Ġof Ġa Ġlarge Ġnumber Ġof Ġunseen Ġwords . ĠTherefore , Ġword 2 vec Ġgenerally Ġachieves Ġa Ġhigh Ġprecision Ġon Ġnews Ġtexts , Ġbut Ġlow Ġrecall Ġon Ġmedical Ġtext . ĠThe Ġletter - tr ig ram Ġapproach Ġ( TR I ) Ġcombines Ġboth Ġword Ġvector Ġgeneral ization Ġand Ġrobust ness Ġtowards Ġidiosyncr atic Ġlanguage . </s> We Ġobserve Ġthat Ġthe Ġcontextual ĠL ST M Ġmodel Ġachieves Ġscores Ġthroughout Ġin Ġthe Ġ85 % âĢĵ 94 % Ġrange Ġand Ġsignificantly Ġoutper forms Ġthe Ġfeed - forward Ġ( FF ) Ġbaseline Ġthat Ġshows Ġa Ġmaximum Ġof Ġ75 %. ĠBid irection al ĠL ST Ms Ġcan Ġfurther Ġimprove Ġlabel Ġclassification Ġin Ġboth Ġprecision Ġand Ġrecall . </s> Discussion Ġand ĠError ĠAnalysis </s> We Ġinvestigate Ġdifferent Ġaspects Ġof Ġthe ĠD ATE X IS - NER Ġcomponents Ġby Ġmanual Ġinspection Ġof Ġclassification Ġerrors Ġin Ġthe Ġcontext Ġof Ġthe Ġdocument . ĠFor Ġthe Ġerror Ġclasses Ġdescribed Ġin Ġthe Ġintroduction Ġ( false Ġnegative Ġdetect ions , Ġfalse Ġpositives Ġand Ġinvalid Ġboundaries ), Ġwe Ġobserve Ġfollowing Ġcauses : </s> In Ġdictionary Ġbased Ġconfigurations Ġ( e . g . Ġ1 - hot Ġword Ġvector Ġencoding ĠD ICT ), Ġwe Ġobserve Ġfalse Ġnegative Ġpredictions Ġcaused Ġby Ġdictionary Ġmisses Ġfor Ġwords Ġthat Ġdo Ġnot Ġexist Ġin Ġthe Ġtraining Ġdata . ĠThe Ġcause Ġcan Ġbe Ġrare Ġunseen Ġor Ġnovel Ġwords Ġ( e . g . ĠT - pro ly mph ocy tic Ġcells ) Ġor Ġmiss pell ings Ġ( e . g . Ġstrength n end ). ĠThese Ġwords Ġyield Ġa Ġnull Ġvector Ġresult Ġfrom Ġthe Ġenc oder Ġand Ġcan Ġtherefore Ġnot Ġbe Ġdistinguished Ġby Ġthe ĠL ST M . ĠThe Ġerror Ġincreases Ġwhen Ġusing Ġword 2 vec , Ġbecause Ġthese Ġmodels Ġare Ġtrained Ġwith Ġstop Ġwords Ġfiltered Ġout . ĠThis Ġimpl icates Ġthat Ġe . g . Ġmentions Ġsurrounded Ġby Ġor Ġcontaining Ġa Ġdetermin er Ġ( e . g . ĠThe ĠSunday ĠTelegraph Ġquoted ĠMajor ie ĠOr r ) Ġare Ġhighly Ġerror Ġprone Ġtowards Ġthe Ġdetection Ġof Ġtheir Ġboundaries . ĠWe Ġresolve Ġthis Ġerror Ġby Ġthe Ġletter - tr ig ram Ġapproach . ĠUn seen Ġtrig rams Ġ( e . g . Ġth h ) Ġmay Ġstill Ġbe Ġmissing Ġin Ġthe Ġword Ġvector , Ġbut Ġonly Ġaffect Ġsingle Ġdimensions Ġas Ġopposed Ġto Ġthe Ġvector Ġas Ġa Ġwhole . </s> Sur face Ġforms Ġencode Ġimportant Ġfeatures Ġfor ĠN ER Ġ( e . g . Ġcapital ization Ġof ĠâĢ ľ new âĢ Ŀ Ġin ĠAlan ĠShe arer Ġwas Ġnamed Ġas Ġthe Ġnew ĠEngland Ġcaptain Ġ/ Ġas ĠNew ĠYork Ġbeat Ġthe ĠAngels ). ĠHowever , Ġcase - sensitive Ġword Ġvector ization Ġmethods Ġyield Ġa Ġlarge Ġamount Ġof Ġfalse Ġpositive Ġpredictions Ġcaused Ġby Ġincorrect Ġcapital ization Ġin Ġthe Ġinput Ġdata . ĠAn Ġu pperc ase Ġheadline Ġ( e . g . ĠT ENN IS Ġ- ĠU . S . ĠTEAM ĠON ĠTHE ĠRO AD ĠFOR Ġ1997 ĠF ED ĠC UP ) Ġis Ġencoded Ġcompletely Ġdifferent Ġthan Ġa Ġlower case Ġone Ġ( e . g . ĠU . S . Ġteam Ġon Ġthe Ġroad Ġfor ĠFed ĠCup ). ĠBecause Ġof Ġthat , Ġwe Ġachieve Ġbest Ġresults Ġwith Ġlower case Ġword Ġvectors Ġand Ġadditional Ġsurface Ġform Ġfeature Ġflags , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 14 Ġ. </s> We Ġobserve Ġmentions Ġthat Ġare Ġcomposed Ġof Ġco - occ urring Ġwords Ġwith Ġhigh Ġambiguity Ġ( e . g . Ġdegradation Ġof ĠIk B Ġalpha Ġin ĠT Ġcell Ġlines ). ĠThese Ġgroups Ġencode Ġstrong Ġsynt ag matic Ġword Ġrelations ĠB IB REF 24 Ġthat Ġcan Ġbe Ġlever aged Ġto Ġresolve Ġword Ġsense Ġand Ġhom onyms Ġfrom Ġsentence Ġcontext . ĠTherefore , Ġcorrect Ġboundaries Ġin Ġthese Ġgroups Ġcan Ġeffectively Ġbe Ġidentified Ġonly Ġwith Ġcontextual Ġmodels Ġsuch Ġas ĠL ST Ms . </s> Or th og onal Ġto Ġthe Ġprevious Ġproblem , Ġdifferent Ġwords Ġin Ġa Ġparad igmatic Ġrelation ĠB IB REF 24 Ġcan Ġoccur Ġin Ġthe Ġsame Ġcontext Ġ( e . g . Ġcycl os por in ĠA - treated Ġcells Ġ/ ĠH U Ġtreated Ġcells ). ĠThese Ġgroups Ġare Ġefficiently Ġrepresented Ġin Ġword 2 vec . ĠHowever , Ġletter - tr ig ram Ġvectors Ġcannot Ġencode Ġparad igmatic Ġgroups Ġand Ġtherefore Ġrequire Ġa Ġlarger Ġtraining Ġsample Ġto Ġcapture Ġthese Ġrelations . </s> Often , Ġsyn onyms Ġcan Ġonly Ġbe Ġresolved Ġregarding Ġa Ġlarger Ġdocument Ġcontext Ġthan Ġthe Ġlocal Ġsentence Ġcontext Ġknown Ġby Ġthe ĠL ST M . ĠIn Ġthese Ġcases , Ġword Ġsense Ġis Ġred efined Ġby Ġa Ġtopic Ġmodel Ġlocal Ġto Ġthe Ġparagraph Ġ( e . g . Ġsports : ĠTiger Ġwas Ġlost Ġin Ġthe Ġwoods Ġafter Ġdivorce .). ĠThis Ġproblem Ġdoes Ġnot Ġheavily Ġaffect ĠN ER Ġrecall , Ġbut Ġis Ġcrucial Ġfor Ġnamed Ġentity Ġdis amb ig uation Ġand Ġcore ference Ġresolution . </s> The Ġproposed ĠD ATE X IS - NER Ġmodel Ġis Ġrestricted Ġto Ġrecognize Ġboundaries Ġof Ġgeneric Ġmentions Ġin Ġtext . ĠWe Ġevaluate Ġthe Ġmodel Ġon Ġannotations Ġof Ġisolated Ġtypes Ġ( e . g . Ġpersons , Ġorganizations , Ġlocations ) Ġfor Ġcomparison Ġpurposes Ġonly , Ġbut Ġwe Ġdo Ġnot Ġapproach ĠN ER - style Ġtyping . ĠContrary , Ġwe Ġapproach Ġto Ġdetect Ġmentions Ġwithout Ġtype Ġinformation . ĠThe Ġdetection Ġof Ġspecific Ġtypes Ġcan Ġbe Ġrealized Ġby Ġtraining Ġmultiple Ġindependent Ġmodels Ġon Ġa Ġselection Ġof Ġlabels Ġper Ġtype Ġand Ġnesting Ġthe Ġresulting Ġannotations Ġusing Ġa Ġlongest - span Ġsemantic Ġtype Ġhe uristic ĠB IB REF 25 Ġ. </s> Summary </s> ling 2015 design Ġshow Ġthat Ġthe Ġtask Ġof ĠN ER Ġis Ġnot Ġclearly Ġdefined Ġand Ġrather Ġdepends Ġon Ġa Ġspecific Ġproblem Ġcontext . ĠContrary , Ġmost ĠN ER Ġapproaches Ġare Ġspecifically Ġtrained Ġon Ġfixed Ġdatasets Ġin Ġa Ġbatch Ġmode . ĠWorse , Ġthey Ġoften Ġsuffer Ġfrom Ġpoor Ġrecall ĠB IB REF 26 Ġ. ĠIdeally , Ġone Ġcould Ġpersonal ize Ġthe Ġtask Ġof Ġrecognizing Ġnamed Ġentities , Ġconcepts Ġor Ġphrases Ġaccording Ġto Ġthe Ġspecific Ġproblem . ĠâĢ ľ Personal izing âĢ Ŀ Ġand Ġadapting Ġsuch Ġannot ators Ġshould Ġhappen Ġwith Ġvery Ġlimited Ġhuman Ġlabeling Ġeffort , Ġin Ġparticular Ġfor Ġidiosyncr atic Ġdomains Ġwith Ġsparse Ġtraining Ġdata . </s> Our Ġwork Ġfollows Ġthis Ġline . ĠFrom Ġour Ġresults Ġwe Ġreport ĠF 1 Ġscores Ġbetween Ġ84 âĢĵ 94 % Ġwhen Ġusing Ġbid irection al Ġmulti - lay ered ĠL ST Ms , Ġletter - tr ig ram Ġword Ġhashing Ġand Ġsurface Ġform Ġfeatures Ġon Ġonly Ġfew Ġhundred Ġtraining Ġexamples . </s> This Ġwork Ġis Ġonly Ġa Ġpreliminary Ġstep Ġtowards Ġthe Ġvision Ġof Ġpersonal izing Ġannotation Ġguidelines Ġfor ĠN ER ĠB IB REF 2 Ġ. ĠIn Ġour Ġfuture Ġwork , Ġwe Ġwill Ġfocus Ġon Ġadditional Ġimportant Ġidiosyncr atic Ġdomains , Ġsuch Ġas Ġhealth , Ġlife Ġscience , Ġfashion , Ġengineering Ġor Ġautomotive . ĠFor Ġthese Ġdomains , Ġwe Ġwill Ġconsider Ġthe Ġprocess Ġof Ġdetecting Ġmentions Ġand Ġlinking Ġthem Ġto Ġan Ġont ology Ġas Ġa Ġjoint Ġtask Ġand Ġwe Ġwill Ġinvestigate Ġsimple Ġand Ġinteractive Ġwork flows Ġfor Ġcreating Ġrobust Ġpersonalized Ġnamed Ġentity Ġlinking Ġsystems .