Stanford NER, spaCy 2.0 , recurrent model with a CRF top layer
EXTRACTIVE
Stanford NER, spaCy 2.0, recurrent model with a CRF top layer
EXTRACTIVE
In this section we describe a number of experiments targeted to compare the performance of popular named entity recognition algorithms on our data. We trained and evaluated Stanford NER, spaCy 2.0, and a recurrent model similar to BIBREF13 , BIBREF14 that uses bidirectional LSTM cells for character-based feature extraction and CRF, described in Guillaume Genthial's Sequence Tagging with Tensorflow blog post BIBREF15 . Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 . spaCy 2.0 uses a CNN-based transition system for named entity recognition. For each token, a Bloom embedding is calculated based on its lowercase form, prefix, suffix and shape, then using residual CNNs, a contextual representation of that token is extracted that potentially draws information from up to 4 tokens from each side BIBREF17 . Each update of the transition system's configuration is a classification task that uses the contextual representation of the top token on the stack, preceding and succeeding tokens, first two tokens of the buffer, and their leftmost, second leftmost, rightmost, second rightmost children. The valid transition with the highest score is applied to the system. This approach reportedly performs within 1% of the current state-of-the-art for English . In our experiments, we tried out 50-, 100-, 200- and 300-dimensional pre-trained GloVe embeddings. Due to time constraints, we did not tune the rest of hyperparameters and used their default values. The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines. The distinctive feature of this approach is the way contextual word embeddings are formed. For each token separately, to capture its word shape features, character-based representation is extracted using a bidirectional LSTM BIBREF18 . This representation gets concatenated with a distributional word vector such as GloVe, forming an intermediate word embedding. Using another bidirectional LSTM cell on these intermediate word embeddings, the contextual representation of tokens is obtained (Figure FIGREF17 ). Finally, a CRF layer labels the sequence of these contextual representations. In our experiments, we used Guillaume Genthial's implementation of the algorithm. We set the size of character-based biLSTM to 100 and the size of second biLSTM network to 300.
Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 . spaCy 2.0 uses a CNN-based transition system for named entity recognition. For each token, a Bloom embedding is calculated based on its lowercase form, prefix, suffix and shape, then using residual CNNs, a contextual representation of that token is extracted that potentially draws information from up to 4 tokens from each side BIBREF17 . Each update of the transition system's configuration is a classification task that uses the contextual representation of the top token on the stack, preceding and succeeding tokens, first two tokens of the buffer, and their leftmost, second leftmost, rightmost, second rightmost children. The valid transition with the highest score is applied to the system. This approach reportedly performs within 1% of the current state-of-the-art for English . In our experiments, we tried out 50-, 100-, 200- and 300-dimensional pre-trained GloVe embeddings. Due to time constraints, we did not tune the rest of hyperparameters and used their default values. The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines. The distinctive feature of this approach is the way contextual word embeddings are formed. For each token separately, to capture its word shape features, character-based representation is extracted using a bidirectional LSTM BIBREF18 . This representation gets concatenated with a distributional word vector such as GloVe, forming an intermediate word embedding. Using another bidirectional LSTM cell on these intermediate word embeddings, the contextual representation of tokens is obtained (Figure FIGREF17 ). Finally, a CRF layer labels the sequence of these contextual representations. In our experiments, we used Guillaume Genthial's implementation of the algorithm. We set the size of character-based biLSTM to 100 and the size of second biLSTM network to 300.
1810.08699
Introduction </s> N amed Ġentity Ġrecognition Ġis Ġan Ġimportant Ġtask Ġof Ġnatural Ġlanguage Ġprocessing , Ġfeaturing Ġin Ġmany Ġpopular Ġtext Ġprocessing Ġtool k its . ĠThis Ġarea Ġof Ġnatural Ġlanguage Ġprocessing Ġhas Ġbeen Ġactively Ġstudied Ġin Ġthe Ġlatest Ġdecades Ġand Ġthe Ġadvent Ġof Ġdeep Ġlearning Ġrein v igor ated Ġthe Ġresearch Ġon Ġmore Ġeffective Ġand Ġaccurate Ġmodels . ĠHowever , Ġmost Ġof Ġexisting Ġapproaches Ġrequire Ġlarge Ġannot ated Ġcorpor a . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġsuch Ġwork Ġhas Ġbeen Ġdone Ġfor Ġthe ĠArmenian Ġlanguage , Ġand Ġin Ġthis Ġwork Ġwe Ġaddress Ġseveral Ġproblems , Ġincluding Ġthe Ġcreation Ġof Ġa Ġcorpus Ġfor Ġtraining Ġmachine Ġlearning Ġmodels , Ġthe Ġdevelopment Ġof Ġgold - standard Ġtest Ġcorpus Ġand Ġevaluation Ġof Ġthe Ġeffectiveness Ġof Ġestablished Ġapproaches Ġfor Ġthe ĠArmenian Ġlanguage . </s> Considering Ġthe Ġcost Ġof Ġcreating Ġmanually Ġannot ated Ġnamed Ġentity Ġcorpus , Ġwe Ġfocused Ġon Ġalternative Ġapproaches . ĠLack Ġof Ġnamed Ġentity Ġcorpor a Ġis Ġa Ġcommon Ġproblem Ġfor Ġmany Ġlanguages , Ġthus Ġbringing Ġthe Ġattention Ġof Ġmany Ġresearchers Ġaround Ġthe Ġglobe . ĠProject ion Ġbased Ġtransfer Ġschemes Ġhave Ġbeen Ġshown Ġto Ġbe Ġvery Ġeffective Ġ( e . g . ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ), Ġusing Ġresource - rich Ġlanguage 's Ġcorpor a Ġto Ġgenerate Ġannot ated Ġdata Ġfor Ġthe Ġlow - resource Ġlanguage . ĠIn Ġthis Ġapproach , Ġthe Ġannotations Ġof Ġhigh - resource Ġlanguage Ġare Ġprojected Ġover Ġthe Ġcorresponding Ġtokens Ġof Ġthe Ġparallel Ġlow - resource Ġlanguage 's Ġtexts . ĠThis Ġstrategy Ġcan Ġbe Ġapplied Ġfor Ġlanguage Ġpairs Ġthat Ġhave Ġparallel Ġcorpor a . ĠHowever , Ġthat Ġapproach Ġwould Ġnot Ġwork Ġfor ĠArmenian Ġas Ġwe Ġdid Ġnot Ġhave Ġaccess Ġto Ġsufficiently Ġlarge Ġparallel Ġcorpus Ġwith Ġa Ġresource - rich Ġlanguage . </s> Another Ġpopular Ġapproach Ġis Ġusing ĠWikipedia . ĠKl est i ĠH ox ha Ġand ĠArt ur ĠB ax h aku Ġemploy Ġg az ette ers Ġextracted Ġfrom ĠWikipedia Ġto Ġgenerate Ġan Ġannot ated Ġcorpus Ġfor ĠAlban ian ĠB IB REF 3 Ġ, Ġand ĠWeber Ġand ĠP Ã¶ tz l Ġpropose Ġa Ġrule - based Ġsystem Ġfor ĠGerman Ġthat Ġlever ages Ġthe Ġinformation Ġfrom ĠWikipedia ĠB IB REF 4 Ġ. ĠHowever , Ġthe Ġlatter Ġrelies Ġon Ġexternal Ġtools Ġsuch Ġas Ġpart - of - speech Ġtag gers , Ġmaking Ġit Ġnon v iable Ġfor Ġthe ĠArmenian Ġlanguage . </s> N oth man Ġet Ġal . Ġgenerated Ġa Ġsilver - standard Ġcorpus Ġfor Ġ9 Ġlanguages Ġby Ġextracting ĠWikipedia Ġarticle Ġtexts Ġwith Ġoutgoing Ġlinks Ġand Ġturning Ġthose Ġlinks Ġinto Ġnamed Ġentity Ġannotations Ġbased Ġon Ġthe Ġtarget Ġarticle 's Ġtype ĠB IB REF 5 Ġ. ĠSy so ev Ġand ĠAnd rian ov Ġused Ġa Ġsimilar Ġapproach Ġfor Ġthe ĠRussian Ġlanguage ĠB IB REF 6 ĠB IB REF 7 Ġ. ĠBased Ġon Ġits Ġsuccess Ġfor Ġa Ġwide Ġrange Ġof Ġlanguages , Ġour Ġchoice Ġfell Ġon Ġthis Ġmodel Ġto Ġtackle Ġautomated Ġdata Ġgeneration Ġand Ġannotation Ġfor Ġthe ĠArmenian Ġlanguage . </s> Aside Ġfrom Ġthe Ġlack Ġof Ġtraining Ġdata , Ġwe Ġalso Ġaddress Ġthe Ġabsence Ġof Ġa Ġbenchmark Ġdataset Ġof ĠArmenian Ġtexts Ġfor Ġnamed Ġentity Ġrecognition . ĠWe Ġpropose Ġa Ġgold - standard Ġcorpus Ġwith Ġmanual Ġannotation Ġof ĠCo N LL Ġnamed Ġentity Ġcategories : Ġperson , Ġlocation , Ġand Ġorganization ĠB IB REF 8 ĠB IB REF 9 Ġ, Ġhoping Ġit Ġwill Ġbe Ġused Ġto Ġevaluate Ġfuture Ġnamed Ġentity Ġrecognition Ġmodels . </s> Furthermore , Ġpopular Ġentity Ġrecognition Ġmodels Ġwere Ġapplied Ġto Ġthe Ġmentioned Ġdata Ġin Ġorder Ġto Ġobtain Ġbaseline Ġresults Ġfor Ġfuture Ġresearch Ġin Ġthe Ġarea . ĠAlong Ġwith Ġthe Ġdatasets , Ġwe Ġdeveloped ĠGl o Ve ĠB IB REF 10 Ġword Ġembed d ings Ġto Ġtrain Ġand Ġevaluate Ġthe Ġdeep Ġlearning Ġmodels Ġin Ġour Ġexperiments . </s> The Ġcontributions Ġof Ġthis Ġwork Ġare Ġ( i ) Ġthe Ġsilver - standard Ġtraining Ġcorpus , Ġ( ii ) Ġthe Ġgold - standard Ġtest Ġcorpus , Ġ( iii ) ĠGl o Ve Ġword Ġembed d ings , Ġ( iv ) Ġbaseline Ġresults Ġfor Ġ3 Ġdifferent Ġmodels Ġon Ġthe Ġproposed Ġbenchmark Ġdata Ġset . ĠAll Ġaforementioned Ġresources Ġare Ġavailable Ġon ĠGitHub . </s> Autom ated Ġtraining Ġcorpus Ġgeneration </s> We Ġused ĠSy so ev Ġand ĠAnd rian ov 's Ġmodification Ġof Ġthe ĠN oth man Ġet Ġal . Ġapproach Ġto Ġautomatically Ġgenerate Ġdata Ġfor Ġtraining Ġa Ġnamed Ġentity Ġrecogn izer . ĠThis Ġapproach Ġuses Ġlinks Ġbetween ĠWikipedia Ġarticles Ġto Ġgenerate Ġsequences Ġof Ġnamed - entity Ġannot ated Ġtokens . </s> Dat as et Ġextraction </s> The Ġmain Ġsteps Ġof Ġthe Ġdataset Ġextraction Ġsystem Ġare Ġdescribed Ġin ĠFigure ĠFIG REF 3 Ġ. </s> First , Ġeach ĠWikipedia Ġarticle Ġis Ġassigned Ġa Ġnamed Ġentity Ġclass Ġ( e . g . Ġthe Ġarticle Ġ Õ Ķ Õ « Õ ´ Ġ Õ Ķ Õ ¡ Õ · Ö Ħ Õ ¡ Õ · Õ µ Õ ¡ Õ ¶ Ġ( Kim ĠKash k ash ian ) Ġis Ġclassified Ġas ĠPER Ġ( person ), Ġ Ô ± Õ ¦ Õ £ Õ ¥ Ö Ģ Õ « Ġ Õ ¬ Õ « Õ £ Õ ¡ ( League Ġof ĠNations ) Ġas ĠOR G Ġ( organ ization ), Ġ Õ į Õ « Ö Ģ Õ « Õ ¡ ( Syria ) Ġas ĠLOC Ġetc ). ĠOne Ġof Ġthe Ġcore Ġdifferences Ġbetween Ġour Ġapproach Ġand ĠN oth man 's Ġsystem Ġis Ġthat Ġwe Ġdo Ġnot Ġrely Ġon Ġmanual Ġclassification Ġof Ġarticles Ġand Ġdo Ġnot Ġuse Ġinter - language Ġlinks Ġto Ġproject Ġarticle Ġclass ifications Ġacross Ġlanguages . ĠInstead , Ġour Ġclassification Ġalgorithm Ġuses Ġonly Ġan Ġarticle 's ĠWik id ata Ġentry 's Ġfirst Ġinstance Ġof Ġlabel 's Ġparent Ġsubclass Ġof Ġlabels , Ġwhich Ġare , Ġincidentally , Ġlanguage Ġindependent Ġand Ġthus Ġcan Ġbe Ġused Ġfor Ġany Ġlanguage . </s> Then , Ġoutgoing Ġlinks Ġin Ġarticles Ġare Ġassigned Ġthe Ġarticle 's Ġtype Ġthey Ġare Ġleading Ġto . ĠSent ences Ġare Ġincluded Ġin Ġthe Ġtraining Ġcorpus Ġonly Ġif Ġthey Ġcontain Ġat Ġleast Ġone Ġnamed Ġentity Ġand Ġall Ġcontained Ġcapital ized Ġwords Ġhave Ġan Ġoutgoing Ġlink Ġto Ġan Ġarticle Ġof Ġknown Ġtype . ĠSince Ġin ĠWikipedia Ġarticles Ġonly Ġthe Ġfirst Ġmention Ġof Ġeach Ġentity Ġis Ġlinked , Ġthis Ġapproach Ġbecomes Ġvery Ġrestrictive Ġand Ġin Ġorder Ġto Ġinclude Ġmore Ġsentences , Ġadditional Ġlinks Ġare Ġinferred . ĠThis Ġis Ġaccomplished Ġby Ġcompiling Ġa Ġlist Ġof Ġcommon Ġaliases Ġfor Ġarticles Ġcorresponding Ġto Ġnamed Ġentities , Ġand Ġthen Ġfinding Ġtext Ġfragments Ġmatching Ġthose Ġaliases Ġto Ġassign Ġa Ġnamed Ġentity Ġlabel . ĠAn Ġarticle 's Ġaliases Ġinclude Ġits Ġtitle , Ġtitles Ġof Ġdis amb ig uation Ġpages Ġwith Ġthe Ġarticle , Ġand Ġtexts Ġof Ġlinks Ġleading Ġto Ġthe Ġarticle Ġ( e . g . Ġ Ô ¼ Õ ¥ Õ ¶ Õ « Õ ¶ Õ £ Ö Ģ Õ ¡ Õ ¤ Ġ( L ening rad ), Ġ Õ Ĭ Õ ¥ Õ ¿ Ö Ģ Õ ¸ Õ £ Ö Ģ Õ ¡ Õ ¤ Ġ( Pet ro grad ), Ġ Õ Ĭ Õ ¥ Õ ¿ Õ ¥ Ö Ģ Õ ¢ Õ ¸ Ö Ĥ Ö Ģ Õ £ Ġ( Peter burg ) Ġare Ġaliases Ġfor Ġ Õ į Õ ¡ Õ ¶ Õ ¯ Õ ¿ Ġ Õ Ĭ Õ ¥ Õ ¿ Õ ¥ Ö Ģ Õ ¢ Õ ¸ Ö Ĥ Ö Ģ Õ £ Ġ( Saint ĠPetersburg )). ĠThe Ġlist Ġof Ġaliases Ġis Ġcompiled Ġfor Ġall ĠPER , ĠOR G , ĠLOC Ġarticles . </s> After Ġthat , Ġlink Ġboundaries Ġare Ġadjusted Ġby Ġremoving Ġthe Ġlabels Ġfor Ġexpressions Ġin Ġparentheses , Ġthe Ġtext Ġafter Ġa Ġcomma , Ġand Ġin Ġsome Ġcases Ġbreaking Ġinto Ġseparate Ġnamed Ġentities Ġif Ġthe Ġlinked Ġtext Ġcontains Ġa Ġcomma . ĠFor Ġexample , Ġ[ LOC Ġ Ô ± Õ ¢ Õ ¸ Õ ¾ Õ µ Õ ¡ Õ ¶ Ġ( Ö Ħ Õ ¡ Õ ² Õ ¡ Ö Ħ )] Ġ( Ab ov yan Ġ( town )) Ġis Ġre worked Ġinto Ġ[ LOC Ġ Ô ± Õ ¢ Õ ¸ Õ ¾ Õ µ Õ ¡ Õ ¶ ] Ġ( Ö Ħ Õ ¡ Õ ² Õ ¡ Ö Ħ ). </s> Using ĠWik id ata Ġto Ġclassify ĠWikipedia </s> Instead Ġof Ġmanually Ġclass ifying ĠWikipedia Ġarticles Ġas Ġit Ġwas Ġdone Ġin ĠN oth man Ġet Ġal ., Ġwe Ġdeveloped Ġa Ġrule - based Ġclass ifier Ġthat Ġused Ġan Ġarticle 's ĠWik id ata Ġinstance Ġof Ġand Ġsubclass Ġof Ġattributes Ġto Ġfind Ġthe Ġcorresponding Ġnamed Ġentity Ġtype . </s> The Ġclassification Ġcould Ġbe Ġdone Ġusing Ġsolely Ġinstance Ġof Ġlabels , Ġbut Ġthese Ġlabels Ġare Ġunnecessarily Ġspecific Ġfor Ġthe Ġtask Ġand Ġbuilding Ġa Ġmapping Ġon Ġit Ġwould Ġrequire Ġa Ġmore Ġtime - consuming Ġand Ġmeticulous Ġwork . ĠTherefore , Ġwe Ġclassified Ġarticles Ġbased Ġon Ġtheir Ġfirst Ġinstance Ġof Ġattribute 's Ġsubclass Ġof Ġvalues . ĠTable ĠT AB REF 4 Ġdisplays Ġthe Ġmapping Ġbetween Ġthese Ġvalues Ġand Ġnamed Ġentity Ġtypes . ĠUsing Ġhigher - level Ġsubclass Ġof Ġvalues Ġwas Ġnot Ġan Ġoption Ġas Ġtheir Ġvalues Ġoften Ġwere Ġtoo Ġgeneral , Ġmaking Ġit Ġimpossible Ġto Ġderive Ġthe Ġcorrect Ġnamed Ġentity Ġcategory . </s> Gener ated Ġdata </s> Using Ġthe Ġalgorithm Ġdescribed Ġabove , Ġwe Ġgenerated Ġ7 455 Ġannot ated Ġsentences Ġwith Ġ163 247 Ġtokens Ġbased Ġon Ġ20 ĠFebruary Ġ2018 Ġdump Ġof ĠArmenian ĠWikipedia . </s> The Ġgenerated Ġdata Ġis Ġstill Ġsignificantly Ġsmaller Ġthan Ġthe Ġmanually Ġannot ated Ġcorpor a Ġfrom ĠCo N LL Ġ2002 Ġand Ġ2003 . ĠFor Ġcomparison , Ġthe Ġtrain Ġset Ġof ĠEnglish ĠCo N LL Ġ2003 Ġcorpus Ġcontains Ġ20 36 21 Ġtokens Ġand Ġthe ĠGerman Ġone Ġ20 69 31 , Ġwhile Ġthe ĠSpanish Ġand ĠDutch Ġcorpor a Ġfrom ĠCo N LL Ġ2002 Ġrespectively Ġ27 30 37 Ġand Ġ2 187 37 Ġlines . ĠThe Ġsmaller Ġsize Ġof Ġour Ġgenerated Ġdata Ġcan Ġbe Ġattributed Ġto Ġthe Ġstrict Ġselection Ġof Ġcandidate Ġsentences Ġas Ġwell Ġas Ġsimply Ġto Ġthe Ġrelatively Ġsmall Ġsize Ġof ĠArmenian ĠWikipedia . </s> The Ġaccuracy Ġof Ġannotation Ġin Ġthe Ġgenerated Ġcorpus Ġheavily Ġrelies Ġon Ġthe Ġquality Ġof Ġlinks Ġin ĠWikipedia Ġarticles . ĠDuring Ġgeneration , Ġwe Ġassumed Ġthat Ġfirst Ġmentions Ġof Ġall Ġnamed Ġentities Ġhave Ġan Ġoutgoing Ġlink Ġto Ġtheir Ġarticle , Ġhowever Ġthis Ġwas Ġnot Ġalways Ġthe Ġcase Ġin Ġactual Ġsource Ġdata Ġand Ġas Ġa Ġresult Ġthe Ġtrain Ġset Ġcontained Ġsentences Ġwhere Ġnot Ġall Ġnamed Ġentities Ġare Ġlabeled . ĠAn notation Ġinaccur acies Ġalso Ġstemmed Ġfrom Ġwrongly Ġassigned Ġlink Ġboundaries Ġ( for Ġexample , Ġin ĠWikipedia Ġarticle Ġ Ô ± Ö Ģ Õ © Õ ¸ Ö Ĥ Ö Ģ Ġ Õ Ī Ö Ĥ Õ ¥ Õ ¬ Õ ½ Õ ¬ Õ « Ġ Õ İ Õ ¥ Õ ¬ Õ « Õ ¶ Õ £ Õ © Õ ¸ Õ ¶ Ġ( Arthur ĠWell esley ) Ġthere Ġis Ġa Ġlink Ġto Ġthe ĠNapoleon Ġarticle Ġwith Ġthe Ġtext Ġ" Õ § Ġ Õ Ĩ Õ ¡ Õ º Õ ¸ Õ ¬ Õ ¥ Õ ¸ Õ ¶ Õ ¨ " Ġ(" Nap oleon Ġis "), Ġwhen Ġit Ġshould Ġbe Ġ" Õ Ĩ Õ ¡ Õ º Õ ¸ Õ ¬ Õ ¥ Õ ¸ Õ ¶ Õ ¨ " Ġ(" Nap oleon ") ). ĠAnother Ġkind Ġof Ġcommon Ġannotation Ġerrors Ġoccurred Ġwhen Ġa Ġnamed Ġentity Ġappeared Ġinside Ġa Ġlink Ġnot Ġtargeting Ġa ĠLOC , ĠOR G , Ġor ĠPER Ġarticle Ġ( e . g . Ġ" Ô ± Õ Ħ Õ Ĩ Ġ Õ ¶ Õ ¡ Õ Ń Õ ¡ Õ £ Õ ¡ Õ ° Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ¨ Õ ¶ Õ ¿ Ö Ģ Õ ¸ Ö Ĥ Õ © Õ µ Õ ¸ Ö Ĥ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Õ ¸ Ö Ĥ Õ ´ " Ġ(" USA Ġpresidential Ġelections ") Ġis Ġlinked Ġto Ġthe Ġarticle Ġ Ô ± Õ Ħ Õ Ĩ Ġ Õ ¶ Õ ¡ Õ Ń Õ ¡ Õ £ Õ ¡ Õ ° Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ¨ Õ ¶ Õ ¿ Ö Ģ Õ ¸ Ö Ĥ Õ © Õ µ Õ ¸ Ö Ĥ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Ġ2016 Ġ( United ĠStates Ġpresidential Ġelection , Ġ2016 ) Ġand Ġas Ġa Ġresult Ġ[ LOC Ġ Ô ± Õ Ħ Õ Ĩ ] Ġ( USA ) Ġis Ġlost ). </s> Test Ġdataset </s> In Ġorder Ġto Ġevaluate Ġthe Ġmodels Ġtrained Ġon Ġgenerated Ġdata , Ġwe Ġmanually Ġannot ated Ġa Ġnamed Ġentities Ġdataset Ġcomprising Ġ53 453 Ġtokens Ġand Ġ25 66 Ġsentences Ġselected Ġfrom Ġover Ġ250 Ġnews Ġtexts Ġfrom Ġil ur . am . ĠThis Ġdataset Ġis Ġcomparable Ġin Ġsize Ġwith Ġthe Ġtest Ġsets Ġof Ġother Ġlanguages Ġ( Table ĠT AB REF 10 Ġ). ĠIncluded Ġsentences Ġare Ġfrom Ġpolitical , Ġsports , Ġlocal Ġand Ġworld Ġnews Ġ( Fig ures ĠFIG REF 8 Ġ, ĠFIG REF 9 Ġ), Ġcovering Ġthe Ġperiod Ġbetween ĠAugust Ġ2012 Ġand ĠJuly Ġ2018 . ĠThe Ġdataset Ġprovides Ġannotations Ġfor Ġ3 Ġpopular Ġnamed Ġentity Ġclasses : Ġpeople Ġ( PER ), Ġorganizations Ġ( OR G ), Ġand Ġlocations Ġ( LOC ), Ġand Ġis Ġreleased Ġin ĠCo N LL 03 Ġformat Ġwith ĠI OB Ġtagging Ġscheme . ĠTokens Ġand Ġsentences Ġwere Ġsegment ed Ġaccording Ġto Ġthe ĠUD Ġstandards Ġfor Ġthe ĠArmenian Ġlanguage ĠB IB REF 11 Ġ. </s> During Ġannotation , Ġwe Ġgenerally Ġrelied Ġon Ġcategories Ġand Ġguidelines Ġassembled Ġby ĠBB N ĠTechnologies Ġfor ĠT REC Ġ2002 Ġquestion Ġanswering Ġtrack . ĠOnly Ġnamed Ġentities Ġcorresponding Ġto ĠBB N 's Ġperson Ġname Ġcategory Ġwere Ġtagged Ġas ĠPER . ĠThose Ġinclude Ġproper Ġnames Ġof Ġpeople , Ġincluding Ġfictional Ġpeople , Ġfirst Ġand Ġlast Ġnames , Ġfamily Ġnames , Ġunique Ġnick names . ĠSimilarly , Ġorganization Ġname Ġcategories , Ġincluding Ġcompany Ġnames , Ġgovernment Ġagencies , Ġeducational Ġand Ġacademic Ġinstitutions , Ġsports Ġclubs , Ġmusical Ġen semb les Ġand Ġother Ġgroups , Ġhospitals , Ġmuseums , Ġnewspaper Ġnames , Ġwere Ġmarked Ġas ĠOR G . ĠHowever , Ġunlike ĠBB N , Ġwe Ġdid Ġnot Ġmark Ġadject ival Ġforms Ġof Ġorganization Ġnames Ġas Ġnamed Ġentities . ĠBB N 's Ġg pe Ġname , Ġfacility Ġname , Ġlocation Ġname Ġcategories Ġwere Ġcombined Ġand Ġannot ated Ġas ĠLOC . </s> We Ġignored Ġentities Ġof Ġother Ġcategories Ġ( e . g . Ġworks Ġof Ġart , Ġlaw , Ġor Ġevents ), Ġincluding Ġthose Ġcases Ġwhere Ġan ĠOR G , ĠLOC Ġor ĠPER Ġentity Ġwas Ġinside Ġan Ġentity Ġof Ġextr aneous Ġtype Ġ( e . g . Ġ Õ Ģ Õ Ģ Ġ( RA ) Ġin Ġ Õ Ģ Õ Ģ Ġ Õ Ķ Ö Ģ Õ ¥ Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ķ Ö Ģ Õ ¥ Õ ¶ Õ ½ Õ £ Õ « Ö Ģ Ö Ħ Ġ( RA ĠCriminal ĠCode ) Ġwas Ġnot Ġannot ated Ġas ĠLOC ). </s> Qu otation Ġmarks Ġaround Ġa Ġnamed Ġentity Ġwere Ġnot Ġannot ated Ġunless Ġthose Ġquotations Ġwere Ġa Ġpart Ġof Ġthat Ġentity 's Ġfull Ġofficial Ġname Ġ( e . g . ĠÂ« Õ Ĩ Õ ¡ Õ « Ö Ģ Õ « Õ ¿ Ġ Õ £ Õ ¸ Ö Ģ Õ ® Õ ¡ Ö Ģ Õ ¡ Õ ¶ Â» Ġ Õ ĵ Ô ² Ô ¸ Ġ(" N air it ĠPlant " ĠCJ SC )). </s> Depending Ġon Ġcontext , Ġmet onyms Ġsuch Ġas Ġ Ô ¿ Ö Ģ Õ ¥ Õ ´ Õ ¬ Ġ( K remlin ), Ġ Ô ² Õ ¡ Õ ² Ö Ģ Õ ¡ Õ ´ Õ µ Õ ¡ Õ ¶ Ġ26 Ġ( B agh ram yan Ġ26 ) Ġwere Ġannot ated Ġas ĠOR G Ġwhen Ġreferring Ġto Ġrespective Ġgovernment Ġagencies . ĠLikewise , Ġcountry Ġor Ġcity Ġnames Ġwere Ġalso Ġtagged Ġas ĠOR G Ġwhen Ġreferring Ġto Ġsports Ġteams Ġrepresenting Ġthem . </s> Word Ġembed d ings </s> Apart Ġfrom Ġthe Ġdatasets , Ġwe Ġalso Ġdeveloped Ġword Ġembed d ings Ġfor Ġthe ĠArmenian Ġlanguage , Ġwhich Ġwe Ġused Ġin Ġour Ġexperiments Ġto Ġtrain Ġand Ġevaluate Ġnamed Ġentity Ġrecognition Ġalgorithms . ĠConsidering Ġtheir Ġability Ġto Ġcapture Ġsemantic Ġregular ities , Ġwe Ġused ĠGl o Ve Ġto Ġtrain Ġword Ġembed d ings . ĠWe Ġassembled Ġa Ġdataset Ġof ĠArmenian Ġtexts Ġcontaining Ġ79 Ġmillion Ġtokens Ġfrom Ġthe Ġarticles Ġof ĠArmenian ĠWikipedia , ĠThe ĠArmenian ĠSoviet ĠEncyclopedia , Ġa Ġsub cor p us Ġof ĠEastern ĠArmenian ĠNational ĠCorpus ĠB IB REF 12 Ġ, Ġover Ġa Ġdozen ĠArmenian Ġnews Ġwebsites Ġand Ġblogs . ĠIncluded Ġtexts Ġcovered Ġtopics Ġsuch Ġas Ġeconomics , Ġpolitics , Ġweather Ġforecast , ĠIT , Ġlaw , Ġsociety Ġand Ġpolitics , Ġcoming Ġfrom Ġnon - fiction Ġas Ġwell Ġas Ġfiction Ġgenres . </s> Similar Ġto Ġthe Ġoriginal Ġembed d ings Ġpublished Ġfor Ġthe ĠEnglish Ġlanguage , Ġwe Ġrelease Ġ50 -, Ġ100 -, Ġ200 - Ġand Ġ300 - dimensional Ġword Ġvectors Ġfor ĠArmenian Ġwith Ġa Ġvocabulary Ġsize Ġof Ġ400 000 . ĠBefore Ġtraining , Ġall Ġthe Ġwords Ġin Ġthe Ġdataset Ġwere Ġlower c ased . ĠFor Ġthe Ġfinal Ġmodels Ġwe Ġused Ġthe Ġfollowing Ġtraining Ġhyper param eters : Ġ15 Ġwindow Ġsize Ġand Ġ20 Ġtraining Ġepoch s . </s> Exper iments </s> In Ġthis Ġsection Ġwe Ġdescribe Ġa Ġnumber Ġof Ġexperiments Ġtargeted Ġto Ġcompare Ġthe Ġperformance Ġof Ġpopular Ġnamed Ġentity Ġrecognition Ġalgorithms Ġon Ġour Ġdata . ĠWe Ġtrained Ġand Ġevaluated ĠStanford ĠN ER , Ġspa Cy Ġ2 . 0 , Ġand Ġa Ġrecurrent Ġmodel Ġsimilar Ġto ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġthat Ġuses Ġbid irection al ĠL ST M Ġcells Ġfor Ġcharacter - based Ġfeature Ġextraction Ġand ĠCR F , Ġdescribed Ġin ĠGu illa ume ĠG enth ial 's ĠSequence ĠT agging Ġwith ĠT ensor flow Ġblog Ġpost ĠB IB REF 15 Ġ. </s> Mod els </s> Stan ford ĠN ER Ġis Ġconditional Ġrandom Ġfields Ġ( CR F ) Ġclass ifier Ġbased Ġon Ġlex ical Ġand Ġcontextual Ġfeatures Ġsuch Ġas Ġthe Ġcurrent Ġword , Ġcharacter - level Ġn - gram s Ġof Ġup Ġto Ġlength Ġ6 Ġat Ġits Ġbeginning Ġand Ġthe Ġend , Ġprevious Ġand Ġnext Ġwords , Ġword Ġshape Ġand Ġsequence Ġfeatures ĠB IB REF 16 Ġ. </s> sp a Cy Ġ2 . 0 Ġuses Ġa ĠCNN - based Ġtransition Ġsystem Ġfor Ġnamed Ġentity Ġrecognition . ĠFor Ġeach Ġtoken , Ġa ĠBloom Ġembed ding Ġis Ġcalculated Ġbased Ġon Ġits Ġlower case Ġform , Ġprefix , Ġsuffix Ġand Ġshape , Ġthen Ġusing Ġresidual ĠCNN s , Ġa Ġcontextual Ġrepresentation Ġof Ġthat Ġtoken Ġis Ġextracted Ġthat Ġpotentially Ġdraws Ġinformation Ġfrom Ġup Ġto Ġ4 Ġtokens Ġfrom Ġeach Ġside ĠB IB REF 17 Ġ. ĠEach Ġupdate Ġof Ġthe Ġtransition Ġsystem 's Ġconfiguration Ġis Ġa Ġclassification Ġtask Ġthat Ġuses Ġthe Ġcontextual Ġrepresentation Ġof Ġthe Ġtop Ġtoken Ġon Ġthe Ġstack , Ġpreceding Ġand Ġsucceeding Ġtokens , Ġfirst Ġtwo Ġtokens Ġof Ġthe Ġbuffer , Ġand Ġtheir Ġleft most , Ġsecond Ġleft most , Ġright most , Ġsecond Ġright most Ġchildren . ĠThe Ġvalid Ġtransition Ġwith Ġthe Ġhighest Ġscore Ġis Ġapplied Ġto Ġthe Ġsystem . ĠThis Ġapproach Ġreportedly Ġperforms Ġwithin Ġ1 % Ġof Ġthe Ġcurrent Ġstate - of - the - art Ġfor ĠEnglish Ġ. ĠIn Ġour Ġexperiments , Ġwe Ġtried Ġout Ġ50 -, Ġ100 -, Ġ200 - Ġand Ġ300 - dimensional Ġpre - trained ĠGl o Ve Ġembed d ings . ĠDue Ġto Ġtime Ġconstraints , Ġwe Ġdid Ġnot Ġtune Ġthe Ġrest Ġof Ġhyper param eters Ġand Ġused Ġtheir Ġdefault Ġvalues . </s> The Ġmain Ġmodel Ġthat Ġwe Ġfocused Ġon Ġwas Ġthe Ġrecurrent Ġmodel Ġwith Ġa ĠCR F Ġtop Ġlayer , Ġand Ġthe Ġabove - mentioned Ġmethods Ġserved Ġmostly Ġas Ġbas elines . ĠThe Ġdistinctive Ġfeature Ġof Ġthis Ġapproach Ġis Ġthe Ġway Ġcontextual Ġword Ġembed d ings Ġare Ġformed . ĠFor Ġeach Ġtoken Ġseparately , Ġto Ġcapture Ġits Ġword Ġshape Ġfeatures , Ġcharacter - based Ġrepresentation Ġis Ġextracted Ġusing Ġa Ġbid irection al ĠL ST M ĠB IB REF 18 Ġ. ĠThis Ġrepresentation Ġgets Ġconc aten ated Ġwith Ġa Ġdistribution al Ġword Ġvector Ġsuch Ġas ĠGl o Ve , Ġforming Ġan Ġintermediate Ġword Ġembed ding . ĠUsing Ġanother Ġbid irection al ĠL ST M Ġcell Ġon Ġthese Ġintermediate Ġword Ġembed d ings , Ġthe Ġcontextual Ġrepresentation Ġof Ġtokens Ġis Ġobtained Ġ( Figure ĠFIG REF 17 Ġ). ĠFinally , Ġa ĠCR F Ġlayer Ġlabels Ġthe Ġsequence Ġof Ġthese Ġcontextual Ġrepresentations . ĠIn Ġour Ġexperiments , Ġwe Ġused ĠGu illa ume ĠG enth ial 's Ġimplementation Ġof Ġthe Ġalgorithm . ĠWe Ġset Ġthe Ġsize Ġof Ġcharacter - based Ġbi L ST M Ġto Ġ100 Ġand Ġthe Ġsize Ġof Ġsecond Ġbi L ST M Ġnetwork Ġto Ġ300 . </s> E val uation </s> Exper iments Ġwere Ġcarried Ġout Ġusing ĠI OB Ġtagging Ġscheme , Ġwith Ġa Ġtotal Ġof Ġ7 Ġclass Ġtags : ĠO , ĠB - PER , ĠI - PER , ĠB - LOC , ĠI - LOC , ĠB - OR G , ĠI - OR G . </s> We Ġrandomly Ġselected Ġ80 % Ġof Ġgenerated Ġannot ated Ġsentences Ġfor Ġtraining Ġand Ġused Ġthe Ġother Ġ20 % Ġas Ġa Ġdevelopment Ġset . ĠThe Ġmodels Ġwith Ġthe Ġbest ĠF 1 Ġscore Ġon Ġthe Ġdevelopment Ġset Ġwere Ġtested Ġon Ġthe Ġmanually Ġannot ated Ġgold Ġdataset . </s> Discussion </s> Table ĠT AB REF 19 Ġshows Ġthe Ġaverage Ġscores Ġof Ġevaluated Ġmodels . ĠThe Ġhighest ĠF 1 Ġscore Ġwas Ġachieved Ġby Ġthe Ġrecurrent Ġmodel Ġusing Ġa Ġbatch Ġsize Ġof Ġ8 Ġand ĠAdam Ġoptim izer Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ0 . 001 . ĠUp dating Ġword Ġembed d ings Ġduring Ġtraining Ġalso Ġnoticeably Ġimproved Ġthe Ġperformance . ĠGl o Ve Ġword Ġvector Ġmodels Ġof Ġfour Ġdifferent Ġsizes Ġ( 50 , Ġ100 , Ġ200 , Ġand Ġ300 ) Ġwere Ġtested , Ġwith Ġvectors Ġof Ġsize Ġ50 Ġproducing Ġthe Ġbest Ġresults Ġ( Table ĠT AB REF 20 Ġ). </s> For Ġspa Cy Ġ2 . 0 Ġnamed Ġentity Ġrecogn izer , Ġthe Ġsame Ġword Ġembed ding Ġmodels Ġwere Ġtested . ĠHowever , Ġin Ġthis Ġcase Ġthe Ġperformance Ġof Ġ200 - dimensional Ġembed d ings Ġwas Ġhighest Ġ( Table ĠT AB REF 21 Ġ). ĠUn surprisingly , Ġboth Ġdeep Ġlearning Ġmodels Ġoutper formed Ġthe Ġfeature - based ĠStanford Ġrecogn izer Ġin Ġrecall , Ġthe Ġlatter Ġhowever Ġdemonstrated Ġnoticeably Ġhigher Ġprecision . </s> It Ġis Ġclear Ġthat Ġthe Ġdevelopment Ġset Ġof Ġautomatically Ġgenerated Ġexamples Ġwas Ġnot Ġan Ġideal Ġindicator Ġof Ġmodels ' Ġperformance Ġon Ġgold - standard Ġtest Ġset . ĠHigher Ġdevelopment Ġset Ġscores Ġoften Ġled Ġto Ġlower Ġtest Ġscores Ġas Ġseen Ġin Ġthe Ġevaluation Ġresults Ġfor Ġspa Cy Ġ2 . 0 Ġand ĠChar - bi L ST M + bi L ST M + CR F Ġ( T ables ĠT AB REF 21 Ġand ĠT AB REF 20 Ġ). ĠAnalysis Ġof Ġerrors Ġon Ġthe Ġdevelopment Ġset Ġrevealed Ġthat Ġmany Ġwere Ġcaused Ġby Ġthe Ġincom ple teness Ġof Ġannotations , Ġwhen Ġnamed Ġentity Ġrecogn izers Ġcorrectly Ġpredicted Ġentities Ġthat Ġwere Ġabsent Ġfrom Ġannotations Ġ( e . g . Ġ[ Ô ½ Õ į Õ Ģ Õ Ħ - Õ « ĠLOC ] Ġ( US SR 's ), Ġ[ Ô ´ Õ « Õ ¶ Õ ¡ Õ ´ Õ ¸ Õ ¶ ĠOR G ] Ġ( the _ D in amo ), Ġ[ Õ Ĭ Õ « Ö Ģ Õ ¥ Õ ¶ Õ ¥ Õ µ Õ ¡ Õ ¶ Ġ Õ © Õ ¥ Ö Ģ Õ ¡ Õ ¯ Õ ² Õ ¦ Õ ¸ Ö Ĥ ĠLOC ] Ġ( I ber ian ĠPeninsula 's ) Ġetc ). ĠSimilarly , Ġthe Ġrecogn izers Ġoften Ġcorrectly Ġignored Ġnon - ent ities Ġthat Ġare Ġincorrectly Ġlabeled Ġin Ġdata Ġ( e . g . Ġ[ Ö ħ Õ ½ Õ ´ Õ ¡ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Õ « ĠPER ], Ġ[ Õ ¯ Õ ¸ Õ ¶ Õ ½ Õ ¥ Ö Ģ Õ ¾ Õ ¡ Õ ¿ Õ ¸ Ö Ģ Õ « Õ ¡ Õ ¶ ĠOR G ] Ġetc ). </s> Generally , Ġtested Ġmodels Ġdemonstrated Ġrelatively Ġhigh Ġprecision Ġof Ġrecognizing Ġtokens Ġthat Ġstarted Ġnamed Ġentities , Ġbut Ġfailed Ġto Ġdo Ġso Ġwith Ġdescriptor Ġwords Ġfor Ġorganizations Ġand , Ġto Ġa Ġcertain Ġdegree , Ġlocations . ĠThe Ġconfusion Ġmatrix Ġfor Ġone Ġof Ġthe Ġtrained Ġrecurrent Ġmodels Ġillustrates Ġthat Ġdifference Ġ( Table ĠT AB REF 22 Ġ). ĠThis Ġcan Ġbe Ġpartly Ġattributed Ġto Ġthe Ġquality Ġof Ġgenerated Ġdata : Ġdescriptor Ġwords Ġare Ġsometimes Ġsuperflu ously Ġlabeled Ġ( e . g . Ġ[ Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ Õ « Ġ Õ ¿ Õ ¥ Õ ² Õ ¡ Õ ¢ Õ ¶ Õ « Õ ¯ Õ ¶ Õ ¥ Ö Ģ Õ ¨ ĠLOC ] Ġ( the Ġindigenous Ġpeople Ġof ĠHawaii )), Ġwhich Ġis Ġlikely Ġcaused Ġby Ġthe Ġinconsistent Ġstyle Ġof Ġlinking Ġin ĠArmenian ĠWikipedia Ġ( in Ġthe Ġarticle Ġ Ô ± Õ Ħ Õ Ĩ Ġ Õ ´ Õ · Õ ¡ Õ ¯ Õ ¸ Ö Ĥ Õ µ Õ © Ġ( C ulture Ġof Ġthe ĠUnited ĠStates ), Ġits Ġlinked Ġtext Ġfragment Ġ" Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ Õ « Ġ Õ ¿ Õ ¥ Õ ² Õ ¡ Õ ¢ Õ ¶ Õ « Õ ¯ Õ ¶ Õ ¥ Ö Ģ Õ ¨ " Ġ(" the Ġindigenous Ġpeople Ġof ĠHawaii ") Ġleads Ġto Ġthe Ġarticle Ġ Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ ( Haw aii )). </s> Conclusion </s> We Ġrelease Ġtwo Ġnamed - entity Ġannot ated Ġdatasets Ġfor Ġthe ĠArmenian Ġlanguage : Ġa Ġsilver - standard Ġcorpus Ġfor Ġtraining ĠN ER Ġmodels , Ġand Ġa Ġgold - standard Ġcorpus Ġfor Ġtesting . ĠIt Ġis Ġworth Ġto Ġunder line Ġthe Ġimportance Ġof Ġthe Ġlatter Ġcorpus , Ġas Ġwe Ġaim Ġit Ġto Ġserve Ġas Ġa Ġbenchmark Ġfor Ġfuture Ġnamed Ġentity Ġrecognition Ġsystems Ġdesigned Ġfor Ġthe ĠArmenian Ġlanguage . ĠAlong Ġwith Ġthe Ġcorpor a , Ġwe Ġpublish ĠGl o Ve Ġword Ġvector Ġmodels Ġtrained Ġon Ġa Ġcollection Ġof ĠArmenian Ġtexts . </s> Additionally , Ġto Ġestablish Ġthe Ġapplic ability Ġof ĠWikipedia - based Ġapproaches Ġfor Ġthe ĠArmenian Ġlanguage , Ġwe Ġprovide Ġevaluation Ġresults Ġfor Ġ3 Ġdifferent Ġnamed Ġentity Ġrecognition Ġsystems Ġtrained Ġand Ġtested Ġon Ġour Ġdatasets . ĠThe Ġresults Ġreinforce Ġthe Ġability Ġof Ġdeep Ġlearning Ġapproaches Ġin Ġachieving Ġrelatively Ġhigh Ġrecall Ġvalues Ġfor Ġthis Ġspecific Ġtask , Ġas Ġwell Ġas Ġthe Ġpower Ġof Ġusing Ġcharacter - ext racted Ġembed d ings Ġalongside Ġconventional Ġword Ġembed d ings . </s> There Ġare Ġseveral Ġavenues Ġof Ġfuture Ġwork . ĠSince ĠN oth man Ġet Ġal . Ġ2013 , Ġmore Ġefficient Ġmethods Ġof Ġexploiting ĠWikipedia Ġhave Ġbeen Ġproposed , Ġnamely ĠWi NER ĠB IB REF 19 Ġ, Ġwhich Ġcould Ġhelp Ġincrease Ġboth Ġthe Ġquantity Ġand Ġquality Ġof Ġthe Ġtraining Ġcorpus . ĠAnother Ġpotential Ġarea Ġof Ġwork Ġis Ġthe Ġfurther Ġenrichment Ġof Ġthe Ġbenchmark Ġtest Ġset Ġwith Ġadditional Ġannotation Ġof Ġother Ġclasses Ġsuch Ġas ĠM ISC Ġor Ġmore Ġfine - gr ained Ġtypes Ġ( e . g . ĠCITY , ĠCOUN TRY , ĠREG ION Ġetc Ġinstead Ġof ĠLOC ).
Introduction Named entity recognition is an important task of natural language processing, featuring in many popular text processing toolkits. This area of natural language processing has been actively studied in the latest decades and the advent of deep learning reinvigorated the research on more effective and accurate models. However, most of existing approaches require large annotated corpora. To the best of our knowledge, no such work has been done for the Armenian language, and in this work we address several problems, including the creation of a corpus for training machine learning models, the development of gold-standard test corpus and evaluation of the effectiveness of established approaches for the Armenian language. Considering the cost of creating manually annotated named entity corpus, we focused on alternative approaches. Lack of named entity corpora is a common problem for many languages, thus bringing the attention of many researchers around the globe. Projection based transfer schemes have been shown to be very effective (e.g. BIBREF0 , BIBREF1 , BIBREF2 ), using resource-rich language's corpora to generate annotated data for the low-resource language. In this approach, the annotations of high-resource language are projected over the corresponding tokens of the parallel low-resource language's texts. This strategy can be applied for language pairs that have parallel corpora. However, that approach would not work for Armenian as we did not have access to sufficiently large parallel corpus with a resource-rich language. Another popular approach is using Wikipedia. Klesti Hoxha and Artur Baxhaku employ gazetteers extracted from Wikipedia to generate an annotated corpus for Albanian BIBREF3 , and Weber and Pötzl propose a rule-based system for German that leverages the information from Wikipedia BIBREF4 . However, the latter relies on external tools such as part-of-speech taggers, making it nonviable for the Armenian language. Nothman et al. generated a silver-standard corpus for 9 languages by extracting Wikipedia article texts with outgoing links and turning those links into named entity annotations based on the target article's type BIBREF5 . Sysoev and Andrianov used a similar approach for the Russian language BIBREF6 BIBREF7 . Based on its success for a wide range of languages, our choice fell on this model to tackle automated data generation and annotation for the Armenian language. Aside from the lack of training data, we also address the absence of a benchmark dataset of Armenian texts for named entity recognition. We propose a gold-standard corpus with manual annotation of CoNLL named entity categories: person, location, and organization BIBREF8 BIBREF9 , hoping it will be used to evaluate future named entity recognition models. Furthermore, popular entity recognition models were applied to the mentioned data in order to obtain baseline results for future research in the area. Along with the datasets, we developed GloVe BIBREF10 word embeddings to train and evaluate the deep learning models in our experiments. The contributions of this work are (i) the silver-standard training corpus, (ii) the gold-standard test corpus, (iii) GloVe word embeddings, (iv) baseline results for 3 different models on the proposed benchmark data set. All aforementioned resources are available on GitHub. Automated training corpus generation We used Sysoev and Andrianov's modification of the Nothman et al. approach to automatically generate data for training a named entity recognizer. This approach uses links between Wikipedia articles to generate sequences of named-entity annotated tokens. Dataset extraction The main steps of the dataset extraction system are described in Figure FIGREF3 . First, each Wikipedia article is assigned a named entity class (e.g. the article Քիմ Քաշքաշյան (Kim Kashkashian) is classified as PER (person), Ազգերի լիգա(League of Nations) as ORG (organization), Սիրիա(Syria) as LOC etc). One of the core differences between our approach and Nothman's system is that we do not rely on manual classification of articles and do not use inter-language links to project article classifications across languages. Instead, our classification algorithm uses only an article's Wikidata entry's first instance of label's parent subclass of labels, which are, incidentally, language independent and thus can be used for any language. Then, outgoing links in articles are assigned the article's type they are leading to. Sentences are included in the training corpus only if they contain at least one named entity and all contained capitalized words have an outgoing link to an article of known type. Since in Wikipedia articles only the first mention of each entity is linked, this approach becomes very restrictive and in order to include more sentences, additional links are inferred. This is accomplished by compiling a list of common aliases for articles corresponding to named entities, and then finding text fragments matching those aliases to assign a named entity label. An article's aliases include its title, titles of disambiguation pages with the article, and texts of links leading to the article (e.g. Լենինգրադ (Leningrad), Պետրոգրադ (Petrograd), Պետերբուրգ (Peterburg) are aliases for Սանկտ Պետերբուրգ (Saint Petersburg)). The list of aliases is compiled for all PER, ORG, LOC articles. After that, link boundaries are adjusted by removing the labels for expressions in parentheses, the text after a comma, and in some cases breaking into separate named entities if the linked text contains a comma. For example, [LOC Աբովյան (քաղաք)] (Abovyan (town)) is reworked into [LOC Աբովյան] (քաղաք). Using Wikidata to classify Wikipedia Instead of manually classifying Wikipedia articles as it was done in Nothman et al., we developed a rule-based classifier that used an article's Wikidata instance of and subclass of attributes to find the corresponding named entity type. The classification could be done using solely instance of labels, but these labels are unnecessarily specific for the task and building a mapping on it would require a more time-consuming and meticulous work. Therefore, we classified articles based on their first instance of attribute's subclass of values. Table TABREF4 displays the mapping between these values and named entity types. Using higher-level subclass of values was not an option as their values often were too general, making it impossible to derive the correct named entity category. Generated data Using the algorithm described above, we generated 7455 annotated sentences with 163247 tokens based on 20 February 2018 dump of Armenian Wikipedia. The generated data is still significantly smaller than the manually annotated corpora from CoNLL 2002 and 2003. For comparison, the train set of English CoNLL 2003 corpus contains 203621 tokens and the German one 206931, while the Spanish and Dutch corpora from CoNLL 2002 respectively 273037 and 218737 lines. The smaller size of our generated data can be attributed to the strict selection of candidate sentences as well as simply to the relatively small size of Armenian Wikipedia. The accuracy of annotation in the generated corpus heavily relies on the quality of links in Wikipedia articles. During generation, we assumed that first mentions of all named entities have an outgoing link to their article, however this was not always the case in actual source data and as a result the train set contained sentences where not all named entities are labeled. Annotation inaccuracies also stemmed from wrongly assigned link boundaries (for example, in Wikipedia article Արթուր Ուելսլի Վելինգթոն (Arthur Wellesley) there is a link to the Napoleon article with the text "է Նապոլեոնը" ("Napoleon is"), when it should be "Նապոլեոնը" ("Napoleon")). Another kind of common annotation errors occurred when a named entity appeared inside a link not targeting a LOC, ORG, or PER article (e.g. "ԱՄՆ նախագահական ընտրություններում" ("USA presidential elections") is linked to the article ԱՄՆ նախագահական ընտրություններ 2016 (United States presidential election, 2016) and as a result [LOC ԱՄՆ] (USA) is lost). Test dataset In order to evaluate the models trained on generated data, we manually annotated a named entities dataset comprising 53453 tokens and 2566 sentences selected from over 250 news texts from ilur.am. This dataset is comparable in size with the test sets of other languages (Table TABREF10 ). Included sentences are from political, sports, local and world news (Figures FIGREF8 , FIGREF9 ), covering the period between August 2012 and July 2018. The dataset provides annotations for 3 popular named entity classes: people (PER), organizations (ORG), and locations (LOC), and is released in CoNLL03 format with IOB tagging scheme. Tokens and sentences were segmented according to the UD standards for the Armenian language BIBREF11 . During annotation, we generally relied on categories and guidelines assembled by BBN Technologies for TREC 2002 question answering track. Only named entities corresponding to BBN's person name category were tagged as PER. Those include proper names of people, including fictional people, first and last names, family names, unique nicknames. Similarly, organization name categories, including company names, government agencies, educational and academic institutions, sports clubs, musical ensembles and other groups, hospitals, museums, newspaper names, were marked as ORG. However, unlike BBN, we did not mark adjectival forms of organization names as named entities. BBN's gpe name, facility name, location name categories were combined and annotated as LOC. We ignored entities of other categories (e.g. works of art, law, or events), including those cases where an ORG, LOC or PER entity was inside an entity of extraneous type (e.g. ՀՀ (RA) in ՀՀ Քրեական Օրենսգիրք (RA Criminal Code) was not annotated as LOC). Quotation marks around a named entity were not annotated unless those quotations were a part of that entity's full official name (e.g. «Նաիրիտ գործարան» ՓԲԸ ("Nairit Plant" CJSC)). Depending on context, metonyms such as Կրեմլ (Kremlin), Բաղրամյան 26 (Baghramyan 26) were annotated as ORG when referring to respective government agencies. Likewise, country or city names were also tagged as ORG when referring to sports teams representing them. Word embeddings Apart from the datasets, we also developed word embeddings for the Armenian language, which we used in our experiments to train and evaluate named entity recognition algorithms. Considering their ability to capture semantic regularities, we used GloVe to train word embeddings. We assembled a dataset of Armenian texts containing 79 million tokens from the articles of Armenian Wikipedia, The Armenian Soviet Encyclopedia, a subcorpus of Eastern Armenian National Corpus BIBREF12 , over a dozen Armenian news websites and blogs. Included texts covered topics such as economics, politics, weather forecast, IT, law, society and politics, coming from non-fiction as well as fiction genres. Similar to the original embeddings published for the English language, we release 50-, 100-, 200- and 300-dimensional word vectors for Armenian with a vocabulary size of 400000. Before training, all the words in the dataset were lowercased. For the final models we used the following training hyperparameters: 15 window size and 20 training epochs. Experiments In this section we describe a number of experiments targeted to compare the performance of popular named entity recognition algorithms on our data. We trained and evaluated Stanford NER, spaCy 2.0, and a recurrent model similar to BIBREF13 , BIBREF14 that uses bidirectional LSTM cells for character-based feature extraction and CRF, described in Guillaume Genthial's Sequence Tagging with Tensorflow blog post BIBREF15 . Models Stanford NER is conditional random fields (CRF) classifier based on lexical and contextual features such as the current word, character-level n-grams of up to length 6 at its beginning and the end, previous and next words, word shape and sequence features BIBREF16 . spaCy 2.0 uses a CNN-based transition system for named entity recognition. For each token, a Bloom embedding is calculated based on its lowercase form, prefix, suffix and shape, then using residual CNNs, a contextual representation of that token is extracted that potentially draws information from up to 4 tokens from each side BIBREF17 . Each update of the transition system's configuration is a classification task that uses the contextual representation of the top token on the stack, preceding and succeeding tokens, first two tokens of the buffer, and their leftmost, second leftmost, rightmost, second rightmost children. The valid transition with the highest score is applied to the system. This approach reportedly performs within 1% of the current state-of-the-art for English . In our experiments, we tried out 50-, 100-, 200- and 300-dimensional pre-trained GloVe embeddings. Due to time constraints, we did not tune the rest of hyperparameters and used their default values. The main model that we focused on was the recurrent model with a CRF top layer, and the above-mentioned methods served mostly as baselines. The distinctive feature of this approach is the way contextual word embeddings are formed. For each token separately, to capture its word shape features, character-based representation is extracted using a bidirectional LSTM BIBREF18 . This representation gets concatenated with a distributional word vector such as GloVe, forming an intermediate word embedding. Using another bidirectional LSTM cell on these intermediate word embeddings, the contextual representation of tokens is obtained (Figure FIGREF17 ). Finally, a CRF layer labels the sequence of these contextual representations. In our experiments, we used Guillaume Genthial's implementation of the algorithm. We set the size of character-based biLSTM to 100 and the size of second biLSTM network to 300. Evaluation Experiments were carried out using IOB tagging scheme, with a total of 7 class tags: O, B-PER, I-PER, B-LOC, I-LOC, B-ORG, I-ORG. We randomly selected 80% of generated annotated sentences for training and used the other 20% as a development set. The models with the best F1 score on the development set were tested on the manually annotated gold dataset. Discussion Table TABREF19 shows the average scores of evaluated models. The highest F1 score was achieved by the recurrent model using a batch size of 8 and Adam optimizer with an initial learning rate of 0.001. Updating word embeddings during training also noticeably improved the performance. GloVe word vector models of four different sizes (50, 100, 200, and 300) were tested, with vectors of size 50 producing the best results (Table TABREF20 ). For spaCy 2.0 named entity recognizer, the same word embedding models were tested. However, in this case the performance of 200-dimensional embeddings was highest (Table TABREF21 ). Unsurprisingly, both deep learning models outperformed the feature-based Stanford recognizer in recall, the latter however demonstrated noticeably higher precision. It is clear that the development set of automatically generated examples was not an ideal indicator of models' performance on gold-standard test set. Higher development set scores often led to lower test scores as seen in the evaluation results for spaCy 2.0 and Char-biLSTM+biLSTM+CRF (Tables TABREF21 and TABREF20 ). Analysis of errors on the development set revealed that many were caused by the incompleteness of annotations, when named entity recognizers correctly predicted entities that were absent from annotations (e.g. [ԽՍՀՄ-ի LOC] (USSR's), [Դինամոն ORG] (the_Dinamo), [Պիրենեյան թերակղզու LOC] (Iberian Peninsula's) etc). Similarly, the recognizers often correctly ignored non-entities that are incorrectly labeled in data (e.g. [օսմանների PER], [կոնսերվատորիան ORG] etc). Generally, tested models demonstrated relatively high precision of recognizing tokens that started named entities, but failed to do so with descriptor words for organizations and, to a certain degree, locations. The confusion matrix for one of the trained recurrent models illustrates that difference (Table TABREF22 ). This can be partly attributed to the quality of generated data: descriptor words are sometimes superfluously labeled (e.g. [Հավայան կղզիների տեղաբնիկները LOC] (the indigenous people of Hawaii)), which is likely caused by the inconsistent style of linking in Armenian Wikipedia (in the article ԱՄՆ մշակույթ (Culture of the United States), its linked text fragment "Հավայան կղզիների տեղաբնիկները" ("the indigenous people of Hawaii") leads to the article Հավայան կղզիներ(Hawaii)). Conclusion We release two named-entity annotated datasets for the Armenian language: a silver-standard corpus for training NER models, and a gold-standard corpus for testing. It is worth to underline the importance of the latter corpus, as we aim it to serve as a benchmark for future named entity recognition systems designed for the Armenian language. Along with the corpora, we publish GloVe word vector models trained on a collection of Armenian texts. Additionally, to establish the applicability of Wikipedia-based approaches for the Armenian language, we provide evaluation results for 3 different named entity recognition systems trained and tested on our datasets. The results reinforce the ability of deep learning approaches in achieving relatively high recall values for this specific task, as well as the power of using character-extracted embeddings alongside conventional word embeddings. There are several avenues of future work. Since Nothman et al. 2013, more efficient methods of exploiting Wikipedia have been proposed, namely WiNER BIBREF19 , which could help increase both the quantity and quality of the training corpus. Another potential area of work is the further enrichment of the benchmark test set with additional annotation of other classes such as MISC or more fine-grained types (e.g. CITY, COUNTRY, REGION etc instead of LOC).
what ner models were evaluated?
18c5d366b1da8447b5404eab71f4cc658ba12e6f
what Ġner Ġmodels Ġwere Ġevaluated ?
Stan ford ĠN ER , Ġspa Cy Ġ2 . 0 Ġ, Ġrecurrent Ġmodel Ġwith Ġa ĠCR F Ġtop Ġlayer
<s> what Ġner Ġmodels Ġwere Ġevaluated ? </s> Introduction </s> N amed Ġentity Ġrecognition Ġis Ġan Ġimportant Ġtask Ġof Ġnatural Ġlanguage Ġprocessing , Ġfeaturing Ġin Ġmany Ġpopular Ġtext Ġprocessing Ġtool k its . ĠThis Ġarea Ġof Ġnatural Ġlanguage Ġprocessing Ġhas Ġbeen Ġactively Ġstudied Ġin Ġthe Ġlatest Ġdecades Ġand Ġthe Ġadvent Ġof Ġdeep Ġlearning Ġrein v igor ated Ġthe Ġresearch Ġon Ġmore Ġeffective Ġand Ġaccurate Ġmodels . ĠHowever , Ġmost Ġof Ġexisting Ġapproaches Ġrequire Ġlarge Ġannot ated Ġcorpor a . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġsuch Ġwork Ġhas Ġbeen Ġdone Ġfor Ġthe ĠArmenian Ġlanguage , Ġand Ġin Ġthis Ġwork Ġwe Ġaddress Ġseveral Ġproblems , Ġincluding Ġthe Ġcreation Ġof Ġa Ġcorpus Ġfor Ġtraining Ġmachine Ġlearning Ġmodels , Ġthe Ġdevelopment Ġof Ġgold - standard Ġtest Ġcorpus Ġand Ġevaluation Ġof Ġthe Ġeffectiveness Ġof Ġestablished Ġapproaches Ġfor Ġthe ĠArmenian Ġlanguage . </s> Considering Ġthe Ġcost Ġof Ġcreating Ġmanually Ġannot ated Ġnamed Ġentity Ġcorpus , Ġwe Ġfocused Ġon Ġalternative Ġapproaches . ĠLack Ġof Ġnamed Ġentity Ġcorpor a Ġis Ġa Ġcommon Ġproblem Ġfor Ġmany Ġlanguages , Ġthus Ġbringing Ġthe Ġattention Ġof Ġmany Ġresearchers Ġaround Ġthe Ġglobe . ĠProject ion Ġbased Ġtransfer Ġschemes Ġhave Ġbeen Ġshown Ġto Ġbe Ġvery Ġeffective Ġ( e . g . ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ), Ġusing Ġresource - rich Ġlanguage 's Ġcorpor a Ġto Ġgenerate Ġannot ated Ġdata Ġfor Ġthe Ġlow - resource Ġlanguage . ĠIn Ġthis Ġapproach , Ġthe Ġannotations Ġof Ġhigh - resource Ġlanguage Ġare Ġprojected Ġover Ġthe Ġcorresponding Ġtokens Ġof Ġthe Ġparallel Ġlow - resource Ġlanguage 's Ġtexts . ĠThis Ġstrategy Ġcan Ġbe Ġapplied Ġfor Ġlanguage Ġpairs Ġthat Ġhave Ġparallel Ġcorpor a . ĠHowever , Ġthat Ġapproach Ġwould Ġnot Ġwork Ġfor ĠArmenian Ġas Ġwe Ġdid Ġnot Ġhave Ġaccess Ġto Ġsufficiently Ġlarge Ġparallel Ġcorpus Ġwith Ġa Ġresource - rich Ġlanguage . </s> Another Ġpopular Ġapproach Ġis Ġusing ĠWikipedia . ĠKl est i ĠH ox ha Ġand ĠArt ur ĠB ax h aku Ġemploy Ġg az ette ers Ġextracted Ġfrom ĠWikipedia Ġto Ġgenerate Ġan Ġannot ated Ġcorpus Ġfor ĠAlban ian ĠB IB REF 3 Ġ, Ġand ĠWeber Ġand ĠP Ã¶ tz l Ġpropose Ġa Ġrule - based Ġsystem Ġfor ĠGerman Ġthat Ġlever ages Ġthe Ġinformation Ġfrom ĠWikipedia ĠB IB REF 4 Ġ. ĠHowever , Ġthe Ġlatter Ġrelies Ġon Ġexternal Ġtools Ġsuch Ġas Ġpart - of - speech Ġtag gers , Ġmaking Ġit Ġnon v iable Ġfor Ġthe ĠArmenian Ġlanguage . </s> N oth man Ġet Ġal . Ġgenerated Ġa Ġsilver - standard Ġcorpus Ġfor Ġ9 Ġlanguages Ġby Ġextracting ĠWikipedia Ġarticle Ġtexts Ġwith Ġoutgoing Ġlinks Ġand Ġturning Ġthose Ġlinks Ġinto Ġnamed Ġentity Ġannotations Ġbased Ġon Ġthe Ġtarget Ġarticle 's Ġtype ĠB IB REF 5 Ġ. ĠSy so ev Ġand ĠAnd rian ov Ġused Ġa Ġsimilar Ġapproach Ġfor Ġthe ĠRussian Ġlanguage ĠB IB REF 6 ĠB IB REF 7 Ġ. ĠBased Ġon Ġits Ġsuccess Ġfor Ġa Ġwide Ġrange Ġof Ġlanguages , Ġour Ġchoice Ġfell Ġon Ġthis Ġmodel Ġto Ġtackle Ġautomated Ġdata Ġgeneration Ġand Ġannotation Ġfor Ġthe ĠArmenian Ġlanguage . </s> Aside Ġfrom Ġthe Ġlack Ġof Ġtraining Ġdata , Ġwe Ġalso Ġaddress Ġthe Ġabsence Ġof Ġa Ġbenchmark Ġdataset Ġof ĠArmenian Ġtexts Ġfor Ġnamed Ġentity Ġrecognition . ĠWe Ġpropose Ġa Ġgold - standard Ġcorpus Ġwith Ġmanual Ġannotation Ġof ĠCo N LL Ġnamed Ġentity Ġcategories : Ġperson , Ġlocation , Ġand Ġorganization ĠB IB REF 8 ĠB IB REF 9 Ġ, Ġhoping Ġit Ġwill Ġbe Ġused Ġto Ġevaluate Ġfuture Ġnamed Ġentity Ġrecognition Ġmodels . </s> Furthermore , Ġpopular Ġentity Ġrecognition Ġmodels Ġwere Ġapplied Ġto Ġthe Ġmentioned Ġdata Ġin Ġorder Ġto Ġobtain Ġbaseline Ġresults Ġfor Ġfuture Ġresearch Ġin Ġthe Ġarea . ĠAlong Ġwith Ġthe Ġdatasets , Ġwe Ġdeveloped ĠGl o Ve ĠB IB REF 10 Ġword Ġembed d ings Ġto Ġtrain Ġand Ġevaluate Ġthe Ġdeep Ġlearning Ġmodels Ġin Ġour Ġexperiments . </s> The Ġcontributions Ġof Ġthis Ġwork Ġare Ġ( i ) Ġthe Ġsilver - standard Ġtraining Ġcorpus , Ġ( ii ) Ġthe Ġgold - standard Ġtest Ġcorpus , Ġ( iii ) ĠGl o Ve Ġword Ġembed d ings , Ġ( iv ) Ġbaseline Ġresults Ġfor Ġ3 Ġdifferent Ġmodels Ġon Ġthe Ġproposed Ġbenchmark Ġdata Ġset . ĠAll Ġaforementioned Ġresources Ġare Ġavailable Ġon ĠGitHub . </s> Autom ated Ġtraining Ġcorpus Ġgeneration </s> We Ġused ĠSy so ev Ġand ĠAnd rian ov 's Ġmodification Ġof Ġthe ĠN oth man Ġet Ġal . Ġapproach Ġto Ġautomatically Ġgenerate Ġdata Ġfor Ġtraining Ġa Ġnamed Ġentity Ġrecogn izer . ĠThis Ġapproach Ġuses Ġlinks Ġbetween ĠWikipedia Ġarticles Ġto Ġgenerate Ġsequences Ġof Ġnamed - entity Ġannot ated Ġtokens . </s> Dat as et Ġextraction </s> The Ġmain Ġsteps Ġof Ġthe Ġdataset Ġextraction Ġsystem Ġare Ġdescribed Ġin ĠFigure ĠFIG REF 3 Ġ. </s> First , Ġeach ĠWikipedia Ġarticle Ġis Ġassigned Ġa Ġnamed Ġentity Ġclass Ġ( e . g . Ġthe Ġarticle Ġ Õ Ķ Õ « Õ ´ Ġ Õ Ķ Õ ¡ Õ · Ö Ħ Õ ¡ Õ · Õ µ Õ ¡ Õ ¶ Ġ( Kim ĠKash k ash ian ) Ġis Ġclassified Ġas ĠPER Ġ( person ), Ġ Ô ± Õ ¦ Õ £ Õ ¥ Ö Ģ Õ « Ġ Õ ¬ Õ « Õ £ Õ ¡ ( League Ġof ĠNations ) Ġas ĠOR G Ġ( organ ization ), Ġ Õ į Õ « Ö Ģ Õ « Õ ¡ ( Syria ) Ġas ĠLOC Ġetc ). ĠOne Ġof Ġthe Ġcore Ġdifferences Ġbetween Ġour Ġapproach Ġand ĠN oth man 's Ġsystem Ġis Ġthat Ġwe Ġdo Ġnot Ġrely Ġon Ġmanual Ġclassification Ġof Ġarticles Ġand Ġdo Ġnot Ġuse Ġinter - language Ġlinks Ġto Ġproject Ġarticle Ġclass ifications Ġacross Ġlanguages . ĠInstead , Ġour Ġclassification Ġalgorithm Ġuses Ġonly Ġan Ġarticle 's ĠWik id ata Ġentry 's Ġfirst Ġinstance Ġof Ġlabel 's Ġparent Ġsubclass Ġof Ġlabels , Ġwhich Ġare , Ġincidentally , Ġlanguage Ġindependent Ġand Ġthus Ġcan Ġbe Ġused Ġfor Ġany Ġlanguage . </s> Then , Ġoutgoing Ġlinks Ġin Ġarticles Ġare Ġassigned Ġthe Ġarticle 's Ġtype Ġthey Ġare Ġleading Ġto . ĠSent ences Ġare Ġincluded Ġin Ġthe Ġtraining Ġcorpus Ġonly Ġif Ġthey Ġcontain Ġat Ġleast Ġone Ġnamed Ġentity Ġand Ġall Ġcontained Ġcapital ized Ġwords Ġhave Ġan Ġoutgoing Ġlink Ġto Ġan Ġarticle Ġof Ġknown Ġtype . ĠSince Ġin ĠWikipedia Ġarticles Ġonly Ġthe Ġfirst Ġmention Ġof Ġeach Ġentity Ġis Ġlinked , Ġthis Ġapproach Ġbecomes Ġvery Ġrestrictive Ġand Ġin Ġorder Ġto Ġinclude Ġmore Ġsentences , Ġadditional Ġlinks Ġare Ġinferred . ĠThis Ġis Ġaccomplished Ġby Ġcompiling Ġa Ġlist Ġof Ġcommon Ġaliases Ġfor Ġarticles Ġcorresponding Ġto Ġnamed Ġentities , Ġand Ġthen Ġfinding Ġtext Ġfragments Ġmatching Ġthose Ġaliases Ġto Ġassign Ġa Ġnamed Ġentity Ġlabel . ĠAn Ġarticle 's Ġaliases Ġinclude Ġits Ġtitle , Ġtitles Ġof Ġdis amb ig uation Ġpages Ġwith Ġthe Ġarticle , Ġand Ġtexts Ġof Ġlinks Ġleading Ġto Ġthe Ġarticle Ġ( e . g . Ġ Ô ¼ Õ ¥ Õ ¶ Õ « Õ ¶ Õ £ Ö Ģ Õ ¡ Õ ¤ Ġ( L ening rad ), Ġ Õ Ĭ Õ ¥ Õ ¿ Ö Ģ Õ ¸ Õ £ Ö Ģ Õ ¡ Õ ¤ Ġ( Pet ro grad ), Ġ Õ Ĭ Õ ¥ Õ ¿ Õ ¥ Ö Ģ Õ ¢ Õ ¸ Ö Ĥ Ö Ģ Õ £ Ġ( Peter burg ) Ġare Ġaliases Ġfor Ġ Õ į Õ ¡ Õ ¶ Õ ¯ Õ ¿ Ġ Õ Ĭ Õ ¥ Õ ¿ Õ ¥ Ö Ģ Õ ¢ Õ ¸ Ö Ĥ Ö Ģ Õ £ Ġ( Saint ĠPetersburg )). ĠThe Ġlist Ġof Ġaliases Ġis Ġcompiled Ġfor Ġall ĠPER , ĠOR G , ĠLOC Ġarticles . </s> After Ġthat , Ġlink Ġboundaries Ġare Ġadjusted Ġby Ġremoving Ġthe Ġlabels Ġfor Ġexpressions Ġin Ġparentheses , Ġthe Ġtext Ġafter Ġa Ġcomma , Ġand Ġin Ġsome Ġcases Ġbreaking Ġinto Ġseparate Ġnamed Ġentities Ġif Ġthe Ġlinked Ġtext Ġcontains Ġa Ġcomma . ĠFor Ġexample , Ġ[ LOC Ġ Ô ± Õ ¢ Õ ¸ Õ ¾ Õ µ Õ ¡ Õ ¶ Ġ( Ö Ħ Õ ¡ Õ ² Õ ¡ Ö Ħ )] Ġ( Ab ov yan Ġ( town )) Ġis Ġre worked Ġinto Ġ[ LOC Ġ Ô ± Õ ¢ Õ ¸ Õ ¾ Õ µ Õ ¡ Õ ¶ ] Ġ( Ö Ħ Õ ¡ Õ ² Õ ¡ Ö Ħ ). </s> Using ĠWik id ata Ġto Ġclassify ĠWikipedia </s> Instead Ġof Ġmanually Ġclass ifying ĠWikipedia Ġarticles Ġas Ġit Ġwas Ġdone Ġin ĠN oth man Ġet Ġal ., Ġwe Ġdeveloped Ġa Ġrule - based Ġclass ifier Ġthat Ġused Ġan Ġarticle 's ĠWik id ata Ġinstance Ġof Ġand Ġsubclass Ġof Ġattributes Ġto Ġfind Ġthe Ġcorresponding Ġnamed Ġentity Ġtype . </s> The Ġclassification Ġcould Ġbe Ġdone Ġusing Ġsolely Ġinstance Ġof Ġlabels , Ġbut Ġthese Ġlabels Ġare Ġunnecessarily Ġspecific Ġfor Ġthe Ġtask Ġand Ġbuilding Ġa Ġmapping Ġon Ġit Ġwould Ġrequire Ġa Ġmore Ġtime - consuming Ġand Ġmeticulous Ġwork . ĠTherefore , Ġwe Ġclassified Ġarticles Ġbased Ġon Ġtheir Ġfirst Ġinstance Ġof Ġattribute 's Ġsubclass Ġof Ġvalues . ĠTable ĠT AB REF 4 Ġdisplays Ġthe Ġmapping Ġbetween Ġthese Ġvalues Ġand Ġnamed Ġentity Ġtypes . ĠUsing Ġhigher - level Ġsubclass Ġof Ġvalues Ġwas Ġnot Ġan Ġoption Ġas Ġtheir Ġvalues Ġoften Ġwere Ġtoo Ġgeneral , Ġmaking Ġit Ġimpossible Ġto Ġderive Ġthe Ġcorrect Ġnamed Ġentity Ġcategory . </s> Gener ated Ġdata </s> Using Ġthe Ġalgorithm Ġdescribed Ġabove , Ġwe Ġgenerated Ġ7 455 Ġannot ated Ġsentences Ġwith Ġ163 247 Ġtokens Ġbased Ġon Ġ20 ĠFebruary Ġ2018 Ġdump Ġof ĠArmenian ĠWikipedia . </s> The Ġgenerated Ġdata Ġis Ġstill Ġsignificantly Ġsmaller Ġthan Ġthe Ġmanually Ġannot ated Ġcorpor a Ġfrom ĠCo N LL Ġ2002 Ġand Ġ2003 . ĠFor Ġcomparison , Ġthe Ġtrain Ġset Ġof ĠEnglish ĠCo N LL Ġ2003 Ġcorpus Ġcontains Ġ20 36 21 Ġtokens Ġand Ġthe ĠGerman Ġone Ġ20 69 31 , Ġwhile Ġthe ĠSpanish Ġand ĠDutch Ġcorpor a Ġfrom ĠCo N LL Ġ2002 Ġrespectively Ġ27 30 37 Ġand Ġ2 187 37 Ġlines . ĠThe Ġsmaller Ġsize Ġof Ġour Ġgenerated Ġdata Ġcan Ġbe Ġattributed Ġto Ġthe Ġstrict Ġselection Ġof Ġcandidate Ġsentences Ġas Ġwell Ġas Ġsimply Ġto Ġthe Ġrelatively Ġsmall Ġsize Ġof ĠArmenian ĠWikipedia . </s> The Ġaccuracy Ġof Ġannotation Ġin Ġthe Ġgenerated Ġcorpus Ġheavily Ġrelies Ġon Ġthe Ġquality Ġof Ġlinks Ġin ĠWikipedia Ġarticles . ĠDuring Ġgeneration , Ġwe Ġassumed Ġthat Ġfirst Ġmentions Ġof Ġall Ġnamed Ġentities Ġhave Ġan Ġoutgoing Ġlink Ġto Ġtheir Ġarticle , Ġhowever Ġthis Ġwas Ġnot Ġalways Ġthe Ġcase Ġin Ġactual Ġsource Ġdata Ġand Ġas Ġa Ġresult Ġthe Ġtrain Ġset Ġcontained Ġsentences Ġwhere Ġnot Ġall Ġnamed Ġentities Ġare Ġlabeled . ĠAn notation Ġinaccur acies Ġalso Ġstemmed Ġfrom Ġwrongly Ġassigned Ġlink Ġboundaries Ġ( for Ġexample , Ġin ĠWikipedia Ġarticle Ġ Ô ± Ö Ģ Õ © Õ ¸ Ö Ĥ Ö Ģ Ġ Õ Ī Ö Ĥ Õ ¥ Õ ¬ Õ ½ Õ ¬ Õ « Ġ Õ İ Õ ¥ Õ ¬ Õ « Õ ¶ Õ £ Õ © Õ ¸ Õ ¶ Ġ( Arthur ĠWell esley ) Ġthere Ġis Ġa Ġlink Ġto Ġthe ĠNapoleon Ġarticle Ġwith Ġthe Ġtext Ġ" Õ § Ġ Õ Ĩ Õ ¡ Õ º Õ ¸ Õ ¬ Õ ¥ Õ ¸ Õ ¶ Õ ¨ " Ġ(" Nap oleon Ġis "), Ġwhen Ġit Ġshould Ġbe Ġ" Õ Ĩ Õ ¡ Õ º Õ ¸ Õ ¬ Õ ¥ Õ ¸ Õ ¶ Õ ¨ " Ġ(" Nap oleon ") ). ĠAnother Ġkind Ġof Ġcommon Ġannotation Ġerrors Ġoccurred Ġwhen Ġa Ġnamed Ġentity Ġappeared Ġinside Ġa Ġlink Ġnot Ġtargeting Ġa ĠLOC , ĠOR G , Ġor ĠPER Ġarticle Ġ( e . g . Ġ" Ô ± Õ Ħ Õ Ĩ Ġ Õ ¶ Õ ¡ Õ Ń Õ ¡ Õ £ Õ ¡ Õ ° Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ¨ Õ ¶ Õ ¿ Ö Ģ Õ ¸ Ö Ĥ Õ © Õ µ Õ ¸ Ö Ĥ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Õ ¸ Ö Ĥ Õ ´ " Ġ(" USA Ġpresidential Ġelections ") Ġis Ġlinked Ġto Ġthe Ġarticle Ġ Ô ± Õ Ħ Õ Ĩ Ġ Õ ¶ Õ ¡ Õ Ń Õ ¡ Õ £ Õ ¡ Õ ° Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ¨ Õ ¶ Õ ¿ Ö Ģ Õ ¸ Ö Ĥ Õ © Õ µ Õ ¸ Ö Ĥ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Ġ2016 Ġ( United ĠStates Ġpresidential Ġelection , Ġ2016 ) Ġand Ġas Ġa Ġresult Ġ[ LOC Ġ Ô ± Õ Ħ Õ Ĩ ] Ġ( USA ) Ġis Ġlost ). </s> Test Ġdataset </s> In Ġorder Ġto Ġevaluate Ġthe Ġmodels Ġtrained Ġon Ġgenerated Ġdata , Ġwe Ġmanually Ġannot ated Ġa Ġnamed Ġentities Ġdataset Ġcomprising Ġ53 453 Ġtokens Ġand Ġ25 66 Ġsentences Ġselected Ġfrom Ġover Ġ250 Ġnews Ġtexts Ġfrom Ġil ur . am . ĠThis Ġdataset Ġis Ġcomparable Ġin Ġsize Ġwith Ġthe Ġtest Ġsets Ġof Ġother Ġlanguages Ġ( Table ĠT AB REF 10 Ġ). ĠIncluded Ġsentences Ġare Ġfrom Ġpolitical , Ġsports , Ġlocal Ġand Ġworld Ġnews Ġ( Fig ures ĠFIG REF 8 Ġ, ĠFIG REF 9 Ġ), Ġcovering Ġthe Ġperiod Ġbetween ĠAugust Ġ2012 Ġand ĠJuly Ġ2018 . ĠThe Ġdataset Ġprovides Ġannotations Ġfor Ġ3 Ġpopular Ġnamed Ġentity Ġclasses : Ġpeople Ġ( PER ), Ġorganizations Ġ( OR G ), Ġand Ġlocations Ġ( LOC ), Ġand Ġis Ġreleased Ġin ĠCo N LL 03 Ġformat Ġwith ĠI OB Ġtagging Ġscheme . ĠTokens Ġand Ġsentences Ġwere Ġsegment ed Ġaccording Ġto Ġthe ĠUD Ġstandards Ġfor Ġthe ĠArmenian Ġlanguage ĠB IB REF 11 Ġ. </s> During Ġannotation , Ġwe Ġgenerally Ġrelied Ġon Ġcategories Ġand Ġguidelines Ġassembled Ġby ĠBB N ĠTechnologies Ġfor ĠT REC Ġ2002 Ġquestion Ġanswering Ġtrack . ĠOnly Ġnamed Ġentities Ġcorresponding Ġto ĠBB N 's Ġperson Ġname Ġcategory Ġwere Ġtagged Ġas ĠPER . ĠThose Ġinclude Ġproper Ġnames Ġof Ġpeople , Ġincluding Ġfictional Ġpeople , Ġfirst Ġand Ġlast Ġnames , Ġfamily Ġnames , Ġunique Ġnick names . ĠSimilarly , Ġorganization Ġname Ġcategories , Ġincluding Ġcompany Ġnames , Ġgovernment Ġagencies , Ġeducational Ġand Ġacademic Ġinstitutions , Ġsports Ġclubs , Ġmusical Ġen semb les Ġand Ġother Ġgroups , Ġhospitals , Ġmuseums , Ġnewspaper Ġnames , Ġwere Ġmarked Ġas ĠOR G . ĠHowever , Ġunlike ĠBB N , Ġwe Ġdid Ġnot Ġmark Ġadject ival Ġforms Ġof Ġorganization Ġnames Ġas Ġnamed Ġentities . ĠBB N 's Ġg pe Ġname , Ġfacility Ġname , Ġlocation Ġname Ġcategories Ġwere Ġcombined Ġand Ġannot ated Ġas ĠLOC . </s> We Ġignored Ġentities Ġof Ġother Ġcategories Ġ( e . g . Ġworks Ġof Ġart , Ġlaw , Ġor Ġevents ), Ġincluding Ġthose Ġcases Ġwhere Ġan ĠOR G , ĠLOC Ġor ĠPER Ġentity Ġwas Ġinside Ġan Ġentity Ġof Ġextr aneous Ġtype Ġ( e . g . Ġ Õ Ģ Õ Ģ Ġ( RA ) Ġin Ġ Õ Ģ Õ Ģ Ġ Õ Ķ Ö Ģ Õ ¥ Õ ¡ Õ ¯ Õ ¡ Õ ¶ Ġ Õ ķ Ö Ģ Õ ¥ Õ ¶ Õ ½ Õ £ Õ « Ö Ģ Ö Ħ Ġ( RA ĠCriminal ĠCode ) Ġwas Ġnot Ġannot ated Ġas ĠLOC ). </s> Qu otation Ġmarks Ġaround Ġa Ġnamed Ġentity Ġwere Ġnot Ġannot ated Ġunless Ġthose Ġquotations Ġwere Ġa Ġpart Ġof Ġthat Ġentity 's Ġfull Ġofficial Ġname Ġ( e . g . ĠÂ« Õ Ĩ Õ ¡ Õ « Ö Ģ Õ « Õ ¿ Ġ Õ £ Õ ¸ Ö Ģ Õ ® Õ ¡ Ö Ģ Õ ¡ Õ ¶ Â» Ġ Õ ĵ Ô ² Ô ¸ Ġ(" N air it ĠPlant " ĠCJ SC )). </s> Depending Ġon Ġcontext , Ġmet onyms Ġsuch Ġas Ġ Ô ¿ Ö Ģ Õ ¥ Õ ´ Õ ¬ Ġ( K remlin ), Ġ Ô ² Õ ¡ Õ ² Ö Ģ Õ ¡ Õ ´ Õ µ Õ ¡ Õ ¶ Ġ26 Ġ( B agh ram yan Ġ26 ) Ġwere Ġannot ated Ġas ĠOR G Ġwhen Ġreferring Ġto Ġrespective Ġgovernment Ġagencies . ĠLikewise , Ġcountry Ġor Ġcity Ġnames Ġwere Ġalso Ġtagged Ġas ĠOR G Ġwhen Ġreferring Ġto Ġsports Ġteams Ġrepresenting Ġthem . </s> Word Ġembed d ings </s> Apart Ġfrom Ġthe Ġdatasets , Ġwe Ġalso Ġdeveloped Ġword Ġembed d ings Ġfor Ġthe ĠArmenian Ġlanguage , Ġwhich Ġwe Ġused Ġin Ġour Ġexperiments Ġto Ġtrain Ġand Ġevaluate Ġnamed Ġentity Ġrecognition Ġalgorithms . ĠConsidering Ġtheir Ġability Ġto Ġcapture Ġsemantic Ġregular ities , Ġwe Ġused ĠGl o Ve Ġto Ġtrain Ġword Ġembed d ings . ĠWe Ġassembled Ġa Ġdataset Ġof ĠArmenian Ġtexts Ġcontaining Ġ79 Ġmillion Ġtokens Ġfrom Ġthe Ġarticles Ġof ĠArmenian ĠWikipedia , ĠThe ĠArmenian ĠSoviet ĠEncyclopedia , Ġa Ġsub cor p us Ġof ĠEastern ĠArmenian ĠNational ĠCorpus ĠB IB REF 12 Ġ, Ġover Ġa Ġdozen ĠArmenian Ġnews Ġwebsites Ġand Ġblogs . ĠIncluded Ġtexts Ġcovered Ġtopics Ġsuch Ġas Ġeconomics , Ġpolitics , Ġweather Ġforecast , ĠIT , Ġlaw , Ġsociety Ġand Ġpolitics , Ġcoming Ġfrom Ġnon - fiction Ġas Ġwell Ġas Ġfiction Ġgenres . </s> Similar Ġto Ġthe Ġoriginal Ġembed d ings Ġpublished Ġfor Ġthe ĠEnglish Ġlanguage , Ġwe Ġrelease Ġ50 -, Ġ100 -, Ġ200 - Ġand Ġ300 - dimensional Ġword Ġvectors Ġfor ĠArmenian Ġwith Ġa Ġvocabulary Ġsize Ġof Ġ400 000 . ĠBefore Ġtraining , Ġall Ġthe Ġwords Ġin Ġthe Ġdataset Ġwere Ġlower c ased . ĠFor Ġthe Ġfinal Ġmodels Ġwe Ġused Ġthe Ġfollowing Ġtraining Ġhyper param eters : Ġ15 Ġwindow Ġsize Ġand Ġ20 Ġtraining Ġepoch s . </s> Exper iments </s> In Ġthis Ġsection Ġwe Ġdescribe Ġa Ġnumber Ġof Ġexperiments Ġtargeted Ġto Ġcompare Ġthe Ġperformance Ġof Ġpopular Ġnamed Ġentity Ġrecognition Ġalgorithms Ġon Ġour Ġdata . ĠWe Ġtrained Ġand Ġevaluated ĠStanford ĠN ER , Ġspa Cy Ġ2 . 0 , Ġand Ġa Ġrecurrent Ġmodel Ġsimilar Ġto ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġthat Ġuses Ġbid irection al ĠL ST M Ġcells Ġfor Ġcharacter - based Ġfeature Ġextraction Ġand ĠCR F , Ġdescribed Ġin ĠGu illa ume ĠG enth ial 's ĠSequence ĠT agging Ġwith ĠT ensor flow Ġblog Ġpost ĠB IB REF 15 Ġ. </s> Mod els </s> Stan ford ĠN ER Ġis Ġconditional Ġrandom Ġfields Ġ( CR F ) Ġclass ifier Ġbased Ġon Ġlex ical Ġand Ġcontextual Ġfeatures Ġsuch Ġas Ġthe Ġcurrent Ġword , Ġcharacter - level Ġn - gram s Ġof Ġup Ġto Ġlength Ġ6 Ġat Ġits Ġbeginning Ġand Ġthe Ġend , Ġprevious Ġand Ġnext Ġwords , Ġword Ġshape Ġand Ġsequence Ġfeatures ĠB IB REF 16 Ġ. </s> sp a Cy Ġ2 . 0 Ġuses Ġa ĠCNN - based Ġtransition Ġsystem Ġfor Ġnamed Ġentity Ġrecognition . ĠFor Ġeach Ġtoken , Ġa ĠBloom Ġembed ding Ġis Ġcalculated Ġbased Ġon Ġits Ġlower case Ġform , Ġprefix , Ġsuffix Ġand Ġshape , Ġthen Ġusing Ġresidual ĠCNN s , Ġa Ġcontextual Ġrepresentation Ġof Ġthat Ġtoken Ġis Ġextracted Ġthat Ġpotentially Ġdraws Ġinformation Ġfrom Ġup Ġto Ġ4 Ġtokens Ġfrom Ġeach Ġside ĠB IB REF 17 Ġ. ĠEach Ġupdate Ġof Ġthe Ġtransition Ġsystem 's Ġconfiguration Ġis Ġa Ġclassification Ġtask Ġthat Ġuses Ġthe Ġcontextual Ġrepresentation Ġof Ġthe Ġtop Ġtoken Ġon Ġthe Ġstack , Ġpreceding Ġand Ġsucceeding Ġtokens , Ġfirst Ġtwo Ġtokens Ġof Ġthe Ġbuffer , Ġand Ġtheir Ġleft most , Ġsecond Ġleft most , Ġright most , Ġsecond Ġright most Ġchildren . ĠThe Ġvalid Ġtransition Ġwith Ġthe Ġhighest Ġscore Ġis Ġapplied Ġto Ġthe Ġsystem . ĠThis Ġapproach Ġreportedly Ġperforms Ġwithin Ġ1 % Ġof Ġthe Ġcurrent Ġstate - of - the - art Ġfor ĠEnglish Ġ. ĠIn Ġour Ġexperiments , Ġwe Ġtried Ġout Ġ50 -, Ġ100 -, Ġ200 - Ġand Ġ300 - dimensional Ġpre - trained ĠGl o Ve Ġembed d ings . ĠDue Ġto Ġtime Ġconstraints , Ġwe Ġdid Ġnot Ġtune Ġthe Ġrest Ġof Ġhyper param eters Ġand Ġused Ġtheir Ġdefault Ġvalues . </s> The Ġmain Ġmodel Ġthat Ġwe Ġfocused Ġon Ġwas Ġthe Ġrecurrent Ġmodel Ġwith Ġa ĠCR F Ġtop Ġlayer , Ġand Ġthe Ġabove - mentioned Ġmethods Ġserved Ġmostly Ġas Ġbas elines . ĠThe Ġdistinctive Ġfeature Ġof Ġthis Ġapproach Ġis Ġthe Ġway Ġcontextual Ġword Ġembed d ings Ġare Ġformed . ĠFor Ġeach Ġtoken Ġseparately , Ġto Ġcapture Ġits Ġword Ġshape Ġfeatures , Ġcharacter - based Ġrepresentation Ġis Ġextracted Ġusing Ġa Ġbid irection al ĠL ST M ĠB IB REF 18 Ġ. ĠThis Ġrepresentation Ġgets Ġconc aten ated Ġwith Ġa Ġdistribution al Ġword Ġvector Ġsuch Ġas ĠGl o Ve , Ġforming Ġan Ġintermediate Ġword Ġembed ding . ĠUsing Ġanother Ġbid irection al ĠL ST M Ġcell Ġon Ġthese Ġintermediate Ġword Ġembed d ings , Ġthe Ġcontextual Ġrepresentation Ġof Ġtokens Ġis Ġobtained Ġ( Figure ĠFIG REF 17 Ġ). ĠFinally , Ġa ĠCR F Ġlayer Ġlabels Ġthe Ġsequence Ġof Ġthese Ġcontextual Ġrepresentations . ĠIn Ġour Ġexperiments , Ġwe Ġused ĠGu illa ume ĠG enth ial 's Ġimplementation Ġof Ġthe Ġalgorithm . ĠWe Ġset Ġthe Ġsize Ġof Ġcharacter - based Ġbi L ST M Ġto Ġ100 Ġand Ġthe Ġsize Ġof Ġsecond Ġbi L ST M Ġnetwork Ġto Ġ300 . </s> E val uation </s> Exper iments Ġwere Ġcarried Ġout Ġusing ĠI OB Ġtagging Ġscheme , Ġwith Ġa Ġtotal Ġof Ġ7 Ġclass Ġtags : ĠO , ĠB - PER , ĠI - PER , ĠB - LOC , ĠI - LOC , ĠB - OR G , ĠI - OR G . </s> We Ġrandomly Ġselected Ġ80 % Ġof Ġgenerated Ġannot ated Ġsentences Ġfor Ġtraining Ġand Ġused Ġthe Ġother Ġ20 % Ġas Ġa Ġdevelopment Ġset . ĠThe Ġmodels Ġwith Ġthe Ġbest ĠF 1 Ġscore Ġon Ġthe Ġdevelopment Ġset Ġwere Ġtested Ġon Ġthe Ġmanually Ġannot ated Ġgold Ġdataset . </s> Discussion </s> Table ĠT AB REF 19 Ġshows Ġthe Ġaverage Ġscores Ġof Ġevaluated Ġmodels . ĠThe Ġhighest ĠF 1 Ġscore Ġwas Ġachieved Ġby Ġthe Ġrecurrent Ġmodel Ġusing Ġa Ġbatch Ġsize Ġof Ġ8 Ġand ĠAdam Ġoptim izer Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ0 . 001 . ĠUp dating Ġword Ġembed d ings Ġduring Ġtraining Ġalso Ġnoticeably Ġimproved Ġthe Ġperformance . ĠGl o Ve Ġword Ġvector Ġmodels Ġof Ġfour Ġdifferent Ġsizes Ġ( 50 , Ġ100 , Ġ200 , Ġand Ġ300 ) Ġwere Ġtested , Ġwith Ġvectors Ġof Ġsize Ġ50 Ġproducing Ġthe Ġbest Ġresults Ġ( Table ĠT AB REF 20 Ġ). </s> For Ġspa Cy Ġ2 . 0 Ġnamed Ġentity Ġrecogn izer , Ġthe Ġsame Ġword Ġembed ding Ġmodels Ġwere Ġtested . ĠHowever , Ġin Ġthis Ġcase Ġthe Ġperformance Ġof Ġ200 - dimensional Ġembed d ings Ġwas Ġhighest Ġ( Table ĠT AB REF 21 Ġ). ĠUn surprisingly , Ġboth Ġdeep Ġlearning Ġmodels Ġoutper formed Ġthe Ġfeature - based ĠStanford Ġrecogn izer Ġin Ġrecall , Ġthe Ġlatter Ġhowever Ġdemonstrated Ġnoticeably Ġhigher Ġprecision . </s> It Ġis Ġclear Ġthat Ġthe Ġdevelopment Ġset Ġof Ġautomatically Ġgenerated Ġexamples Ġwas Ġnot Ġan Ġideal Ġindicator Ġof Ġmodels ' Ġperformance Ġon Ġgold - standard Ġtest Ġset . ĠHigher Ġdevelopment Ġset Ġscores Ġoften Ġled Ġto Ġlower Ġtest Ġscores Ġas Ġseen Ġin Ġthe Ġevaluation Ġresults Ġfor Ġspa Cy Ġ2 . 0 Ġand ĠChar - bi L ST M + bi L ST M + CR F Ġ( T ables ĠT AB REF 21 Ġand ĠT AB REF 20 Ġ). ĠAnalysis Ġof Ġerrors Ġon Ġthe Ġdevelopment Ġset Ġrevealed Ġthat Ġmany Ġwere Ġcaused Ġby Ġthe Ġincom ple teness Ġof Ġannotations , Ġwhen Ġnamed Ġentity Ġrecogn izers Ġcorrectly Ġpredicted Ġentities Ġthat Ġwere Ġabsent Ġfrom Ġannotations Ġ( e . g . Ġ[ Ô ½ Õ į Õ Ģ Õ Ħ - Õ « ĠLOC ] Ġ( US SR 's ), Ġ[ Ô ´ Õ « Õ ¶ Õ ¡ Õ ´ Õ ¸ Õ ¶ ĠOR G ] Ġ( the _ D in amo ), Ġ[ Õ Ĭ Õ « Ö Ģ Õ ¥ Õ ¶ Õ ¥ Õ µ Õ ¡ Õ ¶ Ġ Õ © Õ ¥ Ö Ģ Õ ¡ Õ ¯ Õ ² Õ ¦ Õ ¸ Ö Ĥ ĠLOC ] Ġ( I ber ian ĠPeninsula 's ) Ġetc ). ĠSimilarly , Ġthe Ġrecogn izers Ġoften Ġcorrectly Ġignored Ġnon - ent ities Ġthat Ġare Ġincorrectly Ġlabeled Ġin Ġdata Ġ( e . g . Ġ[ Ö ħ Õ ½ Õ ´ Õ ¡ Õ ¶ Õ ¶ Õ ¥ Ö Ģ Õ « ĠPER ], Ġ[ Õ ¯ Õ ¸ Õ ¶ Õ ½ Õ ¥ Ö Ģ Õ ¾ Õ ¡ Õ ¿ Õ ¸ Ö Ģ Õ « Õ ¡ Õ ¶ ĠOR G ] Ġetc ). </s> Generally , Ġtested Ġmodels Ġdemonstrated Ġrelatively Ġhigh Ġprecision Ġof Ġrecognizing Ġtokens Ġthat Ġstarted Ġnamed Ġentities , Ġbut Ġfailed Ġto Ġdo Ġso Ġwith Ġdescriptor Ġwords Ġfor Ġorganizations Ġand , Ġto Ġa Ġcertain Ġdegree , Ġlocations . ĠThe Ġconfusion Ġmatrix Ġfor Ġone Ġof Ġthe Ġtrained Ġrecurrent Ġmodels Ġillustrates Ġthat Ġdifference Ġ( Table ĠT AB REF 22 Ġ). ĠThis Ġcan Ġbe Ġpartly Ġattributed Ġto Ġthe Ġquality Ġof Ġgenerated Ġdata : Ġdescriptor Ġwords Ġare Ġsometimes Ġsuperflu ously Ġlabeled Ġ( e . g . Ġ[ Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ Õ « Ġ Õ ¿ Õ ¥ Õ ² Õ ¡ Õ ¢ Õ ¶ Õ « Õ ¯ Õ ¶ Õ ¥ Ö Ģ Õ ¨ ĠLOC ] Ġ( the Ġindigenous Ġpeople Ġof ĠHawaii )), Ġwhich Ġis Ġlikely Ġcaused Ġby Ġthe Ġinconsistent Ġstyle Ġof Ġlinking Ġin ĠArmenian ĠWikipedia Ġ( in Ġthe Ġarticle Ġ Ô ± Õ Ħ Õ Ĩ Ġ Õ ´ Õ · Õ ¡ Õ ¯ Õ ¸ Ö Ĥ Õ µ Õ © Ġ( C ulture Ġof Ġthe ĠUnited ĠStates ), Ġits Ġlinked Ġtext Ġfragment Ġ" Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ Õ « Ġ Õ ¿ Õ ¥ Õ ² Õ ¡ Õ ¢ Õ ¶ Õ « Õ ¯ Õ ¶ Õ ¥ Ö Ģ Õ ¨ " Ġ(" the Ġindigenous Ġpeople Ġof ĠHawaii ") Ġleads Ġto Ġthe Ġarticle Ġ Õ Ģ Õ ¡ Õ ¾ Õ ¡ Õ µ Õ ¡ Õ ¶ Ġ Õ ¯ Õ ² Õ ¦ Õ « Õ ¶ Õ ¥ Ö Ģ ( Haw aii )). </s> Conclusion </s> We Ġrelease Ġtwo Ġnamed - entity Ġannot ated Ġdatasets Ġfor Ġthe ĠArmenian Ġlanguage : Ġa Ġsilver - standard Ġcorpus Ġfor Ġtraining ĠN ER Ġmodels , Ġand Ġa Ġgold - standard Ġcorpus Ġfor Ġtesting . ĠIt Ġis Ġworth Ġto Ġunder line Ġthe Ġimportance Ġof Ġthe Ġlatter Ġcorpus , Ġas Ġwe Ġaim Ġit Ġto Ġserve Ġas Ġa Ġbenchmark Ġfor Ġfuture Ġnamed Ġentity Ġrecognition Ġsystems Ġdesigned Ġfor Ġthe ĠArmenian Ġlanguage . ĠAlong Ġwith Ġthe Ġcorpor a , Ġwe Ġpublish ĠGl o Ve Ġword Ġvector Ġmodels Ġtrained Ġon Ġa Ġcollection Ġof ĠArmenian Ġtexts . </s> Additionally , Ġto Ġestablish Ġthe Ġapplic ability Ġof ĠWikipedia - based Ġapproaches Ġfor Ġthe ĠArmenian Ġlanguage , Ġwe Ġprovide Ġevaluation Ġresults Ġfor Ġ3 Ġdifferent Ġnamed Ġentity Ġrecognition Ġsystems Ġtrained Ġand Ġtested Ġon Ġour Ġdatasets . ĠThe Ġresults Ġreinforce Ġthe Ġability Ġof Ġdeep Ġlearning Ġapproaches Ġin Ġachieving Ġrelatively Ġhigh Ġrecall Ġvalues Ġfor Ġthis Ġspecific Ġtask , Ġas Ġwell Ġas Ġthe Ġpower Ġof Ġusing Ġcharacter - ext racted Ġembed d ings Ġalongside Ġconventional Ġword Ġembed d ings . </s> There Ġare Ġseveral Ġavenues Ġof Ġfuture Ġwork . ĠSince ĠN oth man Ġet Ġal . Ġ2013 , Ġmore Ġefficient Ġmethods Ġof Ġexploiting ĠWikipedia Ġhave Ġbeen Ġproposed , Ġnamely ĠWi NER ĠB IB REF 19 Ġ, Ġwhich Ġcould Ġhelp Ġincrease Ġboth Ġthe Ġquantity Ġand Ġquality Ġof Ġthe Ġtraining Ġcorpus . ĠAnother Ġpotential Ġarea Ġof Ġwork Ġis Ġthe Ġfurther Ġenrichment Ġof Ġthe Ġbenchmark Ġtest Ġset Ġwith Ġadditional Ġannotation Ġof Ġother Ġclasses Ġsuch Ġas ĠM ISC Ġor Ġmore Ġfine - gr ained Ġtypes Ġ( e . g . ĠCITY , ĠCOUN TRY , ĠREG ION Ġetc Ġinstead Ġof ĠLOC ).