No
BOOLEAN
No
BOOLEAN
1908.05828
Introduction </s> N amed ĠEntity ĠRecogn ition Ġ( NER ) Ġis Ġa Ġforemost ĠN LP Ġtask Ġto Ġlabel Ġeach Ġatomic Ġelements Ġof Ġa Ġsentence Ġinto Ġspecific Ġcategories Ġlike Ġ" P ERSON ", Ġ" LOC ATION ", Ġ" OR GAN IZ ATION " Ġand Ġothers BI B REF 0 . ĠThere Ġhas Ġbeen Ġan Ġextensive ĠN ER Ġresearch Ġon ĠEnglish , ĠGerman , ĠDutch Ġand ĠSpanish Ġlanguage ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġand Ġnotable Ġresearch Ġon Ġlow Ġresource ĠSouth ĠAsian Ġlanguages Ġlike ĠHindi BI B REF 6 , ĠIndonesian BI B REF 7 Ġand Ġother ĠIndian Ġlanguages Ġ( K ann ada , ĠMal ay alam , ĠTamil Ġand ĠTel ugu ) BI B REF 8 . ĠHowever , Ġthere Ġhas Ġbeen Ġno Ġstudy Ġon Ġdeveloping Ġneural ĠN ER Ġfor ĠNep ali Ġlanguage . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġneural Ġbased ĠNep ali ĠN ER Ġusing Ġlatest Ġstate - of - the - art Ġarchitecture Ġbased Ġon Ġgrap he me - level Ġwhich Ġdoesn 't Ġrequire Ġany Ġhand - crafted Ġfeatures Ġand Ġno Ġdata Ġpre - processing . </s> Recent Ġneural Ġarchitecture Ġlike ĠB IB REF 1 Ġis Ġused Ġto Ġrelax Ġthe Ġneed Ġto Ġhand - craft Ġthe Ġfeatures Ġand Ġneed Ġto Ġuse Ġpart - of - speech Ġtag Ġto Ġdetermine Ġthe Ġcategory Ġof Ġthe Ġentity . ĠHowever , Ġthis Ġarchitecture Ġhave Ġbeen Ġstudied Ġfor Ġlanguages Ġlike ĠEnglish , Ġand ĠGerman Ġand Ġnot Ġbeen Ġapplied Ġto Ġlanguages Ġlike ĠNep ali Ġwhich Ġis Ġa Ġlow Ġresource Ġlanguage Ġi . e Ġlimited Ġdata Ġset Ġto Ġtrain Ġthe Ġmodel . ĠTraditional Ġmethods Ġlike ĠHidden ĠMark ov ĠModel Ġ( H MM ) Ġwith Ġrule Ġbased Ġapproaches BI B REF 9 , BI B REF 10 , Ġand ĠSupport ĠVector ĠMachine Ġ( S VM ) Ġwith Ġmanual Ġfeature - engineering BI B REF 11 Ġhave Ġbeen Ġapplied Ġbut Ġthey Ġperform Ġpoor Ġcompared Ġto Ġneural . ĠHowever , Ġthere Ġhas Ġbeen Ġno Ġresearch Ġin ĠNep ali ĠN ER Ġusing Ġneural Ġnetwork . ĠTherefore , Ġwe Ġcreated Ġthe Ġnamed Ġentity Ġannot ated Ġdataset Ġpartly Ġwith Ġthe Ġhelp Ġof ĠDat atur k Ġto Ġtrain Ġa Ġneural Ġmodel . ĠThe Ġtexts Ġused Ġfor Ġthis Ġdataset Ġare Ġcollected Ġfrom Ġvarious Ġdaily Ġnews Ġsources Ġfrom ĠNepal Ġaround Ġthe Ġyear Ġ2015 - 2016 . </s> Following Ġare Ġour Ġcontributions : </s> We Ġpresent Ġa Ġnovel ĠNamed ĠEntity ĠRecogn izer Ġ( NER ) Ġfor ĠNep ali Ġlanguage . ĠTo Ġbest Ġof Ġour Ġknowledge Ġwe Ġare Ġthe Ġfirst Ġto Ġpropose Ġneural Ġbased ĠNep ali ĠN ER . </s> As Ġthere Ġare Ġnot Ġgood Ġquality Ġdataset Ġto Ġtrain ĠN ER Ġwe Ġrelease Ġa Ġdataset Ġto Ġsupport Ġfuture Ġresearch </s> We Ġperform Ġempirical Ġevaluation Ġof Ġour Ġmodel Ġwith Ġstate - of - the - art Ġmodels Ġwith Ġrelative Ġimprovement Ġof Ġupt o Ġ10 % </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġworks Ġsimilar Ġto Ġours Ġin ĠSection ĠSEC REF 2 . ĠWe Ġdescribe Ġour Ġapproach Ġand Ġdataset Ġstatistics Ġin ĠSection ĠSEC REF 3 Ġand ĠSEC REF 4 , Ġfollowed Ġby Ġour Ġexperiments , Ġevaluation Ġand Ġdiscussion Ġin ĠSection ĠSEC REF 5 , ĠSEC REF 6 , Ġand ĠSEC REF 7 . ĠWe Ġconclude Ġwith Ġour Ġobservations Ġin ĠSection ĠSEC REF 8 . </s> To Ġfacilitate Ġfurther Ġresearch Ġour Ġcode Ġand Ġdataset Ġwill Ġbe Ġmade Ġavailable Ġat Ġgithub . com / link - yet - to - be - updated </s> Related ĠWork </s> There Ġhas Ġbeen Ġa Ġhandful Ġof Ġresearch Ġon ĠNep ali ĠN ER Ġtask Ġbased Ġon Ġapproaches Ġlike ĠSupport ĠVector ĠMachine Ġand Ġg az ette er Ġlist BI B REF 11 Ġand ĠHidden ĠMark ov ĠModel Ġand Ġg az ette er Ġlist BI B REF 9 , BI B REF 10 . </s> BI B REF 11 Ġuses ĠS VM Ġalong Ġwith Ġfeatures Ġlike Ġfirst Ġword , Ġword Ġlength , Ġdigit Ġfeatures Ġand Ġg az ette er Ġ( person , Ġorganization , Ġlocation , Ġmiddle Ġname , Ġverb , Ġdesignation Ġand Ġothers ). ĠIt Ġuses Ġone Ġvs Ġrest Ġclassification Ġmodel Ġto Ġclassify Ġeach Ġword Ġinto Ġdifferent Ġentity Ġclasses . ĠHowever , Ġit Ġdoes Ġnot Ġthe Ġtake Ġcontext Ġword Ġinto Ġaccount Ġwhile Ġtraining Ġthe Ġmodel . ĠSimilarly , ĠB IB REF 9 Ġand ĠB IB REF 10 Ġuses ĠHidden ĠMark ov ĠModel Ġwith Ġn - gram Ġtechnique Ġfor Ġextracting ĠPOS - tags . ĠPOS - tags Ġwith Ġcommon Ġnoun , Ġproper Ġnoun Ġor Ġcombination Ġof Ġboth Ġare Ġcombined Ġtogether , Ġthen Ġuses Ġg az ette er Ġlist Ġas Ġlook - up Ġtable Ġto Ġidentify Ġthe Ġnamed Ġentities . </s> Researchers Ġhave Ġshown Ġthat Ġthe Ġneural Ġnetworks Ġlike ĠCNN BI B REF 12 , ĠR NN BI B REF 13 , ĠL ST M BI B REF 14 , ĠGR U BI B REF 15 Ġcan Ġcapture Ġthe Ġsemantic Ġknowledge Ġof Ġlanguage Ġbetter Ġwith Ġthe Ġhelp Ġof Ġpre - trained Ġemb bed d ings Ġlike Ġword 2 vec BI B REF 16 , Ġglove BI B REF 17 Ġor Ġfast text BI B REF 18 . </s> Similar Ġapproaches Ġhas Ġbeen Ġapplied Ġto Ġmany ĠSouth ĠAsian Ġlanguages Ġlike ĠHindi BI B REF 6 , ĠIndonesian BI B REF 7 , ĠBeng ali BI B REF 19 Ġand ĠIn Ġthis Ġpaper , Ġwe Ġpresent Ġthe Ġneural Ġnetwork Ġarchitecture Ġfor ĠN ER Ġtask Ġin ĠNep ali Ġlanguage , Ġwhich Ġdoesn 't Ġrequire Ġany Ġmanual Ġfeature Ġengineering Ġnor Ġany Ġdata Ġpre - processing Ġduring Ġtraining . ĠFirst Ġwe Ġare Ġcomparing ĠBi L ST M BI B REF 14 , ĠBi L ST M + CNN BI B REF 20 , ĠBi L ST M + CR FBI B REF 1 , ĠBi L ST M + CNN + CR FBI B REF 2 Ġmodels Ġwith ĠCNN Ġmodel BI B REF 0 Ġand ĠStanford ĠCR F Ġmodel BI B REF 21 . ĠSecondly , Ġwe Ġshow Ġthe Ġcomparison Ġbetween Ġmodels Ġtrained Ġon Ġgeneral Ġword Ġembed d ings , Ġword Ġembed ding Ġ+ Ġcharacter - level Ġembed ding , Ġword Ġembed ding Ġ+ Ġpart - of - speech ( POS ) Ġone - hot Ġencoding Ġand Ġword Ġembed ding Ġ+ Ġgrap he me Ġclustered Ġor Ġsub - word Ġembed ding BI B REF 22 . ĠThe Ġexperiments Ġwere Ġperformed Ġon Ġthe Ġdataset Ġthat Ġwe Ġcreated Ġand Ġon Ġthe Ġdataset Ġreceived Ġfrom ĠIL PR L Ġlab . ĠOur Ġextensive Ġstudy Ġshows Ġthat Ġaugment ing Ġword Ġembed ding Ġwith Ġcharacter Ġor Ġgrap he me - level Ġrepresentation Ġand ĠPOS Ġone - hot Ġencoding Ġvector Ġyields Ġbetter Ġresults Ġcompared Ġto Ġusing Ġgeneral Ġword Ġembed ding Ġalone . </s> App roach </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġapproach Ġin Ġbuilding Ġour Ġmodel . ĠThis Ġmodel Ġis Ġpartly Ġinspired Ġfrom Ġmultiple Ġmodels ĠB IB REF 20 , BI B REF 1 , Ġand BI B REF 2 </s> App roach Ġ: :: ĠBid irection al ĠL ST M </s> We Ġused ĠBi - direction al ĠL ST M Ġto Ġcapture Ġthe Ġword Ġrepresentation Ġin Ġforward Ġas Ġwell Ġas Ġreverse Ġdirection Ġof Ġa Ġsentence . ĠGenerally , ĠL ST Ms Ġtake Ġinputs Ġfrom Ġleft Ġ( past ) Ġof Ġthe Ġsentence Ġand Ġcomp utes Ġthe Ġhidden Ġstate . ĠHowever , Ġit Ġis Ġproven Ġbeneficial BI B REF 23 Ġto Ġuse Ġbi - direction al ĠL ST M , Ġwhere , Ġhidden Ġstates Ġare Ġcomputed Ġbased Ġfrom Ġright Ġ( future ) Ġof Ġsentence Ġand Ġboth Ġof Ġthese Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġfinal Ġoutput Ġas Ġ$ h _ t $ =[ $ \ over right arrow { h _ t } $ ; $ \ over left arrow { h _ t } $ ], Ġwhere Ġ$\ over right arrow { h _ t } $, Ġ$\ over left arrow { h _ t } $ Ġ= Ġhidden Ġstate Ġcomputed Ġin Ġforward Ġand Ġbackward Ġdirection Ġrespectively . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠWord Ġembed d ings </s> We Ġhave Ġused ĠWord 2 V ec ĠB IB REF 16 , ĠGl o Ve ĠB IB REF 17 Ġand ĠFast Text ĠB IB REF 18 Ġword Ġvectors Ġof Ġ300 Ġdimensions . ĠThese Ġvectors Ġwere Ġtrained Ġon Ġthe Ġcorpus Ġobtained Ġfrom ĠNep ali ĠNational ĠCorpus . ĠThis Ġpre - le mm at ized Ġcorpus Ġconsists Ġof Ġ14 Ġmillion Ġwords Ġfrom Ġbooks , Ġweb - text s Ġand Ġnews Ġpapers . ĠThis Ġcorpus Ġwas Ġmixed Ġwith Ġthe Ġtexts Ġfrom Ġthe Ġdataset Ġbefore Ġtraining ĠCB OW Ġand Ġskip - gram Ġversion Ġof Ġword 2 vec Ġusing Ġg ens im Ġlibrary BI B REF 24 . ĠThis Ġtrained Ġmodel Ġconsists Ġof Ġvectors Ġfor Ġ7 27 82 Ġunique Ġwords . </s> Light Ġpre - processing Ġwas Ġperformed Ġon Ġthe Ġcorpus Ġbefore Ġtraining Ġit . ĠFor Ġexample , Ġinvalid Ġcharacters Ġor Ġcharacters Ġother Ġthan ĠDev an ag ari Ġwere Ġremoved Ġbut Ġpunct uation Ġand Ġnumbers Ġwere Ġnot Ġremoved . ĠWe Ġset Ġthe Ġwindow Ġcontext Ġat Ġ10 Ġand Ġthe Ġrare Ġwords Ġwhose Ġcount Ġis Ġbelow Ġ5 Ġare Ġdropped . ĠThese Ġword Ġembed d ings Ġwere Ġnot Ġfrozen Ġduring Ġthe Ġtraining Ġsession Ġbecause Ġfine - tun ing Ġword Ġembed ding Ġhelp Ġachieve Ġbetter Ġperformance Ġcompared Ġto Ġfrozen Ġone BI B REF 20 . </s> We Ġhave Ġused Ġfast text Ġembed d ings Ġin Ġparticular Ġbecause Ġof Ġits Ġsub - word Ġrepresentation Ġability , Ġwhich Ġis Ġvery Ġuseful Ġin Ġhighly Ġinf lection al Ġlanguage Ġas Ġshown Ġin ĠTable ĠT AB REF 25 . ĠWe Ġhave Ġtrained Ġthe Ġword Ġembed ding Ġin Ġsuch Ġa Ġway Ġthat Ġthe Ġsub - word Ġsize Ġremains Ġbetween Ġ1 Ġand Ġ4 . ĠWe Ġparticularly Ġchose Ġthis Ġsize Ġbecause Ġin ĠNep ali Ġlanguage Ġa Ġsingle Ġletter Ġcan Ġalso Ġbe Ġa Ġword , Ġfor Ġexample Ġe , Ġt , ĠC , Ġr , Ġl , Ġn , Ġu Ġand Ġa Ġsingle Ġcharacter Ġ( g rap he me ) Ġor Ġsub - word Ġcan Ġbe Ġformed Ġafter Ġmixture Ġof Ġdependent Ġvowel Ġsigns Ġwith Ġconson ant Ġletters Ġfor Ġexample , ĠC Ġ+ ĠO Ġ+ Ġ= ĠCO , Ġhere Ġthree Ġdifferent Ġconson ant Ġletters Ġform Ġa Ġsingle Ġsub - word . </s> The Ġtwo - dimensional Ġvisualization Ġof Ġan Ġexample Ġword Ġnp Al Ġis Ġshown Ġin ĠFIG REF 14 . ĠPrincipal ĠComponent ĠAnalysis Ġ( PC A ) Ġtechnique Ġwas Ġused Ġto Ġgenerate Ġthis Ġvisualization Ġwhich Ġhelps Ġuse Ġto Ġanalyze Ġthe Ġnearest Ġneighbor Ġwords Ġof Ġa Ġgiven Ġsample Ġword . Ġ84 Ġand Ġ104 Ġnearest Ġneighbors Ġwere Ġobserved Ġusing Ġword 2 vec Ġand Ġfast text Ġembed ding Ġrespectively Ġon Ġthe Ġsame Ġcorpus . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠCharacter - level Ġembed d ings </s> BI B REF 20 Ġand ĠB IB REF 2 Ġsuccessfully Ġpresented Ġthat Ġthe Ġcharacter - level Ġembed d ings , Ġextracted Ġusing ĠCNN , Ġwhen Ġcombined Ġwith Ġword Ġembed d ings Ġenhances Ġthe ĠN ER Ġmodel Ġperformance Ġsignificantly , Ġas Ġit Ġis Ġable Ġto Ġcapture Ġmorph ological Ġfeatures Ġof Ġa Ġword . ĠFigure ĠFIG REF 7 Ġshows Ġthe Ġgrap he me - level ĠCNN Ġused Ġin Ġour Ġmodel , Ġwhere Ġinputs Ġto ĠCNN Ġare Ġgrap hem es . ĠCharacter - level ĠCNN Ġis Ġalso Ġbuilt Ġin Ġsimilar Ġfashion , Ġexcept Ġthe Ġinputs Ġare Ġcharacters . ĠG rap he me Ġor ĠCharacter Ġ- level Ġembed d ings Ġare Ġrandomly Ġinitialized Ġfrom Ġ[ 0 , 1 ] Ġwith Ġreal Ġvalues Ġwith Ġuniform Ġdistribution Ġof Ġdimension Ġ30 . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠG rap he me - level Ġembed d ings </s> G rap he me Ġis Ġatomic Ġmeaningful Ġunit Ġin Ġwriting Ġsystem Ġof Ġany Ġlanguages . ĠSince , ĠNep ali Ġlanguage Ġis Ġhighly Ġmorph ologically Ġinf lection al , Ġwe Ġcompared Ġgrap he me - level Ġrepresentation Ġwith Ġcharacter - level Ġrepresentation Ġto Ġevaluate Ġits Ġeffect . ĠFor Ġexample , Ġin Ġcharacter - level Ġembed ding , Ġeach Ġcharacter Ġof Ġa Ġword Ġnp Al Ġresults Ġinto Ġn Ġ+ Ġ+ Ġp Ġ+ ĠA Ġ+ Ġl Ġhas Ġits Ġown Ġembed ding . ĠHowever , Ġin Ġgrap he me Ġlevel , Ġa Ġword Ġnp Al Ġis Ġclustered Ġinto Ġgrap hem es , Ġresulting Ġinto Ġn Ġ+ Ġp A Ġ+ Ġl . ĠHere , Ġeach Ġgrap he me Ġhas Ġits Ġown Ġembed ding . ĠThis Ġgrap he me - level Ġembed ding Ġresults Ġgood Ġscores Ġon Ġpar Ġwith Ġcharacter - level Ġembed ding Ġin Ġhighly Ġinf lection al Ġlanguages Ġlike ĠNep ali , Ġbecause Ġgrap hem es Ġalso Ġcapture Ġsynt actic Ġinformation Ġsimilar Ġto Ġcharacters . ĠWe Ġcreated Ġgrap he me Ġclusters Ġusing Ġun ise g Ġpackage Ġwhich Ġis Ġhelpful Ġin Ġunic ode Ġtext Ġsegment ations . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠPart - of - speech Ġ( POS ) Ġone Ġhot Ġencoding </s> We Ġcreated Ġone - hot Ġencoded Ġvector Ġof ĠPOS Ġtags Ġand Ġthen Ġconc aten ated Ġwith Ġpre - trained Ġword Ġembed d ings Ġbefore Ġpassing Ġit Ġto ĠBi L ST M Ġnetwork . ĠA Ġsample Ġof Ġdata Ġis Ġshown Ġin Ġfigure ĠFIG REF 13 . </s> Dat as et ĠStatistics Ġ: :: ĠOur N ep ali Ġdataset </s> Since , Ġwe Ġthere Ġwas Ġno Ġpublicly Ġavailable Ġstandard ĠNep ali ĠN ER Ġdataset Ġand Ġdid Ġnot Ġreceive Ġany Ġdataset Ġfrom Ġthe Ġprevious Ġresearchers , Ġwe Ġhad Ġto Ġcreate Ġour Ġown Ġdataset . ĠThis Ġdataset Ġcontains Ġthe Ġsentences Ġcollected Ġfrom Ġdaily Ġnewspaper Ġof Ġthe Ġyear Ġ2015 - 2016 . ĠThis Ġdataset Ġhas Ġthree Ġmajor Ġclasses ĠPerson Ġ( PER ), ĠLocation Ġ( LOC ) Ġand ĠOrganization Ġ( OR G ). ĠPre - processing Ġwas Ġperformed Ġon Ġthe Ġtext Ġbefore Ġcreation Ġof Ġthe Ġdataset , Ġfor Ġexample Ġall Ġpunct uations Ġand Ġnumbers Ġbesides Ġ', ', Ġ' - ', Ġ' | ' Ġand Ġ' .' Ġwere Ġremoved . ĠCurrently , Ġthe Ġdataset Ġis Ġin Ġstandard ĠCo N LL - 2003 ĠIO Ġformat BI B REF 25 . </s> Since , Ġthis Ġdataset Ġis Ġnot Ġle mm at ized Ġoriginally , Ġwe Ġle mm at ized Ġonly Ġthe Ġpost - pos itions Ġlike ĠEk , Ġk O , Ġl , Ġm A , Ġm , Ġmy , Ġj F , Ġs g , Ġa EG Ġwhich Ġare Ġjust Ġthe Ġfew Ġexamples Ġamong Ġ299 Ġpost Ġpositions Ġin ĠNep ali Ġlanguage . ĠWe Ġobtained Ġthese Ġpost - pos itions Ġfrom Ġsan ja al cor ps Ġand Ġadded Ġfew Ġmore Ġto Ġmatch Ġour Ġdataset . ĠWe Ġwill Ġbe Ġreleasing Ġthis Ġlist Ġin Ġour Ġgithub Ġrepository . ĠWe Ġfound Ġout Ġthat Ġle mm at izing Ġthe Ġpost - pos itions Ġboosted Ġthe ĠF 1 Ġscore Ġby Ġalmost Ġ10 %. </s> In Ġorder Ġto Ġlabel Ġour Ġdataset Ġwith ĠPOS - tags , Ġwe Ġfirst Ġcreated ĠPOS Ġannot ated Ġdataset Ġof Ġ69 46 Ġsentences Ġand Ġ16 225 Ġunique Ġwords Ġextracted Ġfrom ĠPOS - tag ged ĠNep ali ĠNational ĠCorpus Ġand Ġtrained Ġa ĠBi L ST M Ġmodel Ġwith Ġ95 . 14 % Ġaccuracy Ġwhich Ġwas Ġused Ġto Ġcreate ĠPOS - tags Ġfor Ġour Ġdataset . </s> The Ġdataset Ġreleased Ġin Ġour Ġgithub Ġrepository Ġcontains Ġeach Ġword Ġin Ġnew line Ġwith Ġspace Ġseparated ĠPOS - tags Ġand ĠEntity - tags . ĠThe Ġsentences Ġare Ġseparated Ġby Ġempty Ġnew line . ĠA Ġsample Ġsentence Ġfrom Ġthe Ġdataset Ġis Ġpresented Ġin Ġtable ĠFIG REF 13 . </s> Dat as et ĠStatistics Ġ: :: ĠIL PR L Ġdataset </s> After Ġmuch Ġtime , Ġwe Ġreceived Ġthe Ġdataset Ġfrom ĠBal ĠKrishna ĠBal , ĠIL PR L , ĠK U . ĠThis Ġdataset Ġfollows Ġstandard ĠCo N LL - 2003 ĠI OB Ġformat BI B REF 25 Ġwith ĠPOS Ġtags . ĠThis Ġdataset Ġis Ġprepared Ġby ĠIL PR L ĠLab , ĠK U Ġand ĠKE IV ĠTechnologies . ĠFew Ġcorrections Ġlike Ġcorrecting Ġthe ĠN ER Ġtags Ġhad Ġto Ġbe Ġmade Ġon Ġthe Ġdataset . ĠThe Ġstatistics Ġof Ġboth Ġthe Ġdataset Ġis Ġpresented Ġin Ġtable ĠT AB REF 23 . </s> Table ĠT AB REF 24 Ġpresents Ġthe Ġtotal Ġentities Ġ( PER , ĠLOC , ĠOR G Ġand ĠM ISC ) Ġfrom Ġboth Ġof Ġthe Ġdataset Ġused Ġin Ġour Ġexperiments . ĠThe Ġdataset Ġis Ġdivided Ġinto Ġthree Ġparts Ġwith Ġ64 %, Ġ16 % Ġand Ġ20 % Ġof Ġthe Ġtotal Ġdataset Ġinto Ġtraining Ġset , Ġdevelopment Ġset Ġand Ġtest Ġset Ġrespectively . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġdetails Ġabout Ġtraining Ġour Ġneural Ġnetwork . ĠThe Ġneural Ġnetwork Ġarchitecture Ġare Ġimplemented Ġusing ĠPy Tor ch Ġframework ĠB IB REF 26 . ĠThe Ġtraining Ġis Ġperformed Ġon Ġa Ġsingle ĠNvidia ĠTesla ĠP 100 ĠS XM 2 . ĠWe Ġfirst Ġrun Ġour Ġexperiment Ġon ĠBi L ST M , ĠBi L ST M - CNN , ĠBi L ST M - CR F ĠBi L ST M - CNN - CR F Ġusing Ġthe Ġhyper - param eters Ġmentioned Ġin ĠTable ĠT AB REF 30 . ĠThe Ġtraining Ġand Ġevaluation Ġwas Ġdone Ġon Ġsentence - level . ĠThe ĠR NN Ġvariants Ġare Ġinitialized Ġrandomly Ġfrom Ġ$( - \ sq rt { k }, \ sq rt { k }) $ Ġwhere Ġ$ k = \ frac { 1 }{ hidden \ _ size } $. </s> First Ġwe Ġloaded Ġour Ġdataset Ġand Ġbuilt Ġvocabulary Ġusing Ġtorch text Ġlibrary . ĠThis Ġeased Ġour Ġprocess Ġof Ġdata Ġloading Ġusing Ġits ĠSequence T agging Dat as et Ġclass . ĠWe Ġtrained Ġour Ġmodel Ġwith Ġshuff led Ġtraining Ġset Ġusing ĠAdam Ġoptim izer Ġwith Ġhyper - param eters Ġmentioned Ġin Ġtable ĠT AB REF 30 . ĠAll Ġour Ġmodels Ġwere Ġtrained Ġon Ġsingle Ġlayer Ġof ĠL ST M Ġnetwork . ĠWe Ġfound Ġout ĠAdam Ġwas Ġgiving Ġbetter Ġperformance Ġand Ġfaster Ġconvergence Ġcompared Ġto ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ). ĠWe Ġchose Ġthose Ġhyper - param eters Ġafter Ġmany Ġab lation Ġstudies . ĠThe Ġdrop out Ġof Ġ0 . 5 Ġis Ġapplied Ġafter ĠL ST M Ġlayer . </s> For ĠCNN , Ġwe Ġused Ġ30 Ġdifferent Ġfilters Ġof Ġsizes Ġ3 , Ġ4 Ġand Ġ5 . ĠThe Ġembed d ings Ġof Ġeach Ġcharacter Ġor Ġgrap he me Ġinvolved Ġin Ġa Ġgiven Ġword , Ġwere Ġpassed Ġthrough Ġthe Ġpipeline Ġof ĠConv olution , ĠRect ified ĠLinear ĠUnit Ġand ĠMax - Pool ing . ĠThe Ġresulting Ġvectors Ġwere Ġconc aten ated Ġand Ġapplied Ġdrop out Ġof Ġ0 . 5 Ġbefore Ġpassing Ġinto Ġlinear Ġlayer Ġto Ġobtain Ġthe Ġembed ding Ġsize Ġof Ġ30 Ġfor Ġthe Ġgiven Ġword . ĠThis Ġresulting Ġembed ding Ġis Ġconc aten ated Ġwith Ġword Ġembed d ings , Ġwhich Ġis Ġagain Ġconc aten ated Ġwith Ġone - hot ĠPOS Ġvector . </s> Exper iments Ġ: :: ĠT agging ĠScheme </s> Currently , Ġfor Ġour Ġexperiments Ġwe Ġtrained Ġour Ġmodel Ġon ĠIO Ġ( Inside , ĠOutside ) Ġformat Ġfor Ġboth Ġthe Ġdataset , Ġhence Ġthe Ġdataset Ġdoes Ġnot Ġcontain Ġany ĠB - type Ġannotation Ġunlike Ġin ĠB IO Ġ( Beginning , ĠInside , ĠOutside ) Ġscheme . </s> Exper iments Ġ: :: ĠEarly ĠSto pping </s> We Ġused Ġsimple Ġearly Ġstopping Ġtechnique Ġwhere Ġif Ġthe Ġvalidation Ġloss Ġdoes Ġnot Ġdecrease Ġafter Ġ10 Ġepoch s , Ġthe Ġtraining Ġwas Ġstopped , Ġelse Ġthe Ġtraining Ġwill Ġrun Ġupt o Ġ100 Ġepoch s . ĠIn Ġour Ġexperience , Ġtraining Ġusually Ġstops Ġaround Ġ30 - 50 Ġepoch s . </s> Exper iments Ġ: :: ĠHyper - param eters ĠTun ing </s> We Ġran Ġour Ġexperiment Ġlooking Ġfor Ġthe Ġbest Ġhyper - param eters Ġby Ġchanging Ġlearning Ġrate Ġfrom Ġ( 0 , 1 , Ġ0 . 01 , Ġ0 . 001 , Ġ0 . 0001 ), Ġweight Ġdecay Ġfrom Ġ[ $ 10 ^{ - 1 } $, Ġ$ 10 ^{ - 2 } $, Ġ$ 10 ^{ - 3 } $, Ġ$ 10 ^{ - 4 } $, Ġ$ 10 ^{ - 5 } $, Ġ$ 10 ^{ - 6 } $, Ġ$ 10 ^{ - 7 } $ ], Ġbatch Ġsize Ġfrom Ġ[ 1 , Ġ2 , Ġ4 , Ġ8 , Ġ16 , Ġ32 , Ġ64 , Ġ128 ], Ġhidden Ġsize Ġfrom Ġ[ 8 , Ġ16 , Ġ32 , Ġ64 , Ġ128 , Ġ256 , Ġ512 Ġ1024 ]. ĠTable ĠT AB REF 30 Ġshows Ġall Ġother Ġhyper - param eter Ġused Ġin Ġour Ġexperiment Ġfor Ġboth Ġof Ġthe Ġdataset . </s> Exper iments Ġ: :: ĠEffect Ġof ĠDrop out </s> Figure ĠFIG REF 31 Ġshows Ġhow Ġwe Ġend Ġup Ġchoosing Ġ0 . 5 Ġas Ġdrop out Ġrate . ĠWhen Ġthe Ġdrop out Ġlayer Ġwas Ġnot Ġused , Ġthe ĠF 1 Ġscore Ġare Ġat Ġthe Ġlowest . ĠAs , Ġwe Ġslowly Ġincrease Ġthe Ġdrop out Ġrate , Ġthe ĠF 1 Ġscore Ġalso Ġgradually Ġincreases , Ġhowever Ġafter Ġdrop out Ġrate Ġ= Ġ0 . 5 , Ġthe ĠF 1 Ġscore Ġstarts Ġfalling Ġdown . ĠTherefore , Ġwe Ġhave Ġchosen Ġ0 . 5 Ġas Ġdrop out Ġrate Ġfor Ġall Ġother Ġexperiments Ġperformed . </s> E val uation </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġdetails Ġregarding Ġevaluation Ġand Ġcomparison Ġof Ġour Ġmodels Ġwith Ġother Ġbas elines . </s> Table ĠT AB REF 25 Ġshows Ġthe Ġstudy Ġof Ġvarious Ġembed d ings Ġand Ġcomparison Ġamong Ġeach Ġother Ġin ĠOur N ep ali Ġdataset . ĠHere , Ġraw Ġdataset Ġrepresents Ġsuch Ġdataset Ġwhere Ġpost - pos itions Ġare Ġnot Ġle mm at ized . ĠWe Ġcan Ġobserve Ġthat Ġpre - trained Ġembed d ings Ġsignificantly Ġimproves Ġthe Ġscore Ġcompared Ġto Ġrandomly Ġinitialized Ġembed ding . ĠWe Ġcan Ġded uce Ġthat ĠSkip ĠGram Ġmodels Ġperform Ġbetter Ġcompared ĠCB OW Ġmodels Ġfor Ġword 2 vec Ġand Ġfast text . ĠHere , Ġfast Text _ P ret rained Ġrepresents Ġthe Ġembed ding Ġreadily Ġavailable Ġin Ġfast Text Ġwebsite , Ġwhile Ġother Ġembed d ings Ġare Ġtrained Ġon Ġthe ĠNep ali ĠNational ĠCorpus Ġas Ġmentioned Ġin Ġsub - section ĠSEC REF 11 . ĠFrom Ġthis Ġtable ĠT AB REF 25 , Ġwe Ġcan Ġclearly Ġobserve Ġthat Ġmodel Ġusing Ġfast Text _ Skip ĠGram Ġembed d ings Ġoutper forms Ġall Ġother Ġmodels . </s> Table ĠT AB REF 35 Ġshows Ġthe Ġmodel Ġarchitecture Ġcomparison Ġbetween Ġall Ġthe Ġmodels Ġexperimented . ĠThe Ġfeatures Ġused Ġfor ĠStanford ĠCR F Ġclass ifier Ġare Ġwords , Ġletter Ġn - gram s Ġof Ġupt o Ġlength Ġ6 , Ġprevious Ġword Ġand Ġnext Ġword . ĠThis Ġmodel Ġis Ġtrained Ġtill Ġthe Ġcurrent Ġfunction Ġvalue Ġis Ġless Ġthan Ġ$ 1 \ math rm Ġ{ e }{ - 2 } $. ĠThe Ġhyper - param eters Ġof Ġneural Ġnetwork Ġexperiments Ġare Ġset Ġas Ġshown Ġin Ġtable ĠT AB REF 30 . ĠSince , Ġword Ġembed ding Ġof Ġcharacter - level Ġand Ġgrap he me - level Ġis Ġrandom , Ġtheir Ġscores Ġare Ġnear . </s> All Ġmodels Ġare Ġevaluated Ġusing ĠCo N LL - 2003 Ġevaluation Ġscript BI B REF 25 Ġto Ġcalculate Ġentity - wise Ġprecision , Ġrecall Ġand Ġf 1 Ġscore . </s> Discussion </s> In Ġthis Ġpaper Ġwe Ġpresent Ġthat Ġwe Ġcan Ġexploit Ġthe Ġpower Ġof Ġneural Ġnetwork Ġto Ġtrain Ġthe Ġmodel Ġto Ġperform Ġdownstream ĠN LP Ġtasks Ġlike ĠNamed ĠEntity ĠRecogn ition Ġeven Ġin ĠNep ali Ġlanguage . ĠWe Ġshowed Ġthat Ġthe Ġword Ġvectors Ġlearned Ġthrough Ġfast text Ġskip Ġgram Ġmodel Ġperforms Ġbetter Ġthan Ġother Ġword Ġembed ding Ġbecause Ġof Ġits Ġcapability Ġto Ġrepresent Ġsub - word Ġand Ġthis Ġis Ġparticularly Ġimportant Ġto Ġcapture Ġmorph ological Ġstructure Ġof Ġwords Ġand Ġsentences Ġin Ġhighly Ġinf lection al Ġlike ĠNep ali . ĠThis Ġconcept Ġcan Ġcome Ġhandy Ġin Ġother ĠDev an ag ari Ġlanguages Ġas Ġwell Ġbecause Ġthe Ġwritten Ġscripts Ġhave Ġsimilar Ġsynt act ical Ġstructure . </s> We Ġalso Ġfound Ġout Ġthat Ġstemming Ġpost - pos itions Ġcan Ġhelp Ġa Ġlot Ġin Ġimproving Ġmodel Ġperformance Ġbecause Ġof Ġinf lection al Ġcharacteristics Ġof ĠNep ali Ġlanguage . ĠSo Ġwhen Ġwe Ġseparate Ġout Ġits Ġinf lections Ġor Ġmorp hem es , Ġwe Ġcan Ġminimize Ġthe Ġvariations Ġof Ġsame Ġword Ġwhich Ġgives Ġits Ġroot Ġword Ġa Ġstronger Ġword Ġvector Ġrepresentations Ġcompared Ġto Ġits Ġinf lected Ġversions . </s> We Ġcan Ġclearly Ġimply Ġfrom Ġtables ĠT AB REF 23 , ĠT AB REF 24 , Ġand ĠT AB REF 35 Ġthat Ġwe Ġneed Ġmore Ġdata Ġto Ġget Ġbetter Ġresults Ġbecause ĠOur N ep ali Ġdataset Ġvolume Ġis Ġalmost Ġten Ġtimes Ġbigger Ġcompared Ġto ĠIL PR L Ġdataset Ġin Ġterms Ġof Ġentities . </s> Conclusion Ġand ĠFuture Ġwork </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnovel ĠN ER Ġfor ĠNep ali Ġlanguage Ġand Ġachieved Ġrelative Ġimprovement Ġof Ġupt o Ġ10 % Ġand Ġstudies Ġdifferent Ġfactors Ġeffect ing Ġthe Ġperformance Ġof Ġthe ĠN ER Ġfor ĠNep ali Ġlanguage . </s> We Ġalso Ġpresent Ġa Ġneural Ġarchitecture ĠBi L ST M + CNN ( g rap he me - level ) Ġwhich Ġturns Ġout Ġto Ġbe Ġperforming Ġon Ġpar Ġwith ĠBi L ST M + CNN ( character - level ) Ġunder Ġthe Ġsame Ġconfiguration . ĠWe Ġbelieve Ġthis Ġwill Ġnot Ġonly Ġhelp ĠNep ali Ġlanguage Ġbut Ġalso Ġother Ġlanguages Ġfalling Ġunder Ġthe Ġumb rell as Ġof ĠDev an ag ari Ġlanguages . ĠOur Ġmodel ĠBi L ST M + CNN ( g rap he me - level ) Ġand ĠBi L ST M + CNN ( G )+ POS Ġoutper forms Ġall Ġother Ġmodel Ġexperimented Ġin ĠOur N ep ali Ġand ĠIL PR L Ġdataset Ġrespectively . </s> Since Ġthis Ġis Ġthe Ġfirst Ġnamed Ġentity Ġrecognition Ġresearch Ġin ĠNepal Ġlanguage Ġusing Ġneural Ġnetwork , Ġthere Ġare Ġmany Ġrooms Ġfor Ġimprovement . ĠWe Ġbelieve Ġinitial izing Ġthe Ġgrap he me - level Ġembed ding Ġwith Ġfast text Ġembed d ings Ġmight Ġhelp Ġboosting Ġthe Ġperformance , Ġrather Ġthan Ġrandomly Ġinitial izing Ġit . ĠIn Ġfuture , Ġwe Ġplan Ġto Ġapply Ġother Ġlatest Ġtechniques Ġlike ĠB ERT , ĠEL Mo Ġand ĠFL AIR Ġto Ġstudy Ġits Ġeffect Ġon Ġlow - resource Ġlanguage Ġlike ĠNep ali . ĠWe Ġalso Ġplan Ġto Ġimprove Ġthe Ġmodel Ġusing Ġcross - ling ual Ġor Ġmulti - ling ual Ġparameter Ġsharing Ġtechniques Ġby Ġjointly Ġtraining Ġwith Ġother ĠDev an ag ari Ġlanguages Ġlike ĠHindi Ġand ĠBeng ali . </s> Finally , Ġwe Ġwould Ġlike Ġto Ġcontribute Ġour Ġdataset Ġto ĠNep ali ĠN LP Ġcommunity Ġto Ġmove Ġforward Ġthe Ġresearch Ġgoing Ġon Ġin Ġlanguage Ġunderstanding Ġdomain . ĠWe Ġbelieve Ġthere Ġshould Ġbe Ġspecial Ġcommittee Ġto Ġcreate Ġand Ġmaintain Ġsuch Ġdataset Ġfor ĠNep ali ĠN LP Ġand Ġorganize Ġvarious Ġcompetitions Ġwhich Ġwould Ġelevate Ġthe ĠN LP Ġresearch Ġin ĠNepal . </s> Some Ġof Ġthe Ġfuture Ġworks Ġare Ġlisted Ġbelow : </s> Pro per Ġinitialization Ġof Ġgrap he me Ġlevel Ġembed ding Ġfrom Ġfast text Ġembed d ings . </s> Apply Ġrobust ĠPOS - t agger Ġfor ĠNep ali Ġdataset </s> L em mat ize Ġthe ĠOur N ep ali Ġdataset Ġwith Ġrobust Ġand Ġefficient Ġle mm at izer </s> Improve ĠNep ali Ġlanguage Ġscore Ġwith Ġcross - ling ual Ġlearning Ġtechniques </s> Create Ġmore Ġdataset Ġusing ĠWikipedia / Wik id ata Ġframework </s> Acknowled gments </s> The Ġauthors Ġof Ġthis Ġpaper Ġwould Ġlike Ġto Ġexpress Ġsincere Ġthanks Ġto ĠBal ĠKrishna ĠBal , ĠKath mand u ĠUniversity ĠProfessor Ġfor Ġproviding Ġus Ġthe ĠPOS - tag ged ĠNep ali ĠN ER Ġdata .
Introduction Named Entity Recognition (NER) is a foremost NLP task to label each atomic elements of a sentence into specific categories like "PERSON", "LOCATION", "ORGANIZATION" and othersBIBREF0. There has been an extensive NER research on English, German, Dutch and Spanish language BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, and notable research on low resource South Asian languages like HindiBIBREF6, IndonesianBIBREF7 and other Indian languages (Kannada, Malayalam, Tamil and Telugu)BIBREF8. However, there has been no study on developing neural NER for Nepali language. In this paper, we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing. Recent neural architecture like BIBREF1 is used to relax the need to hand-craft the features and need to use part-of-speech tag to determine the category of the entity. However, this architecture have been studied for languages like English, and German and not been applied to languages like Nepali which is a low resource language i.e limited data set to train the model. Traditional methods like Hidden Markov Model (HMM) with rule based approachesBIBREF9,BIBREF10, and Support Vector Machine (SVM) with manual feature-engineeringBIBREF11 have been applied but they perform poor compared to neural. However, there has been no research in Nepali NER using neural network. Therefore, we created the named entity annotated dataset partly with the help of Dataturk to train a neural model. The texts used for this dataset are collected from various daily news sources from Nepal around the year 2015-2016. Following are our contributions: We present a novel Named Entity Recognizer (NER) for Nepali language. To best of our knowledge we are the first to propose neural based Nepali NER. As there are not good quality dataset to train NER we release a dataset to support future research We perform empirical evaluation of our model with state-of-the-art models with relative improvement of upto 10% In this paper, we present works similar to ours in Section SECREF2. We describe our approach and dataset statistics in Section SECREF3 and SECREF4, followed by our experiments, evaluation and discussion in Section SECREF5, SECREF6, and SECREF7. We conclude with our observations in Section SECREF8. To facilitate further research our code and dataset will be made available at github.com/link-yet-to-be-updated Related Work There has been a handful of research on Nepali NER task based on approaches like Support Vector Machine and gazetteer listBIBREF11 and Hidden Markov Model and gazetteer listBIBREF9,BIBREF10. BIBREF11 uses SVM along with features like first word, word length, digit features and gazetteer (person, organization, location, middle name, verb, designation and others). It uses one vs rest classification model to classify each word into different entity classes. However, it does not the take context word into account while training the model. Similarly, BIBREF9 and BIBREF10 uses Hidden Markov Model with n-gram technique for extracting POS-tags. POS-tags with common noun, proper noun or combination of both are combined together, then uses gazetteer list as look-up table to identify the named entities. Researchers have shown that the neural networks like CNNBIBREF12, RNNBIBREF13, LSTMBIBREF14, GRUBIBREF15 can capture the semantic knowledge of language better with the help of pre-trained embbeddings like word2vecBIBREF16, gloveBIBREF17 or fasttextBIBREF18. Similar approaches has been applied to many South Asian languages like HindiBIBREF6, IndonesianBIBREF7, BengaliBIBREF19 and In this paper, we present the neural network architecture for NER task in Nepali language, which doesn't require any manual feature engineering nor any data pre-processing during training. First we are comparing BiLSTMBIBREF14, BiLSTM+CNNBIBREF20, BiLSTM+CRFBIBREF1, BiLSTM+CNN+CRFBIBREF2 models with CNN modelBIBREF0 and Stanford CRF modelBIBREF21. Secondly, we show the comparison between models trained on general word embeddings, word embedding + character-level embedding, word embedding + part-of-speech(POS) one-hot encoding and word embedding + grapheme clustered or sub-word embeddingBIBREF22. The experiments were performed on the dataset that we created and on the dataset received from ILPRL lab. Our extensive study shows that augmenting word embedding with character or grapheme-level representation and POS one-hot encoding vector yields better results compared to using general word embedding alone. Approach In this section, we describe our approach in building our model. This model is partly inspired from multiple models BIBREF20,BIBREF1, andBIBREF2 Approach ::: Bidirectional LSTM We used Bi-directional LSTM to capture the word representation in forward as well as reverse direction of a sentence. Generally, LSTMs take inputs from left (past) of the sentence and computes the hidden state. However, it is proven beneficialBIBREF23 to use bi-directional LSTM, where, hidden states are computed based from right (future) of sentence and both of these hidden states are concatenated to produce the final output as $h_t$=[$\overrightarrow{h_t}$;$\overleftarrow{h_t}$], where $\overrightarrow{h_t}$, $\overleftarrow{h_t}$ = hidden state computed in forward and backward direction respectively. Approach ::: Features ::: Word embeddings We have used Word2Vec BIBREF16, GloVe BIBREF17 and FastText BIBREF18 word vectors of 300 dimensions. These vectors were trained on the corpus obtained from Nepali National Corpus. This pre-lemmatized corpus consists of 14 million words from books, web-texts and news papers. This corpus was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec using gensim libraryBIBREF24. This trained model consists of vectors for 72782 unique words. Light pre-processing was performed on the corpus before training it. For example, invalid characters or characters other than Devanagari were removed but punctuation and numbers were not removed. We set the window context at 10 and the rare words whose count is below 5 are dropped. These word embeddings were not frozen during the training session because fine-tuning word embedding help achieve better performance compared to frozen oneBIBREF20. We have used fasttext embeddings in particular because of its sub-word representation ability, which is very useful in highly inflectional language as shown in Table TABREF25. We have trained the word embedding in such a way that the sub-word size remains between 1 and 4. We particularly chose this size because in Nepali language a single letter can also be a word, for example e, t, C, r, l, n, u and a single character (grapheme) or sub-word can be formed after mixture of dependent vowel signs with consonant letters for example, C + O + = CO, here three different consonant letters form a single sub-word. The two-dimensional visualization of an example word npAl is shown in FIGREF14. Principal Component Analysis (PCA) technique was used to generate this visualization which helps use to analyze the nearest neighbor words of a given sample word. 84 and 104 nearest neighbors were observed using word2vec and fasttext embedding respectively on the same corpus. Approach ::: Features ::: Character-level embeddings BIBREF20 and BIBREF2 successfully presented that the character-level embeddings, extracted using CNN, when combined with word embeddings enhances the NER model performance significantly, as it is able to capture morphological features of a word. Figure FIGREF7 shows the grapheme-level CNN used in our model, where inputs to CNN are graphemes. Character-level CNN is also built in similar fashion, except the inputs are characters. Grapheme or Character -level embeddings are randomly initialized from [0,1] with real values with uniform distribution of dimension 30. Approach ::: Features ::: Grapheme-level embeddings Grapheme is atomic meaningful unit in writing system of any languages. Since, Nepali language is highly morphologically inflectional, we compared grapheme-level representation with character-level representation to evaluate its effect. For example, in character-level embedding, each character of a word npAl results into n + + p + A + l has its own embedding. However, in grapheme level, a word npAl is clustered into graphemes, resulting into n + pA + l. Here, each grapheme has its own embedding. This grapheme-level embedding results good scores on par with character-level embedding in highly inflectional languages like Nepali, because graphemes also capture syntactic information similar to characters. We created grapheme clusters using uniseg package which is helpful in unicode text segmentations. Approach ::: Features ::: Part-of-speech (POS) one hot encoding We created one-hot encoded vector of POS tags and then concatenated with pre-trained word embeddings before passing it to BiLSTM network. A sample of data is shown in figure FIGREF13. Dataset Statistics ::: OurNepali dataset Since, we there was no publicly available standard Nepali NER dataset and did not receive any dataset from the previous researchers, we had to create our own dataset. This dataset contains the sentences collected from daily newspaper of the year 2015-2016. This dataset has three major classes Person (PER), Location (LOC) and Organization (ORG). Pre-processing was performed on the text before creation of the dataset, for example all punctuations and numbers besides ',', '-', '|' and '.' were removed. Currently, the dataset is in standard CoNLL-2003 IO formatBIBREF25. Since, this dataset is not lemmatized originally, we lemmatized only the post-positions like Ek, kO, l, mA, m, my, jF, sg, aEG which are just the few examples among 299 post positions in Nepali language. We obtained these post-positions from sanjaalcorps and added few more to match our dataset. We will be releasing this list in our github repository. We found out that lemmatizing the post-positions boosted the F1 score by almost 10%. In order to label our dataset with POS-tags, we first created POS annotated dataset of 6946 sentences and 16225 unique words extracted from POS-tagged Nepali National Corpus and trained a BiLSTM model with 95.14% accuracy which was used to create POS-tags for our dataset. The dataset released in our github repository contains each word in newline with space separated POS-tags and Entity-tags. The sentences are separated by empty newline. A sample sentence from the dataset is presented in table FIGREF13. Dataset Statistics ::: ILPRL dataset After much time, we received the dataset from Bal Krishna Bal, ILPRL, KU. This dataset follows standard CoNLL-2003 IOB formatBIBREF25 with POS tags. This dataset is prepared by ILPRL Lab, KU and KEIV Technologies. Few corrections like correcting the NER tags had to be made on the dataset. The statistics of both the dataset is presented in table TABREF23. Table TABREF24 presents the total entities (PER, LOC, ORG and MISC) from both of the dataset used in our experiments. The dataset is divided into three parts with 64%, 16% and 20% of the total dataset into training set, development set and test set respectively. Experiments In this section, we present the details about training our neural network. The neural network architecture are implemented using PyTorch framework BIBREF26. The training is performed on a single Nvidia Tesla P100 SXM2. We first run our experiment on BiLSTM, BiLSTM-CNN, BiLSTM-CRF BiLSTM-CNN-CRF using the hyper-parameters mentioned in Table TABREF30. The training and evaluation was done on sentence-level. The RNN variants are initialized randomly from $(-\sqrt{k},\sqrt{k})$ where $k=\frac{1}{hidden\_size}$. First we loaded our dataset and built vocabulary using torchtext library. This eased our process of data loading using its SequenceTaggingDataset class. We trained our model with shuffled training set using Adam optimizer with hyper-parameters mentioned in table TABREF30. All our models were trained on single layer of LSTM network. We found out Adam was giving better performance and faster convergence compared to Stochastic Gradient Descent (SGD). We chose those hyper-parameters after many ablation studies. The dropout of 0.5 is applied after LSTM layer. For CNN, we used 30 different filters of sizes 3, 4 and 5. The embeddings of each character or grapheme involved in a given word, were passed through the pipeline of Convolution, Rectified Linear Unit and Max-Pooling. The resulting vectors were concatenated and applied dropout of 0.5 before passing into linear layer to obtain the embedding size of 30 for the given word. This resulting embedding is concatenated with word embeddings, which is again concatenated with one-hot POS vector. Experiments ::: Tagging Scheme Currently, for our experiments we trained our model on IO (Inside, Outside) format for both the dataset, hence the dataset does not contain any B-type annotation unlike in BIO (Beginning, Inside, Outside) scheme. Experiments ::: Early Stopping We used simple early stopping technique where if the validation loss does not decrease after 10 epochs, the training was stopped, else the training will run upto 100 epochs. In our experience, training usually stops around 30-50 epochs. Experiments ::: Hyper-parameters Tuning We ran our experiment looking for the best hyper-parameters by changing learning rate from (0,1, 0.01, 0.001, 0.0001), weight decay from [$10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$], batch size from [1, 2, 4, 8, 16, 32, 64, 128], hidden size from [8, 16, 32, 64, 128, 256, 512 1024]. Table TABREF30 shows all other hyper-parameter used in our experiment for both of the dataset. Experiments ::: Effect of Dropout Figure FIGREF31 shows how we end up choosing 0.5 as dropout rate. When the dropout layer was not used, the F1 score are at the lowest. As, we slowly increase the dropout rate, the F1 score also gradually increases, however after dropout rate = 0.5, the F1 score starts falling down. Therefore, we have chosen 0.5 as dropout rate for all other experiments performed. Evaluation In this section, we present the details regarding evaluation and comparison of our models with other baselines. Table TABREF25 shows the study of various embeddings and comparison among each other in OurNepali dataset. Here, raw dataset represents such dataset where post-positions are not lemmatized. We can observe that pre-trained embeddings significantly improves the score compared to randomly initialized embedding. We can deduce that Skip Gram models perform better compared CBOW models for word2vec and fasttext. Here, fastText_Pretrained represents the embedding readily available in fastText website, while other embeddings are trained on the Nepali National Corpus as mentioned in sub-section SECREF11. From this table TABREF25, we can clearly observe that model using fastText_Skip Gram embeddings outperforms all other models. Table TABREF35 shows the model architecture comparison between all the models experimented. The features used for Stanford CRF classifier are words, letter n-grams of upto length 6, previous word and next word. This model is trained till the current function value is less than $1\mathrm {e}{-2}$. The hyper-parameters of neural network experiments are set as shown in table TABREF30. Since, word embedding of character-level and grapheme-level is random, their scores are near. All models are evaluated using CoNLL-2003 evaluation scriptBIBREF25 to calculate entity-wise precision, recall and f1 score. Discussion In this paper we present that we can exploit the power of neural network to train the model to perform downstream NLP tasks like Named Entity Recognition even in Nepali language. We showed that the word vectors learned through fasttext skip gram model performs better than other word embedding because of its capability to represent sub-word and this is particularly important to capture morphological structure of words and sentences in highly inflectional like Nepali. This concept can come handy in other Devanagari languages as well because the written scripts have similar syntactical structure. We also found out that stemming post-positions can help a lot in improving model performance because of inflectional characteristics of Nepali language. So when we separate out its inflections or morphemes, we can minimize the variations of same word which gives its root word a stronger word vector representations compared to its inflected versions. We can clearly imply from tables TABREF23, TABREF24, and TABREF35 that we need more data to get better results because OurNepali dataset volume is almost ten times bigger compared to ILPRL dataset in terms of entities. Conclusion and Future work In this paper, we proposed a novel NER for Nepali language and achieved relative improvement of upto 10% and studies different factors effecting the performance of the NER for Nepali language. We also present a neural architecture BiLSTM+CNN(grapheme-level) which turns out to be performing on par with BiLSTM+CNN(character-level) under the same configuration. We believe this will not only help Nepali language but also other languages falling under the umbrellas of Devanagari languages. Our model BiLSTM+CNN(grapheme-level) and BiLSTM+CNN(G)+POS outperforms all other model experimented in OurNepali and ILPRL dataset respectively. Since this is the first named entity recognition research in Nepal language using neural network, there are many rooms for improvement. We believe initializing the grapheme-level embedding with fasttext embeddings might help boosting the performance, rather than randomly initializing it. In future, we plan to apply other latest techniques like BERT, ELMo and FLAIR to study its effect on low-resource language like Nepali. We also plan to improve the model using cross-lingual or multi-lingual parameter sharing techniques by jointly training with other Devanagari languages like Hindi and Bengali. Finally, we would like to contribute our dataset to Nepali NLP community to move forward the research going on in language understanding domain. We believe there should be special committee to create and maintain such dataset for Nepali NLP and organize various competitions which would elevate the NLP research in Nepal. Some of the future works are listed below: Proper initialization of grapheme level embedding from fasttext embeddings. Apply robust POS-tagger for Nepali dataset Lemmatize the OurNepali dataset with robust and efficient lemmatizer Improve Nepali language score with cross-lingual learning techniques Create more dataset using Wikipedia/Wikidata framework Acknowledgments The authors of this paper would like to express sincere thanks to Bal Krishna Bal, Kathmandu University Professor for providing us the POS-tagged Nepali NER data.
Do they try to use byte-pair encoding representations?
1170e4ee76fa202cabac9f621e8fbeb4a6c5f094
Do Ġthey Ġtry Ġto Ġuse Ġbyte - pair Ġencoding Ġrepresentations ?
No
<s> Do Ġthey Ġtry Ġto Ġuse Ġbyte - pair Ġencoding Ġrepresentations ? </s> Introduction </s> N amed ĠEntity ĠRecogn ition Ġ( NER ) Ġis Ġa Ġforemost ĠN LP Ġtask Ġto Ġlabel Ġeach Ġatomic Ġelements Ġof Ġa Ġsentence Ġinto Ġspecific Ġcategories Ġlike Ġ" P ERSON ", Ġ" LOC ATION ", Ġ" OR GAN IZ ATION " Ġand Ġothers BI B REF 0 . ĠThere Ġhas Ġbeen Ġan Ġextensive ĠN ER Ġresearch Ġon ĠEnglish , ĠGerman , ĠDutch Ġand ĠSpanish Ġlanguage ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġand Ġnotable Ġresearch Ġon Ġlow Ġresource ĠSouth ĠAsian Ġlanguages Ġlike ĠHindi BI B REF 6 , ĠIndonesian BI B REF 7 Ġand Ġother ĠIndian Ġlanguages Ġ( K ann ada , ĠMal ay alam , ĠTamil Ġand ĠTel ugu ) BI B REF 8 . ĠHowever , Ġthere Ġhas Ġbeen Ġno Ġstudy Ġon Ġdeveloping Ġneural ĠN ER Ġfor ĠNep ali Ġlanguage . ĠIn Ġthis Ġpaper , Ġwe Ġpropose Ġa Ġneural Ġbased ĠNep ali ĠN ER Ġusing Ġlatest Ġstate - of - the - art Ġarchitecture Ġbased Ġon Ġgrap he me - level Ġwhich Ġdoesn 't Ġrequire Ġany Ġhand - crafted Ġfeatures Ġand Ġno Ġdata Ġpre - processing . </s> Recent Ġneural Ġarchitecture Ġlike ĠB IB REF 1 Ġis Ġused Ġto Ġrelax Ġthe Ġneed Ġto Ġhand - craft Ġthe Ġfeatures Ġand Ġneed Ġto Ġuse Ġpart - of - speech Ġtag Ġto Ġdetermine Ġthe Ġcategory Ġof Ġthe Ġentity . ĠHowever , Ġthis Ġarchitecture Ġhave Ġbeen Ġstudied Ġfor Ġlanguages Ġlike ĠEnglish , Ġand ĠGerman Ġand Ġnot Ġbeen Ġapplied Ġto Ġlanguages Ġlike ĠNep ali Ġwhich Ġis Ġa Ġlow Ġresource Ġlanguage Ġi . e Ġlimited Ġdata Ġset Ġto Ġtrain Ġthe Ġmodel . ĠTraditional Ġmethods Ġlike ĠHidden ĠMark ov ĠModel Ġ( H MM ) Ġwith Ġrule Ġbased Ġapproaches BI B REF 9 , BI B REF 10 , Ġand ĠSupport ĠVector ĠMachine Ġ( S VM ) Ġwith Ġmanual Ġfeature - engineering BI B REF 11 Ġhave Ġbeen Ġapplied Ġbut Ġthey Ġperform Ġpoor Ġcompared Ġto Ġneural . ĠHowever , Ġthere Ġhas Ġbeen Ġno Ġresearch Ġin ĠNep ali ĠN ER Ġusing Ġneural Ġnetwork . ĠTherefore , Ġwe Ġcreated Ġthe Ġnamed Ġentity Ġannot ated Ġdataset Ġpartly Ġwith Ġthe Ġhelp Ġof ĠDat atur k Ġto Ġtrain Ġa Ġneural Ġmodel . ĠThe Ġtexts Ġused Ġfor Ġthis Ġdataset Ġare Ġcollected Ġfrom Ġvarious Ġdaily Ġnews Ġsources Ġfrom ĠNepal Ġaround Ġthe Ġyear Ġ2015 - 2016 . </s> Following Ġare Ġour Ġcontributions : </s> We Ġpresent Ġa Ġnovel ĠNamed ĠEntity ĠRecogn izer Ġ( NER ) Ġfor ĠNep ali Ġlanguage . ĠTo Ġbest Ġof Ġour Ġknowledge Ġwe Ġare Ġthe Ġfirst Ġto Ġpropose Ġneural Ġbased ĠNep ali ĠN ER . </s> As Ġthere Ġare Ġnot Ġgood Ġquality Ġdataset Ġto Ġtrain ĠN ER Ġwe Ġrelease Ġa Ġdataset Ġto Ġsupport Ġfuture Ġresearch </s> We Ġperform Ġempirical Ġevaluation Ġof Ġour Ġmodel Ġwith Ġstate - of - the - art Ġmodels Ġwith Ġrelative Ġimprovement Ġof Ġupt o Ġ10 % </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġworks Ġsimilar Ġto Ġours Ġin ĠSection ĠSEC REF 2 . ĠWe Ġdescribe Ġour Ġapproach Ġand Ġdataset Ġstatistics Ġin ĠSection ĠSEC REF 3 Ġand ĠSEC REF 4 , Ġfollowed Ġby Ġour Ġexperiments , Ġevaluation Ġand Ġdiscussion Ġin ĠSection ĠSEC REF 5 , ĠSEC REF 6 , Ġand ĠSEC REF 7 . ĠWe Ġconclude Ġwith Ġour Ġobservations Ġin ĠSection ĠSEC REF 8 . </s> To Ġfacilitate Ġfurther Ġresearch Ġour Ġcode Ġand Ġdataset Ġwill Ġbe Ġmade Ġavailable Ġat Ġgithub . com / link - yet - to - be - updated </s> Related ĠWork </s> There Ġhas Ġbeen Ġa Ġhandful Ġof Ġresearch Ġon ĠNep ali ĠN ER Ġtask Ġbased Ġon Ġapproaches Ġlike ĠSupport ĠVector ĠMachine Ġand Ġg az ette er Ġlist BI B REF 11 Ġand ĠHidden ĠMark ov ĠModel Ġand Ġg az ette er Ġlist BI B REF 9 , BI B REF 10 . </s> BI B REF 11 Ġuses ĠS VM Ġalong Ġwith Ġfeatures Ġlike Ġfirst Ġword , Ġword Ġlength , Ġdigit Ġfeatures Ġand Ġg az ette er Ġ( person , Ġorganization , Ġlocation , Ġmiddle Ġname , Ġverb , Ġdesignation Ġand Ġothers ). ĠIt Ġuses Ġone Ġvs Ġrest Ġclassification Ġmodel Ġto Ġclassify Ġeach Ġword Ġinto Ġdifferent Ġentity Ġclasses . ĠHowever , Ġit Ġdoes Ġnot Ġthe Ġtake Ġcontext Ġword Ġinto Ġaccount Ġwhile Ġtraining Ġthe Ġmodel . ĠSimilarly , ĠB IB REF 9 Ġand ĠB IB REF 10 Ġuses ĠHidden ĠMark ov ĠModel Ġwith Ġn - gram Ġtechnique Ġfor Ġextracting ĠPOS - tags . ĠPOS - tags Ġwith Ġcommon Ġnoun , Ġproper Ġnoun Ġor Ġcombination Ġof Ġboth Ġare Ġcombined Ġtogether , Ġthen Ġuses Ġg az ette er Ġlist Ġas Ġlook - up Ġtable Ġto Ġidentify Ġthe Ġnamed Ġentities . </s> Researchers Ġhave Ġshown Ġthat Ġthe Ġneural Ġnetworks Ġlike ĠCNN BI B REF 12 , ĠR NN BI B REF 13 , ĠL ST M BI B REF 14 , ĠGR U BI B REF 15 Ġcan Ġcapture Ġthe Ġsemantic Ġknowledge Ġof Ġlanguage Ġbetter Ġwith Ġthe Ġhelp Ġof Ġpre - trained Ġemb bed d ings Ġlike Ġword 2 vec BI B REF 16 , Ġglove BI B REF 17 Ġor Ġfast text BI B REF 18 . </s> Similar Ġapproaches Ġhas Ġbeen Ġapplied Ġto Ġmany ĠSouth ĠAsian Ġlanguages Ġlike ĠHindi BI B REF 6 , ĠIndonesian BI B REF 7 , ĠBeng ali BI B REF 19 Ġand ĠIn Ġthis Ġpaper , Ġwe Ġpresent Ġthe Ġneural Ġnetwork Ġarchitecture Ġfor ĠN ER Ġtask Ġin ĠNep ali Ġlanguage , Ġwhich Ġdoesn 't Ġrequire Ġany Ġmanual Ġfeature Ġengineering Ġnor Ġany Ġdata Ġpre - processing Ġduring Ġtraining . ĠFirst Ġwe Ġare Ġcomparing ĠBi L ST M BI B REF 14 , ĠBi L ST M + CNN BI B REF 20 , ĠBi L ST M + CR FBI B REF 1 , ĠBi L ST M + CNN + CR FBI B REF 2 Ġmodels Ġwith ĠCNN Ġmodel BI B REF 0 Ġand ĠStanford ĠCR F Ġmodel BI B REF 21 . ĠSecondly , Ġwe Ġshow Ġthe Ġcomparison Ġbetween Ġmodels Ġtrained Ġon Ġgeneral Ġword Ġembed d ings , Ġword Ġembed ding Ġ+ Ġcharacter - level Ġembed ding , Ġword Ġembed ding Ġ+ Ġpart - of - speech ( POS ) Ġone - hot Ġencoding Ġand Ġword Ġembed ding Ġ+ Ġgrap he me Ġclustered Ġor Ġsub - word Ġembed ding BI B REF 22 . ĠThe Ġexperiments Ġwere Ġperformed Ġon Ġthe Ġdataset Ġthat Ġwe Ġcreated Ġand Ġon Ġthe Ġdataset Ġreceived Ġfrom ĠIL PR L Ġlab . ĠOur Ġextensive Ġstudy Ġshows Ġthat Ġaugment ing Ġword Ġembed ding Ġwith Ġcharacter Ġor Ġgrap he me - level Ġrepresentation Ġand ĠPOS Ġone - hot Ġencoding Ġvector Ġyields Ġbetter Ġresults Ġcompared Ġto Ġusing Ġgeneral Ġword Ġembed ding Ġalone . </s> App roach </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġapproach Ġin Ġbuilding Ġour Ġmodel . ĠThis Ġmodel Ġis Ġpartly Ġinspired Ġfrom Ġmultiple Ġmodels ĠB IB REF 20 , BI B REF 1 , Ġand BI B REF 2 </s> App roach Ġ: :: ĠBid irection al ĠL ST M </s> We Ġused ĠBi - direction al ĠL ST M Ġto Ġcapture Ġthe Ġword Ġrepresentation Ġin Ġforward Ġas Ġwell Ġas Ġreverse Ġdirection Ġof Ġa Ġsentence . ĠGenerally , ĠL ST Ms Ġtake Ġinputs Ġfrom Ġleft Ġ( past ) Ġof Ġthe Ġsentence Ġand Ġcomp utes Ġthe Ġhidden Ġstate . ĠHowever , Ġit Ġis Ġproven Ġbeneficial BI B REF 23 Ġto Ġuse Ġbi - direction al ĠL ST M , Ġwhere , Ġhidden Ġstates Ġare Ġcomputed Ġbased Ġfrom Ġright Ġ( future ) Ġof Ġsentence Ġand Ġboth Ġof Ġthese Ġhidden Ġstates Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġfinal Ġoutput Ġas Ġ$ h _ t $ =[ $ \ over right arrow { h _ t } $ ; $ \ over left arrow { h _ t } $ ], Ġwhere Ġ$\ over right arrow { h _ t } $, Ġ$\ over left arrow { h _ t } $ Ġ= Ġhidden Ġstate Ġcomputed Ġin Ġforward Ġand Ġbackward Ġdirection Ġrespectively . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠWord Ġembed d ings </s> We Ġhave Ġused ĠWord 2 V ec ĠB IB REF 16 , ĠGl o Ve ĠB IB REF 17 Ġand ĠFast Text ĠB IB REF 18 Ġword Ġvectors Ġof Ġ300 Ġdimensions . ĠThese Ġvectors Ġwere Ġtrained Ġon Ġthe Ġcorpus Ġobtained Ġfrom ĠNep ali ĠNational ĠCorpus . ĠThis Ġpre - le mm at ized Ġcorpus Ġconsists Ġof Ġ14 Ġmillion Ġwords Ġfrom Ġbooks , Ġweb - text s Ġand Ġnews Ġpapers . ĠThis Ġcorpus Ġwas Ġmixed Ġwith Ġthe Ġtexts Ġfrom Ġthe Ġdataset Ġbefore Ġtraining ĠCB OW Ġand Ġskip - gram Ġversion Ġof Ġword 2 vec Ġusing Ġg ens im Ġlibrary BI B REF 24 . ĠThis Ġtrained Ġmodel Ġconsists Ġof Ġvectors Ġfor Ġ7 27 82 Ġunique Ġwords . </s> Light Ġpre - processing Ġwas Ġperformed Ġon Ġthe Ġcorpus Ġbefore Ġtraining Ġit . ĠFor Ġexample , Ġinvalid Ġcharacters Ġor Ġcharacters Ġother Ġthan ĠDev an ag ari Ġwere Ġremoved Ġbut Ġpunct uation Ġand Ġnumbers Ġwere Ġnot Ġremoved . ĠWe Ġset Ġthe Ġwindow Ġcontext Ġat Ġ10 Ġand Ġthe Ġrare Ġwords Ġwhose Ġcount Ġis Ġbelow Ġ5 Ġare Ġdropped . ĠThese Ġword Ġembed d ings Ġwere Ġnot Ġfrozen Ġduring Ġthe Ġtraining Ġsession Ġbecause Ġfine - tun ing Ġword Ġembed ding Ġhelp Ġachieve Ġbetter Ġperformance Ġcompared Ġto Ġfrozen Ġone BI B REF 20 . </s> We Ġhave Ġused Ġfast text Ġembed d ings Ġin Ġparticular Ġbecause Ġof Ġits Ġsub - word Ġrepresentation Ġability , Ġwhich Ġis Ġvery Ġuseful Ġin Ġhighly Ġinf lection al Ġlanguage Ġas Ġshown Ġin ĠTable ĠT AB REF 25 . ĠWe Ġhave Ġtrained Ġthe Ġword Ġembed ding Ġin Ġsuch Ġa Ġway Ġthat Ġthe Ġsub - word Ġsize Ġremains Ġbetween Ġ1 Ġand Ġ4 . ĠWe Ġparticularly Ġchose Ġthis Ġsize Ġbecause Ġin ĠNep ali Ġlanguage Ġa Ġsingle Ġletter Ġcan Ġalso Ġbe Ġa Ġword , Ġfor Ġexample Ġe , Ġt , ĠC , Ġr , Ġl , Ġn , Ġu Ġand Ġa Ġsingle Ġcharacter Ġ( g rap he me ) Ġor Ġsub - word Ġcan Ġbe Ġformed Ġafter Ġmixture Ġof Ġdependent Ġvowel Ġsigns Ġwith Ġconson ant Ġletters Ġfor Ġexample , ĠC Ġ+ ĠO Ġ+ Ġ= ĠCO , Ġhere Ġthree Ġdifferent Ġconson ant Ġletters Ġform Ġa Ġsingle Ġsub - word . </s> The Ġtwo - dimensional Ġvisualization Ġof Ġan Ġexample Ġword Ġnp Al Ġis Ġshown Ġin ĠFIG REF 14 . ĠPrincipal ĠComponent ĠAnalysis Ġ( PC A ) Ġtechnique Ġwas Ġused Ġto Ġgenerate Ġthis Ġvisualization Ġwhich Ġhelps Ġuse Ġto Ġanalyze Ġthe Ġnearest Ġneighbor Ġwords Ġof Ġa Ġgiven Ġsample Ġword . Ġ84 Ġand Ġ104 Ġnearest Ġneighbors Ġwere Ġobserved Ġusing Ġword 2 vec Ġand Ġfast text Ġembed ding Ġrespectively Ġon Ġthe Ġsame Ġcorpus . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠCharacter - level Ġembed d ings </s> BI B REF 20 Ġand ĠB IB REF 2 Ġsuccessfully Ġpresented Ġthat Ġthe Ġcharacter - level Ġembed d ings , Ġextracted Ġusing ĠCNN , Ġwhen Ġcombined Ġwith Ġword Ġembed d ings Ġenhances Ġthe ĠN ER Ġmodel Ġperformance Ġsignificantly , Ġas Ġit Ġis Ġable Ġto Ġcapture Ġmorph ological Ġfeatures Ġof Ġa Ġword . ĠFigure ĠFIG REF 7 Ġshows Ġthe Ġgrap he me - level ĠCNN Ġused Ġin Ġour Ġmodel , Ġwhere Ġinputs Ġto ĠCNN Ġare Ġgrap hem es . ĠCharacter - level ĠCNN Ġis Ġalso Ġbuilt Ġin Ġsimilar Ġfashion , Ġexcept Ġthe Ġinputs Ġare Ġcharacters . ĠG rap he me Ġor ĠCharacter Ġ- level Ġembed d ings Ġare Ġrandomly Ġinitialized Ġfrom Ġ[ 0 , 1 ] Ġwith Ġreal Ġvalues Ġwith Ġuniform Ġdistribution Ġof Ġdimension Ġ30 . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠG rap he me - level Ġembed d ings </s> G rap he me Ġis Ġatomic Ġmeaningful Ġunit Ġin Ġwriting Ġsystem Ġof Ġany Ġlanguages . ĠSince , ĠNep ali Ġlanguage Ġis Ġhighly Ġmorph ologically Ġinf lection al , Ġwe Ġcompared Ġgrap he me - level Ġrepresentation Ġwith Ġcharacter - level Ġrepresentation Ġto Ġevaluate Ġits Ġeffect . ĠFor Ġexample , Ġin Ġcharacter - level Ġembed ding , Ġeach Ġcharacter Ġof Ġa Ġword Ġnp Al Ġresults Ġinto Ġn Ġ+ Ġ+ Ġp Ġ+ ĠA Ġ+ Ġl Ġhas Ġits Ġown Ġembed ding . ĠHowever , Ġin Ġgrap he me Ġlevel , Ġa Ġword Ġnp Al Ġis Ġclustered Ġinto Ġgrap hem es , Ġresulting Ġinto Ġn Ġ+ Ġp A Ġ+ Ġl . ĠHere , Ġeach Ġgrap he me Ġhas Ġits Ġown Ġembed ding . ĠThis Ġgrap he me - level Ġembed ding Ġresults Ġgood Ġscores Ġon Ġpar Ġwith Ġcharacter - level Ġembed ding Ġin Ġhighly Ġinf lection al Ġlanguages Ġlike ĠNep ali , Ġbecause Ġgrap hem es Ġalso Ġcapture Ġsynt actic Ġinformation Ġsimilar Ġto Ġcharacters . ĠWe Ġcreated Ġgrap he me Ġclusters Ġusing Ġun ise g Ġpackage Ġwhich Ġis Ġhelpful Ġin Ġunic ode Ġtext Ġsegment ations . </s> App roach Ġ: :: ĠFeatures Ġ: :: ĠPart - of - speech Ġ( POS ) Ġone Ġhot Ġencoding </s> We Ġcreated Ġone - hot Ġencoded Ġvector Ġof ĠPOS Ġtags Ġand Ġthen Ġconc aten ated Ġwith Ġpre - trained Ġword Ġembed d ings Ġbefore Ġpassing Ġit Ġto ĠBi L ST M Ġnetwork . ĠA Ġsample Ġof Ġdata Ġis Ġshown Ġin Ġfigure ĠFIG REF 13 . </s> Dat as et ĠStatistics Ġ: :: ĠOur N ep ali Ġdataset </s> Since , Ġwe Ġthere Ġwas Ġno Ġpublicly Ġavailable Ġstandard ĠNep ali ĠN ER Ġdataset Ġand Ġdid Ġnot Ġreceive Ġany Ġdataset Ġfrom Ġthe Ġprevious Ġresearchers , Ġwe Ġhad Ġto Ġcreate Ġour Ġown Ġdataset . ĠThis Ġdataset Ġcontains Ġthe Ġsentences Ġcollected Ġfrom Ġdaily Ġnewspaper Ġof Ġthe Ġyear Ġ2015 - 2016 . ĠThis Ġdataset Ġhas Ġthree Ġmajor Ġclasses ĠPerson Ġ( PER ), ĠLocation Ġ( LOC ) Ġand ĠOrganization Ġ( OR G ). ĠPre - processing Ġwas Ġperformed Ġon Ġthe Ġtext Ġbefore Ġcreation Ġof Ġthe Ġdataset , Ġfor Ġexample Ġall Ġpunct uations Ġand Ġnumbers Ġbesides Ġ', ', Ġ' - ', Ġ' | ' Ġand Ġ' .' Ġwere Ġremoved . ĠCurrently , Ġthe Ġdataset Ġis Ġin Ġstandard ĠCo N LL - 2003 ĠIO Ġformat BI B REF 25 . </s> Since , Ġthis Ġdataset Ġis Ġnot Ġle mm at ized Ġoriginally , Ġwe Ġle mm at ized Ġonly Ġthe Ġpost - pos itions Ġlike ĠEk , Ġk O , Ġl , Ġm A , Ġm , Ġmy , Ġj F , Ġs g , Ġa EG Ġwhich Ġare Ġjust Ġthe Ġfew Ġexamples Ġamong Ġ299 Ġpost Ġpositions Ġin ĠNep ali Ġlanguage . ĠWe Ġobtained Ġthese Ġpost - pos itions Ġfrom Ġsan ja al cor ps Ġand Ġadded Ġfew Ġmore Ġto Ġmatch Ġour Ġdataset . ĠWe Ġwill Ġbe Ġreleasing Ġthis Ġlist Ġin Ġour Ġgithub Ġrepository . ĠWe Ġfound Ġout Ġthat Ġle mm at izing Ġthe Ġpost - pos itions Ġboosted Ġthe ĠF 1 Ġscore Ġby Ġalmost Ġ10 %. </s> In Ġorder Ġto Ġlabel Ġour Ġdataset Ġwith ĠPOS - tags , Ġwe Ġfirst Ġcreated ĠPOS Ġannot ated Ġdataset Ġof Ġ69 46 Ġsentences Ġand Ġ16 225 Ġunique Ġwords Ġextracted Ġfrom ĠPOS - tag ged ĠNep ali ĠNational ĠCorpus Ġand Ġtrained Ġa ĠBi L ST M Ġmodel Ġwith Ġ95 . 14 % Ġaccuracy Ġwhich Ġwas Ġused Ġto Ġcreate ĠPOS - tags Ġfor Ġour Ġdataset . </s> The Ġdataset Ġreleased Ġin Ġour Ġgithub Ġrepository Ġcontains Ġeach Ġword Ġin Ġnew line Ġwith Ġspace Ġseparated ĠPOS - tags Ġand ĠEntity - tags . ĠThe Ġsentences Ġare Ġseparated Ġby Ġempty Ġnew line . ĠA Ġsample Ġsentence Ġfrom Ġthe Ġdataset Ġis Ġpresented Ġin Ġtable ĠFIG REF 13 . </s> Dat as et ĠStatistics Ġ: :: ĠIL PR L Ġdataset </s> After Ġmuch Ġtime , Ġwe Ġreceived Ġthe Ġdataset Ġfrom ĠBal ĠKrishna ĠBal , ĠIL PR L , ĠK U . ĠThis Ġdataset Ġfollows Ġstandard ĠCo N LL - 2003 ĠI OB Ġformat BI B REF 25 Ġwith ĠPOS Ġtags . ĠThis Ġdataset Ġis Ġprepared Ġby ĠIL PR L ĠLab , ĠK U Ġand ĠKE IV ĠTechnologies . ĠFew Ġcorrections Ġlike Ġcorrecting Ġthe ĠN ER Ġtags Ġhad Ġto Ġbe Ġmade Ġon Ġthe Ġdataset . ĠThe Ġstatistics Ġof Ġboth Ġthe Ġdataset Ġis Ġpresented Ġin Ġtable ĠT AB REF 23 . </s> Table ĠT AB REF 24 Ġpresents Ġthe Ġtotal Ġentities Ġ( PER , ĠLOC , ĠOR G Ġand ĠM ISC ) Ġfrom Ġboth Ġof Ġthe Ġdataset Ġused Ġin Ġour Ġexperiments . ĠThe Ġdataset Ġis Ġdivided Ġinto Ġthree Ġparts Ġwith Ġ64 %, Ġ16 % Ġand Ġ20 % Ġof Ġthe Ġtotal Ġdataset Ġinto Ġtraining Ġset , Ġdevelopment Ġset Ġand Ġtest Ġset Ġrespectively . </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġdetails Ġabout Ġtraining Ġour Ġneural Ġnetwork . ĠThe Ġneural Ġnetwork Ġarchitecture Ġare Ġimplemented Ġusing ĠPy Tor ch Ġframework ĠB IB REF 26 . ĠThe Ġtraining Ġis Ġperformed Ġon Ġa Ġsingle ĠNvidia ĠTesla ĠP 100 ĠS XM 2 . ĠWe Ġfirst Ġrun Ġour Ġexperiment Ġon ĠBi L ST M , ĠBi L ST M - CNN , ĠBi L ST M - CR F ĠBi L ST M - CNN - CR F Ġusing Ġthe Ġhyper - param eters Ġmentioned Ġin ĠTable ĠT AB REF 30 . ĠThe Ġtraining Ġand Ġevaluation Ġwas Ġdone Ġon Ġsentence - level . ĠThe ĠR NN Ġvariants Ġare Ġinitialized Ġrandomly Ġfrom Ġ$( - \ sq rt { k }, \ sq rt { k }) $ Ġwhere Ġ$ k = \ frac { 1 }{ hidden \ _ size } $. </s> First Ġwe Ġloaded Ġour Ġdataset Ġand Ġbuilt Ġvocabulary Ġusing Ġtorch text Ġlibrary . ĠThis Ġeased Ġour Ġprocess Ġof Ġdata Ġloading Ġusing Ġits ĠSequence T agging Dat as et Ġclass . ĠWe Ġtrained Ġour Ġmodel Ġwith Ġshuff led Ġtraining Ġset Ġusing ĠAdam Ġoptim izer Ġwith Ġhyper - param eters Ġmentioned Ġin Ġtable ĠT AB REF 30 . ĠAll Ġour Ġmodels Ġwere Ġtrained Ġon Ġsingle Ġlayer Ġof ĠL ST M Ġnetwork . ĠWe Ġfound Ġout ĠAdam Ġwas Ġgiving Ġbetter Ġperformance Ġand Ġfaster Ġconvergence Ġcompared Ġto ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ). ĠWe Ġchose Ġthose Ġhyper - param eters Ġafter Ġmany Ġab lation Ġstudies . ĠThe Ġdrop out Ġof Ġ0 . 5 Ġis Ġapplied Ġafter ĠL ST M Ġlayer . </s> For ĠCNN , Ġwe Ġused Ġ30 Ġdifferent Ġfilters Ġof Ġsizes Ġ3 , Ġ4 Ġand Ġ5 . ĠThe Ġembed d ings Ġof Ġeach Ġcharacter Ġor Ġgrap he me Ġinvolved Ġin Ġa Ġgiven Ġword , Ġwere Ġpassed Ġthrough Ġthe Ġpipeline Ġof ĠConv olution , ĠRect ified ĠLinear ĠUnit Ġand ĠMax - Pool ing . ĠThe Ġresulting Ġvectors Ġwere Ġconc aten ated Ġand Ġapplied Ġdrop out Ġof Ġ0 . 5 Ġbefore Ġpassing Ġinto Ġlinear Ġlayer Ġto Ġobtain Ġthe Ġembed ding Ġsize Ġof Ġ30 Ġfor Ġthe Ġgiven Ġword . ĠThis Ġresulting Ġembed ding Ġis Ġconc aten ated Ġwith Ġword Ġembed d ings , Ġwhich Ġis Ġagain Ġconc aten ated Ġwith Ġone - hot ĠPOS Ġvector . </s> Exper iments Ġ: :: ĠT agging ĠScheme </s> Currently , Ġfor Ġour Ġexperiments Ġwe Ġtrained Ġour Ġmodel Ġon ĠIO Ġ( Inside , ĠOutside ) Ġformat Ġfor Ġboth Ġthe Ġdataset , Ġhence Ġthe Ġdataset Ġdoes Ġnot Ġcontain Ġany ĠB - type Ġannotation Ġunlike Ġin ĠB IO Ġ( Beginning , ĠInside , ĠOutside ) Ġscheme . </s> Exper iments Ġ: :: ĠEarly ĠSto pping </s> We Ġused Ġsimple Ġearly Ġstopping Ġtechnique Ġwhere Ġif Ġthe Ġvalidation Ġloss Ġdoes Ġnot Ġdecrease Ġafter Ġ10 Ġepoch s , Ġthe Ġtraining Ġwas Ġstopped , Ġelse Ġthe Ġtraining Ġwill Ġrun Ġupt o Ġ100 Ġepoch s . ĠIn Ġour Ġexperience , Ġtraining Ġusually Ġstops Ġaround Ġ30 - 50 Ġepoch s . </s> Exper iments Ġ: :: ĠHyper - param eters ĠTun ing </s> We Ġran Ġour Ġexperiment Ġlooking Ġfor Ġthe Ġbest Ġhyper - param eters Ġby Ġchanging Ġlearning Ġrate Ġfrom Ġ( 0 , 1 , Ġ0 . 01 , Ġ0 . 001 , Ġ0 . 0001 ), Ġweight Ġdecay Ġfrom Ġ[ $ 10 ^{ - 1 } $, Ġ$ 10 ^{ - 2 } $, Ġ$ 10 ^{ - 3 } $, Ġ$ 10 ^{ - 4 } $, Ġ$ 10 ^{ - 5 } $, Ġ$ 10 ^{ - 6 } $, Ġ$ 10 ^{ - 7 } $ ], Ġbatch Ġsize Ġfrom Ġ[ 1 , Ġ2 , Ġ4 , Ġ8 , Ġ16 , Ġ32 , Ġ64 , Ġ128 ], Ġhidden Ġsize Ġfrom Ġ[ 8 , Ġ16 , Ġ32 , Ġ64 , Ġ128 , Ġ256 , Ġ512 Ġ1024 ]. ĠTable ĠT AB REF 30 Ġshows Ġall Ġother Ġhyper - param eter Ġused Ġin Ġour Ġexperiment Ġfor Ġboth Ġof Ġthe Ġdataset . </s> Exper iments Ġ: :: ĠEffect Ġof ĠDrop out </s> Figure ĠFIG REF 31 Ġshows Ġhow Ġwe Ġend Ġup Ġchoosing Ġ0 . 5 Ġas Ġdrop out Ġrate . ĠWhen Ġthe Ġdrop out Ġlayer Ġwas Ġnot Ġused , Ġthe ĠF 1 Ġscore Ġare Ġat Ġthe Ġlowest . ĠAs , Ġwe Ġslowly Ġincrease Ġthe Ġdrop out Ġrate , Ġthe ĠF 1 Ġscore Ġalso Ġgradually Ġincreases , Ġhowever Ġafter Ġdrop out Ġrate Ġ= Ġ0 . 5 , Ġthe ĠF 1 Ġscore Ġstarts Ġfalling Ġdown . ĠTherefore , Ġwe Ġhave Ġchosen Ġ0 . 5 Ġas Ġdrop out Ġrate Ġfor Ġall Ġother Ġexperiments Ġperformed . </s> E val uation </s> In Ġthis Ġsection , Ġwe Ġpresent Ġthe Ġdetails Ġregarding Ġevaluation Ġand Ġcomparison Ġof Ġour Ġmodels Ġwith Ġother Ġbas elines . </s> Table ĠT AB REF 25 Ġshows Ġthe Ġstudy Ġof Ġvarious Ġembed d ings Ġand Ġcomparison Ġamong Ġeach Ġother Ġin ĠOur N ep ali Ġdataset . ĠHere , Ġraw Ġdataset Ġrepresents Ġsuch Ġdataset Ġwhere Ġpost - pos itions Ġare Ġnot Ġle mm at ized . ĠWe Ġcan Ġobserve Ġthat Ġpre - trained Ġembed d ings Ġsignificantly Ġimproves Ġthe Ġscore Ġcompared Ġto Ġrandomly Ġinitialized Ġembed ding . ĠWe Ġcan Ġded uce Ġthat ĠSkip ĠGram Ġmodels Ġperform Ġbetter Ġcompared ĠCB OW Ġmodels Ġfor Ġword 2 vec Ġand Ġfast text . ĠHere , Ġfast Text _ P ret rained Ġrepresents Ġthe Ġembed ding Ġreadily Ġavailable Ġin Ġfast Text Ġwebsite , Ġwhile Ġother Ġembed d ings Ġare Ġtrained Ġon Ġthe ĠNep ali ĠNational ĠCorpus Ġas Ġmentioned Ġin Ġsub - section ĠSEC REF 11 . ĠFrom Ġthis Ġtable ĠT AB REF 25 , Ġwe Ġcan Ġclearly Ġobserve Ġthat Ġmodel Ġusing Ġfast Text _ Skip ĠGram Ġembed d ings Ġoutper forms Ġall Ġother Ġmodels . </s> Table ĠT AB REF 35 Ġshows Ġthe Ġmodel Ġarchitecture Ġcomparison Ġbetween Ġall Ġthe Ġmodels Ġexperimented . ĠThe Ġfeatures Ġused Ġfor ĠStanford ĠCR F Ġclass ifier Ġare Ġwords , Ġletter Ġn - gram s Ġof Ġupt o Ġlength Ġ6 , Ġprevious Ġword Ġand Ġnext Ġword . ĠThis Ġmodel Ġis Ġtrained Ġtill Ġthe Ġcurrent Ġfunction Ġvalue Ġis Ġless Ġthan Ġ$ 1 \ math rm Ġ{ e }{ - 2 } $. ĠThe Ġhyper - param eters Ġof Ġneural Ġnetwork Ġexperiments Ġare Ġset Ġas Ġshown Ġin Ġtable ĠT AB REF 30 . ĠSince , Ġword Ġembed ding Ġof Ġcharacter - level Ġand Ġgrap he me - level Ġis Ġrandom , Ġtheir Ġscores Ġare Ġnear . </s> All Ġmodels Ġare Ġevaluated Ġusing ĠCo N LL - 2003 Ġevaluation Ġscript BI B REF 25 Ġto Ġcalculate Ġentity - wise Ġprecision , Ġrecall Ġand Ġf 1 Ġscore . </s> Discussion </s> In Ġthis Ġpaper Ġwe Ġpresent Ġthat Ġwe Ġcan Ġexploit Ġthe Ġpower Ġof Ġneural Ġnetwork Ġto Ġtrain Ġthe Ġmodel Ġto Ġperform Ġdownstream ĠN LP Ġtasks Ġlike ĠNamed ĠEntity ĠRecogn ition Ġeven Ġin ĠNep ali Ġlanguage . ĠWe Ġshowed Ġthat Ġthe Ġword Ġvectors Ġlearned Ġthrough Ġfast text Ġskip Ġgram Ġmodel Ġperforms Ġbetter Ġthan Ġother Ġword Ġembed ding Ġbecause Ġof Ġits Ġcapability Ġto Ġrepresent Ġsub - word Ġand Ġthis Ġis Ġparticularly Ġimportant Ġto Ġcapture Ġmorph ological Ġstructure Ġof Ġwords Ġand Ġsentences Ġin Ġhighly Ġinf lection al Ġlike ĠNep ali . ĠThis Ġconcept Ġcan Ġcome Ġhandy Ġin Ġother ĠDev an ag ari Ġlanguages Ġas Ġwell Ġbecause Ġthe Ġwritten Ġscripts Ġhave Ġsimilar Ġsynt act ical Ġstructure . </s> We Ġalso Ġfound Ġout Ġthat Ġstemming Ġpost - pos itions Ġcan Ġhelp Ġa Ġlot Ġin Ġimproving Ġmodel Ġperformance Ġbecause Ġof Ġinf lection al Ġcharacteristics Ġof ĠNep ali Ġlanguage . ĠSo Ġwhen Ġwe Ġseparate Ġout Ġits Ġinf lections Ġor Ġmorp hem es , Ġwe Ġcan Ġminimize Ġthe Ġvariations Ġof Ġsame Ġword Ġwhich Ġgives Ġits Ġroot Ġword Ġa Ġstronger Ġword Ġvector Ġrepresentations Ġcompared Ġto Ġits Ġinf lected Ġversions . </s> We Ġcan Ġclearly Ġimply Ġfrom Ġtables ĠT AB REF 23 , ĠT AB REF 24 , Ġand ĠT AB REF 35 Ġthat Ġwe Ġneed Ġmore Ġdata Ġto Ġget Ġbetter Ġresults Ġbecause ĠOur N ep ali Ġdataset Ġvolume Ġis Ġalmost Ġten Ġtimes Ġbigger Ġcompared Ġto ĠIL PR L Ġdataset Ġin Ġterms Ġof Ġentities . </s> Conclusion Ġand ĠFuture Ġwork </s> In Ġthis Ġpaper , Ġwe Ġproposed Ġa Ġnovel ĠN ER Ġfor ĠNep ali Ġlanguage Ġand Ġachieved Ġrelative Ġimprovement Ġof Ġupt o Ġ10 % Ġand Ġstudies Ġdifferent Ġfactors Ġeffect ing Ġthe Ġperformance Ġof Ġthe ĠN ER Ġfor ĠNep ali Ġlanguage . </s> We Ġalso Ġpresent Ġa Ġneural Ġarchitecture ĠBi L ST M + CNN ( g rap he me - level ) Ġwhich Ġturns Ġout Ġto Ġbe Ġperforming Ġon Ġpar Ġwith ĠBi L ST M + CNN ( character - level ) Ġunder Ġthe Ġsame Ġconfiguration . ĠWe Ġbelieve Ġthis Ġwill Ġnot Ġonly Ġhelp ĠNep ali Ġlanguage Ġbut Ġalso Ġother Ġlanguages Ġfalling Ġunder Ġthe Ġumb rell as Ġof ĠDev an ag ari Ġlanguages . ĠOur Ġmodel ĠBi L ST M + CNN ( g rap he me - level ) Ġand ĠBi L ST M + CNN ( G )+ POS Ġoutper forms Ġall Ġother Ġmodel Ġexperimented Ġin ĠOur N ep ali Ġand ĠIL PR L Ġdataset Ġrespectively . </s> Since Ġthis Ġis Ġthe Ġfirst Ġnamed Ġentity Ġrecognition Ġresearch Ġin ĠNepal Ġlanguage Ġusing Ġneural Ġnetwork , Ġthere Ġare Ġmany Ġrooms Ġfor Ġimprovement . ĠWe Ġbelieve Ġinitial izing Ġthe Ġgrap he me - level Ġembed ding Ġwith Ġfast text Ġembed d ings Ġmight Ġhelp Ġboosting Ġthe Ġperformance , Ġrather Ġthan Ġrandomly Ġinitial izing Ġit . ĠIn Ġfuture , Ġwe Ġplan Ġto Ġapply Ġother Ġlatest Ġtechniques Ġlike ĠB ERT , ĠEL Mo Ġand ĠFL AIR Ġto Ġstudy Ġits Ġeffect Ġon Ġlow - resource Ġlanguage Ġlike ĠNep ali . ĠWe Ġalso Ġplan Ġto Ġimprove Ġthe Ġmodel Ġusing Ġcross - ling ual Ġor Ġmulti - ling ual Ġparameter Ġsharing Ġtechniques Ġby Ġjointly Ġtraining Ġwith Ġother ĠDev an ag ari Ġlanguages Ġlike ĠHindi Ġand ĠBeng ali . </s> Finally , Ġwe Ġwould Ġlike Ġto Ġcontribute Ġour Ġdataset Ġto ĠNep ali ĠN LP Ġcommunity Ġto Ġmove Ġforward Ġthe Ġresearch Ġgoing Ġon Ġin Ġlanguage Ġunderstanding Ġdomain . ĠWe Ġbelieve Ġthere Ġshould Ġbe Ġspecial Ġcommittee Ġto Ġcreate Ġand Ġmaintain Ġsuch Ġdataset Ġfor ĠNep ali ĠN LP Ġand Ġorganize Ġvarious Ġcompetitions Ġwhich Ġwould Ġelevate Ġthe ĠN LP Ġresearch Ġin ĠNepal . </s> Some Ġof Ġthe Ġfuture Ġworks Ġare Ġlisted Ġbelow : </s> Pro per Ġinitialization Ġof Ġgrap he me Ġlevel Ġembed ding Ġfrom Ġfast text Ġembed d ings . </s> Apply Ġrobust ĠPOS - t agger Ġfor ĠNep ali Ġdataset </s> L em mat ize Ġthe ĠOur N ep ali Ġdataset Ġwith Ġrobust Ġand Ġefficient Ġle mm at izer </s> Improve ĠNep ali Ġlanguage Ġscore Ġwith Ġcross - ling ual Ġlearning Ġtechniques </s> Create Ġmore Ġdataset Ġusing ĠWikipedia / Wik id ata Ġframework </s> Acknowled gments </s> The Ġauthors Ġof Ġthis Ġpaper Ġwould Ġlike Ġto Ġexpress Ġsincere Ġthanks Ġto ĠBal ĠKrishna ĠBal , ĠKath mand u ĠUniversity ĠProfessor Ġfor Ġproviding Ġus Ġthe ĠPOS - tag ged ĠNep ali ĠN ER Ġdata .