5 percent points.
ABSTRACTIVE
0.05 F1
ABSTRACTIVE
FLOAT SELECTED: Table 4: The results of NER evaluation task, averaged over 5 training and evaluation runs. The scores are average F1 score of the three named entity classes. The columns show FastText, ELMo, and the difference between them (∆(E − FT )).
FLOAT SELECTED: Table 4: The results of NER evaluation task, averaged over 5 training and evaluation runs. The scores are average F1 score of the three named entity classes. The columns show FastText, ELMo, and the difference between them (∆(E − FT )).
1911.10049
Introduction </s> Word Ġembed d ings Ġare Ġrepresentations Ġof Ġwords Ġin Ġnumerical Ġform , Ġas Ġvectors Ġof Ġtypically Ġseveral Ġhundred Ġdimensions . ĠThe Ġvectors Ġare Ġused Ġas Ġan Ġinput Ġto Ġmachine Ġlearning Ġmodels ; Ġfor Ġcomplex Ġlanguage Ġprocessing Ġtasks Ġthese Ġare Ġtypically Ġdeep Ġneural Ġnetworks . ĠThe Ġembed ding Ġvectors Ġare Ġobtained Ġfrom Ġspecialized Ġlearning Ġtasks , Ġbased Ġon Ġneural Ġnetworks , Ġe . g ., Ġword 2 vec ĠB IB REF 0 , ĠGl o Ve ĠB IB REF 1 , ĠFast Text ĠB IB REF 2 , ĠEL Mo ĠB IB REF 3 , Ġand ĠB ERT ĠB IB REF 4 . ĠFor Ġtraining , Ġthe Ġembed d ings Ġalgorithms Ġuse Ġlarge Ġmon oling ual Ġcorpor a Ġthat Ġencode Ġimportant Ġinformation Ġabout Ġword Ġmeaning Ġas Ġdistances Ġbetween Ġvectors . ĠIn Ġorder Ġto Ġenable Ġdownstream Ġmachine Ġlearning Ġon Ġtext Ġunderstanding Ġtasks , Ġthe Ġembed d ings Ġshall Ġpreserve Ġsemantic Ġrelations Ġbetween Ġwords , Ġand Ġthis Ġis Ġtrue Ġeven Ġacross Ġlanguages . </s> Probably Ġthe Ġbest Ġknown Ġword Ġembed d ings Ġare Ġproduced Ġby Ġthe Ġword 2 vec Ġmethod ĠB IB REF 5 . ĠThe Ġproblem Ġwith Ġword 2 vec Ġembed d ings Ġis Ġtheir Ġfailure Ġto Ġexpress Ġpoly sem ous Ġwords . ĠDuring Ġtraining Ġof Ġan Ġembed ding , Ġall Ġsenses Ġof Ġa Ġgiven Ġword Ġ( e . g ., Ġpaper Ġas Ġa Ġmaterial , Ġas Ġa Ġnewspaper , Ġas Ġa Ġscientific Ġwork , Ġand Ġas Ġan Ġexam ) Ġcontribute Ġrelevant Ġinformation Ġin Ġproportion Ġto Ġtheir Ġfrequency Ġin Ġthe Ġtraining Ġcorpus . ĠThis Ġcauses Ġthe Ġfinal Ġvector Ġto Ġbe Ġplaced Ġsomewhere Ġin Ġthe Ġweighted Ġmiddle Ġof Ġall Ġwords ' Ġmeanings . ĠConsequently , Ġrare Ġmeanings Ġof Ġwords Ġare Ġpoorly Ġexpressed Ġwith Ġword 2 vec Ġand Ġthe Ġresulting Ġvectors Ġdo Ġnot Ġoffer Ġgood Ġsemantic Ġrepresentations . ĠFor Ġexample , Ġnone Ġof Ġthe Ġ50 Ġclosest Ġvectors Ġof Ġthe Ġword Ġpaper Ġis Ġrelated Ġto Ġscience . </s> The Ġidea Ġof Ġcontextual Ġembed d ings Ġis Ġto Ġgenerate Ġa Ġdifferent Ġvector Ġfor Ġeach Ġcontext Ġa Ġword Ġappears Ġin Ġand Ġthe Ġcontext Ġis Ġtypically Ġdefined Ġsentence - wise . ĠTo Ġa Ġlarge Ġextent , Ġthis Ġsolves Ġthe Ġproblems Ġwith Ġword Ġpoly se my , Ġi . e . Ġthe Ġcontext Ġof Ġa Ġsentence Ġis Ġtypically Ġenough Ġto Ġdis amb ig uate Ġdifferent Ġmeanings Ġof Ġa Ġword Ġfor Ġhumans Ġand Ġso Ġit Ġis Ġfor Ġthe Ġlearning Ġalgorithms . ĠIn Ġthis Ġwork , Ġwe Ġdescribe Ġhigh - quality Ġmodels Ġfor Ġcontextual Ġembed d ings , Ġcalled ĠEL Mo ĠB IB REF 3 , Ġpre com puted Ġfor Ġseven Ġmorph ologically Ġrich , Ġless - res ourced Ġlanguages : ĠSloven ian , ĠCroatian , ĠFinnish , ĠEston ian , ĠLat v ian , ĠLithuan ian , Ġand ĠSwedish . ĠEL Mo Ġis Ġone Ġof Ġthe Ġmost Ġsuccessful Ġapproaches Ġto Ġcontextual Ġword Ġembed d ings . ĠAt Ġtime Ġof Ġits Ġcreation , ĠEL Mo Ġhas Ġbeen Ġshown Ġto Ġoutper form Ġprevious Ġword Ġembed d ings ĠB IB REF 3 Ġlike Ġword 2 vec Ġand ĠGl o Ve Ġon Ġmany ĠN LP Ġtasks , Ġe . g ., Ġquestion Ġanswering , Ġnamed Ġentity Ġextraction , Ġsentiment Ġanalysis , Ġtextual Ġentail ment , Ġsemantic Ġrole Ġlabeling , Ġand Ġcore ference Ġresolution . </s> This Ġreport Ġis Ġsplit Ġinto Ġfurther Ġfive Ġsections . ĠIn Ġsection ĠSEC REF 2 , Ġwe Ġdescribe Ġthe Ġcontextual Ġembed d ings ĠEL Mo . ĠIn ĠSection ĠSEC REF 3 , Ġwe Ġdescribe Ġthe Ġdatasets Ġused Ġand Ġin ĠSection ĠSEC REF 4 Ġwe Ġdescribe Ġpre processing Ġand Ġtraining Ġof Ġthe Ġembed d ings . ĠWe Ġdescribe Ġthe Ġmethodology Ġfor Ġevaluation Ġof Ġcreated Ġvectors Ġand Ġresults Ġin ĠSection ĠSEC REF 5 . ĠWe Ġpresent Ġconclusion Ġin ĠSection ĠSEC REF 6 Ġwhere Ġwe Ġalso Ġoutline Ġplans Ġfor Ġfurther Ġwork . </s> EL Mo </s> Typ ical Ġword Ġembed d ings Ġmodels Ġor Ġrepresentations , Ġsuch Ġas Ġword 2 vec ĠB IB REF 0 , ĠGl o Ve ĠB IB REF 1 , Ġor ĠFast Text ĠB IB REF 2 , Ġare Ġfast Ġto Ġtrain Ġand Ġhave Ġbeen Ġpre - trained Ġfor Ġa Ġnumber Ġof Ġdifferent Ġlanguages . ĠThey Ġdo Ġnot Ġcapture Ġthe Ġcontext , Ġthough , Ġso Ġeach Ġword Ġis Ġalways Ġgiven Ġthe Ġsame Ġvector , Ġregardless Ġof Ġits Ġcontext Ġor Ġmeaning . ĠThis Ġis Ġespecially Ġproblematic Ġfor Ġpoly sem ous Ġwords . ĠEL Mo Ġ( Emb edd ings Ġfrom ĠLanguage ĠModels ) Ġembed ding ĠB IB REF 3 Ġis Ġone Ġof Ġthe Ġstate - of - the - art Ġpret rained Ġtransfer Ġlearning Ġmodels , Ġthat Ġremedies Ġthe Ġproblem Ġand Ġintroduces Ġa Ġcontextual Ġcomponent . </s> EL Mo Ġmodel ` s Ġarchitecture Ġconsists Ġof Ġthree Ġneural Ġnetwork Ġlayers . ĠThe Ġoutput Ġof Ġthe Ġmodel Ġafter Ġeach Ġlayer Ġgives Ġone Ġset Ġof Ġembed d ings , Ġaltogether Ġthree Ġsets . ĠThe Ġfirst Ġlayer Ġis Ġa ĠCNN Ġlayer , Ġwhich Ġoperates Ġon Ġa Ġcharacter Ġlevel . ĠIt Ġis Ġcontext Ġindependent , Ġso Ġeach Ġword Ġalways Ġgets Ġthe Ġsame Ġembed ding , Ġregardless Ġof Ġits Ġcontext . ĠIt Ġis Ġfollowed Ġby Ġtwo Ġbi LM Ġlayers . ĠA Ġbi LM Ġlayer Ġconsists Ġof Ġtwo Ġconc aten ated ĠL ST Ms . ĠIn Ġthe Ġfirst ĠL ST M , Ġwe Ġtry Ġto Ġpredict Ġthe Ġfollowing Ġword , Ġbased Ġon Ġthe Ġgiven Ġpast Ġwords , Ġwhere Ġeach Ġword Ġis Ġrepresented Ġby Ġthe Ġembed d ings Ġfrom Ġthe ĠCNN Ġlayer . ĠIn Ġthe Ġsecond ĠL ST M , Ġwe Ġtry Ġto Ġpredict Ġthe Ġpreceding Ġword , Ġbased Ġon Ġthe Ġgiven Ġfollowing Ġwords . ĠIt Ġis Ġequivalent Ġto Ġthe Ġfirst ĠL ST M , Ġjust Ġreading Ġthe Ġtext Ġin Ġreverse . </s> In ĠN LP Ġtasks , Ġany Ġset Ġof Ġthese Ġembed d ings Ġmay Ġbe Ġused ; Ġhowever , Ġa Ġweighted Ġaverage Ġis Ġusually Ġused . ĠThe Ġweights Ġof Ġthe Ġaverage Ġare Ġlearned Ġduring Ġthe Ġtraining Ġof Ġthe Ġmodel Ġfor Ġthe Ġspecific Ġtask . ĠAdditionally , Ġan Ġentire ĠEL Mo Ġmodel Ġcan Ġbe Ġfine - tun ed Ġon Ġa Ġspecific Ġend Ġtask . </s> Although ĠEL Mo Ġis Ġtrained Ġon Ġcharacter Ġlevel Ġand Ġis Ġable Ġto Ġhandle Ġout - of - voc abulary Ġwords , Ġa Ġvocabulary Ġfile Ġcontaining Ġmost Ġcommon Ġtokens Ġis Ġused Ġfor Ġefficiency Ġduring Ġtraining Ġand Ġembed ding Ġgeneration . ĠThe Ġoriginal ĠEL Mo Ġmodel Ġwas Ġtrained Ġon Ġa Ġone Ġbillion Ġword Ġlarge ĠEnglish Ġcorpus , Ġwith Ġa Ġgiven Ġvocabulary Ġfile Ġof Ġabout Ġ800 , 000 Ġwords . ĠLater , ĠEL Mo Ġmodels Ġfor Ġother Ġlanguages Ġwere Ġtrained Ġas Ġwell , Ġbut Ġlimited Ġto Ġlarger Ġlanguages Ġwith Ġmany Ġresources , Ġlike ĠGerman Ġand ĠJapanese . </s> EL Mo Ġ: :: ĠEL Mo For Many L angs </s> Recently , ĠEL Mo For Many L angs ĠB IB REF 6 Ġproject Ġreleased Ġpre - trained ĠEL Mo Ġmodels Ġfor Ġa Ġnumber Ġof Ġdifferent Ġlanguages ĠB IB REF 7 . ĠThese Ġmodels , Ġhowever , Ġwere Ġtrained Ġon Ġa Ġsignificantly Ġsmaller Ġdatasets . ĠThey Ġused Ġ20 - million - words Ġdata Ġrandomly Ġsampled Ġfrom Ġthe Ġraw Ġtext Ġreleased Ġby Ġthe ĠCo N LL Ġ2017 ĠShared ĠTask Ġ- ĠAutom atically ĠAn not ated ĠRaw ĠText s Ġand ĠWord ĠEmb edd ings ĠB IB REF 8 , Ġwhich Ġis Ġa Ġcombination Ġof ĠWikipedia Ġdump Ġand Ġcommon Ġcrawl . ĠThe Ġquality Ġof Ġthese Ġmodels Ġis Ġquestionable . ĠFor Ġexample , Ġwe Ġcompared Ġthe ĠLat v ian Ġmodel Ġby ĠEL Mo For Many L angs Ġwith Ġa Ġmodel Ġwe Ġtrained Ġon Ġa Ġcomplete Ġ( wik id ump Ġ+ Ġcommon Ġcrawl ) ĠLat v ian Ġcorpus , Ġwhich Ġhas Ġabout Ġ280 Ġmillion Ġtokens . ĠThe Ġdifference Ġof Ġeach Ġmodel Ġon Ġthe Ġword Ġanalogy Ġtask Ġis Ġshown Ġin ĠFigure ĠFIG REF 16 Ġin ĠSection ĠSEC REF 5 . ĠAs Ġthe Ġresults Ġof Ġthe ĠEL Mo For Many L angs Ġembed d ings Ġare Ġsignificantly Ġworse Ġthan Ġusing Ġthe Ġfull Ġcorpus , Ġwe Ġcan Ġconclude Ġthat Ġthese Ġembed d ings Ġare Ġnot Ġof Ġsufficient Ġquality . ĠFor Ġthat Ġreason , Ġwe Ġcomputed ĠEL Mo Ġembed d ings Ġfor Ġseven Ġlanguages Ġon Ġmuch Ġlarger Ġcorpor a . ĠAs Ġthis Ġeffort Ġrequires Ġaccess Ġto Ġlarge Ġamount Ġof Ġtextual Ġdata Ġand Ġconsiderable Ġcomputational Ġresources , Ġwe Ġmade Ġthe Ġpre com puted Ġmodels Ġpublicly Ġavailable Ġby Ġdepos iting Ġthem Ġto ĠClar in Ġrepository . </s> Training ĠData </s> We Ġtrained ĠEL Mo Ġmodels Ġfor Ġseven Ġlanguages : ĠSloven ian , ĠCroatian , ĠFinnish , ĠEston ian , ĠLat v ian , ĠLithuan ian Ġand ĠSwedish . ĠTo Ġobtain Ġhigh - quality Ġembed d ings , Ġwe Ġused Ġlarge Ġmon oling ual Ġcorpor a Ġfrom Ġvarious Ġsources Ġfor Ġeach Ġlanguage . ĠSome Ġcorpor a Ġare Ġavailable Ġonline Ġunder Ġper missive Ġlicences , Ġothers Ġare Ġavailable Ġonly Ġfor Ġresearch Ġpurposes Ġor Ġhave Ġlimited Ġavailability . ĠThe Ġcorpor a Ġused Ġin Ġtraining Ġdatasets Ġare Ġa Ġmix Ġof Ġnews Ġarticles Ġand Ġgeneral Ġweb Ġcrawl , Ġwhich Ġwe Ġpre process ed Ġand Ġded u pl icated . ĠBelow Ġwe Ġshortly Ġdescribe Ġthe Ġused Ġcorpor a Ġin Ġalphabet ical Ġorder Ġof Ġthe Ġinvolved Ġlanguages . ĠTheir Ġnames Ġand Ġsizes Ġare Ġsummarized Ġin ĠTable ĠT AB REF 3 . </s> Cro at ian Ġdataset Ġinclude Ġhr Wa C Ġ2 . 1 Ġcorpus ĠB IB REF 9 , ĠR iz n ica ĠB IB REF 10 , Ġand Ġarticles Ġof ĠCroatian Ġbranch Ġof ĠSty ria Ġmedia Ġhouse , Ġmade Ġavailable Ġto Ġus Ġthrough Ġpartnership Ġin Ġa Ġjoint Ġproject . Ġhr Wa C Ġwas Ġbuilt Ġby Ġcrawling Ġthe Ġ. hr Ġinternet Ġdomain Ġin Ġ2011 Ġand Ġ2014 . ĠR iz n ica Ġis Ġcomposed Ġof ĠCroatian Ġfiction Ġand Ġnon - fiction Ġprose , Ġpoetry , Ġdrama , Ġtextbooks , Ġmanuals , Ġetc . ĠThe ĠSty ria Ġdataset Ġconsists Ġof Ġ570 , 219 Ġnews Ġarticles Ġpublished Ġon Ġthe ĠCroatian Ġ24 s ata Ġnews Ġportal Ġand Ġniche Ġportals Ġrelated Ġto Ġ24 s ata . </s> E ston ian Ġdataset Ġcontains Ġtexts Ġfrom Ġtwo Ġsources , ĠCo N LL Ġ2017 ĠShared ĠTask Ġ- ĠAutom atically ĠAn not ated ĠRaw ĠText s Ġand ĠWord ĠEmb edd ings ĠB IB REF 8 , Ġand Ġnews Ġarticles Ġmade Ġavailable Ġto Ġus Ġby ĠE ks press ĠMe edia Ġdue Ġto Ġpartnership Ġin Ġthe Ġproject . ĠE ks press ĠMe edia Ġdataset Ġis Ġcomposed Ġof ĠEston ian Ġnews Ġarticles Ġbetween Ġyears Ġ2009 Ġand Ġ2019 . ĠThe ĠCo N LL Ġ2017 Ġcorpus Ġis Ġcomposed Ġof ĠEston ian ĠWikipedia Ġand Ġweb c rawl . </s> F inn ish Ġdataset Ġcontains Ġarticles Ġby ĠFinnish Ġnews Ġagency ĠST T , ĠFinnish Ġpart Ġof Ġthe ĠCo N LL Ġ2017 Ġdataset , Ġand ĠY l il aut a Ġdownloadable Ġversion ĠB IB REF 11 . ĠST T Ġnews Ġarticles Ġwere Ġpublished Ġbetween Ġyears Ġ1992 Ġand Ġ2018 . ĠY l il aut a Ġis Ġa ĠFinnish Ġonline Ġdiscussion Ġboard ; Ġthe Ġcorpus Ġcontains Ġparts Ġof Ġthe Ġdiscussions Ġfrom Ġ2012 Ġto Ġ2014 . </s> Lat v ian Ġdataset Ġconsists Ġonly Ġof Ġthe ĠLat v ian Ġportion Ġof Ġthe ĠCon LL Ġ2017 Ġcorpus . </s> L ith uan ian Ġdataset Ġis Ġcomposed Ġof ĠLithuan ian ĠWikipedia Ġarticles Ġfrom Ġ2018 , ĠD GT - UD Ġcorpus , Ġand ĠLt Ten Ten . ĠD GT - UD Ġis Ġa Ġparallel Ġcorpus Ġof Ġ23 Ġofficial Ġlanguages Ġof Ġthe ĠEU , Ġcomposed Ġof ĠJ RC ĠD GT Ġtranslation Ġmemory Ġof ĠEuropean Ġlaw , Ġautomatically Ġannot ated Ġwith ĠUD - P ipe Ġ1 . 2 . ĠLt Ten Ten Ġis ĠLithuan ian Ġweb Ġcorpus Ġmade Ġup Ġof Ġtexts Ġcollected Ġfrom Ġthe Ġinternet Ġin ĠApril Ġ2014 ĠB IB REF 12 . </s> S lo ven e Ġdataset Ġis Ġformed Ġfrom Ġthe ĠGig af ida Ġ2 . 0 Ġcorpus ĠB IB REF 13 . ĠIt Ġis Ġa Ġgeneral Ġlanguage Ġcorpus Ġcomposed Ġof Ġvarious Ġsources , Ġmostly Ġnewspapers , Ġinternet Ġpages , Ġand Ġmagazines , Ġbut Ġalso Ġfiction Ġand Ġnon - fiction Ġprose , Ġtextbooks , Ġetc . </s> Sw ed ish Ġdataset Ġis Ġcomposed Ġof ĠST T ĠSwedish Ġarticles Ġand ĠSwedish Ġpart Ġof ĠCo N LL Ġ2017 . ĠThe ĠFinnish Ġnews Ġagency ĠST T Ġpublishes Ġsome Ġof Ġits Ġarticles Ġin ĠSwedish Ġlanguage . ĠThey Ġwere Ġmade Ġavailable Ġto Ġus Ġthrough Ġpartnership Ġin Ġa Ġjoint Ġproject . ĠThe Ġcorpus Ġcontains Ġthose Ġarticles Ġfrom Ġ1992 Ġto Ġ2017 . </s> Pre processing Ġand ĠTraining </s> Prior Ġto Ġtraining Ġthe ĠEL Mo Ġmodels , Ġwe Ġsentence Ġand Ġword Ġtoken ized Ġall Ġthe Ġdatasets . ĠThe Ġtext Ġwas Ġformatted Ġin Ġsuch Ġa Ġway Ġthat Ġeach Ġsentence Ġwas Ġin Ġits Ġown Ġline Ġwith Ġtokens Ġseparated Ġby Ġwhite Ġspaces . ĠCo N LL Ġ2017 , ĠD GT - UD Ġand ĠLt Ten Ten 14 Ġcorpor a Ġwere Ġalready Ġpre - token ized . ĠWe Ġtoken ized Ġthe Ġothers Ġusing Ġthe ĠNL T K Ġlibrary Ġand Ġits Ġtoken izers Ġfor Ġeach Ġof Ġthe Ġlanguages . ĠThere Ġis Ġno Ġtoken izer Ġfor ĠCroatian Ġin ĠNL T K Ġlibrary , Ġso Ġwe Ġused ĠSloven e Ġtoken izer Ġinstead . ĠAfter Ġtoken ization , Ġwe Ġded u pl icated Ġthe Ġdatasets Ġfor Ġeach Ġlanguage Ġseparately , Ġusing Ġthe ĠOnion Ġ( ON e ĠInst ance ĠON ly ) Ġtool Ġfor Ġtext Ġded u pl ication . ĠWe Ġapplied Ġthe Ġtool Ġon Ġparagraph Ġlevel Ġfor Ġcorpor a Ġthat Ġdid Ġnot Ġhave Ġsentences Ġshuff led Ġand Ġon Ġsentence Ġlevel Ġfor Ġthe Ġrest . ĠWe Ġconsidered Ġ9 - gram s Ġwith Ġduplicate Ġcontent Ġthreshold Ġof Ġ0 . 9 . </s> For Ġeach Ġlanguage Ġwe Ġprepared Ġa Ġvocabulary Ġfile , Ġcontaining Ġroughly Ġone Ġmillion Ġmost Ġcommon Ġtokens , Ġi . e . Ġtokens Ġthat Ġappear Ġat Ġleast Ġ$ n $ Ġtimes Ġin Ġthe Ġcorpus , Ġwhere Ġ$ n $ Ġis Ġbetween Ġ15 Ġand Ġ25 , Ġdepending Ġon Ġthe Ġdataset Ġsize . ĠWe Ġincluded Ġthe Ġpunct uation Ġmarks Ġamong Ġthe Ġtokens . ĠWe Ġtrained Ġeach ĠEL Mo Ġmodel Ġusing Ġdefault Ġvalues Ġused Ġto Ġtrain Ġthe Ġoriginal ĠEnglish ĠEL Mo Ġ( large ) Ġmodel . </s> E val uation </s> We Ġevaluated Ġthe Ġproduced ĠEL Mo Ġmodels Ġfor Ġall Ġlanguages Ġusing Ġtwo Ġevaluation Ġtasks : Ġa Ġword Ġanalogy Ġtask Ġand Ġnamed Ġentity Ġrecognition Ġ( NER ) Ġtask . ĠBelow , Ġwe Ġfirst Ġshortly Ġdescribe Ġeach Ġtask , Ġfollowed Ġby Ġthe Ġevaluation Ġresults . </s> E val uation Ġ: :: ĠWord ĠAn alogy ĠTask </s> The Ġword Ġanalogy Ġtask Ġwas Ġpopular ized Ġby Ġm ik ol ov 2013 dist ributed . ĠThe Ġgoal Ġis Ġto Ġfind Ġa Ġterm Ġ$ y $ Ġfor Ġa Ġgiven Ġterm Ġ$ x $ Ġso Ġthat Ġthe Ġrelationship Ġbetween Ġ$ x $ Ġand Ġ$ y $ Ġbest Ġresembles Ġthe Ġgiven Ġrelationship Ġ$ a Ġ: Ġb $. ĠThere Ġare Ġtwo Ġmain Ġgroups Ġof Ġcategories : Ġ5 Ġsemantic Ġand Ġ10 Ġsynt actic . ĠTo Ġillustrate Ġa Ġsemantic Ġrelationship , Ġconsider Ġfor Ġexample Ġthat Ġthe Ġword Ġpair Ġ$ a Ġ: Ġb $ Ġis Ġgiven Ġas ĠâĢ ľ Fin land Ġ: ĠHelsinki âĢ Ŀ . ĠThe Ġtask Ġis Ġto Ġfind Ġthe Ġterm Ġ$ y $ Ġcorresponding Ġto Ġthe Ġrelationship ĠâĢ ľ Sw eden Ġ: Ġ$ y $ âĢ Ŀ , Ġwith Ġthe Ġexpected Ġanswer Ġbeing Ġ$ y =$ ĠStockholm . ĠIn Ġsynt actic Ġcategories , Ġthe Ġtwo Ġwords Ġin Ġa Ġpair Ġhave Ġa Ġcommon Ġstem Ġ( in Ġsome Ġcases Ġeven Ġsame Ġle mma ), Ġwith Ġall Ġthe Ġpairs Ġin Ġa Ġgiven Ġcategory Ġhaving Ġthe Ġsame Ġmorph ological Ġrelationship . ĠFor Ġexample , Ġgiven Ġthe Ġword Ġpair ĠâĢ ľ long Ġ: Ġlonger âĢ Ŀ , Ġwe Ġsee Ġthat Ġwe Ġhave Ġan Ġadjective Ġin Ġits Ġbase Ġform Ġand Ġthe Ġsame Ġadjective Ġin Ġa Ġcomparative Ġform . ĠThat Ġtask Ġis Ġthen Ġto Ġfind Ġthe Ġterm Ġ$ y $ Ġcorresponding Ġto Ġthe Ġrelationship ĠâĢ ľ dark Ġ: Ġ$ y $ âĢ Ŀ , Ġwith Ġthe Ġexpected Ġanswer Ġbeing Ġ$ y =$ Ġdarker , Ġthat Ġis Ġa Ġcomparative Ġform Ġof Ġthe Ġadjective Ġdark . </s> In Ġthe Ġvector Ġspace , Ġthe Ġanalogy Ġtask Ġis Ġtransformed Ġinto Ġvector Ġarithmetic Ġand Ġsearch Ġfor Ġnearest Ġneighbours , Ġi . e . Ġwe Ġcompute Ġthe Ġdistance Ġbetween Ġvectors : Ġd ( vec ( Fin land ), Ġvec ( H els inki )) Ġand Ġsearch Ġfor Ġword Ġ$ y $ Ġwhich Ġwould Ġgive Ġthe Ġclosest Ġresult Ġin Ġdistance Ġd ( vec ( Sw eden ), Ġvec ($ y $ )). ĠIn Ġthe Ġanalogy Ġdataset Ġthe Ġanalog ies Ġare Ġalready Ġpre - specified , Ġso Ġwe Ġare Ġmeasuring Ġhow Ġclose Ġare Ġthe Ġgiven Ġpairs . ĠIn Ġthe Ġevaluation Ġbelow , Ġwe Ġuse Ġanalogy Ġdatasets Ġfor Ġall Ġtested Ġlanguages Ġbased Ġon Ġthe ĠEnglish Ġdataset Ġby ĠB IB REF 14 Ġ. ĠDue Ġto ĠEnglish - centered Ġbias Ġof Ġthis Ġdataset , Ġwe Ġused Ġa Ġmodified Ġdataset Ġwhich Ġwas Ġfirst Ġwritten Ġin ĠSloven e Ġlanguage Ġand Ġthen Ġtranslated Ġinto Ġother Ġlanguages ĠB IB REF 15 . </s> As Ġeach Ġinstance Ġof Ġanalogy Ġcontains Ġonly Ġfour Ġwords , Ġwithout Ġany Ġcontext , Ġthe Ġcontextual Ġmodels Ġ( such Ġas ĠEL Mo ) Ġdo Ġnot Ġhave Ġenough Ġcontext Ġto Ġgenerate Ġsensible Ġembed d ings . ĠWe Ġtherefore Ġused Ġsome Ġadditional Ġtext Ġto Ġform Ġsimple Ġsentences Ġusing Ġthe Ġfour Ġanalogy Ġwords , Ġwhile Ġtaking Ġcare Ġthat Ġtheir Ġnoun Ġcase Ġstays Ġthe Ġsame . ĠFor Ġexample , Ġfor Ġthe Ġwords Ġ" R ome ", Ġ" Italy ", Ġ" Paris " Ġand Ġ" France " Ġ( forming Ġthe Ġanalogy ĠRome Ġis Ġto ĠItaly Ġas ĠParis Ġis Ġto Ġ$ x $, Ġwhere Ġthe Ġcorrect Ġanswer Ġis Ġ$ x =$ France ), Ġwe Ġformed Ġthe Ġsentence Ġ" If Ġthe Ġword ĠRome Ġcorresponds Ġto Ġthe Ġword ĠItaly , Ġthen Ġthe Ġword ĠParis Ġcorresponds Ġto Ġthe Ġword ĠFrance ". ĠWe Ġgenerated Ġembed d ings Ġfor Ġthose Ġfour Ġwords Ġin Ġthe Ġconstructed Ġsentence , Ġsubstituted Ġthe Ġlast Ġword Ġwith Ġeach Ġword Ġin Ġour Ġvocabulary Ġand Ġgenerated Ġthe Ġembed d ings Ġagain . ĠAs Ġtypical Ġfor Ġnon - context ual Ġanalogy Ġtask , Ġwe Ġmeasure Ġthe Ġcos ine Ġdistance Ġ($ d $ ) Ġbetween Ġthe Ġlast Ġword Ġ($ w _ 4 $ ) Ġand Ġthe Ġcombination Ġof Ġthe Ġfirst Ġthree Ġwords Ġ($ w _ 2 - w _ 1 + w _ 3 $ ). ĠWe Ġuse Ġthe ĠCS LS Ġmetric ĠB IB REF 16 Ġto Ġfind Ġthe Ġclosest Ġcandidate Ġword Ġ($ w _ 4 $ ). ĠIf Ġwe Ġfind Ġthe Ġcorrect Ġword Ġamong Ġthe Ġfive Ġclosest Ġwords , Ġwe Ġconsider Ġthat Ġentry Ġas Ġsuccessfully Ġidentified . ĠThe Ġproportion Ġof Ġcorrectly Ġidentified Ġwords Ġforms Ġa Ġstatistic Ġcalled Ġaccuracy @ 5 , Ġwhich Ġwe Ġreport Ġas Ġthe Ġresult . </s> We Ġfirst Ġcompare Ġexisting ĠLat v ian ĠEL Mo Ġembed d ings Ġfrom ĠEL Mo For Many L angs Ġproject Ġwith Ġour ĠLat v ian Ġembed d ings , Ġfollowed Ġby Ġthe Ġdetailed Ġanalysis Ġof Ġour ĠEL Mo Ġembed d ings . ĠWe Ġtrained ĠLat v ian ĠEL Mo Ġusing Ġonly ĠCo N LL Ġ2017 Ġcorpor a . ĠSince Ġthis Ġis Ġthe Ġonly Ġlanguage , Ġwhere Ġwe Ġtrained Ġthe Ġembed ding Ġmodel Ġon Ġexactly Ġthe Ġsame Ġcorpor a Ġas ĠEL Mo For Many L angs Ġmodels , Ġwe Ġchose Ġit Ġfor Ġcomparison Ġbetween Ġour ĠEL Mo Ġmodel Ġwith ĠEL Mo For Many L angs . ĠIn Ġother Ġlanguages , Ġadditional Ġor Ġother Ġcorpor a Ġwere Ġused , Ġso Ġa Ġdirect Ġcomparison Ġwould Ġalso Ġreflect Ġthe Ġquality Ġof Ġthe Ġcorpor a Ġused Ġfor Ġtraining . ĠIn ĠLat v ian , Ġhowever , Ġonly Ġthe Ġsize Ġof Ġthe Ġtraining Ġdataset Ġis Ġdifferent . ĠEL Mo For Many L angs Ġuses Ġonly Ġ20 Ġmillion Ġtokens Ġand Ġwe Ġuse Ġthe Ġwhole Ġcorpus Ġof Ġ270 Ġmillion Ġtokens . </s> The ĠLat v ian ĠEL Mo Ġmodel Ġfrom ĠEL Mo For Many L angs Ġproject Ġperforms Ġsignificantly Ġworse Ġthan ĠE MB ED D IA ĠEL Mo ĠLat v ian Ġmodel Ġon Ġall Ġcategories Ġof Ġword Ġanalogy Ġtask Ġ( Figure ĠFIG REF 16 ). ĠWe Ġalso Ġinclude Ġthe Ġcomparison Ġwith Ġour ĠEston ian ĠEL Mo Ġembed d ings Ġin Ġthe Ġsame Ġfigure . ĠThis Ġcomparison Ġshows Ġthat Ġwhile Ġdifferences Ġbetween Ġour ĠLat v ian Ġand ĠEston ian Ġembed d ings Ġcan Ġbe Ġsignificant Ġfor Ġcertain Ġcategories , Ġthe Ġaccuracy Ġscore Ġof ĠEL Mo For Many L angs Ġis Ġalways Ġworse Ġthan Ġeither Ġof Ġour Ġmodels . ĠThe Ġcomparison Ġof ĠEston ian Ġand ĠLat v ian Ġmodels Ġleads Ġus Ġto Ġbelieve Ġthat Ġa Ġfew Ġhundred Ġmillion Ġtokens Ġis Ġa Ġsufficiently Ġlarge Ġcorpus Ġto Ġtrain ĠEL Mo Ġmodels Ġ( at Ġleast Ġfor Ġword Ġanalogy Ġtask ), Ġbut Ġ20 - million Ġtoken Ġcorpor a Ġused Ġin ĠEL Mo For Many L angs Ġare Ġtoo Ġsmall . </s> The Ġresults Ġfor Ġall Ġlanguages Ġand Ġall ĠEL Mo Ġlayers , Ġaveraged Ġover Ġsemantic Ġand Ġsynt actic Ġcategories , Ġare Ġshown Ġin ĠTable ĠT AB REF 17 . ĠThe Ġembed d ings Ġafter Ġthe Ġfirst ĠL ST M Ġlayer Ġperform Ġbest Ġin Ġsemantic Ġcategories . ĠIn Ġsynt actic Ġcategories , Ġthe Ġnon - context ual ĠCNN Ġlayer Ġperforms Ġthe Ġbest . ĠSynt actic Ġcategories Ġare Ġless Ġcontext Ġdependent Ġand Ġmuch Ġmore Ġmorphology Ġand Ġsyntax Ġbased , Ġso Ġit Ġis Ġnot Ġsurprising Ġthat Ġthe Ġnon - context ual Ġlayer Ġperforms Ġwell . ĠThe Ġsecond ĠL ST M Ġlayer Ġembed d ings Ġperform Ġthe Ġworst Ġin Ġsynt actic Ġcategories , Ġthough Ġstill Ġoutper forming ĠCNN Ġlayer Ġembed d ings Ġin Ġsemantic Ġcategories . ĠLat v ian ĠEL Mo Ġperforms Ġworse Ġcompared Ġto Ġother Ġlanguages Ġwe Ġtrained , Ġespecially Ġin Ġsemantic Ġcategories , Ġpresumably Ġdue Ġto Ġsmaller Ġtraining Ġdata Ġsize . ĠSurprisingly , Ġthe Ġoriginal ĠEnglish ĠEL Mo Ġperforms Ġvery Ġpoorly Ġin Ġsynt actic Ġcategories Ġand Ġonly Ġoutper forms ĠLat v ian Ġin Ġsemantic Ġcategories . ĠThe Ġlow Ġscore Ġcan Ġbe Ġpartially Ġexplained Ġby ĠEnglish Ġmodel Ġscoring Ġ$ 0 . 00 $ Ġin Ġone Ġsynt actic Ġcategory ĠâĢ ľ opp osite Ġadjective âĢ Ŀ , Ġwhich Ġwe Ġhave Ġnot Ġbeen Ġable Ġto Ġexplain . </s> E val uation Ġ: :: ĠNamed ĠEntity ĠRecogn ition </s> For Ġevaluation Ġof ĠEL Mo Ġmodels Ġon Ġa Ġrelevant Ġdownstream Ġtask , Ġwe Ġused Ġnamed Ġentity Ġrecognition Ġ( NER ) Ġtask . ĠN ER Ġis Ġan Ġinformation Ġextraction Ġtask Ġthat Ġseeks Ġto Ġlocate Ġand Ġclassify Ġnamed Ġentity Ġmentions Ġin Ġun struct ured Ġtext Ġinto Ġpre - defined Ġcategories Ġsuch Ġas Ġthe Ġperson Ġnames , Ġorganizations , Ġlocations , Ġmedical Ġcodes , Ġtime Ġexpressions , Ġquantities , Ġmonetary Ġvalues , Ġpercentages , Ġetc . ĠTo Ġallow Ġcomparison Ġof Ġresults Ġbetween Ġlanguages , Ġwe Ġused Ġan Ġadapted Ġversion Ġof Ġthis Ġtask , Ġwhich Ġuses Ġa Ġreduced Ġset Ġof Ġlabels , Ġavailable Ġin ĠN ER Ġdatasets Ġfor Ġall Ġprocessed Ġlanguages . ĠThe Ġlabels Ġin Ġthe Ġused ĠN ER Ġdatasets Ġare Ġsimplified Ġto Ġa Ġcommon Ġlabel Ġset Ġof Ġthree Ġlabels Ġ( person Ġ- ĠPER , Ġlocation Ġ- ĠLOC , Ġorganization Ġ- ĠOR G ). ĠEach Ġword Ġin Ġthe ĠN ER Ġdataset Ġis Ġlabeled Ġwith Ġone Ġof Ġthe Ġthree Ġmentioned Ġlabels Ġor Ġa Ġlabel Ġ' O ' Ġ( other , Ġi . e . Ġnot Ġa Ġnamed Ġentity ) Ġif Ġit Ġdoes Ġnot Ġfit Ġany Ġof Ġthe Ġother Ġthree Ġlabels . ĠThe Ġnumber Ġof Ġwords Ġhaving Ġeach Ġlabel Ġis Ġshown Ġin ĠTable ĠT AB REF 19 . </s> To Ġmeasure Ġthe Ġperformance Ġof ĠEL Mo Ġembed d ings Ġon Ġthe ĠN ER Ġtask Ġwe Ġproceeded Ġas Ġfollows . ĠWe Ġembedded Ġthe Ġtext Ġin Ġthe Ġdatasets Ġsentence Ġby Ġsentence , Ġproducing Ġthree Ġvectors Ġ( one Ġfrom Ġeach ĠEL Mo Ġlayer ) Ġfor Ġeach Ġtoken Ġin Ġa Ġsentence . ĠWe Ġcalculated Ġthe Ġaverage Ġof Ġthe Ġthree Ġvectors Ġand Ġused Ġit Ġas Ġthe Ġinput Ġof Ġour Ġrecognition Ġmodel . ĠThe Ġinput Ġlayer Ġwas Ġfollowed Ġby Ġa Ġsingle ĠL ST M Ġlayer Ġwith Ġ128 ĠL ST M Ġcells Ġand Ġa Ġdrop out Ġlayer , Ġrandomly Ġdropping Ġ10 % Ġof Ġthe Ġneurons Ġon Ġboth Ġthe Ġoutput Ġand Ġthe Ġrecurrent Ġbranch . ĠThe Ġfinal Ġlayer Ġof Ġour Ġmodel Ġwas Ġa Ġtime Ġdistributed Ġsoft max Ġlayer Ġwith Ġ4 Ġneurons . </s> We Ġused ĠAD AM Ġoptim iser ĠB IB REF 17 Ġwith Ġthe Ġlearning Ġrate Ġ0 . 01 Ġand Ġ$ 10 ^{ - 5 } $ Ġlearning Ġrate Ġdecay . ĠWe Ġused Ġcateg orical Ġcross - ent ropy Ġas Ġa Ġloss Ġfunction Ġand Ġtrained Ġthe Ġmodel Ġfor Ġ3 Ġepoch s . ĠWe Ġpresent Ġthe Ġresults Ġusing Ġthe ĠMacro Ġ$ F _ 1 $ Ġscore , Ġthat Ġis Ġthe Ġaverage Ġof Ġ$ F _ 1 $ - sc ores Ġfor Ġeach Ġof Ġthe Ġthree ĠNE Ġclasses Ġ( the Ġclass ĠOther Ġis Ġexcluded ). </s> Since Ġthe Ġdifferences Ġbetween Ġthe Ġtested Ġlanguages Ġdepend Ġmore Ġon Ġthe Ġproperties Ġof Ġthe ĠN ER Ġdatasets Ġthan Ġon Ġthe Ġquality Ġof Ġembed d ings , Ġwe Ġcan Ġnot Ġdirectly Ġcompare ĠEL Mo Ġmodels . ĠFor Ġthis Ġreason , Ġwe Ġtake Ġthe Ġnon - context ual Ġfast Text Ġembed d ings Ġas Ġa Ġbaseline Ġand Ġpredict Ġnamed Ġentities Ġusing Ġthem . ĠThe Ġarchitecture Ġof Ġthe Ġmodel Ġusing Ġfast Text Ġembed d ings Ġis Ġthe Ġsame Ġas Ġthe Ġone Ġusing ĠEL Mo Ġembed d ings , Ġexcept Ġthat Ġthe Ġinput Ġuses Ġ300 Ġdimensional Ġfast Text Ġembed ding Ġvectors , Ġand Ġthe Ġmodel Ġwas Ġtrained Ġfor Ġ5 Ġepoch s Ġ( instead Ġof Ġ3 Ġas Ġfor ĠEL Mo ). ĠIn Ġboth Ġcases Ġ( EL Mo Ġand Ġfast Text ) Ġwe Ġtrained Ġand Ġevaluated Ġthe Ġmodel Ġfive Ġtimes , Ġbecause Ġthere Ġis Ġsome Ġrandom Ġcomponent Ġinvolved Ġin Ġinitialization Ġof Ġthe Ġneural Ġnetwork Ġmodel . ĠBy Ġtraining Ġand Ġevaluating Ġmultiple Ġtimes , Ġwe Ġminim ise Ġthis Ġrandom Ġcomponent . </s> The Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 21 . ĠWe Ġincluded Ġthe Ġevaluation Ġof Ġthe Ġoriginal ĠEL Mo ĠEnglish Ġmodel Ġin Ġthe Ġsame Ġtable . ĠN ER Ġmodels Ġhave Ġlittle Ġdifficulty Ġdistinguishing Ġbetween Ġtypes Ġof Ġnamed Ġentities , Ġbut Ġrecognizing Ġwhether Ġa Ġword Ġis Ġa Ġnamed Ġentity Ġor Ġnot Ġis Ġmore Ġdifficult . ĠFor Ġlanguages Ġwith Ġthe Ġsmallest ĠN ER Ġdatasets , ĠCroatian Ġand ĠLithuan ian , ĠEL Mo Ġembed d ings Ġshow Ġthe Ġlargest Ġimprovement Ġover Ġfast Text Ġembed d ings . ĠHowever , Ġwe Ġcan Ġobserve Ġsignificant Ġimprovements Ġwith ĠEL Mo Ġalso Ġon ĠEnglish Ġand ĠFinnish , Ġwhich Ġare Ġamong Ġthe Ġlargest Ġdatasets Ġ( English Ġbeing Ġby Ġfar Ġthe Ġlargest ). ĠOnly Ġon ĠSloven ian Ġdataset Ġdid ĠEL Mo Ġperform Ġslightly Ġworse Ġthan Ġfast Text , Ġon Ġall Ġother ĠE MB ED D IA Ġlanguages , Ġthe ĠEL Mo Ġembed d ings Ġimprove Ġthe Ġresults . </s> Conclusion </s> We Ġprepared Ġpre com puted ĠEL Mo Ġcontextual Ġembed d ings Ġfor Ġseven Ġlanguages : ĠCroatian , ĠEston ian , ĠFinnish , ĠLat v ian , ĠLithuan ian , ĠSloven ian , Ġand ĠSwedish . ĠWe Ġpresent Ġthe Ġnecessary Ġbackground Ġon Ġembed d ings Ġand Ġcontextual Ġembed d ings , Ġthe Ġdetails Ġof Ġtraining Ġthe Ġembed ding Ġmodels , Ġand Ġtheir Ġevaluation . ĠWe Ġshow Ġthat Ġthe Ġsize Ġof Ġused Ġtraining Ġsets Ġimportantly Ġaffects Ġthe Ġquality Ġof Ġproduced Ġembed d ings , Ġand Ġtherefore Ġthe Ġexisting Ġpublicly Ġavailable ĠEL Mo Ġembed d ings Ġfor Ġthe Ġprocessed Ġlanguages Ġare Ġinadequate . ĠWe Ġtrained Ġnew ĠEL Mo Ġembed d ings Ġon Ġlarger Ġtraining Ġsets Ġand Ġanalysed Ġtheir Ġproperties Ġon Ġthe Ġanalogy Ġtask Ġand Ġon Ġthe ĠN ER Ġtask . ĠThe Ġresults Ġshow Ġthat Ġthe Ġnewly Ġproduced Ġcontextual Ġembed d ings Ġproduce Ġsubstantially Ġbetter Ġresults Ġcompared Ġto Ġthe Ġnon - context ual Ġfast Text Ġbaseline . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġuse Ġthe Ġproduced Ġcontextual Ġembed d ings Ġon Ġthe Ġproblems Ġof Ġnews Ġmedia Ġindustry . ĠThe Ġpret rained ĠEL Mo Ġmodels Ġwill Ġbe Ġdeposited Ġto Ġthe ĠCL AR IN Ġrepository Ġby Ġthe Ġtime Ġof Ġthe Ġfinal Ġversion Ġof Ġthis Ġpaper . </s> Acknowled gments </s> The Ġwork Ġwas Ġpartially Ġsupported Ġby Ġthe ĠSloven ian ĠResearch ĠAgency Ġ( AR RS ) Ġcore Ġresearch Ġprogramme ĠP 6 - 04 11 . ĠThis Ġpaper Ġis Ġsupported Ġby ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġunder Ġgrant Ġagreement ĠNo Ġ8 25 153 , Ġproject ĠE MB ED D IA Ġ( Cross - L ing ual ĠEmb edd ings Ġfor ĠLess - Represent ed ĠLanguages Ġin ĠEuropean ĠNews ĠMedia ). ĠThe Ġresults Ġof Ġthis Ġpublication Ġreflects Ġonly Ġthe Ġauthors ' Ġview Ġand Ġthe ĠEU ĠCommission Ġis Ġnot Ġresponsible Ġfor Ġany Ġuse Ġthat Ġmay Ġbe Ġmade Ġof Ġthe Ġinformation Ġit Ġcontains .
Introduction Word embeddings are representations of words in numerical form, as vectors of typically several hundred dimensions. The vectors are used as an input to machine learning models; for complex language processing tasks these are typically deep neural networks. The embedding vectors are obtained from specialized learning tasks, based on neural networks, e.g., word2vec BIBREF0, GloVe BIBREF1, FastText BIBREF2, ELMo BIBREF3, and BERT BIBREF4. For training, the embeddings algorithms use large monolingual corpora that encode important information about word meaning as distances between vectors. In order to enable downstream machine learning on text understanding tasks, the embeddings shall preserve semantic relations between words, and this is true even across languages. Probably the best known word embeddings are produced by the word2vec method BIBREF5. The problem with word2vec embeddings is their failure to express polysemous words. During training of an embedding, all senses of a given word (e.g., paper as a material, as a newspaper, as a scientific work, and as an exam) contribute relevant information in proportion to their frequency in the training corpus. This causes the final vector to be placed somewhere in the weighted middle of all words' meanings. Consequently, rare meanings of words are poorly expressed with word2vec and the resulting vectors do not offer good semantic representations. For example, none of the 50 closest vectors of the word paper is related to science. The idea of contextual embeddings is to generate a different vector for each context a word appears in and the context is typically defined sentence-wise. To a large extent, this solves the problems with word polysemy, i.e. the context of a sentence is typically enough to disambiguate different meanings of a word for humans and so it is for the learning algorithms. In this work, we describe high-quality models for contextual embeddings, called ELMo BIBREF3, precomputed for seven morphologically rich, less-resourced languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian, and Swedish. ELMo is one of the most successful approaches to contextual word embeddings. At time of its creation, ELMo has been shown to outperform previous word embeddings BIBREF3 like word2vec and GloVe on many NLP tasks, e.g., question answering, named entity extraction, sentiment analysis, textual entailment, semantic role labeling, and coreference resolution. This report is split into further five sections. In section SECREF2, we describe the contextual embeddings ELMo. In Section SECREF3, we describe the datasets used and in Section SECREF4 we describe preprocessing and training of the embeddings. We describe the methodology for evaluation of created vectors and results in Section SECREF5. We present conclusion in Section SECREF6 where we also outline plans for further work. ELMo Typical word embeddings models or representations, such as word2vec BIBREF0, GloVe BIBREF1, or FastText BIBREF2, are fast to train and have been pre-trained for a number of different languages. They do not capture the context, though, so each word is always given the same vector, regardless of its context or meaning. This is especially problematic for polysemous words. ELMo (Embeddings from Language Models) embedding BIBREF3 is one of the state-of-the-art pretrained transfer learning models, that remedies the problem and introduces a contextual component. ELMo model`s architecture consists of three neural network layers. The output of the model after each layer gives one set of embeddings, altogether three sets. The first layer is a CNN layer, which operates on a character level. It is context independent, so each word always gets the same embedding, regardless of its context. It is followed by two biLM layers. A biLM layer consists of two concatenated LSTMs. In the first LSTM, we try to predict the following word, based on the given past words, where each word is represented by the embeddings from the CNN layer. In the second LSTM, we try to predict the preceding word, based on the given following words. It is equivalent to the first LSTM, just reading the text in reverse. In NLP tasks, any set of these embeddings may be used; however, a weighted average is usually used. The weights of the average are learned during the training of the model for the specific task. Additionally, an entire ELMo model can be fine-tuned on a specific end task. Although ELMo is trained on character level and is able to handle out-of-vocabulary words, a vocabulary file containing most common tokens is used for efficiency during training and embedding generation. The original ELMo model was trained on a one billion word large English corpus, with a given vocabulary file of about 800,000 words. Later, ELMo models for other languages were trained as well, but limited to larger languages with many resources, like German and Japanese. ELMo ::: ELMoForManyLangs Recently, ELMoForManyLangs BIBREF6 project released pre-trained ELMo models for a number of different languages BIBREF7. These models, however, were trained on a significantly smaller datasets. They used 20-million-words data randomly sampled from the raw text released by the CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, which is a combination of Wikipedia dump and common crawl. The quality of these models is questionable. For example, we compared the Latvian model by ELMoForManyLangs with a model we trained on a complete (wikidump + common crawl) Latvian corpus, which has about 280 million tokens. The difference of each model on the word analogy task is shown in Figure FIGREF16 in Section SECREF5. As the results of the ELMoForManyLangs embeddings are significantly worse than using the full corpus, we can conclude that these embeddings are not of sufficient quality. For that reason, we computed ELMo embeddings for seven languages on much larger corpora. As this effort requires access to large amount of textual data and considerable computational resources, we made the precomputed models publicly available by depositing them to Clarin repository. Training Data We trained ELMo models for seven languages: Slovenian, Croatian, Finnish, Estonian, Latvian, Lithuanian and Swedish. To obtain high-quality embeddings, we used large monolingual corpora from various sources for each language. Some corpora are available online under permissive licences, others are available only for research purposes or have limited availability. The corpora used in training datasets are a mix of news articles and general web crawl, which we preprocessed and deduplicated. Below we shortly describe the used corpora in alphabetical order of the involved languages. Their names and sizes are summarized in Table TABREF3. Croatian dataset include hrWaC 2.1 corpus BIBREF9, Riznica BIBREF10, and articles of Croatian branch of Styria media house, made available to us through partnership in a joint project. hrWaC was built by crawling the .hr internet domain in 2011 and 2014. Riznica is composed of Croatian fiction and non-fiction prose, poetry, drama, textbooks, manuals, etc. The Styria dataset consists of 570,219 news articles published on the Croatian 24sata news portal and niche portals related to 24sata. Estonian dataset contains texts from two sources, CoNLL 2017 Shared Task - Automatically Annotated Raw Texts and Word Embeddings BIBREF8, and news articles made available to us by Ekspress Meedia due to partnership in the project. Ekspress Meedia dataset is composed of Estonian news articles between years 2009 and 2019. The CoNLL 2017 corpus is composed of Estonian Wikipedia and webcrawl. Finnish dataset contains articles by Finnish news agency STT, Finnish part of the CoNLL 2017 dataset, and Ylilauta downloadable version BIBREF11. STT news articles were published between years 1992 and 2018. Ylilauta is a Finnish online discussion board; the corpus contains parts of the discussions from 2012 to 2014. Latvian dataset consists only of the Latvian portion of the ConLL 2017 corpus. Lithuanian dataset is composed of Lithuanian Wikipedia articles from 2018, DGT-UD corpus, and LtTenTen. DGT-UD is a parallel corpus of 23 official languages of the EU, composed of JRC DGT translation memory of European law, automatically annotated with UD-Pipe 1.2. LtTenTen is Lithuanian web corpus made up of texts collected from the internet in April 2014 BIBREF12. Slovene dataset is formed from the Gigafida 2.0 corpus BIBREF13. It is a general language corpus composed of various sources, mostly newspapers, internet pages, and magazines, but also fiction and non-fiction prose, textbooks, etc. Swedish dataset is composed of STT Swedish articles and Swedish part of CoNLL 2017. The Finnish news agency STT publishes some of its articles in Swedish language. They were made available to us through partnership in a joint project. The corpus contains those articles from 1992 to 2017. Preprocessing and Training Prior to training the ELMo models, we sentence and word tokenized all the datasets. The text was formatted in such a way that each sentence was in its own line with tokens separated by white spaces. CoNLL 2017, DGT-UD and LtTenTen14 corpora were already pre-tokenized. We tokenized the others using the NLTK library and its tokenizers for each of the languages. There is no tokenizer for Croatian in NLTK library, so we used Slovene tokenizer instead. After tokenization, we deduplicated the datasets for each language separately, using the Onion (ONe Instance ONly) tool for text deduplication. We applied the tool on paragraph level for corpora that did not have sentences shuffled and on sentence level for the rest. We considered 9-grams with duplicate content threshold of 0.9. For each language we prepared a vocabulary file, containing roughly one million most common tokens, i.e. tokens that appear at least $n$ times in the corpus, where $n$ is between 15 and 25, depending on the dataset size. We included the punctuation marks among the tokens. We trained each ELMo model using default values used to train the original English ELMo (large) model. Evaluation We evaluated the produced ELMo models for all languages using two evaluation tasks: a word analogy task and named entity recognition (NER) task. Below, we first shortly describe each task, followed by the evaluation results. Evaluation ::: Word Analogy Task The word analogy task was popularized by mikolov2013distributed. The goal is to find a term $y$ for a given term $x$ so that the relationship between $x$ and $y$ best resembles the given relationship $a : b$. There are two main groups of categories: 5 semantic and 10 syntactic. To illustrate a semantic relationship, consider for example that the word pair $a : b$ is given as “Finland : Helsinki”. The task is to find the term $y$ corresponding to the relationship “Sweden : $y$”, with the expected answer being $y=$ Stockholm. In syntactic categories, the two words in a pair have a common stem (in some cases even same lemma), with all the pairs in a given category having the same morphological relationship. For example, given the word pair “long : longer”, we see that we have an adjective in its base form and the same adjective in a comparative form. That task is then to find the term $y$ corresponding to the relationship “dark : $y$”, with the expected answer being $y=$ darker, that is a comparative form of the adjective dark. In the vector space, the analogy task is transformed into vector arithmetic and search for nearest neighbours, i.e. we compute the distance between vectors: d(vec(Finland), vec(Helsinki)) and search for word $y$ which would give the closest result in distance d(vec(Sweden), vec($y$)). In the analogy dataset the analogies are already pre-specified, so we are measuring how close are the given pairs. In the evaluation below, we use analogy datasets for all tested languages based on the English dataset by BIBREF14 . Due to English-centered bias of this dataset, we used a modified dataset which was first written in Slovene language and then translated into other languages BIBREF15. As each instance of analogy contains only four words, without any context, the contextual models (such as ELMo) do not have enough context to generate sensible embeddings. We therefore used some additional text to form simple sentences using the four analogy words, while taking care that their noun case stays the same. For example, for the words "Rome", "Italy", "Paris" and "France" (forming the analogy Rome is to Italy as Paris is to $x$, where the correct answer is $x=$France), we formed the sentence "If the word Rome corresponds to the word Italy, then the word Paris corresponds to the word France". We generated embeddings for those four words in the constructed sentence, substituted the last word with each word in our vocabulary and generated the embeddings again. As typical for non-contextual analogy task, we measure the cosine distance ($d$) between the last word ($w_4$) and the combination of the first three words ($w_2-w_1+w_3$). We use the CSLS metric BIBREF16 to find the closest candidate word ($w_4$). If we find the correct word among the five closest words, we consider that entry as successfully identified. The proportion of correctly identified words forms a statistic called accuracy@5, which we report as the result. We first compare existing Latvian ELMo embeddings from ELMoForManyLangs project with our Latvian embeddings, followed by the detailed analysis of our ELMo embeddings. We trained Latvian ELMo using only CoNLL 2017 corpora. Since this is the only language, where we trained the embedding model on exactly the same corpora as ELMoForManyLangs models, we chose it for comparison between our ELMo model with ELMoForManyLangs. In other languages, additional or other corpora were used, so a direct comparison would also reflect the quality of the corpora used for training. In Latvian, however, only the size of the training dataset is different. ELMoForManyLangs uses only 20 million tokens and we use the whole corpus of 270 million tokens. The Latvian ELMo model from ELMoForManyLangs project performs significantly worse than EMBEDDIA ELMo Latvian model on all categories of word analogy task (Figure FIGREF16). We also include the comparison with our Estonian ELMo embeddings in the same figure. This comparison shows that while differences between our Latvian and Estonian embeddings can be significant for certain categories, the accuracy score of ELMoForManyLangs is always worse than either of our models. The comparison of Estonian and Latvian models leads us to believe that a few hundred million tokens is a sufficiently large corpus to train ELMo models (at least for word analogy task), but 20-million token corpora used in ELMoForManyLangs are too small. The results for all languages and all ELMo layers, averaged over semantic and syntactic categories, are shown in Table TABREF17. The embeddings after the first LSTM layer perform best in semantic categories. In syntactic categories, the non-contextual CNN layer performs the best. Syntactic categories are less context dependent and much more morphology and syntax based, so it is not surprising that the non-contextual layer performs well. The second LSTM layer embeddings perform the worst in syntactic categories, though still outperforming CNN layer embeddings in semantic categories. Latvian ELMo performs worse compared to other languages we trained, especially in semantic categories, presumably due to smaller training data size. Surprisingly, the original English ELMo performs very poorly in syntactic categories and only outperforms Latvian in semantic categories. The low score can be partially explained by English model scoring $0.00$ in one syntactic category “opposite adjective”, which we have not been able to explain. Evaluation ::: Named Entity Recognition For evaluation of ELMo models on a relevant downstream task, we used named entity recognition (NER) task. NER is an information extraction task that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc. To allow comparison of results between languages, we used an adapted version of this task, which uses a reduced set of labels, available in NER datasets for all processed languages. The labels in the used NER datasets are simplified to a common label set of three labels (person - PER, location - LOC, organization - ORG). Each word in the NER dataset is labeled with one of the three mentioned labels or a label 'O' (other, i.e. not a named entity) if it does not fit any of the other three labels. The number of words having each label is shown in Table TABREF19. To measure the performance of ELMo embeddings on the NER task we proceeded as follows. We embedded the text in the datasets sentence by sentence, producing three vectors (one from each ELMo layer) for each token in a sentence. We calculated the average of the three vectors and used it as the input of our recognition model. The input layer was followed by a single LSTM layer with 128 LSTM cells and a dropout layer, randomly dropping 10% of the neurons on both the output and the recurrent branch. The final layer of our model was a time distributed softmax layer with 4 neurons. We used ADAM optimiser BIBREF17 with the learning rate 0.01 and $10^{-5}$ learning rate decay. We used categorical cross-entropy as a loss function and trained the model for 3 epochs. We present the results using the Macro $F_1$ score, that is the average of $F_1$-scores for each of the three NE classes (the class Other is excluded). Since the differences between the tested languages depend more on the properties of the NER datasets than on the quality of embeddings, we can not directly compare ELMo models. For this reason, we take the non-contextual fastText embeddings as a baseline and predict named entities using them. The architecture of the model using fastText embeddings is the same as the one using ELMo embeddings, except that the input uses 300 dimensional fastText embedding vectors, and the model was trained for 5 epochs (instead of 3 as for ELMo). In both cases (ELMo and fastText) we trained and evaluated the model five times, because there is some random component involved in initialization of the neural network model. By training and evaluating multiple times, we minimise this random component. The results are presented in Table TABREF21. We included the evaluation of the original ELMo English model in the same table. NER models have little difficulty distinguishing between types of named entities, but recognizing whether a word is a named entity or not is more difficult. For languages with the smallest NER datasets, Croatian and Lithuanian, ELMo embeddings show the largest improvement over fastText embeddings. However, we can observe significant improvements with ELMo also on English and Finnish, which are among the largest datasets (English being by far the largest). Only on Slovenian dataset did ELMo perform slightly worse than fastText, on all other EMBEDDIA languages, the ELMo embeddings improve the results. Conclusion We prepared precomputed ELMo contextual embeddings for seven languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Slovenian, and Swedish. We present the necessary background on embeddings and contextual embeddings, the details of training the embedding models, and their evaluation. We show that the size of used training sets importantly affects the quality of produced embeddings, and therefore the existing publicly available ELMo embeddings for the processed languages are inadequate. We trained new ELMo embeddings on larger training sets and analysed their properties on the analogy task and on the NER task. The results show that the newly produced contextual embeddings produce substantially better results compared to the non-contextual fastText baseline. In future work, we plan to use the produced contextual embeddings on the problems of news media industry. The pretrained ELMo models will be deposited to the CLARIN repository by the time of the final version of this paper. Acknowledgments The work was partially supported by the Slovenian Research Agency (ARRS) core research programme P6-0411. This paper is supported by European Union's Horizon 2020 research and innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media). The results of this publication reflects only the authors' view and the EU Commission is not responsible for any use that may be made of the information it contains.
What is the improvement in performance for Estonian in the NER task?
09a1173e971e0fcdbf2fbecb1b077158ab08f497
What Ġis Ġthe Ġimprovement Ġin Ġperformance Ġfor ĠEston ian Ġin Ġthe ĠN ER Ġtask ?
5 Ġpercent Ġpoints .
<s> What Ġis Ġthe Ġimprovement Ġin Ġperformance Ġfor ĠEston ian Ġin Ġthe ĠN ER Ġtask ? </s> Introduction </s> Word Ġembed d ings Ġare Ġrepresentations Ġof Ġwords Ġin Ġnumerical Ġform , Ġas Ġvectors Ġof Ġtypically Ġseveral Ġhundred Ġdimensions . ĠThe Ġvectors Ġare Ġused Ġas Ġan Ġinput Ġto Ġmachine Ġlearning Ġmodels ; Ġfor Ġcomplex Ġlanguage Ġprocessing Ġtasks Ġthese Ġare Ġtypically Ġdeep Ġneural Ġnetworks . ĠThe Ġembed ding Ġvectors Ġare Ġobtained Ġfrom Ġspecialized Ġlearning Ġtasks , Ġbased Ġon Ġneural Ġnetworks , Ġe . g ., Ġword 2 vec ĠB IB REF 0 , ĠGl o Ve ĠB IB REF 1 , ĠFast Text ĠB IB REF 2 , ĠEL Mo ĠB IB REF 3 , Ġand ĠB ERT ĠB IB REF 4 . ĠFor Ġtraining , Ġthe Ġembed d ings Ġalgorithms Ġuse Ġlarge Ġmon oling ual Ġcorpor a Ġthat Ġencode Ġimportant Ġinformation Ġabout Ġword Ġmeaning Ġas Ġdistances Ġbetween Ġvectors . ĠIn Ġorder Ġto Ġenable Ġdownstream Ġmachine Ġlearning Ġon Ġtext Ġunderstanding Ġtasks , Ġthe Ġembed d ings Ġshall Ġpreserve Ġsemantic Ġrelations Ġbetween Ġwords , Ġand Ġthis Ġis Ġtrue Ġeven Ġacross Ġlanguages . </s> Probably Ġthe Ġbest Ġknown Ġword Ġembed d ings Ġare Ġproduced Ġby Ġthe Ġword 2 vec Ġmethod ĠB IB REF 5 . ĠThe Ġproblem Ġwith Ġword 2 vec Ġembed d ings Ġis Ġtheir Ġfailure Ġto Ġexpress Ġpoly sem ous Ġwords . ĠDuring Ġtraining Ġof Ġan Ġembed ding , Ġall Ġsenses Ġof Ġa Ġgiven Ġword Ġ( e . g ., Ġpaper Ġas Ġa Ġmaterial , Ġas Ġa Ġnewspaper , Ġas Ġa Ġscientific Ġwork , Ġand Ġas Ġan Ġexam ) Ġcontribute Ġrelevant Ġinformation Ġin Ġproportion Ġto Ġtheir Ġfrequency Ġin Ġthe Ġtraining Ġcorpus . ĠThis Ġcauses Ġthe Ġfinal Ġvector Ġto Ġbe Ġplaced Ġsomewhere Ġin Ġthe Ġweighted Ġmiddle Ġof Ġall Ġwords ' Ġmeanings . ĠConsequently , Ġrare Ġmeanings Ġof Ġwords Ġare Ġpoorly Ġexpressed Ġwith Ġword 2 vec Ġand Ġthe Ġresulting Ġvectors Ġdo Ġnot Ġoffer Ġgood Ġsemantic Ġrepresentations . ĠFor Ġexample , Ġnone Ġof Ġthe Ġ50 Ġclosest Ġvectors Ġof Ġthe Ġword Ġpaper Ġis Ġrelated Ġto Ġscience . </s> The Ġidea Ġof Ġcontextual Ġembed d ings Ġis Ġto Ġgenerate Ġa Ġdifferent Ġvector Ġfor Ġeach Ġcontext Ġa Ġword Ġappears Ġin Ġand Ġthe Ġcontext Ġis Ġtypically Ġdefined Ġsentence - wise . ĠTo Ġa Ġlarge Ġextent , Ġthis Ġsolves Ġthe Ġproblems Ġwith Ġword Ġpoly se my , Ġi . e . Ġthe Ġcontext Ġof Ġa Ġsentence Ġis Ġtypically Ġenough Ġto Ġdis amb ig uate Ġdifferent Ġmeanings Ġof Ġa Ġword Ġfor Ġhumans Ġand Ġso Ġit Ġis Ġfor Ġthe Ġlearning Ġalgorithms . ĠIn Ġthis Ġwork , Ġwe Ġdescribe Ġhigh - quality Ġmodels Ġfor Ġcontextual Ġembed d ings , Ġcalled ĠEL Mo ĠB IB REF 3 , Ġpre com puted Ġfor Ġseven Ġmorph ologically Ġrich , Ġless - res ourced Ġlanguages : ĠSloven ian , ĠCroatian , ĠFinnish , ĠEston ian , ĠLat v ian , ĠLithuan ian , Ġand ĠSwedish . ĠEL Mo Ġis Ġone Ġof Ġthe Ġmost Ġsuccessful Ġapproaches Ġto Ġcontextual Ġword Ġembed d ings . ĠAt Ġtime Ġof Ġits Ġcreation , ĠEL Mo Ġhas Ġbeen Ġshown Ġto Ġoutper form Ġprevious Ġword Ġembed d ings ĠB IB REF 3 Ġlike Ġword 2 vec Ġand ĠGl o Ve Ġon Ġmany ĠN LP Ġtasks , Ġe . g ., Ġquestion Ġanswering , Ġnamed Ġentity Ġextraction , Ġsentiment Ġanalysis , Ġtextual Ġentail ment , Ġsemantic Ġrole Ġlabeling , Ġand Ġcore ference Ġresolution . </s> This Ġreport Ġis Ġsplit Ġinto Ġfurther Ġfive Ġsections . ĠIn Ġsection ĠSEC REF 2 , Ġwe Ġdescribe Ġthe Ġcontextual Ġembed d ings ĠEL Mo . ĠIn ĠSection ĠSEC REF 3 , Ġwe Ġdescribe Ġthe Ġdatasets Ġused Ġand Ġin ĠSection ĠSEC REF 4 Ġwe Ġdescribe Ġpre processing Ġand Ġtraining Ġof Ġthe Ġembed d ings . ĠWe Ġdescribe Ġthe Ġmethodology Ġfor Ġevaluation Ġof Ġcreated Ġvectors Ġand Ġresults Ġin ĠSection ĠSEC REF 5 . ĠWe Ġpresent Ġconclusion Ġin ĠSection ĠSEC REF 6 Ġwhere Ġwe Ġalso Ġoutline Ġplans Ġfor Ġfurther Ġwork . </s> EL Mo </s> Typ ical Ġword Ġembed d ings Ġmodels Ġor Ġrepresentations , Ġsuch Ġas Ġword 2 vec ĠB IB REF 0 , ĠGl o Ve ĠB IB REF 1 , Ġor ĠFast Text ĠB IB REF 2 , Ġare Ġfast Ġto Ġtrain Ġand Ġhave Ġbeen Ġpre - trained Ġfor Ġa Ġnumber Ġof Ġdifferent Ġlanguages . ĠThey Ġdo Ġnot Ġcapture Ġthe Ġcontext , Ġthough , Ġso Ġeach Ġword Ġis Ġalways Ġgiven Ġthe Ġsame Ġvector , Ġregardless Ġof Ġits Ġcontext Ġor Ġmeaning . ĠThis Ġis Ġespecially Ġproblematic Ġfor Ġpoly sem ous Ġwords . ĠEL Mo Ġ( Emb edd ings Ġfrom ĠLanguage ĠModels ) Ġembed ding ĠB IB REF 3 Ġis Ġone Ġof Ġthe Ġstate - of - the - art Ġpret rained Ġtransfer Ġlearning Ġmodels , Ġthat Ġremedies Ġthe Ġproblem Ġand Ġintroduces Ġa Ġcontextual Ġcomponent . </s> EL Mo Ġmodel ` s Ġarchitecture Ġconsists Ġof Ġthree Ġneural Ġnetwork Ġlayers . ĠThe Ġoutput Ġof Ġthe Ġmodel Ġafter Ġeach Ġlayer Ġgives Ġone Ġset Ġof Ġembed d ings , Ġaltogether Ġthree Ġsets . ĠThe Ġfirst Ġlayer Ġis Ġa ĠCNN Ġlayer , Ġwhich Ġoperates Ġon Ġa Ġcharacter Ġlevel . ĠIt Ġis Ġcontext Ġindependent , Ġso Ġeach Ġword Ġalways Ġgets Ġthe Ġsame Ġembed ding , Ġregardless Ġof Ġits Ġcontext . ĠIt Ġis Ġfollowed Ġby Ġtwo Ġbi LM Ġlayers . ĠA Ġbi LM Ġlayer Ġconsists Ġof Ġtwo Ġconc aten ated ĠL ST Ms . ĠIn Ġthe Ġfirst ĠL ST M , Ġwe Ġtry Ġto Ġpredict Ġthe Ġfollowing Ġword , Ġbased Ġon Ġthe Ġgiven Ġpast Ġwords , Ġwhere Ġeach Ġword Ġis Ġrepresented Ġby Ġthe Ġembed d ings Ġfrom Ġthe ĠCNN Ġlayer . ĠIn Ġthe Ġsecond ĠL ST M , Ġwe Ġtry Ġto Ġpredict Ġthe Ġpreceding Ġword , Ġbased Ġon Ġthe Ġgiven Ġfollowing Ġwords . ĠIt Ġis Ġequivalent Ġto Ġthe Ġfirst ĠL ST M , Ġjust Ġreading Ġthe Ġtext Ġin Ġreverse . </s> In ĠN LP Ġtasks , Ġany Ġset Ġof Ġthese Ġembed d ings Ġmay Ġbe Ġused ; Ġhowever , Ġa Ġweighted Ġaverage Ġis Ġusually Ġused . ĠThe Ġweights Ġof Ġthe Ġaverage Ġare Ġlearned Ġduring Ġthe Ġtraining Ġof Ġthe Ġmodel Ġfor Ġthe Ġspecific Ġtask . ĠAdditionally , Ġan Ġentire ĠEL Mo Ġmodel Ġcan Ġbe Ġfine - tun ed Ġon Ġa Ġspecific Ġend Ġtask . </s> Although ĠEL Mo Ġis Ġtrained Ġon Ġcharacter Ġlevel Ġand Ġis Ġable Ġto Ġhandle Ġout - of - voc abulary Ġwords , Ġa Ġvocabulary Ġfile Ġcontaining Ġmost Ġcommon Ġtokens Ġis Ġused Ġfor Ġefficiency Ġduring Ġtraining Ġand Ġembed ding Ġgeneration . ĠThe Ġoriginal ĠEL Mo Ġmodel Ġwas Ġtrained Ġon Ġa Ġone Ġbillion Ġword Ġlarge ĠEnglish Ġcorpus , Ġwith Ġa Ġgiven Ġvocabulary Ġfile Ġof Ġabout Ġ800 , 000 Ġwords . ĠLater , ĠEL Mo Ġmodels Ġfor Ġother Ġlanguages Ġwere Ġtrained Ġas Ġwell , Ġbut Ġlimited Ġto Ġlarger Ġlanguages Ġwith Ġmany Ġresources , Ġlike ĠGerman Ġand ĠJapanese . </s> EL Mo Ġ: :: ĠEL Mo For Many L angs </s> Recently , ĠEL Mo For Many L angs ĠB IB REF 6 Ġproject Ġreleased Ġpre - trained ĠEL Mo Ġmodels Ġfor Ġa Ġnumber Ġof Ġdifferent Ġlanguages ĠB IB REF 7 . ĠThese Ġmodels , Ġhowever , Ġwere Ġtrained Ġon Ġa Ġsignificantly Ġsmaller Ġdatasets . ĠThey Ġused Ġ20 - million - words Ġdata Ġrandomly Ġsampled Ġfrom Ġthe Ġraw Ġtext Ġreleased Ġby Ġthe ĠCo N LL Ġ2017 ĠShared ĠTask Ġ- ĠAutom atically ĠAn not ated ĠRaw ĠText s Ġand ĠWord ĠEmb edd ings ĠB IB REF 8 , Ġwhich Ġis Ġa Ġcombination Ġof ĠWikipedia Ġdump Ġand Ġcommon Ġcrawl . ĠThe Ġquality Ġof Ġthese Ġmodels Ġis Ġquestionable . ĠFor Ġexample , Ġwe Ġcompared Ġthe ĠLat v ian Ġmodel Ġby ĠEL Mo For Many L angs Ġwith Ġa Ġmodel Ġwe Ġtrained Ġon Ġa Ġcomplete Ġ( wik id ump Ġ+ Ġcommon Ġcrawl ) ĠLat v ian Ġcorpus , Ġwhich Ġhas Ġabout Ġ280 Ġmillion Ġtokens . ĠThe Ġdifference Ġof Ġeach Ġmodel Ġon Ġthe Ġword Ġanalogy Ġtask Ġis Ġshown Ġin ĠFigure ĠFIG REF 16 Ġin ĠSection ĠSEC REF 5 . ĠAs Ġthe Ġresults Ġof Ġthe ĠEL Mo For Many L angs Ġembed d ings Ġare Ġsignificantly Ġworse Ġthan Ġusing Ġthe Ġfull Ġcorpus , Ġwe Ġcan Ġconclude Ġthat Ġthese Ġembed d ings Ġare Ġnot Ġof Ġsufficient Ġquality . ĠFor Ġthat Ġreason , Ġwe Ġcomputed ĠEL Mo Ġembed d ings Ġfor Ġseven Ġlanguages Ġon Ġmuch Ġlarger Ġcorpor a . ĠAs Ġthis Ġeffort Ġrequires Ġaccess Ġto Ġlarge Ġamount Ġof Ġtextual Ġdata Ġand Ġconsiderable Ġcomputational Ġresources , Ġwe Ġmade Ġthe Ġpre com puted Ġmodels Ġpublicly Ġavailable Ġby Ġdepos iting Ġthem Ġto ĠClar in Ġrepository . </s> Training ĠData </s> We Ġtrained ĠEL Mo Ġmodels Ġfor Ġseven Ġlanguages : ĠSloven ian , ĠCroatian , ĠFinnish , ĠEston ian , ĠLat v ian , ĠLithuan ian Ġand ĠSwedish . ĠTo Ġobtain Ġhigh - quality Ġembed d ings , Ġwe Ġused Ġlarge Ġmon oling ual Ġcorpor a Ġfrom Ġvarious Ġsources Ġfor Ġeach Ġlanguage . ĠSome Ġcorpor a Ġare Ġavailable Ġonline Ġunder Ġper missive Ġlicences , Ġothers Ġare Ġavailable Ġonly Ġfor Ġresearch Ġpurposes Ġor Ġhave Ġlimited Ġavailability . ĠThe Ġcorpor a Ġused Ġin Ġtraining Ġdatasets Ġare Ġa Ġmix Ġof Ġnews Ġarticles Ġand Ġgeneral Ġweb Ġcrawl , Ġwhich Ġwe Ġpre process ed Ġand Ġded u pl icated . ĠBelow Ġwe Ġshortly Ġdescribe Ġthe Ġused Ġcorpor a Ġin Ġalphabet ical Ġorder Ġof Ġthe Ġinvolved Ġlanguages . ĠTheir Ġnames Ġand Ġsizes Ġare Ġsummarized Ġin ĠTable ĠT AB REF 3 . </s> Cro at ian Ġdataset Ġinclude Ġhr Wa C Ġ2 . 1 Ġcorpus ĠB IB REF 9 , ĠR iz n ica ĠB IB REF 10 , Ġand Ġarticles Ġof ĠCroatian Ġbranch Ġof ĠSty ria Ġmedia Ġhouse , Ġmade Ġavailable Ġto Ġus Ġthrough Ġpartnership Ġin Ġa Ġjoint Ġproject . Ġhr Wa C Ġwas Ġbuilt Ġby Ġcrawling Ġthe Ġ. hr Ġinternet Ġdomain Ġin Ġ2011 Ġand Ġ2014 . ĠR iz n ica Ġis Ġcomposed Ġof ĠCroatian Ġfiction Ġand Ġnon - fiction Ġprose , Ġpoetry , Ġdrama , Ġtextbooks , Ġmanuals , Ġetc . ĠThe ĠSty ria Ġdataset Ġconsists Ġof Ġ570 , 219 Ġnews Ġarticles Ġpublished Ġon Ġthe ĠCroatian Ġ24 s ata Ġnews Ġportal Ġand Ġniche Ġportals Ġrelated Ġto Ġ24 s ata . </s> E ston ian Ġdataset Ġcontains Ġtexts Ġfrom Ġtwo Ġsources , ĠCo N LL Ġ2017 ĠShared ĠTask Ġ- ĠAutom atically ĠAn not ated ĠRaw ĠText s Ġand ĠWord ĠEmb edd ings ĠB IB REF 8 , Ġand Ġnews Ġarticles Ġmade Ġavailable Ġto Ġus Ġby ĠE ks press ĠMe edia Ġdue Ġto Ġpartnership Ġin Ġthe Ġproject . ĠE ks press ĠMe edia Ġdataset Ġis Ġcomposed Ġof ĠEston ian Ġnews Ġarticles Ġbetween Ġyears Ġ2009 Ġand Ġ2019 . ĠThe ĠCo N LL Ġ2017 Ġcorpus Ġis Ġcomposed Ġof ĠEston ian ĠWikipedia Ġand Ġweb c rawl . </s> F inn ish Ġdataset Ġcontains Ġarticles Ġby ĠFinnish Ġnews Ġagency ĠST T , ĠFinnish Ġpart Ġof Ġthe ĠCo N LL Ġ2017 Ġdataset , Ġand ĠY l il aut a Ġdownloadable Ġversion ĠB IB REF 11 . ĠST T Ġnews Ġarticles Ġwere Ġpublished Ġbetween Ġyears Ġ1992 Ġand Ġ2018 . ĠY l il aut a Ġis Ġa ĠFinnish Ġonline Ġdiscussion Ġboard ; Ġthe Ġcorpus Ġcontains Ġparts Ġof Ġthe Ġdiscussions Ġfrom Ġ2012 Ġto Ġ2014 . </s> Lat v ian Ġdataset Ġconsists Ġonly Ġof Ġthe ĠLat v ian Ġportion Ġof Ġthe ĠCon LL Ġ2017 Ġcorpus . </s> L ith uan ian Ġdataset Ġis Ġcomposed Ġof ĠLithuan ian ĠWikipedia Ġarticles Ġfrom Ġ2018 , ĠD GT - UD Ġcorpus , Ġand ĠLt Ten Ten . ĠD GT - UD Ġis Ġa Ġparallel Ġcorpus Ġof Ġ23 Ġofficial Ġlanguages Ġof Ġthe ĠEU , Ġcomposed Ġof ĠJ RC ĠD GT Ġtranslation Ġmemory Ġof ĠEuropean Ġlaw , Ġautomatically Ġannot ated Ġwith ĠUD - P ipe Ġ1 . 2 . ĠLt Ten Ten Ġis ĠLithuan ian Ġweb Ġcorpus Ġmade Ġup Ġof Ġtexts Ġcollected Ġfrom Ġthe Ġinternet Ġin ĠApril Ġ2014 ĠB IB REF 12 . </s> S lo ven e Ġdataset Ġis Ġformed Ġfrom Ġthe ĠGig af ida Ġ2 . 0 Ġcorpus ĠB IB REF 13 . ĠIt Ġis Ġa Ġgeneral Ġlanguage Ġcorpus Ġcomposed Ġof Ġvarious Ġsources , Ġmostly Ġnewspapers , Ġinternet Ġpages , Ġand Ġmagazines , Ġbut Ġalso Ġfiction Ġand Ġnon - fiction Ġprose , Ġtextbooks , Ġetc . </s> Sw ed ish Ġdataset Ġis Ġcomposed Ġof ĠST T ĠSwedish Ġarticles Ġand ĠSwedish Ġpart Ġof ĠCo N LL Ġ2017 . ĠThe ĠFinnish Ġnews Ġagency ĠST T Ġpublishes Ġsome Ġof Ġits Ġarticles Ġin ĠSwedish Ġlanguage . ĠThey Ġwere Ġmade Ġavailable Ġto Ġus Ġthrough Ġpartnership Ġin Ġa Ġjoint Ġproject . ĠThe Ġcorpus Ġcontains Ġthose Ġarticles Ġfrom Ġ1992 Ġto Ġ2017 . </s> Pre processing Ġand ĠTraining </s> Prior Ġto Ġtraining Ġthe ĠEL Mo Ġmodels , Ġwe Ġsentence Ġand Ġword Ġtoken ized Ġall Ġthe Ġdatasets . ĠThe Ġtext Ġwas Ġformatted Ġin Ġsuch Ġa Ġway Ġthat Ġeach Ġsentence Ġwas Ġin Ġits Ġown Ġline Ġwith Ġtokens Ġseparated Ġby Ġwhite Ġspaces . ĠCo N LL Ġ2017 , ĠD GT - UD Ġand ĠLt Ten Ten 14 Ġcorpor a Ġwere Ġalready Ġpre - token ized . ĠWe Ġtoken ized Ġthe Ġothers Ġusing Ġthe ĠNL T K Ġlibrary Ġand Ġits Ġtoken izers Ġfor Ġeach Ġof Ġthe Ġlanguages . ĠThere Ġis Ġno Ġtoken izer Ġfor ĠCroatian Ġin ĠNL T K Ġlibrary , Ġso Ġwe Ġused ĠSloven e Ġtoken izer Ġinstead . ĠAfter Ġtoken ization , Ġwe Ġded u pl icated Ġthe Ġdatasets Ġfor Ġeach Ġlanguage Ġseparately , Ġusing Ġthe ĠOnion Ġ( ON e ĠInst ance ĠON ly ) Ġtool Ġfor Ġtext Ġded u pl ication . ĠWe Ġapplied Ġthe Ġtool Ġon Ġparagraph Ġlevel Ġfor Ġcorpor a Ġthat Ġdid Ġnot Ġhave Ġsentences Ġshuff led Ġand Ġon Ġsentence Ġlevel Ġfor Ġthe Ġrest . ĠWe Ġconsidered Ġ9 - gram s Ġwith Ġduplicate Ġcontent Ġthreshold Ġof Ġ0 . 9 . </s> For Ġeach Ġlanguage Ġwe Ġprepared Ġa Ġvocabulary Ġfile , Ġcontaining Ġroughly Ġone Ġmillion Ġmost Ġcommon Ġtokens , Ġi . e . Ġtokens Ġthat Ġappear Ġat Ġleast Ġ$ n $ Ġtimes Ġin Ġthe Ġcorpus , Ġwhere Ġ$ n $ Ġis Ġbetween Ġ15 Ġand Ġ25 , Ġdepending Ġon Ġthe Ġdataset Ġsize . ĠWe Ġincluded Ġthe Ġpunct uation Ġmarks Ġamong Ġthe Ġtokens . ĠWe Ġtrained Ġeach ĠEL Mo Ġmodel Ġusing Ġdefault Ġvalues Ġused Ġto Ġtrain Ġthe Ġoriginal ĠEnglish ĠEL Mo Ġ( large ) Ġmodel . </s> E val uation </s> We Ġevaluated Ġthe Ġproduced ĠEL Mo Ġmodels Ġfor Ġall Ġlanguages Ġusing Ġtwo Ġevaluation Ġtasks : Ġa Ġword Ġanalogy Ġtask Ġand Ġnamed Ġentity Ġrecognition Ġ( NER ) Ġtask . ĠBelow , Ġwe Ġfirst Ġshortly Ġdescribe Ġeach Ġtask , Ġfollowed Ġby Ġthe Ġevaluation Ġresults . </s> E val uation Ġ: :: ĠWord ĠAn alogy ĠTask </s> The Ġword Ġanalogy Ġtask Ġwas Ġpopular ized Ġby Ġm ik ol ov 2013 dist ributed . ĠThe Ġgoal Ġis Ġto Ġfind Ġa Ġterm Ġ$ y $ Ġfor Ġa Ġgiven Ġterm Ġ$ x $ Ġso Ġthat Ġthe Ġrelationship Ġbetween Ġ$ x $ Ġand Ġ$ y $ Ġbest Ġresembles Ġthe Ġgiven Ġrelationship Ġ$ a Ġ: Ġb $. ĠThere Ġare Ġtwo Ġmain Ġgroups Ġof Ġcategories : Ġ5 Ġsemantic Ġand Ġ10 Ġsynt actic . ĠTo Ġillustrate Ġa Ġsemantic Ġrelationship , Ġconsider Ġfor Ġexample Ġthat Ġthe Ġword Ġpair Ġ$ a Ġ: Ġb $ Ġis Ġgiven Ġas ĠâĢ ľ Fin land Ġ: ĠHelsinki âĢ Ŀ . ĠThe Ġtask Ġis Ġto Ġfind Ġthe Ġterm Ġ$ y $ Ġcorresponding Ġto Ġthe Ġrelationship ĠâĢ ľ Sw eden Ġ: Ġ$ y $ âĢ Ŀ , Ġwith Ġthe Ġexpected Ġanswer Ġbeing Ġ$ y =$ ĠStockholm . ĠIn Ġsynt actic Ġcategories , Ġthe Ġtwo Ġwords Ġin Ġa Ġpair Ġhave Ġa Ġcommon Ġstem Ġ( in Ġsome Ġcases Ġeven Ġsame Ġle mma ), Ġwith Ġall Ġthe Ġpairs Ġin Ġa Ġgiven Ġcategory Ġhaving Ġthe Ġsame Ġmorph ological Ġrelationship . ĠFor Ġexample , Ġgiven Ġthe Ġword Ġpair ĠâĢ ľ long Ġ: Ġlonger âĢ Ŀ , Ġwe Ġsee Ġthat Ġwe Ġhave Ġan Ġadjective Ġin Ġits Ġbase Ġform Ġand Ġthe Ġsame Ġadjective Ġin Ġa Ġcomparative Ġform . ĠThat Ġtask Ġis Ġthen Ġto Ġfind Ġthe Ġterm Ġ$ y $ Ġcorresponding Ġto Ġthe Ġrelationship ĠâĢ ľ dark Ġ: Ġ$ y $ âĢ Ŀ , Ġwith Ġthe Ġexpected Ġanswer Ġbeing Ġ$ y =$ Ġdarker , Ġthat Ġis Ġa Ġcomparative Ġform Ġof Ġthe Ġadjective Ġdark . </s> In Ġthe Ġvector Ġspace , Ġthe Ġanalogy Ġtask Ġis Ġtransformed Ġinto Ġvector Ġarithmetic Ġand Ġsearch Ġfor Ġnearest Ġneighbours , Ġi . e . Ġwe Ġcompute Ġthe Ġdistance Ġbetween Ġvectors : Ġd ( vec ( Fin land ), Ġvec ( H els inki )) Ġand Ġsearch Ġfor Ġword Ġ$ y $ Ġwhich Ġwould Ġgive Ġthe Ġclosest Ġresult Ġin Ġdistance Ġd ( vec ( Sw eden ), Ġvec ($ y $ )). ĠIn Ġthe Ġanalogy Ġdataset Ġthe Ġanalog ies Ġare Ġalready Ġpre - specified , Ġso Ġwe Ġare Ġmeasuring Ġhow Ġclose Ġare Ġthe Ġgiven Ġpairs . ĠIn Ġthe Ġevaluation Ġbelow , Ġwe Ġuse Ġanalogy Ġdatasets Ġfor Ġall Ġtested Ġlanguages Ġbased Ġon Ġthe ĠEnglish Ġdataset Ġby ĠB IB REF 14 Ġ. ĠDue Ġto ĠEnglish - centered Ġbias Ġof Ġthis Ġdataset , Ġwe Ġused Ġa Ġmodified Ġdataset Ġwhich Ġwas Ġfirst Ġwritten Ġin ĠSloven e Ġlanguage Ġand Ġthen Ġtranslated Ġinto Ġother Ġlanguages ĠB IB REF 15 . </s> As Ġeach Ġinstance Ġof Ġanalogy Ġcontains Ġonly Ġfour Ġwords , Ġwithout Ġany Ġcontext , Ġthe Ġcontextual Ġmodels Ġ( such Ġas ĠEL Mo ) Ġdo Ġnot Ġhave Ġenough Ġcontext Ġto Ġgenerate Ġsensible Ġembed d ings . ĠWe Ġtherefore Ġused Ġsome Ġadditional Ġtext Ġto Ġform Ġsimple Ġsentences Ġusing Ġthe Ġfour Ġanalogy Ġwords , Ġwhile Ġtaking Ġcare Ġthat Ġtheir Ġnoun Ġcase Ġstays Ġthe Ġsame . ĠFor Ġexample , Ġfor Ġthe Ġwords Ġ" R ome ", Ġ" Italy ", Ġ" Paris " Ġand Ġ" France " Ġ( forming Ġthe Ġanalogy ĠRome Ġis Ġto ĠItaly Ġas ĠParis Ġis Ġto Ġ$ x $, Ġwhere Ġthe Ġcorrect Ġanswer Ġis Ġ$ x =$ France ), Ġwe Ġformed Ġthe Ġsentence Ġ" If Ġthe Ġword ĠRome Ġcorresponds Ġto Ġthe Ġword ĠItaly , Ġthen Ġthe Ġword ĠParis Ġcorresponds Ġto Ġthe Ġword ĠFrance ". ĠWe Ġgenerated Ġembed d ings Ġfor Ġthose Ġfour Ġwords Ġin Ġthe Ġconstructed Ġsentence , Ġsubstituted Ġthe Ġlast Ġword Ġwith Ġeach Ġword Ġin Ġour Ġvocabulary Ġand Ġgenerated Ġthe Ġembed d ings Ġagain . ĠAs Ġtypical Ġfor Ġnon - context ual Ġanalogy Ġtask , Ġwe Ġmeasure Ġthe Ġcos ine Ġdistance Ġ($ d $ ) Ġbetween Ġthe Ġlast Ġword Ġ($ w _ 4 $ ) Ġand Ġthe Ġcombination Ġof Ġthe Ġfirst Ġthree Ġwords Ġ($ w _ 2 - w _ 1 + w _ 3 $ ). ĠWe Ġuse Ġthe ĠCS LS Ġmetric ĠB IB REF 16 Ġto Ġfind Ġthe Ġclosest Ġcandidate Ġword Ġ($ w _ 4 $ ). ĠIf Ġwe Ġfind Ġthe Ġcorrect Ġword Ġamong Ġthe Ġfive Ġclosest Ġwords , Ġwe Ġconsider Ġthat Ġentry Ġas Ġsuccessfully Ġidentified . ĠThe Ġproportion Ġof Ġcorrectly Ġidentified Ġwords Ġforms Ġa Ġstatistic Ġcalled Ġaccuracy @ 5 , Ġwhich Ġwe Ġreport Ġas Ġthe Ġresult . </s> We Ġfirst Ġcompare Ġexisting ĠLat v ian ĠEL Mo Ġembed d ings Ġfrom ĠEL Mo For Many L angs Ġproject Ġwith Ġour ĠLat v ian Ġembed d ings , Ġfollowed Ġby Ġthe Ġdetailed Ġanalysis Ġof Ġour ĠEL Mo Ġembed d ings . ĠWe Ġtrained ĠLat v ian ĠEL Mo Ġusing Ġonly ĠCo N LL Ġ2017 Ġcorpor a . ĠSince Ġthis Ġis Ġthe Ġonly Ġlanguage , Ġwhere Ġwe Ġtrained Ġthe Ġembed ding Ġmodel Ġon Ġexactly Ġthe Ġsame Ġcorpor a Ġas ĠEL Mo For Many L angs Ġmodels , Ġwe Ġchose Ġit Ġfor Ġcomparison Ġbetween Ġour ĠEL Mo Ġmodel Ġwith ĠEL Mo For Many L angs . ĠIn Ġother Ġlanguages , Ġadditional Ġor Ġother Ġcorpor a Ġwere Ġused , Ġso Ġa Ġdirect Ġcomparison Ġwould Ġalso Ġreflect Ġthe Ġquality Ġof Ġthe Ġcorpor a Ġused Ġfor Ġtraining . ĠIn ĠLat v ian , Ġhowever , Ġonly Ġthe Ġsize Ġof Ġthe Ġtraining Ġdataset Ġis Ġdifferent . ĠEL Mo For Many L angs Ġuses Ġonly Ġ20 Ġmillion Ġtokens Ġand Ġwe Ġuse Ġthe Ġwhole Ġcorpus Ġof Ġ270 Ġmillion Ġtokens . </s> The ĠLat v ian ĠEL Mo Ġmodel Ġfrom ĠEL Mo For Many L angs Ġproject Ġperforms Ġsignificantly Ġworse Ġthan ĠE MB ED D IA ĠEL Mo ĠLat v ian Ġmodel Ġon Ġall Ġcategories Ġof Ġword Ġanalogy Ġtask Ġ( Figure ĠFIG REF 16 ). ĠWe Ġalso Ġinclude Ġthe Ġcomparison Ġwith Ġour ĠEston ian ĠEL Mo Ġembed d ings Ġin Ġthe Ġsame Ġfigure . ĠThis Ġcomparison Ġshows Ġthat Ġwhile Ġdifferences Ġbetween Ġour ĠLat v ian Ġand ĠEston ian Ġembed d ings Ġcan Ġbe Ġsignificant Ġfor Ġcertain Ġcategories , Ġthe Ġaccuracy Ġscore Ġof ĠEL Mo For Many L angs Ġis Ġalways Ġworse Ġthan Ġeither Ġof Ġour Ġmodels . ĠThe Ġcomparison Ġof ĠEston ian Ġand ĠLat v ian Ġmodels Ġleads Ġus Ġto Ġbelieve Ġthat Ġa Ġfew Ġhundred Ġmillion Ġtokens Ġis Ġa Ġsufficiently Ġlarge Ġcorpus Ġto Ġtrain ĠEL Mo Ġmodels Ġ( at Ġleast Ġfor Ġword Ġanalogy Ġtask ), Ġbut Ġ20 - million Ġtoken Ġcorpor a Ġused Ġin ĠEL Mo For Many L angs Ġare Ġtoo Ġsmall . </s> The Ġresults Ġfor Ġall Ġlanguages Ġand Ġall ĠEL Mo Ġlayers , Ġaveraged Ġover Ġsemantic Ġand Ġsynt actic Ġcategories , Ġare Ġshown Ġin ĠTable ĠT AB REF 17 . ĠThe Ġembed d ings Ġafter Ġthe Ġfirst ĠL ST M Ġlayer Ġperform Ġbest Ġin Ġsemantic Ġcategories . ĠIn Ġsynt actic Ġcategories , Ġthe Ġnon - context ual ĠCNN Ġlayer Ġperforms Ġthe Ġbest . ĠSynt actic Ġcategories Ġare Ġless Ġcontext Ġdependent Ġand Ġmuch Ġmore Ġmorphology Ġand Ġsyntax Ġbased , Ġso Ġit Ġis Ġnot Ġsurprising Ġthat Ġthe Ġnon - context ual Ġlayer Ġperforms Ġwell . ĠThe Ġsecond ĠL ST M Ġlayer Ġembed d ings Ġperform Ġthe Ġworst Ġin Ġsynt actic Ġcategories , Ġthough Ġstill Ġoutper forming ĠCNN Ġlayer Ġembed d ings Ġin Ġsemantic Ġcategories . ĠLat v ian ĠEL Mo Ġperforms Ġworse Ġcompared Ġto Ġother Ġlanguages Ġwe Ġtrained , Ġespecially Ġin Ġsemantic Ġcategories , Ġpresumably Ġdue Ġto Ġsmaller Ġtraining Ġdata Ġsize . ĠSurprisingly , Ġthe Ġoriginal ĠEnglish ĠEL Mo Ġperforms Ġvery Ġpoorly Ġin Ġsynt actic Ġcategories Ġand Ġonly Ġoutper forms ĠLat v ian Ġin Ġsemantic Ġcategories . ĠThe Ġlow Ġscore Ġcan Ġbe Ġpartially Ġexplained Ġby ĠEnglish Ġmodel Ġscoring Ġ$ 0 . 00 $ Ġin Ġone Ġsynt actic Ġcategory ĠâĢ ľ opp osite Ġadjective âĢ Ŀ , Ġwhich Ġwe Ġhave Ġnot Ġbeen Ġable Ġto Ġexplain . </s> E val uation Ġ: :: ĠNamed ĠEntity ĠRecogn ition </s> For Ġevaluation Ġof ĠEL Mo Ġmodels Ġon Ġa Ġrelevant Ġdownstream Ġtask , Ġwe Ġused Ġnamed Ġentity Ġrecognition Ġ( NER ) Ġtask . ĠN ER Ġis Ġan Ġinformation Ġextraction Ġtask Ġthat Ġseeks Ġto Ġlocate Ġand Ġclassify Ġnamed Ġentity Ġmentions Ġin Ġun struct ured Ġtext Ġinto Ġpre - defined Ġcategories Ġsuch Ġas Ġthe Ġperson Ġnames , Ġorganizations , Ġlocations , Ġmedical Ġcodes , Ġtime Ġexpressions , Ġquantities , Ġmonetary Ġvalues , Ġpercentages , Ġetc . ĠTo Ġallow Ġcomparison Ġof Ġresults Ġbetween Ġlanguages , Ġwe Ġused Ġan Ġadapted Ġversion Ġof Ġthis Ġtask , Ġwhich Ġuses Ġa Ġreduced Ġset Ġof Ġlabels , Ġavailable Ġin ĠN ER Ġdatasets Ġfor Ġall Ġprocessed Ġlanguages . ĠThe Ġlabels Ġin Ġthe Ġused ĠN ER Ġdatasets Ġare Ġsimplified Ġto Ġa Ġcommon Ġlabel Ġset Ġof Ġthree Ġlabels Ġ( person Ġ- ĠPER , Ġlocation Ġ- ĠLOC , Ġorganization Ġ- ĠOR G ). ĠEach Ġword Ġin Ġthe ĠN ER Ġdataset Ġis Ġlabeled Ġwith Ġone Ġof Ġthe Ġthree Ġmentioned Ġlabels Ġor Ġa Ġlabel Ġ' O ' Ġ( other , Ġi . e . Ġnot Ġa Ġnamed Ġentity ) Ġif Ġit Ġdoes Ġnot Ġfit Ġany Ġof Ġthe Ġother Ġthree Ġlabels . ĠThe Ġnumber Ġof Ġwords Ġhaving Ġeach Ġlabel Ġis Ġshown Ġin ĠTable ĠT AB REF 19 . </s> To Ġmeasure Ġthe Ġperformance Ġof ĠEL Mo Ġembed d ings Ġon Ġthe ĠN ER Ġtask Ġwe Ġproceeded Ġas Ġfollows . ĠWe Ġembedded Ġthe Ġtext Ġin Ġthe Ġdatasets Ġsentence Ġby Ġsentence , Ġproducing Ġthree Ġvectors Ġ( one Ġfrom Ġeach ĠEL Mo Ġlayer ) Ġfor Ġeach Ġtoken Ġin Ġa Ġsentence . ĠWe Ġcalculated Ġthe Ġaverage Ġof Ġthe Ġthree Ġvectors Ġand Ġused Ġit Ġas Ġthe Ġinput Ġof Ġour Ġrecognition Ġmodel . ĠThe Ġinput Ġlayer Ġwas Ġfollowed Ġby Ġa Ġsingle ĠL ST M Ġlayer Ġwith Ġ128 ĠL ST M Ġcells Ġand Ġa Ġdrop out Ġlayer , Ġrandomly Ġdropping Ġ10 % Ġof Ġthe Ġneurons Ġon Ġboth Ġthe Ġoutput Ġand Ġthe Ġrecurrent Ġbranch . ĠThe Ġfinal Ġlayer Ġof Ġour Ġmodel Ġwas Ġa Ġtime Ġdistributed Ġsoft max Ġlayer Ġwith Ġ4 Ġneurons . </s> We Ġused ĠAD AM Ġoptim iser ĠB IB REF 17 Ġwith Ġthe Ġlearning Ġrate Ġ0 . 01 Ġand Ġ$ 10 ^{ - 5 } $ Ġlearning Ġrate Ġdecay . ĠWe Ġused Ġcateg orical Ġcross - ent ropy Ġas Ġa Ġloss Ġfunction Ġand Ġtrained Ġthe Ġmodel Ġfor Ġ3 Ġepoch s . ĠWe Ġpresent Ġthe Ġresults Ġusing Ġthe ĠMacro Ġ$ F _ 1 $ Ġscore , Ġthat Ġis Ġthe Ġaverage Ġof Ġ$ F _ 1 $ - sc ores Ġfor Ġeach Ġof Ġthe Ġthree ĠNE Ġclasses Ġ( the Ġclass ĠOther Ġis Ġexcluded ). </s> Since Ġthe Ġdifferences Ġbetween Ġthe Ġtested Ġlanguages Ġdepend Ġmore Ġon Ġthe Ġproperties Ġof Ġthe ĠN ER Ġdatasets Ġthan Ġon Ġthe Ġquality Ġof Ġembed d ings , Ġwe Ġcan Ġnot Ġdirectly Ġcompare ĠEL Mo Ġmodels . ĠFor Ġthis Ġreason , Ġwe Ġtake Ġthe Ġnon - context ual Ġfast Text Ġembed d ings Ġas Ġa Ġbaseline Ġand Ġpredict Ġnamed Ġentities Ġusing Ġthem . ĠThe Ġarchitecture Ġof Ġthe Ġmodel Ġusing Ġfast Text Ġembed d ings Ġis Ġthe Ġsame Ġas Ġthe Ġone Ġusing ĠEL Mo Ġembed d ings , Ġexcept Ġthat Ġthe Ġinput Ġuses Ġ300 Ġdimensional Ġfast Text Ġembed ding Ġvectors , Ġand Ġthe Ġmodel Ġwas Ġtrained Ġfor Ġ5 Ġepoch s Ġ( instead Ġof Ġ3 Ġas Ġfor ĠEL Mo ). ĠIn Ġboth Ġcases Ġ( EL Mo Ġand Ġfast Text ) Ġwe Ġtrained Ġand Ġevaluated Ġthe Ġmodel Ġfive Ġtimes , Ġbecause Ġthere Ġis Ġsome Ġrandom Ġcomponent Ġinvolved Ġin Ġinitialization Ġof Ġthe Ġneural Ġnetwork Ġmodel . ĠBy Ġtraining Ġand Ġevaluating Ġmultiple Ġtimes , Ġwe Ġminim ise Ġthis Ġrandom Ġcomponent . </s> The Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 21 . ĠWe Ġincluded Ġthe Ġevaluation Ġof Ġthe Ġoriginal ĠEL Mo ĠEnglish Ġmodel Ġin Ġthe Ġsame Ġtable . ĠN ER Ġmodels Ġhave Ġlittle Ġdifficulty Ġdistinguishing Ġbetween Ġtypes Ġof Ġnamed Ġentities , Ġbut Ġrecognizing Ġwhether Ġa Ġword Ġis Ġa Ġnamed Ġentity Ġor Ġnot Ġis Ġmore Ġdifficult . ĠFor Ġlanguages Ġwith Ġthe Ġsmallest ĠN ER Ġdatasets , ĠCroatian Ġand ĠLithuan ian , ĠEL Mo Ġembed d ings Ġshow Ġthe Ġlargest Ġimprovement Ġover Ġfast Text Ġembed d ings . ĠHowever , Ġwe Ġcan Ġobserve Ġsignificant Ġimprovements Ġwith ĠEL Mo Ġalso Ġon ĠEnglish Ġand ĠFinnish , Ġwhich Ġare Ġamong Ġthe Ġlargest Ġdatasets Ġ( English Ġbeing Ġby Ġfar Ġthe Ġlargest ). ĠOnly Ġon ĠSloven ian Ġdataset Ġdid ĠEL Mo Ġperform Ġslightly Ġworse Ġthan Ġfast Text , Ġon Ġall Ġother ĠE MB ED D IA Ġlanguages , Ġthe ĠEL Mo Ġembed d ings Ġimprove Ġthe Ġresults . </s> Conclusion </s> We Ġprepared Ġpre com puted ĠEL Mo Ġcontextual Ġembed d ings Ġfor Ġseven Ġlanguages : ĠCroatian , ĠEston ian , ĠFinnish , ĠLat v ian , ĠLithuan ian , ĠSloven ian , Ġand ĠSwedish . ĠWe Ġpresent Ġthe Ġnecessary Ġbackground Ġon Ġembed d ings Ġand Ġcontextual Ġembed d ings , Ġthe Ġdetails Ġof Ġtraining Ġthe Ġembed ding Ġmodels , Ġand Ġtheir Ġevaluation . ĠWe Ġshow Ġthat Ġthe Ġsize Ġof Ġused Ġtraining Ġsets Ġimportantly Ġaffects Ġthe Ġquality Ġof Ġproduced Ġembed d ings , Ġand Ġtherefore Ġthe Ġexisting Ġpublicly Ġavailable ĠEL Mo Ġembed d ings Ġfor Ġthe Ġprocessed Ġlanguages Ġare Ġinadequate . ĠWe Ġtrained Ġnew ĠEL Mo Ġembed d ings Ġon Ġlarger Ġtraining Ġsets Ġand Ġanalysed Ġtheir Ġproperties Ġon Ġthe Ġanalogy Ġtask Ġand Ġon Ġthe ĠN ER Ġtask . ĠThe Ġresults Ġshow Ġthat Ġthe Ġnewly Ġproduced Ġcontextual Ġembed d ings Ġproduce Ġsubstantially Ġbetter Ġresults Ġcompared Ġto Ġthe Ġnon - context ual Ġfast Text Ġbaseline . ĠIn Ġfuture Ġwork , Ġwe Ġplan Ġto Ġuse Ġthe Ġproduced Ġcontextual Ġembed d ings Ġon Ġthe Ġproblems Ġof Ġnews Ġmedia Ġindustry . ĠThe Ġpret rained ĠEL Mo Ġmodels Ġwill Ġbe Ġdeposited Ġto Ġthe ĠCL AR IN Ġrepository Ġby Ġthe Ġtime Ġof Ġthe Ġfinal Ġversion Ġof Ġthis Ġpaper . </s> Acknowled gments </s> The Ġwork Ġwas Ġpartially Ġsupported Ġby Ġthe ĠSloven ian ĠResearch ĠAgency Ġ( AR RS ) Ġcore Ġresearch Ġprogramme ĠP 6 - 04 11 . ĠThis Ġpaper Ġis Ġsupported Ġby ĠEuropean ĠUnion 's ĠHorizon Ġ2020 Ġresearch Ġand Ġinnovation Ġprogramme Ġunder Ġgrant Ġagreement ĠNo Ġ8 25 153 , Ġproject ĠE MB ED D IA Ġ( Cross - L ing ual ĠEmb edd ings Ġfor ĠLess - Represent ed ĠLanguages Ġin ĠEuropean ĠNews ĠMedia ). ĠThe Ġresults Ġof Ġthis Ġpublication Ġreflects Ġonly Ġthe Ġauthors ' Ġview Ġand Ġthe ĠEU ĠCommission Ġis Ġnot Ġresponsible Ġfor Ġany Ġuse Ġthat Ġmay Ġbe Ġmade Ġof Ġthe Ġinformation Ġit Ġcontains .