Russian
EXTRACTIVE
Russian
EXTRACTIVE
For the Russian language, with its rich morphology, lemmatizing the training and testing data for ELMo representations yields small but consistent improvements in the WSD task. This is unlike English, where the differences are negligible.
The RUSSE'18 dataset was created in 2018 for the shared task in Russian word sense induction. This dataset contains only nouns; the list of words with their English translations is given in Table TABREF30. To sum up, the RUSSE'18 dataset is morphologically far more complex than the Senseval3, reflecting the properties of the respective languages. In the next section we will see that this leads to substantial differences regarding comparisons between token-based and lemma-based ELMo models.
1909.03135
Introduction </s> Deep Ġcontextual ised Ġrepresentations Ġof Ġlinguistic Ġentities Ġ( words Ġand / or Ġsentences ) Ġare Ġused Ġin Ġmany Ġcurrent Ġstate - of - the - art ĠN LP Ġsystems . ĠThe Ġmost Ġwell - known Ġexamples Ġof Ġsuch Ġmodels Ġare Ġarguably ĠEL Mo ĠB IB REF 0 Ġand ĠB ERT ĠB IB REF 1 . </s> A Ġlong - standing Ġtradition Ġif Ġthe Ġfield Ġof Ġapplying Ġdeep Ġlearning Ġto ĠN LP Ġtasks Ġcan Ġbe Ġsummar ised Ġas Ġfollows : Ġas Ġminimal Ġpre - processing Ġas Ġpossible . ĠIt Ġis Ġwidely Ġbelieved Ġthat Ġle mm at ization Ġor Ġother Ġtext Ġinput Ġnormal isation Ġis Ġnot Ġnecessary . ĠAdvanced Ġneural Ġarchitectures Ġbased Ġon Ġcharacter Ġinput Ġ( CNN s , ĠB PE , Ġetc ) Ġare Ġsupposed Ġto Ġbe Ġable Ġto Ġlearn Ġhow Ġto Ġhandle Ġspelling Ġand Ġmorphology Ġvariations Ġthemselves , Ġeven Ġfor Ġlanguages Ġwith Ġrich Ġmorphology : Ġ` just Ġadd Ġmore Ġlayers ! '. ĠContext ual ised Ġembed ding Ġmodels Ġfollow Ġthis Ġtradition : Ġas Ġa Ġrule , Ġthey Ġare Ġtrained Ġon Ġraw Ġtext Ġcollections , Ġwith Ġminimal Ġlinguistic Ġpre - processing . ĠBelow , Ġwe Ġshow Ġthat Ġthis Ġis Ġnot Ġentirely Ġtrue . </s> It Ġis Ġknown Ġthat Ġfor Ġthe Ġprevious Ġgeneration Ġof Ġword Ġembed ding Ġmodels Ġ( ` static ' Ġones Ġlike Ġword 2 vec ĠB IB REF 2 , Ġwhere Ġa Ġword Ġalways Ġhas Ġthe Ġsame Ġrepresentation Ġregardless Ġof Ġthe Ġcontext Ġin Ġwhich Ġit Ġoccurs ), Ġle mm at ization Ġof Ġthe Ġtraining Ġand Ġtesting Ġdata Ġimproves Ġtheir Ġperformance . ĠB IB REF 3 Ġshowed Ġthat Ġthis Ġis Ġtrue Ġat Ġleast Ġfor Ġsemantic Ġsimilarity Ġand Ġanalogy Ġtasks . </s> In Ġthis Ġpaper , Ġwe Ġdescribe Ġour Ġexperiments Ġin Ġfinding Ġout Ġwhether Ġle mm at ization Ġhelps Ġmodern Ġcontextual ised Ġembed d ings Ġ( on Ġthe Ġexample Ġof ĠEL Mo ). ĠWe Ġcompare Ġthe Ġperformance Ġof ĠEL Mo Ġmodels Ġtrained Ġon Ġthe Ġsame Ġcorpus Ġbefore Ġand Ġafter Ġle mm at ization . ĠIt Ġis Ġimpossible Ġto Ġevaluate Ġcontextual ised Ġmodels Ġon Ġ` static ' Ġtasks Ġlike Ġlex ical Ġsemantic Ġsimilarity Ġor Ġword Ġanalog ies . ĠBecause Ġof Ġthis , Ġwe Ġturned Ġto Ġword Ġsense Ġdis amb ig uation Ġin Ġcontext Ġ( W SD ) Ġas Ġan Ġevaluation Ġtask . </s> In Ġbrief , Ġwe Ġuse Ġcontextual ised Ġrepresentations Ġof Ġambiguous Ġwords Ġfrom Ġthe Ġtop Ġlayer Ġof Ġan ĠEL Mo Ġmodel Ġto Ġtrain Ġword Ġsense Ġclass ifiers Ġand Ġfind Ġout Ġwhether Ġusing Ġle mm as Ġinstead Ġof Ġtokens Ġhelps Ġin Ġthis Ġtask Ġ( see ĠSection ĠSEC REF 5 ). ĠWe Ġexperiment Ġwith Ġthe ĠEnglish Ġand ĠRussian Ġlanguages Ġand Ġshow Ġthat Ġthey Ġdiffer Ġsignificantly Ġin Ġthe Ġinfluence Ġof Ġle mm at ization Ġon Ġthe ĠW SD Ġperformance Ġof ĠEL Mo Ġmodels . </s> Our Ġfindings Ġand Ġthe Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> L ingu istic Ġtext Ġpre - processing Ġstill Ġmatters Ġin Ġsome Ġtasks , Ġeven Ġfor Ġcontemporary Ġdeep Ġrepresentation Ġlearning Ġalgorithms . </s> For Ġthe ĠRussian Ġlanguage , Ġwith Ġits Ġrich Ġmorphology , Ġle mm at izing Ġthe Ġtraining Ġand Ġtesting Ġdata Ġfor ĠEL Mo Ġrepresentations Ġyields Ġsmall Ġbut Ġconsistent Ġimprovements Ġin Ġthe ĠW SD Ġtask . ĠThis Ġis Ġunlike ĠEnglish , Ġwhere Ġthe Ġdifferences Ġare Ġnegligible . </s> Related Ġwork </s> EL Mo Ġcontextual Ġword Ġrepresentations Ġare Ġlearned Ġin Ġan Ġun super vised Ġway Ġthrough Ġlanguage Ġmodelling ĠB IB REF 0 . ĠThe Ġgeneral Ġarchitecture Ġconsists Ġof Ġa Ġtwo - layer ĠBi L ST M Ġon Ġtop Ġof Ġa Ġconv olution al Ġlayer Ġwhich Ġtakes Ġcharacter Ġsequences Ġas Ġits Ġinput . ĠSince Ġthe Ġmodel Ġuses Ġfully Ġcharacter - based Ġtoken Ġrepresentations , Ġit Ġavoids Ġthe Ġproblem Ġof Ġout - of - voc abulary Ġwords . ĠBecause Ġof Ġthis , Ġthe Ġauthors Ġexplicitly Ġrecommend Ġnot Ġto Ġuse Ġany Ġnormal isation Ġexcept Ġtoken ization Ġfor Ġthe Ġinput Ġtext . ĠHowever , Ġas Ġwe Ġshow Ġbelow , Ġwhile Ġthis Ġis Ġtrue Ġfor ĠEnglish , Ġfor Ġother Ġlanguages Ġfeeding ĠEL Mo Ġwith Ġle mm as Ġinstead Ġof Ġraw Ġtokens Ġcan Ġimprove ĠW SD Ġperformance . </s> Word Ġsense Ġdis amb ig uation Ġor ĠW SD ĠB IB REF 4 Ġis Ġthe ĠN LP Ġtask Ġconsisting Ġof Ġchoosing Ġa Ġword Ġsense Ġfrom Ġa Ġpre - defined Ġsense Ġinventory , Ġgiven Ġthe Ġcontext Ġin Ġwhich Ġthe Ġword Ġis Ġused . ĠW SD Ġfits Ġwell Ġinto Ġour Ġaim Ġto Ġintrinsically Ġevaluate ĠEL Mo Ġmodels , Ġsince Ġsolving Ġthe Ġproblem Ġof Ġpoly se my Ġand Ġhom ony my Ġwas Ġone Ġof Ġthe Ġoriginal Ġpromises Ġof Ġcontextual ised Ġembed d ings : Ġtheir Ġprimary Ġdifference Ġfrom Ġthe Ġprevious Ġgeneration Ġof Ġword Ġembed ding Ġmodels Ġis Ġthat Ġcontextual ised Ġapproaches Ġgenerate Ġdifferent Ġrepresentations Ġfor Ġhom ographs Ġdepending Ġon Ġthe Ġcontext . ĠWe Ġuse Ġtwo Ġlex ical Ġsample ĠW SD Ġtest Ġsets , Ġfurther Ġdescribed Ġin ĠSection ĠSEC REF 4 . </s> Training ĠEL Mo </s> For Ġthe Ġexperiments Ġdescribed Ġbelow , Ġwe Ġtrained Ġour Ġown ĠEL Mo Ġmodels Ġfrom Ġscratch . ĠFor ĠEnglish , Ġthe Ġtraining Ġcorpus Ġconsisted Ġof Ġthe ĠEnglish ĠWikipedia Ġdump Ġfrom ĠFebruary Ġ2017 . ĠFor ĠRussian , Ġit Ġwas Ġa Ġconc aten ation Ġof Ġthe ĠRussian ĠWikipedia Ġdump Ġfrom ĠDecember Ġ2018 Ġand Ġthe Ġfull ĠRussian ĠNational ĠCorpus Ġ( R NC ). ĠThe ĠRNC Ġtexts Ġwere Ġadded Ġto Ġthe ĠRussian ĠWikipedia Ġdump Ġso Ġas Ġto Ġmake Ġthe ĠRussian Ġtraining Ġcorpus Ġmore Ġcomparable Ġin Ġsize Ġto Ġthe ĠEnglish Ġone Ġ( Wikipedia Ġtexts Ġwould Ġcomprise Ġonly Ġhalf Ġof Ġthe Ġsize ). ĠAs ĠTable ĠT AB REF 3 Ġshows , Ġthe ĠEnglish ĠWikipedia Ġis Ġstill Ġtwo Ġtimes Ġlarger , Ġbut Ġat Ġleast Ġthe Ġorder Ġis Ġthe Ġsame . </s> The Ġtexts Ġwere Ġtoken ized Ġand Ġle mm at ized Ġwith Ġthe ĠUDP ipe Ġmodels Ġfor Ġthe Ġrespective Ġlanguages Ġtrained Ġon Ġthe ĠUniversal ĠDepend encies Ġ2 . 3 Ġtree banks ĠB IB REF 5 . ĠUDP ipe Ġyields Ġle mm at ization Ġaccuracy Ġabout Ġ96 % Ġfor ĠEnglish Ġand Ġ97 % Ġfor ĠRussian ; Ġthus Ġfor Ġthe Ġtask Ġat Ġhand , Ġwe Ġconsidered Ġit Ġto Ġbe Ġgold Ġand Ġdid Ġnot Ġtry Ġto Ġfurther Ġimprove Ġthe Ġquality Ġof Ġnormal isation Ġitself Ġ( although Ġit Ġis Ġnot Ġentirely Ġerror - free , Ġsee ĠSection ĠSEC REF 4 ). </s> EL Mo Ġmodels Ġwere Ġtrained Ġon Ġthese Ġcorpor a Ġusing Ġthe Ġoriginal ĠT ensor Flow Ġimplementation , Ġfor Ġ3 Ġepoch s Ġwith Ġbatch Ġsize Ġ192 , Ġon Ġtwo ĠGPUs . ĠTo Ġtrain Ġfaster , Ġwe Ġdecreased Ġthe Ġdimension ality Ġof Ġthe ĠL ST M Ġlayers Ġfrom Ġthe Ġdefault Ġ4096 Ġto Ġ2048 Ġfor Ġall Ġthe Ġmodels . </s> Word Ġsense Ġdis amb ig uation Ġtest Ġsets </s> We Ġused Ġtwo ĠW SD Ġdatasets Ġfor Ġevaluation : </s> Sense val - 3 Ġfor ĠEnglish ĠB IB REF 6 </s> R US SE ' 18 Ġfor ĠRussian ĠB IB REF 7 </s> The ĠSense val - 3 Ġdataset Ġconsists Ġof Ġlex ical Ġsamples Ġfor Ġnoun s , Ġverbs Ġand Ġadject ives ; Ġwe Ġused Ġonly Ġnoun Ġtarget Ġwords : </s> argument </s> arm </s> at mosp here </s> aud ience </s> bank </s> degree </s> diff erence </s> diff iculty </s> disc </s> image </s> interest </s> jud gement </s> organ ization </s> paper </s> party </s> performance </s> plan </s> s hel ter </s> sort </s> source </s> An Ġexample Ġfor Ġthe Ġambiguous Ġword Ġargument Ġis Ġgiven Ġbelow : </s> In Ġsome Ġsituations ĠPost script Ġcan Ġbe Ġfaster Ġthan Ġthe Ġescape Ġsequence Ġtype Ġof Ġprinter Ġcontrol Ġfile . ĠIt Ġuses Ġpost Ġfix Ġnotation , Ġwhere Ġarguments Ġcome Ġfirst Ġand Ġoperators Ġfollow . ĠThis Ġis Ġbasically Ġthe Ġsame Ġas ĠReverse ĠPolish ĠNot ation Ġas Ġused Ġon Ġcertain Ġcalcul ators , Ġand Ġfollows Ġdirectly Ġfrom Ġthe Ġstack Ġbased Ġapproach . </s> It Ġthis Ġsentence , Ġthe Ġword Ġ` argument ' Ġis Ġused Ġin Ġthe Ġsense Ġof Ġa Ġmathematical Ġoperator . </s> The ĠR US SE ' 18 Ġdataset Ġwas Ġcreated Ġin Ġ2018 Ġfor Ġthe Ġshared Ġtask Ġin ĠRussian Ġword Ġsense Ġinduction . ĠThis Ġdataset Ġcontains Ġonly Ġnoun s ; Ġthe Ġlist Ġof Ġwords Ġwith Ġtheir ĠEnglish Ġtranslations Ġis Ġgiven Ġin ĠTable ĠT AB REF 30 . </s> Originally , Ġit Ġincludes Ġalso Ġthe Ġwords Ġr ussian Ð ± Ð° Ð ¹ Ðº Ð° Ġ` tale / f lee ce ' Ġand Ġr ussian Ð ³ Ð² Ð¾Ð · Ð´ Ð¸ Ðº Ð° Ġ' cl ove / small Ġnail ', Ġbut Ġtheir Ġsenses Ġare Ġambiguous Ġonly Ġin Ġsome Ġinf lection al Ġforms Ġ( not Ġin Ġle mm as ), Ġtherefore Ġwe Ġdecided Ġto Ġexclude Ġthese Ġwords Ġfrom Ġevaluation . </s> The ĠRussian Ġdataset Ġis Ġmore Ġhom ogeneous Ġcompared Ġto Ġthe ĠEnglish Ġone , Ġas Ġfor Ġall Ġthe Ġtarget Ġwords Ġthere Ġis Ġapproximately Ġthe Ġsame Ġnumber Ġof Ġcontext Ġwords Ġin Ġthe Ġexamples . ĠThis Ġis Ġachieved Ġby Ġapplying Ġthe Ġlex ical Ġwindow Ġ( 25 Ġwords Ġbefore Ġand Ġafter Ġthe Ġtarget Ġword ) Ġand Ġcro pping Ġeverything Ġthat Ġfalls Ġoutside Ġof Ġthat Ġwindow . ĠIn Ġthe ĠEnglish Ġdataset , Ġon Ġthe Ġcontrary , Ġthe Ġwhole Ġparagraph Ġwith Ġthe Ġtarget Ġword Ġis Ġtaken Ġinto Ġaccount . ĠWe Ġhave Ġtried Ġcro pping Ġthe Ġexamples Ġfor ĠEnglish Ġas Ġwell , Ġbut Ġit Ġdid Ġnot Ġresult Ġin Ġany Ġchange Ġin Ġthe Ġquality Ġof Ġclassification . ĠIn Ġthe Ġend , Ġwe Ġdecided Ġnot Ġto Ġapply Ġthe Ġlex ical Ġwindow Ġto Ġthe ĠEnglish Ġdataset Ġso Ġas Ġnot Ġto Ġalter Ġit Ġand Ġrather Ġuse Ġit Ġin Ġthe Ġoriginal Ġform . </s> Here Ġis Ġan Ġexample Ġfrom Ġthe ĠR US SE ' 18 Ġfor Ġthe Ġambiguous Ġword Ġr ussian Ð¼ Ð° Ð½ Ð´ Ð° ÑĢ Ð¸ Ð½ Ġ` mand arin ' Ġin Ġthe Ġsense Ġ` Chinese Ġofficial Ġtitle ': </s> r ussian âĢ ľ ... Ð´ Ð¸ Ð ¿ Ð» Ð¾Ð ¼ Ð° ÑĤ Ð¸ Ñ ĩ Ðµ Ñģ Ðº Ð¾Ð ³ Ð¾ ĠÐ º Ð¾ ÑĢ Ð ¿ Ñĥ Ñģ Ð° ĠÐ ¾ Ñģ ÑĤ Ð° Ð½ Ðº Ð° Ð¼ ĠÐ ± Ð¾Ð ³ Ð´ Ñĭ Ñ ħ Ð° Ð½ Ð° ĠÐ ¸ ĠÐ ¸ Ð¼ Ð ¿ Ðµ ÑĢ Ð° ÑĤ ÑĢ Ð¸ Ñ Ĩ Ñĭ ĠÐ ¾ Ð ± Ñģ ÑĤ Ð° Ð² Ð» Ðµ Ð½ Ð¾ ĠÐ ± Ñĭ Ð» Ð¾ Ġ Ñģ ĠÐ ½ Ðµ Ð¾Ð ± Ñĭ Ñ ĩ Ð° Ð ¹ Ð½ Ð¾Ð ¹ Ġ ÑĤ Ð¾ ÑĢ Ð ¶ Ðµ Ñģ ÑĤ Ð² Ðµ Ð½ Ð½ Ð¾ Ñģ ÑĤ ÑĮ Ñ İ . ĠÐ ¢ Ñĭ Ñģ Ñı Ñ ĩ Ð¸ ĠÐ ¼ Ð° Ð½ Ð´ Ð° ÑĢ Ð¸ Ð½ Ð¾Ð ² ĠÐ ¸ ĠÐ ´ ÑĢ Ñĥ Ð ³ Ð¸ Ñ ħ ĠÐ ² Ñĭ Ñģ Ð¾Ð º Ð¾Ð ¿ Ð¾ Ñģ ÑĤ Ð° Ð² Ð» Ðµ Ð½ Ð½ Ñĭ Ñ ħ ĠÐ » Ð¸ Ñ Ĩ Ġ ÑĢ Ð° Ð · Ð¼ Ðµ Ñģ ÑĤ Ð¸ Ð» Ð¸ Ñģ ÑĮ Ġ Ñ Ī Ð ¿ Ð° Ð» Ðµ ÑĢ Ð° Ð¼ Ð¸ ĠÐ ½ Ð° Ġ ÑĤ ÑĢ Ðµ Ñ ħ ĠÐ ¼ ÑĢ Ð° Ð¼ Ð¾ ÑĢ Ð½ Ñĭ Ñ ħ Ġ ÑĤ Ðµ ÑĢ ÑĢ Ð° Ñģ Ð° Ñ ħ ĠÐ ² Ðµ Ð´ Ñĥ Ñ ī Ð¸ Ñ ħ ĠÐ º ... âĢ Ŀ </s> ` ... the Ġdiplomatic Ġbodies Ġof Ġthe ĠBog d ik han Ġand Ġthe ĠEmpress Ġwas Ġfurnished Ġwith Ġextraordinary Ġsolemn ity . ĠThousands Ġof Ġmand ar ins Ġand Ġother Ġdign it aries Ġwere Ġplaced Ġon Ġthree Ġmarble Ġterr aces Ġleading Ġto ... '. </s> Table ĠT AB REF 31 Ġcompares Ġboth Ġdatasets . ĠBefore Ġusage , Ġthey Ġwere Ġpre - process ed Ġin Ġthe Ġsame Ġway Ġas Ġthe Ġtraining Ġcorpor a Ġfor ĠEL Mo Ġ( see ĠSection ĠSEC REF 3 ), Ġthus Ġproducing Ġa Ġle mm at ized Ġand Ġa Ġnon - le mm at ized Ġversions Ġof Ġeach . </s> As Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 31 , Ġfor Ġ20 Ġtarget Ġwords Ġin ĠEnglish Ġthere Ġare Ġ24 Ġle mm as , Ġand Ġfor Ġ18 Ġtarget Ġwords Ġin ĠRussian Ġthere Ġare Ġ36 Ġdifferent Ġle mm as . ĠThese Ġnumbers Ġare Ġexplained Ġby Ġoccasional Ġerrors Ġin Ġthe ĠUDP ipe Ġle mm at ization . ĠAnother Ġinteresting Ġthing Ġto Ġobserve Ġis Ġthe Ġnumber Ġof Ġdistinct Ġword Ġforms Ġfor Ġevery Ġlanguage . ĠFor ĠEnglish , Ġthere Ġare Ġ39 Ġdistinct Ġforms Ġfor Ġ20 Ġtarget Ġnoun s : Ġsingular Ġand Ġplural Ġfor Ġevery Ġnoun , Ġexcept Ġ` at mosp here ' Ġwhich Ġis Ġused Ġonly Ġin Ġthe Ġsingular Ġform . ĠThus , Ġinf lection al Ġvariability Ġof ĠEnglish Ġnoun s Ġis Ġcovered Ġby Ġthe Ġdataset Ġalmost Ġcompletely . ĠFor ĠRussian , Ġwe Ġobserve Ġ132 Ġdistinct Ġforms Ġfor Ġ18 Ġtarget Ġnoun s , Ġgiving Ġmore Ġthan Ġ7 Ġinf lection al Ġforms Ġper Ġeach Ġword . ĠNote Ġthat Ġthis Ġstill Ġcovers Ġonly Ġhalf Ġof Ġall Ġthe Ġinf lection al Ġvariability Ġof ĠRussian : Ġthis Ġlanguage Ġfeatures Ġ12 Ġdistinct Ġforms Ġfor Ġeach Ġnoun Ġ( 6 Ġcases Ġand Ġ2 Ġnumbers ). </s> To Ġsum Ġup , Ġthe ĠR US SE ' 18 Ġdataset Ġis Ġmorph ologically Ġfar Ġmore Ġcomplex Ġthan Ġthe ĠSense val 3 , Ġreflecting Ġthe Ġproperties Ġof Ġthe Ġrespective Ġlanguages . ĠIn Ġthe Ġnext Ġsection Ġwe Ġwill Ġsee Ġthat Ġthis Ġleads Ġto Ġsubstantial Ġdifferences Ġregarding Ġcomparisons Ġbetween Ġtoken - based Ġand Ġle mma - based ĠEL Mo Ġmodels . </s> Exper iments </s> Following ĠB IB REF 8 , Ġwe Ġdecided Ġto Ġavoid Ġusing Ġany Ġstandard Ġtrain - test Ġsplits Ġfor Ġour ĠW SD Ġdatasets . ĠInstead , Ġwe Ġrely Ġon Ġper - word Ġrandom Ġsplits Ġand Ġ5 - fold Ġcross - valid ation . ĠThis Ġmeans Ġthat Ġfor Ġeach Ġtarget Ġword Ġwe Ġrandomly Ġgenerate Ġ5 Ġdifferent Ġdivisions Ġof Ġits Ġcontext Ġsentences Ġlist Ġinto Ġtrain Ġand Ġtest Ġsets , Ġand Ġthen Ġtrain Ġand Ġtest Ġ5 Ġdifferent Ġclass ifier Ġmodels Ġon Ġthis Ġdata . ĠThe Ġresulting Ġperformance Ġscore Ġfor Ġeach Ġtarget Ġword Ġis Ġthe Ġaverage Ġof Ġ5 Ġmacro - F 1 Ġscores Ġproduced Ġby Ġthese Ġclass ifiers . </s> EL Mo Ġmodels Ġcan Ġbe Ġemployed Ġfor Ġthe ĠW SD Ġtask Ġin Ġtwo Ġdifferent Ġways : Ġeither Ġby Ġfine - tun ing Ġthe Ġmodel Ġor Ġby Ġextracting Ġword Ġrepresentations Ġfrom Ġit Ġand Ġthen Ġusing Ġthem Ġas Ġfeatures Ġin Ġa Ġdownstream Ġclass ifier . ĠWe Ġdecided Ġto Ġstick Ġto Ġthe Ġsecond Ġ( feature Ġextraction ) Ġapproach , Ġsince Ġit Ġis Ġconcept ually Ġand Ġcomput ationally Ġsimpler . ĠAdditionally , ĠB IB REF 9 Ġshowed Ġthat Ġfor Ġmost ĠN LP Ġtasks Ġ( except Ġthose Ġfocused Ġon Ġsentence Ġpairs ) Ġthe Ġperformance Ġof Ġfeature Ġextraction Ġand Ġfine - tun ing Ġis Ġnearly Ġthe Ġsame . ĠThus Ġwe Ġextracted Ġthe Ġsingle Ġvector Ġof Ġthe Ġtarget Ġword Ġfrom Ġthe ĠEL Mo Ġtop Ġlayer Ġ( ` target ' Ġrows Ġin ĠTable ĠT AB REF 32 ) Ġor Ġthe Ġaveraged ĠEL Mo Ġtop Ġlayer Ġvectors Ġof Ġall Ġwords Ġin Ġthe Ġcontext Ġsentence Ġ( ` aver aged ' Ġrows Ġin ĠTable ĠT AB REF 32 ). </s> For Ġcomparison , Ġwe Ġalso Ġreport Ġthe Ġscores Ġof Ġthe Ġ` aver aged Ġvectors ' Ġrepresentations Ġwith ĠContinuous ĠSkip gram ĠB IB REF 2 Ġembed ding Ġmodels Ġtrained Ġon Ġthe ĠEnglish Ġor ĠRussian ĠWikipedia Ġdumps Ġ( ` SG NS ' Ġrows ): Ġbefore Ġthe Ġadvent Ġof Ġcontextual ised Ġmodels , Ġthis Ġwas Ġone Ġof Ġthe Ġmost Ġwidely Ġused Ġways Ġto Ġ` squ ee ze ' Ġthe Ġmeaning Ġof Ġa Ġsentence Ġinto Ġa Ġfixed - size Ġvector . ĠOf Ġcourse Ġit Ġdoes Ġnot Ġmean Ġthat Ġthe Ġmeaning Ġof Ġa Ġsentence Ġalways Ġdetermines Ġthe Ġsenses Ġall Ġits Ġwords Ġare Ġused Ġin . ĠHowever , Ġaveraging Ġrepresentations Ġof Ġwords Ġin Ġcontexts Ġas Ġa Ġproxy Ġto Ġthe Ġsense Ġof Ġone Ġparticular Ġword Ġis Ġa Ġlong Ġestablished Ġtradition Ġin ĠW SD , Ġstarting Ġat Ġleast Ġfrom ĠB IB REF 10 . ĠAlso , Ġsince ĠSG NS Ġis Ġa Ġ` static ' Ġembed ding Ġmodel , Ġit Ġis Ġof Ġcourse Ġnot Ġpossible Ġto Ġuse Ġonly Ġtarget Ġword Ġvectors Ġas Ġfeatures : Ġthey Ġwould Ġbe Ġidentical Ġwhatever Ġthe Ġcontext Ġis . </s> Simple Ġlog istic Ġregression Ġwas Ġused Ġas Ġa Ġclassification Ġalgorithm . ĠWe Ġalso Ġtested Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġclass ifier Ġwith Ġ200 - ne ur ons Ġhidden Ġlayer , Ġwhich Ġyielded Ġessentially Ġthe Ġsame Ġresults . ĠThis Ġleads Ġus Ġto Ġbelieve Ġthat Ġour Ġfindings Ġare Ġnot Ġclass ifier - dependent . </s> Table ĠT AB REF 32 Ġshows Ġthe Ġresults , Ġtogether Ġwith Ġthe Ġrandom Ġand Ġmost Ġfrequent Ġsense Ġ( M FS ) Ġbas elines Ġfor Ġeach Ġdataset . </s> First , ĠEL Mo Ġoutper forms ĠSG NS Ġfor Ġboth Ġlanguages , Ġwhich Ġcomes Ġas Ġno Ġsurprise . ĠSecond , Ġthe Ġapproach Ġwith Ġaveraging Ġrepresentations Ġfrom Ġall Ġwords Ġin Ġthe Ġsentence Ġis Ġnot Ġbeneficial Ġfor ĠW SD Ġwith ĠEL Mo : Ġfor ĠEnglish Ġdata , Ġit Ġclearly Ġloses Ġto Ġa Ġsingle Ġtarget Ġword Ġrepresentation , Ġand Ġfor ĠRussian Ġthere Ġare Ġno Ġsignificant Ġdifferences Ġ( and Ġusing Ġa Ġsingle Ġtarget Ġword Ġis Ġpreferable Ġfrom Ġthe Ġcomputational Ġpoint Ġof Ġview , Ġsince Ġit Ġdoes Ġnot Ġrequire Ġthe Ġaveraging Ġoperation ). ĠThus , Ġbelow Ġwe Ġdiscuss Ġonly Ġthe Ġsingle Ġtarget Ġword Ġusage Ġmode Ġof ĠEL Mo . </s> But Ġthe Ġmost Ġimportant Ġpart Ġis Ġthe Ġcomparison Ġbetween Ġusing Ġtokens Ġor Ġle mm as Ġin Ġthe Ġtrain Ġand Ġtest Ġdata . ĠFor Ġthe Ġ` static ' ĠSG NS Ġembed d ings , Ġit Ġdoes Ġnot Ġsignificantly Ġchange Ġthe ĠW SD Ġscores Ġfor Ġboth Ġlanguages . ĠThe Ġsame Ġis Ġtrue Ġfor ĠEnglish ĠEL Mo Ġmodels , Ġwhere Ġdifferences Ġare Ġnegligible Ġand Ġseem Ġto Ġbe Ġsimple Ġfluctuations . ĠHowever , Ġfor ĠRussian , ĠEL Mo Ġ( target ) Ġon Ġle mm as Ġoutper forms ĠEL Mo Ġon Ġtokens , Ġwith Ġsmall Ġbut Ġsignificant Ġimprovement . ĠThe Ġmost Ġplausible Ġexplanation Ġfor Ġthis Ġis Ġthat Ġ( despite Ġof Ġpurely Ġcharacter - based Ġinput Ġof ĠEL Mo ) Ġthe Ġmodel Ġdoes Ġnot Ġhave Ġto Ġlearn Ġidiosyncr as ies Ġof Ġa Ġparticular Ġlanguage Ġmorphology . ĠInstead , Ġit Ġcan Ġuse Ġits Ġ( limited ) Ġcapacity Ġto Ġbetter Ġlearn Ġlex ical Ġsemantic Ġstructures , Ġleading Ġto Ġbetter ĠW SD Ġperformance . ĠThe Ġbox Ġplots ĠFIG REF 33 Ġand ĠFIG REF 35 Ġillustrate Ġthe Ġscores Ġdisp ersion Ġacross Ġwords Ġin Ġthe Ġtest Ġsets Ġfor ĠEnglish Ġand ĠRussian Ġcorrespond ingly Ġ( orange Ġlines Ġare Ġmed ians ). ĠIn Ġthe Ġnext Ġsection ĠSEC REF 6 Ġwe Ġanalyse Ġthe Ġresults Ġqual itatively . </s> Qual itative Ġanalysis </s> In Ġthis Ġsection Ġwe Ġfocus Ġon Ġthe Ġcomparison Ġof Ġscores Ġfor Ġthe ĠRussian Ġdataset . ĠThe Ġclass ifier Ġfor ĠRussian Ġhad Ġto Ġchoose Ġbetween Ġfewer Ġclasses Ġ( two Ġor Ġthree ), Ġwhich Ġmade Ġthe Ġscores Ġhigher Ġand Ġmore Ġconsistent Ġthan Ġfor Ġthe ĠEnglish Ġdataset . ĠOverall , Ġwe Ġsee Ġimprovements Ġin Ġthe Ġscores Ġfor Ġthe Ġmajority Ġof Ġwords , Ġwhich Ġproves Ġthat Ġle mm at ization Ġfor Ġmorph ologically Ġrich Ġlanguages Ġis Ġbeneficial . </s> We Ġdecided Ġto Ġanalyse Ġmore Ġclosely Ġthose Ġwords Ġfor Ġwhich Ġthe Ġdifference Ġin Ġthe Ġscores Ġbetween Ġle mma - based Ġand Ġtoken - based Ġmodels Ġwas Ġstatistically Ġsignificant . ĠBy Ġ` significant ' Ġwe Ġmean Ġthat Ġthe Ġscores Ġdiffer Ġby Ġmore Ġthat Ġone Ġstandard Ġdeviation Ġ( the Ġlargest Ġstandard Ġdeviation Ġvalue Ġin Ġthe Ġtwo Ġsets Ġwas Ġtaken ). ĠThe Ġresulting Ġlist Ġof Ġtargets Ġwords Ġwith Ġsignificant Ġdifference Ġin Ġscores Ġis Ġgiven Ġin ĠTable ĠT AB REF 36 . </s> We Ġcan Ġsee Ġthat Ġamong Ġ18 Ġwords Ġin Ġthe Ġdataset Ġonly Ġ3 Ġexhibit Ġsignificant Ġimprovement Ġin Ġtheir Ġscores Ġwhen Ġmoving Ġfrom Ġtokens Ġto Ġle mm as Ġin Ġthe Ġinput Ġdata . ĠIt Ġshows Ġthat Ġeven Ġthough Ġthe Ġoverall ĠF 1 Ġscores Ġfor Ġthe ĠRussian Ġdata Ġhave Ġshown Ġthe Ġplaus ibility Ġof Ġle mm at ization , Ġthis Ġimprovement Ġis Ġmostly Ġdriven Ġby Ġa Ġfew Ġwords . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġthese Ġwords ' Ġscores Ġfeature Ġvery Ġlow Ġstandard Ġdeviation Ġvalues Ġ( for Ġother Ġwords , Ġstandard Ġdeviation Ġvalues Ġwere Ġabove Ġ0 . 1 , Ġmaking ĠF 1 Ġdifferences Ġinsignificant ). ĠSuch Ġa Ġbehaviour Ġcan Ġbe Ġcaused Ġby Ġmore Ġconsistent Ġdifferentiation Ġof Ġcontext Ġfor Ġvarious Ġsenses Ġof Ġthese Ġ3 Ġwords . ĠFor Ġexample , Ġwith Ġthe Ġword Ġr ussian Ðº Ð° Ð ± Ð° Ñ ĩ Ð¾Ð º Ġ` squ ash Ġ/ Ġsmall Ġrestaurant ', Ġthe Ġcontexts Ġfor Ġboth Ġsenses Ġcan Ġbe Ġsimilar , Ġsince Ġthey Ġare Ġall Ġrelated Ġto Ġfood . ĠThis Ġmakes Ġthe ĠW SD Ġscores Ġunstable . ĠOn Ġthe Ġother Ġhand , Ġfor Ġr ussian Ð° Ðº Ñ Ĩ Ð¸ Ñı Ġ` stock , Ġshare Ġ/ Ġevent ', Ġr ussian Ðº ÑĢ Ð¾ Ð½ Ð° Ġ` c rown Ġ( tree Ġ/ Ġcoin )' Ġor Ġr ussian Ðº ÑĢ Ñĥ Ð ¿ Ġ` c roup Ġ( horse Ġbody Ġpart Ġ/ Ġillness ) ', Ġtheir Ġsenses Ġare Ġnot Ġrelated , Ġwhich Ġresulted Ġin Ġmore Ġstable Ġresults Ġand Ġsignificant Ġdifference Ġin Ġthe Ġscores Ġ( see ĠTable ĠT AB REF 36 ). </s> There Ġis Ġonly Ġone Ġword Ġin Ġthe ĠR US SE ' 18 Ġdataset Ġfor Ġwhich Ġthe Ġscore Ġhas Ġstrongly Ġdecreased Ġwhen Ġmoving Ġto Ġle mma - based Ġmodels : Ġr ussian Ð´ Ð¾Ð ¼ Ð¸ Ð½ Ð¾ Ġ` dom ino Ġ( game Ġ/ Ġcostume ) '. ĠIn Ġfact , Ġthe Ġscore Ġdifference Ġhere Ġlies Ġon Ġthe Ġborder Ġof Ġone Ġstandard Ġdeviation , Ġso Ġstrictly Ġspeaking Ġit Ġis Ġnot Ġreally Ġsignificant . ĠHowever , Ġthe Ġword Ġstill Ġpresents Ġan Ġinteresting Ġphenomenon . </s> r ussian Ð Ķ Ð¾Ð ¼ Ð¸ Ð½ Ð¾ Ġis Ġthe Ġonly Ġtarget Ġnoun Ġin Ġthe ĠR US SE ' 18 Ġthat Ġhas Ġno Ġinf lected Ġforms , Ġsince Ġit Ġis Ġa Ġborrowed Ġword . ĠThis Ġleaves Ġno Ġroom Ġfor Ġimprovement Ġwhen Ġusing Ġle mma - based ĠEL Mo Ġmodels : Ġall Ġtokens Ġof Ġthis Ġword Ġare Ġalready Ġidentical . ĠAt Ġthe Ġsame Ġtime , Ġsome Ġinformation Ġabout Ġinf lected Ġword Ġforms Ġin Ġthe Ġcontext Ġcan Ġbe Ġuseful , Ġbut Ġit Ġis Ġlost Ġduring Ġle mm at ization , Ġand Ġthis Ġleads Ġto Ġthe Ġdecreased Ġscore . ĠAr gu ably , Ġthis Ġmeans Ġthat Ġle mm at ization Ġbrings Ġalong Ġboth Ġadvantages Ġand Ġdisadvantages Ġfor ĠW SD Ġwith ĠEL Mo . ĠFor Ġinf lected Ġwords Ġ( which Ġconstitute Ġthe Ġmajority Ġof ĠRussian Ġvocabulary ) Ġprofits Ġoutweigh Ġthe Ġlosses , Ġbut Ġfor Ġat yp ical Ġnon - change able Ġwords Ġit Ġcan Ġbe Ġthe Ġopposite . </s> The Ġscores Ġfor Ġthe Ġexcluded Ġtarget Ġwords Ġr ussian Ð ± Ð° Ð ¹ Ðº Ð° Ġ` tale Ġ/ Ġflee ce ' Ġand Ġr ussian Ð ³ Ð² Ð¾Ð · Ð´ Ð¸ Ðº Ð° Ġ' cl ove Ġ/ Ġsmall Ġnail ' Ġare Ġgiven Ġin ĠTable ĠT AB REF 37 Ġ( rec all Ġthat Ġthey Ġwere Ġexcluded Ġbecause Ġof Ġbeing Ġambiguous Ġonly Ġin Ġsome Ġinf lection al Ġforms ). ĠFor Ġthese Ġwords Ġwe Ġcan Ġsee Ġa Ġgreat Ġimprovement Ġwith Ġle mma - based Ġmodels . ĠThis , Ġof Ġcourse Ġstems Ġfrom Ġthe Ġfact Ġthat Ġthese Ġwords Ġin Ġdifferent Ġsenses Ġhave Ġdifferent Ġle mm as . ĠTherefore , Ġthe Ġresults Ġare Ġheavily Ġdependent Ġon Ġthe Ġquality Ġof Ġle mm at ization . </s> Conclusion </s> We Ġevaluated Ġhow Ġthe Ġability Ġof ĠEL Mo Ġcontextual ised Ġword Ġembed ding Ġmodels Ġto Ġdis amb ig uate Ġword Ġsenses Ġdepends Ġon Ġthe Ġnature Ġof Ġthe Ġtraining Ġdata . ĠIn Ġparticular , Ġwe Ġcompared Ġthe Ġmodels Ġtrained Ġon Ġraw Ġtoken ized Ġcorpor a Ġand Ġthose Ġtrained Ġon Ġthe Ġcorpor a Ġwith Ġword Ġtokens Ġreplaced Ġby Ġtheir Ġnormal Ġforms Ġ( le mm as ). ĠThe Ġmodels Ġwe Ġtrained Ġare Ġpublicly Ġavailable Ġvia Ġthe ĠNL PL Ġword Ġembed d ings Ġrepository ĠB IB REF 3 . </s> In Ġthe Ġmajority Ġof Ġresearch Ġpapers Ġon Ġdeep Ġlearning Ġapproaches Ġto ĠN LP , Ġit Ġis Ġassumed Ġthat Ġle mm at ization Ġis Ġnot Ġnecessary , Ġespecially Ġwhen Ġusing Ġpowerful Ġcontextual ised Ġembed d ings . ĠOur Ġexperiments Ġshow Ġthat Ġthis Ġis Ġindeed Ġtrue Ġfor Ġlanguages Ġwith Ġsimple Ġmorphology Ġ( like ĠEnglish ). ĠHowever , Ġfor Ġrich - morph ology Ġlanguages Ġ( like ĠRussian ), Ġusing Ġle mm at ized Ġtraining Ġdata Ġyields Ġsmall Ġbut Ġconsistent Ġimprovements Ġin Ġthe Ġword Ġsense Ġdis amb ig uation Ġtask . ĠThese Ġimprovements Ġare Ġnot Ġobserved Ġfor Ġrare Ġwords Ġwhich Ġlack Ġinf lected Ġforms ; Ġthis Ġfurther Ġsupports Ġour Ġhypothesis Ġthat Ġbetter ĠW SD Ġscores Ġof Ġle mma - based Ġmodels Ġare Ġrelated Ġto Ġthem Ġbetter Ġhandling Ġmultiple Ġword Ġforms Ġin Ġmorphology - rich Ġlanguages . </s> Of Ġcourse , Ġle mm at ization Ġis Ġby Ġall Ġmeans Ġnot Ġa Ġsilver Ġbullet . ĠIn Ġother Ġtasks , Ġwhere Ġinf lection al Ġproperties Ġof Ġwords Ġare Ġimportant , Ġit Ġcan Ġeven Ġhurt Ġthe Ġperformance . ĠBut Ġthis Ġis Ġtrue Ġfor Ġany ĠN LP Ġsystems , Ġnot Ġonly Ġdeep Ġlearning Ġbased Ġones . </s> The Ġtake - home Ġmessage Ġhere Ġis Ġtwo fold : Ġfirst , Ġtext Ġpre - processing Ġstill Ġmatters Ġfor Ġcontemporary Ġdeep Ġlearning Ġalgorithms . ĠTheir Ġimpressive Ġlearning Ġabilities Ġdo Ġnot Ġalways Ġallow Ġthem Ġto Ġinfer Ġnormal isation Ġrules Ġthemselves , Ġfrom Ġsimply Ġoptim ising Ġthe Ġlanguage Ġmodelling Ġtask . ĠSecond , Ġthe Ġnature Ġof Ġlanguage Ġat Ġhand Ġmatters Ġas Ġwell , Ġand Ġdifferences Ġin Ġthis Ġnature Ġcan Ġresult Ġin Ġdifferent Ġdecisions Ġbeing Ġoptimal Ġor Ġsub - opt imal Ġat Ġthe Ġstage Ġof Ġdeep Ġlearning Ġmodels Ġtraining . ĠThe Ġsimple Ġtruth Ġ` English Ġis Ġnot Ġrepresentative Ġof Ġall Ġlanguages Ġon ĠEarth ' Ġstill Ġholds Ġhere . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġextend Ġour Ġwork Ġby Ġincluding Ġmore Ġlanguages Ġinto Ġthe Ġanalysis . ĠUsing ĠRussian Ġand ĠEnglish Ġallowed Ġus Ġto Ġhypothes ise Ġabout Ġthe Ġimportance Ġof Ġmorph ological Ġcharacter Ġof Ġa Ġlanguage . ĠBut Ġwe Ġonly Ġscratched Ġthe Ġsurface Ġof Ġthe Ġlinguistic Ġdiversity . ĠTo Ġverify Ġthis Ġclaim , Ġit Ġis Ġnecessary Ġto Ġanalyse Ġmore Ġstrongly Ġinf lected Ġlanguages Ġlike ĠRussian Ġas Ġwell Ġas Ġmore Ġweak ly Ġinf lected Ġ( analy tical ) Ġlanguages Ġsimilar Ġto ĠEnglish . ĠThis Ġwill Ġhelp Ġto Ġfind Ġout Ġif Ġthe Ġinf lection Ġdifferences Ġare Ġimportant Ġfor Ġtraining Ġdeep Ġlearning Ġmodels Ġacross Ġhuman Ġlanguages Ġin Ġgeneral .
Introduction Deep contextualised representations of linguistic entities (words and/or sentences) are used in many current state-of-the-art NLP systems. The most well-known examples of such models are arguably ELMo BIBREF0 and BERT BIBREF1. A long-standing tradition if the field of applying deep learning to NLP tasks can be summarised as follows: as minimal pre-processing as possible. It is widely believed that lemmatization or other text input normalisation is not necessary. Advanced neural architectures based on character input (CNNs, BPE, etc) are supposed to be able to learn how to handle spelling and morphology variations themselves, even for languages with rich morphology: `just add more layers!'. Contextualised embedding models follow this tradition: as a rule, they are trained on raw text collections, with minimal linguistic pre-processing. Below, we show that this is not entirely true. It is known that for the previous generation of word embedding models (`static' ones like word2vec BIBREF2, where a word always has the same representation regardless of the context in which it occurs), lemmatization of the training and testing data improves their performance. BIBREF3 showed that this is true at least for semantic similarity and analogy tasks. In this paper, we describe our experiments in finding out whether lemmatization helps modern contextualised embeddings (on the example of ELMo). We compare the performance of ELMo models trained on the same corpus before and after lemmatization. It is impossible to evaluate contextualised models on `static' tasks like lexical semantic similarity or word analogies. Because of this, we turned to word sense disambiguation in context (WSD) as an evaluation task. In brief, we use contextualised representations of ambiguous words from the top layer of an ELMo model to train word sense classifiers and find out whether using lemmas instead of tokens helps in this task (see Section SECREF5). We experiment with the English and Russian languages and show that they differ significantly in the influence of lemmatization on the WSD performance of ELMo models. Our findings and the contributions of this paper are: Linguistic text pre-processing still matters in some tasks, even for contemporary deep representation learning algorithms. For the Russian language, with its rich morphology, lemmatizing the training and testing data for ELMo representations yields small but consistent improvements in the WSD task. This is unlike English, where the differences are negligible. Related work ELMo contextual word representations are learned in an unsupervised way through language modelling BIBREF0. The general architecture consists of a two-layer BiLSTM on top of a convolutional layer which takes character sequences as its input. Since the model uses fully character-based token representations, it avoids the problem of out-of-vocabulary words. Because of this, the authors explicitly recommend not to use any normalisation except tokenization for the input text. However, as we show below, while this is true for English, for other languages feeding ELMo with lemmas instead of raw tokens can improve WSD performance. Word sense disambiguation or WSD BIBREF4 is the NLP task consisting of choosing a word sense from a pre-defined sense inventory, given the context in which the word is used. WSD fits well into our aim to intrinsically evaluate ELMo models, since solving the problem of polysemy and homonymy was one of the original promises of contextualised embeddings: their primary difference from the previous generation of word embedding models is that contextualised approaches generate different representations for homographs depending on the context. We use two lexical sample WSD test sets, further described in Section SECREF4. Training ELMo For the experiments described below, we trained our own ELMo models from scratch. For English, the training corpus consisted of the English Wikipedia dump from February 2017. For Russian, it was a concatenation of the Russian Wikipedia dump from December 2018 and the full Russian National Corpus (RNC). The RNC texts were added to the Russian Wikipedia dump so as to make the Russian training corpus more comparable in size to the English one (Wikipedia texts would comprise only half of the size). As Table TABREF3 shows, the English Wikipedia is still two times larger, but at least the order is the same. The texts were tokenized and lemmatized with the UDPipe models for the respective languages trained on the Universal Dependencies 2.3 treebanks BIBREF5. UDPipe yields lemmatization accuracy about 96% for English and 97% for Russian; thus for the task at hand, we considered it to be gold and did not try to further improve the quality of normalisation itself (although it is not entirely error-free, see Section SECREF4). ELMo models were trained on these corpora using the original TensorFlow implementation, for 3 epochs with batch size 192, on two GPUs. To train faster, we decreased the dimensionality of the LSTM layers from the default 4096 to 2048 for all the models. Word sense disambiguation test sets We used two WSD datasets for evaluation: Senseval-3 for English BIBREF6 RUSSE'18 for Russian BIBREF7 The Senseval-3 dataset consists of lexical samples for nouns, verbs and adjectives; we used only noun target words: argument arm atmosphere audience bank degree difference difficulty disc image interest judgement organization paper party performance plan shelter sort source An example for the ambiguous word argument is given below: In some situations Postscript can be faster than the escape sequence type of printer control file. It uses post fix notation, where arguments come first and operators follow. This is basically the same as Reverse Polish Notation as used on certain calculators, and follows directly from the stack based approach. It this sentence, the word `argument' is used in the sense of a mathematical operator. The RUSSE'18 dataset was created in 2018 for the shared task in Russian word sense induction. This dataset contains only nouns; the list of words with their English translations is given in Table TABREF30. Originally, it includes also the words russianбайка `tale/fleece' and russianгвоздика 'clove/small nail', but their senses are ambiguous only in some inflectional forms (not in lemmas), therefore we decided to exclude these words from evaluation. The Russian dataset is more homogeneous compared to the English one, as for all the target words there is approximately the same number of context words in the examples. This is achieved by applying the lexical window (25 words before and after the target word) and cropping everything that falls outside of that window. In the English dataset, on the contrary, the whole paragraph with the target word is taken into account. We have tried cropping the examples for English as well, but it did not result in any change in the quality of classification. In the end, we decided not to apply the lexical window to the English dataset so as not to alter it and rather use it in the original form. Here is an example from the RUSSE'18 for the ambiguous word russianмандарин `mandarin' in the sense `Chinese official title': russian“...дипломатического корпуса останкам богдыхана и императрицы обставлено было с необычайной торжественностью. Тысячи мандаринов и других высокопоставленных лиц разместились шпалерами на трех мраморных террасах ведущих к...” `...the diplomatic bodies of the Bogdikhan and the Empress was furnished with extraordinary solemnity. Thousands of mandarins and other dignitaries were placed on three marble terraces leading to...'. Table TABREF31 compares both datasets. Before usage, they were pre-processed in the same way as the training corpora for ELMo (see Section SECREF3), thus producing a lemmatized and a non-lemmatized versions of each. As we can see from Table TABREF31, for 20 target words in English there are 24 lemmas, and for 18 target words in Russian there are 36 different lemmas. These numbers are explained by occasional errors in the UDPipe lemmatization. Another interesting thing to observe is the number of distinct word forms for every language. For English, there are 39 distinct forms for 20 target nouns: singular and plural for every noun, except `atmosphere' which is used only in the singular form. Thus, inflectional variability of English nouns is covered by the dataset almost completely. For Russian, we observe 132 distinct forms for 18 target nouns, giving more than 7 inflectional forms per each word. Note that this still covers only half of all the inflectional variability of Russian: this language features 12 distinct forms for each noun (6 cases and 2 numbers). To sum up, the RUSSE'18 dataset is morphologically far more complex than the Senseval3, reflecting the properties of the respective languages. In the next section we will see that this leads to substantial differences regarding comparisons between token-based and lemma-based ELMo models. Experiments Following BIBREF8, we decided to avoid using any standard train-test splits for our WSD datasets. Instead, we rely on per-word random splits and 5-fold cross-validation. This means that for each target word we randomly generate 5 different divisions of its context sentences list into train and test sets, and then train and test 5 different classifier models on this data. The resulting performance score for each target word is the average of 5 macro-F1 scores produced by these classifiers. ELMo models can be employed for the WSD task in two different ways: either by fine-tuning the model or by extracting word representations from it and then using them as features in a downstream classifier. We decided to stick to the second (feature extraction) approach, since it is conceptually and computationally simpler. Additionally, BIBREF9 showed that for most NLP tasks (except those focused on sentence pairs) the performance of feature extraction and fine-tuning is nearly the same. Thus we extracted the single vector of the target word from the ELMo top layer (`target' rows in Table TABREF32) or the averaged ELMo top layer vectors of all words in the context sentence (`averaged' rows in Table TABREF32). For comparison, we also report the scores of the `averaged vectors' representations with Continuous Skipgram BIBREF2 embedding models trained on the English or Russian Wikipedia dumps (`SGNS' rows): before the advent of contextualised models, this was one of the most widely used ways to `squeeze' the meaning of a sentence into a fixed-size vector. Of course it does not mean that the meaning of a sentence always determines the senses all its words are used in. However, averaging representations of words in contexts as a proxy to the sense of one particular word is a long established tradition in WSD, starting at least from BIBREF10. Also, since SGNS is a `static' embedding model, it is of course not possible to use only target word vectors as features: they would be identical whatever the context is. Simple logistic regression was used as a classification algorithm. We also tested a multi-layer perceptron (MLP) classifier with 200-neurons hidden layer, which yielded essentially the same results. This leads us to believe that our findings are not classifier-dependent. Table TABREF32 shows the results, together with the random and most frequent sense (MFS) baselines for each dataset. First, ELMo outperforms SGNS for both languages, which comes as no surprise. Second, the approach with averaging representations from all words in the sentence is not beneficial for WSD with ELMo: for English data, it clearly loses to a single target word representation, and for Russian there are no significant differences (and using a single target word is preferable from the computational point of view, since it does not require the averaging operation). Thus, below we discuss only the single target word usage mode of ELMo. But the most important part is the comparison between using tokens or lemmas in the train and test data. For the `static' SGNS embeddings, it does not significantly change the WSD scores for both languages. The same is true for English ELMo models, where differences are negligible and seem to be simple fluctuations. However, for Russian, ELMo (target) on lemmas outperforms ELMo on tokens, with small but significant improvement. The most plausible explanation for this is that (despite of purely character-based input of ELMo) the model does not have to learn idiosyncrasies of a particular language morphology. Instead, it can use its (limited) capacity to better learn lexical semantic structures, leading to better WSD performance. The box plots FIGREF33 and FIGREF35 illustrate the scores dispersion across words in the test sets for English and Russian correspondingly (orange lines are medians). In the next section SECREF6 we analyse the results qualitatively. Qualitative analysis In this section we focus on the comparison of scores for the Russian dataset. The classifier for Russian had to choose between fewer classes (two or three), which made the scores higher and more consistent than for the English dataset. Overall, we see improvements in the scores for the majority of words, which proves that lemmatization for morphologically rich languages is beneficial. We decided to analyse more closely those words for which the difference in the scores between lemma-based and token-based models was statistically significant. By `significant' we mean that the scores differ by more that one standard deviation (the largest standard deviation value in the two sets was taken). The resulting list of targets words with significant difference in scores is given in Table TABREF36. We can see that among 18 words in the dataset only 3 exhibit significant improvement in their scores when moving from tokens to lemmas in the input data. It shows that even though the overall F1 scores for the Russian data have shown the plausibility of lemmatization, this improvement is mostly driven by a few words. It should be noted that these words' scores feature very low standard deviation values (for other words, standard deviation values were above 0.1, making F1 differences insignificant). Such a behaviour can be caused by more consistent differentiation of context for various senses of these 3 words. For example, with the word russianкабачок `squash / small restaurant', the contexts for both senses can be similar, since they are all related to food. This makes the WSD scores unstable. On the other hand, for russianакция `stock, share / event', russianкрона `crown (tree / coin)' or russianкруп `croup (horse body part / illness)', their senses are not related, which resulted in more stable results and significant difference in the scores (see Table TABREF36). There is only one word in the RUSSE'18 dataset for which the score has strongly decreased when moving to lemma-based models: russianдомино `domino (game / costume)'. In fact, the score difference here lies on the border of one standard deviation, so strictly speaking it is not really significant. However, the word still presents an interesting phenomenon. russianДомино is the only target noun in the RUSSE'18 that has no inflected forms, since it is a borrowed word. This leaves no room for improvement when using lemma-based ELMo models: all tokens of this word are already identical. At the same time, some information about inflected word forms in the context can be useful, but it is lost during lemmatization, and this leads to the decreased score. Arguably, this means that lemmatization brings along both advantages and disadvantages for WSD with ELMo. For inflected words (which constitute the majority of Russian vocabulary) profits outweigh the losses, but for atypical non-changeable words it can be the opposite. The scores for the excluded target words russianбайка `tale / fleece' and russianгвоздика 'clove / small nail' are given in Table TABREF37 (recall that they were excluded because of being ambiguous only in some inflectional forms). For these words we can see a great improvement with lemma-based models. This, of course stems from the fact that these words in different senses have different lemmas. Therefore, the results are heavily dependent on the quality of lemmatization. Conclusion We evaluated how the ability of ELMo contextualised word embedding models to disambiguate word senses depends on the nature of the training data. In particular, we compared the models trained on raw tokenized corpora and those trained on the corpora with word tokens replaced by their normal forms (lemmas). The models we trained are publicly available via the NLPL word embeddings repository BIBREF3. In the majority of research papers on deep learning approaches to NLP, it is assumed that lemmatization is not necessary, especially when using powerful contextualised embeddings. Our experiments show that this is indeed true for languages with simple morphology (like English). However, for rich-morphology languages (like Russian), using lemmatized training data yields small but consistent improvements in the word sense disambiguation task. These improvements are not observed for rare words which lack inflected forms; this further supports our hypothesis that better WSD scores of lemma-based models are related to them better handling multiple word forms in morphology-rich languages. Of course, lemmatization is by all means not a silver bullet. In other tasks, where inflectional properties of words are important, it can even hurt the performance. But this is true for any NLP systems, not only deep learning based ones. The take-home message here is twofold: first, text pre-processing still matters for contemporary deep learning algorithms. Their impressive learning abilities do not always allow them to infer normalisation rules themselves, from simply optimising the language modelling task. Second, the nature of language at hand matters as well, and differences in this nature can result in different decisions being optimal or sub-optimal at the stage of deep learning models training. The simple truth `English is not representative of all languages on Earth' still holds here. In the future, we plan to extend our work by including more languages into the analysis. Using Russian and English allowed us to hypothesise about the importance of morphological character of a language. But we only scratched the surface of the linguistic diversity. To verify this claim, it is necessary to analyse more strongly inflected languages like Russian as well as more weakly inflected (analytical) languages similar to English. This will help to find out if the inflection differences are important for training deep learning models across human languages in general.
What other examples of morphologically-rich languages do the authors give?
22815878083ebd2f9e08bc33a5e733063dac7a0f
What Ġother Ġexamples Ġof Ġmorph ologically - rich Ġlanguages Ġdo Ġthe Ġauthors Ġgive ?
Russian
<s> What Ġother Ġexamples Ġof Ġmorph ologically - rich Ġlanguages Ġdo Ġthe Ġauthors Ġgive ? </s> Introduction </s> Deep Ġcontextual ised Ġrepresentations Ġof Ġlinguistic Ġentities Ġ( words Ġand / or Ġsentences ) Ġare Ġused Ġin Ġmany Ġcurrent Ġstate - of - the - art ĠN LP Ġsystems . ĠThe Ġmost Ġwell - known Ġexamples Ġof Ġsuch Ġmodels Ġare Ġarguably ĠEL Mo ĠB IB REF 0 Ġand ĠB ERT ĠB IB REF 1 . </s> A Ġlong - standing Ġtradition Ġif Ġthe Ġfield Ġof Ġapplying Ġdeep Ġlearning Ġto ĠN LP Ġtasks Ġcan Ġbe Ġsummar ised Ġas Ġfollows : Ġas Ġminimal Ġpre - processing Ġas Ġpossible . ĠIt Ġis Ġwidely Ġbelieved Ġthat Ġle mm at ization Ġor Ġother Ġtext Ġinput Ġnormal isation Ġis Ġnot Ġnecessary . ĠAdvanced Ġneural Ġarchitectures Ġbased Ġon Ġcharacter Ġinput Ġ( CNN s , ĠB PE , Ġetc ) Ġare Ġsupposed Ġto Ġbe Ġable Ġto Ġlearn Ġhow Ġto Ġhandle Ġspelling Ġand Ġmorphology Ġvariations Ġthemselves , Ġeven Ġfor Ġlanguages Ġwith Ġrich Ġmorphology : Ġ` just Ġadd Ġmore Ġlayers ! '. ĠContext ual ised Ġembed ding Ġmodels Ġfollow Ġthis Ġtradition : Ġas Ġa Ġrule , Ġthey Ġare Ġtrained Ġon Ġraw Ġtext Ġcollections , Ġwith Ġminimal Ġlinguistic Ġpre - processing . ĠBelow , Ġwe Ġshow Ġthat Ġthis Ġis Ġnot Ġentirely Ġtrue . </s> It Ġis Ġknown Ġthat Ġfor Ġthe Ġprevious Ġgeneration Ġof Ġword Ġembed ding Ġmodels Ġ( ` static ' Ġones Ġlike Ġword 2 vec ĠB IB REF 2 , Ġwhere Ġa Ġword Ġalways Ġhas Ġthe Ġsame Ġrepresentation Ġregardless Ġof Ġthe Ġcontext Ġin Ġwhich Ġit Ġoccurs ), Ġle mm at ization Ġof Ġthe Ġtraining Ġand Ġtesting Ġdata Ġimproves Ġtheir Ġperformance . ĠB IB REF 3 Ġshowed Ġthat Ġthis Ġis Ġtrue Ġat Ġleast Ġfor Ġsemantic Ġsimilarity Ġand Ġanalogy Ġtasks . </s> In Ġthis Ġpaper , Ġwe Ġdescribe Ġour Ġexperiments Ġin Ġfinding Ġout Ġwhether Ġle mm at ization Ġhelps Ġmodern Ġcontextual ised Ġembed d ings Ġ( on Ġthe Ġexample Ġof ĠEL Mo ). ĠWe Ġcompare Ġthe Ġperformance Ġof ĠEL Mo Ġmodels Ġtrained Ġon Ġthe Ġsame Ġcorpus Ġbefore Ġand Ġafter Ġle mm at ization . ĠIt Ġis Ġimpossible Ġto Ġevaluate Ġcontextual ised Ġmodels Ġon Ġ` static ' Ġtasks Ġlike Ġlex ical Ġsemantic Ġsimilarity Ġor Ġword Ġanalog ies . ĠBecause Ġof Ġthis , Ġwe Ġturned Ġto Ġword Ġsense Ġdis amb ig uation Ġin Ġcontext Ġ( W SD ) Ġas Ġan Ġevaluation Ġtask . </s> In Ġbrief , Ġwe Ġuse Ġcontextual ised Ġrepresentations Ġof Ġambiguous Ġwords Ġfrom Ġthe Ġtop Ġlayer Ġof Ġan ĠEL Mo Ġmodel Ġto Ġtrain Ġword Ġsense Ġclass ifiers Ġand Ġfind Ġout Ġwhether Ġusing Ġle mm as Ġinstead Ġof Ġtokens Ġhelps Ġin Ġthis Ġtask Ġ( see ĠSection ĠSEC REF 5 ). ĠWe Ġexperiment Ġwith Ġthe ĠEnglish Ġand ĠRussian Ġlanguages Ġand Ġshow Ġthat Ġthey Ġdiffer Ġsignificantly Ġin Ġthe Ġinfluence Ġof Ġle mm at ization Ġon Ġthe ĠW SD Ġperformance Ġof ĠEL Mo Ġmodels . </s> Our Ġfindings Ġand Ġthe Ġcontributions Ġof Ġthis Ġpaper Ġare : </s> L ingu istic Ġtext Ġpre - processing Ġstill Ġmatters Ġin Ġsome Ġtasks , Ġeven Ġfor Ġcontemporary Ġdeep Ġrepresentation Ġlearning Ġalgorithms . </s> For Ġthe ĠRussian Ġlanguage , Ġwith Ġits Ġrich Ġmorphology , Ġle mm at izing Ġthe Ġtraining Ġand Ġtesting Ġdata Ġfor ĠEL Mo Ġrepresentations Ġyields Ġsmall Ġbut Ġconsistent Ġimprovements Ġin Ġthe ĠW SD Ġtask . ĠThis Ġis Ġunlike ĠEnglish , Ġwhere Ġthe Ġdifferences Ġare Ġnegligible . </s> Related Ġwork </s> EL Mo Ġcontextual Ġword Ġrepresentations Ġare Ġlearned Ġin Ġan Ġun super vised Ġway Ġthrough Ġlanguage Ġmodelling ĠB IB REF 0 . ĠThe Ġgeneral Ġarchitecture Ġconsists Ġof Ġa Ġtwo - layer ĠBi L ST M Ġon Ġtop Ġof Ġa Ġconv olution al Ġlayer Ġwhich Ġtakes Ġcharacter Ġsequences Ġas Ġits Ġinput . ĠSince Ġthe Ġmodel Ġuses Ġfully Ġcharacter - based Ġtoken Ġrepresentations , Ġit Ġavoids Ġthe Ġproblem Ġof Ġout - of - voc abulary Ġwords . ĠBecause Ġof Ġthis , Ġthe Ġauthors Ġexplicitly Ġrecommend Ġnot Ġto Ġuse Ġany Ġnormal isation Ġexcept Ġtoken ization Ġfor Ġthe Ġinput Ġtext . ĠHowever , Ġas Ġwe Ġshow Ġbelow , Ġwhile Ġthis Ġis Ġtrue Ġfor ĠEnglish , Ġfor Ġother Ġlanguages Ġfeeding ĠEL Mo Ġwith Ġle mm as Ġinstead Ġof Ġraw Ġtokens Ġcan Ġimprove ĠW SD Ġperformance . </s> Word Ġsense Ġdis amb ig uation Ġor ĠW SD ĠB IB REF 4 Ġis Ġthe ĠN LP Ġtask Ġconsisting Ġof Ġchoosing Ġa Ġword Ġsense Ġfrom Ġa Ġpre - defined Ġsense Ġinventory , Ġgiven Ġthe Ġcontext Ġin Ġwhich Ġthe Ġword Ġis Ġused . ĠW SD Ġfits Ġwell Ġinto Ġour Ġaim Ġto Ġintrinsically Ġevaluate ĠEL Mo Ġmodels , Ġsince Ġsolving Ġthe Ġproblem Ġof Ġpoly se my Ġand Ġhom ony my Ġwas Ġone Ġof Ġthe Ġoriginal Ġpromises Ġof Ġcontextual ised Ġembed d ings : Ġtheir Ġprimary Ġdifference Ġfrom Ġthe Ġprevious Ġgeneration Ġof Ġword Ġembed ding Ġmodels Ġis Ġthat Ġcontextual ised Ġapproaches Ġgenerate Ġdifferent Ġrepresentations Ġfor Ġhom ographs Ġdepending Ġon Ġthe Ġcontext . ĠWe Ġuse Ġtwo Ġlex ical Ġsample ĠW SD Ġtest Ġsets , Ġfurther Ġdescribed Ġin ĠSection ĠSEC REF 4 . </s> Training ĠEL Mo </s> For Ġthe Ġexperiments Ġdescribed Ġbelow , Ġwe Ġtrained Ġour Ġown ĠEL Mo Ġmodels Ġfrom Ġscratch . ĠFor ĠEnglish , Ġthe Ġtraining Ġcorpus Ġconsisted Ġof Ġthe ĠEnglish ĠWikipedia Ġdump Ġfrom ĠFebruary Ġ2017 . ĠFor ĠRussian , Ġit Ġwas Ġa Ġconc aten ation Ġof Ġthe ĠRussian ĠWikipedia Ġdump Ġfrom ĠDecember Ġ2018 Ġand Ġthe Ġfull ĠRussian ĠNational ĠCorpus Ġ( R NC ). ĠThe ĠRNC Ġtexts Ġwere Ġadded Ġto Ġthe ĠRussian ĠWikipedia Ġdump Ġso Ġas Ġto Ġmake Ġthe ĠRussian Ġtraining Ġcorpus Ġmore Ġcomparable Ġin Ġsize Ġto Ġthe ĠEnglish Ġone Ġ( Wikipedia Ġtexts Ġwould Ġcomprise Ġonly Ġhalf Ġof Ġthe Ġsize ). ĠAs ĠTable ĠT AB REF 3 Ġshows , Ġthe ĠEnglish ĠWikipedia Ġis Ġstill Ġtwo Ġtimes Ġlarger , Ġbut Ġat Ġleast Ġthe Ġorder Ġis Ġthe Ġsame . </s> The Ġtexts Ġwere Ġtoken ized Ġand Ġle mm at ized Ġwith Ġthe ĠUDP ipe Ġmodels Ġfor Ġthe Ġrespective Ġlanguages Ġtrained Ġon Ġthe ĠUniversal ĠDepend encies Ġ2 . 3 Ġtree banks ĠB IB REF 5 . ĠUDP ipe Ġyields Ġle mm at ization Ġaccuracy Ġabout Ġ96 % Ġfor ĠEnglish Ġand Ġ97 % Ġfor ĠRussian ; Ġthus Ġfor Ġthe Ġtask Ġat Ġhand , Ġwe Ġconsidered Ġit Ġto Ġbe Ġgold Ġand Ġdid Ġnot Ġtry Ġto Ġfurther Ġimprove Ġthe Ġquality Ġof Ġnormal isation Ġitself Ġ( although Ġit Ġis Ġnot Ġentirely Ġerror - free , Ġsee ĠSection ĠSEC REF 4 ). </s> EL Mo Ġmodels Ġwere Ġtrained Ġon Ġthese Ġcorpor a Ġusing Ġthe Ġoriginal ĠT ensor Flow Ġimplementation , Ġfor Ġ3 Ġepoch s Ġwith Ġbatch Ġsize Ġ192 , Ġon Ġtwo ĠGPUs . ĠTo Ġtrain Ġfaster , Ġwe Ġdecreased Ġthe Ġdimension ality Ġof Ġthe ĠL ST M Ġlayers Ġfrom Ġthe Ġdefault Ġ4096 Ġto Ġ2048 Ġfor Ġall Ġthe Ġmodels . </s> Word Ġsense Ġdis amb ig uation Ġtest Ġsets </s> We Ġused Ġtwo ĠW SD Ġdatasets Ġfor Ġevaluation : </s> Sense val - 3 Ġfor ĠEnglish ĠB IB REF 6 </s> R US SE ' 18 Ġfor ĠRussian ĠB IB REF 7 </s> The ĠSense val - 3 Ġdataset Ġconsists Ġof Ġlex ical Ġsamples Ġfor Ġnoun s , Ġverbs Ġand Ġadject ives ; Ġwe Ġused Ġonly Ġnoun Ġtarget Ġwords : </s> argument </s> arm </s> at mosp here </s> aud ience </s> bank </s> degree </s> diff erence </s> diff iculty </s> disc </s> image </s> interest </s> jud gement </s> organ ization </s> paper </s> party </s> performance </s> plan </s> s hel ter </s> sort </s> source </s> An Ġexample Ġfor Ġthe Ġambiguous Ġword Ġargument Ġis Ġgiven Ġbelow : </s> In Ġsome Ġsituations ĠPost script Ġcan Ġbe Ġfaster Ġthan Ġthe Ġescape Ġsequence Ġtype Ġof Ġprinter Ġcontrol Ġfile . ĠIt Ġuses Ġpost Ġfix Ġnotation , Ġwhere Ġarguments Ġcome Ġfirst Ġand Ġoperators Ġfollow . ĠThis Ġis Ġbasically Ġthe Ġsame Ġas ĠReverse ĠPolish ĠNot ation Ġas Ġused Ġon Ġcertain Ġcalcul ators , Ġand Ġfollows Ġdirectly Ġfrom Ġthe Ġstack Ġbased Ġapproach . </s> It Ġthis Ġsentence , Ġthe Ġword Ġ` argument ' Ġis Ġused Ġin Ġthe Ġsense Ġof Ġa Ġmathematical Ġoperator . </s> The ĠR US SE ' 18 Ġdataset Ġwas Ġcreated Ġin Ġ2018 Ġfor Ġthe Ġshared Ġtask Ġin ĠRussian Ġword Ġsense Ġinduction . ĠThis Ġdataset Ġcontains Ġonly Ġnoun s ; Ġthe Ġlist Ġof Ġwords Ġwith Ġtheir ĠEnglish Ġtranslations Ġis Ġgiven Ġin ĠTable ĠT AB REF 30 . </s> Originally , Ġit Ġincludes Ġalso Ġthe Ġwords Ġr ussian Ð ± Ð° Ð ¹ Ðº Ð° Ġ` tale / f lee ce ' Ġand Ġr ussian Ð ³ Ð² Ð¾Ð · Ð´ Ð¸ Ðº Ð° Ġ' cl ove / small Ġnail ', Ġbut Ġtheir Ġsenses Ġare Ġambiguous Ġonly Ġin Ġsome Ġinf lection al Ġforms Ġ( not Ġin Ġle mm as ), Ġtherefore Ġwe Ġdecided Ġto Ġexclude Ġthese Ġwords Ġfrom Ġevaluation . </s> The ĠRussian Ġdataset Ġis Ġmore Ġhom ogeneous Ġcompared Ġto Ġthe ĠEnglish Ġone , Ġas Ġfor Ġall Ġthe Ġtarget Ġwords Ġthere Ġis Ġapproximately Ġthe Ġsame Ġnumber Ġof Ġcontext Ġwords Ġin Ġthe Ġexamples . ĠThis Ġis Ġachieved Ġby Ġapplying Ġthe Ġlex ical Ġwindow Ġ( 25 Ġwords Ġbefore Ġand Ġafter Ġthe Ġtarget Ġword ) Ġand Ġcro pping Ġeverything Ġthat Ġfalls Ġoutside Ġof Ġthat Ġwindow . ĠIn Ġthe ĠEnglish Ġdataset , Ġon Ġthe Ġcontrary , Ġthe Ġwhole Ġparagraph Ġwith Ġthe Ġtarget Ġword Ġis Ġtaken Ġinto Ġaccount . ĠWe Ġhave Ġtried Ġcro pping Ġthe Ġexamples Ġfor ĠEnglish Ġas Ġwell , Ġbut Ġit Ġdid Ġnot Ġresult Ġin Ġany Ġchange Ġin Ġthe Ġquality Ġof Ġclassification . ĠIn Ġthe Ġend , Ġwe Ġdecided Ġnot Ġto Ġapply Ġthe Ġlex ical Ġwindow Ġto Ġthe ĠEnglish Ġdataset Ġso Ġas Ġnot Ġto Ġalter Ġit Ġand Ġrather Ġuse Ġit Ġin Ġthe Ġoriginal Ġform . </s> Here Ġis Ġan Ġexample Ġfrom Ġthe ĠR US SE ' 18 Ġfor Ġthe Ġambiguous Ġword Ġr ussian Ð¼ Ð° Ð½ Ð´ Ð° ÑĢ Ð¸ Ð½ Ġ` mand arin ' Ġin Ġthe Ġsense Ġ` Chinese Ġofficial Ġtitle ': </s> r ussian âĢ ľ ... Ð´ Ð¸ Ð ¿ Ð» Ð¾Ð ¼ Ð° ÑĤ Ð¸ Ñ ĩ Ðµ Ñģ Ðº Ð¾Ð ³ Ð¾ ĠÐ º Ð¾ ÑĢ Ð ¿ Ñĥ Ñģ Ð° ĠÐ ¾ Ñģ ÑĤ Ð° Ð½ Ðº Ð° Ð¼ ĠÐ ± Ð¾Ð ³ Ð´ Ñĭ Ñ ħ Ð° Ð½ Ð° ĠÐ ¸ ĠÐ ¸ Ð¼ Ð ¿ Ðµ ÑĢ Ð° ÑĤ ÑĢ Ð¸ Ñ Ĩ Ñĭ ĠÐ ¾ Ð ± Ñģ ÑĤ Ð° Ð² Ð» Ðµ Ð½ Ð¾ ĠÐ ± Ñĭ Ð» Ð¾ Ġ Ñģ ĠÐ ½ Ðµ Ð¾Ð ± Ñĭ Ñ ĩ Ð° Ð ¹ Ð½ Ð¾Ð ¹ Ġ ÑĤ Ð¾ ÑĢ Ð ¶ Ðµ Ñģ ÑĤ Ð² Ðµ Ð½ Ð½ Ð¾ Ñģ ÑĤ ÑĮ Ñ İ . ĠÐ ¢ Ñĭ Ñģ Ñı Ñ ĩ Ð¸ ĠÐ ¼ Ð° Ð½ Ð´ Ð° ÑĢ Ð¸ Ð½ Ð¾Ð ² ĠÐ ¸ ĠÐ ´ ÑĢ Ñĥ Ð ³ Ð¸ Ñ ħ ĠÐ ² Ñĭ Ñģ Ð¾Ð º Ð¾Ð ¿ Ð¾ Ñģ ÑĤ Ð° Ð² Ð» Ðµ Ð½ Ð½ Ñĭ Ñ ħ ĠÐ » Ð¸ Ñ Ĩ Ġ ÑĢ Ð° Ð · Ð¼ Ðµ Ñģ ÑĤ Ð¸ Ð» Ð¸ Ñģ ÑĮ Ġ Ñ Ī Ð ¿ Ð° Ð» Ðµ ÑĢ Ð° Ð¼ Ð¸ ĠÐ ½ Ð° Ġ ÑĤ ÑĢ Ðµ Ñ ħ ĠÐ ¼ ÑĢ Ð° Ð¼ Ð¾ ÑĢ Ð½ Ñĭ Ñ ħ Ġ ÑĤ Ðµ ÑĢ ÑĢ Ð° Ñģ Ð° Ñ ħ ĠÐ ² Ðµ Ð´ Ñĥ Ñ ī Ð¸ Ñ ħ ĠÐ º ... âĢ Ŀ </s> ` ... the Ġdiplomatic Ġbodies Ġof Ġthe ĠBog d ik han Ġand Ġthe ĠEmpress Ġwas Ġfurnished Ġwith Ġextraordinary Ġsolemn ity . ĠThousands Ġof Ġmand ar ins Ġand Ġother Ġdign it aries Ġwere Ġplaced Ġon Ġthree Ġmarble Ġterr aces Ġleading Ġto ... '. </s> Table ĠT AB REF 31 Ġcompares Ġboth Ġdatasets . ĠBefore Ġusage , Ġthey Ġwere Ġpre - process ed Ġin Ġthe Ġsame Ġway Ġas Ġthe Ġtraining Ġcorpor a Ġfor ĠEL Mo Ġ( see ĠSection ĠSEC REF 3 ), Ġthus Ġproducing Ġa Ġle mm at ized Ġand Ġa Ġnon - le mm at ized Ġversions Ġof Ġeach . </s> As Ġwe Ġcan Ġsee Ġfrom ĠTable ĠT AB REF 31 , Ġfor Ġ20 Ġtarget Ġwords Ġin ĠEnglish Ġthere Ġare Ġ24 Ġle mm as , Ġand Ġfor Ġ18 Ġtarget Ġwords Ġin ĠRussian Ġthere Ġare Ġ36 Ġdifferent Ġle mm as . ĠThese Ġnumbers Ġare Ġexplained Ġby Ġoccasional Ġerrors Ġin Ġthe ĠUDP ipe Ġle mm at ization . ĠAnother Ġinteresting Ġthing Ġto Ġobserve Ġis Ġthe Ġnumber Ġof Ġdistinct Ġword Ġforms Ġfor Ġevery Ġlanguage . ĠFor ĠEnglish , Ġthere Ġare Ġ39 Ġdistinct Ġforms Ġfor Ġ20 Ġtarget Ġnoun s : Ġsingular Ġand Ġplural Ġfor Ġevery Ġnoun , Ġexcept Ġ` at mosp here ' Ġwhich Ġis Ġused Ġonly Ġin Ġthe Ġsingular Ġform . ĠThus , Ġinf lection al Ġvariability Ġof ĠEnglish Ġnoun s Ġis Ġcovered Ġby Ġthe Ġdataset Ġalmost Ġcompletely . ĠFor ĠRussian , Ġwe Ġobserve Ġ132 Ġdistinct Ġforms Ġfor Ġ18 Ġtarget Ġnoun s , Ġgiving Ġmore Ġthan Ġ7 Ġinf lection al Ġforms Ġper Ġeach Ġword . ĠNote Ġthat Ġthis Ġstill Ġcovers Ġonly Ġhalf Ġof Ġall Ġthe Ġinf lection al Ġvariability Ġof ĠRussian : Ġthis Ġlanguage Ġfeatures Ġ12 Ġdistinct Ġforms Ġfor Ġeach Ġnoun Ġ( 6 Ġcases Ġand Ġ2 Ġnumbers ). </s> To Ġsum Ġup , Ġthe ĠR US SE ' 18 Ġdataset Ġis Ġmorph ologically Ġfar Ġmore Ġcomplex Ġthan Ġthe ĠSense val 3 , Ġreflecting Ġthe Ġproperties Ġof Ġthe Ġrespective Ġlanguages . ĠIn Ġthe Ġnext Ġsection Ġwe Ġwill Ġsee Ġthat Ġthis Ġleads Ġto Ġsubstantial Ġdifferences Ġregarding Ġcomparisons Ġbetween Ġtoken - based Ġand Ġle mma - based ĠEL Mo Ġmodels . </s> Exper iments </s> Following ĠB IB REF 8 , Ġwe Ġdecided Ġto Ġavoid Ġusing Ġany Ġstandard Ġtrain - test Ġsplits Ġfor Ġour ĠW SD Ġdatasets . ĠInstead , Ġwe Ġrely Ġon Ġper - word Ġrandom Ġsplits Ġand Ġ5 - fold Ġcross - valid ation . ĠThis Ġmeans Ġthat Ġfor Ġeach Ġtarget Ġword Ġwe Ġrandomly Ġgenerate Ġ5 Ġdifferent Ġdivisions Ġof Ġits Ġcontext Ġsentences Ġlist Ġinto Ġtrain Ġand Ġtest Ġsets , Ġand Ġthen Ġtrain Ġand Ġtest Ġ5 Ġdifferent Ġclass ifier Ġmodels Ġon Ġthis Ġdata . ĠThe Ġresulting Ġperformance Ġscore Ġfor Ġeach Ġtarget Ġword Ġis Ġthe Ġaverage Ġof Ġ5 Ġmacro - F 1 Ġscores Ġproduced Ġby Ġthese Ġclass ifiers . </s> EL Mo Ġmodels Ġcan Ġbe Ġemployed Ġfor Ġthe ĠW SD Ġtask Ġin Ġtwo Ġdifferent Ġways : Ġeither Ġby Ġfine - tun ing Ġthe Ġmodel Ġor Ġby Ġextracting Ġword Ġrepresentations Ġfrom Ġit Ġand Ġthen Ġusing Ġthem Ġas Ġfeatures Ġin Ġa Ġdownstream Ġclass ifier . ĠWe Ġdecided Ġto Ġstick Ġto Ġthe Ġsecond Ġ( feature Ġextraction ) Ġapproach , Ġsince Ġit Ġis Ġconcept ually Ġand Ġcomput ationally Ġsimpler . ĠAdditionally , ĠB IB REF 9 Ġshowed Ġthat Ġfor Ġmost ĠN LP Ġtasks Ġ( except Ġthose Ġfocused Ġon Ġsentence Ġpairs ) Ġthe Ġperformance Ġof Ġfeature Ġextraction Ġand Ġfine - tun ing Ġis Ġnearly Ġthe Ġsame . ĠThus Ġwe Ġextracted Ġthe Ġsingle Ġvector Ġof Ġthe Ġtarget Ġword Ġfrom Ġthe ĠEL Mo Ġtop Ġlayer Ġ( ` target ' Ġrows Ġin ĠTable ĠT AB REF 32 ) Ġor Ġthe Ġaveraged ĠEL Mo Ġtop Ġlayer Ġvectors Ġof Ġall Ġwords Ġin Ġthe Ġcontext Ġsentence Ġ( ` aver aged ' Ġrows Ġin ĠTable ĠT AB REF 32 ). </s> For Ġcomparison , Ġwe Ġalso Ġreport Ġthe Ġscores Ġof Ġthe Ġ` aver aged Ġvectors ' Ġrepresentations Ġwith ĠContinuous ĠSkip gram ĠB IB REF 2 Ġembed ding Ġmodels Ġtrained Ġon Ġthe ĠEnglish Ġor ĠRussian ĠWikipedia Ġdumps Ġ( ` SG NS ' Ġrows ): Ġbefore Ġthe Ġadvent Ġof Ġcontextual ised Ġmodels , Ġthis Ġwas Ġone Ġof Ġthe Ġmost Ġwidely Ġused Ġways Ġto Ġ` squ ee ze ' Ġthe Ġmeaning Ġof Ġa Ġsentence Ġinto Ġa Ġfixed - size Ġvector . ĠOf Ġcourse Ġit Ġdoes Ġnot Ġmean Ġthat Ġthe Ġmeaning Ġof Ġa Ġsentence Ġalways Ġdetermines Ġthe Ġsenses Ġall Ġits Ġwords Ġare Ġused Ġin . ĠHowever , Ġaveraging Ġrepresentations Ġof Ġwords Ġin Ġcontexts Ġas Ġa Ġproxy Ġto Ġthe Ġsense Ġof Ġone Ġparticular Ġword Ġis Ġa Ġlong Ġestablished Ġtradition Ġin ĠW SD , Ġstarting Ġat Ġleast Ġfrom ĠB IB REF 10 . ĠAlso , Ġsince ĠSG NS Ġis Ġa Ġ` static ' Ġembed ding Ġmodel , Ġit Ġis Ġof Ġcourse Ġnot Ġpossible Ġto Ġuse Ġonly Ġtarget Ġword Ġvectors Ġas Ġfeatures : Ġthey Ġwould Ġbe Ġidentical Ġwhatever Ġthe Ġcontext Ġis . </s> Simple Ġlog istic Ġregression Ġwas Ġused Ġas Ġa Ġclassification Ġalgorithm . ĠWe Ġalso Ġtested Ġa Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġclass ifier Ġwith Ġ200 - ne ur ons Ġhidden Ġlayer , Ġwhich Ġyielded Ġessentially Ġthe Ġsame Ġresults . ĠThis Ġleads Ġus Ġto Ġbelieve Ġthat Ġour Ġfindings Ġare Ġnot Ġclass ifier - dependent . </s> Table ĠT AB REF 32 Ġshows Ġthe Ġresults , Ġtogether Ġwith Ġthe Ġrandom Ġand Ġmost Ġfrequent Ġsense Ġ( M FS ) Ġbas elines Ġfor Ġeach Ġdataset . </s> First , ĠEL Mo Ġoutper forms ĠSG NS Ġfor Ġboth Ġlanguages , Ġwhich Ġcomes Ġas Ġno Ġsurprise . ĠSecond , Ġthe Ġapproach Ġwith Ġaveraging Ġrepresentations Ġfrom Ġall Ġwords Ġin Ġthe Ġsentence Ġis Ġnot Ġbeneficial Ġfor ĠW SD Ġwith ĠEL Mo : Ġfor ĠEnglish Ġdata , Ġit Ġclearly Ġloses Ġto Ġa Ġsingle Ġtarget Ġword Ġrepresentation , Ġand Ġfor ĠRussian Ġthere Ġare Ġno Ġsignificant Ġdifferences Ġ( and Ġusing Ġa Ġsingle Ġtarget Ġword Ġis Ġpreferable Ġfrom Ġthe Ġcomputational Ġpoint Ġof Ġview , Ġsince Ġit Ġdoes Ġnot Ġrequire Ġthe Ġaveraging Ġoperation ). ĠThus , Ġbelow Ġwe Ġdiscuss Ġonly Ġthe Ġsingle Ġtarget Ġword Ġusage Ġmode Ġof ĠEL Mo . </s> But Ġthe Ġmost Ġimportant Ġpart Ġis Ġthe Ġcomparison Ġbetween Ġusing Ġtokens Ġor Ġle mm as Ġin Ġthe Ġtrain Ġand Ġtest Ġdata . ĠFor Ġthe Ġ` static ' ĠSG NS Ġembed d ings , Ġit Ġdoes Ġnot Ġsignificantly Ġchange Ġthe ĠW SD Ġscores Ġfor Ġboth Ġlanguages . ĠThe Ġsame Ġis Ġtrue Ġfor ĠEnglish ĠEL Mo Ġmodels , Ġwhere Ġdifferences Ġare Ġnegligible Ġand Ġseem Ġto Ġbe Ġsimple Ġfluctuations . ĠHowever , Ġfor ĠRussian , ĠEL Mo Ġ( target ) Ġon Ġle mm as Ġoutper forms ĠEL Mo Ġon Ġtokens , Ġwith Ġsmall Ġbut Ġsignificant Ġimprovement . ĠThe Ġmost Ġplausible Ġexplanation Ġfor Ġthis Ġis Ġthat Ġ( despite Ġof Ġpurely Ġcharacter - based Ġinput Ġof ĠEL Mo ) Ġthe Ġmodel Ġdoes Ġnot Ġhave Ġto Ġlearn Ġidiosyncr as ies Ġof Ġa Ġparticular Ġlanguage Ġmorphology . ĠInstead , Ġit Ġcan Ġuse Ġits Ġ( limited ) Ġcapacity Ġto Ġbetter Ġlearn Ġlex ical Ġsemantic Ġstructures , Ġleading Ġto Ġbetter ĠW SD Ġperformance . ĠThe Ġbox Ġplots ĠFIG REF 33 Ġand ĠFIG REF 35 Ġillustrate Ġthe Ġscores Ġdisp ersion Ġacross Ġwords Ġin Ġthe Ġtest Ġsets Ġfor ĠEnglish Ġand ĠRussian Ġcorrespond ingly Ġ( orange Ġlines Ġare Ġmed ians ). ĠIn Ġthe Ġnext Ġsection ĠSEC REF 6 Ġwe Ġanalyse Ġthe Ġresults Ġqual itatively . </s> Qual itative Ġanalysis </s> In Ġthis Ġsection Ġwe Ġfocus Ġon Ġthe Ġcomparison Ġof Ġscores Ġfor Ġthe ĠRussian Ġdataset . ĠThe Ġclass ifier Ġfor ĠRussian Ġhad Ġto Ġchoose Ġbetween Ġfewer Ġclasses Ġ( two Ġor Ġthree ), Ġwhich Ġmade Ġthe Ġscores Ġhigher Ġand Ġmore Ġconsistent Ġthan Ġfor Ġthe ĠEnglish Ġdataset . ĠOverall , Ġwe Ġsee Ġimprovements Ġin Ġthe Ġscores Ġfor Ġthe Ġmajority Ġof Ġwords , Ġwhich Ġproves Ġthat Ġle mm at ization Ġfor Ġmorph ologically Ġrich Ġlanguages Ġis Ġbeneficial . </s> We Ġdecided Ġto Ġanalyse Ġmore Ġclosely Ġthose Ġwords Ġfor Ġwhich Ġthe Ġdifference Ġin Ġthe Ġscores Ġbetween Ġle mma - based Ġand Ġtoken - based Ġmodels Ġwas Ġstatistically Ġsignificant . ĠBy Ġ` significant ' Ġwe Ġmean Ġthat Ġthe Ġscores Ġdiffer Ġby Ġmore Ġthat Ġone Ġstandard Ġdeviation Ġ( the Ġlargest Ġstandard Ġdeviation Ġvalue Ġin Ġthe Ġtwo Ġsets Ġwas Ġtaken ). ĠThe Ġresulting Ġlist Ġof Ġtargets Ġwords Ġwith Ġsignificant Ġdifference Ġin Ġscores Ġis Ġgiven Ġin ĠTable ĠT AB REF 36 . </s> We Ġcan Ġsee Ġthat Ġamong Ġ18 Ġwords Ġin Ġthe Ġdataset Ġonly Ġ3 Ġexhibit Ġsignificant Ġimprovement Ġin Ġtheir Ġscores Ġwhen Ġmoving Ġfrom Ġtokens Ġto Ġle mm as Ġin Ġthe Ġinput Ġdata . ĠIt Ġshows Ġthat Ġeven Ġthough Ġthe Ġoverall ĠF 1 Ġscores Ġfor Ġthe ĠRussian Ġdata Ġhave Ġshown Ġthe Ġplaus ibility Ġof Ġle mm at ization , Ġthis Ġimprovement Ġis Ġmostly Ġdriven Ġby Ġa Ġfew Ġwords . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġthese Ġwords ' Ġscores Ġfeature Ġvery Ġlow Ġstandard Ġdeviation Ġvalues Ġ( for Ġother Ġwords , Ġstandard Ġdeviation Ġvalues Ġwere Ġabove Ġ0 . 1 , Ġmaking ĠF 1 Ġdifferences Ġinsignificant ). ĠSuch Ġa Ġbehaviour Ġcan Ġbe Ġcaused Ġby Ġmore Ġconsistent Ġdifferentiation Ġof Ġcontext Ġfor Ġvarious Ġsenses Ġof Ġthese Ġ3 Ġwords . ĠFor Ġexample , Ġwith Ġthe Ġword Ġr ussian Ðº Ð° Ð ± Ð° Ñ ĩ Ð¾Ð º Ġ` squ ash Ġ/ Ġsmall Ġrestaurant ', Ġthe Ġcontexts Ġfor Ġboth Ġsenses Ġcan Ġbe Ġsimilar , Ġsince Ġthey Ġare Ġall Ġrelated Ġto Ġfood . ĠThis Ġmakes Ġthe ĠW SD Ġscores Ġunstable . ĠOn Ġthe Ġother Ġhand , Ġfor Ġr ussian Ð° Ðº Ñ Ĩ Ð¸ Ñı Ġ` stock , Ġshare Ġ/ Ġevent ', Ġr ussian Ðº ÑĢ Ð¾ Ð½ Ð° Ġ` c rown Ġ( tree Ġ/ Ġcoin )' Ġor Ġr ussian Ðº ÑĢ Ñĥ Ð ¿ Ġ` c roup Ġ( horse Ġbody Ġpart Ġ/ Ġillness ) ', Ġtheir Ġsenses Ġare Ġnot Ġrelated , Ġwhich Ġresulted Ġin Ġmore Ġstable Ġresults Ġand Ġsignificant Ġdifference Ġin Ġthe Ġscores Ġ( see ĠTable ĠT AB REF 36 ). </s> There Ġis Ġonly Ġone Ġword Ġin Ġthe ĠR US SE ' 18 Ġdataset Ġfor Ġwhich Ġthe Ġscore Ġhas Ġstrongly Ġdecreased Ġwhen Ġmoving Ġto Ġle mma - based Ġmodels : Ġr ussian Ð´ Ð¾Ð ¼ Ð¸ Ð½ Ð¾ Ġ` dom ino Ġ( game Ġ/ Ġcostume ) '. ĠIn Ġfact , Ġthe Ġscore Ġdifference Ġhere Ġlies Ġon Ġthe Ġborder Ġof Ġone Ġstandard Ġdeviation , Ġso Ġstrictly Ġspeaking Ġit Ġis Ġnot Ġreally Ġsignificant . ĠHowever , Ġthe Ġword Ġstill Ġpresents Ġan Ġinteresting Ġphenomenon . </s> r ussian Ð Ķ Ð¾Ð ¼ Ð¸ Ð½ Ð¾ Ġis Ġthe Ġonly Ġtarget Ġnoun Ġin Ġthe ĠR US SE ' 18 Ġthat Ġhas Ġno Ġinf lected Ġforms , Ġsince Ġit Ġis Ġa Ġborrowed Ġword . ĠThis Ġleaves Ġno Ġroom Ġfor Ġimprovement Ġwhen Ġusing Ġle mma - based ĠEL Mo Ġmodels : Ġall Ġtokens Ġof Ġthis Ġword Ġare Ġalready Ġidentical . ĠAt Ġthe Ġsame Ġtime , Ġsome Ġinformation Ġabout Ġinf lected Ġword Ġforms Ġin Ġthe Ġcontext Ġcan Ġbe Ġuseful , Ġbut Ġit Ġis Ġlost Ġduring Ġle mm at ization , Ġand Ġthis Ġleads Ġto Ġthe Ġdecreased Ġscore . ĠAr gu ably , Ġthis Ġmeans Ġthat Ġle mm at ization Ġbrings Ġalong Ġboth Ġadvantages Ġand Ġdisadvantages Ġfor ĠW SD Ġwith ĠEL Mo . ĠFor Ġinf lected Ġwords Ġ( which Ġconstitute Ġthe Ġmajority Ġof ĠRussian Ġvocabulary ) Ġprofits Ġoutweigh Ġthe Ġlosses , Ġbut Ġfor Ġat yp ical Ġnon - change able Ġwords Ġit Ġcan Ġbe Ġthe Ġopposite . </s> The Ġscores Ġfor Ġthe Ġexcluded Ġtarget Ġwords Ġr ussian Ð ± Ð° Ð ¹ Ðº Ð° Ġ` tale Ġ/ Ġflee ce ' Ġand Ġr ussian Ð ³ Ð² Ð¾Ð · Ð´ Ð¸ Ðº Ð° Ġ' cl ove Ġ/ Ġsmall Ġnail ' Ġare Ġgiven Ġin ĠTable ĠT AB REF 37 Ġ( rec all Ġthat Ġthey Ġwere Ġexcluded Ġbecause Ġof Ġbeing Ġambiguous Ġonly Ġin Ġsome Ġinf lection al Ġforms ). ĠFor Ġthese Ġwords Ġwe Ġcan Ġsee Ġa Ġgreat Ġimprovement Ġwith Ġle mma - based Ġmodels . ĠThis , Ġof Ġcourse Ġstems Ġfrom Ġthe Ġfact Ġthat Ġthese Ġwords Ġin Ġdifferent Ġsenses Ġhave Ġdifferent Ġle mm as . ĠTherefore , Ġthe Ġresults Ġare Ġheavily Ġdependent Ġon Ġthe Ġquality Ġof Ġle mm at ization . </s> Conclusion </s> We Ġevaluated Ġhow Ġthe Ġability Ġof ĠEL Mo Ġcontextual ised Ġword Ġembed ding Ġmodels Ġto Ġdis amb ig uate Ġword Ġsenses Ġdepends Ġon Ġthe Ġnature Ġof Ġthe Ġtraining Ġdata . ĠIn Ġparticular , Ġwe Ġcompared Ġthe Ġmodels Ġtrained Ġon Ġraw Ġtoken ized Ġcorpor a Ġand Ġthose Ġtrained Ġon Ġthe Ġcorpor a Ġwith Ġword Ġtokens Ġreplaced Ġby Ġtheir Ġnormal Ġforms Ġ( le mm as ). ĠThe Ġmodels Ġwe Ġtrained Ġare Ġpublicly Ġavailable Ġvia Ġthe ĠNL PL Ġword Ġembed d ings Ġrepository ĠB IB REF 3 . </s> In Ġthe Ġmajority Ġof Ġresearch Ġpapers Ġon Ġdeep Ġlearning Ġapproaches Ġto ĠN LP , Ġit Ġis Ġassumed Ġthat Ġle mm at ization Ġis Ġnot Ġnecessary , Ġespecially Ġwhen Ġusing Ġpowerful Ġcontextual ised Ġembed d ings . ĠOur Ġexperiments Ġshow Ġthat Ġthis Ġis Ġindeed Ġtrue Ġfor Ġlanguages Ġwith Ġsimple Ġmorphology Ġ( like ĠEnglish ). ĠHowever , Ġfor Ġrich - morph ology Ġlanguages Ġ( like ĠRussian ), Ġusing Ġle mm at ized Ġtraining Ġdata Ġyields Ġsmall Ġbut Ġconsistent Ġimprovements Ġin Ġthe Ġword Ġsense Ġdis amb ig uation Ġtask . ĠThese Ġimprovements Ġare Ġnot Ġobserved Ġfor Ġrare Ġwords Ġwhich Ġlack Ġinf lected Ġforms ; Ġthis Ġfurther Ġsupports Ġour Ġhypothesis Ġthat Ġbetter ĠW SD Ġscores Ġof Ġle mma - based Ġmodels Ġare Ġrelated Ġto Ġthem Ġbetter Ġhandling Ġmultiple Ġword Ġforms Ġin Ġmorphology - rich Ġlanguages . </s> Of Ġcourse , Ġle mm at ization Ġis Ġby Ġall Ġmeans Ġnot Ġa Ġsilver Ġbullet . ĠIn Ġother Ġtasks , Ġwhere Ġinf lection al Ġproperties Ġof Ġwords Ġare Ġimportant , Ġit Ġcan Ġeven Ġhurt Ġthe Ġperformance . ĠBut Ġthis Ġis Ġtrue Ġfor Ġany ĠN LP Ġsystems , Ġnot Ġonly Ġdeep Ġlearning Ġbased Ġones . </s> The Ġtake - home Ġmessage Ġhere Ġis Ġtwo fold : Ġfirst , Ġtext Ġpre - processing Ġstill Ġmatters Ġfor Ġcontemporary Ġdeep Ġlearning Ġalgorithms . ĠTheir Ġimpressive Ġlearning Ġabilities Ġdo Ġnot Ġalways Ġallow Ġthem Ġto Ġinfer Ġnormal isation Ġrules Ġthemselves , Ġfrom Ġsimply Ġoptim ising Ġthe Ġlanguage Ġmodelling Ġtask . ĠSecond , Ġthe Ġnature Ġof Ġlanguage Ġat Ġhand Ġmatters Ġas Ġwell , Ġand Ġdifferences Ġin Ġthis Ġnature Ġcan Ġresult Ġin Ġdifferent Ġdecisions Ġbeing Ġoptimal Ġor Ġsub - opt imal Ġat Ġthe Ġstage Ġof Ġdeep Ġlearning Ġmodels Ġtraining . ĠThe Ġsimple Ġtruth Ġ` English Ġis Ġnot Ġrepresentative Ġof Ġall Ġlanguages Ġon ĠEarth ' Ġstill Ġholds Ġhere . </s> In Ġthe Ġfuture , Ġwe Ġplan Ġto Ġextend Ġour Ġwork Ġby Ġincluding Ġmore Ġlanguages Ġinto Ġthe Ġanalysis . ĠUsing ĠRussian Ġand ĠEnglish Ġallowed Ġus Ġto Ġhypothes ise Ġabout Ġthe Ġimportance Ġof Ġmorph ological Ġcharacter Ġof Ġa Ġlanguage . ĠBut Ġwe Ġonly Ġscratched Ġthe Ġsurface Ġof Ġthe Ġlinguistic Ġdiversity . ĠTo Ġverify Ġthis Ġclaim , Ġit Ġis Ġnecessary Ġto Ġanalyse Ġmore Ġstrongly Ġinf lected Ġlanguages Ġlike ĠRussian Ġas Ġwell Ġas Ġmore Ġweak ly Ġinf lected Ġ( analy tical ) Ġlanguages Ġsimilar Ġto ĠEnglish . ĠThis Ġwill Ġhelp Ġto Ġfind Ġout Ġif Ġthe Ġinf lection Ġdifferences Ġare Ġimportant Ġfor Ġtraining Ġdeep Ġlearning Ġmodels Ġacross Ġhuman Ġlanguages Ġin Ġgeneral .