The Näive-Bayes classifier is corrected so it is not biased to most frequent classes
ABSTRACTIVE
Bayesian classifier has been modified, removing the bias towards frequent labels in the training data
EXTRACTIVE
Monosemous relatives have been employed multiple times (see Section 2), but results remain unsatisfactory. The aim of my study is to explore the limitations of this technique by implementing and evaluating such a tool for Polish. Firstly, the method is expanded by waiving the requirement of monosemy and proposing several new sources of relatives. These previously unexplored sources are based on wordnet data and help gather many training cases from the corpus. Secondly, a well-known problem of uneven yet unknown distribution of word senses is alleviated by modifying a naïve Bayesian classifier. Thanks to this correction, the classifier is no longer biased towards senses that have more training data. Finally, a very large corpus (600 million documents), gathered from the web by a Polish search engine NEKST, is used to build models based on training corpora of different sizes. Those experiments show what amount of data is sufficient for such a task. The proposed solution is compared to baselines that use wordnet structure only, with no training corpora. The algorithm works as follows. First, a set of relatives is obtained for each sense of a target word using the Polish wordnet: plWordNet BIBREF18 . Some of the replacements may have multiple senses, however usually one of them covers most cases. Secondly, a set of context features is extracted from occurrences of relatives in the NEKST corpus. Finally, the aggregated feature values corresponding to target word senses are used to build a naïve Bayesian classifier adjusted to a situation of unknown a priori probabilities.
Which could be rewritten as: INLINEFORM0 The expression has been formulated as a product of two factors: INLINEFORM0 , independent from observed features and corresponding to empty word context, and INLINEFORM1 that depends on observed context. To weaken the influence of improper distribution of training cases, we omit INLINEFORM2 , so that when no context features are observed, every word sense is considered equally probable. In this paper the limitations and improvements of unsupervised word sense disambiguation have been investigated. The main problem – insufficient number and quality of replacements has been tackled by adding new rich sources of replacements. The quality of the models has indeed improved, especially thanks to replacements based on sense ordering in plWordNet. To deal with the problem of unknown sense distribution, the Bayesian classifier has been modified, removing the bias towards frequent labels in the training data. Finally, the experiments with very large corpus have shown the sufficient amount of training data for this task, which is only 6 million documents.
1710.07960
Introduction </s> The Ġfocus Ġof Ġthe Ġword Ġsense Ġdis amb ig uation Ġ( W SD ) Ġtask Ġis Ġpoly se my , Ġi . e . Ġwords Ġhaving Ġseveral Ġsubstantially Ġdifferent Ġmeanings . ĠTwo Ġcommon Ġexamples Ġare Ġbank Ġ( ri vers ide Ġor Ġfinancial Ġinstitution ) Ġand Ġbass Ġ( fish Ġor Ġmusical Ġinstrument ), Ġbut Ġusually Ġthe Ġmeanings Ġof Ġa Ġword Ġare Ġclosely Ġrelated , Ġe . g . Ġclass Ġmay Ġrefer Ġto : Ġ( a ) Ġa Ġgroup Ġof Ġstudents , Ġ( b ) Ġthe Ġperiod Ġwhen Ġthey Ġmeet Ġto Ġstudy Ġor Ġ( c ) Ġa Ġroom Ġwhere Ġsuch Ġmeetings Ġoccur . ĠReaders Ġdeal Ġwith Ġthis Ġproblem Ġby Ġusing Ġa Ġword 's Ġcontext Ġand Ġin ĠW SD Ġwe Ġaim Ġat Ġdoing Ġit Ġautomatically . </s> The Ġmost Ġeffective Ġsolution , Ġcalled Ġsupervised ĠW SD , Ġis Ġto Ġuse Ġa Ġlarge Ġnumber Ġof Ġsense - annot ated Ġoccurrences Ġof Ġthe Ġtarget Ġword Ġto Ġbuild Ġa Ġmachine Ġlearning Ġmodel Ġto Ġlabel Ġtest Ġcases . ĠHowever , Ġthis Ġapproach Ġsuffers Ġfrom Ġa Ġknowledge Ġacquisition Ġbottleneck . ĠThe Ġannotation Ġof Ġa Ġseparate Ġtraining Ġcorpus Ġfor Ġevery Ġtarget Ġword Ġdemands Ġa Ġconsiderable Ġamount Ġof Ġhuman Ġlabour . ĠTherefore , Ġthis Ġapproach Ġis Ġunus able Ġin Ġapplications Ġthat Ġrequire ĠW SD Ġacross Ġa Ġwide Ġvocabulary , Ġsuch Ġas Ġopen - domain Ġquestion Ġanswering ĠB IB REF 0 Ġ. </s> The Ġmethod Ġof Ġmon ose m ous Ġrelatives , Ġwhich Ġis Ġthe Ġfocus Ġof Ġthis Ġwork , Ġbypass es Ġthe Ġbottleneck Ġby Ġgathering Ġocc ure nces Ġof Ġwords Ġrelated Ġto Ġthe Ġtarget Ġword , Ġbut Ġfree Ġfrom Ġambiguity , Ġand Ġtreating Ġthem Ġas Ġtraining Ġcases Ġof Ġthe Ġrespective Ġsenses . ĠHuman Ġlabour Ġis Ġeliminated Ġat Ġthe Ġexpense Ġof Ġaccuracy , Ġas Ġthe Ġcontext Ġof Ġeach Ġrelative Ġonly Ġapproximately Ġmatches Ġthe Ġcontext Ġof Ġthe Ġtarget Ġword Ġsense . </s> Mon ose m ous Ġrelatives Ġhave Ġbeen Ġemployed Ġmultiple Ġtimes Ġ( see ĠSection Ġ2 ), Ġbut Ġresults Ġremain Ġunsatisf actory . ĠThe Ġaim Ġof Ġmy Ġstudy Ġis Ġto Ġexplore Ġthe Ġlimitations Ġof Ġthis Ġtechnique Ġby Ġimplementing Ġand Ġevaluating Ġsuch Ġa Ġtool Ġfor ĠPolish . ĠFirstly , Ġthe Ġmethod Ġis Ġexpanded Ġby Ġwa iving Ġthe Ġrequirement Ġof Ġmon ose my Ġand Ġproposing Ġseveral Ġnew Ġsources Ġof Ġrelatives . ĠThese Ġpreviously Ġunexpl ored Ġsources Ġare Ġbased Ġon Ġword net Ġdata Ġand Ġhelp Ġgather Ġmany Ġtraining Ġcases Ġfrom Ġthe Ġcorpus . ĠSecondly , Ġa Ġwell - known Ġproblem Ġof Ġuneven Ġyet Ġunknown Ġdistribution Ġof Ġword Ġsenses Ġis Ġallev iated Ġby Ġmodifying Ġa ĠnaÃ¯ve ĠBay esian Ġclass ifier . ĠThanks Ġto Ġthis Ġcorrection , Ġthe Ġclass ifier Ġis Ġno Ġlonger Ġbiased Ġtowards Ġsenses Ġthat Ġhave Ġmore Ġtraining Ġdata . ĠFinally , Ġa Ġvery Ġlarge Ġcorpus Ġ( 600 Ġmillion Ġdocuments ), Ġgathered Ġfrom Ġthe Ġweb Ġby Ġa ĠPolish Ġsearch Ġengine ĠNE K ST , Ġis Ġused Ġto Ġbuild Ġmodels Ġbased Ġon Ġtraining Ġcorpor a Ġof Ġdifferent Ġsizes . ĠThose Ġexperiments Ġshow Ġwhat Ġamount Ġof Ġdata Ġis Ġsufficient Ġfor Ġsuch Ġa Ġtask . ĠThe Ġproposed Ġsolution Ġis Ġcompared Ġto Ġbas elines Ġthat Ġuse Ġword net Ġstructure Ġonly , Ġwith Ġno Ġtraining Ġcorpor a . </s> This Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠThe Ġnext Ġsection Ġreviews Ġthe Ġprevious Ġresearch Ġin Ġthe Ġarea , Ġfocusing Ġon Ġun super vised ĠW SD Ġusing Ġmon ose m ous Ġrelatives . ĠSection Ġ3 Ġoutlines Ġthe Ġproposed Ġsolution Ġby Ġdescribing Ġthe Ġnew Ġsources Ġof Ġrelatives , Ġthe Ġemployed Ġcorpus , Ġthe Ġfeatures Ġextracted Ġfrom Ġcontext Ġand Ġthe Ġmodified ĠBay esian Ġclass ifier . ĠSection Ġ4 Ġdescribes Ġthe Ġevaluation Ġdata Ġand Ġprocess , Ġwhile Ġsection Ġ5 Ġquotes Ġthe Ġresults . ĠSection Ġ6 Ġis Ġdevoted Ġto Ġdiscussing Ġthe Ġoutcomes Ġand Ġsection Ġ7 Ġconcludes Ġthe Ġpaper . </s> Related Ġwork </s> The Ġproblem Ġof ĠW SD Ġhas Ġreceived Ġa Ġlot Ġof Ġattention Ġsince Ġthe Ġbeginning Ġof Ġnatural Ġlanguage Ġprocessing Ġresearch . ĠW SD Ġis Ġtypically Ġexpected Ġto Ġimprove Ġthe Ġresults Ġof Ġreal - world Ġapplications : Ġoriginally Ġmachine Ġtranslation Ġand Ġrecently Ġinformation Ġretrieval Ġand Ġextraction , Ġespecially Ġquestion Ġanswering ĠB IB REF 0 Ġ. ĠLike Ġmany Ġother Ġareas , ĠW SD Ġhas Ġgreatly Ġbenefited Ġfrom Ġpublicly Ġavailable Ġtest Ġsets Ġand Ġcompetitions . ĠTwo Ġnotable Ġcorpor a Ġare : Ġ1 ) ĠSem Cor ĠB IB REF 1 Ġ, Ġbuilt Ġby Ġlab elling Ġa Ġsubset Ġof ĠBrown Ġcorpus Ġwith ĠPrinceton ĠWord Net Ġsy ns ets Ġand Ġ2 ) Ġthe Ġpublic Ġevaluations Ġof ĠSense val Ġworkshops ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> There Ġare Ġa Ġvariety Ġof Ġapproaches Ġto Ġsolve Ġthe ĠW SD Ġproblem , Ġwhich Ġcan Ġbe Ġgrouped Ġbased Ġupon Ġhow Ġthey Ġuse Ġtheir Ġdata ĠâĢĵ Ġsee Ġreviews ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. ĠIn Ġsupervised Ġsolutions Ġa Ġlarge Ġsense - tag ged Ġcorpus Ġis Ġavailable Ġfor Ġtraining . ĠThis Ġapproach Ġhas Ġbeen Ġapplied Ġto Ġthe Ġthe Ġtest Ġset Ġused Ġin Ġthe Ġcurrent Ġstudy , Ġresulting Ġin Ġan Ġaccuracy Ġvalue Ġof Ġ91 . 5 % ĠB IB REF 6 Ġ. ĠAlthough Ġthis Ġtechnique Ġundoubtedly Ġyields Ġthe Ġbest Ġresults , Ġwe Ġwould Ġneed Ġan Ġimmense Ġamount Ġof Ġhuman Ġlabour Ġto Ġbuild Ġa Ġtraining Ġcorpus Ġof Ġsufficient Ġsize Ġfor Ġdis amb ig uating Ġall Ġwords . ĠThis Ġdoes Ġnot Ġseem Ġpossible , Ġespecially Ġin Ġthe Ġcase Ġof Ġlanguages , Ġsuch Ġas ĠPolish , Ġwhich Ġreceive Ġless Ġattention Ġthan ĠEnglish . </s> In Ġthe Ġminim ally Ġsupervised Ġapproach ĠB IB REF 7 Ġ, Ġa Ġsmall Ġset Ġof Ġinitial Ġtraining Ġexamples , Ġobtained Ġby Ġa Ġhe uristic Ġor Ġhand - tag ging , Ġis Ġused Ġto Ġlabel Ġnew Ġoccurrences . ĠThey Ġin Ġturn Ġserve Ġas Ġa Ġtraining Ġset Ġfor Ġnext Ġiteration , Ġand Ġso Ġon . ĠThis Ġboot stra pping Ġprocedure Ġrequires Ġvery Ġlittle Ġmanual Ġtagging Ġbut Ġneeds Ġto Ġbe Ġcarefully Ġimplemented Ġto Ġavoid Ġlo osing Ġaccuracy Ġin Ġfurther Ġsteps . </s> Un super vised Ġmethods Ġuse Ġno Ġpreviously Ġlabelled Ġexamples . ĠInstead Ġan Ġexternal Ġknowledge Ġsource Ġis Ġemployed , Ġe . g . Ġa Ġmachine - readable Ġdictionary Ġor Ġword net . ĠIn Ġthe Ġsimplest Ġun super vised Ġsolution , Ġcalled Ġthe ĠLes k Ġalgorithm ĠB IB REF 8 Ġ, Ġmeanings Ġof Ġconsecutive Ġambiguous Ġwords Ġare Ġselected Ġby Ġfinding Ġthose Ġsenses Ġwhose Ġdefinitions Ġoverlap Ġthe Ġmost . </s> If Ġlack Ġof Ġdefinitions Ġmake Ġthe ĠLes k Ġalgorithm Ġinf eas ible , Ġwe Ġcan Ġexploit Ġrelations Ġbetween Ġwords . ĠThis Ġstudy Ġfocuses Ġon Ġmon ose m ous Ġrelatives , Ġi . e . Ġwords Ġor Ġcoll ocations , Ġselected Ġusing Ġword net , Ġbeing Ġrelated Ġto Ġa Ġdis amb ig uation Ġtarget , Ġbut Ġfree Ġof Ġambiguity . ĠOne Ġcan Ġeasily Ġfind Ġoccurrences Ġof Ġsuch Ġrelatives Ġin Ġan Ġun annot ated Ġtext Ġand Ġtreat Ġthem Ġas Ġtraining Ġexamples Ġfor Ġthe Ġtarget Ġambiguous Ġword . ĠThe Ġmethod Ġhas Ġbeen Ġsuccessfully Ġapplied Ġin Ġan ĠEnglish ĠW SD Ġtask ĠB IB REF 9 Ġ, Ġbut Ġstill Ġmany Ġproblems Ġremain . ĠOne Ġof Ġthem Ġis Ġchoice Ġof Ġrelatives ĠâĢĵ Ġin Ġfact , Ġeven Ġsyn onyms Ġdiffer Ġin Ġmeaning Ġand Ġusage Ġcontexts ; Ġand Ġthey Ġare Ġnot Ġavailable Ġfor Ġmany Ġwords . ĠThat Ġis Ġwhy Ġalso Ġhyper ny ms Ġand Ġhyp onyms , Ġespecially Ġmulti - word Ġexpressions Ġcontaining Ġthe Ġtarget Ġword , Ġare Ġtaken Ġinto Ġaccount . ĠSome Ġresearchers Ġalso Ġinclude Ġsiblings Ġ( i . e . Ġwords Ġwith Ġa Ġcommon Ġhyper ny m Ġwith Ġthe Ġtarget ) Ġand Ġan ton y ms , Ġbut Ġtheir Ġinfluence Ġis Ġnot Ġalways Ġbeneficiary ĠB IB REF 10 Ġ. ĠOther Ġinteresting Ġsources Ġof Ġmon ose m ous Ġrelatives Ġare Ġparts Ġof Ġdefinition ĠB IB REF 11 Ġ, Ġnamed Ġentities ĠB IB REF 12 Ġ, Ġindirect Ġhyp onyms Ġand Ġhyper ny ms , Ġand Ġfinally Ġmer onyms Ġand Ġhol onyms ĠB IB REF 10 Ġ. </s> The Ġmajority Ġof Ġclassification Ġtechniques Ġare Ġbuilt Ġon Ġan Ġassumption Ġthat Ġthe Ġtraining Ġdata Ġapproximately Ġreflects Ġthe Ġtrue Ġdistribution Ġof Ġthe Ġtarget Ġclasses . ĠHowever , Ġthat Ġis Ġnot Ġthe Ġcase Ġwhen Ġusing Ġmon ose m ous Ġrelatives . ĠThe Ġnumber Ġof Ġtheir Ġoccurrences Ġseldom Ġagrees Ġwith Ġthe Ġprobabilities Ġof Ġcorresponding Ġword Ġsenses . ĠQuite Ġoften Ġit Ġactually Ġis Ġthe Ġopposite : Ġobvious Ġand Ġfrequent Ġmeanings Ġhave Ġvery Ġfew Ġrelatives Ġand Ġvice Ġversa . ĠSome Ġresearchers Ġsimply Ġcopy Ġthe Ġa Ġprior i Ġprobabilities Ġfrom Ġtest Ġdata ĠB IB REF 9 Ġ, Ġothers Ġemploy Ġhe ur istics , Ġbut Ġthey Ġare Ġeasily Ġbeaten Ġby Ġstatistics Ġtaken Ġfrom Ġa Ġreal Ġannot ated Ġcorpus , Ġeven Ġdifferent Ġthan Ġtest Ġset ĠB IB REF 13 Ġ. </s> Prep aring Ġa Ġcorpus Ġfor Ġfinding Ġrelatives Ġposes Ġa Ġchallenge Ġas Ġwell . ĠIt Ġshould Ġcontain Ġa Ġlot Ġof Ġtext , Ġas Ġmany Ġmon ose m ous Ġwords Ġare Ġscarce . ĠSome Ġresearchers Ġuse Ġsnippets Ġretrieved Ġfrom Ġsearch Ġengines , Ġi . e . ĠAl ta V ista ĠB IB REF 11 Ġor ĠGoogle ĠB IB REF 13 Ġ. ĠOne Ġcan Ġalso Ġextend Ġa Ġsearch Ġquery Ġby Ġincluding Ġthe Ġcontext Ġof Ġthe Ġdis amb ig uated Ġword ĠB IB REF 14 Ġ, Ġbut Ġit Ġrequires Ġusing Ġas Ġmany Ġqueries Ġas Ġtest Ġcases . </s> Finally , Ġthe Ġusage Ġof Ġmon ose m ous Ġrelatives Ġhas Ġmore Ġapplications Ġthan Ġclassical ĠW SD . ĠOne Ġcan Ġuse Ġthem Ġto Ġgenerate Ġtopical Ġsignatures Ġfor Ġconcepts ĠB IB REF 15 Ġ, Ġautomatically Ġbuild Ġlarge Ġsense - tag ged Ġcorpor a ĠB IB REF 16 Ġand Ġevaluate Ġthe Ġquality Ġof Ġword net - related Ġsemantic Ġresources ĠB IB REF 17 Ġ. </s> Method </s> The Ġalgorithm Ġworks Ġas Ġfollows . ĠFirst , Ġa Ġset Ġof Ġrelatives Ġis Ġobtained Ġfor Ġeach Ġsense Ġof Ġa Ġtarget Ġword Ġusing Ġthe ĠPolish Ġword net : Ġpl Word Net ĠB IB REF 18 Ġ. ĠSome Ġof Ġthe Ġreplacements Ġmay Ġhave Ġmultiple Ġsenses , Ġhowever Ġusually Ġone Ġof Ġthem Ġcovers Ġmost Ġcases . ĠSecondly , Ġa Ġset Ġof Ġcontext Ġfeatures Ġis Ġextracted Ġfrom Ġoccurrences Ġof Ġrelatives Ġin Ġthe ĠNE K ST Ġcorpus . ĠFinally , Ġthe Ġaggreg ated Ġfeature Ġvalues Ġcorresponding Ġto Ġtarget Ġword Ġsenses Ġare Ġused Ġto Ġbuild Ġa ĠnaÃ¯ve ĠBay esian Ġclass ifier Ġadjusted Ġto Ġa Ġsituation Ġof Ġunknown Ġa Ġprior i Ġprobabilities . </s> Rel atives </s> In Ġorder Ġto Ġobtain Ġtraining Ġcases Ġfrom Ġun annot ated Ġcorpor a , Ġwe Ġaim Ġto Ġselect Ġrelatives Ġwhich Ġare Ġsem antically Ġsimilar Ġto Ġa Ġgiven Ġsense Ġof Ġa Ġtarget Ġword . ĠAn Ġexample Ġof Ġthis Ġprocess , Ġconcerning Ġthe Ġword Ġj Ä Ļ zyk Ġ( t ong ue ) Ġin Ġone Ġof Ġits Ġmeanings Ġ( human Ġor Ġanimal Ġorgan ) Ġis Ġshown Ġin ĠFigure ĠFIG REF 4 Ġ. ĠThis Ġstudy Ġtakes Ġinto Ġaccount Ġonly Ġsyn onyms , Ġhyper ny ms Ġand Ġhyp onyms , Ġas Ġother Ġoptions Ġ( s iblings , Ġan ton y ms , Ġhigher - order Ġrelatives ) Ġhave Ġpreviously Ġgiven Ġunsatisf actory Ġresults ĠB IB REF 10 Ġ. ĠInstead , Ġanother Ġproblem Ġdeserves Ġmore Ġattention : Ġhow Ġdo Ġwe Ġselect Ġthose Ġoccurrences Ġof Ġa Ġpoly sem ous Ġrelative Ġthat Ġcorrespond Ġto Ġa Ġtarget Ġword Ġsense ? ĠSo Ġfar , Ġthe Ġproblem Ġhas Ġbeen Ġcircumvent ed Ġby Ġincluding Ġonly Ġmon ose m ous Ġrelatives Ġ( nar z Ä ħ d Ġand Ġj Ä Ļ z or Ġin Ġthe Ġexample ), Ġwhich Ġgreatly Ġdecreases Ġtheir Ġavailability . ĠInstead , Ġwe Ġemploy Ġthose Ġrelatives , Ġwhose Ġfirst Ġmeaning Ġis Ġrelated Ġto Ġthe Ġconsidered Ġsense Ġ( arty k ulator Ġin Ġthe Ġexample ). ĠThe Ġintuition Ġis Ġthat Ġpl Word Net Ġusually Ġmentions Ġthe Ġmost Ġfrequent Ġmeaning Ġas Ġthe Ġfirst . </s> We Ġalso Ġexploit Ġpl Word Net Ġrelations Ġcalled Ġdetermin er , Ġwhich Ġlinks Ġnom inals Ġwith Ġadject ives Ġthat Ġare Ġfrequently Ġused Ġto Ġdescribe Ġthem . ĠFor Ġexample , Ġconsider Ġa Ġword Ġorgan Ġ( Eng . Ġorgan ). ĠAn Ġadjective Ġnat len ion y Ġ( oxy gen ated ) Ġis Ġa Ġdetermin er Ġof Ġone Ġof Ġthe Ġmeanings Ġof Ġorgan , Ġi . e . Ġpart Ġof Ġbody , Ġbut Ġnot Ġthe Ġothers , Ġi . e . Ġpart Ġof Ġan Ġinstitution . ĠTherefore , Ġthe Ġincluded Ġrelatives Ġconsist Ġof Ġa Ġpoly sem ous Ġrelated Ġword , Ġincluding Ġthe Ġtarget Ġword Ġitself , Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġa Ġmeaning Ġ( w yd ol ny Ġorgan Ġand Ġnat len ion y Ġorgan Ġin Ġthe Ġexample ). ĠThis Ġprocedure Ġis Ġperformed Ġonly Ġin Ġthe Ġcase Ġof Ġrelatives Ġthat Ġweren 't Ġincluded Ġso Ġfar , Ġi . e . Ġwith Ġa Ġsense Ġnumber Ġhigher Ġthan Ġ1 . </s> Finally , Ġwe Ġalso Ġmake Ġuse Ġof Ġa Ġwell - known Ġprinciple Ġcalled Ġone Ġword Ġper Ġdiscourse ĠB IB REF 19 Ġ, Ġwhich Ġstates Ġthat Ġa Ġpoly sem ous Ġword Ġis Ġvery Ġunlikely Ġto Ġtake Ġdifferent Ġmeanings Ġin Ġa Ġsingle Ġdocument . ĠIn Ġthis Ġstudy , Ġthe Ġprinciple Ġis Ġemployed Ġin Ġthe Ġfollowing Ġway : Ġif Ġin Ġa Ġsingle Ġdocument Ġthere Ġappear Ġonly Ġrelatives Ġcorresponding Ġto Ġa Ġsingle Ġtarget Ġword Ġsense , Ġthen Ġall Ġoccurrences Ġof Ġthe Ġtarget Ġword Ġin Ġthis Ġdocument Ġare Ġalso Ġtreated Ġas Ġtraining Ġexamples Ġfor Ġthis Ġmeaning . </s> One Ġcan Ġeasily Ġsee Ġthat Ġthese Ġassumptions Ġare Ġfalse Ġin Ġmany Ġcases , Ġwhich Ġmay Ġintroduce Ġnoise Ġand Ġdeterior ate Ġa Ġresulting Ġmodel . ĠThus , Ġthe Ġpresented Ġsolutions Ġundergo Ġexperimental Ġvalidation Ġusing Ġthe Ġfollowing Ġsets Ġof Ġrelatives : </s> Mon ose m ous Ġchildren ĠâĢĵ Ġmon ose m ous Ġdirect Ġhyp onyms . </s> Mon ose m ous Ġrelatives ĠâĢĵ Ġmon ose m ous Ġsyn onyms , Ġdirect Ġhyp onyms Ġand Ġdirect Ġhyper ny ms . </s> First Ġrelatives ĠâĢĵ Ġwords Ġin Ġthe Ġfirst Ġmeaning Ġbelonging Ġto Ġsy ns ets Ġof Ġsyn onyms , Ġdirect Ġhyp onyms Ġor Ġdirect Ġhyper ny ms . </s> Word Ġdetermin ers ĠâĢĵ Ġcoll ocations Ġmade Ġof Ġtwo Ġwords Ġin Ġany Ġorder : Ġthe Ġtarget Ġword Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġa Ġgiven Ġmeaning . </s> All Ġdetermin ers ĠâĢĵ Ġcoll ocations Ġmade Ġof Ġtwo Ġwords Ġin Ġany Ġorder : Ġa Ġpoly sem ous Ġrelative Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġthe Ġappropriate Ġmeaning . </s> Other Ġwords ĠâĢĵ Ġoccurrences Ġof Ġthe Ġtarget Ġambiguous Ġword Ġin Ġa Ġdocument Ġthat Ġcontains Ġother Ġrelatives Ġcorresponding Ġto Ġexactly Ġone Ġof Ġthe Ġmeanings . </s> Table ĠT AB REF 23 Ġshows Ġhow Ġmany Ġrelatives Ġhave Ġbeen Ġobtained Ġfor Ġeach Ġcategory , Ġas Ġwell Ġas Ġthe Ġnumber Ġof Ġoccurrences Ġin Ġthe Ġcorpus Ġof Ġ6 Ġmillion Ġdocuments Ġ( see Ġnext Ġsection ). </s> Corp us </s> As Ġsome Ġof Ġthe Ġrelatives Ġmay Ġbe Ġvery Ġrare , Ġit Ġis Ġimportant Ġto Ġuse Ġa Ġtraining Ġcorpus Ġof Ġsignificant Ġsize . ĠIn Ġthis Ġcase , Ġwe Ġused Ġ600 Ġmillion Ġweb pages Ġ( 594 Ġbillion Ġtokens ) Ġindexed Ġby Ġa ĠPolish Ġsearch Ġengine ĠNE K ST , Ġdeveloped Ġat Ġthe ĠInstitute Ġof ĠComputer ĠScience , ĠPolish ĠAcademy Ġof ĠSciences . ĠTraining Ġsub - cor pora Ġwere Ġselected Ġwith Ġsize Ġvarying Ġfrom Ġ19 , 000 Ġto Ġ60 Ġmillion Ġdocuments . ĠInstead Ġof Ġusing Ġsnippets Ġreturned Ġby Ġa Ġsearch Ġinterface , Ġwe Ġuse Ġwhole Ġtextual Ġcontents Ġ( with Ġmorph os ynt actic Ġannotation ) Ġof Ġeach Ġdocument , Ġtaken Ġfrom ĠNE K ST Ġdistributed Ġinfrastructure . </s> Unfortunately , Ġa Ġlot Ġof Ġtext Ġon Ġweb Ġpages Ġis Ġnot Ġsuitable Ġfor Ġtraining Ġa ĠW SD Ġclass ifier , Ġfor Ġexample Ġelements Ġof Ġpage Ġstructure Ġor Ġparts Ġunrecogn ised Ġby Ġa Ġtag ger . ĠThus , Ġeach Ġsentence Ġhas Ġto Ġsatisfy Ġthe Ġfollowing Ġrequirements Ġto Ġqualify Ġfor Ġtraining : </s> be Ġat Ġleast Ġ150 - character Ġlong . </s> cont ain Ġat Ġleast Ġfive Ġwords . </s> cont ain Ġat Ġleast Ġfour Ġdifferent Ġparts Ġof Ġspeech Ġ( including Ġpunct uation ). </s> These Ġcriteria Ġhelp Ġto Ġfilter Ġout Ġmost Ġof Ġthe Ġweb Ġcontent Ġof Ġunsatisf actory Ġquality . </s> Features </s> Context Ġfeatures Ġextracted Ġfor Ġclassification Ġare Ġvery Ġsimilar Ġto Ġthose Ġthat Ġhave Ġproven Ġsuccessful Ġin Ġsupervised ĠW SD , Ġincluding Ġexperiments Ġon Ġthe Ġsame Ġevaluation Ġset ĠB IB REF 20 Ġ: </s> words Ġpresent Ġat Ġcertain Ġpositions Ġin Ġa Ġneighbourhood Ġof Ġa Ġtarget Ġword : </s> le mm as Ġat Ġpositions : Ġ- 2 , Ġ- 1 , Ġ1 , Ġ2 Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ), </s> morph os ynt actic Ġinterpretations Ġ( sequ ences Ġof Ġtags ) Ġat Ġpositions : Ġ- 1 , Ġ1 Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ) Ġand Ġ0 Ġ( den oted Ġby ĠIN LINE FORM 1 Ġ), </s> le mm as Ġpresent Ġin Ġthe Ġsentence Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ). </s> Note Ġthat Ġthe Ġmorph os ynt actic Ġinterpretations Ġare Ġassigned Ġto Ġsingle Ġwords Ġonly , Ġtherefore Ġin Ġcase Ġof Ġmulti - word Ġrelatives ĠIN LINE FORM 0 Ġis Ġnot Ġavailable . ĠAlso , Ġa Ġgender Ġtag Ġis Ġremoved Ġfrom ĠIN LINE FORM 1 Ġ. </s> Class ification </s> After Ġgathering Ġthe Ġvalues Ġof Ġfeatures Ġfrom Ġoccurrences Ġof Ġrelatives , Ġa ĠnaÃ¯ve ĠBay esian Ġclassification Ġmodel Ġis Ġbuilt . ĠHowever , Ġas Ġmany Ġresearchers Ġin Ġthe Ġfield Ġhave Ġnoticed ĠB IB REF 13 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, Ġthere Ġis Ġa Ġmajor Ġdrawback Ġof Ġusing Ġrelatives Ġinstead Ġof Ġa Ġtarget Ġword : Ġthe Ġnumber Ġof Ġoccurrences Ġof Ġrelatives Ġis Ġusually Ġnot Ġproportional Ġto Ġthe Ġfrequency Ġof Ġthe Ġtarget Ġword Ġsense . ĠTo Ġbypass Ġthis Ġproblem , Ġwe Ġmodify Ġthe ĠBay esian Ġmodel Ġin Ġthis Ġstudy . ĠThe Ġbasic Ġclass ifier Ġchooses Ġthe Ġclass Ġthat Ġmaxim ises Ġthe Ġfollowing Ġprobability : ĠIN LINE FORM 0 </s> In Ġthe Ġcase Ġof Ġbinary Ġfeatures , Ġwhich Ġrepresent Ġthe Ġoccurrence Ġof Ġa Ġcertain Ġword Ġin Ġcontext , Ġwe Ġhave ĠIN LINE FORM 0 Ġand : ĠIN LINE FORM 1 </s> Which Ġcould Ġbe Ġrewritten Ġas : ĠIN LINE FORM 0 </s> The Ġexpression Ġhas Ġbeen Ġformulated Ġas Ġa Ġproduct Ġof Ġtwo Ġfactors : ĠIN LINE FORM 0 Ġ, Ġindependent Ġfrom Ġobserved Ġfeatures Ġand Ġcorresponding Ġto Ġempty Ġword Ġcontext , Ġand ĠIN LINE FORM 1 Ġthat Ġdepends Ġon Ġobserved Ġcontext . ĠTo Ġweaken Ġthe Ġinfluence Ġof Ġimproper Ġdistribution Ġof Ġtraining Ġcases , Ġwe Ġomit ĠIN LINE FORM 2 Ġ, Ġso Ġthat Ġwhen Ġno Ġcontext Ġfeatures Ġare Ġobserved , Ġevery Ġword Ġsense Ġis Ġconsidered Ġequally Ġprobable . </s> Thus , Ġfor Ġthe Ġgiven Ġcontext Ġfeatures ĠIN LINE FORM 0 Ġ, Ġsense ĠIN LINE FORM 1 Ġis Ġselected Ġwhen : ĠIN LINE FORM 2 </s> The Ġtable Ġwith Ġfinal Ġresults Ġ( ĠT AB REF 27 Ġ) Ġcontains Ġaccur acies Ġof Ġboth Ġoriginal Ġand Ġmodified Ġversions Ġof Ġthe Ġclass ifier . </s> E val uation </s> For Ġexperiments Ġand Ġevaluation Ġa Ġsub - cor p us Ġof Ġthe ĠNational ĠCorpus Ġof ĠPolish , ĠN CP ĠB IB REF 6 Ġwas Ġemployed . ĠThe Ġmanually Ġannot ated Ġsub - cor p us Ġcontains Ġsense Ġlabels Ġof Ġ106 Ġdifferent Ġwords : Ġ50 Ġnoun s , Ġ48 Ġverbs Ġand Ġ8 Ġadject ives . ĠAs Ġverbs Ġhave Ġmuch Ġpoorer Ġconnectivity Ġin Ġpl Word Net , Ġthey Ġhave Ġbeen Ġignored Ġwithin Ġthis Ġstudy . </s> The Ġsenses Ġused Ġfor Ġannotation Ġare Ġcoarse - gr ained ĠâĢĵ Ġwith Ġone Ġsense Ġcovering Ġa Ġrange Ġof Ġrelated Ġus ages . ĠEach Ġword Ġhas Ġbetween Ġtwo Ġand Ġfour Ġsenses . ĠTo Ġemploy Ġthe Ġmethod Ġdescribed Ġin Ġprevious Ġsection , Ġthe ĠN CP Ġsenses Ġhave Ġbeen Ġmanually Ġmapped Ġto Ġfine - gr ained Ġpl Word Net Ġsy ns ets . ĠAs ĠN CP Ġsenses Ġhave Ġbeen Ġcreated Ġindependently Ġfrom Ġword net Ġsenses , Ġa Ġsubstantial Ġpart Ġof Ġthe Ġlatter Ġremain Ġuncovered Ġby Ġthe Ġformer . ĠHowever , Ġwe Ġonly Ġfound Ġfour Ġcases Ġwhere Ġan ĠN CP Ġsense Ġhas Ġno Ġcounterpart Ġin Ġword net ; Ġthose Ġwords Ġare Ġexcluded Ġfrom Ġthe Ġtest Ġset . </s> In Ġtotal , Ġthe Ġtest Ġset Ġincludes Ġ17 , 314 Ġoccurrences Ġof Ġ54 Ġambiguous Ġwords , Ġhaving Ġtwo Ġto Ġfour Ġcoarse - gr ained Ġmeanings . ĠTable ĠT AB REF 26 Ġcontains Ġa Ġdetailed Ġsummary . </s> The Ġalgorithm Ġworks Ġusing Ġpl Word Net Ġsy ns ets Ġand Ġits Ġoutput Ġis Ġmapped Ġto ĠN CP Ġmeaning Ġto Ġmeasure Ġits Ġquality . ĠAccuracy Ġmeasures Ġwhat Ġpercentage Ġof Ġthe Ġprogramme 's Ġguesses Ġagree Ġwith Ġmanually Ġassigned Ġsenses . ĠTo Ġassess Ġthe Ġgeneral Ġperformance Ġof Ġa Ġparticular Ġconfiguration , Ġthe Ġaccuracy Ġhas Ġbeen Ġaveraged Ġover Ġall Ġtarget Ġwords . </s> To Ġproperly Ġjudge Ġthe Ġresults , Ġwe Ġneed Ġto Ġstart Ġwith Ġthe Ġbas elines . ĠWithout Ġknowing Ġthe Ġdistribution Ġof Ġsenses , Ġthree Ġbasic Ġpossibilities Ġseem Ġreasonable : Ġwe Ġcan Ġeither Ġ1 ) Ġselect Ġa Ġmeaning Ġrandomly , Ġ2 ) Ġbase Ġon Ġthe Ġsense Ġnumbering Ġin ĠN CP Ġor Ġ3 ) Ġuse Ġpl Word Net Ġin Ġthe Ġsame Ġway . ĠTo Ġhave Ġa Ġbetter Ġcomparison Ġwith Ġont ology - based Ġmethods , Ġthe Ġresults Ġalso Ġinclude Ġa Ġword Ġsimilarity Ġbaseline Ġconfiguration , Ġwhich Ġselects Ġthe Ġsense Ġwith Ġthe Ġstrongest Ġsimilarity Ġto Ġany Ġof Ġthe Ġwords Ġin Ġcontext Ġ( sent ence ). ĠFor Ġthat Ġpurpose Ġthe ĠLe ac ock & Ch odor ow Ġsimilarity Ġmeasure Ġ( im ple mented Ġusing Ġall Ġrelations Ġbetween Ġsy ns ets Ġin Ġpl Word Net ) Ġis Ġemployed , Ġas Ġit Ġhas Ġbeen Ġpreviously Ġused Ġin ĠW SD ĠB IB REF 21 Ġand Ġalso Ġcorrelates Ġwell Ġwith Ġhuman Ġjudgement Ġof Ġsimilarity ĠB IB REF 22 Ġ. ĠThe Ġbaseline Ġresults , Ġshown Ġin ĠTable ĠT AB REF 27 Ġ, Ġsupport Ġthe Ġclaim Ġof Ġintentional Ġsense Ġordering Ġin Ġpl Word Net . </s> Results </s> The Ġgoal Ġof Ġthe Ġfirst Ġexperiment Ġwas Ġto Ġselect Ġan Ġoptimal Ġfeature Ġset Ġfor Ġthis Ġtask . ĠSeveral Ġmodels Ġwith Ġa Ġcommon Ġbasic Ġconfiguration , Ġi . e . Ġusing Ġall Ġpossible Ġrelatives Ġand Ġ6 Ġmillion Ġdocuments , Ġhave Ġbeen Ġbuilt Ġwith Ġdifferent Ġfeature Ġsets Ġand Ġevaluated . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 22 Ġ. ĠAs Ġwe Ġcan Ġsee , Ġthe Ġlex ical Ġfeatures Ġgive Ġus Ġmore Ġpredictive Ġpower Ġthan Ġmorph ological Ġinterpretations . ĠThe Ġbest Ġsolution , Ġincorporated Ġinto Ġthe Ġbasic Ġconfiguration Ġfor Ġfurther Ġexperiments , Ġincludes Ġall Ġfeatures Ġexcept Ġthese Ġbased Ġon Ġthe Ġinterpretation Ġof Ġthe Ġword Ġin Ġfocus . </s> Secondly , Ġit Ġis Ġnecessary Ġto Ġevaluate Ġdifferent Ġtypes Ġof Ġreplacements , Ġoutlined Ġin Ġsection ĠSEC REF 2 Ġ. ĠTable ĠT AB REF 23 Ġcontains Ġthe Ġaverage Ġnumber Ġof Ġpossible Ġreplacements Ġper Ġtarget Ġword , Ġthe Ġnumber Ġof Ġoccurrences Ġin Ġthe Ġsix - million Ġcorpus Ġand Ġthe Ġaverage Ġclassification Ġaccuracy . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġnumber Ġof Ġreplacements Ġrises Ġafter Ġadding Ġsubsequent Ġsources , Ġbut Ġthe Ġlargest Ġincrease Ġis Ġcaused Ġby Ġincluding Ġpoly sem ous Ġrelatives Ġwith Ġdetermin ers . ĠOn Ġthe Ġother Ġhand , Ġthese Ġcompound Ġrelatives Ġrarely Ġappear Ġin Ġthe Ġcorpus Ġ( 13 , 126 Ġocc ure nces ), Ġwhereas Ġemploying Ġpoly sem ous Ġwords Ġin Ġthe Ġfirst Ġsense Ġresults Ġin Ġ1 , 525 , 7 39 Ġnew Ġtraining Ġcases Ġand Ġa Ġsubstantial Ġgrowth Ġof Ġaccuracy . ĠWhat Ġis Ġmore , Ġalthough Ġthe Ġprofits Ġfrom Ġthese Ġsources Ġof Ġrelatives Ġdiffer , Ġnone Ġof Ġthem Ġdecreases Ġthe Ġaccuracy . </s> The Ġavailability Ġof Ġthe Ġcorpus Ġcontaining Ġ600 Ġmillion Ġdocuments Ġhelps Ġto Ġanswer Ġthe Ġquestion Ġof Ġsufficient Ġcorpus Ġsize Ġfor Ġsuch Ġtask . ĠFigure ĠFIG REF 24 Ġshows Ġmean Ġclassification Ġaccuracy Ġfor Ġmodels Ġbuilt Ġusing Ġdifferent Ġtraining Ġcorpor a , Ġwhich Ġhave Ġbeen Ġcreated Ġby Ġrandomly Ġselecting Ġa Ġsubset Ġof Ġthe Ġoriginal Ġdocument Ġset . ĠThe Ġconsidered Ġsizes Ġare Ġbetween Ġ19 , 000 Ġand Ġ60 , 000 , 000 . ĠAdditionally , Ġa Ġdifferent Ġtype Ġof Ġcorpor a Ġhas Ġbeen Ġcreated Ġby Ġusing Ġonly Ġdocuments Ġfrom Ġthe ĠPolish ĠWikipedia Ġ( s izes Ġ11 , 000 ĠâĢĵ Ġ1 , 09 8 , 000 ). ĠWe Ġsee Ġthat Ġafter Ġa Ġcertain Ġpoint Ġadding Ġin Ġnew Ġdata Ġdoes Ġnot Ġimprove Ġthe Ġaccuracy . ĠSurprisingly , Ġthe Ġsub cor pora Ġof ĠWikipedia Ġyield Ġworse Ġresults Ġthan Ġthose Ġof Ġunlimited Ġorigin . </s> Table ĠT AB REF 26 Ġshows Ġthe Ġaccuracy Ġof Ġdis amb ig uation Ġin Ġthe Ġconfiguration Ġoutlined Ġabove Ġwith Ġrespect Ġto Ġthe Ġtarget Ġword . ĠThe Ġeasiest Ġwords Ġhave Ġmeanings Ġcorresponding Ġto Ġdistinct Ġphysical Ġobjects , Ġe . g . Ġin ĠPolish Ġpi ÅĤ ka Ġ( 100 % Ġaccuracy ) Ġmay Ġmean Ġa Ġball Ġor Ġa Ġsmall Ġsaw . ĠThe Ġhardest Ġcases Ġare Ġthose Ġwith Ġmany Ġabstract Ġand Ġfuzzy Ġmeanings , Ġe . g . Ġin ĠPolish Ġk l asa Ġhas Ġfour Ġmeanings Ġrelated Ġto ĠEnglish Ġclass : Ġ( 1 ) Ġgroup Ġof Ġsimilar Ġobjects , Ġ( 2 ) Ġlevel Ġof Ġquality , Ġ( 3 ) Ġgroup Ġof Ġpupils Ġor Ġ( 4 ) Ġclassroom . ĠThe Ġmeaning Ġ( 1 ) Ġcould Ġbe Ġhard Ġto Ġdistinguish Ġfrom Ġ( 2 ) Ġeven Ġfor Ġa Ġhuman , Ġwhereas Ġ( 3 ) Ġand Ġ( 4 ) Ġmay Ġappear Ġin Ġvery Ġsimilar Ġcontexts . </s> Finally , ĠTable ĠT AB REF 27 Ġcontains Ġthe Ġmean Ġaccuracy Ġof Ġthe Ġbasic Ġconfiguration Ġof Ġthe Ġclass ifier Ġdescribed Ġin Ġthis Ġwork Ġ( with Ġand Ġwithout Ġmodifications Ġto ĠBay esian Ġmodel ). ĠIt Ġis Ġcompared Ġto Ġthe Ġfour Ġpreviously Ġmentioned Ġbas elines . </s> Discussion </s> Although Ġmany Ġdifferent Ġconfigurations Ġhave Ġbeen Ġtested Ġin Ġthis Ġstudy , Ġall Ġof Ġthem Ġremain Ġbelow Ġthe Ġaccuracy Ġlevel Ġof Ġ80 %, Ġapproximately Ġequal Ġto Ġaverage Ġshare Ġof Ġdominating Ġsenses Ġin Ġthis Ġdataset . ĠThis Ġis Ġobviously Ġunsatisf actory Ġand Ġdemands Ġexplanation . </s> First Ġof Ġall , Ġthe Ġnew Ġsources Ġof Ġreplacements Ġproposed Ġin Ġthis Ġwork Ġindeed Ġseem Ġto Ġimprove Ġthe Ġmodels Ġfrom Ġ70 . 86 % Ġ( only Ġtraditional Ġmon ose m ous Ġrelatives ) Ġto Ġ77 . 96 % Ġ( all Ġproposed Ġrelatives ). ĠThe Ġbiggest Ġgain Ġis Ġobtained Ġby Ġincluding Ġthe Ġpoly sem ous Ġrelatives Ġtaking Ġinto Ġaccount Ġonly Ġtheir Ġfirst Ġmeaning . ĠThis Ġtechnique Ġrelies Ġon Ġtwo Ġassumptions : Ġa Ġstrong Ġdomination Ġof Ġone Ġof Ġthe Ġsenses Ġand Ġthat Ġsense Ġbeing Ġlisted Ġfirst Ġin Ġpl Word Net . ĠWhile Ġthe Ġformer Ġis Ġalmost Ġalways Ġtrue , Ġif Ġthe Ġsecond Ġassumption Ġis Ġfalse Ġthen Ġthe Ġcreated Ġmodel Ġare Ġadversely Ġaffected . ĠIn Ġthe Ġcase Ġof Ġtwo Ġtarget Ġwords Ġthe Ġsenses , Ġthe Ġfirst Ġsense Ġin Ġeach Ġcase Ġ( stop ie Å Ħ Ġas Ġa Ġmusical Ġconcept Ġand Ġform a Ġas Ġa Ġsyn onym Ġof Ġpo lyn omial ) Ġwas Ġso Ġpeculiar Ġthat Ġthey Ġwere Ġunknown Ġto Ġthe Ġauthor Ġof Ġthis Ġstudy Ġand Ġcouldn 't Ġbe Ġassigned Ġto Ġany Ġof Ġthe Ġcoarse - gr ained ĠN CP Ġsenses . ĠClearly , Ġnot Ġonly Ġthe Ġmethod Ġof Ġun super vised ĠW SD Ġusing Ġrelatives , Ġbut Ġalso Ġother Ġsolutions Ġrelated Ġto Ġpoly se my Ġwould Ġdefinitely Ġbenefit Ġfrom Ġa Ġreliable Ġordering Ġof Ġsenses Ġin Ġword nets , Ġespecially Ġas Ġincreasingly Ġuncommon Ġsenses Ġare Ġadded Ġto Ġthem Ġwith Ġtime . ĠIt Ġis Ġhowever Ġnot Ġclear Ġhow Ġsuch Ġknowledge Ġcould Ġbe Ġobtained Ġwithout Ġsolving Ġthe ĠW SD Ġtask Ġfirst . ĠWhat Ġis Ġmore , Ġsense Ġdistributions Ġobviously Ġchange Ġwith Ġgenre , Ġtime , Ġauthor , Ġetc . </s> When Ġit Ġcomes Ġto Ġfeature Ġselection , Ġthe Ġmost Ġunexpected Ġphenomenon Ġobserved Ġin Ġthis Ġstudy Ġis Ġlow Ġusefulness Ġof Ġthe Ġinterpretation - based Ġfeatures . ĠAccording Ġto ĠTable ĠT AB REF 22 Ġ, Ġadding Ġinterpretations Ġof Ġneighbouring Ġwords Ġ( ĠIN LINE FORM 0 Ġ) Ġyields Ġvery Ġlittle Ġimprovement , Ġwhile Ġthis Ġtype Ġof Ġinformation Ġregarding Ġreplacements Ġ( ĠIN LINE FORM 1 Ġ) Ġeven Ġlowers Ġthe Ġaccuracy . ĠThis Ġresult Ġcould Ġbe Ġattributed Ġto Ġtwo Ġfactors . ĠFirstly , Ġmore Ġdeveloped Ġreplacement Ġgeneration Ġresults Ġin Ġmore Ġoccurrences , Ġbut Ġalso Ġcauses Ġtheir Ġtags Ġto Ġdiffer Ġfrom Ġthe Ġtarget Ġword Ġby Ġgender Ġor Ġnumber . ĠThey Ġmay Ġeven Ġnot Ġbe Ġavailable Ġat Ġall Ġ( in Ġthe Ġcase Ġof Ġmulti - word Ġreplacements ). ĠThe Ġsecond Ġreason Ġis Ġa Ġdifference Ġin Ġlanguage : Ġwhile Ġin ĠEnglish Ġa Ġword Ġinterpretation Ġis Ġrepresented Ġas Ġone Ġof Ġseveral Ġdozen Ġpart Ġof Ġspeech Ġidentifiers , Ġin ĠSlav onic Ġlanguages , Ġsuch Ġas ĠPolish , Ġwe Ġneed Ġto Ġspecify Ġthe Ġvalues Ġof Ġseveral Ġtags Ġfor Ġeach Ġword , Ġleading Ġto Ġthousands Ġof Ġpossible Ġinterpretations . ĠObviously , Ġthe Ġfeatures Ġbased Ġon Ġthese Ġtags Ġare Ġvery Ġsparse . ĠFinally , Ġthe Ġmorph os ynt actic Ġannotation Ġwas Ġperformed Ġautomatically , Ġwhich Ġmay Ġlead Ġto Ġerrors , Ġespecially Ġin Ġthe Ġcase Ġof Ġnoisy Ġweb Ġtext . </s> One Ġof Ġthe Ġpurposes Ġof Ġthis Ġstudy Ġwas Ġto Ġcheck Ġthe Ġnecessary Ġamount Ġof Ġtraining Ġdata Ġfor Ġsuch Ġa Ġsolution Ġby Ġemploying Ġa Ġvery Ġlarge Ġcollection Ġfrom Ġthe ĠNE K ST Ġsearch Ġengine . ĠThe Ġneed Ġfor Ġlarge Ġcorpor a Ġis Ġobvious Ġwhen Ġusing Ġonly Ġmon ose m ous Ġrelatives ĠâĢĵ Ġthose Ġusually Ġrare Ġwords Ġshould Ġappear Ġin Ġmany Ġcontexts . ĠHowever , Ġaccording Ġto Ġthe Ġresults Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ, Ġthe Ġstrategy Ġfor Ġgenerating Ġrelatives Ġpresented Ġin Ġthis Ġpaper Ġreaches Ġoptimum Ġperformance Ġfor Ġa Ġreasonable Ġamount Ġof Ġtexts ĠâĢĵ Ġ6 Ġmillion Ġdocuments Ġis Ġenough . ĠHowever , Ġone Ġshould Ġkeep Ġin Ġmind Ġthat Ġthis Ġstatement Ġremains Ġtrue Ġassuming Ġa Ġconstant Ġevaluation Ġenvironment ; Ġexpanding Ġa Ġtest Ġset Ġ( currently Ġcontaining Ġ17 , 314 Ġoccurrences ) Ġmay Ġhelp Ġto Ġsee Ġdifferences Ġbetween Ġapparently Ġequivalent Ġmodels Ġand Ġraise Ġthe Ġneed Ġfor Ġbigger Ġcorpor a . </s> Con clusions </s> In Ġthis Ġpaper Ġthe Ġlimitations Ġand Ġimprovements Ġof Ġun super vised Ġword Ġsense Ġdis amb ig uation Ġhave Ġbeen Ġinvestigated . ĠThe Ġmain Ġproblem ĠâĢĵ Ġinsufficient Ġnumber Ġand Ġquality Ġof Ġreplacements Ġhas Ġbeen Ġtackled Ġby Ġadding Ġnew Ġrich Ġsources Ġof Ġreplacements . ĠThe Ġquality Ġof Ġthe Ġmodels Ġhas Ġindeed Ġimproved , Ġespecially Ġthanks Ġto Ġreplacements Ġbased Ġon Ġsense Ġordering Ġin Ġpl Word Net . ĠTo Ġdeal Ġwith Ġthe Ġproblem Ġof Ġunknown Ġsense Ġdistribution , Ġthe ĠBay esian Ġclass ifier Ġhas Ġbeen Ġmodified , Ġremoving Ġthe Ġbias Ġtowards Ġfrequent Ġlabels Ġin Ġthe Ġtraining Ġdata . ĠFinally , Ġthe Ġexperiments Ġwith Ġvery Ġlarge Ġcorpus Ġhave Ġshown Ġthe Ġsufficient Ġamount Ġof Ġtraining Ġdata Ġfor Ġthis Ġtask , Ġwhich Ġis Ġonly Ġ6 Ġmillion Ġdocuments . </s> Acknowled gements </s> This Ġstudy Ġwas Ġconducted Ġat ĠInstitute Ġof ĠComputer ĠScience , ĠPolish ĠAcademy Ġof ĠSciences , Ġand Ġsupported Ġby Ġa Ġresearch Ġfellowship Ġwithin Ġ" Information Ġtechnologies : Ġresearch Ġand Ġtheir Ġinter disciplinary Ġapplications " Ġagreement Ġnumber ĠP OK L . 04 . 01 . 01 - 00 - 05 1 / 10 - 00 . ĠThe Ġauthor Ġwould Ġlike Ġto Ġthank ĠDarius z ĠC zers ki Ġand ĠNE K ST Ġteam Ġfor Ġproviding Ġaccess Ġto Ġthe Ġsearch Ġengine Ġindex Ġand Ġhelpful Ġdiscussions Ġand ĠMatthew ĠShard low Ġfor Ġcomments Ġthat Ġgreatly Ġimproved Ġthe Ġmanuscript .
Introduction The focus of the word sense disambiguation (WSD) task is polysemy, i.e. words having several substantially different meanings. Two common examples are bank (riverside or financial institution) and bass (fish or musical instrument), but usually the meanings of a word are closely related, e.g. class may refer to: (a) a group of students, (b) the period when they meet to study or (c) a room where such meetings occur. Readers deal with this problem by using a word's context and in WSD we aim at doing it automatically. The most effective solution, called supervised WSD, is to use a large number of sense-annotated occurrences of the target word to build a machine learning model to label test cases. However, this approach suffers from a knowledge acquisition bottleneck. The annotation of a separate training corpus for every target word demands a considerable amount of human labour. Therefore, this approach is unusable in applications that require WSD across a wide vocabulary, such as open-domain question answering BIBREF0 . The method of monosemous relatives, which is the focus of this work, bypasses the bottleneck by gathering occurences of words related to the target word, but free from ambiguity, and treating them as training cases of the respective senses. Human labour is eliminated at the expense of accuracy, as the context of each relative only approximately matches the context of the target word sense. Monosemous relatives have been employed multiple times (see Section 2), but results remain unsatisfactory. The aim of my study is to explore the limitations of this technique by implementing and evaluating such a tool for Polish. Firstly, the method is expanded by waiving the requirement of monosemy and proposing several new sources of relatives. These previously unexplored sources are based on wordnet data and help gather many training cases from the corpus. Secondly, a well-known problem of uneven yet unknown distribution of word senses is alleviated by modifying a naïve Bayesian classifier. Thanks to this correction, the classifier is no longer biased towards senses that have more training data. Finally, a very large corpus (600 million documents), gathered from the web by a Polish search engine NEKST, is used to build models based on training corpora of different sizes. Those experiments show what amount of data is sufficient for such a task. The proposed solution is compared to baselines that use wordnet structure only, with no training corpora. This paper is organised as follows. The next section reviews the previous research in the area, focusing on unsupervised WSD using monosemous relatives. Section 3 outlines the proposed solution by describing the new sources of relatives, the employed corpus, the features extracted from context and the modified Bayesian classifier. Section 4 describes the evaluation data and process, while section 5 quotes the results. Section 6 is devoted to discussing the outcomes and section 7 concludes the paper. Related work The problem of WSD has received a lot of attention since the beginning of natural language processing research. WSD is typically expected to improve the results of real-world applications: originally machine translation and recently information retrieval and extraction, especially question answering BIBREF0 . Like many other areas, WSD has greatly benefited from publicly available test sets and competitions. Two notable corpora are: 1) SemCor BIBREF1 , built by labelling a subset of Brown corpus with Princeton WordNet synsets and 2) the public evaluations of Senseval workshops BIBREF2 , BIBREF3 . There are a variety of approaches to solve the WSD problem, which can be grouped based upon how they use their data – see reviews BIBREF4 , BIBREF5 . In supervised solutions a large sense-tagged corpus is available for training. This approach has been applied to the the test set used in the current study, resulting in an accuracy value of 91.5% BIBREF6 . Although this technique undoubtedly yields the best results, we would need an immense amount of human labour to build a training corpus of sufficient size for disambiguating all words. This does not seem possible, especially in the case of languages, such as Polish, which receive less attention than English. In the minimally supervised approach BIBREF7 , a small set of initial training examples, obtained by a heuristic or hand-tagging, is used to label new occurrences. They in turn serve as a training set for next iteration, and so on. This bootstrapping procedure requires very little manual tagging but needs to be carefully implemented to avoid loosing accuracy in further steps. Unsupervised methods use no previously labelled examples. Instead an external knowledge source is employed, e.g. a machine-readable dictionary or wordnet. In the simplest unsupervised solution, called the Lesk algorithm BIBREF8 , meanings of consecutive ambiguous words are selected by finding those senses whose definitions overlap the most. If lack of definitions make the Lesk algorithm infeasible, we can exploit relations between words. This study focuses on monosemous relatives, i.e. words or collocations, selected using wordnet, being related to a disambiguation target, but free of ambiguity. One can easily find occurrences of such relatives in an unannotated text and treat them as training examples for the target ambiguous word. The method has been successfully applied in an English WSD task BIBREF9 , but still many problems remain. One of them is choice of relatives – in fact, even synonyms differ in meaning and usage contexts; and they are not available for many words. That is why also hypernyms and hyponyms, especially multi-word expressions containing the target word, are taken into account. Some researchers also include siblings (i.e. words with a common hypernym with the target) and antonyms, but their influence is not always beneficiary BIBREF10 . Other interesting sources of monosemous relatives are parts of definition BIBREF11 , named entities BIBREF12 , indirect hyponyms and hypernyms, and finally meronyms and holonyms BIBREF10 . The majority of classification techniques are built on an assumption that the training data approximately reflects the true distribution of the target classes. However, that is not the case when using monosemous relatives. The number of their occurrences seldom agrees with the probabilities of corresponding word senses. Quite often it actually is the opposite: obvious and frequent meanings have very few relatives and vice versa. Some researchers simply copy the a priori probabilities from test data BIBREF9 , others employ heuristics, but they are easily beaten by statistics taken from a real annotated corpus, even different than test set BIBREF13 . Preparing a corpus for finding relatives poses a challenge as well. It should contain a lot of text, as many monosemous words are scarce. Some researchers use snippets retrieved from search engines, i.e. AltaVista BIBREF11 or Google BIBREF13 . One can also extend a search query by including the context of the disambiguated word BIBREF14 , but it requires using as many queries as test cases. Finally, the usage of monosemous relatives has more applications than classical WSD. One can use them to generate topical signatures for concepts BIBREF15 , automatically build large sense-tagged corpora BIBREF16 and evaluate the quality of wordnet-related semantic resources BIBREF17 . Method The algorithm works as follows. First, a set of relatives is obtained for each sense of a target word using the Polish wordnet: plWordNet BIBREF18 . Some of the replacements may have multiple senses, however usually one of them covers most cases. Secondly, a set of context features is extracted from occurrences of relatives in the NEKST corpus. Finally, the aggregated feature values corresponding to target word senses are used to build a naïve Bayesian classifier adjusted to a situation of unknown a priori probabilities. Relatives In order to obtain training cases from unannotated corpora, we aim to select relatives which are semantically similar to a given sense of a target word. An example of this process, concerning the word język (tongue) in one of its meanings (human or animal organ) is shown in Figure FIGREF4 . This study takes into account only synonyms, hypernyms and hyponyms, as other options (siblings, antonyms, higher-order relatives) have previously given unsatisfactory results BIBREF10 . Instead, another problem deserves more attention: how do we select those occurrences of a polysemous relative that correspond to a target word sense? So far, the problem has been circumvented by including only monosemous relatives (narząd and jęzor in the example), which greatly decreases their availability. Instead, we employ those relatives, whose first meaning is related to the considered sense (artykulator in the example). The intuition is that plWordNet usually mentions the most frequent meaning as the first. We also exploit plWordNet relations called determiner, which links nominals with adjectives that are frequently used to describe them. For example, consider a word organ (Eng. organ). An adjective natleniony (oxygenated) is a determiner of one of the meanings of organ, i.e. part of body, but not the others, i.e. part of an institution. Therefore, the included relatives consist of a polysemous related word, including the target word itself, and a determiner associated with a meaning (wydolny organ and natleniony organ in the example). This procedure is performed only in the case of relatives that weren't included so far, i.e. with a sense number higher than 1. Finally, we also make use of a well-known principle called one word per discourse BIBREF19 , which states that a polysemous word is very unlikely to take different meanings in a single document. In this study, the principle is employed in the following way: if in a single document there appear only relatives corresponding to a single target word sense, then all occurrences of the target word in this document are also treated as training examples for this meaning. One can easily see that these assumptions are false in many cases, which may introduce noise and deteriorate a resulting model. Thus, the presented solutions undergo experimental validation using the following sets of relatives: Monosemous children – monosemous direct hyponyms. Monosemous relatives – monosemous synonyms, direct hyponyms and direct hypernyms. First relatives – words in the first meaning belonging to synsets of synonyms, direct hyponyms or direct hypernyms. Word determiners – collocations made of two words in any order: the target word and a determiner associated with a given meaning. All determiners – collocations made of two words in any order: a polysemous relative and a determiner associated with the appropriate meaning. Other words – occurrences of the target ambiguous word in a document that contains other relatives corresponding to exactly one of the meanings. Table TABREF23 shows how many relatives have been obtained for each category, as well as the number of occurrences in the corpus of 6 million documents (see next section). Corpus As some of the relatives may be very rare, it is important to use a training corpus of significant size. In this case, we used 600 million webpages (594 billion tokens) indexed by a Polish search engine NEKST, developed at the Institute of Computer Science, Polish Academy of Sciences. Training sub-corpora were selected with size varying from 19,000 to 60 million documents. Instead of using snippets returned by a search interface, we use whole textual contents (with morphosyntactic annotation) of each document, taken from NEKST distributed infrastructure. Unfortunately, a lot of text on web pages is not suitable for training a WSD classifier, for example elements of page structure or parts unrecognised by a tagger. Thus, each sentence has to satisfy the following requirements to qualify for training: be at least 150-character long. contain at least five words. contain at least four different parts of speech (including punctuation). These criteria help to filter out most of the web content of unsatisfactory quality. Features Context features extracted for classification are very similar to those that have proven successful in supervised WSD, including experiments on the same evaluation set BIBREF20 : words present at certain positions in a neighbourhood of a target word: lemmas at positions: -2, -1, 1, 2 (denoted by INLINEFORM0 ), morphosyntactic interpretations (sequences of tags) at positions: -1, 1 (denoted by INLINEFORM0 ) and 0 (denoted by INLINEFORM1 ), lemmas present in the sentence (denoted by INLINEFORM0 ). Note that the morphosyntactic interpretations are assigned to single words only, therefore in case of multi-word relatives INLINEFORM0 is not available. Also, a gender tag is removed from INLINEFORM1 . Classification After gathering the values of features from occurrences of relatives, a naïve Bayesian classification model is built. However, as many researchers in the field have noticed BIBREF13 , BIBREF9 , BIBREF11 , there is a major drawback of using relatives instead of a target word: the number of occurrences of relatives is usually not proportional to the frequency of the target word sense. To bypass this problem, we modify the Bayesian model in this study. The basic classifier chooses the class that maximises the following probability: INLINEFORM0 In the case of binary features, which represent the occurrence of a certain word in context, we have INLINEFORM0 and: INLINEFORM1 Which could be rewritten as: INLINEFORM0 The expression has been formulated as a product of two factors: INLINEFORM0 , independent from observed features and corresponding to empty word context, and INLINEFORM1 that depends on observed context. To weaken the influence of improper distribution of training cases, we omit INLINEFORM2 , so that when no context features are observed, every word sense is considered equally probable. Thus, for the given context features INLINEFORM0 , sense INLINEFORM1 is selected when: INLINEFORM2 The table with final results ( TABREF27 ) contains accuracies of both original and modified versions of the classifier. Evaluation For experiments and evaluation a sub-corpus of the National Corpus of Polish, NCP BIBREF6 was employed. The manually annotated sub-corpus contains sense labels of 106 different words: 50 nouns, 48 verbs and 8 adjectives. As verbs have much poorer connectivity in plWordNet, they have been ignored within this study. The senses used for annotation are coarse-grained – with one sense covering a range of related usages. Each word has between two and four senses. To employ the method described in previous section, the NCP senses have been manually mapped to fine-grained plWordNet synsets. As NCP senses have been created independently from wordnet senses, a substantial part of the latter remain uncovered by the former. However, we only found four cases where an NCP sense has no counterpart in wordnet; those words are excluded from the test set. In total, the test set includes 17,314 occurrences of 54 ambiguous words, having two to four coarse-grained meanings. Table TABREF26 contains a detailed summary. The algorithm works using plWordNet synsets and its output is mapped to NCP meaning to measure its quality. Accuracy measures what percentage of the programme's guesses agree with manually assigned senses. To assess the general performance of a particular configuration, the accuracy has been averaged over all target words. To properly judge the results, we need to start with the baselines. Without knowing the distribution of senses, three basic possibilities seem reasonable: we can either 1) select a meaning randomly, 2) base on the sense numbering in NCP or 3) use plWordNet in the same way. To have a better comparison with ontology-based methods, the results also include a word similarity baseline configuration, which selects the sense with the strongest similarity to any of the words in context (sentence). For that purpose the Leacock&Chodorow similarity measure (implemented using all relations between synsets in plWordNet) is employed, as it has been previously used in WSD BIBREF21 and also correlates well with human judgement of similarity BIBREF22 . The baseline results, shown in Table TABREF27 , support the claim of intentional sense ordering in plWordNet. Results The goal of the first experiment was to select an optimal feature set for this task. Several models with a common basic configuration, i.e. using all possible relatives and 6 million documents, have been built with different feature sets and evaluated. The results are shown in Table TABREF22 . As we can see, the lexical features give us more predictive power than morphological interpretations. The best solution, incorporated into the basic configuration for further experiments, includes all features except these based on the interpretation of the word in focus. Secondly, it is necessary to evaluate different types of replacements, outlined in section SECREF2 . Table TABREF23 contains the average number of possible replacements per target word, the number of occurrences in the six-million corpus and the average classification accuracy. As we can see, the number of replacements rises after adding subsequent sources, but the largest increase is caused by including polysemous relatives with determiners. On the other hand, these compound relatives rarely appear in the corpus (13,126 occurences), whereas employing polysemous words in the first sense results in 1,525,739 new training cases and a substantial growth of accuracy. What is more, although the profits from these sources of relatives differ, none of them decreases the accuracy. The availability of the corpus containing 600 million documents helps to answer the question of sufficient corpus size for such task. Figure FIGREF24 shows mean classification accuracy for models built using different training corpora, which have been created by randomly selecting a subset of the original document set. The considered sizes are between 19,000 and 60,000,000. Additionally, a different type of corpora has been created by using only documents from the Polish Wikipedia (sizes 11,000 – 1,098,000). We see that after a certain point adding in new data does not improve the accuracy. Surprisingly, the subcorpora of Wikipedia yield worse results than those of unlimited origin. Table TABREF26 shows the accuracy of disambiguation in the configuration outlined above with respect to the target word. The easiest words have meanings corresponding to distinct physical objects, e.g. in Polish piłka (100% accuracy) may mean a ball or a small saw. The hardest cases are those with many abstract and fuzzy meanings, e.g. in Polish klasa has four meanings related to English class: (1) group of similar objects, (2) level of quality, (3) group of pupils or (4) classroom. The meaning (1) could be hard to distinguish from (2) even for a human, whereas (3) and (4) may appear in very similar contexts. Finally, Table TABREF27 contains the mean accuracy of the basic configuration of the classifier described in this work (with and without modifications to Bayesian model). It is compared to the four previously mentioned baselines. Discussion Although many different configurations have been tested in this study, all of them remain below the accuracy level of 80%, approximately equal to average share of dominating senses in this dataset. This is obviously unsatisfactory and demands explanation. First of all, the new sources of replacements proposed in this work indeed seem to improve the models from 70.86% (only traditional monosemous relatives) to 77.96% (all proposed relatives). The biggest gain is obtained by including the polysemous relatives taking into account only their first meaning. This technique relies on two assumptions: a strong domination of one of the senses and that sense being listed first in plWordNet. While the former is almost always true, if the second assumption is false then the created model are adversely affected. In the case of two target words the senses, the first sense in each case (stopień as a musical concept and forma as a synonym of polynomial) was so peculiar that they were unknown to the author of this study and couldn't be assigned to any of the coarse-grained NCP senses. Clearly, not only the method of unsupervised WSD using relatives, but also other solutions related to polysemy would definitely benefit from a reliable ordering of senses in wordnets, especially as increasingly uncommon senses are added to them with time. It is however not clear how such knowledge could be obtained without solving the WSD task first. What is more, sense distributions obviously change with genre, time, author, etc. When it comes to feature selection, the most unexpected phenomenon observed in this study is low usefulness of the interpretation-based features. According to Table TABREF22 , adding interpretations of neighbouring words ( INLINEFORM0 ) yields very little improvement, while this type of information regarding replacements ( INLINEFORM1 ) even lowers the accuracy. This result could be attributed to two factors. Firstly, more developed replacement generation results in more occurrences, but also causes their tags to differ from the target word by gender or number. They may even not be available at all (in the case of multi-word replacements). The second reason is a difference in language: while in English a word interpretation is represented as one of several dozen part of speech identifiers, in Slavonic languages, such as Polish, we need to specify the values of several tags for each word, leading to thousands of possible interpretations. Obviously, the features based on these tags are very sparse. Finally, the morphosyntactic annotation was performed automatically, which may lead to errors, especially in the case of noisy web text. One of the purposes of this study was to check the necessary amount of training data for such a solution by employing a very large collection from the NEKST search engine. The need for large corpora is obvious when using only monosemous relatives – those usually rare words should appear in many contexts. However, according to the results shown in Figure FIGREF24 , the strategy for generating relatives presented in this paper reaches optimum performance for a reasonable amount of texts – 6 million documents is enough. However, one should keep in mind that this statement remains true assuming a constant evaluation environment; expanding a test set (currently containing 17,314 occurrences) may help to see differences between apparently equivalent models and raise the need for bigger corpora. Conclusions In this paper the limitations and improvements of unsupervised word sense disambiguation have been investigated. The main problem – insufficient number and quality of replacements has been tackled by adding new rich sources of replacements. The quality of the models has indeed improved, especially thanks to replacements based on sense ordering in plWordNet. To deal with the problem of unknown sense distribution, the Bayesian classifier has been modified, removing the bias towards frequent labels in the training data. Finally, the experiments with very large corpus have shown the sufficient amount of training data for this task, which is only 6 million documents. Acknowledgements This study was conducted at Institute of Computer Science, Polish Academy of Sciences, and supported by a research fellowship within "Information technologies: research and their interdisciplinary applications" agreement number POKL.04.01.01-00-051/10-00. The author would like to thank Dariusz Czerski and NEKST team for providing access to the search engine index and helpful discussions and Matthew Shardlow for comments that greatly improved the manuscript.
How do they deal with unknown distribution senses?
5e324846a99a5573cd2e843d1657e87f4eb22fa6
How Ġdo Ġthey Ġdeal Ġwith Ġunknown Ġdistribution Ġsenses ?
The ĠN Ã¤ ive - Bay es Ġclass ifier Ġis Ġcorrected Ġso Ġit Ġis Ġnot Ġbiased Ġto Ġmost Ġfrequent Ġclasses
<s> How Ġdo Ġthey Ġdeal Ġwith Ġunknown Ġdistribution Ġsenses ? </s> Introduction </s> The Ġfocus Ġof Ġthe Ġword Ġsense Ġdis amb ig uation Ġ( W SD ) Ġtask Ġis Ġpoly se my , Ġi . e . Ġwords Ġhaving Ġseveral Ġsubstantially Ġdifferent Ġmeanings . ĠTwo Ġcommon Ġexamples Ġare Ġbank Ġ( ri vers ide Ġor Ġfinancial Ġinstitution ) Ġand Ġbass Ġ( fish Ġor Ġmusical Ġinstrument ), Ġbut Ġusually Ġthe Ġmeanings Ġof Ġa Ġword Ġare Ġclosely Ġrelated , Ġe . g . Ġclass Ġmay Ġrefer Ġto : Ġ( a ) Ġa Ġgroup Ġof Ġstudents , Ġ( b ) Ġthe Ġperiod Ġwhen Ġthey Ġmeet Ġto Ġstudy Ġor Ġ( c ) Ġa Ġroom Ġwhere Ġsuch Ġmeetings Ġoccur . ĠReaders Ġdeal Ġwith Ġthis Ġproblem Ġby Ġusing Ġa Ġword 's Ġcontext Ġand Ġin ĠW SD Ġwe Ġaim Ġat Ġdoing Ġit Ġautomatically . </s> The Ġmost Ġeffective Ġsolution , Ġcalled Ġsupervised ĠW SD , Ġis Ġto Ġuse Ġa Ġlarge Ġnumber Ġof Ġsense - annot ated Ġoccurrences Ġof Ġthe Ġtarget Ġword Ġto Ġbuild Ġa Ġmachine Ġlearning Ġmodel Ġto Ġlabel Ġtest Ġcases . ĠHowever , Ġthis Ġapproach Ġsuffers Ġfrom Ġa Ġknowledge Ġacquisition Ġbottleneck . ĠThe Ġannotation Ġof Ġa Ġseparate Ġtraining Ġcorpus Ġfor Ġevery Ġtarget Ġword Ġdemands Ġa Ġconsiderable Ġamount Ġof Ġhuman Ġlabour . ĠTherefore , Ġthis Ġapproach Ġis Ġunus able Ġin Ġapplications Ġthat Ġrequire ĠW SD Ġacross Ġa Ġwide Ġvocabulary , Ġsuch Ġas Ġopen - domain Ġquestion Ġanswering ĠB IB REF 0 Ġ. </s> The Ġmethod Ġof Ġmon ose m ous Ġrelatives , Ġwhich Ġis Ġthe Ġfocus Ġof Ġthis Ġwork , Ġbypass es Ġthe Ġbottleneck Ġby Ġgathering Ġocc ure nces Ġof Ġwords Ġrelated Ġto Ġthe Ġtarget Ġword , Ġbut Ġfree Ġfrom Ġambiguity , Ġand Ġtreating Ġthem Ġas Ġtraining Ġcases Ġof Ġthe Ġrespective Ġsenses . ĠHuman Ġlabour Ġis Ġeliminated Ġat Ġthe Ġexpense Ġof Ġaccuracy , Ġas Ġthe Ġcontext Ġof Ġeach Ġrelative Ġonly Ġapproximately Ġmatches Ġthe Ġcontext Ġof Ġthe Ġtarget Ġword Ġsense . </s> Mon ose m ous Ġrelatives Ġhave Ġbeen Ġemployed Ġmultiple Ġtimes Ġ( see ĠSection Ġ2 ), Ġbut Ġresults Ġremain Ġunsatisf actory . ĠThe Ġaim Ġof Ġmy Ġstudy Ġis Ġto Ġexplore Ġthe Ġlimitations Ġof Ġthis Ġtechnique Ġby Ġimplementing Ġand Ġevaluating Ġsuch Ġa Ġtool Ġfor ĠPolish . ĠFirstly , Ġthe Ġmethod Ġis Ġexpanded Ġby Ġwa iving Ġthe Ġrequirement Ġof Ġmon ose my Ġand Ġproposing Ġseveral Ġnew Ġsources Ġof Ġrelatives . ĠThese Ġpreviously Ġunexpl ored Ġsources Ġare Ġbased Ġon Ġword net Ġdata Ġand Ġhelp Ġgather Ġmany Ġtraining Ġcases Ġfrom Ġthe Ġcorpus . ĠSecondly , Ġa Ġwell - known Ġproblem Ġof Ġuneven Ġyet Ġunknown Ġdistribution Ġof Ġword Ġsenses Ġis Ġallev iated Ġby Ġmodifying Ġa ĠnaÃ¯ve ĠBay esian Ġclass ifier . ĠThanks Ġto Ġthis Ġcorrection , Ġthe Ġclass ifier Ġis Ġno Ġlonger Ġbiased Ġtowards Ġsenses Ġthat Ġhave Ġmore Ġtraining Ġdata . ĠFinally , Ġa Ġvery Ġlarge Ġcorpus Ġ( 600 Ġmillion Ġdocuments ), Ġgathered Ġfrom Ġthe Ġweb Ġby Ġa ĠPolish Ġsearch Ġengine ĠNE K ST , Ġis Ġused Ġto Ġbuild Ġmodels Ġbased Ġon Ġtraining Ġcorpor a Ġof Ġdifferent Ġsizes . ĠThose Ġexperiments Ġshow Ġwhat Ġamount Ġof Ġdata Ġis Ġsufficient Ġfor Ġsuch Ġa Ġtask . ĠThe Ġproposed Ġsolution Ġis Ġcompared Ġto Ġbas elines Ġthat Ġuse Ġword net Ġstructure Ġonly , Ġwith Ġno Ġtraining Ġcorpor a . </s> This Ġpaper Ġis Ġorganised Ġas Ġfollows . ĠThe Ġnext Ġsection Ġreviews Ġthe Ġprevious Ġresearch Ġin Ġthe Ġarea , Ġfocusing Ġon Ġun super vised ĠW SD Ġusing Ġmon ose m ous Ġrelatives . ĠSection Ġ3 Ġoutlines Ġthe Ġproposed Ġsolution Ġby Ġdescribing Ġthe Ġnew Ġsources Ġof Ġrelatives , Ġthe Ġemployed Ġcorpus , Ġthe Ġfeatures Ġextracted Ġfrom Ġcontext Ġand Ġthe Ġmodified ĠBay esian Ġclass ifier . ĠSection Ġ4 Ġdescribes Ġthe Ġevaluation Ġdata Ġand Ġprocess , Ġwhile Ġsection Ġ5 Ġquotes Ġthe Ġresults . ĠSection Ġ6 Ġis Ġdevoted Ġto Ġdiscussing Ġthe Ġoutcomes Ġand Ġsection Ġ7 Ġconcludes Ġthe Ġpaper . </s> Related Ġwork </s> The Ġproblem Ġof ĠW SD Ġhas Ġreceived Ġa Ġlot Ġof Ġattention Ġsince Ġthe Ġbeginning Ġof Ġnatural Ġlanguage Ġprocessing Ġresearch . ĠW SD Ġis Ġtypically Ġexpected Ġto Ġimprove Ġthe Ġresults Ġof Ġreal - world Ġapplications : Ġoriginally Ġmachine Ġtranslation Ġand Ġrecently Ġinformation Ġretrieval Ġand Ġextraction , Ġespecially Ġquestion Ġanswering ĠB IB REF 0 Ġ. ĠLike Ġmany Ġother Ġareas , ĠW SD Ġhas Ġgreatly Ġbenefited Ġfrom Ġpublicly Ġavailable Ġtest Ġsets Ġand Ġcompetitions . ĠTwo Ġnotable Ġcorpor a Ġare : Ġ1 ) ĠSem Cor ĠB IB REF 1 Ġ, Ġbuilt Ġby Ġlab elling Ġa Ġsubset Ġof ĠBrown Ġcorpus Ġwith ĠPrinceton ĠWord Net Ġsy ns ets Ġand Ġ2 ) Ġthe Ġpublic Ġevaluations Ġof ĠSense val Ġworkshops ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. </s> There Ġare Ġa Ġvariety Ġof Ġapproaches Ġto Ġsolve Ġthe ĠW SD Ġproblem , Ġwhich Ġcan Ġbe Ġgrouped Ġbased Ġupon Ġhow Ġthey Ġuse Ġtheir Ġdata ĠâĢĵ Ġsee Ġreviews ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ. ĠIn Ġsupervised Ġsolutions Ġa Ġlarge Ġsense - tag ged Ġcorpus Ġis Ġavailable Ġfor Ġtraining . ĠThis Ġapproach Ġhas Ġbeen Ġapplied Ġto Ġthe Ġthe Ġtest Ġset Ġused Ġin Ġthe Ġcurrent Ġstudy , Ġresulting Ġin Ġan Ġaccuracy Ġvalue Ġof Ġ91 . 5 % ĠB IB REF 6 Ġ. ĠAlthough Ġthis Ġtechnique Ġundoubtedly Ġyields Ġthe Ġbest Ġresults , Ġwe Ġwould Ġneed Ġan Ġimmense Ġamount Ġof Ġhuman Ġlabour Ġto Ġbuild Ġa Ġtraining Ġcorpus Ġof Ġsufficient Ġsize Ġfor Ġdis amb ig uating Ġall Ġwords . ĠThis Ġdoes Ġnot Ġseem Ġpossible , Ġespecially Ġin Ġthe Ġcase Ġof Ġlanguages , Ġsuch Ġas ĠPolish , Ġwhich Ġreceive Ġless Ġattention Ġthan ĠEnglish . </s> In Ġthe Ġminim ally Ġsupervised Ġapproach ĠB IB REF 7 Ġ, Ġa Ġsmall Ġset Ġof Ġinitial Ġtraining Ġexamples , Ġobtained Ġby Ġa Ġhe uristic Ġor Ġhand - tag ging , Ġis Ġused Ġto Ġlabel Ġnew Ġoccurrences . ĠThey Ġin Ġturn Ġserve Ġas Ġa Ġtraining Ġset Ġfor Ġnext Ġiteration , Ġand Ġso Ġon . ĠThis Ġboot stra pping Ġprocedure Ġrequires Ġvery Ġlittle Ġmanual Ġtagging Ġbut Ġneeds Ġto Ġbe Ġcarefully Ġimplemented Ġto Ġavoid Ġlo osing Ġaccuracy Ġin Ġfurther Ġsteps . </s> Un super vised Ġmethods Ġuse Ġno Ġpreviously Ġlabelled Ġexamples . ĠInstead Ġan Ġexternal Ġknowledge Ġsource Ġis Ġemployed , Ġe . g . Ġa Ġmachine - readable Ġdictionary Ġor Ġword net . ĠIn Ġthe Ġsimplest Ġun super vised Ġsolution , Ġcalled Ġthe ĠLes k Ġalgorithm ĠB IB REF 8 Ġ, Ġmeanings Ġof Ġconsecutive Ġambiguous Ġwords Ġare Ġselected Ġby Ġfinding Ġthose Ġsenses Ġwhose Ġdefinitions Ġoverlap Ġthe Ġmost . </s> If Ġlack Ġof Ġdefinitions Ġmake Ġthe ĠLes k Ġalgorithm Ġinf eas ible , Ġwe Ġcan Ġexploit Ġrelations Ġbetween Ġwords . ĠThis Ġstudy Ġfocuses Ġon Ġmon ose m ous Ġrelatives , Ġi . e . Ġwords Ġor Ġcoll ocations , Ġselected Ġusing Ġword net , Ġbeing Ġrelated Ġto Ġa Ġdis amb ig uation Ġtarget , Ġbut Ġfree Ġof Ġambiguity . ĠOne Ġcan Ġeasily Ġfind Ġoccurrences Ġof Ġsuch Ġrelatives Ġin Ġan Ġun annot ated Ġtext Ġand Ġtreat Ġthem Ġas Ġtraining Ġexamples Ġfor Ġthe Ġtarget Ġambiguous Ġword . ĠThe Ġmethod Ġhas Ġbeen Ġsuccessfully Ġapplied Ġin Ġan ĠEnglish ĠW SD Ġtask ĠB IB REF 9 Ġ, Ġbut Ġstill Ġmany Ġproblems Ġremain . ĠOne Ġof Ġthem Ġis Ġchoice Ġof Ġrelatives ĠâĢĵ Ġin Ġfact , Ġeven Ġsyn onyms Ġdiffer Ġin Ġmeaning Ġand Ġusage Ġcontexts ; Ġand Ġthey Ġare Ġnot Ġavailable Ġfor Ġmany Ġwords . ĠThat Ġis Ġwhy Ġalso Ġhyper ny ms Ġand Ġhyp onyms , Ġespecially Ġmulti - word Ġexpressions Ġcontaining Ġthe Ġtarget Ġword , Ġare Ġtaken Ġinto Ġaccount . ĠSome Ġresearchers Ġalso Ġinclude Ġsiblings Ġ( i . e . Ġwords Ġwith Ġa Ġcommon Ġhyper ny m Ġwith Ġthe Ġtarget ) Ġand Ġan ton y ms , Ġbut Ġtheir Ġinfluence Ġis Ġnot Ġalways Ġbeneficiary ĠB IB REF 10 Ġ. ĠOther Ġinteresting Ġsources Ġof Ġmon ose m ous Ġrelatives Ġare Ġparts Ġof Ġdefinition ĠB IB REF 11 Ġ, Ġnamed Ġentities ĠB IB REF 12 Ġ, Ġindirect Ġhyp onyms Ġand Ġhyper ny ms , Ġand Ġfinally Ġmer onyms Ġand Ġhol onyms ĠB IB REF 10 Ġ. </s> The Ġmajority Ġof Ġclassification Ġtechniques Ġare Ġbuilt Ġon Ġan Ġassumption Ġthat Ġthe Ġtraining Ġdata Ġapproximately Ġreflects Ġthe Ġtrue Ġdistribution Ġof Ġthe Ġtarget Ġclasses . ĠHowever , Ġthat Ġis Ġnot Ġthe Ġcase Ġwhen Ġusing Ġmon ose m ous Ġrelatives . ĠThe Ġnumber Ġof Ġtheir Ġoccurrences Ġseldom Ġagrees Ġwith Ġthe Ġprobabilities Ġof Ġcorresponding Ġword Ġsenses . ĠQuite Ġoften Ġit Ġactually Ġis Ġthe Ġopposite : Ġobvious Ġand Ġfrequent Ġmeanings Ġhave Ġvery Ġfew Ġrelatives Ġand Ġvice Ġversa . ĠSome Ġresearchers Ġsimply Ġcopy Ġthe Ġa Ġprior i Ġprobabilities Ġfrom Ġtest Ġdata ĠB IB REF 9 Ġ, Ġothers Ġemploy Ġhe ur istics , Ġbut Ġthey Ġare Ġeasily Ġbeaten Ġby Ġstatistics Ġtaken Ġfrom Ġa Ġreal Ġannot ated Ġcorpus , Ġeven Ġdifferent Ġthan Ġtest Ġset ĠB IB REF 13 Ġ. </s> Prep aring Ġa Ġcorpus Ġfor Ġfinding Ġrelatives Ġposes Ġa Ġchallenge Ġas Ġwell . ĠIt Ġshould Ġcontain Ġa Ġlot Ġof Ġtext , Ġas Ġmany Ġmon ose m ous Ġwords Ġare Ġscarce . ĠSome Ġresearchers Ġuse Ġsnippets Ġretrieved Ġfrom Ġsearch Ġengines , Ġi . e . ĠAl ta V ista ĠB IB REF 11 Ġor ĠGoogle ĠB IB REF 13 Ġ. ĠOne Ġcan Ġalso Ġextend Ġa Ġsearch Ġquery Ġby Ġincluding Ġthe Ġcontext Ġof Ġthe Ġdis amb ig uated Ġword ĠB IB REF 14 Ġ, Ġbut Ġit Ġrequires Ġusing Ġas Ġmany Ġqueries Ġas Ġtest Ġcases . </s> Finally , Ġthe Ġusage Ġof Ġmon ose m ous Ġrelatives Ġhas Ġmore Ġapplications Ġthan Ġclassical ĠW SD . ĠOne Ġcan Ġuse Ġthem Ġto Ġgenerate Ġtopical Ġsignatures Ġfor Ġconcepts ĠB IB REF 15 Ġ, Ġautomatically Ġbuild Ġlarge Ġsense - tag ged Ġcorpor a ĠB IB REF 16 Ġand Ġevaluate Ġthe Ġquality Ġof Ġword net - related Ġsemantic Ġresources ĠB IB REF 17 Ġ. </s> Method </s> The Ġalgorithm Ġworks Ġas Ġfollows . ĠFirst , Ġa Ġset Ġof Ġrelatives Ġis Ġobtained Ġfor Ġeach Ġsense Ġof Ġa Ġtarget Ġword Ġusing Ġthe ĠPolish Ġword net : Ġpl Word Net ĠB IB REF 18 Ġ. ĠSome Ġof Ġthe Ġreplacements Ġmay Ġhave Ġmultiple Ġsenses , Ġhowever Ġusually Ġone Ġof Ġthem Ġcovers Ġmost Ġcases . ĠSecondly , Ġa Ġset Ġof Ġcontext Ġfeatures Ġis Ġextracted Ġfrom Ġoccurrences Ġof Ġrelatives Ġin Ġthe ĠNE K ST Ġcorpus . ĠFinally , Ġthe Ġaggreg ated Ġfeature Ġvalues Ġcorresponding Ġto Ġtarget Ġword Ġsenses Ġare Ġused Ġto Ġbuild Ġa ĠnaÃ¯ve ĠBay esian Ġclass ifier Ġadjusted Ġto Ġa Ġsituation Ġof Ġunknown Ġa Ġprior i Ġprobabilities . </s> Rel atives </s> In Ġorder Ġto Ġobtain Ġtraining Ġcases Ġfrom Ġun annot ated Ġcorpor a , Ġwe Ġaim Ġto Ġselect Ġrelatives Ġwhich Ġare Ġsem antically Ġsimilar Ġto Ġa Ġgiven Ġsense Ġof Ġa Ġtarget Ġword . ĠAn Ġexample Ġof Ġthis Ġprocess , Ġconcerning Ġthe Ġword Ġj Ä Ļ zyk Ġ( t ong ue ) Ġin Ġone Ġof Ġits Ġmeanings Ġ( human Ġor Ġanimal Ġorgan ) Ġis Ġshown Ġin ĠFigure ĠFIG REF 4 Ġ. ĠThis Ġstudy Ġtakes Ġinto Ġaccount Ġonly Ġsyn onyms , Ġhyper ny ms Ġand Ġhyp onyms , Ġas Ġother Ġoptions Ġ( s iblings , Ġan ton y ms , Ġhigher - order Ġrelatives ) Ġhave Ġpreviously Ġgiven Ġunsatisf actory Ġresults ĠB IB REF 10 Ġ. ĠInstead , Ġanother Ġproblem Ġdeserves Ġmore Ġattention : Ġhow Ġdo Ġwe Ġselect Ġthose Ġoccurrences Ġof Ġa Ġpoly sem ous Ġrelative Ġthat Ġcorrespond Ġto Ġa Ġtarget Ġword Ġsense ? ĠSo Ġfar , Ġthe Ġproblem Ġhas Ġbeen Ġcircumvent ed Ġby Ġincluding Ġonly Ġmon ose m ous Ġrelatives Ġ( nar z Ä ħ d Ġand Ġj Ä Ļ z or Ġin Ġthe Ġexample ), Ġwhich Ġgreatly Ġdecreases Ġtheir Ġavailability . ĠInstead , Ġwe Ġemploy Ġthose Ġrelatives , Ġwhose Ġfirst Ġmeaning Ġis Ġrelated Ġto Ġthe Ġconsidered Ġsense Ġ( arty k ulator Ġin Ġthe Ġexample ). ĠThe Ġintuition Ġis Ġthat Ġpl Word Net Ġusually Ġmentions Ġthe Ġmost Ġfrequent Ġmeaning Ġas Ġthe Ġfirst . </s> We Ġalso Ġexploit Ġpl Word Net Ġrelations Ġcalled Ġdetermin er , Ġwhich Ġlinks Ġnom inals Ġwith Ġadject ives Ġthat Ġare Ġfrequently Ġused Ġto Ġdescribe Ġthem . ĠFor Ġexample , Ġconsider Ġa Ġword Ġorgan Ġ( Eng . Ġorgan ). ĠAn Ġadjective Ġnat len ion y Ġ( oxy gen ated ) Ġis Ġa Ġdetermin er Ġof Ġone Ġof Ġthe Ġmeanings Ġof Ġorgan , Ġi . e . Ġpart Ġof Ġbody , Ġbut Ġnot Ġthe Ġothers , Ġi . e . Ġpart Ġof Ġan Ġinstitution . ĠTherefore , Ġthe Ġincluded Ġrelatives Ġconsist Ġof Ġa Ġpoly sem ous Ġrelated Ġword , Ġincluding Ġthe Ġtarget Ġword Ġitself , Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġa Ġmeaning Ġ( w yd ol ny Ġorgan Ġand Ġnat len ion y Ġorgan Ġin Ġthe Ġexample ). ĠThis Ġprocedure Ġis Ġperformed Ġonly Ġin Ġthe Ġcase Ġof Ġrelatives Ġthat Ġweren 't Ġincluded Ġso Ġfar , Ġi . e . Ġwith Ġa Ġsense Ġnumber Ġhigher Ġthan Ġ1 . </s> Finally , Ġwe Ġalso Ġmake Ġuse Ġof Ġa Ġwell - known Ġprinciple Ġcalled Ġone Ġword Ġper Ġdiscourse ĠB IB REF 19 Ġ, Ġwhich Ġstates Ġthat Ġa Ġpoly sem ous Ġword Ġis Ġvery Ġunlikely Ġto Ġtake Ġdifferent Ġmeanings Ġin Ġa Ġsingle Ġdocument . ĠIn Ġthis Ġstudy , Ġthe Ġprinciple Ġis Ġemployed Ġin Ġthe Ġfollowing Ġway : Ġif Ġin Ġa Ġsingle Ġdocument Ġthere Ġappear Ġonly Ġrelatives Ġcorresponding Ġto Ġa Ġsingle Ġtarget Ġword Ġsense , Ġthen Ġall Ġoccurrences Ġof Ġthe Ġtarget Ġword Ġin Ġthis Ġdocument Ġare Ġalso Ġtreated Ġas Ġtraining Ġexamples Ġfor Ġthis Ġmeaning . </s> One Ġcan Ġeasily Ġsee Ġthat Ġthese Ġassumptions Ġare Ġfalse Ġin Ġmany Ġcases , Ġwhich Ġmay Ġintroduce Ġnoise Ġand Ġdeterior ate Ġa Ġresulting Ġmodel . ĠThus , Ġthe Ġpresented Ġsolutions Ġundergo Ġexperimental Ġvalidation Ġusing Ġthe Ġfollowing Ġsets Ġof Ġrelatives : </s> Mon ose m ous Ġchildren ĠâĢĵ Ġmon ose m ous Ġdirect Ġhyp onyms . </s> Mon ose m ous Ġrelatives ĠâĢĵ Ġmon ose m ous Ġsyn onyms , Ġdirect Ġhyp onyms Ġand Ġdirect Ġhyper ny ms . </s> First Ġrelatives ĠâĢĵ Ġwords Ġin Ġthe Ġfirst Ġmeaning Ġbelonging Ġto Ġsy ns ets Ġof Ġsyn onyms , Ġdirect Ġhyp onyms Ġor Ġdirect Ġhyper ny ms . </s> Word Ġdetermin ers ĠâĢĵ Ġcoll ocations Ġmade Ġof Ġtwo Ġwords Ġin Ġany Ġorder : Ġthe Ġtarget Ġword Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġa Ġgiven Ġmeaning . </s> All Ġdetermin ers ĠâĢĵ Ġcoll ocations Ġmade Ġof Ġtwo Ġwords Ġin Ġany Ġorder : Ġa Ġpoly sem ous Ġrelative Ġand Ġa Ġdetermin er Ġassociated Ġwith Ġthe Ġappropriate Ġmeaning . </s> Other Ġwords ĠâĢĵ Ġoccurrences Ġof Ġthe Ġtarget Ġambiguous Ġword Ġin Ġa Ġdocument Ġthat Ġcontains Ġother Ġrelatives Ġcorresponding Ġto Ġexactly Ġone Ġof Ġthe Ġmeanings . </s> Table ĠT AB REF 23 Ġshows Ġhow Ġmany Ġrelatives Ġhave Ġbeen Ġobtained Ġfor Ġeach Ġcategory , Ġas Ġwell Ġas Ġthe Ġnumber Ġof Ġoccurrences Ġin Ġthe Ġcorpus Ġof Ġ6 Ġmillion Ġdocuments Ġ( see Ġnext Ġsection ). </s> Corp us </s> As Ġsome Ġof Ġthe Ġrelatives Ġmay Ġbe Ġvery Ġrare , Ġit Ġis Ġimportant Ġto Ġuse Ġa Ġtraining Ġcorpus Ġof Ġsignificant Ġsize . ĠIn Ġthis Ġcase , Ġwe Ġused Ġ600 Ġmillion Ġweb pages Ġ( 594 Ġbillion Ġtokens ) Ġindexed Ġby Ġa ĠPolish Ġsearch Ġengine ĠNE K ST , Ġdeveloped Ġat Ġthe ĠInstitute Ġof ĠComputer ĠScience , ĠPolish ĠAcademy Ġof ĠSciences . ĠTraining Ġsub - cor pora Ġwere Ġselected Ġwith Ġsize Ġvarying Ġfrom Ġ19 , 000 Ġto Ġ60 Ġmillion Ġdocuments . ĠInstead Ġof Ġusing Ġsnippets Ġreturned Ġby Ġa Ġsearch Ġinterface , Ġwe Ġuse Ġwhole Ġtextual Ġcontents Ġ( with Ġmorph os ynt actic Ġannotation ) Ġof Ġeach Ġdocument , Ġtaken Ġfrom ĠNE K ST Ġdistributed Ġinfrastructure . </s> Unfortunately , Ġa Ġlot Ġof Ġtext Ġon Ġweb Ġpages Ġis Ġnot Ġsuitable Ġfor Ġtraining Ġa ĠW SD Ġclass ifier , Ġfor Ġexample Ġelements Ġof Ġpage Ġstructure Ġor Ġparts Ġunrecogn ised Ġby Ġa Ġtag ger . ĠThus , Ġeach Ġsentence Ġhas Ġto Ġsatisfy Ġthe Ġfollowing Ġrequirements Ġto Ġqualify Ġfor Ġtraining : </s> be Ġat Ġleast Ġ150 - character Ġlong . </s> cont ain Ġat Ġleast Ġfive Ġwords . </s> cont ain Ġat Ġleast Ġfour Ġdifferent Ġparts Ġof Ġspeech Ġ( including Ġpunct uation ). </s> These Ġcriteria Ġhelp Ġto Ġfilter Ġout Ġmost Ġof Ġthe Ġweb Ġcontent Ġof Ġunsatisf actory Ġquality . </s> Features </s> Context Ġfeatures Ġextracted Ġfor Ġclassification Ġare Ġvery Ġsimilar Ġto Ġthose Ġthat Ġhave Ġproven Ġsuccessful Ġin Ġsupervised ĠW SD , Ġincluding Ġexperiments Ġon Ġthe Ġsame Ġevaluation Ġset ĠB IB REF 20 Ġ: </s> words Ġpresent Ġat Ġcertain Ġpositions Ġin Ġa Ġneighbourhood Ġof Ġa Ġtarget Ġword : </s> le mm as Ġat Ġpositions : Ġ- 2 , Ġ- 1 , Ġ1 , Ġ2 Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ), </s> morph os ynt actic Ġinterpretations Ġ( sequ ences Ġof Ġtags ) Ġat Ġpositions : Ġ- 1 , Ġ1 Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ) Ġand Ġ0 Ġ( den oted Ġby ĠIN LINE FORM 1 Ġ), </s> le mm as Ġpresent Ġin Ġthe Ġsentence Ġ( den oted Ġby ĠIN LINE FORM 0 Ġ). </s> Note Ġthat Ġthe Ġmorph os ynt actic Ġinterpretations Ġare Ġassigned Ġto Ġsingle Ġwords Ġonly , Ġtherefore Ġin Ġcase Ġof Ġmulti - word Ġrelatives ĠIN LINE FORM 0 Ġis Ġnot Ġavailable . ĠAlso , Ġa Ġgender Ġtag Ġis Ġremoved Ġfrom ĠIN LINE FORM 1 Ġ. </s> Class ification </s> After Ġgathering Ġthe Ġvalues Ġof Ġfeatures Ġfrom Ġoccurrences Ġof Ġrelatives , Ġa ĠnaÃ¯ve ĠBay esian Ġclassification Ġmodel Ġis Ġbuilt . ĠHowever , Ġas Ġmany Ġresearchers Ġin Ġthe Ġfield Ġhave Ġnoticed ĠB IB REF 13 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 11 Ġ, Ġthere Ġis Ġa Ġmajor Ġdrawback Ġof Ġusing Ġrelatives Ġinstead Ġof Ġa Ġtarget Ġword : Ġthe Ġnumber Ġof Ġoccurrences Ġof Ġrelatives Ġis Ġusually Ġnot Ġproportional Ġto Ġthe Ġfrequency Ġof Ġthe Ġtarget Ġword Ġsense . ĠTo Ġbypass Ġthis Ġproblem , Ġwe Ġmodify Ġthe ĠBay esian Ġmodel Ġin Ġthis Ġstudy . ĠThe Ġbasic Ġclass ifier Ġchooses Ġthe Ġclass Ġthat Ġmaxim ises Ġthe Ġfollowing Ġprobability : ĠIN LINE FORM 0 </s> In Ġthe Ġcase Ġof Ġbinary Ġfeatures , Ġwhich Ġrepresent Ġthe Ġoccurrence Ġof Ġa Ġcertain Ġword Ġin Ġcontext , Ġwe Ġhave ĠIN LINE FORM 0 Ġand : ĠIN LINE FORM 1 </s> Which Ġcould Ġbe Ġrewritten Ġas : ĠIN LINE FORM 0 </s> The Ġexpression Ġhas Ġbeen Ġformulated Ġas Ġa Ġproduct Ġof Ġtwo Ġfactors : ĠIN LINE FORM 0 Ġ, Ġindependent Ġfrom Ġobserved Ġfeatures Ġand Ġcorresponding Ġto Ġempty Ġword Ġcontext , Ġand ĠIN LINE FORM 1 Ġthat Ġdepends Ġon Ġobserved Ġcontext . ĠTo Ġweaken Ġthe Ġinfluence Ġof Ġimproper Ġdistribution Ġof Ġtraining Ġcases , Ġwe Ġomit ĠIN LINE FORM 2 Ġ, Ġso Ġthat Ġwhen Ġno Ġcontext Ġfeatures Ġare Ġobserved , Ġevery Ġword Ġsense Ġis Ġconsidered Ġequally Ġprobable . </s> Thus , Ġfor Ġthe Ġgiven Ġcontext Ġfeatures ĠIN LINE FORM 0 Ġ, Ġsense ĠIN LINE FORM 1 Ġis Ġselected Ġwhen : ĠIN LINE FORM 2 </s> The Ġtable Ġwith Ġfinal Ġresults Ġ( ĠT AB REF 27 Ġ) Ġcontains Ġaccur acies Ġof Ġboth Ġoriginal Ġand Ġmodified Ġversions Ġof Ġthe Ġclass ifier . </s> E val uation </s> For Ġexperiments Ġand Ġevaluation Ġa Ġsub - cor p us Ġof Ġthe ĠNational ĠCorpus Ġof ĠPolish , ĠN CP ĠB IB REF 6 Ġwas Ġemployed . ĠThe Ġmanually Ġannot ated Ġsub - cor p us Ġcontains Ġsense Ġlabels Ġof Ġ106 Ġdifferent Ġwords : Ġ50 Ġnoun s , Ġ48 Ġverbs Ġand Ġ8 Ġadject ives . ĠAs Ġverbs Ġhave Ġmuch Ġpoorer Ġconnectivity Ġin Ġpl Word Net , Ġthey Ġhave Ġbeen Ġignored Ġwithin Ġthis Ġstudy . </s> The Ġsenses Ġused Ġfor Ġannotation Ġare Ġcoarse - gr ained ĠâĢĵ Ġwith Ġone Ġsense Ġcovering Ġa Ġrange Ġof Ġrelated Ġus ages . ĠEach Ġword Ġhas Ġbetween Ġtwo Ġand Ġfour Ġsenses . ĠTo Ġemploy Ġthe Ġmethod Ġdescribed Ġin Ġprevious Ġsection , Ġthe ĠN CP Ġsenses Ġhave Ġbeen Ġmanually Ġmapped Ġto Ġfine - gr ained Ġpl Word Net Ġsy ns ets . ĠAs ĠN CP Ġsenses Ġhave Ġbeen Ġcreated Ġindependently Ġfrom Ġword net Ġsenses , Ġa Ġsubstantial Ġpart Ġof Ġthe Ġlatter Ġremain Ġuncovered Ġby Ġthe Ġformer . ĠHowever , Ġwe Ġonly Ġfound Ġfour Ġcases Ġwhere Ġan ĠN CP Ġsense Ġhas Ġno Ġcounterpart Ġin Ġword net ; Ġthose Ġwords Ġare Ġexcluded Ġfrom Ġthe Ġtest Ġset . </s> In Ġtotal , Ġthe Ġtest Ġset Ġincludes Ġ17 , 314 Ġoccurrences Ġof Ġ54 Ġambiguous Ġwords , Ġhaving Ġtwo Ġto Ġfour Ġcoarse - gr ained Ġmeanings . ĠTable ĠT AB REF 26 Ġcontains Ġa Ġdetailed Ġsummary . </s> The Ġalgorithm Ġworks Ġusing Ġpl Word Net Ġsy ns ets Ġand Ġits Ġoutput Ġis Ġmapped Ġto ĠN CP Ġmeaning Ġto Ġmeasure Ġits Ġquality . ĠAccuracy Ġmeasures Ġwhat Ġpercentage Ġof Ġthe Ġprogramme 's Ġguesses Ġagree Ġwith Ġmanually Ġassigned Ġsenses . ĠTo Ġassess Ġthe Ġgeneral Ġperformance Ġof Ġa Ġparticular Ġconfiguration , Ġthe Ġaccuracy Ġhas Ġbeen Ġaveraged Ġover Ġall Ġtarget Ġwords . </s> To Ġproperly Ġjudge Ġthe Ġresults , Ġwe Ġneed Ġto Ġstart Ġwith Ġthe Ġbas elines . ĠWithout Ġknowing Ġthe Ġdistribution Ġof Ġsenses , Ġthree Ġbasic Ġpossibilities Ġseem Ġreasonable : Ġwe Ġcan Ġeither Ġ1 ) Ġselect Ġa Ġmeaning Ġrandomly , Ġ2 ) Ġbase Ġon Ġthe Ġsense Ġnumbering Ġin ĠN CP Ġor Ġ3 ) Ġuse Ġpl Word Net Ġin Ġthe Ġsame Ġway . ĠTo Ġhave Ġa Ġbetter Ġcomparison Ġwith Ġont ology - based Ġmethods , Ġthe Ġresults Ġalso Ġinclude Ġa Ġword Ġsimilarity Ġbaseline Ġconfiguration , Ġwhich Ġselects Ġthe Ġsense Ġwith Ġthe Ġstrongest Ġsimilarity Ġto Ġany Ġof Ġthe Ġwords Ġin Ġcontext Ġ( sent ence ). ĠFor Ġthat Ġpurpose Ġthe ĠLe ac ock & Ch odor ow Ġsimilarity Ġmeasure Ġ( im ple mented Ġusing Ġall Ġrelations Ġbetween Ġsy ns ets Ġin Ġpl Word Net ) Ġis Ġemployed , Ġas Ġit Ġhas Ġbeen Ġpreviously Ġused Ġin ĠW SD ĠB IB REF 21 Ġand Ġalso Ġcorrelates Ġwell Ġwith Ġhuman Ġjudgement Ġof Ġsimilarity ĠB IB REF 22 Ġ. ĠThe Ġbaseline Ġresults , Ġshown Ġin ĠTable ĠT AB REF 27 Ġ, Ġsupport Ġthe Ġclaim Ġof Ġintentional Ġsense Ġordering Ġin Ġpl Word Net . </s> Results </s> The Ġgoal Ġof Ġthe Ġfirst Ġexperiment Ġwas Ġto Ġselect Ġan Ġoptimal Ġfeature Ġset Ġfor Ġthis Ġtask . ĠSeveral Ġmodels Ġwith Ġa Ġcommon Ġbasic Ġconfiguration , Ġi . e . Ġusing Ġall Ġpossible Ġrelatives Ġand Ġ6 Ġmillion Ġdocuments , Ġhave Ġbeen Ġbuilt Ġwith Ġdifferent Ġfeature Ġsets Ġand Ġevaluated . ĠThe Ġresults Ġare Ġshown Ġin ĠTable ĠT AB REF 22 Ġ. ĠAs Ġwe Ġcan Ġsee , Ġthe Ġlex ical Ġfeatures Ġgive Ġus Ġmore Ġpredictive Ġpower Ġthan Ġmorph ological Ġinterpretations . ĠThe Ġbest Ġsolution , Ġincorporated Ġinto Ġthe Ġbasic Ġconfiguration Ġfor Ġfurther Ġexperiments , Ġincludes Ġall Ġfeatures Ġexcept Ġthese Ġbased Ġon Ġthe Ġinterpretation Ġof Ġthe Ġword Ġin Ġfocus . </s> Secondly , Ġit Ġis Ġnecessary Ġto Ġevaluate Ġdifferent Ġtypes Ġof Ġreplacements , Ġoutlined Ġin Ġsection ĠSEC REF 2 Ġ. ĠTable ĠT AB REF 23 Ġcontains Ġthe Ġaverage Ġnumber Ġof Ġpossible Ġreplacements Ġper Ġtarget Ġword , Ġthe Ġnumber Ġof Ġoccurrences Ġin Ġthe Ġsix - million Ġcorpus Ġand Ġthe Ġaverage Ġclassification Ġaccuracy . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġnumber Ġof Ġreplacements Ġrises Ġafter Ġadding Ġsubsequent Ġsources , Ġbut Ġthe Ġlargest Ġincrease Ġis Ġcaused Ġby Ġincluding Ġpoly sem ous Ġrelatives Ġwith Ġdetermin ers . ĠOn Ġthe Ġother Ġhand , Ġthese Ġcompound Ġrelatives Ġrarely Ġappear Ġin Ġthe Ġcorpus Ġ( 13 , 126 Ġocc ure nces ), Ġwhereas Ġemploying Ġpoly sem ous Ġwords Ġin Ġthe Ġfirst Ġsense Ġresults Ġin Ġ1 , 525 , 7 39 Ġnew Ġtraining Ġcases Ġand Ġa Ġsubstantial Ġgrowth Ġof Ġaccuracy . ĠWhat Ġis Ġmore , Ġalthough Ġthe Ġprofits Ġfrom Ġthese Ġsources Ġof Ġrelatives Ġdiffer , Ġnone Ġof Ġthem Ġdecreases Ġthe Ġaccuracy . </s> The Ġavailability Ġof Ġthe Ġcorpus Ġcontaining Ġ600 Ġmillion Ġdocuments Ġhelps Ġto Ġanswer Ġthe Ġquestion Ġof Ġsufficient Ġcorpus Ġsize Ġfor Ġsuch Ġtask . ĠFigure ĠFIG REF 24 Ġshows Ġmean Ġclassification Ġaccuracy Ġfor Ġmodels Ġbuilt Ġusing Ġdifferent Ġtraining Ġcorpor a , Ġwhich Ġhave Ġbeen Ġcreated Ġby Ġrandomly Ġselecting Ġa Ġsubset Ġof Ġthe Ġoriginal Ġdocument Ġset . ĠThe Ġconsidered Ġsizes Ġare Ġbetween Ġ19 , 000 Ġand Ġ60 , 000 , 000 . ĠAdditionally , Ġa Ġdifferent Ġtype Ġof Ġcorpor a Ġhas Ġbeen Ġcreated Ġby Ġusing Ġonly Ġdocuments Ġfrom Ġthe ĠPolish ĠWikipedia Ġ( s izes Ġ11 , 000 ĠâĢĵ Ġ1 , 09 8 , 000 ). ĠWe Ġsee Ġthat Ġafter Ġa Ġcertain Ġpoint Ġadding Ġin Ġnew Ġdata Ġdoes Ġnot Ġimprove Ġthe Ġaccuracy . ĠSurprisingly , Ġthe Ġsub cor pora Ġof ĠWikipedia Ġyield Ġworse Ġresults Ġthan Ġthose Ġof Ġunlimited Ġorigin . </s> Table ĠT AB REF 26 Ġshows Ġthe Ġaccuracy Ġof Ġdis amb ig uation Ġin Ġthe Ġconfiguration Ġoutlined Ġabove Ġwith Ġrespect Ġto Ġthe Ġtarget Ġword . ĠThe Ġeasiest Ġwords Ġhave Ġmeanings Ġcorresponding Ġto Ġdistinct Ġphysical Ġobjects , Ġe . g . Ġin ĠPolish Ġpi ÅĤ ka Ġ( 100 % Ġaccuracy ) Ġmay Ġmean Ġa Ġball Ġor Ġa Ġsmall Ġsaw . ĠThe Ġhardest Ġcases Ġare Ġthose Ġwith Ġmany Ġabstract Ġand Ġfuzzy Ġmeanings , Ġe . g . Ġin ĠPolish Ġk l asa Ġhas Ġfour Ġmeanings Ġrelated Ġto ĠEnglish Ġclass : Ġ( 1 ) Ġgroup Ġof Ġsimilar Ġobjects , Ġ( 2 ) Ġlevel Ġof Ġquality , Ġ( 3 ) Ġgroup Ġof Ġpupils Ġor Ġ( 4 ) Ġclassroom . ĠThe Ġmeaning Ġ( 1 ) Ġcould Ġbe Ġhard Ġto Ġdistinguish Ġfrom Ġ( 2 ) Ġeven Ġfor Ġa Ġhuman , Ġwhereas Ġ( 3 ) Ġand Ġ( 4 ) Ġmay Ġappear Ġin Ġvery Ġsimilar Ġcontexts . </s> Finally , ĠTable ĠT AB REF 27 Ġcontains Ġthe Ġmean Ġaccuracy Ġof Ġthe Ġbasic Ġconfiguration Ġof Ġthe Ġclass ifier Ġdescribed Ġin Ġthis Ġwork Ġ( with Ġand Ġwithout Ġmodifications Ġto ĠBay esian Ġmodel ). ĠIt Ġis Ġcompared Ġto Ġthe Ġfour Ġpreviously Ġmentioned Ġbas elines . </s> Discussion </s> Although Ġmany Ġdifferent Ġconfigurations Ġhave Ġbeen Ġtested Ġin Ġthis Ġstudy , Ġall Ġof Ġthem Ġremain Ġbelow Ġthe Ġaccuracy Ġlevel Ġof Ġ80 %, Ġapproximately Ġequal Ġto Ġaverage Ġshare Ġof Ġdominating Ġsenses Ġin Ġthis Ġdataset . ĠThis Ġis Ġobviously Ġunsatisf actory Ġand Ġdemands Ġexplanation . </s> First Ġof Ġall , Ġthe Ġnew Ġsources Ġof Ġreplacements Ġproposed Ġin Ġthis Ġwork Ġindeed Ġseem Ġto Ġimprove Ġthe Ġmodels Ġfrom Ġ70 . 86 % Ġ( only Ġtraditional Ġmon ose m ous Ġrelatives ) Ġto Ġ77 . 96 % Ġ( all Ġproposed Ġrelatives ). ĠThe Ġbiggest Ġgain Ġis Ġobtained Ġby Ġincluding Ġthe Ġpoly sem ous Ġrelatives Ġtaking Ġinto Ġaccount Ġonly Ġtheir Ġfirst Ġmeaning . ĠThis Ġtechnique Ġrelies Ġon Ġtwo Ġassumptions : Ġa Ġstrong Ġdomination Ġof Ġone Ġof Ġthe Ġsenses Ġand Ġthat Ġsense Ġbeing Ġlisted Ġfirst Ġin Ġpl Word Net . ĠWhile Ġthe Ġformer Ġis Ġalmost Ġalways Ġtrue , Ġif Ġthe Ġsecond Ġassumption Ġis Ġfalse Ġthen Ġthe Ġcreated Ġmodel Ġare Ġadversely Ġaffected . ĠIn Ġthe Ġcase Ġof Ġtwo Ġtarget Ġwords Ġthe Ġsenses , Ġthe Ġfirst Ġsense Ġin Ġeach Ġcase Ġ( stop ie Å Ħ Ġas Ġa Ġmusical Ġconcept Ġand Ġform a Ġas Ġa Ġsyn onym Ġof Ġpo lyn omial ) Ġwas Ġso Ġpeculiar Ġthat Ġthey Ġwere Ġunknown Ġto Ġthe Ġauthor Ġof Ġthis Ġstudy Ġand Ġcouldn 't Ġbe Ġassigned Ġto Ġany Ġof Ġthe Ġcoarse - gr ained ĠN CP Ġsenses . ĠClearly , Ġnot Ġonly Ġthe Ġmethod Ġof Ġun super vised ĠW SD Ġusing Ġrelatives , Ġbut Ġalso Ġother Ġsolutions Ġrelated Ġto Ġpoly se my Ġwould Ġdefinitely Ġbenefit Ġfrom Ġa Ġreliable Ġordering Ġof Ġsenses Ġin Ġword nets , Ġespecially Ġas Ġincreasingly Ġuncommon Ġsenses Ġare Ġadded Ġto Ġthem Ġwith Ġtime . ĠIt Ġis Ġhowever Ġnot Ġclear Ġhow Ġsuch Ġknowledge Ġcould Ġbe Ġobtained Ġwithout Ġsolving Ġthe ĠW SD Ġtask Ġfirst . ĠWhat Ġis Ġmore , Ġsense Ġdistributions Ġobviously Ġchange Ġwith Ġgenre , Ġtime , Ġauthor , Ġetc . </s> When Ġit Ġcomes Ġto Ġfeature Ġselection , Ġthe Ġmost Ġunexpected Ġphenomenon Ġobserved Ġin Ġthis Ġstudy Ġis Ġlow Ġusefulness Ġof Ġthe Ġinterpretation - based Ġfeatures . ĠAccording Ġto ĠTable ĠT AB REF 22 Ġ, Ġadding Ġinterpretations Ġof Ġneighbouring Ġwords Ġ( ĠIN LINE FORM 0 Ġ) Ġyields Ġvery Ġlittle Ġimprovement , Ġwhile Ġthis Ġtype Ġof Ġinformation Ġregarding Ġreplacements Ġ( ĠIN LINE FORM 1 Ġ) Ġeven Ġlowers Ġthe Ġaccuracy . ĠThis Ġresult Ġcould Ġbe Ġattributed Ġto Ġtwo Ġfactors . ĠFirstly , Ġmore Ġdeveloped Ġreplacement Ġgeneration Ġresults Ġin Ġmore Ġoccurrences , Ġbut Ġalso Ġcauses Ġtheir Ġtags Ġto Ġdiffer Ġfrom Ġthe Ġtarget Ġword Ġby Ġgender Ġor Ġnumber . ĠThey Ġmay Ġeven Ġnot Ġbe Ġavailable Ġat Ġall Ġ( in Ġthe Ġcase Ġof Ġmulti - word Ġreplacements ). ĠThe Ġsecond Ġreason Ġis Ġa Ġdifference Ġin Ġlanguage : Ġwhile Ġin ĠEnglish Ġa Ġword Ġinterpretation Ġis Ġrepresented Ġas Ġone Ġof Ġseveral Ġdozen Ġpart Ġof Ġspeech Ġidentifiers , Ġin ĠSlav onic Ġlanguages , Ġsuch Ġas ĠPolish , Ġwe Ġneed Ġto Ġspecify Ġthe Ġvalues Ġof Ġseveral Ġtags Ġfor Ġeach Ġword , Ġleading Ġto Ġthousands Ġof Ġpossible Ġinterpretations . ĠObviously , Ġthe Ġfeatures Ġbased Ġon Ġthese Ġtags Ġare Ġvery Ġsparse . ĠFinally , Ġthe Ġmorph os ynt actic Ġannotation Ġwas Ġperformed Ġautomatically , Ġwhich Ġmay Ġlead Ġto Ġerrors , Ġespecially Ġin Ġthe Ġcase Ġof Ġnoisy Ġweb Ġtext . </s> One Ġof Ġthe Ġpurposes Ġof Ġthis Ġstudy Ġwas Ġto Ġcheck Ġthe Ġnecessary Ġamount Ġof Ġtraining Ġdata Ġfor Ġsuch Ġa Ġsolution Ġby Ġemploying Ġa Ġvery Ġlarge Ġcollection Ġfrom Ġthe ĠNE K ST Ġsearch Ġengine . ĠThe Ġneed Ġfor Ġlarge Ġcorpor a Ġis Ġobvious Ġwhen Ġusing Ġonly Ġmon ose m ous Ġrelatives ĠâĢĵ Ġthose Ġusually Ġrare Ġwords Ġshould Ġappear Ġin Ġmany Ġcontexts . ĠHowever , Ġaccording Ġto Ġthe Ġresults Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ, Ġthe Ġstrategy Ġfor Ġgenerating Ġrelatives Ġpresented Ġin Ġthis Ġpaper Ġreaches Ġoptimum Ġperformance Ġfor Ġa Ġreasonable Ġamount Ġof Ġtexts ĠâĢĵ Ġ6 Ġmillion Ġdocuments Ġis Ġenough . ĠHowever , Ġone Ġshould Ġkeep Ġin Ġmind Ġthat Ġthis Ġstatement Ġremains Ġtrue Ġassuming Ġa Ġconstant Ġevaluation Ġenvironment ; Ġexpanding Ġa Ġtest Ġset Ġ( currently Ġcontaining Ġ17 , 314 Ġoccurrences ) Ġmay Ġhelp Ġto Ġsee Ġdifferences Ġbetween Ġapparently Ġequivalent Ġmodels Ġand Ġraise Ġthe Ġneed Ġfor Ġbigger Ġcorpor a . </s> Con clusions </s> In Ġthis Ġpaper Ġthe Ġlimitations Ġand Ġimprovements Ġof Ġun super vised Ġword Ġsense Ġdis amb ig uation Ġhave Ġbeen Ġinvestigated . ĠThe Ġmain Ġproblem ĠâĢĵ Ġinsufficient Ġnumber Ġand Ġquality Ġof Ġreplacements Ġhas Ġbeen Ġtackled Ġby Ġadding Ġnew Ġrich Ġsources Ġof Ġreplacements . ĠThe Ġquality Ġof Ġthe Ġmodels Ġhas Ġindeed Ġimproved , Ġespecially Ġthanks Ġto Ġreplacements Ġbased Ġon Ġsense Ġordering Ġin Ġpl Word Net . ĠTo Ġdeal Ġwith Ġthe Ġproblem Ġof Ġunknown Ġsense Ġdistribution , Ġthe ĠBay esian Ġclass ifier Ġhas Ġbeen Ġmodified , Ġremoving Ġthe Ġbias Ġtowards Ġfrequent Ġlabels Ġin Ġthe Ġtraining Ġdata . ĠFinally , Ġthe Ġexperiments Ġwith Ġvery Ġlarge Ġcorpus Ġhave Ġshown Ġthe Ġsufficient Ġamount Ġof Ġtraining Ġdata Ġfor Ġthis Ġtask , Ġwhich Ġis Ġonly Ġ6 Ġmillion Ġdocuments . </s> Acknowled gements </s> This Ġstudy Ġwas Ġconducted Ġat ĠInstitute Ġof ĠComputer ĠScience , ĠPolish ĠAcademy Ġof ĠSciences , Ġand Ġsupported Ġby Ġa Ġresearch Ġfellowship Ġwithin Ġ" Information Ġtechnologies : Ġresearch Ġand Ġtheir Ġinter disciplinary Ġapplications " Ġagreement Ġnumber ĠP OK L . 04 . 01 . 01 - 00 - 05 1 / 10 - 00 . ĠThe Ġauthor Ġwould Ġlike Ġto Ġthank ĠDarius z ĠC zers ki Ġand ĠNE K ST Ġteam Ġfor Ġproviding Ġaccess Ġto Ġthe Ġsearch Ġengine Ġindex Ġand Ġhelpful Ġdiscussions Ġand ĠMatthew ĠShard low Ġfor Ġcomments Ġthat Ġgreatly Ġimproved Ġthe Ġmanuscript .