construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem
EXTRACTIVE
construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word
EXTRACTIVE
BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks. In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem.
In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system. Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering. Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI). We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task. In particular, our contribution is two-fold:
1908.07245
Introduction </s> Word ĠSense ĠDis amb ig uation Ġ( W SD ) Ġis Ġa Ġfundamental Ġtask Ġand Ġlong - standing Ġchallenge Ġin ĠNatural ĠLanguage ĠProcessing Ġ( N LP ), Ġwhich Ġaims Ġto Ġfind Ġthe Ġexact Ġsense Ġof Ġan Ġambiguous Ġword Ġin Ġa Ġparticular Ġcontext ĠB IB REF 0 . ĠPrevious ĠW SD Ġapproaches Ġcan Ġbe Ġgrouped Ġinto Ġtwo Ġmain Ġcategories : Ġknowledge - based Ġand Ġsupervised Ġmethods . </s> Know ledge - based ĠW SD Ġmethods Ġrely Ġon Ġlex ical Ġresources Ġlike ĠWord Net ĠB IB REF 1 Ġand Ġusually Ġexploit Ġtwo Ġkinds Ġof Ġlex ical Ġknowledge . ĠThe Ġgloss , Ġwhich Ġdefines Ġa Ġword Ġsense Ġmeaning , Ġis Ġfirst Ġutilized Ġin ĠLes k Ġalgorithm ĠB IB REF 2 Ġand Ġthen Ġwidely Ġtaken Ġinto Ġaccount Ġin Ġmany Ġother Ġapproaches ĠB IB REF 3 , ĠB IB REF 4 . ĠBesides , Ġstructural Ġproperties Ġof Ġsemantic Ġgraphs Ġare Ġmainly Ġused Ġin Ġgraph - based Ġalgorithms ĠB IB REF 5 , ĠB IB REF 6 . </s> Traditional Ġsupervised ĠW SD Ġmethods ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 Ġfocus Ġon Ġextracting Ġmanually Ġdesigned Ġfeatures Ġand Ġthen Ġtrain Ġa Ġdedicated Ġclass ifier Ġ( word Ġexpert ) Ġfor Ġevery Ġtarget Ġle mma . </s> Although Ġword Ġexpert Ġsupervised ĠW SD Ġmethods Ġperform Ġbetter , Ġthey Ġare Ġless Ġflexible Ġthan Ġknowledge - based Ġmethods Ġin Ġthe Ġall - words ĠW SD Ġtask ĠB IB REF 10 . ĠRecent Ġneural - based Ġmethods Ġare Ġdevoted Ġto Ġdealing Ġwith Ġthis Ġproblem . ĠB IB REF 11 Ġpresent Ġa Ġsupervised Ġclass ifier Ġbased Ġon ĠBi - L ST M , Ġwhich Ġshares Ġparameters Ġamong Ġall Ġword Ġtypes Ġexcept Ġthe Ġlast Ġlayer . ĠB IB REF 10 Ġconvert ĠW SD Ġtask Ġto Ġa Ġsequence Ġlabeling Ġtask , Ġthus Ġbuilding Ġa Ġunified Ġmodel Ġfor Ġall Ġpoly sem ous Ġwords . ĠHowever , Ġneither Ġof Ġthem Ġcan Ġtotally Ġbeat Ġthe Ġbest Ġword Ġexpert Ġsupervised Ġmethods . </s> More Ġrecently , ĠB IB REF 12 Ġpropose Ġto Ġleverage Ġthe Ġgloss Ġinformation Ġfrom ĠWord Net Ġand Ġmodel Ġthe Ġsemantic Ġrelationship Ġbetween Ġthe Ġcontext Ġand Ġgloss Ġin Ġan Ġimproved Ġmemory Ġnetwork . ĠSimilarly , ĠB IB REF 13 Ġintroduce Ġa Ġ( h ier arch ical ) Ġco - att ention Ġmechanism Ġto Ġgenerate Ġco - dependent Ġrepresentations Ġfor Ġthe Ġcontext Ġand Ġgloss . ĠTheir Ġattempts Ġprove Ġthat Ġincorporating Ġgloss Ġknowledge Ġinto Ġsupervised ĠW SD Ġapproach Ġis Ġhelpful , Ġbut Ġthey Ġstill Ġhave Ġnot Ġachieved Ġmuch Ġimprovement , Ġbecause Ġthey Ġmay Ġnot Ġmake Ġfull Ġuse Ġof Ġgloss Ġknowledge . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġhow Ġto Ġbetter Ġleverage Ġgloss Ġinformation Ġin Ġa Ġsupervised Ġneural ĠW SD Ġsystem . ĠRecently , Ġthe Ġpre - trained Ġlanguage Ġmodels , Ġsuch Ġas ĠEL Mo ĠB IB REF 14 Ġand ĠB ERT ĠB IB REF 15 , Ġhave Ġshown Ġtheir Ġeffectiveness Ġto Ġalleviate Ġthe Ġeffort Ġof Ġfeature Ġengineering . ĠEspecially , ĠB ERT Ġhas Ġachieved Ġexcellent Ġresults Ġin Ġquestion Ġanswering Ġ( Q A ) Ġand Ġnatural Ġlanguage Ġinference Ġ( N LI ). ĠWe Ġconstruct Ġcontext - gl oss Ġpairs Ġfrom Ġgloss es Ġof Ġall Ġpossible Ġsenses Ġ( in ĠWord Net ) Ġof Ġthe Ġtarget Ġword , Ġthus Ġtreating ĠW SD Ġtask Ġas Ġa Ġsentence - pair Ġclassification Ġproblem . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon ĠW SD Ġtask . ĠIn Ġparticular , Ġour Ġcontribution Ġis Ġtwo - fold : </s> 1 . ĠWe Ġconstruct Ġcontext - gl oss Ġpairs Ġand Ġpropose Ġthree ĠB ERT - based Ġmodels Ġfor ĠW SD . </s> 2 . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel , Ġand Ġthe Ġexperimental Ġresults Ġon Ġseveral ĠEnglish Ġall - words ĠW SD Ġbenchmark Ġdatasets Ġshow Ġthat Ġour Ġapproach Ġsignificantly Ġoutper forms Ġthe Ġstate - of - the - art Ġsystems . </s> Method ology </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġmethod Ġin Ġdetail . </s> Method ology Ġ: :: ĠTask ĠDefinition </s> In ĠW SD , Ġa Ġsentence Ġ$ s $ Ġusually Ġconsists Ġof Ġa Ġseries Ġof Ġwords : Ġ$\ l brace Ġw _ 1 , \ cd ots Ġ, w _ m \ r brace Ġ$ , Ġand Ġsome Ġof Ġthe Ġwords Ġ$\ l brace Ġw _{ i _ 1 }, \ cd ots Ġ, w _{ i _ k }\ r brace Ġ$ Ġare Ġtargets Ġ$\ l brace Ġt _ 1 , \ cd ots Ġ, t _ k \ r brace Ġ$ Ġneed Ġto Ġbe Ġdis amb ig uated . ĠFor Ġeach Ġtarget Ġ$ t $, Ġits Ġcandidate Ġsenses Ġ$\ l brace Ġc _ 1 , \ cd ots Ġ, c _ n \ r brace Ġ$ Ġcome Ġfrom Ġentries Ġof Ġits Ġle mma Ġin Ġa Ġpre - defined Ġsense Ġinventory Ġ( usually ĠWord Net ). ĠTherefore , ĠW SD Ġtask Ġaims Ġto Ġfind Ġthe Ġmost Ġsuitable Ġentry Ġ( sy mbol ized Ġas Ġunique Ġsense Ġkey ) Ġfor Ġeach Ġtarget Ġin Ġa Ġsentence . ĠSee Ġa Ġsentence Ġexample Ġin ĠTable ĠT AB REF 1 . </s> Method ology Ġ: :: ĠB ERT </s> BER T ĠB IB REF 15 Ġis Ġa Ġnew Ġlanguage Ġrepresentation Ġmodel , Ġand Ġits Ġarchitecture Ġis Ġa Ġmulti - layer Ġbid irection al ĠTrans former Ġenc oder . ĠB ERT Ġmodel Ġis Ġpre - trained Ġon Ġa Ġlarge Ġcorpus Ġand Ġtwo Ġnovel Ġun super vised Ġprediction Ġtasks , Ġi . e ., Ġmasked Ġlanguage Ġmodel Ġand Ġnext Ġsentence Ġprediction Ġtasks Ġare Ġused Ġin Ġpre - training . ĠWhen Ġincorporating ĠB ERT Ġinto Ġdownstream Ġtasks , Ġthe Ġfine - tun ing Ġprocedure Ġis Ġrecommended . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġon ĠW SD Ġtask . </s> Method ology Ġ: :: ĠB ERT Ġ: :: ĠB ERT ( Token - CL S ) </s> Since Ġevery Ġtarget Ġin Ġa Ġsentence Ġneeds Ġto Ġbe Ġdis amb ig uated Ġto Ġfind Ġits Ġexact Ġsense , ĠW SD Ġtask Ġcan Ġbe Ġregarded Ġas Ġa Ġtoken - level Ġclassification Ġtask . ĠTo Ġincorporate ĠB ERT Ġto ĠW SD Ġtask , Ġwe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġtoken Ġcorresponding Ġto Ġthe Ġtarget Ġword Ġ( if Ġmore Ġthan Ġone Ġtoken , Ġwe Ġaverage Ġthem ) Ġand Ġadd Ġa Ġclassification Ġlayer Ġfor Ġevery Ġtarget Ġle mma , Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġlast Ġlayer Ġof Ġthe ĠBi - L ST M Ġmodel ĠB IB REF 11 . </s> Method ology Ġ: :: ĠGloss BER T </s> BER T Ġcan Ġexplicitly Ġmodel Ġthe Ġrelationship Ġof Ġa Ġpair Ġof Ġtexts , Ġwhich Ġhas Ġshown Ġto Ġbe Ġbeneficial Ġto Ġmany Ġpair - wise Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠIn Ġorder Ġto Ġfully Ġleverage Ġgloss Ġinformation , Ġwe Ġpropose ĠGloss BER T Ġto Ġconstruct Ġcontext - gl oss Ġpairs Ġfrom Ġall Ġpossible Ġsenses Ġof Ġthe Ġtarget Ġword Ġin ĠWord Net , Ġthus Ġtreating ĠW SD Ġtask Ġas Ġa Ġsentence - pair Ġclassification Ġproblem . </s> We Ġdescribe Ġour Ġconstruction Ġmethod Ġwith Ġan Ġexample Ġ( See ĠTable ĠT AB REF 1 ). ĠThere Ġare Ġfour Ġtargets Ġin Ġthis Ġsentence , Ġand Ġhere Ġwe Ġtake Ġtarget Ġword Ġresearch Ġas Ġan Ġexample : </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠContext - Gl oss ĠP airs </s> The Ġsentence Ġcontaining Ġtarget Ġwords Ġis Ġden oted Ġas Ġcontext Ġsentence . ĠFor Ġeach Ġtarget Ġword , Ġwe Ġextract Ġgloss es Ġof Ġall Ġ$ N $ Ġpossible Ġsenses Ġ( here Ġ$ N = 4 $ ) Ġof Ġthe Ġtarget Ġword Ġ( research ) Ġin ĠWord Net Ġto Ġobtain Ġthe Ġgloss Ġsentence . Ġ[ CL S ] Ġand Ġ[ SE P ] Ġmarks Ġare Ġadded Ġto Ġthe Ġcontext - gl oss Ġpairs Ġto Ġmake Ġit Ġsuitable Ġfor Ġthe Ġinput Ġof ĠB ERT Ġmodel . ĠA Ġsimilar Ġidea Ġis Ġalso Ġused Ġin Ġaspect - based Ġsentiment Ġanalysis ĠB IB REF 16 . </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠContext - Gl oss ĠP airs Ġwith ĠWeak ĠSuper vision </s> Based Ġon Ġthe Ġprevious Ġconstruction Ġmethod , Ġwe Ġadd Ġweak Ġsupervised Ġsignals Ġto Ġthe Ġcontext - gl oss Ġpairs Ġ( see Ġthe Ġhighlighted Ġpart Ġin ĠTable ĠT AB REF 1 ). ĠThe Ġsignal Ġin Ġthe Ġgloss Ġsentence Ġaims Ġto Ġpoint Ġout Ġthe Ġtarget Ġword , Ġand Ġthe Ġsignal Ġin Ġthe Ġcontext Ġsentence Ġaims Ġto Ġemphasize Ġthe Ġtarget Ġword Ġconsidering Ġthe Ġsituation Ġthat Ġa Ġtarget Ġword Ġmay Ġoccur Ġmore Ġthan Ġone Ġtime Ġin Ġthe Ġsame Ġsentence . </s> Therefore , Ġeach Ġtarget Ġword Ġhas Ġ$ N $ Ġcontext - gl oss Ġpair Ġtraining Ġinstances Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ). ĠWhen Ġtesting , Ġwe Ġoutput Ġthe Ġprobability Ġof Ġ$ label = yes $ Ġof Ġeach Ġcontext - gl oss Ġpair Ġand Ġchoose Ġthe Ġsense Ġcorresponding Ġto Ġthe Ġhighest Ġprobability Ġas Ġthe Ġprediction Ġlabel Ġof Ġthe Ġtarget Ġword . ĠWe Ġexperiment Ġwith Ġthree ĠGloss BER T Ġmodels : </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Token - CL S ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġas Ġinput . ĠWe Ġhighlight Ġthe Ġtarget Ġword Ġby Ġtaking Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġtoken Ġcorresponding Ġto Ġthe Ġtarget Ġword Ġ( if Ġmore Ġthan Ġone Ġtoken , Ġwe Ġaverage Ġthem ) Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ). </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Sent - CL S ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġas Ġinput . ĠWe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġfirst Ġtoken Ġ[ CL S ] Ġas Ġthe Ġrepresentation Ġof Ġthe Ġwhole Ġsequence Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ), Ġwhich Ġdoes Ġnot Ġhighlight Ġthe Ġtarget Ġword . </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Sent - CL S - WS ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġwith Ġweak Ġsupervision Ġas Ġinput . ĠWe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġfirst Ġtoken Ġ[ CL S ] Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ), Ġwhich Ġweekly Ġhighlight Ġthe Ġtarget Ġword Ġby Ġthe Ġweak Ġsupervision . </s> Exper iments Ġ: :: ĠDat as ets </s> The Ġstatistics Ġof Ġthe ĠW SD Ġdatasets Ġare Ġshown Ġin ĠTable ĠT AB REF 12 . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠTraining ĠDat as et </s> Following Ġprevious Ġwork ĠB IB REF 13 , ĠB IB REF 12 , ĠB IB REF 10 , ĠB IB REF 17 , ĠB IB REF 9 , ĠB IB REF 7 , Ġwe Ġchoose ĠSem Cor 3 . 0 Ġas Ġtraining Ġcorpus , Ġwhich Ġis Ġthe Ġlargest Ġcorpus Ġmanually Ġannot ated Ġwith ĠWord Net Ġsense Ġfor ĠW SD . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠEvaluation ĠDat as ets </s> We Ġevaluate Ġour Ġmethod Ġon Ġseveral ĠEnglish Ġall - words ĠW SD Ġdatasets . ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġbenchmark Ġdatasets Ġproposed Ġby ĠB IB REF 17 Ġwhich Ġinclude Ġfive Ġstandard Ġall - words Ġfine - gr ained ĠW SD Ġdatasets Ġfrom Ġthe ĠSense val Ġand ĠSem E val Ġcompetitions : ĠSense val - 2 Ġ( SE 2 ), ĠSense val - 3 Ġ( SE 3 ), ĠSem E val - 2007 Ġ( SE 07 ), ĠSem E val - 2013 Ġ( SE 13 ) Ġand ĠSem E val - 2015 Ġ( SE 15 ). ĠFollowing ĠB IB REF 13 , ĠB IB REF 12 Ġand ĠB IB REF 10 , Ġwe Ġchoose ĠSE 07 , Ġthe Ġsmallest Ġamong Ġthese Ġtest Ġsets , Ġas Ġthe Ġdevelopment Ġset . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠWord Net </s> Since ĠB IB REF 17 Ġmap Ġall Ġthe Ġsense Ġannotations Ġin Ġthese Ġdatasets Ġfrom Ġtheir Ġoriginal Ġversions Ġto ĠWord Net Ġ3 . 0 , Ġwe Ġextract Ġword Ġsense Ġgloss es Ġfrom ĠWord Net Ġ3 . 0 . </s> Exper iments Ġ: :: ĠSettings </s> We Ġuse Ġthe Ġpre - trained Ġunc ased ĠB ERT $ _ \ math rm Ġ{ B ASE } $ Ġmodel Ġfor Ġfine - tun ing , Ġbecause Ġwe Ġfind Ġthat ĠB ERT $ _ \ math rm Ġ{ L AR GE } $ Ġmodel Ġperforms Ġslightly Ġworse Ġthan ĠB ERT $ _ \ math rm Ġ{ B ASE } $ Ġin Ġthis Ġtask . ĠThe Ġnumber Ġof ĠTrans former Ġblocks Ġis Ġ12 , Ġthe Ġnumber Ġof Ġthe Ġhidden Ġlayer Ġis Ġ768 , Ġthe Ġnumber Ġof Ġself - att ention Ġheads Ġis Ġ12 , Ġand Ġthe Ġtotal Ġnumber Ġof Ġparameters Ġof Ġthe Ġpre - trained Ġmodel Ġis Ġ110 M . ĠWhen Ġfine - tun ing , Ġwe Ġuse Ġthe Ġdevelopment Ġset Ġ( SE 07 ) Ġto Ġfind Ġthe Ġoptimal Ġsettings Ġfor Ġour Ġexperiments . ĠWe Ġkeep Ġthe Ġdrop out Ġprobability Ġat Ġ0 . 1 , Ġset Ġthe Ġnumber Ġof Ġepoch s Ġto Ġ4 . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ2 e - 5 , Ġand Ġthe Ġbatch Ġsize Ġis Ġ64 . </s> Exper iments Ġ: :: ĠResults </s> Table ĠT AB REF 19 Ġshows Ġthe Ġperformance Ġof Ġour Ġmethod Ġon Ġthe ĠEnglish Ġall - words ĠW SD Ġbenchmark Ġdatasets . ĠWe Ġcompare Ġour Ġapproach Ġwith Ġprevious Ġmethods . </s> The Ġfirst Ġblock Ġshows Ġthe ĠM FS Ġbaseline , Ġwhich Ġselects Ġthe Ġmost Ġfrequent Ġsense Ġin Ġthe Ġtraining Ġcorpus Ġfor Ġeach Ġtarget Ġword . </s> The Ġsecond Ġblock Ġshows Ġtwo Ġknowledge - based Ġsystems . ĠLes k $ _{ ext + emb } $ ĠB IB REF 4 Ġis Ġa Ġvariant Ġof ĠLes k Ġalgorithm ĠB IB REF 2 Ġby Ġcalculating Ġthe Ġgloss - context Ġoverlap Ġof Ġthe Ġtarget Ġword . ĠBab elf y ĠB IB REF 6 Ġis Ġa Ġunified Ġgraph - based Ġapproach Ġwhich Ġexploits Ġthe Ġsemantic Ġnetwork Ġstructure Ġfrom ĠBabel Net . </s> The Ġthird Ġblock Ġshows Ġtwo Ġword Ġexpert Ġtraditional Ġsupervised Ġsystems . ĠI MS ĠB IB REF 7 Ġis Ġa Ġflexible Ġframework Ġwhich Ġtrains ĠS VM Ġclass ifiers Ġand Ġuses Ġlocal Ġfeatures . ĠAnd ĠI MS $ _{ + emb } $ ĠB IB REF 9 Ġis Ġthe Ġbest Ġconfiguration Ġof Ġthe ĠI MS Ġframework , Ġwhich Ġalso Ġintegrates Ġword Ġembed d ings Ġas Ġfeatures . </s> The Ġfourth Ġblock Ġshows Ġseveral Ġrecent Ġneural - based Ġmethods . ĠBi - L ST M ĠB IB REF 11 Ġis Ġa Ġbaseline Ġfor Ġneural Ġmodels . ĠBi - L ST M $ _{ + Ġatt . Ġ+ ĠL EX Ġ+ ĠPOS } $ ĠB IB REF 10 Ġis Ġa Ġmulti - task Ġlearning Ġframework Ġfor ĠW SD , ĠPOS Ġtagging , Ġand ĠL EX Ġwith Ġself - att ention Ġmechanism , Ġwhich Ġconverts ĠW SD Ġto Ġa Ġsequence Ġlearning Ġtask . ĠG AS $ _{ ext } $ ĠB IB REF 12 Ġis Ġa Ġvariant Ġof ĠG AS Ġwhich Ġis Ġa Ġgloss - au gment ed Ġvariant Ġof Ġthe Ġmemory Ġnetwork Ġby Ġextending Ġgloss Ġknowledge . ĠCAN $ ^ s $ Ġand ĠHC AN ĠB IB REF 13 Ġare Ġsentence - level Ġand Ġhierarchical Ġco - att ention Ġneural Ġnetwork Ġmodels Ġwhich Ġleverage Ġgloss Ġknowledge . </s> In Ġthe Ġlast Ġblock , Ġwe Ġreport Ġthe Ġperformance Ġof Ġour Ġmethod . ĠB ERT ( Token - CL S ) Ġis Ġour Ġbaseline , Ġwhich Ġdoes Ġnot Ġincorporate Ġgloss Ġinformation , Ġand Ġit Ġperforms Ġslightly Ġworse Ġthan Ġprevious Ġtraditional Ġsupervised Ġmethods Ġand Ġrecent Ġneural - based Ġmethods . ĠIt Ġproves Ġthat Ġdirectly Ġusing ĠB ERT Ġcannot Ġobtain Ġperformance Ġgrowth . ĠThe Ġother Ġthree Ġmethods Ġoutper form Ġother Ġmodels Ġby Ġa Ġsubstantial Ġmargin , Ġwhich Ġproves Ġthat Ġthe Ġimprovements Ġcome Ġfrom Ġleveraging ĠB ERT Ġto Ġbetter Ġexploit Ġgloss Ġinformation . ĠIt Ġis Ġworth Ġnoting Ġthat Ġour Ġmethod Ġachieves Ġsignificant Ġimprovements Ġin ĠSE 07 Ġand ĠVerb Ġover Ġprevious Ġmethods , Ġwhich Ġhave Ġthe Ġhighest Ġambiguity Ġlevel Ġamong Ġall Ġdatasets Ġand Ġall ĠPOS Ġtags Ġrespectively Ġaccording Ġto ĠB IB REF 17 . </s> Moreover , ĠGloss BER T ( Token - CL S ) Ġperforms Ġbetter Ġthan ĠGloss BER T ( Sent - CL S ), Ġwhich Ġproves Ġthat Ġhighlighting Ġthe Ġtarget Ġword Ġin Ġthe Ġsentence Ġis Ġimportant . ĠHowever , Ġthe Ġweak ly Ġhighlighting Ġmethod ĠGloss BER T ( Sent - CL S - WS ) Ġperforms Ġbest Ġin Ġmost Ġcircumstances , Ġwhich Ġmay Ġresult Ġfrom Ġits Ġcombination Ġof Ġthe Ġadvantages Ġof Ġthe Ġother Ġtwo Ġmethods . </s> Exper iments Ġ: :: ĠDiscussion </s> There Ġare Ġtwo Ġmain Ġreasons Ġfor Ġthe Ġgreat Ġimprovements Ġof Ġour Ġexperimental Ġresults . ĠFirst , Ġwe Ġconstruct Ġcontext - gl oss Ġpairs Ġand Ġconvert ĠW SD Ġproblem Ġto Ġa Ġsentence - pair Ġclassification Ġtask Ġwhich Ġis Ġsimilar Ġto ĠNL I Ġtasks Ġand Ġtrain Ġonly Ġone Ġclass ifier , Ġwhich Ġis Ġequivalent Ġto Ġexpanding Ġthe Ġcorpus . ĠSecond , Ġwe Ġleverage ĠB ERT ĠB IB REF 15 Ġto Ġbetter Ġexploit Ġthe Ġgloss Ġinformation . ĠB ERT Ġmodel Ġshows Ġits Ġadvantage Ġin Ġdealing Ġwith Ġsentence - pair Ġclassification Ġtasks Ġby Ġits Ġamazing Ġimprovement Ġon ĠQ A Ġand ĠNL I Ġtasks . ĠThis Ġadvantage Ġcomes Ġfrom Ġboth Ġof Ġits Ġtwo Ġnovel Ġun super vised Ġprediction Ġtasks . </s> Compared Ġwith Ġtraditional Ġword Ġexpert Ġsupervised Ġmethods , Ġour ĠGloss BER T Ġshows Ġits Ġeffectiveness Ġto Ġalleviate Ġthe Ġeffort Ġof Ġfeature Ġengineering Ġand Ġdoes Ġnot Ġrequire Ġtraining Ġa Ġdedicated Ġclass ifier Ġfor Ġevery Ġtarget Ġle mma . ĠUp Ġto Ġnow , Ġit Ġcan Ġbe Ġsaid Ġthat Ġthe Ġneural Ġnetwork Ġmethod Ġcan Ġtotally Ġbeat Ġthe Ġtraditional Ġword Ġexpert Ġmethod . ĠCompared Ġwith Ġrecent Ġneural - based Ġmethods , Ġour Ġsolution Ġis Ġmore Ġintuitive Ġand Ġcan Ġmake Ġbetter Ġuse Ġof Ġgloss Ġknowledge . ĠBesides , Ġour Ġapproach Ġdemonstrates Ġthat Ġwhen Ġwe Ġfine - t une ĠB ERT Ġon Ġa Ġdownstream Ġtask , Ġconverting Ġit Ġinto Ġa Ġsentence - pair Ġclassification Ġtask Ġmay Ġbe Ġa Ġgood Ġchoice . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġseek Ġto Ġbetter Ġleverage Ġgloss Ġknowledge Ġin Ġa Ġsupervised Ġneural ĠW SD Ġsystem . ĠWe Ġpropose Ġa Ġnew Ġsolution Ġto ĠW SD Ġby Ġconstructing Ġcontext - gl oss Ġpairs Ġand Ġthen Ġconverting ĠW SD Ġto Ġa Ġsentence - pair Ġclassification Ġtask . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon ĠW SD Ġtask . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġcomments . ĠThe Ġresearch Ġwork Ġis Ġsupported Ġby ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( No . Ġ6 175 12 01 Ġand Ġ6 16 72 162 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠCommission Ġ( 16 JC 14 204 01 Ġand Ġ17 JC 14 04 100 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠMajor ĠProject Ġ( No . 2018 SH Z D ZX 01 ) Ġand ĠZ J Lab .
Introduction Word Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0. Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods. Knowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge. The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4. Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6. Traditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma. Although word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10. Recent neural-based methods are devoted to dealing with this problem. BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer. BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words. However, neither of them can totally beat the best word expert supervised methods. More recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network. Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss. Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge. In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system. Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering. Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI). We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task. In particular, our contribution is two-fold: 1. We construct context-gloss pairs and propose three BERT-based models for WSD. 2. We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems. Methodology In this section, we describe our method in detail. Methodology ::: Task Definition In WSD, a sentence $s$ usually consists of a series of words: $\lbrace w_1,\cdots ,w_m\rbrace $, and some of the words $\lbrace w_{i_1},\cdots ,w_{i_k}\rbrace $ are targets $\lbrace t_1,\cdots ,t_k\rbrace $ need to be disambiguated. For each target $t$, its candidate senses $\lbrace c_1,\cdots ,c_n\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet). Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence. See a sentence example in Table TABREF1. Methodology ::: BERT BERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder. BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i.e., masked language model and next sentence prediction tasks are used in pre-training. When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended. We fine-tune the pre-trained BERT model on WSD task. Methodology ::: BERT ::: BERT(Token-CLS) Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task. To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11. Methodology ::: GlossBERT BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks. In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem. We describe our construction method with an example (See Table TABREF1). There are four targets in this sentence, and here we take target word research as an example: Methodology ::: GlossBERT ::: Context-Gloss Pairs The sentence containing target words is denoted as context sentence. For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence. [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model. A similar idea is also used in aspect-based sentiment analysis BIBREF16. Methodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision Based on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1). The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence. Therefore, each target word has $N$ context-gloss pair training instances ($label\in \lbrace yes, no\rbrace $). When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word. We experiment with three GlossBERT models: Methodology ::: GlossBERT ::: GlossBERT(Token-CLS) We use context-gloss pairs as input. We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\in \lbrace yes, no\rbrace $). Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS) We use context-gloss pairs as input. We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\in \lbrace yes, no\rbrace $), which does not highlight the target word. Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS) We use context-gloss pairs with weak supervision as input. We take the final hidden state of the first token [CLS] and add a classification layer ($label\in \lbrace yes, no\rbrace $), which weekly highlight the target word by the weak supervision. Experiments ::: Datasets The statistics of the WSD datasets are shown in Table TABREF12. Experiments ::: Datasets ::: Training Dataset Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3.0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD. Experiments ::: Datasets ::: Evaluation Datasets We evaluate our method on several English all-words WSD datasets. For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15). Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set. Experiments ::: Datasets ::: WordNet Since BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3.0, we extract word sense glosses from WordNet 3.0. Experiments ::: Settings We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task. The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M. When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments. We keep the dropout probability at 0.1, set the number of epochs to 4. The initial learning rate is 2e-5, and the batch size is 64. Experiments ::: Results Table TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets. We compare our approach with previous methods. The first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word. The second block shows two knowledge-based systems. Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word. Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet. The third block shows two word expert traditional supervised systems. IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features. And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features. The fourth block shows several recent neural-based methods. Bi-LSTM BIBREF11 is a baseline for neural models. Bi-LSTM$_{+ att. + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task. GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge. CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge. In the last block, we report the performance of our method. BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods. It proves that directly using BERT cannot obtain performance growth. The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information. It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17. Moreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important. However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods. Experiments ::: Discussion There are two main reasons for the great improvements of our experimental results. First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus. Second, we leverage BERT BIBREF15 to better exploit the gloss information. BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks. This advantage comes from both of its two novel unsupervised prediction tasks. Compared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma. Up to now, it can be said that the neural network method can totally beat the traditional word expert method. Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge. Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice. Conclusion In this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system. We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task. Acknowledgments We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by National Natural Science Foundation of China (No. 61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01) and ZJLab.
Do they incoprorate WordNet into the model?
7ab9c0b4ceca1c142ff068f85015a249b14282d0
Do Ġthey Ġinc op ror ate ĠWord Net Ġinto Ġthe Ġmodel ?
construct Ġcontext - gl oss Ġpairs Ġfrom Ġall Ġpossible Ġsenses Ġof Ġthe Ġtarget Ġword Ġin ĠWord Net , Ġthus Ġtreating ĠW SD Ġtask Ġas Ġa Ġsentence - pair Ġclassification Ġproblem
<s> Do Ġthey Ġinc op ror ate ĠWord Net Ġinto Ġthe Ġmodel ? </s> Introduction </s> Word ĠSense ĠDis amb ig uation Ġ( W SD ) Ġis Ġa Ġfundamental Ġtask Ġand Ġlong - standing Ġchallenge Ġin ĠNatural ĠLanguage ĠProcessing Ġ( N LP ), Ġwhich Ġaims Ġto Ġfind Ġthe Ġexact Ġsense Ġof Ġan Ġambiguous Ġword Ġin Ġa Ġparticular Ġcontext ĠB IB REF 0 . ĠPrevious ĠW SD Ġapproaches Ġcan Ġbe Ġgrouped Ġinto Ġtwo Ġmain Ġcategories : Ġknowledge - based Ġand Ġsupervised Ġmethods . </s> Know ledge - based ĠW SD Ġmethods Ġrely Ġon Ġlex ical Ġresources Ġlike ĠWord Net ĠB IB REF 1 Ġand Ġusually Ġexploit Ġtwo Ġkinds Ġof Ġlex ical Ġknowledge . ĠThe Ġgloss , Ġwhich Ġdefines Ġa Ġword Ġsense Ġmeaning , Ġis Ġfirst Ġutilized Ġin ĠLes k Ġalgorithm ĠB IB REF 2 Ġand Ġthen Ġwidely Ġtaken Ġinto Ġaccount Ġin Ġmany Ġother Ġapproaches ĠB IB REF 3 , ĠB IB REF 4 . ĠBesides , Ġstructural Ġproperties Ġof Ġsemantic Ġgraphs Ġare Ġmainly Ġused Ġin Ġgraph - based Ġalgorithms ĠB IB REF 5 , ĠB IB REF 6 . </s> Traditional Ġsupervised ĠW SD Ġmethods ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 Ġfocus Ġon Ġextracting Ġmanually Ġdesigned Ġfeatures Ġand Ġthen Ġtrain Ġa Ġdedicated Ġclass ifier Ġ( word Ġexpert ) Ġfor Ġevery Ġtarget Ġle mma . </s> Although Ġword Ġexpert Ġsupervised ĠW SD Ġmethods Ġperform Ġbetter , Ġthey Ġare Ġless Ġflexible Ġthan Ġknowledge - based Ġmethods Ġin Ġthe Ġall - words ĠW SD Ġtask ĠB IB REF 10 . ĠRecent Ġneural - based Ġmethods Ġare Ġdevoted Ġto Ġdealing Ġwith Ġthis Ġproblem . ĠB IB REF 11 Ġpresent Ġa Ġsupervised Ġclass ifier Ġbased Ġon ĠBi - L ST M , Ġwhich Ġshares Ġparameters Ġamong Ġall Ġword Ġtypes Ġexcept Ġthe Ġlast Ġlayer . ĠB IB REF 10 Ġconvert ĠW SD Ġtask Ġto Ġa Ġsequence Ġlabeling Ġtask , Ġthus Ġbuilding Ġa Ġunified Ġmodel Ġfor Ġall Ġpoly sem ous Ġwords . ĠHowever , Ġneither Ġof Ġthem Ġcan Ġtotally Ġbeat Ġthe Ġbest Ġword Ġexpert Ġsupervised Ġmethods . </s> More Ġrecently , ĠB IB REF 12 Ġpropose Ġto Ġleverage Ġthe Ġgloss Ġinformation Ġfrom ĠWord Net Ġand Ġmodel Ġthe Ġsemantic Ġrelationship Ġbetween Ġthe Ġcontext Ġand Ġgloss Ġin Ġan Ġimproved Ġmemory Ġnetwork . ĠSimilarly , ĠB IB REF 13 Ġintroduce Ġa Ġ( h ier arch ical ) Ġco - att ention Ġmechanism Ġto Ġgenerate Ġco - dependent Ġrepresentations Ġfor Ġthe Ġcontext Ġand Ġgloss . ĠTheir Ġattempts Ġprove Ġthat Ġincorporating Ġgloss Ġknowledge Ġinto Ġsupervised ĠW SD Ġapproach Ġis Ġhelpful , Ġbut Ġthey Ġstill Ġhave Ġnot Ġachieved Ġmuch Ġimprovement , Ġbecause Ġthey Ġmay Ġnot Ġmake Ġfull Ġuse Ġof Ġgloss Ġknowledge . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġhow Ġto Ġbetter Ġleverage Ġgloss Ġinformation Ġin Ġa Ġsupervised Ġneural ĠW SD Ġsystem . ĠRecently , Ġthe Ġpre - trained Ġlanguage Ġmodels , Ġsuch Ġas ĠEL Mo ĠB IB REF 14 Ġand ĠB ERT ĠB IB REF 15 , Ġhave Ġshown Ġtheir Ġeffectiveness Ġto Ġalleviate Ġthe Ġeffort Ġof Ġfeature Ġengineering . ĠEspecially , ĠB ERT Ġhas Ġachieved Ġexcellent Ġresults Ġin Ġquestion Ġanswering Ġ( Q A ) Ġand Ġnatural Ġlanguage Ġinference Ġ( N LI ). ĠWe Ġconstruct Ġcontext - gl oss Ġpairs Ġfrom Ġgloss es Ġof Ġall Ġpossible Ġsenses Ġ( in ĠWord Net ) Ġof Ġthe Ġtarget Ġword , Ġthus Ġtreating ĠW SD Ġtask Ġas Ġa Ġsentence - pair Ġclassification Ġproblem . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon ĠW SD Ġtask . ĠIn Ġparticular , Ġour Ġcontribution Ġis Ġtwo - fold : </s> 1 . ĠWe Ġconstruct Ġcontext - gl oss Ġpairs Ġand Ġpropose Ġthree ĠB ERT - based Ġmodels Ġfor ĠW SD . </s> 2 . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel , Ġand Ġthe Ġexperimental Ġresults Ġon Ġseveral ĠEnglish Ġall - words ĠW SD Ġbenchmark Ġdatasets Ġshow Ġthat Ġour Ġapproach Ġsignificantly Ġoutper forms Ġthe Ġstate - of - the - art Ġsystems . </s> Method ology </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġmethod Ġin Ġdetail . </s> Method ology Ġ: :: ĠTask ĠDefinition </s> In ĠW SD , Ġa Ġsentence Ġ$ s $ Ġusually Ġconsists Ġof Ġa Ġseries Ġof Ġwords : Ġ$\ l brace Ġw _ 1 , \ cd ots Ġ, w _ m \ r brace Ġ$ , Ġand Ġsome Ġof Ġthe Ġwords Ġ$\ l brace Ġw _{ i _ 1 }, \ cd ots Ġ, w _{ i _ k }\ r brace Ġ$ Ġare Ġtargets Ġ$\ l brace Ġt _ 1 , \ cd ots Ġ, t _ k \ r brace Ġ$ Ġneed Ġto Ġbe Ġdis amb ig uated . ĠFor Ġeach Ġtarget Ġ$ t $, Ġits Ġcandidate Ġsenses Ġ$\ l brace Ġc _ 1 , \ cd ots Ġ, c _ n \ r brace Ġ$ Ġcome Ġfrom Ġentries Ġof Ġits Ġle mma Ġin Ġa Ġpre - defined Ġsense Ġinventory Ġ( usually ĠWord Net ). ĠTherefore , ĠW SD Ġtask Ġaims Ġto Ġfind Ġthe Ġmost Ġsuitable Ġentry Ġ( sy mbol ized Ġas Ġunique Ġsense Ġkey ) Ġfor Ġeach Ġtarget Ġin Ġa Ġsentence . ĠSee Ġa Ġsentence Ġexample Ġin ĠTable ĠT AB REF 1 . </s> Method ology Ġ: :: ĠB ERT </s> BER T ĠB IB REF 15 Ġis Ġa Ġnew Ġlanguage Ġrepresentation Ġmodel , Ġand Ġits Ġarchitecture Ġis Ġa Ġmulti - layer Ġbid irection al ĠTrans former Ġenc oder . ĠB ERT Ġmodel Ġis Ġpre - trained Ġon Ġa Ġlarge Ġcorpus Ġand Ġtwo Ġnovel Ġun super vised Ġprediction Ġtasks , Ġi . e ., Ġmasked Ġlanguage Ġmodel Ġand Ġnext Ġsentence Ġprediction Ġtasks Ġare Ġused Ġin Ġpre - training . ĠWhen Ġincorporating ĠB ERT Ġinto Ġdownstream Ġtasks , Ġthe Ġfine - tun ing Ġprocedure Ġis Ġrecommended . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġon ĠW SD Ġtask . </s> Method ology Ġ: :: ĠB ERT Ġ: :: ĠB ERT ( Token - CL S ) </s> Since Ġevery Ġtarget Ġin Ġa Ġsentence Ġneeds Ġto Ġbe Ġdis amb ig uated Ġto Ġfind Ġits Ġexact Ġsense , ĠW SD Ġtask Ġcan Ġbe Ġregarded Ġas Ġa Ġtoken - level Ġclassification Ġtask . ĠTo Ġincorporate ĠB ERT Ġto ĠW SD Ġtask , Ġwe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġtoken Ġcorresponding Ġto Ġthe Ġtarget Ġword Ġ( if Ġmore Ġthan Ġone Ġtoken , Ġwe Ġaverage Ġthem ) Ġand Ġadd Ġa Ġclassification Ġlayer Ġfor Ġevery Ġtarget Ġle mma , Ġwhich Ġis Ġthe Ġsame Ġas Ġthe Ġlast Ġlayer Ġof Ġthe ĠBi - L ST M Ġmodel ĠB IB REF 11 . </s> Method ology Ġ: :: ĠGloss BER T </s> BER T Ġcan Ġexplicitly Ġmodel Ġthe Ġrelationship Ġof Ġa Ġpair Ġof Ġtexts , Ġwhich Ġhas Ġshown Ġto Ġbe Ġbeneficial Ġto Ġmany Ġpair - wise Ġnatural Ġlanguage Ġunderstanding Ġtasks . ĠIn Ġorder Ġto Ġfully Ġleverage Ġgloss Ġinformation , Ġwe Ġpropose ĠGloss BER T Ġto Ġconstruct Ġcontext - gl oss Ġpairs Ġfrom Ġall Ġpossible Ġsenses Ġof Ġthe Ġtarget Ġword Ġin ĠWord Net , Ġthus Ġtreating ĠW SD Ġtask Ġas Ġa Ġsentence - pair Ġclassification Ġproblem . </s> We Ġdescribe Ġour Ġconstruction Ġmethod Ġwith Ġan Ġexample Ġ( See ĠTable ĠT AB REF 1 ). ĠThere Ġare Ġfour Ġtargets Ġin Ġthis Ġsentence , Ġand Ġhere Ġwe Ġtake Ġtarget Ġword Ġresearch Ġas Ġan Ġexample : </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠContext - Gl oss ĠP airs </s> The Ġsentence Ġcontaining Ġtarget Ġwords Ġis Ġden oted Ġas Ġcontext Ġsentence . ĠFor Ġeach Ġtarget Ġword , Ġwe Ġextract Ġgloss es Ġof Ġall Ġ$ N $ Ġpossible Ġsenses Ġ( here Ġ$ N = 4 $ ) Ġof Ġthe Ġtarget Ġword Ġ( research ) Ġin ĠWord Net Ġto Ġobtain Ġthe Ġgloss Ġsentence . Ġ[ CL S ] Ġand Ġ[ SE P ] Ġmarks Ġare Ġadded Ġto Ġthe Ġcontext - gl oss Ġpairs Ġto Ġmake Ġit Ġsuitable Ġfor Ġthe Ġinput Ġof ĠB ERT Ġmodel . ĠA Ġsimilar Ġidea Ġis Ġalso Ġused Ġin Ġaspect - based Ġsentiment Ġanalysis ĠB IB REF 16 . </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠContext - Gl oss ĠP airs Ġwith ĠWeak ĠSuper vision </s> Based Ġon Ġthe Ġprevious Ġconstruction Ġmethod , Ġwe Ġadd Ġweak Ġsupervised Ġsignals Ġto Ġthe Ġcontext - gl oss Ġpairs Ġ( see Ġthe Ġhighlighted Ġpart Ġin ĠTable ĠT AB REF 1 ). ĠThe Ġsignal Ġin Ġthe Ġgloss Ġsentence Ġaims Ġto Ġpoint Ġout Ġthe Ġtarget Ġword , Ġand Ġthe Ġsignal Ġin Ġthe Ġcontext Ġsentence Ġaims Ġto Ġemphasize Ġthe Ġtarget Ġword Ġconsidering Ġthe Ġsituation Ġthat Ġa Ġtarget Ġword Ġmay Ġoccur Ġmore Ġthan Ġone Ġtime Ġin Ġthe Ġsame Ġsentence . </s> Therefore , Ġeach Ġtarget Ġword Ġhas Ġ$ N $ Ġcontext - gl oss Ġpair Ġtraining Ġinstances Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ). ĠWhen Ġtesting , Ġwe Ġoutput Ġthe Ġprobability Ġof Ġ$ label = yes $ Ġof Ġeach Ġcontext - gl oss Ġpair Ġand Ġchoose Ġthe Ġsense Ġcorresponding Ġto Ġthe Ġhighest Ġprobability Ġas Ġthe Ġprediction Ġlabel Ġof Ġthe Ġtarget Ġword . ĠWe Ġexperiment Ġwith Ġthree ĠGloss BER T Ġmodels : </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Token - CL S ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġas Ġinput . ĠWe Ġhighlight Ġthe Ġtarget Ġword Ġby Ġtaking Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġtoken Ġcorresponding Ġto Ġthe Ġtarget Ġword Ġ( if Ġmore Ġthan Ġone Ġtoken , Ġwe Ġaverage Ġthem ) Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ). </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Sent - CL S ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġas Ġinput . ĠWe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġfirst Ġtoken Ġ[ CL S ] Ġas Ġthe Ġrepresentation Ġof Ġthe Ġwhole Ġsequence Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ), Ġwhich Ġdoes Ġnot Ġhighlight Ġthe Ġtarget Ġword . </s> Method ology Ġ: :: ĠGloss BER T Ġ: :: ĠGloss BER T ( Sent - CL S - WS ) </s> We Ġuse Ġcontext - gl oss Ġpairs Ġwith Ġweak Ġsupervision Ġas Ġinput . ĠWe Ġtake Ġthe Ġfinal Ġhidden Ġstate Ġof Ġthe Ġfirst Ġtoken Ġ[ CL S ] Ġand Ġadd Ġa Ġclassification Ġlayer Ġ($ label \ in Ġ\ l brace Ġyes , Ġno \ r brace Ġ$ ), Ġwhich Ġweekly Ġhighlight Ġthe Ġtarget Ġword Ġby Ġthe Ġweak Ġsupervision . </s> Exper iments Ġ: :: ĠDat as ets </s> The Ġstatistics Ġof Ġthe ĠW SD Ġdatasets Ġare Ġshown Ġin ĠTable ĠT AB REF 12 . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠTraining ĠDat as et </s> Following Ġprevious Ġwork ĠB IB REF 13 , ĠB IB REF 12 , ĠB IB REF 10 , ĠB IB REF 17 , ĠB IB REF 9 , ĠB IB REF 7 , Ġwe Ġchoose ĠSem Cor 3 . 0 Ġas Ġtraining Ġcorpus , Ġwhich Ġis Ġthe Ġlargest Ġcorpus Ġmanually Ġannot ated Ġwith ĠWord Net Ġsense Ġfor ĠW SD . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠEvaluation ĠDat as ets </s> We Ġevaluate Ġour Ġmethod Ġon Ġseveral ĠEnglish Ġall - words ĠW SD Ġdatasets . ĠFor Ġa Ġfair Ġcomparison , Ġwe Ġuse Ġthe Ġbenchmark Ġdatasets Ġproposed Ġby ĠB IB REF 17 Ġwhich Ġinclude Ġfive Ġstandard Ġall - words Ġfine - gr ained ĠW SD Ġdatasets Ġfrom Ġthe ĠSense val Ġand ĠSem E val Ġcompetitions : ĠSense val - 2 Ġ( SE 2 ), ĠSense val - 3 Ġ( SE 3 ), ĠSem E val - 2007 Ġ( SE 07 ), ĠSem E val - 2013 Ġ( SE 13 ) Ġand ĠSem E val - 2015 Ġ( SE 15 ). ĠFollowing ĠB IB REF 13 , ĠB IB REF 12 Ġand ĠB IB REF 10 , Ġwe Ġchoose ĠSE 07 , Ġthe Ġsmallest Ġamong Ġthese Ġtest Ġsets , Ġas Ġthe Ġdevelopment Ġset . </s> Exper iments Ġ: :: ĠDat as ets Ġ: :: ĠWord Net </s> Since ĠB IB REF 17 Ġmap Ġall Ġthe Ġsense Ġannotations Ġin Ġthese Ġdatasets Ġfrom Ġtheir Ġoriginal Ġversions Ġto ĠWord Net Ġ3 . 0 , Ġwe Ġextract Ġword Ġsense Ġgloss es Ġfrom ĠWord Net Ġ3 . 0 . </s> Exper iments Ġ: :: ĠSettings </s> We Ġuse Ġthe Ġpre - trained Ġunc ased ĠB ERT $ _ \ math rm Ġ{ B ASE } $ Ġmodel Ġfor Ġfine - tun ing , Ġbecause Ġwe Ġfind Ġthat ĠB ERT $ _ \ math rm Ġ{ L AR GE } $ Ġmodel Ġperforms Ġslightly Ġworse Ġthan ĠB ERT $ _ \ math rm Ġ{ B ASE } $ Ġin Ġthis Ġtask . ĠThe Ġnumber Ġof ĠTrans former Ġblocks Ġis Ġ12 , Ġthe Ġnumber Ġof Ġthe Ġhidden Ġlayer Ġis Ġ768 , Ġthe Ġnumber Ġof Ġself - att ention Ġheads Ġis Ġ12 , Ġand Ġthe Ġtotal Ġnumber Ġof Ġparameters Ġof Ġthe Ġpre - trained Ġmodel Ġis Ġ110 M . ĠWhen Ġfine - tun ing , Ġwe Ġuse Ġthe Ġdevelopment Ġset Ġ( SE 07 ) Ġto Ġfind Ġthe Ġoptimal Ġsettings Ġfor Ġour Ġexperiments . ĠWe Ġkeep Ġthe Ġdrop out Ġprobability Ġat Ġ0 . 1 , Ġset Ġthe Ġnumber Ġof Ġepoch s Ġto Ġ4 . ĠThe Ġinitial Ġlearning Ġrate Ġis Ġ2 e - 5 , Ġand Ġthe Ġbatch Ġsize Ġis Ġ64 . </s> Exper iments Ġ: :: ĠResults </s> Table ĠT AB REF 19 Ġshows Ġthe Ġperformance Ġof Ġour Ġmethod Ġon Ġthe ĠEnglish Ġall - words ĠW SD Ġbenchmark Ġdatasets . ĠWe Ġcompare Ġour Ġapproach Ġwith Ġprevious Ġmethods . </s> The Ġfirst Ġblock Ġshows Ġthe ĠM FS Ġbaseline , Ġwhich Ġselects Ġthe Ġmost Ġfrequent Ġsense Ġin Ġthe Ġtraining Ġcorpus Ġfor Ġeach Ġtarget Ġword . </s> The Ġsecond Ġblock Ġshows Ġtwo Ġknowledge - based Ġsystems . ĠLes k $ _{ ext + emb } $ ĠB IB REF 4 Ġis Ġa Ġvariant Ġof ĠLes k Ġalgorithm ĠB IB REF 2 Ġby Ġcalculating Ġthe Ġgloss - context Ġoverlap Ġof Ġthe Ġtarget Ġword . ĠBab elf y ĠB IB REF 6 Ġis Ġa Ġunified Ġgraph - based Ġapproach Ġwhich Ġexploits Ġthe Ġsemantic Ġnetwork Ġstructure Ġfrom ĠBabel Net . </s> The Ġthird Ġblock Ġshows Ġtwo Ġword Ġexpert Ġtraditional Ġsupervised Ġsystems . ĠI MS ĠB IB REF 7 Ġis Ġa Ġflexible Ġframework Ġwhich Ġtrains ĠS VM Ġclass ifiers Ġand Ġuses Ġlocal Ġfeatures . ĠAnd ĠI MS $ _{ + emb } $ ĠB IB REF 9 Ġis Ġthe Ġbest Ġconfiguration Ġof Ġthe ĠI MS Ġframework , Ġwhich Ġalso Ġintegrates Ġword Ġembed d ings Ġas Ġfeatures . </s> The Ġfourth Ġblock Ġshows Ġseveral Ġrecent Ġneural - based Ġmethods . ĠBi - L ST M ĠB IB REF 11 Ġis Ġa Ġbaseline Ġfor Ġneural Ġmodels . ĠBi - L ST M $ _{ + Ġatt . Ġ+ ĠL EX Ġ+ ĠPOS } $ ĠB IB REF 10 Ġis Ġa Ġmulti - task Ġlearning Ġframework Ġfor ĠW SD , ĠPOS Ġtagging , Ġand ĠL EX Ġwith Ġself - att ention Ġmechanism , Ġwhich Ġconverts ĠW SD Ġto Ġa Ġsequence Ġlearning Ġtask . ĠG AS $ _{ ext } $ ĠB IB REF 12 Ġis Ġa Ġvariant Ġof ĠG AS Ġwhich Ġis Ġa Ġgloss - au gment ed Ġvariant Ġof Ġthe Ġmemory Ġnetwork Ġby Ġextending Ġgloss Ġknowledge . ĠCAN $ ^ s $ Ġand ĠHC AN ĠB IB REF 13 Ġare Ġsentence - level Ġand Ġhierarchical Ġco - att ention Ġneural Ġnetwork Ġmodels Ġwhich Ġleverage Ġgloss Ġknowledge . </s> In Ġthe Ġlast Ġblock , Ġwe Ġreport Ġthe Ġperformance Ġof Ġour Ġmethod . ĠB ERT ( Token - CL S ) Ġis Ġour Ġbaseline , Ġwhich Ġdoes Ġnot Ġincorporate Ġgloss Ġinformation , Ġand Ġit Ġperforms Ġslightly Ġworse Ġthan Ġprevious Ġtraditional Ġsupervised Ġmethods Ġand Ġrecent Ġneural - based Ġmethods . ĠIt Ġproves Ġthat Ġdirectly Ġusing ĠB ERT Ġcannot Ġobtain Ġperformance Ġgrowth . ĠThe Ġother Ġthree Ġmethods Ġoutper form Ġother Ġmodels Ġby Ġa Ġsubstantial Ġmargin , Ġwhich Ġproves Ġthat Ġthe Ġimprovements Ġcome Ġfrom Ġleveraging ĠB ERT Ġto Ġbetter Ġexploit Ġgloss Ġinformation . ĠIt Ġis Ġworth Ġnoting Ġthat Ġour Ġmethod Ġachieves Ġsignificant Ġimprovements Ġin ĠSE 07 Ġand ĠVerb Ġover Ġprevious Ġmethods , Ġwhich Ġhave Ġthe Ġhighest Ġambiguity Ġlevel Ġamong Ġall Ġdatasets Ġand Ġall ĠPOS Ġtags Ġrespectively Ġaccording Ġto ĠB IB REF 17 . </s> Moreover , ĠGloss BER T ( Token - CL S ) Ġperforms Ġbetter Ġthan ĠGloss BER T ( Sent - CL S ), Ġwhich Ġproves Ġthat Ġhighlighting Ġthe Ġtarget Ġword Ġin Ġthe Ġsentence Ġis Ġimportant . ĠHowever , Ġthe Ġweak ly Ġhighlighting Ġmethod ĠGloss BER T ( Sent - CL S - WS ) Ġperforms Ġbest Ġin Ġmost Ġcircumstances , Ġwhich Ġmay Ġresult Ġfrom Ġits Ġcombination Ġof Ġthe Ġadvantages Ġof Ġthe Ġother Ġtwo Ġmethods . </s> Exper iments Ġ: :: ĠDiscussion </s> There Ġare Ġtwo Ġmain Ġreasons Ġfor Ġthe Ġgreat Ġimprovements Ġof Ġour Ġexperimental Ġresults . ĠFirst , Ġwe Ġconstruct Ġcontext - gl oss Ġpairs Ġand Ġconvert ĠW SD Ġproblem Ġto Ġa Ġsentence - pair Ġclassification Ġtask Ġwhich Ġis Ġsimilar Ġto ĠNL I Ġtasks Ġand Ġtrain Ġonly Ġone Ġclass ifier , Ġwhich Ġis Ġequivalent Ġto Ġexpanding Ġthe Ġcorpus . ĠSecond , Ġwe Ġleverage ĠB ERT ĠB IB REF 15 Ġto Ġbetter Ġexploit Ġthe Ġgloss Ġinformation . ĠB ERT Ġmodel Ġshows Ġits Ġadvantage Ġin Ġdealing Ġwith Ġsentence - pair Ġclassification Ġtasks Ġby Ġits Ġamazing Ġimprovement Ġon ĠQ A Ġand ĠNL I Ġtasks . ĠThis Ġadvantage Ġcomes Ġfrom Ġboth Ġof Ġits Ġtwo Ġnovel Ġun super vised Ġprediction Ġtasks . </s> Compared Ġwith Ġtraditional Ġword Ġexpert Ġsupervised Ġmethods , Ġour ĠGloss BER T Ġshows Ġits Ġeffectiveness Ġto Ġalleviate Ġthe Ġeffort Ġof Ġfeature Ġengineering Ġand Ġdoes Ġnot Ġrequire Ġtraining Ġa Ġdedicated Ġclass ifier Ġfor Ġevery Ġtarget Ġle mma . ĠUp Ġto Ġnow , Ġit Ġcan Ġbe Ġsaid Ġthat Ġthe Ġneural Ġnetwork Ġmethod Ġcan Ġtotally Ġbeat Ġthe Ġtraditional Ġword Ġexpert Ġmethod . ĠCompared Ġwith Ġrecent Ġneural - based Ġmethods , Ġour Ġsolution Ġis Ġmore Ġintuitive Ġand Ġcan Ġmake Ġbetter Ġuse Ġof Ġgloss Ġknowledge . ĠBesides , Ġour Ġapproach Ġdemonstrates Ġthat Ġwhen Ġwe Ġfine - t une ĠB ERT Ġon Ġa Ġdownstream Ġtask , Ġconverting Ġit Ġinto Ġa Ġsentence - pair Ġclassification Ġtask Ġmay Ġbe Ġa Ġgood Ġchoice . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġseek Ġto Ġbetter Ġleverage Ġgloss Ġknowledge Ġin Ġa Ġsupervised Ġneural ĠW SD Ġsystem . ĠWe Ġpropose Ġa Ġnew Ġsolution Ġto ĠW SD Ġby Ġconstructing Ġcontext - gl oss Ġpairs Ġand Ġthen Ġconverting ĠW SD Ġto Ġa Ġsentence - pair Ġclassification Ġtask . ĠWe Ġfine - t une Ġthe Ġpre - trained ĠB ERT Ġmodel Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon ĠW SD Ġtask . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġvaluable Ġcomments . ĠThe Ġresearch Ġwork Ġis Ġsupported Ġby ĠNational ĠNatural ĠScience ĠFoundation Ġof ĠChina Ġ( No . Ġ6 175 12 01 Ġand Ġ6 16 72 162 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠCommission Ġ( 16 JC 14 204 01 Ġand Ġ17 JC 14 04 100 ), ĠShanghai ĠMunicipal ĠScience Ġand ĠTechnology ĠMajor ĠProject Ġ( No . 2018 SH Z D ZX 01 ) Ġand ĠZ J Lab .