GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.
EXTRACTIVE
Table TABREF17 compares the performance of the approaches on the SCWS dataset. It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.
1911.06118
Introduction </s> Language Ġmodelling Ġin Ġits Ġinception Ġhad Ġone - hot Ġvector Ġencoding Ġof Ġwords . ĠHowever , Ġit Ġcaptures Ġonly Ġal ph abetic Ġordering Ġbut Ġnot Ġthe Ġword Ġsemantic Ġsimilarity . ĠVector Ġspace Ġmodels Ġhelps Ġto Ġlearn Ġword Ġrepresentations Ġin Ġa Ġlower Ġdimensional Ġspace Ġand Ġalso Ġcaptures Ġsemantic Ġsimilarity . ĠLearning Ġword Ġembed ding Ġaids Ġin Ġnatural Ġlanguage Ġprocessing Ġtasks Ġsuch Ġas Ġquestion Ġanswering Ġand Ġreasoning ĠB IB REF 0 , Ġstance Ġdetection ĠB IB REF 1 , Ġclaim Ġverification ĠB IB REF 2 . </s> Recent Ġmodels ĠB IB REF 3 , ĠB IB REF 4 Ġwork Ġon Ġthe Ġbasis Ġthat Ġwords Ġwith Ġsimilar Ġcontext Ġshare Ġsemantic Ġsimilarity . ĠB IB REF 4 Ġproposes Ġa Ġneural Ġprob abil istic Ġmodel Ġwhich Ġmodels Ġthe Ġtarget Ġword Ġprobability Ġconditioned Ġon Ġthe Ġprevious Ġwords Ġusing Ġa Ġrecurrent Ġneural Ġnetwork . ĠWord 2 V ec Ġmodels ĠB IB REF 3 Ġsuch Ġas Ġcontinuous Ġbag - of - words Ġ( CB OW ) Ġpredict Ġthe Ġtarget Ġword Ġgiven Ġthe Ġcontext , Ġand Ġskip - gram Ġmodel Ġworks Ġin Ġreverse Ġof Ġpredicting Ġthe Ġcontext Ġgiven Ġthe Ġtarget Ġword . ĠWhile , ĠGl o Ve Ġembed d ings Ġwere Ġbased Ġon Ġa ĠGlobal Ġmatrix Ġfactor ization Ġon Ġlocal Ġcontexts ĠB IB REF 5 . ĠHowever , Ġthe Ġaforementioned Ġmodels Ġdo Ġnot Ġhandle Ġwords Ġwith Ġmultiple Ġmeanings Ġ( poly sem ies ). </s> BI B REF 6 Ġproposes Ġa Ġneural Ġnetwork Ġapproach Ġconsidering Ġboth Ġlocal Ġand Ġglobal Ġcontexts Ġin Ġlearning Ġword Ġembed d ings Ġ( point Ġestimates ). ĠTheir Ġmultiple Ġprototype Ġmodel Ġhandles Ġpoly sem ous Ġwords Ġby Ġproviding Ġap rior i Ġhe ur istics Ġabout Ġword Ġsenses Ġin Ġthe Ġdataset . ĠB IB REF 7 Ġproposes Ġan Ġalternative Ġto Ġhandle Ġpoly sem ous Ġwords Ġby Ġa Ġmodified Ġskip - gram Ġmodel Ġand ĠEM Ġalgorithm . ĠB IB REF 8 Ġpresents Ġa Ġnon - param etric Ġbased Ġalternative Ġto Ġhandle Ġpoly sem ies . ĠHowever , Ġthese Ġapproaches Ġfail Ġto Ġconsider Ġentail ment Ġrelations Ġamong Ġthe Ġwords . ĠB IB REF 9 Ġlearn Ġa ĠGa ussian Ġdistribution Ġper Ġword Ġusing Ġthe Ġexpected Ġlikelihood Ġkernel . ĠHowever , Ġfor Ġpoly sem ous Ġwords , Ġthis Ġmay Ġlead Ġto Ġword Ġdistributions Ġwith Ġlarger Ġvar iances Ġas Ġit Ġmay Ġhave Ġto Ġcover Ġvarious Ġsenses . </s> BI B REF 10 Ġproposes Ġmultim od al Ġword Ġdistribution Ġapproach . ĠIt Ġcaptures Ġpoly se my . ĠHowever , Ġthe Ġenergy Ġbased Ġobjective Ġfunction Ġfails Ġto Ġconsider Ġasymm etry Ġand Ġhence Ġentail ment . ĠText ual Ġentail ment Ġrecognition Ġis Ġnecessary Ġto Ġcapture Ġlex ical Ġinference Ġrelations Ġsuch Ġas Ġcaus ality Ġ( for Ġexample , Ġmosquito Ġ$\ right arrow Ġ$ Ġmalaria ), Ġhyper ny my Ġ( for Ġexample , Ġdog Ġ$\ models Ġ$ Ġanimal ) Ġetc . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġobtain Ġmulti - sense Ġword Ġembed ding Ġdistributions Ġby Ġusing Ġa Ġvariant Ġof Ġmax Ġmargin Ġobjective Ġbased Ġon Ġthe Ġasymm etric ĠKL Ġdivergence Ġenergy Ġfunction Ġto Ġcapture Ġtextual Ġentail ment . ĠMulti - sense Ġdistributions Ġare Ġadvantageous Ġin Ġcapturing Ġpoly sem ous Ġnature Ġof Ġwords Ġand Ġin Ġreducing Ġthe Ġuncertainty Ġper Ġword Ġby Ġdistributing Ġit Ġacross Ġsenses . ĠHowever , Ġcomputing ĠKL Ġdivergence Ġbetween Ġm ixtures Ġof ĠGa uss ians Ġis Ġint ract able , Ġand Ġwe Ġuse Ġa ĠKL Ġdivergence Ġapproximation Ġbased Ġon Ġstricter Ġupper Ġand Ġlower Ġbounds . ĠWhile Ġcapturing Ġtextual Ġentail ment Ġ( asy mm etry ), Ġwe Ġhave Ġalso Ġnot Ġcompromised Ġon Ġcapturing Ġsymm etrical Ġsimilarity Ġbetween Ġwords Ġ( for Ġexample , Ġfunny Ġand Ġhilarious ) Ġwhich Ġwill Ġbe Ġeluc idated Ġin ĠSection Ġ$ 3 . 1 $. ĠWe Ġalso Ġshow Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġapproach Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġand Ġentail ment Ġdatasets Ġin Ġthe Ġexperimental Ġsection . </s> Method ology Ġ: :: ĠWord ĠRepresent ation </s> Pro b abil istic Ġrepresentation Ġof Ġwords Ġhelps Ġone Ġmodel Ġuncertainty Ġin Ġword Ġrepresentation , Ġand Ġpoly se my . ĠGiven Ġa Ġcorpus Ġ$ V $, Ġcontaining Ġa Ġlist Ġof Ġwords Ġeach Ġrepresented Ġas Ġ$ w $, Ġthe Ġprobability Ġdensity Ġfor Ġa Ġword Ġ$ w $ Ġcan Ġbe Ġrepresented Ġas Ġa Ġmixture Ġof ĠGa uss ians Ġwith Ġ$ C $ Ġcomponents ĠB IB REF 10 . </s> Here , Ġ$ p _{ w , j } $ Ġrepresents Ġthe Ġprobability Ġof Ġword Ġ$ w $ Ġbelonging Ġto Ġthe Ġcomponent Ġ$ j $, Ġ$\ oper at orn ame {\ math bf Ġ{\ mu Ġ}} _{ w , j } $ Ġrepresents Ġ$ D $ Ġdimensional Ġword Ġrepresentation Ġcorresponding Ġto Ġthe Ġ$ j ^{ th } $ Ġcomponent Ġsense Ġof Ġthe Ġword Ġ$ w $, Ġand Ġ$\ S igma Ġ_ { w , j } $ Ġrepresents Ġthe Ġuncertainty Ġin Ġrepresentation Ġfor Ġword Ġ$ w $ Ġbelonging Ġto Ġcomponent Ġ$ j $. </s> Object ive Ġfunction </s> The Ġmodel Ġparameters Ġ( me ans , Ġcov arian ces Ġand Ġmixture Ġweights ) Ġ$\ the ta Ġ$ Ġcan Ġbe Ġlearnt Ġusing Ġa Ġvariant Ġof Ġmax - margin Ġobjective ĠB IB REF 11 . </s> Here Ġ$ E _ \ the ta Ġ( \ cd ot Ġ, Ġ\ cd ot Ġ) $ Ġrepresents Ġan Ġenergy Ġfunction Ġwhich Ġassigns Ġa Ġscore Ġto Ġthe Ġpair Ġof Ġwords , Ġ$ w $ Ġis Ġa Ġparticular Ġword Ġunder Ġconsideration , Ġ$ c $ Ġits Ġpositive Ġcontext Ġ( same Ġcontext ), Ġand Ġ$ c ^ {\ prime Ġ} $ Ġthe Ġnegative Ġcontext . ĠThe Ġobjective Ġaims Ġto Ġpush Ġthe Ġmargin Ġof Ġthe Ġdifference Ġbetween Ġthe Ġenergy Ġfunction Ġof Ġa Ġword Ġ$ w $ Ġto Ġits Ġpositive Ġcontext Ġ$ c $ Ġhigher Ġthan Ġits Ġnegative Ġcontext Ġ$ c $ Ġby Ġa Ġthreshold Ġof Ġ$ m $. ĠThus , Ġword Ġpairs Ġin Ġthe Ġsame Ġcontext Ġgets Ġa Ġhigher Ġenergy Ġthan Ġthe Ġword Ġpairs Ġin Ġthe Ġdiss imilar Ġcontext . ĠB IB REF 10 Ġconsider Ġthe Ġenergy Ġfunction Ġto Ġbe Ġan Ġexpected Ġlikelihood Ġkernel Ġwhich Ġis Ġdefined Ġas Ġfollows . </s> This Ġis Ġsimilar Ġto Ġthe Ġcos ine Ġsimilarity Ġmetric Ġover Ġvectors Ġand Ġthe Ġenergy Ġbetween Ġtwo Ġwords Ġis Ġmaximum Ġwhen Ġthey Ġhave Ġsimilar Ġdistributions . ĠBut , Ġthe Ġexpected Ġlikelihood Ġkernel Ġis Ġa Ġsymm etric Ġmetric Ġwhich Ġwill Ġnot Ġbe Ġsuitable Ġfor Ġcapturing Ġordering Ġamong Ġwords Ġand Ġhence Ġentail ment . </s> Object ive Ġfunction Ġ: :: ĠProp osed ĠEnergy Ġfunction </s> As Ġeach Ġword Ġis Ġrepresented Ġby Ġa Ġmixture Ġof ĠGa ussian Ġdistributions , ĠKL Ġdivergence Ġis Ġa Ġbetter Ġchoice Ġof Ġenergy Ġfunction Ġto Ġcapture Ġdistance Ġbetween Ġdistributions . ĠSince , ĠKL Ġdivergence Ġis Ġminimum Ġwhen Ġthe Ġdistributions Ġare Ġsimilar Ġand Ġmaximum Ġwhen Ġthey Ġare Ġdiss imilar , Ġenergy Ġfunction Ġis Ġtaken Ġas Ġexponent iated Ġnegative ĠKL Ġdivergence . </s> However , Ġcomputing ĠKL Ġdivergence Ġbetween ĠGa ussian Ġm ixtures Ġis Ġint ract able Ġand Ġobtaining Ġexact ĠKL Ġvalue Ġis Ġnot Ġpossible . ĠOne Ġway Ġof Ġapprox imating Ġthe ĠKL Ġis Ġby ĠMonte - Car lo Ġapproximation Ġbut Ġit Ġrequires Ġlarge Ġnumber Ġof Ġsamples Ġto Ġget Ġa Ġgood Ġapproximation Ġand Ġis Ġcomput ationally Ġexpensive Ġon Ġhigh Ġdimensional Ġembed ding Ġspace . </s> Alternatively , ĠB IB REF 12 Ġpresents Ġa ĠKL Ġapproximation Ġbetween ĠGa ussian Ġm ixtures Ġwhere Ġthey Ġobtain Ġan Ġupper Ġbound Ġthrough Ġproduct Ġof ĠGa ussian Ġapproximation Ġmethod Ġand Ġa Ġlower Ġbound Ġthrough Ġvari ational Ġapproximation Ġmethod . ĠIn ĠB IB REF 13 , Ġthe Ġauthors Ġcombine Ġthe Ġlower Ġand Ġupper Ġbounds Ġfrom Ġapproximation Ġmethods Ġof ĠB IB REF 12 Ġto Ġprovide Ġa Ġstricter Ġbound Ġon ĠKL Ġbetween ĠGa ussian Ġm ixtures . ĠLets Ġconsider ĠGa ussian Ġm ixtures Ġfor Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $ Ġas Ġfollows . </s> The Ġapproximate ĠKL Ġdivergence Ġbetween Ġthe ĠGa ussian Ġmixture Ġrepresentations Ġover Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $ Ġis Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 8 . ĠMore Ġdetails Ġon Ġapproximation Ġis Ġincluded Ġin Ġthe ĠSupplementary ĠMaterial . </s> where Ġ$ EL _{ ik } ( w , w ) Ġ= Ġ\ int Ġf _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġf _{ w , k } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġd \ oper at orn ame {\ math bf Ġ{ x }} $ Ġand Ġ$ EL _{ ij } ( w , v ) Ġ= Ġ\ int Ġf _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġf _{ v , k } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġd \ oper at orn ame {\ math bf Ġ{ x }} $. ĠNote Ġthat Ġthe Ġexpected Ġlikelihood Ġkernel Ġappears Ġcomponent Ġwise Ġinside Ġthe Ġapproximate ĠKL Ġdivergence Ġderiv ation . </s> One Ġadvantage Ġof Ġusing ĠKL Ġas Ġenergy Ġfunction Ġis Ġthat Ġit Ġenables Ġto Ġcapture Ġasymm etry Ġin Ġentail ment Ġdatasets . ĠFor Ġeg ., Ġlet Ġus Ġconsider Ġthe Ġwords Ġ' chair ' Ġwith Ġtwo Ġsenses Ġas Ġ' bench ' Ġand Ġ' s ling ', Ġand Ġ' wood ' Ġwith Ġtwo Ġsenses Ġas Ġ' t rees ' Ġand Ġ' f urn iture '. ĠThe Ġword Ġchair Ġ($ w $ ) Ġis Ġent ailed Ġwithin Ġwood Ġ($ v $ ), Ġi . e . Ġchair Ġ$\ models Ġ$ Ġwood . ĠNow , Ġminimizing Ġthe ĠKL Ġdivergence Ġnecess itates Ġmaximizing Ġ$\ log Ġ{\ sum Ġ_ j Ġp _{ v , j } Ġ\ exp Ġ({ - K L ( f _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} )| | f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} )) }) } $ Ġwhich Ġin Ġturn Ġminim izes Ġ$ K L ( f _{ w , i } (\ oper at orn ame {\ math bf Ġ{ x }} )| | f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} )) $. ĠThis Ġwill Ġresult Ġin Ġthe Ġsupport Ġof Ġthe Ġ$ i ^{ th } $ Ġcomponent Ġof Ġ$ w $ Ġto Ġbe Ġwithin Ġthe Ġ$ j ^{ th } $ Ġcomponent Ġof Ġ$ v $, Ġand Ġholds Ġfor Ġall Ġcomponent Ġpairs Ġleading Ġto Ġthe Ġentail ment Ġof Ġ$ w $ Ġwithin Ġ$ v $. ĠConsequently , Ġwe Ġcan Ġsee Ġthat Ġbench Ġ$\ models Ġ$ Ġtrees , Ġbench Ġ$\ models Ġ$ Ġfurniture , Ġsling Ġ$\ models Ġ$ Ġtrees , Ġand Ġsling Ġ$\ models Ġ$ Ġfurniture . ĠThus , Ġit Ġintroduces Ġlex ical Ġrelationship Ġbetween Ġthe Ġsenses Ġof Ġchild Ġword Ġand Ġthat Ġof Ġthe Ġparent Ġword . ĠMin im izing Ġthe ĠKL Ġalso Ġnecess itates Ġmaximizing Ġ$\ log Ġ{\ sum Ġ_ j Ġ{ p _{ v , j }} ĠEL _{ ij } ( w , v )} $ Ġterm Ġfor Ġall Ġcomponent Ġpairs Ġamong Ġ$ w $ Ġand Ġ$ v $. ĠThis Ġis Ġsimilar Ġto Ġmaximizing Ġexpected Ġlikelihood Ġkernel , Ġwhich Ġbrings Ġthe Ġmeans Ġof Ġ$ f _{ w , i } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġand Ġ$ f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġcloser Ġ( weight ed Ġby Ġtheir Ġco - v arian ces ) Ġas Ġdiscussed Ġin ĠB IB REF 10 . ĠHence , Ġthe Ġproposed Ġapproach Ġcaptures Ġthe Ġbest Ġof Ġboth Ġworlds , Ġthereby Ġcatering Ġto Ġboth Ġword Ġsimilarity Ġand Ġentail ment . </s> We Ġalso Ġnote Ġthat Ġminimizing Ġthe ĠKL Ġdivergence Ġnecess itates Ġminimizing Ġ$\ log Ġ{\ sum Ġ_ k Ġp _{ w , k } Ġ\ exp Ġ({ - K L ( f _{ w , i } || f _{ w , k }) }) } $ Ġwhich Ġin Ġturn Ġmaxim izes Ġ$ K L ( f _{ w , i } || f _{ w , k }) $. ĠThis Ġprevents Ġthe Ġdifferent Ġmixture Ġcomponents Ġof Ġa Ġword Ġconver ging Ġto Ġsingle ĠGa ussian Ġand Ġencourages Ġcapturing Ġdifferent Ġpossible Ġsenses Ġof Ġthe Ġword . ĠThe Ġsame Ġis Ġalso Ġachieved Ġby Ġminimizing Ġ$\ sum Ġ_ k Ġ{ p _{ w , k }} ĠEL _{ ik } ( w , w ) $ Ġterm Ġand Ġact Ġas Ġa Ġregular ization Ġterm Ġwhich Ġpromotes Ġdiversity Ġin Ġlearning Ġsenses Ġof Ġa Ġword . </s> Exper iment ation Ġand ĠResults </s> We Ġtrain Ġour Ġproposed Ġmodel ĠGM $ \ _ $ K L Ġ( Ga ussian ĠM ixture Ġusing ĠKL ĠDiver gence ) Ġon Ġthe ĠText 8 Ġdataset ĠB IB REF 14 Ġwhich Ġis Ġa Ġpre - process ed Ġdata Ġof Ġ$ 17 M $ Ġwords Ġfrom Ġwik ipedia . ĠOf Ġwhich , Ġ7 12 90 Ġunique Ġand Ġfrequent Ġwords Ġare Ġchosen Ġusing Ġthe Ġsubs am pling Ġtrick Ġin ĠB IB REF 15 . ĠWe Ġcompare ĠGM $ \ _ $ K L Ġwith Ġthe Ġprevious Ġapproaches Ġw 2 g ĠB IB REF 9 Ġ( Ġsingle ĠGa ussian Ġmodel ) Ġand Ġw 2 gm ĠB IB REF 10 Ġ( m ixture Ġof ĠGa ussian Ġmodel Ġwith Ġexpected Ġlikelihood Ġkernel ). ĠFor Ġall Ġthe Ġmodels Ġused Ġfor Ġexperimentation , Ġthe Ġembed ding Ġsize Ġ($ D $ ) Ġwas Ġset Ġto Ġ50 , Ġnumber Ġof Ġm ixtures Ġto Ġ2 , Ġcontext Ġwindow Ġlength Ġto Ġ10 , Ġbatch Ġsize Ġto Ġ128 . ĠThe Ġword Ġembed d ings Ġwere Ġinitialized Ġusing Ġa Ġuniform Ġdistribution Ġin Ġthe Ġrange Ġof Ġ$ [ - \ sq rt {\ frac { 3 }{ D }} $, Ġ$\ sq rt {\ frac { 3 }{ D }} ] $ Ġsuch Ġthat Ġthe Ġexpectation Ġof Ġvariance Ġis Ġ1 Ġand Ġmean Ġ0 ĠB IB REF 16 . ĠOne Ġcould Ġalso Ġconsider Ġinitial izing Ġthe Ġword Ġembed d ings Ġusing Ġother Ġcontextual Ġrepresentations Ġsuch Ġas ĠB ERT ĠB IB REF 17 Ġand ĠEL Mo ĠB IB REF 18 Ġin Ġthe Ġproposed Ġapproach . ĠIn Ġorder Ġto Ġpurely Ġanalyze Ġthe Ġperformance Ġof Ġ$\ em ph Ġ{ GM \ _ K L } $ Ġover Ġthe Ġother Ġmodels , Ġwe Ġhave Ġchosen Ġinitialization Ġusing Ġuniform Ġdistribution Ġfor Ġexperiments . ĠFor Ġcomputational Ġbenefits , Ġdiagonal Ġcovari ance Ġis Ġused Ġsimilar Ġto ĠB IB REF 10 . ĠEach Ġmixture Ġprobability Ġis Ġconstrained Ġin Ġthe Ġrange Ġ$ [ 0 , 1 ] $, Ġsum ming Ġto Ġ1 Ġby Ġoptimizing Ġover Ġuncon str ained Ġscores Ġin Ġthe Ġrange Ġ$( - \ in fty Ġ, \ in fty Ġ) $ Ġand Ġconverting Ġscores Ġto Ġprobability Ġusing Ġsoft max Ġfunction . ĠThe Ġmixture Ġscores Ġare Ġinitialized Ġto Ġ0 Ġto Ġensure Ġfairness Ġamong Ġall Ġthe Ġcomponents . ĠThe Ġthreshold Ġfor Ġnegative Ġsampling Ġwas Ġset Ġto Ġ$ 10 ^{ - 5 } $, Ġas Ġrecommended Ġin ĠB IB REF 3 . ĠMini - batch Ġgradient Ġdescent Ġwith ĠAd ag rad Ġoptim izer ĠB IB REF 19 Ġwas Ġused Ġwith Ġinitial Ġlearning Ġrate Ġset Ġto Ġ$ 0 . 05 $. </s> Table ĠT AB REF 9 Ġshows Ġthe Ġqualitative Ġresults Ġof ĠGM $ \ _ $ K L . ĠGiven Ġa Ġquery Ġword Ġand Ġcomponent Ġid , Ġthe Ġset Ġof Ġnearest Ġneighbours Ġalong Ġwith Ġtheir Ġrespective Ġcomponent Ġ ids Ġare Ġlisted . ĠFor Ġeg ., Ġthe Ġword Ġ` plane ' Ġin Ġits Ġ0 th Ġcomponent Ġcaptures Ġthe Ġ` ge ometry ' Ġsense Ġand Ġso Ġare Ġits Ġneighbours , Ġand Ġits Ġ1 st Ġcomponent Ġcaptures Ġ` veh icle ' Ġsense Ġand Ġso Ġare Ġits Ġcorresponding Ġneighbours . ĠOther Ġwords Ġsuch Ġas Ġ` rock ' Ġcaptures Ġboth Ġthe Ġ` metal ' Ġand Ġ` music ' Ġsenses , Ġ` star ' Ġcaptures Ġ` cele br ity ' Ġand Ġ` ast ron omical ' Ġsenses , Ġand Ġ` phone ' Ġcaptures Ġ` tele phony ' Ġand Ġ` internet ' Ġsenses . </s> We Ġquant itatively Ġcompare Ġthe Ġperformance Ġof Ġthe ĠGM $ \ _ $ K L , Ġw 2 g , Ġand Ġw 2 gm Ġapproaches Ġon Ġthe ĠSC WS Ġdataset ĠB IB REF 6 . ĠThe Ġdataset Ġconsists Ġof Ġ2003 Ġword Ġpairs Ġof Ġpoly sem ous Ġand Ġhom onymous Ġwords Ġwith Ġlabels Ġobtained Ġby Ġan Ġaverage Ġof Ġ10 Ġhuman Ġscores . ĠThe ĠSpear man Ġcorrelation Ġbetween Ġthe Ġhuman Ġscores Ġand Ġthe Ġmodel Ġscores Ġare Ġcomputed . ĠTo Ġobtain Ġthe Ġmodel Ġscore , Ġthe Ġfollowing Ġmetrics Ġare Ġused : </s> Max Cos : ĠMaximum Ġcos ine Ġsimilarity Ġamong Ġall Ġcomponent Ġpairs Ġof Ġwords Ġ$ w $ Ġand Ġ$ v $ : </s> Avg Cos : ĠAverage Ġcomponent - wise Ġcos ine Ġsimilarity Ġbetween Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $. </s> K L $ \ _ $ app rox : ĠForm ulated Ġas Ġshown Ġin Ġ( DIS PLAY _ FORM 8 ) Ġbetween Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $. </s> K L $ \ _ $ comp : ĠMaximum Ġcomponent - wise Ġnegative ĠKL Ġbetween Ġwords Ġ$ w $ Ġand Ġ$ v $ : </s> Table ĠT AB REF 17 Ġcompares Ġthe Ġperformance Ġof Ġthe Ġapproaches Ġon Ġthe ĠSC WS Ġdataset . ĠIt Ġis Ġevident Ġfrom ĠTable ĠT AB REF 17 Ġthat ĠGM $ \ _ $ K L Ġachieves Ġbetter Ġcorrelation Ġthan Ġexisting Ġapproaches Ġfor Ġvarious Ġmetrics Ġon ĠSC WS Ġdataset . </s> Table ĠT AB REF 18 Ġshows Ġthe ĠSpear man Ġcorrelation Ġvalues Ġof ĠGM $ \ _ $ K L Ġmodel Ġevaluated Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġdatasets : ĠSL ĠB IB REF 20 , ĠWS , ĠWS - R , ĠWS - S ĠB IB REF 21 , ĠMEN ĠB IB REF 22 , ĠMC ĠB IB REF 23 , ĠRG ĠB IB REF 24 , ĠY P ĠB IB REF 25 , ĠM Tur k - 287 Ġand ĠM Tur k - 771 ĠB IB REF 26 , ĠB IB REF 27 , Ġand ĠRW ĠB IB REF 28 . ĠThe Ġmetric Ġused Ġfor Ġcomparison Ġis Ġ' Avg Cos '. ĠIt Ġcan Ġbe Ġseen Ġthat Ġfor Ġmost Ġof Ġthe Ġdatasets , ĠGM $ \ _ $ K L Ġachieves Ġsignificantly Ġbetter Ġcorrelation Ġscore Ġthan Ġw 2 g Ġand Ġw 2 gm Ġapproaches . ĠOther Ġdatasets Ġsuch Ġas ĠMC Ġand ĠRW Ġconsist Ġof Ġonly Ġa Ġsingle Ġsense , Ġand Ġhence Ġw 2 g Ġmodel Ġperforms Ġbetter Ġand ĠGM $ \ _ $ K L Ġachieves Ġnext Ġbetter Ġperformance . ĠThe ĠY P Ġdataset Ġhave Ġmultiple Ġsenses Ġbut Ġdoes Ġnot Ġcontain Ġent ailed Ġdata Ġand Ġhence Ġcould Ġnot Ġmake Ġuse Ġof Ġentail ment Ġbenefits Ġof ĠGM $ \ _ $ K L . </s> Table ĠT AB REF 19 Ġshows Ġthe Ġevaluation Ġresults Ġof ĠGM $ \ _ $ K L Ġmodel Ġon Ġthe Ġentail ment Ġdatasets Ġsuch Ġas Ġentail ment Ġpairs Ġdataset ĠB IB REF 29 Ġcreated Ġfrom ĠWord Net Ġwith Ġboth Ġpositive Ġand Ġnegative Ġlabels , Ġa Ġcrowds ourced Ġdataset ĠB IB REF 30 Ġof Ġ79 Ġsemantic Ġrelations Ġlabelled Ġas Ġent ailed Ġor Ġnot Ġand Ġannot ated Ġdistribution ally Ġsimilar Ġnoun s Ġdataset ĠB IB REF 31 . ĠThe Ġ' Max Cos ' Ġsimilarity Ġmetric Ġis Ġused Ġfor Ġevaluation Ġand Ġthe Ġbest Ġprecision Ġand Ġbest ĠF 1 - score Ġis Ġshown , Ġby Ġpicking Ġthe Ġoptimal Ġthreshold . ĠOverall , ĠGM $ \ _ $ K L Ġperforms Ġbetter Ġthan Ġboth Ġw 2 g Ġand Ġw 2 gm Ġapproaches . </s> Conclusion </s> We Ġproposed Ġa ĠKL Ġdivergence Ġbased Ġenergy Ġfunction Ġfor Ġlearning Ġmulti - sense Ġword Ġembed ding Ġdistributions Ġmod elled Ġas ĠGa ussian Ġm ixtures . ĠDue Ġto Ġthe Ġint ract ability Ġof Ġthe ĠGa ussian Ġm ixtures Ġfor Ġthe ĠKL Ġdivergence Ġmeasure , Ġwe Ġuse Ġan Ġapproximate ĠKL Ġdivergence Ġfunction . ĠWe Ġalso Ġdemonstrated Ġthat Ġthe Ġproposed ĠGM $ \ _ $ K L Ġapproaches Ġperformed Ġbetter Ġthan Ġother Ġapproaches Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġand Ġentail ment Ġdatasets . </s> toc section App end ices </s> App rox imation Ġfor ĠKL Ġdivergence Ġbetween Ġm ixtures Ġof Ġga uss ians </s> K L Ġbetween Ġga ussian Ġm ixtures Ġ$ f _{ w } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġand Ġ$ f _{ v } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġcan Ġbe Ġdecomp osed Ġas : </s> BI B REF 12 Ġpresents ĠKL Ġapproximation Ġbetween Ġga ussian Ġm ixtures Ġusing </s> product Ġof Ġga ussian Ġapproximation Ġmethod Ġwhere ĠKL Ġis Ġapprox imated Ġusing Ġproduct Ġof Ġcomponent Ġga uss ians Ġand </s> vari ational Ġapproximation Ġmethod Ġwhere ĠKL Ġis Ġapprox imated Ġby Ġintroducing Ġsome Ġvari ational Ġparameters . </s> The Ġproduct Ġof Ġcomponent Ġga ussian Ġapproximation Ġmethod Ġusing ĠJensen 's Ġinequality Ġprovides Ġupper Ġbounds Ġas Ġshown Ġin Ġequations ĠDIS PLAY _ FORM 23 Ġand Ġ. </s> The Ġvari ational Ġapproximation Ġmethod Ġprovides Ġlower Ġbounds Ġas Ġshown Ġin Ġequations ĠDIS PLAY _ FORM 24 Ġand ĠDIS PLAY _ FORM 25 . </s> where Ġ$ H $ Ġrepresents Ġthe Ġentropy Ġterm Ġand Ġthe Ġentropy Ġof Ġ$ i ^{ th } $ Ġcomponent Ġof Ġword Ġ$ w $ Ġwith Ġdimension Ġ$ D $ Ġis Ġgiven Ġas </s> In ĠB IB REF 13 , Ġthe Ġauthors Ġcombine Ġthe Ġlower Ġand Ġupper Ġbounds Ġfrom Ġapproximation Ġmethods Ġof ĠB IB REF 12 Ġto Ġformulate Ġa Ġstricter Ġbound Ġon ĠKL Ġbetween Ġga ussian Ġm ixtures . </s> From Ġequations ĠDIS PLAY _ FORM 23 Ġand ĠDIS PLAY _ FORM 25 , Ġa Ġstricter Ġlower Ġbound Ġfor ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġobtained Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 26 </s> From Ġequations Ġand ĠDIS PLAY _ FORM 24 , Ġa Ġstricter Ġupper Ġbound Ġfor ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġobtained Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 27 </s> Finally , Ġthe ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġtaken Ġas Ġthe Ġmean Ġof ĠKL Ġupper Ġand Ġlower Ġbounds Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 28 .
Introduction Language modelling in its inception had one-hot vector encoding of words. However, it captures only alphabetic ordering but not the word semantic similarity. Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity. Learning word embedding aids in natural language processing tasks such as question answering and reasoning BIBREF0, stance detection BIBREF1, claim verification BIBREF2. Recent models BIBREF3, BIBREF4 work on the basis that words with similar context share semantic similarity. BIBREF4 proposes a neural probabilistic model which models the target word probability conditioned on the previous words using a recurrent neural network. Word2Vec models BIBREF3 such as continuous bag-of-words (CBOW) predict the target word given the context, and skip-gram model works in reverse of predicting the context given the target word. While, GloVe embeddings were based on a Global matrix factorization on local contexts BIBREF5. However, the aforementioned models do not handle words with multiple meanings (polysemies). BIBREF6 proposes a neural network approach considering both local and global contexts in learning word embeddings (point estimates). Their multiple prototype model handles polysemous words by providing apriori heuristics about word senses in the dataset. BIBREF7 proposes an alternative to handle polysemous words by a modified skip-gram model and EM algorithm. BIBREF8 presents a non-parametric based alternative to handle polysemies. However, these approaches fail to consider entailment relations among the words. BIBREF9 learn a Gaussian distribution per word using the expected likelihood kernel. However, for polysemous words, this may lead to word distributions with larger variances as it may have to cover various senses. BIBREF10 proposes multimodal word distribution approach. It captures polysemy. However, the energy based objective function fails to consider asymmetry and hence entailment. Textual entailment recognition is necessary to capture lexical inference relations such as causality (for example, mosquito $\rightarrow $ malaria), hypernymy (for example, dog $\models $ animal) etc. In this paper, we propose to obtain multi-sense word embedding distributions by using a variant of max margin objective based on the asymmetric KL divergence energy function to capture textual entailment. Multi-sense distributions are advantageous in capturing polysemous nature of words and in reducing the uncertainty per word by distributing it across senses. However, computing KL divergence between mixtures of Gaussians is intractable, and we use a KL divergence approximation based on stricter upper and lower bounds. While capturing textual entailment (asymmetry), we have also not compromised on capturing symmetrical similarity between words (for example, funny and hilarious) which will be elucidated in Section $3.1$. We also show the effectiveness of the proposed approach on the benchmark word similarity and entailment datasets in the experimental section. Methodology ::: Word Representation Probabilistic representation of words helps one model uncertainty in word representation, and polysemy. Given a corpus $V$, containing a list of words each represented as $w$, the probability density for a word $w$ can be represented as a mixture of Gaussians with $C$ components BIBREF10. Here, $p_{w,j}$ represents the probability of word $w$ belonging to the component $j$, $\operatorname{\mathbf {\mu }}_{w,j}$ represents $D$ dimensional word representation corresponding to the $j^{th}$ component sense of the word $w$, and $\Sigma _{w,j}$ represents the uncertainty in representation for word $w$ belonging to component $j$. Objective function The model parameters (means, covariances and mixture weights) $\theta $ can be learnt using a variant of max-margin objective BIBREF11. Here $E_\theta (\cdot , \cdot )$ represents an energy function which assigns a score to the pair of words, $w$ is a particular word under consideration, $c$ its positive context (same context), and $c^{\prime }$ the negative context. The objective aims to push the margin of the difference between the energy function of a word $w$ to its positive context $c$ higher than its negative context $c$ by a threshold of $m$. Thus, word pairs in the same context gets a higher energy than the word pairs in the dissimilar context. BIBREF10 consider the energy function to be an expected likelihood kernel which is defined as follows. This is similar to the cosine similarity metric over vectors and the energy between two words is maximum when they have similar distributions. But, the expected likelihood kernel is a symmetric metric which will not be suitable for capturing ordering among words and hence entailment. Objective function ::: Proposed Energy function As each word is represented by a mixture of Gaussian distributions, KL divergence is a better choice of energy function to capture distance between distributions. Since, KL divergence is minimum when the distributions are similar and maximum when they are dissimilar, energy function is taken as exponentiated negative KL divergence. However, computing KL divergence between Gaussian mixtures is intractable and obtaining exact KL value is not possible. One way of approximating the KL is by Monte-Carlo approximation but it requires large number of samples to get a good approximation and is computationally expensive on high dimensional embedding space. Alternatively, BIBREF12 presents a KL approximation between Gaussian mixtures where they obtain an upper bound through product of Gaussian approximation method and a lower bound through variational approximation method. In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to provide a stricter bound on KL between Gaussian mixtures. Lets consider Gaussian mixtures for the words $w$ and $v$ as follows. The approximate KL divergence between the Gaussian mixture representations over the words $w$ and $v$ is shown in equation DISPLAY_FORM8. More details on approximation is included in the Supplementary Material. where $EL_{ik}(w,w) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{w,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$ and $EL_{ij}(w,v) = \int f_{w,i} (\operatorname{\mathbf {x}}) f_{v,k} (\operatorname{\mathbf {x}}) d\operatorname{\mathbf {x}}$. Note that the expected likelihood kernel appears component wise inside the approximate KL divergence derivation. One advantage of using KL as energy function is that it enables to capture asymmetry in entailment datasets. For eg., let us consider the words 'chair' with two senses as 'bench' and 'sling', and 'wood' with two senses as 'trees' and 'furniture'. The word chair ($w$) is entailed within wood ($v$), i.e. chair $\models $ wood. Now, minimizing the KL divergence necessitates maximizing $\log {\sum _j p_{v,j} \exp ({-KL(f_{w,i} (\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))})}$ which in turn minimizes $KL(f_{w,i}(\operatorname{\mathbf {x}})||f_{v,j}(\operatorname{\mathbf {x}}))$. This will result in the support of the $i^{th}$ component of $w$ to be within the $j^{th}$ component of $v$, and holds for all component pairs leading to the entailment of $w$ within $v$. Consequently, we can see that bench $\models $ trees, bench $\models $ furniture, sling $\models $ trees, and sling $\models $ furniture. Thus, it introduces lexical relationship between the senses of child word and that of the parent word. Minimizing the KL also necessitates maximizing $\log {\sum _j {p_{v,j}} EL_{ij}(w,v)}$ term for all component pairs among $w$ and $v$. This is similar to maximizing expected likelihood kernel, which brings the means of $f_{w,i}(\operatorname{\mathbf {x}})$ and $f_{v,j}(\operatorname{\mathbf {x}})$ closer (weighted by their co-variances) as discussed in BIBREF10. Hence, the proposed approach captures the best of both worlds, thereby catering to both word similarity and entailment. We also note that minimizing the KL divergence necessitates minimizing $\log {\sum _k p_{w,k} \exp ({-KL(f_{w,i}||f_{w,k})})}$ which in turn maximizes $KL(f_{w,i}||f_{w,k})$. This prevents the different mixture components of a word converging to single Gaussian and encourages capturing different possible senses of the word. The same is also achieved by minimizing $\sum _k {p_{w,k}} EL_{ik}(w,w)$ term and act as a regularization term which promotes diversity in learning senses of a word. Experimentation and Results We train our proposed model GM$\_$KL (Gaussian Mixture using KL Divergence) on the Text8 dataset BIBREF14 which is a pre-processed data of $17M$ words from wikipedia. Of which, 71290 unique and frequent words are chosen using the subsampling trick in BIBREF15. We compare GM$\_$KL with the previous approaches w2g BIBREF9 ( single Gaussian model) and w2gm BIBREF10 (mixture of Gaussian model with expected likelihood kernel). For all the models used for experimentation, the embedding size ($D$) was set to 50, number of mixtures to 2, context window length to 10, batch size to 128. The word embeddings were initialized using a uniform distribution in the range of $[-\sqrt{\frac{3}{D}}$, $\sqrt{\frac{3}{D}}]$ such that the expectation of variance is 1 and mean 0 BIBREF16. One could also consider initializing the word embeddings using other contextual representations such as BERT BIBREF17 and ELMo BIBREF18 in the proposed approach. In order to purely analyze the performance of $\emph {GM\_KL}$ over the other models, we have chosen initialization using uniform distribution for experiments. For computational benefits, diagonal covariance is used similar to BIBREF10. Each mixture probability is constrained in the range $[0,1]$, summing to 1 by optimizing over unconstrained scores in the range $(-\infty ,\infty )$ and converting scores to probability using softmax function. The mixture scores are initialized to 0 to ensure fairness among all the components. The threshold for negative sampling was set to $10^{-5}$, as recommended in BIBREF3. Mini-batch gradient descent with Adagrad optimizer BIBREF19 was used with initial learning rate set to $0.05$. Table TABREF9 shows the qualitative results of GM$\_$KL. Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed. For eg., the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours. Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses. We quantitatively compare the performance of the GM$\_$KL, w2g, and w2gm approaches on the SCWS dataset BIBREF6. The dataset consists of 2003 word pairs of polysemous and homonymous words with labels obtained by an average of 10 human scores. The Spearman correlation between the human scores and the model scores are computed. To obtain the model score, the following metrics are used: MaxCos: Maximum cosine similarity among all component pairs of words $w$ and $v$: AvgCos: Average component-wise cosine similarity between the words $w$ and $v$. KL$\_$approx: Formulated as shown in (DISPLAY_FORM8) between the words $w$ and $v$. KL$\_$comp: Maximum component-wise negative KL between words $w$ and $v$: Table TABREF17 compares the performance of the approaches on the SCWS dataset. It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset. Table TABREF18 shows the Spearman correlation values of GM$\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28. The metric used for comparison is 'AvgCos'. It can be seen that for most of the datasets, GM$\_$KL achieves significantly better correlation score than w2g and w2gm approaches. Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\_$KL achieves next better performance. The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\_$KL. Table TABREF19 shows the evaluation results of GM$\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31. The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold. Overall, GM$\_$KL performs better than both w2g and w2gm approaches. Conclusion We proposed a KL divergence based energy function for learning multi-sense word embedding distributions modelled as Gaussian mixtures. Due to the intractability of the Gaussian mixtures for the KL divergence measure, we use an approximate KL divergence function. We also demonstrated that the proposed GM$\_$KL approaches performed better than other approaches on the benchmark word similarity and entailment datasets. tocsectionAppendices Approximation for KL divergence between mixtures of gaussians KL between gaussian mixtures $f_{w}(\operatorname{\mathbf {x}})$ and $f_{v}(\operatorname{\mathbf {x}})$ can be decomposed as: BIBREF12 presents KL approximation between gaussian mixtures using product of gaussian approximation method where KL is approximated using product of component gaussians and variational approximation method where KL is approximated by introducing some variational parameters. The product of component gaussian approximation method using Jensen's inequality provides upper bounds as shown in equations DISPLAY_FORM23 and . The variational approximation method provides lower bounds as shown in equations DISPLAY_FORM24 and DISPLAY_FORM25. where $H$ represents the entropy term and the entropy of $i^{th}$ component of word $w$ with dimension $D$ is given as In BIBREF13, the authors combine the lower and upper bounds from approximation methods of BIBREF12 to formulate a stricter bound on KL between gaussian mixtures. From equations DISPLAY_FORM23 and DISPLAY_FORM25, a stricter lower bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM26 From equations and DISPLAY_FORM24, a stricter upper bound for KL between gaussian mixtures is obtained as shown in equation DISPLAY_FORM27 Finally, the KL between gaussian mixtures is taken as the mean of KL upper and lower bounds as shown in equation DISPLAY_FORM28.
How does this approach compare to other WSD approaches employing word embeddings?
660284b0a21fe3801e64dc9e0e51da5400223fe3
How Ġdoes Ġthis Ġapproach Ġcompare Ġto Ġother ĠW SD Ġapproaches Ġemploying Ġword Ġembed d ings ?
GM $ \ _ $ K L Ġachieves Ġbetter Ġcorrelation Ġthan Ġexisting Ġapproaches Ġfor Ġvarious Ġmetrics Ġon ĠSC WS Ġdataset .
<s> How Ġdoes Ġthis Ġapproach Ġcompare Ġto Ġother ĠW SD Ġapproaches Ġemploying Ġword Ġembed d ings ? </s> Introduction </s> Language Ġmodelling Ġin Ġits Ġinception Ġhad Ġone - hot Ġvector Ġencoding Ġof Ġwords . ĠHowever , Ġit Ġcaptures Ġonly Ġal ph abetic Ġordering Ġbut Ġnot Ġthe Ġword Ġsemantic Ġsimilarity . ĠVector Ġspace Ġmodels Ġhelps Ġto Ġlearn Ġword Ġrepresentations Ġin Ġa Ġlower Ġdimensional Ġspace Ġand Ġalso Ġcaptures Ġsemantic Ġsimilarity . ĠLearning Ġword Ġembed ding Ġaids Ġin Ġnatural Ġlanguage Ġprocessing Ġtasks Ġsuch Ġas Ġquestion Ġanswering Ġand Ġreasoning ĠB IB REF 0 , Ġstance Ġdetection ĠB IB REF 1 , Ġclaim Ġverification ĠB IB REF 2 . </s> Recent Ġmodels ĠB IB REF 3 , ĠB IB REF 4 Ġwork Ġon Ġthe Ġbasis Ġthat Ġwords Ġwith Ġsimilar Ġcontext Ġshare Ġsemantic Ġsimilarity . ĠB IB REF 4 Ġproposes Ġa Ġneural Ġprob abil istic Ġmodel Ġwhich Ġmodels Ġthe Ġtarget Ġword Ġprobability Ġconditioned Ġon Ġthe Ġprevious Ġwords Ġusing Ġa Ġrecurrent Ġneural Ġnetwork . ĠWord 2 V ec Ġmodels ĠB IB REF 3 Ġsuch Ġas Ġcontinuous Ġbag - of - words Ġ( CB OW ) Ġpredict Ġthe Ġtarget Ġword Ġgiven Ġthe Ġcontext , Ġand Ġskip - gram Ġmodel Ġworks Ġin Ġreverse Ġof Ġpredicting Ġthe Ġcontext Ġgiven Ġthe Ġtarget Ġword . ĠWhile , ĠGl o Ve Ġembed d ings Ġwere Ġbased Ġon Ġa ĠGlobal Ġmatrix Ġfactor ization Ġon Ġlocal Ġcontexts ĠB IB REF 5 . ĠHowever , Ġthe Ġaforementioned Ġmodels Ġdo Ġnot Ġhandle Ġwords Ġwith Ġmultiple Ġmeanings Ġ( poly sem ies ). </s> BI B REF 6 Ġproposes Ġa Ġneural Ġnetwork Ġapproach Ġconsidering Ġboth Ġlocal Ġand Ġglobal Ġcontexts Ġin Ġlearning Ġword Ġembed d ings Ġ( point Ġestimates ). ĠTheir Ġmultiple Ġprototype Ġmodel Ġhandles Ġpoly sem ous Ġwords Ġby Ġproviding Ġap rior i Ġhe ur istics Ġabout Ġword Ġsenses Ġin Ġthe Ġdataset . ĠB IB REF 7 Ġproposes Ġan Ġalternative Ġto Ġhandle Ġpoly sem ous Ġwords Ġby Ġa Ġmodified Ġskip - gram Ġmodel Ġand ĠEM Ġalgorithm . ĠB IB REF 8 Ġpresents Ġa Ġnon - param etric Ġbased Ġalternative Ġto Ġhandle Ġpoly sem ies . ĠHowever , Ġthese Ġapproaches Ġfail Ġto Ġconsider Ġentail ment Ġrelations Ġamong Ġthe Ġwords . ĠB IB REF 9 Ġlearn Ġa ĠGa ussian Ġdistribution Ġper Ġword Ġusing Ġthe Ġexpected Ġlikelihood Ġkernel . ĠHowever , Ġfor Ġpoly sem ous Ġwords , Ġthis Ġmay Ġlead Ġto Ġword Ġdistributions Ġwith Ġlarger Ġvar iances Ġas Ġit Ġmay Ġhave Ġto Ġcover Ġvarious Ġsenses . </s> BI B REF 10 Ġproposes Ġmultim od al Ġword Ġdistribution Ġapproach . ĠIt Ġcaptures Ġpoly se my . ĠHowever , Ġthe Ġenergy Ġbased Ġobjective Ġfunction Ġfails Ġto Ġconsider Ġasymm etry Ġand Ġhence Ġentail ment . ĠText ual Ġentail ment Ġrecognition Ġis Ġnecessary Ġto Ġcapture Ġlex ical Ġinference Ġrelations Ġsuch Ġas Ġcaus ality Ġ( for Ġexample , Ġmosquito Ġ$\ right arrow Ġ$ Ġmalaria ), Ġhyper ny my Ġ( for Ġexample , Ġdog Ġ$\ models Ġ$ Ġanimal ) Ġetc . </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġto Ġobtain Ġmulti - sense Ġword Ġembed ding Ġdistributions Ġby Ġusing Ġa Ġvariant Ġof Ġmax Ġmargin Ġobjective Ġbased Ġon Ġthe Ġasymm etric ĠKL Ġdivergence Ġenergy Ġfunction Ġto Ġcapture Ġtextual Ġentail ment . ĠMulti - sense Ġdistributions Ġare Ġadvantageous Ġin Ġcapturing Ġpoly sem ous Ġnature Ġof Ġwords Ġand Ġin Ġreducing Ġthe Ġuncertainty Ġper Ġword Ġby Ġdistributing Ġit Ġacross Ġsenses . ĠHowever , Ġcomputing ĠKL Ġdivergence Ġbetween Ġm ixtures Ġof ĠGa uss ians Ġis Ġint ract able , Ġand Ġwe Ġuse Ġa ĠKL Ġdivergence Ġapproximation Ġbased Ġon Ġstricter Ġupper Ġand Ġlower Ġbounds . ĠWhile Ġcapturing Ġtextual Ġentail ment Ġ( asy mm etry ), Ġwe Ġhave Ġalso Ġnot Ġcompromised Ġon Ġcapturing Ġsymm etrical Ġsimilarity Ġbetween Ġwords Ġ( for Ġexample , Ġfunny Ġand Ġhilarious ) Ġwhich Ġwill Ġbe Ġeluc idated Ġin ĠSection Ġ$ 3 . 1 $. ĠWe Ġalso Ġshow Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġapproach Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġand Ġentail ment Ġdatasets Ġin Ġthe Ġexperimental Ġsection . </s> Method ology Ġ: :: ĠWord ĠRepresent ation </s> Pro b abil istic Ġrepresentation Ġof Ġwords Ġhelps Ġone Ġmodel Ġuncertainty Ġin Ġword Ġrepresentation , Ġand Ġpoly se my . ĠGiven Ġa Ġcorpus Ġ$ V $, Ġcontaining Ġa Ġlist Ġof Ġwords Ġeach Ġrepresented Ġas Ġ$ w $, Ġthe Ġprobability Ġdensity Ġfor Ġa Ġword Ġ$ w $ Ġcan Ġbe Ġrepresented Ġas Ġa Ġmixture Ġof ĠGa uss ians Ġwith Ġ$ C $ Ġcomponents ĠB IB REF 10 . </s> Here , Ġ$ p _{ w , j } $ Ġrepresents Ġthe Ġprobability Ġof Ġword Ġ$ w $ Ġbelonging Ġto Ġthe Ġcomponent Ġ$ j $, Ġ$\ oper at orn ame {\ math bf Ġ{\ mu Ġ}} _{ w , j } $ Ġrepresents Ġ$ D $ Ġdimensional Ġword Ġrepresentation Ġcorresponding Ġto Ġthe Ġ$ j ^{ th } $ Ġcomponent Ġsense Ġof Ġthe Ġword Ġ$ w $, Ġand Ġ$\ S igma Ġ_ { w , j } $ Ġrepresents Ġthe Ġuncertainty Ġin Ġrepresentation Ġfor Ġword Ġ$ w $ Ġbelonging Ġto Ġcomponent Ġ$ j $. </s> Object ive Ġfunction </s> The Ġmodel Ġparameters Ġ( me ans , Ġcov arian ces Ġand Ġmixture Ġweights ) Ġ$\ the ta Ġ$ Ġcan Ġbe Ġlearnt Ġusing Ġa Ġvariant Ġof Ġmax - margin Ġobjective ĠB IB REF 11 . </s> Here Ġ$ E _ \ the ta Ġ( \ cd ot Ġ, Ġ\ cd ot Ġ) $ Ġrepresents Ġan Ġenergy Ġfunction Ġwhich Ġassigns Ġa Ġscore Ġto Ġthe Ġpair Ġof Ġwords , Ġ$ w $ Ġis Ġa Ġparticular Ġword Ġunder Ġconsideration , Ġ$ c $ Ġits Ġpositive Ġcontext Ġ( same Ġcontext ), Ġand Ġ$ c ^ {\ prime Ġ} $ Ġthe Ġnegative Ġcontext . ĠThe Ġobjective Ġaims Ġto Ġpush Ġthe Ġmargin Ġof Ġthe Ġdifference Ġbetween Ġthe Ġenergy Ġfunction Ġof Ġa Ġword Ġ$ w $ Ġto Ġits Ġpositive Ġcontext Ġ$ c $ Ġhigher Ġthan Ġits Ġnegative Ġcontext Ġ$ c $ Ġby Ġa Ġthreshold Ġof Ġ$ m $. ĠThus , Ġword Ġpairs Ġin Ġthe Ġsame Ġcontext Ġgets Ġa Ġhigher Ġenergy Ġthan Ġthe Ġword Ġpairs Ġin Ġthe Ġdiss imilar Ġcontext . ĠB IB REF 10 Ġconsider Ġthe Ġenergy Ġfunction Ġto Ġbe Ġan Ġexpected Ġlikelihood Ġkernel Ġwhich Ġis Ġdefined Ġas Ġfollows . </s> This Ġis Ġsimilar Ġto Ġthe Ġcos ine Ġsimilarity Ġmetric Ġover Ġvectors Ġand Ġthe Ġenergy Ġbetween Ġtwo Ġwords Ġis Ġmaximum Ġwhen Ġthey Ġhave Ġsimilar Ġdistributions . ĠBut , Ġthe Ġexpected Ġlikelihood Ġkernel Ġis Ġa Ġsymm etric Ġmetric Ġwhich Ġwill Ġnot Ġbe Ġsuitable Ġfor Ġcapturing Ġordering Ġamong Ġwords Ġand Ġhence Ġentail ment . </s> Object ive Ġfunction Ġ: :: ĠProp osed ĠEnergy Ġfunction </s> As Ġeach Ġword Ġis Ġrepresented Ġby Ġa Ġmixture Ġof ĠGa ussian Ġdistributions , ĠKL Ġdivergence Ġis Ġa Ġbetter Ġchoice Ġof Ġenergy Ġfunction Ġto Ġcapture Ġdistance Ġbetween Ġdistributions . ĠSince , ĠKL Ġdivergence Ġis Ġminimum Ġwhen Ġthe Ġdistributions Ġare Ġsimilar Ġand Ġmaximum Ġwhen Ġthey Ġare Ġdiss imilar , Ġenergy Ġfunction Ġis Ġtaken Ġas Ġexponent iated Ġnegative ĠKL Ġdivergence . </s> However , Ġcomputing ĠKL Ġdivergence Ġbetween ĠGa ussian Ġm ixtures Ġis Ġint ract able Ġand Ġobtaining Ġexact ĠKL Ġvalue Ġis Ġnot Ġpossible . ĠOne Ġway Ġof Ġapprox imating Ġthe ĠKL Ġis Ġby ĠMonte - Car lo Ġapproximation Ġbut Ġit Ġrequires Ġlarge Ġnumber Ġof Ġsamples Ġto Ġget Ġa Ġgood Ġapproximation Ġand Ġis Ġcomput ationally Ġexpensive Ġon Ġhigh Ġdimensional Ġembed ding Ġspace . </s> Alternatively , ĠB IB REF 12 Ġpresents Ġa ĠKL Ġapproximation Ġbetween ĠGa ussian Ġm ixtures Ġwhere Ġthey Ġobtain Ġan Ġupper Ġbound Ġthrough Ġproduct Ġof ĠGa ussian Ġapproximation Ġmethod Ġand Ġa Ġlower Ġbound Ġthrough Ġvari ational Ġapproximation Ġmethod . ĠIn ĠB IB REF 13 , Ġthe Ġauthors Ġcombine Ġthe Ġlower Ġand Ġupper Ġbounds Ġfrom Ġapproximation Ġmethods Ġof ĠB IB REF 12 Ġto Ġprovide Ġa Ġstricter Ġbound Ġon ĠKL Ġbetween ĠGa ussian Ġm ixtures . ĠLets Ġconsider ĠGa ussian Ġm ixtures Ġfor Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $ Ġas Ġfollows . </s> The Ġapproximate ĠKL Ġdivergence Ġbetween Ġthe ĠGa ussian Ġmixture Ġrepresentations Ġover Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $ Ġis Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 8 . ĠMore Ġdetails Ġon Ġapproximation Ġis Ġincluded Ġin Ġthe ĠSupplementary ĠMaterial . </s> where Ġ$ EL _{ ik } ( w , w ) Ġ= Ġ\ int Ġf _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġf _{ w , k } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġd \ oper at orn ame {\ math bf Ġ{ x }} $ Ġand Ġ$ EL _{ ij } ( w , v ) Ġ= Ġ\ int Ġf _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġf _{ v , k } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} ) Ġd \ oper at orn ame {\ math bf Ġ{ x }} $. ĠNote Ġthat Ġthe Ġexpected Ġlikelihood Ġkernel Ġappears Ġcomponent Ġwise Ġinside Ġthe Ġapproximate ĠKL Ġdivergence Ġderiv ation . </s> One Ġadvantage Ġof Ġusing ĠKL Ġas Ġenergy Ġfunction Ġis Ġthat Ġit Ġenables Ġto Ġcapture Ġasymm etry Ġin Ġentail ment Ġdatasets . ĠFor Ġeg ., Ġlet Ġus Ġconsider Ġthe Ġwords Ġ' chair ' Ġwith Ġtwo Ġsenses Ġas Ġ' bench ' Ġand Ġ' s ling ', Ġand Ġ' wood ' Ġwith Ġtwo Ġsenses Ġas Ġ' t rees ' Ġand Ġ' f urn iture '. ĠThe Ġword Ġchair Ġ($ w $ ) Ġis Ġent ailed Ġwithin Ġwood Ġ($ v $ ), Ġi . e . Ġchair Ġ$\ models Ġ$ Ġwood . ĠNow , Ġminimizing Ġthe ĠKL Ġdivergence Ġnecess itates Ġmaximizing Ġ$\ log Ġ{\ sum Ġ_ j Ġp _{ v , j } Ġ\ exp Ġ({ - K L ( f _{ w , i } Ġ( \ oper at orn ame {\ math bf Ġ{ x }} )| | f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} )) }) } $ Ġwhich Ġin Ġturn Ġminim izes Ġ$ K L ( f _{ w , i } (\ oper at orn ame {\ math bf Ġ{ x }} )| | f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} )) $. ĠThis Ġwill Ġresult Ġin Ġthe Ġsupport Ġof Ġthe Ġ$ i ^{ th } $ Ġcomponent Ġof Ġ$ w $ Ġto Ġbe Ġwithin Ġthe Ġ$ j ^{ th } $ Ġcomponent Ġof Ġ$ v $, Ġand Ġholds Ġfor Ġall Ġcomponent Ġpairs Ġleading Ġto Ġthe Ġentail ment Ġof Ġ$ w $ Ġwithin Ġ$ v $. ĠConsequently , Ġwe Ġcan Ġsee Ġthat Ġbench Ġ$\ models Ġ$ Ġtrees , Ġbench Ġ$\ models Ġ$ Ġfurniture , Ġsling Ġ$\ models Ġ$ Ġtrees , Ġand Ġsling Ġ$\ models Ġ$ Ġfurniture . ĠThus , Ġit Ġintroduces Ġlex ical Ġrelationship Ġbetween Ġthe Ġsenses Ġof Ġchild Ġword Ġand Ġthat Ġof Ġthe Ġparent Ġword . ĠMin im izing Ġthe ĠKL Ġalso Ġnecess itates Ġmaximizing Ġ$\ log Ġ{\ sum Ġ_ j Ġ{ p _{ v , j }} ĠEL _{ ij } ( w , v )} $ Ġterm Ġfor Ġall Ġcomponent Ġpairs Ġamong Ġ$ w $ Ġand Ġ$ v $. ĠThis Ġis Ġsimilar Ġto Ġmaximizing Ġexpected Ġlikelihood Ġkernel , Ġwhich Ġbrings Ġthe Ġmeans Ġof Ġ$ f _{ w , i } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġand Ġ$ f _{ v , j } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġcloser Ġ( weight ed Ġby Ġtheir Ġco - v arian ces ) Ġas Ġdiscussed Ġin ĠB IB REF 10 . ĠHence , Ġthe Ġproposed Ġapproach Ġcaptures Ġthe Ġbest Ġof Ġboth Ġworlds , Ġthereby Ġcatering Ġto Ġboth Ġword Ġsimilarity Ġand Ġentail ment . </s> We Ġalso Ġnote Ġthat Ġminimizing Ġthe ĠKL Ġdivergence Ġnecess itates Ġminimizing Ġ$\ log Ġ{\ sum Ġ_ k Ġp _{ w , k } Ġ\ exp Ġ({ - K L ( f _{ w , i } || f _{ w , k }) }) } $ Ġwhich Ġin Ġturn Ġmaxim izes Ġ$ K L ( f _{ w , i } || f _{ w , k }) $. ĠThis Ġprevents Ġthe Ġdifferent Ġmixture Ġcomponents Ġof Ġa Ġword Ġconver ging Ġto Ġsingle ĠGa ussian Ġand Ġencourages Ġcapturing Ġdifferent Ġpossible Ġsenses Ġof Ġthe Ġword . ĠThe Ġsame Ġis Ġalso Ġachieved Ġby Ġminimizing Ġ$\ sum Ġ_ k Ġ{ p _{ w , k }} ĠEL _{ ik } ( w , w ) $ Ġterm Ġand Ġact Ġas Ġa Ġregular ization Ġterm Ġwhich Ġpromotes Ġdiversity Ġin Ġlearning Ġsenses Ġof Ġa Ġword . </s> Exper iment ation Ġand ĠResults </s> We Ġtrain Ġour Ġproposed Ġmodel ĠGM $ \ _ $ K L Ġ( Ga ussian ĠM ixture Ġusing ĠKL ĠDiver gence ) Ġon Ġthe ĠText 8 Ġdataset ĠB IB REF 14 Ġwhich Ġis Ġa Ġpre - process ed Ġdata Ġof Ġ$ 17 M $ Ġwords Ġfrom Ġwik ipedia . ĠOf Ġwhich , Ġ7 12 90 Ġunique Ġand Ġfrequent Ġwords Ġare Ġchosen Ġusing Ġthe Ġsubs am pling Ġtrick Ġin ĠB IB REF 15 . ĠWe Ġcompare ĠGM $ \ _ $ K L Ġwith Ġthe Ġprevious Ġapproaches Ġw 2 g ĠB IB REF 9 Ġ( Ġsingle ĠGa ussian Ġmodel ) Ġand Ġw 2 gm ĠB IB REF 10 Ġ( m ixture Ġof ĠGa ussian Ġmodel Ġwith Ġexpected Ġlikelihood Ġkernel ). ĠFor Ġall Ġthe Ġmodels Ġused Ġfor Ġexperimentation , Ġthe Ġembed ding Ġsize Ġ($ D $ ) Ġwas Ġset Ġto Ġ50 , Ġnumber Ġof Ġm ixtures Ġto Ġ2 , Ġcontext Ġwindow Ġlength Ġto Ġ10 , Ġbatch Ġsize Ġto Ġ128 . ĠThe Ġword Ġembed d ings Ġwere Ġinitialized Ġusing Ġa Ġuniform Ġdistribution Ġin Ġthe Ġrange Ġof Ġ$ [ - \ sq rt {\ frac { 3 }{ D }} $, Ġ$\ sq rt {\ frac { 3 }{ D }} ] $ Ġsuch Ġthat Ġthe Ġexpectation Ġof Ġvariance Ġis Ġ1 Ġand Ġmean Ġ0 ĠB IB REF 16 . ĠOne Ġcould Ġalso Ġconsider Ġinitial izing Ġthe Ġword Ġembed d ings Ġusing Ġother Ġcontextual Ġrepresentations Ġsuch Ġas ĠB ERT ĠB IB REF 17 Ġand ĠEL Mo ĠB IB REF 18 Ġin Ġthe Ġproposed Ġapproach . ĠIn Ġorder Ġto Ġpurely Ġanalyze Ġthe Ġperformance Ġof Ġ$\ em ph Ġ{ GM \ _ K L } $ Ġover Ġthe Ġother Ġmodels , Ġwe Ġhave Ġchosen Ġinitialization Ġusing Ġuniform Ġdistribution Ġfor Ġexperiments . ĠFor Ġcomputational Ġbenefits , Ġdiagonal Ġcovari ance Ġis Ġused Ġsimilar Ġto ĠB IB REF 10 . ĠEach Ġmixture Ġprobability Ġis Ġconstrained Ġin Ġthe Ġrange Ġ$ [ 0 , 1 ] $, Ġsum ming Ġto Ġ1 Ġby Ġoptimizing Ġover Ġuncon str ained Ġscores Ġin Ġthe Ġrange Ġ$( - \ in fty Ġ, \ in fty Ġ) $ Ġand Ġconverting Ġscores Ġto Ġprobability Ġusing Ġsoft max Ġfunction . ĠThe Ġmixture Ġscores Ġare Ġinitialized Ġto Ġ0 Ġto Ġensure Ġfairness Ġamong Ġall Ġthe Ġcomponents . ĠThe Ġthreshold Ġfor Ġnegative Ġsampling Ġwas Ġset Ġto Ġ$ 10 ^{ - 5 } $, Ġas Ġrecommended Ġin ĠB IB REF 3 . ĠMini - batch Ġgradient Ġdescent Ġwith ĠAd ag rad Ġoptim izer ĠB IB REF 19 Ġwas Ġused Ġwith Ġinitial Ġlearning Ġrate Ġset Ġto Ġ$ 0 . 05 $. </s> Table ĠT AB REF 9 Ġshows Ġthe Ġqualitative Ġresults Ġof ĠGM $ \ _ $ K L . ĠGiven Ġa Ġquery Ġword Ġand Ġcomponent Ġid , Ġthe Ġset Ġof Ġnearest Ġneighbours Ġalong Ġwith Ġtheir Ġrespective Ġcomponent Ġ ids Ġare Ġlisted . ĠFor Ġeg ., Ġthe Ġword Ġ` plane ' Ġin Ġits Ġ0 th Ġcomponent Ġcaptures Ġthe Ġ` ge ometry ' Ġsense Ġand Ġso Ġare Ġits Ġneighbours , Ġand Ġits Ġ1 st Ġcomponent Ġcaptures Ġ` veh icle ' Ġsense Ġand Ġso Ġare Ġits Ġcorresponding Ġneighbours . ĠOther Ġwords Ġsuch Ġas Ġ` rock ' Ġcaptures Ġboth Ġthe Ġ` metal ' Ġand Ġ` music ' Ġsenses , Ġ` star ' Ġcaptures Ġ` cele br ity ' Ġand Ġ` ast ron omical ' Ġsenses , Ġand Ġ` phone ' Ġcaptures Ġ` tele phony ' Ġand Ġ` internet ' Ġsenses . </s> We Ġquant itatively Ġcompare Ġthe Ġperformance Ġof Ġthe ĠGM $ \ _ $ K L , Ġw 2 g , Ġand Ġw 2 gm Ġapproaches Ġon Ġthe ĠSC WS Ġdataset ĠB IB REF 6 . ĠThe Ġdataset Ġconsists Ġof Ġ2003 Ġword Ġpairs Ġof Ġpoly sem ous Ġand Ġhom onymous Ġwords Ġwith Ġlabels Ġobtained Ġby Ġan Ġaverage Ġof Ġ10 Ġhuman Ġscores . ĠThe ĠSpear man Ġcorrelation Ġbetween Ġthe Ġhuman Ġscores Ġand Ġthe Ġmodel Ġscores Ġare Ġcomputed . ĠTo Ġobtain Ġthe Ġmodel Ġscore , Ġthe Ġfollowing Ġmetrics Ġare Ġused : </s> Max Cos : ĠMaximum Ġcos ine Ġsimilarity Ġamong Ġall Ġcomponent Ġpairs Ġof Ġwords Ġ$ w $ Ġand Ġ$ v $ : </s> Avg Cos : ĠAverage Ġcomponent - wise Ġcos ine Ġsimilarity Ġbetween Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $. </s> K L $ \ _ $ app rox : ĠForm ulated Ġas Ġshown Ġin Ġ( DIS PLAY _ FORM 8 ) Ġbetween Ġthe Ġwords Ġ$ w $ Ġand Ġ$ v $. </s> K L $ \ _ $ comp : ĠMaximum Ġcomponent - wise Ġnegative ĠKL Ġbetween Ġwords Ġ$ w $ Ġand Ġ$ v $ : </s> Table ĠT AB REF 17 Ġcompares Ġthe Ġperformance Ġof Ġthe Ġapproaches Ġon Ġthe ĠSC WS Ġdataset . ĠIt Ġis Ġevident Ġfrom ĠTable ĠT AB REF 17 Ġthat ĠGM $ \ _ $ K L Ġachieves Ġbetter Ġcorrelation Ġthan Ġexisting Ġapproaches Ġfor Ġvarious Ġmetrics Ġon ĠSC WS Ġdataset . </s> Table ĠT AB REF 18 Ġshows Ġthe ĠSpear man Ġcorrelation Ġvalues Ġof ĠGM $ \ _ $ K L Ġmodel Ġevaluated Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġdatasets : ĠSL ĠB IB REF 20 , ĠWS , ĠWS - R , ĠWS - S ĠB IB REF 21 , ĠMEN ĠB IB REF 22 , ĠMC ĠB IB REF 23 , ĠRG ĠB IB REF 24 , ĠY P ĠB IB REF 25 , ĠM Tur k - 287 Ġand ĠM Tur k - 771 ĠB IB REF 26 , ĠB IB REF 27 , Ġand ĠRW ĠB IB REF 28 . ĠThe Ġmetric Ġused Ġfor Ġcomparison Ġis Ġ' Avg Cos '. ĠIt Ġcan Ġbe Ġseen Ġthat Ġfor Ġmost Ġof Ġthe Ġdatasets , ĠGM $ \ _ $ K L Ġachieves Ġsignificantly Ġbetter Ġcorrelation Ġscore Ġthan Ġw 2 g Ġand Ġw 2 gm Ġapproaches . ĠOther Ġdatasets Ġsuch Ġas ĠMC Ġand ĠRW Ġconsist Ġof Ġonly Ġa Ġsingle Ġsense , Ġand Ġhence Ġw 2 g Ġmodel Ġperforms Ġbetter Ġand ĠGM $ \ _ $ K L Ġachieves Ġnext Ġbetter Ġperformance . ĠThe ĠY P Ġdataset Ġhave Ġmultiple Ġsenses Ġbut Ġdoes Ġnot Ġcontain Ġent ailed Ġdata Ġand Ġhence Ġcould Ġnot Ġmake Ġuse Ġof Ġentail ment Ġbenefits Ġof ĠGM $ \ _ $ K L . </s> Table ĠT AB REF 19 Ġshows Ġthe Ġevaluation Ġresults Ġof ĠGM $ \ _ $ K L Ġmodel Ġon Ġthe Ġentail ment Ġdatasets Ġsuch Ġas Ġentail ment Ġpairs Ġdataset ĠB IB REF 29 Ġcreated Ġfrom ĠWord Net Ġwith Ġboth Ġpositive Ġand Ġnegative Ġlabels , Ġa Ġcrowds ourced Ġdataset ĠB IB REF 30 Ġof Ġ79 Ġsemantic Ġrelations Ġlabelled Ġas Ġent ailed Ġor Ġnot Ġand Ġannot ated Ġdistribution ally Ġsimilar Ġnoun s Ġdataset ĠB IB REF 31 . ĠThe Ġ' Max Cos ' Ġsimilarity Ġmetric Ġis Ġused Ġfor Ġevaluation Ġand Ġthe Ġbest Ġprecision Ġand Ġbest ĠF 1 - score Ġis Ġshown , Ġby Ġpicking Ġthe Ġoptimal Ġthreshold . ĠOverall , ĠGM $ \ _ $ K L Ġperforms Ġbetter Ġthan Ġboth Ġw 2 g Ġand Ġw 2 gm Ġapproaches . </s> Conclusion </s> We Ġproposed Ġa ĠKL Ġdivergence Ġbased Ġenergy Ġfunction Ġfor Ġlearning Ġmulti - sense Ġword Ġembed ding Ġdistributions Ġmod elled Ġas ĠGa ussian Ġm ixtures . ĠDue Ġto Ġthe Ġint ract ability Ġof Ġthe ĠGa ussian Ġm ixtures Ġfor Ġthe ĠKL Ġdivergence Ġmeasure , Ġwe Ġuse Ġan Ġapproximate ĠKL Ġdivergence Ġfunction . ĠWe Ġalso Ġdemonstrated Ġthat Ġthe Ġproposed ĠGM $ \ _ $ K L Ġapproaches Ġperformed Ġbetter Ġthan Ġother Ġapproaches Ġon Ġthe Ġbenchmark Ġword Ġsimilarity Ġand Ġentail ment Ġdatasets . </s> toc section App end ices </s> App rox imation Ġfor ĠKL Ġdivergence Ġbetween Ġm ixtures Ġof Ġga uss ians </s> K L Ġbetween Ġga ussian Ġm ixtures Ġ$ f _{ w } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġand Ġ$ f _{ v } (\ oper at orn ame {\ math bf Ġ{ x }} ) $ Ġcan Ġbe Ġdecomp osed Ġas : </s> BI B REF 12 Ġpresents ĠKL Ġapproximation Ġbetween Ġga ussian Ġm ixtures Ġusing </s> product Ġof Ġga ussian Ġapproximation Ġmethod Ġwhere ĠKL Ġis Ġapprox imated Ġusing Ġproduct Ġof Ġcomponent Ġga uss ians Ġand </s> vari ational Ġapproximation Ġmethod Ġwhere ĠKL Ġis Ġapprox imated Ġby Ġintroducing Ġsome Ġvari ational Ġparameters . </s> The Ġproduct Ġof Ġcomponent Ġga ussian Ġapproximation Ġmethod Ġusing ĠJensen 's Ġinequality Ġprovides Ġupper Ġbounds Ġas Ġshown Ġin Ġequations ĠDIS PLAY _ FORM 23 Ġand Ġ. </s> The Ġvari ational Ġapproximation Ġmethod Ġprovides Ġlower Ġbounds Ġas Ġshown Ġin Ġequations ĠDIS PLAY _ FORM 24 Ġand ĠDIS PLAY _ FORM 25 . </s> where Ġ$ H $ Ġrepresents Ġthe Ġentropy Ġterm Ġand Ġthe Ġentropy Ġof Ġ$ i ^{ th } $ Ġcomponent Ġof Ġword Ġ$ w $ Ġwith Ġdimension Ġ$ D $ Ġis Ġgiven Ġas </s> In ĠB IB REF 13 , Ġthe Ġauthors Ġcombine Ġthe Ġlower Ġand Ġupper Ġbounds Ġfrom Ġapproximation Ġmethods Ġof ĠB IB REF 12 Ġto Ġformulate Ġa Ġstricter Ġbound Ġon ĠKL Ġbetween Ġga ussian Ġm ixtures . </s> From Ġequations ĠDIS PLAY _ FORM 23 Ġand ĠDIS PLAY _ FORM 25 , Ġa Ġstricter Ġlower Ġbound Ġfor ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġobtained Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 26 </s> From Ġequations Ġand ĠDIS PLAY _ FORM 24 , Ġa Ġstricter Ġupper Ġbound Ġfor ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġobtained Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 27 </s> Finally , Ġthe ĠKL Ġbetween Ġga ussian Ġm ixtures Ġis Ġtaken Ġas Ġthe Ġmean Ġof ĠKL Ġupper Ġand Ġlower Ġbounds Ġas Ġshown Ġin Ġequation ĠDIS PLAY _ FORM 28 .