$\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus
EXTRACTIVE
A finite corpora may entirely omit rare word combinations
ABSTRACTIVE
Unfortunately, $\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus. Due to unreliable statistics, this happens very frequently in finite corpora. Many models work around this issue by clipping negative $\mathit {PMI}$ values at 0, a measure known as Positive $\mathit {PMI}$ ($\mathit {PPMI}$), which works very well in practice. An unanswered question is: “What is lost/gained by collapsing the negative $\mathit {PMI}$ spectrum to 0?”. Understanding which type of information is captured by $\mathit {\texttt {-}PMI}$ can help in tailoring models for optimal performance.
Unfortunately, $\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus. Due to unreliable statistics, this happens very frequently in finite corpora. Many models work around this issue by clipping negative $\mathit {PMI}$ values at 0, a measure known as Positive $\mathit {PMI}$ ($\mathit {PPMI}$), which works very well in practice. An unanswered question is: “What is lost/gained by collapsing the negative $\mathit {PMI}$ spectrum to 0?”. Understanding which type of information is captured by $\mathit {\texttt {-}PMI}$ can help in tailoring models for optimal performance.
1908.06941
Introduction </s> D ense Ġword Ġvectors Ġ( or Ġembed d ings ) Ġare Ġa Ġkey Ġcomponent Ġin Ġmodern ĠN LP Ġarchitectures Ġfor Ġtasks Ġsuch Ġas Ġsentiment Ġanalysis , Ġparsing , Ġand Ġmachine Ġtranslation . ĠThese Ġvectors Ġcan Ġbe Ġlearned Ġby Ġexploiting Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 0 , Ġparaph r ased Ġby ĠB IB REF 1 Ġas ĠâĢ ľ a Ġword Ġis Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps âĢ Ŀ , Ġusually Ġby Ġconstructing Ġa Ġco occ urrence Ġmatrix Ġover Ġa Ġtraining Ġcorpus , Ġre - weight ing Ġit Ġusing ĠPoint wise ĠMutual ĠInformation Ġ($ \ math it Ġ{ PM I } $ ) ĠB IB REF 2 , Ġand Ġperforming Ġa Ġlow - rank Ġfactor ization Ġto Ġobtain Ġdense Ġvectors . </s> Unfortunately , Ġ$\ math it Ġ{ PM I } ( w , c ) $ Ġgoes Ġto Ġnegative Ġinfinity Ġwhen Ġthe Ġword - context Ġpair Ġ$( w , c ) $ Ġdoes Ġnot Ġappear Ġin Ġthe Ġtraining Ġcorpus . ĠDue Ġto Ġunreliable Ġstatistics , Ġthis Ġhappens Ġvery Ġfrequently Ġin Ġfinite Ġcorpor a . ĠMany Ġmodels Ġwork Ġaround Ġthis Ġissue Ġby Ġclipping Ġnegative Ġ$\ math it Ġ{ PM I } $ Ġvalues Ġat Ġ0 , Ġa Ġmeasure Ġknown Ġas ĠPositive Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{ P PM I } $ ), Ġwhich Ġworks Ġvery Ġwell Ġin Ġpractice . ĠAn Ġunanswered Ġquestion Ġis : ĠâĢ ľ What Ġis Ġlost / g ained Ġby Ġcollapsing Ġthe Ġnegative Ġ$\ math it Ġ{ PM I } $ Ġspectrum Ġto Ġ0 ? âĢ Ŀ . ĠUnderstanding Ġwhich Ġtype Ġof Ġinformation Ġis Ġcaptured Ġby Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġcan Ġhelp Ġin Ġtail oring Ġmodels Ġfor Ġoptimal Ġperformance . </s> In Ġthis Ġwork , Ġwe Ġattempt Ġto Ġanswer Ġthis Ġquestion Ġby Ġstudying Ġthe Ġkind Ġof Ġinformation Ġcontained Ġin Ġthe Ġnegative Ġand Ġpositive Ġspect rums Ġof Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ ). ĠWe Ġevaluate Ġweighted Ġfactor ization Ġof Ġdifferent Ġmat rices Ġwhich Ġuse Ġeither Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $, Ġor Ġboth Ġon Ġvarious Ġsemantic Ġand Ġsynt actic Ġtasks . ĠResults Ġshow Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġalone Ġperforms Ġquite Ġwell Ġon Ġmost Ġtasks , Ġcapturing Ġboth Ġsemantics Ġand Ġsyntax , Ġin Ġcontrast Ġto Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġwhich Ġperforms Ġpoorly Ġon Ġnearly Ġall Ġtasks , Ġexcept Ġthose Ġthat Ġtest Ġfor Ġsyntax . ĠOur Ġmain Ġcontribution Ġis Ġdeepening Ġour Ġunderstanding Ġof Ġdistribution al Ġsemantics Ġby Ġextending ĠB IB REF 1 's Ġparaph rase Ġof Ġthe Ġdistribution al Ġhypothesis Ġto ĠâĢ ľ a Ġword Ġis Ġnot Ġonly Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps , Ġbut Ġalso Ġby Ġthe Ġcompany Ġit Ġrejects âĢ Ŀ . ĠOur Ġsecondary Ġcontributions Ġare Ġthe Ġproposal Ġof Ġtwo Ġ$ PM I $ Ġvariants Ġthat Ġaccount Ġfor Ġthe Ġspectrum Ġof Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġand Ġthe Ġjustification Ġof Ġthe Ġpopular Ġ$ P PM I $ Ġmeasure . </s> In Ġthis Ġpaper , Ġwe Ġfirst Ġlook Ġat Ġrelated Ġwork Ġ($ Â§ $ SEC REF 2 ), Ġthen Ġstudy Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġways Ġof Ġaccounting Ġfor Ġit Ġ($ Â§ $ SEC REF 3 ), Ġdescribe Ġexperiments Ġ($ Â§ $ SEC REF 4 ), Ġanalyze Ġresults Ġ($ Â§ $ SEC REF 5 ), Ġand Ġclose Ġwith Ġideas Ġfor Ġfuture Ġwork Ġ($ Â§ $ SEC REF 6 ). </s> Related ĠWork </s> There Ġis Ġa Ġlong Ġhistory Ġof Ġstudying Ġweight ings Ġ( also Ġknown Ġas Ġassociation Ġmeasures ) Ġof Ġgeneral Ġ( not Ġonly Ġword - context ) Ġco occ urrence Ġmat rices ; Ġsee ĠB IB REF 3 , ĠB IB REF 4 Ġfor Ġan Ġoverview Ġand ĠB IB REF 5 Ġfor Ġcomparison Ġof Ġdifferent Ġweight ings . ĠB IB REF 6 Ġshow Ġthat Ġword Ġvectors Ġderived Ġfrom Ġ$\ math it Ġ{ P PM I } $ Ġmat rices Ġperform Ġbetter Ġthan Ġalternative Ġweight ings Ġfor Ġword - context Ġco occ urrence . ĠIn Ġthe Ġfield Ġof Ġcoll ocation Ġextraction , ĠB IB REF 7 Ġaddress Ġthe Ġnegative Ġinfinity Ġissue Ġwith Ġ$\ math it Ġ{ PM I } $ Ġby Ġintroducing Ġthe Ġnormalized Ġ$\ math it Ġ{ PM I } $ Ġmetric . ĠB IB REF 8 Ġshow Ġtheoretically Ġthat Ġthe Ġpopular ĠSkip - gram Ġmodel ĠB IB REF 9 Ġperforms Ġimplicit Ġfactor ization Ġof Ġshifted Ġ$\ math it Ġ{ PM I } $. </s> Recently , Ġwork Ġin Ġexplicit Ġlow - rank Ġmatrix Ġfactor ization Ġof Ġ$\ math it Ġ{ PM I } $ Ġvariants Ġhas Ġachieved Ġstate Ġof Ġthe Ġart Ġresults Ġin Ġword Ġembed ding . ĠGl o Ve ĠB IB REF 10 Ġperforms Ġweighted Ġfactor ization Ġof Ġthe Ġlog Ġco occ urrence Ġmatrix Ġwith Ġadded Ġbias Ġterms , Ġbut Ġdoes Ġnot Ġaccount Ġfor Ġzero Ġcells . ĠB IB REF 11 Ġpoint Ġout Ġthat ĠGl o Ve 's Ġbias Ġterms Ġcorrelate Ġstrongly Ġwith Ġun ig ram Ġlog Ġcounts , Ġsuggesting Ġthat ĠGl o Ve Ġis Ġfactor izing Ġa Ġvariant Ġof Ġ$\ math it Ġ{ PM I } $. ĠTheir ĠSw i Vel Ġmodel Ġmod ifies Ġthe ĠGl o Ve Ġobjective Ġto Ġuse ĠLa place Ġsmoot hing Ġand Ġhinge Ġloss Ġfor Ġzero Ġcounts Ġof Ġthe Ġco occ urrence Ġmatrix , Ġdirectly Ġfactor izing Ġthe Ġ$\ math it Ġ{ PM I } $ Ġmatrix , Ġsid este pping Ġthe Ġnegative Ġinfinity Ġissue . ĠAn Ġalternative Ġis Ġto Ġuse Ġ$\ math it Ġ{ P PM I } $ Ġand Ġvariants Ġas Ġin ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 . ĠHowever , Ġit Ġis Ġnot Ġclear Ġwhat Ġis Ġlost Ġby Ġclipping Ġthe Ġnegative Ġspectrum Ġof Ġ$\ math it Ġ{ PM I } $, Ġwhich Ġmakes Ġthe Ġuse Ġof Ġ$\ math it Ġ{ P PM I } $, Ġthough Ġit Ġworks Ġwell Ġin Ġpractice , Ġseem Ġun pr in cipled . </s> In Ġthe Ġstudy Ġof Ġlanguage Ġacquisition , ĠB IB REF 17 Ġargue Ġthat Ġindirect Ġnegative Ġevidence Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġhuman Ġacquisition Ġof Ġgrammar , Ġbut Ġdo Ġnot Ġlink Ġthis Ġidea Ġto Ġdistribution al Ġsemantics . </s> PM I Ġ& ĠMatrix ĠFactor ization </s> PM I : ĠA Ġco occ urrence Ġmatrix Ġ$ M $ Ġis Ġconstructed Ġby Ġsliding Ġa Ġsymm etric Ġwindow Ġover Ġthe Ġsubs am pled ĠB IB REF 9 Ġtraining Ġcorpus Ġand Ġfor Ġeach Ġcenter Ġword Ġ$ w $ Ġand Ġcontext Ġword Ġ$ c $ Ġwithin Ġthe Ġwindow , Ġincrement ing Ġ$ M _{ w c } $. Ġ$\ math it Ġ{ PM I } $ Ġis Ġthen Ġequal Ġto : </s> where Ġ* Ġdenotes Ġsumm ation Ġover Ġthe Ġcorresponding Ġindex . ĠTo Ġdeal Ġwith Ġnegative Ġvalues , Ġwe Ġpropose Ġclipped Ġ$\ math it Ġ{ PM I } $, </s> which Ġis Ġequivalent Ġto Ġ$\ math it Ġ{ P PM I } $ Ġwhen Ġ$ z Ġ= Ġ0 $. </s> Matrix Ġfactor ization : ĠLex V ec ĠB IB REF 15 Ġperforms Ġthe Ġfactor ization Ġ$ M ^ {\ prime Ġ} Ġ= ĠWC ^ \ top Ġ$ , Ġwhere Ġ$ M ^ {\ prime Ġ} $ Ġis Ġany Ġtransformation Ġof Ġ$ M $ Ġ( such Ġas Ġ$\ math it Ġ{ P PM I } $ ), Ġand Ġ$ W , ĠC $ Ġare Ġthe Ġword Ġand Ġcontext Ġembed d ings Ġrespectively . ĠBy Ġsliding Ġa Ġsymm etric Ġwindow Ġover Ġthe Ġtraining Ġcorpus Ġ( window Ġsampling ), ĠLex V ec Ġperforms Ġone ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġstep Ġevery Ġtime Ġa Ġ$( w , c ) $ Ġpair Ġis Ġobserved , Ġminimizing </s> Additionally , Ġfor Ġevery Ġcenter Ġword Ġ$ w $, Ġ$ k $ Ġnegative Ġwords ĠB IB REF 9 Ġare Ġdrawn Ġfrom Ġthe Ġun ig ram Ġcontext Ġdistribution Ġ$ P _ n $ Ġ( negative Ġsampling ) Ġand ĠSG D Ġsteps Ġtaken Ġto Ġminimize : </s> Thus Ġthe Ġloss Ġfunction Ġprior itiz es Ġthe Ġcorrect Ġapproximation Ġof Ġfrequently Ġco occ urring Ġpairs Ġand Ġof Ġpairs Ġwhere Ġeither Ġword Ġoccurs Ġwith Ġhigh Ġfrequency ; Ġthese Ġare Ġpairs Ġfor Ġwhich Ġwe Ġhave Ġmore Ġreliable Ġstatistics . </s> In Ġour Ġexperiments , Ġwe Ġuse ĠLex V ec Ġover ĠSing ular ĠValue ĠDec om position Ġ( S VD ) Ġbecause Ġa ) ĠEm pir ical Ġresults Ġshows Ġit Ġoutper forms ĠS VD ĠB IB REF 15 . Ġb ) ĠThe Ġweight ing Ġof Ġreconstruction Ġerrors Ġby Ġstatistical Ġconfidence Ġis Ġparticularly Ġimportant Ġfor Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġwhere Ġnegative Ġco occ urrence Ġbetween Ġa Ġpair Ġof Ġfrequent Ġwords Ġis Ġmore Ġsignificant Ġand Ġshould Ġbe Ġbetter Ġapprox imated Ġthan Ġthat Ġbetween Ġa Ġpair Ġof Ġrare Ġwords . ĠGl o Ve 's Ġmatrix Ġfactor ization Ġis Ġeven Ġmore Ġunsu itable Ġfor Ġour Ġexperiments Ġas Ġits Ġloss Ġweight ing ĠâĢĶ Ġa Ġmon oton ically Ġincreasing Ġfunction Ġof Ġ$ M _{ w c } $ ĠâĢĶ Ġignores Ġreconstruction Ġerrors Ġof Ġnon - co occ urring Ġpairs . </s> Spect rum Ġof ĠPM I : ĠTo Ġbetter Ġunderstand Ġthe Ġdistribution Ġof Ġ$\ math it Ġ{ C PM I } $ Ġvalues , Ġwe Ġplot Ġa Ġhist ogram Ġof Ġ$ 10 ^ 5 $ Ġpairs Ġrandomly Ġsampled Ġby Ġwindow Ġsampling Ġand Ġnegative Ġsampling Ġin Ġfig : hist , Ġsetting Ġ$ z =- 5 $. ĠWe Ġcan Ġclearly Ġsee Ġthe Ġspectrum Ġof Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġthat Ġis Ġcollapsed Ġwhen Ġwe Ġuse Ġ$\ math it Ġ{ P PM I } $ Ġ($ z = 0 $ ). ĠIn Ġpractice Ġwe Ġfind Ġthat Ġ$ z =- 2 $ Ġcaptures Ġmost Ġof Ġthe Ġnegative Ġspectrum Ġand Ġconsistently Ġgives Ġbetter Ġresults Ġthan Ġsmaller Ġvalues Ġso Ġwe Ġuse Ġthis Ġvalue Ġfor Ġthe Ġrest Ġof Ġthis Ġpaper . ĠWe Ġsuspect Ġthis Ġis Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġnon - co occ urring Ġpairs Ġ($ 41 . 7 \ % $ Ġin Ġthis Ġsample ) Ġwhich Ġend Ġup Ġdominating Ġthe Ġloss Ġfunction Ġwhen Ġ$ z $ Ġis Ġtoo Ġsmall . </s> Normal ization : ĠWe Ġalso Ġexperiment Ġwith Ġnormalized Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{ N PM I } $ ) ĠB IB REF 7 : </s> such Ġthat Ġ$ N PM I ( w , c ) Ġ= Ġ- 1 $ Ġwhen Ġ$( w , c ) $ Ġnever Ġco occ ur , Ġ$ N PM I ( w , c ) Ġ= Ġ0 $ Ġwhen Ġthey Ġare Ġindependent , Ġand Ġ$ N PM I ( w , c ) Ġ= Ġ1 $ Ġwhen Ġthey Ġalways Ġco occ ur Ġtogether . ĠThis Ġeffectively Ġcaptures Ġthe Ġentire Ġnegative Ġspectrum , Ġbut Ġhas Ġthe Ġdownside Ġof Ġnormal ization Ġwhich Ġdisc ards Ġscale Ġinformation . ĠIn Ġpractice Ġwe Ġfind Ġthis Ġworks Ġpoorly Ġif Ġdone Ġsymm etr ically , Ġso Ġwe Ġintroduce Ġa Ġvariant Ġcalled Ġ$\ math it Ġ{ NN EG PM I } $ Ġwhich Ġonly Ġnormal izes Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ : </s> We Ġalso Ġexperimented Ġwith ĠLa place Ġsmoot hing Ġas Ġin ĠB IB REF 18 Ġfor Ġvarious Ġpseud oc ount s Ġbut Ġfound Ġit Ġto Ġwork Ġconsistently Ġworse Ġthan Ġboth Ġ$\ math it Ġ{ C PM I _ z } $ Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġso Ġwe Ġomit Ġfurther Ġdiscussion Ġin Ġthis Ġpaper . </s> Materials </s> In Ġorder Ġto Ġidentify Ġthe Ġrole Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġplay Ġin Ġdistribution al Ġsemantics , Ġwe Ġtrain ĠLex V ec Ġmodels Ġthat Ġskip ĠSG D Ġsteps Ġwhen Ġtarget Ġcell Ġvalues Ġare Ġ$ > 0 $ Ġor Ġ$\ le Ġ0 $, Ġrespectively . ĠFor Ġexample , Ġ$ - \ math it Ġ{ C PM I } _{ \ text tt Ġ{ - } 2 } $ Ġsk ips Ġsteps Ġwhen Ġ$\ math it Ġ{ C PM I } _{ \ text tt Ġ{ - } 2 } ( w , c ) Ġ> Ġ0 $. ĠSimilarly , Ġthe Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġmodel Ġsk ips ĠSG D Ġsteps Ġwhen Ġ$\ math it Ġ{ P PM I } ( w , c ) Ġ\ le Ġ0 $. ĠWe Ġcompare Ġthese Ġto Ġmodels Ġthat Ġinclude Ġboth Ġnegative Ġand Ġpositive Ġinformation Ġto Ġsee Ġhow Ġthe Ġtwo Ġinteract . </s> We Ġuse Ġthe Ġdefault ĠLex V ec Ġconfiguration Ġfor Ġall Ġ$\ math it Ġ{ PM I } $ Ġvariants : Ġfixed Ġwindow Ġof Ġsize Ġ2 , Ġembed ding Ġdimension Ġof Ġ300 , Ġ5 Ġnegative Ġsamples , Ġpositional Ġcontexts , Ġcontext Ġdistribution Ġsmoot hing Ġof Ġ$ . 75 $, Ġlearning Ġrate Ġof Ġ$ . 025 $, Ġno Ġsub word Ġinformation , Ġand Ġnegative Ġdistribution Ġpower Ġof Ġ$ . 75 $. ĠWe Ġtrain Ġon Ġa Ġlower c ased , Ġal phan umer ical Ġ2015 ĠWikipedia Ġdump Ġwith Ġ$ 3 . 8 $ B Ġtokens , Ġdisc arding Ġtokens Ġwith Ġfrequency Ġ$ < Ġ100 $, Ġfor Ġa Ġvocabulary Ġsize Ġof Ġ$ 303 , 517 $ Ġwords . </s> For Ġcomparison , Ġwe Ġinclude Ġresults Ġfor Ġa Ġrandomly Ġinitialized , Ġnon - trained Ġembed ding Ġto Ġestablish Ġtask Ġbas elines . </s> Sem antics : ĠTo Ġevaluate Ġword - level Ġsemantics , Ġwe Ġuse Ġthe ĠSim Lex ĠB IB REF 19 Ġand ĠRare ĠWord Ġ( RW ) ĠB IB REF 20 Ġword Ġsimilarity Ġdatasets , Ġand Ġthe ĠGoogle ĠSem antic Ġ( G Sem ) Ġanalog ies ĠB IB REF 9 . ĠWe Ġevaluate Ġsentence - level Ġsemantics Ġusing Ġaveraged Ġbag Ġof Ġvectors Ġ( Bo V ) Ġrepresentations Ġon Ġthe ĠSem antic ĠText ual ĠSimilar ity Ġ( ST SB ) Ġtask ĠB IB REF 21 Ġand ĠWord ĠContent Ġ( WC ) Ġprobing Ġtask Ġ( ident ify Ġfrom Ġa Ġlist Ġof Ġwords Ġwhich Ġis Ġcontained Ġin Ġthe Ġsentence Ġrepresentation ) Ġfrom ĠSent E val ĠB IB REF 22 . </s> Sy ntax : ĠSimilarly , Ġwe Ġuse Ġthe ĠGoogle ĠSynt actic Ġanalog ies Ġ( GS yn ) ĠB IB REF 9 Ġto Ġevaluate Ġword - level Ġsynt actic Ġinformation , Ġand ĠDepth Ġ( Dep ) Ġand ĠTop ĠConst itu ent Ġ( Top C ) Ġ( of Ġthe Ġinput Ġsentence 's Ġconstituent Ġparse Ġtree ) Ġprobing Ġtasks Ġfrom ĠSent E val ĠB IB REF 22 Ġfor Ġsentence - level Ġsyntax . ĠClass ifiers Ġfor Ġall ĠSent E val Ġprobing Ġtasks Ġare Ġmult il ayer Ġpercept rons Ġwith Ġa Ġsingle Ġhidden Ġlayer Ġof Ġ100 Ġunits Ġand Ġdrop out Ġof Ġ$ . 1 $. ĠOur Ġfinal Ġsynt actic Ġtask Ġis Ġpart - of - speech Ġ( POS ) Ġtagging Ġusing Ġthe Ġsame ĠBi L ST M - CR F Ġsetup Ġas ĠB IB REF 23 Ġbut Ġusing Ġonly Ġword Ġembed d ings Ġ( no Ġhand - engine ered Ġfeatures ) Ġas Ġinput , Ġtrained Ġon Ġthe ĠWS J Ġsection Ġof Ġthe ĠPenn ĠTree bank ĠB IB REF 24 . </s> Results </s> All Ġresults Ġare Ġshown Ġin Ġtab : sent eval . </s> Neg ative ĠPM I : ĠWe Ġobserve Ġthat Ġusing Ġonly Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġ( rows Ġ$\ math it Ġ{\ text tt Ġ{ - } C PM I _{ \ text tt Ġ{ - } 2 }} $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ - } NN EG PM I } $ ) Ġperforms Ġsimilarly Ġto Ġall Ġother Ġmodels Ġin ĠPOS Ġtagging Ġand Ġboth Ġsynt actic Ġprobing Ġtasks , Ġbut Ġvery Ġpoorly Ġon Ġall Ġsemantic Ġtasks , Ġstrongly Ġsupporting Ġour Ġmain Ġclaim Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġmostly Ġenc odes Ġsynt actic Ġinformation . </s> Our Ġhypothesis Ġfor Ġthis Ġis Ġthat Ġthe Ġgrammar Ġthat Ġgenerates Ġlanguage Ġimplicitly Ġcreates Ġnegative Ġco occ urrence Ġand Ġso Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġenc odes Ġthis Ġsynt actic Ġinformation . ĠInterestingly , Ġthis Ġidea Ġcreates Ġa Ġbridge Ġbetween Ġdistribution al Ġsemantics Ġand Ġthe Ġargument Ġby ĠB IB REF 17 Ġthat Ġindirect Ġnegative Ġevidence Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġhuman Ġlanguage Ġacquisition Ġof Ġgrammar . ĠPositive ĠPM I : ĠThe Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġmodel Ġperforms Ġas Ġwell Ġor Ġbetter Ġas Ġthe Ġfull Ġspectrum Ġmodels Ġon Ġnearly Ġall Ġtasks , Ġclearly Ġindicating Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġenc odes Ġboth Ġsemantic Ġand Ġsynt actic Ġinformation . </s> Why Ġincorporate Ġ- PM I ? Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġonly Ġfal ters Ġon Ġthe ĠRW Ġand Ġanalogy Ġtasks , Ġand Ġwe Ġhypothes ize Ġthis Ġis Ġwhere Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġis Ġuseful : Ġin Ġthe Ġabsence Ġof Ġpositive Ġinformation , Ġnegative Ġinformation Ġcan Ġbe Ġused Ġto Ġimprove Ġrare Ġword Ġrepresentations Ġand Ġword Ġanalog ies . ĠAnalog ies Ġare Ġsolved Ġusing Ġnearest Ġneighbor Ġlook ups Ġin Ġthe Ġvector Ġspace , Ġand Ġso Ġaccounting Ġfor Ġnegative Ġco occ urrence Ġeffectively Ġrep els Ġwords Ġwith Ġwhich Ġno Ġpositive Ġco occ urrence Ġwas Ġobserved . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġexplore Ġincorporating Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġonly Ġfor Ġrare Ġwords Ġ( where Ġit Ġis Ġmost Ġneeded ). </s> Full Ġspectrum Ġmodels : ĠThe Ġ$\ math it Ġ{ P PM I } $, Ġ$\ math it Ġ{ C PM I _{ \ text tt Ġ{ - } 2 }} $, Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġmodels Ġperform Ġsimilarly , Ġwhereas Ġthe Ġ$\ math it Ġ{ N PM I } $ Ġmodel Ġis Ġsignificantly Ġworst Ġon Ġnearly Ġall Ġsemantic Ġtasks . ĠWe Ġthus Ġconclude Ġthat Ġaccounting Ġfor Ġscale Ġin Ġthe Ġpositive Ġspectrum Ġis Ġmore Ġimportant Ġthan Ġin Ġthe Ġnegative Ġspectrum . ĠWe Ġhypothes ize Ġthis Ġis Ġbecause Ġscale Ġhelps Ġto Ġuniquely Ġidentify Ġwords , Ġwhich Ġis Ġcritical Ġfor Ġsemantics Ġ( results Ġon Ġ$ WC $ Ġtask Ġcorrelate Ġstrongly Ġwith Ġperformance Ġon Ġsemantic Ġtasks ), Ġbut Ġin Ġsyntax , Ġwords Ġwith Ġthe Ġsame Ġfunction Ġshould Ġbe Ġindistinguishable . ĠSince Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġenc odes Ġboth Ġsemantics Ġand Ġsyntax , Ġscale Ġmust Ġbe Ġpreserved , Ġwhereas Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġenc odes Ġmostly Ġsyntax , Ġand Ġso Ġscale Ġinformation Ġcan Ġbe Ġdiscarded . </s> Coll aps ing Ġthe Ġnegative Ġspectrum : ĠThe Ġ$\ math it Ġ{ P PM I } $ Ġmodel , Ġwhich Ġcollapses Ġthe Ġnegative Ġspectrum Ġto Ġzero , Ġperforms Ġalmost Ġident ically Ġto Ġthe Ġ$\ math it Ġ{ C PM I _{ \ text tt Ġ{ - } 2 }} $ Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġmodels Ġthat Ġaccount Ġfor Ġthe Ġrange Ġof Ġnegative Ġvalues . ĠThis Ġis Ġjustified Ġby Ġ1 ) ĠOur Ġdiscussion Ġwhich Ġshows Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġis Ġfar Ġmore Ġinformative Ġthan Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ2 ) ĠLooking Ġat Ġfig : hist , Ġwe Ġsee Ġthat Ġcollapsed Ġvalues ĠâĢĶ Ġinterval Ġ$( - 5 , 0 ] $ ĠâĢĶ Ġaccount Ġfor Ġonly Ġ$ 11 \ % $ Ġof Ġsamples Ġcompared Ġto Ġ$ 41 . 7 \ % $ Ġfor Ġnon - coll apsed Ġnegative Ġvalues . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġevaluated Ġexisting Ġand Ġnovel Ġways Ġof Ġincorporating Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġinto Ġword Ġembed ding Ġmodels Ġbased Ġon Ġexplicit Ġweighted Ġmatrix Ġfactor ization , Ġand , Ġmore Ġimportantly , Ġstudied Ġthe Ġrole Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġeach Ġplay Ġin Ġdistribution al Ġsemantics , Ġfinding Ġthat ĠâĢ ľ a Ġword Ġis Ġnot Ġonly Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps , Ġbut Ġalso Ġby Ġthe Ġcompany Ġit Ġrejects âĢ Ŀ . ĠIn Ġfuture Ġwork , Ġwe Ġwish Ġto Ġfurther Ġstudy Ġthe Ġlink Ġbetween Ġour Ġwork Ġand Ġlanguage Ġacquisition , Ġand Ġexplore Ġthe Ġfact Ġthe Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġis Ġalmost Ġpurely Ġsynt actic Ġto Ġ( possibly ) Ġsubtract Ġsyntax Ġfrom Ġthe Ġfull Ġspectrum Ġmodels , Ġstudying Ġthe Ġfrontier Ġ( if Ġthere Ġis Ġone ) Ġbetween Ġsemantics Ġand Ġsyntax . </s> Acknowled gments </s> This Ġresearch Ġwas Ġpartly Ġsupported Ġby ĠCAP ES Ġand ĠC NP q Ġ( projects Ġ312 114 / 2015 - 0 , Ġ4 238 43 / 2016 - 8 , Ġand Ġ140 402 / 2018 - 7 ).
Introduction Dense word vectors (or embeddings) are a key component in modern NLP architectures for tasks such as sentiment analysis, parsing, and machine translation. These vectors can be learned by exploiting the distributional hypothesis BIBREF0, paraphrased by BIBREF1 as “a word is characterized by the company that it keeps”, usually by constructing a cooccurrence matrix over a training corpus, re-weighting it using Pointwise Mutual Information ($\mathit {PMI}$) BIBREF2, and performing a low-rank factorization to obtain dense vectors. Unfortunately, $\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus. Due to unreliable statistics, this happens very frequently in finite corpora. Many models work around this issue by clipping negative $\mathit {PMI}$ values at 0, a measure known as Positive $\mathit {PMI}$ ($\mathit {PPMI}$), which works very well in practice. An unanswered question is: “What is lost/gained by collapsing the negative $\mathit {PMI}$ spectrum to 0?”. Understanding which type of information is captured by $\mathit {\texttt {-}PMI}$ can help in tailoring models for optimal performance. In this work, we attempt to answer this question by studying the kind of information contained in the negative and positive spectrums of $\mathit {PMI}$ ($\mathit {\texttt {-}PMI}$ and $\mathit {\texttt {+}PMI}$). We evaluate weighted factorization of different matrices which use either $\mathit {\texttt {-}PMI}$, $\mathit {\texttt {+}PMI}$, or both on various semantic and syntactic tasks. Results show that $\mathit {\texttt {+}PMI}$ alone performs quite well on most tasks, capturing both semantics and syntax, in contrast to $\mathit {\texttt {-}PMI}$, which performs poorly on nearly all tasks, except those that test for syntax. Our main contribution is deepening our understanding of distributional semantics by extending BIBREF1's paraphrase of the distributional hypothesis to “a word is not only characterized by the company that it keeps, but also by the company it rejects”. Our secondary contributions are the proposal of two $PMI$ variants that account for the spectrum of $\mathit {\texttt {-}PMI}$, and the justification of the popular $PPMI$ measure. In this paper, we first look at related work ($§$SECREF2), then study $\mathit {\texttt {-}PMI}$ and ways of accounting for it ($§$SECREF3), describe experiments ($§$SECREF4), analyze results ($§$SECREF5), and close with ideas for future work ($§$SECREF6). Related Work There is a long history of studying weightings (also known as association measures) of general (not only word-context) cooccurrence matrices; see BIBREF3, BIBREF4 for an overview and BIBREF5 for comparison of different weightings. BIBREF6 show that word vectors derived from $\mathit {PPMI}$ matrices perform better than alternative weightings for word-context cooccurrence. In the field of collocation extraction, BIBREF7 address the negative infinity issue with $\mathit {PMI}$ by introducing the normalized $\mathit {PMI}$ metric. BIBREF8 show theoretically that the popular Skip-gram model BIBREF9 performs implicit factorization of shifted $\mathit {PMI}$. Recently, work in explicit low-rank matrix factorization of $\mathit {PMI}$ variants has achieved state of the art results in word embedding. GloVe BIBREF10 performs weighted factorization of the log cooccurrence matrix with added bias terms, but does not account for zero cells. BIBREF11 point out that GloVe's bias terms correlate strongly with unigram log counts, suggesting that GloVe is factorizing a variant of $\mathit {PMI}$. Their SwiVel model modifies the GloVe objective to use Laplace smoothing and hinge loss for zero counts of the cooccurrence matrix, directly factorizing the $\mathit {PMI}$ matrix, sidestepping the negative infinity issue. An alternative is to use $\mathit {PPMI}$ and variants as in BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16. However, it is not clear what is lost by clipping the negative spectrum of $\mathit {PMI}$, which makes the use of $\mathit {PPMI}$, though it works well in practice, seem unprincipled. In the study of language acquisition, BIBREF17 argue that indirect negative evidence might play an important role in human acquisition of grammar, but do not link this idea to distributional semantics. PMI & Matrix Factorization PMI: A cooccurrence matrix $M$ is constructed by sliding a symmetric window over the subsampled BIBREF9 training corpus and for each center word $w$ and context word $c$ within the window, incrementing $M_{wc}$. $\mathit {PMI}$ is then equal to: where * denotes summation over the corresponding index. To deal with negative values, we propose clipped $\mathit {PMI}$, which is equivalent to $\mathit {PPMI}$ when $z = 0$. Matrix factorization: LexVec BIBREF15 performs the factorization $M^{\prime } = WC^\top $, where $M^{\prime }$ is any transformation of $M$ (such as $\mathit {PPMI}$), and $W, C$ are the word and context embeddings respectively. By sliding a symmetric window over the training corpus (window sampling), LexVec performs one Stochastic Gradient Descent (SGD) step every time a $(w,c)$ pair is observed, minimizing Additionally, for every center word $w$, $k$ negative words BIBREF9 are drawn from the unigram context distribution $P_n$ (negative sampling) and SGD steps taken to minimize: Thus the loss function prioritizes the correct approximation of frequently cooccurring pairs and of pairs where either word occurs with high frequency; these are pairs for which we have more reliable statistics. In our experiments, we use LexVec over Singular Value Decomposition (SVD) because a) Empirical results shows it outperforms SVD BIBREF15. b) The weighting of reconstruction errors by statistical confidence is particularly important for $\mathit {\texttt {-}PMI}$, where negative cooccurrence between a pair of frequent words is more significant and should be better approximated than that between a pair of rare words. GloVe's matrix factorization is even more unsuitable for our experiments as its loss weighting — a monotonically increasing function of $M_{wc}$ — ignores reconstruction errors of non-cooccurring pairs. Spectrum of PMI: To better understand the distribution of $\mathit {CPMI}$ values, we plot a histogram of $10^5$ pairs randomly sampled by window sampling and negative sampling in fig:hist, setting $z=-5$. We can clearly see the spectrum of $\mathit {\texttt {-}PMI}$ that is collapsed when we use $\mathit {PPMI}$ ($z=0$). In practice we find that $z=-2$ captures most of the negative spectrum and consistently gives better results than smaller values so we use this value for the rest of this paper. We suspect this is due to the large number of non-cooccurring pairs ($41.7\%$ in this sample) which end up dominating the loss function when $z$ is too small. Normalization: We also experiment with normalized $\mathit {PMI}$ ($\mathit {NPMI}$) BIBREF7: such that $NPMI(w,c) = -1$ when $(w,c)$ never cooccur, $NPMI(w,c) = 0$ when they are independent, and $NPMI(w,c) = 1$ when they always cooccur together. This effectively captures the entire negative spectrum, but has the downside of normalization which discards scale information. In practice we find this works poorly if done symmetrically, so we introduce a variant called $\mathit {NNEGPMI}$ which only normalizes $\mathit {\texttt {-}PMI}$: We also experimented with Laplace smoothing as in BIBREF18 for various pseudocounts but found it to work consistently worse than both $\mathit {CPMI_z}$ and $\mathit {NNEGPMI}$ so we omit further discussion in this paper. Materials In order to identify the role that $\mathit {\texttt {-}PMI}$ and $\mathit {\texttt {+}PMI}$ play in distributional semantics, we train LexVec models that skip SGD steps when target cell values are $>0$ or $\le 0$, respectively. For example, $-\mathit {CPMI}_{\texttt {-}2}$ skips steps when $\mathit {CPMI}_{\texttt {-}2}(w,c) > 0$. Similarly, the $\mathit {\texttt {+}PPMI}$ model skips SGD steps when $\mathit {PPMI}(w,c) \le 0$. We compare these to models that include both negative and positive information to see how the two interact. We use the default LexVec configuration for all $\mathit {PMI}$ variants: fixed window of size 2, embedding dimension of 300, 5 negative samples, positional contexts, context distribution smoothing of $.75$, learning rate of $.025$, no subword information, and negative distribution power of $.75$. We train on a lowercased, alphanumerical 2015 Wikipedia dump with $3.8$B tokens, discarding tokens with frequency $< 100$, for a vocabulary size of $303,517$ words. For comparison, we include results for a randomly initialized, non-trained embedding to establish task baselines. Semantics: To evaluate word-level semantics, we use the SimLex BIBREF19 and Rare Word (RW) BIBREF20 word similarity datasets, and the Google Semantic (GSem) analogies BIBREF9. We evaluate sentence-level semantics using averaged bag of vectors (BoV) representations on the Semantic Textual Similarity (STSB) task BIBREF21 and Word Content (WC) probing task (identify from a list of words which is contained in the sentence representation) from SentEval BIBREF22. Syntax: Similarly, we use the Google Syntactic analogies (GSyn) BIBREF9 to evaluate word-level syntactic information, and Depth (Dep) and Top Constituent (TopC) (of the input sentence's constituent parse tree) probing tasks from SentEval BIBREF22 for sentence-level syntax. Classifiers for all SentEval probing tasks are multilayer perceptrons with a single hidden layer of 100 units and dropout of $.1$. Our final syntactic task is part-of-speech (POS) tagging using the same BiLSTM-CRF setup as BIBREF23 but using only word embeddings (no hand-engineered features) as input, trained on the WSJ section of the Penn Treebank BIBREF24. Results All results are shown in tab:senteval. Negative PMI: We observe that using only $\mathit {\texttt {-}PMI}$ (rows $\mathit {\texttt {-}CPMI_{\texttt {-}2}}$ and $\mathit {\texttt {-}NNEGPMI}$) performs similarly to all other models in POS tagging and both syntactic probing tasks, but very poorly on all semantic tasks, strongly supporting our main claim that $\mathit {\texttt {-}PMI}$ mostly encodes syntactic information. Our hypothesis for this is that the grammar that generates language implicitly creates negative cooccurrence and so $\mathit {\texttt {-}PMI}$ encodes this syntactic information. Interestingly, this idea creates a bridge between distributional semantics and the argument by BIBREF17 that indirect negative evidence might play an important role in human language acquisition of grammar. Positive PMI: The $\mathit {\texttt {+}PPMI}$ model performs as well or better as the full spectrum models on nearly all tasks, clearly indicating that $\mathit {\texttt {+}PMI}$ encodes both semantic and syntactic information. Why incorporate -PMI? $\mathit {\texttt {+}PPMI}$ only falters on the RW and analogy tasks, and we hypothesize this is where $\mathit {\texttt {-}PMI}$ is useful: in the absence of positive information, negative information can be used to improve rare word representations and word analogies. Analogies are solved using nearest neighbor lookups in the vector space, and so accounting for negative cooccurrence effectively repels words with which no positive cooccurrence was observed. In future work, we will explore incorporating $\mathit {\texttt {-}PMI}$ only for rare words (where it is most needed). Full spectrum models: The $\mathit {PPMI}$, $\mathit {CPMI_{\texttt {-}2}}$, and $\mathit {NNEGPMI}$ models perform similarly, whereas the $\mathit {NPMI}$ model is significantly worst on nearly all semantic tasks. We thus conclude that accounting for scale in the positive spectrum is more important than in the negative spectrum. We hypothesize this is because scale helps to uniquely identify words, which is critical for semantics (results on $WC$ task correlate strongly with performance on semantic tasks), but in syntax, words with the same function should be indistinguishable. Since $\mathit {\texttt {+}PMI}$ encodes both semantics and syntax, scale must be preserved, whereas $\mathit {\texttt {-}PMI}$ encodes mostly syntax, and so scale information can be discarded. Collapsing the negative spectrum: The $\mathit {PPMI}$ model, which collapses the negative spectrum to zero, performs almost identically to the $\mathit {CPMI_{\texttt {-}2}}$ and $\mathit {NNEGPMI}$ models that account for the range of negative values. This is justified by 1) Our discussion which shows that $\mathit {\texttt {+}PMI}$ is far more informative than $\mathit {\texttt {-}PMI}$ and 2) Looking at fig:hist, we see that collapsed values — interval $(-5,0]$ — account for only $11\%$ of samples compared to $41.7\%$ for non-collapsed negative values. Conclusions and Future Work In this paper, we evaluated existing and novel ways of incorporating $\mathit {\texttt {-}PMI}$ into word embedding models based on explicit weighted matrix factorization, and, more importantly, studied the role that $\mathit {\texttt {-}PMI}$ and $\mathit {\texttt {+}PMI}$ each play in distributional semantics, finding that “a word is not only characterized by the company that it keeps, but also by the company it rejects”. In future work, we wish to further study the link between our work and language acquisition, and explore the fact the $\mathit {\texttt {-}PMI}$ is almost purely syntactic to (possibly) subtract syntax from the full spectrum models, studying the frontier (if there is one) between semantics and syntax. Acknowledgments This research was partly supported by CAPES and CNPq (projects 312114/2015-0, 423843/2016-8, and 140402/2018-7).
Why are statistics from finite corpora unreliable?
6844683935d0d8f588fa06530f5068bf3e1ed0c0
Why Ġare Ġstatistics Ġfrom Ġfinite Ġcorpor a Ġunreliable ?
$ \ math it Ġ{ PM I } ( w , c ) $ Ġgoes Ġto Ġnegative Ġinfinity Ġwhen Ġthe Ġword - context Ġpair Ġ$( w , c ) $ Ġdoes Ġnot Ġappear Ġin Ġthe Ġtraining Ġcorpus
<s> Why Ġare Ġstatistics Ġfrom Ġfinite Ġcorpor a Ġunreliable ? </s> Introduction </s> D ense Ġword Ġvectors Ġ( or Ġembed d ings ) Ġare Ġa Ġkey Ġcomponent Ġin Ġmodern ĠN LP Ġarchitectures Ġfor Ġtasks Ġsuch Ġas Ġsentiment Ġanalysis , Ġparsing , Ġand Ġmachine Ġtranslation . ĠThese Ġvectors Ġcan Ġbe Ġlearned Ġby Ġexploiting Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 0 , Ġparaph r ased Ġby ĠB IB REF 1 Ġas ĠâĢ ľ a Ġword Ġis Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps âĢ Ŀ , Ġusually Ġby Ġconstructing Ġa Ġco occ urrence Ġmatrix Ġover Ġa Ġtraining Ġcorpus , Ġre - weight ing Ġit Ġusing ĠPoint wise ĠMutual ĠInformation Ġ($ \ math it Ġ{ PM I } $ ) ĠB IB REF 2 , Ġand Ġperforming Ġa Ġlow - rank Ġfactor ization Ġto Ġobtain Ġdense Ġvectors . </s> Unfortunately , Ġ$\ math it Ġ{ PM I } ( w , c ) $ Ġgoes Ġto Ġnegative Ġinfinity Ġwhen Ġthe Ġword - context Ġpair Ġ$( w , c ) $ Ġdoes Ġnot Ġappear Ġin Ġthe Ġtraining Ġcorpus . ĠDue Ġto Ġunreliable Ġstatistics , Ġthis Ġhappens Ġvery Ġfrequently Ġin Ġfinite Ġcorpor a . ĠMany Ġmodels Ġwork Ġaround Ġthis Ġissue Ġby Ġclipping Ġnegative Ġ$\ math it Ġ{ PM I } $ Ġvalues Ġat Ġ0 , Ġa Ġmeasure Ġknown Ġas ĠPositive Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{ P PM I } $ ), Ġwhich Ġworks Ġvery Ġwell Ġin Ġpractice . ĠAn Ġunanswered Ġquestion Ġis : ĠâĢ ľ What Ġis Ġlost / g ained Ġby Ġcollapsing Ġthe Ġnegative Ġ$\ math it Ġ{ PM I } $ Ġspectrum Ġto Ġ0 ? âĢ Ŀ . ĠUnderstanding Ġwhich Ġtype Ġof Ġinformation Ġis Ġcaptured Ġby Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġcan Ġhelp Ġin Ġtail oring Ġmodels Ġfor Ġoptimal Ġperformance . </s> In Ġthis Ġwork , Ġwe Ġattempt Ġto Ġanswer Ġthis Ġquestion Ġby Ġstudying Ġthe Ġkind Ġof Ġinformation Ġcontained Ġin Ġthe Ġnegative Ġand Ġpositive Ġspect rums Ġof Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ ). ĠWe Ġevaluate Ġweighted Ġfactor ization Ġof Ġdifferent Ġmat rices Ġwhich Ġuse Ġeither Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $, Ġor Ġboth Ġon Ġvarious Ġsemantic Ġand Ġsynt actic Ġtasks . ĠResults Ġshow Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġalone Ġperforms Ġquite Ġwell Ġon Ġmost Ġtasks , Ġcapturing Ġboth Ġsemantics Ġand Ġsyntax , Ġin Ġcontrast Ġto Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġwhich Ġperforms Ġpoorly Ġon Ġnearly Ġall Ġtasks , Ġexcept Ġthose Ġthat Ġtest Ġfor Ġsyntax . ĠOur Ġmain Ġcontribution Ġis Ġdeepening Ġour Ġunderstanding Ġof Ġdistribution al Ġsemantics Ġby Ġextending ĠB IB REF 1 's Ġparaph rase Ġof Ġthe Ġdistribution al Ġhypothesis Ġto ĠâĢ ľ a Ġword Ġis Ġnot Ġonly Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps , Ġbut Ġalso Ġby Ġthe Ġcompany Ġit Ġrejects âĢ Ŀ . ĠOur Ġsecondary Ġcontributions Ġare Ġthe Ġproposal Ġof Ġtwo Ġ$ PM I $ Ġvariants Ġthat Ġaccount Ġfor Ġthe Ġspectrum Ġof Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġand Ġthe Ġjustification Ġof Ġthe Ġpopular Ġ$ P PM I $ Ġmeasure . </s> In Ġthis Ġpaper , Ġwe Ġfirst Ġlook Ġat Ġrelated Ġwork Ġ($ Â§ $ SEC REF 2 ), Ġthen Ġstudy Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġways Ġof Ġaccounting Ġfor Ġit Ġ($ Â§ $ SEC REF 3 ), Ġdescribe Ġexperiments Ġ($ Â§ $ SEC REF 4 ), Ġanalyze Ġresults Ġ($ Â§ $ SEC REF 5 ), Ġand Ġclose Ġwith Ġideas Ġfor Ġfuture Ġwork Ġ($ Â§ $ SEC REF 6 ). </s> Related ĠWork </s> There Ġis Ġa Ġlong Ġhistory Ġof Ġstudying Ġweight ings Ġ( also Ġknown Ġas Ġassociation Ġmeasures ) Ġof Ġgeneral Ġ( not Ġonly Ġword - context ) Ġco occ urrence Ġmat rices ; Ġsee ĠB IB REF 3 , ĠB IB REF 4 Ġfor Ġan Ġoverview Ġand ĠB IB REF 5 Ġfor Ġcomparison Ġof Ġdifferent Ġweight ings . ĠB IB REF 6 Ġshow Ġthat Ġword Ġvectors Ġderived Ġfrom Ġ$\ math it Ġ{ P PM I } $ Ġmat rices Ġperform Ġbetter Ġthan Ġalternative Ġweight ings Ġfor Ġword - context Ġco occ urrence . ĠIn Ġthe Ġfield Ġof Ġcoll ocation Ġextraction , ĠB IB REF 7 Ġaddress Ġthe Ġnegative Ġinfinity Ġissue Ġwith Ġ$\ math it Ġ{ PM I } $ Ġby Ġintroducing Ġthe Ġnormalized Ġ$\ math it Ġ{ PM I } $ Ġmetric . ĠB IB REF 8 Ġshow Ġtheoretically Ġthat Ġthe Ġpopular ĠSkip - gram Ġmodel ĠB IB REF 9 Ġperforms Ġimplicit Ġfactor ization Ġof Ġshifted Ġ$\ math it Ġ{ PM I } $. </s> Recently , Ġwork Ġin Ġexplicit Ġlow - rank Ġmatrix Ġfactor ization Ġof Ġ$\ math it Ġ{ PM I } $ Ġvariants Ġhas Ġachieved Ġstate Ġof Ġthe Ġart Ġresults Ġin Ġword Ġembed ding . ĠGl o Ve ĠB IB REF 10 Ġperforms Ġweighted Ġfactor ization Ġof Ġthe Ġlog Ġco occ urrence Ġmatrix Ġwith Ġadded Ġbias Ġterms , Ġbut Ġdoes Ġnot Ġaccount Ġfor Ġzero Ġcells . ĠB IB REF 11 Ġpoint Ġout Ġthat ĠGl o Ve 's Ġbias Ġterms Ġcorrelate Ġstrongly Ġwith Ġun ig ram Ġlog Ġcounts , Ġsuggesting Ġthat ĠGl o Ve Ġis Ġfactor izing Ġa Ġvariant Ġof Ġ$\ math it Ġ{ PM I } $. ĠTheir ĠSw i Vel Ġmodel Ġmod ifies Ġthe ĠGl o Ve Ġobjective Ġto Ġuse ĠLa place Ġsmoot hing Ġand Ġhinge Ġloss Ġfor Ġzero Ġcounts Ġof Ġthe Ġco occ urrence Ġmatrix , Ġdirectly Ġfactor izing Ġthe Ġ$\ math it Ġ{ PM I } $ Ġmatrix , Ġsid este pping Ġthe Ġnegative Ġinfinity Ġissue . ĠAn Ġalternative Ġis Ġto Ġuse Ġ$\ math it Ġ{ P PM I } $ Ġand Ġvariants Ġas Ġin ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 . ĠHowever , Ġit Ġis Ġnot Ġclear Ġwhat Ġis Ġlost Ġby Ġclipping Ġthe Ġnegative Ġspectrum Ġof Ġ$\ math it Ġ{ PM I } $, Ġwhich Ġmakes Ġthe Ġuse Ġof Ġ$\ math it Ġ{ P PM I } $, Ġthough Ġit Ġworks Ġwell Ġin Ġpractice , Ġseem Ġun pr in cipled . </s> In Ġthe Ġstudy Ġof Ġlanguage Ġacquisition , ĠB IB REF 17 Ġargue Ġthat Ġindirect Ġnegative Ġevidence Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġhuman Ġacquisition Ġof Ġgrammar , Ġbut Ġdo Ġnot Ġlink Ġthis Ġidea Ġto Ġdistribution al Ġsemantics . </s> PM I Ġ& ĠMatrix ĠFactor ization </s> PM I : ĠA Ġco occ urrence Ġmatrix Ġ$ M $ Ġis Ġconstructed Ġby Ġsliding Ġa Ġsymm etric Ġwindow Ġover Ġthe Ġsubs am pled ĠB IB REF 9 Ġtraining Ġcorpus Ġand Ġfor Ġeach Ġcenter Ġword Ġ$ w $ Ġand Ġcontext Ġword Ġ$ c $ Ġwithin Ġthe Ġwindow , Ġincrement ing Ġ$ M _{ w c } $. Ġ$\ math it Ġ{ PM I } $ Ġis Ġthen Ġequal Ġto : </s> where Ġ* Ġdenotes Ġsumm ation Ġover Ġthe Ġcorresponding Ġindex . ĠTo Ġdeal Ġwith Ġnegative Ġvalues , Ġwe Ġpropose Ġclipped Ġ$\ math it Ġ{ PM I } $, </s> which Ġis Ġequivalent Ġto Ġ$\ math it Ġ{ P PM I } $ Ġwhen Ġ$ z Ġ= Ġ0 $. </s> Matrix Ġfactor ization : ĠLex V ec ĠB IB REF 15 Ġperforms Ġthe Ġfactor ization Ġ$ M ^ {\ prime Ġ} Ġ= ĠWC ^ \ top Ġ$ , Ġwhere Ġ$ M ^ {\ prime Ġ} $ Ġis Ġany Ġtransformation Ġof Ġ$ M $ Ġ( such Ġas Ġ$\ math it Ġ{ P PM I } $ ), Ġand Ġ$ W , ĠC $ Ġare Ġthe Ġword Ġand Ġcontext Ġembed d ings Ġrespectively . ĠBy Ġsliding Ġa Ġsymm etric Ġwindow Ġover Ġthe Ġtraining Ġcorpus Ġ( window Ġsampling ), ĠLex V ec Ġperforms Ġone ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġstep Ġevery Ġtime Ġa Ġ$( w , c ) $ Ġpair Ġis Ġobserved , Ġminimizing </s> Additionally , Ġfor Ġevery Ġcenter Ġword Ġ$ w $, Ġ$ k $ Ġnegative Ġwords ĠB IB REF 9 Ġare Ġdrawn Ġfrom Ġthe Ġun ig ram Ġcontext Ġdistribution Ġ$ P _ n $ Ġ( negative Ġsampling ) Ġand ĠSG D Ġsteps Ġtaken Ġto Ġminimize : </s> Thus Ġthe Ġloss Ġfunction Ġprior itiz es Ġthe Ġcorrect Ġapproximation Ġof Ġfrequently Ġco occ urring Ġpairs Ġand Ġof Ġpairs Ġwhere Ġeither Ġword Ġoccurs Ġwith Ġhigh Ġfrequency ; Ġthese Ġare Ġpairs Ġfor Ġwhich Ġwe Ġhave Ġmore Ġreliable Ġstatistics . </s> In Ġour Ġexperiments , Ġwe Ġuse ĠLex V ec Ġover ĠSing ular ĠValue ĠDec om position Ġ( S VD ) Ġbecause Ġa ) ĠEm pir ical Ġresults Ġshows Ġit Ġoutper forms ĠS VD ĠB IB REF 15 . Ġb ) ĠThe Ġweight ing Ġof Ġreconstruction Ġerrors Ġby Ġstatistical Ġconfidence Ġis Ġparticularly Ġimportant Ġfor Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $, Ġwhere Ġnegative Ġco occ urrence Ġbetween Ġa Ġpair Ġof Ġfrequent Ġwords Ġis Ġmore Ġsignificant Ġand Ġshould Ġbe Ġbetter Ġapprox imated Ġthan Ġthat Ġbetween Ġa Ġpair Ġof Ġrare Ġwords . ĠGl o Ve 's Ġmatrix Ġfactor ization Ġis Ġeven Ġmore Ġunsu itable Ġfor Ġour Ġexperiments Ġas Ġits Ġloss Ġweight ing ĠâĢĶ Ġa Ġmon oton ically Ġincreasing Ġfunction Ġof Ġ$ M _{ w c } $ ĠâĢĶ Ġignores Ġreconstruction Ġerrors Ġof Ġnon - co occ urring Ġpairs . </s> Spect rum Ġof ĠPM I : ĠTo Ġbetter Ġunderstand Ġthe Ġdistribution Ġof Ġ$\ math it Ġ{ C PM I } $ Ġvalues , Ġwe Ġplot Ġa Ġhist ogram Ġof Ġ$ 10 ^ 5 $ Ġpairs Ġrandomly Ġsampled Ġby Ġwindow Ġsampling Ġand Ġnegative Ġsampling Ġin Ġfig : hist , Ġsetting Ġ$ z =- 5 $. ĠWe Ġcan Ġclearly Ġsee Ġthe Ġspectrum Ġof Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġthat Ġis Ġcollapsed Ġwhen Ġwe Ġuse Ġ$\ math it Ġ{ P PM I } $ Ġ($ z = 0 $ ). ĠIn Ġpractice Ġwe Ġfind Ġthat Ġ$ z =- 2 $ Ġcaptures Ġmost Ġof Ġthe Ġnegative Ġspectrum Ġand Ġconsistently Ġgives Ġbetter Ġresults Ġthan Ġsmaller Ġvalues Ġso Ġwe Ġuse Ġthis Ġvalue Ġfor Ġthe Ġrest Ġof Ġthis Ġpaper . ĠWe Ġsuspect Ġthis Ġis Ġdue Ġto Ġthe Ġlarge Ġnumber Ġof Ġnon - co occ urring Ġpairs Ġ($ 41 . 7 \ % $ Ġin Ġthis Ġsample ) Ġwhich Ġend Ġup Ġdominating Ġthe Ġloss Ġfunction Ġwhen Ġ$ z $ Ġis Ġtoo Ġsmall . </s> Normal ization : ĠWe Ġalso Ġexperiment Ġwith Ġnormalized Ġ$\ math it Ġ{ PM I } $ Ġ($ \ math it Ġ{ N PM I } $ ) ĠB IB REF 7 : </s> such Ġthat Ġ$ N PM I ( w , c ) Ġ= Ġ- 1 $ Ġwhen Ġ$( w , c ) $ Ġnever Ġco occ ur , Ġ$ N PM I ( w , c ) Ġ= Ġ0 $ Ġwhen Ġthey Ġare Ġindependent , Ġand Ġ$ N PM I ( w , c ) Ġ= Ġ1 $ Ġwhen Ġthey Ġalways Ġco occ ur Ġtogether . ĠThis Ġeffectively Ġcaptures Ġthe Ġentire Ġnegative Ġspectrum , Ġbut Ġhas Ġthe Ġdownside Ġof Ġnormal ization Ġwhich Ġdisc ards Ġscale Ġinformation . ĠIn Ġpractice Ġwe Ġfind Ġthis Ġworks Ġpoorly Ġif Ġdone Ġsymm etr ically , Ġso Ġwe Ġintroduce Ġa Ġvariant Ġcalled Ġ$\ math it Ġ{ NN EG PM I } $ Ġwhich Ġonly Ġnormal izes Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ : </s> We Ġalso Ġexperimented Ġwith ĠLa place Ġsmoot hing Ġas Ġin ĠB IB REF 18 Ġfor Ġvarious Ġpseud oc ount s Ġbut Ġfound Ġit Ġto Ġwork Ġconsistently Ġworse Ġthan Ġboth Ġ$\ math it Ġ{ C PM I _ z } $ Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġso Ġwe Ġomit Ġfurther Ġdiscussion Ġin Ġthis Ġpaper . </s> Materials </s> In Ġorder Ġto Ġidentify Ġthe Ġrole Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġplay Ġin Ġdistribution al Ġsemantics , Ġwe Ġtrain ĠLex V ec Ġmodels Ġthat Ġskip ĠSG D Ġsteps Ġwhen Ġtarget Ġcell Ġvalues Ġare Ġ$ > 0 $ Ġor Ġ$\ le Ġ0 $, Ġrespectively . ĠFor Ġexample , Ġ$ - \ math it Ġ{ C PM I } _{ \ text tt Ġ{ - } 2 } $ Ġsk ips Ġsteps Ġwhen Ġ$\ math it Ġ{ C PM I } _{ \ text tt Ġ{ - } 2 } ( w , c ) Ġ> Ġ0 $. ĠSimilarly , Ġthe Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġmodel Ġsk ips ĠSG D Ġsteps Ġwhen Ġ$\ math it Ġ{ P PM I } ( w , c ) Ġ\ le Ġ0 $. ĠWe Ġcompare Ġthese Ġto Ġmodels Ġthat Ġinclude Ġboth Ġnegative Ġand Ġpositive Ġinformation Ġto Ġsee Ġhow Ġthe Ġtwo Ġinteract . </s> We Ġuse Ġthe Ġdefault ĠLex V ec Ġconfiguration Ġfor Ġall Ġ$\ math it Ġ{ PM I } $ Ġvariants : Ġfixed Ġwindow Ġof Ġsize Ġ2 , Ġembed ding Ġdimension Ġof Ġ300 , Ġ5 Ġnegative Ġsamples , Ġpositional Ġcontexts , Ġcontext Ġdistribution Ġsmoot hing Ġof Ġ$ . 75 $, Ġlearning Ġrate Ġof Ġ$ . 025 $, Ġno Ġsub word Ġinformation , Ġand Ġnegative Ġdistribution Ġpower Ġof Ġ$ . 75 $. ĠWe Ġtrain Ġon Ġa Ġlower c ased , Ġal phan umer ical Ġ2015 ĠWikipedia Ġdump Ġwith Ġ$ 3 . 8 $ B Ġtokens , Ġdisc arding Ġtokens Ġwith Ġfrequency Ġ$ < Ġ100 $, Ġfor Ġa Ġvocabulary Ġsize Ġof Ġ$ 303 , 517 $ Ġwords . </s> For Ġcomparison , Ġwe Ġinclude Ġresults Ġfor Ġa Ġrandomly Ġinitialized , Ġnon - trained Ġembed ding Ġto Ġestablish Ġtask Ġbas elines . </s> Sem antics : ĠTo Ġevaluate Ġword - level Ġsemantics , Ġwe Ġuse Ġthe ĠSim Lex ĠB IB REF 19 Ġand ĠRare ĠWord Ġ( RW ) ĠB IB REF 20 Ġword Ġsimilarity Ġdatasets , Ġand Ġthe ĠGoogle ĠSem antic Ġ( G Sem ) Ġanalog ies ĠB IB REF 9 . ĠWe Ġevaluate Ġsentence - level Ġsemantics Ġusing Ġaveraged Ġbag Ġof Ġvectors Ġ( Bo V ) Ġrepresentations Ġon Ġthe ĠSem antic ĠText ual ĠSimilar ity Ġ( ST SB ) Ġtask ĠB IB REF 21 Ġand ĠWord ĠContent Ġ( WC ) Ġprobing Ġtask Ġ( ident ify Ġfrom Ġa Ġlist Ġof Ġwords Ġwhich Ġis Ġcontained Ġin Ġthe Ġsentence Ġrepresentation ) Ġfrom ĠSent E val ĠB IB REF 22 . </s> Sy ntax : ĠSimilarly , Ġwe Ġuse Ġthe ĠGoogle ĠSynt actic Ġanalog ies Ġ( GS yn ) ĠB IB REF 9 Ġto Ġevaluate Ġword - level Ġsynt actic Ġinformation , Ġand ĠDepth Ġ( Dep ) Ġand ĠTop ĠConst itu ent Ġ( Top C ) Ġ( of Ġthe Ġinput Ġsentence 's Ġconstituent Ġparse Ġtree ) Ġprobing Ġtasks Ġfrom ĠSent E val ĠB IB REF 22 Ġfor Ġsentence - level Ġsyntax . ĠClass ifiers Ġfor Ġall ĠSent E val Ġprobing Ġtasks Ġare Ġmult il ayer Ġpercept rons Ġwith Ġa Ġsingle Ġhidden Ġlayer Ġof Ġ100 Ġunits Ġand Ġdrop out Ġof Ġ$ . 1 $. ĠOur Ġfinal Ġsynt actic Ġtask Ġis Ġpart - of - speech Ġ( POS ) Ġtagging Ġusing Ġthe Ġsame ĠBi L ST M - CR F Ġsetup Ġas ĠB IB REF 23 Ġbut Ġusing Ġonly Ġword Ġembed d ings Ġ( no Ġhand - engine ered Ġfeatures ) Ġas Ġinput , Ġtrained Ġon Ġthe ĠWS J Ġsection Ġof Ġthe ĠPenn ĠTree bank ĠB IB REF 24 . </s> Results </s> All Ġresults Ġare Ġshown Ġin Ġtab : sent eval . </s> Neg ative ĠPM I : ĠWe Ġobserve Ġthat Ġusing Ġonly Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġ( rows Ġ$\ math it Ġ{\ text tt Ġ{ - } C PM I _{ \ text tt Ġ{ - } 2 }} $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ - } NN EG PM I } $ ) Ġperforms Ġsimilarly Ġto Ġall Ġother Ġmodels Ġin ĠPOS Ġtagging Ġand Ġboth Ġsynt actic Ġprobing Ġtasks , Ġbut Ġvery Ġpoorly Ġon Ġall Ġsemantic Ġtasks , Ġstrongly Ġsupporting Ġour Ġmain Ġclaim Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġmostly Ġenc odes Ġsynt actic Ġinformation . </s> Our Ġhypothesis Ġfor Ġthis Ġis Ġthat Ġthe Ġgrammar Ġthat Ġgenerates Ġlanguage Ġimplicitly Ġcreates Ġnegative Ġco occ urrence Ġand Ġso Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġenc odes Ġthis Ġsynt actic Ġinformation . ĠInterestingly , Ġthis Ġidea Ġcreates Ġa Ġbridge Ġbetween Ġdistribution al Ġsemantics Ġand Ġthe Ġargument Ġby ĠB IB REF 17 Ġthat Ġindirect Ġnegative Ġevidence Ġmight Ġplay Ġan Ġimportant Ġrole Ġin Ġhuman Ġlanguage Ġacquisition Ġof Ġgrammar . ĠPositive ĠPM I : ĠThe Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġmodel Ġperforms Ġas Ġwell Ġor Ġbetter Ġas Ġthe Ġfull Ġspectrum Ġmodels Ġon Ġnearly Ġall Ġtasks , Ġclearly Ġindicating Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġenc odes Ġboth Ġsemantic Ġand Ġsynt actic Ġinformation . </s> Why Ġincorporate Ġ- PM I ? Ġ$\ math it Ġ{\ text tt Ġ{ + } P PM I } $ Ġonly Ġfal ters Ġon Ġthe ĠRW Ġand Ġanalogy Ġtasks , Ġand Ġwe Ġhypothes ize Ġthis Ġis Ġwhere Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġis Ġuseful : Ġin Ġthe Ġabsence Ġof Ġpositive Ġinformation , Ġnegative Ġinformation Ġcan Ġbe Ġused Ġto Ġimprove Ġrare Ġword Ġrepresentations Ġand Ġword Ġanalog ies . ĠAnalog ies Ġare Ġsolved Ġusing Ġnearest Ġneighbor Ġlook ups Ġin Ġthe Ġvector Ġspace , Ġand Ġso Ġaccounting Ġfor Ġnegative Ġco occ urrence Ġeffectively Ġrep els Ġwords Ġwith Ġwhich Ġno Ġpositive Ġco occ urrence Ġwas Ġobserved . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġexplore Ġincorporating Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġonly Ġfor Ġrare Ġwords Ġ( where Ġit Ġis Ġmost Ġneeded ). </s> Full Ġspectrum Ġmodels : ĠThe Ġ$\ math it Ġ{ P PM I } $, Ġ$\ math it Ġ{ C PM I _{ \ text tt Ġ{ - } 2 }} $, Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġmodels Ġperform Ġsimilarly , Ġwhereas Ġthe Ġ$\ math it Ġ{ N PM I } $ Ġmodel Ġis Ġsignificantly Ġworst Ġon Ġnearly Ġall Ġsemantic Ġtasks . ĠWe Ġthus Ġconclude Ġthat Ġaccounting Ġfor Ġscale Ġin Ġthe Ġpositive Ġspectrum Ġis Ġmore Ġimportant Ġthan Ġin Ġthe Ġnegative Ġspectrum . ĠWe Ġhypothes ize Ġthis Ġis Ġbecause Ġscale Ġhelps Ġto Ġuniquely Ġidentify Ġwords , Ġwhich Ġis Ġcritical Ġfor Ġsemantics Ġ( results Ġon Ġ$ WC $ Ġtask Ġcorrelate Ġstrongly Ġwith Ġperformance Ġon Ġsemantic Ġtasks ), Ġbut Ġin Ġsyntax , Ġwords Ġwith Ġthe Ġsame Ġfunction Ġshould Ġbe Ġindistinguishable . ĠSince Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġenc odes Ġboth Ġsemantics Ġand Ġsyntax , Ġscale Ġmust Ġbe Ġpreserved , Ġwhereas Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġenc odes Ġmostly Ġsyntax , Ġand Ġso Ġscale Ġinformation Ġcan Ġbe Ġdiscarded . </s> Coll aps ing Ġthe Ġnegative Ġspectrum : ĠThe Ġ$\ math it Ġ{ P PM I } $ Ġmodel , Ġwhich Ġcollapses Ġthe Ġnegative Ġspectrum Ġto Ġzero , Ġperforms Ġalmost Ġident ically Ġto Ġthe Ġ$\ math it Ġ{ C PM I _{ \ text tt Ġ{ - } 2 }} $ Ġand Ġ$\ math it Ġ{ NN EG PM I } $ Ġmodels Ġthat Ġaccount Ġfor Ġthe Ġrange Ġof Ġnegative Ġvalues . ĠThis Ġis Ġjustified Ġby Ġ1 ) ĠOur Ġdiscussion Ġwhich Ġshows Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġis Ġfar Ġmore Ġinformative Ġthan Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ2 ) ĠLooking Ġat Ġfig : hist , Ġwe Ġsee Ġthat Ġcollapsed Ġvalues ĠâĢĶ Ġinterval Ġ$( - 5 , 0 ] $ ĠâĢĶ Ġaccount Ġfor Ġonly Ġ$ 11 \ % $ Ġof Ġsamples Ġcompared Ġto Ġ$ 41 . 7 \ % $ Ġfor Ġnon - coll apsed Ġnegative Ġvalues . </s> Con clusions Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġevaluated Ġexisting Ġand Ġnovel Ġways Ġof Ġincorporating Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġinto Ġword Ġembed ding Ġmodels Ġbased Ġon Ġexplicit Ġweighted Ġmatrix Ġfactor ization , Ġand , Ġmore Ġimportantly , Ġstudied Ġthe Ġrole Ġthat Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġand Ġ$\ math it Ġ{\ text tt Ġ{ + } PM I } $ Ġeach Ġplay Ġin Ġdistribution al Ġsemantics , Ġfinding Ġthat ĠâĢ ľ a Ġword Ġis Ġnot Ġonly Ġcharacterized Ġby Ġthe Ġcompany Ġthat Ġit Ġkeeps , Ġbut Ġalso Ġby Ġthe Ġcompany Ġit Ġrejects âĢ Ŀ . ĠIn Ġfuture Ġwork , Ġwe Ġwish Ġto Ġfurther Ġstudy Ġthe Ġlink Ġbetween Ġour Ġwork Ġand Ġlanguage Ġacquisition , Ġand Ġexplore Ġthe Ġfact Ġthe Ġ$\ math it Ġ{\ text tt Ġ{ - } PM I } $ Ġis Ġalmost Ġpurely Ġsynt actic Ġto Ġ( possibly ) Ġsubtract Ġsyntax Ġfrom Ġthe Ġfull Ġspectrum Ġmodels , Ġstudying Ġthe Ġfrontier Ġ( if Ġthere Ġis Ġone ) Ġbetween Ġsemantics Ġand Ġsyntax . </s> Acknowled gments </s> This Ġresearch Ġwas Ġpartly Ġsupported Ġby ĠCAP ES Ġand ĠC NP q Ġ( projects Ġ312 114 / 2015 - 0 , Ġ4 238 43 / 2016 - 8 , Ġand Ġ140 402 / 2018 - 7 ).