Unanswerable
NONE
2001.07786
Introduction </s> Natural Ġlanguages Ġevolve Ġand Ġwords Ġhave Ġalways Ġbeen Ġsubject Ġto Ġsemantic Ġchange Ġover Ġtime ĠB IB REF 1 . ĠWith Ġthe Ġrise Ġof Ġlarge Ġdigit ized Ġtext Ġresources Ġrecent ĠN LP Ġtechnologies Ġhave Ġmade Ġit Ġpossible Ġto Ġcapture Ġsuch Ġchange Ġwith Ġvector Ġspace Ġmodels ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġtopic Ġmodels ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , Ġand Ġsense Ġclust ering Ġmodels ĠB IB REF 9 . ĠHowever , Ġmany Ġapproaches Ġfor Ġdetecting ĠL SC Ġdiffer Ġprofoundly Ġfrom Ġeach Ġother Ġand Ġtherefore Ġdrawing Ġcomparisons Ġbetween Ġthem Ġcan Ġbe Ġchallenging ĠB IB REF 10 . ĠNot Ġonly Ġdo Ġarchitectures Ġfor Ġdetecting ĠL SC Ġvary , Ġtheir Ġperformance Ġis Ġalso Ġoften Ġevaluated Ġwithout Ġaccess Ġto Ġevaluation Ġdata Ġor Ġtoo Ġsparse Ġdata Ġsets . ĠIn Ġcases Ġwhere Ġevaluation Ġdata Ġis Ġavailable , Ġoft entimes ĠL SC D Ġsystems Ġare Ġnot Ġevaluated Ġon Ġthe Ġsame Ġdata Ġset Ġwhich Ġhind ers Ġthe Ġresearch Ġcommunity Ġto Ġdraw Ġcomparisons . </s> For Ġthis Ġreason Ġwe Ġreport Ġthe Ġresults Ġof Ġthe Ġfirst Ġshared Ġtask Ġon Ġun super vised Ġlex ical Ġsemantic Ġchange Ġdetection Ġin ĠGerman Ġthat Ġis Ġbased Ġon Ġan Ġannot ated Ġdata Ġset Ġto Ġguarantee Ġobjective Ġreasoning Ġthroughout Ġdifferent Ġapproaches . ĠThe Ġtask Ġwas Ġorganized Ġas Ġpart Ġof Ġthe Ġseminar Ġ' Lex ical ĠSem antic ĠChange ĠDetection ' Ġat Ġthe ĠI MS ĠSt utt gart Ġin Ġthe Ġsummer Ġterm Ġof Ġ2019 . </s> Task </s> The Ġgoal Ġof Ġthe Ġshared Ġtask Ġwas Ġto Ġcreate Ġan Ġarchitecture Ġto Ġdetect Ġsemantic Ġchange Ġand Ġto Ġrank Ġwords Ġaccording Ġto Ġtheir Ġdegree Ġof Ġchange Ġbetween Ġtwo Ġdifferent Ġtime Ġperiods . ĠGiven Ġtwo Ġcorpor a ĠCa Ġand ĠC b , Ġthe Ġtarget Ġwords Ġhad Ġto Ġbe Ġranked Ġaccording Ġto Ġtheir Ġdegree Ġof Ġlex ical Ġsemantic Ġchange Ġbetween ĠCa Ġand ĠC b Ġas Ġannot ated Ġby Ġhuman Ġjudges . ĠA Ġcompetition Ġwas Ġset Ġup Ġon ĠCod al ab Ġand Ġteams Ġmostly Ġconsisting Ġof Ġ2 Ġpeople Ġwere Ġformed Ġto Ġtake Ġpart Ġin Ġthe Ġtask . ĠThere Ġwas Ġone Ġgroup Ġconsisting Ġof Ġ3 Ġteam Ġmembers Ġand Ġtwo Ġindividuals Ġwho Ġentered Ġthe Ġtask Ġon Ġtheir Ġown . ĠIn Ġtotal Ġthere Ġwere Ġ12 ĠL SC D Ġsystems Ġparticipating Ġin Ġthe Ġshared Ġtask . </s> The Ġshared Ġtask Ġwas Ġdivided Ġinto Ġthree Ġphases , Ġi . e ., Ġdevelopment , Ġtesting Ġand Ġanalysis Ġphase . ĠIn Ġthe Ġdevelopment Ġphase Ġeach Ġteam Ġimplemented Ġa Ġfirst Ġversion Ġof Ġtheir Ġmodel Ġbased Ġon Ġa Ġtrial Ġdata Ġset Ġand Ġsubmitted Ġit Ġsubsequently . ĠIn Ġthe Ġtesting Ġphase Ġthe Ġtesting Ġdata Ġwas Ġmade Ġpublic Ġand Ġparticipants Ġapplied Ġtheir Ġmodels Ġto Ġthe Ġtest Ġdata Ġwith Ġa Ġrestriction Ġof Ġpossible Ġresult Ġupload s Ġto Ġ30 . ĠThe Ġleader board Ġwas Ġpublic Ġat Ġall Ġtimes . ĠEventually , Ġthe Ġanalysis Ġphase Ġwas Ġentered Ġand Ġthe Ġmodels Ġof Ġthe Ġtesting Ġphase Ġwere Ġevaluated Ġin Ġterms Ġof Ġthe Ġpredictions Ġthey Ġmade Ġand Ġparameters Ġcould Ġbe Ġtuned Ġfurther . ĠThe Ġmodels Ġand Ġresults Ġwill Ġbe Ġdiscussed Ġin Ġdetail Ġin Ġsections Ġ7 Ġand Ġ8 . </s> Cor pora </s> The Ġtask , Ġas Ġframed Ġabove , Ġrequires Ġto Ġdetect Ġthe Ġsemantic Ġchange Ġbetween Ġtwo Ġcorpor a . ĠThe Ġtwo Ġcorpor a Ġused Ġin Ġthe Ġshared Ġtask Ġcorrespond Ġto Ġthe Ġdi ach ronic Ġcorpus Ġpair Ġfrom ĠB IB REF 0 : ĠD TA 18 Ġand ĠD TA 19 . ĠThey Ġconsist Ġof Ġsub parts Ġof ĠD TA Ġcorpus ĠB IB REF 11 Ġwhich Ġis Ġa Ġfreely Ġavailable Ġle mm at ized , ĠPOS - tag ged Ġand Ġspelling - normal ized Ġdi ach ronic Ġcorpus Ġof ĠGerman Ġcontaining Ġtexts Ġfrom Ġthe Ġ16 th Ġto Ġthe Ġ20 th Ġcentury . ĠD TA 18 Ġcontains Ġ26 Ġmillion Ġsentences Ġpublished Ġbetween Ġ17 50 - 17 99 Ġand ĠD TA 19 Ġ40 Ġmillion Ġbetween Ġ1850 - 18 99 . ĠThe Ġcorpus Ġversion Ġused Ġin Ġthe Ġtask Ġhas Ġthe Ġfollowing Ġformat : Ġ" year Ġ[ tab ] Ġle mma 1 Ġle mma 2 Ġle mma 3 Ġ..." . </s> E val uation </s> The ĠDi ach ronic ĠUsage ĠRelated ness Ġ( D UR el ) Ġgold Ġstandard Ġdata Ġset Ġincludes Ġ22 Ġtarget Ġwords Ġand Ġtheir Ġvarying Ġdegrees Ġof Ġsemantic Ġchange ĠB IB REF 12 . ĠFor Ġeach Ġof Ġthese Ġtarget Ġwords Ġa Ġrandom Ġsample Ġof Ġuse Ġpairs Ġfrom Ġthe ĠD TA Ġcorpus Ġwas Ġretrieved Ġand Ġannot ated . ĠThe Ġannot ators Ġwere Ġrequired Ġto Ġrate Ġthe Ġpairs Ġaccording Ġto Ġtheir Ġsemantic Ġrelated ness Ġon Ġa Ġscale Ġfrom Ġ1 Ġto Ġ4 Ġ( un related Ġ- Ġidentical Ġmeanings ) Ġfor Ġtwo Ġtime Ġperiods . ĠThe Ġaverage ĠSpear man 's Ġ$\ r ho Ġ$ Ġbetween Ġthe Ġfive Ġannot ators Ġwas Ġ0 . 66 Ġfor Ġ1 , 320 Ġuse Ġpar is . ĠThe Ġresulting Ġword Ġranking Ġof Ġthe ĠD UR el Ġdata Ġset Ġis Ġdetermined Ġby Ġthe Ġmean Ġusage Ġrelated ness Ġacross Ġtwo Ġtime Ġperiods Ġand Ġis Ġused Ġas Ġthe Ġbenchmark Ġto Ġcompare Ġthe Ġmodels âĢ Ļ Ġperformances Ġin Ġthe Ġshared Ġtask . </s> E val uation Ġ: :: ĠMet ric </s> The Ġoutput Ġof Ġa Ġsystem Ġwith Ġthe Ġtarget Ġwords Ġin Ġthe Ġpredicted Ġorder Ġis Ġcompared Ġto Ġthe Ġgold Ġranking Ġof Ġthe ĠD UR el Ġdata Ġset . ĠAs Ġthe Ġmetric Ġto Ġassess Ġhow Ġwell Ġthe Ġmodel 's Ġoutput Ġfits Ġthe Ġgold Ġranking ĠSpear man 's Ġ$\ r ho Ġ$ Ġwas Ġused . ĠThe Ġhigher ĠSpear man 's Ġrank - order Ġcorrelation Ġthe Ġbetter Ġthe Ġsystem 's Ġperformance . </s> E val uation Ġ: :: ĠBas elines </s> Mod els Ġwere Ġcompared Ġto Ġtwo Ġbas elines Ġfor Ġthe Ġshared Ġtask : </s> log - trans formed Ġnormalized Ġfrequency Ġdifference Ġ( FD ) </s> count Ġvectors Ġwith Ġcolumn Ġintersection Ġand Ġcos ine Ġdistance Ġ( C NT Ġ+ ĠCI Ġ+ ĠCD ) </s> The Ġwindow Ġsize Ġfor ĠC NT Ġ+ ĠCI Ġ+ ĠCD Ġwas Ġ10 . ĠFind Ġmore Ġinformation Ġon Ġthese Ġmodels Ġin ĠB IB REF 0 . </s> Particip ating ĠSystems </s> Particip ants Ġmostly Ġrely Ġon Ġthe Ġmodels Ġcompared Ġin ĠB IB REF 0 Ġand Ġapply Ġmodifications Ġto Ġimprove Ġthem . ĠIn Ġparticular , Ġmost Ġteams Ġmake Ġuse Ġof Ġskip - gram Ġwith Ġnegative Ġsampling Ġ( SG NS ) Ġbased Ġon ĠB IB REF 13 Ġto Ġlearn Ġthe Ġsemantic Ġspaces Ġof Ġthe Ġtwo Ġtime Ġperiods Ġand Ġorth og onal Ġpro cr ust es Ġ( OP ) Ġto Ġalign Ġthese Ġvector Ġspaces , Ġsimilar Ġto Ġthe Ġapproach Ġby ĠB IB REF 14 . ĠDifferent Ġmeaning Ġrepresentations Ġsuch Ġas Ġsense Ġclusters Ġare Ġused Ġas Ġwell . ĠAs Ġmeasure Ġto Ġdetect Ġthe Ġdegree Ġof ĠL SC Ġall Ġteams Ġexcept Ġone Ġchoose Ġcos ine Ġdistance Ġ( CD ). ĠThis Ġteam Ġuses ĠJensen - Sh annon Ġdistance Ġ( J SD ) Ġinstead , Ġwhich Ġcomp utes Ġthe Ġdistance Ġbetween Ġprobability Ġdistributions ĠB IB REF 15 . </s> The Ġmodels Ġof Ġeach Ġteam Ġwill Ġbe Ġbriefly Ġintroduced Ġin Ġthis Ġsection . </s> Particip ating ĠSystems Ġ: :: Ġsore ns bn </s> Team Ġsore ns bn Ġmakes Ġuse Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġto Ġdetect ĠL SC . ĠThey Ġuse Ġsimilar Ġhyper param eters Ġas Ġin ĠB IB REF 0 Ġto Ġtune Ġthe ĠSG NS Ġmodel . ĠThey Ġuse Ġan Ġopen - s ourced Ġnoise - aware Ġimplementation Ġto Ġimprove Ġthe ĠOP Ġalignment ĠB IB REF 16 . </s> Particip ating ĠSystems Ġ: :: Ġtid oe </s> Team Ġtid oe Ġbuilds Ġon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD , Ġbut Ġthey Ġadd Ġa Ġtransformation Ġstep Ġto Ġreceive Ġbin ar ized Ġrepresentations Ġof Ġmat rices ĠB IB REF 17 . ĠThis Ġstep Ġis Ġtaken Ġto Ġcounter Ġthe Ġbias Ġthat Ġcan Ġoccur Ġin Ġvector - space Ġmodels Ġbased Ġon Ġfrequencies ĠB IB REF 18 . </s> Particip ating ĠSystems Ġ: :: Ġin Ġvain </s> The Ġteam Ġapplies Ġa Ġmodel Ġbased Ġon ĠSG NS Ġwith Ġvector Ġinitialization Ġalignment Ġand Ġcos ine Ġdistance Ġ( SG NS Ġ+ ĠVI Ġ+ ĠCD ). ĠVector Ġinitialization Ġis Ġan Ġalignment Ġstrategy Ġwhere Ġthe Ġvector Ġspace Ġlearning Ġmodel Ġfor Ġ$ t _ 2 $ Ġis Ġinitialized Ġwith Ġthe Ġvectors Ġfrom Ġ$ t _ 1 $ ĠB IB REF 19 . ĠSince ĠSG NS Ġ+ ĠVI Ġ+ ĠOP Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġother Ġmodels Ġin ĠB IB REF 0 , Ġthey Ġalter Ġthe Ġvector Ġinitialization Ġprocess Ġby Ġinitial izing Ġon Ġthe Ġcomplete Ġmodel Ġinstead Ġof Ġonly Ġthe Ġword Ġmatrix Ġof Ġ$ t _ 1 $ Ġto Ġreceive Ġimproved Ġresults . </s> Particip ating ĠSystems Ġ: :: ĠEv illy </s> In Ġline Ġwith Ġprevious Ġapproaches , Ġteam ĠEv illy Ġbuilds Ġupon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠThey Ġalter Ġthe ĠOP Ġstep Ġby Ġusing Ġonly Ġhigh - frequency Ġwords Ġfor Ġalignment . </s> Particip ating ĠSystems Ġ: :: ĠD AF </s> Team ĠD AF Ġuses Ġan Ġarchitecture Ġbased Ġon Ġlearning Ġvectors Ġwith Ġfast Text , Ġalignment Ġwith Ġun super vised Ġand Ġsupervised Ġvariations Ġof ĠOP , Ġand ĠCD , Ġusing Ġthe ĠM USE Ġpackage ĠB IB REF 20 , ĠB IB REF 21 . ĠFor Ġthe Ġsupervised Ġalignment Ġstop Ġwords Ġare Ġused . ĠThe Ġunderlying Ġassumption Ġis Ġthat Ġstop Ġwords Ġserve Ġas Ġfunctional Ġunits Ġof Ġlanguage Ġand Ġtheir Ġusage Ġshould Ġbe Ġconsistent Ġover Ġtime . </s> Particip ating ĠSystems Ġ: :: ĠSn akes On AP lane </s> The Ġteam Ġlearns Ġvector Ġspaces Ġwith Ġcount Ġvectors , Ġpositive Ġpoint wise Ġmutual Ġinformation Ġ( P PM I ), ĠSG NS Ġand Ġuses Ġcolumn Ġintersection Ġ( CI ) Ġand ĠOP Ġas Ġalignment Ġtechniques Ġwhere Ġapplicable . ĠThen Ġthey Ġcompare Ġtwo Ġdistance Ġmeasures Ġ( CD Ġand ĠJ SD ) Ġfor Ġthe Ġdifferent Ġmodels ĠC NT Ġ+ ĠCI , ĠP PM I Ġ+ ĠCI Ġand ĠSG NS Ġ+ ĠOP Ġto Ġidentify Ġwhich Ġmeasure Ġperforms Ġbetter Ġfor Ġthese Ġmodels . ĠThey Ġalso Ġexperiment Ġwith Ġdifferent Ġways Ġto Ġremove Ġnegative Ġvalues Ġfrom ĠSG NS Ġvectors , Ġwhich Ġis Ġneeded Ġfor ĠJ SD . </s> Particip ating ĠSystems Ġ: :: ĠTeam K ulk arn i 15 </s> Team K ulk arn i 15 Ġuses ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġwith Ġthe Ġmodification Ġof Ġlocal Ġalignment Ġwith Ġk Ġnearest Ġneighbors , Ġsince Ġother Ġmodels Ġoften Ġuse Ġglobal Ġalignment Ġthat Ġcan Ġbe Ġprone Ġto Ġnoise ĠB IB REF 22 . </s> Particip ating ĠSystems Ġ: :: ĠBash ma ist ori </s> They Ġuse Ġword Ġinjection Ġ( WI ) Ġalignment Ġon ĠP PM I Ġvectors Ġwith ĠCD . ĠThis Ġapproach Ġavoids Ġthe Ġcomplex Ġalignment Ġprocedure Ġfor Ġembed d ings Ġand Ġis Ġapplicable Ġto Ġembed d ings Ġand Ġcount - based Ġmethods . ĠThey Ġcompare Ġtwo Ġimplementations Ġof Ġword Ġinjection ĠB IB REF 23 , ĠB IB REF 0 Ġas Ġthese Ġshowed Ġdifferent Ġresults Ġon Ġdifferent Ġdata Ġsets . </s> Particip ating ĠSystems Ġ: :: Ġg iki </s> Team Ġg iki Ġuses ĠP PM I Ġ+ ĠCI Ġ+ ĠCD Ġto Ġdetect ĠL SC . ĠThey Ġstate Ġthat Ġa Ġword Ġsense Ġis Ġdetermined Ġby Ġits Ġcontext , Ġbut Ġrelevant Ġcontext Ġwords Ġcan Ġalso Ġbe Ġfound Ġoutside Ġa Ġpred efined Ġwindow . ĠTherefore , Ġthey Ġuse Ġtf - id f Ġto Ġselect Ġrelevant Ġcontext ĠB IB REF 24 . </s> Particip ating ĠSystems Ġ: :: ĠEdu - Phil </s> Similar Ġto Ġteam ĠD AF Ġthey Ġalso Ġuse Ġfast Text Ġ+ ĠOP Ġ+ ĠCD . ĠTheir Ġhypothesis Ġis Ġthat Ġfast Text Ġmay Ġincrease Ġthe Ġperformance Ġfor Ġless Ġfrequent Ġwords Ġin Ġthe Ġcorpus Ġsince Ġgenerating Ġword Ġembed d ings Ġin Ġfast text Ġis Ġbased Ġon Ġcharacter Ġn - gram s . </s> Particip ating ĠSystems Ġ: :: Ġorange fox es </s> They Ġuse Ġthe Ġmodel Ġby ĠB IB REF 5 Ġwhich Ġis Ġbased Ġon ĠSG NS , Ġbut Ġavoids Ġalignment Ġby Ġtreating Ġtime Ġas Ġa Ġvector Ġthat Ġmay Ġbe Ġcombined Ġwith Ġword Ġvectors Ġto Ġget Ġtime - specific Ġword Ġvectors . </s> Particip ating ĠSystems Ġ: :: ĠLoud ĠWhis per </s> L oud ĠWhis per Ġbase Ġtheir Ġapproach Ġon ĠB IB REF 9 Ġwhich Ġis Ġa Ġgraph - based Ġsense Ġclust ering Ġmodel . ĠThey Ġprocess Ġthe Ġdata Ġset Ġto Ġreceive Ġbig rams , Ġcreate Ġa Ġco - occ ure nce Ġgraph Ġrepresentation Ġand Ġafter Ġclust ering Ġassess Ġthe Ġtype Ġof Ġchange Ġper Ġword Ġby Ġcomparing Ġthe Ġresults Ġagainst Ġan Ġintersection Ġtable . ĠTheir Ġmotivation Ġis Ġnot Ġonly Ġto Ġuse Ġa Ġgraph - based Ġapproach , Ġbut Ġto Ġextend Ġthe Ġapproach Ġby Ġenabling Ġchange Ġdetection Ġfor Ġall Ġparts Ġof Ġspeech Ġas Ġopposed Ġto Ġthe Ġoriginal Ġmodel . </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 8 Ġshows Ġthe Ġresults Ġof Ġthe Ġshared Ġtask . ĠAll Ġteams Ġreceive Ġbetter Ġresults Ġthan Ġbaseline Ġ1 Ġ( FD ), Ġof Ġwhich Ġa Ġtotal Ġof Ġ8 Ġteams Ġoutper form Ġbaseline Ġ2 Ġ( C NT Ġ+ ĠCI Ġ+ ĠCD ). ĠThe Ġ4 Ġtop Ġscores Ġwith Ġ$\ r ho Ġ$ Ġ$ > $ Ġ0 . 7 Ġare Ġeither Ġmodified Ġversions Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġor Ġuse ĠSG NS Ġ+ ĠVI Ġ+ ĠCD . ĠThe Ġfollowing Ġ4 Ġscores Ġin Ġthe Ġrange Ġof Ġ0 . 5 Ġ$ < $ Ġ$\ r ho Ġ$ Ġ$ < $ Ġ0 . 6 Ġare Ġgenerated Ġby Ġthe Ġmodels Ġfast Text Ġ+ ĠOP Ġ+ ĠCD , ĠSG NS Ġ+ ĠOP Ġ+ ĠCD / J SD , Ġand ĠP PM I Ġ+ ĠWI Ġ+ ĠCD . </s> Cont rary Ġto Ġthe Ġresults Ġby ĠB IB REF 0 Ġthe Ġmodified Ġversion Ġof Ġvector Ġinitialization Ġshows Ġhigh Ġperformance Ġsimilar Ġto ĠOP Ġalignment , Ġas Ġpreviously Ġreported Ġby ĠB IB REF 14 . ĠSome Ġmodifications Ġto Ġthe ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġapproach Ġare Ġable Ġto Ġyield Ġbetter Ġresults Ġthan Ġothers , Ġe . g . Ġnoise - aware Ġalignment Ġand Ġbin ar ized Ġmat rices Ġas Ġcompared Ġto Ġfrequency - driven ĠOP Ġalignment Ġor Ġlocal Ġalignment Ġwith ĠK NN . ĠTeam ĠSn akes On AP lane Ġcompare Ġtwo Ġdistance Ġmeasures Ġand Ġtheir Ġresults Ġshow Ġthat ĠJ SD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 561 ) Ġperforms Ġminim ally Ġworse Ġthan ĠCD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 565 ) Ġas Ġthe Ġsemantic Ġchange Ġmeasure Ġfor Ġtheir Ġmodel . </s> The Ġoverall Ġbest - performing Ġmodel Ġis ĠSkip - G ram Ġwith Ġorth og onal Ġalignment Ġand Ġcos ine Ġdistance Ġ( SG NS Ġ+ ĠOP Ġ+ ĠCD ) Ġwith Ġsimilar Ġhyper param eters Ġas Ġin Ġthe Ġmodel Ġarchitecture Ġdescribed Ġpreviously ĠB IB REF 0 . ĠSaid Ġarchitecture Ġwas Ġused Ġas Ġthe Ġbasis Ġfor Ġthe Ġtwo Ġbest Ġperforming Ġmodels . ĠTeam Ġtid oe Ġreports Ġthat Ġbin ar izing Ġmat rices Ġleads Ġto Ġa Ġgenerally Ġworse Ġperformance Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 8 11 ) Ġcompared Ġto Ġthe Ġun modified Ġversion Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ0 . 9 ). ĠThe Ġnoise Ġaware Ġalignment Ġapproach Ġapplied Ġby Ġteam Ġsore ns bn Ġob tains Ġa Ġhigher Ġscore Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 8 54 ) Ġcompared Ġto Ġthe Ġresult Ġreported Ġby Ġtid oe , Ġbut Ġis Ġunable Ġto Ġexceed Ġthe Ġperformance Ġof Ġthe Ġun modified ĠS NG S Ġ+ ĠOP Ġ+ ĠCD Ġfor Ġthe Ġsame Ġset Ġof Ġhyper param eters Ġ( window Ġsize Ġ= Ġ10 , Ġnegative Ġsampling Ġ= Ġ1 ; Ġsubs am pling Ġ= ĠNone ). ĠOf Ġthe Ġ8 Ġscores Ġabove Ġthe Ġsecond Ġbaseline , Ġ5 Ġuse Ġan Ġarchitecture Ġthat Ġbuilds Ġupon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠWhereas Ġin Ġthe Ġlower Ġscore Ġsegment Ġ$\ r ho Ġ$ Ġ$ < $ Ġ0 . 5 Ġnone Ġof Ġthe Ġmodels Ġuse ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠThese Ġfindings Ġare Ġin Ġline Ġwith Ġthe Ġresults Ġreported Ġby ĠB IB REF 0 , Ġhowever Ġthe Ġoverall Ġbest Ġresults Ġare Ġlower Ġin Ġthis Ġshared Ġtask , Ġwhich Ġis Ġexpected Ġfrom Ġthe Ġsmaller Ġnumber Ġof Ġparameter Ġcombinations Ġexplored . ĠAdditionally , Ġin Ġthe Ġshared Ġtask Ġthe Ġobjective Ġwas Ġto Ġreport Ġthe Ġbest Ġscore Ġand Ġnot Ġto Ġcalculate Ġthe Ġmean Ġwhich Ġmakes Ġit Ġmore Ġdifficult Ġto Ġcompare Ġthe Ġrobust ness Ġof Ġthe Ġmodels Ġpresented Ġhere .
Introduction Natural languages evolve and words have always been subject to semantic change over time BIBREF1. With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9. However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10. Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets. In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons. For this reason we report the results of the first shared task on unsupervised lexical semantic change detection in German that is based on an annotated data set to guarantee objective reasoning throughout different approaches. The task was organized as part of the seminar 'Lexical Semantic Change Detection' at the IMS Stuttgart in the summer term of 2019. Task The goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods. Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges. A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task. There was one group consisting of 3 team members and two individuals who entered the task on their own. In total there were 12 LSCD systems participating in the shared task. The shared task was divided into three phases, i.e., development, testing and analysis phase. In the development phase each team implemented a first version of their model based on a trial data set and submitted it subsequently. In the testing phase the testing data was made public and participants applied their models to the test data with a restriction of possible result uploads to 30. The leaderboard was public at all times. Eventually, the analysis phase was entered and the models of the testing phase were evaluated in terms of the predictions they made and parameters could be tuned further. The models and results will be discussed in detail in sections 7 and 8. Corpora The task, as framed above, requires to detect the semantic change between two corpora. The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19. They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century. DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899. The corpus version used in the task has the following format: "year [tab] lemma1 lemma2 lemma3 ...". Evaluation The Diachronic Usage Relatedness (DURel) gold standard data set includes 22 target words and their varying degrees of semantic change BIBREF12. For each of these target words a random sample of use pairs from the DTA corpus was retrieved and annotated. The annotators were required to rate the pairs according to their semantic relatedness on a scale from 1 to 4 (unrelated - identical meanings) for two time periods. The average Spearman's $\rho $ between the five annotators was 0.66 for 1,320 use paris. The resulting word ranking of the DURel data set is determined by the mean usage relatedness across two time periods and is used as the benchmark to compare the models’ performances in the shared task. Evaluation ::: Metric The output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set. As the metric to assess how well the model's output fits the gold ranking Spearman's $\rho $ was used. The higher Spearman's rank-order correlation the better the system's performance. Evaluation ::: Baselines Models were compared to two baselines for the shared task: log-transformed normalized frequency difference (FD) count vectors with column intersection and cosine distance (CNT + CI + CD) The window size for CNT + CI + CD was 10. Find more information on these models in BIBREF0. Participating Systems Participants mostly rely on the models compared in BIBREF0 and apply modifications to improve them. In particular, most teams make use of skip-gram with negative sampling (SGNS) based on BIBREF13 to learn the semantic spaces of the two time periods and orthogonal procrustes (OP) to align these vector spaces, similar to the approach by BIBREF14. Different meaning representations such as sense clusters are used as well. As measure to detect the degree of LSC all teams except one choose cosine distance (CD). This team uses Jensen-Shannon distance (JSD) instead, which computes the distance between probability distributions BIBREF15. The models of each team will be briefly introduced in this section. Participating Systems ::: sorensbn Team sorensbn makes use of SGNS + OP + CD to detect LSC. They use similar hyperparameters as in BIBREF0 to tune the SGNS model. They use an open-sourced noise-aware implementation to improve the OP alignment BIBREF16. Participating Systems ::: tidoe Team tidoe builds on SGNS + OP + CD, but they add a transformation step to receive binarized representations of matrices BIBREF17. This step is taken to counter the bias that can occur in vector-space models based on frequencies BIBREF18. Participating Systems ::: in vain The team applies a model based on SGNS with vector initialization alignment and cosine distance (SGNS + VI + CD). Vector initialization is an alignment strategy where the vector space learning model for $t_2$ is initialized with the vectors from $t_1$ BIBREF19. Since SGNS + VI + OP does not perform as well as other models in BIBREF0, they alter the vector initialization process by initializing on the complete model instead of only the word matrix of $t_1$ to receive improved results. Participating Systems ::: Evilly In line with previous approaches, team Evilly builds upon SGNS + OP + CD. They alter the OP step by using only high-frequency words for alignment. Participating Systems ::: DAF Team DAF uses an architecture based on learning vectors with fastText, alignment with unsupervised and supervised variations of OP, and CD, using the MUSE package BIBREF20, BIBREF21. For the supervised alignment stop words are used. The underlying assumption is that stop words serve as functional units of language and their usage should be consistent over time. Participating Systems ::: SnakesOnAPlane The team learns vector spaces with count vectors, positive pointwise mutual information (PPMI), SGNS and uses column intersection (CI) and OP as alignment techniques where applicable. Then they compare two distance measures (CD and JSD) for the different models CNT + CI, PPMI + CI and SGNS + OP to identify which measure performs better for these models. They also experiment with different ways to remove negative values from SGNS vectors, which is needed for JSD. Participating Systems ::: TeamKulkarni15 TeamKulkarni15 uses SGNS + OP + CD with the modification of local alignment with k nearest neighbors, since other models often use global alignment that can be prone to noise BIBREF22. Participating Systems ::: Bashmaistori They use word injection (WI) alignment on PPMI vectors with CD. This approach avoids the complex alignment procedure for embeddings and is applicable to embeddings and count-based methods. They compare two implementations of word injection BIBREF23, BIBREF0 as these showed different results on different data sets. Participating Systems ::: giki Team giki uses PPMI + CI + CD to detect LSC. They state that a word sense is determined by its context, but relevant context words can also be found outside a predefined window. Therefore, they use tf-idf to select relevant context BIBREF24. Participating Systems ::: Edu-Phil Similar to team DAF they also use fastText + OP + CD. Their hypothesis is that fastText may increase the performance for less frequent words in the corpus since generating word embeddings in fasttext is based on character n-grams. Participating Systems ::: orangefoxes They use the model by BIBREF5 which is based on SGNS, but avoids alignment by treating time as a vector that may be combined with word vectors to get time-specific word vectors. Participating Systems ::: Loud Whisper Loud Whisper base their approach on BIBREF9 which is a graph-based sense clustering model. They process the data set to receive bigrams, create a co-occurence graph representation and after clustering assess the type of change per word by comparing the results against an intersection table. Their motivation is not only to use a graph-based approach, but to extend the approach by enabling change detection for all parts of speech as opposed to the original model. Results and Discussion Table TABREF8 shows the results of the shared task. All teams receive better results than baseline 1 (FD), of which a total of 8 teams outperform baseline 2 (CNT + CI + CD). The 4 top scores with $\rho $ $>$ 0.7 are either modified versions of SGNS + OP + CD or use SGNS + VI + CD. The following 4 scores in the range of 0.5 $<$ $\rho $ $<$ 0.6 are generated by the models fastText + OP + CD, SGNS + OP + CD/JSD, and PPMI + WI + CD. Contrary to the results by BIBREF0 the modified version of vector initialization shows high performance similar to OP alignment, as previously reported by BIBREF14. Some modifications to the SGNS + OP + CD approach are able to yield better results than others, e.g. noise-aware alignment and binarized matrices as compared to frequency-driven OP alignment or local alignment with KNN. Team SnakesOnAPlane compare two distance measures and their results show that JSD ($\rho $ $=$ .561) performs minimally worse than CD ($\rho $ $=$ .565) as the semantic change measure for their model. The overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS + OP + CD) with similar hyperparameters as in the model architecture described previously BIBREF0. Said architecture was used as the basis for the two best performing models. Team tidoe reports that binarizing matrices leads to a generally worse performance ($\rho $ $=$ .811) compared to the unmodified version of SGNS + OP + CD ($\rho $ $=$ 0.9). The noise aware alignment approach applied by team sorensbn obtains a higher score ($\rho $ $=$ .854) compared to the result reported by tidoe, but is unable to exceed the performance of the unmodified SNGS + OP + CD for the same set of hyperparameters (window size = 10, negative sampling = 1; subsampling = None). Of the 8 scores above the second baseline, 5 use an architecture that builds upon SGNS + OP + CD. Whereas in the lower score segment $\rho $ $<$ 0.5 none of the models use SGNS + OP + CD. These findings are in line with the results reported by BIBREF0, however the overall best results are lower in this shared task, which is expected from the smaller number of parameter combinations explored. Additionally, in the shared task the objective was to report the best score and not to calculate the mean which makes it more difficult to compare the robustness of the models presented here.
What is the algorithm used to create word embeddings?
902b3123aec0f3a39319ffa9d05ab8e08a2eb567
What Ġis Ġthe Ġalgorithm Ġused Ġto Ġcreate Ġword Ġembed d ings ?
Un answer able
<s> What Ġis Ġthe Ġalgorithm Ġused Ġto Ġcreate Ġword Ġembed d ings ? </s> Introduction </s> Natural Ġlanguages Ġevolve Ġand Ġwords Ġhave Ġalways Ġbeen Ġsubject Ġto Ġsemantic Ġchange Ġover Ġtime ĠB IB REF 1 . ĠWith Ġthe Ġrise Ġof Ġlarge Ġdigit ized Ġtext Ġresources Ġrecent ĠN LP Ġtechnologies Ġhave Ġmade Ġit Ġpossible Ġto Ġcapture Ġsuch Ġchange Ġwith Ġvector Ġspace Ġmodels ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġtopic Ġmodels ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , Ġand Ġsense Ġclust ering Ġmodels ĠB IB REF 9 . ĠHowever , Ġmany Ġapproaches Ġfor Ġdetecting ĠL SC Ġdiffer Ġprofoundly Ġfrom Ġeach Ġother Ġand Ġtherefore Ġdrawing Ġcomparisons Ġbetween Ġthem Ġcan Ġbe Ġchallenging ĠB IB REF 10 . ĠNot Ġonly Ġdo Ġarchitectures Ġfor Ġdetecting ĠL SC Ġvary , Ġtheir Ġperformance Ġis Ġalso Ġoften Ġevaluated Ġwithout Ġaccess Ġto Ġevaluation Ġdata Ġor Ġtoo Ġsparse Ġdata Ġsets . ĠIn Ġcases Ġwhere Ġevaluation Ġdata Ġis Ġavailable , Ġoft entimes ĠL SC D Ġsystems Ġare Ġnot Ġevaluated Ġon Ġthe Ġsame Ġdata Ġset Ġwhich Ġhind ers Ġthe Ġresearch Ġcommunity Ġto Ġdraw Ġcomparisons . </s> For Ġthis Ġreason Ġwe Ġreport Ġthe Ġresults Ġof Ġthe Ġfirst Ġshared Ġtask Ġon Ġun super vised Ġlex ical Ġsemantic Ġchange Ġdetection Ġin ĠGerman Ġthat Ġis Ġbased Ġon Ġan Ġannot ated Ġdata Ġset Ġto Ġguarantee Ġobjective Ġreasoning Ġthroughout Ġdifferent Ġapproaches . ĠThe Ġtask Ġwas Ġorganized Ġas Ġpart Ġof Ġthe Ġseminar Ġ' Lex ical ĠSem antic ĠChange ĠDetection ' Ġat Ġthe ĠI MS ĠSt utt gart Ġin Ġthe Ġsummer Ġterm Ġof Ġ2019 . </s> Task </s> The Ġgoal Ġof Ġthe Ġshared Ġtask Ġwas Ġto Ġcreate Ġan Ġarchitecture Ġto Ġdetect Ġsemantic Ġchange Ġand Ġto Ġrank Ġwords Ġaccording Ġto Ġtheir Ġdegree Ġof Ġchange Ġbetween Ġtwo Ġdifferent Ġtime Ġperiods . ĠGiven Ġtwo Ġcorpor a ĠCa Ġand ĠC b , Ġthe Ġtarget Ġwords Ġhad Ġto Ġbe Ġranked Ġaccording Ġto Ġtheir Ġdegree Ġof Ġlex ical Ġsemantic Ġchange Ġbetween ĠCa Ġand ĠC b Ġas Ġannot ated Ġby Ġhuman Ġjudges . ĠA Ġcompetition Ġwas Ġset Ġup Ġon ĠCod al ab Ġand Ġteams Ġmostly Ġconsisting Ġof Ġ2 Ġpeople Ġwere Ġformed Ġto Ġtake Ġpart Ġin Ġthe Ġtask . ĠThere Ġwas Ġone Ġgroup Ġconsisting Ġof Ġ3 Ġteam Ġmembers Ġand Ġtwo Ġindividuals Ġwho Ġentered Ġthe Ġtask Ġon Ġtheir Ġown . ĠIn Ġtotal Ġthere Ġwere Ġ12 ĠL SC D Ġsystems Ġparticipating Ġin Ġthe Ġshared Ġtask . </s> The Ġshared Ġtask Ġwas Ġdivided Ġinto Ġthree Ġphases , Ġi . e ., Ġdevelopment , Ġtesting Ġand Ġanalysis Ġphase . ĠIn Ġthe Ġdevelopment Ġphase Ġeach Ġteam Ġimplemented Ġa Ġfirst Ġversion Ġof Ġtheir Ġmodel Ġbased Ġon Ġa Ġtrial Ġdata Ġset Ġand Ġsubmitted Ġit Ġsubsequently . ĠIn Ġthe Ġtesting Ġphase Ġthe Ġtesting Ġdata Ġwas Ġmade Ġpublic Ġand Ġparticipants Ġapplied Ġtheir Ġmodels Ġto Ġthe Ġtest Ġdata Ġwith Ġa Ġrestriction Ġof Ġpossible Ġresult Ġupload s Ġto Ġ30 . ĠThe Ġleader board Ġwas Ġpublic Ġat Ġall Ġtimes . ĠEventually , Ġthe Ġanalysis Ġphase Ġwas Ġentered Ġand Ġthe Ġmodels Ġof Ġthe Ġtesting Ġphase Ġwere Ġevaluated Ġin Ġterms Ġof Ġthe Ġpredictions Ġthey Ġmade Ġand Ġparameters Ġcould Ġbe Ġtuned Ġfurther . ĠThe Ġmodels Ġand Ġresults Ġwill Ġbe Ġdiscussed Ġin Ġdetail Ġin Ġsections Ġ7 Ġand Ġ8 . </s> Cor pora </s> The Ġtask , Ġas Ġframed Ġabove , Ġrequires Ġto Ġdetect Ġthe Ġsemantic Ġchange Ġbetween Ġtwo Ġcorpor a . ĠThe Ġtwo Ġcorpor a Ġused Ġin Ġthe Ġshared Ġtask Ġcorrespond Ġto Ġthe Ġdi ach ronic Ġcorpus Ġpair Ġfrom ĠB IB REF 0 : ĠD TA 18 Ġand ĠD TA 19 . ĠThey Ġconsist Ġof Ġsub parts Ġof ĠD TA Ġcorpus ĠB IB REF 11 Ġwhich Ġis Ġa Ġfreely Ġavailable Ġle mm at ized , ĠPOS - tag ged Ġand Ġspelling - normal ized Ġdi ach ronic Ġcorpus Ġof ĠGerman Ġcontaining Ġtexts Ġfrom Ġthe Ġ16 th Ġto Ġthe Ġ20 th Ġcentury . ĠD TA 18 Ġcontains Ġ26 Ġmillion Ġsentences Ġpublished Ġbetween Ġ17 50 - 17 99 Ġand ĠD TA 19 Ġ40 Ġmillion Ġbetween Ġ1850 - 18 99 . ĠThe Ġcorpus Ġversion Ġused Ġin Ġthe Ġtask Ġhas Ġthe Ġfollowing Ġformat : Ġ" year Ġ[ tab ] Ġle mma 1 Ġle mma 2 Ġle mma 3 Ġ..." . </s> E val uation </s> The ĠDi ach ronic ĠUsage ĠRelated ness Ġ( D UR el ) Ġgold Ġstandard Ġdata Ġset Ġincludes Ġ22 Ġtarget Ġwords Ġand Ġtheir Ġvarying Ġdegrees Ġof Ġsemantic Ġchange ĠB IB REF 12 . ĠFor Ġeach Ġof Ġthese Ġtarget Ġwords Ġa Ġrandom Ġsample Ġof Ġuse Ġpairs Ġfrom Ġthe ĠD TA Ġcorpus Ġwas Ġretrieved Ġand Ġannot ated . ĠThe Ġannot ators Ġwere Ġrequired Ġto Ġrate Ġthe Ġpairs Ġaccording Ġto Ġtheir Ġsemantic Ġrelated ness Ġon Ġa Ġscale Ġfrom Ġ1 Ġto Ġ4 Ġ( un related Ġ- Ġidentical Ġmeanings ) Ġfor Ġtwo Ġtime Ġperiods . ĠThe Ġaverage ĠSpear man 's Ġ$\ r ho Ġ$ Ġbetween Ġthe Ġfive Ġannot ators Ġwas Ġ0 . 66 Ġfor Ġ1 , 320 Ġuse Ġpar is . ĠThe Ġresulting Ġword Ġranking Ġof Ġthe ĠD UR el Ġdata Ġset Ġis Ġdetermined Ġby Ġthe Ġmean Ġusage Ġrelated ness Ġacross Ġtwo Ġtime Ġperiods Ġand Ġis Ġused Ġas Ġthe Ġbenchmark Ġto Ġcompare Ġthe Ġmodels âĢ Ļ Ġperformances Ġin Ġthe Ġshared Ġtask . </s> E val uation Ġ: :: ĠMet ric </s> The Ġoutput Ġof Ġa Ġsystem Ġwith Ġthe Ġtarget Ġwords Ġin Ġthe Ġpredicted Ġorder Ġis Ġcompared Ġto Ġthe Ġgold Ġranking Ġof Ġthe ĠD UR el Ġdata Ġset . ĠAs Ġthe Ġmetric Ġto Ġassess Ġhow Ġwell Ġthe Ġmodel 's Ġoutput Ġfits Ġthe Ġgold Ġranking ĠSpear man 's Ġ$\ r ho Ġ$ Ġwas Ġused . ĠThe Ġhigher ĠSpear man 's Ġrank - order Ġcorrelation Ġthe Ġbetter Ġthe Ġsystem 's Ġperformance . </s> E val uation Ġ: :: ĠBas elines </s> Mod els Ġwere Ġcompared Ġto Ġtwo Ġbas elines Ġfor Ġthe Ġshared Ġtask : </s> log - trans formed Ġnormalized Ġfrequency Ġdifference Ġ( FD ) </s> count Ġvectors Ġwith Ġcolumn Ġintersection Ġand Ġcos ine Ġdistance Ġ( C NT Ġ+ ĠCI Ġ+ ĠCD ) </s> The Ġwindow Ġsize Ġfor ĠC NT Ġ+ ĠCI Ġ+ ĠCD Ġwas Ġ10 . ĠFind Ġmore Ġinformation Ġon Ġthese Ġmodels Ġin ĠB IB REF 0 . </s> Particip ating ĠSystems </s> Particip ants Ġmostly Ġrely Ġon Ġthe Ġmodels Ġcompared Ġin ĠB IB REF 0 Ġand Ġapply Ġmodifications Ġto Ġimprove Ġthem . ĠIn Ġparticular , Ġmost Ġteams Ġmake Ġuse Ġof Ġskip - gram Ġwith Ġnegative Ġsampling Ġ( SG NS ) Ġbased Ġon ĠB IB REF 13 Ġto Ġlearn Ġthe Ġsemantic Ġspaces Ġof Ġthe Ġtwo Ġtime Ġperiods Ġand Ġorth og onal Ġpro cr ust es Ġ( OP ) Ġto Ġalign Ġthese Ġvector Ġspaces , Ġsimilar Ġto Ġthe Ġapproach Ġby ĠB IB REF 14 . ĠDifferent Ġmeaning Ġrepresentations Ġsuch Ġas Ġsense Ġclusters Ġare Ġused Ġas Ġwell . ĠAs Ġmeasure Ġto Ġdetect Ġthe Ġdegree Ġof ĠL SC Ġall Ġteams Ġexcept Ġone Ġchoose Ġcos ine Ġdistance Ġ( CD ). ĠThis Ġteam Ġuses ĠJensen - Sh annon Ġdistance Ġ( J SD ) Ġinstead , Ġwhich Ġcomp utes Ġthe Ġdistance Ġbetween Ġprobability Ġdistributions ĠB IB REF 15 . </s> The Ġmodels Ġof Ġeach Ġteam Ġwill Ġbe Ġbriefly Ġintroduced Ġin Ġthis Ġsection . </s> Particip ating ĠSystems Ġ: :: Ġsore ns bn </s> Team Ġsore ns bn Ġmakes Ġuse Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġto Ġdetect ĠL SC . ĠThey Ġuse Ġsimilar Ġhyper param eters Ġas Ġin ĠB IB REF 0 Ġto Ġtune Ġthe ĠSG NS Ġmodel . ĠThey Ġuse Ġan Ġopen - s ourced Ġnoise - aware Ġimplementation Ġto Ġimprove Ġthe ĠOP Ġalignment ĠB IB REF 16 . </s> Particip ating ĠSystems Ġ: :: Ġtid oe </s> Team Ġtid oe Ġbuilds Ġon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD , Ġbut Ġthey Ġadd Ġa Ġtransformation Ġstep Ġto Ġreceive Ġbin ar ized Ġrepresentations Ġof Ġmat rices ĠB IB REF 17 . ĠThis Ġstep Ġis Ġtaken Ġto Ġcounter Ġthe Ġbias Ġthat Ġcan Ġoccur Ġin Ġvector - space Ġmodels Ġbased Ġon Ġfrequencies ĠB IB REF 18 . </s> Particip ating ĠSystems Ġ: :: Ġin Ġvain </s> The Ġteam Ġapplies Ġa Ġmodel Ġbased Ġon ĠSG NS Ġwith Ġvector Ġinitialization Ġalignment Ġand Ġcos ine Ġdistance Ġ( SG NS Ġ+ ĠVI Ġ+ ĠCD ). ĠVector Ġinitialization Ġis Ġan Ġalignment Ġstrategy Ġwhere Ġthe Ġvector Ġspace Ġlearning Ġmodel Ġfor Ġ$ t _ 2 $ Ġis Ġinitialized Ġwith Ġthe Ġvectors Ġfrom Ġ$ t _ 1 $ ĠB IB REF 19 . ĠSince ĠSG NS Ġ+ ĠVI Ġ+ ĠOP Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġother Ġmodels Ġin ĠB IB REF 0 , Ġthey Ġalter Ġthe Ġvector Ġinitialization Ġprocess Ġby Ġinitial izing Ġon Ġthe Ġcomplete Ġmodel Ġinstead Ġof Ġonly Ġthe Ġword Ġmatrix Ġof Ġ$ t _ 1 $ Ġto Ġreceive Ġimproved Ġresults . </s> Particip ating ĠSystems Ġ: :: ĠEv illy </s> In Ġline Ġwith Ġprevious Ġapproaches , Ġteam ĠEv illy Ġbuilds Ġupon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠThey Ġalter Ġthe ĠOP Ġstep Ġby Ġusing Ġonly Ġhigh - frequency Ġwords Ġfor Ġalignment . </s> Particip ating ĠSystems Ġ: :: ĠD AF </s> Team ĠD AF Ġuses Ġan Ġarchitecture Ġbased Ġon Ġlearning Ġvectors Ġwith Ġfast Text , Ġalignment Ġwith Ġun super vised Ġand Ġsupervised Ġvariations Ġof ĠOP , Ġand ĠCD , Ġusing Ġthe ĠM USE Ġpackage ĠB IB REF 20 , ĠB IB REF 21 . ĠFor Ġthe Ġsupervised Ġalignment Ġstop Ġwords Ġare Ġused . ĠThe Ġunderlying Ġassumption Ġis Ġthat Ġstop Ġwords Ġserve Ġas Ġfunctional Ġunits Ġof Ġlanguage Ġand Ġtheir Ġusage Ġshould Ġbe Ġconsistent Ġover Ġtime . </s> Particip ating ĠSystems Ġ: :: ĠSn akes On AP lane </s> The Ġteam Ġlearns Ġvector Ġspaces Ġwith Ġcount Ġvectors , Ġpositive Ġpoint wise Ġmutual Ġinformation Ġ( P PM I ), ĠSG NS Ġand Ġuses Ġcolumn Ġintersection Ġ( CI ) Ġand ĠOP Ġas Ġalignment Ġtechniques Ġwhere Ġapplicable . ĠThen Ġthey Ġcompare Ġtwo Ġdistance Ġmeasures Ġ( CD Ġand ĠJ SD ) Ġfor Ġthe Ġdifferent Ġmodels ĠC NT Ġ+ ĠCI , ĠP PM I Ġ+ ĠCI Ġand ĠSG NS Ġ+ ĠOP Ġto Ġidentify Ġwhich Ġmeasure Ġperforms Ġbetter Ġfor Ġthese Ġmodels . ĠThey Ġalso Ġexperiment Ġwith Ġdifferent Ġways Ġto Ġremove Ġnegative Ġvalues Ġfrom ĠSG NS Ġvectors , Ġwhich Ġis Ġneeded Ġfor ĠJ SD . </s> Particip ating ĠSystems Ġ: :: ĠTeam K ulk arn i 15 </s> Team K ulk arn i 15 Ġuses ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġwith Ġthe Ġmodification Ġof Ġlocal Ġalignment Ġwith Ġk Ġnearest Ġneighbors , Ġsince Ġother Ġmodels Ġoften Ġuse Ġglobal Ġalignment Ġthat Ġcan Ġbe Ġprone Ġto Ġnoise ĠB IB REF 22 . </s> Particip ating ĠSystems Ġ: :: ĠBash ma ist ori </s> They Ġuse Ġword Ġinjection Ġ( WI ) Ġalignment Ġon ĠP PM I Ġvectors Ġwith ĠCD . ĠThis Ġapproach Ġavoids Ġthe Ġcomplex Ġalignment Ġprocedure Ġfor Ġembed d ings Ġand Ġis Ġapplicable Ġto Ġembed d ings Ġand Ġcount - based Ġmethods . ĠThey Ġcompare Ġtwo Ġimplementations Ġof Ġword Ġinjection ĠB IB REF 23 , ĠB IB REF 0 Ġas Ġthese Ġshowed Ġdifferent Ġresults Ġon Ġdifferent Ġdata Ġsets . </s> Particip ating ĠSystems Ġ: :: Ġg iki </s> Team Ġg iki Ġuses ĠP PM I Ġ+ ĠCI Ġ+ ĠCD Ġto Ġdetect ĠL SC . ĠThey Ġstate Ġthat Ġa Ġword Ġsense Ġis Ġdetermined Ġby Ġits Ġcontext , Ġbut Ġrelevant Ġcontext Ġwords Ġcan Ġalso Ġbe Ġfound Ġoutside Ġa Ġpred efined Ġwindow . ĠTherefore , Ġthey Ġuse Ġtf - id f Ġto Ġselect Ġrelevant Ġcontext ĠB IB REF 24 . </s> Particip ating ĠSystems Ġ: :: ĠEdu - Phil </s> Similar Ġto Ġteam ĠD AF Ġthey Ġalso Ġuse Ġfast Text Ġ+ ĠOP Ġ+ ĠCD . ĠTheir Ġhypothesis Ġis Ġthat Ġfast Text Ġmay Ġincrease Ġthe Ġperformance Ġfor Ġless Ġfrequent Ġwords Ġin Ġthe Ġcorpus Ġsince Ġgenerating Ġword Ġembed d ings Ġin Ġfast text Ġis Ġbased Ġon Ġcharacter Ġn - gram s . </s> Particip ating ĠSystems Ġ: :: Ġorange fox es </s> They Ġuse Ġthe Ġmodel Ġby ĠB IB REF 5 Ġwhich Ġis Ġbased Ġon ĠSG NS , Ġbut Ġavoids Ġalignment Ġby Ġtreating Ġtime Ġas Ġa Ġvector Ġthat Ġmay Ġbe Ġcombined Ġwith Ġword Ġvectors Ġto Ġget Ġtime - specific Ġword Ġvectors . </s> Particip ating ĠSystems Ġ: :: ĠLoud ĠWhis per </s> L oud ĠWhis per Ġbase Ġtheir Ġapproach Ġon ĠB IB REF 9 Ġwhich Ġis Ġa Ġgraph - based Ġsense Ġclust ering Ġmodel . ĠThey Ġprocess Ġthe Ġdata Ġset Ġto Ġreceive Ġbig rams , Ġcreate Ġa Ġco - occ ure nce Ġgraph Ġrepresentation Ġand Ġafter Ġclust ering Ġassess Ġthe Ġtype Ġof Ġchange Ġper Ġword Ġby Ġcomparing Ġthe Ġresults Ġagainst Ġan Ġintersection Ġtable . ĠTheir Ġmotivation Ġis Ġnot Ġonly Ġto Ġuse Ġa Ġgraph - based Ġapproach , Ġbut Ġto Ġextend Ġthe Ġapproach Ġby Ġenabling Ġchange Ġdetection Ġfor Ġall Ġparts Ġof Ġspeech Ġas Ġopposed Ġto Ġthe Ġoriginal Ġmodel . </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 8 Ġshows Ġthe Ġresults Ġof Ġthe Ġshared Ġtask . ĠAll Ġteams Ġreceive Ġbetter Ġresults Ġthan Ġbaseline Ġ1 Ġ( FD ), Ġof Ġwhich Ġa Ġtotal Ġof Ġ8 Ġteams Ġoutper form Ġbaseline Ġ2 Ġ( C NT Ġ+ ĠCI Ġ+ ĠCD ). ĠThe Ġ4 Ġtop Ġscores Ġwith Ġ$\ r ho Ġ$ Ġ$ > $ Ġ0 . 7 Ġare Ġeither Ġmodified Ġversions Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġor Ġuse ĠSG NS Ġ+ ĠVI Ġ+ ĠCD . ĠThe Ġfollowing Ġ4 Ġscores Ġin Ġthe Ġrange Ġof Ġ0 . 5 Ġ$ < $ Ġ$\ r ho Ġ$ Ġ$ < $ Ġ0 . 6 Ġare Ġgenerated Ġby Ġthe Ġmodels Ġfast Text Ġ+ ĠOP Ġ+ ĠCD , ĠSG NS Ġ+ ĠOP Ġ+ ĠCD / J SD , Ġand ĠP PM I Ġ+ ĠWI Ġ+ ĠCD . </s> Cont rary Ġto Ġthe Ġresults Ġby ĠB IB REF 0 Ġthe Ġmodified Ġversion Ġof Ġvector Ġinitialization Ġshows Ġhigh Ġperformance Ġsimilar Ġto ĠOP Ġalignment , Ġas Ġpreviously Ġreported Ġby ĠB IB REF 14 . ĠSome Ġmodifications Ġto Ġthe ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġapproach Ġare Ġable Ġto Ġyield Ġbetter Ġresults Ġthan Ġothers , Ġe . g . Ġnoise - aware Ġalignment Ġand Ġbin ar ized Ġmat rices Ġas Ġcompared Ġto Ġfrequency - driven ĠOP Ġalignment Ġor Ġlocal Ġalignment Ġwith ĠK NN . ĠTeam ĠSn akes On AP lane Ġcompare Ġtwo Ġdistance Ġmeasures Ġand Ġtheir Ġresults Ġshow Ġthat ĠJ SD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 561 ) Ġperforms Ġminim ally Ġworse Ġthan ĠCD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 565 ) Ġas Ġthe Ġsemantic Ġchange Ġmeasure Ġfor Ġtheir Ġmodel . </s> The Ġoverall Ġbest - performing Ġmodel Ġis ĠSkip - G ram Ġwith Ġorth og onal Ġalignment Ġand Ġcos ine Ġdistance Ġ( SG NS Ġ+ ĠOP Ġ+ ĠCD ) Ġwith Ġsimilar Ġhyper param eters Ġas Ġin Ġthe Ġmodel Ġarchitecture Ġdescribed Ġpreviously ĠB IB REF 0 . ĠSaid Ġarchitecture Ġwas Ġused Ġas Ġthe Ġbasis Ġfor Ġthe Ġtwo Ġbest Ġperforming Ġmodels . ĠTeam Ġtid oe Ġreports Ġthat Ġbin ar izing Ġmat rices Ġleads Ġto Ġa Ġgenerally Ġworse Ġperformance Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 8 11 ) Ġcompared Ġto Ġthe Ġun modified Ġversion Ġof ĠSG NS Ġ+ ĠOP Ġ+ ĠCD Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ0 . 9 ). ĠThe Ġnoise Ġaware Ġalignment Ġapproach Ġapplied Ġby Ġteam Ġsore ns bn Ġob tains Ġa Ġhigher Ġscore Ġ($ \ r ho Ġ$ Ġ$ =$ Ġ. 8 54 ) Ġcompared Ġto Ġthe Ġresult Ġreported Ġby Ġtid oe , Ġbut Ġis Ġunable Ġto Ġexceed Ġthe Ġperformance Ġof Ġthe Ġun modified ĠS NG S Ġ+ ĠOP Ġ+ ĠCD Ġfor Ġthe Ġsame Ġset Ġof Ġhyper param eters Ġ( window Ġsize Ġ= Ġ10 , Ġnegative Ġsampling Ġ= Ġ1 ; Ġsubs am pling Ġ= ĠNone ). ĠOf Ġthe Ġ8 Ġscores Ġabove Ġthe Ġsecond Ġbaseline , Ġ5 Ġuse Ġan Ġarchitecture Ġthat Ġbuilds Ġupon ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠWhereas Ġin Ġthe Ġlower Ġscore Ġsegment Ġ$\ r ho Ġ$ Ġ$ < $ Ġ0 . 5 Ġnone Ġof Ġthe Ġmodels Ġuse ĠSG NS Ġ+ ĠOP Ġ+ ĠCD . ĠThese Ġfindings Ġare Ġin Ġline Ġwith Ġthe Ġresults Ġreported Ġby ĠB IB REF 0 , Ġhowever Ġthe Ġoverall Ġbest Ġresults Ġare Ġlower Ġin Ġthis Ġshared Ġtask , Ġwhich Ġis Ġexpected Ġfrom Ġthe Ġsmaller Ġnumber Ġof Ġparameter Ġcombinations Ġexplored . ĠAdditionally , Ġin Ġthe Ġshared Ġtask Ġthe Ġobjective Ġwas Ġto Ġreport Ġthe Ġbest Ġscore Ġand Ġnot Ġto Ġcalculate Ġthe Ġmean Ġwhich Ġmakes Ġit Ġmore Ġdifficult Ġto Ġcompare Ġthe Ġrobust ness Ġof Ġthe Ġmodels Ġpresented Ġhere .