a benchmark dataset created by ceccarelli2013learning from the CoNLL 2003 data
EXTRACTIVE
dataset created by ceccarelli2013learning from the CoNLL 2003 data
EXTRACTIVE
We use a benchmark dataset created by ceccarelli2013learning from the CoNLL 2003 data. As in previous studies BIBREF16 , BIBREF15 , we model measuring entity relatedness as a ranking problem. We use the test split of the dataset to create 3,314 queries. Each query has a query entity and INLINEFORM0 91 response entities labeled as related or unrelated. The quality is measured by the ability of the system to rank related entities on top of unrelated ones.
We use a benchmark dataset created by ceccarelli2013learning from the CoNLL 2003 data. As in previous studies BIBREF16 , BIBREF15 , we model measuring entity relatedness as a ranking problem. We use the test split of the dataset to create 3,314 queries. Each query has a query entity and INLINEFORM0 91 response entities labeled as related or unrelated. The quality is measured by the ability of the system to rank related entities on top of unrelated ones.
1702.03342
Introduction </s> Vector - based Ġsemantic Ġmapping Ġmodels Ġare Ġused Ġto Ġrepresent Ġtextual Ġstructures Ġ( words , Ġphrases , Ġand Ġdocuments ) Ġas Ġhigh - dimensional Ġmeaning Ġvectors . ĠTypically , Ġthese Ġmodels Ġutilize Ġtextual Ġcorpor a Ġand / or ĠKnowledge ĠB ases Ġ( KB s ) Ġto Ġacquire Ġworld Ġknowledge , Ġwhich Ġis Ġthen Ġused Ġto Ġgenerate Ġa Ġvector Ġrepresentation Ġfor Ġthe Ġgiven Ġtext Ġin Ġthe Ġsemantic Ġspace . ĠThe Ġgoal Ġis Ġthus Ġto Ġaccurately Ġplace Ġsem antically Ġsimilar Ġstructures Ġclose Ġto Ġeach Ġother Ġin Ġthat Ġsemantic Ġspace . ĠOn Ġthe Ġother Ġhand , Ġdiss imilar Ġstructures Ġshould Ġbe Ġfar Ġapart . </s> Expl icit Ġconcept Ġspace Ġmodels Ġare Ġmotivated Ġby Ġthe Ġidea Ġthat Ġhigh Ġlevel Ġcognitive Ġtasks Ġsuch Ġlearning Ġand Ġreasoning Ġare Ġsupported Ġby Ġthe Ġknowledge Ġwe Ġacquire Ġfrom Ġconcepts ĠB IB REF 0 Ġ. ĠTherefore , Ġsuch Ġmodels Ġutilize Ġconcept Ġvectors Ġ( a . k . a Ġbag - of - concept s Ġ( B OC )) Ġas Ġthe Ġunderlying Ġsemantic Ġrepresentation Ġof Ġa Ġgiven Ġtext Ġthrough Ġa Ġprocess Ġcalled Ġconceptual ization , Ġwhich Ġis Ġmapping Ġthe Ġtext Ġinto Ġrelevant Ġconcepts Ġcapturing Ġits Ġmain Ġtopics . ĠThe Ġconcept Ġspace Ġtypically Ġinclude Ġconcepts Ġobtained Ġfrom ĠKB s Ġsuch Ġas ĠWikipedia , ĠPro base ĠB IB REF 1 Ġ, Ġand Ġothers . ĠOnce Ġthe Ġconcept Ġvectors Ġare Ġgenerated , Ġsimilarity Ġbetween Ġtwo Ġconcept Ġvectors Ġcan Ġbe Ġcomputed Ġusing Ġa Ġsuitable Ġsimilarity / distance Ġmeasure Ġsuch Ġas Ġcos ine . </s> The ĠB OC Ġrepresentation Ġhas Ġproven Ġefficacy Ġfor Ġsemantic Ġanalysis Ġof Ġtextual Ġdata Ġespecially Ġshort Ġtexts Ġwhere Ġcontextual Ġinformation Ġis Ġmissing Ġor Ġinsufficient . ĠFor Ġexample , Ġmeasuring Ġsemantic Ġsimilarity / related ness ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġdat al ess Ġclassification ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġshort Ġtext Ġclust ering ĠB IB REF 0 Ġ, Ġsearch Ġand Ġrele v ancy Ġranking ĠB IB REF 9 Ġ, Ġevent Ġdetection Ġand Ġcore ference Ġresolution ĠB IB REF 10 Ġ. </s> Similar Ġto Ġthe Ġtraditional Ġbag - of - words Ġrepresentation , Ġthe ĠB OC Ġvector Ġis Ġa Ġhigh Ġdimensional Ġsparse Ġvector Ġwhose Ġdimension ality Ġis Ġthe Ġsame Ġas Ġthe Ġnumber Ġof Ġconcepts Ġin Ġthe Ġemployed ĠKB Ġ( typically Ġmillions ). ĠConsequently , Ġit Ġsuffers Ġfrom Ġdata Ġsp arsity Ġcausing Ġlow Ġsimilarity Ġscores Ġbetween Ġsimilar Ġtexts Ġdue Ġto Ġlow Ġconcept Ġoverlap . ĠForm ally , Ġgiven Ġa Ġtext Ġsnippet ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġterms Ġwhere ĠIN LINE FORM 2 Ġ, Ġand Ġa Ġconcept Ġspace ĠIN LINE FORM 3 Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠThe ĠB OC Ġvector ĠIN LINE FORM 5 Ġ= ĠIN LINE FORM 6 Ġof ĠIN LINE FORM 7 Ġis Ġa Ġvector Ġof Ġweights Ġof Ġeach Ġconcept Ġwhere Ġeach ĠIN LINE FORM 8 Ġof Ġconcept ĠIN LINE FORM 9 Ġis Ġcalculated Ġas Ġin Ġequation Ġ1 : ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġa Ġscoring Ġfunction Ġwhich Ġindicates Ġthe Ġdegree Ġof Ġassociation Ġbetween Ġterm ĠIN LINE FORM 1 Ġand Ġconcept ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , Ġg ab ril ovich 2007 com puting Ġproposed ĠExplicit ĠSem antic ĠAnalysis Ġ( ESA ) Ġwhich Ġuses ĠWikipedia Ġarticles Ġas Ġconcepts Ġand Ġthe ĠTF - ID F Ġscore Ġof Ġthe Ġterms Ġin Ġthese Ġarticle Ġas Ġthe Ġassociation Ġscore . ĠAnother Ġscoring Ġfunction Ġmight Ġbe Ġthe Ġco - occ urrence Ġcount Ġor ĠPearson Ġcorrelation Ġscore Ġbetween ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. ĠAs Ġwe Ġcan Ġnotice , Ġonly Ġvery Ġsmall Ġsubset Ġof Ġthe Ġconcept Ġspace Ġwould Ġhave Ġnon - zero Ġscores Ġwith Ġthe Ġgiven Ġterms . ĠMoreover , Ġthe ĠB OC Ġvector Ġis Ġgenerated Ġfrom Ġthe ĠIN LINE FORM 5 Ġconcepts Ġwhich Ġhave Ġrelatively Ġhigh Ġassociation Ġscores Ġwith Ġthe Ġinput Ġterms Ġ( typically Ġfew Ġhundreds ). ĠThus Ġeach Ġtext Ġsnippet Ġis Ġmapped Ġto Ġa Ġvery Ġsparse Ġvector Ġof Ġmillions Ġof Ġdimensions Ġhaving Ġonly Ġfew Ġhundreds Ġnon - zero Ġvalues ĠB IB REF 10 Ġ. </s> Typically , Ġthe Ġcos ine Ġsimilarity Ġmeasure Ġis Ġused Ġcompute Ġthe Ġsimilarity Ġbetween Ġa Ġpair Ġof ĠB OC Ġvectors ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠBecause Ġthe Ġconcept Ġvectors Ġare Ġvery Ġsparse , Ġwe Ġcan Ġrewrite Ġeach Ġvector Ġas Ġa Ġvector Ġof Ġtu ples ĠIN LINE FORM 2 Ġ. ĠSuppose Ġthat ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġthe Ġcorresponding Ġweights Ġof Ġconcepts ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġrespectively . ĠAnd ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġare Ġthe Ġindices Ġof Ġthese Ġconcepts Ġin Ġthe Ġconcept Ġspace ĠIN LINE FORM 11 Ġsuch Ġthat ĠIN LINE FORM 12 Ġ. ĠThen , Ġthe Ġrelated ness Ġscore Ġcan Ġbe Ġwritten Ġas Ġin Ġequation Ġ2 : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġindicator Ġfunction Ġwhich Ġreturns Ġ1 Ġif ĠIN LINE FORM 1 Ġand Ġ0 Ġotherwise . ĠHaving Ġsuch Ġsparse Ġrepresentation Ġand Ġusing Ġexact Ġmatch Ġsimilarity Ġscoring Ġmeasure , Ġwe Ġcan Ġexpect Ġthat Ġtwo Ġvery Ġsimilar Ġtext Ġsnippets Ġmight Ġhave Ġzero Ġsimilarity Ġscore Ġif Ġthey Ġmap Ġto Ġdifferent Ġbut Ġvery Ġrelated Ġset Ġof Ġconcepts ĠB IB REF 7 Ġ. </s> Ne ural Ġembed ding Ġmodels Ġhave Ġbeen Ġproposed Ġto Ġovercome Ġthe ĠB OC Ġsp arsity Ġproblem . ĠThe Ġbasic Ġidea Ġis Ġto Ġlearn Ġfixed Ġsize Ġcontinuous Ġvectors Ġfor Ġeach Ġconcept . ĠThese Ġvectors Ġcan Ġthen Ġbe Ġused Ġto Ġcompute Ġconcept - concept Ġsimilarity Ġand Ġthus Ġovercome Ġthe Ġconcept Ġmismatch Ġproblem . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġtwo Ġneural Ġembed ding Ġmodels Ġin Ġorder Ġto Ġlearn Ġcontinuous Ġconcept Ġvectors Ġbased Ġon Ġthe Ġskip - gram Ġmodel ĠB IB REF 11 Ġ. ĠOur Ġfirst Ġmodel Ġis Ġthe ĠConcept ĠRaw ĠContext Ġmodel Ġ( C RC ) Ġwhich Ġutilizes Ġconcept Ġmentions Ġin Ġa Ġlarge Ġscale ĠKB Ġto Ġjointly Ġlearn Ġembed d ings Ġof Ġboth Ġwords Ġand Ġconcepts . ĠOur Ġsecond Ġmodel Ġis Ġthe ĠConcept - Con cept ĠContext Ġmodel Ġ( 3 C ) Ġwhich Ġlearns Ġthe Ġembed d ings Ġof Ġconcepts Ġfrom Ġtheir Ġconceptual Ġcontexts Ġ( i . e ., Ġcontexts Ġcontaining Ġsurrounding Ġconcepts Ġonly ). ĠAfter Ġlearning Ġthe Ġconcept Ġvectors , Ġwe Ġpropose Ġan Ġefficient Ġconcept Ġvector Ġaggregation Ġmethod Ġto Ġgenerate Ġfully Ġdense ĠB OC Ġrepresentations . ĠOur Ġefficient Ġaggregation Ġmethod Ġallows Ġmeasuring Ġthe Ġsimilarity Ġbetween Ġpairs Ġof ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠThis Ġis Ġmore Ġefficient Ġthan Ġprior Ġmethods Ġwhich Ġrequire Ġquad r atic Ġtime Ġor Ġat Ġleast Ġlog - linear Ġtime Ġif Ġoptimized Ġ( see Ġequation Ġ2 ). </s> We Ġevaluate Ġour Ġembed ding Ġmodels Ġon Ġtwo Ġtasks : </s> The Ġcontributions Ġof Ġthis Ġpaper Ġare Ġthree fold : ĠFirst , Ġwe Ġpropose Ġtwo Ġlow Ġcost Ġconcept Ġembed ding Ġmodels Ġwhich Ġrequires Ġfew Ġhours Ġrather Ġthan Ġdays Ġto Ġtrain . ĠSecond , Ġwe Ġpropose Ġsimple Ġand Ġefficient Ġvector Ġaggregation Ġmethod Ġto Ġobtain Ġfully Ġdens ified ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠThird , Ġwe Ġdemonstrate Ġthrough Ġexperiments Ġthat Ġwe Ġcan Ġobtain Ġsame Ġor Ġbetter Ġaccuracy Ġusing Ġthe Ġdens ified ĠB OC Ġrepresentation Ġwith Ġmuch Ġless Ġdimensions Ġ( few Ġin Ġmost Ġcases ), Ġreducing Ġthe Ġcomputational Ġcost Ġof Ġgenerating Ġthe ĠB OC Ġvector Ġsignificantly . </s> Related ĠWork </s> Con cept / Entity ĠEmb edd ings : Ġneural Ġembed ding Ġmodels Ġhave Ġbeen Ġproposed Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġconcepts / ent ities . Ġsong un super vised Ġproposed Ġusing Ġthe Ġpopular ĠWord 2 V ec Ġmodel ĠB IB REF 12 Ġto Ġobtain Ġthe Ġembed d ings Ġof Ġeach Ġconcept Ġby Ġaveraging Ġthe Ġvectors Ġof Ġthe Ġconcept 's Ġindividual Ġwords . ĠFor Ġexample , Ġthe Ġembed d ings Ġof ĠMicrosoft ĠOffice Ġwould Ġbe Ġobtained Ġby Ġaveraging Ġthe Ġembed d ings Ġof ĠMicrosoft Ġand ĠOffice Ġobtained Ġfrom Ġthe ĠWord 2 V ec Ġmodel . ĠClearly , Ġthis Ġmethod Ġdisreg ards Ġthe Ġfact Ġthat Ġthe Ġsemantics Ġof Ġmulti - word Ġconcepts Ġis Ġdifferent Ġfrom Ġthe Ġsemantics Ġof Ġtheir Ġindividual Ġwords . ĠMore Ġrobust Ġconcept Ġembed d ings Ġcan Ġbe Ġlearned Ġfrom Ġthe Ġconcept 's Ġcorresponding Ġarticle Ġand / or Ġfrom Ġthe Ġstructure Ġof Ġthe Ġemployed ĠKB Ġ( e . g ., Ġits Ġlink Ġgraph ). ĠSuch Ġconcept Ġembed ding Ġmodels Ġwere Ġproposed Ġby Ġh u 2015 entity , li 2016 j oint , y am ada 2016 j oint Ġwho Ġall Ġutilize Ġthe Ġskip - gram Ġmodel ĠB IB REF 11 Ġ, Ġbut Ġdiffer Ġin Ġhow Ġthey Ġdefine Ġthe Ġcontext Ġof Ġthe Ġtarget Ġconcept . </s> li 2016 j oint Ġextended Ġthe Ġembed ding Ġmodel Ġproposed Ġby Ġh u 2015 entity Ġby Ġjointly Ġlearning Ġconcept Ġand Ġcategory Ġembed d ings Ġfrom Ġcontexts Ġdefined Ġby Ġall Ġother Ġconcepts Ġin Ġthe Ġtarget Ġconcept 's Ġarticle Ġas Ġwell Ġas Ġits Ġcategory Ġhierarchy Ġin ĠWikipedia . ĠThis Ġmethod Ġhas Ġthe Ġadvantage Ġof Ġlearning Ġembed d ings Ġof Ġboth Ġconcepts Ġand Ġcategories Ġsimultaneously . ĠHowever , Ġdefining Ġthe Ġconcept Ġcontexts Ġas Ġpairs Ġof Ġthe Ġtarget Ġconcept Ġand Ġall Ġother Ġconcepts Ġappearing Ġin Ġits Ġcorresponding Ġarticle Ġmight Ġintroduce Ġnoisy Ġcontexts , Ġespecially Ġfor Ġlong Ġarticles . ĠFor Ġexample , Ġthe ĠWikipedia Ġarticle Ġfor ĠUnited ĠStates Ġcontains Ġlinks Ġto ĠKinder g arten , ĠFirst Ġgrade , Ġand ĠSecondary Ġschool Ġunder Ġthe ĠEducation Ġsection . </s> y am ada 2016 j oint Ġproposed Ġa Ġmethod Ġbased Ġon Ġthe Ġskip - gram Ġmodel Ġto Ġjointly Ġlearn Ġembed d ings Ġof Ġwords Ġand Ġconcepts Ġusing Ġcontexts Ġgenerated Ġfrom Ġsurrounding Ġwords Ġof Ġthe Ġtarget Ġconcept Ġor Ġword . ĠThe Ġauthors Ġalso Ġproposed Ġincorporating Ġthe ĠKB Ġlink Ġgraph Ġby Ġgenerating Ġcontexts Ġfrom Ġall Ġconcepts Ġwith Ġoutgoing Ġlink Ġto Ġthe Ġtarget Ġconcept Ġto Ġbetter Ġmodel Ġconcept - concept Ġrelated ness . </s> Unlike Ġli 2016 j oint Ġand Ġh u 2015 entity Ġwho Ġlearn Ġconcept Ġembed d ings Ġonly , Ġour ĠCRC Ġmodel Ġ( described Ġin ĠSection Ġ3 ), Ġmaps Ġboth Ġwords Ġand Ġconcepts Ġinto Ġthe Ġsame Ġsemantic Ġspace . ĠTherefore Ġwe Ġcan Ġeasily Ġmeasure Ġword - word , Ġword - concept , Ġand Ġconcept - concept Ġsemantic Ġsimilarities . ĠIn Ġaddition , Ġcompared Ġto Ġy am ada 2016 j oint Ġmodel , Ġwe Ġutilize Ġcontexts Ġgenerated Ġfrom Ġboth Ġsurrounding Ġwords Ġand Ġconcepts . ĠTherefore , Ġwe Ġcan Ġbetter Ġcapture Ġlocal Ġcontextual Ġinformation Ġof Ġeach Ġtarget Ġword / concept . ĠMoreover , Ġour Ġproposed Ġmodels Ġare Ġcomput ationally Ġless Ġcostly Ġthan Ġh u 2015 entity Ġand Ġy am ada 2016 j oint Ġmodels Ġas Ġthey Ġrequire Ġfew Ġhours Ġrather Ġthan Ġdays Ġto Ġtrain Ġon Ġsimilar Ġcomputing Ġresources . </s> B OC ĠD ens ification : Ġdistributed Ġconcept Ġvectors Ġhave Ġbeen Ġused Ġby ĠB OC Ġdens ification Ġmechanisms Ġto Ġovercome Ġthe ĠB OC Ġsp arsity Ġproblem . Ġsong un super vised Ġproposed Ġthree Ġdifferent Ġmechanisms Ġfor Ġalign ing Ġthe Ġconcepts Ġat Ġdifferent Ġindices Ġgiven Ġa Ġsparse ĠB OC Ġpair Ġ( ĠIN LINE FORM 0 Ġ) Ġin Ġorder Ġto Ġincrease Ġtheir Ġsimilarity Ġscore . </s> The Ġmany - to - many Ġmechanism Ġworks Ġby Ġaveraging Ġall Ġpair wise Ġsimilarities . ĠThe Ġmany - to - one Ġmechanism Ġworks Ġby Ġalign ing Ġeach Ġconcept Ġin ĠIN LINE FORM 0 Ġwith Ġthe Ġmost Ġsimilar Ġconcept Ġin ĠIN LINE FORM 1 Ġ( i . e ., Ġits Ġbest Ġmatch ). ĠClearly , Ġthe Ġcomplexity Ġof Ġthese Ġtwo Ġmechanisms Ġis Ġquad r atic . ĠThe Ġthird Ġmechanism Ġis Ġthe Ġone - to - one . ĠIt Ġutilizes Ġthe ĠHungarian Ġmethod Ġin Ġorder Ġto Ġfind Ġan Ġoptimal Ġalignment Ġon Ġa Ġone - to - one Ġbasis ĠB IB REF 13 Ġ. ĠThis Ġmechanism Ġperformed Ġthe Ġbest Ġon Ġdat al ess Ġclassification Ġand Ġwas Ġalso Ġutilized Ġby Ġli 2016 j oint . ĠHowever , Ġthe ĠHungarian Ġmethod Ġis Ġa Ġcomb inator ial Ġoptimization Ġalgorithm Ġwhose Ġcomplexity Ġis Ġpo lyn omial . ĠOur Ġproposed Ġdens ification Ġmechanism Ġis Ġmore Ġefficient Ġthan Ġthese Ġthree Ġmechanisms Ġas Ġits Ġcomplexity Ġis Ġlinear Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġnon - zero Ġelements Ġin Ġthe ĠB OC Ġvector . ĠAdditionally , Ġit Ġis Ġsimpler Ġas Ġit Ġdoes Ġnot Ġrequire Ġtuning Ġa Ġcut Ġoff Ġthreshold Ġfor Ġthe Ġminimum Ġsimilarity Ġbetween Ġtwo Ġaligned Ġconcepts Ġas Ġin Ġprior Ġwork . </s> Con cept ĠEmb edd ings Ġfor ĠB OC ĠD ens ification </s> A Ġmain Ġobjective Ġof Ġlearning Ġconcept Ġembed d ings Ġis Ġto Ġovercome Ġthe Ġinherent Ġproblem Ġof Ġdata Ġsp arsity Ġassociated Ġwith Ġthe ĠB OC Ġrepresentation . ĠHere Ġwe Ġtry Ġto Ġlearn Ġcontinuous Ġconcept Ġvectors Ġby Ġbuilding Ġupon Ġthe Ġskip - gram Ġembed ding Ġmodel ĠB IB REF 11 Ġ. ĠIn Ġthe Ġconventional Ġskip - gram Ġmodel , Ġa Ġset Ġof Ġcontexts Ġare Ġgenerated Ġby Ġsliding Ġa Ġcontext Ġwindow Ġof Ġpred efined Ġsize Ġover Ġsentences Ġof Ġa Ġgiven Ġtext Ġcorpus . ĠVector Ġrepresentation Ġof Ġa Ġtarget Ġword Ġis Ġlearned Ġwith Ġthe Ġobjective Ġto Ġmaximize Ġthe Ġability Ġof Ġpredicting Ġsurrounding Ġwords Ġof Ġthat Ġtarget Ġword . </s> Form ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠThe Ġskip - gram Ġmodel Ġaims Ġto Ġmaximize Ġthe Ġaverage Ġlog Ġprobability : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġword , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġcontext Ġword . ĠThe Ġsoft max Ġfunction Ġis Ġused Ġto Ġestimate Ġthe Ġprobability ĠIN LINE FORM 3 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġinput Ġand Ġoutput Ġvectors Ġrespectively Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġvocabulary Ġsize . Ġm ik ol ov 2013 dist ributed Ġproposed Ġhierarchical Ġsoft max Ġand Ġnegative Ġsampling Ġas Ġefficient Ġalternatives Ġto Ġapproximate Ġthe Ġsoft max Ġfunction Ġwhich Ġbecomes Ġcomput ationally Ġint ract able Ġwhen ĠIN LINE FORM 3 Ġbecomes Ġhuge . </s> Our Ġapproach Ġgenuinely Ġlearns Ġdistributed Ġconcept Ġrepresentations Ġby Ġgenerating Ġconcept Ġcontexts Ġfrom Ġmentions Ġof Ġthose Ġconcepts Ġin Ġlarge Ġen cycl oped ic ĠKB s Ġsuch Ġas ĠWikipedia . ĠUt il izing Ġsuch Ġannot ated ĠKB s Ġeliminates Ġthe Ġneed Ġto Ġmanually Ġannot ate Ġconcept Ġmentions Ġand Ġthus Ġcomes Ġat Ġno Ġcost . </s> Con cept ĠRaw ĠContext ĠModel Ġ( C RC ) </s> In Ġthis Ġmodel , Ġwe Ġjointly Ġlearn Ġthe Ġembed d ings Ġof Ġboth Ġwords Ġand Ġconcepts . ĠFirst , Ġall Ġconcept Ġmentions Ġare Ġidentified Ġin Ġthe Ġgiven Ġcorpus . ĠSecond , Ġcontexts Ġare Ġgenerated Ġfor Ġboth Ġwords Ġand Ġconcepts Ġfrom Ġboth Ġother Ġsurrounding Ġwords Ġand Ġother Ġsurrounding Ġconcepts Ġas Ġwell . ĠAfter Ġgenerating Ġall Ġthe Ġcontexts , Ġwe Ġuse Ġthe Ġskip - gram Ġmodel Ġto Ġjointly Ġlearn Ġwords Ġand Ġconcepts Ġembed d ings . ĠForm ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠWe Ġiter ate Ġover Ġthe Ġcorpus Ġidentifying Ġwords Ġand Ġconcept Ġmentions Ġand Ġthus Ġgenerating Ġa Ġsequence Ġof ĠIN LINE FORM 2 Ġtokens ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ( as Ġmulti - word Ġconcepts Ġwill Ġbe Ġcounted Ġas Ġone Ġtoken ). ĠAfterwards Ġwe Ġtrain Ġthe Ġa Ġskip - gram Ġmodel Ġaiming Ġto Ġmaximize : ĠDIS PLAY FORM 0 </s> where Ġas Ġin Ġthe Ġconventional Ġskip - gram Ġmodel , ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize . ĠIn Ġthis Ġmodel , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġtoken Ġwhich Ġwould Ġbe Ġeither Ġa Ġword Ġor Ġa Ġconcept Ġmention , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġcontext Ġword Ġor Ġconcept Ġmention . </s> This Ġmodel Ġis Ġdifferent Ġfrom Ġy am ada 2016 j oint Ġanchor Ġcontext Ġmodel Ġin Ġthree Ġaspects : Ġ1 ) Ġwhile Ġgenerating Ġtarget Ġconcept Ġcontexts , Ġwe Ġutilize Ġnot Ġonly Ġsurrounding Ġwords Ġbut Ġother Ġsurrounding Ġconcepts Ġas Ġwell , Ġ2 ) Ġour Ġmodel Ġaims Ġto Ġmaximize ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġcould Ġbe Ġa Ġword Ġor Ġa Ġconcept , Ġwhile Ġy am ada 2016 j oint Ġmodel Ġmaxim izes ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġconcept / entity Ġ( see Ġy am ada 2016 j oint ĠE q . Ġ6 ), Ġand Ġ3 ) Ġin Ġcase ĠIN LINE FORM 4 Ġis Ġa Ġconcept , Ġour Ġmodel Ġcaptures Ġall Ġthe Ġcontexts Ġin Ġwhich Ġit Ġappeared , Ġwhile Ġy am ada 2016 j oint Ġmodel Ġgenerates Ġfor Ġeach Ġentity Ġone Ġcontext Ġof ĠIN LINE FORM 5 Ġprevious Ġand ĠIN LINE FORM 6 Ġnext Ġwords . ĠWe Ġhypothes ize Ġthat Ġconsidering Ġboth Ġconcepts Ġand Ġindividual Ġwords Ġin Ġthe Ġoptimization Ġfunction Ġwould Ġgenerate Ġmore Ġrobust Ġembed d ings . </s> Con cept - Con cept ĠContext ĠModel Ġ( 3 C ) </s> Insp ired Ġby Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 14 Ġ, Ġwe , Ġin Ġthis Ġmodel , Ġhypothes ize Ġthat Ġ" similar Ġconcepts Ġtend Ġto Ġappear Ġin Ġsimilar Ġconceptual Ġcontexts ". ĠIn Ġorder Ġto Ġtest Ġthis Ġhypothesis , Ġwe Ġlearn Ġconcept Ġembed d ings Ġby Ġtraining Ġa Ġskip - gram Ġmodel Ġon Ġcontexts Ġgenerated Ġsolely Ġfrom Ġconcept Ġmentions . ĠAs Ġin Ġthe ĠCRC Ġmodel , Ġwe Ġstart Ġby Ġidentifying Ġall Ġconcept Ġmentions Ġin Ġthe Ġgiven Ġcorpus . ĠThen , Ġcontexts Ġare Ġgenerated Ġfrom Ġonly Ġsurrounding Ġconcepts . ĠForm ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠWe Ġiter ate Ġover Ġthe Ġcorpus Ġidentifying Ġconcept Ġmentions Ġand Ġthus Ġgenerating Ġa Ġsequence Ġof ĠIN LINE FORM 2 Ġconcept Ġtokens ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ. ĠAfterwards Ġwe Ġtrain Ġthe Ġskip - gram Ġmodel Ġaiming Ġto Ġmaximize : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġconcept , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġconcept Ġmention Ġwithin ĠIN LINE FORM 3 Ġmentions . </s> This Ġmodel Ġis Ġdifferent Ġfrom Ġli 2016 j oint Ġand Ġh u 2015 entity Ġas Ġthey Ġdefine Ġthe Ġcontext Ġof Ġa Ġtarget Ġconcept Ġby Ġall Ġthe Ġother Ġconcepts Ġwhich Ġappear Ġin Ġthe Ġconcept 's Ġcorresponding Ġarticle . ĠClearly , Ġsome Ġof Ġthese Ġconcepts Ġmight Ġbe Ġirrelevant Ġespecially Ġfor Ġvery Ġlong Ġarticles Ġwhich Ġcite Ġhundreds Ġof Ġother Ġconcepts . ĠOur Ġ3 C Ġmodel , Ġalternatively , Ġlearns Ġconcept Ġsemantics Ġfrom Ġsurrounding Ġconcepts Ġand Ġnot Ġonly Ġfrom Ġthose Ġthat Ġare Ġcited Ġin Ġits Ġarticle . ĠWe Ġalso Ġextend Ġthe Ġcontext Ġwindow Ġbeyond Ġpairs Ġof Ġconcepts Ġallowing Ġmore Ġinfluence Ġto Ġother Ġnearby Ġconcepts . </s> The Ġmain Ġadvantage Ġof Ġthe Ġ3 C Ġmodel Ġover Ġthe ĠCRC Ġmodel Ġis Ġits Ġcomputational Ġefficiency Ġwhere Ġthe Ġmodel Ġvocabulary Ġis Ġlimited Ġto Ġthe Ġcorpus Ġconcepts Ġ( Wikipedia Ġin Ġour Ġcase ). ĠOne Ġthe Ġother Ġhand , Ġthe ĠCRC Ġmodel Ġis Ġadvantageous Ġbecause Ġit Ġjointly Ġlearns Ġthe Ġembed d ings Ġof Ġwords Ġand Ġconcepts Ġand Ġis Ġtherefore Ġexpected Ġto Ġgenerate Ġhigher Ġquality Ġvectors . ĠIn Ġother Ġwords , Ġthe ĠCRC Ġmodel Ġcan Ġcapture Ġmore Ġcontextual Ġsignals Ġsuch Ġas Ġactions , Ġtimes , Ġand Ġrelationships Ġat Ġthe Ġexpense Ġof Ġtraining Ġcomputational Ġcost . </s> Training </s> We Ġutilize Ġa Ġrecent ĠWikipedia Ġdump Ġof ĠAugust Ġ2016 , Ġwhich Ġhas Ġabout Ġ7 Ġmillion Ġarticles . ĠWe Ġextract Ġarticles Ġplain Ġtext Ġdisc arding Ġimages Ġand Ġtables . ĠWe Ġalso Ġdiscard ĠReferences Ġand ĠExternal Ġlinks Ġsections Ġ( if Ġany ). ĠWe Ġpr uned Ġboth Ġarticles Ġnot Ġunder Ġthe Ġmain Ġnamespace Ġand Ġpr uned Ġall Ġredirect Ġpages Ġas Ġwell . ĠEventually , Ġour Ġcorpus Ġcontained Ġabout Ġ5 Ġmillion Ġarticles Ġin Ġtotal . </s> We Ġpre process Ġeach Ġarticle Ġreplacing Ġall Ġits Ġreferences Ġto Ġother ĠWikipedia Ġarticles Ġwith Ġthe Ġtheir Ġcorresponding Ġpage ĠIDs . ĠIn Ġcase Ġany Ġof Ġthe Ġreferences Ġis Ġa Ġtitle Ġof Ġa Ġredirect Ġpage , Ġwe Ġuse Ġthe Ġpage ĠID Ġof Ġthe Ġoriginal Ġpage Ġto Ġensure Ġthat Ġall Ġconcept Ġmentions Ġare Ġnormalized . </s> Following Ġm ik ol ov 2013 dist ributed , Ġwe Ġutilize Ġnegative Ġsampling Ġto Ġapproximate Ġthe Ġsoft max Ġfunction Ġby Ġreplacing Ġevery ĠIN LINE FORM 0 Ġterm Ġin Ġthe Ġsoft max Ġfunction Ġ( equ ation Ġ4 ) Ġwith : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġnegative Ġsamples Ġdrawn Ġfor Ġeach Ġword Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġs igm oid Ġfunction Ġ( ĠIN LINE FORM 2 Ġ). ĠIn Ġthe Ġcase Ġof Ġthe ĠCRC Ġmodel ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġwould Ġbe Ġreplaced Ġwith ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġrespectively . ĠAnd Ġin Ġthe Ġcase Ġof Ġthe Ġ3 C Ġmodel ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġwould Ġbe Ġreplaced Ġwith ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġrespectively . </s> For Ġboth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġwith Ġuse Ġa Ġcontext Ġwindow Ġof Ġsize Ġ9 Ġand Ġa Ġvector Ġof Ġ500 Ġdimensions . ĠWe Ġtrain Ġthe Ġskip - gram Ġmodel Ġfor Ġ10 Ġiterations Ġusing Ġ12 Ġcores Ġmachine Ġwith Ġ64 GB Ġof ĠRAM . ĠThe ĠCRC Ġmodel Ġtook ĠIN LINE FORM 0 Ġ15 Ġhours Ġto Ġtrain Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 1 Ġ12 . 7 Ġmillion Ġtokens . ĠThe Ġ3 C Ġmodel Ġtook ĠIN LINE FORM 2 Ġ1 . 5 Ġhours Ġto Ġtrain Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 3 Ġ4 . 5 Ġmillion Ġconcepts . </s> B OC ĠD ens ification </s> As Ġwe Ġmentioned Ġin Ġthe Ġrelated Ġwork Ġsection , Ġthe Ġcurrent Ġmechanisms Ġfor ĠB OC Ġdens ification Ġare Ġinefficient Ġas Ġtheir Ġcomplexity Ġis Ġleast Ġquad r atic Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġnon - zero Ġelements Ġin Ġthe ĠB OC Ġvector . ĠHere , Ġwe Ġpropose Ġsimple Ġand Ġefficient Ġvector Ġaggregation Ġmethod Ġto Ġobtain Ġfully Ġdens ified ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠOur Ġmechanism Ġworks Ġby Ġperforming Ġa Ġweighted Ġaverage Ġof Ġthe Ġembed ding Ġvectors Ġof Ġall Ġconcepts Ġin Ġthe Ġgiven ĠB OC . ĠThis Ġoperation Ġscales Ġlin early Ġwith Ġthe Ġnumber Ġof Ġnon - zero Ġdimensions Ġin Ġthe ĠB OC Ġvector . ĠIn Ġaddition , Ġit Ġproduces Ġa Ġfully Ġdense Ġvector Ġrepresenting Ġthe Ġsemantics Ġof Ġthe Ġoriginal Ġconcepts Ġand Ġconsidering Ġtheir Ġweights . ĠForm ally , Ġgiven Ġa Ġsparse ĠB OC Ġvector ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġweight Ġof Ġconcept ĠIN LINE FORM 2 Ġ. ĠWe Ġcan Ġobtain Ġthe Ġdense Ġrepresentation Ġof ĠIN LINE FORM 3 Ġas Ġin Ġequation Ġ8 : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġembed ding Ġvector Ġof Ġconcept ĠIN LINE FORM 1 Ġ. ĠOnce Ġwe Ġhave Ġthis Ġdense ĠB OC Ġvector , Ġwe Ġcan Ġapply Ġthe Ġcos ine Ġmeasure Ġto Ġcompute Ġthe Ġsimilarity Ġbetween Ġa Ġpair Ġof Ġdense ĠB OC Ġvectors . </s> As Ġwe Ġcan Ġnotice , Ġthis Ġweighted Ġaverage Ġis Ġdone Ġonce Ġand Ġfor Ġall Ġfor Ġa Ġgiven ĠB OC Ġvector . ĠOther Ġmechanisms Ġthat Ġrely Ġon Ġconcept Ġalignment ĠB IB REF 7 Ġ, Ġrequire Ġreal ignment Ġevery Ġtime Ġa Ġgiven ĠB OC Ġvector Ġis Ġcompared Ġto Ġanother ĠB OC Ġvector . ĠOur Ġapproach Ġimproves Ġthe Ġefficiency Ġespecially Ġin Ġthe Ġcontext Ġof Ġdat al ess Ġclassification Ġwith Ġlarge Ġnumber Ġof Ġclasses . ĠUsing Ġour Ġdens ification Ġmechanism , Ġwe Ġapply Ġthe Ġweighted Ġaverage Ġfor Ġeach Ġclass Ġvector Ġand Ġfor Ġeach Ġinstance Ġonce . </s> Interestingly , Ġour Ġdens ification Ġmechanism Ġallows Ġus Ġto Ġdens ify Ġthe Ġsparse ĠB OC Ġvector Ġusing Ġonly Ġthe Ġtop Ġfew Ġdimensions . ĠAs Ġwe Ġwill Ġshow Ġin Ġthe Ġexperiments Ġsection , Ġwe Ġcan Ġget Ġ( near -) best Ġresults Ġusing Ġthese Ġfew Ġdimensions Ġcompared Ġto Ġdens ifying Ġwith Ġall Ġthe Ġdimensions Ġin Ġthe Ġoriginal Ġsparse Ġvector . ĠThis Ġproperty Ġreduces Ġthe Ġcost Ġof Ġobtaining Ġa ĠB OC Ġvector Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġin Ġthe Ġfirst Ġplace . </s> Entity ĠSem antic ĠRelated ness </s> We Ġevaluate Ġthe Ġ" good ness " Ġof Ġour Ġconcept Ġembed d ings Ġon Ġmeasuring Ġentity Ġsemantic Ġrelated ness Ġas Ġan Ġintrinsic Ġevaluation . ĠEntity Ġrelated ness Ġhas Ġbeen Ġrecently Ġused Ġto Ġmodel Ġentity Ġco herence Ġin Ġmany Ġnamed Ġentity Ġdis amb ig uation Ġsystems . </s> We Ġuse Ġa Ġbenchmark Ġdataset Ġcreated Ġby Ġce cc are lli 2013 learning Ġfrom Ġthe ĠCo N LL Ġ2003 Ġdata . ĠAs Ġin Ġprevious Ġstudies ĠB IB REF 16 Ġ, ĠB IB REF 15 Ġ, Ġwe Ġmodel Ġmeasuring Ġentity Ġrelated ness Ġas Ġa Ġranking Ġproblem . ĠWe Ġuse Ġthe Ġtest Ġsplit Ġof Ġthe Ġdataset Ġto Ġcreate Ġ3 , 314 Ġqueries . ĠEach Ġquery Ġhas Ġa Ġquery Ġentity Ġand ĠIN LINE FORM 0 Ġ91 Ġresponse Ġentities Ġlabeled Ġas Ġrelated Ġor Ġunrelated . ĠThe Ġquality Ġis Ġmeasured Ġby Ġthe Ġability Ġof Ġthe Ġsystem Ġto Ġrank Ġrelated Ġentities Ġon Ġtop Ġof Ġunrelated Ġones . </s> We Ġcompare Ġour Ġmodels Ġwith Ġtwo Ġprior Ġmethods : </s> y am ada 2016 j oint Ġwho Ġused Ġthe Ġskip - gram Ġmodel Ġto Ġlearn Ġembed d ings Ġof Ġwords Ġand Ġentities Ġjointly . ĠThe Ġauthors Ġalso Ġutilized ĠWikipedia Ġlink Ġgraph Ġto Ġbetter Ġmodel Ġentity - entity Ġrelated ness . </s> w itten 2008 effective Ġwho Ġproposed ĠWikipedia ĠLink - based ĠMeasure Ġ( W LM ) Ġas Ġa Ġsimple Ġmechanism Ġfor Ġmodeling Ġthe Ġsemantic Ġrelated ness Ġbetween ĠWikipedia Ġconcepts . ĠThe Ġauthors Ġutilized ĠWikipedia Ġlink Ġstructure Ġunder Ġthe Ġassumption Ġthat Ġrelated Ġconcepts Ġwould Ġhave Ġsimilar Ġincoming Ġlinks . </s> We Ġreport ĠMean ĠAverage ĠPrecision Ġ( MAP ) Ġand ĠNormal ized ĠDiscount ed ĠCum ulative ĠGain Ġ( n DC G ) Ġscores Ġas Ġcommonly Ġused Ġmeasures Ġfor Ġevaluating Ġthe Ġranking Ġquality . ĠTable Ġ1 Ġshows Ġthe Ġperformance Ġof Ġour ĠCRC Ġand Ġ3 C Ġmodels Ġcompared Ġto Ġprevious Ġmodels . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġ3 C Ġmodel Ġperforms Ġpoorly Ġon Ġthis Ġtask Ġcompared Ġto Ġprior Ġmodels . ĠOn Ġthe Ġother Ġhand , Ġour ĠCRC Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġmethods Ġby Ġ2 - 4 % Ġin Ġterms Ġof Ġn DC G Ġand Ġby Ġ3 % Ġpercent Ġin Ġterms Ġof ĠMAP . </s> D atal ess ĠClassification </s> ch ang 2008 import ance Ġproposed Ġdat al ess Ġclassification Ġas Ġa Ġlearning Ġprotocol Ġto Ġperform Ġtext Ġcategor ization Ġwithout Ġthe Ġneed Ġfor Ġlabeled Ġdata Ġto Ġtrain Ġa Ġclass ifier . ĠGiven Ġonly Ġlabel Ġnames Ġand Ġfew Ġdescriptive Ġkeywords Ġof Ġeach Ġlabel , Ġclassification Ġis Ġperformed Ġon Ġthe Ġfly Ġby Ġmapping Ġeach Ġlabel Ġinto Ġa ĠB OC Ġrepresentation Ġusing ĠESA . ĠLikewise , Ġeach Ġdata Ġinstance Ġis Ġmapped Ġinto Ġthe Ġsame ĠB OC Ġsemantic Ġspace Ġand Ġassigned Ġto Ġthe Ġmost Ġsimilar Ġlabel Ġusing Ġa Ġproper Ġsimilarity Ġmeasure Ġsuch Ġas Ġcos ine . ĠForm ally , Ġgiven Ġa Ġset Ġof ĠIN LINE FORM 0 Ġlabels ĠIN LINE FORM 1 Ġ, Ġa Ġtext Ġdocument ĠIN LINE FORM 2 Ġ, Ġa ĠB OC Ġmapping Ġmodel ĠIN LINE FORM 3 Ġ, Ġand Ġa Ġsimilarity Ġfunction ĠIN LINE FORM 4 Ġ, Ġthen ĠIN LINE FORM 5 Ġis Ġassigned Ġto Ġthe ĠIN LINE FORM 6 Ġth Ġlabel ĠIN LINE FORM 7 Ġsuch Ġthat ĠIN LINE FORM 8 Ġ. ĠWe Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġconcept Ġembed ding Ġmodels Ġon Ġthe Ġdat al ess Ġclassification Ġtask Ġas Ġan Ġextr ins ic Ġevaluation . ĠWe Ġdemonstrate Ġthrough Ġempirical Ġresults Ġthe Ġefficiency Ġand Ġeffectiveness Ġof Ġour Ġproposed ĠB OC Ġdens ification Ġscheme Ġin Ġobtaining Ġbetter Ġclassification Ġresults Ġcompared Ġto Ġthe Ġoriginal Ġsparse ĠB OC Ġrepresentation . </s> We Ġuse Ġthe Ġ20 ĠNews groups Ġdataset Ġ( 20 NG ) ĠB IB REF 17 Ġwhich Ġis Ġcommonly Ġused Ġfor Ġbenchmark ing Ġtext Ġclassification Ġalgorithms . ĠThe Ġdataset Ġcontains Ġ20 Ġcategories Ġeach Ġhas ĠIN LINE FORM 0 Ġ1000 Ġnews Ġposts . ĠWe Ġobtained Ġthe ĠB OC Ġrepresentations Ġusing ĠESA Ġfrom Ġsong 2014 d atal ess Ġwho Ġutilized Ġa ĠWikipedia Ġindex Ġcontaining Ġpages Ġwith Ġ100 + Ġwords Ġand Ġ5 + Ġoutgoing Ġlinks Ġto Ġcreate ĠESA Ġm appings Ġof Ġ500 Ġdimensions Ġfor Ġboth Ġthe Ġcategories Ġand Ġnews Ġposts Ġof Ġthe Ġ20 NG . ĠWe Ġdesigned Ġtwo Ġtypes Ġof Ġclassification Ġtasks : Ġ1 ) Ġfine - gr ained Ġclassification Ġinvolving Ġclosely Ġrelated Ġclasses Ġsuch Ġas ĠHockey Ġvs . ĠBaseball , ĠAut os Ġvs . ĠMotor cycles , Ġand ĠGuns Ġvs . ĠM ide ast Ġvs . ĠMisc , Ġand Ġ2 ) Ġcoarse - gr ained Ġclassification Ġinvolving Ġtop - level Ġcategories Ġsuch Ġas ĠSport Ġvs . ĠPolitics Ġand ĠSport Ġvs . ĠReligion . ĠThe Ġtop - level Ġcategories Ġare Ġcreated Ġby Ġcombining Ġinstances Ġof Ġthe Ġfine - gr ained Ġcategories Ġas Ġshown Ġin ĠTable Ġ2 . </s> We Ġcompare Ġour Ġmodels Ġwith Ġthree Ġprior Ġmethods : </s> ESA Ġwhich Ġcomp utes Ġthe Ġcos ine Ġsimilarity Ġusing Ġthe Ġsparse ĠB OC Ġvectors . </s> WE ĠIN LINE FORM 0 Ġ& ĠWE ĠIN LINE FORM 1 Ġwhich Ġwere Ġproposed Ġby Ġsong un super vised Ġfor ĠB OC Ġdens ification Ġusing Ġembed d ings Ġobtained Ġfrom ĠWord 2 V ec . ĠAs Ġthe Ġauthors Ġreported , Ġwe Ġfix Ġthe Ġminimum Ġsimilarity Ġthreshold Ġto Ġ0 . 85 . ĠWE ĠIN LINE FORM 2 Ġfinds Ġthe Ġbest Ġmatch Ġfor Ġeach Ġconcept , Ġwhile ĠWE ĠIN LINE FORM 3 Ġutilizes Ġthe ĠHungarian Ġalgorithm Ġto Ġfind Ġthe Ġbest Ġconcept - concept Ġalignment Ġon Ġone - to - one Ġbasis . ĠBoth Ġmechanisms Ġhave Ġpo lyn omial Ġtime Ġcomplexity . </s> Table Ġ3 Ġpresents Ġthe Ġresults Ġof Ġfine - gr ained Ġdat al ess Ġclassification Ġmeasured Ġin Ġmicro - aver aged ĠF 1 . ĠAs Ġwe Ġcan Ġnotice , ĠESA Ġachieves Ġits Ġpeak Ġperformance Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġof Ġthe Ġsparse ĠB OC Ġvector . ĠUsing Ġour Ġdens ification Ġmechanism , Ġboth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġachieve Ġequal Ġperformance Ġto ĠESA Ġat Ġmuch Ġless Ġdimensions . ĠD ens ification Ġusing Ġthe ĠCRC Ġmodel Ġembed d ings Ġgives Ġthe Ġbest ĠF 1 Ġscores Ġon Ġthe Ġthree Ġtasks . ĠInterestingly , Ġthe ĠCRC Ġmodel Ġimproves Ġthe ĠF 1 Ġscore Ġby ĠIN LINE FORM 0 Ġ7 % Ġusing Ġonly Ġ14 Ġconcepts Ġon ĠAut os Ġvs . ĠMotor cycles , Ġand Ġby ĠIN LINE FORM 1 Ġ3 % Ġusing Ġ70 Ġconcepts Ġon ĠGuns Ġvs . ĠM ide ast Ġvs . ĠMisc . ĠThe Ġ3 C Ġmodel , Ġstill Ġperforms Ġbetter Ġthan ĠESA Ġon Ġ2 Ġout Ġof Ġthe Ġ3 Ġtasks . ĠBoth ĠWE ĠIN LINE FORM 2 Ġand ĠWE ĠIN LINE FORM 3 Ġimprove Ġthe Ġperformance Ġover ĠESA Ġbut Ġnot Ġas Ġour ĠCRC Ġmodel . </s> In Ġorder Ġto Ġbetter Ġillustrate Ġthe Ġrobust ness Ġof Ġour Ġdens ification Ġmechanism Ġwhen Ġvarying Ġthe Ġ# Ġof ĠB OC Ġdimensions , Ġwe Ġmeasured ĠF 1 Ġscores Ġof Ġeach Ġtask Ġas Ġa Ġfunction Ġof Ġthe Ġ# Ġof ĠB OC Ġdimensions Ġused Ġfor Ġdens ification . ĠAs Ġwe Ġsee Ġin ĠFigure Ġ1 , Ġwith Ġone Ġconcept Ġwe Ġcan Ġachieve Ġhigh ĠF 1 Ġscores Ġcompared Ġto ĠESA Ġwhich Ġachieves Ġzero Ġor Ġvery Ġlow ĠF 1 . ĠMoreover , Ġnear - peak Ġperformance Ġis Ġachievable Ġwith Ġthe Ġtop Ġ50 Ġor Ġless Ġdimensions . ĠWe Ġcan Ġalso Ġnotice Ġthat , Ġas Ġwe Ġincrease Ġthe Ġ# Ġof Ġdimensions , Ġboth ĠWE ĠIN LINE FORM 0 Ġand ĠWE ĠIN LINE FORM 1 Ġdens ification Ġmethods Ġhave Ġthe Ġsame Ġundes ired Ġmon ot onic Ġpattern Ġlike ĠESA . ĠActually , Ġthe Ġimposed Ġthreshold Ġby Ġthese Ġmethods Ġdoes Ġnot Ġallow Ġfor Ġfull Ġdense Ġrepresentation Ġof Ġthe ĠB OC Ġvector Ġand Ġtherefore Ġat Ġlow Ġdimensions Ġwe Ġstill Ġsee Ġlow Ġoverall ĠF 1 Ġscore . ĠOur Ġproposed Ġdens ification Ġmechanisms Ġbesides Ġtheir Ġlow Ġcost , Ġproduce Ġfully Ġdens ified Ġrepresentations Ġallowing Ġgood Ġsimilarities Ġat Ġlow Ġdimensions . </s> Results Ġof Ġcoarse - gr ained Ġclassification Ġare Ġpresented Ġin ĠTable Ġ4 . ĠClassification Ġat Ġthe Ġtop Ġlevel Ġis Ġeasier Ġthan Ġthe Ġfine - gr ained Ġlevel . ĠNevertheless , Ġas Ġwith Ġfine - gr ained Ġclassification , ĠESA Ġstill Ġpeaks Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġof Ġthe Ġsparse ĠB OC Ġvector . ĠBoth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġachieve Ġequal Ġperformance Ġto ĠESA Ġat Ġvery Ġfew Ġdimensions Ġ( ĠIN LINE FORM 0 Ġ). ĠD ens ification Ġusing Ġthe ĠCRC Ġmodel Ġembed d ings Ġstill Ġperforms Ġthe Ġbest Ġon Ġboth Ġtasks . ĠInterestingly , Ġthe Ġ3 C Ġmodel Ġgives Ġvery Ġclose ĠF 1 Ġscores Ġto Ġthe ĠCRC Ġmodel Ġat Ġless Ġdimensions Ġ(@ 4 Ġwith ĠSport Ġvs . ĠPolitics , Ġand Ġ@ 60 Ġwith ĠSport Ġvs . ĠReligion ) Ġindicating Ġits Ġcompetitive Ġadvantage Ġwhen Ġcomputational Ġcost Ġis Ġa Ġdecisive Ġcriteria . ĠThe Ġ3 C Ġmodel , Ġstill Ġperforms Ġbetter Ġthan ĠESA , ĠWE ĠIN LINE FORM 1 Ġ, Ġand ĠWE ĠIN LINE FORM 2 Ġon Ġboth Ġtasks . </s> Figure Ġ2 Ġshows ĠF 1 Ġscores Ġof Ġcoarse - gr ained Ġclassification Ġwhen Ġvarying Ġthe Ġ# Ġof ĠB OC Ġdimensions Ġused Ġfor Ġdens ification . ĠThe Ġsame Ġpattern Ġof Ġachieving Ġnear - peak Ġperformance Ġat Ġvery Ġfew Ġdimensions Ġrec ur Ġwith Ġthe ĠCRC Ġ& Ġ3 C Ġmodels . ĠESA Ġusing Ġthe Ġsparse ĠB OC Ġvectors Ġachieves Ġlow ĠF 1 Ġup Ġuntil Ġfew Ġhundred Ġdimensions Ġare Ġconsidered . ĠEven Ġwith Ġthe Ġcostly ĠWE ĠIN LINE FORM 0 Ġand ĠWE ĠIN LINE FORM 1 Ġdens ifications , Ġperformance Ġsometimes Ġdecreases . </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġproposed Ġtwo Ġmodels Ġfor Ġlearning Ġconcept Ġembed d ings Ġbased Ġon Ġthe Ġskip - gram Ġmodel . ĠWe Ġalso Ġproposed Ġan Ġefficient Ġand Ġeffective Ġmechanism Ġfor ĠB OC Ġdens ification Ġwhich Ġoutper formed Ġthe Ġprior Ġproposed Ġdens ification Ġschemes Ġon Ġdat al ess Ġclassification . ĠUnlike Ġthese Ġprior Ġdens ification Ġmechanisms , Ġour Ġmethod Ġscales Ġlin early Ġwith Ġthe Ġ# Ġof Ġthe ĠB OC Ġdimensions . ĠIn Ġaddition , Ġwe Ġdemonstrated Ġthrough Ġthe Ġresults Ġhow Ġthis Ġefficient Ġmechanism Ġallows Ġgenerating Ġhigh Ġquality Ġdense ĠB OC Ġvectors Ġfrom Ġfew Ġconcepts Ġallev iating Ġthe Ġneed Ġof Ġobtaining Ġhundreds Ġof Ġconcepts Ġwhen Ġgenerating Ġthe Ġconcept Ġvector .
Introduction Vector-based semantic mapping models are used to represent textual structures (words, phrases, and documents) as high-dimensional meaning vectors. Typically, these models utilize textual corpora and/or Knowledge Bases (KBs) to acquire world knowledge, which is then used to generate a vector representation for the given text in the semantic space. The goal is thus to accurately place semantically similar structures close to each other in that semantic space. On the other hand, dissimilar structures should be far apart. Explicit concept space models are motivated by the idea that high level cognitive tasks such learning and reasoning are supported by the knowledge we acquire from concepts BIBREF0 . Therefore, such models utilize concept vectors (a.k.a bag-of-concepts (BOC)) as the underlying semantic representation of a given text through a process called conceptualization, which is mapping the text into relevant concepts capturing its main topics. The concept space typically include concepts obtained from KBs such as Wikipedia, Probase BIBREF1 , and others. Once the concept vectors are generated, similarity between two concept vectors can be computed using a suitable similarity/distance measure such as cosine. The BOC representation has proven efficacy for semantic analysis of textual data especially short texts where contextual information is missing or insufficient. For example, measuring semantic similarity/relatedness BIBREF2 , BIBREF3 , BIBREF4 , dataless classification BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , short text clustering BIBREF0 , search and relevancy ranking BIBREF9 , event detection and coreference resolution BIBREF10 . Similar to the traditional bag-of-words representation, the BOC vector is a high dimensional sparse vector whose dimensionality is the same as the number of concepts in the employed KB (typically millions). Consequently, it suffers from data sparsity causing low similarity scores between similar texts due to low concept overlap. Formally, given a text snippet INLINEFORM0 of INLINEFORM1 terms where INLINEFORM2 , and a concept space INLINEFORM3 of size INLINEFORM4 . The BOC vector INLINEFORM5 = INLINEFORM6 of INLINEFORM7 is a vector of weights of each concept where each INLINEFORM8 of concept INLINEFORM9 is calculated as in equation 1: DISPLAYFORM0 Here INLINEFORM0 is a scoring function which indicates the degree of association between term INLINEFORM1 and concept INLINEFORM2 . For example, gabrilovich2007computing proposed Explicit Semantic Analysis (ESA) which uses Wikipedia articles as concepts and the TF-IDF score of the terms in these article as the association score. Another scoring function might be the co-occurrence count or Pearson correlation score between INLINEFORM3 and INLINEFORM4 . As we can notice, only very small subset of the concept space would have non-zero scores with the given terms. Moreover, the BOC vector is generated from the INLINEFORM5 concepts which have relatively high association scores with the input terms (typically few hundreds). Thus each text snippet is mapped to a very sparse vector of millions of dimensions having only few hundreds non-zero values BIBREF10 . Typically, the cosine similarity measure is used compute the similarity between a pair of BOC vectors INLINEFORM0 and INLINEFORM1 . Because the concept vectors are very sparse, we can rewrite each vector as a vector of tuples INLINEFORM2 . Suppose that INLINEFORM3 and INLINEFORM4 , where INLINEFORM5 and INLINEFORM6 are the corresponding weights of concepts INLINEFORM7 and INLINEFORM8 respectively. And INLINEFORM9 , INLINEFORM10 are the indices of these concepts in the concept space INLINEFORM11 such that INLINEFORM12 . Then, the relatedness score can be written as in equation 2: DISPLAYFORM0 where INLINEFORM0 is the indicator function which returns 1 if INLINEFORM1 and 0 otherwise. Having such sparse representation and using exact match similarity scoring measure, we can expect that two very similar text snippets might have zero similarity score if they map to different but very related set of concepts BIBREF7 . Neural embedding models have been proposed to overcome the BOC sparsity problem. The basic idea is to learn fixed size continuous vectors for each concept. These vectors can then be used to compute concept-concept similarity and thus overcome the concept mismatch problem. In this paper we propose two neural embedding models in order to learn continuous concept vectors based on the skip-gram model BIBREF11 . Our first model is the Concept Raw Context model (CRC) which utilizes concept mentions in a large scale KB to jointly learn embeddings of both words and concepts. Our second model is the Concept-Concept Context model (3C) which learns the embeddings of concepts from their conceptual contexts (i.e., contexts containing surrounding concepts only). After learning the concept vectors, we propose an efficient concept vector aggregation method to generate fully dense BOC representations. Our efficient aggregation method allows measuring the similarity between pairs of BOC vectors in linear time. This is more efficient than prior methods which require quadratic time or at least log-linear time if optimized (see equation 2). We evaluate our embedding models on two tasks: The contributions of this paper are threefold: First, we propose two low cost concept embedding models which requires few hours rather than days to train. Second, we propose simple and efficient vector aggregation method to obtain fully densified BOC vectors in linear time. Third, we demonstrate through experiments that we can obtain same or better accuracy using the densified BOC representation with much less dimensions (few in most cases), reducing the computational cost of generating the BOC vector significantly. Related Work Concept/Entity Embeddings: neural embedding models have been proposed to learn distributed representations of concepts/entities. songunsupervised proposed using the popular Word2Vec model BIBREF12 to obtain the embeddings of each concept by averaging the vectors of the concept's individual words. For example, the embeddings of Microsoft Office would be obtained by averaging the embeddings of Microsoft and Office obtained from the Word2Vec model. Clearly, this method disregards the fact that the semantics of multi-word concepts is different from the semantics of their individual words. More robust concept embeddings can be learned from the concept's corresponding article and/or from the structure of the employed KB (e.g., its link graph). Such concept embedding models were proposed by hu2015entity,li2016joint,yamada2016joint who all utilize the skip-gram model BIBREF11 , but differ in how they define the context of the target concept. li2016joint extended the embedding model proposed by hu2015entity by jointly learning concept and category embeddings from contexts defined by all other concepts in the target concept's article as well as its category hierarchy in Wikipedia. This method has the advantage of learning embeddings of both concepts and categories simultaneously. However, defining the concept contexts as pairs of the target concept and all other concepts appearing in its corresponding article might introduce noisy contexts, especially for long articles. For example, the Wikipedia article for United States contains links to Kindergarten, First grade, and Secondary school under the Education section. yamada2016joint proposed a method based on the skip-gram model to jointly learn embeddings of words and concepts using contexts generated from surrounding words of the target concept or word. The authors also proposed incorporating the KB link graph by generating contexts from all concepts with outgoing link to the target concept to better model concept-concept relatedness. Unlike li2016joint and hu2015entity who learn concept embeddings only, our CRC model (described in Section 3), maps both words and concepts into the same semantic space. Therefore we can easily measure word-word, word-concept, and concept-concept semantic similarities. In addition, compared to yamada2016joint model, we utilize contexts generated from both surrounding words and concepts. Therefore, we can better capture local contextual information of each target word/concept. Moreover, our proposed models are computationally less costly than hu2015entity and yamada2016joint models as they require few hours rather than days to train on similar computing resources. BOC Densification: distributed concept vectors have been used by BOC densification mechanisms to overcome the BOC sparsity problem. songunsupervised proposed three different mechanisms for aligning the concepts at different indices given a sparse BOC pair ( INLINEFORM0 ) in order to increase their similarity score. The many-to-many mechanism works by averaging all pairwise similarities. The many-to-one mechanism works by aligning each concept in INLINEFORM0 with the most similar concept in INLINEFORM1 (i.e., its best match). Clearly, the complexity of these two mechanisms is quadratic. The third mechanism is the one-to-one. It utilizes the Hungarian method in order to find an optimal alignment on a one-to-one basis BIBREF13 . This mechanism performed the best on dataless classification and was also utilized by li2016joint. However, the Hungarian method is a combinatorial optimization algorithm whose complexity is polynomial. Our proposed densification mechanism is more efficient than these three mechanisms as its complexity is linear with respect to the number of non-zero elements in the BOC vector. Additionally, it is simpler as it does not require tuning a cut off threshold for the minimum similarity between two aligned concepts as in prior work. Concept Embeddings for BOC Densification A main objective of learning concept embeddings is to overcome the inherent problem of data sparsity associated with the BOC representation. Here we try to learn continuous concept vectors by building upon the skip-gram embedding model BIBREF11 . In the conventional skip-gram model, a set of contexts are generated by sliding a context window of predefined size over sentences of a given text corpus. Vector representation of a target word is learned with the objective to maximize the ability of predicting surrounding words of that target word. Formally, given a training corpus of INLINEFORM0 words INLINEFORM1 . The skip-gram model aims to maximize the average log probability: DISPLAYFORM0 where INLINEFORM0 is the context window size, INLINEFORM1 is the target word, and INLINEFORM2 is a surrounding context word. The softmax function is used to estimate the probability INLINEFORM3 as follows: DISPLAYFORM0 where INLINEFORM0 and INLINEFORM1 are the input and output vectors respectively and INLINEFORM2 is the vocabulary size. mikolov2013distributed proposed hierarchical softmax and negative sampling as efficient alternatives to approximate the softmax function which becomes computationally intractable when INLINEFORM3 becomes huge. Our approach genuinely learns distributed concept representations by generating concept contexts from mentions of those concepts in large encyclopedic KBs such as Wikipedia. Utilizing such annotated KBs eliminates the need to manually annotate concept mentions and thus comes at no cost. Concept Raw Context Model (CRC) In this model, we jointly learn the embeddings of both words and concepts. First, all concept mentions are identified in the given corpus. Second, contexts are generated for both words and concepts from both other surrounding words and other surrounding concepts as well. After generating all the contexts, we use the skip-gram model to jointly learn words and concepts embeddings. Formally, given a training corpus of INLINEFORM0 words INLINEFORM1 . We iterate over the corpus identifying words and concept mentions and thus generating a sequence of INLINEFORM2 tokens INLINEFORM3 where INLINEFORM4 (as multi-word concepts will be counted as one token). Afterwards we train the a skip-gram model aiming to maximize: DISPLAYFORM0 where as in the conventional skip-gram model, INLINEFORM0 is the context window size. In this model, INLINEFORM1 is the target token which would be either a word or a concept mention, and INLINEFORM2 is a surrounding context word or concept mention. This model is different from yamada2016joint anchor context model in three aspects: 1) while generating target concept contexts, we utilize not only surrounding words but other surrounding concepts as well, 2) our model aims to maximize INLINEFORM0 where INLINEFORM1 could be a word or a concept, while yamada2016joint model maximizes INLINEFORM2 where INLINEFORM3 is the target concept/entity (see yamada2016joint Eq. 6), and 3) in case INLINEFORM4 is a concept, our model captures all the contexts in which it appeared, while yamada2016joint model generates for each entity one context of INLINEFORM5 previous and INLINEFORM6 next words. We hypothesize that considering both concepts and individual words in the optimization function would generate more robust embeddings. Concept-Concept Context Model (3C) Inspired by the distributional hypothesis BIBREF14 , we, in this model, hypothesize that "similar concepts tend to appear in similar conceptual contexts". In order to test this hypothesis, we learn concept embeddings by training a skip-gram model on contexts generated solely from concept mentions. As in the CRC model, we start by identifying all concept mentions in the given corpus. Then, contexts are generated from only surrounding concepts. Formally, given a training corpus of INLINEFORM0 words INLINEFORM1 . We iterate over the corpus identifying concept mentions and thus generating a sequence of INLINEFORM2 concept tokens INLINEFORM3 where INLINEFORM4 . Afterwards we train the skip-gram model aiming to maximize: DISPLAYFORM0 where INLINEFORM0 is the context window size, INLINEFORM1 is the target concept, and INLINEFORM2 is a surrounding concept mention within INLINEFORM3 mentions. This model is different from li2016joint and hu2015entity as they define the context of a target concept by all the other concepts which appear in the concept's corresponding article. Clearly, some of these concepts might be irrelevant especially for very long articles which cite hundreds of other concepts. Our 3C model, alternatively, learns concept semantics from surrounding concepts and not only from those that are cited in its article. We also extend the context window beyond pairs of concepts allowing more influence to other nearby concepts. The main advantage of the 3C model over the CRC model is its computational efficiency where the model vocabulary is limited to the corpus concepts (Wikipedia in our case). One the other hand, the CRC model is advantageous because it jointly learns the embeddings of words and concepts and is therefore expected to generate higher quality vectors. In other words, the CRC model can capture more contextual signals such as actions, times, and relationships at the expense of training computational cost. Training We utilize a recent Wikipedia dump of August 2016, which has about 7 million articles. We extract articles plain text discarding images and tables. We also discard References and External links sections (if any). We pruned both articles not under the main namespace and pruned all redirect pages as well. Eventually, our corpus contained about 5 million articles in total. We preprocess each article replacing all its references to other Wikipedia articles with the their corresponding page IDs. In case any of the references is a title of a redirect page, we use the page ID of the original page to ensure that all concept mentions are normalized. Following mikolov2013distributed, we utilize negative sampling to approximate the softmax function by replacing every INLINEFORM0 term in the softmax function (equation 4) with: DISPLAYFORM0 where INLINEFORM0 is the number of negative samples drawn for each word and INLINEFORM1 is the sigmoid function ( INLINEFORM2 ). In the case of the CRC model INLINEFORM3 and INLINEFORM4 would be replaced with INLINEFORM5 and INLINEFORM6 respectively. And in the case of the 3C model INLINEFORM7 and INLINEFORM8 would be replaced with INLINEFORM9 and INLINEFORM10 respectively. For both the CRC & 3C models with use a context window of size 9 and a vector of 500 dimensions. We train the skip-gram model for 10 iterations using 12 cores machine with 64GB of RAM. The CRC model took INLINEFORM0 15 hours to train for a total of INLINEFORM1 12.7 million tokens. The 3C model took INLINEFORM2 1.5 hours to train for a total of INLINEFORM3 4.5 million concepts. BOC Densification As we mentioned in the related work section, the current mechanisms for BOC densification are inefficient as their complexity is least quadratic with respect to the number of non-zero elements in the BOC vector. Here, we propose simple and efficient vector aggregation method to obtain fully densified BOC vectors in linear time. Our mechanism works by performing a weighted average of the embedding vectors of all concepts in the given BOC. This operation scales linearly with the number of non-zero dimensions in the BOC vector. In addition, it produces a fully dense vector representing the semantics of the original concepts and considering their weights. Formally, given a sparse BOC vector INLINEFORM0 where INLINEFORM1 is weight of concept INLINEFORM2 . We can obtain the dense representation of INLINEFORM3 as in equation 8: DISPLAYFORM0 where INLINEFORM0 is the embedding vector of concept INLINEFORM1 . Once we have this dense BOC vector, we can apply the cosine measure to compute the similarity between a pair of dense BOC vectors. As we can notice, this weighted average is done once and for all for a given BOC vector. Other mechanisms that rely on concept alignment BIBREF7 , require realignment every time a given BOC vector is compared to another BOC vector. Our approach improves the efficiency especially in the context of dataless classification with large number of classes. Using our densification mechanism, we apply the weighted average for each class vector and for each instance once. Interestingly, our densification mechanism allows us to densify the sparse BOC vector using only the top few dimensions. As we will show in the experiments section, we can get (near-)best results using these few dimensions compared to densifying with all the dimensions in the original sparse vector. This property reduces the cost of obtaining a BOC vector with a few hundred dimensions in the first place. Entity Semantic Relatedness We evaluate the "goodness" of our concept embeddings on measuring entity semantic relatedness as an intrinsic evaluation. Entity relatedness has been recently used to model entity coherence in many named entity disambiguation systems. We use a benchmark dataset created by ceccarelli2013learning from the CoNLL 2003 data. As in previous studies BIBREF16 , BIBREF15 , we model measuring entity relatedness as a ranking problem. We use the test split of the dataset to create 3,314 queries. Each query has a query entity and INLINEFORM0 91 response entities labeled as related or unrelated. The quality is measured by the ability of the system to rank related entities on top of unrelated ones. We compare our models with two prior methods: yamada2016joint who used the skip-gram model to learn embeddings of words and entities jointly. The authors also utilized Wikipedia link graph to better model entity-entity relatedness. witten2008effective who proposed Wikipedia Link-based Measure (WLM) as a simple mechanism for modeling the semantic relatedness between Wikipedia concepts. The authors utilized Wikipedia link structure under the assumption that related concepts would have similar incoming links. We report Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (nDCG) scores as commonly used measures for evaluating the ranking quality. Table 1 shows the performance of our CRC and 3C models compared to previous models. As we can see, the 3C model performs poorly on this task compared to prior models. On the other hand, our CRC model outperforms all the other methods by 2-4% in terms of nDCG and by 3% percent in terms of MAP. Dataless Classification chang2008importance proposed dataless classification as a learning protocol to perform text categorization without the need for labeled data to train a classifier. Given only label names and few descriptive keywords of each label, classification is performed on the fly by mapping each label into a BOC representation using ESA. Likewise, each data instance is mapped into the same BOC semantic space and assigned to the most similar label using a proper similarity measure such as cosine. Formally, given a set of INLINEFORM0 labels INLINEFORM1 , a text document INLINEFORM2 , a BOC mapping model INLINEFORM3 , and a similarity function INLINEFORM4 , then INLINEFORM5 is assigned to the INLINEFORM6 th label INLINEFORM7 such that INLINEFORM8 . We evaluate the effectiveness of our concept embedding models on the dataless classification task as an extrinsic evaluation. We demonstrate through empirical results the efficiency and effectiveness of our proposed BOC densification scheme in obtaining better classification results compared to the original sparse BOC representation. We use the 20 Newsgroups dataset (20NG) BIBREF17 which is commonly used for benchmarking text classification algorithms. The dataset contains 20 categories each has INLINEFORM0 1000 news posts. We obtained the BOC representations using ESA from song2014dataless who utilized a Wikipedia index containing pages with 100+ words and 5+ outgoing links to create ESA mappings of 500 dimensions for both the categories and news posts of the 20NG. We designed two types of classification tasks: 1) fine-grained classification involving closely related classes such as Hockey vs. Baseball, Autos vs. Motorcycles, and Guns vs. Mideast vs. Misc, and 2) coarse-grained classification involving top-level categories such as Sport vs. Politics and Sport vs. Religion. The top-level categories are created by combining instances of the fine-grained categories as shown in Table 2. We compare our models with three prior methods: ESA which computes the cosine similarity using the sparse BOC vectors. WE INLINEFORM0 & WE INLINEFORM1 which were proposed by songunsupervised for BOC densification using embeddings obtained from Word2Vec. As the authors reported, we fix the minimum similarity threshold to 0.85. WE INLINEFORM2 finds the best match for each concept, while WE INLINEFORM3 utilizes the Hungarian algorithm to find the best concept-concept alignment on one-to-one basis. Both mechanisms have polynomial time complexity. Table 3 presents the results of fine-grained dataless classification measured in micro-averaged F1. As we can notice, ESA achieves its peak performance with a few hundred dimensions of the sparse BOC vector. Using our densification mechanism, both the CRC & 3C models achieve equal performance to ESA at much less dimensions. Densification using the CRC model embeddings gives the best F1 scores on the three tasks. Interestingly, the CRC model improves the F1 score by INLINEFORM0 7% using only 14 concepts on Autos vs. Motorcycles, and by INLINEFORM1 3% using 70 concepts on Guns vs. Mideast vs. Misc. The 3C model, still performs better than ESA on 2 out of the 3 tasks. Both WE INLINEFORM2 and WE INLINEFORM3 improve the performance over ESA but not as our CRC model. In order to better illustrate the robustness of our densification mechanism when varying the # of BOC dimensions, we measured F1 scores of each task as a function of the # of BOC dimensions used for densification. As we see in Figure 1, with one concept we can achieve high F1 scores compared to ESA which achieves zero or very low F1. Moreover, near-peak performance is achievable with the top 50 or less dimensions. We can also notice that, as we increase the # of dimensions, both WE INLINEFORM0 and WE INLINEFORM1 densification methods have the same undesired monotonic pattern like ESA. Actually, the imposed threshold by these methods does not allow for full dense representation of the BOC vector and therefore at low dimensions we still see low overall F1 score. Our proposed densification mechanisms besides their low cost, produce fully densified representations allowing good similarities at low dimensions. Results of coarse-grained classification are presented in Table 4. Classification at the top level is easier than the fine-grained level. Nevertheless, as with fine-grained classification, ESA still peaks with a few hundred dimensions of the sparse BOC vector. Both the CRC & 3C models achieve equal performance to ESA at very few dimensions ( INLINEFORM0 ). Densification using the CRC model embeddings still performs the best on both tasks. Interestingly, the 3C model gives very close F1 scores to the CRC model at less dimensions (@4 with Sport vs. Politics, and @60 with Sport vs. Religion) indicating its competitive advantage when computational cost is a decisive criteria. The 3C model, still performs better than ESA, WE INLINEFORM1 , and WE INLINEFORM2 on both tasks. Figure 2 shows F1 scores of coarse-grained classification when varying the # of BOC dimensions used for densification. The same pattern of achieving near-peak performance at very few dimensions recur with the CRC & 3C models. ESA using the sparse BOC vectors achieves low F1 up until few hundred dimensions are considered. Even with the costly WE INLINEFORM0 and WE INLINEFORM1 densifications, performance sometimes decreases. Conclusion In this paper we proposed two models for learning concept embeddings based on the skip-gram model. We also proposed an efficient and effective mechanism for BOC densification which outperformed the prior proposed densification schemes on dataless classification. Unlike these prior densification mechanisms, our method scales linearly with the # of the BOC dimensions. In addition, we demonstrated through the results how this efficient mechanism allows generating high quality dense BOC vectors from few concepts alleviating the need of obtaining hundreds of concepts when generating the concept vector.
What is the benchmark dataset?
f4496316ddd35ee2f0ccc6475d73a66abf87b611
What Ġis Ġthe Ġbenchmark Ġdataset ?
a Ġbenchmark Ġdataset Ġcreated Ġby Ġce cc are lli 2013 learning Ġfrom Ġthe ĠCo N LL Ġ2003 Ġdata
<s> What Ġis Ġthe Ġbenchmark Ġdataset ? </s> Introduction </s> Vector - based Ġsemantic Ġmapping Ġmodels Ġare Ġused Ġto Ġrepresent Ġtextual Ġstructures Ġ( words , Ġphrases , Ġand Ġdocuments ) Ġas Ġhigh - dimensional Ġmeaning Ġvectors . ĠTypically , Ġthese Ġmodels Ġutilize Ġtextual Ġcorpor a Ġand / or ĠKnowledge ĠB ases Ġ( KB s ) Ġto Ġacquire Ġworld Ġknowledge , Ġwhich Ġis Ġthen Ġused Ġto Ġgenerate Ġa Ġvector Ġrepresentation Ġfor Ġthe Ġgiven Ġtext Ġin Ġthe Ġsemantic Ġspace . ĠThe Ġgoal Ġis Ġthus Ġto Ġaccurately Ġplace Ġsem antically Ġsimilar Ġstructures Ġclose Ġto Ġeach Ġother Ġin Ġthat Ġsemantic Ġspace . ĠOn Ġthe Ġother Ġhand , Ġdiss imilar Ġstructures Ġshould Ġbe Ġfar Ġapart . </s> Expl icit Ġconcept Ġspace Ġmodels Ġare Ġmotivated Ġby Ġthe Ġidea Ġthat Ġhigh Ġlevel Ġcognitive Ġtasks Ġsuch Ġlearning Ġand Ġreasoning Ġare Ġsupported Ġby Ġthe Ġknowledge Ġwe Ġacquire Ġfrom Ġconcepts ĠB IB REF 0 Ġ. ĠTherefore , Ġsuch Ġmodels Ġutilize Ġconcept Ġvectors Ġ( a . k . a Ġbag - of - concept s Ġ( B OC )) Ġas Ġthe Ġunderlying Ġsemantic Ġrepresentation Ġof Ġa Ġgiven Ġtext Ġthrough Ġa Ġprocess Ġcalled Ġconceptual ization , Ġwhich Ġis Ġmapping Ġthe Ġtext Ġinto Ġrelevant Ġconcepts Ġcapturing Ġits Ġmain Ġtopics . ĠThe Ġconcept Ġspace Ġtypically Ġinclude Ġconcepts Ġobtained Ġfrom ĠKB s Ġsuch Ġas ĠWikipedia , ĠPro base ĠB IB REF 1 Ġ, Ġand Ġothers . ĠOnce Ġthe Ġconcept Ġvectors Ġare Ġgenerated , Ġsimilarity Ġbetween Ġtwo Ġconcept Ġvectors Ġcan Ġbe Ġcomputed Ġusing Ġa Ġsuitable Ġsimilarity / distance Ġmeasure Ġsuch Ġas Ġcos ine . </s> The ĠB OC Ġrepresentation Ġhas Ġproven Ġefficacy Ġfor Ġsemantic Ġanalysis Ġof Ġtextual Ġdata Ġespecially Ġshort Ġtexts Ġwhere Ġcontextual Ġinformation Ġis Ġmissing Ġor Ġinsufficient . ĠFor Ġexample , Ġmeasuring Ġsemantic Ġsimilarity / related ness ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġdat al ess Ġclassification ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġshort Ġtext Ġclust ering ĠB IB REF 0 Ġ, Ġsearch Ġand Ġrele v ancy Ġranking ĠB IB REF 9 Ġ, Ġevent Ġdetection Ġand Ġcore ference Ġresolution ĠB IB REF 10 Ġ. </s> Similar Ġto Ġthe Ġtraditional Ġbag - of - words Ġrepresentation , Ġthe ĠB OC Ġvector Ġis Ġa Ġhigh Ġdimensional Ġsparse Ġvector Ġwhose Ġdimension ality Ġis Ġthe Ġsame Ġas Ġthe Ġnumber Ġof Ġconcepts Ġin Ġthe Ġemployed ĠKB Ġ( typically Ġmillions ). ĠConsequently , Ġit Ġsuffers Ġfrom Ġdata Ġsp arsity Ġcausing Ġlow Ġsimilarity Ġscores Ġbetween Ġsimilar Ġtexts Ġdue Ġto Ġlow Ġconcept Ġoverlap . ĠForm ally , Ġgiven Ġa Ġtext Ġsnippet ĠIN LINE FORM 0 Ġof ĠIN LINE FORM 1 Ġterms Ġwhere ĠIN LINE FORM 2 Ġ, Ġand Ġa Ġconcept Ġspace ĠIN LINE FORM 3 Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠThe ĠB OC Ġvector ĠIN LINE FORM 5 Ġ= ĠIN LINE FORM 6 Ġof ĠIN LINE FORM 7 Ġis Ġa Ġvector Ġof Ġweights Ġof Ġeach Ġconcept Ġwhere Ġeach ĠIN LINE FORM 8 Ġof Ġconcept ĠIN LINE FORM 9 Ġis Ġcalculated Ġas Ġin Ġequation Ġ1 : ĠDIS PLAY FORM 0 </s> Here ĠIN LINE FORM 0 Ġis Ġa Ġscoring Ġfunction Ġwhich Ġindicates Ġthe Ġdegree Ġof Ġassociation Ġbetween Ġterm ĠIN LINE FORM 1 Ġand Ġconcept ĠIN LINE FORM 2 Ġ. ĠFor Ġexample , Ġg ab ril ovich 2007 com puting Ġproposed ĠExplicit ĠSem antic ĠAnalysis Ġ( ESA ) Ġwhich Ġuses ĠWikipedia Ġarticles Ġas Ġconcepts Ġand Ġthe ĠTF - ID F Ġscore Ġof Ġthe Ġterms Ġin Ġthese Ġarticle Ġas Ġthe Ġassociation Ġscore . ĠAnother Ġscoring Ġfunction Ġmight Ġbe Ġthe Ġco - occ urrence Ġcount Ġor ĠPearson Ġcorrelation Ġscore Ġbetween ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ. ĠAs Ġwe Ġcan Ġnotice , Ġonly Ġvery Ġsmall Ġsubset Ġof Ġthe Ġconcept Ġspace Ġwould Ġhave Ġnon - zero Ġscores Ġwith Ġthe Ġgiven Ġterms . ĠMoreover , Ġthe ĠB OC Ġvector Ġis Ġgenerated Ġfrom Ġthe ĠIN LINE FORM 5 Ġconcepts Ġwhich Ġhave Ġrelatively Ġhigh Ġassociation Ġscores Ġwith Ġthe Ġinput Ġterms Ġ( typically Ġfew Ġhundreds ). ĠThus Ġeach Ġtext Ġsnippet Ġis Ġmapped Ġto Ġa Ġvery Ġsparse Ġvector Ġof Ġmillions Ġof Ġdimensions Ġhaving Ġonly Ġfew Ġhundreds Ġnon - zero Ġvalues ĠB IB REF 10 Ġ. </s> Typically , Ġthe Ġcos ine Ġsimilarity Ġmeasure Ġis Ġused Ġcompute Ġthe Ġsimilarity Ġbetween Ġa Ġpair Ġof ĠB OC Ġvectors ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠBecause Ġthe Ġconcept Ġvectors Ġare Ġvery Ġsparse , Ġwe Ġcan Ġrewrite Ġeach Ġvector Ġas Ġa Ġvector Ġof Ġtu ples ĠIN LINE FORM 2 Ġ. ĠSuppose Ġthat ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġthe Ġcorresponding Ġweights Ġof Ġconcepts ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġrespectively . ĠAnd ĠIN LINE FORM 9 Ġ, ĠIN LINE FORM 10 Ġare Ġthe Ġindices Ġof Ġthese Ġconcepts Ġin Ġthe Ġconcept Ġspace ĠIN LINE FORM 11 Ġsuch Ġthat ĠIN LINE FORM 12 Ġ. ĠThen , Ġthe Ġrelated ness Ġscore Ġcan Ġbe Ġwritten Ġas Ġin Ġequation Ġ2 : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġindicator Ġfunction Ġwhich Ġreturns Ġ1 Ġif ĠIN LINE FORM 1 Ġand Ġ0 Ġotherwise . ĠHaving Ġsuch Ġsparse Ġrepresentation Ġand Ġusing Ġexact Ġmatch Ġsimilarity Ġscoring Ġmeasure , Ġwe Ġcan Ġexpect Ġthat Ġtwo Ġvery Ġsimilar Ġtext Ġsnippets Ġmight Ġhave Ġzero Ġsimilarity Ġscore Ġif Ġthey Ġmap Ġto Ġdifferent Ġbut Ġvery Ġrelated Ġset Ġof Ġconcepts ĠB IB REF 7 Ġ. </s> Ne ural Ġembed ding Ġmodels Ġhave Ġbeen Ġproposed Ġto Ġovercome Ġthe ĠB OC Ġsp arsity Ġproblem . ĠThe Ġbasic Ġidea Ġis Ġto Ġlearn Ġfixed Ġsize Ġcontinuous Ġvectors Ġfor Ġeach Ġconcept . ĠThese Ġvectors Ġcan Ġthen Ġbe Ġused Ġto Ġcompute Ġconcept - concept Ġsimilarity Ġand Ġthus Ġovercome Ġthe Ġconcept Ġmismatch Ġproblem . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġtwo Ġneural Ġembed ding Ġmodels Ġin Ġorder Ġto Ġlearn Ġcontinuous Ġconcept Ġvectors Ġbased Ġon Ġthe Ġskip - gram Ġmodel ĠB IB REF 11 Ġ. ĠOur Ġfirst Ġmodel Ġis Ġthe ĠConcept ĠRaw ĠContext Ġmodel Ġ( C RC ) Ġwhich Ġutilizes Ġconcept Ġmentions Ġin Ġa Ġlarge Ġscale ĠKB Ġto Ġjointly Ġlearn Ġembed d ings Ġof Ġboth Ġwords Ġand Ġconcepts . ĠOur Ġsecond Ġmodel Ġis Ġthe ĠConcept - Con cept ĠContext Ġmodel Ġ( 3 C ) Ġwhich Ġlearns Ġthe Ġembed d ings Ġof Ġconcepts Ġfrom Ġtheir Ġconceptual Ġcontexts Ġ( i . e ., Ġcontexts Ġcontaining Ġsurrounding Ġconcepts Ġonly ). ĠAfter Ġlearning Ġthe Ġconcept Ġvectors , Ġwe Ġpropose Ġan Ġefficient Ġconcept Ġvector Ġaggregation Ġmethod Ġto Ġgenerate Ġfully Ġdense ĠB OC Ġrepresentations . ĠOur Ġefficient Ġaggregation Ġmethod Ġallows Ġmeasuring Ġthe Ġsimilarity Ġbetween Ġpairs Ġof ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠThis Ġis Ġmore Ġefficient Ġthan Ġprior Ġmethods Ġwhich Ġrequire Ġquad r atic Ġtime Ġor Ġat Ġleast Ġlog - linear Ġtime Ġif Ġoptimized Ġ( see Ġequation Ġ2 ). </s> We Ġevaluate Ġour Ġembed ding Ġmodels Ġon Ġtwo Ġtasks : </s> The Ġcontributions Ġof Ġthis Ġpaper Ġare Ġthree fold : ĠFirst , Ġwe Ġpropose Ġtwo Ġlow Ġcost Ġconcept Ġembed ding Ġmodels Ġwhich Ġrequires Ġfew Ġhours Ġrather Ġthan Ġdays Ġto Ġtrain . ĠSecond , Ġwe Ġpropose Ġsimple Ġand Ġefficient Ġvector Ġaggregation Ġmethod Ġto Ġobtain Ġfully Ġdens ified ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠThird , Ġwe Ġdemonstrate Ġthrough Ġexperiments Ġthat Ġwe Ġcan Ġobtain Ġsame Ġor Ġbetter Ġaccuracy Ġusing Ġthe Ġdens ified ĠB OC Ġrepresentation Ġwith Ġmuch Ġless Ġdimensions Ġ( few Ġin Ġmost Ġcases ), Ġreducing Ġthe Ġcomputational Ġcost Ġof Ġgenerating Ġthe ĠB OC Ġvector Ġsignificantly . </s> Related ĠWork </s> Con cept / Entity ĠEmb edd ings : Ġneural Ġembed ding Ġmodels Ġhave Ġbeen Ġproposed Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġconcepts / ent ities . Ġsong un super vised Ġproposed Ġusing Ġthe Ġpopular ĠWord 2 V ec Ġmodel ĠB IB REF 12 Ġto Ġobtain Ġthe Ġembed d ings Ġof Ġeach Ġconcept Ġby Ġaveraging Ġthe Ġvectors Ġof Ġthe Ġconcept 's Ġindividual Ġwords . ĠFor Ġexample , Ġthe Ġembed d ings Ġof ĠMicrosoft ĠOffice Ġwould Ġbe Ġobtained Ġby Ġaveraging Ġthe Ġembed d ings Ġof ĠMicrosoft Ġand ĠOffice Ġobtained Ġfrom Ġthe ĠWord 2 V ec Ġmodel . ĠClearly , Ġthis Ġmethod Ġdisreg ards Ġthe Ġfact Ġthat Ġthe Ġsemantics Ġof Ġmulti - word Ġconcepts Ġis Ġdifferent Ġfrom Ġthe Ġsemantics Ġof Ġtheir Ġindividual Ġwords . ĠMore Ġrobust Ġconcept Ġembed d ings Ġcan Ġbe Ġlearned Ġfrom Ġthe Ġconcept 's Ġcorresponding Ġarticle Ġand / or Ġfrom Ġthe Ġstructure Ġof Ġthe Ġemployed ĠKB Ġ( e . g ., Ġits Ġlink Ġgraph ). ĠSuch Ġconcept Ġembed ding Ġmodels Ġwere Ġproposed Ġby Ġh u 2015 entity , li 2016 j oint , y am ada 2016 j oint Ġwho Ġall Ġutilize Ġthe Ġskip - gram Ġmodel ĠB IB REF 11 Ġ, Ġbut Ġdiffer Ġin Ġhow Ġthey Ġdefine Ġthe Ġcontext Ġof Ġthe Ġtarget Ġconcept . </s> li 2016 j oint Ġextended Ġthe Ġembed ding Ġmodel Ġproposed Ġby Ġh u 2015 entity Ġby Ġjointly Ġlearning Ġconcept Ġand Ġcategory Ġembed d ings Ġfrom Ġcontexts Ġdefined Ġby Ġall Ġother Ġconcepts Ġin Ġthe Ġtarget Ġconcept 's Ġarticle Ġas Ġwell Ġas Ġits Ġcategory Ġhierarchy Ġin ĠWikipedia . ĠThis Ġmethod Ġhas Ġthe Ġadvantage Ġof Ġlearning Ġembed d ings Ġof Ġboth Ġconcepts Ġand Ġcategories Ġsimultaneously . ĠHowever , Ġdefining Ġthe Ġconcept Ġcontexts Ġas Ġpairs Ġof Ġthe Ġtarget Ġconcept Ġand Ġall Ġother Ġconcepts Ġappearing Ġin Ġits Ġcorresponding Ġarticle Ġmight Ġintroduce Ġnoisy Ġcontexts , Ġespecially Ġfor Ġlong Ġarticles . ĠFor Ġexample , Ġthe ĠWikipedia Ġarticle Ġfor ĠUnited ĠStates Ġcontains Ġlinks Ġto ĠKinder g arten , ĠFirst Ġgrade , Ġand ĠSecondary Ġschool Ġunder Ġthe ĠEducation Ġsection . </s> y am ada 2016 j oint Ġproposed Ġa Ġmethod Ġbased Ġon Ġthe Ġskip - gram Ġmodel Ġto Ġjointly Ġlearn Ġembed d ings Ġof Ġwords Ġand Ġconcepts Ġusing Ġcontexts Ġgenerated Ġfrom Ġsurrounding Ġwords Ġof Ġthe Ġtarget Ġconcept Ġor Ġword . ĠThe Ġauthors Ġalso Ġproposed Ġincorporating Ġthe ĠKB Ġlink Ġgraph Ġby Ġgenerating Ġcontexts Ġfrom Ġall Ġconcepts Ġwith Ġoutgoing Ġlink Ġto Ġthe Ġtarget Ġconcept Ġto Ġbetter Ġmodel Ġconcept - concept Ġrelated ness . </s> Unlike Ġli 2016 j oint Ġand Ġh u 2015 entity Ġwho Ġlearn Ġconcept Ġembed d ings Ġonly , Ġour ĠCRC Ġmodel Ġ( described Ġin ĠSection Ġ3 ), Ġmaps Ġboth Ġwords Ġand Ġconcepts Ġinto Ġthe Ġsame Ġsemantic Ġspace . ĠTherefore Ġwe Ġcan Ġeasily Ġmeasure Ġword - word , Ġword - concept , Ġand Ġconcept - concept Ġsemantic Ġsimilarities . ĠIn Ġaddition , Ġcompared Ġto Ġy am ada 2016 j oint Ġmodel , Ġwe Ġutilize Ġcontexts Ġgenerated Ġfrom Ġboth Ġsurrounding Ġwords Ġand Ġconcepts . ĠTherefore , Ġwe Ġcan Ġbetter Ġcapture Ġlocal Ġcontextual Ġinformation Ġof Ġeach Ġtarget Ġword / concept . ĠMoreover , Ġour Ġproposed Ġmodels Ġare Ġcomput ationally Ġless Ġcostly Ġthan Ġh u 2015 entity Ġand Ġy am ada 2016 j oint Ġmodels Ġas Ġthey Ġrequire Ġfew Ġhours Ġrather Ġthan Ġdays Ġto Ġtrain Ġon Ġsimilar Ġcomputing Ġresources . </s> B OC ĠD ens ification : Ġdistributed Ġconcept Ġvectors Ġhave Ġbeen Ġused Ġby ĠB OC Ġdens ification Ġmechanisms Ġto Ġovercome Ġthe ĠB OC Ġsp arsity Ġproblem . Ġsong un super vised Ġproposed Ġthree Ġdifferent Ġmechanisms Ġfor Ġalign ing Ġthe Ġconcepts Ġat Ġdifferent Ġindices Ġgiven Ġa Ġsparse ĠB OC Ġpair Ġ( ĠIN LINE FORM 0 Ġ) Ġin Ġorder Ġto Ġincrease Ġtheir Ġsimilarity Ġscore . </s> The Ġmany - to - many Ġmechanism Ġworks Ġby Ġaveraging Ġall Ġpair wise Ġsimilarities . ĠThe Ġmany - to - one Ġmechanism Ġworks Ġby Ġalign ing Ġeach Ġconcept Ġin ĠIN LINE FORM 0 Ġwith Ġthe Ġmost Ġsimilar Ġconcept Ġin ĠIN LINE FORM 1 Ġ( i . e ., Ġits Ġbest Ġmatch ). ĠClearly , Ġthe Ġcomplexity Ġof Ġthese Ġtwo Ġmechanisms Ġis Ġquad r atic . ĠThe Ġthird Ġmechanism Ġis Ġthe Ġone - to - one . ĠIt Ġutilizes Ġthe ĠHungarian Ġmethod Ġin Ġorder Ġto Ġfind Ġan Ġoptimal Ġalignment Ġon Ġa Ġone - to - one Ġbasis ĠB IB REF 13 Ġ. ĠThis Ġmechanism Ġperformed Ġthe Ġbest Ġon Ġdat al ess Ġclassification Ġand Ġwas Ġalso Ġutilized Ġby Ġli 2016 j oint . ĠHowever , Ġthe ĠHungarian Ġmethod Ġis Ġa Ġcomb inator ial Ġoptimization Ġalgorithm Ġwhose Ġcomplexity Ġis Ġpo lyn omial . ĠOur Ġproposed Ġdens ification Ġmechanism Ġis Ġmore Ġefficient Ġthan Ġthese Ġthree Ġmechanisms Ġas Ġits Ġcomplexity Ġis Ġlinear Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġnon - zero Ġelements Ġin Ġthe ĠB OC Ġvector . ĠAdditionally , Ġit Ġis Ġsimpler Ġas Ġit Ġdoes Ġnot Ġrequire Ġtuning Ġa Ġcut Ġoff Ġthreshold Ġfor Ġthe Ġminimum Ġsimilarity Ġbetween Ġtwo Ġaligned Ġconcepts Ġas Ġin Ġprior Ġwork . </s> Con cept ĠEmb edd ings Ġfor ĠB OC ĠD ens ification </s> A Ġmain Ġobjective Ġof Ġlearning Ġconcept Ġembed d ings Ġis Ġto Ġovercome Ġthe Ġinherent Ġproblem Ġof Ġdata Ġsp arsity Ġassociated Ġwith Ġthe ĠB OC Ġrepresentation . ĠHere Ġwe Ġtry Ġto Ġlearn Ġcontinuous Ġconcept Ġvectors Ġby Ġbuilding Ġupon Ġthe Ġskip - gram Ġembed ding Ġmodel ĠB IB REF 11 Ġ. ĠIn Ġthe Ġconventional Ġskip - gram Ġmodel , Ġa Ġset Ġof Ġcontexts Ġare Ġgenerated Ġby Ġsliding Ġa Ġcontext Ġwindow Ġof Ġpred efined Ġsize Ġover Ġsentences Ġof Ġa Ġgiven Ġtext Ġcorpus . ĠVector Ġrepresentation Ġof Ġa Ġtarget Ġword Ġis Ġlearned Ġwith Ġthe Ġobjective Ġto Ġmaximize Ġthe Ġability Ġof Ġpredicting Ġsurrounding Ġwords Ġof Ġthat Ġtarget Ġword . </s> Form ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠThe Ġskip - gram Ġmodel Ġaims Ġto Ġmaximize Ġthe Ġaverage Ġlog Ġprobability : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġword , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġcontext Ġword . ĠThe Ġsoft max Ġfunction Ġis Ġused Ġto Ġestimate Ġthe Ġprobability ĠIN LINE FORM 3 Ġas Ġfollows : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġare Ġthe Ġinput Ġand Ġoutput Ġvectors Ġrespectively Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġvocabulary Ġsize . Ġm ik ol ov 2013 dist ributed Ġproposed Ġhierarchical Ġsoft max Ġand Ġnegative Ġsampling Ġas Ġefficient Ġalternatives Ġto Ġapproximate Ġthe Ġsoft max Ġfunction Ġwhich Ġbecomes Ġcomput ationally Ġint ract able Ġwhen ĠIN LINE FORM 3 Ġbecomes Ġhuge . </s> Our Ġapproach Ġgenuinely Ġlearns Ġdistributed Ġconcept Ġrepresentations Ġby Ġgenerating Ġconcept Ġcontexts Ġfrom Ġmentions Ġof Ġthose Ġconcepts Ġin Ġlarge Ġen cycl oped ic ĠKB s Ġsuch Ġas ĠWikipedia . ĠUt il izing Ġsuch Ġannot ated ĠKB s Ġeliminates Ġthe Ġneed Ġto Ġmanually Ġannot ate Ġconcept Ġmentions Ġand Ġthus Ġcomes Ġat Ġno Ġcost . </s> Con cept ĠRaw ĠContext ĠModel Ġ( C RC ) </s> In Ġthis Ġmodel , Ġwe Ġjointly Ġlearn Ġthe Ġembed d ings Ġof Ġboth Ġwords Ġand Ġconcepts . ĠFirst , Ġall Ġconcept Ġmentions Ġare Ġidentified Ġin Ġthe Ġgiven Ġcorpus . ĠSecond , Ġcontexts Ġare Ġgenerated Ġfor Ġboth Ġwords Ġand Ġconcepts Ġfrom Ġboth Ġother Ġsurrounding Ġwords Ġand Ġother Ġsurrounding Ġconcepts Ġas Ġwell . ĠAfter Ġgenerating Ġall Ġthe Ġcontexts , Ġwe Ġuse Ġthe Ġskip - gram Ġmodel Ġto Ġjointly Ġlearn Ġwords Ġand Ġconcepts Ġembed d ings . ĠForm ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠWe Ġiter ate Ġover Ġthe Ġcorpus Ġidentifying Ġwords Ġand Ġconcept Ġmentions Ġand Ġthus Ġgenerating Ġa Ġsequence Ġof ĠIN LINE FORM 2 Ġtokens ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ( as Ġmulti - word Ġconcepts Ġwill Ġbe Ġcounted Ġas Ġone Ġtoken ). ĠAfterwards Ġwe Ġtrain Ġthe Ġa Ġskip - gram Ġmodel Ġaiming Ġto Ġmaximize : ĠDIS PLAY FORM 0 </s> where Ġas Ġin Ġthe Ġconventional Ġskip - gram Ġmodel , ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize . ĠIn Ġthis Ġmodel , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġtoken Ġwhich Ġwould Ġbe Ġeither Ġa Ġword Ġor Ġa Ġconcept Ġmention , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġcontext Ġword Ġor Ġconcept Ġmention . </s> This Ġmodel Ġis Ġdifferent Ġfrom Ġy am ada 2016 j oint Ġanchor Ġcontext Ġmodel Ġin Ġthree Ġaspects : Ġ1 ) Ġwhile Ġgenerating Ġtarget Ġconcept Ġcontexts , Ġwe Ġutilize Ġnot Ġonly Ġsurrounding Ġwords Ġbut Ġother Ġsurrounding Ġconcepts Ġas Ġwell , Ġ2 ) Ġour Ġmodel Ġaims Ġto Ġmaximize ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġcould Ġbe Ġa Ġword Ġor Ġa Ġconcept , Ġwhile Ġy am ada 2016 j oint Ġmodel Ġmaxim izes ĠIN LINE FORM 2 Ġwhere ĠIN LINE FORM 3 Ġis Ġthe Ġtarget Ġconcept / entity Ġ( see Ġy am ada 2016 j oint ĠE q . Ġ6 ), Ġand Ġ3 ) Ġin Ġcase ĠIN LINE FORM 4 Ġis Ġa Ġconcept , Ġour Ġmodel Ġcaptures Ġall Ġthe Ġcontexts Ġin Ġwhich Ġit Ġappeared , Ġwhile Ġy am ada 2016 j oint Ġmodel Ġgenerates Ġfor Ġeach Ġentity Ġone Ġcontext Ġof ĠIN LINE FORM 5 Ġprevious Ġand ĠIN LINE FORM 6 Ġnext Ġwords . ĠWe Ġhypothes ize Ġthat Ġconsidering Ġboth Ġconcepts Ġand Ġindividual Ġwords Ġin Ġthe Ġoptimization Ġfunction Ġwould Ġgenerate Ġmore Ġrobust Ġembed d ings . </s> Con cept - Con cept ĠContext ĠModel Ġ( 3 C ) </s> Insp ired Ġby Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 14 Ġ, Ġwe , Ġin Ġthis Ġmodel , Ġhypothes ize Ġthat Ġ" similar Ġconcepts Ġtend Ġto Ġappear Ġin Ġsimilar Ġconceptual Ġcontexts ". ĠIn Ġorder Ġto Ġtest Ġthis Ġhypothesis , Ġwe Ġlearn Ġconcept Ġembed d ings Ġby Ġtraining Ġa Ġskip - gram Ġmodel Ġon Ġcontexts Ġgenerated Ġsolely Ġfrom Ġconcept Ġmentions . ĠAs Ġin Ġthe ĠCRC Ġmodel , Ġwe Ġstart Ġby Ġidentifying Ġall Ġconcept Ġmentions Ġin Ġthe Ġgiven Ġcorpus . ĠThen , Ġcontexts Ġare Ġgenerated Ġfrom Ġonly Ġsurrounding Ġconcepts . ĠForm ally , Ġgiven Ġa Ġtraining Ġcorpus Ġof ĠIN LINE FORM 0 Ġwords ĠIN LINE FORM 1 Ġ. ĠWe Ġiter ate Ġover Ġthe Ġcorpus Ġidentifying Ġconcept Ġmentions Ġand Ġthus Ġgenerating Ġa Ġsequence Ġof ĠIN LINE FORM 2 Ġconcept Ġtokens ĠIN LINE FORM 3 Ġwhere ĠIN LINE FORM 4 Ġ. ĠAfterwards Ġwe Ġtrain Ġthe Ġskip - gram Ġmodel Ġaiming Ġto Ġmaximize : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġwindow Ġsize , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġconcept , Ġand ĠIN LINE FORM 2 Ġis Ġa Ġsurrounding Ġconcept Ġmention Ġwithin ĠIN LINE FORM 3 Ġmentions . </s> This Ġmodel Ġis Ġdifferent Ġfrom Ġli 2016 j oint Ġand Ġh u 2015 entity Ġas Ġthey Ġdefine Ġthe Ġcontext Ġof Ġa Ġtarget Ġconcept Ġby Ġall Ġthe Ġother Ġconcepts Ġwhich Ġappear Ġin Ġthe Ġconcept 's Ġcorresponding Ġarticle . ĠClearly , Ġsome Ġof Ġthese Ġconcepts Ġmight Ġbe Ġirrelevant Ġespecially Ġfor Ġvery Ġlong Ġarticles Ġwhich Ġcite Ġhundreds Ġof Ġother Ġconcepts . ĠOur Ġ3 C Ġmodel , Ġalternatively , Ġlearns Ġconcept Ġsemantics Ġfrom Ġsurrounding Ġconcepts Ġand Ġnot Ġonly Ġfrom Ġthose Ġthat Ġare Ġcited Ġin Ġits Ġarticle . ĠWe Ġalso Ġextend Ġthe Ġcontext Ġwindow Ġbeyond Ġpairs Ġof Ġconcepts Ġallowing Ġmore Ġinfluence Ġto Ġother Ġnearby Ġconcepts . </s> The Ġmain Ġadvantage Ġof Ġthe Ġ3 C Ġmodel Ġover Ġthe ĠCRC Ġmodel Ġis Ġits Ġcomputational Ġefficiency Ġwhere Ġthe Ġmodel Ġvocabulary Ġis Ġlimited Ġto Ġthe Ġcorpus Ġconcepts Ġ( Wikipedia Ġin Ġour Ġcase ). ĠOne Ġthe Ġother Ġhand , Ġthe ĠCRC Ġmodel Ġis Ġadvantageous Ġbecause Ġit Ġjointly Ġlearns Ġthe Ġembed d ings Ġof Ġwords Ġand Ġconcepts Ġand Ġis Ġtherefore Ġexpected Ġto Ġgenerate Ġhigher Ġquality Ġvectors . ĠIn Ġother Ġwords , Ġthe ĠCRC Ġmodel Ġcan Ġcapture Ġmore Ġcontextual Ġsignals Ġsuch Ġas Ġactions , Ġtimes , Ġand Ġrelationships Ġat Ġthe Ġexpense Ġof Ġtraining Ġcomputational Ġcost . </s> Training </s> We Ġutilize Ġa Ġrecent ĠWikipedia Ġdump Ġof ĠAugust Ġ2016 , Ġwhich Ġhas Ġabout Ġ7 Ġmillion Ġarticles . ĠWe Ġextract Ġarticles Ġplain Ġtext Ġdisc arding Ġimages Ġand Ġtables . ĠWe Ġalso Ġdiscard ĠReferences Ġand ĠExternal Ġlinks Ġsections Ġ( if Ġany ). ĠWe Ġpr uned Ġboth Ġarticles Ġnot Ġunder Ġthe Ġmain Ġnamespace Ġand Ġpr uned Ġall Ġredirect Ġpages Ġas Ġwell . ĠEventually , Ġour Ġcorpus Ġcontained Ġabout Ġ5 Ġmillion Ġarticles Ġin Ġtotal . </s> We Ġpre process Ġeach Ġarticle Ġreplacing Ġall Ġits Ġreferences Ġto Ġother ĠWikipedia Ġarticles Ġwith Ġthe Ġtheir Ġcorresponding Ġpage ĠIDs . ĠIn Ġcase Ġany Ġof Ġthe Ġreferences Ġis Ġa Ġtitle Ġof Ġa Ġredirect Ġpage , Ġwe Ġuse Ġthe Ġpage ĠID Ġof Ġthe Ġoriginal Ġpage Ġto Ġensure Ġthat Ġall Ġconcept Ġmentions Ġare Ġnormalized . </s> Following Ġm ik ol ov 2013 dist ributed , Ġwe Ġutilize Ġnegative Ġsampling Ġto Ġapproximate Ġthe Ġsoft max Ġfunction Ġby Ġreplacing Ġevery ĠIN LINE FORM 0 Ġterm Ġin Ġthe Ġsoft max Ġfunction Ġ( equ ation Ġ4 ) Ġwith : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġnegative Ġsamples Ġdrawn Ġfor Ġeach Ġword Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġs igm oid Ġfunction Ġ( ĠIN LINE FORM 2 Ġ). ĠIn Ġthe Ġcase Ġof Ġthe ĠCRC Ġmodel ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġwould Ġbe Ġreplaced Ġwith ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġrespectively . ĠAnd Ġin Ġthe Ġcase Ġof Ġthe Ġ3 C Ġmodel ĠIN LINE FORM 7 Ġand ĠIN LINE FORM 8 Ġwould Ġbe Ġreplaced Ġwith ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġrespectively . </s> For Ġboth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġwith Ġuse Ġa Ġcontext Ġwindow Ġof Ġsize Ġ9 Ġand Ġa Ġvector Ġof Ġ500 Ġdimensions . ĠWe Ġtrain Ġthe Ġskip - gram Ġmodel Ġfor Ġ10 Ġiterations Ġusing Ġ12 Ġcores Ġmachine Ġwith Ġ64 GB Ġof ĠRAM . ĠThe ĠCRC Ġmodel Ġtook ĠIN LINE FORM 0 Ġ15 Ġhours Ġto Ġtrain Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 1 Ġ12 . 7 Ġmillion Ġtokens . ĠThe Ġ3 C Ġmodel Ġtook ĠIN LINE FORM 2 Ġ1 . 5 Ġhours Ġto Ġtrain Ġfor Ġa Ġtotal Ġof ĠIN LINE FORM 3 Ġ4 . 5 Ġmillion Ġconcepts . </s> B OC ĠD ens ification </s> As Ġwe Ġmentioned Ġin Ġthe Ġrelated Ġwork Ġsection , Ġthe Ġcurrent Ġmechanisms Ġfor ĠB OC Ġdens ification Ġare Ġinefficient Ġas Ġtheir Ġcomplexity Ġis Ġleast Ġquad r atic Ġwith Ġrespect Ġto Ġthe Ġnumber Ġof Ġnon - zero Ġelements Ġin Ġthe ĠB OC Ġvector . ĠHere , Ġwe Ġpropose Ġsimple Ġand Ġefficient Ġvector Ġaggregation Ġmethod Ġto Ġobtain Ġfully Ġdens ified ĠB OC Ġvectors Ġin Ġlinear Ġtime . ĠOur Ġmechanism Ġworks Ġby Ġperforming Ġa Ġweighted Ġaverage Ġof Ġthe Ġembed ding Ġvectors Ġof Ġall Ġconcepts Ġin Ġthe Ġgiven ĠB OC . ĠThis Ġoperation Ġscales Ġlin early Ġwith Ġthe Ġnumber Ġof Ġnon - zero Ġdimensions Ġin Ġthe ĠB OC Ġvector . ĠIn Ġaddition , Ġit Ġproduces Ġa Ġfully Ġdense Ġvector Ġrepresenting Ġthe Ġsemantics Ġof Ġthe Ġoriginal Ġconcepts Ġand Ġconsidering Ġtheir Ġweights . ĠForm ally , Ġgiven Ġa Ġsparse ĠB OC Ġvector ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġweight Ġof Ġconcept ĠIN LINE FORM 2 Ġ. ĠWe Ġcan Ġobtain Ġthe Ġdense Ġrepresentation Ġof ĠIN LINE FORM 3 Ġas Ġin Ġequation Ġ8 : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġembed ding Ġvector Ġof Ġconcept ĠIN LINE FORM 1 Ġ. ĠOnce Ġwe Ġhave Ġthis Ġdense ĠB OC Ġvector , Ġwe Ġcan Ġapply Ġthe Ġcos ine Ġmeasure Ġto Ġcompute Ġthe Ġsimilarity Ġbetween Ġa Ġpair Ġof Ġdense ĠB OC Ġvectors . </s> As Ġwe Ġcan Ġnotice , Ġthis Ġweighted Ġaverage Ġis Ġdone Ġonce Ġand Ġfor Ġall Ġfor Ġa Ġgiven ĠB OC Ġvector . ĠOther Ġmechanisms Ġthat Ġrely Ġon Ġconcept Ġalignment ĠB IB REF 7 Ġ, Ġrequire Ġreal ignment Ġevery Ġtime Ġa Ġgiven ĠB OC Ġvector Ġis Ġcompared Ġto Ġanother ĠB OC Ġvector . ĠOur Ġapproach Ġimproves Ġthe Ġefficiency Ġespecially Ġin Ġthe Ġcontext Ġof Ġdat al ess Ġclassification Ġwith Ġlarge Ġnumber Ġof Ġclasses . ĠUsing Ġour Ġdens ification Ġmechanism , Ġwe Ġapply Ġthe Ġweighted Ġaverage Ġfor Ġeach Ġclass Ġvector Ġand Ġfor Ġeach Ġinstance Ġonce . </s> Interestingly , Ġour Ġdens ification Ġmechanism Ġallows Ġus Ġto Ġdens ify Ġthe Ġsparse ĠB OC Ġvector Ġusing Ġonly Ġthe Ġtop Ġfew Ġdimensions . ĠAs Ġwe Ġwill Ġshow Ġin Ġthe Ġexperiments Ġsection , Ġwe Ġcan Ġget Ġ( near -) best Ġresults Ġusing Ġthese Ġfew Ġdimensions Ġcompared Ġto Ġdens ifying Ġwith Ġall Ġthe Ġdimensions Ġin Ġthe Ġoriginal Ġsparse Ġvector . ĠThis Ġproperty Ġreduces Ġthe Ġcost Ġof Ġobtaining Ġa ĠB OC Ġvector Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġin Ġthe Ġfirst Ġplace . </s> Entity ĠSem antic ĠRelated ness </s> We Ġevaluate Ġthe Ġ" good ness " Ġof Ġour Ġconcept Ġembed d ings Ġon Ġmeasuring Ġentity Ġsemantic Ġrelated ness Ġas Ġan Ġintrinsic Ġevaluation . ĠEntity Ġrelated ness Ġhas Ġbeen Ġrecently Ġused Ġto Ġmodel Ġentity Ġco herence Ġin Ġmany Ġnamed Ġentity Ġdis amb ig uation Ġsystems . </s> We Ġuse Ġa Ġbenchmark Ġdataset Ġcreated Ġby Ġce cc are lli 2013 learning Ġfrom Ġthe ĠCo N LL Ġ2003 Ġdata . ĠAs Ġin Ġprevious Ġstudies ĠB IB REF 16 Ġ, ĠB IB REF 15 Ġ, Ġwe Ġmodel Ġmeasuring Ġentity Ġrelated ness Ġas Ġa Ġranking Ġproblem . ĠWe Ġuse Ġthe Ġtest Ġsplit Ġof Ġthe Ġdataset Ġto Ġcreate Ġ3 , 314 Ġqueries . ĠEach Ġquery Ġhas Ġa Ġquery Ġentity Ġand ĠIN LINE FORM 0 Ġ91 Ġresponse Ġentities Ġlabeled Ġas Ġrelated Ġor Ġunrelated . ĠThe Ġquality Ġis Ġmeasured Ġby Ġthe Ġability Ġof Ġthe Ġsystem Ġto Ġrank Ġrelated Ġentities Ġon Ġtop Ġof Ġunrelated Ġones . </s> We Ġcompare Ġour Ġmodels Ġwith Ġtwo Ġprior Ġmethods : </s> y am ada 2016 j oint Ġwho Ġused Ġthe Ġskip - gram Ġmodel Ġto Ġlearn Ġembed d ings Ġof Ġwords Ġand Ġentities Ġjointly . ĠThe Ġauthors Ġalso Ġutilized ĠWikipedia Ġlink Ġgraph Ġto Ġbetter Ġmodel Ġentity - entity Ġrelated ness . </s> w itten 2008 effective Ġwho Ġproposed ĠWikipedia ĠLink - based ĠMeasure Ġ( W LM ) Ġas Ġa Ġsimple Ġmechanism Ġfor Ġmodeling Ġthe Ġsemantic Ġrelated ness Ġbetween ĠWikipedia Ġconcepts . ĠThe Ġauthors Ġutilized ĠWikipedia Ġlink Ġstructure Ġunder Ġthe Ġassumption Ġthat Ġrelated Ġconcepts Ġwould Ġhave Ġsimilar Ġincoming Ġlinks . </s> We Ġreport ĠMean ĠAverage ĠPrecision Ġ( MAP ) Ġand ĠNormal ized ĠDiscount ed ĠCum ulative ĠGain Ġ( n DC G ) Ġscores Ġas Ġcommonly Ġused Ġmeasures Ġfor Ġevaluating Ġthe Ġranking Ġquality . ĠTable Ġ1 Ġshows Ġthe Ġperformance Ġof Ġour ĠCRC Ġand Ġ3 C Ġmodels Ġcompared Ġto Ġprevious Ġmodels . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġ3 C Ġmodel Ġperforms Ġpoorly Ġon Ġthis Ġtask Ġcompared Ġto Ġprior Ġmodels . ĠOn Ġthe Ġother Ġhand , Ġour ĠCRC Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġmethods Ġby Ġ2 - 4 % Ġin Ġterms Ġof Ġn DC G Ġand Ġby Ġ3 % Ġpercent Ġin Ġterms Ġof ĠMAP . </s> D atal ess ĠClassification </s> ch ang 2008 import ance Ġproposed Ġdat al ess Ġclassification Ġas Ġa Ġlearning Ġprotocol Ġto Ġperform Ġtext Ġcategor ization Ġwithout Ġthe Ġneed Ġfor Ġlabeled Ġdata Ġto Ġtrain Ġa Ġclass ifier . ĠGiven Ġonly Ġlabel Ġnames Ġand Ġfew Ġdescriptive Ġkeywords Ġof Ġeach Ġlabel , Ġclassification Ġis Ġperformed Ġon Ġthe Ġfly Ġby Ġmapping Ġeach Ġlabel Ġinto Ġa ĠB OC Ġrepresentation Ġusing ĠESA . ĠLikewise , Ġeach Ġdata Ġinstance Ġis Ġmapped Ġinto Ġthe Ġsame ĠB OC Ġsemantic Ġspace Ġand Ġassigned Ġto Ġthe Ġmost Ġsimilar Ġlabel Ġusing Ġa Ġproper Ġsimilarity Ġmeasure Ġsuch Ġas Ġcos ine . ĠForm ally , Ġgiven Ġa Ġset Ġof ĠIN LINE FORM 0 Ġlabels ĠIN LINE FORM 1 Ġ, Ġa Ġtext Ġdocument ĠIN LINE FORM 2 Ġ, Ġa ĠB OC Ġmapping Ġmodel ĠIN LINE FORM 3 Ġ, Ġand Ġa Ġsimilarity Ġfunction ĠIN LINE FORM 4 Ġ, Ġthen ĠIN LINE FORM 5 Ġis Ġassigned Ġto Ġthe ĠIN LINE FORM 6 Ġth Ġlabel ĠIN LINE FORM 7 Ġsuch Ġthat ĠIN LINE FORM 8 Ġ. ĠWe Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġconcept Ġembed ding Ġmodels Ġon Ġthe Ġdat al ess Ġclassification Ġtask Ġas Ġan Ġextr ins ic Ġevaluation . ĠWe Ġdemonstrate Ġthrough Ġempirical Ġresults Ġthe Ġefficiency Ġand Ġeffectiveness Ġof Ġour Ġproposed ĠB OC Ġdens ification Ġscheme Ġin Ġobtaining Ġbetter Ġclassification Ġresults Ġcompared Ġto Ġthe Ġoriginal Ġsparse ĠB OC Ġrepresentation . </s> We Ġuse Ġthe Ġ20 ĠNews groups Ġdataset Ġ( 20 NG ) ĠB IB REF 17 Ġwhich Ġis Ġcommonly Ġused Ġfor Ġbenchmark ing Ġtext Ġclassification Ġalgorithms . ĠThe Ġdataset Ġcontains Ġ20 Ġcategories Ġeach Ġhas ĠIN LINE FORM 0 Ġ1000 Ġnews Ġposts . ĠWe Ġobtained Ġthe ĠB OC Ġrepresentations Ġusing ĠESA Ġfrom Ġsong 2014 d atal ess Ġwho Ġutilized Ġa ĠWikipedia Ġindex Ġcontaining Ġpages Ġwith Ġ100 + Ġwords Ġand Ġ5 + Ġoutgoing Ġlinks Ġto Ġcreate ĠESA Ġm appings Ġof Ġ500 Ġdimensions Ġfor Ġboth Ġthe Ġcategories Ġand Ġnews Ġposts Ġof Ġthe Ġ20 NG . ĠWe Ġdesigned Ġtwo Ġtypes Ġof Ġclassification Ġtasks : Ġ1 ) Ġfine - gr ained Ġclassification Ġinvolving Ġclosely Ġrelated Ġclasses Ġsuch Ġas ĠHockey Ġvs . ĠBaseball , ĠAut os Ġvs . ĠMotor cycles , Ġand ĠGuns Ġvs . ĠM ide ast Ġvs . ĠMisc , Ġand Ġ2 ) Ġcoarse - gr ained Ġclassification Ġinvolving Ġtop - level Ġcategories Ġsuch Ġas ĠSport Ġvs . ĠPolitics Ġand ĠSport Ġvs . ĠReligion . ĠThe Ġtop - level Ġcategories Ġare Ġcreated Ġby Ġcombining Ġinstances Ġof Ġthe Ġfine - gr ained Ġcategories Ġas Ġshown Ġin ĠTable Ġ2 . </s> We Ġcompare Ġour Ġmodels Ġwith Ġthree Ġprior Ġmethods : </s> ESA Ġwhich Ġcomp utes Ġthe Ġcos ine Ġsimilarity Ġusing Ġthe Ġsparse ĠB OC Ġvectors . </s> WE ĠIN LINE FORM 0 Ġ& ĠWE ĠIN LINE FORM 1 Ġwhich Ġwere Ġproposed Ġby Ġsong un super vised Ġfor ĠB OC Ġdens ification Ġusing Ġembed d ings Ġobtained Ġfrom ĠWord 2 V ec . ĠAs Ġthe Ġauthors Ġreported , Ġwe Ġfix Ġthe Ġminimum Ġsimilarity Ġthreshold Ġto Ġ0 . 85 . ĠWE ĠIN LINE FORM 2 Ġfinds Ġthe Ġbest Ġmatch Ġfor Ġeach Ġconcept , Ġwhile ĠWE ĠIN LINE FORM 3 Ġutilizes Ġthe ĠHungarian Ġalgorithm Ġto Ġfind Ġthe Ġbest Ġconcept - concept Ġalignment Ġon Ġone - to - one Ġbasis . ĠBoth Ġmechanisms Ġhave Ġpo lyn omial Ġtime Ġcomplexity . </s> Table Ġ3 Ġpresents Ġthe Ġresults Ġof Ġfine - gr ained Ġdat al ess Ġclassification Ġmeasured Ġin Ġmicro - aver aged ĠF 1 . ĠAs Ġwe Ġcan Ġnotice , ĠESA Ġachieves Ġits Ġpeak Ġperformance Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġof Ġthe Ġsparse ĠB OC Ġvector . ĠUsing Ġour Ġdens ification Ġmechanism , Ġboth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġachieve Ġequal Ġperformance Ġto ĠESA Ġat Ġmuch Ġless Ġdimensions . ĠD ens ification Ġusing Ġthe ĠCRC Ġmodel Ġembed d ings Ġgives Ġthe Ġbest ĠF 1 Ġscores Ġon Ġthe Ġthree Ġtasks . ĠInterestingly , Ġthe ĠCRC Ġmodel Ġimproves Ġthe ĠF 1 Ġscore Ġby ĠIN LINE FORM 0 Ġ7 % Ġusing Ġonly Ġ14 Ġconcepts Ġon ĠAut os Ġvs . ĠMotor cycles , Ġand Ġby ĠIN LINE FORM 1 Ġ3 % Ġusing Ġ70 Ġconcepts Ġon ĠGuns Ġvs . ĠM ide ast Ġvs . ĠMisc . ĠThe Ġ3 C Ġmodel , Ġstill Ġperforms Ġbetter Ġthan ĠESA Ġon Ġ2 Ġout Ġof Ġthe Ġ3 Ġtasks . ĠBoth ĠWE ĠIN LINE FORM 2 Ġand ĠWE ĠIN LINE FORM 3 Ġimprove Ġthe Ġperformance Ġover ĠESA Ġbut Ġnot Ġas Ġour ĠCRC Ġmodel . </s> In Ġorder Ġto Ġbetter Ġillustrate Ġthe Ġrobust ness Ġof Ġour Ġdens ification Ġmechanism Ġwhen Ġvarying Ġthe Ġ# Ġof ĠB OC Ġdimensions , Ġwe Ġmeasured ĠF 1 Ġscores Ġof Ġeach Ġtask Ġas Ġa Ġfunction Ġof Ġthe Ġ# Ġof ĠB OC Ġdimensions Ġused Ġfor Ġdens ification . ĠAs Ġwe Ġsee Ġin ĠFigure Ġ1 , Ġwith Ġone Ġconcept Ġwe Ġcan Ġachieve Ġhigh ĠF 1 Ġscores Ġcompared Ġto ĠESA Ġwhich Ġachieves Ġzero Ġor Ġvery Ġlow ĠF 1 . ĠMoreover , Ġnear - peak Ġperformance Ġis Ġachievable Ġwith Ġthe Ġtop Ġ50 Ġor Ġless Ġdimensions . ĠWe Ġcan Ġalso Ġnotice Ġthat , Ġas Ġwe Ġincrease Ġthe Ġ# Ġof Ġdimensions , Ġboth ĠWE ĠIN LINE FORM 0 Ġand ĠWE ĠIN LINE FORM 1 Ġdens ification Ġmethods Ġhave Ġthe Ġsame Ġundes ired Ġmon ot onic Ġpattern Ġlike ĠESA . ĠActually , Ġthe Ġimposed Ġthreshold Ġby Ġthese Ġmethods Ġdoes Ġnot Ġallow Ġfor Ġfull Ġdense Ġrepresentation Ġof Ġthe ĠB OC Ġvector Ġand Ġtherefore Ġat Ġlow Ġdimensions Ġwe Ġstill Ġsee Ġlow Ġoverall ĠF 1 Ġscore . ĠOur Ġproposed Ġdens ification Ġmechanisms Ġbesides Ġtheir Ġlow Ġcost , Ġproduce Ġfully Ġdens ified Ġrepresentations Ġallowing Ġgood Ġsimilarities Ġat Ġlow Ġdimensions . </s> Results Ġof Ġcoarse - gr ained Ġclassification Ġare Ġpresented Ġin ĠTable Ġ4 . ĠClassification Ġat Ġthe Ġtop Ġlevel Ġis Ġeasier Ġthan Ġthe Ġfine - gr ained Ġlevel . ĠNevertheless , Ġas Ġwith Ġfine - gr ained Ġclassification , ĠESA Ġstill Ġpeaks Ġwith Ġa Ġfew Ġhundred Ġdimensions Ġof Ġthe Ġsparse ĠB OC Ġvector . ĠBoth Ġthe ĠCRC Ġ& Ġ3 C Ġmodels Ġachieve Ġequal Ġperformance Ġto ĠESA Ġat Ġvery Ġfew Ġdimensions Ġ( ĠIN LINE FORM 0 Ġ). ĠD ens ification Ġusing Ġthe ĠCRC Ġmodel Ġembed d ings Ġstill Ġperforms Ġthe Ġbest Ġon Ġboth Ġtasks . ĠInterestingly , Ġthe Ġ3 C Ġmodel Ġgives Ġvery Ġclose ĠF 1 Ġscores Ġto Ġthe ĠCRC Ġmodel Ġat Ġless Ġdimensions Ġ(@ 4 Ġwith ĠSport Ġvs . ĠPolitics , Ġand Ġ@ 60 Ġwith ĠSport Ġvs . ĠReligion ) Ġindicating Ġits Ġcompetitive Ġadvantage Ġwhen Ġcomputational Ġcost Ġis Ġa Ġdecisive Ġcriteria . ĠThe Ġ3 C Ġmodel , Ġstill Ġperforms Ġbetter Ġthan ĠESA , ĠWE ĠIN LINE FORM 1 Ġ, Ġand ĠWE ĠIN LINE FORM 2 Ġon Ġboth Ġtasks . </s> Figure Ġ2 Ġshows ĠF 1 Ġscores Ġof Ġcoarse - gr ained Ġclassification Ġwhen Ġvarying Ġthe Ġ# Ġof ĠB OC Ġdimensions Ġused Ġfor Ġdens ification . ĠThe Ġsame Ġpattern Ġof Ġachieving Ġnear - peak Ġperformance Ġat Ġvery Ġfew Ġdimensions Ġrec ur Ġwith Ġthe ĠCRC Ġ& Ġ3 C Ġmodels . ĠESA Ġusing Ġthe Ġsparse ĠB OC Ġvectors Ġachieves Ġlow ĠF 1 Ġup Ġuntil Ġfew Ġhundred Ġdimensions Ġare Ġconsidered . ĠEven Ġwith Ġthe Ġcostly ĠWE ĠIN LINE FORM 0 Ġand ĠWE ĠIN LINE FORM 1 Ġdens ifications , Ġperformance Ġsometimes Ġdecreases . </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġproposed Ġtwo Ġmodels Ġfor Ġlearning Ġconcept Ġembed d ings Ġbased Ġon Ġthe Ġskip - gram Ġmodel . ĠWe Ġalso Ġproposed Ġan Ġefficient Ġand Ġeffective Ġmechanism Ġfor ĠB OC Ġdens ification Ġwhich Ġoutper formed Ġthe Ġprior Ġproposed Ġdens ification Ġschemes Ġon Ġdat al ess Ġclassification . ĠUnlike Ġthese Ġprior Ġdens ification Ġmechanisms , Ġour Ġmethod Ġscales Ġlin early Ġwith Ġthe Ġ# Ġof Ġthe ĠB OC Ġdimensions . ĠIn Ġaddition , Ġwe Ġdemonstrated Ġthrough Ġthe Ġresults Ġhow Ġthis Ġefficient Ġmechanism Ġallows Ġgenerating Ġhigh Ġquality Ġdense ĠB OC Ġvectors Ġfrom Ġfew Ġconcepts Ġallev iating Ġthe Ġneed Ġof Ġobtaining Ġhundreds Ġof Ġconcepts Ġwhen Ġgenerating Ġthe Ġconcept Ġvector .