Human evaluation for interpretability using the word intrusion test and automated evaluation for interpretability using a semantic category-based approach based on the method and category dataset (SEMCAT).
ABSTRACTIVE
semantic category-based approach
EXTRACTIVE
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon. A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT. Interpretability scores are calculated using Interpretability Score (IS) as given below:
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon. A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT. Interpretability scores are calculated using Interpretability Score (IS) as given below:
1807.07279
Introduction </s> Dist ributed Ġword Ġrepresentations , Ġcommonly Ġreferred Ġto Ġas Ġword Ġembed d ings ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġserve Ġas Ġelementary Ġbuilding Ġblocks Ġin Ġthe Ġcourse Ġof Ġalgorithm Ġdesign Ġfor Ġan Ġexpanding Ġrange Ġof Ġapplications Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġincluding Ġnamed Ġentity Ġrecognition ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġparsing ĠB IB REF 6 Ġ, Ġsentiment Ġanalysis ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġword - sense Ġdis amb ig uation ĠB IB REF 9 Ġ. ĠAlthough Ġthe Ġempirical Ġutility Ġof Ġword Ġembed d ings Ġas Ġan Ġun super vised Ġmethod Ġfor Ġcapturing Ġthe Ġsemantic Ġor Ġsynt actic Ġfeatures Ġof Ġa Ġcertain Ġword Ġas Ġit Ġis Ġused Ġin Ġa Ġgiven Ġlex ical Ġresource Ġis Ġwell - established ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġan Ġunderstanding Ġof Ġwhat Ġthese Ġfeatures Ġmean Ġremains Ġan Ġopen Ġproblem ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġas Ġsuch Ġword Ġembed d ings Ġmostly Ġremain Ġa Ġblack Ġbox . ĠIt Ġis Ġdesirable Ġto Ġbe Ġable Ġto Ġdevelop Ġinsight Ġinto Ġthis Ġblack Ġbox Ġand Ġbe Ġable Ġto Ġinterpret Ġwhat Ġit Ġmeans , Ġwhile Ġretaining Ġthe Ġutility Ġof Ġword Ġembed d ings Ġas Ġsem antically - rich Ġintermediate Ġrepresentations . ĠOther Ġthan Ġthe Ġintrinsic Ġvalue Ġof Ġthis Ġinsight , Ġthis Ġwould Ġnot Ġonly Ġallow Ġus Ġto Ġexplain Ġand Ġunderstand Ġhow Ġalgorithms Ġwork ĠB IB REF 15 Ġ, Ġbut Ġalso Ġset Ġa Ġground Ġthat Ġwould Ġfacilitate Ġthe Ġdesign Ġof Ġnew Ġalgorithms Ġin Ġa Ġmore Ġdeliberate Ġway . </s> Recent Ġapproaches Ġto Ġgenerating Ġword Ġembed d ings Ġ( e . g . ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ) Ġare Ġrooted Ġlingu istically Ġin Ġthe Ġfield Ġof Ġdistributed Ġsemantics ĠB IB REF 16 Ġ, Ġwhere Ġwords Ġare Ġtaken Ġto Ġassume Ġmeaning Ġmainly Ġby Ġtheir Ġdegree Ġof Ġinteraction Ġ( or Ġlack Ġthereof ) Ġwith Ġother Ġwords Ġin Ġthe Ġlex icon ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠUnder Ġthis Ġparadigm , Ġdense , Ġcontinuous Ġvector Ġrepresentations Ġare Ġlearned Ġin Ġan Ġun super vised Ġmanner Ġfrom Ġa Ġlarge Ġcorpus , Ġusing Ġthe Ġword Ġco occ urrence Ġstatistics Ġdirectly Ġor Ġindirectly , Ġand Ġsuch Ġan Ġapproach Ġis Ġshown Ġto Ġresult Ġin Ġvector Ġrepresentations Ġthat Ġmat hematically Ġcapture Ġvarious Ġsemantic Ġand Ġsynt actic Ġrelations Ġbetween Ġwords ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġthe Ġdense Ġnature Ġof Ġthe Ġlearned Ġembed d ings Ġobfusc ate Ġthe Ġdistinct Ġconcepts Ġencoded Ġin Ġthe Ġdifferent Ġdimensions , Ġwhich Ġrenders Ġthe Ġresulting Ġvectors Ġvirtually Ġun interpret able . ĠThe Ġlearned Ġembed d ings Ġmake Ġsense Ġonly Ġin Ġrelation Ġto Ġeach Ġother Ġand Ġtheir Ġspecific Ġdimensions Ġdo Ġnot Ġcarry Ġexplicit Ġinformation Ġthat Ġcan Ġbe Ġinterpreted . ĠHowever , Ġbeing Ġable Ġto Ġinterpret Ġa Ġword Ġembed ding Ġwould Ġilluminate Ġthe Ġsemantic Ġconcepts Ġimplicitly Ġrepresented Ġalong Ġthe Ġvarious Ġdimensions Ġof Ġthe Ġembed ding , Ġand Ġreveal Ġits Ġhidden Ġsemantic Ġstructures . </s> In Ġthe Ġliterature , Ġresearchers Ġtackled Ġinterpret ability Ġproblem Ġof Ġthe Ġword Ġembed d ings Ġusing Ġdifferent Ġapproaches . ĠSeveral Ġresearchers ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġproposed Ġalgorithms Ġbased Ġon Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġapplied Ġto Ġco occ urrence Ġvariant Ġmat rices . ĠOther Ġresearchers Ġsuggested Ġto Ġobtain Ġinterpret able Ġword Ġvectors Ġfrom Ġexisting Ġun interpret able Ġword Ġvectors Ġby Ġapplying Ġsparse Ġcoding ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, Ġby Ġtraining Ġa Ġsparse Ġauto - enc oder Ġto Ġtransform Ġthe Ġembed ding Ġspace ĠB IB REF 24 Ġ, Ġby Ġrotating Ġthe Ġoriginal Ġembed d ings ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġor Ġby Ġapplying Ġtransformations Ġbased Ġon Ġexternal Ġsemantic Ġdatasets ĠB IB REF 27 Ġ. </s> Although Ġthe Ġabove - mentioned Ġapproaches Ġprovide Ġbetter Ġinterpret ability Ġthat Ġis Ġmeasured Ġusing Ġa Ġparticular Ġmethod Ġsuch Ġas Ġword Ġintrusion Ġtest , Ġusually Ġthe Ġimproved Ġinterpret ability Ġcomes Ġwith Ġa Ġcost Ġof Ġperformance Ġin Ġthe Ġbenchmark Ġtests Ġsuch Ġas Ġword Ġsimilarity Ġor Ġword Ġanalogy . ĠOne Ġpossible Ġexplanation Ġfor Ġthis Ġperformance Ġdecrease Ġis Ġthat Ġthe Ġproposed Ġtransformations Ġfrom Ġthe Ġoriginal Ġembed ding Ġspace Ġdistort Ġthe Ġunderlying Ġsemantic Ġstructure Ġconstructed Ġby Ġthe Ġoriginal Ġembed ding Ġalgorithm . ĠTherefore , Ġit Ġcan Ġbe Ġclaimed Ġthat Ġa Ġmethod Ġthat Ġlearns Ġdense Ġand Ġinterpret able Ġword Ġembed d ings Ġwithout Ġinflicting Ġany Ġdamage Ġto Ġthe Ġunderlying Ġsemantic Ġlearning Ġmechanism Ġis Ġthe Ġkey Ġto Ġachieve Ġboth Ġhigh Ġperforming Ġand Ġinterpret able Ġword Ġembed d ings . </s> Especially Ġafter Ġthe Ġintroduction Ġof Ġthe Ġword 2 vec Ġalgorithm Ġby ĠMik ol ov ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġthere Ġhas Ġbeen Ġa Ġgrowing Ġinterest Ġin Ġalgorithms Ġthat Ġgenerate Ġimproved Ġword Ġrepresentations Ġunder Ġsome Ġperformance Ġmetric . ĠSignificant Ġeffort Ġis Ġspent Ġon Ġappropriately Ġmodifying Ġthe Ġobjective Ġfunctions Ġof Ġthe Ġalgorithms Ġin Ġorder Ġto Ġincorporate Ġknowledge Ġfrom Ġexternal Ġresources , Ġwith Ġthe Ġpurpose Ġof Ġincreasing Ġthe Ġperformance Ġof Ġthe Ġresulting Ġword Ġrepresentations ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ. ĠInspired Ġby Ġthe Ġline Ġof Ġwork Ġreported Ġin Ġthese Ġstudies , Ġwe Ġpropose Ġto Ġuse Ġmodified Ġobjective Ġfunctions Ġfor Ġa Ġdifferent Ġpurpose : Ġlearning Ġmore Ġinterpret able Ġdense Ġword Ġembed d ings . ĠBy Ġdoing Ġthis , Ġwe Ġaim Ġto Ġincorporate Ġsemantic Ġinformation Ġfrom Ġan Ġexternal Ġlex ical Ġresource Ġinto Ġthe Ġword Ġembed ding Ġso Ġthat Ġthe Ġembed ding Ġdimensions Ġare Ġaligned Ġalong Ġpred efined Ġconcepts . ĠThis Ġalignment Ġis Ġachieved Ġby Ġintroducing Ġa Ġmodification Ġto Ġthe Ġembed ding Ġlearning Ġprocess . ĠIn Ġour Ġproposed Ġmethod , Ġwhich Ġis Ġbuilt Ġon Ġtop Ġof Ġthe ĠGl o Ve Ġalgorithm ĠB IB REF 2 Ġ, Ġthe Ġcost Ġfunction Ġfor Ġany Ġone Ġof Ġthe Ġwords Ġof Ġconcept Ġword - groups Ġis Ġmodified Ġby Ġthe Ġintroduction Ġof Ġan Ġadditive Ġterm Ġto Ġthe Ġcost Ġfunction . ĠEach Ġembed ding Ġvector Ġdimension Ġis Ġfirst Ġassociated Ġwith Ġa Ġconcept . ĠFor Ġa Ġword Ġbelonging Ġto Ġany Ġone Ġof Ġthe Ġword - groups Ġrepresenting Ġthese Ġconcepts , Ġthe Ġmodified Ġcost Ġterm Ġfavors Ġan Ġincrease Ġfor Ġthe Ġvalue Ġof Ġthis Ġword 's Ġembed ding Ġvector Ġdimension Ġcorresponding Ġto Ġthe Ġconcept Ġthat Ġthe Ġparticular Ġword Ġbelongs Ġto . ĠFor Ġwords Ġthat Ġdo Ġnot Ġbelong Ġto Ġany Ġone Ġof Ġthe Ġword - groups , Ġthe Ġcost Ġterm Ġis Ġleft Ġuntouched . ĠSpecifically , ĠRog et 's ĠThe s aurus ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġis Ġused Ġto Ġderive Ġthe Ġconcepts Ġand Ġconcept Ġword - groups Ġto Ġbe Ġused Ġas Ġthe Ġexternal Ġlex ical Ġresource Ġfor Ġour Ġproposed Ġmethod . ĠWe Ġquant itatively Ġdemonstrate Ġthe Ġincrease Ġin Ġinterpret ability Ġby Ġusing Ġthe Ġmeasure Ġgiven Ġin ĠB IB REF 27 Ġ, ĠB IB REF 40 Ġas Ġwell Ġas Ġdemonstrating Ġqualitative Ġresults . ĠWe Ġalso Ġshow Ġthat Ġthe Ġsemantic Ġstructure Ġof Ġthe Ġoriginal Ġembed ding Ġhas Ġnot Ġbeen Ġharmed Ġin Ġthe Ġprocess Ġsince Ġthere Ġis Ġno Ġperformance Ġloss Ġwith Ġstandard Ġword - similar ity Ġor Ġword - an alogy Ġtests . </s> The Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection ĠSEC REF 2 Ġ, Ġwe Ġdiscuss Ġprevious Ġstudies Ġrelated Ġto Ġour Ġwork Ġunder Ġtwo Ġmain Ġcategories : Ġinterpret ability Ġof Ġword Ġembed d ings Ġand Ġjoint - learning Ġframeworks Ġwhere Ġthe Ġobjective Ġfunction Ġis Ġmodified . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġwe Ġpresent Ġthe Ġproblem Ġframework Ġand Ġprovide Ġthe Ġformulation Ġwithin Ġthe ĠGl o Ve ĠB IB REF 2 Ġalgorithm Ġsetting . ĠIn ĠSection ĠSEC REF 4 Ġwhere Ġour Ġapproach Ġis Ġproposed , Ġwe Ġmotivate Ġand Ġdevelop Ġa Ġmodification Ġto Ġthe Ġoriginal Ġobjective Ġfunction Ġwith Ġthe Ġaim Ġof Ġincreasing Ġrepresentation Ġinterpret ability . ĠIn ĠSection ĠSEC REF 5 Ġ, Ġexperimental Ġresults Ġare Ġprovided Ġand Ġthe Ġproposed Ġmethod Ġis Ġquant itatively Ġand Ġqual itatively Ġevaluated . ĠAdditionally , Ġin ĠSection ĠSEC REF 5 Ġ, Ġresults Ġdemonstrating Ġthe Ġextent Ġto Ġwhich Ġthe Ġoriginal Ġsemantic Ġstructure Ġof Ġthe Ġembed ding Ġspace Ġis Ġaffected Ġare Ġpresented Ġby Ġusing Ġword - an alogy Ġand Ġword - similar ity Ġtests . ĠWe Ġconclude Ġthe Ġpaper Ġin ĠSection ĠSEC REF 6 Ġ. </s> Related ĠWork </s> Method ologically , Ġour Ġwork Ġis Ġrelated Ġto Ġprior Ġstudies Ġthat Ġaim Ġto Ġobtain ĠâĢ ľ impro ved âĢ Ŀ Ġword Ġembed d ings Ġusing Ġexternal Ġlex ical Ġresources , Ġunder Ġsome Ġperformance Ġmetric . ĠPrevious Ġwork Ġin Ġthis Ġarea Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġmain Ġcategories : Ġworks Ġthat Ġi ) Ġmodify Ġthe Ġword Ġembed ding Ġlearning Ġalgorithm Ġto Ġincorporate Ġlex ical Ġinformation , Ġii ) Ġoperate Ġon Ġpre - trained Ġembed d ings Ġwith Ġa Ġpost - processing Ġstep . </s> Among Ġworks Ġthat Ġfollow Ġthe Ġfirst Ġapproach , ĠB IB REF 28 Ġextend Ġthe ĠSkip - G ram Ġmodel Ġby Ġincorporating Ġthe Ġword Ġsimilarity Ġrelations Ġextracted Ġfrom Ġthe ĠPar aph rase ĠDatabase Ġ( P PD B ) Ġand ĠWord Net ĠB IB REF 29 Ġ, Ġinto Ġthe ĠSkip - G ram Ġpredictive Ġmodel Ġas Ġan Ġadditional Ġcost Ġterm . ĠIn ĠB IB REF 30 Ġ, Ġthe Ġauthors Ġextend Ġthe ĠCB OW Ġmodel Ġby Ġconsidering Ġtwo Ġtypes Ġof Ġsemantic Ġinformation , Ġtermed Ġrelational Ġand Ġcateg orical , Ġto Ġbe Ġincorporated Ġinto Ġthe Ġembed d ings Ġduring Ġtraining . ĠFor Ġthe Ġformer Ġtype Ġof Ġsemantic Ġinformation , Ġthe Ġauthors Ġpropose Ġthe Ġlearning Ġof Ġexplicit Ġvectors Ġfor Ġthe Ġdifferent Ġrelations Ġextracted Ġfrom Ġa Ġsemantic Ġlex icon Ġsuch Ġthat Ġthe Ġword Ġpairs Ġthat Ġsatisfy Ġthe Ġsame Ġrelation Ġare Ġdistributed Ġmore Ġhom ogene ously . ĠFor Ġthe Ġlatter , Ġthe Ġauthors Ġmodify Ġthe Ġlearning Ġobjective Ġsuch Ġthat Ġsome Ġweighted Ġaverage Ġdistance Ġis Ġminimized Ġfor Ġwords Ġunder Ġthe Ġsame Ġsemantic Ġcategory . ĠIn ĠB IB REF 31 Ġ, Ġthe Ġauthors Ġrepresent Ġthe Ġsyn ony my Ġand Ġhyper ny my - hyp ony my Ġrelations Ġin Ġterms Ġof Ġinequality Ġconstraints , Ġwhere Ġthe Ġpair wise Ġsimilarity Ġrankings Ġover Ġword Ġtriple ts Ġare Ġforced Ġto Ġfollow Ġan Ġorder Ġextracted Ġfrom Ġa Ġlex ical Ġresource . ĠFollowing Ġtheir Ġextraction Ġfrom ĠWord Net , Ġthe Ġauthors Ġimpose Ġthese Ġconstraints Ġin Ġthe Ġform Ġof Ġan Ġadditive Ġcost Ġterm Ġto Ġthe ĠSkip - G ram Ġformulation . ĠFinally , ĠB IB REF 32 Ġbuilds Ġon Ġtop Ġof Ġthe ĠGl o Ve Ġalgorithm Ġby Ġintroducing Ġa Ġregular ization Ġterm Ġto Ġthe Ġobjective Ġfunction Ġthat Ġencourages Ġthe Ġvector Ġrepresentations Ġof Ġsimilar Ġwords Ġas Ġdictated Ġby ĠWord Net Ġto Ġbe Ġsimilar Ġas Ġwell . </s> Turn ing Ġour Ġattention Ġto Ġthe Ġpost - processing Ġapproach Ġfor Ġenrich ing Ġword Ġembed d ings Ġwith Ġexternal Ġlex ical Ġknowledge , ĠB IB REF 33 Ġhas Ġintroduced Ġthe Ġretro fitting Ġalgorithm Ġthat Ġacts Ġon Ġpre - trained Ġembed d ings Ġsuch Ġas ĠSkip - G ram Ġor ĠGl o Ve . ĠThe Ġauthors Ġpropose Ġan Ġobjective Ġfunction Ġthat Ġaims Ġto Ġbalance Ġout Ġthe Ġsemantic Ġinformation Ġcaptured Ġin Ġthe Ġpre - trained Ġembed d ings Ġwith Ġthe Ġconstraints Ġderived Ġfrom Ġlex ical Ġresources Ġsuch Ġas ĠWord Net , ĠP PD B Ġand ĠFrame Net . ĠOne Ġof Ġthe Ġmodels Ġproposed Ġin ĠB IB REF 34 Ġextends Ġthe Ġretro fitting Ġapproach Ġto Ġincorporate Ġthe Ġword Ġsense Ġinformation Ġfrom ĠWord Net . ĠSimilarly , ĠB IB REF 35 Ġcreates Ġmulti - sense Ġembed d ings Ġby Ġgathering Ġthe Ġword Ġsense Ġinformation Ġfrom Ġa Ġlex ical Ġresource Ġand Ġlearning Ġto Ġdecom pose Ġthe Ġpre - trained Ġembed d ings Ġinto Ġa Ġconve x Ġcombination Ġof Ġsense Ġembed d ings . ĠIn ĠB IB REF 36 Ġ, Ġthe Ġauthors Ġfocus Ġon Ġimproving Ġword Ġembed d ings Ġfor Ġcapturing Ġword Ġsimilarity , Ġas Ġopposed Ġto Ġmere Ġrelated ness . ĠTo Ġthis Ġend , Ġthey Ġintroduce Ġthe Ġcounter - fitting Ġtechnique Ġwhich Ġacts Ġon Ġthe Ġinput Ġword Ġvectors Ġsuch Ġthat Ġsynonymous Ġwords Ġare Ġattracted Ġto Ġone Ġanother Ġwhereas Ġan ton ym ous Ġwords Ġare Ġrepe lled , Ġwhere Ġthe Ġsyn ony my - ant ony my Ġrelations Ġare Ġextracted Ġfrom Ġa Ġlex ical Ġresource . ĠMore Ġrecently , Ġthe ĠAT TR ACT - REP EL Ġalgorithm Ġproposed Ġby ĠB IB REF 37 Ġimproves Ġon Ġcounter - fitting Ġby Ġa Ġformulation Ġwhich Ġimp arts Ġthe Ġword Ġvectors Ġwith Ġexternal Ġlex ical Ġinformation Ġin Ġmini - bat ches . </s> Most Ġof Ġthe Ġstudies Ġdiscussed Ġabove Ġ( ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ) Ġreport Ġperformance Ġimprovements Ġin Ġbenchmark Ġtests Ġsuch Ġas Ġword Ġsimilarity Ġor Ġword Ġanalogy , Ġwhile ĠB IB REF 29 Ġuses Ġa Ġdifferent Ġanalysis Ġmethod Ġ( mean Ġreciprocal Ġrank ). ĠIn Ġsum , Ġthe Ġliterature Ġis Ġrich Ġwith Ġstudies Ġaiming Ġto Ġobtain Ġword Ġembed d ings Ġthat Ġperform Ġbetter Ġunder Ġspecific Ġperformance Ġmetrics . ĠHowever , Ġless Ġattention Ġhas Ġbeen Ġdirected Ġto Ġthe Ġissue Ġof Ġinterpret ability Ġof Ġthe Ġword Ġembed d ings . ĠIn Ġthe Ġliterature , Ġthe Ġproblem Ġof Ġinterpret ability Ġhas Ġbeen Ġtackled Ġusing Ġdifferent Ġapproaches . ĠB IB REF 19 Ġproposed Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġfor Ġlearning Ġsparse , Ġinterpret able Ġword Ġvectors Ġfrom Ġco - occ urrence Ġvariant Ġmat rices Ġwhere Ġthe Ġresulting Ġvector Ġspace Ġis Ġcalled Ġnon - negative Ġsparse Ġembed d ign s Ġ( NN SE ). ĠHowever , Ġsince ĠNM F Ġmethods Ġrequire Ġmaintaining Ġa Ġglobal Ġmatrix Ġfor Ġlearning , Ġthey Ġsuffer Ġfrom Ġmemory Ġand Ġscale Ġissue . ĠThis Ġproblem Ġhas Ġbeen Ġaddressed Ġin ĠB IB REF 20 Ġwhere Ġan Ġonline Ġmethod Ġof Ġlearning Ġinterpret able Ġword Ġembed d ings Ġfrom Ġcorpor a Ġusing Ġa Ġmodified Ġversion Ġof Ġskip - gram Ġmodel ĠB IB REF 0 Ġis Ġproposed . ĠAs Ġa Ġdifferent Ġapproach , ĠB IB REF 21 Ġcombined Ġtext - based Ġsimilarity Ġinformation Ġamong Ġwords Ġwith Ġbrain Ġactivity Ġbased Ġsimilarity Ġinformation Ġto Ġimprove Ġinterpret ability Ġusing Ġjoint Ġnon - negative Ġsparse Ġembed ding Ġ( J NN SE ). </s> A Ġcommon Ġalternative Ġapproach Ġfor Ġlearning Ġinterpret able Ġembed d ings Ġis Ġto Ġlearn Ġtransformations Ġthat Ġmap Ġpre - trained Ġstate - of - the - art Ġembed d ings Ġto Ġnew Ġinterpret able Ġsemantic Ġspaces . ĠTo Ġobtain Ġsparse , Ġhigher Ġdimensional Ġand Ġmore Ġinterpret able Ġvector Ġspaces , ĠB IB REF 22 Ġand ĠB IB REF 23 Ġuse Ġsparse Ġcoding Ġon Ġconventional Ġdense Ġword Ġembed d ings . ĠHowever , Ġthese Ġmethods Ġlearn Ġthe Ġprojection Ġvectors Ġthat Ġare Ġused Ġfor Ġthe Ġtransformation Ġfrom Ġthe Ġword Ġembed d ings Ġwithout Ġsupervision . ĠFor Ġthis Ġreason , Ġlabels Ġdescribing Ġthe Ġcorresponding Ġsemantic Ġcategories Ġcannot Ġbe Ġprovided . ĠAn Ġalternative Ġapproach Ġwas Ġproposed Ġin ĠB IB REF 25 Ġ, Ġwhere Ġorth og onal Ġtransformations Ġwere Ġutilized Ġto Ġincrease Ġinterpret ability Ġwhile Ġpreserving Ġthe Ġperformance Ġof Ġthe Ġunderlying Ġembed ding . ĠHowever , ĠB IB REF 25 Ġhas Ġalso Ġshown Ġthat Ġtotal Ġinterpret ability Ġof Ġan Ġembed ding Ġis Ġkept Ġconstant Ġunder Ġany Ġorth og onal Ġtransformation Ġand Ġit Ġcan Ġonly Ġbe Ġredistributed Ġacross Ġthe Ġdimensions . ĠR otation Ġalgorithms Ġbased Ġon Ġexplor atory Ġfactor Ġanalysis Ġ( E FA ) Ġto Ġpreserve Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġword Ġembed d ings Ġwhile Ġimproving Ġtheir Ġinterpret ability Ġwas Ġproposed Ġin ĠB IB REF 26 Ġ. ĠB IB REF 24 Ġproposed Ġto Ġdeploy Ġa Ġsparse Ġauto - enc oder Ġusing Ġpre - trained Ġdense Ġword Ġembed d ings Ġto Ġimprove Ġinterpret ability . ĠMore Ġdetailed Ġinvestigation Ġof Ġsemantic Ġstructure Ġand Ġinterpret ability Ġof Ġword Ġembed d ings Ġcan Ġbe Ġfound Ġin ĠB IB REF 27 Ġ, Ġwhere Ġa Ġmetric Ġwas Ġproposed Ġto Ġquant itatively Ġmeasure Ġthe Ġdegree Ġof Ġinterpret ability Ġalready Ġpresent Ġin Ġthe Ġembed ding Ġvector Ġspaces . </s> Previous Ġworks Ġon Ġinterpret ability Ġmentioned Ġabove , Ġexcept ĠB IB REF 21 Ġ, ĠB IB REF 27 Ġand Ġour Ġproposed Ġmethod , Ġdo Ġnot Ġneed Ġexternal Ġresources , Ġutilization Ġof Ġwhich Ġhas Ġboth Ġadvantages Ġand Ġdisadvantages . ĠMethods Ġthat Ġdo Ġnot Ġuse Ġexternal Ġresources Ġrequire Ġfewer Ġresources Ġbut Ġthey Ġalso Ġlack Ġthe Ġaid Ġof Ġinformation Ġextracted Ġfrom Ġthese Ġresources . </s> Problem ĠDescription </s> For Ġthe Ġtask Ġof Ġun super vised Ġword Ġembed ding Ġextraction , Ġwe Ġoperate Ġon Ġa Ġdiscrete Ġcollection Ġof Ġlex ical Ġunits Ġ( words ) ĠIN LINE FORM 0 Ġthat Ġis Ġpart Ġof Ġan Ġinput Ġcorpus ĠIN LINE FORM 1 Ġ, Ġwith Ġnumber Ġof Ġtokens ĠIN LINE FORM 2 Ġ, Ġsourced Ġfrom Ġa Ġvocabulary ĠIN LINE FORM 3 Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠIn Ġthe Ġsetting Ġof Ġdistribution al Ġsemantics , Ġthe Ġobjective Ġof Ġa Ġword Ġembed ding Ġalgorithm Ġis Ġto Ġmaximize Ġsome Ġaggregate Ġutility Ġover Ġthe Ġentire Ġcorpus Ġso Ġthat Ġsome Ġmeasure Ġof ĠâĢ ľ cl osen ess âĢ Ŀ Ġis Ġmaxim ized Ġfor Ġpairs Ġof Ġvector Ġrepresentations ĠIN LINE FORM 14 Ġfor Ġwords Ġwhich , Ġon Ġthe Ġaverage , Ġappear Ġin Ġproximity Ġto Ġone Ġanother . ĠIn Ġthe ĠGl o Ve Ġalgorithm ĠB IB REF 2 Ġ, Ġwhich Ġwe Ġbase Ġour Ġimprovements Ġupon , Ġthe Ġfollowing Ġobjective Ġfunction Ġis Ġconsidered : ĠDIS PLAY FORM 0 </s> In Ġ( ĠEQ REF 6 Ġ), ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġstand Ġfor Ġword Ġand Ġcontext Ġvector Ġrepresentations , Ġrespectively , Ġfor Ġwords ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġwhile ĠIN LINE FORM 4 Ġrepresents Ġthe Ġ( possibly Ġweighted ) Ġco occ urrence Ġcount Ġfor Ġthe Ġword Ġpair ĠIN LINE FORM 5 Ġ. ĠInt uitive ly , Ġ( ĠEQ REF 6 Ġ) Ġrepresents Ġthe Ġrequirement Ġthat Ġif Ġsome Ġword ĠIN LINE FORM 6 Ġoccurs Ġoften Ġenough Ġin Ġthe Ġcontext Ġ( or Ġvicinity ) Ġof Ġanother Ġword ĠIN LINE FORM 7 Ġ, Ġthen Ġthe Ġcorresponding Ġword Ġrepresentations Ġshould Ġhave Ġa Ġlarge Ġenough Ġinner Ġproduct Ġin Ġkeeping Ġwith Ġtheir Ġlarge ĠIN LINE FORM 8 Ġvalue , Ġup Ġto Ġsome Ġbias Ġterms ĠIN LINE FORM 9 Ġ; Ġand Ġvice Ġversa . ĠIN LINE FORM 10 Ġin Ġ( ĠEQ REF 6 Ġ) Ġis Ġused Ġas Ġa Ġdiscount ing Ġfactor Ġthat Ġprohibits Ġrare Ġco occ ur rences Ġfrom Ġdisproportionately Ġinfluencing Ġthe Ġresulting Ġembed d ings . </s> The Ġobjective Ġ( ĠEQ REF 6 Ġ) Ġis Ġminimized Ġusing Ġsto ch astic Ġgradient Ġdescent Ġby Ġiter ating Ġover Ġthe Ġmatrix Ġof Ġco occ urrence Ġrecords ĠIN LINE FORM 0 Ġ. ĠIn Ġthe ĠGl o Ve Ġalgorithm , Ġfor Ġa Ġgiven Ġword ĠIN LINE FORM 1 Ġ, Ġthe Ġfinal Ġword Ġrepresentation Ġis Ġtaken Ġto Ġbe Ġthe Ġaverage Ġof Ġthe Ġtwo Ġintermediate Ġvector Ġrepresentations Ġobtained Ġfrom Ġ( ĠEQ REF 6 Ġ); Ġi . e , ĠIN LINE FORM 2 Ġ. ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdetail Ġthe Ġenhancements Ġmade Ġto Ġ( ĠEQ REF 6 Ġ) Ġfor Ġthe Ġpurposes Ġof Ġenhanced Ġinterpret ability , Ġusing Ġthe Ġaforementioned Ġframework Ġas Ġour Ġbasis . </s> Imp art ing ĠInterpret ability </s> Our Ġapproach Ġfalls Ġinto Ġa Ġjoint - learning Ġframework Ġwhere Ġthe Ġdistribution al Ġinformation Ġextracted Ġfrom Ġthe Ġcorpus Ġis Ġallowed Ġto Ġfuse Ġwith Ġthe Ġexternal Ġlex icon - based Ġinformation . ĠWord - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus Ġare Ġdirectly Ġmapped Ġto Ġindividual Ġdimensions Ġof Ġword Ġembed d ings . ĠSpecifically , Ġthe Ġvector Ġrepresentations Ġof Ġwords Ġthat Ġbelong Ġto Ġa Ġparticular Ġgroup Ġare Ġencouraged Ġto Ġhave Ġdeliberately Ġincreased Ġvalues Ġin Ġa Ġparticular Ġdimension Ġthat Ġcorresponds Ġto Ġthe Ġword - group Ġunder Ġconsideration . ĠThis Ġcan Ġbe Ġachieved Ġby Ġmodifying Ġthe Ġobjective Ġfunction Ġof Ġthe Ġembed ding Ġalgorithm Ġto Ġpartially Ġinfluence Ġvector Ġrepresentation Ġdistributions Ġacross Ġtheir Ġdimensions Ġover Ġan Ġinput Ġvocabulary . ĠTo Ġdo Ġthis , Ġwe Ġpropose Ġthe Ġfollowing Ġmodification Ġto Ġthe ĠGl o Ve Ġobjective Ġin Ġ( ĠEQ REF 6 Ġ): Ġr Cl ĠJ Ġ= Ġi , j = 1 V Ġf ( X ij )[ Ġ( wi Tw j Ġ+ Ġbi Ġ+ Ġb j Ġ- X ij ) 2 </s> + Ġk ( l = 1 D ĠIN LINE FORM 0 Ġi Fl Ġg ( wi , l ) Ġ+ Ġl = 1 D ĠIN LINE FORM 1 Ġj ĠFl Ġg ( w j , l ) Ġ) Ġ]. ĠIn Ġ( ĠSEC REF 4 Ġ), ĠIN LINE FORM 2 Ġdenotes Ġthe Ġindices Ġfor Ġthe Ġelements Ġof Ġthe ĠIN LINE FORM 3 Ġth Ġconcept Ġword - group Ġwhich Ġwe Ġwish Ġto Ġassign Ġin Ġthe Ġvector Ġdimension ĠIN LINE FORM 4 Ġ. ĠThe Ġobjective Ġ( ĠSEC REF 4 Ġ) Ġis Ġdesigned Ġas Ġa Ġmixture Ġof Ġtwo Ġindividual Ġcost Ġterms : Ġthe Ġoriginal ĠGl o Ve Ġcost Ġterm Ġalong Ġwith Ġa Ġsecond Ġterm Ġthat Ġencourages Ġembed ding Ġvectors Ġof Ġa Ġgiven Ġconcept Ġword - group Ġto Ġachieve Ġdeliberately Ġincreased Ġvalues Ġalong Ġan Ġassociated Ġdimension ĠIN LINE FORM 5 Ġ. ĠThe Ġrelative Ġweight Ġof Ġthe Ġsecond Ġterm Ġis Ġcontrolled Ġby Ġthe Ġparameter ĠIN LINE FORM 6 Ġ. ĠThe Ġsimultaneous Ġminim ization Ġof Ġboth Ġobjectives Ġensures Ġthat Ġwords Ġthat Ġare Ġsimilar Ġto , Ġbut Ġnot Ġincluded Ġin , Ġone Ġof Ġthese Ġconcept Ġword - groups Ġare Ġalso ĠâĢ ľ n ud ged âĢ Ŀ Ġtowards Ġthe Ġassociated Ġdimension ĠIN LINE FORM 7 Ġ. ĠThe Ġtrained Ġword Ġvectors Ġare Ġthus Ġencouraged Ġto Ġform Ġa Ġdistribution Ġwhere Ġthe Ġindividual Ġvector Ġdimensions Ġalign Ġwith Ġcertain Ġsemantic Ġconcepts Ġrepresented Ġby Ġa Ġcollection Ġof Ġconcept Ġword - groups , Ġone Ġassigned Ġto Ġeach Ġvector Ġdimension . ĠTo Ġfacilitate Ġthis Ġbehaviour , Ġ( ĠSEC REF 4 Ġ) Ġintroduces Ġa Ġmon ot one Ġdecreasing Ġfunction ĠIN LINE FORM 8 Ġdefined Ġas ĠIN LINE FORM 9 </s> which Ġserves Ġto Ġincrease Ġthe Ġtotal Ġcost Ġincurred Ġif Ġthe Ġvalue Ġof Ġthe ĠIN LINE FORM 0 Ġth Ġdimension Ġfor Ġthe Ġtwo Ġvector Ġrepresentations ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġfor Ġa Ġconcept Ġword ĠIN LINE FORM 3 Ġwith ĠIN LINE FORM 4 Ġfails Ġto Ġbe Ġlarge Ġenough . ĠIN LINE FORM 5 Ġis Ġalso Ġshown Ġin ĠFig . ĠFIG REF 7 Ġ. </s> The Ġobjective Ġ( ĠSEC REF 4 Ġ) Ġis Ġminimized Ġusing Ġsto ch astic Ġgradient Ġdescent Ġover Ġthe Ġco occ urrence Ġrecords ĠIN LINE FORM 0 Ġ. ĠInt uitive ly , Ġthe Ġterms Ġadded Ġto Ġ( ĠSEC REF 4 Ġ) Ġin Ġcomparison Ġwith Ġ( ĠEQ REF 6 Ġ) Ġintroduce Ġthe Ġeffect Ġof Ġselectively Ġapplying Ġa Ġpositive Ġstep - type Ġinput Ġto Ġthe Ġoriginal Ġdescent Ġupdates Ġof Ġ( ĠEQ REF 6 Ġ) Ġfor Ġconcept Ġwords Ġalong Ġtheir Ġrespective Ġvector Ġdimensions , Ġwhich Ġinfluences Ġthe Ġdimension Ġvalue Ġin Ġthe Ġpositive Ġdirection . ĠThe Ġparameter ĠIN LINE FORM 1 Ġin Ġ( ĠSEC REF 4 Ġ) Ġallows Ġfor Ġthe Ġadjustment Ġof Ġthe Ġmagnitude Ġof Ġthis Ġinfluence Ġas Ġneeded . </s> In Ġthe Ġnext Ġsection , Ġwe Ġdemonstrate Ġthe Ġfeasibility Ġof Ġthis Ġapproach Ġby Ġexperiments Ġwith Ġan Ġexample Ġcollection Ġof Ġconcept Ġword - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus . </s> Exper iments Ġand ĠResults </s> We Ġfirst Ġidentified Ġ300 Ġconcepts , Ġone Ġfor Ġeach Ġdimension Ġof Ġthe Ġ300 - dimensional Ġvector Ġrepresentation , Ġby Ġemploying ĠRog et 's ĠThe s aurus . ĠThis Ġthe s aurus Ġfollows Ġa Ġtree Ġstructure Ġwhich Ġstarts Ġwith Ġa ĠRoot Ġnode Ġthat Ġcontains Ġall Ġthe Ġwords Ġand Ġphrases Ġin Ġthe Ġthe s aurus . ĠThe Ġroot Ġnode Ġis Ġsuccess ively Ġsplit Ġinto ĠClasses Ġand ĠSections , Ġwhich Ġare Ġthen Ġ( option ally ) Ġsplit Ġinto ĠSub sections Ġof Ġvarious Ġdepths , Ġfinally Ġending Ġin ĠCategories , Ġwhich Ġconstitute Ġthe Ġsmallest Ġunit Ġof Ġword / phrase Ġcollections Ġin Ġthe Ġstructure . ĠThe Ġactual Ġwords Ġand Ġphrases Ġdescend Ġfrom Ġthese ĠCategories , Ġand Ġmake Ġup Ġthe Ġleaves Ġof Ġthe Ġtree Ġstructure . ĠWe Ġnote Ġthat Ġa Ġgiven Ġword Ġtypically Ġappears Ġin Ġmultiple Ġcategories Ġcorresponding Ġto Ġthe Ġdifferent Ġsenses Ġof Ġthe Ġword . ĠWe Ġconstructed Ġconcept Ġword - groups Ġfrom ĠRog et 's ĠThe s aurus Ġas Ġfollows : ĠWe Ġfirst Ġfiltered Ġout Ġthe Ġmulti - word Ġphrases Ġand Ġthe Ġrelatively Ġobscure Ġterms Ġfrom Ġthe Ġthe s aurus . ĠThe Ġobscure Ġterms Ġwere Ġidentified Ġby Ġchecking Ġthem Ġagainst Ġa Ġvocabulary Ġextracted Ġfrom ĠWikipedia . ĠWe Ġthen Ġobtained Ġ300 Ġword - groups Ġas Ġthe Ġresult Ġof Ġa Ġpartition ing Ġoperation Ġapplied Ġto Ġthe Ġsubt ree Ġthat Ġends Ġwith Ġcategories Ġas Ġits Ġleaves . ĠThe Ġpartition Ġboundaries , Ġhence Ġthe Ġresulting Ġword - groups , Ġcan Ġbe Ġchosen Ġin Ġmany Ġdifferent Ġways . ĠIn Ġour Ġproposed Ġapproach , Ġwe Ġhave Ġchosen Ġto Ġdetermine Ġthis Ġpartition ing Ġby Ġtravers ing Ġthis Ġtree Ġstructure Ġfrom Ġthe Ġroot Ġnode Ġin Ġbreadth - first Ġorder , Ġand Ġby Ġemploying Ġa Ġparameter ĠIN LINE FORM 0 Ġfor Ġthe Ġmaximum Ġsize Ġof Ġa Ġnode . ĠHere , Ġthe Ġsize Ġof Ġa Ġnode Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġunique Ġwords Ġthat Ġever - desc end Ġfrom Ġthat Ġnode . ĠDuring Ġthe Ġtravers al , Ġif Ġthe Ġsize Ġof Ġa Ġgiven Ġnode Ġis Ġless Ġthan Ġthis Ġthreshold , Ġwe Ġdesignate Ġthe Ġwords Ġthat Ġultimately Ġdescend Ġfrom Ġthat Ġnode Ġas Ġa Ġconcept Ġword - group . ĠOtherwise , Ġif Ġthe Ġnode Ġhas Ġchildren , Ġwe Ġdiscard Ġthe Ġnode , Ġand Ġqueue Ġup Ġall Ġits Ġchildren Ġfor Ġfurther Ġconsideration . ĠIf Ġthis Ġnode Ġdoes Ġnot Ġhave Ġany Ġchildren , Ġon Ġthe Ġother Ġhand , Ġthe Ġnode Ġis Ġtrunc ated Ġto ĠIN LINE FORM 1 Ġelements Ġwith Ġthe Ġhighest Ġfrequency - r anks , Ġand Ġthe Ġresulting Ġwords Ġare Ġdesignated Ġas Ġa Ġconcept Ġword - group . ĠWe Ġnote Ġthat Ġthe Ġchoice Ġof ĠIN LINE FORM 2 Ġgreatly Ġaffects Ġthe Ġresulting Ġcollection Ġof Ġword - groups : ĠEx cess ively Ġlarge Ġvalues Ġresult Ġin Ġfew Ġword - groups Ġthat Ġgreatly Ġoverlap Ġwith Ġone Ġanother , Ġwhile Ġoverly Ġsmall Ġvalues Ġresult Ġin Ġnumerous Ġtiny Ġword - groups Ġthat Ġfail Ġto Ġadequately Ġrepresent Ġa Ġconcept . ĠWe Ġexperiment ally Ġdetermined Ġthat Ġa ĠIN LINE FORM 3 Ġvalue Ġof Ġ45 2 Ġresults Ġin Ġthe Ġmost Ġhealthy Ġnumber Ġof Ġrelatively Ġlarge Ġword - groups Ġ( 113 Ġgroups Ġwith Ġsize ĠIN LINE FORM 4 Ġ100 ), Ġwhile Ġyielding Ġa Ġpreferably Ġsmall Ġoverlap Ġamongst Ġthe Ġresulting Ġword - groups Ġ( with Ġaverage Ġoverlap Ġsize Ġnot Ġexceeding Ġ3 Ġwords ). ĠA Ġtotal Ġof Ġ5 66 Ġword - groups Ġwere Ġthus Ġobtained . Ġ259 Ġsmallest Ġword - groups Ġ( with Ġsize ĠIN LINE FORM 5 Ġ38 ) Ġwere Ġdiscarded Ġto Ġbring Ġdown Ġthe Ġnumber Ġof Ġword - groups Ġto Ġ307 . ĠOut Ġof Ġthese , Ġ7 Ġgroups Ġwith Ġthe Ġlowest Ġmedian Ġfrequency - rank Ġwere Ġfurther Ġdiscarded , Ġwhich Ġyields Ġthe Ġfinal Ġ300 Ġconcept Ġword - groups Ġused Ġin Ġthe Ġexperiments . ĠWe Ġpresent Ġsome Ġof Ġthe Ġresulting Ġword - groups Ġin ĠTable ĠT AB REF 9 Ġ. </s> By Ġusing Ġthe Ġconcept Ġword - groups , Ġwe Ġhave Ġtrained Ġthe ĠGl o Ve Ġalgorithm Ġwith Ġthe Ġproposed Ġmodification Ġgiven Ġin ĠSection ĠSEC REF 4 Ġon Ġa Ġsnapshot Ġof ĠEnglish ĠWikipedia Ġmeasuring Ġ8 GB Ġin Ġsize , Ġwith Ġthe Ġstop - words Ġfiltered Ġout . ĠUsing Ġthe Ġparameters Ġgiven Ġin ĠTable ĠT AB REF 10 Ġ, Ġthis Ġresulted Ġin Ġa Ġvocabulary Ġsize Ġof Ġ287 , 8 47 . ĠFor Ġthe Ġweight ing Ġparameter Ġin ĠE q . ĠSEC REF 4 Ġ, Ġwe Ġused Ġa Ġvalue Ġof ĠIN LINE FORM 0 Ġ. ĠThe Ġalgorithm Ġwas Ġtrained Ġover Ġ20 Ġiterations . ĠThe ĠGl o Ve Ġalgorithm Ġwithout Ġany Ġmodifications Ġwas Ġalso Ġtrained Ġas Ġa Ġbaseline Ġwith Ġthe Ġsame Ġparameters . ĠIn Ġaddition Ġto Ġthe Ġoriginal ĠGl o Ve Ġalgorithm , Ġwe Ġcompare Ġour Ġproposed Ġmethod Ġwith Ġprevious Ġstudies Ġthat Ġaim Ġto Ġobtain Ġinterpret able Ġword Ġvectors . ĠWe Ġtrain Ġthe Ġimproved Ġprojected Ġgradient Ġmodel Ġproposed Ġin ĠB IB REF 20 Ġto Ġobtain Ġword Ġvectors Ġ( called ĠO I WE - IP G ) Ġusing Ġthe Ġsame Ġcorpus Ġwe Ġuse Ġto Ġtrain ĠGl o Ve Ġand Ġour Ġproposed Ġmethod . ĠUsing Ġthe Ġmethods Ġproposed Ġin ĠB IB REF 23 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 24 Ġon Ġour Ġbaseline ĠGl o Ve Ġembed d ings , Ġwe Ġobtain ĠS OV , ĠSP INE Ġand ĠPars im ax Ġ( orth og onal ) Ġword Ġrepresentations , Ġrespectively . ĠWe Ġtrain Ġall Ġthe Ġmodels Ġwith Ġthe Ġproposed Ġparameters . ĠHowever , Ġin ĠB IB REF 26 Ġ, Ġthe Ġauthors Ġshow Ġresults Ġfor Ġa Ġrelatively Ġsmall Ġvocabulary Ġof Ġ15 , 000 Ġwords . ĠWhen Ġwe Ġtrained Ġtheir Ġmodel Ġon Ġour Ġbaseline ĠGl o Ve Ġembed d ings Ġwith Ġa Ġlarge Ġvocabulary Ġof Ġsize Ġ287 , 8 47 , Ġthe Ġresulting Ġvectors Ġperformed Ġsignificantly Ġpoor Ġon Ġword Ġsimilarity Ġtasks Ġcompared Ġto Ġthe Ġresults Ġpresented Ġin Ġtheir Ġpaper . ĠIn Ġaddition , ĠPars im ax Ġ( orth og onal ) Ġword Ġvectors Ġobtained Ġusing Ġmethod Ġin ĠB IB REF 26 Ġare Ġnearly Ġidentical Ġto Ġthe Ġbaseline Ġvectors Ġ( i . e . Ġlearned Ġorth og onal Ġtransformation Ġmatrix Ġis Ġvery Ġclose Ġto Ġidentity ). ĠTherefore , ĠPars im ax Ġ( orth og onal ) Ġyields Ġalmost Ġsame Ġresults Ġwith Ġbaseline Ġvectors Ġin Ġall Ġevaluations . ĠWe Ġevaluate Ġthe Ġinterpret ability Ġof Ġthe Ġresulting Ġembed d ings Ġqual itatively Ġand Ġquant itatively . ĠWe Ġalso Ġtest Ġthe Ġperformance Ġof Ġthe Ġembed d ings Ġon Ġword Ġsimilarity Ġand Ġword Ġanalogy Ġtests . </s> In Ġour Ġexperiments , Ġvocabulary Ġsize Ġis Ġclose Ġto Ġ300 , 000 Ġwhile Ġonly Ġ16 , 242 Ġunique Ġwords Ġof Ġthe Ġvocabulary Ġare Ġpresent Ġin Ġthe Ġconcept Ġgroups . ĠFurthermore , Ġonly Ġdimensions Ġthat Ġcorrespond Ġto Ġthe Ġconcept Ġgroup Ġof Ġthe Ġword Ġwill Ġbe Ġupdated Ġdue Ġto Ġthe Ġadditional Ġcost Ġterm . ĠGiven Ġthat Ġthese Ġconcept Ġwords Ġcan Ġbelong Ġto Ġmultiple Ġconcept Ġgroups Ġ( 2 Ġon Ġaverage ), Ġonly Ġ33 , 319 Ġparameters Ġare Ġupdated . ĠThere Ġare Ġ90 Ġmillion Ġindividual Ġparameters Ġpresent Ġfor Ġthe Ġ300 , 000 Ġword Ġvectors Ġof Ġsize Ġ300 . ĠOf Ġthese Ġparameters , Ġonly Ġapproximately Ġ33 , 000 Ġare Ġupdated Ġby Ġthe Ġadditional Ġcost Ġterm . </s> Qual itative ĠEvaluation Ġfor ĠInterpret ability </s> In ĠFig . ĠFIG REF 13 Ġ, Ġwe Ġdemonstrate Ġthe Ġparticular Ġway Ġin Ġwhich Ġthe Ġproposed Ġalgorithm Ġ( ĠSEC REF 4 Ġ) Ġinfluences Ġthe Ġvector Ġrepresentation Ġdistributions . ĠSpecifically , Ġwe Ġconsider , Ġfor Ġillustration , Ġthe Ġ32 nd Ġdimension Ġvalues Ġfor Ġthe Ġoriginal ĠGl o Ve Ġalgorithm Ġand Ġour Ġmodified Ġversion , Ġrestricting Ġthe Ġplots Ġto Ġthe Ġtop - 1000 Ġwords Ġwith Ġrespect Ġto Ġtheir Ġfrequency Ġranks Ġfor Ġclarity Ġof Ġpresentation . ĠIn ĠFig . ĠFIG REF 13 Ġ, Ġthe Ġwords Ġin Ġthe Ġhorizontal Ġaxis Ġare Ġsorted Ġin Ġdescending Ġorder Ġwith Ġrespect Ġto Ġthe Ġvalues Ġat Ġthe Ġ32 nd Ġdimension Ġof Ġtheir Ġword Ġembed ding Ġvectors Ġcoming Ġfrom Ġthe Ġoriginal ĠGl o Ve Ġalgorithm . ĠThe Ġdimension Ġvalues Ġare Ġden oted Ġwith Ġblue Ġand Ġred / green Ġmarkers Ġfor Ġthe Ġoriginal Ġand Ġthe Ġproposed Ġalgorithms , Ġrespectively . ĠAdditionally , Ġthe Ġtop - 50 Ġwords Ġthat Ġachieve Ġthe Ġgreatest Ġ32 nd Ġdimension Ġvalues Ġamong Ġthe Ġconsidered Ġ1000 Ġwords Ġare Ġemphasized Ġwith Ġenlarged Ġmarkers , Ġalong Ġwith Ġtext Ġannotations . ĠIn Ġthe Ġpresented Ġsimulation Ġof Ġthe Ġproposed Ġalgorithm , Ġthe Ġ32 nd Ġdimension Ġvalues Ġare Ġencoded Ġwith Ġthe Ġconcept ĠJ UD G MENT , Ġwhich Ġis Ġreflected Ġas Ġan Ġincrease Ġin Ġthe Ġdimension Ġvalues Ġfor Ġwords Ġsuch Ġas Ġcommittee , Ġacademy , Ġand Ġarticle . ĠWe Ġnote Ġthat Ġthese Ġwords Ġ( red ) Ġare Ġnot Ġpart Ġof Ġthe Ġpre - d etermined Ġword - group Ġfor Ġthe Ġconcept ĠJ UD G MENT , Ġin Ġcontrast Ġto Ġwords Ġsuch Ġas Ġaward , Ġreview Ġand Ġaccount Ġ( green ) Ġwhich Ġare . ĠThis Ġimplies Ġthat Ġthe Ġincrease Ġin Ġthe Ġcorresponding Ġdimension Ġvalues Ġseen Ġfor Ġthese Ġwords Ġis Ġattributable Ġto Ġthe Ġjoint Ġeffect Ġof Ġthe Ġfirst Ġterm Ġin Ġ( ĠSEC REF 4 Ġ) Ġwhich Ġis Ġinherited Ġfrom Ġthe Ġoriginal ĠGl o Ve Ġalgorithm , Ġin Ġconjunction Ġwith Ġthe Ġremaining Ġterms Ġin Ġthe Ġproposed Ġobjective Ġexpression Ġ( ĠSEC REF 4 Ġ). ĠThis Ġexperiment Ġillustrates Ġthat Ġthe Ġproposed Ġalgorithm Ġis Ġable Ġto Ġimpart Ġthe Ġconcept Ġof ĠJ UD G MENT Ġon Ġits Ġdesignated Ġvector Ġdimension Ġabove Ġand Ġbeyond Ġthe Ġsupplied Ġlist Ġof Ġwords Ġbelonging Ġto Ġthe Ġconcept Ġword - group Ġfor Ġthat Ġdimension . ĠWe Ġalso Ġpresent Ġthe Ġlist Ġof Ġwords Ġwith Ġthe Ġgreatest Ġdimension Ġvalue Ġfor Ġthe Ġdimensions Ġ11 , Ġ13 , Ġ16 , Ġ31 , Ġ36 , Ġ39 , Ġ41 , Ġ43 Ġand Ġ79 Ġin ĠTable ĠT AB REF 11 Ġ. ĠThese Ġdimensions Ġare Ġaligned / im part ed Ġwith Ġthe Ġconcepts Ġthat Ġare Ġgiven Ġin Ġthe Ġcolumn Ġheaders . ĠIn ĠTable ĠT AB REF 11 Ġ, Ġthe Ġwords Ġthat Ġare Ġhighlighted Ġwith Ġgreen Ġdenote Ġthe Ġwords Ġthat Ġexist Ġin Ġthe Ġcorresponding Ġword - group Ġobtained Ġfrom ĠRog et 's ĠThe s aurus Ġ( and Ġare Ġthus Ġexplicitly Ġforced Ġto Ġachieve Ġincreased Ġdimension Ġvalues ), Ġwhile Ġthe Ġred Ġwords Ġdenote Ġthe Ġwords Ġthat Ġachieve Ġincreased Ġdimension Ġvalues Ġby Ġvirtue Ġof Ġtheir Ġco occ urrence Ġstatistics Ġwith Ġthe Ġthe s aurus - based Ġwords Ġ( ind irect ly , Ġwithout Ġbeing Ġexplicitly Ġforced ). ĠThis Ġagain Ġillustrates Ġthat Ġa Ġsemantic Ġconcept Ġcan Ġindeed Ġbe Ġcoded Ġto Ġa Ġvector Ġdimension Ġprovided Ġthat Ġa Ġsensible Ġlex ical Ġresource Ġis Ġused Ġto Ġguide Ġsem antically Ġrelated Ġwords Ġto Ġthe Ġdesired Ġvector Ġdimension Ġvia Ġthe Ġproposed Ġobjective Ġfunction Ġin Ġ( ĠSEC REF 4 Ġ). ĠEven Ġthe Ġwords Ġthat Ġdo Ġnot Ġappear Ġin , Ġbut Ġare Ġsem antically Ġrelated Ġto , Ġthe Ġword - groups Ġthat Ġwe Ġformed Ġusing ĠRog et 's ĠThe s aurus , Ġare Ġindirectly Ġaffected Ġby Ġthe Ġproposed Ġalgorithm . ĠThey Ġalso Ġreflect Ġthe Ġassociated Ġconcepts Ġat Ġtheir Ġrespective Ġdimensions Ġeven Ġthough Ġthe Ġobjective Ġfunctions Ġfor Ġtheir Ġparticular Ġvectors Ġare Ġnot Ġmodified . ĠThis Ġpoint Ġcannot Ġbe Ġove rem phas ized . ĠAlthough Ġthe Ġword - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus Ġimpose Ġa Ġdegree Ġof Ġsupervision Ġto Ġthe Ġprocess , Ġthe Ġfact Ġthat Ġthe Ġremaining Ġwords Ġin Ġthe Ġentire Ġvocabulary Ġare Ġalso Ġindirectly Ġaffected Ġmakes Ġthe Ġproposed Ġmethod Ġa Ġsemi - super vised Ġapproach Ġthat Ġcan Ġhandle Ġwords Ġthat Ġare Ġnot Ġin Ġthese Ġchosen Ġword - groups . ĠA Ġqualitative Ġexample Ġof Ġthis Ġresult Ġcan Ġbe Ġseen Ġin Ġthe Ġlast Ġcolumn Ġof ĠTable ĠT AB REF 11 Ġ. ĠIt Ġis Ġinteresting Ġto Ġnote Ġthe Ġappearance Ġof Ġwords Ġsuch Ġas Ġgu er illa , Ġinsurgency , Ġm ujah ide en , ĠWe hr m acht Ġand ĠLu ft w affe Ġin Ġaddition Ġto Ġthe Ġmore Ġobvious Ġand Ġstraightforward Ġarmy , Ġsoldiers Ġand Ġtroops , Ġall Ġof Ġwhich Ġare Ġnot Ġpresent Ġin Ġthe Ġassociated Ġword - group ĠWAR F ARE . </s> Most Ġof Ġthe Ġdimensions Ġwe Ġinvestigated Ġexhibit Ġsimilar Ġbehaviour Ġto Ġthe Ġones Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ. ĠThus Ġgenerally Ġspeaking , Ġwe Ġcan Ġsay Ġthat Ġthe Ġentries Ġin ĠTable ĠT AB REF 11 Ġare Ġrepresentative Ġof Ġthe Ġgreat Ġmajority . ĠHowever , Ġwe Ġhave Ġalso Ġspecifically Ġlooked Ġfor Ġdimensions Ġthat Ġmake Ġless Ġsense Ġand Ġdetermined Ġa Ġfew Ġsuch Ġdimensions Ġwhich Ġare Ġrelatively Ġless Ġsatisfactory . ĠThese Ġless Ġsatisfactory Ġexamples Ġare Ġgiven Ġin ĠTable ĠT AB REF 14 Ġ. ĠThese Ġexamples Ġare Ġalso Ġinteresting Ġin Ġthat Ġthey Ġshed Ġinsight Ġinto Ġthe Ġlimitations Ġposed Ġby Ġpoly se my Ġand Ġexistence Ġof Ġvery Ġrare Ġout lier Ġwords . </s> Quant itative ĠEvaluation Ġfor ĠInterpret ability </s> One Ġof Ġthe Ġmain Ġgoals Ġof Ġthis Ġstudy Ġis Ġto Ġimprove Ġthe Ġinterpret ability Ġof Ġdense Ġword Ġembed d ings Ġby Ġalign ing Ġthe Ġdimensions Ġwith Ġpred efined Ġconcepts Ġfrom Ġa Ġsuitable Ġlex icon . ĠA Ġquantitative Ġmeasure Ġis Ġrequired Ġto Ġreliably Ġevaluate Ġthe Ġachieved Ġimprovement . ĠOne Ġof Ġthe Ġmethods Ġproposed Ġto Ġmeasure Ġthe Ġinterpret ability Ġis Ġthe Ġword Ġintrusion Ġtest ĠB IB REF 41 Ġ. ĠBut , Ġthis Ġmethod Ġis Ġexpensive Ġto Ġapply Ġsince Ġit Ġrequires Ġevaluations Ġfrom Ġmultiple Ġhuman Ġeval u ators Ġfor Ġeach Ġembed ding Ġdimension . ĠIn Ġthis Ġstudy , Ġwe Ġuse Ġa Ġsemantic Ġcategory - based Ġapproach Ġbased Ġon Ġthe Ġmethod Ġand Ġcategory Ġdataset Ġ( S EM C AT ) Ġintroduced Ġin ĠB IB REF 27 Ġto Ġquantify Ġinterpret ability . ĠSpecifically , Ġwe Ġapply Ġa Ġmodified Ġversion Ġof Ġthe Ġapproach Ġpresented Ġin ĠB IB REF 40 Ġin Ġorder Ġto Ġconsider Ġpossible Ġsub - group ings Ġwithin Ġthe Ġcategories Ġin ĠSEM C AT . ĠInterpret ability Ġscores Ġare Ġcalculated Ġusing ĠInterpret ability ĠScore Ġ( IS ) Ġas Ġgiven Ġbelow : </s> DIS PLAY FORM 0 </s> In Ġ( ĠEQ REF 17 Ġ), ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrepresents Ġthe Ġinterpret ability Ġscores Ġin Ġthe Ġpositive Ġand Ġnegative Ġdirections Ġof Ġthe ĠIN LINE FORM 2 Ġdimension Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġnumber Ġof Ġdimensions Ġin Ġthe Ġembed ding Ġspace ) Ġof Ġword Ġembed ding Ġspace Ġfor Ġthe ĠIN LINE FORM 5 Ġcategory Ġ( ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġis Ġnumber Ġof Ġcategories Ġin ĠSEM C AT , ĠIN LINE FORM 8 Ġ) Ġin ĠSEM C AT Ġrespectively . ĠIN LINE FORM 9 Ġis Ġthe Ġset Ġof Ġwords Ġin Ġthe ĠIN LINE FORM 10 Ġcategory Ġin ĠSEM C AT Ġand ĠIN LINE FORM 11 Ġis Ġthe Ġnumber Ġof Ġwords Ġin ĠIN LINE FORM 12 Ġ. ĠIN LINE FORM 13 Ġcorresponds Ġto Ġthe Ġminimum Ġnumber Ġof Ġwords Ġrequired Ġto Ġconstruct Ġa Ġsemantic Ġcategory Ġ( i . e . Ġrepresent Ġa Ġconcept ). ĠIN LINE FORM 14 Ġrepresents Ġthe Ġset Ġof ĠIN LINE FORM 15 Ġwords Ġthat Ġhave Ġthe Ġhighest Ġ( ĠIN LINE FORM 16 Ġ) Ġand Ġlowest Ġ( ĠIN LINE FORM 17 Ġ) Ġvalues Ġin ĠIN LINE FORM 18 Ġdimension Ġof Ġthe Ġembed ding Ġspace . ĠIN LINE FORM 19 Ġis Ġthe Ġintersection Ġoperator Ġand ĠIN LINE FORM 20 Ġis Ġthe Ġcardinal ity Ġoperator Ġ( number Ġof Ġelements ) Ġfor Ġthe Ġintersect ing Ġset . ĠIn Ġ( ĠEQ REF 17 Ġ), ĠIN LINE FORM 21 Ġgives Ġthe Ġinterpret ability Ġscore Ġfor Ġthe ĠIN LINE FORM 22 Ġdimension Ġand ĠIN LINE FORM 23 Ġgives Ġthe Ġaverage Ġinterpret ability Ġscore Ġof Ġthe Ġembed ding Ġspace . </s> Fig . ĠFIG REF 18 Ġpresents Ġthe Ġmeasured Ġaverage Ġinterpret ability Ġscores Ġacross Ġdimensions Ġfor Ġoriginal ĠGl o Ve Ġembed d ings , Ġfor Ġthe Ġproposed Ġmethod Ġand Ġfor Ġthe Ġother Ġfour Ġmethods Ġwe Ġcompare , Ġalong Ġwith Ġa Ġrandomly Ġgenerated Ġembed ding . ĠResults Ġare Ġcalculated Ġfor Ġthe Ġparameters ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠOur Ġproposed Ġmethod Ġsignificantly Ġimproves Ġthe Ġinterpret ability Ġfor Ġall ĠIN LINE FORM 2 Ġcompared Ġto Ġthe Ġoriginal ĠGl o Ve Ġapproach . ĠOur Ġproposed Ġmethod Ġis Ġsecond Ġto Ġonly ĠSP INE Ġin Ġincreasing Ġinterpret ability . ĠHowever , Ġas Ġwe Ġwill Ġexperiment ally Ġdemonstrate Ġin Ġthe Ġnext Ġsubsection , Ġin Ġdoing Ġthis , ĠSP INE Ġalmost Ġentirely Ġdestroys Ġthe Ġunderlying Ġsemantic Ġstructure Ġof Ġthe Ġword Ġembed d ings , Ġwhich Ġis Ġthe Ġprimary Ġfunction Ġof Ġa Ġword Ġembed ding . </s> The Ġproposed Ġmethod Ġand Ġinterpret ability Ġmeasurements Ġare Ġboth Ġbased Ġon Ġutilizing Ġconcepts Ġrepresented Ġby Ġword - groups . ĠTherefore Ġit Ġis Ġexpected Ġthat Ġthere Ġwill Ġbe Ġhigher Ġinterpret ability Ġscores Ġfor Ġsome Ġof Ġthe Ġdimensions Ġfor Ġwhich Ġthe Ġimp arted Ġconcepts Ġare Ġalso Ġcontained Ġin ĠSEM C AT . ĠHowever , Ġby Ġdesign , Ġword Ġgroups Ġthat Ġthey Ġuse Ġare Ġformed Ġby Ġusing Ġdifferent Ġsources Ġand Ġare Ġindependent . ĠInterpret ability Ġmeasurements Ġuse ĠSEM C AT Ġwhile Ġour Ġproposed Ġmethod Ġutilizes ĠRog et 's ĠThe s aurus . </s> Int r ins ic ĠEvaluation Ġof Ġthe ĠEmb edd ings </s> It Ġis Ġnecessary Ġto Ġshow Ġthat Ġthe Ġsemantic Ġstructure Ġof Ġthe Ġoriginal Ġembed ding Ġhas Ġnot Ġbeen Ġdamaged Ġor Ġdistorted Ġas Ġa Ġresult Ġof Ġalign ing Ġthe Ġdimensions Ġwith Ġgiven Ġconcepts , Ġand Ġthat Ġthere Ġis Ġno Ġsubstantial Ġsacrifice Ġinvolved Ġfrom Ġthe Ġperformance Ġthat Ġcan Ġbe Ġobtained Ġwith Ġthe Ġoriginal ĠGl o Ve . ĠTo Ġcheck Ġthis , Ġwe Ġevaluate Ġperformances Ġof Ġthe Ġproposed Ġembed d ings Ġon Ġword Ġsimilarity ĠB IB REF 42 Ġand Ġword Ġanalogy ĠB IB REF 0 Ġtests . ĠWe Ġcompare Ġthe Ġresults Ġwith Ġthe Ġoriginal Ġembed d ings Ġand Ġthe Ġthree Ġalternatives Ġexcluding ĠPars im ax ĠB IB REF 26 Ġsince Ġorth og onal Ġtransformations Ġwill Ġnot Ġaffect Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġembed d ings Ġon Ġthese Ġtests . </s> Word Ġsimilarity Ġtest Ġmeasures Ġthe Ġcorrelation Ġbetween Ġword Ġsimilarity Ġscores Ġobtained Ġfrom Ġhuman Ġevaluation Ġ( i . e . Ġtrue Ġsimilarities ) Ġand Ġfrom Ġword Ġembed d ings Ġ( usually Ġusing Ġcos ine Ġsimilarity ). ĠIn Ġother Ġwords , Ġthis Ġtest Ġquant ifies Ġhow Ġwell Ġthe Ġembed ding Ġspace Ġreflects Ġhuman Ġjud gements Ġin Ġterms Ġof Ġsimilarities Ġbetween Ġdifferent Ġwords . ĠThe Ġcorrelation Ġscores Ġfor Ġ13 Ġdifferent Ġsimilarity Ġtest Ġsets Ġare Ġreported Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġobserve Ġthat , Ġlet Ġalone Ġa Ġreduction Ġin Ġperformance , Ġthe Ġobtained Ġscores Ġindicate Ġan Ġalmost Ġuniform Ġimprovement Ġin Ġthe Ġcorrelation Ġvalues Ġfor Ġthe Ġproposed Ġalgorithm , Ġoutper forming Ġall Ġthe Ġalternatives Ġin Ġalmost Ġall Ġtest Ġsets . ĠCategories Ġfrom ĠRog et 's Ġthe s aurus Ġare Ġgroup ings Ġof Ġwords Ġthat Ġare Ġsimilar Ġin Ġsome Ġsense Ġwhich Ġthe Ġoriginal Ġembed ding Ġalgorithm Ġmay Ġfail Ġto Ġcapture . ĠThese Ġtest Ġresults Ġsignify Ġthat Ġthe Ġsemantic Ġinformation Ġinjected Ġinto Ġthe Ġalgorithm Ġby Ġthe Ġadditional Ġcost Ġterm Ġis Ġsignificant Ġenough Ġto Ġresult Ġin Ġa Ġmeasurable Ġimprovement . ĠIt Ġshould Ġalso Ġbe Ġnoted Ġthat Ġscores Ġobtained Ġby ĠSP INE Ġis Ġun accept ably Ġlow Ġon Ġalmost Ġall Ġtests Ġindicating Ġthat Ġit Ġhas Ġachieved Ġits Ġinterpret ability Ġperformance Ġat Ġthe Ġcost Ġof Ġlosing Ġits Ġsemantic Ġfunctions . </s> Word Ġanalogy Ġtest Ġis Ġintroduced Ġin ĠB IB REF 1 Ġand Ġlooks Ġfor Ġthe Ġanswers Ġof Ġthe Ġquestions Ġthat Ġare Ġin Ġthe Ġform Ġof Ġ" X Ġis Ġto ĠY , Ġwhat ĠZ Ġis Ġto Ġ ?" Ġby Ġapplying Ġsimple Ġarithmetic Ġoperations Ġto Ġvectors Ġof Ġwords ĠX , ĠY Ġand ĠZ . ĠWe Ġpresent Ġprecision Ġscores Ġfor Ġthe Ġword Ġanalogy Ġtests Ġin ĠTable ĠT AB REF 21 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe Ġalternative Ġapproaches Ġthat Ġaim Ġto Ġimprove Ġinterpret ability , Ġhave Ġpoor Ġperformance Ġon Ġthe Ġword Ġanalogy Ġtests . ĠHowever , Ġour Ġproposed Ġmethod Ġhas Ġcomparable Ġperformance Ġwith Ġthe Ġoriginal ĠGl o Ve Ġembed d ings . ĠOur Ġmethod Ġoutper forms ĠGl o Ve Ġin Ġsemantic Ġanalogy Ġtest Ġset Ġand Ġin Ġoverall Ġresults , Ġwhile ĠGl o Ve Ġperforms Ġslightly Ġbetter Ġin Ġsynt actic Ġtest Ġset . ĠThis Ġcomparable Ġperformance Ġis Ġmainly Ġdue Ġto Ġthe Ġcost Ġfunction Ġof Ġour Ġproposed Ġmethod Ġthat Ġincludes Ġthe Ġoriginal Ġobjective Ġof Ġthe ĠGl o Ve . </s> To Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġadditional Ġcost Ġterm Ġon Ġthe Ġperformance Ġimprovement Ġin Ġthe Ġsemantic Ġanalogy Ġtest , Ġwe Ġpresent ĠTable ĠT AB REF 22 Ġ. ĠIn Ġparticular , Ġwe Ġpresent Ġresults Ġfor Ġthe Ġcases Ġwhere Ġi ) Ġall Ġquestions Ġin Ġthe Ġdataset Ġare Ġconsidered , Ġii ) Ġonly Ġthe Ġquestions Ġthat Ġcontains Ġat Ġleast Ġone Ġconcept Ġword Ġare Ġconsidered , Ġiii ) Ġonly Ġthe Ġquestions Ġthat Ġconsist Ġentirely Ġof Ġconcept Ġwords Ġare Ġconsidered . ĠWe Ġnote Ġspecifically Ġthat Ġfor Ġthe Ġlast Ġcase , Ġonly Ġa Ġsubset Ġof Ġthe Ġquestions Ġunder Ġthe Ġsemantic Ġcategory Ġfamily . txt Ġended Ġup Ġbeing Ġincluded . ĠWe Ġobserve Ġthat Ġfor Ġall Ġthree Ġscenarios , Ġour Ġproposed Ġalgorithm Ġresults Ġin Ġan Ġimprovement Ġin Ġthe Ġprecision Ġscores . ĠHowever , Ġthe Ġgreatest Ġperformance Ġincrease Ġis Ġseen Ġfor Ġthe Ġlast Ġscenario , Ġwhich Ġunderscores Ġthe Ġextent Ġto Ġwhich Ġthe Ġsemantic Ġfeatures Ġcaptured Ġby Ġembed d ings Ġcan Ġbe Ġimproved Ġwith Ġa Ġreasonable Ġselection Ġof Ġthe Ġlex ical Ġresource Ġfrom Ġwhich Ġthe Ġconcept Ġword - groups Ġwere Ġderived . </s> Conclusion </s> We Ġpresented Ġa Ġnovel Ġapproach Ġto Ġimpart Ġinterpret ability Ġinto Ġword Ġembed d ings . ĠWe Ġachieved Ġthis Ġby Ġencouraging Ġdifferent Ġdimensions Ġof Ġthe Ġvector Ġrepresentation Ġto Ġalign Ġwith Ġpred efined Ġconcepts , Ġthrough Ġthe Ġaddition Ġof Ġan Ġadditional Ġcost Ġterm Ġin Ġthe Ġoptimization Ġobjective Ġof Ġthe ĠGl o Ve Ġalgorithm Ġthat Ġfavors Ġa Ġselective Ġincrease Ġfor Ġa Ġpre - specified Ġinput Ġof Ġconcept Ġwords Ġalong Ġeach Ġdimension . </s> We Ġdemonstrated Ġthe Ġefficacy Ġof Ġthis Ġapproach Ġby Ġapplying Ġqualitative Ġand Ġquantitative Ġevaluations Ġfor Ġinterpret ability . ĠWe Ġalso Ġshowed Ġvia Ġstandard Ġword - an alogy Ġand Ġword - similar ity Ġtests Ġthat Ġthe Ġsemantic Ġco herence Ġof Ġthe Ġoriginal Ġvector Ġspace Ġis Ġpreserved , Ġeven Ġslightly Ġimproved . ĠWe Ġhave Ġalso Ġperformed Ġand Ġreported Ġquantitative Ġcomparisons Ġwith Ġseveral Ġother Ġmethods Ġfor Ġboth Ġinterpret ab ilty Ġincrease Ġand Ġpreservation Ġof Ġsemantic Ġco herence . ĠUpon Ġinspection Ġof ĠFig . ĠFIG REF 18 Ġand ĠTables ĠT AB REF 20 Ġ, ĠT AB REF 21 Ġ, Ġand ĠT AB REF 22 Ġaltogether , Ġit Ġshould Ġbe Ġnoted Ġthat Ġour Ġproposed Ġmethod Ġachieves Ġboth Ġof Ġthe Ġobjectives Ġsimultaneously , Ġincreased Ġinterpret ability Ġand Ġpreservation Ġof Ġthe Ġintrinsic Ġsemantic Ġstructure . </s> An Ġimportant Ġpoint Ġwas Ġthat , Ġwhile Ġit Ġis Ġexpected Ġfor Ġwords Ġthat Ġare Ġalready Ġincluded Ġin Ġthe Ġconcept Ġword - groups Ġto Ġbe Ġaligned Ġtogether Ġsince Ġtheir Ġdimensions Ġare Ġdirectly Ġupdated Ġwith Ġthe Ġproposed Ġcost Ġterm , Ġit Ġwas Ġalso Ġobserved Ġthat Ġwords Ġnot Ġin Ġthese Ġgroups Ġalso Ġaligned Ġin Ġa Ġmeaningful Ġmanner Ġwithout Ġany Ġdirect Ġmodification Ġto Ġtheir Ġcost Ġfunction . ĠThis Ġindicates Ġthat Ġthe Ġcost Ġterm Ġwe Ġadded Ġworks Ġproduct ively Ġwith Ġthe Ġoriginal Ġcost Ġfunction Ġof ĠGl o Ve Ġto Ġhandle Ġwords Ġthat Ġare Ġnot Ġincluded Ġin Ġthe Ġoriginal Ġconcept Ġword - groups , Ġbut Ġare Ġsem antically Ġrelated Ġto Ġthose Ġword - groups . ĠThe Ġunderlying Ġmechanism Ġcan Ġbe Ġexplained Ġas Ġfollows . ĠWhile Ġthe Ġoutside Ġlex ical Ġresource Ġwe Ġintroduce Ġcontains Ġa Ġrelatively Ġsmall Ġnumber Ġof Ġwords Ġcompared Ġto Ġthe Ġtotal Ġnumber Ġof Ġwords , Ġthese Ġwords Ġand Ġthe Ġcategories Ġthey Ġrepresent Ġhave Ġbeen Ġcarefully Ġchosen Ġand Ġin Ġa Ġsense , Ġ" d ense ly Ġspan " Ġall Ġthe Ġwords Ġin Ġthe Ġlanguage . ĠBy Ġsaying Ġ" span ", Ġwe Ġmean Ġthey Ġcover Ġmost Ġof Ġthe Ġconcepts Ġand Ġideas Ġin Ġthe Ġlanguage Ġwithout Ġleaving Ġtoo Ġmany Ġuncovered Ġareas . ĠWith Ġ" d ense ly " Ġwe Ġmean Ġall Ġareas Ġare Ġcovered Ġwith Ġsufficient Ġstrength . ĠIn Ġother Ġwords , Ġthis Ġsubset Ġof Ġwords Ġis Ġable Ġto Ġconstitute Ġa Ġsufficiently Ġstrong Ġskeleton , Ġor Ġscaff old . ĠNow Ġremember Ġthat ĠGl o Ve Ġworks Ġto Ġalign Ġor Ġbring Ġcloser Ġrelated Ġgroups Ġof Ġwords , Ġwhich Ġwill Ġinclude Ġwords Ġfrom Ġthe Ġlex ical Ġsource . ĠSo Ġthe Ġjoint Ġaction Ġof Ġalign ing Ġthe Ġwords Ġwith Ġthe Ġpred efined Ġcategories Ġ( introdu ced Ġby Ġus ) Ġand Ġalign ing Ġrelated Ġwords Ġ( handled Ġby ĠGl o Ve ) Ġallows Ġwords Ġnot Ġin Ġthe Ġlex ical Ġgroups Ġto Ġalso Ġbe Ġaligned Ġmeaning fully . ĠWe Ġmay Ġsay Ġthat Ġthe Ġnon - in cluded Ġwords Ġare Ġ" pull ed Ġalong " Ġwith Ġthe Ġincluded Ġwords Ġby Ġvirtue Ġof Ġthe Ġ" strings " Ġor Ġ" gl ue " Ġthat Ġis Ġprovided Ġby ĠGl o Ve . ĠIn Ġnumbers , Ġthe Ġdesired Ġeffect Ġis Ġachieved Ġby Ġmanipulating Ġless Ġthan Ġonly Ġ0 . 05 % Ġof Ġparameters Ġof Ġthe Ġentire Ġword Ġvectors . ĠThus , Ġwhile Ġthere Ġis Ġa Ġdegree Ġof Ġsupervision Ġcoming Ġfrom Ġthe Ġexternal Ġlex ical Ġresource , Ġthe Ġrest Ġof Ġthe Ġvocabulary Ġis Ġalso Ġaligned Ġindirectly Ġin Ġan Ġun super vised Ġway . ĠThis Ġmay Ġbe Ġthe Ġreason Ġwhy , Ġunlike Ġearlier Ġproposed Ġapproaches , Ġour Ġmethod Ġis Ġable Ġto Ġachieve Ġincreasing Ġinterpret ability Ġwithout Ġdestroying Ġunderlying Ġsemantic Ġstructure , Ġand Ġconsequently Ġwithout Ġsacrificing Ġperformance Ġin Ġbenchmark Ġtests . </s> Upon Ġinspecting Ġthe Ġ2 nd Ġcolumn Ġof ĠTable ĠT AB REF 14 Ġ, Ġwhere Ġqualitative Ġresults Ġfor Ġconcept ĠT AS TE Ġare Ġpresented , Ġanother Ġinsight Ġregarding Ġthe Ġlearning Ġmechanism Ġof Ġour Ġproposed Ġapproach Ġcan Ġbe Ġmade . ĠHere Ġit Ġseems Ġunderstandable Ġthat Ġour Ġproposed Ġapproach , Ġalong Ġwith ĠGl o Ve , Ġbrought Ġtogether Ġthe Ġwords Ġtaste Ġand Ġpolish , Ġand Ġthen Ġthe Ġwords ĠPolish Ġand , Ġfor Ġinstance , ĠWarsaw Ġare Ġbrought Ġtogether Ġby ĠGl o Ve . ĠThese Ġexamples Ġare Ġinteresting Ġin Ġthat Ġthey Ġshed Ġinsight Ġinto Ġhow ĠGl o Ve Ġworks Ġand Ġthe Ġlimitations Ġposed Ġby Ġpoly se my . ĠIt Ġshould Ġbe Ġunder lined Ġthat Ġthe Ġpresent Ġapproach Ġis Ġnot Ġtotally Ġincapable Ġof Ġhandling Ġpoly se my , Ġbut Ġcannot Ġdo Ġso Ġperfectly . ĠSince Ġrelated Ġwords Ġare Ġbeing Ġclustered , Ġsufficiently Ġwell - connected Ġwords Ġthat Ġdo Ġnot Ġmeaning fully Ġbelong Ġalong Ġwith Ġothers Ġwill Ġbe Ġappropriately Ġ" pull ed Ġaway " Ġfrom Ġthat Ġgroup Ġby Ġseveral Ġwords , Ġagainst Ġthe Ġless Ġeffective , Ġinappropriate Ġpull Ġof Ġa Ġparticular Ġword . ĠEven Ġthough Ġpolish Ġwith Ġlower case Ġ" p " Ġbelongs Ġwhere Ġit Ġis , Ġit Ġis Ġattracting ĠWarsaw Ġto Ġitself Ġthrough Ġpoly se my Ġand Ġthis Ġis Ġnot Ġmeaningful . ĠPerhaps Ġbecause ĠWarsaw Ġis Ġnot Ġa Ġsufficiently Ġwell - connected Ġword , Ġit Ġends Ġbeing Ġdragged Ġalong , Ġalthough Ġwords Ġwith Ġgreater Ġconnected ness Ġto Ġa Ġconcept Ġgroup Ġmight Ġhave Ġbetter Ġresisted Ġsuch Ġinappropriate Ġattractions . </s> In Ġthis Ġstudy , Ġwe Ġused Ġthe ĠGl o Ve Ġalgorithm Ġas Ġthe Ġunderlying Ġdense Ġword Ġembed ding Ġscheme Ġto Ġdemonstrate Ġour Ġapproach . ĠHowever , Ġwe Ġstress Ġthat Ġit Ġis Ġpossible Ġfor Ġour Ġapproach Ġto Ġbe Ġextended Ġto Ġother Ġword Ġembed ding Ġalgorithms Ġwhich Ġhave Ġa Ġlearning Ġroutine Ġconsisting Ġof Ġiterations Ġover Ġco occ urrence Ġrecords , Ġby Ġmaking Ġsuitable Ġadjustments Ġin Ġthe Ġobjective Ġfunction . ĠSince Ġword 2 vec Ġmodel Ġis Ġalso Ġbased Ġon Ġthe Ġco oc ur rences Ġof Ġwords Ġin Ġa Ġsliding Ġwindow Ġthrough Ġa Ġlarge Ġcorpus , Ġwe Ġexpect Ġthat Ġour Ġapproach Ġcan Ġalso Ġbe Ġapplied Ġto Ġword 2 vec Ġafter Ġmaking Ġsuitable Ġadjustments , Ġwhich Ġcan Ġbe Ġconsidered Ġas Ġan Ġimmediate Ġfuture Ġwork Ġfor Ġour Ġapproach . ĠAlthough Ġthe Ġsemantic Ġconcepts Ġare Ġencoded Ġin Ġonly Ġone Ġdirection Ġ( positive ) Ġwithin Ġthe Ġembed ding Ġdimensions , Ġit Ġmight Ġbe Ġbeneficial Ġto Ġpursue Ġfuture Ġwork Ġthat Ġalso Ġenc odes Ġopposite Ġconcepts , Ġsuch Ġas Ġgood Ġand Ġbad , Ġin Ġtwo Ġopposite Ġdirections Ġof Ġthe Ġsame Ġdimension . </s> The Ġproposed Ġmethodology Ġcan Ġalso Ġbe Ġhelpful Ġin Ġcomputational Ġcross - ling ual Ġstudies , Ġwhere Ġthe Ġsimilarities Ġare Ġexplored Ġacross Ġthe Ġvector Ġspaces Ġof Ġdifferent Ġlanguages ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ.
Introduction Distributed word representations, commonly referred to as word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , serve as elementary building blocks in the course of algorithm design for an expanding range of applications in natural language processing (NLP), including named entity recognition BIBREF4 , BIBREF5 , parsing BIBREF6 , sentiment analysis BIBREF7 , BIBREF8 , and word-sense disambiguation BIBREF9 . Although the empirical utility of word embeddings as an unsupervised method for capturing the semantic or syntactic features of a certain word as it is used in a given lexical resource is well-established BIBREF10 , BIBREF11 , BIBREF12 , an understanding of what these features mean remains an open problem BIBREF13 , BIBREF14 and as such word embeddings mostly remain a black box. It is desirable to be able to develop insight into this black box and be able to interpret what it means, while retaining the utility of word embeddings as semantically-rich intermediate representations. Other than the intrinsic value of this insight, this would not only allow us to explain and understand how algorithms work BIBREF15 , but also set a ground that would facilitate the design of new algorithms in a more deliberate way. Recent approaches to generating word embeddings (e.g. BIBREF0 , BIBREF2 ) are rooted linguistically in the field of distributed semantics BIBREF16 , where words are taken to assume meaning mainly by their degree of interaction (or lack thereof) with other words in the lexicon BIBREF17 , BIBREF18 . Under this paradigm, dense, continuous vector representations are learned in an unsupervised manner from a large corpus, using the word cooccurrence statistics directly or indirectly, and such an approach is shown to result in vector representations that mathematically capture various semantic and syntactic relations between words BIBREF0 , BIBREF2 , BIBREF3 . However, the dense nature of the learned embeddings obfuscate the distinct concepts encoded in the different dimensions, which renders the resulting vectors virtually uninterpretable. The learned embeddings make sense only in relation to each other and their specific dimensions do not carry explicit information that can be interpreted. However, being able to interpret a word embedding would illuminate the semantic concepts implicitly represented along the various dimensions of the embedding, and reveal its hidden semantic structures. In the literature, researchers tackled interpretability problem of the word embeddings using different approaches. Several researchers BIBREF19 , BIBREF20 , BIBREF21 proposed algorithms based on non-negative matrix factorization (NMF) applied to cooccurrence variant matrices. Other researchers suggested to obtain interpretable word vectors from existing uninterpretable word vectors by applying sparse coding BIBREF22 , BIBREF23 , by training a sparse auto-encoder to transform the embedding space BIBREF24 , by rotating the original embeddings BIBREF25 , BIBREF26 or by applying transformations based on external semantic datasets BIBREF27 . Although the above-mentioned approaches provide better interpretability that is measured using a particular method such as word intrusion test, usually the improved interpretability comes with a cost of performance in the benchmark tests such as word similarity or word analogy. One possible explanation for this performance decrease is that the proposed transformations from the original embedding space distort the underlying semantic structure constructed by the original embedding algorithm. Therefore, it can be claimed that a method that learns dense and interpretable word embeddings without inflicting any damage to the underlying semantic learning mechanism is the key to achieve both high performing and interpretable word embeddings. Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric. Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 . Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings. By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts. This alignment is achieved by introducing a modification to the embedding learning process. In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function. Each embedding vector dimension is first associated with a concept. For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to. For words that do not belong to any one of the word-groups, the cost term is left untouched. Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method. We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results. We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests. The paper is organized as follows. In Section SECREF2 , we discuss previous studies related to our work under two main categories: interpretability of word embeddings and joint-learning frameworks where the objective function is modified. In Section SECREF3 , we present the problem framework and provide the formulation within the GloVe BIBREF2 algorithm setting. In Section SECREF4 where our approach is proposed, we motivate and develop a modification to the original objective function with the aim of increasing representation interpretability. In Section SECREF5 , experimental results are provided and the proposed method is quantitatively and qualitatively evaluated. Additionally, in Section SECREF5 , results demonstrating the extent to which the original semantic structure of the embedding space is affected are presented by using word-analogy and word-similarity tests. We conclude the paper in Section SECREF6 . Related Work Methodologically, our work is related to prior studies that aim to obtain “improved” word embeddings using external lexical resources, under some performance metric. Previous work in this area can be divided into two main categories: works that i) modify the word embedding learning algorithm to incorporate lexical information, ii) operate on pre-trained embeddings with a post-processing step. Among works that follow the first approach, BIBREF28 extend the Skip-Gram model by incorporating the word similarity relations extracted from the Paraphrase Database (PPDB) and WordNet BIBREF29 , into the Skip-Gram predictive model as an additional cost term. In BIBREF30 , the authors extend the CBOW model by considering two types of semantic information, termed relational and categorical, to be incorporated into the embeddings during training. For the former type of semantic information, the authors propose the learning of explicit vectors for the different relations extracted from a semantic lexicon such that the word pairs that satisfy the same relation are distributed more homogeneously. For the latter, the authors modify the learning objective such that some weighted average distance is minimized for words under the same semantic category. In BIBREF31 , the authors represent the synonymy and hypernymy-hyponymy relations in terms of inequality constraints, where the pairwise similarity rankings over word triplets are forced to follow an order extracted from a lexical resource. Following their extraction from WordNet, the authors impose these constraints in the form of an additive cost term to the Skip-Gram formulation. Finally, BIBREF32 builds on top of the GloVe algorithm by introducing a regularization term to the objective function that encourages the vector representations of similar words as dictated by WordNet to be similar as well. Turning our attention to the post-processing approach for enriching word embeddings with external lexical knowledge, BIBREF33 has introduced the retrofitting algorithm that acts on pre-trained embeddings such as Skip-Gram or GloVe. The authors propose an objective function that aims to balance out the semantic information captured in the pre-trained embeddings with the constraints derived from lexical resources such as WordNet, PPDB and FrameNet. One of the models proposed in BIBREF34 extends the retrofitting approach to incorporate the word sense information from WordNet. Similarly, BIBREF35 creates multi-sense embeddings by gathering the word sense information from a lexical resource and learning to decompose the pre-trained embeddings into a convex combination of sense embeddings. In BIBREF36 , the authors focus on improving word embeddings for capturing word similarity, as opposed to mere relatedness. To this end, they introduce the counter-fitting technique which acts on the input word vectors such that synonymous words are attracted to one another whereas antonymous words are repelled, where the synonymy-antonymy relations are extracted from a lexical resource. More recently, the ATTRACT-REPEL algorithm proposed by BIBREF37 improves on counter-fitting by a formulation which imparts the word vectors with external lexical information in mini-batches. Most of the studies discussed above ( BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF36 , BIBREF37 ) report performance improvements in benchmark tests such as word similarity or word analogy, while BIBREF29 uses a different analysis method (mean reciprocal rank). In sum, the literature is rich with studies aiming to obtain word embeddings that perform better under specific performance metrics. However, less attention has been directed to the issue of interpretability of the word embeddings. In the literature, the problem of interpretability has been tackled using different approaches. BIBREF19 proposed non-negative matrix factorization (NMF) for learning sparse, interpretable word vectors from co-occurrence variant matrices where the resulting vector space is called non-negative sparse embeddigns (NNSE). However, since NMF methods require maintaining a global matrix for learning, they suffer from memory and scale issue. This problem has been addressed in BIBREF20 where an online method of learning interpretable word embeddings from corpora using a modified version of skip-gram model BIBREF0 is proposed. As a different approach, BIBREF21 combined text-based similarity information among words with brain activity based similarity information to improve interpretability using joint non-negative sparse embedding (JNNSE). A common alternative approach for learning interpretable embeddings is to learn transformations that map pre-trained state-of-the-art embeddings to new interpretable semantic spaces. To obtain sparse, higher dimensional and more interpretable vector spaces, BIBREF22 and BIBREF23 use sparse coding on conventional dense word embeddings. However, these methods learn the projection vectors that are used for the transformation from the word embeddings without supervision. For this reason, labels describing the corresponding semantic categories cannot be provided. An alternative approach was proposed in BIBREF25 , where orthogonal transformations were utilized to increase interpretability while preserving the performance of the underlying embedding. However, BIBREF25 has also shown that total interpretability of an embedding is kept constant under any orthogonal transformation and it can only be redistributed across the dimensions. Rotation algorithms based on exploratory factor analysis (EFA) to preserve the performance of the original word embeddings while improving their interpretability was proposed in BIBREF26 . BIBREF24 proposed to deploy a sparse auto-encoder using pre-trained dense word embeddings to improve interpretability. More detailed investigation of semantic structure and interpretability of word embeddings can be found in BIBREF27 , where a metric was proposed to quantitatively measure the degree of interpretability already present in the embedding vector spaces. Previous works on interpretability mentioned above, except BIBREF21 , BIBREF27 and our proposed method, do not need external resources, utilization of which has both advantages and disadvantages. Methods that do not use external resources require fewer resources but they also lack the aid of information extracted from these resources. Problem Description For the task of unsupervised word embedding extraction, we operate on a discrete collection of lexical units (words) INLINEFORM0 that is part of an input corpus INLINEFORM1 , with number of tokens INLINEFORM2 , sourced from a vocabulary INLINEFORM3 of size INLINEFORM4 . In the setting of distributional semantics, the objective of a word embedding algorithm is to maximize some aggregate utility over the entire corpus so that some measure of “closeness” is maximized for pairs of vector representations INLINEFORM14 for words which, on the average, appear in proximity to one another. In the GloVe algorithm BIBREF2 , which we base our improvements upon, the following objective function is considered: DISPLAYFORM0 In ( EQREF6 ), INLINEFORM0 and INLINEFORM1 stand for word and context vector representations, respectively, for words INLINEFORM2 and INLINEFORM3 , while INLINEFORM4 represents the (possibly weighted) cooccurrence count for the word pair INLINEFORM5 . Intuitively, ( EQREF6 ) represents the requirement that if some word INLINEFORM6 occurs often enough in the context (or vicinity) of another word INLINEFORM7 , then the corresponding word representations should have a large enough inner product in keeping with their large INLINEFORM8 value, up to some bias terms INLINEFORM9 ; and vice versa. INLINEFORM10 in ( EQREF6 ) is used as a discounting factor that prohibits rare cooccurrences from disproportionately influencing the resulting embeddings. The objective ( EQREF6 ) is minimized using stochastic gradient descent by iterating over the matrix of cooccurrence records INLINEFORM0 . In the GloVe algorithm, for a given word INLINEFORM1 , the final word representation is taken to be the average of the two intermediate vector representations obtained from ( EQREF6 ); i.e, INLINEFORM2 . In the next section, we detail the enhancements made to ( EQREF6 ) for the purposes of enhanced interpretability, using the aforementioned framework as our basis. Imparting Interpretability Our approach falls into a joint-learning framework where the distributional information extracted from the corpus is allowed to fuse with the external lexicon-based information. Word-groups extracted from Roget's Thesaurus are directly mapped to individual dimensions of word embeddings. Specifically, the vector representations of words that belong to a particular group are encouraged to have deliberately increased values in a particular dimension that corresponds to the word-group under consideration. This can be achieved by modifying the objective function of the embedding algorithm to partially influence vector representation distributions across their dimensions over an input vocabulary. To do this, we propose the following modification to the GloVe objective in ( EQREF6 ): rCl J = i,j=1V f(Xij)[ (wiTwj + bi + bj -Xij)2 + k(l=1D INLINEFORM0 iFl g(wi,l) + l=1D INLINEFORM1 j Fl g(wj,l) ) ]. In ( SECREF4 ), INLINEFORM2 denotes the indices for the elements of the INLINEFORM3 th concept word-group which we wish to assign in the vector dimension INLINEFORM4 . The objective ( SECREF4 ) is designed as a mixture of two individual cost terms: the original GloVe cost term along with a second term that encourages embedding vectors of a given concept word-group to achieve deliberately increased values along an associated dimension INLINEFORM5 . The relative weight of the second term is controlled by the parameter INLINEFORM6 . The simultaneous minimization of both objectives ensures that words that are similar to, but not included in, one of these concept word-groups are also “nudged” towards the associated dimension INLINEFORM7 . The trained word vectors are thus encouraged to form a distribution where the individual vector dimensions align with certain semantic concepts represented by a collection of concept word-groups, one assigned to each vector dimension. To facilitate this behaviour, ( SECREF4 ) introduces a monotone decreasing function INLINEFORM8 defined as INLINEFORM9 which serves to increase the total cost incurred if the value of the INLINEFORM0 th dimension for the two vector representations INLINEFORM1 and INLINEFORM2 for a concept word INLINEFORM3 with INLINEFORM4 fails to be large enough. INLINEFORM5 is also shown in Fig. FIGREF7 . The objective ( SECREF4 ) is minimized using stochastic gradient descent over the cooccurrence records INLINEFORM0 . Intuitively, the terms added to ( SECREF4 ) in comparison with ( EQREF6 ) introduce the effect of selectively applying a positive step-type input to the original descent updates of ( EQREF6 ) for concept words along their respective vector dimensions, which influences the dimension value in the positive direction. The parameter INLINEFORM1 in ( SECREF4 ) allows for the adjustment of the magnitude of this influence as needed. In the next section, we demonstrate the feasibility of this approach by experiments with an example collection of concept word-groups extracted from Roget's Thesaurus. Experiments and Results We first identified 300 concepts, one for each dimension of the 300-dimensional vector representation, by employing Roget's Thesaurus. This thesaurus follows a tree structure which starts with a Root node that contains all the words and phrases in the thesaurus. The root node is successively split into Classes and Sections, which are then (optionally) split into Subsections of various depths, finally ending in Categories, which constitute the smallest unit of word/phrase collections in the structure. The actual words and phrases descend from these Categories, and make up the leaves of the tree structure. We note that a given word typically appears in multiple categories corresponding to the different senses of the word. We constructed concept word-groups from Roget's Thesaurus as follows: We first filtered out the multi-word phrases and the relatively obscure terms from the thesaurus. The obscure terms were identified by checking them against a vocabulary extracted from Wikipedia. We then obtained 300 word-groups as the result of a partitioning operation applied to the subtree that ends with categories as its leaves. The partition boundaries, hence the resulting word-groups, can be chosen in many different ways. In our proposed approach, we have chosen to determine this partitioning by traversing this tree structure from the root node in breadth-first order, and by employing a parameter INLINEFORM0 for the maximum size of a node. Here, the size of a node is defined as the number of unique words that ever-descend from that node. During the traversal, if the size of a given node is less than this threshold, we designate the words that ultimately descend from that node as a concept word-group. Otherwise, if the node has children, we discard the node, and queue up all its children for further consideration. If this node does not have any children, on the other hand, the node is truncated to INLINEFORM1 elements with the highest frequency-ranks, and the resulting words are designated as a concept word-group. We note that the choice of INLINEFORM2 greatly affects the resulting collection of word-groups: Excessively large values result in few word-groups that greatly overlap with one another, while overly small values result in numerous tiny word-groups that fail to adequately represent a concept. We experimentally determined that a INLINEFORM3 value of 452 results in the most healthy number of relatively large word-groups (113 groups with size INLINEFORM4 100), while yielding a preferably small overlap amongst the resulting word-groups (with average overlap size not exceeding 3 words). A total of 566 word-groups were thus obtained. 259 smallest word-groups (with size INLINEFORM5 38) were discarded to bring down the number of word-groups to 307. Out of these, 7 groups with the lowest median frequency-rank were further discarded, which yields the final 300 concept word-groups used in the experiments. We present some of the resulting word-groups in Table TABREF9 . By using the concept word-groups, we have trained the GloVe algorithm with the proposed modification given in Section SECREF4 on a snapshot of English Wikipedia measuring 8GB in size, with the stop-words filtered out. Using the parameters given in Table TABREF10 , this resulted in a vocabulary size of 287,847. For the weighting parameter in Eq. SECREF4 , we used a value of INLINEFORM0 . The algorithm was trained over 20 iterations. The GloVe algorithm without any modifications was also trained as a baseline with the same parameters. In addition to the original GloVe algorithm, we compare our proposed method with previous studies that aim to obtain interpretable word vectors. We train the improved projected gradient model proposed in BIBREF20 to obtain word vectors (called OIWE-IPG) using the same corpus we use to train GloVe and our proposed method. Using the methods proposed in BIBREF23 , BIBREF26 , BIBREF24 on our baseline GloVe embeddings, we obtain SOV, SPINE and Parsimax (orthogonal) word representations, respectively. We train all the models with the proposed parameters. However, in BIBREF26 , the authors show results for a relatively small vocabulary of 15,000 words. When we trained their model on our baseline GloVe embeddings with a large vocabulary of size 287,847, the resulting vectors performed significantly poor on word similarity tasks compared to the results presented in their paper. In addition, Parsimax (orthogonal) word vectors obtained using method in BIBREF26 are nearly identical to the baseline vectors (i.e. learned orthogonal transformation matrix is very close to identity). Therefore, Parsimax (orthogonal) yields almost same results with baseline vectors in all evaluations. We evaluate the interpretability of the resulting embeddings qualitatively and quantitatively. We also test the performance of the embeddings on word similarity and word analogy tests. In our experiments, vocabulary size is close to 300,000 while only 16,242 unique words of the vocabulary are present in the concept groups. Furthermore, only dimensions that correspond to the concept group of the word will be updated due to the additional cost term. Given that these concept words can belong to multiple concept groups (2 on average), only 33,319 parameters are updated. There are 90 million individual parameters present for the 300,000 word vectors of size 300. Of these parameters, only approximately 33,000 are updated by the additional cost term. Qualitative Evaluation for Interpretability In Fig. FIGREF13 , we demonstrate the particular way in which the proposed algorithm ( SECREF4 ) influences the vector representation distributions. Specifically, we consider, for illustration, the 32nd dimension values for the original GloVe algorithm and our modified version, restricting the plots to the top-1000 words with respect to their frequency ranks for clarity of presentation. In Fig. FIGREF13 , the words in the horizontal axis are sorted in descending order with respect to the values at the 32nd dimension of their word embedding vectors coming from the original GloVe algorithm. The dimension values are denoted with blue and red/green markers for the original and the proposed algorithms, respectively. Additionally, the top-50 words that achieve the greatest 32nd dimension values among the considered 1000 words are emphasized with enlarged markers, along with text annotations. In the presented simulation of the proposed algorithm, the 32nd dimension values are encoded with the concept JUDGMENT, which is reflected as an increase in the dimension values for words such as committee, academy, and article. We note that these words (red) are not part of the pre-determined word-group for the concept JUDGMENT, in contrast to words such as award, review and account (green) which are. This implies that the increase in the corresponding dimension values seen for these words is attributable to the joint effect of the first term in ( SECREF4 ) which is inherited from the original GloVe algorithm, in conjunction with the remaining terms in the proposed objective expression ( SECREF4 ). This experiment illustrates that the proposed algorithm is able to impart the concept of JUDGMENT on its designated vector dimension above and beyond the supplied list of words belonging to the concept word-group for that dimension. We also present the list of words with the greatest dimension value for the dimensions 11, 13, 16, 31, 36, 39, 41, 43 and 79 in Table TABREF11 . These dimensions are aligned/imparted with the concepts that are given in the column headers. In Table TABREF11 , the words that are highlighted with green denote the words that exist in the corresponding word-group obtained from Roget's Thesaurus (and are thus explicitly forced to achieve increased dimension values), while the red words denote the words that achieve increased dimension values by virtue of their cooccurrence statistics with the thesaurus-based words (indirectly, without being explicitly forced). This again illustrates that a semantic concept can indeed be coded to a vector dimension provided that a sensible lexical resource is used to guide semantically related words to the desired vector dimension via the proposed objective function in ( SECREF4 ). Even the words that do not appear in, but are semantically related to, the word-groups that we formed using Roget's Thesaurus, are indirectly affected by the proposed algorithm. They also reflect the associated concepts at their respective dimensions even though the objective functions for their particular vectors are not modified. This point cannot be overemphasized. Although the word-groups extracted from Roget's Thesaurus impose a degree of supervision to the process, the fact that the remaining words in the entire vocabulary are also indirectly affected makes the proposed method a semi-supervised approach that can handle words that are not in these chosen word-groups. A qualitative example of this result can be seen in the last column of Table TABREF11 . It is interesting to note the appearance of words such as guerilla, insurgency, mujahideen, Wehrmacht and Luftwaffe in addition to the more obvious and straightforward army, soldiers and troops, all of which are not present in the associated word-group WARFARE. Most of the dimensions we investigated exhibit similar behaviour to the ones presented in Table TABREF11 . Thus generally speaking, we can say that the entries in Table TABREF11 are representative of the great majority. However, we have also specifically looked for dimensions that make less sense and determined a few such dimensions which are relatively less satisfactory. These less satisfactory examples are given in Table TABREF14 . These examples are also interesting in that they shed insight into the limitations posed by polysemy and existence of very rare outlier words. Quantitative Evaluation for Interpretability One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon. A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT. Interpretability scores are calculated using Interpretability Score (IS) as given below: DISPLAYFORM0 In ( EQREF17 ), INLINEFORM0 and INLINEFORM1 represents the interpretability scores in the positive and negative directions of the INLINEFORM2 dimension ( INLINEFORM3 , INLINEFORM4 number of dimensions in the embedding space) of word embedding space for the INLINEFORM5 category ( INLINEFORM6 , INLINEFORM7 is number of categories in SEMCAT, INLINEFORM8 ) in SEMCAT respectively. INLINEFORM9 is the set of words in the INLINEFORM10 category in SEMCAT and INLINEFORM11 is the number of words in INLINEFORM12 . INLINEFORM13 corresponds to the minimum number of words required to construct a semantic category (i.e. represent a concept). INLINEFORM14 represents the set of INLINEFORM15 words that have the highest ( INLINEFORM16 ) and lowest ( INLINEFORM17 ) values in INLINEFORM18 dimension of the embedding space. INLINEFORM19 is the intersection operator and INLINEFORM20 is the cardinality operator (number of elements) for the intersecting set. In ( EQREF17 ), INLINEFORM21 gives the interpretability score for the INLINEFORM22 dimension and INLINEFORM23 gives the average interpretability score of the embedding space. Fig. FIGREF18 presents the measured average interpretability scores across dimensions for original GloVe embeddings, for the proposed method and for the other four methods we compare, along with a randomly generated embedding. Results are calculated for the parameters INLINEFORM0 and INLINEFORM1 . Our proposed method significantly improves the interpretability for all INLINEFORM2 compared to the original GloVe approach. Our proposed method is second to only SPINE in increasing interpretability. However, as we will experimentally demonstrate in the next subsection, in doing this, SPINE almost entirely destroys the underlying semantic structure of the word embeddings, which is the primary function of a word embedding. The proposed method and interpretability measurements are both based on utilizing concepts represented by word-groups. Therefore it is expected that there will be higher interpretability scores for some of the dimensions for which the imparted concepts are also contained in SEMCAT. However, by design, word groups that they use are formed by using different sources and are independent. Interpretability measurements use SEMCAT while our proposed method utilizes Roget's Thesaurus. Intrinsic Evaluation of the Embeddings It is necessary to show that the semantic structure of the original embedding has not been damaged or distorted as a result of aligning the dimensions with given concepts, and that there is no substantial sacrifice involved from the performance that can be obtained with the original GloVe. To check this, we evaluate performances of the proposed embeddings on word similarity BIBREF42 and word analogy BIBREF0 tests. We compare the results with the original embeddings and the three alternatives excluding Parsimax BIBREF26 since orthogonal transformations will not affect the performance of the original embeddings on these tests. Word similarity test measures the correlation between word similarity scores obtained from human evaluation (i.e. true similarities) and from word embeddings (usually using cosine similarity). In other words, this test quantifies how well the embedding space reflects human judgements in terms of similarities between different words. The correlation scores for 13 different similarity test sets are reported in Table TABREF20 . We observe that, let alone a reduction in performance, the obtained scores indicate an almost uniform improvement in the correlation values for the proposed algorithm, outperforming all the alternatives in almost all test sets. Categories from Roget's thesaurus are groupings of words that are similar in some sense which the original embedding algorithm may fail to capture. These test results signify that the semantic information injected into the algorithm by the additional cost term is significant enough to result in a measurable improvement. It should also be noted that scores obtained by SPINE is unacceptably low on almost all tests indicating that it has achieved its interpretability performance at the cost of losing its semantic functions. Word analogy test is introduced in BIBREF1 and looks for the answers of the questions that are in the form of "X is to Y, what Z is to ?" by applying simple arithmetic operations to vectors of words X, Y and Z. We present precision scores for the word analogy tests in Table TABREF21 . It can be seen that the alternative approaches that aim to improve interpretability, have poor performance on the word analogy tests. However, our proposed method has comparable performance with the original GloVe embeddings. Our method outperforms GloVe in semantic analogy test set and in overall results, while GloVe performs slightly better in syntactic test set. This comparable performance is mainly due to the cost function of our proposed method that includes the original objective of the GloVe. To investigate the effect of the additional cost term on the performance improvement in the semantic analogy test, we present Table TABREF22 . In particular, we present results for the cases where i) all questions in the dataset are considered, ii) only the questions that contains at least one concept word are considered, iii) only the questions that consist entirely of concept words are considered. We note specifically that for the last case, only a subset of the questions under the semantic category family.txt ended up being included. We observe that for all three scenarios, our proposed algorithm results in an improvement in the precision scores. However, the greatest performance increase is seen for the last scenario, which underscores the extent to which the semantic features captured by embeddings can be improved with a reasonable selection of the lexical resource from which the concept word-groups were derived. Conclusion We presented a novel approach to impart interpretability into word embeddings. We achieved this by encouraging different dimensions of the vector representation to align with predefined concepts, through the addition of an additional cost term in the optimization objective of the GloVe algorithm that favors a selective increase for a pre-specified input of concept words along each dimension. We demonstrated the efficacy of this approach by applying qualitative and quantitative evaluations for interpretability. We also showed via standard word-analogy and word-similarity tests that the semantic coherence of the original vector space is preserved, even slightly improved. We have also performed and reported quantitative comparisons with several other methods for both interpretabilty increase and preservation of semantic coherence. Upon inspection of Fig. FIGREF18 and Tables TABREF20 , TABREF21 , and TABREF22 altogether, it should be noted that our proposed method achieves both of the objectives simultaneously, increased interpretability and preservation of the intrinsic semantic structure. An important point was that, while it is expected for words that are already included in the concept word-groups to be aligned together since their dimensions are directly updated with the proposed cost term, it was also observed that words not in these groups also aligned in a meaningful manner without any direct modification to their cost function. This indicates that the cost term we added works productively with the original cost function of GloVe to handle words that are not included in the original concept word-groups, but are semantically related to those word-groups. The underlying mechanism can be explained as follows. While the outside lexical resource we introduce contains a relatively small number of words compared to the total number of words, these words and the categories they represent have been carefully chosen and in a sense, "densely span" all the words in the language. By saying "span", we mean they cover most of the concepts and ideas in the language without leaving too many uncovered areas. With "densely" we mean all areas are covered with sufficient strength. In other words, this subset of words is able to constitute a sufficiently strong skeleton, or scaffold. Now remember that GloVe works to align or bring closer related groups of words, which will include words from the lexical source. So the joint action of aligning the words with the predefined categories (introduced by us) and aligning related words (handled by GloVe) allows words not in the lexical groups to also be aligned meaningfully. We may say that the non-included words are "pulled along" with the included words by virtue of the "strings" or "glue" that is provided by GloVe. In numbers, the desired effect is achieved by manipulating less than only 0.05% of parameters of the entire word vectors. Thus, while there is a degree of supervision coming from the external lexical resource, the rest of the vocabulary is also aligned indirectly in an unsupervised way. This may be the reason why, unlike earlier proposed approaches, our method is able to achieve increasing interpretability without destroying underlying semantic structure, and consequently without sacrificing performance in benchmark tests. Upon inspecting the 2nd column of Table TABREF14 , where qualitative results for concept TASTE are presented, another insight regarding the learning mechanism of our proposed approach can be made. Here it seems understandable that our proposed approach, along with GloVe, brought together the words taste and polish, and then the words Polish and, for instance, Warsaw are brought together by GloVe. These examples are interesting in that they shed insight into how GloVe works and the limitations posed by polysemy. It should be underlined that the present approach is not totally incapable of handling polysemy, but cannot do so perfectly. Since related words are being clustered, sufficiently well-connected words that do not meaningfully belong along with others will be appropriately "pulled away" from that group by several words, against the less effective, inappropriate pull of a particular word. Even though polish with lowercase "p" belongs where it is, it is attracting Warsaw to itself through polysemy and this is not meaningful. Perhaps because Warsaw is not a sufficiently well-connected word, it ends being dragged along, although words with greater connectedness to a concept group might have better resisted such inappropriate attractions. In this study, we used the GloVe algorithm as the underlying dense word embedding scheme to demonstrate our approach. However, we stress that it is possible for our approach to be extended to other word embedding algorithms which have a learning routine consisting of iterations over cooccurrence records, by making suitable adjustments in the objective function. Since word2vec model is also based on the coocurrences of words in a sliding window through a large corpus, we expect that our approach can also be applied to word2vec after making suitable adjustments, which can be considered as an immediate future work for our approach. Although the semantic concepts are encoded in only one direction (positive) within the embedding dimensions, it might be beneficial to pursue future work that also encodes opposite concepts, such as good and bad, in two opposite directions of the same dimension. The proposed methodology can also be helpful in computational cross-lingual studies, where the similarities are explored across the vector spaces of different languages BIBREF43 , BIBREF44 .
What experiments do they use to quantify the extent of interpretability?
17a1eff7993c47c54eddc7344e7454fbe64191cd
What Ġexperiments Ġdo Ġthey Ġuse Ġto Ġquantify Ġthe Ġextent Ġof Ġinterpret ability ?
Human Ġevaluation Ġfor Ġinterpret ability Ġusing Ġthe Ġword Ġintrusion Ġtest Ġand Ġautomated Ġevaluation Ġfor Ġinterpret ability Ġusing Ġa Ġsemantic Ġcategory - based Ġapproach Ġbased Ġon Ġthe Ġmethod Ġand Ġcategory Ġdataset Ġ( S EM C AT ).
<s> What Ġexperiments Ġdo Ġthey Ġuse Ġto Ġquantify Ġthe Ġextent Ġof Ġinterpret ability ? </s> Introduction </s> Dist ributed Ġword Ġrepresentations , Ġcommonly Ġreferred Ġto Ġas Ġword Ġembed d ings ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġserve Ġas Ġelementary Ġbuilding Ġblocks Ġin Ġthe Ġcourse Ġof Ġalgorithm Ġdesign Ġfor Ġan Ġexpanding Ġrange Ġof Ġapplications Ġin Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġincluding Ġnamed Ġentity Ġrecognition ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġparsing ĠB IB REF 6 Ġ, Ġsentiment Ġanalysis ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, Ġand Ġword - sense Ġdis amb ig uation ĠB IB REF 9 Ġ. ĠAlthough Ġthe Ġempirical Ġutility Ġof Ġword Ġembed d ings Ġas Ġan Ġun super vised Ġmethod Ġfor Ġcapturing Ġthe Ġsemantic Ġor Ġsynt actic Ġfeatures Ġof Ġa Ġcertain Ġword Ġas Ġit Ġis Ġused Ġin Ġa Ġgiven Ġlex ical Ġresource Ġis Ġwell - established ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, Ġan Ġunderstanding Ġof Ġwhat Ġthese Ġfeatures Ġmean Ġremains Ġan Ġopen Ġproblem ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġas Ġsuch Ġword Ġembed d ings Ġmostly Ġremain Ġa Ġblack Ġbox . ĠIt Ġis Ġdesirable Ġto Ġbe Ġable Ġto Ġdevelop Ġinsight Ġinto Ġthis Ġblack Ġbox Ġand Ġbe Ġable Ġto Ġinterpret Ġwhat Ġit Ġmeans , Ġwhile Ġretaining Ġthe Ġutility Ġof Ġword Ġembed d ings Ġas Ġsem antically - rich Ġintermediate Ġrepresentations . ĠOther Ġthan Ġthe Ġintrinsic Ġvalue Ġof Ġthis Ġinsight , Ġthis Ġwould Ġnot Ġonly Ġallow Ġus Ġto Ġexplain Ġand Ġunderstand Ġhow Ġalgorithms Ġwork ĠB IB REF 15 Ġ, Ġbut Ġalso Ġset Ġa Ġground Ġthat Ġwould Ġfacilitate Ġthe Ġdesign Ġof Ġnew Ġalgorithms Ġin Ġa Ġmore Ġdeliberate Ġway . </s> Recent Ġapproaches Ġto Ġgenerating Ġword Ġembed d ings Ġ( e . g . ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ) Ġare Ġrooted Ġlingu istically Ġin Ġthe Ġfield Ġof Ġdistributed Ġsemantics ĠB IB REF 16 Ġ, Ġwhere Ġwords Ġare Ġtaken Ġto Ġassume Ġmeaning Ġmainly Ġby Ġtheir Ġdegree Ġof Ġinteraction Ġ( or Ġlack Ġthereof ) Ġwith Ġother Ġwords Ġin Ġthe Ġlex icon ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠUnder Ġthis Ġparadigm , Ġdense , Ġcontinuous Ġvector Ġrepresentations Ġare Ġlearned Ġin Ġan Ġun super vised Ġmanner Ġfrom Ġa Ġlarge Ġcorpus , Ġusing Ġthe Ġword Ġco occ urrence Ġstatistics Ġdirectly Ġor Ġindirectly , Ġand Ġsuch Ġan Ġapproach Ġis Ġshown Ġto Ġresult Ġin Ġvector Ġrepresentations Ġthat Ġmat hematically Ġcapture Ġvarious Ġsemantic Ġand Ġsynt actic Ġrelations Ġbetween Ġwords ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠHowever , Ġthe Ġdense Ġnature Ġof Ġthe Ġlearned Ġembed d ings Ġobfusc ate Ġthe Ġdistinct Ġconcepts Ġencoded Ġin Ġthe Ġdifferent Ġdimensions , Ġwhich Ġrenders Ġthe Ġresulting Ġvectors Ġvirtually Ġun interpret able . ĠThe Ġlearned Ġembed d ings Ġmake Ġsense Ġonly Ġin Ġrelation Ġto Ġeach Ġother Ġand Ġtheir Ġspecific Ġdimensions Ġdo Ġnot Ġcarry Ġexplicit Ġinformation Ġthat Ġcan Ġbe Ġinterpreted . ĠHowever , Ġbeing Ġable Ġto Ġinterpret Ġa Ġword Ġembed ding Ġwould Ġilluminate Ġthe Ġsemantic Ġconcepts Ġimplicitly Ġrepresented Ġalong Ġthe Ġvarious Ġdimensions Ġof Ġthe Ġembed ding , Ġand Ġreveal Ġits Ġhidden Ġsemantic Ġstructures . </s> In Ġthe Ġliterature , Ġresearchers Ġtackled Ġinterpret ability Ġproblem Ġof Ġthe Ġword Ġembed d ings Ġusing Ġdifferent Ġapproaches . ĠSeveral Ġresearchers ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġproposed Ġalgorithms Ġbased Ġon Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġapplied Ġto Ġco occ urrence Ġvariant Ġmat rices . ĠOther Ġresearchers Ġsuggested Ġto Ġobtain Ġinterpret able Ġword Ġvectors Ġfrom Ġexisting Ġun interpret able Ġword Ġvectors Ġby Ġapplying Ġsparse Ġcoding ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, Ġby Ġtraining Ġa Ġsparse Ġauto - enc oder Ġto Ġtransform Ġthe Ġembed ding Ġspace ĠB IB REF 24 Ġ, Ġby Ġrotating Ġthe Ġoriginal Ġembed d ings ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġor Ġby Ġapplying Ġtransformations Ġbased Ġon Ġexternal Ġsemantic Ġdatasets ĠB IB REF 27 Ġ. </s> Although Ġthe Ġabove - mentioned Ġapproaches Ġprovide Ġbetter Ġinterpret ability Ġthat Ġis Ġmeasured Ġusing Ġa Ġparticular Ġmethod Ġsuch Ġas Ġword Ġintrusion Ġtest , Ġusually Ġthe Ġimproved Ġinterpret ability Ġcomes Ġwith Ġa Ġcost Ġof Ġperformance Ġin Ġthe Ġbenchmark Ġtests Ġsuch Ġas Ġword Ġsimilarity Ġor Ġword Ġanalogy . ĠOne Ġpossible Ġexplanation Ġfor Ġthis Ġperformance Ġdecrease Ġis Ġthat Ġthe Ġproposed Ġtransformations Ġfrom Ġthe Ġoriginal Ġembed ding Ġspace Ġdistort Ġthe Ġunderlying Ġsemantic Ġstructure Ġconstructed Ġby Ġthe Ġoriginal Ġembed ding Ġalgorithm . ĠTherefore , Ġit Ġcan Ġbe Ġclaimed Ġthat Ġa Ġmethod Ġthat Ġlearns Ġdense Ġand Ġinterpret able Ġword Ġembed d ings Ġwithout Ġinflicting Ġany Ġdamage Ġto Ġthe Ġunderlying Ġsemantic Ġlearning Ġmechanism Ġis Ġthe Ġkey Ġto Ġachieve Ġboth Ġhigh Ġperforming Ġand Ġinterpret able Ġword Ġembed d ings . </s> Especially Ġafter Ġthe Ġintroduction Ġof Ġthe Ġword 2 vec Ġalgorithm Ġby ĠMik ol ov ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġthere Ġhas Ġbeen Ġa Ġgrowing Ġinterest Ġin Ġalgorithms Ġthat Ġgenerate Ġimproved Ġword Ġrepresentations Ġunder Ġsome Ġperformance Ġmetric . ĠSignificant Ġeffort Ġis Ġspent Ġon Ġappropriately Ġmodifying Ġthe Ġobjective Ġfunctions Ġof Ġthe Ġalgorithms Ġin Ġorder Ġto Ġincorporate Ġknowledge Ġfrom Ġexternal Ġresources , Ġwith Ġthe Ġpurpose Ġof Ġincreasing Ġthe Ġperformance Ġof Ġthe Ġresulting Ġword Ġrepresentations ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ. ĠInspired Ġby Ġthe Ġline Ġof Ġwork Ġreported Ġin Ġthese Ġstudies , Ġwe Ġpropose Ġto Ġuse Ġmodified Ġobjective Ġfunctions Ġfor Ġa Ġdifferent Ġpurpose : Ġlearning Ġmore Ġinterpret able Ġdense Ġword Ġembed d ings . ĠBy Ġdoing Ġthis , Ġwe Ġaim Ġto Ġincorporate Ġsemantic Ġinformation Ġfrom Ġan Ġexternal Ġlex ical Ġresource Ġinto Ġthe Ġword Ġembed ding Ġso Ġthat Ġthe Ġembed ding Ġdimensions Ġare Ġaligned Ġalong Ġpred efined Ġconcepts . ĠThis Ġalignment Ġis Ġachieved Ġby Ġintroducing Ġa Ġmodification Ġto Ġthe Ġembed ding Ġlearning Ġprocess . ĠIn Ġour Ġproposed Ġmethod , Ġwhich Ġis Ġbuilt Ġon Ġtop Ġof Ġthe ĠGl o Ve Ġalgorithm ĠB IB REF 2 Ġ, Ġthe Ġcost Ġfunction Ġfor Ġany Ġone Ġof Ġthe Ġwords Ġof Ġconcept Ġword - groups Ġis Ġmodified Ġby Ġthe Ġintroduction Ġof Ġan Ġadditive Ġterm Ġto Ġthe Ġcost Ġfunction . ĠEach Ġembed ding Ġvector Ġdimension Ġis Ġfirst Ġassociated Ġwith Ġa Ġconcept . ĠFor Ġa Ġword Ġbelonging Ġto Ġany Ġone Ġof Ġthe Ġword - groups Ġrepresenting Ġthese Ġconcepts , Ġthe Ġmodified Ġcost Ġterm Ġfavors Ġan Ġincrease Ġfor Ġthe Ġvalue Ġof Ġthis Ġword 's Ġembed ding Ġvector Ġdimension Ġcorresponding Ġto Ġthe Ġconcept Ġthat Ġthe Ġparticular Ġword Ġbelongs Ġto . ĠFor Ġwords Ġthat Ġdo Ġnot Ġbelong Ġto Ġany Ġone Ġof Ġthe Ġword - groups , Ġthe Ġcost Ġterm Ġis Ġleft Ġuntouched . ĠSpecifically , ĠRog et 's ĠThe s aurus ĠB IB REF 38 Ġ, ĠB IB REF 39 Ġis Ġused Ġto Ġderive Ġthe Ġconcepts Ġand Ġconcept Ġword - groups Ġto Ġbe Ġused Ġas Ġthe Ġexternal Ġlex ical Ġresource Ġfor Ġour Ġproposed Ġmethod . ĠWe Ġquant itatively Ġdemonstrate Ġthe Ġincrease Ġin Ġinterpret ability Ġby Ġusing Ġthe Ġmeasure Ġgiven Ġin ĠB IB REF 27 Ġ, ĠB IB REF 40 Ġas Ġwell Ġas Ġdemonstrating Ġqualitative Ġresults . ĠWe Ġalso Ġshow Ġthat Ġthe Ġsemantic Ġstructure Ġof Ġthe Ġoriginal Ġembed ding Ġhas Ġnot Ġbeen Ġharmed Ġin Ġthe Ġprocess Ġsince Ġthere Ġis Ġno Ġperformance Ġloss Ġwith Ġstandard Ġword - similar ity Ġor Ġword - an alogy Ġtests . </s> The Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠIn ĠSection ĠSEC REF 2 Ġ, Ġwe Ġdiscuss Ġprevious Ġstudies Ġrelated Ġto Ġour Ġwork Ġunder Ġtwo Ġmain Ġcategories : Ġinterpret ability Ġof Ġword Ġembed d ings Ġand Ġjoint - learning Ġframeworks Ġwhere Ġthe Ġobjective Ġfunction Ġis Ġmodified . ĠIn ĠSection ĠSEC REF 3 Ġ, Ġwe Ġpresent Ġthe Ġproblem Ġframework Ġand Ġprovide Ġthe Ġformulation Ġwithin Ġthe ĠGl o Ve ĠB IB REF 2 Ġalgorithm Ġsetting . ĠIn ĠSection ĠSEC REF 4 Ġwhere Ġour Ġapproach Ġis Ġproposed , Ġwe Ġmotivate Ġand Ġdevelop Ġa Ġmodification Ġto Ġthe Ġoriginal Ġobjective Ġfunction Ġwith Ġthe Ġaim Ġof Ġincreasing Ġrepresentation Ġinterpret ability . ĠIn ĠSection ĠSEC REF 5 Ġ, Ġexperimental Ġresults Ġare Ġprovided Ġand Ġthe Ġproposed Ġmethod Ġis Ġquant itatively Ġand Ġqual itatively Ġevaluated . ĠAdditionally , Ġin ĠSection ĠSEC REF 5 Ġ, Ġresults Ġdemonstrating Ġthe Ġextent Ġto Ġwhich Ġthe Ġoriginal Ġsemantic Ġstructure Ġof Ġthe Ġembed ding Ġspace Ġis Ġaffected Ġare Ġpresented Ġby Ġusing Ġword - an alogy Ġand Ġword - similar ity Ġtests . ĠWe Ġconclude Ġthe Ġpaper Ġin ĠSection ĠSEC REF 6 Ġ. </s> Related ĠWork </s> Method ologically , Ġour Ġwork Ġis Ġrelated Ġto Ġprior Ġstudies Ġthat Ġaim Ġto Ġobtain ĠâĢ ľ impro ved âĢ Ŀ Ġword Ġembed d ings Ġusing Ġexternal Ġlex ical Ġresources , Ġunder Ġsome Ġperformance Ġmetric . ĠPrevious Ġwork Ġin Ġthis Ġarea Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġmain Ġcategories : Ġworks Ġthat Ġi ) Ġmodify Ġthe Ġword Ġembed ding Ġlearning Ġalgorithm Ġto Ġincorporate Ġlex ical Ġinformation , Ġii ) Ġoperate Ġon Ġpre - trained Ġembed d ings Ġwith Ġa Ġpost - processing Ġstep . </s> Among Ġworks Ġthat Ġfollow Ġthe Ġfirst Ġapproach , ĠB IB REF 28 Ġextend Ġthe ĠSkip - G ram Ġmodel Ġby Ġincorporating Ġthe Ġword Ġsimilarity Ġrelations Ġextracted Ġfrom Ġthe ĠPar aph rase ĠDatabase Ġ( P PD B ) Ġand ĠWord Net ĠB IB REF 29 Ġ, Ġinto Ġthe ĠSkip - G ram Ġpredictive Ġmodel Ġas Ġan Ġadditional Ġcost Ġterm . ĠIn ĠB IB REF 30 Ġ, Ġthe Ġauthors Ġextend Ġthe ĠCB OW Ġmodel Ġby Ġconsidering Ġtwo Ġtypes Ġof Ġsemantic Ġinformation , Ġtermed Ġrelational Ġand Ġcateg orical , Ġto Ġbe Ġincorporated Ġinto Ġthe Ġembed d ings Ġduring Ġtraining . ĠFor Ġthe Ġformer Ġtype Ġof Ġsemantic Ġinformation , Ġthe Ġauthors Ġpropose Ġthe Ġlearning Ġof Ġexplicit Ġvectors Ġfor Ġthe Ġdifferent Ġrelations Ġextracted Ġfrom Ġa Ġsemantic Ġlex icon Ġsuch Ġthat Ġthe Ġword Ġpairs Ġthat Ġsatisfy Ġthe Ġsame Ġrelation Ġare Ġdistributed Ġmore Ġhom ogene ously . ĠFor Ġthe Ġlatter , Ġthe Ġauthors Ġmodify Ġthe Ġlearning Ġobjective Ġsuch Ġthat Ġsome Ġweighted Ġaverage Ġdistance Ġis Ġminimized Ġfor Ġwords Ġunder Ġthe Ġsame Ġsemantic Ġcategory . ĠIn ĠB IB REF 31 Ġ, Ġthe Ġauthors Ġrepresent Ġthe Ġsyn ony my Ġand Ġhyper ny my - hyp ony my Ġrelations Ġin Ġterms Ġof Ġinequality Ġconstraints , Ġwhere Ġthe Ġpair wise Ġsimilarity Ġrankings Ġover Ġword Ġtriple ts Ġare Ġforced Ġto Ġfollow Ġan Ġorder Ġextracted Ġfrom Ġa Ġlex ical Ġresource . ĠFollowing Ġtheir Ġextraction Ġfrom ĠWord Net , Ġthe Ġauthors Ġimpose Ġthese Ġconstraints Ġin Ġthe Ġform Ġof Ġan Ġadditive Ġcost Ġterm Ġto Ġthe ĠSkip - G ram Ġformulation . ĠFinally , ĠB IB REF 32 Ġbuilds Ġon Ġtop Ġof Ġthe ĠGl o Ve Ġalgorithm Ġby Ġintroducing Ġa Ġregular ization Ġterm Ġto Ġthe Ġobjective Ġfunction Ġthat Ġencourages Ġthe Ġvector Ġrepresentations Ġof Ġsimilar Ġwords Ġas Ġdictated Ġby ĠWord Net Ġto Ġbe Ġsimilar Ġas Ġwell . </s> Turn ing Ġour Ġattention Ġto Ġthe Ġpost - processing Ġapproach Ġfor Ġenrich ing Ġword Ġembed d ings Ġwith Ġexternal Ġlex ical Ġknowledge , ĠB IB REF 33 Ġhas Ġintroduced Ġthe Ġretro fitting Ġalgorithm Ġthat Ġacts Ġon Ġpre - trained Ġembed d ings Ġsuch Ġas ĠSkip - G ram Ġor ĠGl o Ve . ĠThe Ġauthors Ġpropose Ġan Ġobjective Ġfunction Ġthat Ġaims Ġto Ġbalance Ġout Ġthe Ġsemantic Ġinformation Ġcaptured Ġin Ġthe Ġpre - trained Ġembed d ings Ġwith Ġthe Ġconstraints Ġderived Ġfrom Ġlex ical Ġresources Ġsuch Ġas ĠWord Net , ĠP PD B Ġand ĠFrame Net . ĠOne Ġof Ġthe Ġmodels Ġproposed Ġin ĠB IB REF 34 Ġextends Ġthe Ġretro fitting Ġapproach Ġto Ġincorporate Ġthe Ġword Ġsense Ġinformation Ġfrom ĠWord Net . ĠSimilarly , ĠB IB REF 35 Ġcreates Ġmulti - sense Ġembed d ings Ġby Ġgathering Ġthe Ġword Ġsense Ġinformation Ġfrom Ġa Ġlex ical Ġresource Ġand Ġlearning Ġto Ġdecom pose Ġthe Ġpre - trained Ġembed d ings Ġinto Ġa Ġconve x Ġcombination Ġof Ġsense Ġembed d ings . ĠIn ĠB IB REF 36 Ġ, Ġthe Ġauthors Ġfocus Ġon Ġimproving Ġword Ġembed d ings Ġfor Ġcapturing Ġword Ġsimilarity , Ġas Ġopposed Ġto Ġmere Ġrelated ness . ĠTo Ġthis Ġend , Ġthey Ġintroduce Ġthe Ġcounter - fitting Ġtechnique Ġwhich Ġacts Ġon Ġthe Ġinput Ġword Ġvectors Ġsuch Ġthat Ġsynonymous Ġwords Ġare Ġattracted Ġto Ġone Ġanother Ġwhereas Ġan ton ym ous Ġwords Ġare Ġrepe lled , Ġwhere Ġthe Ġsyn ony my - ant ony my Ġrelations Ġare Ġextracted Ġfrom Ġa Ġlex ical Ġresource . ĠMore Ġrecently , Ġthe ĠAT TR ACT - REP EL Ġalgorithm Ġproposed Ġby ĠB IB REF 37 Ġimproves Ġon Ġcounter - fitting Ġby Ġa Ġformulation Ġwhich Ġimp arts Ġthe Ġword Ġvectors Ġwith Ġexternal Ġlex ical Ġinformation Ġin Ġmini - bat ches . </s> Most Ġof Ġthe Ġstudies Ġdiscussed Ġabove Ġ( ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, ĠB IB REF 36 Ġ, ĠB IB REF 37 Ġ) Ġreport Ġperformance Ġimprovements Ġin Ġbenchmark Ġtests Ġsuch Ġas Ġword Ġsimilarity Ġor Ġword Ġanalogy , Ġwhile ĠB IB REF 29 Ġuses Ġa Ġdifferent Ġanalysis Ġmethod Ġ( mean Ġreciprocal Ġrank ). ĠIn Ġsum , Ġthe Ġliterature Ġis Ġrich Ġwith Ġstudies Ġaiming Ġto Ġobtain Ġword Ġembed d ings Ġthat Ġperform Ġbetter Ġunder Ġspecific Ġperformance Ġmetrics . ĠHowever , Ġless Ġattention Ġhas Ġbeen Ġdirected Ġto Ġthe Ġissue Ġof Ġinterpret ability Ġof Ġthe Ġword Ġembed d ings . ĠIn Ġthe Ġliterature , Ġthe Ġproblem Ġof Ġinterpret ability Ġhas Ġbeen Ġtackled Ġusing Ġdifferent Ġapproaches . ĠB IB REF 19 Ġproposed Ġnon - negative Ġmatrix Ġfactor ization Ġ( NM F ) Ġfor Ġlearning Ġsparse , Ġinterpret able Ġword Ġvectors Ġfrom Ġco - occ urrence Ġvariant Ġmat rices Ġwhere Ġthe Ġresulting Ġvector Ġspace Ġis Ġcalled Ġnon - negative Ġsparse Ġembed d ign s Ġ( NN SE ). ĠHowever , Ġsince ĠNM F Ġmethods Ġrequire Ġmaintaining Ġa Ġglobal Ġmatrix Ġfor Ġlearning , Ġthey Ġsuffer Ġfrom Ġmemory Ġand Ġscale Ġissue . ĠThis Ġproblem Ġhas Ġbeen Ġaddressed Ġin ĠB IB REF 20 Ġwhere Ġan Ġonline Ġmethod Ġof Ġlearning Ġinterpret able Ġword Ġembed d ings Ġfrom Ġcorpor a Ġusing Ġa Ġmodified Ġversion Ġof Ġskip - gram Ġmodel ĠB IB REF 0 Ġis Ġproposed . ĠAs Ġa Ġdifferent Ġapproach , ĠB IB REF 21 Ġcombined Ġtext - based Ġsimilarity Ġinformation Ġamong Ġwords Ġwith Ġbrain Ġactivity Ġbased Ġsimilarity Ġinformation Ġto Ġimprove Ġinterpret ability Ġusing Ġjoint Ġnon - negative Ġsparse Ġembed ding Ġ( J NN SE ). </s> A Ġcommon Ġalternative Ġapproach Ġfor Ġlearning Ġinterpret able Ġembed d ings Ġis Ġto Ġlearn Ġtransformations Ġthat Ġmap Ġpre - trained Ġstate - of - the - art Ġembed d ings Ġto Ġnew Ġinterpret able Ġsemantic Ġspaces . ĠTo Ġobtain Ġsparse , Ġhigher Ġdimensional Ġand Ġmore Ġinterpret able Ġvector Ġspaces , ĠB IB REF 22 Ġand ĠB IB REF 23 Ġuse Ġsparse Ġcoding Ġon Ġconventional Ġdense Ġword Ġembed d ings . ĠHowever , Ġthese Ġmethods Ġlearn Ġthe Ġprojection Ġvectors Ġthat Ġare Ġused Ġfor Ġthe Ġtransformation Ġfrom Ġthe Ġword Ġembed d ings Ġwithout Ġsupervision . ĠFor Ġthis Ġreason , Ġlabels Ġdescribing Ġthe Ġcorresponding Ġsemantic Ġcategories Ġcannot Ġbe Ġprovided . ĠAn Ġalternative Ġapproach Ġwas Ġproposed Ġin ĠB IB REF 25 Ġ, Ġwhere Ġorth og onal Ġtransformations Ġwere Ġutilized Ġto Ġincrease Ġinterpret ability Ġwhile Ġpreserving Ġthe Ġperformance Ġof Ġthe Ġunderlying Ġembed ding . ĠHowever , ĠB IB REF 25 Ġhas Ġalso Ġshown Ġthat Ġtotal Ġinterpret ability Ġof Ġan Ġembed ding Ġis Ġkept Ġconstant Ġunder Ġany Ġorth og onal Ġtransformation Ġand Ġit Ġcan Ġonly Ġbe Ġredistributed Ġacross Ġthe Ġdimensions . ĠR otation Ġalgorithms Ġbased Ġon Ġexplor atory Ġfactor Ġanalysis Ġ( E FA ) Ġto Ġpreserve Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġword Ġembed d ings Ġwhile Ġimproving Ġtheir Ġinterpret ability Ġwas Ġproposed Ġin ĠB IB REF 26 Ġ. ĠB IB REF 24 Ġproposed Ġto Ġdeploy Ġa Ġsparse Ġauto - enc oder Ġusing Ġpre - trained Ġdense Ġword Ġembed d ings Ġto Ġimprove Ġinterpret ability . ĠMore Ġdetailed Ġinvestigation Ġof Ġsemantic Ġstructure Ġand Ġinterpret ability Ġof Ġword Ġembed d ings Ġcan Ġbe Ġfound Ġin ĠB IB REF 27 Ġ, Ġwhere Ġa Ġmetric Ġwas Ġproposed Ġto Ġquant itatively Ġmeasure Ġthe Ġdegree Ġof Ġinterpret ability Ġalready Ġpresent Ġin Ġthe Ġembed ding Ġvector Ġspaces . </s> Previous Ġworks Ġon Ġinterpret ability Ġmentioned Ġabove , Ġexcept ĠB IB REF 21 Ġ, ĠB IB REF 27 Ġand Ġour Ġproposed Ġmethod , Ġdo Ġnot Ġneed Ġexternal Ġresources , Ġutilization Ġof Ġwhich Ġhas Ġboth Ġadvantages Ġand Ġdisadvantages . ĠMethods Ġthat Ġdo Ġnot Ġuse Ġexternal Ġresources Ġrequire Ġfewer Ġresources Ġbut Ġthey Ġalso Ġlack Ġthe Ġaid Ġof Ġinformation Ġextracted Ġfrom Ġthese Ġresources . </s> Problem ĠDescription </s> For Ġthe Ġtask Ġof Ġun super vised Ġword Ġembed ding Ġextraction , Ġwe Ġoperate Ġon Ġa Ġdiscrete Ġcollection Ġof Ġlex ical Ġunits Ġ( words ) ĠIN LINE FORM 0 Ġthat Ġis Ġpart Ġof Ġan Ġinput Ġcorpus ĠIN LINE FORM 1 Ġ, Ġwith Ġnumber Ġof Ġtokens ĠIN LINE FORM 2 Ġ, Ġsourced Ġfrom Ġa Ġvocabulary ĠIN LINE FORM 3 Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠIn Ġthe Ġsetting Ġof Ġdistribution al Ġsemantics , Ġthe Ġobjective Ġof Ġa Ġword Ġembed ding Ġalgorithm Ġis Ġto Ġmaximize Ġsome Ġaggregate Ġutility Ġover Ġthe Ġentire Ġcorpus Ġso Ġthat Ġsome Ġmeasure Ġof ĠâĢ ľ cl osen ess âĢ Ŀ Ġis Ġmaxim ized Ġfor Ġpairs Ġof Ġvector Ġrepresentations ĠIN LINE FORM 14 Ġfor Ġwords Ġwhich , Ġon Ġthe Ġaverage , Ġappear Ġin Ġproximity Ġto Ġone Ġanother . ĠIn Ġthe ĠGl o Ve Ġalgorithm ĠB IB REF 2 Ġ, Ġwhich Ġwe Ġbase Ġour Ġimprovements Ġupon , Ġthe Ġfollowing Ġobjective Ġfunction Ġis Ġconsidered : ĠDIS PLAY FORM 0 </s> In Ġ( ĠEQ REF 6 Ġ), ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġstand Ġfor Ġword Ġand Ġcontext Ġvector Ġrepresentations , Ġrespectively , Ġfor Ġwords ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġwhile ĠIN LINE FORM 4 Ġrepresents Ġthe Ġ( possibly Ġweighted ) Ġco occ urrence Ġcount Ġfor Ġthe Ġword Ġpair ĠIN LINE FORM 5 Ġ. ĠInt uitive ly , Ġ( ĠEQ REF 6 Ġ) Ġrepresents Ġthe Ġrequirement Ġthat Ġif Ġsome Ġword ĠIN LINE FORM 6 Ġoccurs Ġoften Ġenough Ġin Ġthe Ġcontext Ġ( or Ġvicinity ) Ġof Ġanother Ġword ĠIN LINE FORM 7 Ġ, Ġthen Ġthe Ġcorresponding Ġword Ġrepresentations Ġshould Ġhave Ġa Ġlarge Ġenough Ġinner Ġproduct Ġin Ġkeeping Ġwith Ġtheir Ġlarge ĠIN LINE FORM 8 Ġvalue , Ġup Ġto Ġsome Ġbias Ġterms ĠIN LINE FORM 9 Ġ; Ġand Ġvice Ġversa . ĠIN LINE FORM 10 Ġin Ġ( ĠEQ REF 6 Ġ) Ġis Ġused Ġas Ġa Ġdiscount ing Ġfactor Ġthat Ġprohibits Ġrare Ġco occ ur rences Ġfrom Ġdisproportionately Ġinfluencing Ġthe Ġresulting Ġembed d ings . </s> The Ġobjective Ġ( ĠEQ REF 6 Ġ) Ġis Ġminimized Ġusing Ġsto ch astic Ġgradient Ġdescent Ġby Ġiter ating Ġover Ġthe Ġmatrix Ġof Ġco occ urrence Ġrecords ĠIN LINE FORM 0 Ġ. ĠIn Ġthe ĠGl o Ve Ġalgorithm , Ġfor Ġa Ġgiven Ġword ĠIN LINE FORM 1 Ġ, Ġthe Ġfinal Ġword Ġrepresentation Ġis Ġtaken Ġto Ġbe Ġthe Ġaverage Ġof Ġthe Ġtwo Ġintermediate Ġvector Ġrepresentations Ġobtained Ġfrom Ġ( ĠEQ REF 6 Ġ); Ġi . e , ĠIN LINE FORM 2 Ġ. ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdetail Ġthe Ġenhancements Ġmade Ġto Ġ( ĠEQ REF 6 Ġ) Ġfor Ġthe Ġpurposes Ġof Ġenhanced Ġinterpret ability , Ġusing Ġthe Ġaforementioned Ġframework Ġas Ġour Ġbasis . </s> Imp art ing ĠInterpret ability </s> Our Ġapproach Ġfalls Ġinto Ġa Ġjoint - learning Ġframework Ġwhere Ġthe Ġdistribution al Ġinformation Ġextracted Ġfrom Ġthe Ġcorpus Ġis Ġallowed Ġto Ġfuse Ġwith Ġthe Ġexternal Ġlex icon - based Ġinformation . ĠWord - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus Ġare Ġdirectly Ġmapped Ġto Ġindividual Ġdimensions Ġof Ġword Ġembed d ings . ĠSpecifically , Ġthe Ġvector Ġrepresentations Ġof Ġwords Ġthat Ġbelong Ġto Ġa Ġparticular Ġgroup Ġare Ġencouraged Ġto Ġhave Ġdeliberately Ġincreased Ġvalues Ġin Ġa Ġparticular Ġdimension Ġthat Ġcorresponds Ġto Ġthe Ġword - group Ġunder Ġconsideration . ĠThis Ġcan Ġbe Ġachieved Ġby Ġmodifying Ġthe Ġobjective Ġfunction Ġof Ġthe Ġembed ding Ġalgorithm Ġto Ġpartially Ġinfluence Ġvector Ġrepresentation Ġdistributions Ġacross Ġtheir Ġdimensions Ġover Ġan Ġinput Ġvocabulary . ĠTo Ġdo Ġthis , Ġwe Ġpropose Ġthe Ġfollowing Ġmodification Ġto Ġthe ĠGl o Ve Ġobjective Ġin Ġ( ĠEQ REF 6 Ġ): Ġr Cl ĠJ Ġ= Ġi , j = 1 V Ġf ( X ij )[ Ġ( wi Tw j Ġ+ Ġbi Ġ+ Ġb j Ġ- X ij ) 2 </s> + Ġk ( l = 1 D ĠIN LINE FORM 0 Ġi Fl Ġg ( wi , l ) Ġ+ Ġl = 1 D ĠIN LINE FORM 1 Ġj ĠFl Ġg ( w j , l ) Ġ) Ġ]. ĠIn Ġ( ĠSEC REF 4 Ġ), ĠIN LINE FORM 2 Ġdenotes Ġthe Ġindices Ġfor Ġthe Ġelements Ġof Ġthe ĠIN LINE FORM 3 Ġth Ġconcept Ġword - group Ġwhich Ġwe Ġwish Ġto Ġassign Ġin Ġthe Ġvector Ġdimension ĠIN LINE FORM 4 Ġ. ĠThe Ġobjective Ġ( ĠSEC REF 4 Ġ) Ġis Ġdesigned Ġas Ġa Ġmixture Ġof Ġtwo Ġindividual Ġcost Ġterms : Ġthe Ġoriginal ĠGl o Ve Ġcost Ġterm Ġalong Ġwith Ġa Ġsecond Ġterm Ġthat Ġencourages Ġembed ding Ġvectors Ġof Ġa Ġgiven Ġconcept Ġword - group Ġto Ġachieve Ġdeliberately Ġincreased Ġvalues Ġalong Ġan Ġassociated Ġdimension ĠIN LINE FORM 5 Ġ. ĠThe Ġrelative Ġweight Ġof Ġthe Ġsecond Ġterm Ġis Ġcontrolled Ġby Ġthe Ġparameter ĠIN LINE FORM 6 Ġ. ĠThe Ġsimultaneous Ġminim ization Ġof Ġboth Ġobjectives Ġensures Ġthat Ġwords Ġthat Ġare Ġsimilar Ġto , Ġbut Ġnot Ġincluded Ġin , Ġone Ġof Ġthese Ġconcept Ġword - groups Ġare Ġalso ĠâĢ ľ n ud ged âĢ Ŀ Ġtowards Ġthe Ġassociated Ġdimension ĠIN LINE FORM 7 Ġ. ĠThe Ġtrained Ġword Ġvectors Ġare Ġthus Ġencouraged Ġto Ġform Ġa Ġdistribution Ġwhere Ġthe Ġindividual Ġvector Ġdimensions Ġalign Ġwith Ġcertain Ġsemantic Ġconcepts Ġrepresented Ġby Ġa Ġcollection Ġof Ġconcept Ġword - groups , Ġone Ġassigned Ġto Ġeach Ġvector Ġdimension . ĠTo Ġfacilitate Ġthis Ġbehaviour , Ġ( ĠSEC REF 4 Ġ) Ġintroduces Ġa Ġmon ot one Ġdecreasing Ġfunction ĠIN LINE FORM 8 Ġdefined Ġas ĠIN LINE FORM 9 </s> which Ġserves Ġto Ġincrease Ġthe Ġtotal Ġcost Ġincurred Ġif Ġthe Ġvalue Ġof Ġthe ĠIN LINE FORM 0 Ġth Ġdimension Ġfor Ġthe Ġtwo Ġvector Ġrepresentations ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġfor Ġa Ġconcept Ġword ĠIN LINE FORM 3 Ġwith ĠIN LINE FORM 4 Ġfails Ġto Ġbe Ġlarge Ġenough . ĠIN LINE FORM 5 Ġis Ġalso Ġshown Ġin ĠFig . ĠFIG REF 7 Ġ. </s> The Ġobjective Ġ( ĠSEC REF 4 Ġ) Ġis Ġminimized Ġusing Ġsto ch astic Ġgradient Ġdescent Ġover Ġthe Ġco occ urrence Ġrecords ĠIN LINE FORM 0 Ġ. ĠInt uitive ly , Ġthe Ġterms Ġadded Ġto Ġ( ĠSEC REF 4 Ġ) Ġin Ġcomparison Ġwith Ġ( ĠEQ REF 6 Ġ) Ġintroduce Ġthe Ġeffect Ġof Ġselectively Ġapplying Ġa Ġpositive Ġstep - type Ġinput Ġto Ġthe Ġoriginal Ġdescent Ġupdates Ġof Ġ( ĠEQ REF 6 Ġ) Ġfor Ġconcept Ġwords Ġalong Ġtheir Ġrespective Ġvector Ġdimensions , Ġwhich Ġinfluences Ġthe Ġdimension Ġvalue Ġin Ġthe Ġpositive Ġdirection . ĠThe Ġparameter ĠIN LINE FORM 1 Ġin Ġ( ĠSEC REF 4 Ġ) Ġallows Ġfor Ġthe Ġadjustment Ġof Ġthe Ġmagnitude Ġof Ġthis Ġinfluence Ġas Ġneeded . </s> In Ġthe Ġnext Ġsection , Ġwe Ġdemonstrate Ġthe Ġfeasibility Ġof Ġthis Ġapproach Ġby Ġexperiments Ġwith Ġan Ġexample Ġcollection Ġof Ġconcept Ġword - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus . </s> Exper iments Ġand ĠResults </s> We Ġfirst Ġidentified Ġ300 Ġconcepts , Ġone Ġfor Ġeach Ġdimension Ġof Ġthe Ġ300 - dimensional Ġvector Ġrepresentation , Ġby Ġemploying ĠRog et 's ĠThe s aurus . ĠThis Ġthe s aurus Ġfollows Ġa Ġtree Ġstructure Ġwhich Ġstarts Ġwith Ġa ĠRoot Ġnode Ġthat Ġcontains Ġall Ġthe Ġwords Ġand Ġphrases Ġin Ġthe Ġthe s aurus . ĠThe Ġroot Ġnode Ġis Ġsuccess ively Ġsplit Ġinto ĠClasses Ġand ĠSections , Ġwhich Ġare Ġthen Ġ( option ally ) Ġsplit Ġinto ĠSub sections Ġof Ġvarious Ġdepths , Ġfinally Ġending Ġin ĠCategories , Ġwhich Ġconstitute Ġthe Ġsmallest Ġunit Ġof Ġword / phrase Ġcollections Ġin Ġthe Ġstructure . ĠThe Ġactual Ġwords Ġand Ġphrases Ġdescend Ġfrom Ġthese ĠCategories , Ġand Ġmake Ġup Ġthe Ġleaves Ġof Ġthe Ġtree Ġstructure . ĠWe Ġnote Ġthat Ġa Ġgiven Ġword Ġtypically Ġappears Ġin Ġmultiple Ġcategories Ġcorresponding Ġto Ġthe Ġdifferent Ġsenses Ġof Ġthe Ġword . ĠWe Ġconstructed Ġconcept Ġword - groups Ġfrom ĠRog et 's ĠThe s aurus Ġas Ġfollows : ĠWe Ġfirst Ġfiltered Ġout Ġthe Ġmulti - word Ġphrases Ġand Ġthe Ġrelatively Ġobscure Ġterms Ġfrom Ġthe Ġthe s aurus . ĠThe Ġobscure Ġterms Ġwere Ġidentified Ġby Ġchecking Ġthem Ġagainst Ġa Ġvocabulary Ġextracted Ġfrom ĠWikipedia . ĠWe Ġthen Ġobtained Ġ300 Ġword - groups Ġas Ġthe Ġresult Ġof Ġa Ġpartition ing Ġoperation Ġapplied Ġto Ġthe Ġsubt ree Ġthat Ġends Ġwith Ġcategories Ġas Ġits Ġleaves . ĠThe Ġpartition Ġboundaries , Ġhence Ġthe Ġresulting Ġword - groups , Ġcan Ġbe Ġchosen Ġin Ġmany Ġdifferent Ġways . ĠIn Ġour Ġproposed Ġapproach , Ġwe Ġhave Ġchosen Ġto Ġdetermine Ġthis Ġpartition ing Ġby Ġtravers ing Ġthis Ġtree Ġstructure Ġfrom Ġthe Ġroot Ġnode Ġin Ġbreadth - first Ġorder , Ġand Ġby Ġemploying Ġa Ġparameter ĠIN LINE FORM 0 Ġfor Ġthe Ġmaximum Ġsize Ġof Ġa Ġnode . ĠHere , Ġthe Ġsize Ġof Ġa Ġnode Ġis Ġdefined Ġas Ġthe Ġnumber Ġof Ġunique Ġwords Ġthat Ġever - desc end Ġfrom Ġthat Ġnode . ĠDuring Ġthe Ġtravers al , Ġif Ġthe Ġsize Ġof Ġa Ġgiven Ġnode Ġis Ġless Ġthan Ġthis Ġthreshold , Ġwe Ġdesignate Ġthe Ġwords Ġthat Ġultimately Ġdescend Ġfrom Ġthat Ġnode Ġas Ġa Ġconcept Ġword - group . ĠOtherwise , Ġif Ġthe Ġnode Ġhas Ġchildren , Ġwe Ġdiscard Ġthe Ġnode , Ġand Ġqueue Ġup Ġall Ġits Ġchildren Ġfor Ġfurther Ġconsideration . ĠIf Ġthis Ġnode Ġdoes Ġnot Ġhave Ġany Ġchildren , Ġon Ġthe Ġother Ġhand , Ġthe Ġnode Ġis Ġtrunc ated Ġto ĠIN LINE FORM 1 Ġelements Ġwith Ġthe Ġhighest Ġfrequency - r anks , Ġand Ġthe Ġresulting Ġwords Ġare Ġdesignated Ġas Ġa Ġconcept Ġword - group . ĠWe Ġnote Ġthat Ġthe Ġchoice Ġof ĠIN LINE FORM 2 Ġgreatly Ġaffects Ġthe Ġresulting Ġcollection Ġof Ġword - groups : ĠEx cess ively Ġlarge Ġvalues Ġresult Ġin Ġfew Ġword - groups Ġthat Ġgreatly Ġoverlap Ġwith Ġone Ġanother , Ġwhile Ġoverly Ġsmall Ġvalues Ġresult Ġin Ġnumerous Ġtiny Ġword - groups Ġthat Ġfail Ġto Ġadequately Ġrepresent Ġa Ġconcept . ĠWe Ġexperiment ally Ġdetermined Ġthat Ġa ĠIN LINE FORM 3 Ġvalue Ġof Ġ45 2 Ġresults Ġin Ġthe Ġmost Ġhealthy Ġnumber Ġof Ġrelatively Ġlarge Ġword - groups Ġ( 113 Ġgroups Ġwith Ġsize ĠIN LINE FORM 4 Ġ100 ), Ġwhile Ġyielding Ġa Ġpreferably Ġsmall Ġoverlap Ġamongst Ġthe Ġresulting Ġword - groups Ġ( with Ġaverage Ġoverlap Ġsize Ġnot Ġexceeding Ġ3 Ġwords ). ĠA Ġtotal Ġof Ġ5 66 Ġword - groups Ġwere Ġthus Ġobtained . Ġ259 Ġsmallest Ġword - groups Ġ( with Ġsize ĠIN LINE FORM 5 Ġ38 ) Ġwere Ġdiscarded Ġto Ġbring Ġdown Ġthe Ġnumber Ġof Ġword - groups Ġto Ġ307 . ĠOut Ġof Ġthese , Ġ7 Ġgroups Ġwith Ġthe Ġlowest Ġmedian Ġfrequency - rank Ġwere Ġfurther Ġdiscarded , Ġwhich Ġyields Ġthe Ġfinal Ġ300 Ġconcept Ġword - groups Ġused Ġin Ġthe Ġexperiments . ĠWe Ġpresent Ġsome Ġof Ġthe Ġresulting Ġword - groups Ġin ĠTable ĠT AB REF 9 Ġ. </s> By Ġusing Ġthe Ġconcept Ġword - groups , Ġwe Ġhave Ġtrained Ġthe ĠGl o Ve Ġalgorithm Ġwith Ġthe Ġproposed Ġmodification Ġgiven Ġin ĠSection ĠSEC REF 4 Ġon Ġa Ġsnapshot Ġof ĠEnglish ĠWikipedia Ġmeasuring Ġ8 GB Ġin Ġsize , Ġwith Ġthe Ġstop - words Ġfiltered Ġout . ĠUsing Ġthe Ġparameters Ġgiven Ġin ĠTable ĠT AB REF 10 Ġ, Ġthis Ġresulted Ġin Ġa Ġvocabulary Ġsize Ġof Ġ287 , 8 47 . ĠFor Ġthe Ġweight ing Ġparameter Ġin ĠE q . ĠSEC REF 4 Ġ, Ġwe Ġused Ġa Ġvalue Ġof ĠIN LINE FORM 0 Ġ. ĠThe Ġalgorithm Ġwas Ġtrained Ġover Ġ20 Ġiterations . ĠThe ĠGl o Ve Ġalgorithm Ġwithout Ġany Ġmodifications Ġwas Ġalso Ġtrained Ġas Ġa Ġbaseline Ġwith Ġthe Ġsame Ġparameters . ĠIn Ġaddition Ġto Ġthe Ġoriginal ĠGl o Ve Ġalgorithm , Ġwe Ġcompare Ġour Ġproposed Ġmethod Ġwith Ġprevious Ġstudies Ġthat Ġaim Ġto Ġobtain Ġinterpret able Ġword Ġvectors . ĠWe Ġtrain Ġthe Ġimproved Ġprojected Ġgradient Ġmodel Ġproposed Ġin ĠB IB REF 20 Ġto Ġobtain Ġword Ġvectors Ġ( called ĠO I WE - IP G ) Ġusing Ġthe Ġsame Ġcorpus Ġwe Ġuse Ġto Ġtrain ĠGl o Ve Ġand Ġour Ġproposed Ġmethod . ĠUsing Ġthe Ġmethods Ġproposed Ġin ĠB IB REF 23 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 24 Ġon Ġour Ġbaseline ĠGl o Ve Ġembed d ings , Ġwe Ġobtain ĠS OV , ĠSP INE Ġand ĠPars im ax Ġ( orth og onal ) Ġword Ġrepresentations , Ġrespectively . ĠWe Ġtrain Ġall Ġthe Ġmodels Ġwith Ġthe Ġproposed Ġparameters . ĠHowever , Ġin ĠB IB REF 26 Ġ, Ġthe Ġauthors Ġshow Ġresults Ġfor Ġa Ġrelatively Ġsmall Ġvocabulary Ġof Ġ15 , 000 Ġwords . ĠWhen Ġwe Ġtrained Ġtheir Ġmodel Ġon Ġour Ġbaseline ĠGl o Ve Ġembed d ings Ġwith Ġa Ġlarge Ġvocabulary Ġof Ġsize Ġ287 , 8 47 , Ġthe Ġresulting Ġvectors Ġperformed Ġsignificantly Ġpoor Ġon Ġword Ġsimilarity Ġtasks Ġcompared Ġto Ġthe Ġresults Ġpresented Ġin Ġtheir Ġpaper . ĠIn Ġaddition , ĠPars im ax Ġ( orth og onal ) Ġword Ġvectors Ġobtained Ġusing Ġmethod Ġin ĠB IB REF 26 Ġare Ġnearly Ġidentical Ġto Ġthe Ġbaseline Ġvectors Ġ( i . e . Ġlearned Ġorth og onal Ġtransformation Ġmatrix Ġis Ġvery Ġclose Ġto Ġidentity ). ĠTherefore , ĠPars im ax Ġ( orth og onal ) Ġyields Ġalmost Ġsame Ġresults Ġwith Ġbaseline Ġvectors Ġin Ġall Ġevaluations . ĠWe Ġevaluate Ġthe Ġinterpret ability Ġof Ġthe Ġresulting Ġembed d ings Ġqual itatively Ġand Ġquant itatively . ĠWe Ġalso Ġtest Ġthe Ġperformance Ġof Ġthe Ġembed d ings Ġon Ġword Ġsimilarity Ġand Ġword Ġanalogy Ġtests . </s> In Ġour Ġexperiments , Ġvocabulary Ġsize Ġis Ġclose Ġto Ġ300 , 000 Ġwhile Ġonly Ġ16 , 242 Ġunique Ġwords Ġof Ġthe Ġvocabulary Ġare Ġpresent Ġin Ġthe Ġconcept Ġgroups . ĠFurthermore , Ġonly Ġdimensions Ġthat Ġcorrespond Ġto Ġthe Ġconcept Ġgroup Ġof Ġthe Ġword Ġwill Ġbe Ġupdated Ġdue Ġto Ġthe Ġadditional Ġcost Ġterm . ĠGiven Ġthat Ġthese Ġconcept Ġwords Ġcan Ġbelong Ġto Ġmultiple Ġconcept Ġgroups Ġ( 2 Ġon Ġaverage ), Ġonly Ġ33 , 319 Ġparameters Ġare Ġupdated . ĠThere Ġare Ġ90 Ġmillion Ġindividual Ġparameters Ġpresent Ġfor Ġthe Ġ300 , 000 Ġword Ġvectors Ġof Ġsize Ġ300 . ĠOf Ġthese Ġparameters , Ġonly Ġapproximately Ġ33 , 000 Ġare Ġupdated Ġby Ġthe Ġadditional Ġcost Ġterm . </s> Qual itative ĠEvaluation Ġfor ĠInterpret ability </s> In ĠFig . ĠFIG REF 13 Ġ, Ġwe Ġdemonstrate Ġthe Ġparticular Ġway Ġin Ġwhich Ġthe Ġproposed Ġalgorithm Ġ( ĠSEC REF 4 Ġ) Ġinfluences Ġthe Ġvector Ġrepresentation Ġdistributions . ĠSpecifically , Ġwe Ġconsider , Ġfor Ġillustration , Ġthe Ġ32 nd Ġdimension Ġvalues Ġfor Ġthe Ġoriginal ĠGl o Ve Ġalgorithm Ġand Ġour Ġmodified Ġversion , Ġrestricting Ġthe Ġplots Ġto Ġthe Ġtop - 1000 Ġwords Ġwith Ġrespect Ġto Ġtheir Ġfrequency Ġranks Ġfor Ġclarity Ġof Ġpresentation . ĠIn ĠFig . ĠFIG REF 13 Ġ, Ġthe Ġwords Ġin Ġthe Ġhorizontal Ġaxis Ġare Ġsorted Ġin Ġdescending Ġorder Ġwith Ġrespect Ġto Ġthe Ġvalues Ġat Ġthe Ġ32 nd Ġdimension Ġof Ġtheir Ġword Ġembed ding Ġvectors Ġcoming Ġfrom Ġthe Ġoriginal ĠGl o Ve Ġalgorithm . ĠThe Ġdimension Ġvalues Ġare Ġden oted Ġwith Ġblue Ġand Ġred / green Ġmarkers Ġfor Ġthe Ġoriginal Ġand Ġthe Ġproposed Ġalgorithms , Ġrespectively . ĠAdditionally , Ġthe Ġtop - 50 Ġwords Ġthat Ġachieve Ġthe Ġgreatest Ġ32 nd Ġdimension Ġvalues Ġamong Ġthe Ġconsidered Ġ1000 Ġwords Ġare Ġemphasized Ġwith Ġenlarged Ġmarkers , Ġalong Ġwith Ġtext Ġannotations . ĠIn Ġthe Ġpresented Ġsimulation Ġof Ġthe Ġproposed Ġalgorithm , Ġthe Ġ32 nd Ġdimension Ġvalues Ġare Ġencoded Ġwith Ġthe Ġconcept ĠJ UD G MENT , Ġwhich Ġis Ġreflected Ġas Ġan Ġincrease Ġin Ġthe Ġdimension Ġvalues Ġfor Ġwords Ġsuch Ġas Ġcommittee , Ġacademy , Ġand Ġarticle . ĠWe Ġnote Ġthat Ġthese Ġwords Ġ( red ) Ġare Ġnot Ġpart Ġof Ġthe Ġpre - d etermined Ġword - group Ġfor Ġthe Ġconcept ĠJ UD G MENT , Ġin Ġcontrast Ġto Ġwords Ġsuch Ġas Ġaward , Ġreview Ġand Ġaccount Ġ( green ) Ġwhich Ġare . ĠThis Ġimplies Ġthat Ġthe Ġincrease Ġin Ġthe Ġcorresponding Ġdimension Ġvalues Ġseen Ġfor Ġthese Ġwords Ġis Ġattributable Ġto Ġthe Ġjoint Ġeffect Ġof Ġthe Ġfirst Ġterm Ġin Ġ( ĠSEC REF 4 Ġ) Ġwhich Ġis Ġinherited Ġfrom Ġthe Ġoriginal ĠGl o Ve Ġalgorithm , Ġin Ġconjunction Ġwith Ġthe Ġremaining Ġterms Ġin Ġthe Ġproposed Ġobjective Ġexpression Ġ( ĠSEC REF 4 Ġ). ĠThis Ġexperiment Ġillustrates Ġthat Ġthe Ġproposed Ġalgorithm Ġis Ġable Ġto Ġimpart Ġthe Ġconcept Ġof ĠJ UD G MENT Ġon Ġits Ġdesignated Ġvector Ġdimension Ġabove Ġand Ġbeyond Ġthe Ġsupplied Ġlist Ġof Ġwords Ġbelonging Ġto Ġthe Ġconcept Ġword - group Ġfor Ġthat Ġdimension . ĠWe Ġalso Ġpresent Ġthe Ġlist Ġof Ġwords Ġwith Ġthe Ġgreatest Ġdimension Ġvalue Ġfor Ġthe Ġdimensions Ġ11 , Ġ13 , Ġ16 , Ġ31 , Ġ36 , Ġ39 , Ġ41 , Ġ43 Ġand Ġ79 Ġin ĠTable ĠT AB REF 11 Ġ. ĠThese Ġdimensions Ġare Ġaligned / im part ed Ġwith Ġthe Ġconcepts Ġthat Ġare Ġgiven Ġin Ġthe Ġcolumn Ġheaders . ĠIn ĠTable ĠT AB REF 11 Ġ, Ġthe Ġwords Ġthat Ġare Ġhighlighted Ġwith Ġgreen Ġdenote Ġthe Ġwords Ġthat Ġexist Ġin Ġthe Ġcorresponding Ġword - group Ġobtained Ġfrom ĠRog et 's ĠThe s aurus Ġ( and Ġare Ġthus Ġexplicitly Ġforced Ġto Ġachieve Ġincreased Ġdimension Ġvalues ), Ġwhile Ġthe Ġred Ġwords Ġdenote Ġthe Ġwords Ġthat Ġachieve Ġincreased Ġdimension Ġvalues Ġby Ġvirtue Ġof Ġtheir Ġco occ urrence Ġstatistics Ġwith Ġthe Ġthe s aurus - based Ġwords Ġ( ind irect ly , Ġwithout Ġbeing Ġexplicitly Ġforced ). ĠThis Ġagain Ġillustrates Ġthat Ġa Ġsemantic Ġconcept Ġcan Ġindeed Ġbe Ġcoded Ġto Ġa Ġvector Ġdimension Ġprovided Ġthat Ġa Ġsensible Ġlex ical Ġresource Ġis Ġused Ġto Ġguide Ġsem antically Ġrelated Ġwords Ġto Ġthe Ġdesired Ġvector Ġdimension Ġvia Ġthe Ġproposed Ġobjective Ġfunction Ġin Ġ( ĠSEC REF 4 Ġ). ĠEven Ġthe Ġwords Ġthat Ġdo Ġnot Ġappear Ġin , Ġbut Ġare Ġsem antically Ġrelated Ġto , Ġthe Ġword - groups Ġthat Ġwe Ġformed Ġusing ĠRog et 's ĠThe s aurus , Ġare Ġindirectly Ġaffected Ġby Ġthe Ġproposed Ġalgorithm . ĠThey Ġalso Ġreflect Ġthe Ġassociated Ġconcepts Ġat Ġtheir Ġrespective Ġdimensions Ġeven Ġthough Ġthe Ġobjective Ġfunctions Ġfor Ġtheir Ġparticular Ġvectors Ġare Ġnot Ġmodified . ĠThis Ġpoint Ġcannot Ġbe Ġove rem phas ized . ĠAlthough Ġthe Ġword - groups Ġextracted Ġfrom ĠRog et 's ĠThe s aurus Ġimpose Ġa Ġdegree Ġof Ġsupervision Ġto Ġthe Ġprocess , Ġthe Ġfact Ġthat Ġthe Ġremaining Ġwords Ġin Ġthe Ġentire Ġvocabulary Ġare Ġalso Ġindirectly Ġaffected Ġmakes Ġthe Ġproposed Ġmethod Ġa Ġsemi - super vised Ġapproach Ġthat Ġcan Ġhandle Ġwords Ġthat Ġare Ġnot Ġin Ġthese Ġchosen Ġword - groups . ĠA Ġqualitative Ġexample Ġof Ġthis Ġresult Ġcan Ġbe Ġseen Ġin Ġthe Ġlast Ġcolumn Ġof ĠTable ĠT AB REF 11 Ġ. ĠIt Ġis Ġinteresting Ġto Ġnote Ġthe Ġappearance Ġof Ġwords Ġsuch Ġas Ġgu er illa , Ġinsurgency , Ġm ujah ide en , ĠWe hr m acht Ġand ĠLu ft w affe Ġin Ġaddition Ġto Ġthe Ġmore Ġobvious Ġand Ġstraightforward Ġarmy , Ġsoldiers Ġand Ġtroops , Ġall Ġof Ġwhich Ġare Ġnot Ġpresent Ġin Ġthe Ġassociated Ġword - group ĠWAR F ARE . </s> Most Ġof Ġthe Ġdimensions Ġwe Ġinvestigated Ġexhibit Ġsimilar Ġbehaviour Ġto Ġthe Ġones Ġpresented Ġin ĠTable ĠT AB REF 11 Ġ. ĠThus Ġgenerally Ġspeaking , Ġwe Ġcan Ġsay Ġthat Ġthe Ġentries Ġin ĠTable ĠT AB REF 11 Ġare Ġrepresentative Ġof Ġthe Ġgreat Ġmajority . ĠHowever , Ġwe Ġhave Ġalso Ġspecifically Ġlooked Ġfor Ġdimensions Ġthat Ġmake Ġless Ġsense Ġand Ġdetermined Ġa Ġfew Ġsuch Ġdimensions Ġwhich Ġare Ġrelatively Ġless Ġsatisfactory . ĠThese Ġless Ġsatisfactory Ġexamples Ġare Ġgiven Ġin ĠTable ĠT AB REF 14 Ġ. ĠThese Ġexamples Ġare Ġalso Ġinteresting Ġin Ġthat Ġthey Ġshed Ġinsight Ġinto Ġthe Ġlimitations Ġposed Ġby Ġpoly se my Ġand Ġexistence Ġof Ġvery Ġrare Ġout lier Ġwords . </s> Quant itative ĠEvaluation Ġfor ĠInterpret ability </s> One Ġof Ġthe Ġmain Ġgoals Ġof Ġthis Ġstudy Ġis Ġto Ġimprove Ġthe Ġinterpret ability Ġof Ġdense Ġword Ġembed d ings Ġby Ġalign ing Ġthe Ġdimensions Ġwith Ġpred efined Ġconcepts Ġfrom Ġa Ġsuitable Ġlex icon . ĠA Ġquantitative Ġmeasure Ġis Ġrequired Ġto Ġreliably Ġevaluate Ġthe Ġachieved Ġimprovement . ĠOne Ġof Ġthe Ġmethods Ġproposed Ġto Ġmeasure Ġthe Ġinterpret ability Ġis Ġthe Ġword Ġintrusion Ġtest ĠB IB REF 41 Ġ. ĠBut , Ġthis Ġmethod Ġis Ġexpensive Ġto Ġapply Ġsince Ġit Ġrequires Ġevaluations Ġfrom Ġmultiple Ġhuman Ġeval u ators Ġfor Ġeach Ġembed ding Ġdimension . ĠIn Ġthis Ġstudy , Ġwe Ġuse Ġa Ġsemantic Ġcategory - based Ġapproach Ġbased Ġon Ġthe Ġmethod Ġand Ġcategory Ġdataset Ġ( S EM C AT ) Ġintroduced Ġin ĠB IB REF 27 Ġto Ġquantify Ġinterpret ability . ĠSpecifically , Ġwe Ġapply Ġa Ġmodified Ġversion Ġof Ġthe Ġapproach Ġpresented Ġin ĠB IB REF 40 Ġin Ġorder Ġto Ġconsider Ġpossible Ġsub - group ings Ġwithin Ġthe Ġcategories Ġin ĠSEM C AT . ĠInterpret ability Ġscores Ġare Ġcalculated Ġusing ĠInterpret ability ĠScore Ġ( IS ) Ġas Ġgiven Ġbelow : </s> DIS PLAY FORM 0 </s> In Ġ( ĠEQ REF 17 Ġ), ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġrepresents Ġthe Ġinterpret ability Ġscores Ġin Ġthe Ġpositive Ġand Ġnegative Ġdirections Ġof Ġthe ĠIN LINE FORM 2 Ġdimension Ġ( ĠIN LINE FORM 3 Ġ, ĠIN LINE FORM 4 Ġnumber Ġof Ġdimensions Ġin Ġthe Ġembed ding Ġspace ) Ġof Ġword Ġembed ding Ġspace Ġfor Ġthe ĠIN LINE FORM 5 Ġcategory Ġ( ĠIN LINE FORM 6 Ġ, ĠIN LINE FORM 7 Ġis Ġnumber Ġof Ġcategories Ġin ĠSEM C AT , ĠIN LINE FORM 8 Ġ) Ġin ĠSEM C AT Ġrespectively . ĠIN LINE FORM 9 Ġis Ġthe Ġset Ġof Ġwords Ġin Ġthe ĠIN LINE FORM 10 Ġcategory Ġin ĠSEM C AT Ġand ĠIN LINE FORM 11 Ġis Ġthe Ġnumber Ġof Ġwords Ġin ĠIN LINE FORM 12 Ġ. ĠIN LINE FORM 13 Ġcorresponds Ġto Ġthe Ġminimum Ġnumber Ġof Ġwords Ġrequired Ġto Ġconstruct Ġa Ġsemantic Ġcategory Ġ( i . e . Ġrepresent Ġa Ġconcept ). ĠIN LINE FORM 14 Ġrepresents Ġthe Ġset Ġof ĠIN LINE FORM 15 Ġwords Ġthat Ġhave Ġthe Ġhighest Ġ( ĠIN LINE FORM 16 Ġ) Ġand Ġlowest Ġ( ĠIN LINE FORM 17 Ġ) Ġvalues Ġin ĠIN LINE FORM 18 Ġdimension Ġof Ġthe Ġembed ding Ġspace . ĠIN LINE FORM 19 Ġis Ġthe Ġintersection Ġoperator Ġand ĠIN LINE FORM 20 Ġis Ġthe Ġcardinal ity Ġoperator Ġ( number Ġof Ġelements ) Ġfor Ġthe Ġintersect ing Ġset . ĠIn Ġ( ĠEQ REF 17 Ġ), ĠIN LINE FORM 21 Ġgives Ġthe Ġinterpret ability Ġscore Ġfor Ġthe ĠIN LINE FORM 22 Ġdimension Ġand ĠIN LINE FORM 23 Ġgives Ġthe Ġaverage Ġinterpret ability Ġscore Ġof Ġthe Ġembed ding Ġspace . </s> Fig . ĠFIG REF 18 Ġpresents Ġthe Ġmeasured Ġaverage Ġinterpret ability Ġscores Ġacross Ġdimensions Ġfor Ġoriginal ĠGl o Ve Ġembed d ings , Ġfor Ġthe Ġproposed Ġmethod Ġand Ġfor Ġthe Ġother Ġfour Ġmethods Ġwe Ġcompare , Ġalong Ġwith Ġa Ġrandomly Ġgenerated Ġembed ding . ĠResults Ġare Ġcalculated Ġfor Ġthe Ġparameters ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ. ĠOur Ġproposed Ġmethod Ġsignificantly Ġimproves Ġthe Ġinterpret ability Ġfor Ġall ĠIN LINE FORM 2 Ġcompared Ġto Ġthe Ġoriginal ĠGl o Ve Ġapproach . ĠOur Ġproposed Ġmethod Ġis Ġsecond Ġto Ġonly ĠSP INE Ġin Ġincreasing Ġinterpret ability . ĠHowever , Ġas Ġwe Ġwill Ġexperiment ally Ġdemonstrate Ġin Ġthe Ġnext Ġsubsection , Ġin Ġdoing Ġthis , ĠSP INE Ġalmost Ġentirely Ġdestroys Ġthe Ġunderlying Ġsemantic Ġstructure Ġof Ġthe Ġword Ġembed d ings , Ġwhich Ġis Ġthe Ġprimary Ġfunction Ġof Ġa Ġword Ġembed ding . </s> The Ġproposed Ġmethod Ġand Ġinterpret ability Ġmeasurements Ġare Ġboth Ġbased Ġon Ġutilizing Ġconcepts Ġrepresented Ġby Ġword - groups . ĠTherefore Ġit Ġis Ġexpected Ġthat Ġthere Ġwill Ġbe Ġhigher Ġinterpret ability Ġscores Ġfor Ġsome Ġof Ġthe Ġdimensions Ġfor Ġwhich Ġthe Ġimp arted Ġconcepts Ġare Ġalso Ġcontained Ġin ĠSEM C AT . ĠHowever , Ġby Ġdesign , Ġword Ġgroups Ġthat Ġthey Ġuse Ġare Ġformed Ġby Ġusing Ġdifferent Ġsources Ġand Ġare Ġindependent . ĠInterpret ability Ġmeasurements Ġuse ĠSEM C AT Ġwhile Ġour Ġproposed Ġmethod Ġutilizes ĠRog et 's ĠThe s aurus . </s> Int r ins ic ĠEvaluation Ġof Ġthe ĠEmb edd ings </s> It Ġis Ġnecessary Ġto Ġshow Ġthat Ġthe Ġsemantic Ġstructure Ġof Ġthe Ġoriginal Ġembed ding Ġhas Ġnot Ġbeen Ġdamaged Ġor Ġdistorted Ġas Ġa Ġresult Ġof Ġalign ing Ġthe Ġdimensions Ġwith Ġgiven Ġconcepts , Ġand Ġthat Ġthere Ġis Ġno Ġsubstantial Ġsacrifice Ġinvolved Ġfrom Ġthe Ġperformance Ġthat Ġcan Ġbe Ġobtained Ġwith Ġthe Ġoriginal ĠGl o Ve . ĠTo Ġcheck Ġthis , Ġwe Ġevaluate Ġperformances Ġof Ġthe Ġproposed Ġembed d ings Ġon Ġword Ġsimilarity ĠB IB REF 42 Ġand Ġword Ġanalogy ĠB IB REF 0 Ġtests . ĠWe Ġcompare Ġthe Ġresults Ġwith Ġthe Ġoriginal Ġembed d ings Ġand Ġthe Ġthree Ġalternatives Ġexcluding ĠPars im ax ĠB IB REF 26 Ġsince Ġorth og onal Ġtransformations Ġwill Ġnot Ġaffect Ġthe Ġperformance Ġof Ġthe Ġoriginal Ġembed d ings Ġon Ġthese Ġtests . </s> Word Ġsimilarity Ġtest Ġmeasures Ġthe Ġcorrelation Ġbetween Ġword Ġsimilarity Ġscores Ġobtained Ġfrom Ġhuman Ġevaluation Ġ( i . e . Ġtrue Ġsimilarities ) Ġand Ġfrom Ġword Ġembed d ings Ġ( usually Ġusing Ġcos ine Ġsimilarity ). ĠIn Ġother Ġwords , Ġthis Ġtest Ġquant ifies Ġhow Ġwell Ġthe Ġembed ding Ġspace Ġreflects Ġhuman Ġjud gements Ġin Ġterms Ġof Ġsimilarities Ġbetween Ġdifferent Ġwords . ĠThe Ġcorrelation Ġscores Ġfor Ġ13 Ġdifferent Ġsimilarity Ġtest Ġsets Ġare Ġreported Ġin ĠTable ĠT AB REF 20 Ġ. ĠWe Ġobserve Ġthat , Ġlet Ġalone Ġa Ġreduction Ġin Ġperformance , Ġthe Ġobtained Ġscores Ġindicate Ġan Ġalmost Ġuniform Ġimprovement Ġin Ġthe Ġcorrelation Ġvalues Ġfor Ġthe Ġproposed Ġalgorithm , Ġoutper forming Ġall Ġthe Ġalternatives Ġin Ġalmost Ġall Ġtest Ġsets . ĠCategories Ġfrom ĠRog et 's Ġthe s aurus Ġare Ġgroup ings Ġof Ġwords Ġthat Ġare Ġsimilar Ġin Ġsome Ġsense Ġwhich Ġthe Ġoriginal Ġembed ding Ġalgorithm Ġmay Ġfail Ġto Ġcapture . ĠThese Ġtest Ġresults Ġsignify Ġthat Ġthe Ġsemantic Ġinformation Ġinjected Ġinto Ġthe Ġalgorithm Ġby Ġthe Ġadditional Ġcost Ġterm Ġis Ġsignificant Ġenough Ġto Ġresult Ġin Ġa Ġmeasurable Ġimprovement . ĠIt Ġshould Ġalso Ġbe Ġnoted Ġthat Ġscores Ġobtained Ġby ĠSP INE Ġis Ġun accept ably Ġlow Ġon Ġalmost Ġall Ġtests Ġindicating Ġthat Ġit Ġhas Ġachieved Ġits Ġinterpret ability Ġperformance Ġat Ġthe Ġcost Ġof Ġlosing Ġits Ġsemantic Ġfunctions . </s> Word Ġanalogy Ġtest Ġis Ġintroduced Ġin ĠB IB REF 1 Ġand Ġlooks Ġfor Ġthe Ġanswers Ġof Ġthe Ġquestions Ġthat Ġare Ġin Ġthe Ġform Ġof Ġ" X Ġis Ġto ĠY , Ġwhat ĠZ Ġis Ġto Ġ ?" Ġby Ġapplying Ġsimple Ġarithmetic Ġoperations Ġto Ġvectors Ġof Ġwords ĠX , ĠY Ġand ĠZ . ĠWe Ġpresent Ġprecision Ġscores Ġfor Ġthe Ġword Ġanalogy Ġtests Ġin ĠTable ĠT AB REF 21 Ġ. ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe Ġalternative Ġapproaches Ġthat Ġaim Ġto Ġimprove Ġinterpret ability , Ġhave Ġpoor Ġperformance Ġon Ġthe Ġword Ġanalogy Ġtests . ĠHowever , Ġour Ġproposed Ġmethod Ġhas Ġcomparable Ġperformance Ġwith Ġthe Ġoriginal ĠGl o Ve Ġembed d ings . ĠOur Ġmethod Ġoutper forms ĠGl o Ve Ġin Ġsemantic Ġanalogy Ġtest Ġset Ġand Ġin Ġoverall Ġresults , Ġwhile ĠGl o Ve Ġperforms Ġslightly Ġbetter Ġin Ġsynt actic Ġtest Ġset . ĠThis Ġcomparable Ġperformance Ġis Ġmainly Ġdue Ġto Ġthe Ġcost Ġfunction Ġof Ġour Ġproposed Ġmethod Ġthat Ġincludes Ġthe Ġoriginal Ġobjective Ġof Ġthe ĠGl o Ve . </s> To Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġadditional Ġcost Ġterm Ġon Ġthe Ġperformance Ġimprovement Ġin Ġthe Ġsemantic Ġanalogy Ġtest , Ġwe Ġpresent ĠTable ĠT AB REF 22 Ġ. ĠIn Ġparticular , Ġwe Ġpresent Ġresults Ġfor Ġthe Ġcases Ġwhere Ġi ) Ġall Ġquestions Ġin Ġthe Ġdataset Ġare Ġconsidered , Ġii ) Ġonly Ġthe Ġquestions Ġthat Ġcontains Ġat Ġleast Ġone Ġconcept Ġword Ġare Ġconsidered , Ġiii ) Ġonly Ġthe Ġquestions Ġthat Ġconsist Ġentirely Ġof Ġconcept Ġwords Ġare Ġconsidered . ĠWe Ġnote Ġspecifically Ġthat Ġfor Ġthe Ġlast Ġcase , Ġonly Ġa Ġsubset Ġof Ġthe Ġquestions Ġunder Ġthe Ġsemantic Ġcategory Ġfamily . txt Ġended Ġup Ġbeing Ġincluded . ĠWe Ġobserve Ġthat Ġfor Ġall Ġthree Ġscenarios , Ġour Ġproposed Ġalgorithm Ġresults Ġin Ġan Ġimprovement Ġin Ġthe Ġprecision Ġscores . ĠHowever , Ġthe Ġgreatest Ġperformance Ġincrease Ġis Ġseen Ġfor Ġthe Ġlast Ġscenario , Ġwhich Ġunderscores Ġthe Ġextent Ġto Ġwhich Ġthe Ġsemantic Ġfeatures Ġcaptured Ġby Ġembed d ings Ġcan Ġbe Ġimproved Ġwith Ġa Ġreasonable Ġselection Ġof Ġthe Ġlex ical Ġresource Ġfrom Ġwhich Ġthe Ġconcept Ġword - groups Ġwere Ġderived . </s> Conclusion </s> We Ġpresented Ġa Ġnovel Ġapproach Ġto Ġimpart Ġinterpret ability Ġinto Ġword Ġembed d ings . ĠWe Ġachieved Ġthis Ġby Ġencouraging Ġdifferent Ġdimensions Ġof Ġthe Ġvector Ġrepresentation Ġto Ġalign Ġwith Ġpred efined Ġconcepts , Ġthrough Ġthe Ġaddition Ġof Ġan Ġadditional Ġcost Ġterm Ġin Ġthe Ġoptimization Ġobjective Ġof Ġthe ĠGl o Ve Ġalgorithm Ġthat Ġfavors Ġa Ġselective Ġincrease Ġfor Ġa Ġpre - specified Ġinput Ġof Ġconcept Ġwords Ġalong Ġeach Ġdimension . </s> We Ġdemonstrated Ġthe Ġefficacy Ġof Ġthis Ġapproach Ġby Ġapplying Ġqualitative Ġand Ġquantitative Ġevaluations Ġfor Ġinterpret ability . ĠWe Ġalso Ġshowed Ġvia Ġstandard Ġword - an alogy Ġand Ġword - similar ity Ġtests Ġthat Ġthe Ġsemantic Ġco herence Ġof Ġthe Ġoriginal Ġvector Ġspace Ġis Ġpreserved , Ġeven Ġslightly Ġimproved . ĠWe Ġhave Ġalso Ġperformed Ġand Ġreported Ġquantitative Ġcomparisons Ġwith Ġseveral Ġother Ġmethods Ġfor Ġboth Ġinterpret ab ilty Ġincrease Ġand Ġpreservation Ġof Ġsemantic Ġco herence . ĠUpon Ġinspection Ġof ĠFig . ĠFIG REF 18 Ġand ĠTables ĠT AB REF 20 Ġ, ĠT AB REF 21 Ġ, Ġand ĠT AB REF 22 Ġaltogether , Ġit Ġshould Ġbe Ġnoted Ġthat Ġour Ġproposed Ġmethod Ġachieves Ġboth Ġof Ġthe Ġobjectives Ġsimultaneously , Ġincreased Ġinterpret ability Ġand Ġpreservation Ġof Ġthe Ġintrinsic Ġsemantic Ġstructure . </s> An Ġimportant Ġpoint Ġwas Ġthat , Ġwhile Ġit Ġis Ġexpected Ġfor Ġwords Ġthat Ġare Ġalready Ġincluded Ġin Ġthe Ġconcept Ġword - groups Ġto Ġbe Ġaligned Ġtogether Ġsince Ġtheir Ġdimensions Ġare Ġdirectly Ġupdated Ġwith Ġthe Ġproposed Ġcost Ġterm , Ġit Ġwas Ġalso Ġobserved Ġthat Ġwords Ġnot Ġin Ġthese Ġgroups Ġalso Ġaligned Ġin Ġa Ġmeaningful Ġmanner Ġwithout Ġany Ġdirect Ġmodification Ġto Ġtheir Ġcost Ġfunction . ĠThis Ġindicates Ġthat Ġthe Ġcost Ġterm Ġwe Ġadded Ġworks Ġproduct ively Ġwith Ġthe Ġoriginal Ġcost Ġfunction Ġof ĠGl o Ve Ġto Ġhandle Ġwords Ġthat Ġare Ġnot Ġincluded Ġin Ġthe Ġoriginal Ġconcept Ġword - groups , Ġbut Ġare Ġsem antically Ġrelated Ġto Ġthose Ġword - groups . ĠThe Ġunderlying Ġmechanism Ġcan Ġbe Ġexplained Ġas Ġfollows . ĠWhile Ġthe Ġoutside Ġlex ical Ġresource Ġwe Ġintroduce Ġcontains Ġa Ġrelatively Ġsmall Ġnumber Ġof Ġwords Ġcompared Ġto Ġthe Ġtotal Ġnumber Ġof Ġwords , Ġthese Ġwords Ġand Ġthe Ġcategories Ġthey Ġrepresent Ġhave Ġbeen Ġcarefully Ġchosen Ġand Ġin Ġa Ġsense , Ġ" d ense ly Ġspan " Ġall Ġthe Ġwords Ġin Ġthe Ġlanguage . ĠBy Ġsaying Ġ" span ", Ġwe Ġmean Ġthey Ġcover Ġmost Ġof Ġthe Ġconcepts Ġand Ġideas Ġin Ġthe Ġlanguage Ġwithout Ġleaving Ġtoo Ġmany Ġuncovered Ġareas . ĠWith Ġ" d ense ly " Ġwe Ġmean Ġall Ġareas Ġare Ġcovered Ġwith Ġsufficient Ġstrength . ĠIn Ġother Ġwords , Ġthis Ġsubset Ġof Ġwords Ġis Ġable Ġto Ġconstitute Ġa Ġsufficiently Ġstrong Ġskeleton , Ġor Ġscaff old . ĠNow Ġremember Ġthat ĠGl o Ve Ġworks Ġto Ġalign Ġor Ġbring Ġcloser Ġrelated Ġgroups Ġof Ġwords , Ġwhich Ġwill Ġinclude Ġwords Ġfrom Ġthe Ġlex ical Ġsource . ĠSo Ġthe Ġjoint Ġaction Ġof Ġalign ing Ġthe Ġwords Ġwith Ġthe Ġpred efined Ġcategories Ġ( introdu ced Ġby Ġus ) Ġand Ġalign ing Ġrelated Ġwords Ġ( handled Ġby ĠGl o Ve ) Ġallows Ġwords Ġnot Ġin Ġthe Ġlex ical Ġgroups Ġto Ġalso Ġbe Ġaligned Ġmeaning fully . ĠWe Ġmay Ġsay Ġthat Ġthe Ġnon - in cluded Ġwords Ġare Ġ" pull ed Ġalong " Ġwith Ġthe Ġincluded Ġwords Ġby Ġvirtue Ġof Ġthe Ġ" strings " Ġor Ġ" gl ue " Ġthat Ġis Ġprovided Ġby ĠGl o Ve . ĠIn Ġnumbers , Ġthe Ġdesired Ġeffect Ġis Ġachieved Ġby Ġmanipulating Ġless Ġthan Ġonly Ġ0 . 05 % Ġof Ġparameters Ġof Ġthe Ġentire Ġword Ġvectors . ĠThus , Ġwhile Ġthere Ġis Ġa Ġdegree Ġof Ġsupervision Ġcoming Ġfrom Ġthe Ġexternal Ġlex ical Ġresource , Ġthe Ġrest Ġof Ġthe Ġvocabulary Ġis Ġalso Ġaligned Ġindirectly Ġin Ġan Ġun super vised Ġway . ĠThis Ġmay Ġbe Ġthe Ġreason Ġwhy , Ġunlike Ġearlier Ġproposed Ġapproaches , Ġour Ġmethod Ġis Ġable Ġto Ġachieve Ġincreasing Ġinterpret ability Ġwithout Ġdestroying Ġunderlying Ġsemantic Ġstructure , Ġand Ġconsequently Ġwithout Ġsacrificing Ġperformance Ġin Ġbenchmark Ġtests . </s> Upon Ġinspecting Ġthe Ġ2 nd Ġcolumn Ġof ĠTable ĠT AB REF 14 Ġ, Ġwhere Ġqualitative Ġresults Ġfor Ġconcept ĠT AS TE Ġare Ġpresented , Ġanother Ġinsight Ġregarding Ġthe Ġlearning Ġmechanism Ġof Ġour Ġproposed Ġapproach Ġcan Ġbe Ġmade . ĠHere Ġit Ġseems Ġunderstandable Ġthat Ġour Ġproposed Ġapproach , Ġalong Ġwith ĠGl o Ve , Ġbrought Ġtogether Ġthe Ġwords Ġtaste Ġand Ġpolish , Ġand Ġthen Ġthe Ġwords ĠPolish Ġand , Ġfor Ġinstance , ĠWarsaw Ġare Ġbrought Ġtogether Ġby ĠGl o Ve . ĠThese Ġexamples Ġare Ġinteresting Ġin Ġthat Ġthey Ġshed Ġinsight Ġinto Ġhow ĠGl o Ve Ġworks Ġand Ġthe Ġlimitations Ġposed Ġby Ġpoly se my . ĠIt Ġshould Ġbe Ġunder lined Ġthat Ġthe Ġpresent Ġapproach Ġis Ġnot Ġtotally Ġincapable Ġof Ġhandling Ġpoly se my , Ġbut Ġcannot Ġdo Ġso Ġperfectly . ĠSince Ġrelated Ġwords Ġare Ġbeing Ġclustered , Ġsufficiently Ġwell - connected Ġwords Ġthat Ġdo Ġnot Ġmeaning fully Ġbelong Ġalong Ġwith Ġothers Ġwill Ġbe Ġappropriately Ġ" pull ed Ġaway " Ġfrom Ġthat Ġgroup Ġby Ġseveral Ġwords , Ġagainst Ġthe Ġless Ġeffective , Ġinappropriate Ġpull Ġof Ġa Ġparticular Ġword . ĠEven Ġthough Ġpolish Ġwith Ġlower case Ġ" p " Ġbelongs Ġwhere Ġit Ġis , Ġit Ġis Ġattracting ĠWarsaw Ġto Ġitself Ġthrough Ġpoly se my Ġand Ġthis Ġis Ġnot Ġmeaningful . ĠPerhaps Ġbecause ĠWarsaw Ġis Ġnot Ġa Ġsufficiently Ġwell - connected Ġword , Ġit Ġends Ġbeing Ġdragged Ġalong , Ġalthough Ġwords Ġwith Ġgreater Ġconnected ness Ġto Ġa Ġconcept Ġgroup Ġmight Ġhave Ġbetter Ġresisted Ġsuch Ġinappropriate Ġattractions . </s> In Ġthis Ġstudy , Ġwe Ġused Ġthe ĠGl o Ve Ġalgorithm Ġas Ġthe Ġunderlying Ġdense Ġword Ġembed ding Ġscheme Ġto Ġdemonstrate Ġour Ġapproach . ĠHowever , Ġwe Ġstress Ġthat Ġit Ġis Ġpossible Ġfor Ġour Ġapproach Ġto Ġbe Ġextended Ġto Ġother Ġword Ġembed ding Ġalgorithms Ġwhich Ġhave Ġa Ġlearning Ġroutine Ġconsisting Ġof Ġiterations Ġover Ġco occ urrence Ġrecords , Ġby Ġmaking Ġsuitable Ġadjustments Ġin Ġthe Ġobjective Ġfunction . ĠSince Ġword 2 vec Ġmodel Ġis Ġalso Ġbased Ġon Ġthe Ġco oc ur rences Ġof Ġwords Ġin Ġa Ġsliding Ġwindow Ġthrough Ġa Ġlarge Ġcorpus , Ġwe Ġexpect Ġthat Ġour Ġapproach Ġcan Ġalso Ġbe Ġapplied Ġto Ġword 2 vec Ġafter Ġmaking Ġsuitable Ġadjustments , Ġwhich Ġcan Ġbe Ġconsidered Ġas Ġan Ġimmediate Ġfuture Ġwork Ġfor Ġour Ġapproach . ĠAlthough Ġthe Ġsemantic Ġconcepts Ġare Ġencoded Ġin Ġonly Ġone Ġdirection Ġ( positive ) Ġwithin Ġthe Ġembed ding Ġdimensions , Ġit Ġmight Ġbe Ġbeneficial Ġto Ġpursue Ġfuture Ġwork Ġthat Ġalso Ġenc odes Ġopposite Ġconcepts , Ġsuch Ġas Ġgood Ġand Ġbad , Ġin Ġtwo Ġopposite Ġdirections Ġof Ġthe Ġsame Ġdimension . </s> The Ġproposed Ġmethodology Ġcan Ġalso Ġbe Ġhelpful Ġin Ġcomputational Ġcross - ling ual Ġstudies , Ġwhere Ġthe Ġsimilarities Ġare Ġexplored Ġacross Ġthe Ġvector Ġspaces Ġof Ġdifferent Ġlanguages ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ.