Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese
ABSTRACTIVE
Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese
ABSTRACTIVE
Language Selection. Multi-SimLex comprises eleven languages in addition to English. The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family). Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex. We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families. This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili. We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project. FLOAT SELECTED: Table 1: The list of 12 languages in the Multi-SimLex multilingual suite along with their corresponding language family (IE = Indo-European), broad morphological type, and their ISO 639-3 code. The number of speakers is based on the total count of L1 and L2 speakers, according to ethnologue.com.
FLOAT SELECTED: Table 1: The list of 12 languages in the Multi-SimLex multilingual suite along with their corresponding language family (IE = Indo-European), broad morphological type, and their ISO 639-3 code. The number of speakers is based on the total count of L1 and L2 speakers, according to ethnologue.com.
2003.04866
Introduction </s> The Ġlack Ġof Ġannot ated Ġtraining Ġand Ġevaluation Ġdata Ġfor Ġmany Ġtasks Ġand Ġdomains Ġhind ers Ġthe Ġdevelopment Ġof Ġcomputational Ġmodels Ġfor Ġthe Ġmajority Ġof Ġthe Ġworld 's Ġlanguages ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . ĠThe Ġnecessity Ġto Ġguide Ġand Ġadvance Ġmult ilingual Ġand Ġcross - ling ual ĠN LP Ġthrough Ġannotation Ġefforts Ġthat Ġfollow Ġcross - ling ually Ġconsistent Ġguidelines Ġhas Ġbeen Ġrecently Ġrecognized Ġby Ġcollaborative Ġinitiatives Ġsuch Ġas Ġthe ĠUniversal ĠDepend ency Ġ( UD ) Ġproject ĠB IB REF 3 . ĠThe Ġlatest Ġversion Ġof ĠUD Ġ( as Ġof ĠMarch Ġ2020 ) Ġcovers Ġmore Ġthan Ġ70 Ġlanguages . ĠCru cially , Ġthis Ġresource Ġcontinues Ġto Ġsteadily Ġgrow Ġand Ġevolve Ġthrough Ġthe Ġcontributions Ġof Ġannot ators Ġfrom Ġacross Ġthe Ġworld , Ġextending Ġthe ĠUD 's Ġreach Ġto Ġa Ġwide Ġarray Ġof Ġtyp ologically Ġdiverse Ġlanguages . ĠBesides Ġsteering Ġresearch Ġin Ġmult ilingual Ġparsing ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġcross - ling ual Ġparser Ġtransfer ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , Ġthe Ġconsistent Ġannotations Ġand Ġguidelines Ġhave Ġalso Ġenabled Ġa Ġrange Ġof Ġinsightful Ġcomparative Ġstudies Ġfocused Ġon Ġthe Ġlanguages ' Ġsynt actic Ġ( dis ) similar ities ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 . </s> Insp ired Ġby Ġthe ĠUD Ġwork Ġand Ġits Ġsubstantial Ġimpact Ġon Ġresearch Ġin Ġ( mult ilingual ) Ġsyntax , Ġin Ġthis Ġarticle Ġwe Ġintroduce ĠMulti - Sim Lex , Ġa Ġsuite Ġof Ġmanually Ġand Ġconsistently Ġannot ated Ġsemantic Ġdatasets Ġfor Ġ12 Ġdifferent Ġlanguages , Ġfocused Ġon Ġthe Ġfundamental Ġlex ical Ġrelation Ġof Ġsemantic Ġsimilarity ĠB IB REF 13 , ĠB IB REF 14 . ĠFor Ġany Ġpair Ġof Ġwords , Ġthis Ġrelation Ġmeasures Ġwhether Ġtheir Ġrefere nt s Ġshare Ġthe Ġsame Ġ( functional ) Ġfeatures , Ġas Ġopposed Ġto Ġgeneral Ġcognitive Ġassociation Ġcaptured Ġby Ġco - occ urrence Ġpatterns Ġin Ġtexts Ġ( i . e ., Ġthe Ġdistribution al Ġinformation ). ĠDat as ets Ġthat Ġquantify Ġthe Ġstrength Ġof Ġtrue Ġsemantic Ġsimilarity Ġbetween Ġconcept Ġpairs Ġsuch Ġas ĠSim Lex - 999 ĠB IB REF 14 Ġor ĠSim Ver b - 35 00 ĠB IB REF 15 Ġhave Ġbeen Ġinstrumental Ġin Ġimproving Ġmodels Ġfor Ġdistribution al Ġsemantics Ġand Ġrepresentation Ġlearning . ĠDis cerning Ġbetween Ġsemantic Ġsimilarity Ġand Ġrelated ness / ass ociation Ġis Ġnot Ġonly Ġcrucial Ġfor Ġtheoretical Ġstudies Ġon Ġlex ical Ġsemantics Ġ( see ĠÂ§ SEC REF 2 ), Ġbut Ġhas Ġalso Ġbeen Ġshown Ġto Ġbenefit Ġa Ġrange Ġof Ġlanguage Ġunderstanding Ġtasks Ġin ĠN LP . ĠExamples Ġinclude Ġdialog Ġstate Ġtracking ĠB IB REF 16 , ĠB IB REF 17 , Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 18 , ĠB IB REF 19 , Ġtext Ġsimpl ification ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 , Ġdictionary Ġand Ġthe s aurus Ġconstruction ĠB IB REF 23 , ĠB IB REF 24 . </s> Despite Ġthe Ġproven Ġusefulness Ġof Ġsemantic Ġsimilarity Ġdatasets , Ġthey Ġare Ġavailable Ġonly Ġfor Ġa Ġsmall Ġand Ġtyp ologically Ġnarrow Ġsample Ġof Ġresource - rich Ġlanguages Ġsuch Ġas ĠGerman , ĠItalian , Ġand ĠRussian ĠB IB REF 25 , Ġwhereas Ġsome Ġlanguage Ġtypes Ġand Ġlow - resource Ġlanguages Ġtypically Ġlack Ġsimilar Ġevaluation Ġdata . ĠEven Ġif Ġsome Ġresources Ġdo Ġexist , Ġthey Ġare Ġlimited Ġin Ġtheir Ġsize Ġ( e . g ., Ġ500 Ġpairs Ġin ĠTurkish ĠB IB REF 26 , Ġ500 Ġin ĠF ars i ĠB IB REF 27 , Ġor Ġ300 Ġin ĠFinnish ĠB IB REF 28 ) Ġand Ġcoverage Ġ( e . g ., Ġall Ġdatasets Ġwhich Ġoriginated Ġfrom Ġthe Ġoriginal ĠEnglish ĠSim Lex - 999 Ġcontain Ġonly Ġhigh - f requent Ġconcepts , Ġand Ġare Ġdominated Ġby Ġnoun s ). ĠThis Ġis Ġwhy , Ġas Ġour Ġdeparture Ġpoint , Ġwe Ġintroduce Ġa Ġlarger Ġand Ġmore Ġcomprehensive ĠEnglish Ġword Ġsimilarity Ġdataset Ġspanning Ġ1 , 888 Ġconcept Ġpairs Ġ( see ĠÂ§ SEC REF 4 ). </s> Most Ġimportantly , Ġsemantic Ġsimilarity Ġdatasets Ġin Ġdifferent Ġlanguages Ġhave Ġbeen Ġcreated Ġusing Ġheter ogeneous Ġconstruction Ġprocedures Ġwith Ġdifferent Ġguidelines Ġfor Ġtranslation Ġand Ġannotation , Ġas Ġwell Ġas Ġdifferent Ġrating Ġscales . ĠFor Ġinstance , Ġsome Ġdatasets Ġwere Ġobtained Ġby Ġdirectly Ġtranslating Ġthe ĠEnglish ĠSim Lex - 999 Ġin Ġits Ġentirety ĠB IB REF 25 , ĠB IB REF 16 Ġor Ġin Ġpart ĠB IB REF 28 . ĠOther Ġdatasets Ġwere Ġcreated Ġfrom Ġscratch ĠB IB REF 26 Ġand Ġyet Ġothers Ġsampled ĠEnglish Ġconcept Ġpairs Ġdifferently Ġfrom ĠSim Lex - 999 Ġand Ġthen Ġtranslated Ġand Ġre annot ated Ġthem Ġin Ġtarget Ġlanguages ĠB IB REF 27 . ĠThis Ġheterogeneity Ġmakes Ġthese Ġdatasets Ġincom parable Ġand Ġpre cludes Ġsystematic Ġcross - ling u istic Ġanalyses . ĠIn Ġthis Ġarticle , Ġconsolid ating Ġthe Ġlessons Ġlearned Ġfrom Ġprevious Ġdataset Ġconstruction Ġparad ig ms , Ġwe Ġpropose Ġa Ġcarefully Ġdesigned Ġtranslation Ġand Ġannotation Ġprotocol Ġfor Ġdeveloping Ġmon oling ual ĠMulti - Sim Lex Ġdatasets Ġwith Ġaligned Ġconcept Ġpairs Ġfor Ġtyp ologically Ġdiverse Ġlanguages . ĠWe Ġapply Ġthis Ġprotocol Ġto Ġa Ġset Ġof Ġ12 Ġlanguages , Ġincluding Ġa Ġmixture Ġof Ġmajor Ġlanguages Ġ( e . g ., ĠMandarin , ĠRussian , Ġand ĠFrench ) Ġas Ġwell Ġas Ġseveral Ġlow - resource Ġones Ġ( e . g ., ĠKis w ah ili , ĠWelsh , Ġand ĠYue ĠChinese ). ĠWe Ġdemonstrate Ġthat Ġour Ġproposed Ġdataset Ġcreation Ġprocedure Ġyields Ġdata Ġwith Ġhigh Ġinter - annot ator Ġagreement Ġrates Ġ( e . g ., Ġthe Ġaverage Ġmean Ġinter - annot ator Ġagreement Ġfor ĠWelsh Ġis Ġ0 . 7 42 ). </s> The Ġunified Ġconstruction Ġprotocol Ġand Ġalignment Ġbetween Ġconcept Ġpairs Ġenables Ġa Ġseries Ġof Ġquantitative Ġanalyses . ĠPrel iminary Ġstudies Ġon Ġthe Ġinfluence Ġthat Ġpoly se my Ġand Ġcross - ling ual Ġvariation Ġin Ġlex ical Ġcategories Ġ( see ĠÂ§ SEC REF 6 ) Ġhave Ġon Ġsimilarity Ġjudgments Ġare Ġprovided Ġin ĠÂ§ SEC REF 5 . ĠData Ġcreated Ġaccording Ġto ĠMulti - Sim Lex Ġprotocol Ġalso Ġallow Ġfor Ġprobing Ġinto Ġwhether Ġsimilarity Ġjudgments Ġare Ġuniversal Ġacross Ġlanguages , Ġor Ġrather Ġdepend Ġon Ġlinguistic Ġaffinity Ġ( in Ġterms Ġof Ġlinguistic Ġfeatures , Ġphylogen y , Ġand Ġgeographical Ġlocation ). ĠWe Ġinvestigate Ġthis Ġquestion Ġin ĠÂ§ SEC REF 25 . ĠNaturally , ĠMulti - Sim Lex Ġdatasets Ġcan Ġbe Ġused Ġas Ġan Ġintrinsic Ġevaluation Ġbenchmark Ġto Ġassess Ġthe Ġquality Ġof Ġlex ical Ġrepresentations Ġbased Ġon Ġmon oling ual , Ġjoint Ġmult ilingual , Ġand Ġtransfer Ġlearning Ġparad ig ms . ĠWe Ġconduct Ġa Ġsystematic Ġevaluation Ġof Ġseveral Ġstate - of - the - art Ġrepresentation Ġmodels Ġin ĠÂ§ SEC REF 7 , Ġshowing Ġthat Ġthere Ġare Ġlarge Ġgaps Ġbetween Ġhuman Ġand Ġsystem Ġperformance Ġin Ġall Ġlanguages . ĠThe Ġproposed Ġconstruction Ġparadigm Ġalso Ġsupports Ġthe Ġautomatic Ġcreation Ġof Ġ66 Ġcross - ling ual ĠMulti - Sim Lex Ġdatasets Ġby Ġinter le aving Ġthe Ġmon oling ual Ġones . ĠWe Ġoutline Ġthe Ġconstruction Ġof Ġthe Ġcross - ling ual Ġdatasets Ġin ĠÂ§ SEC REF 6 , Ġand Ġthen Ġpresent Ġa Ġquantitative Ġevaluation Ġof Ġa Ġseries Ġof Ġcutting - edge Ġcross - ling ual Ġrepresentation Ġmodels Ġon Ġthis Ġbenchmark Ġin ĠÂ§ SEC REF 8 . </s> Contribut ions . ĠWe Ġnow Ġsummarize Ġthe Ġmain Ġcontributions Ġof Ġthis Ġwork : </s> 1 ) ĠBuilding Ġon Ġlessons Ġlearned Ġfrom Ġprior Ġwork , Ġwe Ġcreate Ġa Ġmore Ġcomprehensive Ġlex ical Ġsemantic Ġsimilarity Ġdataset Ġfor Ġthe ĠEnglish Ġlanguage Ġspanning Ġa Ġtotal Ġof Ġ1 , 888 Ġconcept Ġpairs Ġbalanced Ġwith Ġrespect Ġto Ġsimilarity , Ġfrequency , Ġand Ġconc ret eness , Ġand Ġcovering Ġfour Ġword Ġclasses : Ġnoun s , Ġverbs , Ġadject ives Ġand , Ġfor Ġthe Ġfirst Ġtime , Ġad verbs . ĠThis Ġdataset Ġserves Ġas Ġthe Ġmain Ġsource Ġfor Ġthe Ġcreation Ġof Ġequivalent Ġdatasets Ġin Ġseveral Ġother Ġlanguages . </s> 2 ) ĠWe Ġpresent Ġa Ġcarefully Ġdesigned Ġand Ġrigorous Ġlanguage - agn ostic Ġtranslation Ġand Ġannotation Ġprotocol . ĠThese Ġwell - defined Ġguidelines Ġwill Ġfacilitate Ġthe Ġdevelopment Ġof Ġfuture ĠMulti - Sim Lex Ġdatasets Ġfor Ġother Ġlanguages . ĠThe Ġproposed Ġprotocol Ġeliminates Ġsome Ġcrucial Ġissues Ġwith Ġprior Ġefforts Ġfocused Ġon Ġthe Ġcreation Ġof Ġmulti - ling ual Ġsemantic Ġresources , Ġnamely : Ġi ) Ġlimited Ġcoverage ; Ġii ) Ġheter ogeneous Ġannotation Ġguidelines ; Ġand Ġiii ) Ġconcept Ġpairs Ġwhich Ġare Ġsem antically Ġincom parable Ġacross Ġdifferent Ġlanguages . </s> 3 ) ĠWe Ġoffer Ġto Ġthe Ġcommunity Ġmanually Ġannot ated Ġevaluation Ġsets Ġof Ġ1 , 888 Ġconcept Ġpairs Ġacross Ġ12 Ġtyp ologically Ġdiverse Ġlanguages , Ġand Ġ66 Ġlarge Ġcross - ling ual Ġevaluation Ġsets . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , ĠMulti - Sim Lex Ġis Ġthe Ġmost Ġcomprehensive Ġevaluation Ġresource Ġto Ġdate Ġfocused Ġon Ġthe Ġrelation Ġof Ġsemantic Ġsimilarity . </s> 4 ) ĠWe Ġbenchmark Ġa Ġwide Ġarray Ġof Ġrecent Ġstate - of - the - art Ġmon oling ual Ġand Ġcross - ling ual Ġword Ġrepresentation Ġmodels Ġacross Ġour Ġsample Ġof Ġlanguages . ĠThe Ġresults Ġcan Ġserve Ġas Ġstrong Ġbas elines Ġthat Ġlay Ġthe Ġfoundation Ġfor Ġfuture Ġimprovements . </s> 5 ) ĠWe Ġpresent Ġa Ġfirst Ġlarge - scale Ġevaluation Ġstudy Ġon Ġthe Ġability Ġof Ġenc od ers Ġpret rained Ġon Ġlanguage Ġmodeling Ġ( such Ġas Ġb ert ĠB IB REF 29 Ġand Ġx l m ĠB IB REF 30 ) Ġto Ġreason Ġover Ġword - level Ġsemantic Ġsimilarity Ġin Ġdifferent Ġlanguages . ĠTo Ġour Ġown Ġsurprise , Ġthe Ġresults Ġshow Ġthat Ġmon oling ual Ġpret rained Ġenc od ers , Ġeven Ġwhen Ġpresented Ġwith Ġword Ġtypes Ġout Ġof Ġcontext , Ġare Ġsometimes Ġcompetitive Ġwith Ġstatic Ġword Ġembed ding Ġmodels Ġsuch Ġas Ġfast Text ĠB IB REF 31 Ġor Ġword 2 vec ĠB IB REF 32 . ĠThe Ġresults Ġalso Ġreveal Ġa Ġhuge Ġgap Ġin Ġperformance Ġbetween Ġmassively Ġmult ilingual Ġpret rained Ġenc od ers Ġand Ġlanguage - specific Ġenc od ers Ġin Ġfavor Ġof Ġthe Ġlatter : Ġour Ġfindings Ġsupport Ġother Ġrecent Ġempirical Ġevidence Ġrelated Ġto Ġthe ĠâĢ ľ c urse Ġof Ġmult iling uality âĢ Ŀ ĠB IB REF 33 , ĠB IB REF 34 Ġin Ġrepresentation Ġlearning . </s> 6 ) ĠWe Ġmake Ġall Ġof Ġthese Ġresources Ġavailable Ġon Ġa Ġwebsite Ġwhich Ġfacilitates Ġeasy Ġcreation , Ġsubmission Ġand Ġsharing Ġof ĠMulti - Sim lex - style Ġdatasets Ġfor Ġa Ġlarger Ġnumber Ġof Ġlanguages . ĠWe Ġhope Ġthat Ġthis Ġwill Ġyield Ġan Ġeven Ġlarger Ġrepository Ġof Ġsemantic Ġresources Ġthat Ġinspire Ġfuture Ġadvances Ġin ĠN LP Ġwithin Ġand Ġacross Ġlanguages . </s> In Ġlight Ġof Ġthe Ġsuccess Ġof ĠUniversal ĠDepend encies ĠB IB REF 3 , Ġwe Ġhope Ġthat Ġour Ġinitiative Ġwill Ġinst igate Ġa Ġcollaborative Ġpublic Ġeffort Ġwith Ġestablished Ġand Ġclear - cut Ġguidelines Ġthat Ġwill Ġresult Ġin Ġadditional ĠMulti - Sim Lex Ġdatasets Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages Ġin Ġthe Ġnear Ġfuture . ĠMoreover , Ġwe Ġhope Ġthat Ġit Ġwill Ġprovide Ġmeans Ġto Ġadvance Ġour Ġunderstanding Ġof Ġdistribution al Ġand Ġlex ical Ġsemantics Ġacross Ġa Ġlarge Ġnumber Ġof Ġlanguages . ĠAll Ġmon oling ual Ġand Ġcross - ling ual ĠMulti - Sim Lex Ġdatasets âĢĵ along Ġwith Ġdetailed Ġtranslation Ġand Ġannotation Ġguidelines âĢĵ are Ġavailable Ġonline Ġat : Ġhttps :// mult is im lex . com /. </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġand ĠAssociation </s> The Ġfocus Ġof Ġthe ĠMulti - Sim Lex Ġinitiative Ġis Ġon Ġthe Ġlex ical Ġrelation Ġof Ġpure Ġsemantic Ġsimilarity . ĠFor Ġany Ġpair Ġof Ġwords , Ġthis Ġrelation Ġmeasures Ġwhether Ġtheir Ġrefere nt s Ġshare Ġthe Ġsame Ġfeatures . ĠFor Ġinstance , Ġgraffiti Ġand Ġfres cos Ġare Ġsimilar Ġto Ġthe Ġextent Ġthat Ġthey Ġare Ġboth Ġforms Ġof Ġpainting Ġand Ġappear Ġon Ġwalls . ĠThis Ġrelation Ġcan Ġbe Ġcontrasted Ġwith Ġthe Ġcognitive Ġassociation Ġbetween Ġtwo Ġwords , Ġwhich Ġoften Ġdepends Ġon Ġhow Ġmuch Ġtheir Ġrefere nt s Ġinteract Ġin Ġthe Ġreal Ġworld , Ġor Ġare Ġfound Ġin Ġthe Ġsame Ġsituations . ĠFor Ġinstance , Ġa Ġpainter Ġis Ġeasily Ġassociated Ġwith Ġfres cos , Ġalthough Ġthey Ġlack Ġany Ġphysical Ġcommon alities . ĠAssociation Ġis Ġalso Ġknown Ġin Ġthe Ġliterature Ġunder Ġother Ġnames : Ġrelated ness ĠB IB REF 13 , Ġtopical Ġsimilarity ĠB IB REF 35 , Ġand Ġdomain Ġsimilarity ĠB IB REF 36 . </s> Sem antic Ġsimilarity Ġand Ġassociation Ġoverlap Ġto Ġsome Ġdegree , Ġbut Ġdo Ġnot Ġcoincide ĠB IB REF 37 , ĠB IB REF 38 . ĠIn Ġfact , Ġthere Ġexist Ġplenty Ġof Ġpairs Ġthat Ġare Ġintuitive ly Ġassociated Ġbut Ġnot Ġsimilar . ĠP airs Ġwhere Ġthe Ġcon verse Ġis Ġtrue Ġcan Ġalso Ġbe Ġencountered , Ġalthough Ġmore Ġrarely . ĠAn Ġexample Ġare Ġsyn onyms Ġwhere Ġa Ġword Ġis Ġcommon Ġand Ġthe Ġother Ġinf requent , Ġsuch Ġas Ġto Ġseize Ġand Ġto Ġcommand eer . ĠB IB REF 14 Ġrevealed Ġthat Ġwhile Ġsimilarity Ġmeasures Ġbased Ġon Ġthe ĠWord Net Ġgraph ĠB IB REF 39 Ġand Ġhuman Ġjudgments Ġof Ġassociation Ġin Ġthe ĠUniversity Ġof ĠSouth ĠFlorida ĠFree ĠAssociation ĠDatabase ĠB IB REF 40 Ġdo Ġcorrelate , Ġa Ġnumber Ġof Ġpairs Ġfollow Ġopposite Ġtrends . ĠSeveral Ġstudies Ġon Ġhuman Ġcognition Ġalso Ġpoint Ġin Ġthe Ġsame Ġdirection . ĠFor Ġinstance , Ġsemantic Ġprim ing Ġcan Ġbe Ġtriggered Ġby Ġsimilar Ġwords Ġwithout Ġassociation ĠB IB REF 41 . ĠOn Ġthe Ġother Ġhand , Ġa Ġconnection Ġwith Ġcue Ġwords Ġis Ġestablished Ġmore Ġquickly Ġfor Ġtop ically Ġrelated Ġwords Ġrather Ġthan Ġfor Ġsimilar Ġwords Ġin Ġfree Ġassociation Ġtasks ĠB IB REF 42 . </s> A Ġkey Ġproperty Ġof Ġsemantic Ġsimilarity Ġis Ġits Ġgrad ience : Ġpairs Ġof Ġwords Ġcan Ġbe Ġsimilar Ġto Ġa Ġdifferent Ġdegree . ĠOn Ġthe Ġother Ġhand , Ġthe Ġrelation Ġof Ġsyn ony my Ġis Ġbinary : Ġpairs Ġof Ġwords Ġare Ġsyn onyms Ġif Ġthey Ġcan Ġbe Ġsubstituted Ġin Ġall Ġcontexts Ġ( or Ġmost Ġcontexts , Ġin Ġa Ġlo oser Ġsense ), Ġotherwise Ġthey Ġare Ġnot . ĠWhile Ġsyn onyms Ġcan Ġbe Ġconceived Ġas Ġlying Ġon Ġone Ġextreme Ġof Ġthe Ġsemantic Ġsimilarity Ġcontinuum , Ġit Ġis Ġcrucial Ġto Ġnote Ġthat Ġtheir Ġdefinition Ġis Ġstated Ġin Ġpurely Ġrelational Ġterms , Ġrather Ġthan Ġinvoking Ġtheir Ġrefere nt ial Ġproperties ĠB IB REF 43 , ĠB IB REF 44 , ĠB IB REF 45 . ĠThis Ġmakes Ġbehavioral Ġstudies Ġon Ġsemantic Ġsimilarity Ġfundamentally Ġdifferent Ġfrom Ġlex ical Ġresources Ġlike ĠWord Net ĠB IB REF 46 , Ġwhich Ġinclude Ġparad igmatic Ġrelations Ġ( such Ġas Ġsyn ony my ). </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġfor ĠN LP : ĠInt r ins ic ĠEvaluation Ġand ĠSem antic ĠSpecial ization </s> The Ġramifications Ġof Ġthe Ġdistinction Ġbetween Ġsimilarity Ġand Ġassociation Ġare Ġprofound Ġfor Ġdistribution al Ġsemantics . ĠThis Ġparadigm Ġof Ġlex ical Ġsemantics Ġis Ġgrounded Ġin Ġthe Ġdistribution al Ġhypothesis , Ġformulated Ġby ĠB IB REF 47 Ġand ĠB IB REF 48 . ĠAccording Ġto Ġthis Ġhypothesis , Ġthe Ġmeaning Ġof Ġa Ġword Ġcan Ġbe Ġrecovered Ġempir ically Ġfrom Ġthe Ġcontexts Ġin Ġwhich Ġit Ġoccurs Ġwithin Ġa Ġcollection Ġof Ġtexts . ĠSince Ġboth Ġpairs Ġof Ġtop ically Ġrelated Ġwords Ġand Ġpairs Ġof Ġpurely Ġsimilar Ġwords Ġtend Ġto Ġappear Ġin Ġthe Ġsame Ġcontexts , Ġtheir Ġassociated Ġmeaning Ġconf ounds Ġthe Ġtwo Ġdistinct Ġrelations ĠB IB REF 14 , ĠB IB REF 49 , ĠB IB REF 50 . ĠAs Ġa Ġresult , Ġdistribution al Ġmethods Ġobscure Ġa Ġcrucial Ġfacet Ġof Ġlex ical Ġmeaning . ĠThis Ġlimitation Ġalso Ġreflects Ġonto Ġword Ġembed d ings Ġ( WE s ), Ġrepresentations Ġof Ġwords Ġas Ġlow - dimensional Ġvectors Ġthat Ġhave Ġbecome Ġindispensable Ġfor Ġa Ġwide Ġrange Ġof ĠN LP Ġapplications ĠB IB REF 51 , ĠB IB REF 52 , ĠB IB REF 53 . ĠIn Ġparticular , Ġit Ġinvolves Ġboth Ġstatic ĠWE s Ġlearned Ġfrom Ġco - occ urrence Ġpatterns ĠB IB REF 32 , ĠB IB REF 54 , ĠB IB REF 31 Ġand Ġcontextual ized ĠWE s Ġlearned Ġfrom Ġmodeling Ġword Ġsequences ĠB IB REF 55 , ĠB IB REF 29 . ĠAs Ġa Ġresult , Ġin Ġthe Ġinduced Ġrepresentations , Ġge omet rical Ġclos eness Ġ( me asured Ġe . g . Ġthrough Ġcos ine Ġdistance ) Ġconf l ates Ġgenuine Ġsimilarity Ġwith Ġbroad Ġrelated ness . ĠFor Ġinstance , Ġthe Ġvectors Ġfor Ġan ton y ms Ġsuch Ġas Ġsober Ġand Ġdrunk , Ġby Ġdefinition Ġdiss imilar , Ġmight Ġbe Ġneighbors Ġin Ġthe Ġsemantic Ġspace Ġunder Ġthe Ġdistribution al Ġhypothesis . ĠB IB REF 36 , ĠB IB REF 56 , Ġand ĠB IB REF 53 Ġdemonstrated Ġthat Ġdifferent Ġchoices Ġof Ġhyper - param eters Ġin ĠWE Ġalgorithms Ġ( such Ġas Ġcontext Ġwindow ) Ġemphasize Ġdifferent Ġrelations Ġin Ġthe Ġresulting Ġrepresentations . ĠLikewise , ĠB IB REF 57 Ġand ĠB IB REF 54 Ġdiscovered Ġthat ĠWE s Ġlearned Ġfrom Ġtexts Ġannot ated Ġwith Ġsynt actic Ġinformation Ġmirror Ġsimilarity Ġbetter Ġthan Ġsimple Ġlocal Ġbag - of - words Ġneighborhoods . </s> The Ġfailure Ġof ĠWE s Ġto Ġcapture Ġsemantic Ġsimilarity , Ġin Ġturn , Ġaffects Ġmodel Ġperformance Ġin Ġseveral ĠN LP Ġapplications Ġwhere Ġsuch Ġknowledge Ġis Ġcrucial . ĠIn Ġparticular , ĠNatural ĠLanguage ĠUnderstanding Ġtasks Ġsuch Ġas Ġstatistical Ġdialog Ġmodeling , Ġtext Ġsimpl ification , Ġor Ġsemantic Ġtext Ġsimilarity ĠB IB REF 58 , ĠB IB REF 18 , ĠB IB REF 59 , Ġamong Ġothers , Ġsuffer Ġthe Ġmost . ĠAs Ġa Ġconsequence , Ġresources Ġproviding Ġclean Ġinformation Ġon Ġsemantic Ġsimilarity Ġare Ġkey Ġin Ġmitigating Ġthe Ġside Ġeffects Ġof Ġthe Ġdistribution al Ġsignal . ĠIn Ġparticular , Ġsuch Ġdatabases Ġcan Ġbe Ġemployed Ġfor Ġthe Ġintrinsic Ġevaluations Ġof Ġspecific ĠWE Ġmodels Ġas Ġa Ġproxy Ġof Ġtheir Ġreliability Ġfor Ġdownstream Ġapplications ĠB IB REF 60 , ĠB IB REF 61 , ĠB IB REF 14 ; Ġintuitive ly , Ġthe Ġmore ĠWE s Ġare Ġmis aligned Ġwith Ġhuman Ġjudgments Ġof Ġsimilarity , Ġthe Ġmore Ġtheir Ġperformance Ġon Ġactual Ġtasks Ġis Ġexpected Ġto Ġbe Ġdegraded . ĠMoreover , Ġword Ġrepresentations Ġcan Ġbe Ġspecialized Ġ( a . k . a . Ġretro fitted ) Ġby Ġdis ent angling Ġword Ġrelations Ġof Ġsimilarity Ġand Ġassociation . ĠIn Ġparticular , Ġlinguistic Ġconstraints Ġsourced Ġfrom Ġexternal Ġdatabases Ġ( such Ġas Ġsyn onyms Ġfrom ĠWord Net ) Ġcan Ġbe Ġinjected Ġinto ĠWE s ĠB IB REF 62 , ĠB IB REF 63 , ĠB IB REF 16 , ĠB IB REF 22 , ĠB IB REF 64 Ġin Ġorder Ġto Ġenforce Ġa Ġparticular Ġrelation Ġin Ġa Ġdistribution al Ġsemantic Ġspace Ġwhile Ġpreserving Ġthe Ġoriginal Ġadj ac ency Ġproperties . </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġand ĠLanguage ĠVari ation : ĠSem antic ĠTyp ology </s> In Ġthis Ġwork , Ġwe Ġtackle Ġthe Ġconcept Ġof Ġ( true ) Ġsemantic Ġsimilarity Ġfrom Ġa Ġmult ilingual Ġperspective . ĠWhile Ġthe Ġsame Ġmeaning Ġrepresentations Ġmay Ġbe Ġshared Ġby Ġall Ġhuman Ġspeakers Ġat Ġa Ġdeep Ġcognitive Ġlevel , Ġthere Ġis Ġno Ġone - to - one Ġmapping Ġbetween Ġthe Ġwords Ġin Ġthe Ġlex icons Ġof Ġdifferent Ġlanguages . ĠThis Ġmakes Ġthe Ġcomparison Ġof Ġsimilarity Ġjudgments Ġacross Ġlanguages Ġdifficult , Ġsince Ġthe Ġmeaning Ġoverlap Ġof Ġtranslation ally Ġequivalent Ġwords Ġis Ġsometimes Ġfar Ġless Ġthan Ġexact . ĠThis Ġresults Ġfrom Ġthe Ġfact Ġthat Ġthe Ġway Ġlanguages Ġ` part ition ' Ġsemantic Ġfields Ġis Ġpartially Ġarbitrary ĠB IB REF 65 , Ġalthough Ġconstrained Ġcross - ling ually Ġby Ġcommon Ġcognitive Ġbiases ĠB IB REF 66 . ĠFor Ġinstance , Ġconsider Ġthe Ġfield Ġof Ġcolors : ĠEnglish Ġdistinguishes Ġbetween Ġgreen Ġand Ġblue , Ġwhereas ĠMur le Ġ( South ĠSudan ) Ġhas Ġa Ġsingle Ġword Ġfor Ġboth ĠB IB REF 67 . </s> In Ġgeneral , Ġsemantic Ġtyp ology Ġstudies Ġthe Ġvariation Ġin Ġlex ical Ġsemantics Ġacross Ġthe Ġworld 's Ġlanguages . ĠAccording Ġto ĠB IB REF 68 , Ġthe Ġways Ġlanguages Ġcategor ize Ġconcepts Ġinto Ġthe Ġlex icon Ġfollow Ġthree Ġmain Ġaxes : Ġ1 ) Ġgran ularity : Ġwhat Ġis Ġthe Ġnumber Ġof Ġcategories Ġin Ġa Ġspecific Ġdomain ? ; Ġ2 ) Ġboundary Ġlocation : Ġwhere Ġdo Ġthe Ġlines Ġmarking Ġdifferent Ġcategories Ġlie ? ; Ġ3 ) Ġgrouping Ġand Ġdis section : Ġwhat Ġare Ġthe Ġmembership Ġcriteria Ġof Ġa Ġcategory ; Ġwhich Ġinstances Ġare Ġconsidered Ġto Ġbe Ġmore Ġprototyp ical ? ĠDifferent Ġchoices Ġwith Ġrespect Ġto Ġthese Ġaxes Ġlead Ġto Ġdifferent Ġlex ical ization Ġpatterns . ĠFor Ġinstance , Ġdistinct Ġsenses Ġin Ġa Ġpoly sem ous Ġword Ġin ĠEnglish , Ġsuch Ġas Ġskin Ġ( re fer ring Ġto Ġboth Ġthe Ġbody Ġand Ġfruit ), Ġmay Ġbe Ġassigned Ġseparate Ġwords Ġin Ġother Ġlanguages Ġsuch Ġas ĠItalian Ġp elle Ġand Ġbu cc ia , Ġrespectively ĠB IB REF 70 . ĠWe Ġlater Ġanalyze Ġwhether Ġsimilarity Ġscores Ġobtained Ġfrom Ġnative Ġspeakers Ġalso Ġloosely Ġfollow Ġthe Ġpatterns Ġdescribed Ġby Ġsemantic Ġtyp ology . </s> Previous ĠWork Ġand ĠEvaluation ĠData </s> Word ĠPair ĠDat as ets . ĠRich Ġexpert - created Ġresources Ġsuch Ġas ĠWord Net ĠB IB REF 46 , ĠB IB REF 71 , ĠVerb Net ĠB IB REF 72 , ĠB IB REF 73 , Ġor ĠFrame Net ĠB IB REF 74 Ġencode Ġa Ġwealth Ġof Ġsemantic Ġand Ġsynt actic Ġinformation , Ġbut Ġare Ġexpensive Ġand Ġtime - consuming Ġto Ġcreate . ĠThe Ġscale Ġof Ġthis Ġproblem Ġgets Ġmultiplied Ġby Ġthe Ġnumber Ġof Ġlanguages Ġin Ġconsideration . ĠTherefore , Ġcrowd - s ourcing Ġwith Ġnon - ex pert Ġannot ators Ġhas Ġbeen Ġadopted Ġas Ġa Ġquicker Ġalternative Ġto Ġproduce Ġsmaller Ġand Ġmore Ġfocused Ġsemantic Ġresources Ġand Ġevaluation Ġbenchmarks . ĠThis Ġalternative Ġpractice Ġhas Ġhad Ġa Ġprofound Ġimpact Ġon Ġdistribution al Ġsemantics Ġand Ġrepresentation Ġlearning ĠB IB REF 14 . ĠWhile Ġsome Ġprominent ĠEnglish Ġword Ġpair Ġdatasets Ġsuch Ġas ĠWord Sim - 353 ĠB IB REF 75 , ĠMEN ĠB IB REF 76 , Ġor ĠStanford ĠRare ĠWords ĠB IB REF 77 Ġdid Ġnot Ġdiscriminate Ġbetween Ġsimilarity Ġand Ġrelated ness , Ġthe Ġimportance Ġof Ġthis Ġdistinction Ġwas Ġestablished Ġby ĠB IB REF 14 Ġthrough Ġthe Ġcreation Ġof ĠSim Lex - 999 . ĠThis Ġinspired Ġother Ġsimilar Ġdatasets Ġwhich Ġfocused Ġon Ġdifferent Ġlex ical Ġproperties . ĠFor Ġinstance , ĠSim Ver b - 35 00 ĠB IB REF 15 Ġprovided Ġsimilarity Ġratings Ġfor Ġ3 , 500 ĠEnglish Ġverbs , Ġwhereas ĠCARD - 660 ĠB IB REF 78 Ġaimed Ġat Ġmeasuring Ġthe Ġsemantic Ġsimilarity Ġof Ġinf requent Ġconcepts . </s> Sem antic ĠSimilar ity ĠDat as ets Ġin ĠOther ĠLanguages . ĠMot ivated Ġby Ġthe Ġimpact Ġof Ġdatasets Ġsuch Ġas ĠSim Lex - 999 Ġand ĠSim Ver b - 35 00 Ġon Ġrepresentation Ġlearning Ġin ĠEnglish , Ġa Ġline Ġof Ġrelated Ġwork Ġfocused Ġon Ġcreating Ġsimilar Ġresources Ġin Ġother Ġlanguages . ĠThe Ġdominant Ġapproach Ġis Ġtranslating Ġand Ġre annot ating Ġthe Ġentire Ġoriginal ĠEnglish ĠSim Lex - 999 Ġdataset , Ġas Ġdone Ġpreviously Ġfor ĠGerman , ĠItalian , Ġand ĠRussian ĠB IB REF 25 , ĠHebrew Ġand ĠCroatian ĠB IB REF 16 , Ġand ĠPolish ĠB IB REF 79 . ĠVen ek os ki : 2017 n od al ida Ġapply Ġthis Ġprocess Ġonly Ġto Ġa Ġsubset Ġof Ġ300 Ġconcept Ġpairs Ġfrom Ġthe ĠEnglish ĠSim Lex - 999 . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 27 Ġsampled Ġa Ġnew Ġset Ġof Ġ500 ĠEnglish Ġconcept Ġpairs Ġto Ġensure Ġwider Ġtopical Ġcoverage Ġand Ġbalance Ġacross Ġsimilarity Ġspect ra , Ġand Ġthen Ġtranslated Ġthose Ġpairs Ġto ĠGerman , ĠItalian , ĠSpanish , Ġand ĠF ars i Ġ( S EM EV AL - 500 ). ĠA Ġsimilar Ġapproach Ġwas Ġfollowed Ġby ĠB IB REF 26 Ġfor ĠTurkish , Ġby ĠB IB REF 80 Ġfor ĠMandarin ĠChinese , Ġand Ġby ĠB IB REF 81 Ġfor ĠJapanese . ĠB IB REF 82 Ġtranslated Ġthe Ġconc aten ation Ġof ĠSim Lex - 999 , ĠWord Sim - 353 , Ġand Ġthe ĠEnglish ĠSEM EV AL - 500 Ġinto ĠThai Ġand Ġthen Ġre annot ated Ġit . ĠFinally , ĠB IB REF 83 Ġtranslated ĠEnglish ĠSim Lex - 999 Ġand ĠWord Sim - 353 Ġto Ġ11 Ġresource - rich Ġtarget Ġlanguages Ġ( German , ĠFrench , ĠRussian , ĠItalian , ĠDutch , ĠChinese , ĠPortuguese , ĠSwedish , ĠSpanish , ĠArabic , ĠF ars i ), Ġbut Ġthey Ġdid Ġnot Ġprovide Ġdetails Ġconcerning Ġthe Ġtranslation Ġprocess Ġand Ġthe Ġresolution Ġof Ġtranslation Ġdisagreements . ĠMore Ġimportantly , Ġthey Ġalso Ġdid Ġnot Ġre annot ate Ġthe Ġtranslated Ġpairs Ġin Ġthe Ġtarget Ġlanguages . ĠAs Ġwe Ġdiscussed Ġin ĠÂ§ ĠSEC REF 6 Ġand Ġreiterate Ġlater Ġin ĠÂ§ SEC REF 5 , Ġsemantic Ġdifferences Ġamong Ġlanguages Ġcan Ġhave Ġa Ġprofound Ġimpact Ġon Ġthe Ġannotation Ġscores ; Ġpartic ul ary , Ġwe Ġshow Ġin ĠÂ§ SEC REF 25 Ġthat Ġthese Ġdifferences Ġeven Ġroughly Ġdefine Ġlanguage Ġclusters Ġbased Ġon Ġlanguage Ġaffinity . </s> A Ġcore Ġissue Ġwith Ġthe Ġcurrent Ġdatasets Ġconcerns Ġa Ġlack Ġof Ġone Ġunified Ġprocedure Ġthat Ġensures Ġthe Ġcompar ability Ġof Ġresources Ġin Ġdifferent Ġlanguages . ĠFurther , Ġconcept Ġpairs Ġfor Ġdifferent Ġlanguages Ġare Ġsourced Ġfrom Ġdifferent Ġcorpor a Ġ( e . g ., Ġdirect Ġtranslation Ġof Ġthe ĠEnglish Ġdata Ġversus Ġsampling Ġfrom Ġscratch Ġin Ġthe Ġtarget Ġlanguage ). ĠMoreover , Ġthe Ġprevious ĠSim Lex - based Ġmult ilingual Ġdatasets Ġinherit Ġthe Ġmain Ġdeficiencies Ġof Ġthe ĠEnglish Ġoriginal Ġversion , Ġsuch Ġas Ġthe Ġfocus Ġon Ġnoun s Ġand Ġhighly Ġfrequent Ġconcepts . ĠFinally , Ġprior Ġwork Ġmostly Ġfocused Ġon Ġlanguages Ġthat Ġare Ġwidely Ġspoken Ġand Ġdo Ġnot Ġaccount Ġfor Ġthe Ġvariety Ġof Ġthe Ġworld 's Ġlanguages . ĠOur Ġlong - term Ġgoal Ġis Ġdev ising Ġa Ġstandardized Ġmethodology Ġto Ġextend Ġthe Ġcoverage Ġalso Ġto Ġlanguages Ġthat Ġare Ġresource - lean Ġand / or Ġtyp ologically Ġdiverse Ġ( e . g ., ĠWelsh , ĠKis w ah ili Ġas Ġin Ġthis Ġwork ). </s> Mult ilingual ĠDat as ets Ġfor ĠNatural ĠLanguage ĠUnderstanding . ĠThe ĠMulti - Sim Lex Ġinitiative Ġand Ġcorresponding Ġdatasets Ġare Ġalso Ġaligned Ġwith Ġthe Ġrecent Ġefforts Ġon Ġproc uring Ġmult ilingual Ġbenchmarks Ġthat Ġcan Ġhelp Ġadvance Ġcomputational Ġmodeling Ġof Ġnatural Ġlanguage Ġunderstanding Ġacross Ġdifferent Ġlanguages . ĠFor Ġinstance , Ġpret rained Ġmult ilingual Ġlanguage Ġmodels Ġsuch Ġas Ġmult ilingual Ġb ert ĠB IB REF 29 Ġor Ġx l m ĠB IB REF 30 Ġare Ġtypically Ġprob ed Ġon ĠX N LI Ġtest Ġdata ĠB IB REF 84 Ġfor Ġcross - ling ual Ġnatural Ġlanguage Ġinference . ĠX N LI Ġwas Ġcreated Ġby Ġtranslating Ġexamples Ġfrom Ġthe ĠEnglish ĠMulti N LI Ġdataset , Ġand Ġprojecting Ġits Ġsentence Ġlabels ĠB IB REF 85 . ĠOther Ġrecent Ġmult ilingual Ġdatasets Ġtarget Ġthe Ġtask Ġof Ġquestion Ġanswering Ġbased Ġon Ġreading Ġcomprehension : Ġi ) ĠML Q A ĠB IB REF 86 Ġincludes Ġ7 Ġlanguages Ġii ) ĠX Qu AD ĠB IB REF 87 Ġ10 Ġlanguages ; Ġiii ) ĠTy Di Q A ĠB IB REF 88 Ġ9 Ġwidely Ġspoken Ġtyp ologically Ġdiverse Ġlanguages . ĠWhile ĠML Q A Ġand ĠX Qu AD Ġresult Ġfrom Ġthe Ġtranslation Ġfrom Ġan ĠEnglish Ġdataset , ĠTy Di Q A Ġwas Ġbuilt Ġindependently Ġin Ġeach Ġlanguage . ĠAnother Ġmult ilingual Ġdataset , ĠPA WS - X ĠB IB REF 89 , Ġfocused Ġon Ġthe Ġparaph rase Ġidentification Ġtask Ġand Ġwas Ġcreated Ġtranslating Ġthe Ġoriginal ĠEnglish ĠPA WS ĠB IB REF 90 Ġinto Ġ6 Ġlanguages . ĠWe Ġbelieve Ġthat ĠMulti - Sim Lex Ġcan Ġsubstantially Ġcontribute Ġto Ġthis Ġendeavor Ġby Ġoffering Ġa Ġcomprehensive Ġmult ilingual Ġbenchmark Ġfor Ġthe Ġfundamental Ġlex ical Ġlevel Ġrelation Ġof Ġsemantic Ġsimilarity . ĠIn Ġfuture Ġwork , ĠMulti - Sim Lex Ġalso Ġoffers Ġan Ġopportunity Ġto Ġinvestigate Ġthe Ġcorrelations Ġbetween Ġword - level Ġsemantic Ġsimilarity Ġand Ġperformance Ġin Ġdownstream Ġtasks Ġsuch Ġas ĠQ A Ġand ĠNL I Ġacross Ġdifferent Ġlanguages . </s> The ĠBase Ġfor ĠMulti - Sim Lex : ĠExt ending ĠEnglish ĠSim Lex - 999 </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġdesign Ġprinciples Ġbehind Ġthe ĠEnglish Ġ( eng ) ĠMulti - Sim Lex Ġdataset , Ġwhich Ġis Ġthe Ġbasis Ġfor Ġall Ġthe ĠMulti - Sim Lex Ġdatasets Ġin Ġother Ġlanguages , Ġas Ġdetailed Ġin ĠÂ§ SEC REF 5 . ĠWe Ġfirst Ġargue Ġthat Ġa Ġnew , Ġmore Ġbalanced , Ġand Ġmore Ġcomprehensive Ġevaluation Ġresource Ġfor Ġlex ical Ġsemantic Ġsimilarity Ġin ĠEnglish Ġis Ġnecessary . ĠWe Ġthen Ġdescribe Ġhow Ġthe Ġ1 , 888 Ġword Ġpairs Ġcontained Ġin Ġthe Ġeng ĠMulti - Sim Lex Ġwere Ġselected Ġin Ġsuch Ġa Ġway Ġas Ġto Ġrepresent Ġvarious Ġlinguistic Ġphenomena Ġwithin Ġa Ġsingle Ġintegrated Ġresource . </s> Construction ĠCrit eria . ĠThe Ġfollowing Ġcriteria Ġhave Ġto Ġbe Ġsatisfied Ġby Ġany Ġhigh - quality Ġsemantic Ġevaluation Ġresource , Ġas Ġargued Ġby Ġprevious Ġstudies Ġfocused Ġon Ġthe Ġcreation Ġof Ġsuch Ġresources ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 91 , ĠB IB REF 27 : </s> ( C 1 ) ĠRepresentative Ġand Ġdiverse . ĠThe Ġresource Ġmust Ġcover Ġthe Ġfull Ġrange Ġof Ġdiverse Ġconcepts Ġoccurring Ġin Ġnatural Ġlanguage , Ġincluding Ġdifferent Ġword Ġclasses Ġ( e . g ., Ġnoun s , Ġverbs , Ġadject ives , Ġad verbs ), Ġconcrete Ġand Ġabstract Ġconcepts , Ġa Ġvariety Ġof Ġlex ical Ġfields , Ġand Ġdifferent Ġfrequency Ġranges . </s> ( C 2 ) ĠClearly Ġdefined . ĠThe Ġresource Ġmust Ġprovide Ġa Ġclear Ġunderstanding Ġof Ġwhich Ġsemantic Ġrelation Ġexactly Ġis Ġannot ated Ġand Ġmeasured , Ġpossibly Ġcontrasting Ġit Ġwith Ġother Ġrelations . ĠFor Ġinstance , Ġthe Ġoriginal ĠSim Lex - 999 Ġand ĠSim Ver b - 35 00 Ġexplicitly Ġfocus Ġon Ġtrue Ġsemantic Ġsimilarity Ġand Ġdistinguish Ġit Ġfrom Ġbroader Ġrelated ness Ġcaptured Ġby Ġdatasets Ġsuch Ġas ĠMEN ĠB IB REF 76 Ġor ĠWord Sim - 353 ĠB IB REF 75 . </s> ( C 3 ) ĠCons istent Ġand Ġreliable . ĠThe Ġresource Ġmust Ġensure Ġconsistent Ġannotations Ġobtained Ġfrom Ġnon - ex pert Ġnative Ġspeakers Ġfollowing Ġsimple Ġand Ġprecise Ġannotation Ġguidelines . </s> In Ġchoosing Ġthe Ġword Ġpairs Ġand Ġconstructing Ġeng ĠMulti - Sim Lex , Ġwe Ġadhere Ġto Ġthese Ġrequirements . ĠMoreover , Ġwe Ġfollow Ġgood Ġpractices Ġestablished Ġby Ġthe Ġresearch Ġon Ġrelated Ġresources . ĠIn Ġparticular , Ġsince Ġthe Ġintroduction Ġof Ġthe Ġoriginal ĠSim Lex - 999 Ġdataset ĠB IB REF 14 , Ġfollow - up Ġworks Ġhave Ġimproved Ġits Ġconstruction Ġprotocol Ġacross Ġseveral Ġaspects , Ġincluding : Ġ1 ) Ġcoverage Ġof Ġmore Ġlex ical Ġfields , Ġe . g ., Ġby Ġrelying Ġon Ġa Ġdiverse Ġset Ġof ĠWikipedia Ġcategories ĠB IB REF 27 , Ġ2 ) Ġinf requent / ra re Ġwords ĠB IB REF 78 , Ġ3 ) Ġfocus Ġon Ġparticular Ġword Ġclasses , Ġe . g ., Ġverbs ĠB IB REF 15 , Ġ4 ) Ġannotation Ġquality Ġcontrol ĠB IB REF 78 . ĠOur Ġgoal Ġis Ġto Ġmake Ġuse Ġof Ġthese Ġimprovements Ġtowards Ġa Ġlarger , Ġmore Ġrepresentative , Ġand Ġmore Ġreliable Ġlex ical Ġsimilarity Ġdataset Ġin ĠEnglish Ġand , Ġconsequently , Ġin Ġall Ġother Ġlanguages . </s> The ĠFinal ĠOutput : ĠEnglish ĠMulti - Sim Lex . ĠIn Ġorder Ġto Ġensure Ġthat Ġthe Ġcriterion ĠC 1 Ġis Ġsatisfied , Ġwe Ġconsolidate Ġand Ġintegrate Ġthe Ġdata Ġalready Ġcarefully Ġsampled Ġin Ġprior Ġwork Ġinto Ġa Ġsingle , Ġcomprehensive , Ġand Ġrepresentative Ġdataset . ĠThis Ġway , Ġwe Ġcan Ġcontrol Ġfor Ġdiversity , Ġfrequency , Ġand Ġother Ġproperties Ġwhile Ġavoiding Ġto Ġperform Ġthis Ġtime - consuming Ġselection Ġprocess Ġfrom Ġscratch . ĠNote Ġthat , Ġon Ġthe Ġother Ġhand , Ġthe Ġword Ġpairs Ġchosen Ġfor ĠEnglish Ġare Ġscored Ġfrom Ġscratch Ġas Ġpart Ġof Ġthe Ġentire ĠMulti - Sim Lex Ġannotation Ġprocess , Ġintroduced Ġlater Ġin ĠÂ§ SEC REF 5 . ĠWe Ġnow Ġdescribe Ġthe Ġexternal Ġdata Ġsources Ġfor Ġthe Ġfinal Ġset Ġof Ġword Ġpairs : </s> 1 ) ĠSource : ĠSim Lex - 999 . ĠB IB REF 14 . ĠThe ĠEnglish ĠMulti - Sim Lex Ġhas Ġbeen Ġinitially Ġconceived Ġas Ġan Ġextension Ġof Ġthe Ġoriginal ĠSim Lex - 999 Ġdataset . ĠTherefore , Ġwe Ġinclude Ġall Ġ999 Ġword Ġpairs Ġfrom ĠSim Lex , Ġwhich Ġspan Ġ666 Ġnoun Ġpairs , Ġ222 Ġverb Ġpairs , Ġand Ġ111 Ġadjective Ġpairs . ĠWhile ĠSim Lex - 999 Ġalready Ġprovides Ġexamples Ġrepresenting Ġdifferent ĠPOS Ġclasses , Ġit Ġdoes Ġnot Ġhave Ġa Ġsufficient Ġcoverage Ġof Ġdifferent Ġlinguistic Ġphenomena : Ġfor Ġinstance , Ġit Ġcontains Ġonly Ġvery Ġfrequent Ġconcepts , Ġand Ġit Ġdoes Ġnot Ġprovide Ġa Ġrepresentative Ġset Ġof Ġverbs ĠB IB REF 15 . </s> 2 ) ĠSource : ĠSem E val - 17 : ĠTask Ġ2 ĠB IB REF 92 . ĠWe Ġstart Ġfrom Ġthe Ġfull Ġdataset Ġof Ġ500 Ġconcept Ġpairs Ġto Ġextract Ġa Ġtotal Ġof Ġ334 Ġconcept Ġpairs Ġfor ĠEnglish ĠMulti - Sim Lex Ġa ) Ġwhich Ġcontain Ġonly Ġsingle - word Ġconcepts , Ġb ) Ġwhich Ġare Ġnot Ġnamed Ġentities , Ġc ) Ġwhere ĠPOS Ġtags Ġof Ġthe Ġtwo Ġconcepts Ġare Ġthe Ġsame , Ġd ) Ġwhere Ġboth Ġconcepts Ġoccur Ġin Ġthe Ġtop Ġ250 K Ġmost Ġfrequent Ġword Ġtypes Ġin Ġthe ĠEnglish ĠWikipedia , Ġand Ġe ) Ġdo Ġnot Ġalready Ġoccur Ġin ĠSim Lex - 999 . ĠThe Ġoriginal Ġconcepts Ġwere Ġsampled Ġas Ġto Ġspan Ġall Ġthe Ġ34 Ġdomains Ġavailable Ġas Ġpart Ġof ĠBabel Dom ains ĠB IB REF 93 , Ġwhich Ġroughly Ġcorrespond Ġto Ġthe Ġmain Ġhigh - level ĠWikipedia Ġcategories . ĠThis Ġensures Ġtopical Ġdiversity Ġin Ġour Ġsub - sample . </s> 3 ) ĠSource : ĠCARD - 660 ĠB IB REF 78 . Ġ67 Ġword Ġpairs Ġare Ġtaken Ġfrom Ġthis Ġdataset Ġfocused Ġon Ġrare Ġword Ġsimilarity , Ġapplying Ġthe Ġsame Ġselection Ġcriteria Ġa ) Ġto Ġe ) Ġemployed Ġfor ĠSEM EV AL - 500 . ĠWords Ġare Ġcontrolled Ġfor Ġfrequency Ġbased Ġon Ġtheir Ġoccurrence Ġcounts Ġfrom Ġthe ĠGoogle ĠNews Ġdata Ġand Ġthe Ġu k Wa C Ġcorpus ĠB IB REF 94 . ĠCARD - 660 Ġcontains Ġsome Ġwords Ġthat Ġare Ġvery Ġrare Ġ( log boat ), Ġdomain - specific Ġ( ery th role ukemia ) Ġand Ġslang Ġ( 2 m rw ), Ġwhich Ġmight Ġbe Ġdifficult Ġto Ġtranslate Ġand Ġannot ate Ġacross Ġa Ġwide Ġarray Ġof Ġlanguages . ĠHence , Ġwe Ġopt Ġfor Ġretaining Ġonly Ġthe Ġconcept Ġpairs Ġabove Ġthe Ġthreshold Ġof Ġtop Ġ250 K Ġmost Ġfrequent ĠWikipedia Ġconcepts , Ġas Ġabove . </s> 4 ) ĠSource : ĠSim Ver b - 35 00 ĠB IB REF 15 ĠSince Ġboth ĠCARD - 660 Ġand ĠSEM EV AL - 500 Ġare Ġheavily Ġskewed Ġtowards Ġnoun Ġpairs , Ġand Ġnoun s Ġalso Ġdominate Ġthe Ġoriginal ĠSim Lex - 999 , Ġwe Ġalso Ġextract Ġadditional Ġverb Ġpairs Ġfrom Ġthe Ġverb - specific Ġsimilarity Ġdataset ĠSim Ver b - 35 00 . ĠWe Ġrandomly Ġsample Ġ244 Ġverb Ġpairs Ġfrom ĠSim Ver b - 35 00 Ġthat Ġrepresent Ġall Ġsimilarity Ġspect ra . ĠIn Ġparticular , Ġwe Ġadd Ġ61 Ġverb Ġpairs Ġfor Ġeach Ġof Ġthe Ġsimilarity Ġintervals : Ġ$ [ 0 , 1 . 5 ), Ġ[ 1 . 5 , 3 ), Ġ[ 3 , 4 . 5 ), Ġ[ 4 . 5 , Ġ6 ] $. ĠSince Ġverbs Ġin ĠSim Ver b - 35 00 Ġwere Ġoriginally Ġchosen Ġfrom ĠVerb Net ĠB IB REF 95 , ĠB IB REF 73 , Ġthey Ġcover Ġa Ġwide Ġrange Ġof Ġverb Ġclasses Ġand Ġtheir Ġrelated Ġlinguistic Ġphenomena . </s> 5 ) ĠSource : ĠUniversity Ġof ĠSouth ĠFlorida ĠB IB REF 96 Ġnorms , Ġthe Ġlargest Ġdatabase Ġof Ġfree Ġassociation Ġfor ĠEnglish . ĠIn Ġorder Ġto Ġimprove Ġthe Ġrepresentation Ġof Ġdifferent ĠPOS Ġclasses , Ġwe Ġsample Ġadditional Ġadject ives Ġand Ġad verbs Ġfrom Ġthe ĠUS F Ġnorms Ġfollowing Ġthe Ġprocedure Ġestablished Ġby ĠB IB REF 14 , ĠB IB REF 15 . ĠThis Ġyields Ġadditional Ġ122 Ġadjective Ġpairs , Ġbut Ġonly Ġa Ġlimited Ġnumber Ġof Ġad verb Ġpairs Ġ( e . g ., Ġlater ĠâĢĵ Ġnever , Ġnow ĠâĢĵ Ġhere , Ġonce ĠâĢĵ Ġtwice ). ĠTherefore , Ġwe Ġalso Ġcreate Ġa Ġset Ġof Ġad verb Ġpairs Ġsemi - aut om atically Ġby Ġsampling Ġadject ives Ġthat Ġcan Ġbe Ġderiv ationally Ġtransformed Ġinto Ġad verbs Ġ( e . g . Ġadding Ġthe Ġsuffix Ġ- ly ) Ġfrom Ġthe ĠUS F , Ġand Ġassessing Ġthe Ġcorrectness Ġof Ġsuch Ġderiv ation Ġin ĠWord Net . ĠThe Ġresulting Ġpairs Ġinclude , Ġfor Ġinstance , Ġprimarily ĠâĢĵ Ġmainly , Ġsoftly ĠâĢĵ Ġfirmly , Ġroughly ĠâĢĵ Ġreliably , Ġetc . ĠWe Ġinclude Ġa Ġtotal Ġof Ġ123 Ġad verb Ġpairs Ġinto Ġthe Ġfinal ĠEnglish ĠMulti - Sim Lex . ĠNote Ġthat Ġthis Ġis Ġthe Ġfirst Ġtime Ġad verbs Ġare Ġincluded Ġinto Ġany Ġsemantic Ġsimilarity Ġdataset . </s> F ulf ill ment Ġof ĠConstruction ĠCrit eria . ĠThe Ġfinal Ġeng ĠMulti - Sim Lex Ġdataset Ġspans Ġ1 , 05 1 Ġnoun Ġpairs , Ġ4 69 Ġverb Ġpairs , Ġ245 Ġadjective Ġpairs , Ġand Ġ123 Ġad verb Ġpairs . ĠAs Ġmentioned Ġabove , Ġthe Ġcriterion ĠC 1 Ġhas Ġbeen Ġfulfilled Ġby Ġrelying Ġonly Ġon Ġword Ġpairs Ġthat Ġalready Ġunderwent Ġmeticulous Ġsampling Ġprocesses Ġin Ġprior Ġwork , Ġintegrating Ġthem Ġinto Ġa Ġsingle Ġresource . ĠAs Ġa Ġconsequence , ĠMulti - Sim Lex Ġallows Ġfor Ġfine - gr ained Ġanalyses Ġover Ġdifferent ĠPOS Ġclasses , Ġconc ret eness Ġlevels , Ġsimilarity Ġspect ra , Ġfrequency Ġintervals , Ġrelation Ġtypes , Ġmorphology , Ġlex ical Ġfields , Ġand Ġit Ġalso Ġincludes Ġsome Ġchallenging Ġorth ographically Ġsimilar Ġexamples Ġ( e . g ., Ġinfection ĠâĢĵ Ġinf lection ). ĠWe Ġensure Ġthat Ġthe Ġcriteria ĠC 2 Ġand ĠC 3 Ġare Ġsatisfied Ġby Ġusing Ġsimilar Ġannotation Ġguidelines Ġas ĠSim lex - 999 , ĠSim Ver b - 35 00 , Ġand ĠSEM EV AL - 500 Ġthat Ġexplicitly Ġtarget Ġsemantic Ġsimilarity . ĠIn Ġwhat Ġfollows , Ġwe Ġoutline Ġthe Ġcarefully Ġtailored Ġprocess Ġof Ġtranslating Ġand Ġannot ating ĠMulti - Sim Lex Ġdatasets Ġin Ġall Ġtarget Ġlanguages . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation </s> We Ġnow Ġdetail Ġthe Ġdevelopment Ġof Ġthe Ġfinal ĠMulti - Sim Lex Ġresource , Ġdescribing Ġour Ġlanguage Ġselection Ġprocess , Ġas Ġwell Ġas Ġtranslation Ġand Ġannotation Ġof Ġthe Ġresource , Ġincluding Ġthe Ġsteps Ġtaken Ġto Ġensure Ġand Ġmeasure Ġthe Ġquality Ġof Ġthis Ġresource . ĠWe Ġalso Ġprovide Ġkey Ġdata Ġstatistics Ġand Ġpreliminary Ġcross - ling ual Ġcomparative Ġanalyses . </s> Language ĠSelection . ĠMulti - Sim Lex Ġcomprises Ġeleven Ġlanguages Ġin Ġaddition Ġto ĠEnglish . ĠThe Ġmain Ġobjective Ġfor Ġour Ġinclusion Ġcriteria Ġhas Ġbeen Ġto Ġbalance Ġlanguage Ġprominence Ġ( by Ġnumber Ġof Ġspeakers Ġof Ġthe Ġlanguage ) Ġfor Ġmaximum Ġimpact Ġof Ġthe Ġresource , Ġwhile Ġsimultaneously Ġhaving Ġa Ġdiverse Ġsuite Ġof Ġlanguages Ġbased Ġon Ġtheir Ġtyp ological Ġfeatures Ġ( such Ġas Ġmorph ological Ġtype Ġand Ġlanguage Ġfamily ). ĠTable ĠT AB REF 10 Ġsummarizes Ġkey Ġinformation Ġabout Ġthe Ġlanguages Ġcurrently Ġincluded Ġin ĠMulti - Sim Lex . ĠWe Ġhave Ġincluded Ġa Ġmixture Ġof Ġf usional , Ġagg l ut in ative , Ġisol ating , Ġand Ġintro flex ive Ġlanguages Ġthat Ġcome Ġfrom Ġeight Ġdifferent Ġlanguage Ġfamilies . ĠThis Ġincludes Ġlanguages Ġthat Ġare Ġvery Ġwidely Ġused Ġsuch Ġas ĠChinese ĠMandarin Ġand ĠSpanish , Ġand Ġlow - resource Ġlanguages Ġsuch Ġas ĠWelsh Ġand ĠKis w ah ili . ĠWe Ġhope Ġto Ġfurther Ġinclude Ġadditional Ġlanguages Ġand Ġinspire Ġother Ġresearchers Ġto Ġcontribute Ġto Ġthe Ġeffort Ġover Ġthe Ġlifetime Ġof Ġthis Ġproject . </s> The Ġwork Ġon Ġdata Ġcollection Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġcrucial Ġphases : Ġ1 ) Ġa Ġtranslation Ġphase Ġwhere Ġthe Ġextended ĠEnglish Ġlanguage Ġdataset Ġwith Ġ1 , 888 Ġpairs Ġ( described Ġin ĠÂ§ SEC REF 4 ) Ġis Ġtranslated Ġinto Ġeleven Ġtarget Ġlanguages , Ġand Ġ2 ) Ġan Ġannotation Ġphase Ġwhere Ġhuman Ġrat ers Ġscored Ġeach Ġpair Ġin Ġthe Ġtranslated Ġset Ġas Ġwell Ġas Ġthe ĠEnglish Ġset . ĠDet ailed Ġguidelines Ġfor Ġboth Ġphases Ġare Ġavailable Ġonline Ġat : Ġhttps :// mult is im lex . com . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠWord ĠPair ĠTranslation </s> Trans l ators Ġfor Ġeach Ġtarget Ġlanguage Ġwere Ġinstructed Ġto Ġfind Ġdirect Ġor Ġapproximate Ġtranslations Ġfor Ġthe Ġ1 , 888 Ġword Ġpairs Ġthat Ġsatisfy Ġthe Ġfollowing Ġrules . Ġ( 1 ) ĠAll Ġpairs Ġin Ġthe Ġtranslated Ġset Ġmust Ġbe Ġunique Ġ( i . e ., Ġno Ġduplicate Ġpairs ); Ġ( 2 ) ĠTrans l ating Ġtwo Ġwords Ġfrom Ġthe Ġsame ĠEnglish Ġpair Ġinto Ġthe Ġsame Ġword Ġin Ġthe Ġtarget Ġlanguage Ġis Ġnot Ġallowed Ġ( e . g ., Ġit Ġis Ġnot Ġallowed Ġto Ġtranslate Ġcar Ġand Ġautomobile Ġto Ġthe Ġsame ĠSpanish Ġword Ġcoc he ). Ġ( 3 ) ĠThe Ġtranslated Ġpairs Ġmust Ġpreserve Ġthe Ġsemantic Ġrelations Ġbetween Ġthe Ġtwo Ġwords Ġwhen Ġpossible . ĠThis Ġmeans Ġthat , Ġwhen Ġmultiple Ġtranslations Ġare Ġpossible , Ġthe Ġtranslation Ġthat Ġbest Ġconve ys Ġthe Ġsemantic Ġrelation Ġbetween Ġthe Ġtwo Ġwords Ġfound Ġin Ġthe Ġoriginal ĠEnglish Ġpair Ġis Ġselected . Ġ( 4 ) ĠIf Ġit Ġis Ġnot Ġpossible Ġto Ġuse Ġa Ġsingle - word Ġtranslation Ġin Ġthe Ġtarget Ġlanguage , Ġthen Ġa Ġmulti - word Ġexpression Ġ( M WE ) Ġcan Ġbe Ġused Ġto Ġconvey Ġthe Ġnearest Ġpossible Ġsemantics Ġgiven Ġthe Ġabove Ġpoints Ġ( e . g ., Ġthe ĠEnglish Ġword Ġhomework Ġis Ġtranslated Ġinto Ġthe ĠPolish ĠM WE Ġpr aca Ġdom owa ). </s> S atisf ying Ġthe Ġabove Ġrules Ġwhen Ġfinding Ġappropriate Ġtranslations Ġfor Ġeach Ġpair âĢĵ while Ġkeeping Ġto Ġthe Ġspirit Ġof Ġthe Ġintended Ġsemantic Ġrelation Ġin Ġthe ĠEnglish Ġversion âĢĵ is Ġnot Ġalways Ġstraightforward . ĠFor Ġinstance , Ġkins hip Ġterminology Ġin ĠSin itic Ġlanguages Ġ( Mand arin Ġand ĠYue ) Ġuses Ġdifferent Ġterms Ġdepending Ġon Ġwhether Ġthe Ġfamily Ġmember Ġis Ġolder Ġor Ġyounger , Ġand Ġwhether Ġthe Ġfamily Ġmember Ġcomes Ġfrom Ġthe Ġmother âĢ Ļ s Ġside Ġor Ġthe Ġfather âĢ Ļ s Ġside . ĠUTF 8 g bs n ĠIn ĠMandarin , Ġbrother Ġhas Ġno Ġdirect Ġtranslation Ġand Ġcan Ġbe Ġtranslated Ġas Ġeither : Ġå ĵ ¥ å ĵ ¥ Ġ( older Ġbrother ) Ġor Ġå ¼ Ł å¼ Ł Ġ( young er Ġbrother ). ĠTherefore , Ġin Ġsuch Ġcases , Ġthe Ġtransl ators Ġare Ġasked Ġto Ġchoose Ġthe Ġbest Ġoption Ġgiven Ġthe Ġsemantic Ġcontext Ġ( relation ) Ġexpressed Ġby Ġthe Ġpair Ġin ĠEnglish , Ġotherwise Ġselect Ġone Ġof Ġthe Ġtranslations Ġarbitrarily . ĠThis Ġis Ġalso Ġused Ġto Ġremove Ġduplicate Ġpairs Ġin Ġthe Ġtranslated Ġset , Ġby Ġdifferent iating Ġthe Ġdupl icates Ġusing Ġa Ġvariant Ġat Ġeach Ġinstance . ĠFurther , Ġmany Ġtranslation Ġinstances Ġwere Ġresolved Ġusing Ġnear - syn onymous Ġterms Ġin Ġthe Ġtranslation . ĠFor Ġexample , Ġthe Ġwords Ġin Ġthe Ġpair : Ġwood ĠâĢĵ Ġtimber Ġcan Ġonly Ġbe Ġdirectly Ġtranslated Ġin ĠEston ian Ġto Ġp uit , Ġand Ġare Ġnot Ġdistinguish able . ĠTherefore , Ġthe Ġtransl ators Ġapprox imated Ġthe Ġtranslation Ġfor Ġtimber Ġto Ġthe Ġcompound Ġnoun Ġp uit m ater j al Ġ( literally : Ġwood Ġmaterial ) Ġin Ġorder Ġto Ġproduce Ġa Ġvalid Ġpair Ġin Ġthe Ġtarget Ġlanguage . ĠIn Ġsome Ġcases , Ġa Ġdirect Ġtransl iter ation Ġfrom ĠEnglish Ġis Ġused . ĠFor Ġexample , Ġthe Ġpair : Ġphysician Ġand Ġdoctor Ġboth Ġtranslate Ġto Ġthe Ġsame Ġword Ġin ĠEston ian Ġ( ar st ); Ġthe Ġless Ġformal Ġword Ġdo k tor Ġis Ġused Ġas Ġa Ġtranslation Ġof Ġdoctor Ġto Ġgenerate Ġa Ġvalid Ġpair . </s> We Ġmeasure Ġthe Ġquality Ġof Ġthe Ġtranslated Ġpairs Ġby Ġusing Ġa Ġrandom Ġsample Ġset Ġof Ġ100 Ġpairs Ġ( from Ġthe Ġ1 , 888 Ġpairs ) Ġto Ġbe Ġtranslated Ġby Ġan Ġindependent Ġtranslator Ġfor Ġeach Ġtarget Ġlanguage . ĠThe Ġsample Ġis Ġproportion ally Ġstrat ified Ġaccording Ġto Ġthe Ġpart - of - speech Ġcategories . ĠThe Ġindependent Ġtranslator Ġis Ġgiven Ġidentical Ġinstructions Ġto Ġthe Ġmain Ġtranslator ; Ġwe Ġthen Ġmeasure Ġthe Ġpercentage Ġof Ġmatched Ġtranslated Ġwords Ġbetween Ġthe Ġtwo Ġtranslations Ġof Ġthe Ġsample Ġset . ĠTable ĠT AB REF 12 Ġsummarizes Ġthe Ġinter - trans lator Ġagreement Ġresults Ġfor Ġall Ġlanguages Ġand Ġby Ġpart - of - speech Ġsubs ets . ĠOverall Ġacross Ġall Ġlanguages , Ġthe Ġagreement Ġis Ġ84 . 8 %, Ġwhich Ġis Ġsimilar Ġto Ġprior Ġwork ĠB IB REF 27 , ĠB IB REF 97 . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠGuidelines Ġand ĠWord ĠPair ĠSc oring </s> Across Ġall Ġlanguages , Ġ145 Ġhuman Ġannot ators Ġwere Ġasked Ġto Ġscore Ġall Ġ1 , 888 Ġpairs Ġ( in Ġtheir Ġgiven Ġlanguage ). ĠWe Ġfinally Ġcollect Ġat Ġleast Ġten Ġvalid Ġannotations Ġfor Ġeach Ġword Ġpair Ġin Ġeach Ġlanguage . ĠAll Ġannot ators Ġwere Ġrequired Ġto Ġabide Ġby Ġthe Ġfollowing Ġinstructions : </s> 1 . ĠEach Ġannot ator Ġmust Ġassign Ġan Ġinteger Ġscore Ġbetween Ġ0 Ġand Ġ6 Ġ( in clusive ) Ġindicating Ġhow Ġsem antically Ġsimilar Ġthe Ġtwo Ġwords Ġin Ġa Ġgiven Ġpair Ġare . ĠA Ġscore Ġof Ġ6 Ġindicates Ġvery Ġhigh Ġsimilarity Ġ( i . e ., Ġperfect Ġsyn ony my ), Ġwhile Ġzero Ġindicates Ġno Ġsimilarity . </s> 2 . ĠEach Ġannot ator Ġmust Ġscore Ġthe Ġentire Ġset Ġof Ġ1 , 888 Ġpairs Ġin Ġthe Ġdataset . ĠThe Ġpairs Ġmust Ġnot Ġbe Ġshared Ġbetween Ġdifferent Ġannot ators . </s> 3 . ĠAn not ators Ġare Ġable Ġto Ġbreak Ġthe Ġworkload Ġover Ġa Ġperiod Ġof Ġapproximately Ġ2 - 3 Ġweeks , Ġand Ġare Ġable Ġto Ġuse Ġexternal Ġsources Ġ( e . g . Ġdiction aries , Ġthe s auri , ĠWord Net ) Ġif Ġrequired . </s> 4 . ĠAn not ators Ġare Ġkept Ġanonymous , Ġand Ġare Ġnot Ġable Ġto Ġcommunicate Ġwith Ġeach Ġother Ġduring Ġthe Ġannotation Ġprocess . </s> The Ġselection Ġcriteria Ġfor Ġthe Ġannot ators Ġrequired Ġthat Ġall Ġannot ators Ġmust Ġbe Ġnative Ġspeakers Ġof Ġthe Ġtarget Ġlanguage . ĠPre ference Ġto Ġannot ators Ġwith Ġuniversity Ġeducation Ġwas Ġgiven , Ġbut Ġnot Ġrequired . ĠAn not ators Ġwere Ġasked Ġto Ġcomplete Ġa Ġspreadsheet Ġcontaining Ġthe Ġtranslated Ġpairs Ġof Ġwords , Ġas Ġwell Ġas Ġthe Ġpart - of - speech , Ġand Ġa Ġcolumn Ġto Ġenter Ġthe Ġscore . ĠThe Ġannot ators Ġdid Ġnot Ġhave Ġaccess Ġto Ġthe Ġoriginal Ġpairs Ġin ĠEnglish . </s> To Ġensure Ġthe Ġquality Ġof Ġthe Ġcollected Ġratings , Ġwe Ġhave Ġemployed Ġan Ġadjud ication Ġprotocol Ġsimilar Ġto Ġthe Ġone Ġproposed Ġand Ġvalidated Ġby ĠP ile h var : 2018 em nl p . ĠIt Ġconsists Ġof Ġthe Ġfollowing Ġthree Ġrounds : </s> Round Ġ1 : ĠAll Ġannot ators Ġare Ġasked Ġto Ġfollow Ġthe Ġinstructions Ġoutlined Ġabove , Ġand Ġto Ġrate Ġall Ġ1 , 888 Ġpairs Ġwith Ġinteger Ġscores Ġbetween Ġ0 Ġand Ġ6 . </s> Round Ġ2 : ĠWe Ġcompare Ġthe Ġscores Ġof Ġall Ġannot ators Ġand Ġidentify Ġthe Ġpairs Ġfor Ġeach Ġannot ator Ġthat Ġhave Ġshown Ġthe Ġmost Ġdisagreement . ĠWe Ġask Ġthe Ġannot ators Ġto Ġreconsider Ġthe Ġassigned Ġscores Ġfor Ġthose Ġpairs Ġonly . ĠThe Ġannot ators Ġmay Ġchose Ġto Ġeither Ġchange Ġor Ġkeep Ġthe Ġscores . ĠAs Ġin Ġthe Ġcase Ġwith ĠRound Ġ1 , Ġthe Ġannot ators Ġhave Ġno Ġaccess Ġto Ġthe Ġscores Ġof Ġthe Ġother Ġannot ators , Ġand Ġthe Ġprocess Ġis Ġanonymous . ĠThis Ġprocess Ġgives Ġa Ġchance Ġfor Ġannot ators Ġto Ġcorrect Ġerrors Ġor Ġreconsider Ġtheir Ġjudgments , Ġand Ġhas Ġbeen Ġshown Ġto Ġbe Ġvery Ġeffective Ġin Ġreaching Ġconsensus , Ġas Ġreported Ġby ĠB IB REF 78 . ĠWe Ġused Ġa Ġvery Ġsimilar Ġprocedure Ġas ĠB IB REF 78 Ġto Ġidentify Ġthe Ġpairs Ġwith Ġthe Ġmost Ġdisagreement ; Ġfor Ġeach Ġannot ator , Ġwe Ġmarked Ġthe Ġ$ i $ th Ġpair Ġif Ġthe Ġrated Ġscore Ġ$ s _ i $ Ġfalls Ġwithin : Ġ$ s _ i Ġ\ ge Ġ\ mu Ġ_ i Ġ+ Ġ1 . 5 $ Ġor Ġ$ s _ i Ġ\ le Ġ\ mu Ġ_ i Ġ- Ġ1 . 5 $, Ġwhere Ġ$\ mu Ġ_ i $ Ġis Ġthe Ġmean Ġof Ġthe Ġother Ġannot ators ' Ġscores . </s> Round Ġ3 : ĠWe Ġcompute Ġthe Ġaverage Ġagreement Ġfor Ġeach Ġannot ator Ġ( with Ġthe Ġother Ġannot ators ), Ġby Ġmeasuring Ġthe Ġaverage ĠSpear man 's Ġcorrelation Ġagainst Ġall Ġother Ġannot ators . ĠWe Ġdiscard Ġthe Ġscores Ġof Ġannot ators Ġthat Ġhave Ġshown Ġthe Ġleast Ġaverage Ġagreement Ġwith Ġall Ġother Ġannot ators , Ġwhile Ġwe Ġmaintain Ġat Ġleast Ġten Ġannot ators Ġper Ġlanguage Ġby Ġthe Ġend Ġof Ġthis Ġround . ĠThe Ġactual Ġprocess Ġis Ġdone Ġin Ġmultiple Ġiterations : Ġ( S 1 ) Ġwe Ġmeasure Ġthe Ġaverage Ġagreement Ġfor Ġeach Ġannot ator Ġwith Ġevery Ġother Ġannot ator Ġ( this Ġcorresponds Ġto Ġthe ĠAP IA A Ġmeasure , Ġsee Ġlater ); Ġ( S 2 ) Ġif Ġwe Ġstill Ġhave Ġmore Ġthan Ġ10 Ġvalid Ġannot ators Ġand Ġthe Ġlowest Ġaverage Ġscore Ġis Ġhigher Ġthan Ġin Ġthe Ġprevious Ġiteration , Ġwe Ġremove Ġthe Ġlowest Ġone , Ġand Ġre run ĠS 1 . ĠTable ĠT AB REF 14 Ġshows Ġthe Ġnumber Ġof Ġannot ators Ġat Ġboth Ġthe Ġstart Ġ( Round Ġ1 ) Ġand Ġend Ġ( Round Ġ3 ) Ġof Ġour Ġprocess Ġfor Ġeach Ġlanguage . </s> We Ġmeasure Ġthe Ġagreement Ġbetween Ġannot ators Ġusing Ġtwo Ġmetrics , Ġaverage Ġpair wise Ġinter - annot ator Ġagreement Ġ( AP IA A ), Ġand Ġaverage Ġmean Ġinter - annot ator Ġagreement Ġ( AM IA A ). ĠBoth Ġof Ġthese Ġuse ĠSpear man 's Ġcorrelation Ġ($ \ r ho Ġ$ ) Ġbetween Ġannot ators Ġscores , Ġthe Ġonly Ġdifference Ġis Ġhow Ġthey Ġare Ġaveraged . ĠThey Ġare Ġcomputed Ġas Ġfollows : </s> where Ġ$\ r ho Ġ( s _ i , s _ j ) $ Ġis Ġthe ĠSpear man 's Ġcorrelation Ġbetween Ġannot ators Ġ$ i $ Ġand Ġ$ j $ ' s Ġscores Ġ($ s _ i $, $ s _ j $ ) Ġfor Ġall Ġpairs Ġin Ġthe Ġdataset , Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġannot ators . ĠAP IA A Ġhas Ġbeen Ġused Ġwidely Ġas Ġthe Ġstandard Ġmeasure Ġfor Ġinter - annot ator Ġagreement , Ġincluding Ġin Ġthe Ġoriginal ĠSim Lex Ġpaper ĠB IB REF 14 . ĠIt Ġsimply Ġaverages Ġthe Ġpair wise ĠSpear man 's Ġcorrelation Ġbetween Ġall Ġannot ators . ĠOn Ġthe Ġother Ġhand , ĠAM IA A Ġcompares Ġthe Ġaverage ĠSpear man 's Ġcorrelation Ġof Ġone Ġheld - out Ġannot ator Ġwith Ġthe Ġaverage Ġof Ġall Ġthe Ġother Ġ$ N - 1 $ Ġannot ators , Ġand Ġthen Ġaverages Ġacross Ġall Ġ$ N $ Ġ` held - out ' Ġannot ators . ĠIt Ġsmooth s Ġindividual Ġannot ator Ġeffects Ġand Ġarguably Ġserves Ġas Ġa Ġbetter Ġupper Ġbound Ġthan ĠAP IA A ĠB IB REF 15 , ĠB IB REF 91 , ĠB IB REF 78 . </s> We Ġpresent Ġthe Ġrespective ĠAP IA A Ġand ĠAM IA A Ġscores Ġin ĠTable ĠT AB REF 16 Ġand ĠTable ĠT AB REF 17 Ġfor Ġall Ġpart - of - speech Ġsubs ets , Ġas Ġwell Ġas Ġthe Ġagreement Ġfor Ġthe Ġfull Ġdatasets . ĠAs Ġreported Ġin Ġprior Ġwork ĠB IB REF 15 , ĠB IB REF 91 , ĠAM IA A Ġscores Ġare Ġtypically Ġhigher Ġthan ĠAP IA A Ġscores . ĠCru cially , Ġthe Ġresults Ġindicate Ġ` strong Ġagreement ' Ġ( ac ross Ġall Ġlanguages ) Ġusing Ġboth Ġmeasurements . ĠThe Ġlanguages Ġwith Ġthe Ġhighest Ġannot ator Ġagreement Ġwere ĠFrench Ġ( f ra ) Ġand ĠYue ĠChinese Ġ( y ue ), Ġwhile ĠRussian Ġ( rus ) Ġhad Ġthe Ġlowest Ġoverall ĠI AA Ġscores . ĠThese Ġscores , Ġhowever , Ġare Ġstill Ġconsidered Ġto Ġbe Ġ` mod er ately Ġstrong Ġagreement '. </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠData ĠAnalysis </s> Similar ity ĠScore ĠDistribut ions . ĠAcross Ġall Ġlanguages , Ġthe Ġaverage Ġscore Ġ( mean Ġ$ = 1 . 61 $, Ġmedian $ = 1 . 1 $ ) Ġis Ġon Ġthe Ġlower Ġside Ġof Ġthe Ġsimilarity Ġscale . ĠHowever , Ġlooking Ġcloser Ġat Ġthe Ġscores Ġof Ġeach Ġlanguage Ġin ĠTable ĠT AB REF 19 , Ġwe Ġindicate Ġnotable Ġdifferences Ġin Ġboth Ġthe Ġaverages Ġand Ġthe Ġspread Ġof Ġscores . ĠNotably , ĠFrench Ġhas Ġthe Ġhighest Ġaverage Ġof Ġsimilarity Ġscores Ġ( mean $ = 2 . 61 $, Ġmedian $ = 2 . 5 $ ), Ġwhile ĠKis w ah ili Ġhas Ġthe Ġlowest Ġaverage Ġ( mean $ = 1 . 28 $, Ġmedian $ = 0 . 5 $ ). ĠRussian Ġhas Ġthe Ġlowest Ġspread Ġ($ \ s igma Ġ= 1 . 37 $ ), Ġwhile ĠPolish Ġhas Ġthe Ġlargest Ġ($ \ s igma Ġ= 1 . 62 $ ). ĠAll Ġof Ġthe Ġlanguages Ġare Ġstrongly Ġcorrelated Ġwith Ġeach Ġother , Ġas Ġshown Ġin ĠFigure ĠFIG REF 20 , Ġwhere Ġall Ġof Ġthe ĠSpear man 's Ġcorrelation Ġcoefficients Ġare Ġgreater Ġthan Ġ0 . 6 Ġfor Ġall Ġlanguage Ġpairs . ĠLanguages Ġthat Ġshare Ġthe Ġsame Ġlanguage Ġfamily Ġare Ġhighly Ġcorrelated Ġ( e . g , Ġc mn - y ue , Ġr us - pol , Ġest - fin ). ĠIn Ġaddition , Ġwe Ġobserve Ġhigh Ġcorrelations Ġbetween ĠEnglish Ġand Ġmost Ġother Ġlanguages , Ġas Ġexpected . ĠThis Ġis Ġdue Ġto Ġthe Ġeffect Ġof Ġusing ĠEnglish Ġas Ġthe Ġbase / anch or Ġlanguage Ġto Ġcreate Ġthe Ġdataset . ĠIn Ġsimple Ġwords , Ġif Ġone Ġtranslates Ġto Ġtwo Ġlanguages Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġstarting Ġfrom Ġthe Ġsame Ġset Ġof Ġpairs Ġin ĠEnglish , Ġit Ġis Ġhig ly Ġlikely Ġthat Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġwill Ġdiver ge Ġfrom ĠEnglish Ġin Ġdifferent Ġways . ĠTherefore , Ġthe Ġsimilarity Ġbetween Ġ$ L _ 1 $ - eng Ġand Ġ$ L _ 2 $ - eng Ġis Ġexpected Ġto Ġbe Ġhigher Ġthan Ġbetween Ġ$ L _ 1 $ -$ L _ 2 $, Ġespecially Ġif Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġare Ġtyp ologically Ġdiss imilar Ġlanguages Ġ( e . g ., Ġhe b - c mn , Ġsee ĠFigure ĠFIG REF 20 ). ĠThis Ġphenomenon Ġis Ġwell Ġdocumented Ġin Ġrelated Ġprior Ġwork ĠB IB REF 25 , ĠB IB REF 27 , ĠB IB REF 16 , ĠB IB REF 97 . ĠWhile Ġwe Ġacknowledge Ġthis Ġas Ġa Ġslight Ġartifact Ġof Ġthe Ġdataset Ġdesign , Ġit Ġwould Ġotherwise Ġbe Ġimpossible Ġto Ġconstruct Ġa Ġsem antically Ġaligned Ġand Ġcomprehensive Ġdataset Ġacross Ġa Ġlarge Ġnumber Ġof Ġlanguages . </s> We Ġalso Ġreport Ġdifferences Ġin Ġthe Ġdistribution Ġof Ġthe Ġfrequency Ġof Ġwords Ġamong Ġthe Ġlanguages Ġin ĠMulti - Sim Lex . ĠFigure ĠFIG REF 22 Ġshows Ġsix Ġexample Ġlanguages , Ġwhere Ġeach Ġbar Ġsegment Ġshows Ġthe Ġproportion Ġof Ġwords Ġin Ġeach Ġlanguage Ġthat Ġoccur Ġin Ġthe Ġgiven Ġfrequency Ġrange . ĠFor Ġexample , Ġthe Ġ10 K - 20 K Ġsegment Ġof Ġthe Ġbars Ġrepresents Ġthe Ġproportion Ġof Ġwords Ġin Ġthe Ġdataset Ġthat Ġoccur Ġin Ġthe Ġlist Ġof Ġmost Ġfrequent Ġwords Ġbetween Ġthe Ġfrequency Ġrank Ġof Ġ10 , 000 Ġand Ġ20 , 000 Ġin Ġthat Ġlanguage ; Ġlikewise Ġwith Ġother Ġintervals . ĠFrequency Ġlists Ġfor Ġthe Ġpresented Ġlanguages Ġare Ġderived Ġfrom ĠWikipedia Ġand ĠCommon ĠC rawl Ġcorpor a . ĠWhile Ġmany Ġconcept Ġpairs Ġare Ġdirect Ġor Ġapproximate Ġtranslations Ġof ĠEnglish Ġpairs , Ġwe Ġcan Ġsee Ġthat Ġthe Ġfrequency Ġdistribution Ġdoes Ġvary Ġacross Ġdifferent Ġlanguages , Ġand Ġis Ġalso Ġrelated Ġto Ġinherent Ġlanguage Ġproperties . ĠFor Ġinstance , Ġin ĠFinnish Ġand ĠRussian , Ġwhile Ġwe Ġuse Ġinf in itive Ġforms Ġof Ġall Ġverbs , Ġconj ug ated Ġverb Ġinf lections Ġare Ġoften Ġmore Ġfrequent Ġin Ġraw Ġcorpor a Ġthan Ġthe Ġcorresponding Ġinf in itive Ġforms . ĠThe Ġvariance Ġcan Ġalso Ġbe Ġpartially Ġexplained Ġby Ġthe Ġdifference Ġin Ġmon oling ual Ġcorpor a Ġsize Ġused Ġto Ġderive Ġthe Ġfrequency Ġrankings Ġin Ġthe Ġfirst Ġplace : Ġabsolute Ġvocabulary Ġsizes Ġare Ġexpected Ġto Ġfluct uate Ġacross Ġdifferent Ġlanguages . ĠHowever , Ġit Ġis Ġalso Ġimportant Ġto Ġnote Ġthat Ġthe Ġdatasets Ġalso Ġcontain Ġsubs ets Ġof Ġlower - frequency Ġand Ġrare Ġwords , Ġwhich Ġcan Ġbe Ġused Ġfor Ġrare Ġword Ġevaluations Ġin Ġmultiple Ġlanguages , Ġin Ġthe Ġspirit Ġof ĠP ile h var : 2018 em nl p 's ĠEnglish Ġrare Ġword Ġdataset . </s> Cross - L ingu istic ĠDifferences . ĠTable ĠT AB REF 23 Ġshows Ġsome Ġexamples Ġof Ġaverage Ġsimilarity Ġscores Ġof ĠEnglish , ĠSpanish , ĠKis w ah ili Ġand ĠWelsh Ġconcept Ġpairs . ĠRemember Ġthat Ġthe Ġscores Ġrange Ġfrom Ġ0 Ġto Ġ6 : Ġthe Ġhigher Ġthe Ġscore , Ġthe Ġmore Ġsimilar Ġthe Ġparticipants Ġfound Ġthe Ġconcepts Ġin Ġthe Ġpair . ĠThe Ġexamples Ġfrom ĠTable ĠT AB REF 23 Ġshow Ġevidence Ġof Ġboth Ġthe Ġstability Ġof Ġaverage Ġsimilarity Ġscores Ġacross Ġlanguages Ġ( un likely ĠâĢĵ Ġfriendly , Ġbook ĠâĢĵ Ġliterature , Ġand Ġvanish ĠâĢĵ Ġdisappear ), Ġas Ġwell Ġas Ġlanguage - specific Ġdifferences Ġ( care ĠâĢĵ Ġcaution ). ĠSome Ġdifferences Ġin Ġsimilarity Ġscores Ġseem Ġto Ġgroup Ġlanguages Ġinto Ġclusters . ĠFor Ġexample , Ġthe Ġword Ġpair Ġregular ĠâĢĵ Ġaverage Ġhas Ġan Ġaverage Ġsimilarity Ġscore Ġof Ġ4 . 0 Ġand Ġ4 . 1 Ġin ĠEnglish Ġand ĠSpanish , Ġrespectively , Ġwhereas Ġin ĠKis w ah ili Ġand ĠWelsh Ġthe Ġaverage Ġsimilarity Ġscore Ġof Ġthis Ġpair Ġis Ġ0 . 5 Ġand Ġ0 . 8 . ĠWe Ġanalyze Ġthis Ġphenomenon Ġin Ġmore Ġdetail Ġin ĠÂ§ SEC REF 25 . </s> There Ġare Ġalso Ġexamples Ġfor Ġeach Ġof Ġthe Ġfour Ġlanguages Ġhaving Ġa Ġnotably Ġhigher Ġor Ġlower Ġsimilarity Ġscore Ġfor Ġthe Ġsame Ġconcept Ġpair Ġthan Ġthe Ġthree Ġother Ġlanguages . ĠFor Ġexample , Ġlarge ĠâĢĵ Ġbig Ġin ĠEnglish Ġhas Ġan Ġaverage Ġsimilarity Ġscore Ġof Ġ5 . 9 , Ġwhereas ĠSpanish , ĠKis w ah ili Ġand ĠWelsh Ġspeakers Ġrate Ġthe Ġclosest Ġconcept Ġpair Ġin Ġtheir Ġnative Ġlanguage Ġto Ġhave Ġa Ġsimilarity Ġscore Ġbetween Ġ2 . 7 Ġand Ġ3 . 8 . ĠWhat Ġis Ġmore , Ġwoman ĠâĢĵ Ġwife Ġreceives Ġan Ġaverage Ġsimilarity Ġof Ġ0 . 9 Ġin ĠEnglish , Ġ2 . 9 Ġin ĠSpanish , Ġand Ġgreater Ġthan Ġ4 . 0 Ġin ĠKis w ah ili Ġand ĠWelsh . ĠThe Ġexamples Ġfrom ĠSpanish Ġinclude Ġban co ĠâĢĵ Ġas ient o Ġ( bank ĠâĢĵ Ġseat ) Ġwhich Ġreceives Ġan Ġaverage Ġsimilarity Ġscore Ġ5 . 1 , Ġwhile Ġin Ġthe Ġother Ġthree Ġlanguages Ġthe Ġsimilarity Ġscore Ġfor Ġthis Ġword Ġpair Ġdoes Ġnot Ġexceed Ġ0 . 1 . ĠAt Ġthe Ġsame Ġtime , Ġthe Ġaverage Ġsimilarity Ġscore Ġof Ġesp ant os ament e ĠâĢĵ Ġfant Ã¡ stic ament e Ġ( am azing ly ĠâĢĵ Ġfant astically ) Ġis Ġmuch Ġlower Ġin ĠSpanish Ġ( 0 . 4 ) Ġthan Ġin Ġother Ġlanguages Ġ( 4 . 1 ĠâĢĵ Ġ5 . 1 ). ĠIn ĠKis w ah ili , Ġan Ġexample Ġof Ġa Ġword Ġpair Ġwith Ġa Ġhigher Ġsimilarity Ġscore Ġthan Ġthe Ġrest Ġwould Ġbe Ġmach we o ĠâĢĵ Ġj ion i Ġ( sun set ĠâĢĵ Ġevening ), Ġhaving Ġan Ġaverage Ġscore Ġof Ġ5 . 5 , Ġwhile Ġthe Ġother Ġlanguages Ġreceive Ġ2 . 8 Ġor Ġless , Ġand Ġa Ġnotably Ġlower Ġsimilarity Ġscore Ġis Ġgiven Ġto Ġwa Ġa jab u Ġ- Ġmk ub wa Ġs ana Ġ( w onder ful ĠâĢĵ Ġterrific ), Ġgetting Ġ0 . 9 , Ġwhile Ġthe Ġother Ġlanguages Ġreceive Ġ5 . 3 Ġor Ġmore . ĠWelsh Ġexamples Ġinclude Ġy n Ġll w yr Ġ- Ġy n Ġgy fan Ġg w bl Ġ( pure ly ĠâĢĵ Ġcompletely ), Ġwhich Ġscores Ġ5 . 4 Ġamong ĠWelsh Ġspeakers Ġbut Ġ2 . 3 Ġor Ġless Ġin Ġother Ġlanguages , Ġwhile Ġadd o ĠâĢĵ Ġty ng u Ġ( prom ise ĠâĢĵ Ġswear ) Ġis Ġrated Ġas Ġ0 Ġby Ġall ĠWelsh Ġannot ators , Ġbut Ġin Ġthe Ġother Ġthree Ġlanguages Ġ4 . 3 Ġor Ġmore Ġon Ġaverage . </s> There Ġcan Ġbe Ġseveral Ġexplanations Ġfor Ġthe Ġdifferences Ġin Ġsimilarity Ġscores Ġacross Ġlanguages , Ġincluding Ġbut Ġnot Ġlimited Ġto Ġcultural Ġcontext , Ġpoly se my , Ġmet ony my , Ġtranslation , Ġregional Ġand Ġgenerational Ġdifferences , Ġand Ġmost Ġcommonly , Ġthe Ġfact Ġthat Ġwords Ġand Ġmeanings Ġdo Ġnot Ġexactly Ġmap Ġonto Ġeach Ġother Ġacross Ġlanguages . ĠFor Ġexample , Ġit Ġis Ġlikely Ġthat Ġthe Ġother Ġthree Ġlanguages Ġdo Ġnot Ġhave Ġtwo Ġseparate Ġwords Ġfor Ġdescribing Ġthe Ġconcepts Ġin Ġthe Ġconcept Ġpair : Ġbig ĠâĢĵ Ġlarge , Ġand Ġthe Ġtransl ators Ġhad Ġto Ġopt Ġfor Ġsimilar Ġlex ical Ġitems Ġthat Ġwere Ġmore Ġdistant Ġin Ġmeaning , Ġexplaining Ġwhy Ġin ĠEnglish Ġthe Ġconcept Ġpair Ġreceived Ġa Ġmuch Ġhigher Ġaverage Ġsimilarity Ġscore Ġthan Ġin Ġother Ġlanguages . ĠA Ġsimilar Ġissue Ġrelated Ġto Ġthe Ġmapping Ġproblem Ġacross Ġlanguages Ġarose Ġin Ġthe ĠWelsh Ġconcept Ġpair Ġy n Ġll w ye ĠâĢĵ Ġy n Ġgy fan Ġg w bl , Ġwhere ĠWelsh Ġspeakers Ġagreed Ġthat Ġthe Ġtwo Ġconcepts Ġare Ġvery Ġsimilar . ĠWhen Ġasked , Ġbilingual Ġspeakers Ġconsidered Ġthe Ġtwo ĠWelsh Ġconcepts Ġmore Ġsimilar Ġthan ĠEnglish Ġequivalents Ġpurely ĠâĢĵ Ġcompletely , Ġpotentially Ġexplaining Ġwhy Ġa Ġhigher Ġaverage Ġsimilarity Ġscore Ġwas Ġreached Ġin ĠWelsh . ĠThe Ġexample Ġof Ġwoman ĠâĢĵ Ġwife Ġcan Ġillustrate Ġcultural Ġdifferences Ġor Ġanother Ġtranslation - related Ġissue Ġwhere Ġthe Ġword Ġ` wife ' Ġdid Ġnot Ġexist Ġin Ġsome Ġlanguages Ġ( for Ġexample , ĠEston ian ), Ġand Ġtherefore Ġhad Ġto Ġbe Ġdescribed Ġusing Ġother Ġwords , Ġaffecting Ġthe Ġcompar ability Ġof Ġthe Ġsimilarity Ġscores . ĠThis Ġwas Ġalso Ġthe Ġcase Ġwith Ġthe Ġfootball ĠâĢĵ Ġsoccer Ġconcept Ġpair . ĠThe Ġpair Ġbank ĠâĢĵ Ġseat Ġdemonstrates Ġthe Ġeffect Ġof Ġthe Ġpoly se my Ġmismatch Ġacross Ġlanguages : Ġwhile Ġ` bank ' Ġhas Ġtwo Ġdifferent Ġmeanings Ġin ĠEnglish , Ġneither Ġof Ġthem Ġis Ġsimilar Ġto Ġthe Ġword Ġ` seat ', Ġbut Ġin ĠSpanish , Ġ` ban co ' Ġcan Ġmean Ġ` bank ', Ġbut Ġit Ġcan Ġalso Ġmean Ġ` bench '. ĠQuite Ġnaturally , ĠSpanish Ġspeakers Ġgave Ġthe Ġpair Ġban co ĠâĢĵ Ġas ient o Ġa Ġhigher Ġsimilarity Ġscore Ġthan Ġthe Ġspeakers Ġof Ġlanguages Ġwhere Ġthis Ġpoly se my Ġdid Ġnot Ġoccur . </s> An Ġexample Ġof Ġmet ony my Ġaffecting Ġthe Ġaverage Ġsimilarity Ġscore Ġcan Ġbe Ġseen Ġin Ġthe ĠKis w ah ili Ġversion Ġof Ġthe Ġword Ġpair : Ġsunset ĠâĢĵ Ġevening Ġ( m ach we o ĠâĢĵ Ġj ion i ). ĠThe Ġaverage Ġsimilarity Ġscore Ġfor Ġthis Ġpair Ġis Ġmuch Ġhigher Ġin ĠKis w ah ili , Ġlikely Ġbecause Ġthe Ġword Ġ` sun set ' Ġcan Ġact Ġas Ġa Ġmet onym Ġof Ġ` even ing '. ĠThe Ġlow Ġsimilarity Ġscore Ġof Ġwonderful ĠâĢĵ Ġterrific Ġin ĠKis w ah ili Ġ( wa Ġa jab u Ġ- Ġmk ub wa Ġs ana ) Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġwhile Ġ` mk ub wa Ġs ana ' Ġcan Ġbe Ġused Ġas Ġ` ter r ific ' Ġin ĠKis w ah ili , Ġit Ġtechnically Ġmeans Ġ` very Ġbig ', Ġadding Ġto Ġthe Ġexamples Ġof Ġtranslation - Ġand Ġmapping - related Ġeffects . ĠThe Ġword Ġpair Ġamazingly ĠâĢĵ Ġfant astically Ġ( esp ant os ament e ĠâĢĵ Ġfant Ã¡ stic ament e ) Ġbrings Ġout Ġanother Ġtranslation - related Ġproblem : Ġthe Ġaccuracy Ġof Ġthe Ġtranslation . ĠWhile Ġ` esp ant os ament e ' Ġcould Ġarguably Ġbe Ġtranslated Ġto Ġ` am azing ly ', Ġmore Ġcommon Ġmeanings Ġinclude : Ġ` f right fully ', Ġ` ter r ifying ly ', Ġand Ġ` shock ingly ', Ġexplaining Ġwhy Ġthe Ġaverage Ġsimilarity Ġscore Ġdiffers Ġfrom Ġthe Ġrest Ġof Ġthe Ġlanguages . ĠAnother Ġproblem Ġwas Ġbrought Ġout Ġby Ġadd o ĠâĢĵ Ġty ng u Ġ( prom ise ĠâĢĵ Ġswear ) Ġin ĠWelsh , Ġwhere Ġthe Ġ` ty ng u ' Ġmay Ġnot Ġhave Ġbeen Ġa Ġcommonly Ġused Ġor Ġeven Ġa Ġknown Ġword Ġchoice Ġfor Ġannot ators , Ġpointing Ġout Ġpotential Ġregional Ġor Ġgenerational Ġdifferences Ġin Ġlanguage Ġuse . </s> Table ĠT AB REF 24 Ġpresents Ġexamples Ġof Ġconcept Ġpairs Ġfrom ĠEnglish , ĠSpanish , ĠKis w ah ili , Ġand ĠWelsh Ġon Ġwhich Ġthe Ġparticipants Ġagreed Ġthe Ġmost . ĠFor Ġexample , Ġin ĠEnglish Ġall Ġparticipants Ġrated Ġthe Ġsimilarity Ġof Ġtrial ĠâĢĵ Ġtest Ġto Ġbe Ġ4 Ġor Ġ5 . ĠIn ĠSpanish Ġand ĠWelsh , Ġall Ġparticipants Ġrated Ġstart ĠâĢĵ Ġbegin Ġto Ġcorrespond Ġto Ġa Ġscore Ġof Ġ5 Ġor Ġ6 . ĠIn ĠKis w ah ili , Ġmoney ĠâĢĵ Ġcash Ġreceived Ġa Ġsimilarity Ġrating Ġof Ġ6 Ġfrom Ġevery Ġparticipant . ĠWhile Ġthere Ġare Ġnumerous Ġexamples Ġof Ġconcept Ġpairs Ġin Ġthese Ġlanguages Ġwhere Ġthe Ġparticipants Ġagreed Ġon Ġa Ġsimilarity Ġscore Ġof Ġ4 Ġor Ġhigher , Ġit Ġis Ġworth Ġnoting Ġthat Ġnone Ġof Ġthese Ġlanguages Ġhad Ġa Ġsingle Ġpair Ġwhere Ġall Ġparticipants Ġagreed Ġon Ġeither Ġ1 - 2 , Ġ2 - 3 , Ġor Ġ3 - 4 Ġsimilarity Ġrating . ĠInterestingly , Ġin ĠEnglish Ġall Ġpairs Ġwhere Ġall Ġthe Ġparticipants Ġagreed Ġon Ġa Ġ5 - 6 Ġsimilarity Ġscore Ġwere Ġadject ives . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠEffect Ġof ĠLanguage ĠAff inity Ġon ĠSimilar ity ĠScores </s> Based Ġon Ġthe Ġanalysis Ġin ĠFigure ĠFIG REF 20 Ġand Ġinspecting Ġthe Ġanecdotal Ġexamples Ġin Ġthe Ġprevious Ġsection , Ġit Ġis Ġevident Ġthat Ġthe Ġcorrelation Ġbetween Ġsimilarity Ġscores Ġacross Ġlanguages Ġis Ġnot Ġrandom . ĠTo Ġcorrobor ate Ġthis Ġintuition , Ġwe Ġvisualize Ġthe Ġvectors Ġof Ġsimilarity Ġscores Ġfor Ġeach Ġsingle Ġlanguage Ġby Ġreducing Ġtheir Ġdimension ality Ġto Ġ2 Ġvia ĠPrincipal ĠComponent ĠAnalysis ĠB IB REF 98 . ĠThe Ġresulting Ġscatter Ġplot Ġin ĠFigure ĠFIG REF 26 Ġreveals Ġthat Ġlanguages Ġfrom Ġthe Ġsame Ġfamily Ġor Ġbranch Ġhave Ġsimilar Ġpatterns Ġin Ġthe Ġscores . ĠIn Ġparticular , ĠRussian Ġand ĠPolish Ġ( both ĠSlav ic ), ĠFinnish Ġand ĠEston ian Ġ( both ĠU ral ic ), ĠCant ones e Ġand ĠMandarin ĠChinese Ġ( both ĠSin itic ), Ġand ĠSpanish Ġand ĠFrench Ġ( both ĠRomance ) Ġare Ġall Ġneighbors . </s> In Ġorder Ġto Ġquantify Ġexactly Ġthe Ġeffect Ġof Ġlanguage Ġaffinity Ġon Ġthe Ġsimilarity Ġscores , Ġwe Ġrun Ġcorrelation Ġanalyses Ġbetween Ġthese Ġand Ġlanguage Ġfeatures . ĠIn Ġparticular , Ġwe Ġextract Ġfeature Ġvectors Ġfrom ĠURI EL ĠB IB REF 99 , Ġa Ġmassively Ġmult ilingual Ġtyp ological Ġdatabase Ġthat Ġcollects Ġand Ġnormal izes Ġinformation Ġcompiled Ġby Ġgrammar ians Ġand Ġfield Ġlingu ists Ġabout Ġthe Ġworld 's Ġlanguages . ĠIn Ġparticular , Ġwe Ġfocus Ġon Ġinformation Ġabout Ġgeography Ġ( the Ġareas Ġwhere Ġthe Ġlanguage Ġspeakers Ġare Ġconcentrated ), Ġfamily Ġ( the Ġphylogen etic Ġtree Ġeach Ġlanguage Ġbelongs Ġto ), Ġand Ġtyp ology Ġ( including Ġsyntax , Ġphon ological Ġinventory , Ġand Ġphon ology ). ĠMoreover , Ġwe Ġconsider Ġtyp ological Ġrepresentations Ġof Ġlanguages Ġthat Ġare Ġnot Ġmanually Ġcrafted Ġby Ġexperts , Ġbut Ġrather Ġlearned Ġfrom Ġtexts . ĠB IB REF 100 Ġproposed Ġto Ġconstruct Ġsuch Ġrepresentations Ġby Ġtraining Ġlanguage - ident ifying Ġvectors Ġend - to - end Ġas Ġpart Ġof Ġneural Ġmachine Ġtranslation Ġmodels . </s> The Ġvector Ġfor Ġsimilarity Ġjudgments Ġand Ġthe Ġvector Ġof Ġlinguistic Ġfeatures Ġfor Ġa Ġgiven Ġlanguage Ġhave Ġdifferent Ġdimension ality . ĠHence , Ġwe Ġfirst Ġconstruct Ġa Ġdistance Ġmatrix Ġfor Ġeach Ġvector Ġspace , Ġsuch Ġthat Ġboth Ġcolumns Ġand Ġrows Ġare Ġlanguage Ġindices , Ġand Ġeach Ġcell Ġvalue Ġis Ġthe Ġcos ine Ġdistance Ġbetween Ġthe Ġvectors Ġof Ġthe Ġcorresponding Ġlanguage Ġpair . ĠGiven Ġa Ġset Ġof ĠL Ġlanguages , Ġeach Ġresulting Ġmatrix Ġ$ S $ Ġhas Ġdimension ality Ġof Ġ$\ math bb Ġ{ R } ^{ | L | Ġ\ times Ġ| L | } $ Ġand Ġis Ġsymm etrical . ĠTo Ġestimate Ġthe Ġcorrelation Ġbetween Ġthe Ġmatrix Ġfor Ġsimilarity Ġjudgments Ġand Ġeach Ġof Ġthe Ġmat rices Ġfor Ġlinguistic Ġfeatures , Ġwe Ġrun Ġa ĠMant el Ġtest ĠB IB REF 101 , Ġa Ġnon - param etric Ġstatistical Ġtest Ġbased Ġon Ġmatrix Ġperm utations Ġthat Ġtakes Ġinto Ġaccount Ġinter - depend encies Ġamong Ġpair wise Ġdistances . </s> The Ġresults Ġof Ġthe ĠMant el Ġtest Ġreported Ġin ĠTable ĠFIG REF 26 Ġshow Ġthat Ġthere Ġexist Ġstatistically Ġsignificant Ġcorrelations Ġbetween Ġsimilarity Ġjudgments Ġand Ġgeography , Ġfamily , Ġand Ġsyntax , Ġgiven Ġthat Ġ$ p Ġ< Ġ0 . 05 $ Ġand Ġ$ z Ġ> Ġ1 . 96 $. ĠThe Ġcorrelation Ġcoefficient Ġis Ġparticularly Ġstrong Ġfor Ġgeography Ġ($ r Ġ= Ġ0 . 647 $ ) Ġand Ġsyntax Ġ($ r Ġ= Ġ0 . 649 $ ). ĠThe Ġformer Ġresult Ġis Ġintuitive , Ġbecause Ġlanguages Ġin Ġcontact Ġeasily Ġborrow Ġand Ġloan Ġlex ical Ġunits , Ġand Ġcultural Ġinteractions Ġmay Ġresult Ġin Ġsimilar Ġcognitive Ġcategor izations . ĠThe Ġresult Ġfor Ġsyntax , Ġinstead , Ġcannot Ġbe Ġexplained Ġso Ġeasily , Ġas Ġformal Ġproperties Ġof Ġlanguage Ġdo Ġnot Ġaffect Ġlex ical Ġsemantics . ĠInstead , Ġwe Ġconjecture Ġthat , Ġwhile Ġno Ġcausal Ġrelation Ġis Ġpresent , Ġboth Ġsynt actic Ġfeatures Ġand Ġsimilarity Ġjudgments Ġmight Ġbe Ġlinked Ġto Ġa Ġcommon Ġexplanatory Ġvariable Ġ( such Ġas Ġgeography ). ĠIn Ġfact , Ġseveral Ġsynt actic Ġproperties Ġare Ġnot Ġuniformly Ġspread Ġacross Ġthe Ġglobe . ĠFor Ġinstance , Ġverbs Ġwith ĠVerb âĢĵ Object âĢĵ Subject Ġword Ġorder Ġare Ġmostly Ġconcentrated Ġin ĠO ce ania ĠB IB REF 102 . ĠIn Ġturn , Ġgeographical Ġproximity Ġleads Ġto Ġsimilar Ġjudgment Ġpatterns , Ġas Ġmentioned Ġabove . ĠOn Ġthe Ġother Ġhand , Ġwe Ġfind Ġno Ġcorrelation Ġwith Ġphon ology Ġand Ġinventory , Ġas Ġexpected , Ġnor Ġwith Ġthe Ġbottom - up Ġtyp ological Ġfeatures Ġfrom ĠB IB REF 100 . </s> Cross - L ing ual ĠMulti - Sim Lex ĠDat as ets </s> A Ġcrucial Ġadvantage Ġof Ġhaving Ġsem antically Ġaligned Ġmon oling ual Ġdatasets Ġacross Ġdifferent Ġlanguages Ġis Ġthe Ġpotential Ġto Ġcreate Ġcross - ling ual Ġsemantic Ġsimilarity Ġdatasets . ĠSuch Ġdatasets Ġallow Ġfor Ġprobing Ġthe Ġquality Ġof Ġcross - ling ual Ġrepresentation Ġlearning Ġalgorithms ĠB IB REF 27 , ĠB IB REF 103 , ĠB IB REF 104 , ĠB IB REF 105 , ĠB IB REF 106 , ĠB IB REF 30 , ĠB IB REF 107 Ġas Ġan Ġintrinsic Ġevaluation Ġtask . ĠHowever , Ġthe Ġcross - ling ual Ġdatasets Ġprevious Ġwork Ġrelied Ġupon ĠB IB REF 27 Ġwere Ġlimited Ġto Ġa Ġhom ogeneous Ġset Ġof Ġhigh - resource Ġlanguages Ġ( e . g ., ĠEnglish , ĠGerman , ĠItalian , ĠSpanish ) Ġand Ġa Ġsmall Ġnumber Ġof Ġconcept Ġpairs Ġ( all Ġless Ġthan Ġ1 K Ġpairs ). ĠWe Ġaddress Ġboth Ġproblems Ġby Ġ1 ) Ġusing Ġa Ġtyp ologically Ġmore Ġdiverse Ġlanguage Ġsample , Ġand Ġ2 ) Ġrelying Ġon Ġa Ġsubstantially Ġlarger ĠEnglish Ġdataset Ġas Ġa Ġsource Ġfor Ġthe Ġcross - ling ual Ġdatasets : Ġ1 , 888 Ġpairs Ġin Ġthis Ġwork Ġversus Ġ500 Ġpairs Ġin Ġthe Ġwork Ġof ĠCam acho : 2017 se me val . ĠAs Ġa Ġresult , Ġeach Ġof Ġour Ġcross - ling ual Ġdatasets Ġcontains Ġa Ġsubstantially Ġlarger Ġnumber Ġof Ġconcept Ġpairs , Ġas Ġshown Ġin ĠTable ĠT AB REF 30 . ĠThe Ġcross - ling ual ĠMulti - Sim lex Ġdatasets Ġare Ġconstructed Ġautomatically , Ġleveraging Ġword Ġpair Ġtranslations Ġand Ġannotations Ġcollected Ġin Ġall Ġ12 Ġlanguages . ĠThis Ġyields Ġa Ġtotal Ġof Ġ66 Ġcross - ling ual Ġdatasets , Ġone Ġfor Ġeach Ġpossible Ġcombination Ġof Ġlanguages . ĠTable ĠT AB REF 30 Ġprovides Ġthe Ġfinal Ġnumber Ġof Ġconcept Ġpairs , Ġwhich Ġlie Ġbetween Ġ2 , 031 Ġand Ġ3 , 480 Ġpairs Ġfor Ġeach Ġcross - ling ual Ġdataset , Ġwhereas ĠTable ĠT AB REF 29 Ġshows Ġsome Ġsample Ġpairs Ġwith Ġtheir Ġcorresponding Ġsimilarity Ġscores . </s> The Ġautomatic Ġcreation Ġand Ġverification Ġof Ġcross - ling ual Ġdatasets Ġclosely Ġfollows Ġthe Ġprocedure Ġfirst Ġoutlined Ġby ĠCam acho : 2015 acl Ġand Ġlater Ġadopted Ġby ĠCam acho : 2017 se me val Ġ( for Ġsemantic Ġsimilarity ) Ġand ĠVul ic : 2019 acl Ġ( for Ġgraded Ġlex ical Ġentail ment ). ĠFirst , Ġgiven Ġtwo Ġlanguages , Ġwe Ġintersect Ġtheir Ġaligned Ġconcept Ġpairs Ġobtained Ġthrough Ġtranslation . ĠFor Ġinstance , Ġstarting Ġfrom Ġthe Ġaligned Ġpairs Ġatt roup ement ĠâĢĵ Ġfou le Ġin ĠFrench Ġand Ġr ah vas umm ĠâĢĵ Ġr ah v ah ulk Ġin ĠEston ian , Ġwe Ġconstruct Ġtwo Ġcross - ling ual Ġpairs Ġatt roup ement ĠâĢĵ Ġr ah val uk Ġand Ġr ah vas umm ĠâĢĵ Ġfou le . ĠThe Ġscores Ġof Ġcross - ling ual Ġpairs Ġare Ġthen Ġcomputed Ġas Ġaverages Ġof Ġthe Ġtwo Ġcorresponding Ġmon oling ual Ġscores . ĠFinally , Ġin Ġorder Ġto Ġfilter Ġout Ġconcept Ġpairs Ġwhose Ġsemantic Ġmeaning Ġwas Ġnot Ġpreserved Ġduring Ġthis Ġoperation , Ġwe Ġretain Ġonly Ġcross - ling ual Ġpairs Ġfor Ġwhich Ġthe Ġcorresponding Ġmon oling ual Ġscores Ġ$( s _ s , Ġs _ t ) $ Ġdiffer Ġat Ġmost Ġby Ġone Ġfifth Ġof Ġthe Ġfull Ġscale Ġ( i . e ., Ġ$\ mid Ġs _ s Ġ- Ġs _ t Ġ\ mid Ġ\ le Ġ1 . 2 $ ). ĠThis Ġhe uristic Ġmit ig ates Ġthe Ġnoise Ġdue Ġto Ġcross - ling ual Ġsemantic Ġshifts ĠB IB REF 27 , ĠB IB REF 97 . ĠWe Ġrefer Ġthe Ġreader Ġto Ġthe Ġwork Ġof ĠCam acho : 2015 acl Ġfor Ġa Ġdetailed Ġtechnical Ġdescription Ġof Ġthe Ġprocedure . ĠUTF 8 g bs n </s> To Ġassess Ġthe Ġquality Ġof Ġthe Ġresulting Ġcross - ling ual Ġdatasets , Ġwe Ġhave Ġconducted Ġa Ġverification Ġexperiment Ġsimilar Ġto ĠVul ic : 2019 acl . ĠWe Ġrandomly Ġsampled Ġ300 Ġconcept Ġpairs Ġin Ġthe ĠEnglish - Spanish , ĠEnglish - French , Ġand ĠEnglish - Mand arin Ġcross - ling ual Ġdatasets . ĠSub sequently , Ġwe Ġasked Ġbilingual Ġnative Ġspeakers Ġto Ġprovide Ġsimilarity Ġjudgments Ġof Ġeach Ġpair . ĠThe ĠSpear man 's Ġcorrelation Ġscore Ġ$\ r ho Ġ$ Ġbetween Ġautomatically Ġinduced Ġand Ġmanually Ġcollected Ġratings Ġachieves Ġ$\ r ho Ġ\ ge Ġ0 . 90 $ Ġon Ġall Ġsamples , Ġwhich Ġconfirms Ġthe Ġviability Ġof Ġthe Ġautomatic Ġconstruction Ġprocedure . </s> Score Ġand ĠClass ĠDistribut ions . ĠThe Ġsummary Ġof Ġscore Ġand Ġclass Ġdistributions Ġacross Ġall Ġ66 Ġcross - ling ual Ġdatasets Ġare Ġprovided Ġin ĠFigure ĠFIG REF 31 Ġand ĠFigure ĠFIG REF 31 , Ġrespectively . ĠFirst , Ġit Ġis Ġobvious Ġthat Ġthe Ġdistribution Ġover Ġthe Ġfour ĠPOS Ġclasses Ġlargely Ġad heres Ġto Ġthat Ġof Ġthe Ġoriginal Ġmon oling ual ĠMulti - Sim Lex Ġdatasets , Ġand Ġthat Ġthe Ġvariance Ġis Ġquite Ġlow : Ġe . g ., Ġthe Ġeng - f ra Ġdataset Ġcontains Ġthe Ġlowest Ġproportion Ġof Ġnoun s Ġ( 49 . 21 %) Ġand Ġthe Ġhighest Ġproportion Ġof Ġverbs Ġ( 27 . 1 %), Ġadject ives Ġ( 15 . 28 %), Ġand Ġad verbs Ġ( 8 . 41 %). ĠOn Ġthe Ġother Ġhand , Ġthe Ġdistribution Ġover Ġsimilarity Ġintervals Ġin ĠFigure ĠFIG REF 31 Ġshows Ġa Ġmuch Ġgreater Ġvariance . ĠThis Ġis Ġagain Ġexpected Ġas Ġthis Ġpattern Ġresur faces Ġin Ġmon oling ual Ġdatasets Ġ( see ĠTable ĠT AB REF 19 ). ĠIt Ġis Ġalso Ġevident Ġthat Ġthe Ġdata Ġare Ġskewed Ġtowards Ġlower - similar ity Ġconcept Ġpairs . ĠHowever , Ġdue Ġto Ġthe Ġjoint Ġsize Ġof Ġall Ġcross - ling ual Ġdatasets Ġ( see ĠTable ĠT AB REF 30 ), Ġeven Ġthe Ġleast Ġrepresented Ġintervals Ġcontain Ġa Ġsubstantial Ġnumber Ġof Ġconcept Ġpairs . ĠFor Ġinstance , Ġthe Ġr us - y ue Ġdataset Ġcontains Ġthe Ġleast Ġhighly Ġsimilar Ġconcept Ġpairs Ġ( in Ġthe Ġinterval Ġ$ [ 4 , 6 ] $ ) Ġof Ġall Ġ66 Ġcross - ling ual Ġdatasets . ĠNonetheless , Ġthe Ġabsolute Ġnumber Ġof Ġpairs Ġ( 138 ) Ġin Ġthat Ġinterval Ġfor Ġr us - y ue Ġis Ġstill Ġsubstantial . ĠIf Ġneeded , Ġthis Ġmakes Ġit Ġpossible Ġto Ġcreate Ġsmaller Ġdatasets Ġwhich Ġare Ġbalanced Ġacross Ġthe Ġsimilarity Ġspect ra Ġthrough Ġsub - sam pling . </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels </s> After Ġthe Ġnumerical Ġand Ġqualitative Ġanalyses Ġof Ġthe ĠMulti - Sim Lex Ġdatasets Ġprovided Ġin ĠÂ§Â§ ĠSEC REF 18 âĢĵ SEC REF 25 , Ġwe Ġnow Ġbenchmark Ġa Ġseries Ġof Ġrepresentation Ġlearning Ġmodels Ġon Ġthe Ġnew Ġevaluation Ġdata . ĠWe Ġevaluate Ġstandard Ġstatic Ġword Ġembed ding Ġalgorithms Ġsuch Ġas Ġfast Text ĠB IB REF 31 , Ġas Ġwell Ġas Ġa Ġrange Ġof Ġmore Ġrecent Ġtext Ġenc od ers Ġpret rained Ġon Ġlanguage Ġmodeling Ġsuch Ġas Ġmult ilingual ĠB ERT ĠB IB REF 29 . ĠThese Ġexperiments Ġprovide Ġstrong Ġbaseline Ġscores Ġon Ġthe Ġnew ĠMulti - Sim Lex Ġdatasets Ġand Ġoffer Ġa Ġfirst Ġlarge - scale Ġanalysis Ġof Ġpret rained Ġenc od ers Ġon Ġword - level Ġsemantic Ġsimilarity Ġacross Ġdiverse Ġlanguages . ĠIn Ġaddition , Ġthe Ġexperiments Ġnow Ġenabled Ġby ĠMulti - Sim Lex Ġaim Ġto Ġanswer Ġseveral Ġimportant Ġquestions . Ġ( Q 1 ) ĠIs Ġit Ġviable Ġto Ġextract Ġhigh - quality Ġword - level Ġrepresentations Ġfrom Ġpret rained Ġenc od ers Ġreceiving Ġsub word - level Ġtokens Ġas Ġinput ? ĠAre Ġsuch Ġrepresentations Ġcompetitive Ġwith Ġstandard Ġstatic Ġword - level Ġembed d ings ? Ġ( Q 2 ) ĠWhat Ġare Ġthe Ġimplications Ġof Ġmon oling ual Ġpret raining Ġversus Ġ( mass ively ) Ġmult ilingual Ġpret raining Ġfor Ġperformance ? Ġ( Q 3 ) ĠDo Ġlightweight Ġun super vised Ġpost - processing Ġtechniques Ġimprove Ġword Ġrepresentations Ġconsistently Ġacross Ġdifferent Ġlanguages ? Ġ( Q 4 ) ĠCan Ġwe Ġeffectively Ġtransfer Ġavailable Ġexternal Ġlex ical Ġknowledge Ġfrom Ġresource - rich Ġlanguages Ġto Ġresource - lean Ġlanguages Ġin Ġorder Ġto Ġlearn Ġword Ġrepresentations Ġthat Ġdistinguish Ġbetween Ġtrue Ġsimilarity Ġand Ġconceptual Ġrelated ness Ġ( see Ġthe Ġdiscussion Ġin ĠÂ§ SEC REF 6 )? </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels Ġ: :: ĠModels Ġin ĠComparison </s> Static ĠWord ĠEmb edd ings Ġin ĠDifferent ĠLanguages . ĠFirst , Ġwe Ġevaluate Ġa Ġstandard Ġmethod Ġfor Ġinducing Ġnon - context ual ized Ġ( i . e ., Ġstatic ) Ġword Ġembed d ings Ġacross Ġa Ġplethora Ġof Ġdifferent Ġlanguages : Ġfast Text Ġ( ft ) Ġvectors ĠB IB REF 31 Ġare Ġcurrently Ġthe Ġmost Ġpopular Ġand Ġrobust Ġchoice Ġgiven Ġ1 ) Ġthe Ġavailability Ġof Ġpret rained Ġvectors Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages ĠB IB REF 108 Ġtrained Ġon Ġlarge ĠCommon ĠC rawl Ġ( CC ) Ġplus ĠWikipedia Ġ( Wiki ) Ġdata , Ġand Ġ2 ) Ġtheir Ġsuperior Ġperformance Ġacross Ġa Ġrange Ġof ĠN LP Ġtasks ĠB IB REF 109 . ĠIn Ġfact , Ġfast Text Ġis Ġan Ġextension Ġof Ġthe Ġstandard Ġword - level ĠCB OW Ġand Ġskip - gram Ġword 2 vec Ġmodels ĠB IB REF 32 Ġthat Ġtakes Ġinto Ġaccount Ġsub word - level Ġinformation , Ġi . e . Ġthe Ġcont itu ent Ġcharacter Ġn - gram s Ġof Ġeach Ġword ĠB IB REF 110 . ĠFor Ġthis Ġreason , Ġfast Text Ġis Ġalso Ġmore Ġsuited Ġfor Ġmodeling Ġrare Ġwords Ġand Ġmorph ologically Ġrich Ġlanguages . </s> We Ġrely Ġon Ġ300 - dimensional Ġft Ġword Ġvectors Ġtrained Ġon ĠCC + Wiki Ġand Ġavailable Ġonline Ġfor Ġ157 Ġlanguages . ĠThe Ġword Ġvectors Ġfor Ġall Ġlanguages Ġare Ġobtained Ġby ĠCB OW Ġwith Ġposition - weights , Ġwith Ġcharacter Ġn - gram s Ġof Ġlength Ġ5 , Ġa Ġwindow Ġof Ġsize Ġ5 , Ġ10 Ġnegative Ġexamples , Ġand Ġ10 Ġtraining Ġepoch s . ĠWe Ġalso Ġprobe Ġanother Ġ( older ) Ġcollection Ġof Ġft Ġvectors , Ġpret rained Ġon Ġfull ĠWikipedia Ġdumps Ġof Ġeach Ġlanguage .. ĠThe Ġvectors Ġare Ġ300 - dimensional , Ġtrained Ġwith Ġthe Ġskip - gram Ġobjective Ġfor Ġ5 Ġepoch s , Ġwith Ġ5 Ġnegative Ġexamples , Ġa Ġwindow Ġsize Ġset Ġto Ġ5 , Ġand Ġrelying Ġon Ġall Ġcharacter Ġn - gram s Ġfrom Ġlength Ġ3 Ġto Ġ6 . ĠFollowing Ġprior Ġwork , Ġwe Ġtrim Ġthe Ġvoc ab ular ies Ġfor Ġall Ġlanguages Ġto Ġthe Ġ200 K Ġmost Ġfrequent Ġwords Ġand Ġcompute Ġrepresentations Ġfor Ġmulti - word Ġexpressions Ġby Ġaveraging Ġthe Ġvectors Ġof Ġtheir Ġconstituent Ġwords . </s> Un super vised ĠPost - Process ing . ĠFurther , Ġwe Ġconsider Ġa Ġvariety Ġof Ġun super vised Ġpost - processing Ġsteps Ġthat Ġcan Ġbe Ġapplied Ġpost - training Ġon Ġtop Ġof Ġany Ġpret rained Ġinput Ġword Ġembed ding Ġspace Ġwithout Ġany Ġexternal Ġlex ical Ġsemantic Ġresource . ĠSo Ġfar , Ġthe Ġusefulness Ġof Ġsuch Ġmethods Ġhas Ġbeen Ġverified Ġonly Ġon Ġthe ĠEnglish Ġlanguage Ġthrough Ġbenchmarks Ġfor Ġlex ical Ġsemantics Ġand Ġsentence - level Ġtasks ĠB IB REF 113 . ĠIn Ġthis Ġpaper , Ġwe Ġassess Ġif Ġun super vised Ġpost - processing Ġis Ġbeneficial Ġalso Ġin Ġother Ġlanguages . ĠTo Ġthis Ġend , Ġwe Ġapply Ġthe Ġfollowing Ġpost - h oc Ġtransformations Ġon Ġthe Ġinitial Ġword Ġembed d ings : </s> 1 ) ĠMean Ġcent ering Ġ( mc ) Ġis Ġapplied Ġafter Ġunit Ġlength Ġnormal ization Ġto Ġensure Ġthat Ġall Ġvectors Ġhave Ġa Ġzero Ġmean , Ġand Ġis Ġcommonly Ġapplied Ġin Ġdata Ġmining Ġand Ġanalysis ĠB IB REF 114 , ĠB IB REF 115 . </s> 2 ) ĠAll - but - the - top Ġ( ab tt ) ĠB IB REF 113 , ĠB IB REF 116 Ġeliminates Ġthe Ġcommon Ġmean Ġvector Ġand Ġa Ġfew Ġtop Ġdominating Ġdirections Ġ( according Ġto ĠPC A ) Ġfrom Ġthe Ġinput Ġdistribution al Ġword Ġvectors , Ġsince Ġthey Ġdo Ġnot Ġcontribute Ġtowards Ġdistinguishing Ġthe Ġactual Ġsemantic Ġmeaning Ġof Ġdifferent Ġwords . ĠThe Ġmethod Ġcontains Ġa Ġsingle Ġ( tun able ) Ġhyper - param eter Ġ$ dd _{ A } $ Ġwhich Ġdenotes Ġthe Ġnumber Ġof Ġthe Ġdominating Ġdirections Ġto Ġremove Ġfrom Ġthe Ġinitial Ġrepresentations . ĠPrevious Ġwork Ġhas Ġverified Ġthe Ġusefulness Ġof Ġab tt Ġin Ġseveral ĠEnglish Ġlex ical Ġsemantic Ġtasks Ġsuch Ġas Ġsemantic Ġsimilarity , Ġword Ġanalog ies , Ġand Ġconcept Ġcategor ization , Ġas Ġwell Ġas Ġin Ġsentence - level Ġtext Ġclassification Ġtasks ĠB IB REF 113 . </s> 3 ) Ġunc ove c ĠB IB REF 117 Ġadjusts Ġthe Ġsimilarity Ġorder Ġof Ġan Ġarbitrary Ġinput Ġword Ġembed ding Ġspace , Ġand Ġcan Ġemphasize Ġeither Ġsynt actic Ġor Ġsemantic Ġinformation Ġin Ġthe Ġtransformed Ġvectors . ĠIn Ġshort , Ġit Ġtransforms Ġthe Ġinput Ġspace Ġ$\ math bf Ġ{ X } $ Ġinto Ġan Ġadjusted Ġspace Ġ$\ math bf Ġ{ X }\ math bf Ġ{ W } _{ \ alpha Ġ} $ Ġthrough Ġa Ġlinear Ġmap Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} $ Ġcontrolled Ġby Ġa Ġsingle Ġhyper - param eter Ġ$\ alpha Ġ$ . ĠThe Ġ$ n ^ {\ text { th }} $ - order Ġsimilarity Ġtransformation Ġof Ġthe Ġinput Ġword Ġvector Ġspace Ġ$\ math bf Ġ{ X } $ Ġ( for Ġwhich Ġ$ n = 1 $ ) Ġcan Ġbe Ġobtained Ġas Ġ$\ math bf Ġ{ M } _{ n } (\ math bf Ġ{ X }) Ġ= Ġ\ math bf Ġ{ M } _ 1 (\ math bf Ġ{ X }\ math bf Ġ{ W } _{ ( n - 1 )/ 2 }) $, Ġwith Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} = \ math bf Ġ{ Q }\ math bf Ġ{\ Gam ma Ġ} ^ {\ alpha Ġ} $, Ġwhere Ġ$\ math bf Ġ{ Q } $ Ġand Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġare Ġthe Ġmat rices Ġobtained Ġvia Ġe ig end ec om position Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } = \ math bf Ġ{ Q }\ math bf Ġ{\ Gam ma Ġ} \ math bf Ġ{ Q } ^ T $. Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġis Ġa Ġdiagonal Ġmatrix Ġcontaining Ġe igen values Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } $ ; Ġ$\ math bf Ġ{ Q } $ Ġis Ġan Ġorth og onal Ġmatrix Ġwith Ġe igen ve ctors Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } $ Ġas Ġcolumns . ĠWhile Ġthe Ġmotivation Ġfor Ġthe Ġunc ove c Ġmethods Ġdoes Ġoriginate Ġfrom Ġadjusting Ġdiscrete Ġsimilarity Ġorders , Ġnote Ġthat Ġ$\ alpha Ġ$ Ġis Ġin Ġfact Ġa Ġcontinuous Ġreal - valued Ġhyper - param eter Ġwhich Ġcan Ġbe Ġcarefully Ġtuned . ĠFor Ġmore Ġtechnical Ġdetails Ġwe Ġrefer Ġthe Ġreader Ġto Ġthe Ġoriginal Ġwork Ġof ĠB IB REF 117 . </s> As Ġmentioned , Ġall Ġpost - processing Ġmethods Ġcan Ġbe Ġseen Ġas Ġun super vised Ġretro fitting Ġmethods Ġthat , Ġgiven Ġan Ġarbitrary Ġinput Ġvector Ġspace Ġ$\ math bf Ġ{ X } $, Ġproduce Ġa Ġpert urbed / trans formed Ġoutput Ġvector Ġspace Ġ$\ math bf Ġ{ X } ^ {\ prime Ġ} $, Ġbut Ġunlike Ġcommon Ġretro fitting Ġmethods ĠB IB REF 62 , ĠB IB REF 16 , Ġthe Ġpert urb ation Ġis Ġcompletely Ġun super vised Ġ( i . e ., Ġself - contained ) Ġand Ġdoes Ġnot Ġinject Ġany Ġexternal Ġ( sem antic Ġsimilarity Ġoriented ) Ġknowledge Ġinto Ġthe Ġvector Ġspace . ĠNote Ġthat Ġdifferent Ġpert urb ations Ġcan Ġalso Ġbe Ġstacked : Ġe . g ., Ġwe Ġcan Ġapply Ġunc ove c Ġand Ġthen Ġuse Ġab tt Ġon Ġtop Ġthe Ġoutput Ġunc ove c Ġvectors . ĠWhen Ġusing Ġunc ove c Ġand Ġab tt Ġwe Ġalways Ġlength - normal ize Ġand Ġmean - center Ġthe Ġdata Ġfirst Ġ( i . e ., Ġwe Ġapply Ġthe Ġsimple Ġmc Ġnormal ization ). ĠFinally , Ġwe Ġtune Ġthe Ġtwo Ġhyper - param eters Ġ$ d _ A $ Ġ( for Ġab tt ) Ġand Ġ$\ alpha Ġ$ Ġ( unc ove c ) Ġon Ġthe ĠEnglish ĠMulti - Sim Lex Ġand Ġuse Ġthe Ġsame Ġvalues Ġon Ġthe Ġdatasets Ġof Ġall Ġother Ġlanguages ; Ġwe Ġreport Ġresults Ġwith Ġ$ dd _ A Ġ= Ġ3 $ Ġor Ġ$ dd _ A Ġ= Ġ10 $, Ġand Ġ$\ alpha Ġ= - 0 . 3 $. </s> Context ual ized ĠWord ĠEmb edd ings . ĠWe Ġalso Ġevaluate Ġthe Ġcapacity Ġof Ġun super vised Ġpret raining Ġarchitectures Ġbased Ġon Ġlanguage Ġmodeling Ġobjectives Ġto Ġreason Ġover Ġlex ical Ġsemantic Ġsimilarity . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġarticle Ġis Ġthe Ġfirst Ġstudy Ġperforming Ġsuch Ġanalyses . ĠState - of - the - art Ġmodels Ġsuch Ġas Ġb ert ĠB IB REF 29 , Ġx l m ĠB IB REF 30 , Ġor Ġro bert a ĠB IB REF 118 Ġare Ġtypically Ġvery Ġdeep Ġneural Ġnetworks Ġbased Ġon Ġthe ĠTrans former Ġarchitecture ĠB IB REF 119 . ĠThey Ġreceive Ġsub word - level Ġtokens Ġas Ġinputs Ġ( such Ġas ĠWord Pie ces ĠB IB REF 120 ) Ġto Ġtackle Ġdata Ġsp arsity . ĠIn Ġoutput , Ġthey Ġreturn Ġcontextual ized Ġembed d ings , Ġdynamic Ġrepresentations Ġfor Ġwords Ġin Ġcontext . </s> To Ġrepresent Ġwords Ġor Ġmulti - word Ġexpressions Ġthrough Ġa Ġpret rained Ġmodel , Ġwe Ġfollow Ġprior Ġwork ĠB IB REF 121 Ġand Ġcompute Ġan Ġinput Ġitem 's Ġrepresentation Ġby Ġ1 ) Ġfeeding Ġit Ġto Ġa Ġpret rained Ġmodel Ġin Ġisolation ; Ġthen Ġ2 ) Ġaveraging Ġthe Ġ$ H $ Ġlast Ġhidden Ġrepresentations Ġfor Ġeach Ġof Ġthe Ġitem âĢ Ļ s Ġconstituent Ġsub words ; Ġand Ġthen Ġfinally Ġ3 ) Ġaveraging Ġthe Ġresulting Ġsub word Ġrepresentations Ġto Ġproduce Ġthe Ġfinal Ġ$ d $ - dimensional Ġrepresentation , Ġwhere Ġ$ d $ Ġis Ġthe Ġembed ding Ġand Ġhidden - layer Ġdimension ality Ġ( e . g ., Ġ$ d = 768 $ Ġwith Ġb ert ). ĠWe Ġopt Ġfor Ġthis Ġapproach Ġdue Ġto Ġits Ġproven Ġviability Ġand Ġsimplicity ĠB IB REF 121 , Ġas Ġit Ġdoes Ġnot Ġrequire Ġany Ġadditional Ġcorpor a Ġto Ġcondition Ġthe Ġinduction Ġof Ġcontextual ized Ġembed d ings . ĠOther Ġways Ġto Ġextract Ġthe Ġrepresentations Ġfrom Ġpret rained Ġmodels ĠB IB REF 122 , ĠB IB REF 123 , ĠB IB REF 124 Ġare Ġbeyond Ġthe Ġscope Ġof Ġthis Ġwork , Ġand Ġwe Ġwill Ġexperiment Ġwith Ġthem Ġin Ġthe Ġfuture . </s> In Ġother Ġwords , Ġwe Ġtreat Ġeach Ġpret rained Ġenc oder Ġenc Ġas Ġa Ġblack - box Ġfunction Ġto Ġencode Ġa Ġsingle Ġword Ġor Ġa Ġmulti - word Ġexpression Ġ$ x $ Ġin Ġeach Ġlanguage Ġinto Ġa Ġ$ d $ - dimensional Ġcontextual ized Ġrepresentation Ġ$\ math bf Ġ{ x } _{ \ text sc Ġ{ enc }} Ġ\ in Ġ\ math bb Ġ{ R } ^ d Ġ= Ġ\ text sc Ġ{ enc } ( x ) $ Ġ( e . g ., Ġ$ d = 768 $ Ġwith Ġb ert ). ĠAs Ġmult ilingual Ġpret rained Ġenc od ers , Ġwe Ġexperiment Ġwith Ġthe Ġmult ilingual Ġb ert Ġmodel Ġ( m - bert ) ĠB IB REF 29 Ġand Ġx l m ĠB IB REF 30 . Ġm - bert Ġis Ġpret rained Ġon Ġmon oling ual ĠWikipedia Ġcorpor a Ġof Ġ102 Ġlanguages Ġ( com prising Ġall ĠMulti - Sim Lex Ġlanguages ) Ġwith Ġa Ġ12 - layer ĠTrans former Ġnetwork , Ġand Ġyields Ġ768 - dimensional Ġrepresentations . ĠSince Ġthe Ġconcept Ġpairs Ġin ĠMulti - Sim Lex Ġare Ġlower c ased , Ġwe Ġuse Ġthe Ġunc ased Ġversion Ġof Ġm - bert . Ġm - bert Ġcomprises Ġall ĠMulti - Sim Lex Ġlanguages , Ġand Ġits Ġevident Ġability Ġto Ġperform Ġcross - ling ual Ġtransfer ĠB IB REF 12 , ĠB IB REF 125 , ĠB IB REF 126 Ġalso Ġmakes Ġit Ġa Ġconvenient Ġbaseline Ġmodel Ġfor Ġcross - ling ual Ġexperiments Ġlater Ġin ĠÂ§ SEC REF 8 . ĠThe Ġsecond Ġmult ilingual Ġmodel Ġwe Ġconsider , Ġx l m - 100 , Ġis Ġpret rained Ġon ĠWikipedia Ġdumps Ġof Ġ100 Ġlanguages , Ġand Ġenc odes Ġeach Ġconcept Ġinto Ġa Ġ$ 1 , 280 $ - dimensional Ġrepresentation . ĠIn Ġcontrast Ġto Ġm - bert , Ġx l m - 100 Ġdrops Ġthe Ġnext - sent ence Ġprediction Ġobjective Ġand Ġadds Ġa Ġcross - ling ual Ġmasked Ġlanguage Ġmodeling Ġobjective . ĠFor Ġboth Ġenc od ers , Ġthe Ġrepresentations Ġof Ġeach Ġconcept Ġare Ġcomputed Ġas Ġaverages Ġover Ġthe Ġlast Ġ$ H = 4 $ Ġhidden Ġlayers Ġin Ġall Ġexperiments , Ġas Ġsuggested Ġby ĠWu : 2019 ar x iv . </s> Besides Ġm - bert Ġand Ġx l m , Ġcovering Ġmultiple Ġlanguages , Ġwe Ġalso Ġanalyze Ġthe Ġperformance Ġof ĠâĢ ľ language - specific âĢ Ŀ Ġb ert Ġand Ġx l m Ġmodels Ġfor Ġthe Ġlanguages Ġwhere Ġthey Ġare Ġavailable : ĠFinnish , ĠSpanish , ĠEnglish , ĠMandarin ĠChinese , Ġand ĠFrench . ĠThe Ġmain Ġgoal Ġof Ġthis Ġcomparison Ġis Ġto Ġstudy Ġthe Ġdifferences Ġin Ġperformance Ġbetween Ġmult ilingual ĠâĢ ľ one - size - fits - all âĢ Ŀ Ġenc od ers Ġand Ġlanguage - specific Ġenc od ers . ĠFor Ġall Ġexperiments , Ġwe Ġrely Ġon Ġthe Ġpret rained Ġmodels Ġreleased Ġin Ġthe ĠTransformers Ġrepository ĠB IB REF 127 . </s> Un super vised Ġpost - processing Ġsteps Ġdevised Ġfor Ġstatic Ġword Ġembed d ings Ġ( i . e ., Ġmean - cent ering , Ġab tt , Ġunc ove c ) Ġcan Ġalso Ġbe Ġapplied Ġon Ġtop Ġof Ġcontextual ized Ġembed d ings Ġif Ġwe Ġpred ef ine Ġa Ġvocabulary Ġof Ġword Ġtypes Ġ$ V $ Ġthat Ġwill Ġbe Ġrepresented Ġin Ġa Ġword Ġvector Ġspace Ġ$\ math bf Ġ{ X } $. ĠWe Ġconstruct Ġsuch Ġ$ V $ Ġfor Ġeach Ġlanguage Ġas Ġthe Ġintersection Ġof Ġword Ġtypes Ġcovered Ġby Ġthe Ġcorresponding ĠCC + Wiki Ġfast Text Ġvectors Ġand Ġthe Ġ( single - word Ġor Ġmulti - word ) Ġexpressions Ġappearing Ġin Ġthe Ġcorresponding ĠMulti - Sim Lex Ġdataset . </s> Finally , Ġnote Ġthat Ġit Ġis Ġnot Ġfeasible Ġto Ġevaluate Ġa Ġfull Ġrange Ġof Ġavailable Ġpret rained Ġenc od ers Ġwithin Ġthe Ġscope Ġof Ġthis Ġwork . ĠOur Ġmain Ġintention Ġis Ġto Ġprovide Ġthe Ġfirst Ġset Ġof Ġbaseline Ġresults Ġon ĠMulti - Sim Lex Ġby Ġbenchmark ing Ġa Ġsample Ġof Ġmost Ġpopular Ġenc od ers , Ġat Ġthe Ġsame Ġtime Ġalso Ġinvestigating Ġother Ġimportant Ġquestions Ġsuch Ġas Ġperformance Ġof Ġstatic Ġversus Ġcontextual ized Ġword Ġembed d ings , Ġor Ġmult ilingual Ġversus Ġlanguage - specific Ġpret raining . ĠAnother Ġpurpose Ġof Ġthe Ġexperiments Ġis Ġto Ġoutline Ġthe Ġwide Ġpotential Ġand Ġapplic ability Ġof Ġthe ĠMulti - Sim Lex Ġdatasets Ġfor Ġmult ilingual Ġand Ġcross - ling ual Ġrepresentation Ġlearning Ġevaluation . </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels Ġ: :: ĠResults Ġand ĠDiscussion </s> The Ġresults Ġwe Ġreport Ġare ĠSpear man 's Ġ$\ r ho Ġ$ Ġcoefficients Ġof Ġthe Ġcorrelation Ġbetween Ġthe Ġranks Ġderived Ġfrom Ġthe Ġscores Ġof Ġthe Ġevaluated Ġmodels Ġand Ġthe Ġhuman Ġscores Ġprovided Ġin Ġeach ĠMulti - Sim Lex Ġdataset . ĠThe Ġmain Ġresults Ġwith Ġstatic Ġand Ġcontextual ized Ġword Ġvectors Ġfor Ġall Ġtest Ġlanguages Ġare Ġsummarized Ġin ĠTable ĠT AB REF 43 . ĠThe Ġscores Ġreveal Ġseveral Ġinteresting Ġpatterns , Ġand Ġalso Ġpinpoint Ġthe Ġmain Ġchallenges Ġfor Ġfuture Ġwork . </s> State - of - the - Art ĠRepresent ation ĠModels . ĠThe Ġabsolute Ġscores Ġof ĠCC + Wiki Ġft , ĠWiki Ġft , Ġand Ġm - bert Ġare Ġnot Ġdirectly Ġcomparable , Ġbecause Ġthese Ġmodels Ġhave Ġdifferent Ġcoverage . ĠIn Ġparticular , ĠMulti - Sim Lex Ġcontains Ġsome Ġout - of - voc abulary Ġ( OO V ) Ġwords Ġwhose Ġstatic Ġft Ġembed d ings Ġare Ġnot Ġavailable . ĠOn Ġthe Ġother Ġhand , Ġm - bert Ġhas Ġperfect Ġcoverage . ĠA Ġgeneral Ġcomparison Ġbetween ĠCC + Wiki Ġand ĠWiki Ġft Ġvectors , Ġhowever , Ġsupports Ġthe Ġintuition Ġthat Ġlarger Ġcorpor a Ġ( such Ġas ĠCC + Wiki ) Ġyield Ġhigher Ġcorrelations . ĠAnother Ġfinding Ġis Ġthat Ġa Ġsingle Ġmassively Ġmult ilingual Ġmodel Ġsuch Ġas Ġm - bert Ġcannot Ġproduce Ġsem antically Ġrich Ġword - level Ġrepresentations . ĠWhether Ġthis Ġactually Ġhappens Ġbecause Ġthe Ġtraining Ġobjective Ġis Ġdifferent âĢĶ or Ġbecause Ġthe Ġneed Ġto Ġrepresent Ġ100 + Ġlanguages Ġreduces Ġits Ġlanguage - specific Ġcapacity âĢĶ is Ġinvestigated Ġfurther Ġbelow . </s> The Ġoverall Ġresults Ġalso Ġclearly Ġindicate Ġthat Ġ( i ) Ġthere Ġare Ġdifferences Ġin Ġperformance Ġacross Ġdifferent Ġmon oling ual ĠMulti - Sim Lex Ġdatasets , Ġand Ġ( ii ) Ġun super vised Ġpost - processing Ġis Ġuniversally Ġuseful , Ġand Ġcan Ġlead Ġto Ġhuge Ġimprovements Ġin Ġcorrelation Ġscores Ġfor Ġmany Ġlanguages . ĠIn Ġwhat Ġfollows , Ġwe Ġalso Ġdelve Ġdeeper Ġinto Ġthese Ġanalyses . </s> Imp act Ġof ĠUn super vised ĠPost - Process ing . ĠFirst , Ġthe Ġresults Ġin ĠTable ĠT AB REF 43 Ġsuggest Ġthat Ġapplying Ġdimension - wise Ġmean Ġcent ering Ġto Ġthe Ġinitial Ġvector Ġspaces Ġhas Ġpositive Ġimpact Ġon Ġword Ġsimilarity Ġscores Ġin Ġall Ġtest Ġlanguages Ġand Ġfor Ġall Ġmodels , Ġboth Ġstatic Ġand Ġcontextual ized Ġ( see Ġthe Ġ+ mc Ġrows Ġin ĠTable ĠT AB REF 43 ). ĠB IB REF 128 Ġshow Ġthat Ġdistribution al Ġword Ġvectors Ġhave Ġa Ġtendency Ġtowards Ġnarrow Ġclusters Ġin Ġthe Ġvector Ġspace Ġ( i . e ., Ġthey Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace Ġand Ġare Ġtherefore Ġan is otropic ĠB IB REF 113 , ĠB IB REF 129 ), Ġand Ġare Ġprone Ġto Ġthe Ġundes ired Ġeffect Ġof Ġhub ness ĠB IB REF 130 , ĠB IB REF 131 . ĠApp lying Ġdimension - wise Ġmean Ġcent ering Ġhas Ġthe Ġeffect Ġof Ġspreading Ġthe Ġvectors Ġacross Ġthe Ġhyper - plane Ġand Ġmitigating Ġthe Ġhub ness Ġissue , Ġwhich Ġconsequently Ġimproves Ġword - level Ġsimilarity , Ġas Ġit Ġemerges Ġfrom Ġthe Ġreported Ġresults . ĠPrevious Ġwork Ġhas Ġalready Ġvalidated Ġthe Ġimportance Ġof Ġmean Ġcent ering Ġfor Ġclust ering - based Ġtasks ĠB IB REF 132 , Ġbilingual Ġlex icon Ġinduction Ġwith Ġcross - ling ual Ġword Ġembed d ings ĠB IB REF 133 , ĠB IB REF 134 , ĠB IB REF 135 , Ġand Ġfor Ġmodeling Ġlex ical Ġsemantic Ġchange ĠB IB REF 136 . ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġresults Ġsummarized Ġin ĠTable ĠT AB REF 43 Ġare Ġthe Ġfirst Ġevidence Ġthat Ġalso Ġconfirms Ġits Ġimportance Ġfor Ġsemantic Ġsimilarity Ġin Ġa Ġwide Ġarray Ġof Ġlanguages . ĠIn Ġsum , Ġas Ġa Ġgeneral Ġrule Ġof Ġthumb , Ġwe Ġsuggest Ġto Ġalways Ġmean - center Ġrepresentations Ġfor Ġsemantic Ġtasks . </s> The Ġresults Ġfurther Ġindicate Ġthat Ġadditional Ġpost - processing Ġmethods Ġsuch Ġas Ġab tt Ġand Ġunc ove c Ġon Ġtop Ġof Ġmean - centered Ġvector Ġspaces Ġcan Ġlead Ġto Ġfurther Ġgains Ġin Ġmost Ġlanguages . ĠThe Ġgains Ġare Ġeven Ġvisible Ġfor Ġlanguages Ġwhich Ġstart Ġfrom Ġhigh Ġcorrelation Ġscores : Ġfor Ġinstance ., Ġc mn Ġwith ĠCC + Wiki Ġft Ġincreases Ġfrom Ġ0 . 5 34 Ġto Ġ0 . 583 , Ġfrom Ġ0 . 315 Ġto Ġ0 . 526 Ġwith ĠWiki Ġft , Ġand Ġfrom Ġ0 . 408 Ġto Ġ0 . 487 Ġwith Ġm - bert . ĠSimilarly , Ġfor Ġr us Ġwith ĠCC + Wiki Ġft Ġwe Ġcan Ġimprove Ġfrom Ġ0 . 422 Ġto Ġ0 . 500 , Ġand Ġfor Ġfra Ġthe Ġscores Ġimprove Ġfrom Ġ0 . 578 Ġto Ġ0 . 613 . ĠThere Ġare Ġadditional Ġsimilar Ġcases Ġreported Ġin ĠTable ĠT AB REF 43 . </s> Overall , Ġthe Ġun super vised Ġpost - processing Ġtechniques Ġseem Ġuniversally Ġuseful Ġacross Ġlanguages , Ġbut Ġtheir Ġefficacy Ġand Ġrelative Ġperformance Ġdoes Ġvary Ġacross Ġdifferent Ġlanguages . ĠNote Ġthat Ġwe Ġhave Ġnot Ġcarefully Ġfine - tun ed Ġthe Ġhyper - param eters Ġof Ġthe Ġevaluated Ġpost - processing Ġmethods , Ġso Ġadditional Ġsmall Ġimprovements Ġcan Ġbe Ġexpected Ġfor Ġsome Ġlanguages . ĠThe Ġmain Ġfinding , Ġhowever , Ġis Ġthat Ġthese Ġpost - processing Ġtechniques Ġare Ġrobust Ġto Ġsemantic Ġsimilarity Ġcomput ations Ġbeyond ĠEnglish , Ġand Ġare Ġtruly Ġlanguage Ġindependent . ĠFor Ġinstance , Ġremoving Ġdominant Ġlatent Ġ( PC A - based ) Ġcomponents Ġfrom Ġword Ġvectors Ġemphasizes Ġsemantic Ġdifferences Ġbetween Ġdifferent Ġconcepts , Ġas Ġonly Ġshared Ġnon - in form ative Ġlatent Ġsemantic Ġknowledge Ġis Ġremoved Ġfrom Ġthe Ġrepresentations . </s> In Ġsummary , Ġpret rained Ġword Ġembed d ings Ġdo Ġcontain Ġmore Ġinformation Ġpertaining Ġto Ġsemantic Ġsimilarity Ġthan Ġrevealed Ġin Ġthe Ġinitial Ġvectors . ĠThis Ġway , Ġwe Ġhave Ġcorrobor ated Ġthe Ġhypotheses Ġfrom Ġprior Ġwork ĠB IB REF 113 , ĠB IB REF 117 Ġwhich Ġwere Ġnot Ġpreviously Ġempir ically Ġverified Ġon Ġother Ġlanguages Ġdue Ġto Ġa Ġshortage Ġof Ġevaluation Ġdata ; Ġthis Ġgap Ġhas Ġnow Ġbeen Ġfilled Ġwith Ġthe Ġintroduction Ġof Ġthe ĠMulti - Sim Lex Ġdatasets . ĠIn Ġall Ġfollow - up Ġexperiments , Ġwe Ġalways Ġexplicitly Ġdenote Ġwhich Ġpost - processing Ġconfiguration Ġis Ġused Ġin Ġevaluation . </s> POS - Specific ĠSub sets . ĠWe Ġpresent Ġthe Ġresults Ġfor Ġsubs ets Ġof Ġword Ġpairs Ġgrouped Ġby ĠPOS Ġclass Ġin ĠTable ĠT AB REF 46 . ĠPrior Ġwork Ġbased Ġon ĠEnglish Ġdata Ġshowed Ġthat Ġrepresentations Ġfor Ġnoun s Ġare Ġtypically Ġof Ġhigher Ġquality Ġthan Ġthose Ġfor Ġthe Ġother ĠPOS Ġclasses ĠB IB REF 49 , ĠB IB REF 137 , ĠB IB REF 50 . ĠWe Ġobserve Ġa Ġsimilar Ġtrend Ġin Ġother Ġlanguages Ġas Ġwell . ĠThis Ġpattern Ġis Ġconsistent Ġacross Ġdifferent Ġrepresentation Ġmodels Ġand Ġcan Ġbe Ġattributed Ġto Ġseveral Ġreasons . ĠFirst , Ġverb Ġrepresentations Ġneed Ġto Ġexpress Ġa Ġrich Ġrange Ġof Ġsynt actic Ġand Ġsemantic Ġbehaviors Ġrather Ġthan Ġpurely Ġrefere nt ial Ġfeatures ĠB IB REF 138 , ĠB IB REF 139 , ĠB IB REF 73 . ĠSecond , Ġlow Ġcorrelation Ġscores Ġon Ġthe Ġadjective Ġand Ġad verb Ġsubs ets Ġin Ġsome Ġlanguages Ġ( e . g ., Ġpol , Ġcy m , Ġsw a ) Ġmight Ġbe Ġdue Ġto Ġtheir Ġlow Ġfrequency Ġin Ġmon oling ual Ġtexts , Ġwhich Ġyields Ġunreliable Ġrepresentations . ĠIn Ġgeneral , Ġthe Ġvariance Ġin Ġperformance Ġacross Ġdifferent Ġword Ġclasses Ġwarrants Ġfurther Ġresearch Ġin Ġclass - specific Ġrepresentation Ġlearning ĠB IB REF 140 , ĠB IB REF 50 . ĠThe Ġscores Ġfurther Ġattest Ġthe Ġusefulness Ġof Ġun super vised Ġpost - processing Ġas Ġalmost Ġall Ġclass - specific Ġcorrelation Ġscores Ġare Ġimproved Ġby Ġapplying Ġmean - cent ering Ġand Ġab tt . ĠFinally , Ġthe Ġresults Ġfor Ġm - bert Ġand Ġx l m - 100 Ġin ĠTable ĠT AB REF 46 Ġfurther Ġconfirm Ġthat Ġmassively Ġmult ilingual Ġpret raining Ġcannot Ġyield Ġreasonable Ġsemantic Ġrepresentations Ġfor Ġmany Ġlanguages : Ġin Ġfact , Ġfor Ġsome Ġclasses Ġthey Ġdisplay Ġno Ġcorrelation Ġwith Ġhuman Ġratings Ġat Ġall . </s> Diff erences Ġacross ĠLanguages . ĠNaturally , Ġthe Ġresults Ġfrom ĠTables ĠT AB REF 43 Ġand ĠT AB REF 46 Ġalso Ġreveal Ġthat Ġthere Ġis Ġvariation Ġin Ġperformance Ġof Ġboth Ġstatic Ġword Ġembed d ings Ġand Ġpret rained Ġenc od ers Ġacross Ġdifferent Ġlanguages . ĠAmong Ġother Ġcauses , Ġthe Ġlowest Ġabsolute Ġscores Ġwith Ġft Ġare Ġreported Ġfor Ġlanguages Ġwith Ġleast Ġresources Ġavailable Ġto Ġtrain Ġmon oling
Introduction The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2. The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3. The latest version of UD (as of March 2020) covers more than 70 languages. Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages. Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12. Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14. For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i.e., the distributional information). Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning. Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP. Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24. Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data. Even if some resources do exist, they are limited in their size (e.g., 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e.g., all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns). This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4). Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales. For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28. Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27. This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses. In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages. We apply this protocol to a set of 12 languages, including a mixture of major languages (e.g., Mandarin, Russian, and French) as well as several low-resource ones (e.g., Kiswahili, Welsh, and Yue Chinese). We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e.g., the average mean inter-annotator agreement for Welsh is 0.742). The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses. Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5. Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location). We investigate this question in §SECREF25. Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms. We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages. The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones. We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8. Contributions. We now summarize the main contributions of this work: 1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs. This dataset serves as the main source for the creation of equivalent datasets in several other languages. 2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol. These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages. The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages. 3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets. To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity. 4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages. The results can serve as strong baselines that lay the foundation for future improvements. 5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages. To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32. The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning. 6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages. We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages. In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future. Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages. All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex.com/. Lexical Semantic Similarity ::: Similarity and Association The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity. For any pair of words, this relation measures whether their referents share the same features. For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls. This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations. For instance, a painter is easily associated with frescos, although they lack any physical commonalities. Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36. Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38. In fact, there exist plenty of pairs that are intuitively associated but not similar. Pairs where the converse is true can also be encountered, although more rarely. An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer. BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends. Several studies on human cognition also point in the same direction. For instance, semantic priming can be triggered by similar words without association BIBREF41. On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42. A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree. On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not. While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45. This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy). Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization The ramifications of the distinction between similarity and association are profound for distributional semantics. This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48. According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts. Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50. As a result, distributional methods obscure a crucial facet of lexical meaning. This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53. In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29. As a result, in the induced representations, geometrical closeness (measured e.g. through cosine distance) conflates genuine similarity with broad relatedness. For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis. BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations. Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods. The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial. In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most. As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal. In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded. Moreover, word representations can be specialized (a.k.a. retrofitted) by disentangling word relations of similarity and association. In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties. Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective. While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages. This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact. This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66. For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67. In general, semantic typology studies the variation in lexical semantics across the world's languages. According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns. For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70. We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology. Previous Work and Evaluation Data Word Pair Datasets. Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create. The scale of this problem gets multiplied by the number of languages in consideration. Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks. This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14. While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999. This inspired other similar datasets which focused on different lexical properties. For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts. Semantic Similarity Datasets in Other Languages. Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages. The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79. Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999. On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500). A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese. BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it. Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements. More importantly, they also did not reannotate the translated pairs in the target languages. As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity. A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages. Further, concept pairs for different languages are sourced from different corpora (e.g., direct translation of the English data versus sampling from scratch in the target language). Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts. Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages. Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e.g., Welsh, Kiswahili as in this work). Multilingual Datasets for Natural Language Understanding. The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages. For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference. XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85. Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages. While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language. Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages. We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity. In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages. The Base for Multi-SimLex: Extending English SimLex-999 In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5. We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary. We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource. Construction Criteria. The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27: (C1) Representative and diverse. The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e.g., nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges. (C2) Clearly defined. The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations. For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75. (C3) Consistent and reliable. The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines. In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements. Moreover, we follow good practices established by the research on related resources. In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e.g., by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e.g., verbs BIBREF15, 4) annotation quality control BIBREF78. Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages. The Final Output: English Multi-SimLex. In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset. This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch. Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5. We now describe the external data sources for the final set of word pairs: 1) Source: SimLex-999. BIBREF14. The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset. Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs. While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15. 2) Source: SemEval-17: Task 2 BIBREF92. We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999. The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories. This ensures topical diversity in our sub-sample. 3) Source: CARD-660 BIBREF78. 67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500. Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94. CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages. Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above. 4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500. We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra. In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1.5), [1.5,3), [3,4.5), [4.5, 6]$. Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena. 5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English. In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15. This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e.g., later – never, now – here, once – twice). Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e.g. adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet. The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc. We include a total of 123 adverb pairs into the final English Multi-SimLex. Note that this is the first time adverbs are included into any semantic similarity dataset. Fulfillment of Construction Criteria. The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs. As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource. As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e.g., infection – inflection). We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity. In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages. Multi-SimLex: Translation and Annotation We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource. We also provide key data statistics and preliminary cross-lingual comparative analyses. Language Selection. Multi-SimLex comprises eleven languages in addition to English. The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family). Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex. We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families. This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili. We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project. The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set. Detailed guidelines for both phases are available online at: https://multisimlex.com. Multi-SimLex: Translation and Annotation ::: Word Pair Translation Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules. (1) All pairs in the translated set must be unique (i.e., no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e.g., it is not allowed to translate car and automobile to the same Spanish word coche). (3) The translated pairs must preserve the semantic relations between the two words when possible. This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected. (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e.g., the English word homework is translated into the Polish MWE praca domowa). Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward. For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side. UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother). Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily. This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance. Further, many translation instances were resolved using near-synonymous terms in the translation. For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable. Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language. In some cases, a direct transliteration from English is used. For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair. We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language. The sample is proportionally stratified according to the part-of-speech categories. The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set. Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets. Overall across all languages, the agreement is 84.8%, which is similar to prior work BIBREF27, BIBREF97. Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language). We finally collect at least ten valid annotations for each word pair in each language. All annotators were required to abide by the following instructions: 1. Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are. A score of 6 indicates very high similarity (i.e., perfect synonymy), while zero indicates no similarity. 2. Each annotator must score the entire set of 1,888 pairs in the dataset. The pairs must not be shared between different annotators. 3. Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e.g. dictionaries, thesauri, WordNet) if required. 4. Annotators are kept anonymous, and are not able to communicate with each other during the annotation process. The selection criteria for the annotators required that all annotators must be native speakers of the target language. Preference to annotators with university education was given, but not required. Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score. The annotators did not have access to the original pairs in English. To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp. It consists of the following three rounds: Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6. Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement. We ask the annotators to reconsider the assigned scores for those pairs only. The annotators may chose to either change or keep the scores. As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous. This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78. We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \ge \mu _i + 1.5$ or $s_i \le \mu _i - 1.5$, where $\mu _i$ is the mean of the other annotators' scores. Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators. We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round. The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1. Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language. We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA). Both of these use Spearman's correlation ($\rho $) between annotators scores, the only difference is how they are averaged. They are computed as follows: where $\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators. APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14. It simply averages the pairwise Spearman's correlation between all annotators. On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators. It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78. We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets. As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores. Crucially, the results indicate `strong agreement' (across all languages) using both measurements. The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores. These scores, however, are still considered to be `moderately strong agreement'. Multi-SimLex: Translation and Annotation ::: Data Analysis Similarity Score Distributions. Across all languages, the average score (mean $=1.61$, median$=1.1$) is on the lower side of the similarity scale. However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores. Notably, French has the highest average of similarity scores (mean$=2.61$, median$=2.5$), while Kiswahili has the lowest average (mean$=1.28$, median$=0.5$). Russian has the lowest spread ($\sigma =1.37$), while Polish has the largest ($\sigma =1.62$). All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0.6 for all language pairs. Languages that share the same language family are highly correlated (e.g, cmn-yue, rus-pol, est-fin). In addition, we observe high correlations between English and most other languages, as expected. This is due to the effect of using English as the base/anchor language to create the dataset. In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways. Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e.g., heb-cmn, see Figure FIGREF20). This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97. While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages. We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex. Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range. For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals. Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora. While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties. For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms. The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages. However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset. Cross-Linguistic Differences. Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs. Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair. The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution). Some differences in similarity scores seem to group languages into clusters. For example, the word pair regular – average has an average similarity score of 4.0 and 4.1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0.5 and 0.8. We analyze this phenomenon in more detail in §SECREF25. There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages. For example, large – big in English has an average similarity score of 5.9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2.7 and 3.8. What is more, woman – wife receives an average similarity of 0.9 in English, 2.9 in Spanish, and greater than 4.0 in Kiswahili and Welsh. The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5.1, while in the other three languages the similarity score for this word pair does not exceed 0.1. At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0.4) than in other languages (4.1 – 5.1). In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5.5, while the other languages receive 2.8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0.9, while the other languages receive 5.3 or more. Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5.4 among Welsh speakers but 2.3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4.3 or more on average. There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages. For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages. A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar. When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh. The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores. This was also the case with the football – soccer concept pair. The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'. Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur. An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni). The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'. The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects. The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation. While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages. Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use. Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most. For example, in English all participants rated the similarity of trial – test to be 4 or 5. In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6. In Kiswahili, money – cash received a similarity rating of 6 from every participant. While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating. Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives. Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random. To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98. The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores. In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors. In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features. In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages. In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology). Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts. BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models. The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality. Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair. Given a set of L languages, each resulting matrix $S$ has dimensionality of $\mathbb {R}^{|L| \times |L|}$ and is symmetrical. To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances. The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0.05$ and $z > 1.96$. The correlation coefficient is particularly strong for geography ($r = 0.647$) and syntax ($r = 0.649$). The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations. The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics. Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography). In fact, several syntactic properties are not uniformly spread across the globe. For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102. In turn, geographical proximity leads to similar judgment patterns, as mentioned above. On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100. Cross-Lingual Multi-SimLex Datasets A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets. Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task. However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e.g., English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs). We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval. As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30. The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages. This yields a total of 66 cross-lingual datasets, one for each possible combination of languages. Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores. The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment). First, given two languages, we intersect their aligned concept pairs obtained through translation. For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule. The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores. Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i.e., $\mid s_s - s_t \mid \le 1.2$). This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97. We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure. UTF8gbsn To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl. We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets. Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair. The Spearman's correlation score $\rho $ between automatically induced and manually collected ratings achieves $\rho \ge 0.90$ on all samples, which confirms the viability of the automatic construction procedure. Score and Class Distributions. The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively. First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e.g., the eng-fra dataset contains the lowest proportion of nouns (49.21%) and the highest proportion of verbs (27.1%), adjectives (15.28%), and adverbs (8.41%). On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance. This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19). It is also evident that the data are skewed towards lower-similarity concept pairs. However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs. For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets. Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial. If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling. Monolingual Evaluation of Representation Learning Models After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data. We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29. These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages. In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions. (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)? Monolingual Evaluation of Representation Learning Models ::: Models in Comparison Static Word Embeddings in Different Languages. First, we evaluate a standard method for inducing non-contextualized (i.e., static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109. In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i.e. the contituent character n-grams of each word BIBREF110. For this reason, fastText is also more suited for modeling rare words and morphologically rich languages. We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages. The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs. We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language.. The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6. Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words. Unsupervised Post-Processing. Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource. So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113. In this paper, we assess if unsupervised post-processing is beneficial also in other languages. To this end, we apply the following post-hoc transformations on the initial word embeddings: 1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115. 2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words. The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations. Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113. 3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors. In short, it transforms the input space $\mathbf {X}$ into an adjusted space $\mathbf {X}\mathbf {W}_{\alpha }$ through a linear map $\mathbf {W}_{\alpha }$ controlled by a single hyper-parameter $\alpha $. The $n^{\text{th}}$-order similarity transformation of the input word vector space $\mathbf {X}$ (for which $n=1$) can be obtained as $\mathbf {M}_{n}(\mathbf {X}) = \mathbf {M}_1(\mathbf {X}\mathbf {W}_{(n-1)/2})$, with $\mathbf {W}_{\alpha }=\mathbf {Q}\mathbf {\Gamma }^{\alpha }$, where $\mathbf {Q}$ and $\mathbf {\Gamma }$ are the matrices obtained via eigendecomposition of $\mathbf {X}^T\mathbf {X}=\mathbf {Q}\mathbf {\Gamma }\mathbf {Q}^T$. $\mathbf {\Gamma }$ is a diagonal matrix containing eigenvalues of $\mathbf {X}^T\mathbf {X}$; $\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\mathbf {X}^T\mathbf {X}$ as columns. While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned. For more technical details we refer the reader to the original work of BIBREF117. As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\mathbf {X}$, produce a perturbed/transformed output vector space $\mathbf {X}^{\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i.e., self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space. Note that different perturbations can also be stacked: e.g., we can apply uncovec and then use abtt on top the output uncovec vectors. When using uncovec and abtt we always length-normalize and mean-center the data first (i.e., we apply the simple mc normalization). Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\alpha =-0.3$. Contextualized Word Embeddings. We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity. To the best of our knowledge, our article is the first study performing such analyses. State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119. They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity. In output, they return contextualized embeddings, dynamic representations for words in context. To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e.g., $d=768$ with bert). We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings. Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future. In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\mathbf {x}_{\textsc {enc}} \in \mathbb {R}^d = \textsc {enc}(x)$ (e.g., $d=768$ with bert). As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30. m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations. Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert. m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8. The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation. In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective. For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv. Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French. The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders. For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127. Unsupervised post-processing steps devised for static word embeddings (i.e., mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\mathbf {X}$. We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset. Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work. Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining. Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation. Monolingual Evaluation of Representation Learning Models ::: Results and Discussion The results we report are Spearman's $\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset. The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43. The scores reveal several interesting patterns, and also pinpoint the main challenges for future work. State-of-the-Art Representation Models. The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage. In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available. On the other hand, m-bert has perfect coverage. A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations. Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations. Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below. The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages. In what follows, we also delve deeper into these analyses. Impact of Unsupervised Post-Processing. First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43). BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i.e., they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131. Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results. Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136. However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages. In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks. The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages. The gains are even visible for languages which start from high correlation scores: for instance., cmn with CC+Wiki ft increases from 0.534 to 0.583, from 0.315 to 0.526 with Wiki ft, and from 0.408 to 0.487 with m-bert. Similarly, for rus with CC+Wiki ft we can improve from 0.422 to 0.500, and for fra the scores improve from 0.578 to 0.613. There are additional similar cases reported in Table TABREF43. Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages. Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages. The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent. For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations. In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors. This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets. In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation. POS-Specific Subsets. We present the results for subsets of word pairs grouped by POS class in Table TABREF46. Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50. We observe a similar trend in other languages as well. This pattern is consistent across different representation models and can be attributed to several reasons. First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73. Second, low correlation scores on the adjective and adverb subsets in some languages (e.g., pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations. In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50. The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt. Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all. Differences across Languages. Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages. Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian. The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings. Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect. The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages. Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed. This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations. Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data. We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish. Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e.g., their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages. Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations. This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity. We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work. Multilingual vs. Language-Specific Contextualized Embeddings. Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data. In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models. Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased). In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\times $ reduction in the number of parameters BIBREF143. From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations. The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages. This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available. Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i.e., lexical-level) masking.Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b. Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge. Similarity-Specialized Word Embeddings. Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53. Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145. Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4. Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106. Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed. We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp. In a nutshell, their li-postspec method is a multi-step procedure that operates as follows. First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e.g., synonymy or antonymy). The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart. In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space. To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors. Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process. This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space. Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space. The scores are summarized in Table TABREF56. Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning. However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue. We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations. In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier. On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training. On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization. In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est. The critical problem indeed seems to be insufficient raw data for monolingual distributional training. Cross-Lingual Evaluation Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets. Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article. We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings. Cross-Lingual Evaluation ::: Models in Comparison Static Word Embeddings. We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148. The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106. Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107. More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154. In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157. Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice. Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons. For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually. For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135. We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used. The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work. Contextualized Cross-Lingual Word Embeddings. We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity. Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs. We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34. As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings. Cross-Lingual Evaluation ::: Results and Discussion Main Results and Differences across Language Pairs. A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60. The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity. First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135. The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili. It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin. However, typological differences (e.g., morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus. Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning. The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings. Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e.g., eng, fra, spa, cmn) which dominate the multilingual m-bert training. Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap. This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary. At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus. In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model. Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings. We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces. A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs. Fully Unsupervised vs. Weakly Supervised Cross-Lingual Embeddings. The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs. However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs. Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59. In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets. First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all. We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores. The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs). Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs. We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision. We again note large gains with self-learning. Multilingual vs. Bilingual Contextualized Embeddings. Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair. Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++). We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details. The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs. We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model. Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity. In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e.g., word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm. Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67). We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area. Conclusion and Future Work We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets. The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili. The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field. In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource. We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages. This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research. The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models. Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments. In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages. Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex. This enables future endeavors to improve multilingual representation learning with challenging baselines. In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations: 1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned. 2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs). This confirms previous generalizations based on experiments on English. 3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features). 4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31. Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally. 5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity. In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data. Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages. For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance. However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue. In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100. Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages. Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding. In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification. Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate? In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages. We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work. In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex.com/. This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909). Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship ("Investissements d'avenir" program, reference ANR-19-P3IA-0001).
What are the 12 languages covered?
a616a3f0d244368ec588f04dfbc37d77fda01b4c
What Ġare Ġthe Ġ12 Ġlanguages Ġcovered ?
Chinese ĠMandarin , ĠWelsh , ĠEnglish , ĠEston ian , ĠFinnish , ĠFrench , ĠHebrew , ĠPolish , ĠRussian , ĠSpanish , ĠKis w ah ili , ĠYue ĠChinese
<s> What Ġare Ġthe Ġ12 Ġlanguages Ġcovered ? </s> Introduction </s> The Ġlack Ġof Ġannot ated Ġtraining Ġand Ġevaluation Ġdata Ġfor Ġmany Ġtasks Ġand Ġdomains Ġhind ers Ġthe Ġdevelopment Ġof Ġcomputational Ġmodels Ġfor Ġthe Ġmajority Ġof Ġthe Ġworld 's Ġlanguages ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . ĠThe Ġnecessity Ġto Ġguide Ġand Ġadvance Ġmult ilingual Ġand Ġcross - ling ual ĠN LP Ġthrough Ġannotation Ġefforts Ġthat Ġfollow Ġcross - ling ually Ġconsistent Ġguidelines Ġhas Ġbeen Ġrecently Ġrecognized Ġby Ġcollaborative Ġinitiatives Ġsuch Ġas Ġthe ĠUniversal ĠDepend ency Ġ( UD ) Ġproject ĠB IB REF 3 . ĠThe Ġlatest Ġversion Ġof ĠUD Ġ( as Ġof ĠMarch Ġ2020 ) Ġcovers Ġmore Ġthan Ġ70 Ġlanguages . ĠCru cially , Ġthis Ġresource Ġcontinues Ġto Ġsteadily Ġgrow Ġand Ġevolve Ġthrough Ġthe Ġcontributions Ġof Ġannot ators Ġfrom Ġacross Ġthe Ġworld , Ġextending Ġthe ĠUD 's Ġreach Ġto Ġa Ġwide Ġarray Ġof Ġtyp ologically Ġdiverse Ġlanguages . ĠBesides Ġsteering Ġresearch Ġin Ġmult ilingual Ġparsing ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġand Ġcross - ling ual Ġparser Ġtransfer ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , Ġthe Ġconsistent Ġannotations Ġand Ġguidelines Ġhave Ġalso Ġenabled Ġa Ġrange Ġof Ġinsightful Ġcomparative Ġstudies Ġfocused Ġon Ġthe Ġlanguages ' Ġsynt actic Ġ( dis ) similar ities ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 . </s> Insp ired Ġby Ġthe ĠUD Ġwork Ġand Ġits Ġsubstantial Ġimpact Ġon Ġresearch Ġin Ġ( mult ilingual ) Ġsyntax , Ġin Ġthis Ġarticle Ġwe Ġintroduce ĠMulti - Sim Lex , Ġa Ġsuite Ġof Ġmanually Ġand Ġconsistently Ġannot ated Ġsemantic Ġdatasets Ġfor Ġ12 Ġdifferent Ġlanguages , Ġfocused Ġon Ġthe Ġfundamental Ġlex ical Ġrelation Ġof Ġsemantic Ġsimilarity ĠB IB REF 13 , ĠB IB REF 14 . ĠFor Ġany Ġpair Ġof Ġwords , Ġthis Ġrelation Ġmeasures Ġwhether Ġtheir Ġrefere nt s Ġshare Ġthe Ġsame Ġ( functional ) Ġfeatures , Ġas Ġopposed Ġto Ġgeneral Ġcognitive Ġassociation Ġcaptured Ġby Ġco - occ urrence Ġpatterns Ġin Ġtexts Ġ( i . e ., Ġthe Ġdistribution al Ġinformation ). ĠDat as ets Ġthat Ġquantify Ġthe Ġstrength Ġof Ġtrue Ġsemantic Ġsimilarity Ġbetween Ġconcept Ġpairs Ġsuch Ġas ĠSim Lex - 999 ĠB IB REF 14 Ġor ĠSim Ver b - 35 00 ĠB IB REF 15 Ġhave Ġbeen Ġinstrumental Ġin Ġimproving Ġmodels Ġfor Ġdistribution al Ġsemantics Ġand Ġrepresentation Ġlearning . ĠDis cerning Ġbetween Ġsemantic Ġsimilarity Ġand Ġrelated ness / ass ociation Ġis Ġnot Ġonly Ġcrucial Ġfor Ġtheoretical Ġstudies Ġon Ġlex ical Ġsemantics Ġ( see ĠÂ§ SEC REF 2 ), Ġbut Ġhas Ġalso Ġbeen Ġshown Ġto Ġbenefit Ġa Ġrange Ġof Ġlanguage Ġunderstanding Ġtasks Ġin ĠN LP . ĠExamples Ġinclude Ġdialog Ġstate Ġtracking ĠB IB REF 16 , ĠB IB REF 17 , Ġspoken Ġlanguage Ġunderstanding ĠB IB REF 18 , ĠB IB REF 19 , Ġtext Ġsimpl ification ĠB IB REF 20 , ĠB IB REF 21 , ĠB IB REF 22 , Ġdictionary Ġand Ġthe s aurus Ġconstruction ĠB IB REF 23 , ĠB IB REF 24 . </s> Despite Ġthe Ġproven Ġusefulness Ġof Ġsemantic Ġsimilarity Ġdatasets , Ġthey Ġare Ġavailable Ġonly Ġfor Ġa Ġsmall Ġand Ġtyp ologically Ġnarrow Ġsample Ġof Ġresource - rich Ġlanguages Ġsuch Ġas ĠGerman , ĠItalian , Ġand ĠRussian ĠB IB REF 25 , Ġwhereas Ġsome Ġlanguage Ġtypes Ġand Ġlow - resource Ġlanguages Ġtypically Ġlack Ġsimilar Ġevaluation Ġdata . ĠEven Ġif Ġsome Ġresources Ġdo Ġexist , Ġthey Ġare Ġlimited Ġin Ġtheir Ġsize Ġ( e . g ., Ġ500 Ġpairs Ġin ĠTurkish ĠB IB REF 26 , Ġ500 Ġin ĠF ars i ĠB IB REF 27 , Ġor Ġ300 Ġin ĠFinnish ĠB IB REF 28 ) Ġand Ġcoverage Ġ( e . g ., Ġall Ġdatasets Ġwhich Ġoriginated Ġfrom Ġthe Ġoriginal ĠEnglish ĠSim Lex - 999 Ġcontain Ġonly Ġhigh - f requent Ġconcepts , Ġand Ġare Ġdominated Ġby Ġnoun s ). ĠThis Ġis Ġwhy , Ġas Ġour Ġdeparture Ġpoint , Ġwe Ġintroduce Ġa Ġlarger Ġand Ġmore Ġcomprehensive ĠEnglish Ġword Ġsimilarity Ġdataset Ġspanning Ġ1 , 888 Ġconcept Ġpairs Ġ( see ĠÂ§ SEC REF 4 ). </s> Most Ġimportantly , Ġsemantic Ġsimilarity Ġdatasets Ġin Ġdifferent Ġlanguages Ġhave Ġbeen Ġcreated Ġusing Ġheter ogeneous Ġconstruction Ġprocedures Ġwith Ġdifferent Ġguidelines Ġfor Ġtranslation Ġand Ġannotation , Ġas Ġwell Ġas Ġdifferent Ġrating Ġscales . ĠFor Ġinstance , Ġsome Ġdatasets Ġwere Ġobtained Ġby Ġdirectly Ġtranslating Ġthe ĠEnglish ĠSim Lex - 999 Ġin Ġits Ġentirety ĠB IB REF 25 , ĠB IB REF 16 Ġor Ġin Ġpart ĠB IB REF 28 . ĠOther Ġdatasets Ġwere Ġcreated Ġfrom Ġscratch ĠB IB REF 26 Ġand Ġyet Ġothers Ġsampled ĠEnglish Ġconcept Ġpairs Ġdifferently Ġfrom ĠSim Lex - 999 Ġand Ġthen Ġtranslated Ġand Ġre annot ated Ġthem Ġin Ġtarget Ġlanguages ĠB IB REF 27 . ĠThis Ġheterogeneity Ġmakes Ġthese Ġdatasets Ġincom parable Ġand Ġpre cludes Ġsystematic Ġcross - ling u istic Ġanalyses . ĠIn Ġthis Ġarticle , Ġconsolid ating Ġthe Ġlessons Ġlearned Ġfrom Ġprevious Ġdataset Ġconstruction Ġparad ig ms , Ġwe Ġpropose Ġa Ġcarefully Ġdesigned Ġtranslation Ġand Ġannotation Ġprotocol Ġfor Ġdeveloping Ġmon oling ual ĠMulti - Sim Lex Ġdatasets Ġwith Ġaligned Ġconcept Ġpairs Ġfor Ġtyp ologically Ġdiverse Ġlanguages . ĠWe Ġapply Ġthis Ġprotocol Ġto Ġa Ġset Ġof Ġ12 Ġlanguages , Ġincluding Ġa Ġmixture Ġof Ġmajor Ġlanguages Ġ( e . g ., ĠMandarin , ĠRussian , Ġand ĠFrench ) Ġas Ġwell Ġas Ġseveral Ġlow - resource Ġones Ġ( e . g ., ĠKis w ah ili , ĠWelsh , Ġand ĠYue ĠChinese ). ĠWe Ġdemonstrate Ġthat Ġour Ġproposed Ġdataset Ġcreation Ġprocedure Ġyields Ġdata Ġwith Ġhigh Ġinter - annot ator Ġagreement Ġrates Ġ( e . g ., Ġthe Ġaverage Ġmean Ġinter - annot ator Ġagreement Ġfor ĠWelsh Ġis Ġ0 . 7 42 ). </s> The Ġunified Ġconstruction Ġprotocol Ġand Ġalignment Ġbetween Ġconcept Ġpairs Ġenables Ġa Ġseries Ġof Ġquantitative Ġanalyses . ĠPrel iminary Ġstudies Ġon Ġthe Ġinfluence Ġthat Ġpoly se my Ġand Ġcross - ling ual Ġvariation Ġin Ġlex ical Ġcategories Ġ( see ĠÂ§ SEC REF 6 ) Ġhave Ġon Ġsimilarity Ġjudgments Ġare Ġprovided Ġin ĠÂ§ SEC REF 5 . ĠData Ġcreated Ġaccording Ġto ĠMulti - Sim Lex Ġprotocol Ġalso Ġallow Ġfor Ġprobing Ġinto Ġwhether Ġsimilarity Ġjudgments Ġare Ġuniversal Ġacross Ġlanguages , Ġor Ġrather Ġdepend Ġon Ġlinguistic Ġaffinity Ġ( in Ġterms Ġof Ġlinguistic Ġfeatures , Ġphylogen y , Ġand Ġgeographical Ġlocation ). ĠWe Ġinvestigate Ġthis Ġquestion Ġin ĠÂ§ SEC REF 25 . ĠNaturally , ĠMulti - Sim Lex Ġdatasets Ġcan Ġbe Ġused Ġas Ġan Ġintrinsic Ġevaluation Ġbenchmark Ġto Ġassess Ġthe Ġquality Ġof Ġlex ical Ġrepresentations Ġbased Ġon Ġmon oling ual , Ġjoint Ġmult ilingual , Ġand Ġtransfer Ġlearning Ġparad ig ms . ĠWe Ġconduct Ġa Ġsystematic Ġevaluation Ġof Ġseveral Ġstate - of - the - art Ġrepresentation Ġmodels Ġin ĠÂ§ SEC REF 7 , Ġshowing Ġthat Ġthere Ġare Ġlarge Ġgaps Ġbetween Ġhuman Ġand Ġsystem Ġperformance Ġin Ġall Ġlanguages . ĠThe Ġproposed Ġconstruction Ġparadigm Ġalso Ġsupports Ġthe Ġautomatic Ġcreation Ġof Ġ66 Ġcross - ling ual ĠMulti - Sim Lex Ġdatasets Ġby Ġinter le aving Ġthe Ġmon oling ual Ġones . ĠWe Ġoutline Ġthe Ġconstruction Ġof Ġthe Ġcross - ling ual Ġdatasets Ġin ĠÂ§ SEC REF 6 , Ġand Ġthen Ġpresent Ġa Ġquantitative Ġevaluation Ġof Ġa Ġseries Ġof Ġcutting - edge Ġcross - ling ual Ġrepresentation Ġmodels Ġon Ġthis Ġbenchmark Ġin ĠÂ§ SEC REF 8 . </s> Contribut ions . ĠWe Ġnow Ġsummarize Ġthe Ġmain Ġcontributions Ġof Ġthis Ġwork : </s> 1 ) ĠBuilding Ġon Ġlessons Ġlearned Ġfrom Ġprior Ġwork , Ġwe Ġcreate Ġa Ġmore Ġcomprehensive Ġlex ical Ġsemantic Ġsimilarity Ġdataset Ġfor Ġthe ĠEnglish Ġlanguage Ġspanning Ġa Ġtotal Ġof Ġ1 , 888 Ġconcept Ġpairs Ġbalanced Ġwith Ġrespect Ġto Ġsimilarity , Ġfrequency , Ġand Ġconc ret eness , Ġand Ġcovering Ġfour Ġword Ġclasses : Ġnoun s , Ġverbs , Ġadject ives Ġand , Ġfor Ġthe Ġfirst Ġtime , Ġad verbs . ĠThis Ġdataset Ġserves Ġas Ġthe Ġmain Ġsource Ġfor Ġthe Ġcreation Ġof Ġequivalent Ġdatasets Ġin Ġseveral Ġother Ġlanguages . </s> 2 ) ĠWe Ġpresent Ġa Ġcarefully Ġdesigned Ġand Ġrigorous Ġlanguage - agn ostic Ġtranslation Ġand Ġannotation Ġprotocol . ĠThese Ġwell - defined Ġguidelines Ġwill Ġfacilitate Ġthe Ġdevelopment Ġof Ġfuture ĠMulti - Sim Lex Ġdatasets Ġfor Ġother Ġlanguages . ĠThe Ġproposed Ġprotocol Ġeliminates Ġsome Ġcrucial Ġissues Ġwith Ġprior Ġefforts Ġfocused Ġon Ġthe Ġcreation Ġof Ġmulti - ling ual Ġsemantic Ġresources , Ġnamely : Ġi ) Ġlimited Ġcoverage ; Ġii ) Ġheter ogeneous Ġannotation Ġguidelines ; Ġand Ġiii ) Ġconcept Ġpairs Ġwhich Ġare Ġsem antically Ġincom parable Ġacross Ġdifferent Ġlanguages . </s> 3 ) ĠWe Ġoffer Ġto Ġthe Ġcommunity Ġmanually Ġannot ated Ġevaluation Ġsets Ġof Ġ1 , 888 Ġconcept Ġpairs Ġacross Ġ12 Ġtyp ologically Ġdiverse Ġlanguages , Ġand Ġ66 Ġlarge Ġcross - ling ual Ġevaluation Ġsets . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , ĠMulti - Sim Lex Ġis Ġthe Ġmost Ġcomprehensive Ġevaluation Ġresource Ġto Ġdate Ġfocused Ġon Ġthe Ġrelation Ġof Ġsemantic Ġsimilarity . </s> 4 ) ĠWe Ġbenchmark Ġa Ġwide Ġarray Ġof Ġrecent Ġstate - of - the - art Ġmon oling ual Ġand Ġcross - ling ual Ġword Ġrepresentation Ġmodels Ġacross Ġour Ġsample Ġof Ġlanguages . ĠThe Ġresults Ġcan Ġserve Ġas Ġstrong Ġbas elines Ġthat Ġlay Ġthe Ġfoundation Ġfor Ġfuture Ġimprovements . </s> 5 ) ĠWe Ġpresent Ġa Ġfirst Ġlarge - scale Ġevaluation Ġstudy Ġon Ġthe Ġability Ġof Ġenc od ers Ġpret rained Ġon Ġlanguage Ġmodeling Ġ( such Ġas Ġb ert ĠB IB REF 29 Ġand Ġx l m ĠB IB REF 30 ) Ġto Ġreason Ġover Ġword - level Ġsemantic Ġsimilarity Ġin Ġdifferent Ġlanguages . ĠTo Ġour Ġown Ġsurprise , Ġthe Ġresults Ġshow Ġthat Ġmon oling ual Ġpret rained Ġenc od ers , Ġeven Ġwhen Ġpresented Ġwith Ġword Ġtypes Ġout Ġof Ġcontext , Ġare Ġsometimes Ġcompetitive Ġwith Ġstatic Ġword Ġembed ding Ġmodels Ġsuch Ġas Ġfast Text ĠB IB REF 31 Ġor Ġword 2 vec ĠB IB REF 32 . ĠThe Ġresults Ġalso Ġreveal Ġa Ġhuge Ġgap Ġin Ġperformance Ġbetween Ġmassively Ġmult ilingual Ġpret rained Ġenc od ers Ġand Ġlanguage - specific Ġenc od ers Ġin Ġfavor Ġof Ġthe Ġlatter : Ġour Ġfindings Ġsupport Ġother Ġrecent Ġempirical Ġevidence Ġrelated Ġto Ġthe ĠâĢ ľ c urse Ġof Ġmult iling uality âĢ Ŀ ĠB IB REF 33 , ĠB IB REF 34 Ġin Ġrepresentation Ġlearning . </s> 6 ) ĠWe Ġmake Ġall Ġof Ġthese Ġresources Ġavailable Ġon Ġa Ġwebsite Ġwhich Ġfacilitates Ġeasy Ġcreation , Ġsubmission Ġand Ġsharing Ġof ĠMulti - Sim lex - style Ġdatasets Ġfor Ġa Ġlarger Ġnumber Ġof Ġlanguages . ĠWe Ġhope Ġthat Ġthis Ġwill Ġyield Ġan Ġeven Ġlarger Ġrepository Ġof Ġsemantic Ġresources Ġthat Ġinspire Ġfuture Ġadvances Ġin ĠN LP Ġwithin Ġand Ġacross Ġlanguages . </s> In Ġlight Ġof Ġthe Ġsuccess Ġof ĠUniversal ĠDepend encies ĠB IB REF 3 , Ġwe Ġhope Ġthat Ġour Ġinitiative Ġwill Ġinst igate Ġa Ġcollaborative Ġpublic Ġeffort Ġwith Ġestablished Ġand Ġclear - cut Ġguidelines Ġthat Ġwill Ġresult Ġin Ġadditional ĠMulti - Sim Lex Ġdatasets Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages Ġin Ġthe Ġnear Ġfuture . ĠMoreover , Ġwe Ġhope Ġthat Ġit Ġwill Ġprovide Ġmeans Ġto Ġadvance Ġour Ġunderstanding Ġof Ġdistribution al Ġand Ġlex ical Ġsemantics Ġacross Ġa Ġlarge Ġnumber Ġof Ġlanguages . ĠAll Ġmon oling ual Ġand Ġcross - ling ual ĠMulti - Sim Lex Ġdatasets âĢĵ along Ġwith Ġdetailed Ġtranslation Ġand Ġannotation Ġguidelines âĢĵ are Ġavailable Ġonline Ġat : Ġhttps :// mult is im lex . com /. </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġand ĠAssociation </s> The Ġfocus Ġof Ġthe ĠMulti - Sim Lex Ġinitiative Ġis Ġon Ġthe Ġlex ical Ġrelation Ġof Ġpure Ġsemantic Ġsimilarity . ĠFor Ġany Ġpair Ġof Ġwords , Ġthis Ġrelation Ġmeasures Ġwhether Ġtheir Ġrefere nt s Ġshare Ġthe Ġsame Ġfeatures . ĠFor Ġinstance , Ġgraffiti Ġand Ġfres cos Ġare Ġsimilar Ġto Ġthe Ġextent Ġthat Ġthey Ġare Ġboth Ġforms Ġof Ġpainting Ġand Ġappear Ġon Ġwalls . ĠThis Ġrelation Ġcan Ġbe Ġcontrasted Ġwith Ġthe Ġcognitive Ġassociation Ġbetween Ġtwo Ġwords , Ġwhich Ġoften Ġdepends Ġon Ġhow Ġmuch Ġtheir Ġrefere nt s Ġinteract Ġin Ġthe Ġreal Ġworld , Ġor Ġare Ġfound Ġin Ġthe Ġsame Ġsituations . ĠFor Ġinstance , Ġa Ġpainter Ġis Ġeasily Ġassociated Ġwith Ġfres cos , Ġalthough Ġthey Ġlack Ġany Ġphysical Ġcommon alities . ĠAssociation Ġis Ġalso Ġknown Ġin Ġthe Ġliterature Ġunder Ġother Ġnames : Ġrelated ness ĠB IB REF 13 , Ġtopical Ġsimilarity ĠB IB REF 35 , Ġand Ġdomain Ġsimilarity ĠB IB REF 36 . </s> Sem antic Ġsimilarity Ġand Ġassociation Ġoverlap Ġto Ġsome Ġdegree , Ġbut Ġdo Ġnot Ġcoincide ĠB IB REF 37 , ĠB IB REF 38 . ĠIn Ġfact , Ġthere Ġexist Ġplenty Ġof Ġpairs Ġthat Ġare Ġintuitive ly Ġassociated Ġbut Ġnot Ġsimilar . ĠP airs Ġwhere Ġthe Ġcon verse Ġis Ġtrue Ġcan Ġalso Ġbe Ġencountered , Ġalthough Ġmore Ġrarely . ĠAn Ġexample Ġare Ġsyn onyms Ġwhere Ġa Ġword Ġis Ġcommon Ġand Ġthe Ġother Ġinf requent , Ġsuch Ġas Ġto Ġseize Ġand Ġto Ġcommand eer . ĠB IB REF 14 Ġrevealed Ġthat Ġwhile Ġsimilarity Ġmeasures Ġbased Ġon Ġthe ĠWord Net Ġgraph ĠB IB REF 39 Ġand Ġhuman Ġjudgments Ġof Ġassociation Ġin Ġthe ĠUniversity Ġof ĠSouth ĠFlorida ĠFree ĠAssociation ĠDatabase ĠB IB REF 40 Ġdo Ġcorrelate , Ġa Ġnumber Ġof Ġpairs Ġfollow Ġopposite Ġtrends . ĠSeveral Ġstudies Ġon Ġhuman Ġcognition Ġalso Ġpoint Ġin Ġthe Ġsame Ġdirection . ĠFor Ġinstance , Ġsemantic Ġprim ing Ġcan Ġbe Ġtriggered Ġby Ġsimilar Ġwords Ġwithout Ġassociation ĠB IB REF 41 . ĠOn Ġthe Ġother Ġhand , Ġa Ġconnection Ġwith Ġcue Ġwords Ġis Ġestablished Ġmore Ġquickly Ġfor Ġtop ically Ġrelated Ġwords Ġrather Ġthan Ġfor Ġsimilar Ġwords Ġin Ġfree Ġassociation Ġtasks ĠB IB REF 42 . </s> A Ġkey Ġproperty Ġof Ġsemantic Ġsimilarity Ġis Ġits Ġgrad ience : Ġpairs Ġof Ġwords Ġcan Ġbe Ġsimilar Ġto Ġa Ġdifferent Ġdegree . ĠOn Ġthe Ġother Ġhand , Ġthe Ġrelation Ġof Ġsyn ony my Ġis Ġbinary : Ġpairs Ġof Ġwords Ġare Ġsyn onyms Ġif Ġthey Ġcan Ġbe Ġsubstituted Ġin Ġall Ġcontexts Ġ( or Ġmost Ġcontexts , Ġin Ġa Ġlo oser Ġsense ), Ġotherwise Ġthey Ġare Ġnot . ĠWhile Ġsyn onyms Ġcan Ġbe Ġconceived Ġas Ġlying Ġon Ġone Ġextreme Ġof Ġthe Ġsemantic Ġsimilarity Ġcontinuum , Ġit Ġis Ġcrucial Ġto Ġnote Ġthat Ġtheir Ġdefinition Ġis Ġstated Ġin Ġpurely Ġrelational Ġterms , Ġrather Ġthan Ġinvoking Ġtheir Ġrefere nt ial Ġproperties ĠB IB REF 43 , ĠB IB REF 44 , ĠB IB REF 45 . ĠThis Ġmakes Ġbehavioral Ġstudies Ġon Ġsemantic Ġsimilarity Ġfundamentally Ġdifferent Ġfrom Ġlex ical Ġresources Ġlike ĠWord Net ĠB IB REF 46 , Ġwhich Ġinclude Ġparad igmatic Ġrelations Ġ( such Ġas Ġsyn ony my ). </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġfor ĠN LP : ĠInt r ins ic ĠEvaluation Ġand ĠSem antic ĠSpecial ization </s> The Ġramifications Ġof Ġthe Ġdistinction Ġbetween Ġsimilarity Ġand Ġassociation Ġare Ġprofound Ġfor Ġdistribution al Ġsemantics . ĠThis Ġparadigm Ġof Ġlex ical Ġsemantics Ġis Ġgrounded Ġin Ġthe Ġdistribution al Ġhypothesis , Ġformulated Ġby ĠB IB REF 47 Ġand ĠB IB REF 48 . ĠAccording Ġto Ġthis Ġhypothesis , Ġthe Ġmeaning Ġof Ġa Ġword Ġcan Ġbe Ġrecovered Ġempir ically Ġfrom Ġthe Ġcontexts Ġin Ġwhich Ġit Ġoccurs Ġwithin Ġa Ġcollection Ġof Ġtexts . ĠSince Ġboth Ġpairs Ġof Ġtop ically Ġrelated Ġwords Ġand Ġpairs Ġof Ġpurely Ġsimilar Ġwords Ġtend Ġto Ġappear Ġin Ġthe Ġsame Ġcontexts , Ġtheir Ġassociated Ġmeaning Ġconf ounds Ġthe Ġtwo Ġdistinct Ġrelations ĠB IB REF 14 , ĠB IB REF 49 , ĠB IB REF 50 . ĠAs Ġa Ġresult , Ġdistribution al Ġmethods Ġobscure Ġa Ġcrucial Ġfacet Ġof Ġlex ical Ġmeaning . ĠThis Ġlimitation Ġalso Ġreflects Ġonto Ġword Ġembed d ings Ġ( WE s ), Ġrepresentations Ġof Ġwords Ġas Ġlow - dimensional Ġvectors Ġthat Ġhave Ġbecome Ġindispensable Ġfor Ġa Ġwide Ġrange Ġof ĠN LP Ġapplications ĠB IB REF 51 , ĠB IB REF 52 , ĠB IB REF 53 . ĠIn Ġparticular , Ġit Ġinvolves Ġboth Ġstatic ĠWE s Ġlearned Ġfrom Ġco - occ urrence Ġpatterns ĠB IB REF 32 , ĠB IB REF 54 , ĠB IB REF 31 Ġand Ġcontextual ized ĠWE s Ġlearned Ġfrom Ġmodeling Ġword Ġsequences ĠB IB REF 55 , ĠB IB REF 29 . ĠAs Ġa Ġresult , Ġin Ġthe Ġinduced Ġrepresentations , Ġge omet rical Ġclos eness Ġ( me asured Ġe . g . Ġthrough Ġcos ine Ġdistance ) Ġconf l ates Ġgenuine Ġsimilarity Ġwith Ġbroad Ġrelated ness . ĠFor Ġinstance , Ġthe Ġvectors Ġfor Ġan ton y ms Ġsuch Ġas Ġsober Ġand Ġdrunk , Ġby Ġdefinition Ġdiss imilar , Ġmight Ġbe Ġneighbors Ġin Ġthe Ġsemantic Ġspace Ġunder Ġthe Ġdistribution al Ġhypothesis . ĠB IB REF 36 , ĠB IB REF 56 , Ġand ĠB IB REF 53 Ġdemonstrated Ġthat Ġdifferent Ġchoices Ġof Ġhyper - param eters Ġin ĠWE Ġalgorithms Ġ( such Ġas Ġcontext Ġwindow ) Ġemphasize Ġdifferent Ġrelations Ġin Ġthe Ġresulting Ġrepresentations . ĠLikewise , ĠB IB REF 57 Ġand ĠB IB REF 54 Ġdiscovered Ġthat ĠWE s Ġlearned Ġfrom Ġtexts Ġannot ated Ġwith Ġsynt actic Ġinformation Ġmirror Ġsimilarity Ġbetter Ġthan Ġsimple Ġlocal Ġbag - of - words Ġneighborhoods . </s> The Ġfailure Ġof ĠWE s Ġto Ġcapture Ġsemantic Ġsimilarity , Ġin Ġturn , Ġaffects Ġmodel Ġperformance Ġin Ġseveral ĠN LP Ġapplications Ġwhere Ġsuch Ġknowledge Ġis Ġcrucial . ĠIn Ġparticular , ĠNatural ĠLanguage ĠUnderstanding Ġtasks Ġsuch Ġas Ġstatistical Ġdialog Ġmodeling , Ġtext Ġsimpl ification , Ġor Ġsemantic Ġtext Ġsimilarity ĠB IB REF 58 , ĠB IB REF 18 , ĠB IB REF 59 , Ġamong Ġothers , Ġsuffer Ġthe Ġmost . ĠAs Ġa Ġconsequence , Ġresources Ġproviding Ġclean Ġinformation Ġon Ġsemantic Ġsimilarity Ġare Ġkey Ġin Ġmitigating Ġthe Ġside Ġeffects Ġof Ġthe Ġdistribution al Ġsignal . ĠIn Ġparticular , Ġsuch Ġdatabases Ġcan Ġbe Ġemployed Ġfor Ġthe Ġintrinsic Ġevaluations Ġof Ġspecific ĠWE Ġmodels Ġas Ġa Ġproxy Ġof Ġtheir Ġreliability Ġfor Ġdownstream Ġapplications ĠB IB REF 60 , ĠB IB REF 61 , ĠB IB REF 14 ; Ġintuitive ly , Ġthe Ġmore ĠWE s Ġare Ġmis aligned Ġwith Ġhuman Ġjudgments Ġof Ġsimilarity , Ġthe Ġmore Ġtheir Ġperformance Ġon Ġactual Ġtasks Ġis Ġexpected Ġto Ġbe Ġdegraded . ĠMoreover , Ġword Ġrepresentations Ġcan Ġbe Ġspecialized Ġ( a . k . a . Ġretro fitted ) Ġby Ġdis ent angling Ġword Ġrelations Ġof Ġsimilarity Ġand Ġassociation . ĠIn Ġparticular , Ġlinguistic Ġconstraints Ġsourced Ġfrom Ġexternal Ġdatabases Ġ( such Ġas Ġsyn onyms Ġfrom ĠWord Net ) Ġcan Ġbe Ġinjected Ġinto ĠWE s ĠB IB REF 62 , ĠB IB REF 63 , ĠB IB REF 16 , ĠB IB REF 22 , ĠB IB REF 64 Ġin Ġorder Ġto Ġenforce Ġa Ġparticular Ġrelation Ġin Ġa Ġdistribution al Ġsemantic Ġspace Ġwhile Ġpreserving Ġthe Ġoriginal Ġadj ac ency Ġproperties . </s> Lex ical ĠSem antic ĠSimilar ity Ġ: :: ĠSimilar ity Ġand ĠLanguage ĠVari ation : ĠSem antic ĠTyp ology </s> In Ġthis Ġwork , Ġwe Ġtackle Ġthe Ġconcept Ġof Ġ( true ) Ġsemantic Ġsimilarity Ġfrom Ġa Ġmult ilingual Ġperspective . ĠWhile Ġthe Ġsame Ġmeaning Ġrepresentations Ġmay Ġbe Ġshared Ġby Ġall Ġhuman Ġspeakers Ġat Ġa Ġdeep Ġcognitive Ġlevel , Ġthere Ġis Ġno Ġone - to - one Ġmapping Ġbetween Ġthe Ġwords Ġin Ġthe Ġlex icons Ġof Ġdifferent Ġlanguages . ĠThis Ġmakes Ġthe Ġcomparison Ġof Ġsimilarity Ġjudgments Ġacross Ġlanguages Ġdifficult , Ġsince Ġthe Ġmeaning Ġoverlap Ġof Ġtranslation ally Ġequivalent Ġwords Ġis Ġsometimes Ġfar Ġless Ġthan Ġexact . ĠThis Ġresults Ġfrom Ġthe Ġfact Ġthat Ġthe Ġway Ġlanguages Ġ` part ition ' Ġsemantic Ġfields Ġis Ġpartially Ġarbitrary ĠB IB REF 65 , Ġalthough Ġconstrained Ġcross - ling ually Ġby Ġcommon Ġcognitive Ġbiases ĠB IB REF 66 . ĠFor Ġinstance , Ġconsider Ġthe Ġfield Ġof Ġcolors : ĠEnglish Ġdistinguishes Ġbetween Ġgreen Ġand Ġblue , Ġwhereas ĠMur le Ġ( South ĠSudan ) Ġhas Ġa Ġsingle Ġword Ġfor Ġboth ĠB IB REF 67 . </s> In Ġgeneral , Ġsemantic Ġtyp ology Ġstudies Ġthe Ġvariation Ġin Ġlex ical Ġsemantics Ġacross Ġthe Ġworld 's Ġlanguages . ĠAccording Ġto ĠB IB REF 68 , Ġthe Ġways Ġlanguages Ġcategor ize Ġconcepts Ġinto Ġthe Ġlex icon Ġfollow Ġthree Ġmain Ġaxes : Ġ1 ) Ġgran ularity : Ġwhat Ġis Ġthe Ġnumber Ġof Ġcategories Ġin Ġa Ġspecific Ġdomain ? ; Ġ2 ) Ġboundary Ġlocation : Ġwhere Ġdo Ġthe Ġlines Ġmarking Ġdifferent Ġcategories Ġlie ? ; Ġ3 ) Ġgrouping Ġand Ġdis section : Ġwhat Ġare Ġthe Ġmembership Ġcriteria Ġof Ġa Ġcategory ; Ġwhich Ġinstances Ġare Ġconsidered Ġto Ġbe Ġmore Ġprototyp ical ? ĠDifferent Ġchoices Ġwith Ġrespect Ġto Ġthese Ġaxes Ġlead Ġto Ġdifferent Ġlex ical ization Ġpatterns . ĠFor Ġinstance , Ġdistinct Ġsenses Ġin Ġa Ġpoly sem ous Ġword Ġin ĠEnglish , Ġsuch Ġas Ġskin Ġ( re fer ring Ġto Ġboth Ġthe Ġbody Ġand Ġfruit ), Ġmay Ġbe Ġassigned Ġseparate Ġwords Ġin Ġother Ġlanguages Ġsuch Ġas ĠItalian Ġp elle Ġand Ġbu cc ia , Ġrespectively ĠB IB REF 70 . ĠWe Ġlater Ġanalyze Ġwhether Ġsimilarity Ġscores Ġobtained Ġfrom Ġnative Ġspeakers Ġalso Ġloosely Ġfollow Ġthe Ġpatterns Ġdescribed Ġby Ġsemantic Ġtyp ology . </s> Previous ĠWork Ġand ĠEvaluation ĠData </s> Word ĠPair ĠDat as ets . ĠRich Ġexpert - created Ġresources Ġsuch Ġas ĠWord Net ĠB IB REF 46 , ĠB IB REF 71 , ĠVerb Net ĠB IB REF 72 , ĠB IB REF 73 , Ġor ĠFrame Net ĠB IB REF 74 Ġencode Ġa Ġwealth Ġof Ġsemantic Ġand Ġsynt actic Ġinformation , Ġbut Ġare Ġexpensive Ġand Ġtime - consuming Ġto Ġcreate . ĠThe Ġscale Ġof Ġthis Ġproblem Ġgets Ġmultiplied Ġby Ġthe Ġnumber Ġof Ġlanguages Ġin Ġconsideration . ĠTherefore , Ġcrowd - s ourcing Ġwith Ġnon - ex pert Ġannot ators Ġhas Ġbeen Ġadopted Ġas Ġa Ġquicker Ġalternative Ġto Ġproduce Ġsmaller Ġand Ġmore Ġfocused Ġsemantic Ġresources Ġand Ġevaluation Ġbenchmarks . ĠThis Ġalternative Ġpractice Ġhas Ġhad Ġa Ġprofound Ġimpact Ġon Ġdistribution al Ġsemantics Ġand Ġrepresentation Ġlearning ĠB IB REF 14 . ĠWhile Ġsome Ġprominent ĠEnglish Ġword Ġpair Ġdatasets Ġsuch Ġas ĠWord Sim - 353 ĠB IB REF 75 , ĠMEN ĠB IB REF 76 , Ġor ĠStanford ĠRare ĠWords ĠB IB REF 77 Ġdid Ġnot Ġdiscriminate Ġbetween Ġsimilarity Ġand Ġrelated ness , Ġthe Ġimportance Ġof Ġthis Ġdistinction Ġwas Ġestablished Ġby ĠB IB REF 14 Ġthrough Ġthe Ġcreation Ġof ĠSim Lex - 999 . ĠThis Ġinspired Ġother Ġsimilar Ġdatasets Ġwhich Ġfocused Ġon Ġdifferent Ġlex ical Ġproperties . ĠFor Ġinstance , ĠSim Ver b - 35 00 ĠB IB REF 15 Ġprovided Ġsimilarity Ġratings Ġfor Ġ3 , 500 ĠEnglish Ġverbs , Ġwhereas ĠCARD - 660 ĠB IB REF 78 Ġaimed Ġat Ġmeasuring Ġthe Ġsemantic Ġsimilarity Ġof Ġinf requent Ġconcepts . </s> Sem antic ĠSimilar ity ĠDat as ets Ġin ĠOther ĠLanguages . ĠMot ivated Ġby Ġthe Ġimpact Ġof Ġdatasets Ġsuch Ġas ĠSim Lex - 999 Ġand ĠSim Ver b - 35 00 Ġon Ġrepresentation Ġlearning Ġin ĠEnglish , Ġa Ġline Ġof Ġrelated Ġwork Ġfocused Ġon Ġcreating Ġsimilar Ġresources Ġin Ġother Ġlanguages . ĠThe Ġdominant Ġapproach Ġis Ġtranslating Ġand Ġre annot ating Ġthe Ġentire Ġoriginal ĠEnglish ĠSim Lex - 999 Ġdataset , Ġas Ġdone Ġpreviously Ġfor ĠGerman , ĠItalian , Ġand ĠRussian ĠB IB REF 25 , ĠHebrew Ġand ĠCroatian ĠB IB REF 16 , Ġand ĠPolish ĠB IB REF 79 . ĠVen ek os ki : 2017 n od al ida Ġapply Ġthis Ġprocess Ġonly Ġto Ġa Ġsubset Ġof Ġ300 Ġconcept Ġpairs Ġfrom Ġthe ĠEnglish ĠSim Lex - 999 . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 27 Ġsampled Ġa Ġnew Ġset Ġof Ġ500 ĠEnglish Ġconcept Ġpairs Ġto Ġensure Ġwider Ġtopical Ġcoverage Ġand Ġbalance Ġacross Ġsimilarity Ġspect ra , Ġand Ġthen Ġtranslated Ġthose Ġpairs Ġto ĠGerman , ĠItalian , ĠSpanish , Ġand ĠF ars i Ġ( S EM EV AL - 500 ). ĠA Ġsimilar Ġapproach Ġwas Ġfollowed Ġby ĠB IB REF 26 Ġfor ĠTurkish , Ġby ĠB IB REF 80 Ġfor ĠMandarin ĠChinese , Ġand Ġby ĠB IB REF 81 Ġfor ĠJapanese . ĠB IB REF 82 Ġtranslated Ġthe Ġconc aten ation Ġof ĠSim Lex - 999 , ĠWord Sim - 353 , Ġand Ġthe ĠEnglish ĠSEM EV AL - 500 Ġinto ĠThai Ġand Ġthen Ġre annot ated Ġit . ĠFinally , ĠB IB REF 83 Ġtranslated ĠEnglish ĠSim Lex - 999 Ġand ĠWord Sim - 353 Ġto Ġ11 Ġresource - rich Ġtarget Ġlanguages Ġ( German , ĠFrench , ĠRussian , ĠItalian , ĠDutch , ĠChinese , ĠPortuguese , ĠSwedish , ĠSpanish , ĠArabic , ĠF ars i ), Ġbut Ġthey Ġdid Ġnot Ġprovide Ġdetails Ġconcerning Ġthe Ġtranslation Ġprocess Ġand Ġthe Ġresolution Ġof Ġtranslation Ġdisagreements . ĠMore Ġimportantly , Ġthey Ġalso Ġdid Ġnot Ġre annot ate Ġthe Ġtranslated Ġpairs Ġin Ġthe Ġtarget Ġlanguages . ĠAs Ġwe Ġdiscussed Ġin ĠÂ§ ĠSEC REF 6 Ġand Ġreiterate Ġlater Ġin ĠÂ§ SEC REF 5 , Ġsemantic Ġdifferences Ġamong Ġlanguages Ġcan Ġhave Ġa Ġprofound Ġimpact Ġon Ġthe Ġannotation Ġscores ; Ġpartic ul ary , Ġwe Ġshow Ġin ĠÂ§ SEC REF 25 Ġthat Ġthese Ġdifferences Ġeven Ġroughly Ġdefine Ġlanguage Ġclusters Ġbased Ġon Ġlanguage Ġaffinity . </s> A Ġcore Ġissue Ġwith Ġthe Ġcurrent Ġdatasets Ġconcerns Ġa Ġlack Ġof Ġone Ġunified Ġprocedure Ġthat Ġensures Ġthe Ġcompar ability Ġof Ġresources Ġin Ġdifferent Ġlanguages . ĠFurther , Ġconcept Ġpairs Ġfor Ġdifferent Ġlanguages Ġare Ġsourced Ġfrom Ġdifferent Ġcorpor a Ġ( e . g ., Ġdirect Ġtranslation Ġof Ġthe ĠEnglish Ġdata Ġversus Ġsampling Ġfrom Ġscratch Ġin Ġthe Ġtarget Ġlanguage ). ĠMoreover , Ġthe Ġprevious ĠSim Lex - based Ġmult ilingual Ġdatasets Ġinherit Ġthe Ġmain Ġdeficiencies Ġof Ġthe ĠEnglish Ġoriginal Ġversion , Ġsuch Ġas Ġthe Ġfocus Ġon Ġnoun s Ġand Ġhighly Ġfrequent Ġconcepts . ĠFinally , Ġprior Ġwork Ġmostly Ġfocused Ġon Ġlanguages Ġthat Ġare Ġwidely Ġspoken Ġand Ġdo Ġnot Ġaccount Ġfor Ġthe Ġvariety Ġof Ġthe Ġworld 's Ġlanguages . ĠOur Ġlong - term Ġgoal Ġis Ġdev ising Ġa Ġstandardized Ġmethodology Ġto Ġextend Ġthe Ġcoverage Ġalso Ġto Ġlanguages Ġthat Ġare Ġresource - lean Ġand / or Ġtyp ologically Ġdiverse Ġ( e . g ., ĠWelsh , ĠKis w ah ili Ġas Ġin Ġthis Ġwork ). </s> Mult ilingual ĠDat as ets Ġfor ĠNatural ĠLanguage ĠUnderstanding . ĠThe ĠMulti - Sim Lex Ġinitiative Ġand Ġcorresponding Ġdatasets Ġare Ġalso Ġaligned Ġwith Ġthe Ġrecent Ġefforts Ġon Ġproc uring Ġmult ilingual Ġbenchmarks Ġthat Ġcan Ġhelp Ġadvance Ġcomputational Ġmodeling Ġof Ġnatural Ġlanguage Ġunderstanding Ġacross Ġdifferent Ġlanguages . ĠFor Ġinstance , Ġpret rained Ġmult ilingual Ġlanguage Ġmodels Ġsuch Ġas Ġmult ilingual Ġb ert ĠB IB REF 29 Ġor Ġx l m ĠB IB REF 30 Ġare Ġtypically Ġprob ed Ġon ĠX N LI Ġtest Ġdata ĠB IB REF 84 Ġfor Ġcross - ling ual Ġnatural Ġlanguage Ġinference . ĠX N LI Ġwas Ġcreated Ġby Ġtranslating Ġexamples Ġfrom Ġthe ĠEnglish ĠMulti N LI Ġdataset , Ġand Ġprojecting Ġits Ġsentence Ġlabels ĠB IB REF 85 . ĠOther Ġrecent Ġmult ilingual Ġdatasets Ġtarget Ġthe Ġtask Ġof Ġquestion Ġanswering Ġbased Ġon Ġreading Ġcomprehension : Ġi ) ĠML Q A ĠB IB REF 86 Ġincludes Ġ7 Ġlanguages Ġii ) ĠX Qu AD ĠB IB REF 87 Ġ10 Ġlanguages ; Ġiii ) ĠTy Di Q A ĠB IB REF 88 Ġ9 Ġwidely Ġspoken Ġtyp ologically Ġdiverse Ġlanguages . ĠWhile ĠML Q A Ġand ĠX Qu AD Ġresult Ġfrom Ġthe Ġtranslation Ġfrom Ġan ĠEnglish Ġdataset , ĠTy Di Q A Ġwas Ġbuilt Ġindependently Ġin Ġeach Ġlanguage . ĠAnother Ġmult ilingual Ġdataset , ĠPA WS - X ĠB IB REF 89 , Ġfocused Ġon Ġthe Ġparaph rase Ġidentification Ġtask Ġand Ġwas Ġcreated Ġtranslating Ġthe Ġoriginal ĠEnglish ĠPA WS ĠB IB REF 90 Ġinto Ġ6 Ġlanguages . ĠWe Ġbelieve Ġthat ĠMulti - Sim Lex Ġcan Ġsubstantially Ġcontribute Ġto Ġthis Ġendeavor Ġby Ġoffering Ġa Ġcomprehensive Ġmult ilingual Ġbenchmark Ġfor Ġthe Ġfundamental Ġlex ical Ġlevel Ġrelation Ġof Ġsemantic Ġsimilarity . ĠIn Ġfuture Ġwork , ĠMulti - Sim Lex Ġalso Ġoffers Ġan Ġopportunity Ġto Ġinvestigate Ġthe Ġcorrelations Ġbetween Ġword - level Ġsemantic Ġsimilarity Ġand Ġperformance Ġin Ġdownstream Ġtasks Ġsuch Ġas ĠQ A Ġand ĠNL I Ġacross Ġdifferent Ġlanguages . </s> The ĠBase Ġfor ĠMulti - Sim Lex : ĠExt ending ĠEnglish ĠSim Lex - 999 </s> In Ġthis Ġsection , Ġwe Ġdiscuss Ġthe Ġdesign Ġprinciples Ġbehind Ġthe ĠEnglish Ġ( eng ) ĠMulti - Sim Lex Ġdataset , Ġwhich Ġis Ġthe Ġbasis Ġfor Ġall Ġthe ĠMulti - Sim Lex Ġdatasets Ġin Ġother Ġlanguages , Ġas Ġdetailed Ġin ĠÂ§ SEC REF 5 . ĠWe Ġfirst Ġargue Ġthat Ġa Ġnew , Ġmore Ġbalanced , Ġand Ġmore Ġcomprehensive Ġevaluation Ġresource Ġfor Ġlex ical Ġsemantic Ġsimilarity Ġin ĠEnglish Ġis Ġnecessary . ĠWe Ġthen Ġdescribe Ġhow Ġthe Ġ1 , 888 Ġword Ġpairs Ġcontained Ġin Ġthe Ġeng ĠMulti - Sim Lex Ġwere Ġselected Ġin Ġsuch Ġa Ġway Ġas Ġto Ġrepresent Ġvarious Ġlinguistic Ġphenomena Ġwithin Ġa Ġsingle Ġintegrated Ġresource . </s> Construction ĠCrit eria . ĠThe Ġfollowing Ġcriteria Ġhave Ġto Ġbe Ġsatisfied Ġby Ġany Ġhigh - quality Ġsemantic Ġevaluation Ġresource , Ġas Ġargued Ġby Ġprevious Ġstudies Ġfocused Ġon Ġthe Ġcreation Ġof Ġsuch Ġresources ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 91 , ĠB IB REF 27 : </s> ( C 1 ) ĠRepresentative Ġand Ġdiverse . ĠThe Ġresource Ġmust Ġcover Ġthe Ġfull Ġrange Ġof Ġdiverse Ġconcepts Ġoccurring Ġin Ġnatural Ġlanguage , Ġincluding Ġdifferent Ġword Ġclasses Ġ( e . g ., Ġnoun s , Ġverbs , Ġadject ives , Ġad verbs ), Ġconcrete Ġand Ġabstract Ġconcepts , Ġa Ġvariety Ġof Ġlex ical Ġfields , Ġand Ġdifferent Ġfrequency Ġranges . </s> ( C 2 ) ĠClearly Ġdefined . ĠThe Ġresource Ġmust Ġprovide Ġa Ġclear Ġunderstanding Ġof Ġwhich Ġsemantic Ġrelation Ġexactly Ġis Ġannot ated Ġand Ġmeasured , Ġpossibly Ġcontrasting Ġit Ġwith Ġother Ġrelations . ĠFor Ġinstance , Ġthe Ġoriginal ĠSim Lex - 999 Ġand ĠSim Ver b - 35 00 Ġexplicitly Ġfocus Ġon Ġtrue Ġsemantic Ġsimilarity Ġand Ġdistinguish Ġit Ġfrom Ġbroader Ġrelated ness Ġcaptured Ġby Ġdatasets Ġsuch Ġas ĠMEN ĠB IB REF 76 Ġor ĠWord Sim - 353 ĠB IB REF 75 . </s> ( C 3 ) ĠCons istent Ġand Ġreliable . ĠThe Ġresource Ġmust Ġensure Ġconsistent Ġannotations Ġobtained Ġfrom Ġnon - ex pert Ġnative Ġspeakers Ġfollowing Ġsimple Ġand Ġprecise Ġannotation Ġguidelines . </s> In Ġchoosing Ġthe Ġword Ġpairs Ġand Ġconstructing Ġeng ĠMulti - Sim Lex , Ġwe Ġadhere Ġto Ġthese Ġrequirements . ĠMoreover , Ġwe Ġfollow Ġgood Ġpractices Ġestablished Ġby Ġthe Ġresearch Ġon Ġrelated Ġresources . ĠIn Ġparticular , Ġsince Ġthe Ġintroduction Ġof Ġthe Ġoriginal ĠSim Lex - 999 Ġdataset ĠB IB REF 14 , Ġfollow - up Ġworks Ġhave Ġimproved Ġits Ġconstruction Ġprotocol Ġacross Ġseveral Ġaspects , Ġincluding : Ġ1 ) Ġcoverage Ġof Ġmore Ġlex ical Ġfields , Ġe . g ., Ġby Ġrelying Ġon Ġa Ġdiverse Ġset Ġof ĠWikipedia Ġcategories ĠB IB REF 27 , Ġ2 ) Ġinf requent / ra re Ġwords ĠB IB REF 78 , Ġ3 ) Ġfocus Ġon Ġparticular Ġword Ġclasses , Ġe . g ., Ġverbs ĠB IB REF 15 , Ġ4 ) Ġannotation Ġquality Ġcontrol ĠB IB REF 78 . ĠOur Ġgoal Ġis Ġto Ġmake Ġuse Ġof Ġthese Ġimprovements Ġtowards Ġa Ġlarger , Ġmore Ġrepresentative , Ġand Ġmore Ġreliable Ġlex ical Ġsimilarity Ġdataset Ġin ĠEnglish Ġand , Ġconsequently , Ġin Ġall Ġother Ġlanguages . </s> The ĠFinal ĠOutput : ĠEnglish ĠMulti - Sim Lex . ĠIn Ġorder Ġto Ġensure Ġthat Ġthe Ġcriterion ĠC 1 Ġis Ġsatisfied , Ġwe Ġconsolidate Ġand Ġintegrate Ġthe Ġdata Ġalready Ġcarefully Ġsampled Ġin Ġprior Ġwork Ġinto Ġa Ġsingle , Ġcomprehensive , Ġand Ġrepresentative Ġdataset . ĠThis Ġway , Ġwe Ġcan Ġcontrol Ġfor Ġdiversity , Ġfrequency , Ġand Ġother Ġproperties Ġwhile Ġavoiding Ġto Ġperform Ġthis Ġtime - consuming Ġselection Ġprocess Ġfrom Ġscratch . ĠNote Ġthat , Ġon Ġthe Ġother Ġhand , Ġthe Ġword Ġpairs Ġchosen Ġfor ĠEnglish Ġare Ġscored Ġfrom Ġscratch Ġas Ġpart Ġof Ġthe Ġentire ĠMulti - Sim Lex Ġannotation Ġprocess , Ġintroduced Ġlater Ġin ĠÂ§ SEC REF 5 . ĠWe Ġnow Ġdescribe Ġthe Ġexternal Ġdata Ġsources Ġfor Ġthe Ġfinal Ġset Ġof Ġword Ġpairs : </s> 1 ) ĠSource : ĠSim Lex - 999 . ĠB IB REF 14 . ĠThe ĠEnglish ĠMulti - Sim Lex Ġhas Ġbeen Ġinitially Ġconceived Ġas Ġan Ġextension Ġof Ġthe Ġoriginal ĠSim Lex - 999 Ġdataset . ĠTherefore , Ġwe Ġinclude Ġall Ġ999 Ġword Ġpairs Ġfrom ĠSim Lex , Ġwhich Ġspan Ġ666 Ġnoun Ġpairs , Ġ222 Ġverb Ġpairs , Ġand Ġ111 Ġadjective Ġpairs . ĠWhile ĠSim Lex - 999 Ġalready Ġprovides Ġexamples Ġrepresenting Ġdifferent ĠPOS Ġclasses , Ġit Ġdoes Ġnot Ġhave Ġa Ġsufficient Ġcoverage Ġof Ġdifferent Ġlinguistic Ġphenomena : Ġfor Ġinstance , Ġit Ġcontains Ġonly Ġvery Ġfrequent Ġconcepts , Ġand Ġit Ġdoes Ġnot Ġprovide Ġa Ġrepresentative Ġset Ġof Ġverbs ĠB IB REF 15 . </s> 2 ) ĠSource : ĠSem E val - 17 : ĠTask Ġ2 ĠB IB REF 92 . ĠWe Ġstart Ġfrom Ġthe Ġfull Ġdataset Ġof Ġ500 Ġconcept Ġpairs Ġto Ġextract Ġa Ġtotal Ġof Ġ334 Ġconcept Ġpairs Ġfor ĠEnglish ĠMulti - Sim Lex Ġa ) Ġwhich Ġcontain Ġonly Ġsingle - word Ġconcepts , Ġb ) Ġwhich Ġare Ġnot Ġnamed Ġentities , Ġc ) Ġwhere ĠPOS Ġtags Ġof Ġthe Ġtwo Ġconcepts Ġare Ġthe Ġsame , Ġd ) Ġwhere Ġboth Ġconcepts Ġoccur Ġin Ġthe Ġtop Ġ250 K Ġmost Ġfrequent Ġword Ġtypes Ġin Ġthe ĠEnglish ĠWikipedia , Ġand Ġe ) Ġdo Ġnot Ġalready Ġoccur Ġin ĠSim Lex - 999 . ĠThe Ġoriginal Ġconcepts Ġwere Ġsampled Ġas Ġto Ġspan Ġall Ġthe Ġ34 Ġdomains Ġavailable Ġas Ġpart Ġof ĠBabel Dom ains ĠB IB REF 93 , Ġwhich Ġroughly Ġcorrespond Ġto Ġthe Ġmain Ġhigh - level ĠWikipedia Ġcategories . ĠThis Ġensures Ġtopical Ġdiversity Ġin Ġour Ġsub - sample . </s> 3 ) ĠSource : ĠCARD - 660 ĠB IB REF 78 . Ġ67 Ġword Ġpairs Ġare Ġtaken Ġfrom Ġthis Ġdataset Ġfocused Ġon Ġrare Ġword Ġsimilarity , Ġapplying Ġthe Ġsame Ġselection Ġcriteria Ġa ) Ġto Ġe ) Ġemployed Ġfor ĠSEM EV AL - 500 . ĠWords Ġare Ġcontrolled Ġfor Ġfrequency Ġbased Ġon Ġtheir Ġoccurrence Ġcounts Ġfrom Ġthe ĠGoogle ĠNews Ġdata Ġand Ġthe Ġu k Wa C Ġcorpus ĠB IB REF 94 . ĠCARD - 660 Ġcontains Ġsome Ġwords Ġthat Ġare Ġvery Ġrare Ġ( log boat ), Ġdomain - specific Ġ( ery th role ukemia ) Ġand Ġslang Ġ( 2 m rw ), Ġwhich Ġmight Ġbe Ġdifficult Ġto Ġtranslate Ġand Ġannot ate Ġacross Ġa Ġwide Ġarray Ġof Ġlanguages . ĠHence , Ġwe Ġopt Ġfor Ġretaining Ġonly Ġthe Ġconcept Ġpairs Ġabove Ġthe Ġthreshold Ġof Ġtop Ġ250 K Ġmost Ġfrequent ĠWikipedia Ġconcepts , Ġas Ġabove . </s> 4 ) ĠSource : ĠSim Ver b - 35 00 ĠB IB REF 15 ĠSince Ġboth ĠCARD - 660 Ġand ĠSEM EV AL - 500 Ġare Ġheavily Ġskewed Ġtowards Ġnoun Ġpairs , Ġand Ġnoun s Ġalso Ġdominate Ġthe Ġoriginal ĠSim Lex - 999 , Ġwe Ġalso Ġextract Ġadditional Ġverb Ġpairs Ġfrom Ġthe Ġverb - specific Ġsimilarity Ġdataset ĠSim Ver b - 35 00 . ĠWe Ġrandomly Ġsample Ġ244 Ġverb Ġpairs Ġfrom ĠSim Ver b - 35 00 Ġthat Ġrepresent Ġall Ġsimilarity Ġspect ra . ĠIn Ġparticular , Ġwe Ġadd Ġ61 Ġverb Ġpairs Ġfor Ġeach Ġof Ġthe Ġsimilarity Ġintervals : Ġ$ [ 0 , 1 . 5 ), Ġ[ 1 . 5 , 3 ), Ġ[ 3 , 4 . 5 ), Ġ[ 4 . 5 , Ġ6 ] $. ĠSince Ġverbs Ġin ĠSim Ver b - 35 00 Ġwere Ġoriginally Ġchosen Ġfrom ĠVerb Net ĠB IB REF 95 , ĠB IB REF 73 , Ġthey Ġcover Ġa Ġwide Ġrange Ġof Ġverb Ġclasses Ġand Ġtheir Ġrelated Ġlinguistic Ġphenomena . </s> 5 ) ĠSource : ĠUniversity Ġof ĠSouth ĠFlorida ĠB IB REF 96 Ġnorms , Ġthe Ġlargest Ġdatabase Ġof Ġfree Ġassociation Ġfor ĠEnglish . ĠIn Ġorder Ġto Ġimprove Ġthe Ġrepresentation Ġof Ġdifferent ĠPOS Ġclasses , Ġwe Ġsample Ġadditional Ġadject ives Ġand Ġad verbs Ġfrom Ġthe ĠUS F Ġnorms Ġfollowing Ġthe Ġprocedure Ġestablished Ġby ĠB IB REF 14 , ĠB IB REF 15 . ĠThis Ġyields Ġadditional Ġ122 Ġadjective Ġpairs , Ġbut Ġonly Ġa Ġlimited Ġnumber Ġof Ġad verb Ġpairs Ġ( e . g ., Ġlater ĠâĢĵ Ġnever , Ġnow ĠâĢĵ Ġhere , Ġonce ĠâĢĵ Ġtwice ). ĠTherefore , Ġwe Ġalso Ġcreate Ġa Ġset Ġof Ġad verb Ġpairs Ġsemi - aut om atically Ġby Ġsampling Ġadject ives Ġthat Ġcan Ġbe Ġderiv ationally Ġtransformed Ġinto Ġad verbs Ġ( e . g . Ġadding Ġthe Ġsuffix Ġ- ly ) Ġfrom Ġthe ĠUS F , Ġand Ġassessing Ġthe Ġcorrectness Ġof Ġsuch Ġderiv ation Ġin ĠWord Net . ĠThe Ġresulting Ġpairs Ġinclude , Ġfor Ġinstance , Ġprimarily ĠâĢĵ Ġmainly , Ġsoftly ĠâĢĵ Ġfirmly , Ġroughly ĠâĢĵ Ġreliably , Ġetc . ĠWe Ġinclude Ġa Ġtotal Ġof Ġ123 Ġad verb Ġpairs Ġinto Ġthe Ġfinal ĠEnglish ĠMulti - Sim Lex . ĠNote Ġthat Ġthis Ġis Ġthe Ġfirst Ġtime Ġad verbs Ġare Ġincluded Ġinto Ġany Ġsemantic Ġsimilarity Ġdataset . </s> F ulf ill ment Ġof ĠConstruction ĠCrit eria . ĠThe Ġfinal Ġeng ĠMulti - Sim Lex Ġdataset Ġspans Ġ1 , 05 1 Ġnoun Ġpairs , Ġ4 69 Ġverb Ġpairs , Ġ245 Ġadjective Ġpairs , Ġand Ġ123 Ġad verb Ġpairs . ĠAs Ġmentioned Ġabove , Ġthe Ġcriterion ĠC 1 Ġhas Ġbeen Ġfulfilled Ġby Ġrelying Ġonly Ġon Ġword Ġpairs Ġthat Ġalready Ġunderwent Ġmeticulous Ġsampling Ġprocesses Ġin Ġprior Ġwork , Ġintegrating Ġthem Ġinto Ġa Ġsingle Ġresource . ĠAs Ġa Ġconsequence , ĠMulti - Sim Lex Ġallows Ġfor Ġfine - gr ained Ġanalyses Ġover Ġdifferent ĠPOS Ġclasses , Ġconc ret eness Ġlevels , Ġsimilarity Ġspect ra , Ġfrequency Ġintervals , Ġrelation Ġtypes , Ġmorphology , Ġlex ical Ġfields , Ġand Ġit Ġalso Ġincludes Ġsome Ġchallenging Ġorth ographically Ġsimilar Ġexamples Ġ( e . g ., Ġinfection ĠâĢĵ Ġinf lection ). ĠWe Ġensure Ġthat Ġthe Ġcriteria ĠC 2 Ġand ĠC 3 Ġare Ġsatisfied Ġby Ġusing Ġsimilar Ġannotation Ġguidelines Ġas ĠSim lex - 999 , ĠSim Ver b - 35 00 , Ġand ĠSEM EV AL - 500 Ġthat Ġexplicitly Ġtarget Ġsemantic Ġsimilarity . ĠIn Ġwhat Ġfollows , Ġwe Ġoutline Ġthe Ġcarefully Ġtailored Ġprocess Ġof Ġtranslating Ġand Ġannot ating ĠMulti - Sim Lex Ġdatasets Ġin Ġall Ġtarget Ġlanguages . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation </s> We Ġnow Ġdetail Ġthe Ġdevelopment Ġof Ġthe Ġfinal ĠMulti - Sim Lex Ġresource , Ġdescribing Ġour Ġlanguage Ġselection Ġprocess , Ġas Ġwell Ġas Ġtranslation Ġand Ġannotation Ġof Ġthe Ġresource , Ġincluding Ġthe Ġsteps Ġtaken Ġto Ġensure Ġand Ġmeasure Ġthe Ġquality Ġof Ġthis Ġresource . ĠWe Ġalso Ġprovide Ġkey Ġdata Ġstatistics Ġand Ġpreliminary Ġcross - ling ual Ġcomparative Ġanalyses . </s> Language ĠSelection . ĠMulti - Sim Lex Ġcomprises Ġeleven Ġlanguages Ġin Ġaddition Ġto ĠEnglish . ĠThe Ġmain Ġobjective Ġfor Ġour Ġinclusion Ġcriteria Ġhas Ġbeen Ġto Ġbalance Ġlanguage Ġprominence Ġ( by Ġnumber Ġof Ġspeakers Ġof Ġthe Ġlanguage ) Ġfor Ġmaximum Ġimpact Ġof Ġthe Ġresource , Ġwhile Ġsimultaneously Ġhaving Ġa Ġdiverse Ġsuite Ġof Ġlanguages Ġbased Ġon Ġtheir Ġtyp ological Ġfeatures Ġ( such Ġas Ġmorph ological Ġtype Ġand Ġlanguage Ġfamily ). ĠTable ĠT AB REF 10 Ġsummarizes Ġkey Ġinformation Ġabout Ġthe Ġlanguages Ġcurrently Ġincluded Ġin ĠMulti - Sim Lex . ĠWe Ġhave Ġincluded Ġa Ġmixture Ġof Ġf usional , Ġagg l ut in ative , Ġisol ating , Ġand Ġintro flex ive Ġlanguages Ġthat Ġcome Ġfrom Ġeight Ġdifferent Ġlanguage Ġfamilies . ĠThis Ġincludes Ġlanguages Ġthat Ġare Ġvery Ġwidely Ġused Ġsuch Ġas ĠChinese ĠMandarin Ġand ĠSpanish , Ġand Ġlow - resource Ġlanguages Ġsuch Ġas ĠWelsh Ġand ĠKis w ah ili . ĠWe Ġhope Ġto Ġfurther Ġinclude Ġadditional Ġlanguages Ġand Ġinspire Ġother Ġresearchers Ġto Ġcontribute Ġto Ġthe Ġeffort Ġover Ġthe Ġlifetime Ġof Ġthis Ġproject . </s> The Ġwork Ġon Ġdata Ġcollection Ġcan Ġbe Ġdivided Ġinto Ġtwo Ġcrucial Ġphases : Ġ1 ) Ġa Ġtranslation Ġphase Ġwhere Ġthe Ġextended ĠEnglish Ġlanguage Ġdataset Ġwith Ġ1 , 888 Ġpairs Ġ( described Ġin ĠÂ§ SEC REF 4 ) Ġis Ġtranslated Ġinto Ġeleven Ġtarget Ġlanguages , Ġand Ġ2 ) Ġan Ġannotation Ġphase Ġwhere Ġhuman Ġrat ers Ġscored Ġeach Ġpair Ġin Ġthe Ġtranslated Ġset Ġas Ġwell Ġas Ġthe ĠEnglish Ġset . ĠDet ailed Ġguidelines Ġfor Ġboth Ġphases Ġare Ġavailable Ġonline Ġat : Ġhttps :// mult is im lex . com . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠWord ĠPair ĠTranslation </s> Trans l ators Ġfor Ġeach Ġtarget Ġlanguage Ġwere Ġinstructed Ġto Ġfind Ġdirect Ġor Ġapproximate Ġtranslations Ġfor Ġthe Ġ1 , 888 Ġword Ġpairs Ġthat Ġsatisfy Ġthe Ġfollowing Ġrules . Ġ( 1 ) ĠAll Ġpairs Ġin Ġthe Ġtranslated Ġset Ġmust Ġbe Ġunique Ġ( i . e ., Ġno Ġduplicate Ġpairs ); Ġ( 2 ) ĠTrans l ating Ġtwo Ġwords Ġfrom Ġthe Ġsame ĠEnglish Ġpair Ġinto Ġthe Ġsame Ġword Ġin Ġthe Ġtarget Ġlanguage Ġis Ġnot Ġallowed Ġ( e . g ., Ġit Ġis Ġnot Ġallowed Ġto Ġtranslate Ġcar Ġand Ġautomobile Ġto Ġthe Ġsame ĠSpanish Ġword Ġcoc he ). Ġ( 3 ) ĠThe Ġtranslated Ġpairs Ġmust Ġpreserve Ġthe Ġsemantic Ġrelations Ġbetween Ġthe Ġtwo Ġwords Ġwhen Ġpossible . ĠThis Ġmeans Ġthat , Ġwhen Ġmultiple Ġtranslations Ġare Ġpossible , Ġthe Ġtranslation Ġthat Ġbest Ġconve ys Ġthe Ġsemantic Ġrelation Ġbetween Ġthe Ġtwo Ġwords Ġfound Ġin Ġthe Ġoriginal ĠEnglish Ġpair Ġis Ġselected . Ġ( 4 ) ĠIf Ġit Ġis Ġnot Ġpossible Ġto Ġuse Ġa Ġsingle - word Ġtranslation Ġin Ġthe Ġtarget Ġlanguage , Ġthen Ġa Ġmulti - word Ġexpression Ġ( M WE ) Ġcan Ġbe Ġused Ġto Ġconvey Ġthe Ġnearest Ġpossible Ġsemantics Ġgiven Ġthe Ġabove Ġpoints Ġ( e . g ., Ġthe ĠEnglish Ġword Ġhomework Ġis Ġtranslated Ġinto Ġthe ĠPolish ĠM WE Ġpr aca Ġdom owa ). </s> S atisf ying Ġthe Ġabove Ġrules Ġwhen Ġfinding Ġappropriate Ġtranslations Ġfor Ġeach Ġpair âĢĵ while Ġkeeping Ġto Ġthe Ġspirit Ġof Ġthe Ġintended Ġsemantic Ġrelation Ġin Ġthe ĠEnglish Ġversion âĢĵ is Ġnot Ġalways Ġstraightforward . ĠFor Ġinstance , Ġkins hip Ġterminology Ġin ĠSin itic Ġlanguages Ġ( Mand arin Ġand ĠYue ) Ġuses Ġdifferent Ġterms Ġdepending Ġon Ġwhether Ġthe Ġfamily Ġmember Ġis Ġolder Ġor Ġyounger , Ġand Ġwhether Ġthe Ġfamily Ġmember Ġcomes Ġfrom Ġthe Ġmother âĢ Ļ s Ġside Ġor Ġthe Ġfather âĢ Ļ s Ġside . ĠUTF 8 g bs n ĠIn ĠMandarin , Ġbrother Ġhas Ġno Ġdirect Ġtranslation Ġand Ġcan Ġbe Ġtranslated Ġas Ġeither : Ġå ĵ ¥ å ĵ ¥ Ġ( older Ġbrother ) Ġor Ġå ¼ Ł å¼ Ł Ġ( young er Ġbrother ). ĠTherefore , Ġin Ġsuch Ġcases , Ġthe Ġtransl ators Ġare Ġasked Ġto Ġchoose Ġthe Ġbest Ġoption Ġgiven Ġthe Ġsemantic Ġcontext Ġ( relation ) Ġexpressed Ġby Ġthe Ġpair Ġin ĠEnglish , Ġotherwise Ġselect Ġone Ġof Ġthe Ġtranslations Ġarbitrarily . ĠThis Ġis Ġalso Ġused Ġto Ġremove Ġduplicate Ġpairs Ġin Ġthe Ġtranslated Ġset , Ġby Ġdifferent iating Ġthe Ġdupl icates Ġusing Ġa Ġvariant Ġat Ġeach Ġinstance . ĠFurther , Ġmany Ġtranslation Ġinstances Ġwere Ġresolved Ġusing Ġnear - syn onymous Ġterms Ġin Ġthe Ġtranslation . ĠFor Ġexample , Ġthe Ġwords Ġin Ġthe Ġpair : Ġwood ĠâĢĵ Ġtimber Ġcan Ġonly Ġbe Ġdirectly Ġtranslated Ġin ĠEston ian Ġto Ġp uit , Ġand Ġare Ġnot Ġdistinguish able . ĠTherefore , Ġthe Ġtransl ators Ġapprox imated Ġthe Ġtranslation Ġfor Ġtimber Ġto Ġthe Ġcompound Ġnoun Ġp uit m ater j al Ġ( literally : Ġwood Ġmaterial ) Ġin Ġorder Ġto Ġproduce Ġa Ġvalid Ġpair Ġin Ġthe Ġtarget Ġlanguage . ĠIn Ġsome Ġcases , Ġa Ġdirect Ġtransl iter ation Ġfrom ĠEnglish Ġis Ġused . ĠFor Ġexample , Ġthe Ġpair : Ġphysician Ġand Ġdoctor Ġboth Ġtranslate Ġto Ġthe Ġsame Ġword Ġin ĠEston ian Ġ( ar st ); Ġthe Ġless Ġformal Ġword Ġdo k tor Ġis Ġused Ġas Ġa Ġtranslation Ġof Ġdoctor Ġto Ġgenerate Ġa Ġvalid Ġpair . </s> We Ġmeasure Ġthe Ġquality Ġof Ġthe Ġtranslated Ġpairs Ġby Ġusing Ġa Ġrandom Ġsample Ġset Ġof Ġ100 Ġpairs Ġ( from Ġthe Ġ1 , 888 Ġpairs ) Ġto Ġbe Ġtranslated Ġby Ġan Ġindependent Ġtranslator Ġfor Ġeach Ġtarget Ġlanguage . ĠThe Ġsample Ġis Ġproportion ally Ġstrat ified Ġaccording Ġto Ġthe Ġpart - of - speech Ġcategories . ĠThe Ġindependent Ġtranslator Ġis Ġgiven Ġidentical Ġinstructions Ġto Ġthe Ġmain Ġtranslator ; Ġwe Ġthen Ġmeasure Ġthe Ġpercentage Ġof Ġmatched Ġtranslated Ġwords Ġbetween Ġthe Ġtwo Ġtranslations Ġof Ġthe Ġsample Ġset . ĠTable ĠT AB REF 12 Ġsummarizes Ġthe Ġinter - trans lator Ġagreement Ġresults Ġfor Ġall Ġlanguages Ġand Ġby Ġpart - of - speech Ġsubs ets . ĠOverall Ġacross Ġall Ġlanguages , Ġthe Ġagreement Ġis Ġ84 . 8 %, Ġwhich Ġis Ġsimilar Ġto Ġprior Ġwork ĠB IB REF 27 , ĠB IB REF 97 . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠGuidelines Ġand ĠWord ĠPair ĠSc oring </s> Across Ġall Ġlanguages , Ġ145 Ġhuman Ġannot ators Ġwere Ġasked Ġto Ġscore Ġall Ġ1 , 888 Ġpairs Ġ( in Ġtheir Ġgiven Ġlanguage ). ĠWe Ġfinally Ġcollect Ġat Ġleast Ġten Ġvalid Ġannotations Ġfor Ġeach Ġword Ġpair Ġin Ġeach Ġlanguage . ĠAll Ġannot ators Ġwere Ġrequired Ġto Ġabide Ġby Ġthe Ġfollowing Ġinstructions : </s> 1 . ĠEach Ġannot ator Ġmust Ġassign Ġan Ġinteger Ġscore Ġbetween Ġ0 Ġand Ġ6 Ġ( in clusive ) Ġindicating Ġhow Ġsem antically Ġsimilar Ġthe Ġtwo Ġwords Ġin Ġa Ġgiven Ġpair Ġare . ĠA Ġscore Ġof Ġ6 Ġindicates Ġvery Ġhigh Ġsimilarity Ġ( i . e ., Ġperfect Ġsyn ony my ), Ġwhile Ġzero Ġindicates Ġno Ġsimilarity . </s> 2 . ĠEach Ġannot ator Ġmust Ġscore Ġthe Ġentire Ġset Ġof Ġ1 , 888 Ġpairs Ġin Ġthe Ġdataset . ĠThe Ġpairs Ġmust Ġnot Ġbe Ġshared Ġbetween Ġdifferent Ġannot ators . </s> 3 . ĠAn not ators Ġare Ġable Ġto Ġbreak Ġthe Ġworkload Ġover Ġa Ġperiod Ġof Ġapproximately Ġ2 - 3 Ġweeks , Ġand Ġare Ġable Ġto Ġuse Ġexternal Ġsources Ġ( e . g . Ġdiction aries , Ġthe s auri , ĠWord Net ) Ġif Ġrequired . </s> 4 . ĠAn not ators Ġare Ġkept Ġanonymous , Ġand Ġare Ġnot Ġable Ġto Ġcommunicate Ġwith Ġeach Ġother Ġduring Ġthe Ġannotation Ġprocess . </s> The Ġselection Ġcriteria Ġfor Ġthe Ġannot ators Ġrequired Ġthat Ġall Ġannot ators Ġmust Ġbe Ġnative Ġspeakers Ġof Ġthe Ġtarget Ġlanguage . ĠPre ference Ġto Ġannot ators Ġwith Ġuniversity Ġeducation Ġwas Ġgiven , Ġbut Ġnot Ġrequired . ĠAn not ators Ġwere Ġasked Ġto Ġcomplete Ġa Ġspreadsheet Ġcontaining Ġthe Ġtranslated Ġpairs Ġof Ġwords , Ġas Ġwell Ġas Ġthe Ġpart - of - speech , Ġand Ġa Ġcolumn Ġto Ġenter Ġthe Ġscore . ĠThe Ġannot ators Ġdid Ġnot Ġhave Ġaccess Ġto Ġthe Ġoriginal Ġpairs Ġin ĠEnglish . </s> To Ġensure Ġthe Ġquality Ġof Ġthe Ġcollected Ġratings , Ġwe Ġhave Ġemployed Ġan Ġadjud ication Ġprotocol Ġsimilar Ġto Ġthe Ġone Ġproposed Ġand Ġvalidated Ġby ĠP ile h var : 2018 em nl p . ĠIt Ġconsists Ġof Ġthe Ġfollowing Ġthree Ġrounds : </s> Round Ġ1 : ĠAll Ġannot ators Ġare Ġasked Ġto Ġfollow Ġthe Ġinstructions Ġoutlined Ġabove , Ġand Ġto Ġrate Ġall Ġ1 , 888 Ġpairs Ġwith Ġinteger Ġscores Ġbetween Ġ0 Ġand Ġ6 . </s> Round Ġ2 : ĠWe Ġcompare Ġthe Ġscores Ġof Ġall Ġannot ators Ġand Ġidentify Ġthe Ġpairs Ġfor Ġeach Ġannot ator Ġthat Ġhave Ġshown Ġthe Ġmost Ġdisagreement . ĠWe Ġask Ġthe Ġannot ators Ġto Ġreconsider Ġthe Ġassigned Ġscores Ġfor Ġthose Ġpairs Ġonly . ĠThe Ġannot ators Ġmay Ġchose Ġto Ġeither Ġchange Ġor Ġkeep Ġthe Ġscores . ĠAs Ġin Ġthe Ġcase Ġwith ĠRound Ġ1 , Ġthe Ġannot ators Ġhave Ġno Ġaccess Ġto Ġthe Ġscores Ġof Ġthe Ġother Ġannot ators , Ġand Ġthe Ġprocess Ġis Ġanonymous . ĠThis Ġprocess Ġgives Ġa Ġchance Ġfor Ġannot ators Ġto Ġcorrect Ġerrors Ġor Ġreconsider Ġtheir Ġjudgments , Ġand Ġhas Ġbeen Ġshown Ġto Ġbe Ġvery Ġeffective Ġin Ġreaching Ġconsensus , Ġas Ġreported Ġby ĠB IB REF 78 . ĠWe Ġused Ġa Ġvery Ġsimilar Ġprocedure Ġas ĠB IB REF 78 Ġto Ġidentify Ġthe Ġpairs Ġwith Ġthe Ġmost Ġdisagreement ; Ġfor Ġeach Ġannot ator , Ġwe Ġmarked Ġthe Ġ$ i $ th Ġpair Ġif Ġthe Ġrated Ġscore Ġ$ s _ i $ Ġfalls Ġwithin : Ġ$ s _ i Ġ\ ge Ġ\ mu Ġ_ i Ġ+ Ġ1 . 5 $ Ġor Ġ$ s _ i Ġ\ le Ġ\ mu Ġ_ i Ġ- Ġ1 . 5 $, Ġwhere Ġ$\ mu Ġ_ i $ Ġis Ġthe Ġmean Ġof Ġthe Ġother Ġannot ators ' Ġscores . </s> Round Ġ3 : ĠWe Ġcompute Ġthe Ġaverage Ġagreement Ġfor Ġeach Ġannot ator Ġ( with Ġthe Ġother Ġannot ators ), Ġby Ġmeasuring Ġthe Ġaverage ĠSpear man 's Ġcorrelation Ġagainst Ġall Ġother Ġannot ators . ĠWe Ġdiscard Ġthe Ġscores Ġof Ġannot ators Ġthat Ġhave Ġshown Ġthe Ġleast Ġaverage Ġagreement Ġwith Ġall Ġother Ġannot ators , Ġwhile Ġwe Ġmaintain Ġat Ġleast Ġten Ġannot ators Ġper Ġlanguage Ġby Ġthe Ġend Ġof Ġthis Ġround . ĠThe Ġactual Ġprocess Ġis Ġdone Ġin Ġmultiple Ġiterations : Ġ( S 1 ) Ġwe Ġmeasure Ġthe Ġaverage Ġagreement Ġfor Ġeach Ġannot ator Ġwith Ġevery Ġother Ġannot ator Ġ( this Ġcorresponds Ġto Ġthe ĠAP IA A Ġmeasure , Ġsee Ġlater ); Ġ( S 2 ) Ġif Ġwe Ġstill Ġhave Ġmore Ġthan Ġ10 Ġvalid Ġannot ators Ġand Ġthe Ġlowest Ġaverage Ġscore Ġis Ġhigher Ġthan Ġin Ġthe Ġprevious Ġiteration , Ġwe Ġremove Ġthe Ġlowest Ġone , Ġand Ġre run ĠS 1 . ĠTable ĠT AB REF 14 Ġshows Ġthe Ġnumber Ġof Ġannot ators Ġat Ġboth Ġthe Ġstart Ġ( Round Ġ1 ) Ġand Ġend Ġ( Round Ġ3 ) Ġof Ġour Ġprocess Ġfor Ġeach Ġlanguage . </s> We Ġmeasure Ġthe Ġagreement Ġbetween Ġannot ators Ġusing Ġtwo Ġmetrics , Ġaverage Ġpair wise Ġinter - annot ator Ġagreement Ġ( AP IA A ), Ġand Ġaverage Ġmean Ġinter - annot ator Ġagreement Ġ( AM IA A ). ĠBoth Ġof Ġthese Ġuse ĠSpear man 's Ġcorrelation Ġ($ \ r ho Ġ$ ) Ġbetween Ġannot ators Ġscores , Ġthe Ġonly Ġdifference Ġis Ġhow Ġthey Ġare Ġaveraged . ĠThey Ġare Ġcomputed Ġas Ġfollows : </s> where Ġ$\ r ho Ġ( s _ i , s _ j ) $ Ġis Ġthe ĠSpear man 's Ġcorrelation Ġbetween Ġannot ators Ġ$ i $ Ġand Ġ$ j $ ' s Ġscores Ġ($ s _ i $, $ s _ j $ ) Ġfor Ġall Ġpairs Ġin Ġthe Ġdataset , Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġannot ators . ĠAP IA A Ġhas Ġbeen Ġused Ġwidely Ġas Ġthe Ġstandard Ġmeasure Ġfor Ġinter - annot ator Ġagreement , Ġincluding Ġin Ġthe Ġoriginal ĠSim Lex Ġpaper ĠB IB REF 14 . ĠIt Ġsimply Ġaverages Ġthe Ġpair wise ĠSpear man 's Ġcorrelation Ġbetween Ġall Ġannot ators . ĠOn Ġthe Ġother Ġhand , ĠAM IA A Ġcompares Ġthe Ġaverage ĠSpear man 's Ġcorrelation Ġof Ġone Ġheld - out Ġannot ator Ġwith Ġthe Ġaverage Ġof Ġall Ġthe Ġother Ġ$ N - 1 $ Ġannot ators , Ġand Ġthen Ġaverages Ġacross Ġall Ġ$ N $ Ġ` held - out ' Ġannot ators . ĠIt Ġsmooth s Ġindividual Ġannot ator Ġeffects Ġand Ġarguably Ġserves Ġas Ġa Ġbetter Ġupper Ġbound Ġthan ĠAP IA A ĠB IB REF 15 , ĠB IB REF 91 , ĠB IB REF 78 . </s> We Ġpresent Ġthe Ġrespective ĠAP IA A Ġand ĠAM IA A Ġscores Ġin ĠTable ĠT AB REF 16 Ġand ĠTable ĠT AB REF 17 Ġfor Ġall Ġpart - of - speech Ġsubs ets , Ġas Ġwell Ġas Ġthe Ġagreement Ġfor Ġthe Ġfull Ġdatasets . ĠAs Ġreported Ġin Ġprior Ġwork ĠB IB REF 15 , ĠB IB REF 91 , ĠAM IA A Ġscores Ġare Ġtypically Ġhigher Ġthan ĠAP IA A Ġscores . ĠCru cially , Ġthe Ġresults Ġindicate Ġ` strong Ġagreement ' Ġ( ac ross Ġall Ġlanguages ) Ġusing Ġboth Ġmeasurements . ĠThe Ġlanguages Ġwith Ġthe Ġhighest Ġannot ator Ġagreement Ġwere ĠFrench Ġ( f ra ) Ġand ĠYue ĠChinese Ġ( y ue ), Ġwhile ĠRussian Ġ( rus ) Ġhad Ġthe Ġlowest Ġoverall ĠI AA Ġscores . ĠThese Ġscores , Ġhowever , Ġare Ġstill Ġconsidered Ġto Ġbe Ġ` mod er ately Ġstrong Ġagreement '. </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠData ĠAnalysis </s> Similar ity ĠScore ĠDistribut ions . ĠAcross Ġall Ġlanguages , Ġthe Ġaverage Ġscore Ġ( mean Ġ$ = 1 . 61 $, Ġmedian $ = 1 . 1 $ ) Ġis Ġon Ġthe Ġlower Ġside Ġof Ġthe Ġsimilarity Ġscale . ĠHowever , Ġlooking Ġcloser Ġat Ġthe Ġscores Ġof Ġeach Ġlanguage Ġin ĠTable ĠT AB REF 19 , Ġwe Ġindicate Ġnotable Ġdifferences Ġin Ġboth Ġthe Ġaverages Ġand Ġthe Ġspread Ġof Ġscores . ĠNotably , ĠFrench Ġhas Ġthe Ġhighest Ġaverage Ġof Ġsimilarity Ġscores Ġ( mean $ = 2 . 61 $, Ġmedian $ = 2 . 5 $ ), Ġwhile ĠKis w ah ili Ġhas Ġthe Ġlowest Ġaverage Ġ( mean $ = 1 . 28 $, Ġmedian $ = 0 . 5 $ ). ĠRussian Ġhas Ġthe Ġlowest Ġspread Ġ($ \ s igma Ġ= 1 . 37 $ ), Ġwhile ĠPolish Ġhas Ġthe Ġlargest Ġ($ \ s igma Ġ= 1 . 62 $ ). ĠAll Ġof Ġthe Ġlanguages Ġare Ġstrongly Ġcorrelated Ġwith Ġeach Ġother , Ġas Ġshown Ġin ĠFigure ĠFIG REF 20 , Ġwhere Ġall Ġof Ġthe ĠSpear man 's Ġcorrelation Ġcoefficients Ġare Ġgreater Ġthan Ġ0 . 6 Ġfor Ġall Ġlanguage Ġpairs . ĠLanguages Ġthat Ġshare Ġthe Ġsame Ġlanguage Ġfamily Ġare Ġhighly Ġcorrelated Ġ( e . g , Ġc mn - y ue , Ġr us - pol , Ġest - fin ). ĠIn Ġaddition , Ġwe Ġobserve Ġhigh Ġcorrelations Ġbetween ĠEnglish Ġand Ġmost Ġother Ġlanguages , Ġas Ġexpected . ĠThis Ġis Ġdue Ġto Ġthe Ġeffect Ġof Ġusing ĠEnglish Ġas Ġthe Ġbase / anch or Ġlanguage Ġto Ġcreate Ġthe Ġdataset . ĠIn Ġsimple Ġwords , Ġif Ġone Ġtranslates Ġto Ġtwo Ġlanguages Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġstarting Ġfrom Ġthe Ġsame Ġset Ġof Ġpairs Ġin ĠEnglish , Ġit Ġis Ġhig ly Ġlikely Ġthat Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġwill Ġdiver ge Ġfrom ĠEnglish Ġin Ġdifferent Ġways . ĠTherefore , Ġthe Ġsimilarity Ġbetween Ġ$ L _ 1 $ - eng Ġand Ġ$ L _ 2 $ - eng Ġis Ġexpected Ġto Ġbe Ġhigher Ġthan Ġbetween Ġ$ L _ 1 $ -$ L _ 2 $, Ġespecially Ġif Ġ$ L _ 1 $ Ġand Ġ$ L _ 2 $ Ġare Ġtyp ologically Ġdiss imilar Ġlanguages Ġ( e . g ., Ġhe b - c mn , Ġsee ĠFigure ĠFIG REF 20 ). ĠThis Ġphenomenon Ġis Ġwell Ġdocumented Ġin Ġrelated Ġprior Ġwork ĠB IB REF 25 , ĠB IB REF 27 , ĠB IB REF 16 , ĠB IB REF 97 . ĠWhile Ġwe Ġacknowledge Ġthis Ġas Ġa Ġslight Ġartifact Ġof Ġthe Ġdataset Ġdesign , Ġit Ġwould Ġotherwise Ġbe Ġimpossible Ġto Ġconstruct Ġa Ġsem antically Ġaligned Ġand Ġcomprehensive Ġdataset Ġacross Ġa Ġlarge Ġnumber Ġof Ġlanguages . </s> We Ġalso Ġreport Ġdifferences Ġin Ġthe Ġdistribution Ġof Ġthe Ġfrequency Ġof Ġwords Ġamong Ġthe Ġlanguages Ġin ĠMulti - Sim Lex . ĠFigure ĠFIG REF 22 Ġshows Ġsix Ġexample Ġlanguages , Ġwhere Ġeach Ġbar Ġsegment Ġshows Ġthe Ġproportion Ġof Ġwords Ġin Ġeach Ġlanguage Ġthat Ġoccur Ġin Ġthe Ġgiven Ġfrequency Ġrange . ĠFor Ġexample , Ġthe Ġ10 K - 20 K Ġsegment Ġof Ġthe Ġbars Ġrepresents Ġthe Ġproportion Ġof Ġwords Ġin Ġthe Ġdataset Ġthat Ġoccur Ġin Ġthe Ġlist Ġof Ġmost Ġfrequent Ġwords Ġbetween Ġthe Ġfrequency Ġrank Ġof Ġ10 , 000 Ġand Ġ20 , 000 Ġin Ġthat Ġlanguage ; Ġlikewise Ġwith Ġother Ġintervals . ĠFrequency Ġlists Ġfor Ġthe Ġpresented Ġlanguages Ġare Ġderived Ġfrom ĠWikipedia Ġand ĠCommon ĠC rawl Ġcorpor a . ĠWhile Ġmany Ġconcept Ġpairs Ġare Ġdirect Ġor Ġapproximate Ġtranslations Ġof ĠEnglish Ġpairs , Ġwe Ġcan Ġsee Ġthat Ġthe Ġfrequency Ġdistribution Ġdoes Ġvary Ġacross Ġdifferent Ġlanguages , Ġand Ġis Ġalso Ġrelated Ġto Ġinherent Ġlanguage Ġproperties . ĠFor Ġinstance , Ġin ĠFinnish Ġand ĠRussian , Ġwhile Ġwe Ġuse Ġinf in itive Ġforms Ġof Ġall Ġverbs , Ġconj ug ated Ġverb Ġinf lections Ġare Ġoften Ġmore Ġfrequent Ġin Ġraw Ġcorpor a Ġthan Ġthe Ġcorresponding Ġinf in itive Ġforms . ĠThe Ġvariance Ġcan Ġalso Ġbe Ġpartially Ġexplained Ġby Ġthe Ġdifference Ġin Ġmon oling ual Ġcorpor a Ġsize Ġused Ġto Ġderive Ġthe Ġfrequency Ġrankings Ġin Ġthe Ġfirst Ġplace : Ġabsolute Ġvocabulary Ġsizes Ġare Ġexpected Ġto Ġfluct uate Ġacross Ġdifferent Ġlanguages . ĠHowever , Ġit Ġis Ġalso Ġimportant Ġto Ġnote Ġthat Ġthe Ġdatasets Ġalso Ġcontain Ġsubs ets Ġof Ġlower - frequency Ġand Ġrare Ġwords , Ġwhich Ġcan Ġbe Ġused Ġfor Ġrare Ġword Ġevaluations Ġin Ġmultiple Ġlanguages , Ġin Ġthe Ġspirit Ġof ĠP ile h var : 2018 em nl p 's ĠEnglish Ġrare Ġword Ġdataset . </s> Cross - L ingu istic ĠDifferences . ĠTable ĠT AB REF 23 Ġshows Ġsome Ġexamples Ġof Ġaverage Ġsimilarity Ġscores Ġof ĠEnglish , ĠSpanish , ĠKis w ah ili Ġand ĠWelsh Ġconcept Ġpairs . ĠRemember Ġthat Ġthe Ġscores Ġrange Ġfrom Ġ0 Ġto Ġ6 : Ġthe Ġhigher Ġthe Ġscore , Ġthe Ġmore Ġsimilar Ġthe Ġparticipants Ġfound Ġthe Ġconcepts Ġin Ġthe Ġpair . ĠThe Ġexamples Ġfrom ĠTable ĠT AB REF 23 Ġshow Ġevidence Ġof Ġboth Ġthe Ġstability Ġof Ġaverage Ġsimilarity Ġscores Ġacross Ġlanguages Ġ( un likely ĠâĢĵ Ġfriendly , Ġbook ĠâĢĵ Ġliterature , Ġand Ġvanish ĠâĢĵ Ġdisappear ), Ġas Ġwell Ġas Ġlanguage - specific Ġdifferences Ġ( care ĠâĢĵ Ġcaution ). ĠSome Ġdifferences Ġin Ġsimilarity Ġscores Ġseem Ġto Ġgroup Ġlanguages Ġinto Ġclusters . ĠFor Ġexample , Ġthe Ġword Ġpair Ġregular ĠâĢĵ Ġaverage Ġhas Ġan Ġaverage Ġsimilarity Ġscore Ġof Ġ4 . 0 Ġand Ġ4 . 1 Ġin ĠEnglish Ġand ĠSpanish , Ġrespectively , Ġwhereas Ġin ĠKis w ah ili Ġand ĠWelsh Ġthe Ġaverage Ġsimilarity Ġscore Ġof Ġthis Ġpair Ġis Ġ0 . 5 Ġand Ġ0 . 8 . ĠWe Ġanalyze Ġthis Ġphenomenon Ġin Ġmore Ġdetail Ġin ĠÂ§ SEC REF 25 . </s> There Ġare Ġalso Ġexamples Ġfor Ġeach Ġof Ġthe Ġfour Ġlanguages Ġhaving Ġa Ġnotably Ġhigher Ġor Ġlower Ġsimilarity Ġscore Ġfor Ġthe Ġsame Ġconcept Ġpair Ġthan Ġthe Ġthree Ġother Ġlanguages . ĠFor Ġexample , Ġlarge ĠâĢĵ Ġbig Ġin ĠEnglish Ġhas Ġan Ġaverage Ġsimilarity Ġscore Ġof Ġ5 . 9 , Ġwhereas ĠSpanish , ĠKis w ah ili Ġand ĠWelsh Ġspeakers Ġrate Ġthe Ġclosest Ġconcept Ġpair Ġin Ġtheir Ġnative Ġlanguage Ġto Ġhave Ġa Ġsimilarity Ġscore Ġbetween Ġ2 . 7 Ġand Ġ3 . 8 . ĠWhat Ġis Ġmore , Ġwoman ĠâĢĵ Ġwife Ġreceives Ġan Ġaverage Ġsimilarity Ġof Ġ0 . 9 Ġin ĠEnglish , Ġ2 . 9 Ġin ĠSpanish , Ġand Ġgreater Ġthan Ġ4 . 0 Ġin ĠKis w ah ili Ġand ĠWelsh . ĠThe Ġexamples Ġfrom ĠSpanish Ġinclude Ġban co ĠâĢĵ Ġas ient o Ġ( bank ĠâĢĵ Ġseat ) Ġwhich Ġreceives Ġan Ġaverage Ġsimilarity Ġscore Ġ5 . 1 , Ġwhile Ġin Ġthe Ġother Ġthree Ġlanguages Ġthe Ġsimilarity Ġscore Ġfor Ġthis Ġword Ġpair Ġdoes Ġnot Ġexceed Ġ0 . 1 . ĠAt Ġthe Ġsame Ġtime , Ġthe Ġaverage Ġsimilarity Ġscore Ġof Ġesp ant os ament e ĠâĢĵ Ġfant Ã¡ stic ament e Ġ( am azing ly ĠâĢĵ Ġfant astically ) Ġis Ġmuch Ġlower Ġin ĠSpanish Ġ( 0 . 4 ) Ġthan Ġin Ġother Ġlanguages Ġ( 4 . 1 ĠâĢĵ Ġ5 . 1 ). ĠIn ĠKis w ah ili , Ġan Ġexample Ġof Ġa Ġword Ġpair Ġwith Ġa Ġhigher Ġsimilarity Ġscore Ġthan Ġthe Ġrest Ġwould Ġbe Ġmach we o ĠâĢĵ Ġj ion i Ġ( sun set ĠâĢĵ Ġevening ), Ġhaving Ġan Ġaverage Ġscore Ġof Ġ5 . 5 , Ġwhile Ġthe Ġother Ġlanguages Ġreceive Ġ2 . 8 Ġor Ġless , Ġand Ġa Ġnotably Ġlower Ġsimilarity Ġscore Ġis Ġgiven Ġto Ġwa Ġa jab u Ġ- Ġmk ub wa Ġs ana Ġ( w onder ful ĠâĢĵ Ġterrific ), Ġgetting Ġ0 . 9 , Ġwhile Ġthe Ġother Ġlanguages Ġreceive Ġ5 . 3 Ġor Ġmore . ĠWelsh Ġexamples Ġinclude Ġy n Ġll w yr Ġ- Ġy n Ġgy fan Ġg w bl Ġ( pure ly ĠâĢĵ Ġcompletely ), Ġwhich Ġscores Ġ5 . 4 Ġamong ĠWelsh Ġspeakers Ġbut Ġ2 . 3 Ġor Ġless Ġin Ġother Ġlanguages , Ġwhile Ġadd o ĠâĢĵ Ġty ng u Ġ( prom ise ĠâĢĵ Ġswear ) Ġis Ġrated Ġas Ġ0 Ġby Ġall ĠWelsh Ġannot ators , Ġbut Ġin Ġthe Ġother Ġthree Ġlanguages Ġ4 . 3 Ġor Ġmore Ġon Ġaverage . </s> There Ġcan Ġbe Ġseveral Ġexplanations Ġfor Ġthe Ġdifferences Ġin Ġsimilarity Ġscores Ġacross Ġlanguages , Ġincluding Ġbut Ġnot Ġlimited Ġto Ġcultural Ġcontext , Ġpoly se my , Ġmet ony my , Ġtranslation , Ġregional Ġand Ġgenerational Ġdifferences , Ġand Ġmost Ġcommonly , Ġthe Ġfact Ġthat Ġwords Ġand Ġmeanings Ġdo Ġnot Ġexactly Ġmap Ġonto Ġeach Ġother Ġacross Ġlanguages . ĠFor Ġexample , Ġit Ġis Ġlikely Ġthat Ġthe Ġother Ġthree Ġlanguages Ġdo Ġnot Ġhave Ġtwo Ġseparate Ġwords Ġfor Ġdescribing Ġthe Ġconcepts Ġin Ġthe Ġconcept Ġpair : Ġbig ĠâĢĵ Ġlarge , Ġand Ġthe Ġtransl ators Ġhad Ġto Ġopt Ġfor Ġsimilar Ġlex ical Ġitems Ġthat Ġwere Ġmore Ġdistant Ġin Ġmeaning , Ġexplaining Ġwhy Ġin ĠEnglish Ġthe Ġconcept Ġpair Ġreceived Ġa Ġmuch Ġhigher Ġaverage Ġsimilarity Ġscore Ġthan Ġin Ġother Ġlanguages . ĠA Ġsimilar Ġissue Ġrelated Ġto Ġthe Ġmapping Ġproblem Ġacross Ġlanguages Ġarose Ġin Ġthe ĠWelsh Ġconcept Ġpair Ġy n Ġll w ye ĠâĢĵ Ġy n Ġgy fan Ġg w bl , Ġwhere ĠWelsh Ġspeakers Ġagreed Ġthat Ġthe Ġtwo Ġconcepts Ġare Ġvery Ġsimilar . ĠWhen Ġasked , Ġbilingual Ġspeakers Ġconsidered Ġthe Ġtwo ĠWelsh Ġconcepts Ġmore Ġsimilar Ġthan ĠEnglish Ġequivalents Ġpurely ĠâĢĵ Ġcompletely , Ġpotentially Ġexplaining Ġwhy Ġa Ġhigher Ġaverage Ġsimilarity Ġscore Ġwas Ġreached Ġin ĠWelsh . ĠThe Ġexample Ġof Ġwoman ĠâĢĵ Ġwife Ġcan Ġillustrate Ġcultural Ġdifferences Ġor Ġanother Ġtranslation - related Ġissue Ġwhere Ġthe Ġword Ġ` wife ' Ġdid Ġnot Ġexist Ġin Ġsome Ġlanguages Ġ( for Ġexample , ĠEston ian ), Ġand Ġtherefore Ġhad Ġto Ġbe Ġdescribed Ġusing Ġother Ġwords , Ġaffecting Ġthe Ġcompar ability Ġof Ġthe Ġsimilarity Ġscores . ĠThis Ġwas Ġalso Ġthe Ġcase Ġwith Ġthe Ġfootball ĠâĢĵ Ġsoccer Ġconcept Ġpair . ĠThe Ġpair Ġbank ĠâĢĵ Ġseat Ġdemonstrates Ġthe Ġeffect Ġof Ġthe Ġpoly se my Ġmismatch Ġacross Ġlanguages : Ġwhile Ġ` bank ' Ġhas Ġtwo Ġdifferent Ġmeanings Ġin ĠEnglish , Ġneither Ġof Ġthem Ġis Ġsimilar Ġto Ġthe Ġword Ġ` seat ', Ġbut Ġin ĠSpanish , Ġ` ban co ' Ġcan Ġmean Ġ` bank ', Ġbut Ġit Ġcan Ġalso Ġmean Ġ` bench '. ĠQuite Ġnaturally , ĠSpanish Ġspeakers Ġgave Ġthe Ġpair Ġban co ĠâĢĵ Ġas ient o Ġa Ġhigher Ġsimilarity Ġscore Ġthan Ġthe Ġspeakers Ġof Ġlanguages Ġwhere Ġthis Ġpoly se my Ġdid Ġnot Ġoccur . </s> An Ġexample Ġof Ġmet ony my Ġaffecting Ġthe Ġaverage Ġsimilarity Ġscore Ġcan Ġbe Ġseen Ġin Ġthe ĠKis w ah ili Ġversion Ġof Ġthe Ġword Ġpair : Ġsunset ĠâĢĵ Ġevening Ġ( m ach we o ĠâĢĵ Ġj ion i ). ĠThe Ġaverage Ġsimilarity Ġscore Ġfor Ġthis Ġpair Ġis Ġmuch Ġhigher Ġin ĠKis w ah ili , Ġlikely Ġbecause Ġthe Ġword Ġ` sun set ' Ġcan Ġact Ġas Ġa Ġmet onym Ġof Ġ` even ing '. ĠThe Ġlow Ġsimilarity Ġscore Ġof Ġwonderful ĠâĢĵ Ġterrific Ġin ĠKis w ah ili Ġ( wa Ġa jab u Ġ- Ġmk ub wa Ġs ana ) Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġwhile Ġ` mk ub wa Ġs ana ' Ġcan Ġbe Ġused Ġas Ġ` ter r ific ' Ġin ĠKis w ah ili , Ġit Ġtechnically Ġmeans Ġ` very Ġbig ', Ġadding Ġto Ġthe Ġexamples Ġof Ġtranslation - Ġand Ġmapping - related Ġeffects . ĠThe Ġword Ġpair Ġamazingly ĠâĢĵ Ġfant astically Ġ( esp ant os ament e ĠâĢĵ Ġfant Ã¡ stic ament e ) Ġbrings Ġout Ġanother Ġtranslation - related Ġproblem : Ġthe Ġaccuracy Ġof Ġthe Ġtranslation . ĠWhile Ġ` esp ant os ament e ' Ġcould Ġarguably Ġbe Ġtranslated Ġto Ġ` am azing ly ', Ġmore Ġcommon Ġmeanings Ġinclude : Ġ` f right fully ', Ġ` ter r ifying ly ', Ġand Ġ` shock ingly ', Ġexplaining Ġwhy Ġthe Ġaverage Ġsimilarity Ġscore Ġdiffers Ġfrom Ġthe Ġrest Ġof Ġthe Ġlanguages . ĠAnother Ġproblem Ġwas Ġbrought Ġout Ġby Ġadd o ĠâĢĵ Ġty ng u Ġ( prom ise ĠâĢĵ Ġswear ) Ġin ĠWelsh , Ġwhere Ġthe Ġ` ty ng u ' Ġmay Ġnot Ġhave Ġbeen Ġa Ġcommonly Ġused Ġor Ġeven Ġa Ġknown Ġword Ġchoice Ġfor Ġannot ators , Ġpointing Ġout Ġpotential Ġregional Ġor Ġgenerational Ġdifferences Ġin Ġlanguage Ġuse . </s> Table ĠT AB REF 24 Ġpresents Ġexamples Ġof Ġconcept Ġpairs Ġfrom ĠEnglish , ĠSpanish , ĠKis w ah ili , Ġand ĠWelsh Ġon Ġwhich Ġthe Ġparticipants Ġagreed Ġthe Ġmost . ĠFor Ġexample , Ġin ĠEnglish Ġall Ġparticipants Ġrated Ġthe Ġsimilarity Ġof Ġtrial ĠâĢĵ Ġtest Ġto Ġbe Ġ4 Ġor Ġ5 . ĠIn ĠSpanish Ġand ĠWelsh , Ġall Ġparticipants Ġrated Ġstart ĠâĢĵ Ġbegin Ġto Ġcorrespond Ġto Ġa Ġscore Ġof Ġ5 Ġor Ġ6 . ĠIn ĠKis w ah ili , Ġmoney ĠâĢĵ Ġcash Ġreceived Ġa Ġsimilarity Ġrating Ġof Ġ6 Ġfrom Ġevery Ġparticipant . ĠWhile Ġthere Ġare Ġnumerous Ġexamples Ġof Ġconcept Ġpairs Ġin Ġthese Ġlanguages Ġwhere Ġthe Ġparticipants Ġagreed Ġon Ġa Ġsimilarity Ġscore Ġof Ġ4 Ġor Ġhigher , Ġit Ġis Ġworth Ġnoting Ġthat Ġnone Ġof Ġthese Ġlanguages Ġhad Ġa Ġsingle Ġpair Ġwhere Ġall Ġparticipants Ġagreed Ġon Ġeither Ġ1 - 2 , Ġ2 - 3 , Ġor Ġ3 - 4 Ġsimilarity Ġrating . ĠInterestingly , Ġin ĠEnglish Ġall Ġpairs Ġwhere Ġall Ġthe Ġparticipants Ġagreed Ġon Ġa Ġ5 - 6 Ġsimilarity Ġscore Ġwere Ġadject ives . </s> Multi - Sim Lex : ĠTranslation Ġand ĠAn notation Ġ: :: ĠEffect Ġof ĠLanguage ĠAff inity Ġon ĠSimilar ity ĠScores </s> Based Ġon Ġthe Ġanalysis Ġin ĠFigure ĠFIG REF 20 Ġand Ġinspecting Ġthe Ġanecdotal Ġexamples Ġin Ġthe Ġprevious Ġsection , Ġit Ġis Ġevident Ġthat Ġthe Ġcorrelation Ġbetween Ġsimilarity Ġscores Ġacross Ġlanguages Ġis Ġnot Ġrandom . ĠTo Ġcorrobor ate Ġthis Ġintuition , Ġwe Ġvisualize Ġthe Ġvectors Ġof Ġsimilarity Ġscores Ġfor Ġeach Ġsingle Ġlanguage Ġby Ġreducing Ġtheir Ġdimension ality Ġto Ġ2 Ġvia ĠPrincipal ĠComponent ĠAnalysis ĠB IB REF 98 . ĠThe Ġresulting Ġscatter Ġplot Ġin ĠFigure ĠFIG REF 26 Ġreveals Ġthat Ġlanguages Ġfrom Ġthe Ġsame Ġfamily Ġor Ġbranch Ġhave Ġsimilar Ġpatterns Ġin Ġthe Ġscores . ĠIn Ġparticular , ĠRussian Ġand ĠPolish Ġ( both ĠSlav ic ), ĠFinnish Ġand ĠEston ian Ġ( both ĠU ral ic ), ĠCant ones e Ġand ĠMandarin ĠChinese Ġ( both ĠSin itic ), Ġand ĠSpanish Ġand ĠFrench Ġ( both ĠRomance ) Ġare Ġall Ġneighbors . </s> In Ġorder Ġto Ġquantify Ġexactly Ġthe Ġeffect Ġof Ġlanguage Ġaffinity Ġon Ġthe Ġsimilarity Ġscores , Ġwe Ġrun Ġcorrelation Ġanalyses Ġbetween Ġthese Ġand Ġlanguage Ġfeatures . ĠIn Ġparticular , Ġwe Ġextract Ġfeature Ġvectors Ġfrom ĠURI EL ĠB IB REF 99 , Ġa Ġmassively Ġmult ilingual Ġtyp ological Ġdatabase Ġthat Ġcollects Ġand Ġnormal izes Ġinformation Ġcompiled Ġby Ġgrammar ians Ġand Ġfield Ġlingu ists Ġabout Ġthe Ġworld 's Ġlanguages . ĠIn Ġparticular , Ġwe Ġfocus Ġon Ġinformation Ġabout Ġgeography Ġ( the Ġareas Ġwhere Ġthe Ġlanguage Ġspeakers Ġare Ġconcentrated ), Ġfamily Ġ( the Ġphylogen etic Ġtree Ġeach Ġlanguage Ġbelongs Ġto ), Ġand Ġtyp ology Ġ( including Ġsyntax , Ġphon ological Ġinventory , Ġand Ġphon ology ). ĠMoreover , Ġwe Ġconsider Ġtyp ological Ġrepresentations Ġof Ġlanguages Ġthat Ġare Ġnot Ġmanually Ġcrafted Ġby Ġexperts , Ġbut Ġrather Ġlearned Ġfrom Ġtexts . ĠB IB REF 100 Ġproposed Ġto Ġconstruct Ġsuch Ġrepresentations Ġby Ġtraining Ġlanguage - ident ifying Ġvectors Ġend - to - end Ġas Ġpart Ġof Ġneural Ġmachine Ġtranslation Ġmodels . </s> The Ġvector Ġfor Ġsimilarity Ġjudgments Ġand Ġthe Ġvector Ġof Ġlinguistic Ġfeatures Ġfor Ġa Ġgiven Ġlanguage Ġhave Ġdifferent Ġdimension ality . ĠHence , Ġwe Ġfirst Ġconstruct Ġa Ġdistance Ġmatrix Ġfor Ġeach Ġvector Ġspace , Ġsuch Ġthat Ġboth Ġcolumns Ġand Ġrows Ġare Ġlanguage Ġindices , Ġand Ġeach Ġcell Ġvalue Ġis Ġthe Ġcos ine Ġdistance Ġbetween Ġthe Ġvectors Ġof Ġthe Ġcorresponding Ġlanguage Ġpair . ĠGiven Ġa Ġset Ġof ĠL Ġlanguages , Ġeach Ġresulting Ġmatrix Ġ$ S $ Ġhas Ġdimension ality Ġof Ġ$\ math bb Ġ{ R } ^{ | L | Ġ\ times Ġ| L | } $ Ġand Ġis Ġsymm etrical . ĠTo Ġestimate Ġthe Ġcorrelation Ġbetween Ġthe Ġmatrix Ġfor Ġsimilarity Ġjudgments Ġand Ġeach Ġof Ġthe Ġmat rices Ġfor Ġlinguistic Ġfeatures , Ġwe Ġrun Ġa ĠMant el Ġtest ĠB IB REF 101 , Ġa Ġnon - param etric Ġstatistical Ġtest Ġbased Ġon Ġmatrix Ġperm utations Ġthat Ġtakes Ġinto Ġaccount Ġinter - depend encies Ġamong Ġpair wise Ġdistances . </s> The Ġresults Ġof Ġthe ĠMant el Ġtest Ġreported Ġin ĠTable ĠFIG REF 26 Ġshow Ġthat Ġthere Ġexist Ġstatistically Ġsignificant Ġcorrelations Ġbetween Ġsimilarity Ġjudgments Ġand Ġgeography , Ġfamily , Ġand Ġsyntax , Ġgiven Ġthat Ġ$ p Ġ< Ġ0 . 05 $ Ġand Ġ$ z Ġ> Ġ1 . 96 $. ĠThe Ġcorrelation Ġcoefficient Ġis Ġparticularly Ġstrong Ġfor Ġgeography Ġ($ r Ġ= Ġ0 . 647 $ ) Ġand Ġsyntax Ġ($ r Ġ= Ġ0 . 649 $ ). ĠThe Ġformer Ġresult Ġis Ġintuitive , Ġbecause Ġlanguages Ġin Ġcontact Ġeasily Ġborrow Ġand Ġloan Ġlex ical Ġunits , Ġand Ġcultural Ġinteractions Ġmay Ġresult Ġin Ġsimilar Ġcognitive Ġcategor izations . ĠThe Ġresult Ġfor Ġsyntax , Ġinstead , Ġcannot Ġbe Ġexplained Ġso Ġeasily , Ġas Ġformal Ġproperties Ġof Ġlanguage Ġdo Ġnot Ġaffect Ġlex ical Ġsemantics . ĠInstead , Ġwe Ġconjecture Ġthat , Ġwhile Ġno Ġcausal Ġrelation Ġis Ġpresent , Ġboth Ġsynt actic Ġfeatures Ġand Ġsimilarity Ġjudgments Ġmight Ġbe Ġlinked Ġto Ġa Ġcommon Ġexplanatory Ġvariable Ġ( such Ġas Ġgeography ). ĠIn Ġfact , Ġseveral Ġsynt actic Ġproperties Ġare Ġnot Ġuniformly Ġspread Ġacross Ġthe Ġglobe . ĠFor Ġinstance , Ġverbs Ġwith ĠVerb âĢĵ Object âĢĵ Subject Ġword Ġorder Ġare Ġmostly Ġconcentrated Ġin ĠO ce ania ĠB IB REF 102 . ĠIn Ġturn , Ġgeographical Ġproximity Ġleads Ġto Ġsimilar Ġjudgment Ġpatterns , Ġas Ġmentioned Ġabove . ĠOn Ġthe Ġother Ġhand , Ġwe Ġfind Ġno Ġcorrelation Ġwith Ġphon ology Ġand Ġinventory , Ġas Ġexpected , Ġnor Ġwith Ġthe Ġbottom - up Ġtyp ological Ġfeatures Ġfrom ĠB IB REF 100 . </s> Cross - L ing ual ĠMulti - Sim Lex ĠDat as ets </s> A Ġcrucial Ġadvantage Ġof Ġhaving Ġsem antically Ġaligned Ġmon oling ual Ġdatasets Ġacross Ġdifferent Ġlanguages Ġis Ġthe Ġpotential Ġto Ġcreate Ġcross - ling ual Ġsemantic Ġsimilarity Ġdatasets . ĠSuch Ġdatasets Ġallow Ġfor Ġprobing Ġthe Ġquality Ġof Ġcross - ling ual Ġrepresentation Ġlearning Ġalgorithms ĠB IB REF 27 , ĠB IB REF 103 , ĠB IB REF 104 , ĠB IB REF 105 , ĠB IB REF 106 , ĠB IB REF 30 , ĠB IB REF 107 Ġas Ġan Ġintrinsic Ġevaluation Ġtask . ĠHowever , Ġthe Ġcross - ling ual Ġdatasets Ġprevious Ġwork Ġrelied Ġupon ĠB IB REF 27 Ġwere Ġlimited Ġto Ġa Ġhom ogeneous Ġset Ġof Ġhigh - resource Ġlanguages Ġ( e . g ., ĠEnglish , ĠGerman , ĠItalian , ĠSpanish ) Ġand Ġa Ġsmall Ġnumber Ġof Ġconcept Ġpairs Ġ( all Ġless Ġthan Ġ1 K Ġpairs ). ĠWe Ġaddress Ġboth Ġproblems Ġby Ġ1 ) Ġusing Ġa Ġtyp ologically Ġmore Ġdiverse Ġlanguage Ġsample , Ġand Ġ2 ) Ġrelying Ġon Ġa Ġsubstantially Ġlarger ĠEnglish Ġdataset Ġas Ġa Ġsource Ġfor Ġthe Ġcross - ling ual Ġdatasets : Ġ1 , 888 Ġpairs Ġin Ġthis Ġwork Ġversus Ġ500 Ġpairs Ġin Ġthe Ġwork Ġof ĠCam acho : 2017 se me val . ĠAs Ġa Ġresult , Ġeach Ġof Ġour Ġcross - ling ual Ġdatasets Ġcontains Ġa Ġsubstantially Ġlarger Ġnumber Ġof Ġconcept Ġpairs , Ġas Ġshown Ġin ĠTable ĠT AB REF 30 . ĠThe Ġcross - ling ual ĠMulti - Sim lex Ġdatasets Ġare Ġconstructed Ġautomatically , Ġleveraging Ġword Ġpair Ġtranslations Ġand Ġannotations Ġcollected Ġin Ġall Ġ12 Ġlanguages . ĠThis Ġyields Ġa Ġtotal Ġof Ġ66 Ġcross - ling ual Ġdatasets , Ġone Ġfor Ġeach Ġpossible Ġcombination Ġof Ġlanguages . ĠTable ĠT AB REF 30 Ġprovides Ġthe Ġfinal Ġnumber Ġof Ġconcept Ġpairs , Ġwhich Ġlie Ġbetween Ġ2 , 031 Ġand Ġ3 , 480 Ġpairs Ġfor Ġeach Ġcross - ling ual Ġdataset , Ġwhereas ĠTable ĠT AB REF 29 Ġshows Ġsome Ġsample Ġpairs Ġwith Ġtheir Ġcorresponding Ġsimilarity Ġscores . </s> The Ġautomatic Ġcreation Ġand Ġverification Ġof Ġcross - ling ual Ġdatasets Ġclosely Ġfollows Ġthe Ġprocedure Ġfirst Ġoutlined Ġby ĠCam acho : 2015 acl Ġand Ġlater Ġadopted Ġby ĠCam acho : 2017 se me val Ġ( for Ġsemantic Ġsimilarity ) Ġand ĠVul ic : 2019 acl Ġ( for Ġgraded Ġlex ical Ġentail ment ). ĠFirst , Ġgiven Ġtwo Ġlanguages , Ġwe Ġintersect Ġtheir Ġaligned Ġconcept Ġpairs Ġobtained Ġthrough Ġtranslation . ĠFor Ġinstance , Ġstarting Ġfrom Ġthe Ġaligned Ġpairs Ġatt roup ement ĠâĢĵ Ġfou le Ġin ĠFrench Ġand Ġr ah vas umm ĠâĢĵ Ġr ah v ah ulk Ġin ĠEston ian , Ġwe Ġconstruct Ġtwo Ġcross - ling ual Ġpairs Ġatt roup ement ĠâĢĵ Ġr ah val uk Ġand Ġr ah vas umm ĠâĢĵ Ġfou le . ĠThe Ġscores Ġof Ġcross - ling ual Ġpairs Ġare Ġthen Ġcomputed Ġas Ġaverages Ġof Ġthe Ġtwo Ġcorresponding Ġmon oling ual Ġscores . ĠFinally , Ġin Ġorder Ġto Ġfilter Ġout Ġconcept Ġpairs Ġwhose Ġsemantic Ġmeaning Ġwas Ġnot Ġpreserved Ġduring Ġthis Ġoperation , Ġwe Ġretain Ġonly Ġcross - ling ual Ġpairs Ġfor Ġwhich Ġthe Ġcorresponding Ġmon oling ual Ġscores Ġ$( s _ s , Ġs _ t ) $ Ġdiffer Ġat Ġmost Ġby Ġone Ġfifth Ġof Ġthe Ġfull Ġscale Ġ( i . e ., Ġ$\ mid Ġs _ s Ġ- Ġs _ t Ġ\ mid Ġ\ le Ġ1 . 2 $ ). ĠThis Ġhe uristic Ġmit ig ates Ġthe Ġnoise Ġdue Ġto Ġcross - ling ual Ġsemantic Ġshifts ĠB IB REF 27 , ĠB IB REF 97 . ĠWe Ġrefer Ġthe Ġreader Ġto Ġthe Ġwork Ġof ĠCam acho : 2015 acl Ġfor Ġa Ġdetailed Ġtechnical Ġdescription Ġof Ġthe Ġprocedure . ĠUTF 8 g bs n </s> To Ġassess Ġthe Ġquality Ġof Ġthe Ġresulting Ġcross - ling ual Ġdatasets , Ġwe Ġhave Ġconducted Ġa Ġverification Ġexperiment Ġsimilar Ġto ĠVul ic : 2019 acl . ĠWe Ġrandomly Ġsampled Ġ300 Ġconcept Ġpairs Ġin Ġthe ĠEnglish - Spanish , ĠEnglish - French , Ġand ĠEnglish - Mand arin Ġcross - ling ual Ġdatasets . ĠSub sequently , Ġwe Ġasked Ġbilingual Ġnative Ġspeakers Ġto Ġprovide Ġsimilarity Ġjudgments Ġof Ġeach Ġpair . ĠThe ĠSpear man 's Ġcorrelation Ġscore Ġ$\ r ho Ġ$ Ġbetween Ġautomatically Ġinduced Ġand Ġmanually Ġcollected Ġratings Ġachieves Ġ$\ r ho Ġ\ ge Ġ0 . 90 $ Ġon Ġall Ġsamples , Ġwhich Ġconfirms Ġthe Ġviability Ġof Ġthe Ġautomatic Ġconstruction Ġprocedure . </s> Score Ġand ĠClass ĠDistribut ions . ĠThe Ġsummary Ġof Ġscore Ġand Ġclass Ġdistributions Ġacross Ġall Ġ66 Ġcross - ling ual Ġdatasets Ġare Ġprovided Ġin ĠFigure ĠFIG REF 31 Ġand ĠFigure ĠFIG REF 31 , Ġrespectively . ĠFirst , Ġit Ġis Ġobvious Ġthat Ġthe Ġdistribution Ġover Ġthe Ġfour ĠPOS Ġclasses Ġlargely Ġad heres Ġto Ġthat Ġof Ġthe Ġoriginal Ġmon oling ual ĠMulti - Sim Lex Ġdatasets , Ġand Ġthat Ġthe Ġvariance Ġis Ġquite Ġlow : Ġe . g ., Ġthe Ġeng - f ra Ġdataset Ġcontains Ġthe Ġlowest Ġproportion Ġof Ġnoun s Ġ( 49 . 21 %) Ġand Ġthe Ġhighest Ġproportion Ġof Ġverbs Ġ( 27 . 1 %), Ġadject ives Ġ( 15 . 28 %), Ġand Ġad verbs Ġ( 8 . 41 %). ĠOn Ġthe Ġother Ġhand , Ġthe Ġdistribution Ġover Ġsimilarity Ġintervals Ġin ĠFigure ĠFIG REF 31 Ġshows Ġa Ġmuch Ġgreater Ġvariance . ĠThis Ġis Ġagain Ġexpected Ġas Ġthis Ġpattern Ġresur faces Ġin Ġmon oling ual Ġdatasets Ġ( see ĠTable ĠT AB REF 19 ). ĠIt Ġis Ġalso Ġevident Ġthat Ġthe Ġdata Ġare Ġskewed Ġtowards Ġlower - similar ity Ġconcept Ġpairs . ĠHowever , Ġdue Ġto Ġthe Ġjoint Ġsize Ġof Ġall Ġcross - ling ual Ġdatasets Ġ( see ĠTable ĠT AB REF 30 ), Ġeven Ġthe Ġleast Ġrepresented Ġintervals Ġcontain Ġa Ġsubstantial Ġnumber Ġof Ġconcept Ġpairs . ĠFor Ġinstance , Ġthe Ġr us - y ue Ġdataset Ġcontains Ġthe Ġleast Ġhighly Ġsimilar Ġconcept Ġpairs Ġ( in Ġthe Ġinterval Ġ$ [ 4 , 6 ] $ ) Ġof Ġall Ġ66 Ġcross - ling ual Ġdatasets . ĠNonetheless , Ġthe Ġabsolute Ġnumber Ġof Ġpairs Ġ( 138 ) Ġin Ġthat Ġinterval Ġfor Ġr us - y ue Ġis Ġstill Ġsubstantial . ĠIf Ġneeded , Ġthis Ġmakes Ġit Ġpossible Ġto Ġcreate Ġsmaller Ġdatasets Ġwhich Ġare Ġbalanced Ġacross Ġthe Ġsimilarity Ġspect ra Ġthrough Ġsub - sam pling . </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels </s> After Ġthe Ġnumerical Ġand Ġqualitative Ġanalyses Ġof Ġthe ĠMulti - Sim Lex Ġdatasets Ġprovided Ġin ĠÂ§Â§ ĠSEC REF 18 âĢĵ SEC REF 25 , Ġwe Ġnow Ġbenchmark Ġa Ġseries Ġof Ġrepresentation Ġlearning Ġmodels Ġon Ġthe Ġnew Ġevaluation Ġdata . ĠWe Ġevaluate Ġstandard Ġstatic Ġword Ġembed ding Ġalgorithms Ġsuch Ġas Ġfast Text ĠB IB REF 31 , Ġas Ġwell Ġas Ġa Ġrange Ġof Ġmore Ġrecent Ġtext Ġenc od ers Ġpret rained Ġon Ġlanguage Ġmodeling Ġsuch Ġas Ġmult ilingual ĠB ERT ĠB IB REF 29 . ĠThese Ġexperiments Ġprovide Ġstrong Ġbaseline Ġscores Ġon Ġthe Ġnew ĠMulti - Sim Lex Ġdatasets Ġand Ġoffer Ġa Ġfirst Ġlarge - scale Ġanalysis Ġof Ġpret rained Ġenc od ers Ġon Ġword - level Ġsemantic Ġsimilarity Ġacross Ġdiverse Ġlanguages . ĠIn Ġaddition , Ġthe Ġexperiments Ġnow Ġenabled Ġby ĠMulti - Sim Lex Ġaim Ġto Ġanswer Ġseveral Ġimportant Ġquestions . Ġ( Q 1 ) ĠIs Ġit Ġviable Ġto Ġextract Ġhigh - quality Ġword - level Ġrepresentations Ġfrom Ġpret rained Ġenc od ers Ġreceiving Ġsub word - level Ġtokens Ġas Ġinput ? ĠAre Ġsuch Ġrepresentations Ġcompetitive Ġwith Ġstandard Ġstatic Ġword - level Ġembed d ings ? Ġ( Q 2 ) ĠWhat Ġare Ġthe Ġimplications Ġof Ġmon oling ual Ġpret raining Ġversus Ġ( mass ively ) Ġmult ilingual Ġpret raining Ġfor Ġperformance ? Ġ( Q 3 ) ĠDo Ġlightweight Ġun super vised Ġpost - processing Ġtechniques Ġimprove Ġword Ġrepresentations Ġconsistently Ġacross Ġdifferent Ġlanguages ? Ġ( Q 4 ) ĠCan Ġwe Ġeffectively Ġtransfer Ġavailable Ġexternal Ġlex ical Ġknowledge Ġfrom Ġresource - rich Ġlanguages Ġto Ġresource - lean Ġlanguages Ġin Ġorder Ġto Ġlearn Ġword Ġrepresentations Ġthat Ġdistinguish Ġbetween Ġtrue Ġsimilarity Ġand Ġconceptual Ġrelated ness Ġ( see Ġthe Ġdiscussion Ġin ĠÂ§ SEC REF 6 )? </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels Ġ: :: ĠModels Ġin ĠComparison </s> Static ĠWord ĠEmb edd ings Ġin ĠDifferent ĠLanguages . ĠFirst , Ġwe Ġevaluate Ġa Ġstandard Ġmethod Ġfor Ġinducing Ġnon - context ual ized Ġ( i . e ., Ġstatic ) Ġword Ġembed d ings Ġacross Ġa Ġplethora Ġof Ġdifferent Ġlanguages : Ġfast Text Ġ( ft ) Ġvectors ĠB IB REF 31 Ġare Ġcurrently Ġthe Ġmost Ġpopular Ġand Ġrobust Ġchoice Ġgiven Ġ1 ) Ġthe Ġavailability Ġof Ġpret rained Ġvectors Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages ĠB IB REF 108 Ġtrained Ġon Ġlarge ĠCommon ĠC rawl Ġ( CC ) Ġplus ĠWikipedia Ġ( Wiki ) Ġdata , Ġand Ġ2 ) Ġtheir Ġsuperior Ġperformance Ġacross Ġa Ġrange Ġof ĠN LP Ġtasks ĠB IB REF 109 . ĠIn Ġfact , Ġfast Text Ġis Ġan Ġextension Ġof Ġthe Ġstandard Ġword - level ĠCB OW Ġand Ġskip - gram Ġword 2 vec Ġmodels ĠB IB REF 32 Ġthat Ġtakes Ġinto Ġaccount Ġsub word - level Ġinformation , Ġi . e . Ġthe Ġcont itu ent Ġcharacter Ġn - gram s Ġof Ġeach Ġword ĠB IB REF 110 . ĠFor Ġthis Ġreason , Ġfast Text Ġis Ġalso Ġmore Ġsuited Ġfor Ġmodeling Ġrare Ġwords Ġand Ġmorph ologically Ġrich Ġlanguages . </s> We Ġrely Ġon Ġ300 - dimensional Ġft Ġword Ġvectors Ġtrained Ġon ĠCC + Wiki Ġand Ġavailable Ġonline Ġfor Ġ157 Ġlanguages . ĠThe Ġword Ġvectors Ġfor Ġall Ġlanguages Ġare Ġobtained Ġby ĠCB OW Ġwith Ġposition - weights , Ġwith Ġcharacter Ġn - gram s Ġof Ġlength Ġ5 , Ġa Ġwindow Ġof Ġsize Ġ5 , Ġ10 Ġnegative Ġexamples , Ġand Ġ10 Ġtraining Ġepoch s . ĠWe Ġalso Ġprobe Ġanother Ġ( older ) Ġcollection Ġof Ġft Ġvectors , Ġpret rained Ġon Ġfull ĠWikipedia Ġdumps Ġof Ġeach Ġlanguage .. ĠThe Ġvectors Ġare Ġ300 - dimensional , Ġtrained Ġwith Ġthe Ġskip - gram Ġobjective Ġfor Ġ5 Ġepoch s , Ġwith Ġ5 Ġnegative Ġexamples , Ġa Ġwindow Ġsize Ġset Ġto Ġ5 , Ġand Ġrelying Ġon Ġall Ġcharacter Ġn - gram s Ġfrom Ġlength Ġ3 Ġto Ġ6 . ĠFollowing Ġprior Ġwork , Ġwe Ġtrim Ġthe Ġvoc ab ular ies Ġfor Ġall Ġlanguages Ġto Ġthe Ġ200 K Ġmost Ġfrequent Ġwords Ġand Ġcompute Ġrepresentations Ġfor Ġmulti - word Ġexpressions Ġby Ġaveraging Ġthe Ġvectors Ġof Ġtheir Ġconstituent Ġwords . </s> Un super vised ĠPost - Process ing . ĠFurther , Ġwe Ġconsider Ġa Ġvariety Ġof Ġun super vised Ġpost - processing Ġsteps Ġthat Ġcan Ġbe Ġapplied Ġpost - training Ġon Ġtop Ġof Ġany Ġpret rained Ġinput Ġword Ġembed ding Ġspace Ġwithout Ġany Ġexternal Ġlex ical Ġsemantic Ġresource . ĠSo Ġfar , Ġthe Ġusefulness Ġof Ġsuch Ġmethods Ġhas Ġbeen Ġverified Ġonly Ġon Ġthe ĠEnglish Ġlanguage Ġthrough Ġbenchmarks Ġfor Ġlex ical Ġsemantics Ġand Ġsentence - level Ġtasks ĠB IB REF 113 . ĠIn Ġthis Ġpaper , Ġwe Ġassess Ġif Ġun super vised Ġpost - processing Ġis Ġbeneficial Ġalso Ġin Ġother Ġlanguages . ĠTo Ġthis Ġend , Ġwe Ġapply Ġthe Ġfollowing Ġpost - h oc Ġtransformations Ġon Ġthe Ġinitial Ġword Ġembed d ings : </s> 1 ) ĠMean Ġcent ering Ġ( mc ) Ġis Ġapplied Ġafter Ġunit Ġlength Ġnormal ization Ġto Ġensure Ġthat Ġall Ġvectors Ġhave Ġa Ġzero Ġmean , Ġand Ġis Ġcommonly Ġapplied Ġin Ġdata Ġmining Ġand Ġanalysis ĠB IB REF 114 , ĠB IB REF 115 . </s> 2 ) ĠAll - but - the - top Ġ( ab tt ) ĠB IB REF 113 , ĠB IB REF 116 Ġeliminates Ġthe Ġcommon Ġmean Ġvector Ġand Ġa Ġfew Ġtop Ġdominating Ġdirections Ġ( according Ġto ĠPC A ) Ġfrom Ġthe Ġinput Ġdistribution al Ġword Ġvectors , Ġsince Ġthey Ġdo Ġnot Ġcontribute Ġtowards Ġdistinguishing Ġthe Ġactual Ġsemantic Ġmeaning Ġof Ġdifferent Ġwords . ĠThe Ġmethod Ġcontains Ġa Ġsingle Ġ( tun able ) Ġhyper - param eter Ġ$ dd _{ A } $ Ġwhich Ġdenotes Ġthe Ġnumber Ġof Ġthe Ġdominating Ġdirections Ġto Ġremove Ġfrom Ġthe Ġinitial Ġrepresentations . ĠPrevious Ġwork Ġhas Ġverified Ġthe Ġusefulness Ġof Ġab tt Ġin Ġseveral ĠEnglish Ġlex ical Ġsemantic Ġtasks Ġsuch Ġas Ġsemantic Ġsimilarity , Ġword Ġanalog ies , Ġand Ġconcept Ġcategor ization , Ġas Ġwell Ġas Ġin Ġsentence - level Ġtext Ġclassification Ġtasks ĠB IB REF 113 . </s> 3 ) Ġunc ove c ĠB IB REF 117 Ġadjusts Ġthe Ġsimilarity Ġorder Ġof Ġan Ġarbitrary Ġinput Ġword Ġembed ding Ġspace , Ġand Ġcan Ġemphasize Ġeither Ġsynt actic Ġor Ġsemantic Ġinformation Ġin Ġthe Ġtransformed Ġvectors . ĠIn Ġshort , Ġit Ġtransforms Ġthe Ġinput Ġspace Ġ$\ math bf Ġ{ X } $ Ġinto Ġan Ġadjusted Ġspace Ġ$\ math bf Ġ{ X }\ math bf Ġ{ W } _{ \ alpha Ġ} $ Ġthrough Ġa Ġlinear Ġmap Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} $ Ġcontrolled Ġby Ġa Ġsingle Ġhyper - param eter Ġ$\ alpha Ġ$ . ĠThe Ġ$ n ^ {\ text { th }} $ - order Ġsimilarity Ġtransformation Ġof Ġthe Ġinput Ġword Ġvector Ġspace Ġ$\ math bf Ġ{ X } $ Ġ( for Ġwhich Ġ$ n = 1 $ ) Ġcan Ġbe Ġobtained Ġas Ġ$\ math bf Ġ{ M } _{ n } (\ math bf Ġ{ X }) Ġ= Ġ\ math bf Ġ{ M } _ 1 (\ math bf Ġ{ X }\ math bf Ġ{ W } _{ ( n - 1 )/ 2 }) $, Ġwith Ġ$\ math bf Ġ{ W } _{ \ alpha Ġ} = \ math bf Ġ{ Q }\ math bf Ġ{\ Gam ma Ġ} ^ {\ alpha Ġ} $, Ġwhere Ġ$\ math bf Ġ{ Q } $ Ġand Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġare Ġthe Ġmat rices Ġobtained Ġvia Ġe ig end ec om position Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } = \ math bf Ġ{ Q }\ math bf Ġ{\ Gam ma Ġ} \ math bf Ġ{ Q } ^ T $. Ġ$\ math bf Ġ{\ Gam ma Ġ} $ Ġis Ġa Ġdiagonal Ġmatrix Ġcontaining Ġe igen values Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } $ ; Ġ$\ math bf Ġ{ Q } $ Ġis Ġan Ġorth og onal Ġmatrix Ġwith Ġe igen ve ctors Ġof Ġ$\ math bf Ġ{ X } ^ T \ math bf Ġ{ X } $ Ġas Ġcolumns . ĠWhile Ġthe Ġmotivation Ġfor Ġthe Ġunc ove c Ġmethods Ġdoes Ġoriginate Ġfrom Ġadjusting Ġdiscrete Ġsimilarity Ġorders , Ġnote Ġthat Ġ$\ alpha Ġ$ Ġis Ġin Ġfact Ġa Ġcontinuous Ġreal - valued Ġhyper - param eter Ġwhich Ġcan Ġbe Ġcarefully Ġtuned . ĠFor Ġmore Ġtechnical Ġdetails Ġwe Ġrefer Ġthe Ġreader Ġto Ġthe Ġoriginal Ġwork Ġof ĠB IB REF 117 . </s> As Ġmentioned , Ġall Ġpost - processing Ġmethods Ġcan Ġbe Ġseen Ġas Ġun super vised Ġretro fitting Ġmethods Ġthat , Ġgiven Ġan Ġarbitrary Ġinput Ġvector Ġspace Ġ$\ math bf Ġ{ X } $, Ġproduce Ġa Ġpert urbed / trans formed Ġoutput Ġvector Ġspace Ġ$\ math bf Ġ{ X } ^ {\ prime Ġ} $, Ġbut Ġunlike Ġcommon Ġretro fitting Ġmethods ĠB IB REF 62 , ĠB IB REF 16 , Ġthe Ġpert urb ation Ġis Ġcompletely Ġun super vised Ġ( i . e ., Ġself - contained ) Ġand Ġdoes Ġnot Ġinject Ġany Ġexternal Ġ( sem antic Ġsimilarity Ġoriented ) Ġknowledge Ġinto Ġthe Ġvector Ġspace . ĠNote Ġthat Ġdifferent Ġpert urb ations Ġcan Ġalso Ġbe Ġstacked : Ġe . g ., Ġwe Ġcan Ġapply Ġunc ove c Ġand Ġthen Ġuse Ġab tt Ġon Ġtop Ġthe Ġoutput Ġunc ove c Ġvectors . ĠWhen Ġusing Ġunc ove c Ġand Ġab tt Ġwe Ġalways Ġlength - normal ize Ġand Ġmean - center Ġthe Ġdata Ġfirst Ġ( i . e ., Ġwe Ġapply Ġthe Ġsimple Ġmc Ġnormal ization ). ĠFinally , Ġwe Ġtune Ġthe Ġtwo Ġhyper - param eters Ġ$ d _ A $ Ġ( for Ġab tt ) Ġand Ġ$\ alpha Ġ$ Ġ( unc ove c ) Ġon Ġthe ĠEnglish ĠMulti - Sim Lex Ġand Ġuse Ġthe Ġsame Ġvalues Ġon Ġthe Ġdatasets Ġof Ġall Ġother Ġlanguages ; Ġwe Ġreport Ġresults Ġwith Ġ$ dd _ A Ġ= Ġ3 $ Ġor Ġ$ dd _ A Ġ= Ġ10 $, Ġand Ġ$\ alpha Ġ= - 0 . 3 $. </s> Context ual ized ĠWord ĠEmb edd ings . ĠWe Ġalso Ġevaluate Ġthe Ġcapacity Ġof Ġun super vised Ġpret raining Ġarchitectures Ġbased Ġon Ġlanguage Ġmodeling Ġobjectives Ġto Ġreason Ġover Ġlex ical Ġsemantic Ġsimilarity . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġour Ġarticle Ġis Ġthe Ġfirst Ġstudy Ġperforming Ġsuch Ġanalyses . ĠState - of - the - art Ġmodels Ġsuch Ġas Ġb ert ĠB IB REF 29 , Ġx l m ĠB IB REF 30 , Ġor Ġro bert a ĠB IB REF 118 Ġare Ġtypically Ġvery Ġdeep Ġneural Ġnetworks Ġbased Ġon Ġthe ĠTrans former Ġarchitecture ĠB IB REF 119 . ĠThey Ġreceive Ġsub word - level Ġtokens Ġas Ġinputs Ġ( such Ġas ĠWord Pie ces ĠB IB REF 120 ) Ġto Ġtackle Ġdata Ġsp arsity . ĠIn Ġoutput , Ġthey Ġreturn Ġcontextual ized Ġembed d ings , Ġdynamic Ġrepresentations Ġfor Ġwords Ġin Ġcontext . </s> To Ġrepresent Ġwords Ġor Ġmulti - word Ġexpressions Ġthrough Ġa Ġpret rained Ġmodel , Ġwe Ġfollow Ġprior Ġwork ĠB IB REF 121 Ġand Ġcompute Ġan Ġinput Ġitem 's Ġrepresentation Ġby Ġ1 ) Ġfeeding Ġit Ġto Ġa Ġpret rained Ġmodel Ġin Ġisolation ; Ġthen Ġ2 ) Ġaveraging Ġthe Ġ$ H $ Ġlast Ġhidden Ġrepresentations Ġfor Ġeach Ġof Ġthe Ġitem âĢ Ļ s Ġconstituent Ġsub words ; Ġand Ġthen Ġfinally Ġ3 ) Ġaveraging Ġthe Ġresulting Ġsub word Ġrepresentations Ġto Ġproduce Ġthe Ġfinal Ġ$ d $ - dimensional Ġrepresentation , Ġwhere Ġ$ d $ Ġis Ġthe Ġembed ding Ġand Ġhidden - layer Ġdimension ality Ġ( e . g ., Ġ$ d = 768 $ Ġwith Ġb ert ). ĠWe Ġopt Ġfor Ġthis Ġapproach Ġdue Ġto Ġits Ġproven Ġviability Ġand Ġsimplicity ĠB IB REF 121 , Ġas Ġit Ġdoes Ġnot Ġrequire Ġany Ġadditional Ġcorpor a Ġto Ġcondition Ġthe Ġinduction Ġof Ġcontextual ized Ġembed d ings . ĠOther Ġways Ġto Ġextract Ġthe Ġrepresentations Ġfrom Ġpret rained Ġmodels ĠB IB REF 122 , ĠB IB REF 123 , ĠB IB REF 124 Ġare Ġbeyond Ġthe Ġscope Ġof Ġthis Ġwork , Ġand Ġwe Ġwill Ġexperiment Ġwith Ġthem Ġin Ġthe Ġfuture . </s> In Ġother Ġwords , Ġwe Ġtreat Ġeach Ġpret rained Ġenc oder Ġenc Ġas Ġa Ġblack - box Ġfunction Ġto Ġencode Ġa Ġsingle Ġword Ġor Ġa Ġmulti - word Ġexpression Ġ$ x $ Ġin Ġeach Ġlanguage Ġinto Ġa Ġ$ d $ - dimensional Ġcontextual ized Ġrepresentation Ġ$\ math bf Ġ{ x } _{ \ text sc Ġ{ enc }} Ġ\ in Ġ\ math bb Ġ{ R } ^ d Ġ= Ġ\ text sc Ġ{ enc } ( x ) $ Ġ( e . g ., Ġ$ d = 768 $ Ġwith Ġb ert ). ĠAs Ġmult ilingual Ġpret rained Ġenc od ers , Ġwe Ġexperiment Ġwith Ġthe Ġmult ilingual Ġb ert Ġmodel Ġ( m - bert ) ĠB IB REF 29 Ġand Ġx l m ĠB IB REF 30 . Ġm - bert Ġis Ġpret rained Ġon Ġmon oling ual ĠWikipedia Ġcorpor a Ġof Ġ102 Ġlanguages Ġ( com prising Ġall ĠMulti - Sim Lex Ġlanguages ) Ġwith Ġa Ġ12 - layer ĠTrans former Ġnetwork , Ġand Ġyields Ġ768 - dimensional Ġrepresentations . ĠSince Ġthe Ġconcept Ġpairs Ġin ĠMulti - Sim Lex Ġare Ġlower c ased , Ġwe Ġuse Ġthe Ġunc ased Ġversion Ġof Ġm - bert . Ġm - bert Ġcomprises Ġall ĠMulti - Sim Lex Ġlanguages , Ġand Ġits Ġevident Ġability Ġto Ġperform Ġcross - ling ual Ġtransfer ĠB IB REF 12 , ĠB IB REF 125 , ĠB IB REF 126 Ġalso Ġmakes Ġit Ġa Ġconvenient Ġbaseline Ġmodel Ġfor Ġcross - ling ual Ġexperiments Ġlater Ġin ĠÂ§ SEC REF 8 . ĠThe Ġsecond Ġmult ilingual Ġmodel Ġwe Ġconsider , Ġx l m - 100 , Ġis Ġpret rained Ġon ĠWikipedia Ġdumps Ġof Ġ100 Ġlanguages , Ġand Ġenc odes Ġeach Ġconcept Ġinto Ġa Ġ$ 1 , 280 $ - dimensional Ġrepresentation . ĠIn Ġcontrast Ġto Ġm - bert , Ġx l m - 100 Ġdrops Ġthe Ġnext - sent ence Ġprediction Ġobjective Ġand Ġadds Ġa Ġcross - ling ual Ġmasked Ġlanguage Ġmodeling Ġobjective . ĠFor Ġboth Ġenc od ers , Ġthe Ġrepresentations Ġof Ġeach Ġconcept Ġare Ġcomputed Ġas Ġaverages Ġover Ġthe Ġlast Ġ$ H = 4 $ Ġhidden Ġlayers Ġin Ġall Ġexperiments , Ġas Ġsuggested Ġby ĠWu : 2019 ar x iv . </s> Besides Ġm - bert Ġand Ġx l m , Ġcovering Ġmultiple Ġlanguages , Ġwe Ġalso Ġanalyze Ġthe Ġperformance Ġof ĠâĢ ľ language - specific âĢ Ŀ Ġb ert Ġand Ġx l m Ġmodels Ġfor Ġthe Ġlanguages Ġwhere Ġthey Ġare Ġavailable : ĠFinnish , ĠSpanish , ĠEnglish , ĠMandarin ĠChinese , Ġand ĠFrench . ĠThe Ġmain Ġgoal Ġof Ġthis Ġcomparison Ġis Ġto Ġstudy Ġthe Ġdifferences Ġin Ġperformance Ġbetween Ġmult ilingual ĠâĢ ľ one - size - fits - all âĢ Ŀ Ġenc od ers Ġand Ġlanguage - specific Ġenc od ers . ĠFor Ġall Ġexperiments , Ġwe Ġrely Ġon Ġthe Ġpret rained Ġmodels Ġreleased Ġin Ġthe ĠTransformers Ġrepository ĠB IB REF 127 . </s> Un super vised Ġpost - processing Ġsteps Ġdevised Ġfor Ġstatic Ġword Ġembed d ings Ġ( i . e ., Ġmean - cent ering , Ġab tt , Ġunc ove c ) Ġcan Ġalso Ġbe Ġapplied Ġon Ġtop Ġof Ġcontextual ized Ġembed d ings Ġif Ġwe Ġpred ef ine Ġa Ġvocabulary Ġof Ġword Ġtypes Ġ$ V $ Ġthat Ġwill Ġbe Ġrepresented Ġin Ġa Ġword Ġvector Ġspace Ġ$\ math bf Ġ{ X } $. ĠWe Ġconstruct Ġsuch Ġ$ V $ Ġfor Ġeach Ġlanguage Ġas Ġthe Ġintersection Ġof Ġword Ġtypes Ġcovered Ġby Ġthe Ġcorresponding ĠCC + Wiki Ġfast Text Ġvectors Ġand Ġthe Ġ( single - word Ġor Ġmulti - word ) Ġexpressions Ġappearing Ġin Ġthe Ġcorresponding ĠMulti - Sim Lex Ġdataset . </s> Finally , Ġnote Ġthat Ġit Ġis Ġnot Ġfeasible Ġto Ġevaluate Ġa Ġfull Ġrange Ġof Ġavailable Ġpret rained Ġenc od ers Ġwithin Ġthe Ġscope Ġof Ġthis Ġwork . ĠOur Ġmain Ġintention Ġis Ġto Ġprovide Ġthe Ġfirst Ġset Ġof Ġbaseline Ġresults Ġon ĠMulti - Sim Lex Ġby Ġbenchmark ing Ġa Ġsample Ġof Ġmost Ġpopular Ġenc od ers , Ġat Ġthe Ġsame Ġtime Ġalso Ġinvestigating Ġother Ġimportant Ġquestions Ġsuch Ġas Ġperformance Ġof Ġstatic Ġversus Ġcontextual ized Ġword Ġembed d ings , Ġor Ġmult ilingual Ġversus Ġlanguage - specific Ġpret raining . ĠAnother Ġpurpose Ġof Ġthe Ġexperiments Ġis Ġto Ġoutline Ġthe Ġwide Ġpotential Ġand Ġapplic ability Ġof Ġthe ĠMulti - Sim Lex Ġdatasets Ġfor Ġmult ilingual Ġand Ġcross - ling ual Ġrepresentation Ġlearning Ġevaluation . </s> Mon oling ual ĠEvaluation Ġof ĠRepresent ation ĠLearning ĠModels Ġ: :: ĠResults Ġand ĠDiscussion </s> The Ġresults Ġwe Ġreport Ġare ĠSpear man 's Ġ$\ r ho Ġ$ Ġcoefficients Ġof Ġthe Ġcorrelation Ġbetween Ġthe Ġranks Ġderived Ġfrom Ġthe Ġscores Ġof Ġthe Ġevaluated Ġmodels Ġand Ġthe Ġhuman Ġscores Ġprovided Ġin Ġeach ĠMulti - Sim Lex Ġdataset . ĠThe Ġmain Ġresults Ġwith Ġstatic Ġand Ġcontextual ized Ġword Ġvectors Ġfor Ġall Ġtest Ġlanguages Ġare Ġsummarized Ġin ĠTable ĠT AB REF 43 . ĠThe Ġscores Ġreveal Ġseveral Ġinteresting Ġpatterns , Ġand Ġalso Ġpinpoint Ġthe Ġmain Ġchallenges Ġfor Ġfuture Ġwork . </s> State - of - the - Art ĠRepresent ation ĠModels . ĠThe Ġabsolute Ġscores Ġof ĠCC + Wiki Ġft , ĠWiki Ġft , Ġand Ġm - bert Ġare Ġnot Ġdirectly Ġcomparable , Ġbecause Ġthese Ġmodels Ġhave Ġdifferent Ġcoverage . ĠIn Ġparticular , ĠMulti - Sim Lex Ġcontains Ġsome Ġout - of - voc abulary Ġ( OO V ) Ġwords Ġwhose Ġstatic Ġft Ġembed d ings Ġare Ġnot Ġavailable . ĠOn Ġthe Ġother Ġhand , Ġm - bert Ġhas Ġperfect Ġcoverage . ĠA Ġgeneral Ġcomparison Ġbetween ĠCC + Wiki Ġand ĠWiki Ġft Ġvectors , Ġhowever , Ġsupports Ġthe Ġintuition Ġthat Ġlarger Ġcorpor a Ġ( such Ġas ĠCC + Wiki ) Ġyield Ġhigher Ġcorrelations . ĠAnother Ġfinding Ġis Ġthat Ġa Ġsingle Ġmassively Ġmult ilingual Ġmodel Ġsuch Ġas Ġm - bert Ġcannot Ġproduce Ġsem antically Ġrich Ġword - level Ġrepresentations . ĠWhether Ġthis Ġactually Ġhappens Ġbecause Ġthe Ġtraining Ġobjective Ġis Ġdifferent âĢĶ or Ġbecause Ġthe Ġneed Ġto Ġrepresent Ġ100 + Ġlanguages Ġreduces Ġits Ġlanguage - specific Ġcapacity âĢĶ is Ġinvestigated Ġfurther Ġbelow . </s> The Ġoverall Ġresults Ġalso Ġclearly Ġindicate Ġthat Ġ( i ) Ġthere Ġare Ġdifferences Ġin Ġperformance Ġacross Ġdifferent Ġmon oling ual ĠMulti - Sim Lex Ġdatasets , Ġand Ġ( ii ) Ġun super vised Ġpost - processing Ġis Ġuniversally Ġuseful , Ġand Ġcan Ġlead Ġto Ġhuge Ġimprovements Ġin Ġcorrelation Ġscores Ġfor Ġmany Ġlanguages . ĠIn Ġwhat Ġfollows , Ġwe Ġalso Ġdelve Ġdeeper Ġinto Ġthese Ġanalyses . </s> Imp act Ġof ĠUn super vised ĠPost - Process ing . ĠFirst , Ġthe Ġresults Ġin ĠTable ĠT AB REF 43 Ġsuggest Ġthat Ġapplying Ġdimension - wise Ġmean Ġcent ering Ġto Ġthe Ġinitial Ġvector Ġspaces Ġhas Ġpositive Ġimpact Ġon Ġword Ġsimilarity Ġscores Ġin Ġall Ġtest Ġlanguages Ġand Ġfor Ġall Ġmodels , Ġboth Ġstatic Ġand Ġcontextual ized Ġ( see Ġthe Ġ+ mc Ġrows Ġin ĠTable ĠT AB REF 43 ). ĠB IB REF 128 Ġshow Ġthat Ġdistribution al Ġword Ġvectors Ġhave Ġa Ġtendency Ġtowards Ġnarrow Ġclusters Ġin Ġthe Ġvector Ġspace Ġ( i . e ., Ġthey Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace Ġand Ġare Ġtherefore Ġan is otropic ĠB IB REF 113 , ĠB IB REF 129 ), Ġand Ġare Ġprone Ġto Ġthe Ġundes ired Ġeffect Ġof Ġhub ness ĠB IB REF 130 , ĠB IB REF 131 . ĠApp lying Ġdimension - wise Ġmean Ġcent ering Ġhas Ġthe Ġeffect Ġof Ġspreading Ġthe Ġvectors Ġacross Ġthe Ġhyper - plane Ġand Ġmitigating Ġthe Ġhub ness Ġissue , Ġwhich Ġconsequently Ġimproves Ġword - level Ġsimilarity , Ġas Ġit Ġemerges Ġfrom Ġthe Ġreported Ġresults . ĠPrevious Ġwork Ġhas Ġalready Ġvalidated Ġthe Ġimportance Ġof Ġmean Ġcent ering Ġfor Ġclust ering - based Ġtasks ĠB IB REF 132 , Ġbilingual Ġlex icon Ġinduction Ġwith Ġcross - ling ual Ġword Ġembed d ings ĠB IB REF 133 , ĠB IB REF 134 , ĠB IB REF 135 , Ġand Ġfor Ġmodeling Ġlex ical Ġsemantic Ġchange ĠB IB REF 136 . ĠHowever , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġresults Ġsummarized Ġin ĠTable ĠT AB REF 43 Ġare Ġthe Ġfirst Ġevidence Ġthat Ġalso Ġconfirms Ġits Ġimportance Ġfor Ġsemantic Ġsimilarity Ġin Ġa Ġwide Ġarray Ġof Ġlanguages . ĠIn Ġsum , Ġas Ġa Ġgeneral Ġrule Ġof Ġthumb , Ġwe Ġsuggest Ġto Ġalways Ġmean - center Ġrepresentations Ġfor Ġsemantic Ġtasks . </s> The Ġresults Ġfurther Ġindicate Ġthat Ġadditional Ġpost - processing Ġmethods Ġsuch Ġas Ġab tt Ġand Ġunc ove c Ġon Ġtop Ġof Ġmean - centered Ġvector Ġspaces Ġcan Ġlead Ġto Ġfurther Ġgains Ġin Ġmost Ġlanguages . ĠThe Ġgains Ġare Ġeven Ġvisible Ġfor Ġlanguages Ġwhich Ġstart Ġfrom Ġhigh Ġcorrelation Ġscores : Ġfor Ġinstance ., Ġc mn Ġwith ĠCC + Wiki Ġft Ġincreases Ġfrom Ġ0 . 5 34 Ġto Ġ0 . 583 , Ġfrom Ġ0 . 315 Ġto Ġ0 . 526 Ġwith ĠWiki Ġft , Ġand Ġfrom Ġ0 . 408 Ġto Ġ0 . 487 Ġwith Ġm - bert . ĠSimilarly , Ġfor Ġr us Ġwith ĠCC + Wiki Ġft Ġwe Ġcan Ġimprove Ġfrom Ġ0 . 422 Ġto Ġ0 . 500 , Ġand Ġfor Ġfra Ġthe Ġscores Ġimprove Ġfrom Ġ0 . 578 Ġto Ġ0 . 613 . ĠThere Ġare Ġadditional Ġsimilar Ġcases Ġreported Ġin ĠTable ĠT AB REF 43 . </s> Overall , Ġthe Ġun super vised Ġpost - processing Ġtechniques Ġseem Ġuniversally Ġuseful Ġacross Ġlanguages , Ġbut Ġtheir Ġefficacy Ġand Ġrelative Ġperformance Ġdoes Ġvary Ġacross Ġdifferent Ġlanguages . ĠNote Ġthat Ġwe Ġhave Ġnot Ġcarefully Ġfine - tun ed Ġthe Ġhyper - param eters Ġof Ġthe Ġevaluated Ġpost - processing Ġmethods , Ġso Ġadditional Ġsmall Ġimprovements Ġcan Ġbe Ġexpected Ġfor Ġsome Ġlanguages . ĠThe Ġmain Ġfinding , Ġhowever , Ġis Ġthat Ġthese Ġpost - processing Ġtechniques Ġare Ġrobust Ġto Ġsemantic Ġsimilarity Ġcomput ations Ġbeyond ĠEnglish , Ġand Ġare Ġtruly Ġlanguage Ġindependent . ĠFor Ġinstance , Ġremoving Ġdominant Ġlatent Ġ( PC A - based ) Ġcomponents Ġfrom Ġword Ġvectors Ġemphasizes Ġsemantic Ġdifferences Ġbetween Ġdifferent Ġconcepts , Ġas Ġonly Ġshared Ġnon - in form ative Ġlatent Ġsemantic Ġknowledge Ġis Ġremoved Ġfrom Ġthe Ġrepresentations . </s> In Ġsummary , Ġpret rained Ġword Ġembed d ings Ġdo Ġcontain Ġmore Ġinformation Ġpertaining Ġto Ġsemantic Ġsimilarity Ġthan Ġrevealed Ġin Ġthe Ġinitial Ġvectors . ĠThis Ġway , Ġwe Ġhave Ġcorrobor ated Ġthe Ġhypotheses Ġfrom Ġprior Ġwork ĠB IB REF 113 , ĠB IB REF 117 Ġwhich Ġwere Ġnot Ġpreviously Ġempir ically Ġverified Ġon Ġother Ġlanguages Ġdue Ġto Ġa Ġshortage Ġof Ġevaluation Ġdata ; Ġthis Ġgap Ġhas Ġnow Ġbeen Ġfilled Ġwith Ġthe Ġintroduction Ġof Ġthe ĠMulti - Sim Lex Ġdatasets . ĠIn Ġall Ġfollow - up Ġexperiments , Ġwe Ġalways Ġexplicitly Ġdenote Ġwhich Ġpost - processing Ġconfiguration Ġis Ġused Ġin Ġevaluation . </s> POS - Specific ĠSub sets . ĠWe Ġpresent Ġthe Ġresults Ġfor Ġsubs ets Ġof Ġword Ġpairs Ġgrouped Ġby ĠPOS Ġclass Ġin ĠTable ĠT AB REF 46 . ĠPrior Ġwork Ġbased Ġon ĠEnglish Ġdata Ġshowed Ġthat Ġrepresentations Ġfor Ġnoun s Ġare Ġtypically Ġof Ġhigher Ġquality Ġthan Ġthose Ġfor Ġthe Ġother ĠPOS Ġclasses ĠB IB REF 49 , ĠB IB REF 137 , ĠB IB REF 50 . ĠWe Ġobserve Ġa Ġsimilar Ġtrend Ġin Ġother Ġlanguages Ġas Ġwell . ĠThis Ġpattern Ġis Ġconsistent Ġacross Ġdifferent Ġrepresentation Ġmodels Ġand Ġcan Ġbe Ġattributed Ġto Ġseveral Ġreasons . ĠFirst , Ġverb Ġrepresentations Ġneed Ġto Ġexpress Ġa Ġrich Ġrange Ġof Ġsynt actic Ġand Ġsemantic Ġbehaviors Ġrather Ġthan Ġpurely Ġrefere nt ial Ġfeatures ĠB IB REF 138 , ĠB IB REF 139 , ĠB IB REF 73 . ĠSecond , Ġlow Ġcorrelation Ġscores Ġon Ġthe Ġadjective Ġand Ġad verb Ġsubs ets Ġin Ġsome Ġlanguages Ġ( e . g ., Ġpol , Ġcy m , Ġsw a ) Ġmight Ġbe Ġdue Ġto Ġtheir Ġlow Ġfrequency Ġin Ġmon oling ual Ġtexts , Ġwhich Ġyields Ġunreliable Ġrepresentations . ĠIn Ġgeneral , Ġthe Ġvariance Ġin Ġperformance Ġacross Ġdifferent Ġword Ġclasses Ġwarrants Ġfurther Ġresearch Ġin Ġclass - specific Ġrepresentation Ġlearning ĠB IB REF 140 , ĠB IB REF 50 . ĠThe Ġscores Ġfurther Ġattest Ġthe Ġusefulness Ġof Ġun super vised Ġpost - processing Ġas Ġalmost Ġall Ġclass - specific Ġcorrelation Ġscores Ġare Ġimproved Ġby Ġapplying Ġmean - cent ering Ġand Ġab tt . ĠFinally , Ġthe Ġresults Ġfor Ġm - bert Ġand Ġx l m - 100 Ġin ĠTable ĠT AB REF 46 Ġfurther Ġconfirm Ġthat Ġmassively Ġmult ilingual Ġpret raining Ġcannot Ġyield Ġreasonable Ġsemantic Ġrepresentations Ġfor Ġmany Ġlanguages : Ġin Ġfact , Ġfor Ġsome Ġclasses Ġthey Ġdisplay Ġno Ġcorrelation Ġwith Ġhuman Ġratings Ġat Ġall . </s> Diff erences Ġacross ĠLanguages . ĠNaturally , Ġthe Ġresults Ġfrom ĠTables ĠT AB REF 43 Ġand ĠT AB REF 46 Ġalso Ġreveal Ġthat Ġthere Ġis Ġvariation Ġin Ġperformance Ġof Ġboth Ġstatic Ġword Ġembed d ings Ġand Ġpret rained Ġenc od ers Ġacross Ġdifferent Ġlanguages . ĠAmong Ġother Ġcauses , Ġthe Ġlowest Ġabsolute Ġscores Ġwith Ġft Ġare Ġreported Ġfor Ġlanguages Ġwith Ġleast Ġresources Ġavailable Ġto Ġtrain Ġmon oling