Skip–gram, CBOW
EXTRACTIVE
integrated vector-res, vector-faith, Skip–gram, CBOW
EXTRACTIVE
muneeb2015evalutating trained both the Skip–gram and CBOW models over the PubMed Central Open Access (PMC) corpus of approximately 1.25 million articles. They evaluated the models on a subset of the UMNSRS data, removing word pairs that did not occur in their training corpus more than ten times. chiu2016how evaluated both the the Skip–gram and CBOW models over the PMC corpus and PubMed. They also evaluated the models on a subset of the UMNSRS ignoring those words that did not appear in their training corpus. Pakhomov2016corpus trained CBOW model over three different types of corpora: clinical (clinical notes from the Fairview Health System), biomedical (PMC corpus), and general English (Wikipedia). They evaluated their method using a subset of the UMNSRS restricting to single word term pairs and removing those not found within their training corpus. sajad2015domain trained the Skip–gram model over CUIs identified by MetaMap on the OHSUMED corpus, a collection of 348,566 biomedical research articles. They evaluated the method on the complete UMNSRS, MiniMayoSRS and the MayoSRS datasets; any subset information about the dataset was not explicitly stated therefore we believe a direct comparison may be possible. FLOAT SELECTED: Table 4: Comparison with Previous Work
Table TABREF31 shows a comparison to the top correlation scores reported by each of these works on the respective datasets (or subsets) they evaluated their methods on. N refers to the number of term pairs in the dataset the authors report they evaluated their method. The table also includes our top scoring results: the integrated vector-res and vector-faith. The results show that integrating semantic similarity measures into second–order co–occurrence vectors obtains a higher or on–par correlation with human judgments as the previous works reported results with the exception of the UMNSRS rel dataset. The results reported by Pakhomov2016corpus and chiu2016how obtain a higher correlation although the results can not be directly compared because both works used different subsets of the term pairs from the UMNSRS dataset. muneeb2015evalutating trained both the Skip–gram and CBOW models over the PubMed Central Open Access (PMC) corpus of approximately 1.25 million articles. They evaluated the models on a subset of the UMNSRS data, removing word pairs that did not occur in their training corpus more than ten times. chiu2016how evaluated both the the Skip–gram and CBOW models over the PMC corpus and PubMed. They also evaluated the models on a subset of the UMNSRS ignoring those words that did not appear in their training corpus. Pakhomov2016corpus trained CBOW model over three different types of corpora: clinical (clinical notes from the Fairview Health System), biomedical (PMC corpus), and general English (Wikipedia). They evaluated their method using a subset of the UMNSRS restricting to single word term pairs and removing those not found within their training corpus. sajad2015domain trained the Skip–gram model over CUIs identified by MetaMap on the OHSUMED corpus, a collection of 348,566 biomedical research articles. They evaluated the method on the complete UMNSRS, MiniMayoSRS and the MayoSRS datasets; any subset information about the dataset was not explicitly stated therefore we believe a direct comparison may be possible.
1609.00559
Introduction </s> Me asures Ġof Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġquantify Ġthe Ġdegree Ġto Ġwhich Ġtwo Ġconcepts Ġare Ġsimilar Ġ( e . g ., ĠIN LINE FORM 0 ĠâĢĵ ĠIN LINE FORM 1 Ġ) Ġor Ġrelated Ġ( e . g ., ĠIN LINE FORM 2 ĠâĢĵ ĠIN LINE FORM 3 Ġ). ĠSem antic Ġsimilarity Ġcan Ġbe Ġviewed Ġas Ġa Ġspecial Ġcase Ġof Ġsemantic Ġrelated ness ĠâĢĵ Ġto Ġbe Ġsimilar Ġis Ġone Ġof Ġmany Ġways Ġthat Ġa Ġpair Ġof Ġconcepts Ġmay Ġbe Ġrelated . ĠThe Ġautomated Ġdiscovery Ġof Ġgroups Ġof Ġsem antically Ġsimilar Ġor Ġrelated Ġterms Ġis Ġcritical Ġto Ġimproving Ġthe Ġretrieval ĠB IB REF 0 Ġand Ġclust ering ĠB IB REF 1 Ġof Ġbiomedical Ġand Ġclinical Ġdocuments , Ġand Ġthe Ġdevelopment Ġof Ġbiomedical Ġtermin ologies Ġand Ġont ologies ĠB IB REF 2 Ġ. </s> There Ġis Ġa Ġlong Ġhistory Ġin Ġusing Ġdistribution al Ġmethods Ġto Ġdiscover Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġ( e . g ., ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ). ĠThese Ġmethods Ġare Ġall Ġbased Ġon Ġthe Ġdistribution al Ġhypothesis , Ġwhich Ġholds Ġthat Ġtwo Ġterms Ġthat Ġare Ġdistribution ally Ġsimilar Ġ( i . e ., Ġused Ġin Ġthe Ġsame Ġcontext ) Ġwill Ġalso Ġbe Ġsem antically Ġsimilar ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠRecently Ġword Ġembed ding Ġtechniques Ġsuch Ġas Ġword 2 vec ĠB IB REF 9 Ġhave Ġbecome Ġvery Ġpopular . ĠDespite Ġthe Ġprominent Ġrole Ġthat Ġneural Ġnetworks Ġplay Ġin Ġmany Ġof Ġthese Ġapproaches , Ġat Ġtheir Ġcore Ġthey Ġremain Ġdistribution al Ġtechniques Ġthat Ġtypically Ġstart Ġwith Ġa Ġword Ġby Ġword Ġco âĢĵ occ urrence Ġmatrix , Ġmuch Ġlike Ġmany Ġof Ġthe Ġmore Ġtraditional Ġapproaches . </s> However , Ġdespite Ġthese Ġsuccesses Ġdistribution al Ġmethods Ġdo Ġnot Ġperform Ġwell Ġwhen Ġdata Ġis Ġvery Ġsparse Ġ( which Ġis Ġcommon ). ĠOne Ġpossible Ġsolution Ġis Ġto Ġuse Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠIn Ġthis Ġapproach Ġthe Ġsimilarity Ġbetween Ġtwo Ġwords Ġis Ġnot Ġstrictly Ġbased Ġon Ġtheir Ġco âĢĵ occ urrence Ġfrequencies , Ġbut Ġrather Ġon Ġthe Ġfrequencies Ġof Ġthe Ġother Ġwords Ġwhich Ġoccur Ġwith Ġboth Ġof Ġthem Ġ( i . e ., Ġsecond Ġorder Ġco âĢĵ occ ur rences ). ĠThis Ġapproach Ġhas Ġbeen Ġshown Ġto Ġbe Ġsuccessful Ġin Ġquant ifying Ġsemantic Ġrelated ness ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠHowever , Ġwhile Ġmore Ġrobust Ġin Ġthe Ġface Ġof Ġsp arsity , Ġsecond âĢĵ order Ġmethods Ġcan Ġresult Ġin Ġsignificant Ġamounts Ġof Ġnoise , Ġwhere Ġcontextual Ġinformation Ġthat Ġis Ġoverly Ġgeneral Ġis Ġincluded Ġand Ġdoes Ġnot Ġcontribute Ġto Ġquant ifying Ġthe Ġsemantic Ġrelated ness Ġbetween Ġthe Ġtwo Ġconcepts . </s> Our Ġgoal Ġthen Ġis Ġto Ġdiscover Ġmethods Ġthat Ġautomatically Ġreduce Ġthe Ġamount Ġof Ġnoise Ġin Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector . ĠWe Ġachieve Ġthis Ġby Ġincorporating Ġpair wise Ġsemantic Ġsimilarity Ġscores Ġderived Ġfrom Ġa Ġtax onomy Ġinto Ġour Ġsecond âĢĵ order Ġvectors , Ġand Ġthen Ġusing Ġthese Ġscores Ġto Ġselect Ġonly Ġthe Ġmost Ġsem antically Ġsimilar Ġco âĢĵ occ ur rences Ġ( there by Ġreducing Ġnoise ). </s> We Ġevaluate Ġour Ġmethod Ġon Ġtwo Ġdatasets Ġthat Ġhave Ġbeen Ġannot ated Ġin Ġmultiple Ġways . ĠOne Ġhas Ġbeen Ġannot ated Ġfor Ġboth Ġsimilarity Ġand Ġrelated ness , Ġand Ġthe Ġother Ġhas Ġbeen Ġannot ated Ġfor Ġrelated ness Ġby Ġtwo Ġdifferent Ġtypes Ġof Ġexperts Ġ( medical Ġdoctors Ġand Ġmedical Ġcod ers ). ĠOur Ġresults Ġshow Ġthat Ġintegrating Ġsecond Ġorder Ġco âĢĵ occ ur rences Ġwith Ġmeasures Ġof Ġsemantic Ġsimilarity Ġincreases Ġcorrelation Ġwith Ġour Ġhuman Ġreference Ġstandards . ĠWe Ġalso Ġcompare Ġour Ġresult Ġto Ġa Ġnumber Ġof Ġother Ġstudies Ġwhich Ġhave Ġapplied Ġvarious Ġword Ġembed ding Ġmethods Ġto Ġthe Ġsame Ġreference Ġstandards Ġwe Ġhave Ġused . ĠWe Ġfind Ġthat Ġour Ġmethod Ġoften Ġperforms Ġat Ġa Ġcomparable Ġor Ġhigher Ġlevel Ġthan Ġthese Ġapproaches . ĠThese Ġresults Ġsuggest Ġthat Ġour Ġmethods Ġof Ġintegrating Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġvalues Ġhave Ġthe Ġpotential Ġto Ġimprove Ġperformance Ġof Ġpurely Ġdistribution al Ġmethods . </s> Similar ity Ġand ĠRelated ness ĠMeasures </s> This Ġsection Ġdescribes Ġthe Ġsimilarity Ġand Ġrelated ness Ġmeasures Ġwe Ġintegrate Ġin Ġour Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors . ĠWe Ġuse Ġtwo Ġtax onom ies Ġin Ġthis Ġstudy , ĠSN OM ED âĢĵ CT Ġand ĠMe SH . ĠSN OM ED âĢĵ CT Ġ( System at ized ĠN omen cl ature Ġof ĠMedicine ĠClinical ĠTerms ) Ġis Ġa Ġcomprehensive Ġclinical Ġterminology Ġcreated Ġfor Ġthe Ġelectronic Ġrepresentation Ġof Ġclinical Ġhealth Ġinformation . ĠMe SH Ġ( Medical ĠSubject ĠHead ings ) Ġis Ġa Ġtax onomy Ġof Ġbiomedical Ġterms Ġdeveloped Ġfor Ġindex ing Ġbiomedical Ġjournal Ġarticles . </s> We Ġobtain ĠSN OM ED âĢĵ CT Ġand ĠMe SH Ġvia Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem Ġ( U ML S ) ĠMet at hes aurus Ġ( version Ġ2016 AA ). ĠThe ĠMet at hes aurus Ġcontains Ġapproximately Ġ2 Ġmillion Ġbiomedical Ġand Ġclinical Ġconcepts Ġfrom Ġover Ġ150 Ġdifferent Ġtermin ologies Ġthat Ġhave Ġbeen Ġsemi âĢĵ aut om atically Ġintegrated Ġinto Ġa Ġsingle Ġsource . ĠConcepts Ġin Ġthe ĠMet at hes aurus Ġare Ġconnected Ġlargely Ġby Ġtwo Ġtypes Ġof Ġhierarchical Ġrelations : ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġ( PAR / CH D ) Ġand ĠIN LINE FORM 2 Ġ/ ĠIN LINE FORM 3 Ġ( RB / RN ). </s> Similar ity ĠMeasures </s> Me asures Ġof Ġsemantic Ġsimilarity Ġcan Ġbe Ġclassified Ġinto Ġthree Ġbroad Ġcategories Ġ: Ġpath âĢĵ based , Ġfeature âĢĵ based Ġand Ġinformation Ġcontent Ġ( IC ). ĠPath âĢĵ based Ġsimilarity Ġmeasures Ġuse Ġthe Ġstructure Ġof Ġa Ġtax onomy Ġto Ġmeasure Ġsimilarity ĠâĢĵ Ġconcepts Ġpositioned Ġclose Ġto Ġeach Ġother Ġare Ġmore Ġsimilar Ġthan Ġthose Ġfurther Ġapart . ĠFeature âĢĵ based Ġmethods Ġrely Ġon Ġset Ġtheoret ic Ġmeasures Ġof Ġoverlap Ġbetween Ġfeatures Ġ( union Ġand Ġintersection ). ĠThe Ġinformation Ġcontent Ġmeasures Ġquantify Ġthe Ġamount Ġof Ġinformation Ġthat Ġa Ġconcept Ġprovides ĠâĢĵ Ġmore Ġspecific Ġconcepts Ġhave Ġa Ġhigher Ġamount Ġof Ġinformation Ġcontent . </s> R ada MB B 89 Ġintroduce Ġthe ĠConcept ual ĠDistance Ġmeasure . ĠThis Ġmeasure Ġis Ġsimply Ġthe Ġlength Ġof Ġthe Ġshortest Ġpath Ġbetween Ġtwo Ġconcepts Ġ( ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ) Ġin Ġthe ĠMe SH Ġhierarchy . ĠPath s Ġare Ġbased Ġon Ġbroader Ġthan Ġ( RB ) Ġand Ġnarrower Ġthan Ġ( RN ) Ġrelations . ĠCav ied es C 04 Ġextends Ġthis Ġmeasure Ġto Ġuse Ġparent Ġ( PAR ) Ġand Ġchild Ġ( CH D ) Ġrelations . ĠOur ĠIN LINE FORM 2 Ġmeasure Ġis Ġsimply Ġthe Ġreciprocal Ġof Ġthis Ġshortest Ġpath Ġvalue Ġ( Equ ation ĠEQ REF 3 Ġ), Ġso Ġthat Ġlarger Ġvalues Ġ( appro aching Ġ1 ) Ġindicate Ġa Ġhigh Ġdegree Ġof Ġsimilarity . ĠDIS PLAY FORM 0 </s> While Ġthe Ġsimplicity Ġof ĠIN LINE FORM 0 Ġis Ġappealing , Ġit Ġcan Ġbe Ġmisleading Ġwhen Ġconcepts Ġare Ġat Ġdifferent Ġlevels Ġof Ġspecificity . ĠTwo Ġvery Ġgeneral Ġconcepts Ġmay Ġhave Ġthe Ġsame Ġpath Ġlength Ġas Ġtwo Ġvery Ġspecific Ġconcepts . ĠWu P 94 Ġintroduce Ġa Ġcorrection Ġto ĠIN LINE FORM 1 Ġthat Ġincorporates Ġthe Ġdepth Ġof Ġthe Ġconcepts , Ġand Ġthe Ġdepth Ġof Ġtheir ĠLe ast ĠCommon ĠSub s umer Ġ( LCS ). ĠThis Ġis Ġthe Ġmost Ġspecific Ġancestor Ġtwo Ġconcepts Ġshare . ĠIn Ġthis Ġmeasure , Ġsimilarity Ġis Ġtwice Ġthe Ġdepth Ġof Ġthe Ġtwo Ġconcept 's ĠLCS Ġdivided Ġby Ġthe Ġproduct Ġof Ġthe Ġdepths Ġof Ġthe Ġindividual Ġconcepts Ġ( Equ ation ĠEQ REF 4 Ġ). ĠNote Ġthat Ġif Ġthere Ġare Ġmultiple ĠLCS s Ġfor Ġa Ġpair Ġof Ġconcepts , Ġthe Ġdeepest Ġof Ġthem Ġis Ġused Ġin Ġthis Ġmeasure . ĠDIS PLAY FORM 0 </s> Z h ong Z LY 02 Ġtake Ġa Ġvery Ġsimilar Ġapproach Ġand Ġagain Ġscale Ġthe Ġdepth Ġof Ġthe ĠLCS Ġby Ġthe Ġsum Ġof Ġthe Ġdepths Ġof Ġthe Ġtwo Ġconcepts Ġ( Equ ation ĠEQ REF 5 Ġ), Ġwhere ĠIN LINE FORM 0 Ġ. ĠThe Ġvalue Ġof ĠIN LINE FORM 1 Ġwas Ġset Ġto Ġ2 Ġbased Ġon Ġtheir Ġrecommendations . ĠDIS PLAY FORM 0 </s> P ek ar S 02 Ġoffer Ġanother Ġvariation Ġon ĠIN LINE FORM 0 Ġ, Ġwhere Ġthe Ġshortest Ġpath Ġof Ġthe Ġtwo Ġconcepts Ġto Ġthe ĠLCS Ġis Ġused , Ġin Ġaddition Ġto Ġthe Ġshortest Ġbath Ġbetween Ġthe ĠLCS Ġand Ġthe Ġroot Ġof Ġthe Ġtax onomy Ġ( Equ ation ĠEQ REF 6 Ġ). ĠDIS PLAY FORM 0 </s> Feature âĢĵ based Ġmethods Ġrepresent Ġeach Ġconcept Ġas Ġa Ġset Ġof Ġfeatures Ġand Ġthen Ġmeasure Ġthe Ġoverlap Ġor Ġsharing Ġof Ġfeatures Ġto Ġmeasure Ġsimilarity . ĠIn Ġparticular , Ġeach Ġconcept Ġis Ġrepresented Ġas Ġthe Ġset Ġof Ġtheir Ġancestors , Ġand Ġsimilarity Ġis Ġa Ġratio Ġof Ġthe Ġintersection Ġand Ġunion Ġof Ġthese Ġfeatures . </s> M aed che S 01 Ġquantify Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġratio Ġof Ġthe Ġintersection Ġover Ġtheir Ġunion Ġas Ġshown Ġin ĠEqu ation ĠEQ REF 8 Ġ. ĠDIS PLAY FORM 0 </s> Bat et S V 11 Ġextend Ġthis Ġby Ġexcluding Ġany Ġshared Ġfeatures Ġ( in Ġthe Ġnumer ator ) Ġas Ġshown Ġin ĠEqu ation ĠEQ REF 9 Ġ. ĠDIS PLAY FORM 0 </s> Information Ġcontent Ġis Ġformally Ġdefined Ġas Ġthe Ġnegative Ġlog Ġof Ġthe Ġprobability Ġof Ġa Ġconcept . ĠThe Ġeffect Ġof Ġthis Ġis Ġto Ġassign Ġrare Ġ( low Ġprobability ) Ġconcepts Ġa Ġhigh Ġmeasure Ġof Ġinformation Ġcontent , Ġsince Ġthe Ġunderlying Ġassumption Ġis Ġthat Ġmore Ġspecific Ġconcepts Ġare Ġless Ġfrequently Ġused Ġthan Ġmore Ġcommon Ġones . </s> Res nik 95 Ġmodified Ġthis Ġnotion Ġof Ġinformation Ġcontent Ġin Ġorder Ġto Ġuse Ġit Ġas Ġa Ġsimilarity Ġmeasure . ĠHe Ġdefines Ġthe Ġsimilarity Ġof Ġtwo Ġconcepts Ġto Ġbe Ġthe Ġinformation Ġcontent Ġof Ġtheir ĠLCS Ġ( Equ ation ĠEQ REF 11 Ġ). ĠDIS PLAY FORM 0 </s> J iang C 97 , ĠLin 98 , Ġand ĠPir ro E 10 Ġextend ĠIN LINE FORM 0 Ġby Ġincorporating Ġthe Ġinformation Ġcontent Ġof Ġthe Ġindividual Ġconcepts Ġin Ġvarious Ġdifferent Ġways . ĠLin 98 Ġdefines Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġratio Ġof Ġinformation Ġcontent Ġof Ġthe ĠLCS Ġwith Ġthe Ġsum Ġof Ġthe Ġindividual Ġconcept 's Ġinformation Ġcontent Ġ( Equ ation ĠEQ REF 12 Ġ). ĠNote Ġthat ĠIN LINE FORM 1 Ġhas Ġthe Ġsame Ġform Ġas ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġand Ġis Ġin Ġeffect Ġusing Ġinformation Ġcontent Ġas Ġa Ġmeasure Ġof Ġspecificity Ġ( rather Ġthan Ġdepth ). ĠIf Ġthere Ġis Ġmore Ġthan Ġone Ġpossible ĠLCS , Ġthe ĠLCS Ġwith Ġthe Ġgreatest ĠIC Ġis Ġchosen . ĠDIS PLAY FORM 0 </s> J iang C 97 Ġdefine Ġthe Ġdistance Ġbetween Ġtwo Ġconcepts Ġto Ġbe Ġthe Ġsum Ġof Ġthe Ġinformation Ġcontent Ġof Ġthe Ġtwo Ġconcepts Ġminus Ġtwice Ġthe Ġinformation Ġcontent Ġof Ġthe Ġconcepts ' ĠLCS . ĠWe Ġmodify Ġthis Ġfrom Ġa Ġdistance Ġto Ġa Ġsimilarity Ġmeasure Ġby Ġtaking Ġthe Ġreciprocal Ġof Ġthe Ġdistance Ġ( Equ ation ĠEQ REF 13 Ġ). ĠNote Ġthat Ġthe Ġdenomin ator Ġof ĠIN LINE FORM 0 Ġis Ġvery Ġsimilar Ġto Ġthe Ġnumer ator Ġof ĠIN LINE FORM 1 Ġ. ĠDIS PLAY FORM 0 </s> Pir ro E 10 Ġdefine Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġinformation Ġcontent Ġof Ġthe Ġtwo Ġconcept 's ĠLCS Ġdivided Ġby Ġthe Ġsum Ġof Ġtheir Ġindividual Ġinformation Ġcontent Ġvalues Ġminus Ġthe Ġinformation Ġcontent Ġof Ġtheir ĠLCS Ġ( Equ ation ĠEQ REF 14 Ġ). ĠNote Ġthat ĠIN LINE FORM 0 Ġcan Ġbe Ġviewed Ġas Ġa Ġset âĢĵ the oret ic Ġversion Ġof ĠIN LINE FORM 1 Ġ. ĠDIS PLAY FORM 0 </s> Information ĠContent </s> The Ġinformation Ġcontent Ġof Ġa Ġconcept Ġmay Ġbe Ġderived Ġfrom Ġa Ġcorpus Ġ( cor p us âĢĵ based ) Ġor Ġdirectly Ġfrom Ġa Ġtax onomy Ġ( int r ins ic âĢĵ based ). ĠIn Ġthis Ġwork Ġwe Ġfocus Ġon Ġcorpus âĢĵ based Ġtechniques . </s> For Ġcorpus âĢĵ based Ġinformation Ġcontent , Ġwe Ġestimate Ġthe Ġprobability Ġof Ġa Ġconcept ĠIN LINE FORM 0 Ġby Ġtaking Ġthe Ġsum Ġof Ġthe Ġprobability Ġof Ġthe Ġconcept ĠIN LINE FORM 1 Ġand Ġthe Ġprobability Ġits Ġdescendants ĠIN LINE FORM 2 Ġ( Equ ation ĠEQ REF 16 Ġ). ĠDIS PLAY FORM 0 </s> The Ġinitial Ġprobabilities Ġof Ġa Ġconcept Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġits Ġdescendants Ġ( ĠIN LINE FORM 1 Ġ) Ġare Ġobtained Ġby Ġdividing Ġthe Ġnumber Ġof Ġtimes Ġeach Ġconcept Ġand Ġdescendant Ġoccurs Ġin Ġthe Ġcorpus , Ġand Ġdividing Ġthat Ġby Ġthe Ġtotal Ġnumbers Ġof Ġconcepts Ġ( ĠIN LINE FORM 2 Ġ). </s> Ide ally Ġthe Ġcorpus Ġfrom Ġwhich Ġwe Ġare Ġestimating Ġthe Ġprobabilities Ġof Ġconcepts Ġwill Ġbe Ġsense âĢĵ tag ged . ĠHowever , Ġsense âĢĵ tag ging Ġis Ġa Ġchallenging Ġproblem Ġin Ġits Ġown Ġright , Ġand Ġit Ġis Ġnot Ġalways Ġpossible Ġto Ġcarry Ġout Ġreliably Ġon Ġlarger Ġamounts Ġof Ġtext . ĠIn Ġfact Ġin Ġthis Ġpaper Ġwe Ġdid Ġnot Ġuse Ġany Ġsense âĢĵ tag ging Ġof Ġthe Ġcorpus Ġwe Ġderived Ġinformation Ġcontent Ġfrom . </s> Instead , Ġwe Ġestimated Ġthe Ġprobability Ġof Ġa Ġconcept Ġby Ġusing Ġthe ĠU ML Son Med line Ġdataset . ĠThis Ġwas Ġcreated Ġby Ġthe ĠNational ĠLibrary Ġof ĠMedicine Ġand Ġconsists Ġof Ġconcepts Ġfrom Ġthe Ġ2009 AB ĠU ML S Ġand Ġthe Ġcounts Ġof Ġthe Ġnumber Ġof Ġtimes Ġthey Ġoccurred Ġin Ġa Ġsnapshot Ġof ĠMedline Ġtaken Ġon Ġ12 ĠJanuary , Ġ2009 . ĠThese Ġcounts Ġwere Ġobtained Ġby Ġusing Ġthe ĠEss ie ĠSearch ĠEngine ĠB IB REF 14 Ġwhich Ġquer ied ĠMedline Ġwith Ġnormalized Ġstrings Ġfrom Ġthe Ġ2009 AB ĠM RC ON SO Ġtable Ġin Ġthe ĠU ML S . ĠThe Ġfrequency Ġof Ġa ĠC UI Ġwas Ġobtained Ġby Ġaggreg ating Ġthe Ġfrequency Ġcounts Ġof Ġthe Ġterms Ġassociated Ġwith Ġthe ĠC UI Ġto Ġprovide Ġa Ġrough Ġestimate Ġof Ġits Ġfrequency . ĠThe Ġinformation Ġcontent Ġmeasures Ġthen Ġuse Ġthis Ġinformation Ġto Ġcalculate Ġthe Ġprobability Ġof Ġa Ġconcept . </s> Another Ġalternative Ġis Ġthe Ġuse Ġof ĠInt r ins ic ĠInformation ĠContent . ĠIt Ġassess Ġthe Ġinform at iveness Ġof Ġconcept Ġbased Ġon Ġits Ġplacement Ġwithin Ġa Ġtax onomy Ġby Ġconsidering Ġthe Ġnumber Ġof Ġincoming Ġ( anc est ors ) Ġrelative Ġto Ġoutgoing Ġ( desc endant ) Ġlinks ĠB IB REF 15 Ġ( Equ ation ĠEQ REF 17 Ġ). ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġthe Ġnumber Ġof Ġdescendants Ġof Ġconcept ĠIN LINE FORM 1 Ġthat Ġare Ġleaf Ġnodes , ĠIN LINE FORM 2 Ġare Ġthe Ġnumber Ġof Ġconcept ĠIN LINE FORM 3 Ġ' s Ġancestors Ġand ĠIN LINE FORM 4 Ġare Ġthe Ġtotal Ġnumber Ġof Ġleaf Ġnodes Ġin Ġthe Ġtax onomy . </s> Related ness ĠMeasures </s> Les k 86 Ġobserved Ġthat Ġconcepts Ġthat Ġare Ġrelated Ġshould Ġshare Ġmore Ġwords Ġin Ġtheir Ġrespective Ġdefinitions Ġthan Ġconcepts Ġthat Ġare Ġless Ġconnected . ĠHe Ġwas Ġable Ġto Ġperform Ġword Ġsense Ġdis amb ig uation Ġby Ġidentifying Ġthe Ġsenses Ġof Ġwords Ġin Ġa Ġsentence Ġwith Ġthe Ġlargest Ġnumber Ġof Ġoverl aps Ġbetween Ġtheir Ġdefinitions . ĠAn Ġoverlap Ġis Ġthe Ġlongest Ġsequence Ġof Ġone Ġor Ġmore Ġconsecutive Ġwords Ġthat Ġoccur Ġin Ġboth Ġdefinitions . ĠBan er jee P 03 Ġextended Ġthis Ġidea Ġto ĠWord Net , Ġbut Ġobserved Ġthat ĠWord Net Ġgloss es Ġare Ġoften Ġvery Ġshort , Ġand Ġdid Ġnot Ġcontain Ġenough Ġinformation Ġto Ġdistinguish Ġbetween Ġmultiple Ġconcepts . ĠTherefore , Ġthey Ġcreated Ġa Ġsuper âĢĵ gl oss Ġfor Ġeach Ġconcept Ġby Ġadding Ġthe Ġgloss es Ġof Ġrelated Ġconcepts Ġto Ġthe Ġgloss Ġof Ġthe Ġconcept Ġitself Ġ( and Ġthen Ġfinding Ġoverl aps ). </s> Pat ward han P 06 Ġadapted Ġthis Ġmeasure Ġto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors . ĠIn Ġthis Ġapproach , Ġa Ġvector Ġis Ġcreated Ġfor Ġeach Ġword Ġin Ġa Ġconcept 's Ġdefinition Ġthat Ġshows Ġwhich Ġwords Ġco âĢĵ occ ur Ġwith Ġit Ġin Ġa Ġcorpus . ĠThese Ġword Ġvectors Ġare Ġaveraged Ġto Ġcreate Ġa Ġsingle Ġco - occ urrence Ġvector Ġfor Ġthe Ġconcept . ĠThe Ġsimilarity Ġbetween Ġthe Ġconcepts Ġis Ġcalculated Ġby Ġtaking Ġthe Ġcos ine Ġbetween Ġthe Ġconcepts Ġsecond âĢĵ order Ġvectors . ĠLiu M PM P 12 Ġmodified Ġand Ġextended Ġthis Ġmeasure Ġto Ġbe Ġused Ġto Ġquantify Ġthe Ġrelated ness Ġbetween Ġbiomedical Ġand Ġclinical Ġterms Ġin Ġthe ĠU ML S . ĠThe Ġwork Ġin Ġthis Ġpaper Ġcan Ġbe Ġseen Ġas Ġa Ġfurther Ġextension Ġof ĠPat ward han P 06 Ġand ĠLiu M PM P 12 . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġsecond âĢĵ order Ġsimilarity Ġvector Ġmeasure . ĠThis Ġincorporates Ġboth Ġcontextual Ġinformation Ġusing Ġthe Ġterm Ġpair 's Ġdefinition Ġand Ġtheir Ġpair wise Ġsemantic Ġsimilarity Ġscores Ġderived Ġfrom Ġa Ġtax onomy . ĠThere Ġare Ġtwo Ġstages Ġto Ġour Ġapproach . ĠFirst , Ġa Ġco âĢĵ occ urrence Ġmatrix Ġmust Ġbe Ġconstructed . ĠSecond , Ġthis Ġmatrix Ġis Ġused Ġto Ġconstruct Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġfor Ġeach Ġconcept Ġin Ġa Ġpair Ġof Ġconcepts Ġto Ġbe Ġmeasured Ġfor Ġrelated ness . </s> Co âĢĵ occ urrence ĠMatrix ĠConstruction </s> We Ġbuild Ġan ĠIN LINE FORM 0 Ġsimilarity Ġmatrix Ġusing Ġan Ġexternal Ġcorpus Ġwhere Ġthe Ġrows Ġand Ġcolumns Ġrepresent Ġwords Ġwithin Ġthe Ġcorpus Ġand Ġthe Ġelement Ġcontains Ġthe Ġsimilarity Ġscore Ġbetween Ġthe Ġrow Ġword Ġand Ġcolumn Ġword Ġusing Ġthe Ġsimilarity Ġmeasures Ġdiscussed Ġabove . ĠIf Ġa Ġword Ġmaps Ġto Ġmore Ġthan Ġone Ġpossible Ġsense , Ġwe Ġuse Ġthe Ġsense Ġthat Ġreturns Ġthe Ġhighest Ġsimilarity Ġscore . </s> For Ġthis Ġpaper Ġour Ġexternal Ġcorpus Ġwas Ġthe ĠNL M Ġ2015 ĠMedline Ġbaseline . ĠMedline Ġis Ġa Ġb ibli ographic Ġdatabase Ġcontaining Ġover Ġ23 Ġmillion Ġcitations Ġto Ġjournal Ġarticles Ġin Ġthe Ġbiomedical Ġdomain Ġand Ġis Ġmaintained Ġby ĠNational ĠLibrary Ġof ĠMedicine . ĠThe Ġ2015 ĠMedline ĠBas eline Ġencompasses Ġapproximately Ġ5 , 600 Ġjournals Ġstarting Ġfrom Ġ1948 Ġand Ġcontains Ġ23 , 343 , 329 Ġcitations , Ġof Ġwhich Ġ2 , 579 , 239 Ġcontain Ġabstract s . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠMedline Ġtitles Ġand Ġabstract s Ġfrom Ġ1975 Ġto Ġpresent Ġday . ĠPrior Ġto Ġ1975 , Ġonly Ġ2 % Ġof Ġthe Ġcitations Ġcontained Ġan Ġabstract . ĠWe Ġthen Ġcalculate Ġthe Ġsimilarity Ġfor Ġeach Ġbig ram Ġin Ġthis Ġdataset Ġand Ġinclude Ġthose Ġthat Ġhave Ġa Ġsimilarity Ġscore Ġgreater Ġthan Ġa Ġspecified Ġthreshold Ġon Ġthese Ġexperiments . </s> Measure ĠTerm ĠP airs Ġfor ĠRelated ness </s> We Ġobtain Ġdefinitions Ġfor Ġeach Ġof Ġthe Ġtwo Ġterms Ġwe Ġwish Ġto Ġmeasure . ĠDue Ġto Ġthe Ġsp arsity Ġand Ġinconsistencies Ġof Ġthe Ġdefinitions Ġin Ġthe ĠU ML S , Ġwe Ġnot Ġonly Ġuse Ġthe Ġdefinition Ġof Ġthe Ġterm Ġ( C UI ) Ġbut Ġalso Ġinclude Ġthe Ġdefinition Ġof Ġits Ġrelated Ġconcepts . ĠThis Ġfollows Ġthe Ġmethod Ġproposed Ġby ĠPat ward han P 06 Ġfor Ġgeneral ĠEnglish Ġand ĠWord Net , Ġand Ġwhich Ġwas Ġadapted Ġfor Ġthe ĠU ML S Ġand Ġthe Ġmedical Ġdomain Ġby ĠLiu M PM P 12 . ĠIn Ġparticular Ġwe Ġadd Ġthe Ġdefinitions Ġof Ġany Ġconcepts Ġconnected Ġvia Ġa Ġparent Ġ( PAR ), Ġchild Ġ( CH D ), ĠRB Ġ( bro ader Ġthan ), ĠRN Ġ( n arrow er Ġthan ) Ġor ĠTER M Ġ( terms Ġassociated Ġwith ĠC UI ) Ġrelation . ĠAll Ġof Ġthe Ġdefinitions Ġfor Ġa Ġterm Ġare Ġcombined Ġinto Ġa Ġsingle Ġsuper âĢĵ gl oss . ĠAt Ġthe Ġend Ġof Ġthis Ġprocess Ġwe Ġshould Ġhave Ġtwo Ġsuper âĢĵ gl oss es , Ġone Ġfor Ġeach Ġterm Ġto Ġbe Ġmeasured Ġfor Ġrelated ness . </s> Next , Ġwe Ġprocess Ġeach Ġsuper âĢĵ gl oss Ġas Ġfollows : </s> We Ġextract Ġa Ġfirst âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġfor Ġeach Ġterm Ġin Ġthe Ġsuper âĢĵ gl oss Ġfrom Ġthe Ġco âĢĵ occ urrence Ġmatrix Ġcreated Ġpreviously . </s> We Ġtake Ġthe Ġaverage Ġof Ġthe Ġfirst Ġorder Ġco âĢĵ occ urrence Ġvectors Ġassociated Ġwith Ġthe Ġterms Ġin Ġa Ġsuper âĢĵ gl oss Ġand Ġuse Ġthat Ġto Ġrepresent Ġthe Ġmeaning Ġof Ġthe Ġterm . ĠThis Ġis Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector . </s> After Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġhas Ġbeen Ġconstructed Ġfor Ġeach Ġterm , Ġthen Ġwe Ġcalculate Ġthe Ġcos ine Ġbetween Ġthese Ġtwo Ġvectors Ġto Ġmeasure Ġthe Ġrelated ness Ġof Ġthe Ġterms . </s> Data </s> We Ġuse Ġtwo Ġreference Ġstandards Ġto Ġevaluate Ġthe Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġmeasures Ġ. ĠUM NS RS Ġwas Ġannot ated Ġfor Ġboth Ġsimilarity Ġand Ġrelated ness Ġby Ġmedical Ġresidents . ĠMini May oS RS Ġwas Ġannot ated Ġfor Ġrelated ness Ġby Ġmedical Ġdoctors Ġ( MD ) Ġand Ġmedical Ġcod ers Ġ( c oder ). ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġthese Ġdata Ġsets Ġand Ġdescribe Ġa Ġfew Ġof Ġtheir Ġdifferences . </s> Mini May oS RS : ĠThe ĠMayo S RS , Ġdeveloped Ġby ĠPak hom ov PM M RC 10 , Ġconsists Ġof Ġ101 Ġclinical Ġterm Ġpairs Ġwhose Ġrelated ness Ġwas Ġdetermined Ġby Ġnine Ġmedical Ġcod ers Ġand Ġthree Ġphysicians Ġfrom Ġthe ĠMayo ĠClinic . ĠThe Ġrelated ness Ġof Ġeach Ġterm Ġpair Ġwas Ġassessed Ġbased Ġon Ġa Ġfour Ġpoint Ġscale : Ġ( 4 . 0 ) Ġpractically Ġsynonymous , Ġ( 3 . 0 ) Ġrelated , Ġ( 2 . 0 ) Ġmarginally Ġrelated Ġand Ġ( 1 . 0 ) Ġunrelated . ĠMini May oS RS Ġis Ġa Ġsubset Ġof Ġthe ĠMayo S RS Ġand Ġconsists Ġof Ġ30 Ġterm Ġpairs Ġon Ġwhich Ġa Ġhigher Ġinter âĢĵ annot ator Ġagreement Ġwas Ġachieved . ĠThe Ġaverage Ġcorrelation Ġbetween Ġphysicians Ġis Ġ0 . 68 . ĠThe Ġaverage Ġcorrelation Ġbetween Ġmedical Ġcod ers Ġis Ġ0 . 78 . ĠWe Ġevaluate Ġour Ġmethod Ġon Ġthe Ġmean Ġof Ġthe Ġphysician Ġscores , Ġand Ġthe Ġmean Ġof Ġthe Ġcod ers Ġscores Ġin Ġthis Ġsubset Ġin Ġthe Ġsame Ġmanner Ġas Ġreported Ġby ĠPed ersen P PC 07 . </s> UM NS RS : ĠThe ĠUniversity Ġof ĠMinnesota ĠSem antic ĠRelated ness ĠSet Ġ( UM NS RS ) Ġwas Ġdeveloped Ġby ĠPak hom ov MAL PM 10 , Ġand Ġconsists Ġof Ġ7 25 Ġclinical Ġterm Ġpairs Ġwhose Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġwas Ġdetermined Ġindependently Ġby Ġfour Ġmedical Ġresidents Ġfrom Ġthe ĠUniversity Ġof ĠMinnesota ĠMedical ĠSchool . ĠThe Ġsimilarity Ġand Ġrelated ness Ġof Ġeach Ġterm Ġpair Ġwas Ġannot ated Ġbased Ġon Ġa Ġcontinuous Ġscale Ġby Ġhaving Ġthe Ġresident Ġtouch Ġa Ġbar Ġon Ġa Ġtouch Ġsensitive Ġcomputer Ġscreen Ġto Ġindicate Ġthe Ġdegree Ġof Ġsimilarity Ġor Ġrelated ness . ĠThe ĠInt rac lass ĠCor relation ĠCo efficient Ġ( IC C ) Ġfor Ġthe Ġreference Ġstandard Ġtagged Ġfor Ġsimilarity Ġwas Ġ0 . 47 , Ġand Ġ0 . 50 Ġfor Ġrelated ness . ĠTherefore , Ġas Ġsuggested Ġby ĠPak hom ov Ġand Ġcolleagues , we Ġuse Ġa Ġsubset Ġof Ġthe Ġratings Ġconsisting Ġof Ġ401 Ġpairs Ġfor Ġthe Ġsimilarity Ġset Ġand Ġ430 Ġpairs Ġfor Ġthe Ġrelated ness Ġset Ġwhich Ġeach Ġhave Ġan ĠICC Ġof Ġ0 . 73 . </s> Exper imental ĠFramework </s> We Ġconducted Ġour Ġexperiments Ġusing Ġthe Ġfreely Ġavailable Ġopen Ġsource Ġsoftware Ġpackage ĠU ML S :: Similar ity ĠB IB REF 16 Ġversion Ġ1 . 47 . ĠThis Ġpackage Ġtakes Ġas Ġinput Ġtwo Ġterms Ġ( or ĠU ML S Ġconcepts ) Ġand Ġreturns Ġtheir Ġsimilarity Ġor Ġrelated ness Ġusing Ġthe Ġmeasures Ġdiscussed Ġin ĠSection ĠSEC REF 2 Ġ. </s> Cor relation Ġbetween Ġthe Ġsimilarity Ġmeasures Ġand Ġhuman Ġjudgments Ġwere Ġestimated Ġusing ĠSpear man 's ĠRank ĠCor relation Ġ( ĠIN LINE FORM 0 Ġ). ĠSpear man 's Ġmeasures Ġthe Ġstatistical Ġdependence Ġbetween Ġtwo Ġvariables Ġto Ġassess Ġhow Ġwell Ġthe Ġrelationship Ġbetween Ġthe Ġrankings Ġof Ġthe Ġvariables Ġcan Ġbe Ġdescribed Ġusing Ġa Ġmon ot onic Ġfunction . ĠWe Ġused ĠFisher 's Ġr - to - z Ġtransformation ĠB IB REF 17 Ġto Ġcalculate Ġthe Ġsignificance Ġbetween Ġthe Ġcorrelation Ġresults . </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 26 Ġshows Ġthe ĠSpear man 's ĠRank ĠCor relation Ġbetween Ġthe Ġhuman Ġscores Ġfrom Ġthe Ġfour Ġreference Ġstandards Ġand Ġthe Ġscores Ġfrom Ġthe Ġvarious Ġmeasures Ġof Ġsimilarity Ġintroduced Ġin ĠSection ĠSEC REF 2 Ġ. ĠEach Ġclass Ġof Ġmeasure Ġis Ġfollowed Ġby Ġthe Ġscores Ġobtained Ġwhen Ġintegrating Ġour Ġsecond Ġorder Ġvector Ġapproach Ġwith Ġthese Ġmeasures Ġof Ġsemantic Ġsimilarity . </s> Results ĠComparison </s> The Ġresults Ġfor ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġ( ĠIN LINE FORM 0 Ġ) Ġand ĠMini May oS RS Ġtagged Ġby Ġcod ers Ġshow Ġthat Ġall Ġof Ġthe Ġsecond - order Ġsimilarity Ġvector Ġmeasures Ġ( ĠIN LINE FORM 1 Ġ) Ġexcept Ġfor ĠIN LINE FORM 2 Ġ- ĠIN LINE FORM 3 Ġobtain Ġa Ġhigher Ġcorrelation Ġthan Ġthe Ġoriginal Ġmeasures . ĠWe Ġfound Ġthat ĠIN LINE FORM 4 Ġ- ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ- ĠIN LINE FORM 7 Ġobtain Ġthe Ġhighest Ġcorrelations Ġof Ġall Ġthese Ġresults Ġwith Ġhuman Ġjudgments . </s> For Ġthe ĠUM NS RS Ġdataset Ġtagged Ġfor Ġrelated ness Ġand ĠMini May oS RS Ġtagged Ġby Ġphysicians Ġ( MD ), Ġthe Ġoriginal ĠIN LINE FORM 0 Ġmeasure Ġob tains Ġa Ġhigher Ġcorrelation Ġthan Ġour Ġmeasure Ġ( ĠIN LINE FORM 1 Ġ) Ġalthough Ġthe Ġdifference Ġis Ġnot Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 2 Ġ). </s> In Ġorder Ġto Ġanalyze Ġand Ġbetter Ġunderstand Ġthese Ġresults , Ġwe Ġfiltered Ġthe Ġbig ram Ġpairs Ġused Ġto Ġcreate Ġthe Ġinitial Ġsimilarity Ġmatrix Ġbased Ġon Ġthe Ġstrength Ġof Ġtheir Ġsimilarity Ġusing Ġthe ĠIN LINE FORM 0 Ġand Ġthe ĠIN LINE FORM 1 Ġmeasures . ĠNote Ġthat Ġthe ĠIN LINE FORM 2 Ġmeasure Ġholds Ġto Ġa Ġ0 Ġto Ġ1 Ġscale , Ġwhile ĠIN LINE FORM 3 Ġranges Ġfrom Ġ0 Ġto Ġan Ġunspecified Ġupper Ġbound Ġthat Ġis Ġdependent Ġon Ġthe Ġsize Ġof Ġthe Ġcorpus Ġfrom Ġwhich Ġinformation Ġcontent Ġis Ġestimated . ĠAs Ġsuch Ġwe Ġuse Ġa Ġdifferent Ġrange Ġof Ġthreshold Ġvalues Ġfor Ġeach Ġmeasure . ĠWe Ġdiscuss Ġthe Ġresults Ġof Ġthis Ġfiltering Ġbelow . </s> Th reshold ing ĠExper iments </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults Ġof Ġapplying Ġthe Ġthreshold Ġparameter Ġon Ġeach Ġof Ġthe Ġreference Ġstandards Ġusing Ġthe ĠIN LINE FORM 0 Ġmeasure . ĠFor Ġexample , Ġa Ġthreshold Ġof Ġ0 Ġindicates Ġthat Ġall Ġof Ġthe Ġbig rams Ġwere Ġincluded Ġin Ġthe Ġsimilarity Ġmatrix ; Ġand Ġa Ġthreshold Ġof Ġ1 Ġindicates Ġthat Ġonly Ġthe Ġbig ram Ġpairs Ġwith Ġa Ġsimilarity Ġscore Ġgreater Ġthan Ġone Ġwere Ġincluded . </s> These Ġresults Ġshow Ġthat Ġusing Ġa Ġthreshold Ġcutoff Ġof Ġ2 Ġob tains Ġthe Ġhighest Ġcorrelation Ġfor Ġthe ĠUM NS RS Ġdataset , Ġand Ġthat Ġa Ġthreshold Ġcutoff Ġof Ġ4 Ġob tains Ġthe Ġhighest Ġcorrelation Ġfor Ġthe ĠMini May oS RS Ġdataset . ĠAll Ġof Ġthe Ġresults Ġshow Ġan Ġincrease Ġin Ġcorrelation Ġwith Ġhuman Ġjudgments Ġwhen Ġincorporating Ġa Ġthreshold Ġcutoff Ġover Ġall Ġof Ġthe Ġoriginal Ġmeasures . ĠThe Ġincrease Ġin Ġthe Ġcorrelation Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġis Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ), Ġhowever Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġrelated ness Ġnor Ġfor Ġthe ĠMini May oS RS Ġdata . </s> Similarly , ĠTable ĠT AB REF 30 Ġshows Ġthe Ġresults Ġof Ġapplying Ġthe Ġthreshold Ġparameter Ġ( T ) Ġon Ġeach Ġof Ġthe Ġreference Ġstandards Ġusing Ġthe ĠIN LINE FORM 0 Ġmeasure . ĠAlthough , Ġunlike ĠIN LINE FORM 1 Ġwhose Ġscores Ġare Ġgreater Ġthan Ġor Ġequal Ġto Ġ0 Ġwithout Ġan Ġupper Ġlimit , Ġthe ĠIN LINE FORM 2 Ġmeasure Ġreturns Ġscores Ġbetween Ġ0 Ġand Ġ1 Ġ( in clusive ). ĠTherefore , Ġhere Ġa Ġthreshold Ġof Ġ0 Ġindicates Ġthat Ġall Ġof Ġthe Ġbig rams Ġwere Ġincluded Ġin Ġthe Ġsimilarity Ġmatrix ; Ġand Ġa Ġthreshold Ġof ĠIN LINE FORM 3 Ġindicates Ġthat Ġonly Ġthe Ġbig ram Ġpairs Ġwith Ġa Ġsimilarity Ġscore Ġgreater Ġthan ĠIN LINE FORM 4 Ġwere Ġincluded . ĠThe Ġresults Ġshow Ġan Ġincrease Ġin Ġaccuracy Ġfor Ġall Ġof Ġthe Ġdatasets Ġexcept Ġfor Ġthe ĠMini May oS RS Ġtagged Ġfor Ġphysicians . ĠThe Ġincrease Ġin Ġthe Ġresults Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġand Ġthe ĠMayo S RS Ġis Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 5 Ġ). ĠThis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġrelated ness Ġnor Ġthe ĠMini May oS RS . </s> Overall , Ġthese Ġresults Ġindicate Ġthat Ġincluding Ġonly Ġthose Ġbig rams Ġthat Ġhave Ġa Ġsufficiently Ġhigh Ġsimilarity Ġscore Ġincreases Ġthe Ġcorrelation Ġresults Ġwith Ġhuman Ġjudgments , Ġbut Ġwhat Ġquant ifies Ġas Ġsufficiently Ġhigh Ġvaries Ġdepending Ġon Ġthe Ġdataset Ġand Ġmeasure . </s> Compar ison Ġwith ĠPrevious ĠWork </s> Recently , Ġword Ġembed d ings ĠB IB REF 9 Ġhave Ġbecome Ġa Ġpopular Ġmethod Ġfor Ġmeasuring Ġsemantic Ġrelated ness Ġin Ġthe Ġbiomedical Ġdomain . ĠThis Ġis Ġa Ġneural Ġnetwork Ġbased Ġapproach Ġthat Ġlearns Ġa Ġrepresentation Ġof Ġa Ġword Ġby Ġword Ġco âĢĵ occ urrence Ġmatrix . ĠThe Ġbasic Ġidea Ġis Ġthat Ġthe Ġneural Ġnetwork Ġlearns Ġa Ġseries Ġof Ġweights Ġ( the Ġhidden Ġlayer Ġwithin Ġthe Ġneural Ġnetwork ) Ġthat Ġeither Ġmaxim izes Ġthe Ġprobability Ġof Ġa Ġword Ġgiven Ġits Ġcontext , Ġreferred Ġto Ġas Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġapproach , Ġor Ġthat Ġmaxim izes Ġthe Ġprobability Ġof Ġthe Ġcontext Ġgiven Ġa Ġword , Ġreferred Ġto Ġas Ġthe ĠSkip âĢĵ gram Ġapproach . ĠThese Ġapproaches Ġhave Ġbeen Ġused Ġin Ġnumerous Ġrecent Ġpapers . </s> mun ee b 2015 eval ut ating Ġtrained Ġboth Ġthe ĠSkip âĢĵ gram Ġand ĠCB OW Ġmodels Ġover Ġthe ĠPubMed ĠCentral ĠOpen ĠAccess Ġ( PM C ) Ġcorpus Ġof Ġapproximately Ġ1 . 25 Ġmillion Ġarticles . ĠThey Ġevaluated Ġthe Ġmodels Ġon Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġdata , Ġremoving Ġword Ġpairs Ġthat Ġdid Ġnot Ġoccur Ġin Ġtheir Ġtraining Ġcorpus Ġmore Ġthan Ġten Ġtimes . Ġch iu 2016 how Ġevaluated Ġboth Ġthe Ġthe ĠSkip âĢĵ gram Ġand ĠCB OW Ġmodels Ġover Ġthe ĠPM C Ġcorpus Ġand ĠPubMed . ĠThey Ġalso Ġevaluated Ġthe Ġmodels Ġon Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġignoring Ġthose Ġwords Ġthat Ġdid Ġnot Ġappear Ġin Ġtheir Ġtraining Ġcorpus . ĠPak hom ov 2016 cor p us Ġtrained ĠCB OW Ġmodel Ġover Ġthree Ġdifferent Ġtypes Ġof Ġcorpor a : Ġclinical Ġ( clinical Ġnotes Ġfrom Ġthe ĠFair view ĠHealth ĠSystem ), Ġbiomedical Ġ( PM C Ġcorpus ), Ġand Ġgeneral ĠEnglish Ġ( Wikipedia ). ĠThey Ġevaluated Ġtheir Ġmethod Ġusing Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġrestricting Ġto Ġsingle Ġword Ġterm Ġpairs Ġand Ġremoving Ġthose Ġnot Ġfound Ġwithin Ġtheir Ġtraining Ġcorpus . Ġsa jad 2015 domain Ġtrained Ġthe ĠSkip âĢĵ gram Ġmodel Ġover ĠCU Is Ġidentified Ġby ĠMeta Map Ġon Ġthe ĠO HS UM ED Ġcorpus , Ġa Ġcollection Ġof Ġ348 , 5 66 Ġbiomedical Ġresearch Ġarticles . ĠThey Ġevaluated Ġthe Ġmethod Ġon Ġthe Ġcomplete ĠUM NS RS , ĠMini May oS RS Ġand Ġthe ĠMayo S RS Ġdatasets ; Ġany Ġsubset Ġinformation Ġabout Ġthe Ġdataset Ġwas Ġnot Ġexplicitly Ġstated Ġtherefore Ġwe Ġbelieve Ġa Ġdirect Ġcomparison Ġmay Ġbe Ġpossible . </s> In Ġaddition , Ġa Ġprevious Ġwork Ġvery Ġclosely Ġrelated Ġto Ġours Ġis Ġa Ġretro fitting Ġvector Ġmethod Ġproposed Ġby ĠYu CB J W 16 Ġthat Ġincorporates Ġont ological Ġinformation Ġinto Ġa Ġvector Ġrepresentation Ġby Ġincluding Ġsem antically Ġrelated Ġwords . ĠIn Ġtheir Ġmeasure , Ġthey Ġfirst Ġmap Ġa Ġbiomedical Ġterm Ġto ĠMe SH Ġterms , Ġand Ġsecond Ġbuild Ġa Ġword Ġvector Ġbased Ġon Ġthe Ġdocuments Ġassigned Ġto Ġthe Ġrespective ĠMe SH Ġterm . ĠThey Ġthen Ġretro fit Ġthe Ġvector Ġby Ġincluding Ġsem antically Ġrelated Ġwords Ġfound Ġin Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem . ĠThey Ġevaluate Ġtheir Ġmethod Ġon Ġthe ĠMini May oS RS Ġdataset . </s> Table ĠT AB REF 31 Ġshows Ġa Ġcomparison Ġto Ġthe Ġtop Ġcorrelation Ġscores Ġreported Ġby Ġeach Ġof Ġthese Ġworks Ġon Ġthe Ġrespective Ġdatasets Ġ( or Ġsubs ets ) Ġthey Ġevaluated Ġtheir Ġmethods Ġon . ĠN Ġrefers Ġto Ġthe Ġnumber Ġof Ġterm Ġpairs Ġin Ġthe Ġdataset Ġthe Ġauthors Ġreport Ġthey Ġevaluated Ġtheir Ġmethod . ĠThe Ġtable Ġalso Ġincludes Ġour Ġtop Ġscoring Ġresults : Ġthe Ġintegrated Ġvector - res Ġand Ġvector - faith . ĠThe Ġresults Ġshow Ġthat Ġintegrating Ġsemantic Ġsimilarity Ġmeasures Ġinto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors Ġob tains Ġa Ġhigher Ġor Ġon âĢĵ par Ġcorrelation Ġwith Ġhuman Ġjudgments Ġas Ġthe Ġprevious Ġworks Ġreported Ġresults Ġwith Ġthe Ġexception Ġof Ġthe ĠUM NS RS Ġrel Ġdataset . ĠThe Ġresults Ġreported Ġby ĠPak hom ov 2016 cor p us Ġand Ġch iu 2016 how Ġobtain Ġa Ġhigher Ġcorrelation Ġalthough Ġthe Ġresults Ġcan Ġnot Ġbe Ġdirectly Ġcompared Ġbecause Ġboth Ġworks Ġused Ġdifferent Ġsubs ets Ġof Ġthe Ġterm Ġpairs Ġfrom Ġthe ĠUM NS RS Ġdataset . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġhave Ġpresented Ġa Ġmethod Ġfor Ġquant ifying Ġthe Ġsimilarity Ġand Ġrelated ness Ġbetween Ġtwo Ġterms Ġthat Ġintegrates Ġpair âĢĵ wise Ġsimilarity Ġscores Ġinto Ġsecond âĢĵ order Ġvectors . ĠThe Ġgoal Ġof Ġthis Ġapproach Ġis Ġtwo âĢĵ fold . ĠFirst , Ġwe Ġrestrict Ġthe Ġcontext Ġused Ġby Ġthe Ġvector Ġmeasure Ġto Ġwords Ġthat Ġexist Ġin Ġthe Ġbiomedical Ġdomain , Ġand Ġsecond , Ġwe Ġapply Ġlarger Ġweights Ġto Ġthose Ġword Ġpairs Ġthat Ġare Ġmore Ġsimilar Ġto Ġeach Ġother . ĠOur Ġhypothesis Ġwas Ġthat Ġthis Ġcombination Ġwould Ġreduce Ġthe Ġamount Ġof Ġnoise Ġin Ġthe Ġvectors Ġand Ġtherefore Ġincrease Ġtheir Ġcorrelation Ġwith Ġhuman Ġjudgments . ĠWe Ġevaluated Ġour Ġmethod Ġon Ġdatasets Ġthat Ġhave Ġbeen Ġmanually Ġannot ated Ġfor Ġrelated ness Ġand Ġsimilarity Ġand Ġfound Ġevidence Ġto Ġsupport Ġthis Ġhypothesis . ĠIn Ġparticular Ġwe Ġdiscovered Ġthat Ġguiding Ġthe Ġcreation Ġof Ġa Ġsecond âĢĵ order Ġcontext Ġvector Ġby Ġselecting Ġterm Ġpairs Ġfrom Ġbiomedical Ġtext Ġbased Ġon Ġtheir Ġsemantic Ġsimilarity Ġled Ġto Ġimproved Ġlevels Ġof Ġcorrelation Ġwith Ġhuman Ġjudgment . </s> We Ġalso Ġexplored Ġusing Ġa Ġthreshold Ġcutoff Ġto Ġinclude Ġonly Ġthose Ġterm Ġpairs Ġthat Ġobtained Ġa Ġsufficiently Ġlarge Ġlevel Ġof Ġsimilarity . ĠWe Ġfound Ġthat Ġeliminating Ġless Ġsimilar Ġpairs Ġimproved Ġthe Ġoverall Ġresults Ġ( to Ġa Ġpoint ). ĠIn Ġthe Ġfuture , Ġwe Ġplan Ġto Ġexplore Ġmetrics Ġto Ġautomatically Ġdetermine Ġthe Ġthreshold Ġcutoff Ġappropriate Ġfor Ġa Ġgiven Ġdataset Ġand Ġmeasure . ĠWe Ġalso Ġplan Ġto Ġexplore Ġadditional Ġfeatures Ġthat Ġcan Ġbe Ġintegrated Ġwith Ġa Ġsecond âĢĵ order Ġvector Ġmeasure Ġthat Ġwill Ġreduce Ġthe Ġnoise Ġbut Ġstill Ġprovide Ġsufficient Ġinformation Ġto Ġquantify Ġrelated ness . ĠWe Ġare Ġparticularly Ġinterested Ġin Ġapproaches Ġthat Ġlearn Ġword , Ġphrase , Ġand Ġsentence Ġembed d ings Ġfrom Ġstructured Ġcorpor a Ġsuch Ġas Ġliterature ĠB IB REF 23 Ġand Ġdictionary Ġentries ĠB IB REF 24 Ġ. ĠSuch Ġembed d ings Ġcould Ġbe Ġintegrated Ġinto Ġa Ġsecond âĢĵ order Ġvector Ġor Ġbe Ġused Ġon Ġtheir Ġown . </s> Finally , Ġwe Ġcompared Ġour Ġproposed Ġmethod Ġto Ġother Ġdistribution al Ġapproaches , Ġfocusing Ġon Ġthose Ġthat Ġused Ġword Ġembed d ings . ĠOur Ġresults Ġshowed Ġthat Ġintegrating Ġsemantic Ġsimilarity Ġmeasures Ġinto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors Ġob tains Ġthe Ġsame Ġor Ġhigher Ġcorrelation Ġwith Ġhuman Ġjudgments Ġas Ġdo Ġvarious Ġdifferent Ġword Ġembed ding Ġapproaches . ĠHowever , Ġa Ġdirect Ġcomparison Ġwas Ġnot Ġpossible Ġdue Ġto Ġvariations Ġin Ġthe Ġsubs ets Ġof Ġthe ĠUM NS RS Ġevaluation Ġdataset Ġused . ĠIn Ġthe Ġfuture , Ġwe Ġwould Ġnot Ġonly Ġlike Ġto Ġconduct Ġa Ġdirect Ġcomparison Ġbut Ġalso Ġexplore Ġintegrating Ġsemantic Ġsimilarity Ġinto Ġvarious Ġkinds Ġof Ġword Ġembed d ings Ġby Ġtraining Ġon Ġpair âĢĵ wise Ġvalues Ġof Ġsemantic Ġsimilarity Ġas Ġwell Ġas Ġco âĢĵ occ urrence Ġstatistics .
Introduction Measures of semantic similarity and relatedness quantify the degree to which two concepts are similar (e.g., INLINEFORM0 – INLINEFORM1 ) or related (e.g., INLINEFORM2 – INLINEFORM3 ). Semantic similarity can be viewed as a special case of semantic relatedness – to be similar is one of many ways that a pair of concepts may be related. The automated discovery of groups of semantically similar or related terms is critical to improving the retrieval BIBREF0 and clustering BIBREF1 of biomedical and clinical documents, and the development of biomedical terminologies and ontologies BIBREF2 . There is a long history in using distributional methods to discover semantic similarity and relatedness (e.g., BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 ). These methods are all based on the distributional hypothesis, which holds that two terms that are distributionally similar (i.e., used in the same context) will also be semantically similar BIBREF7 , BIBREF8 . Recently word embedding techniques such as word2vec BIBREF9 have become very popular. Despite the prominent role that neural networks play in many of these approaches, at their core they remain distributional techniques that typically start with a word by word co–occurrence matrix, much like many of the more traditional approaches. However, despite these successes distributional methods do not perform well when data is very sparse (which is common). One possible solution is to use second–order co–occurrence vectors BIBREF10 , BIBREF11 . In this approach the similarity between two words is not strictly based on their co–occurrence frequencies, but rather on the frequencies of the other words which occur with both of them (i.e., second order co–occurrences). This approach has been shown to be successful in quantifying semantic relatedness BIBREF12 , BIBREF13 . However, while more robust in the face of sparsity, second–order methods can result in significant amounts of noise, where contextual information that is overly general is included and does not contribute to quantifying the semantic relatedness between the two concepts. Our goal then is to discover methods that automatically reduce the amount of noise in a second–order co–occurrence vector. We achieve this by incorporating pairwise semantic similarity scores derived from a taxonomy into our second–order vectors, and then using these scores to select only the most semantically similar co–occurrences (thereby reducing noise). We evaluate our method on two datasets that have been annotated in multiple ways. One has been annotated for both similarity and relatedness, and the other has been annotated for relatedness by two different types of experts (medical doctors and medical coders). Our results show that integrating second order co–occurrences with measures of semantic similarity increases correlation with our human reference standards. We also compare our result to a number of other studies which have applied various word embedding methods to the same reference standards we have used. We find that our method often performs at a comparable or higher level than these approaches. These results suggest that our methods of integrating semantic similarity and relatedness values have the potential to improve performance of purely distributional methods. Similarity and Relatedness Measures This section describes the similarity and relatedness measures we integrate in our second–order co–occurrence vectors. We use two taxonomies in this study, SNOMED–CT and MeSH. SNOMED–CT (Systematized Nomenclature of Medicine Clinical Terms) is a comprehensive clinical terminology created for the electronic representation of clinical health information. MeSH (Medical Subject Headings) is a taxonomy of biomedical terms developed for indexing biomedical journal articles. We obtain SNOMED–CT and MeSH via the Unified Medical Language System (UMLS) Metathesaurus (version 2016AA). The Metathesaurus contains approximately 2 million biomedical and clinical concepts from over 150 different terminologies that have been semi–automatically integrated into a single source. Concepts in the Metathesaurus are connected largely by two types of hierarchical relations: INLINEFORM0 / INLINEFORM1 (PAR/CHD) and INLINEFORM2 / INLINEFORM3 (RB/RN). Similarity Measures Measures of semantic similarity can be classified into three broad categories : path–based, feature–based and information content (IC). Path–based similarity measures use the structure of a taxonomy to measure similarity – concepts positioned close to each other are more similar than those further apart. Feature–based methods rely on set theoretic measures of overlap between features (union and intersection). The information content measures quantify the amount of information that a concept provides – more specific concepts have a higher amount of information content. RadaMBB89 introduce the Conceptual Distance measure. This measure is simply the length of the shortest path between two concepts ( INLINEFORM0 and INLINEFORM1 ) in the MeSH hierarchy. Paths are based on broader than (RB) and narrower than (RN) relations. CaviedesC04 extends this measure to use parent (PAR) and child (CHD) relations. Our INLINEFORM2 measure is simply the reciprocal of this shortest path value (Equation EQREF3 ), so that larger values (approaching 1) indicate a high degree of similarity. DISPLAYFORM0 While the simplicity of INLINEFORM0 is appealing, it can be misleading when concepts are at different levels of specificity. Two very general concepts may have the same path length as two very specific concepts. WuP94 introduce a correction to INLINEFORM1 that incorporates the depth of the concepts, and the depth of their Least Common Subsumer (LCS). This is the most specific ancestor two concepts share. In this measure, similarity is twice the depth of the two concept's LCS divided by the product of the depths of the individual concepts (Equation EQREF4 ). Note that if there are multiple LCSs for a pair of concepts, the deepest of them is used in this measure. DISPLAYFORM0 ZhongZLY02 take a very similar approach and again scale the depth of the LCS by the sum of the depths of the two concepts (Equation EQREF5 ), where INLINEFORM0 . The value of INLINEFORM1 was set to 2 based on their recommendations. DISPLAYFORM0 PekarS02 offer another variation on INLINEFORM0 , where the shortest path of the two concepts to the LCS is used, in addition to the shortest bath between the LCS and the root of the taxonomy (Equation EQREF6 ). DISPLAYFORM0 Feature–based methods represent each concept as a set of features and then measure the overlap or sharing of features to measure similarity. In particular, each concept is represented as the set of their ancestors, and similarity is a ratio of the intersection and union of these features. MaedcheS01 quantify the similarity between two concepts as the ratio of the intersection over their union as shown in Equation EQREF8 . DISPLAYFORM0 BatetSV11 extend this by excluding any shared features (in the numerator) as shown in Equation EQREF9 . DISPLAYFORM0 Information content is formally defined as the negative log of the probability of a concept. The effect of this is to assign rare (low probability) concepts a high measure of information content, since the underlying assumption is that more specific concepts are less frequently used than more common ones. Resnik95 modified this notion of information content in order to use it as a similarity measure. He defines the similarity of two concepts to be the information content of their LCS (Equation EQREF11 ). DISPLAYFORM0 JiangC97, Lin98, and PirroE10 extend INLINEFORM0 by incorporating the information content of the individual concepts in various different ways. Lin98 defines the similarity between two concepts as the ratio of information content of the LCS with the sum of the individual concept's information content (Equation EQREF12 ). Note that INLINEFORM1 has the same form as INLINEFORM2 and INLINEFORM3 , and is in effect using information content as a measure of specificity (rather than depth). If there is more than one possible LCS, the LCS with the greatest IC is chosen. DISPLAYFORM0 JiangC97 define the distance between two concepts to be the sum of the information content of the two concepts minus twice the information content of the concepts' LCS. We modify this from a distance to a similarity measure by taking the reciprocal of the distance (Equation EQREF13 ). Note that the denominator of INLINEFORM0 is very similar to the numerator of INLINEFORM1 . DISPLAYFORM0 PirroE10 define the similarity between two concepts as the information content of the two concept's LCS divided by the sum of their individual information content values minus the information content of their LCS (Equation EQREF14 ). Note that INLINEFORM0 can be viewed as a set–theoretic version of INLINEFORM1 . DISPLAYFORM0 Information Content The information content of a concept may be derived from a corpus (corpus–based) or directly from a taxonomy (intrinsic–based). In this work we focus on corpus–based techniques. For corpus–based information content, we estimate the probability of a concept INLINEFORM0 by taking the sum of the probability of the concept INLINEFORM1 and the probability its descendants INLINEFORM2 (Equation EQREF16 ). DISPLAYFORM0 The initial probabilities of a concept ( INLINEFORM0 ) and its descendants ( INLINEFORM1 ) are obtained by dividing the number of times each concept and descendant occurs in the corpus, and dividing that by the total numbers of concepts ( INLINEFORM2 ). Ideally the corpus from which we are estimating the probabilities of concepts will be sense–tagged. However, sense–tagging is a challenging problem in its own right, and it is not always possible to carry out reliably on larger amounts of text. In fact in this paper we did not use any sense–tagging of the corpus we derived information content from. Instead, we estimated the probability of a concept by using the UMLSonMedline dataset. This was created by the National Library of Medicine and consists of concepts from the 2009AB UMLS and the counts of the number of times they occurred in a snapshot of Medline taken on 12 January, 2009. These counts were obtained by using the Essie Search Engine BIBREF14 which queried Medline with normalized strings from the 2009AB MRCONSO table in the UMLS. The frequency of a CUI was obtained by aggregating the frequency counts of the terms associated with the CUI to provide a rough estimate of its frequency. The information content measures then use this information to calculate the probability of a concept. Another alternative is the use of Intrinsic Information Content. It assess the informativeness of concept based on its placement within a taxonomy by considering the number of incoming (ancestors) relative to outgoing (descendant) links BIBREF15 (Equation EQREF17 ). DISPLAYFORM0 where INLINEFORM0 are the number of descendants of concept INLINEFORM1 that are leaf nodes, INLINEFORM2 are the number of concept INLINEFORM3 's ancestors and INLINEFORM4 are the total number of leaf nodes in the taxonomy. Relatedness Measures Lesk86 observed that concepts that are related should share more words in their respective definitions than concepts that are less connected. He was able to perform word sense disambiguation by identifying the senses of words in a sentence with the largest number of overlaps between their definitions. An overlap is the longest sequence of one or more consecutive words that occur in both definitions. BanerjeeP03 extended this idea to WordNet, but observed that WordNet glosses are often very short, and did not contain enough information to distinguish between multiple concepts. Therefore, they created a super–gloss for each concept by adding the glosses of related concepts to the gloss of the concept itself (and then finding overlaps). PatwardhanP06 adapted this measure to second–order co–occurrence vectors. In this approach, a vector is created for each word in a concept's definition that shows which words co–occur with it in a corpus. These word vectors are averaged to create a single co-occurrence vector for the concept. The similarity between the concepts is calculated by taking the cosine between the concepts second–order vectors. LiuMPMP12 modified and extended this measure to be used to quantify the relatedness between biomedical and clinical terms in the UMLS. The work in this paper can be seen as a further extension of PatwardhanP06 and LiuMPMP12. Method In this section, we describe our second–order similarity vector measure. This incorporates both contextual information using the term pair's definition and their pairwise semantic similarity scores derived from a taxonomy. There are two stages to our approach. First, a co–occurrence matrix must be constructed. Second, this matrix is used to construct a second–order co–occurrence vector for each concept in a pair of concepts to be measured for relatedness. Co–occurrence Matrix Construction We build an INLINEFORM0 similarity matrix using an external corpus where the rows and columns represent words within the corpus and the element contains the similarity score between the row word and column word using the similarity measures discussed above. If a word maps to more than one possible sense, we use the sense that returns the highest similarity score. For this paper our external corpus was the NLM 2015 Medline baseline. Medline is a bibliographic database containing over 23 million citations to journal articles in the biomedical domain and is maintained by National Library of Medicine. The 2015 Medline Baseline encompasses approximately 5,600 journals starting from 1948 and contains 23,343,329 citations, of which 2,579,239 contain abstracts. In this work, we use Medline titles and abstracts from 1975 to present day. Prior to 1975, only 2% of the citations contained an abstract. We then calculate the similarity for each bigram in this dataset and include those that have a similarity score greater than a specified threshold on these experiments. Measure Term Pairs for Relatedness We obtain definitions for each of the two terms we wish to measure. Due to the sparsity and inconsistencies of the definitions in the UMLS, we not only use the definition of the term (CUI) but also include the definition of its related concepts. This follows the method proposed by PatwardhanP06 for general English and WordNet, and which was adapted for the UMLS and the medical domain by LiuMPMP12. In particular we add the definitions of any concepts connected via a parent (PAR), child (CHD), RB (broader than), RN (narrower than) or TERM (terms associated with CUI) relation. All of the definitions for a term are combined into a single super–gloss. At the end of this process we should have two super–glosses, one for each term to be measured for relatedness. Next, we process each super–gloss as follows: We extract a first–order co–occurrence vector for each term in the super–gloss from the co–occurrence matrix created previously. We take the average of the first order co–occurrence vectors associated with the terms in a super–gloss and use that to represent the meaning of the term. This is a second–order co–occurrence vector. After a second–order co–occurrence vector has been constructed for each term, then we calculate the cosine between these two vectors to measure the relatedness of the terms. Data We use two reference standards to evaluate the semantic similarity and relatedness measures . UMNSRS was annotated for both similarity and relatedness by medical residents. MiniMayoSRS was annotated for relatedness by medical doctors (MD) and medical coders (coder). In this section, we describe these data sets and describe a few of their differences. MiniMayoSRS: The MayoSRS, developed by PakhomovPMMRC10, consists of 101 clinical term pairs whose relatedness was determined by nine medical coders and three physicians from the Mayo Clinic. The relatedness of each term pair was assessed based on a four point scale: (4.0) practically synonymous, (3.0) related, (2.0) marginally related and (1.0) unrelated. MiniMayoSRS is a subset of the MayoSRS and consists of 30 term pairs on which a higher inter–annotator agreement was achieved. The average correlation between physicians is 0.68. The average correlation between medical coders is 0.78. We evaluate our method on the mean of the physician scores, and the mean of the coders scores in this subset in the same manner as reported by PedersenPPC07. UMNSRS: The University of Minnesota Semantic Relatedness Set (UMNSRS) was developed by PakhomovMALPM10, and consists of 725 clinical term pairs whose semantic similarity and relatedness was determined independently by four medical residents from the University of Minnesota Medical School. The similarity and relatedness of each term pair was annotated based on a continuous scale by having the resident touch a bar on a touch sensitive computer screen to indicate the degree of similarity or relatedness. The Intraclass Correlation Coefficient (ICC) for the reference standard tagged for similarity was 0.47, and 0.50 for relatedness. Therefore, as suggested by Pakhomov and colleagues,we use a subset of the ratings consisting of 401 pairs for the similarity set and 430 pairs for the relatedness set which each have an ICC of 0.73. Experimental Framework We conducted our experiments using the freely available open source software package UMLS::Similarity BIBREF16 version 1.47. This package takes as input two terms (or UMLS concepts) and returns their similarity or relatedness using the measures discussed in Section SECREF2 . Correlation between the similarity measures and human judgments were estimated using Spearman's Rank Correlation ( INLINEFORM0 ). Spearman's measures the statistical dependence between two variables to assess how well the relationship between the rankings of the variables can be described using a monotonic function. We used Fisher's r-to-z transformation BIBREF17 to calculate the significance between the correlation results. Results and Discussion Table TABREF26 shows the Spearman's Rank Correlation between the human scores from the four reference standards and the scores from the various measures of similarity introduced in Section SECREF2 . Each class of measure is followed by the scores obtained when integrating our second order vector approach with these measures of semantic similarity. Results Comparison The results for UMNSRS tagged for similarity ( INLINEFORM0 ) and MiniMayoSRS tagged by coders show that all of the second-order similarity vector measures ( INLINEFORM1 ) except for INLINEFORM2 - INLINEFORM3 obtain a higher correlation than the original measures. We found that INLINEFORM4 - INLINEFORM5 and INLINEFORM6 - INLINEFORM7 obtain the highest correlations of all these results with human judgments. For the UMNSRS dataset tagged for relatedness and MiniMayoSRS tagged by physicians (MD), the original INLINEFORM0 measure obtains a higher correlation than our measure ( INLINEFORM1 ) although the difference is not statistically significant ( INLINEFORM2 ). In order to analyze and better understand these results, we filtered the bigram pairs used to create the initial similarity matrix based on the strength of their similarity using the INLINEFORM0 and the INLINEFORM1 measures. Note that the INLINEFORM2 measure holds to a 0 to 1 scale, while INLINEFORM3 ranges from 0 to an unspecified upper bound that is dependent on the size of the corpus from which information content is estimated. As such we use a different range of threshold values for each measure. We discuss the results of this filtering below. Thresholding Experiments Table TABREF29 shows the results of applying the threshold parameter on each of the reference standards using the INLINEFORM0 measure. For example, a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of 1 indicates that only the bigram pairs with a similarity score greater than one were included. These results show that using a threshold cutoff of 2 obtains the highest correlation for the UMNSRS dataset, and that a threshold cutoff of 4 obtains the highest correlation for the MiniMayoSRS dataset. All of the results show an increase in correlation with human judgments when incorporating a threshold cutoff over all of the original measures. The increase in the correlation for the UMNSRS tagged for similarity is statistically significant ( INLINEFORM0 ), however this is not the case for the UMNSRS tagged for relatedness nor for the MiniMayoSRS data. Similarly, Table TABREF30 shows the results of applying the threshold parameter (T) on each of the reference standards using the INLINEFORM0 measure. Although, unlike INLINEFORM1 whose scores are greater than or equal to 0 without an upper limit, the INLINEFORM2 measure returns scores between 0 and 1 (inclusive). Therefore, here a threshold of 0 indicates that all of the bigrams were included in the similarity matrix; and a threshold of INLINEFORM3 indicates that only the bigram pairs with a similarity score greater than INLINEFORM4 were included. The results show an increase in accuracy for all of the datasets except for the MiniMayoSRS tagged for physicians. The increase in the results for the UMNSRS tagged for similarity and the MayoSRS is statistically significant ( INLINEFORM5 ). This is not the case for the UMNSRS tagged for relatedness nor the MiniMayoSRS. Overall, these results indicate that including only those bigrams that have a sufficiently high similarity score increases the correlation results with human judgments, but what quantifies as sufficiently high varies depending on the dataset and measure. Comparison with Previous Work Recently, word embeddings BIBREF9 have become a popular method for measuring semantic relatedness in the biomedical domain. This is a neural network based approach that learns a representation of a word by word co–occurrence matrix. The basic idea is that the neural network learns a series of weights (the hidden layer within the neural network) that either maximizes the probability of a word given its context, referred to as the continuous bag of words (CBOW) approach, or that maximizes the probability of the context given a word, referred to as the Skip–gram approach. These approaches have been used in numerous recent papers. muneeb2015evalutating trained both the Skip–gram and CBOW models over the PubMed Central Open Access (PMC) corpus of approximately 1.25 million articles. They evaluated the models on a subset of the UMNSRS data, removing word pairs that did not occur in their training corpus more than ten times. chiu2016how evaluated both the the Skip–gram and CBOW models over the PMC corpus and PubMed. They also evaluated the models on a subset of the UMNSRS ignoring those words that did not appear in their training corpus. Pakhomov2016corpus trained CBOW model over three different types of corpora: clinical (clinical notes from the Fairview Health System), biomedical (PMC corpus), and general English (Wikipedia). They evaluated their method using a subset of the UMNSRS restricting to single word term pairs and removing those not found within their training corpus. sajad2015domain trained the Skip–gram model over CUIs identified by MetaMap on the OHSUMED corpus, a collection of 348,566 biomedical research articles. They evaluated the method on the complete UMNSRS, MiniMayoSRS and the MayoSRS datasets; any subset information about the dataset was not explicitly stated therefore we believe a direct comparison may be possible. In addition, a previous work very closely related to ours is a retrofitting vector method proposed by YuCBJW16 that incorporates ontological information into a vector representation by including semantically related words. In their measure, they first map a biomedical term to MeSH terms, and second build a word vector based on the documents assigned to the respective MeSH term. They then retrofit the vector by including semantically related words found in the Unified Medical Language System. They evaluate their method on the MiniMayoSRS dataset. Table TABREF31 shows a comparison to the top correlation scores reported by each of these works on the respective datasets (or subsets) they evaluated their methods on. N refers to the number of term pairs in the dataset the authors report they evaluated their method. The table also includes our top scoring results: the integrated vector-res and vector-faith. The results show that integrating semantic similarity measures into second–order co–occurrence vectors obtains a higher or on–par correlation with human judgments as the previous works reported results with the exception of the UMNSRS rel dataset. The results reported by Pakhomov2016corpus and chiu2016how obtain a higher correlation although the results can not be directly compared because both works used different subsets of the term pairs from the UMNSRS dataset. Conclusion and Future Work We have presented a method for quantifying the similarity and relatedness between two terms that integrates pair–wise similarity scores into second–order vectors. The goal of this approach is two–fold. First, we restrict the context used by the vector measure to words that exist in the biomedical domain, and second, we apply larger weights to those word pairs that are more similar to each other. Our hypothesis was that this combination would reduce the amount of noise in the vectors and therefore increase their correlation with human judgments. We evaluated our method on datasets that have been manually annotated for relatedness and similarity and found evidence to support this hypothesis. In particular we discovered that guiding the creation of a second–order context vector by selecting term pairs from biomedical text based on their semantic similarity led to improved levels of correlation with human judgment. We also explored using a threshold cutoff to include only those term pairs that obtained a sufficiently large level of similarity. We found that eliminating less similar pairs improved the overall results (to a point). In the future, we plan to explore metrics to automatically determine the threshold cutoff appropriate for a given dataset and measure. We also plan to explore additional features that can be integrated with a second–order vector measure that will reduce the noise but still provide sufficient information to quantify relatedness. We are particularly interested in approaches that learn word, phrase, and sentence embeddings from structured corpora such as literature BIBREF23 and dictionary entries BIBREF24 . Such embeddings could be integrated into a second–order vector or be used on their own. Finally, we compared our proposed method to other distributional approaches, focusing on those that used word embeddings. Our results showed that integrating semantic similarity measures into second–order co–occurrence vectors obtains the same or higher correlation with human judgments as do various different word embedding approaches. However, a direct comparison was not possible due to variations in the subsets of the UMNSRS evaluation dataset used. In the future, we would not only like to conduct a direct comparison but also explore integrating semantic similarity into various kinds of word embeddings by training on pair–wise values of semantic similarity as well as co–occurrence statistics.
What embedding techniques are explored in the paper?
8b3d3953454c88bde88181897a7a2c0c8dd87e23
What Ġembed ding Ġtechniques Ġare Ġexplored Ġin Ġthe Ġpaper ?
Skip âĢĵ gram , ĠCB OW
<s> What Ġembed ding Ġtechniques Ġare Ġexplored Ġin Ġthe Ġpaper ? </s> Introduction </s> Me asures Ġof Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġquantify Ġthe Ġdegree Ġto Ġwhich Ġtwo Ġconcepts Ġare Ġsimilar Ġ( e . g ., ĠIN LINE FORM 0 ĠâĢĵ ĠIN LINE FORM 1 Ġ) Ġor Ġrelated Ġ( e . g ., ĠIN LINE FORM 2 ĠâĢĵ ĠIN LINE FORM 3 Ġ). ĠSem antic Ġsimilarity Ġcan Ġbe Ġviewed Ġas Ġa Ġspecial Ġcase Ġof Ġsemantic Ġrelated ness ĠâĢĵ Ġto Ġbe Ġsimilar Ġis Ġone Ġof Ġmany Ġways Ġthat Ġa Ġpair Ġof Ġconcepts Ġmay Ġbe Ġrelated . ĠThe Ġautomated Ġdiscovery Ġof Ġgroups Ġof Ġsem antically Ġsimilar Ġor Ġrelated Ġterms Ġis Ġcritical Ġto Ġimproving Ġthe Ġretrieval ĠB IB REF 0 Ġand Ġclust ering ĠB IB REF 1 Ġof Ġbiomedical Ġand Ġclinical Ġdocuments , Ġand Ġthe Ġdevelopment Ġof Ġbiomedical Ġtermin ologies Ġand Ġont ologies ĠB IB REF 2 Ġ. </s> There Ġis Ġa Ġlong Ġhistory Ġin Ġusing Ġdistribution al Ġmethods Ġto Ġdiscover Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġ( e . g ., ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ). ĠThese Ġmethods Ġare Ġall Ġbased Ġon Ġthe Ġdistribution al Ġhypothesis , Ġwhich Ġholds Ġthat Ġtwo Ġterms Ġthat Ġare Ġdistribution ally Ġsimilar Ġ( i . e ., Ġused Ġin Ġthe Ġsame Ġcontext ) Ġwill Ġalso Ġbe Ġsem antically Ġsimilar ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠRecently Ġword Ġembed ding Ġtechniques Ġsuch Ġas Ġword 2 vec ĠB IB REF 9 Ġhave Ġbecome Ġvery Ġpopular . ĠDespite Ġthe Ġprominent Ġrole Ġthat Ġneural Ġnetworks Ġplay Ġin Ġmany Ġof Ġthese Ġapproaches , Ġat Ġtheir Ġcore Ġthey Ġremain Ġdistribution al Ġtechniques Ġthat Ġtypically Ġstart Ġwith Ġa Ġword Ġby Ġword Ġco âĢĵ occ urrence Ġmatrix , Ġmuch Ġlike Ġmany Ġof Ġthe Ġmore Ġtraditional Ġapproaches . </s> However , Ġdespite Ġthese Ġsuccesses Ġdistribution al Ġmethods Ġdo Ġnot Ġperform Ġwell Ġwhen Ġdata Ġis Ġvery Ġsparse Ġ( which Ġis Ġcommon ). ĠOne Ġpossible Ġsolution Ġis Ġto Ġuse Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. ĠIn Ġthis Ġapproach Ġthe Ġsimilarity Ġbetween Ġtwo Ġwords Ġis Ġnot Ġstrictly Ġbased Ġon Ġtheir Ġco âĢĵ occ urrence Ġfrequencies , Ġbut Ġrather Ġon Ġthe Ġfrequencies Ġof Ġthe Ġother Ġwords Ġwhich Ġoccur Ġwith Ġboth Ġof Ġthem Ġ( i . e ., Ġsecond Ġorder Ġco âĢĵ occ ur rences ). ĠThis Ġapproach Ġhas Ġbeen Ġshown Ġto Ġbe Ġsuccessful Ġin Ġquant ifying Ġsemantic Ġrelated ness ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠHowever , Ġwhile Ġmore Ġrobust Ġin Ġthe Ġface Ġof Ġsp arsity , Ġsecond âĢĵ order Ġmethods Ġcan Ġresult Ġin Ġsignificant Ġamounts Ġof Ġnoise , Ġwhere Ġcontextual Ġinformation Ġthat Ġis Ġoverly Ġgeneral Ġis Ġincluded Ġand Ġdoes Ġnot Ġcontribute Ġto Ġquant ifying Ġthe Ġsemantic Ġrelated ness Ġbetween Ġthe Ġtwo Ġconcepts . </s> Our Ġgoal Ġthen Ġis Ġto Ġdiscover Ġmethods Ġthat Ġautomatically Ġreduce Ġthe Ġamount Ġof Ġnoise Ġin Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector . ĠWe Ġachieve Ġthis Ġby Ġincorporating Ġpair wise Ġsemantic Ġsimilarity Ġscores Ġderived Ġfrom Ġa Ġtax onomy Ġinto Ġour Ġsecond âĢĵ order Ġvectors , Ġand Ġthen Ġusing Ġthese Ġscores Ġto Ġselect Ġonly Ġthe Ġmost Ġsem antically Ġsimilar Ġco âĢĵ occ ur rences Ġ( there by Ġreducing Ġnoise ). </s> We Ġevaluate Ġour Ġmethod Ġon Ġtwo Ġdatasets Ġthat Ġhave Ġbeen Ġannot ated Ġin Ġmultiple Ġways . ĠOne Ġhas Ġbeen Ġannot ated Ġfor Ġboth Ġsimilarity Ġand Ġrelated ness , Ġand Ġthe Ġother Ġhas Ġbeen Ġannot ated Ġfor Ġrelated ness Ġby Ġtwo Ġdifferent Ġtypes Ġof Ġexperts Ġ( medical Ġdoctors Ġand Ġmedical Ġcod ers ). ĠOur Ġresults Ġshow Ġthat Ġintegrating Ġsecond Ġorder Ġco âĢĵ occ ur rences Ġwith Ġmeasures Ġof Ġsemantic Ġsimilarity Ġincreases Ġcorrelation Ġwith Ġour Ġhuman Ġreference Ġstandards . ĠWe Ġalso Ġcompare Ġour Ġresult Ġto Ġa Ġnumber Ġof Ġother Ġstudies Ġwhich Ġhave Ġapplied Ġvarious Ġword Ġembed ding Ġmethods Ġto Ġthe Ġsame Ġreference Ġstandards Ġwe Ġhave Ġused . ĠWe Ġfind Ġthat Ġour Ġmethod Ġoften Ġperforms Ġat Ġa Ġcomparable Ġor Ġhigher Ġlevel Ġthan Ġthese Ġapproaches . ĠThese Ġresults Ġsuggest Ġthat Ġour Ġmethods Ġof Ġintegrating Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġvalues Ġhave Ġthe Ġpotential Ġto Ġimprove Ġperformance Ġof Ġpurely Ġdistribution al Ġmethods . </s> Similar ity Ġand ĠRelated ness ĠMeasures </s> This Ġsection Ġdescribes Ġthe Ġsimilarity Ġand Ġrelated ness Ġmeasures Ġwe Ġintegrate Ġin Ġour Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors . ĠWe Ġuse Ġtwo Ġtax onom ies Ġin Ġthis Ġstudy , ĠSN OM ED âĢĵ CT Ġand ĠMe SH . ĠSN OM ED âĢĵ CT Ġ( System at ized ĠN omen cl ature Ġof ĠMedicine ĠClinical ĠTerms ) Ġis Ġa Ġcomprehensive Ġclinical Ġterminology Ġcreated Ġfor Ġthe Ġelectronic Ġrepresentation Ġof Ġclinical Ġhealth Ġinformation . ĠMe SH Ġ( Medical ĠSubject ĠHead ings ) Ġis Ġa Ġtax onomy Ġof Ġbiomedical Ġterms Ġdeveloped Ġfor Ġindex ing Ġbiomedical Ġjournal Ġarticles . </s> We Ġobtain ĠSN OM ED âĢĵ CT Ġand ĠMe SH Ġvia Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem Ġ( U ML S ) ĠMet at hes aurus Ġ( version Ġ2016 AA ). ĠThe ĠMet at hes aurus Ġcontains Ġapproximately Ġ2 Ġmillion Ġbiomedical Ġand Ġclinical Ġconcepts Ġfrom Ġover Ġ150 Ġdifferent Ġtermin ologies Ġthat Ġhave Ġbeen Ġsemi âĢĵ aut om atically Ġintegrated Ġinto Ġa Ġsingle Ġsource . ĠConcepts Ġin Ġthe ĠMet at hes aurus Ġare Ġconnected Ġlargely Ġby Ġtwo Ġtypes Ġof Ġhierarchical Ġrelations : ĠIN LINE FORM 0 Ġ/ ĠIN LINE FORM 1 Ġ( PAR / CH D ) Ġand ĠIN LINE FORM 2 Ġ/ ĠIN LINE FORM 3 Ġ( RB / RN ). </s> Similar ity ĠMeasures </s> Me asures Ġof Ġsemantic Ġsimilarity Ġcan Ġbe Ġclassified Ġinto Ġthree Ġbroad Ġcategories Ġ: Ġpath âĢĵ based , Ġfeature âĢĵ based Ġand Ġinformation Ġcontent Ġ( IC ). ĠPath âĢĵ based Ġsimilarity Ġmeasures Ġuse Ġthe Ġstructure Ġof Ġa Ġtax onomy Ġto Ġmeasure Ġsimilarity ĠâĢĵ Ġconcepts Ġpositioned Ġclose Ġto Ġeach Ġother Ġare Ġmore Ġsimilar Ġthan Ġthose Ġfurther Ġapart . ĠFeature âĢĵ based Ġmethods Ġrely Ġon Ġset Ġtheoret ic Ġmeasures Ġof Ġoverlap Ġbetween Ġfeatures Ġ( union Ġand Ġintersection ). ĠThe Ġinformation Ġcontent Ġmeasures Ġquantify Ġthe Ġamount Ġof Ġinformation Ġthat Ġa Ġconcept Ġprovides ĠâĢĵ Ġmore Ġspecific Ġconcepts Ġhave Ġa Ġhigher Ġamount Ġof Ġinformation Ġcontent . </s> R ada MB B 89 Ġintroduce Ġthe ĠConcept ual ĠDistance Ġmeasure . ĠThis Ġmeasure Ġis Ġsimply Ġthe Ġlength Ġof Ġthe Ġshortest Ġpath Ġbetween Ġtwo Ġconcepts Ġ( ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġ) Ġin Ġthe ĠMe SH Ġhierarchy . ĠPath s Ġare Ġbased Ġon Ġbroader Ġthan Ġ( RB ) Ġand Ġnarrower Ġthan Ġ( RN ) Ġrelations . ĠCav ied es C 04 Ġextends Ġthis Ġmeasure Ġto Ġuse Ġparent Ġ( PAR ) Ġand Ġchild Ġ( CH D ) Ġrelations . ĠOur ĠIN LINE FORM 2 Ġmeasure Ġis Ġsimply Ġthe Ġreciprocal Ġof Ġthis Ġshortest Ġpath Ġvalue Ġ( Equ ation ĠEQ REF 3 Ġ), Ġso Ġthat Ġlarger Ġvalues Ġ( appro aching Ġ1 ) Ġindicate Ġa Ġhigh Ġdegree Ġof Ġsimilarity . ĠDIS PLAY FORM 0 </s> While Ġthe Ġsimplicity Ġof ĠIN LINE FORM 0 Ġis Ġappealing , Ġit Ġcan Ġbe Ġmisleading Ġwhen Ġconcepts Ġare Ġat Ġdifferent Ġlevels Ġof Ġspecificity . ĠTwo Ġvery Ġgeneral Ġconcepts Ġmay Ġhave Ġthe Ġsame Ġpath Ġlength Ġas Ġtwo Ġvery Ġspecific Ġconcepts . ĠWu P 94 Ġintroduce Ġa Ġcorrection Ġto ĠIN LINE FORM 1 Ġthat Ġincorporates Ġthe Ġdepth Ġof Ġthe Ġconcepts , Ġand Ġthe Ġdepth Ġof Ġtheir ĠLe ast ĠCommon ĠSub s umer Ġ( LCS ). ĠThis Ġis Ġthe Ġmost Ġspecific Ġancestor Ġtwo Ġconcepts Ġshare . ĠIn Ġthis Ġmeasure , Ġsimilarity Ġis Ġtwice Ġthe Ġdepth Ġof Ġthe Ġtwo Ġconcept 's ĠLCS Ġdivided Ġby Ġthe Ġproduct Ġof Ġthe Ġdepths Ġof Ġthe Ġindividual Ġconcepts Ġ( Equ ation ĠEQ REF 4 Ġ). ĠNote Ġthat Ġif Ġthere Ġare Ġmultiple ĠLCS s Ġfor Ġa Ġpair Ġof Ġconcepts , Ġthe Ġdeepest Ġof Ġthem Ġis Ġused Ġin Ġthis Ġmeasure . ĠDIS PLAY FORM 0 </s> Z h ong Z LY 02 Ġtake Ġa Ġvery Ġsimilar Ġapproach Ġand Ġagain Ġscale Ġthe Ġdepth Ġof Ġthe ĠLCS Ġby Ġthe Ġsum Ġof Ġthe Ġdepths Ġof Ġthe Ġtwo Ġconcepts Ġ( Equ ation ĠEQ REF 5 Ġ), Ġwhere ĠIN LINE FORM 0 Ġ. ĠThe Ġvalue Ġof ĠIN LINE FORM 1 Ġwas Ġset Ġto Ġ2 Ġbased Ġon Ġtheir Ġrecommendations . ĠDIS PLAY FORM 0 </s> P ek ar S 02 Ġoffer Ġanother Ġvariation Ġon ĠIN LINE FORM 0 Ġ, Ġwhere Ġthe Ġshortest Ġpath Ġof Ġthe Ġtwo Ġconcepts Ġto Ġthe ĠLCS Ġis Ġused , Ġin Ġaddition Ġto Ġthe Ġshortest Ġbath Ġbetween Ġthe ĠLCS Ġand Ġthe Ġroot Ġof Ġthe Ġtax onomy Ġ( Equ ation ĠEQ REF 6 Ġ). ĠDIS PLAY FORM 0 </s> Feature âĢĵ based Ġmethods Ġrepresent Ġeach Ġconcept Ġas Ġa Ġset Ġof Ġfeatures Ġand Ġthen Ġmeasure Ġthe Ġoverlap Ġor Ġsharing Ġof Ġfeatures Ġto Ġmeasure Ġsimilarity . ĠIn Ġparticular , Ġeach Ġconcept Ġis Ġrepresented Ġas Ġthe Ġset Ġof Ġtheir Ġancestors , Ġand Ġsimilarity Ġis Ġa Ġratio Ġof Ġthe Ġintersection Ġand Ġunion Ġof Ġthese Ġfeatures . </s> M aed che S 01 Ġquantify Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġratio Ġof Ġthe Ġintersection Ġover Ġtheir Ġunion Ġas Ġshown Ġin ĠEqu ation ĠEQ REF 8 Ġ. ĠDIS PLAY FORM 0 </s> Bat et S V 11 Ġextend Ġthis Ġby Ġexcluding Ġany Ġshared Ġfeatures Ġ( in Ġthe Ġnumer ator ) Ġas Ġshown Ġin ĠEqu ation ĠEQ REF 9 Ġ. ĠDIS PLAY FORM 0 </s> Information Ġcontent Ġis Ġformally Ġdefined Ġas Ġthe Ġnegative Ġlog Ġof Ġthe Ġprobability Ġof Ġa Ġconcept . ĠThe Ġeffect Ġof Ġthis Ġis Ġto Ġassign Ġrare Ġ( low Ġprobability ) Ġconcepts Ġa Ġhigh Ġmeasure Ġof Ġinformation Ġcontent , Ġsince Ġthe Ġunderlying Ġassumption Ġis Ġthat Ġmore Ġspecific Ġconcepts Ġare Ġless Ġfrequently Ġused Ġthan Ġmore Ġcommon Ġones . </s> Res nik 95 Ġmodified Ġthis Ġnotion Ġof Ġinformation Ġcontent Ġin Ġorder Ġto Ġuse Ġit Ġas Ġa Ġsimilarity Ġmeasure . ĠHe Ġdefines Ġthe Ġsimilarity Ġof Ġtwo Ġconcepts Ġto Ġbe Ġthe Ġinformation Ġcontent Ġof Ġtheir ĠLCS Ġ( Equ ation ĠEQ REF 11 Ġ). ĠDIS PLAY FORM 0 </s> J iang C 97 , ĠLin 98 , Ġand ĠPir ro E 10 Ġextend ĠIN LINE FORM 0 Ġby Ġincorporating Ġthe Ġinformation Ġcontent Ġof Ġthe Ġindividual Ġconcepts Ġin Ġvarious Ġdifferent Ġways . ĠLin 98 Ġdefines Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġratio Ġof Ġinformation Ġcontent Ġof Ġthe ĠLCS Ġwith Ġthe Ġsum Ġof Ġthe Ġindividual Ġconcept 's Ġinformation Ġcontent Ġ( Equ ation ĠEQ REF 12 Ġ). ĠNote Ġthat ĠIN LINE FORM 1 Ġhas Ġthe Ġsame Ġform Ġas ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ, Ġand Ġis Ġin Ġeffect Ġusing Ġinformation Ġcontent Ġas Ġa Ġmeasure Ġof Ġspecificity Ġ( rather Ġthan Ġdepth ). ĠIf Ġthere Ġis Ġmore Ġthan Ġone Ġpossible ĠLCS , Ġthe ĠLCS Ġwith Ġthe Ġgreatest ĠIC Ġis Ġchosen . ĠDIS PLAY FORM 0 </s> J iang C 97 Ġdefine Ġthe Ġdistance Ġbetween Ġtwo Ġconcepts Ġto Ġbe Ġthe Ġsum Ġof Ġthe Ġinformation Ġcontent Ġof Ġthe Ġtwo Ġconcepts Ġminus Ġtwice Ġthe Ġinformation Ġcontent Ġof Ġthe Ġconcepts ' ĠLCS . ĠWe Ġmodify Ġthis Ġfrom Ġa Ġdistance Ġto Ġa Ġsimilarity Ġmeasure Ġby Ġtaking Ġthe Ġreciprocal Ġof Ġthe Ġdistance Ġ( Equ ation ĠEQ REF 13 Ġ). ĠNote Ġthat Ġthe Ġdenomin ator Ġof ĠIN LINE FORM 0 Ġis Ġvery Ġsimilar Ġto Ġthe Ġnumer ator Ġof ĠIN LINE FORM 1 Ġ. ĠDIS PLAY FORM 0 </s> Pir ro E 10 Ġdefine Ġthe Ġsimilarity Ġbetween Ġtwo Ġconcepts Ġas Ġthe Ġinformation Ġcontent Ġof Ġthe Ġtwo Ġconcept 's ĠLCS Ġdivided Ġby Ġthe Ġsum Ġof Ġtheir Ġindividual Ġinformation Ġcontent Ġvalues Ġminus Ġthe Ġinformation Ġcontent Ġof Ġtheir ĠLCS Ġ( Equ ation ĠEQ REF 14 Ġ). ĠNote Ġthat ĠIN LINE FORM 0 Ġcan Ġbe Ġviewed Ġas Ġa Ġset âĢĵ the oret ic Ġversion Ġof ĠIN LINE FORM 1 Ġ. ĠDIS PLAY FORM 0 </s> Information ĠContent </s> The Ġinformation Ġcontent Ġof Ġa Ġconcept Ġmay Ġbe Ġderived Ġfrom Ġa Ġcorpus Ġ( cor p us âĢĵ based ) Ġor Ġdirectly Ġfrom Ġa Ġtax onomy Ġ( int r ins ic âĢĵ based ). ĠIn Ġthis Ġwork Ġwe Ġfocus Ġon Ġcorpus âĢĵ based Ġtechniques . </s> For Ġcorpus âĢĵ based Ġinformation Ġcontent , Ġwe Ġestimate Ġthe Ġprobability Ġof Ġa Ġconcept ĠIN LINE FORM 0 Ġby Ġtaking Ġthe Ġsum Ġof Ġthe Ġprobability Ġof Ġthe Ġconcept ĠIN LINE FORM 1 Ġand Ġthe Ġprobability Ġits Ġdescendants ĠIN LINE FORM 2 Ġ( Equ ation ĠEQ REF 16 Ġ). ĠDIS PLAY FORM 0 </s> The Ġinitial Ġprobabilities Ġof Ġa Ġconcept Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġits Ġdescendants Ġ( ĠIN LINE FORM 1 Ġ) Ġare Ġobtained Ġby Ġdividing Ġthe Ġnumber Ġof Ġtimes Ġeach Ġconcept Ġand Ġdescendant Ġoccurs Ġin Ġthe Ġcorpus , Ġand Ġdividing Ġthat Ġby Ġthe Ġtotal Ġnumbers Ġof Ġconcepts Ġ( ĠIN LINE FORM 2 Ġ). </s> Ide ally Ġthe Ġcorpus Ġfrom Ġwhich Ġwe Ġare Ġestimating Ġthe Ġprobabilities Ġof Ġconcepts Ġwill Ġbe Ġsense âĢĵ tag ged . ĠHowever , Ġsense âĢĵ tag ging Ġis Ġa Ġchallenging Ġproblem Ġin Ġits Ġown Ġright , Ġand Ġit Ġis Ġnot Ġalways Ġpossible Ġto Ġcarry Ġout Ġreliably Ġon Ġlarger Ġamounts Ġof Ġtext . ĠIn Ġfact Ġin Ġthis Ġpaper Ġwe Ġdid Ġnot Ġuse Ġany Ġsense âĢĵ tag ging Ġof Ġthe Ġcorpus Ġwe Ġderived Ġinformation Ġcontent Ġfrom . </s> Instead , Ġwe Ġestimated Ġthe Ġprobability Ġof Ġa Ġconcept Ġby Ġusing Ġthe ĠU ML Son Med line Ġdataset . ĠThis Ġwas Ġcreated Ġby Ġthe ĠNational ĠLibrary Ġof ĠMedicine Ġand Ġconsists Ġof Ġconcepts Ġfrom Ġthe Ġ2009 AB ĠU ML S Ġand Ġthe Ġcounts Ġof Ġthe Ġnumber Ġof Ġtimes Ġthey Ġoccurred Ġin Ġa Ġsnapshot Ġof ĠMedline Ġtaken Ġon Ġ12 ĠJanuary , Ġ2009 . ĠThese Ġcounts Ġwere Ġobtained Ġby Ġusing Ġthe ĠEss ie ĠSearch ĠEngine ĠB IB REF 14 Ġwhich Ġquer ied ĠMedline Ġwith Ġnormalized Ġstrings Ġfrom Ġthe Ġ2009 AB ĠM RC ON SO Ġtable Ġin Ġthe ĠU ML S . ĠThe Ġfrequency Ġof Ġa ĠC UI Ġwas Ġobtained Ġby Ġaggreg ating Ġthe Ġfrequency Ġcounts Ġof Ġthe Ġterms Ġassociated Ġwith Ġthe ĠC UI Ġto Ġprovide Ġa Ġrough Ġestimate Ġof Ġits Ġfrequency . ĠThe Ġinformation Ġcontent Ġmeasures Ġthen Ġuse Ġthis Ġinformation Ġto Ġcalculate Ġthe Ġprobability Ġof Ġa Ġconcept . </s> Another Ġalternative Ġis Ġthe Ġuse Ġof ĠInt r ins ic ĠInformation ĠContent . ĠIt Ġassess Ġthe Ġinform at iveness Ġof Ġconcept Ġbased Ġon Ġits Ġplacement Ġwithin Ġa Ġtax onomy Ġby Ġconsidering Ġthe Ġnumber Ġof Ġincoming Ġ( anc est ors ) Ġrelative Ġto Ġoutgoing Ġ( desc endant ) Ġlinks ĠB IB REF 15 Ġ( Equ ation ĠEQ REF 17 Ġ). ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġthe Ġnumber Ġof Ġdescendants Ġof Ġconcept ĠIN LINE FORM 1 Ġthat Ġare Ġleaf Ġnodes , ĠIN LINE FORM 2 Ġare Ġthe Ġnumber Ġof Ġconcept ĠIN LINE FORM 3 Ġ' s Ġancestors Ġand ĠIN LINE FORM 4 Ġare Ġthe Ġtotal Ġnumber Ġof Ġleaf Ġnodes Ġin Ġthe Ġtax onomy . </s> Related ness ĠMeasures </s> Les k 86 Ġobserved Ġthat Ġconcepts Ġthat Ġare Ġrelated Ġshould Ġshare Ġmore Ġwords Ġin Ġtheir Ġrespective Ġdefinitions Ġthan Ġconcepts Ġthat Ġare Ġless Ġconnected . ĠHe Ġwas Ġable Ġto Ġperform Ġword Ġsense Ġdis amb ig uation Ġby Ġidentifying Ġthe Ġsenses Ġof Ġwords Ġin Ġa Ġsentence Ġwith Ġthe Ġlargest Ġnumber Ġof Ġoverl aps Ġbetween Ġtheir Ġdefinitions . ĠAn Ġoverlap Ġis Ġthe Ġlongest Ġsequence Ġof Ġone Ġor Ġmore Ġconsecutive Ġwords Ġthat Ġoccur Ġin Ġboth Ġdefinitions . ĠBan er jee P 03 Ġextended Ġthis Ġidea Ġto ĠWord Net , Ġbut Ġobserved Ġthat ĠWord Net Ġgloss es Ġare Ġoften Ġvery Ġshort , Ġand Ġdid Ġnot Ġcontain Ġenough Ġinformation Ġto Ġdistinguish Ġbetween Ġmultiple Ġconcepts . ĠTherefore , Ġthey Ġcreated Ġa Ġsuper âĢĵ gl oss Ġfor Ġeach Ġconcept Ġby Ġadding Ġthe Ġgloss es Ġof Ġrelated Ġconcepts Ġto Ġthe Ġgloss Ġof Ġthe Ġconcept Ġitself Ġ( and Ġthen Ġfinding Ġoverl aps ). </s> Pat ward han P 06 Ġadapted Ġthis Ġmeasure Ġto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors . ĠIn Ġthis Ġapproach , Ġa Ġvector Ġis Ġcreated Ġfor Ġeach Ġword Ġin Ġa Ġconcept 's Ġdefinition Ġthat Ġshows Ġwhich Ġwords Ġco âĢĵ occ ur Ġwith Ġit Ġin Ġa Ġcorpus . ĠThese Ġword Ġvectors Ġare Ġaveraged Ġto Ġcreate Ġa Ġsingle Ġco - occ urrence Ġvector Ġfor Ġthe Ġconcept . ĠThe Ġsimilarity Ġbetween Ġthe Ġconcepts Ġis Ġcalculated Ġby Ġtaking Ġthe Ġcos ine Ġbetween Ġthe Ġconcepts Ġsecond âĢĵ order Ġvectors . ĠLiu M PM P 12 Ġmodified Ġand Ġextended Ġthis Ġmeasure Ġto Ġbe Ġused Ġto Ġquantify Ġthe Ġrelated ness Ġbetween Ġbiomedical Ġand Ġclinical Ġterms Ġin Ġthe ĠU ML S . ĠThe Ġwork Ġin Ġthis Ġpaper Ġcan Ġbe Ġseen Ġas Ġa Ġfurther Ġextension Ġof ĠPat ward han P 06 Ġand ĠLiu M PM P 12 . </s> Method </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour Ġsecond âĢĵ order Ġsimilarity Ġvector Ġmeasure . ĠThis Ġincorporates Ġboth Ġcontextual Ġinformation Ġusing Ġthe Ġterm Ġpair 's Ġdefinition Ġand Ġtheir Ġpair wise Ġsemantic Ġsimilarity Ġscores Ġderived Ġfrom Ġa Ġtax onomy . ĠThere Ġare Ġtwo Ġstages Ġto Ġour Ġapproach . ĠFirst , Ġa Ġco âĢĵ occ urrence Ġmatrix Ġmust Ġbe Ġconstructed . ĠSecond , Ġthis Ġmatrix Ġis Ġused Ġto Ġconstruct Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġfor Ġeach Ġconcept Ġin Ġa Ġpair Ġof Ġconcepts Ġto Ġbe Ġmeasured Ġfor Ġrelated ness . </s> Co âĢĵ occ urrence ĠMatrix ĠConstruction </s> We Ġbuild Ġan ĠIN LINE FORM 0 Ġsimilarity Ġmatrix Ġusing Ġan Ġexternal Ġcorpus Ġwhere Ġthe Ġrows Ġand Ġcolumns Ġrepresent Ġwords Ġwithin Ġthe Ġcorpus Ġand Ġthe Ġelement Ġcontains Ġthe Ġsimilarity Ġscore Ġbetween Ġthe Ġrow Ġword Ġand Ġcolumn Ġword Ġusing Ġthe Ġsimilarity Ġmeasures Ġdiscussed Ġabove . ĠIf Ġa Ġword Ġmaps Ġto Ġmore Ġthan Ġone Ġpossible Ġsense , Ġwe Ġuse Ġthe Ġsense Ġthat Ġreturns Ġthe Ġhighest Ġsimilarity Ġscore . </s> For Ġthis Ġpaper Ġour Ġexternal Ġcorpus Ġwas Ġthe ĠNL M Ġ2015 ĠMedline Ġbaseline . ĠMedline Ġis Ġa Ġb ibli ographic Ġdatabase Ġcontaining Ġover Ġ23 Ġmillion Ġcitations Ġto Ġjournal Ġarticles Ġin Ġthe Ġbiomedical Ġdomain Ġand Ġis Ġmaintained Ġby ĠNational ĠLibrary Ġof ĠMedicine . ĠThe Ġ2015 ĠMedline ĠBas eline Ġencompasses Ġapproximately Ġ5 , 600 Ġjournals Ġstarting Ġfrom Ġ1948 Ġand Ġcontains Ġ23 , 343 , 329 Ġcitations , Ġof Ġwhich Ġ2 , 579 , 239 Ġcontain Ġabstract s . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠMedline Ġtitles Ġand Ġabstract s Ġfrom Ġ1975 Ġto Ġpresent Ġday . ĠPrior Ġto Ġ1975 , Ġonly Ġ2 % Ġof Ġthe Ġcitations Ġcontained Ġan Ġabstract . ĠWe Ġthen Ġcalculate Ġthe Ġsimilarity Ġfor Ġeach Ġbig ram Ġin Ġthis Ġdataset Ġand Ġinclude Ġthose Ġthat Ġhave Ġa Ġsimilarity Ġscore Ġgreater Ġthan Ġa Ġspecified Ġthreshold Ġon Ġthese Ġexperiments . </s> Measure ĠTerm ĠP airs Ġfor ĠRelated ness </s> We Ġobtain Ġdefinitions Ġfor Ġeach Ġof Ġthe Ġtwo Ġterms Ġwe Ġwish Ġto Ġmeasure . ĠDue Ġto Ġthe Ġsp arsity Ġand Ġinconsistencies Ġof Ġthe Ġdefinitions Ġin Ġthe ĠU ML S , Ġwe Ġnot Ġonly Ġuse Ġthe Ġdefinition Ġof Ġthe Ġterm Ġ( C UI ) Ġbut Ġalso Ġinclude Ġthe Ġdefinition Ġof Ġits Ġrelated Ġconcepts . ĠThis Ġfollows Ġthe Ġmethod Ġproposed Ġby ĠPat ward han P 06 Ġfor Ġgeneral ĠEnglish Ġand ĠWord Net , Ġand Ġwhich Ġwas Ġadapted Ġfor Ġthe ĠU ML S Ġand Ġthe Ġmedical Ġdomain Ġby ĠLiu M PM P 12 . ĠIn Ġparticular Ġwe Ġadd Ġthe Ġdefinitions Ġof Ġany Ġconcepts Ġconnected Ġvia Ġa Ġparent Ġ( PAR ), Ġchild Ġ( CH D ), ĠRB Ġ( bro ader Ġthan ), ĠRN Ġ( n arrow er Ġthan ) Ġor ĠTER M Ġ( terms Ġassociated Ġwith ĠC UI ) Ġrelation . ĠAll Ġof Ġthe Ġdefinitions Ġfor Ġa Ġterm Ġare Ġcombined Ġinto Ġa Ġsingle Ġsuper âĢĵ gl oss . ĠAt Ġthe Ġend Ġof Ġthis Ġprocess Ġwe Ġshould Ġhave Ġtwo Ġsuper âĢĵ gl oss es , Ġone Ġfor Ġeach Ġterm Ġto Ġbe Ġmeasured Ġfor Ġrelated ness . </s> Next , Ġwe Ġprocess Ġeach Ġsuper âĢĵ gl oss Ġas Ġfollows : </s> We Ġextract Ġa Ġfirst âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġfor Ġeach Ġterm Ġin Ġthe Ġsuper âĢĵ gl oss Ġfrom Ġthe Ġco âĢĵ occ urrence Ġmatrix Ġcreated Ġpreviously . </s> We Ġtake Ġthe Ġaverage Ġof Ġthe Ġfirst Ġorder Ġco âĢĵ occ urrence Ġvectors Ġassociated Ġwith Ġthe Ġterms Ġin Ġa Ġsuper âĢĵ gl oss Ġand Ġuse Ġthat Ġto Ġrepresent Ġthe Ġmeaning Ġof Ġthe Ġterm . ĠThis Ġis Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector . </s> After Ġa Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvector Ġhas Ġbeen Ġconstructed Ġfor Ġeach Ġterm , Ġthen Ġwe Ġcalculate Ġthe Ġcos ine Ġbetween Ġthese Ġtwo Ġvectors Ġto Ġmeasure Ġthe Ġrelated ness Ġof Ġthe Ġterms . </s> Data </s> We Ġuse Ġtwo Ġreference Ġstandards Ġto Ġevaluate Ġthe Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġmeasures Ġ. ĠUM NS RS Ġwas Ġannot ated Ġfor Ġboth Ġsimilarity Ġand Ġrelated ness Ġby Ġmedical Ġresidents . ĠMini May oS RS Ġwas Ġannot ated Ġfor Ġrelated ness Ġby Ġmedical Ġdoctors Ġ( MD ) Ġand Ġmedical Ġcod ers Ġ( c oder ). ĠIn Ġthis Ġsection , Ġwe Ġdescribe Ġthese Ġdata Ġsets Ġand Ġdescribe Ġa Ġfew Ġof Ġtheir Ġdifferences . </s> Mini May oS RS : ĠThe ĠMayo S RS , Ġdeveloped Ġby ĠPak hom ov PM M RC 10 , Ġconsists Ġof Ġ101 Ġclinical Ġterm Ġpairs Ġwhose Ġrelated ness Ġwas Ġdetermined Ġby Ġnine Ġmedical Ġcod ers Ġand Ġthree Ġphysicians Ġfrom Ġthe ĠMayo ĠClinic . ĠThe Ġrelated ness Ġof Ġeach Ġterm Ġpair Ġwas Ġassessed Ġbased Ġon Ġa Ġfour Ġpoint Ġscale : Ġ( 4 . 0 ) Ġpractically Ġsynonymous , Ġ( 3 . 0 ) Ġrelated , Ġ( 2 . 0 ) Ġmarginally Ġrelated Ġand Ġ( 1 . 0 ) Ġunrelated . ĠMini May oS RS Ġis Ġa Ġsubset Ġof Ġthe ĠMayo S RS Ġand Ġconsists Ġof Ġ30 Ġterm Ġpairs Ġon Ġwhich Ġa Ġhigher Ġinter âĢĵ annot ator Ġagreement Ġwas Ġachieved . ĠThe Ġaverage Ġcorrelation Ġbetween Ġphysicians Ġis Ġ0 . 68 . ĠThe Ġaverage Ġcorrelation Ġbetween Ġmedical Ġcod ers Ġis Ġ0 . 78 . ĠWe Ġevaluate Ġour Ġmethod Ġon Ġthe Ġmean Ġof Ġthe Ġphysician Ġscores , Ġand Ġthe Ġmean Ġof Ġthe Ġcod ers Ġscores Ġin Ġthis Ġsubset Ġin Ġthe Ġsame Ġmanner Ġas Ġreported Ġby ĠPed ersen P PC 07 . </s> UM NS RS : ĠThe ĠUniversity Ġof ĠMinnesota ĠSem antic ĠRelated ness ĠSet Ġ( UM NS RS ) Ġwas Ġdeveloped Ġby ĠPak hom ov MAL PM 10 , Ġand Ġconsists Ġof Ġ7 25 Ġclinical Ġterm Ġpairs Ġwhose Ġsemantic Ġsimilarity Ġand Ġrelated ness Ġwas Ġdetermined Ġindependently Ġby Ġfour Ġmedical Ġresidents Ġfrom Ġthe ĠUniversity Ġof ĠMinnesota ĠMedical ĠSchool . ĠThe Ġsimilarity Ġand Ġrelated ness Ġof Ġeach Ġterm Ġpair Ġwas Ġannot ated Ġbased Ġon Ġa Ġcontinuous Ġscale Ġby Ġhaving Ġthe Ġresident Ġtouch Ġa Ġbar Ġon Ġa Ġtouch Ġsensitive Ġcomputer Ġscreen Ġto Ġindicate Ġthe Ġdegree Ġof Ġsimilarity Ġor Ġrelated ness . ĠThe ĠInt rac lass ĠCor relation ĠCo efficient Ġ( IC C ) Ġfor Ġthe Ġreference Ġstandard Ġtagged Ġfor Ġsimilarity Ġwas Ġ0 . 47 , Ġand Ġ0 . 50 Ġfor Ġrelated ness . ĠTherefore , Ġas Ġsuggested Ġby ĠPak hom ov Ġand Ġcolleagues , we Ġuse Ġa Ġsubset Ġof Ġthe Ġratings Ġconsisting Ġof Ġ401 Ġpairs Ġfor Ġthe Ġsimilarity Ġset Ġand Ġ430 Ġpairs Ġfor Ġthe Ġrelated ness Ġset Ġwhich Ġeach Ġhave Ġan ĠICC Ġof Ġ0 . 73 . </s> Exper imental ĠFramework </s> We Ġconducted Ġour Ġexperiments Ġusing Ġthe Ġfreely Ġavailable Ġopen Ġsource Ġsoftware Ġpackage ĠU ML S :: Similar ity ĠB IB REF 16 Ġversion Ġ1 . 47 . ĠThis Ġpackage Ġtakes Ġas Ġinput Ġtwo Ġterms Ġ( or ĠU ML S Ġconcepts ) Ġand Ġreturns Ġtheir Ġsimilarity Ġor Ġrelated ness Ġusing Ġthe Ġmeasures Ġdiscussed Ġin ĠSection ĠSEC REF 2 Ġ. </s> Cor relation Ġbetween Ġthe Ġsimilarity Ġmeasures Ġand Ġhuman Ġjudgments Ġwere Ġestimated Ġusing ĠSpear man 's ĠRank ĠCor relation Ġ( ĠIN LINE FORM 0 Ġ). ĠSpear man 's Ġmeasures Ġthe Ġstatistical Ġdependence Ġbetween Ġtwo Ġvariables Ġto Ġassess Ġhow Ġwell Ġthe Ġrelationship Ġbetween Ġthe Ġrankings Ġof Ġthe Ġvariables Ġcan Ġbe Ġdescribed Ġusing Ġa Ġmon ot onic Ġfunction . ĠWe Ġused ĠFisher 's Ġr - to - z Ġtransformation ĠB IB REF 17 Ġto Ġcalculate Ġthe Ġsignificance Ġbetween Ġthe Ġcorrelation Ġresults . </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 26 Ġshows Ġthe ĠSpear man 's ĠRank ĠCor relation Ġbetween Ġthe Ġhuman Ġscores Ġfrom Ġthe Ġfour Ġreference Ġstandards Ġand Ġthe Ġscores Ġfrom Ġthe Ġvarious Ġmeasures Ġof Ġsimilarity Ġintroduced Ġin ĠSection ĠSEC REF 2 Ġ. ĠEach Ġclass Ġof Ġmeasure Ġis Ġfollowed Ġby Ġthe Ġscores Ġobtained Ġwhen Ġintegrating Ġour Ġsecond Ġorder Ġvector Ġapproach Ġwith Ġthese Ġmeasures Ġof Ġsemantic Ġsimilarity . </s> Results ĠComparison </s> The Ġresults Ġfor ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġ( ĠIN LINE FORM 0 Ġ) Ġand ĠMini May oS RS Ġtagged Ġby Ġcod ers Ġshow Ġthat Ġall Ġof Ġthe Ġsecond - order Ġsimilarity Ġvector Ġmeasures Ġ( ĠIN LINE FORM 1 Ġ) Ġexcept Ġfor ĠIN LINE FORM 2 Ġ- ĠIN LINE FORM 3 Ġobtain Ġa Ġhigher Ġcorrelation Ġthan Ġthe Ġoriginal Ġmeasures . ĠWe Ġfound Ġthat ĠIN LINE FORM 4 Ġ- ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ- ĠIN LINE FORM 7 Ġobtain Ġthe Ġhighest Ġcorrelations Ġof Ġall Ġthese Ġresults Ġwith Ġhuman Ġjudgments . </s> For Ġthe ĠUM NS RS Ġdataset Ġtagged Ġfor Ġrelated ness Ġand ĠMini May oS RS Ġtagged Ġby Ġphysicians Ġ( MD ), Ġthe Ġoriginal ĠIN LINE FORM 0 Ġmeasure Ġob tains Ġa Ġhigher Ġcorrelation Ġthan Ġour Ġmeasure Ġ( ĠIN LINE FORM 1 Ġ) Ġalthough Ġthe Ġdifference Ġis Ġnot Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 2 Ġ). </s> In Ġorder Ġto Ġanalyze Ġand Ġbetter Ġunderstand Ġthese Ġresults , Ġwe Ġfiltered Ġthe Ġbig ram Ġpairs Ġused Ġto Ġcreate Ġthe Ġinitial Ġsimilarity Ġmatrix Ġbased Ġon Ġthe Ġstrength Ġof Ġtheir Ġsimilarity Ġusing Ġthe ĠIN LINE FORM 0 Ġand Ġthe ĠIN LINE FORM 1 Ġmeasures . ĠNote Ġthat Ġthe ĠIN LINE FORM 2 Ġmeasure Ġholds Ġto Ġa Ġ0 Ġto Ġ1 Ġscale , Ġwhile ĠIN LINE FORM 3 Ġranges Ġfrom Ġ0 Ġto Ġan Ġunspecified Ġupper Ġbound Ġthat Ġis Ġdependent Ġon Ġthe Ġsize Ġof Ġthe Ġcorpus Ġfrom Ġwhich Ġinformation Ġcontent Ġis Ġestimated . ĠAs Ġsuch Ġwe Ġuse Ġa Ġdifferent Ġrange Ġof Ġthreshold Ġvalues Ġfor Ġeach Ġmeasure . ĠWe Ġdiscuss Ġthe Ġresults Ġof Ġthis Ġfiltering Ġbelow . </s> Th reshold ing ĠExper iments </s> Table ĠT AB REF 29 Ġshows Ġthe Ġresults Ġof Ġapplying Ġthe Ġthreshold Ġparameter Ġon Ġeach Ġof Ġthe Ġreference Ġstandards Ġusing Ġthe ĠIN LINE FORM 0 Ġmeasure . ĠFor Ġexample , Ġa Ġthreshold Ġof Ġ0 Ġindicates Ġthat Ġall Ġof Ġthe Ġbig rams Ġwere Ġincluded Ġin Ġthe Ġsimilarity Ġmatrix ; Ġand Ġa Ġthreshold Ġof Ġ1 Ġindicates Ġthat Ġonly Ġthe Ġbig ram Ġpairs Ġwith Ġa Ġsimilarity Ġscore Ġgreater Ġthan Ġone Ġwere Ġincluded . </s> These Ġresults Ġshow Ġthat Ġusing Ġa Ġthreshold Ġcutoff Ġof Ġ2 Ġob tains Ġthe Ġhighest Ġcorrelation Ġfor Ġthe ĠUM NS RS Ġdataset , Ġand Ġthat Ġa Ġthreshold Ġcutoff Ġof Ġ4 Ġob tains Ġthe Ġhighest Ġcorrelation Ġfor Ġthe ĠMini May oS RS Ġdataset . ĠAll Ġof Ġthe Ġresults Ġshow Ġan Ġincrease Ġin Ġcorrelation Ġwith Ġhuman Ġjudgments Ġwhen Ġincorporating Ġa Ġthreshold Ġcutoff Ġover Ġall Ġof Ġthe Ġoriginal Ġmeasures . ĠThe Ġincrease Ġin Ġthe Ġcorrelation Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġis Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ), Ġhowever Ġthis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġrelated ness Ġnor Ġfor Ġthe ĠMini May oS RS Ġdata . </s> Similarly , ĠTable ĠT AB REF 30 Ġshows Ġthe Ġresults Ġof Ġapplying Ġthe Ġthreshold Ġparameter Ġ( T ) Ġon Ġeach Ġof Ġthe Ġreference Ġstandards Ġusing Ġthe ĠIN LINE FORM 0 Ġmeasure . ĠAlthough , Ġunlike ĠIN LINE FORM 1 Ġwhose Ġscores Ġare Ġgreater Ġthan Ġor Ġequal Ġto Ġ0 Ġwithout Ġan Ġupper Ġlimit , Ġthe ĠIN LINE FORM 2 Ġmeasure Ġreturns Ġscores Ġbetween Ġ0 Ġand Ġ1 Ġ( in clusive ). ĠTherefore , Ġhere Ġa Ġthreshold Ġof Ġ0 Ġindicates Ġthat Ġall Ġof Ġthe Ġbig rams Ġwere Ġincluded Ġin Ġthe Ġsimilarity Ġmatrix ; Ġand Ġa Ġthreshold Ġof ĠIN LINE FORM 3 Ġindicates Ġthat Ġonly Ġthe Ġbig ram Ġpairs Ġwith Ġa Ġsimilarity Ġscore Ġgreater Ġthan ĠIN LINE FORM 4 Ġwere Ġincluded . ĠThe Ġresults Ġshow Ġan Ġincrease Ġin Ġaccuracy Ġfor Ġall Ġof Ġthe Ġdatasets Ġexcept Ġfor Ġthe ĠMini May oS RS Ġtagged Ġfor Ġphysicians . ĠThe Ġincrease Ġin Ġthe Ġresults Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġsimilarity Ġand Ġthe ĠMayo S RS Ġis Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 5 Ġ). ĠThis Ġis Ġnot Ġthe Ġcase Ġfor Ġthe ĠUM NS RS Ġtagged Ġfor Ġrelated ness Ġnor Ġthe ĠMini May oS RS . </s> Overall , Ġthese Ġresults Ġindicate Ġthat Ġincluding Ġonly Ġthose Ġbig rams Ġthat Ġhave Ġa Ġsufficiently Ġhigh Ġsimilarity Ġscore Ġincreases Ġthe Ġcorrelation Ġresults Ġwith Ġhuman Ġjudgments , Ġbut Ġwhat Ġquant ifies Ġas Ġsufficiently Ġhigh Ġvaries Ġdepending Ġon Ġthe Ġdataset Ġand Ġmeasure . </s> Compar ison Ġwith ĠPrevious ĠWork </s> Recently , Ġword Ġembed d ings ĠB IB REF 9 Ġhave Ġbecome Ġa Ġpopular Ġmethod Ġfor Ġmeasuring Ġsemantic Ġrelated ness Ġin Ġthe Ġbiomedical Ġdomain . ĠThis Ġis Ġa Ġneural Ġnetwork Ġbased Ġapproach Ġthat Ġlearns Ġa Ġrepresentation Ġof Ġa Ġword Ġby Ġword Ġco âĢĵ occ urrence Ġmatrix . ĠThe Ġbasic Ġidea Ġis Ġthat Ġthe Ġneural Ġnetwork Ġlearns Ġa Ġseries Ġof Ġweights Ġ( the Ġhidden Ġlayer Ġwithin Ġthe Ġneural Ġnetwork ) Ġthat Ġeither Ġmaxim izes Ġthe Ġprobability Ġof Ġa Ġword Ġgiven Ġits Ġcontext , Ġreferred Ġto Ġas Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġapproach , Ġor Ġthat Ġmaxim izes Ġthe Ġprobability Ġof Ġthe Ġcontext Ġgiven Ġa Ġword , Ġreferred Ġto Ġas Ġthe ĠSkip âĢĵ gram Ġapproach . ĠThese Ġapproaches Ġhave Ġbeen Ġused Ġin Ġnumerous Ġrecent Ġpapers . </s> mun ee b 2015 eval ut ating Ġtrained Ġboth Ġthe ĠSkip âĢĵ gram Ġand ĠCB OW Ġmodels Ġover Ġthe ĠPubMed ĠCentral ĠOpen ĠAccess Ġ( PM C ) Ġcorpus Ġof Ġapproximately Ġ1 . 25 Ġmillion Ġarticles . ĠThey Ġevaluated Ġthe Ġmodels Ġon Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġdata , Ġremoving Ġword Ġpairs Ġthat Ġdid Ġnot Ġoccur Ġin Ġtheir Ġtraining Ġcorpus Ġmore Ġthan Ġten Ġtimes . Ġch iu 2016 how Ġevaluated Ġboth Ġthe Ġthe ĠSkip âĢĵ gram Ġand ĠCB OW Ġmodels Ġover Ġthe ĠPM C Ġcorpus Ġand ĠPubMed . ĠThey Ġalso Ġevaluated Ġthe Ġmodels Ġon Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġignoring Ġthose Ġwords Ġthat Ġdid Ġnot Ġappear Ġin Ġtheir Ġtraining Ġcorpus . ĠPak hom ov 2016 cor p us Ġtrained ĠCB OW Ġmodel Ġover Ġthree Ġdifferent Ġtypes Ġof Ġcorpor a : Ġclinical Ġ( clinical Ġnotes Ġfrom Ġthe ĠFair view ĠHealth ĠSystem ), Ġbiomedical Ġ( PM C Ġcorpus ), Ġand Ġgeneral ĠEnglish Ġ( Wikipedia ). ĠThey Ġevaluated Ġtheir Ġmethod Ġusing Ġa Ġsubset Ġof Ġthe ĠUM NS RS Ġrestricting Ġto Ġsingle Ġword Ġterm Ġpairs Ġand Ġremoving Ġthose Ġnot Ġfound Ġwithin Ġtheir Ġtraining Ġcorpus . Ġsa jad 2015 domain Ġtrained Ġthe ĠSkip âĢĵ gram Ġmodel Ġover ĠCU Is Ġidentified Ġby ĠMeta Map Ġon Ġthe ĠO HS UM ED Ġcorpus , Ġa Ġcollection Ġof Ġ348 , 5 66 Ġbiomedical Ġresearch Ġarticles . ĠThey Ġevaluated Ġthe Ġmethod Ġon Ġthe Ġcomplete ĠUM NS RS , ĠMini May oS RS Ġand Ġthe ĠMayo S RS Ġdatasets ; Ġany Ġsubset Ġinformation Ġabout Ġthe Ġdataset Ġwas Ġnot Ġexplicitly Ġstated Ġtherefore Ġwe Ġbelieve Ġa Ġdirect Ġcomparison Ġmay Ġbe Ġpossible . </s> In Ġaddition , Ġa Ġprevious Ġwork Ġvery Ġclosely Ġrelated Ġto Ġours Ġis Ġa Ġretro fitting Ġvector Ġmethod Ġproposed Ġby ĠYu CB J W 16 Ġthat Ġincorporates Ġont ological Ġinformation Ġinto Ġa Ġvector Ġrepresentation Ġby Ġincluding Ġsem antically Ġrelated Ġwords . ĠIn Ġtheir Ġmeasure , Ġthey Ġfirst Ġmap Ġa Ġbiomedical Ġterm Ġto ĠMe SH Ġterms , Ġand Ġsecond Ġbuild Ġa Ġword Ġvector Ġbased Ġon Ġthe Ġdocuments Ġassigned Ġto Ġthe Ġrespective ĠMe SH Ġterm . ĠThey Ġthen Ġretro fit Ġthe Ġvector Ġby Ġincluding Ġsem antically Ġrelated Ġwords Ġfound Ġin Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem . ĠThey Ġevaluate Ġtheir Ġmethod Ġon Ġthe ĠMini May oS RS Ġdataset . </s> Table ĠT AB REF 31 Ġshows Ġa Ġcomparison Ġto Ġthe Ġtop Ġcorrelation Ġscores Ġreported Ġby Ġeach Ġof Ġthese Ġworks Ġon Ġthe Ġrespective Ġdatasets Ġ( or Ġsubs ets ) Ġthey Ġevaluated Ġtheir Ġmethods Ġon . ĠN Ġrefers Ġto Ġthe Ġnumber Ġof Ġterm Ġpairs Ġin Ġthe Ġdataset Ġthe Ġauthors Ġreport Ġthey Ġevaluated Ġtheir Ġmethod . ĠThe Ġtable Ġalso Ġincludes Ġour Ġtop Ġscoring Ġresults : Ġthe Ġintegrated Ġvector - res Ġand Ġvector - faith . ĠThe Ġresults Ġshow Ġthat Ġintegrating Ġsemantic Ġsimilarity Ġmeasures Ġinto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors Ġob tains Ġa Ġhigher Ġor Ġon âĢĵ par Ġcorrelation Ġwith Ġhuman Ġjudgments Ġas Ġthe Ġprevious Ġworks Ġreported Ġresults Ġwith Ġthe Ġexception Ġof Ġthe ĠUM NS RS Ġrel Ġdataset . ĠThe Ġresults Ġreported Ġby ĠPak hom ov 2016 cor p us Ġand Ġch iu 2016 how Ġobtain Ġa Ġhigher Ġcorrelation Ġalthough Ġthe Ġresults Ġcan Ġnot Ġbe Ġdirectly Ġcompared Ġbecause Ġboth Ġworks Ġused Ġdifferent Ġsubs ets Ġof Ġthe Ġterm Ġpairs Ġfrom Ġthe ĠUM NS RS Ġdataset . </s> Conclusion Ġand ĠFuture ĠWork </s> We Ġhave Ġpresented Ġa Ġmethod Ġfor Ġquant ifying Ġthe Ġsimilarity Ġand Ġrelated ness Ġbetween Ġtwo Ġterms Ġthat Ġintegrates Ġpair âĢĵ wise Ġsimilarity Ġscores Ġinto Ġsecond âĢĵ order Ġvectors . ĠThe Ġgoal Ġof Ġthis Ġapproach Ġis Ġtwo âĢĵ fold . ĠFirst , Ġwe Ġrestrict Ġthe Ġcontext Ġused Ġby Ġthe Ġvector Ġmeasure Ġto Ġwords Ġthat Ġexist Ġin Ġthe Ġbiomedical Ġdomain , Ġand Ġsecond , Ġwe Ġapply Ġlarger Ġweights Ġto Ġthose Ġword Ġpairs Ġthat Ġare Ġmore Ġsimilar Ġto Ġeach Ġother . ĠOur Ġhypothesis Ġwas Ġthat Ġthis Ġcombination Ġwould Ġreduce Ġthe Ġamount Ġof Ġnoise Ġin Ġthe Ġvectors Ġand Ġtherefore Ġincrease Ġtheir Ġcorrelation Ġwith Ġhuman Ġjudgments . ĠWe Ġevaluated Ġour Ġmethod Ġon Ġdatasets Ġthat Ġhave Ġbeen Ġmanually Ġannot ated Ġfor Ġrelated ness Ġand Ġsimilarity Ġand Ġfound Ġevidence Ġto Ġsupport Ġthis Ġhypothesis . ĠIn Ġparticular Ġwe Ġdiscovered Ġthat Ġguiding Ġthe Ġcreation Ġof Ġa Ġsecond âĢĵ order Ġcontext Ġvector Ġby Ġselecting Ġterm Ġpairs Ġfrom Ġbiomedical Ġtext Ġbased Ġon Ġtheir Ġsemantic Ġsimilarity Ġled Ġto Ġimproved Ġlevels Ġof Ġcorrelation Ġwith Ġhuman Ġjudgment . </s> We Ġalso Ġexplored Ġusing Ġa Ġthreshold Ġcutoff Ġto Ġinclude Ġonly Ġthose Ġterm Ġpairs Ġthat Ġobtained Ġa Ġsufficiently Ġlarge Ġlevel Ġof Ġsimilarity . ĠWe Ġfound Ġthat Ġeliminating Ġless Ġsimilar Ġpairs Ġimproved Ġthe Ġoverall Ġresults Ġ( to Ġa Ġpoint ). ĠIn Ġthe Ġfuture , Ġwe Ġplan Ġto Ġexplore Ġmetrics Ġto Ġautomatically Ġdetermine Ġthe Ġthreshold Ġcutoff Ġappropriate Ġfor Ġa Ġgiven Ġdataset Ġand Ġmeasure . ĠWe Ġalso Ġplan Ġto Ġexplore Ġadditional Ġfeatures Ġthat Ġcan Ġbe Ġintegrated Ġwith Ġa Ġsecond âĢĵ order Ġvector Ġmeasure Ġthat Ġwill Ġreduce Ġthe Ġnoise Ġbut Ġstill Ġprovide Ġsufficient Ġinformation Ġto Ġquantify Ġrelated ness . ĠWe Ġare Ġparticularly Ġinterested Ġin Ġapproaches Ġthat Ġlearn Ġword , Ġphrase , Ġand Ġsentence Ġembed d ings Ġfrom Ġstructured Ġcorpor a Ġsuch Ġas Ġliterature ĠB IB REF 23 Ġand Ġdictionary Ġentries ĠB IB REF 24 Ġ. ĠSuch Ġembed d ings Ġcould Ġbe Ġintegrated Ġinto Ġa Ġsecond âĢĵ order Ġvector Ġor Ġbe Ġused Ġon Ġtheir Ġown . </s> Finally , Ġwe Ġcompared Ġour Ġproposed Ġmethod Ġto Ġother Ġdistribution al Ġapproaches , Ġfocusing Ġon Ġthose Ġthat Ġused Ġword Ġembed d ings . ĠOur Ġresults Ġshowed Ġthat Ġintegrating Ġsemantic Ġsimilarity Ġmeasures Ġinto Ġsecond âĢĵ order Ġco âĢĵ occ urrence Ġvectors Ġob tains Ġthe Ġsame Ġor Ġhigher Ġcorrelation Ġwith Ġhuman Ġjudgments Ġas Ġdo Ġvarious Ġdifferent Ġword Ġembed ding Ġapproaches . ĠHowever , Ġa Ġdirect Ġcomparison Ġwas Ġnot Ġpossible Ġdue Ġto Ġvariations Ġin Ġthe Ġsubs ets Ġof Ġthe ĠUM NS RS Ġevaluation Ġdataset Ġused . ĠIn Ġthe Ġfuture , Ġwe Ġwould Ġnot Ġonly Ġlike Ġto Ġconduct Ġa Ġdirect Ġcomparison Ġbut Ġalso Ġexplore Ġintegrating Ġsemantic Ġsimilarity Ġinto Ġvarious Ġkinds Ġof Ġword Ġembed d ings Ġby Ġtraining Ġon Ġpair âĢĵ wise Ġvalues Ġof Ġsemantic Ġsimilarity Ġas Ġwell Ġas Ġco âĢĵ occ urrence Ġstatistics .