clinical notes from the CE task in 2010 i2b2/VA
EXTRACTIVE
clinical notes from the CE task in 2010 i2b2/VA 
EXTRACTIVE
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. The synthesized user queries are generated using the aforementioned dermatology glossary. Tagged sentences are extracted from the clinical notes. Sentences with no clinical entity present are ignored. 22,489 tagged sentences are extracted from the clinical notes. We will refer to these tagged sentences interchangeably as the i2b2 data. The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1. The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order. When combining the clinical terms, we attach the BIO tags to their constituent words. The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio. Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively. This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same.
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. The synthesized user queries are generated using the aforementioned dermatology glossary. Tagged sentences are extracted from the clinical notes. Sentences with no clinical entity present are ignored. 22,489 tagged sentences are extracted from the clinical notes. We will refer to these tagged sentences interchangeably as the i2b2 data. The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1. The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order. When combining the clinical terms, we attach the BIO tags to their constituent words. The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio. Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively. This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same.
1912.06262
Introduction </s> Medical Ġsearch Ġengines Ġare Ġan Ġessential Ġcomponent Ġfor Ġmany Ġonline Ġmedical Ġapplications , Ġsuch Ġas Ġonline Ġdiagnosis Ġsystems Ġand Ġmedical Ġdocument Ġdatabases . ĠA Ġtypical Ġonline Ġdiagnosis Ġsystem , Ġfor Ġinstance , Ġrelies Ġon Ġa Ġmedical Ġsearch Ġengine . ĠThe Ġsearch Ġengine Ġtakes Ġas Ġinput Ġa Ġuser Ġquery Ġthat Ġdescribes Ġsome Ġsymptoms Ġand Ġthen Ġoutputs Ġclinical Ġconcept Ġentries Ġthat Ġprovide Ġrelevant Ġinformation Ġto Ġassist Ġin Ġdiagn osing Ġthe Ġproblem . ĠOne Ġchallenge Ġmedical Ġsearch Ġengines Ġface Ġis Ġthe Ġsegment ation Ġof Ġindividual Ġclinical Ġentities . ĠWhen Ġa Ġuser Ġquery Ġconsists Ġof Ġmultiple Ġclinical Ġentities , Ġa Ġsearch Ġengine Ġwould Ġoften Ġfail Ġto Ġrecognize Ġthem Ġas Ġseparate Ġentities . ĠFor Ġexample , Ġthe Ġuser Ġquery ĠâĢ ľ f ever Ġjoint Ġpain Ġweight Ġloss Ġheadache âĢ Ŀ Ġcontains Ġfour Ġseparate Ġclinical Ġentities : ĠâĢ ľ f ever âĢ Ŀ , ĠâĢ ľ j oint Ġpain âĢ Ŀ , ĠâĢ ľ weight Ġloss âĢ Ŀ , Ġand ĠâĢ ľ head ache âĢ Ŀ . ĠBut Ġwhen Ġthe Ġsearch Ġengine Ġdoes Ġnot Ġrecognize Ġthem Ġas Ġseparate Ġentities Ġand Ġproceeds Ġto Ġretrieve Ġresults Ġfor Ġeach Ġword Ġin Ġthe Ġquery , Ġit Ġmay Ġfind Ġ" pain " Ġin Ġbody Ġlocations Ġother Ġthan Ġ" j oint Ġpain ", Ġor Ġit Ġmay Ġmiss Ġ" head ache " Ġaltogether , Ġfor Ġexample . ĠSome Ġsearch Ġengines Ġallow Ġthe Ġusers Ġto Ġenter Ġa Ġsingle Ġclinical Ġconcept Ġby Ġselecting Ġfrom Ġan Ġauto - com pletion Ġpick Ġlist . ĠBut Ġthis Ġcould Ġalso Ġresult Ġin Ġretrieving Ġinaccurate Ġor Ġpartial Ġresults Ġand Ġlead Ġto Ġpoor Ġuser Ġexperience . </s> We Ġwant Ġto Ġimprove Ġthe Ġmedical Ġsearch Ġengine Ġso Ġthat Ġit Ġcan Ġaccurately Ġretrieve Ġall Ġthe Ġrelevant Ġclinical Ġconcepts Ġmentioned Ġin Ġa Ġuser Ġquery , Ġwhere Ġrelevant Ġclinical Ġconcepts Ġare Ġdefined Ġwith Ġrespect Ġto Ġthe Ġtermin ologies Ġthe Ġsearch Ġengine Ġuses . ĠThe Ġproblem Ġof Ġextracting Ġclinical Ġconcept Ġmentions Ġfrom Ġa Ġuser Ġquery Ġcan Ġbe Ġseen Ġas Ġa Ġvariant Ġof Ġthe ĠConcept ĠExt raction Ġ( CE ) Ġtask Ġin Ġthe Ġfrequently - c ited ĠN LP Ġchallenges Ġin Ġhealthcare , Ġsuch Ġas Ġ2010 Ġi 2 b 2 / VA ĠB IB REF 0 Ġand Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 ĠB IB REF 1 . ĠBoth ĠCE Ġtasks Ġin Ġ2010 Ġi 2 b 2 / VA Ġand Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 Ġask Ġthe Ġparticipants Ġto Ġdesign Ġan Ġalgorithm Ġto Ġtag Ġa Ġset Ġof Ġpred efined Ġentities Ġof Ġinterest Ġin Ġclinical Ġnotes . ĠThese Ġentity Ġtagging Ġtasks Ġare Ġalso Ġknown Ġas Ġclinical ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠFor Ġexample , Ġthe ĠCE Ġtask Ġin Ġ2010 Ġi 2 b 2 / VA Ġdefines Ġthree Ġtypes Ġof Ġentities : ĠâĢ ľ problem âĢ Ŀ , ĠâĢ ľ treatment âĢ Ŀ , Ġand ĠâĢ ľ test âĢ Ŀ . ĠThe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġdefines Ġvarious Ġtypes Ġof Ġdisorder Ġsuch Ġas ĠâĢ ľ in jury Ġor Ġpoisoning âĢ Ŀ , Ġ" d ise ase Ġor Ġsyndrome âĢ Ŀ , Ġetc . ĠIn Ġaddition Ġto Ġtagging , Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġhas Ġan Ġencoding Ġcomponent Ġwhich Ġrequires Ġselecting Ġone Ġand Ġonly Ġone ĠConcept ĠUnique ĠIdent ifier Ġ( C UI ) Ġfrom ĠSystem at ized ĠN omen cl ature ĠOf ĠMedicine ĠClinical ĠTerms Ġ( SN OM ED - CT ) Ġfor Ġeach Ġdisorder Ġentity Ġtagged . ĠOur Ġproblem , Ġsimilar Ġto Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F , Ġalso Ġcontains Ġtwo Ġsub - pro blems : Ġtagging Ġmentions Ġof Ġentities Ġof Ġinterest Ġ( entity Ġtagging ), Ġand Ġselecting Ġappropriate Ġterms Ġfrom Ġa Ġgloss ary Ġto Ġmatch Ġthe Ġmentions Ġ( term Ġmatching ). ĠHowever , Ġseveral Ġmajor Ġdifferences Ġexist . ĠFirst , Ġcompared Ġto Ġclinical Ġnotes , Ġthe Ġuser Ġqueries Ġare Ġmuch Ġshorter , Ġless Ġtechnical , Ġand Ġoften Ġless Ġcoherent . ĠSecond , Ġinstead Ġof Ġencoding , Ġwe Ġare Ġdealing Ġwith Ġterm Ġmatching Ġwhere Ġwe Ġrank Ġa Ġfew Ġbest Ġterms Ġthat Ġmatch Ġan Ġentity , Ġinstead Ġof Ġselecting Ġonly Ġone . ĠThis Ġis Ġbecause Ġthe Ġusers Ġwho Ġtype Ġthe Ġqueries Ġmay Ġnot Ġhave Ġa Ġclear Ġidea Ġabout Ġwhat Ġthey Ġare Ġlooking Ġfor , Ġor Ġcould Ġbe Ġlay men Ġwho Ġknow Ġlittle Ġterminology , Ġit Ġmay Ġbe Ġmore Ġhelpful Ġto Ġprovide Ġa Ġset Ġof Ġlikely Ġresults Ġand Ġlet Ġthe Ġusers Ġchoose . ĠThird , Ġthe Ġtypes Ġof Ġentities Ġare Ġdifferent . ĠEach Ġmedical Ġsearch Ġengine Ġmay Ġhave Ġits Ġown Ġtypes Ġof Ġentities Ġto Ġtag . ĠThere Ġis Ġalso Ġone Ġminor Ġdifference Ġin Ġthe Ġtagging Ġscheme Ġbetween Ġour Ġproblem Ġand Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġ- ĠWe Ġlimit Ġour Ġscope Ġto Ġdealing Ġwith Ġentities Ġof Ġconsecutive Ġwords Ġand Ġnot Ġdis j oint Ġentities Ġ. ĠWe Ġuse Ġonly ĠBeginning , ĠInside , ĠOutside Ġ( BI O ) Ġtags . ĠGiven Ġthe Ġdifferences Ġlisted Ġabove , Ġwe Ġneed Ġto Ġcustomize Ġa Ġframework Ġconsisting Ġof Ġan Ġentity Ġtagging Ġand Ġterm Ġmatching Ġcomponent Ġfor Ġour ĠCE Ġproblem . </s> Related ĠWork </s> An Ġeffective Ġmodel Ġthat Ġhas Ġbeen Ġcommonly Ġused Ġfor ĠN ER Ġproblem Ġis Ġa ĠBi - direction al ĠL ST M Ġwith Ġa ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġon Ġthe Ġtop Ġlayer Ġ( Bi L ST M - CR F ), Ġwhich Ġis Ġdescribed Ġin Ġthe Ġnext Ġsection . ĠComb ining ĠL ST M âĢ Ļ s Ġpower Ġof Ġrepresenting Ġrelations Ġbetween Ġwords Ġand ĠCR F âĢ Ļ s Ġcapability Ġof Ġaccounting Ġfor Ġtag Ġsequence Ġconstraints , ĠHuang Ġet Ġal . ĠB IB REF 2 Ġproposed Ġthe ĠBi L ST M - CR F Ġmodel Ġand Ġused Ġhand crafted Ġword Ġfeatures Ġas Ġthe Ġinput Ġto Ġthe Ġmodel . ĠL ample Ġet Ġal . ĠB IB REF 3 Ġused Ġa Ġcombination Ġof Ġcharacter - level Ġand Ġword - level Ġword Ġembed d ings Ġas Ġthe Ġinput Ġto ĠBi L ST M - CR F . ĠSince Ġthen , Ġsimilar Ġmodels Ġwith Ġvariation Ġin Ġtypes Ġof Ġword Ġembed d ings Ġhave Ġbeen Ġused Ġextensively Ġfor Ġclinical ĠCE Ġtasks Ġand Ġproduced Ġstate - of - the - art Ġresults ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠWord Ġembed d ings Ġhave Ġbecome Ġthe Ġcornerstone Ġof Ġthe Ġneural Ġmodels Ġin ĠN LP Ġsince Ġthe Ġfamous ĠWord 2 vec ĠB IB REF 8 Ġmodel Ġdemonstrated Ġits Ġpower Ġin Ġword Ġanalogy Ġtasks . ĠOne Ġwell - known Ġexample Ġis Ġthat Ġafter Ġtraining ĠWord 2 vec Ġon Ġa Ġlarge Ġamount Ġof Ġnews Ġdata , Ġwe Ġcan Ġget Ġword Ġrelations Ġsuch Ġas Ġ$ vector ( ^ {\ prime Ġ} king ^ {\ prime Ġ}) Ġ- Ġvector ( ^ {\ prime Ġ} que en ^ {\ prime Ġ}) Ġ+ Ġvector ( ^ {\ prime Ġ} woman ^ {\ prime Ġ}) Ġ\ app rox Ġvector ( ^ {\ prime Ġ} man ^ {\ prime Ġ}) $. ĠMore Ġsophisticated Ġword Ġembed ding Ġtechnique Ġemerged Ġsince ĠWord 2 vec . ĠIt Ġhas Ġbeen Ġshown Ġempir ically Ġthat Ġbetter Ġquality Ġin Ġword Ġembed d ings Ġleads Ġto Ġbetter Ġperformance Ġin Ġmany Ġdownstream ĠN LP Ġincluding Ġentity Ġtagging ĠB IB REF 9 , ĠB IB REF 10 . ĠRecently , Ġcontextual ized Ġword Ġembed d ings Ġgenerated Ġby Ġdeep Ġlearning Ġmodels , Ġsuch Ġas ĠEL Mo ĠB IB REF 11 , ĠB ERT ĠB IB REF 12 , Ġand ĠFl air ĠB IB REF 13 , Ġhave Ġbeen Ġshown Ġto Ġbe Ġmore Ġeffective Ġin Ġvarious ĠN LP Ġtasks . ĠIn Ġour Ġproject , Ġwe Ġmake Ġuse Ġof Ġa Ġfine - tun ed ĠEL Mo Ġmodel Ġand Ġa Ġfine - tun ed ĠFl air Ġmodel Ġin Ġthe Ġmedical Ġdomain . ĠWe Ġexperiment Ġwith Ġthe Ġword Ġembed d ings Ġfrom Ġthe Ġtwo Ġfine - tun ed Ġmodels Ġas Ġthe Ġinput Ġto Ġthe ĠBi L ST M - CR F Ġmodel Ġseparately Ġand Ġcompare Ġthe Ġresults . </s> Tang Ġet Ġal . ĠB IB REF 14 Ġprovided Ġstraightforward Ġalgorithm Ġfor Ġterm Ġmatching . ĠThe Ġalgorithm Ġstarts Ġwith Ġfinding Ġcandidate Ġterms Ġthat Ġcontain ĠALL Ġthe Ġentity Ġwords , Ġwith Ġterm Ġfrequency Ġ- Ġinverse Ġdocument Ġfrequency Ġ( tf - id f ) Ġweight ing . ĠThen Ġthe Ġcandidates Ġare Ġranked Ġbased Ġon Ġthe Ġpair wise Ġcos ine Ġdistance Ġbetween Ġthe Ġword Ġembed d ings Ġof Ġthe Ġcandidates Ġand Ġthe Ġentity . </s> Fram ework </s> We Ġadopt Ġthe Ġtagging Ġ- Ġencoding Ġpipeline Ġframework Ġfrom Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F . ĠWe Ġfirst Ġtag Ġthe Ġclinical Ġentities Ġin Ġthe Ġuser Ġquery Ġand Ġthen Ġselect Ġrelevant Ġterms Ġfrom Ġa Ġgloss ary Ġin Ġdermat ology Ġto Ġmatch Ġthe Ġentities . </s> Fram ework Ġ: :: ĠEntity ĠT agging </s> We Ġuse Ġthe Ġsame ĠBi L ST M - CR F Ġmodel Ġproposed Ġby ĠHuang Ġet Ġal . ĠB IB REF 2 . ĠAn Ġillustration Ġof Ġthe Ġarchitecture Ġis Ġshown Ġin ĠFigure ĠFIG REF 6 Ġ. ĠGiven Ġa Ġsequence Ġ( or Ġsentence ) Ġof Ġn Ġtokens , Ġ$ r Ġ= Ġ( w _ 1 , Ġw _ 2 ,... , Ġw _ n ) $, Ġwe Ġuse Ġa Ġfine - tun ed ĠEL Mo Ġmodel Ġto Ġgenerate Ġcontextual Ġword Ġembed d ings Ġfor Ġall Ġthe Ġtokens Ġin Ġthe Ġsentence , Ġwhere Ġa Ġtoken Ġrefers Ġto Ġa Ġword Ġor Ġpunct uation . ĠWe Ġdenote Ġthe ĠEL Mo Ġembed ding , Ġ$ x $, Ġfor Ġa Ġtoken Ġ$ w $ Ġin Ġthe Ġsentence Ġ$ r $ Ġby Ġ$ x Ġ= ĠEL Mo ( w | r ) $. ĠThe Ġnotation Ġand Ġthe Ġprocedure Ġdescribed Ġhere Ġcan Ġbe Ġadopted Ġfor ĠFl air Ġembed d ings Ġor Ġother Ġembed d ings . ĠNow , Ġgiven Ġa Ġsequence Ġof Ġtokens Ġin ĠEL Mo Ġembed d ings , Ġ$ X Ġ= ( x _ 1 , Ġx _ 2 , Ġ... , Ġx _ n ) $, Ġthe ĠBi L ST M Ġlayer Ġgenerates Ġa Ġmatrix Ġof Ġscores , Ġ$ P (\ the ta Ġ) $ Ġof Ġsize Ġ$ n Ġ\ times Ġk $, Ġwhere Ġ$ k $ Ġis Ġthe Ġnumber Ġof Ġtag Ġtypes , Ġand Ġ$\ the ta Ġ$ Ġis Ġthe Ġparameters Ġof Ġthe ĠBi L ST M . ĠTo Ġsimplify Ġnotation , Ġwe Ġwill Ġomit Ġthe Ġ$\ the ta Ġ$ Ġand Ġwrite Ġ$ P $. ĠThen , Ġ$ P _{ i , j } $ Ġdenotes Ġthe Ġscore Ġof Ġthe Ġtoken , Ġ$ x _ i $, Ġbeing Ġassigned Ġto Ġthe Ġ$ j $ th Ġtag . ĠSince Ġcertain Ġconstraints Ġmay Ġexist Ġin Ġthe Ġtransition Ġbetween Ġtags , Ġan Ġ" O " Ġtag Ġshould Ġnot Ġbe Ġfollowed Ġby Ġan Ġ" I " Ġtag , Ġfor Ġexample , Ġa Ġtransition Ġmatrix , Ġ$ A $, Ġof Ġdimension Ġ$( k + 2 )\ times Ġ( k + 2 ) $, Ġis Ġinitialized Ġto Ġmodel Ġthe Ġconstraints . ĠThe Ġlearn able Ġparameters , Ġ$ A _{ i , j } $, Ġrepresent Ġthe Ġprobability Ġof Ġthe Ġ$ j $ th Ġtag Ġfollows Ġthe Ġ$ i $ th Ġtag Ġin Ġa Ġsequence . ĠFor Ġexample , Ġif Ġwe Ġindex Ġthe Ġtags Ġby : Ġ1 : âĢ ľ B âĢ Ŀ , Ġ2 : âĢ ľ I âĢ Ŀ , Ġand Ġ3 : âĢ ľ O âĢ Ŀ , Ġthen Ġ$ A _{ 1 , 3 } $ Ġwould Ġbe Ġthe Ġprobability Ġthat Ġan ĠâĢ ľ O âĢ Ŀ Ġtag Ġfollows Ġa ĠâĢ ľ B âĢ Ŀ Ġtag . ĠA Ġbeginning Ġtransition Ġand Ġan Ġend Ġtransition Ġare Ġinserted Ġin Ġ$ A $ Ġand Ġhence Ġ$ A $ Ġis Ġof Ġdimension Ġ$( k + 2 )\ times Ġ( k + 2 ) $. </s> Given Ġa Ġsequence Ġof Ġtags , Ġ$ Y =( y _ 1 , y _ 2 ,... , y _ n ) $, Ġwhere Ġeach Ġ$ y _ i $, Ġ$ 1 \ le Ġi Ġ\ le Ġn $, Ġcorresponds Ġto Ġan Ġindex Ġof Ġthe Ġtags , Ġthe Ġscore Ġof Ġthe Ġsequence Ġis Ġthen Ġgiven Ġby </s> The Ġprobability Ġof Ġthe Ġsequence Ġof Ġtags Ġis Ġthen Ġcalculated Ġby Ġa Ġsoft max , </s> where Ġ$\ l brace ĠY _ x \ r brace Ġ$ Ġdenotes Ġthe Ġset Ġof Ġall Ġpossible Ġtag Ġsequences . ĠDuring Ġtraining , Ġthe Ġobjective Ġfunction Ġis Ġto Ġmaximize Ġ$\ log Ġ( P ( Y | X )) $ Ġby Ġadjusting Ġ$ A $ Ġand Ġ$ P $. </s> Fram ework Ġ: :: ĠTerm ĠMatch ing </s> The Ġterm Ġmatching Ġalgorithm Ġof ĠTang Ġet Ġal . ĠB IB REF 14 Ġis Ġadopted Ġwith Ġsome Ġmajor Ġmodifications . ĠFirst , Ġto Ġidentify Ġcandidate Ġterms , Ġwe Ġuse Ġa Ġmuch Ġlo oser Ġstring Ġsearch Ġalgorithm Ġwhere Ġwe Ġstem Ġthe Ġentity Ġwords Ġwith Ġsnowball Ġstem mer Ġand Ġthen Ġfind Ġcandidate Ġterms Ġwhich Ġcontain ĠANY Ġnon - stop word Ġwords Ġin Ġthe Ġentity . ĠThe Ġstemming Ġis Ġmainly Ġused Ġto Ġrepresent Ġa Ġword Ġin Ġits Ġstem . ĠFor Ġexample , ĠâĢ ľ leg s âĢ Ŀ Ġbecomes ĠâĢ ľ leg âĢ Ŀ , Ġand ĠâĢ ľ j ammed âĢ Ŀ Ġbecomes ĠâĢ ľ jam âĢ Ŀ . ĠThus , Ġstemming Ġcan Ġprovide Ġmore Ġtolerance Ġwhen Ġfinding Ġcandidates . ĠSimilarly , Ġfinding Ġcandidates Ġusing Ġthe Ġcondition ĠANY Ġ( instead Ġof ĠALL ) Ġalso Ġincreases Ġthe Ġtolerance . ĠHowever , Ġif Ġthe Ġtagged Ġentity Ġcontains Ġstop words Ġsuch Ġas ĠâĢ ľ in âĢ Ŀ , ĠâĢ ľ on âĢ Ŀ , Ġetc ., Ġthe Ġpool Ġof Ġcandidates Ġwill Ġnaturally Ġgrow Ġvery Ġlarge Ġand Ġincrease Ġthe Ġcomputation Ġcost Ġfor Ġlater Ġpart , Ġbecause Ġeven Ġcompletely Ġirrelevant Ġterms Ġmay Ġcontain Ġstop words . ĠTherefore , Ġwe Ġmatch Ġbased Ġon Ġnon - stop words Ġonly . ĠTo Ġillustrate Ġthe Ġpoints Ġabove , Ġsuppose Ġa Ġquery Ġis Ġtagged Ġwith Ġthe Ġentity </s> Ex . Ġ3 . 1 ĠâĢ ľ severe Ġburns Ġon Ġlegs âĢ Ŀ , </s> and Ġone Ġrelevant Ġterm Ġis ĠâĢ ľ leg Ġburn âĢ Ŀ . ĠAfter Ġstemming , ĠâĢ ľ burn s âĢ Ŀ Ġand ĠâĢ ľ leg s âĢ Ŀ Ġin ĠEx . UN K REF 12 Ġbecome ĠâĢ ľ burn âĢ Ŀ Ġand ĠâĢ ľ leg âĢ Ŀ , Ġrespectively , Ġallowing Ġ" leg Ġburn " Ġto Ġbe Ġconsidered Ġas Ġa Ġcandidate . ĠAlthough Ġthe Ġword ĠâĢ ľ severe âĢ Ŀ Ġis Ġnot Ġin Ġthe Ġterm ĠâĢ ľ leg Ġburn âĢ Ŀ , Ġthe Ġterm Ġis Ġstill Ġconsidered Ġa Ġcandidate Ġbecause Ġwe Ġselected Ġusing ĠANY . ĠThe Ġstop word ĠâĢ ľ on âĢ Ŀ Ġis Ġignored Ġwhen Ġfinding Ġcandidate Ġterms Ġso Ġthat Ġnot Ġevery Ġterm Ġthat Ġcontains Ġthe Ġword ĠâĢ ľ on âĢ Ŀ Ġis Ġadded Ġto Ġthe Ġcandidate Ġpool . ĠWhen Ġa Ġcandidate Ġterm , Ġ$ C $, Ġis Ġfound Ġin Ġthis Ġmanner Ġfor Ġthe Ġtagged Ġentity , Ġ$ E $, Ġwe Ġcalculate Ġthe Ġsemantic Ġsimilarity Ġscore , Ġ$ s $, Ġbetween Ġ$ C $ Ġand Ġ$ E $ Ġin Ġtwo Ġsteps . ĠIn Ġthe Ġfirst Ġstep , Ġcalculate Ġthe Ġmaximum Ġsimilarity Ġscore Ġfor Ġeach Ġword Ġin Ġ$ C $ Ġas Ġshown Ġin ĠFigure ĠFIG REF 10 . ĠGiven Ġa Ġword Ġin Ġthe Ġcandidate Ġterm , Ġ$ C _ i $ Ġ($ 1 Ġ\ le Ġi Ġ\ le Ġm $, Ġ$ m $ Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġcandidate Ġterm ) Ġand Ġa Ġword Ġin Ġthe Ġtagged Ġentity , Ġ$ E _ j $. Their Ġsimilarity Ġscore , Ġ$ s _{ ij } $ Ġ( shown Ġas Ġthe Ġelement Ġin Ġthe Ġboxed Ġmatrix Ġin ĠFigure ĠFIG REF 10 ), Ġis Ġgiven Ġby </s> where Ġ$ EL Mo ( C _ i | C ) $ Ġand Ġ$ EL Mo ( E _ j | E ) $ Ġare Ġthe ĠEL Mo Ġembed d ings Ġfor Ġthe Ġword Ġ$ C _ i $ Ġand Ġ$ E _ j $, Ġrespectively . ĠThe ĠEL Mo Ġembed d ings Ġhave Ġthe Ġsame Ġdimension Ġfor Ġall Ġwords Ġwhen Ġusing Ġthe Ġsame Ġfine - tun ed ĠEL Mo Ġmodel . ĠThus , Ġwe Ġcan Ġuse Ġa Ġdistance Ġfunction Ġ( e . g ., Ġthe Ġcos ine Ġdistance ), Ġden oted Ġ$ d (\ cd ot Ġ) $ Ġin Ġequation ĠDIS PLAY _ FORM 13 , Ġto Ġcompute Ġthe Ġsemantic Ġsimilarity Ġbetween Ġwords . ĠIn Ġstep Ġ2 , Ġwe Ġcalculate Ġthe Ġcandidate - entity Ġrelevance Ġscore Ġ( similar ity ) Ġusing Ġthe Ġformula </s> where Ġ$ s _ c $ Ġis Ġa Ġscore Ġthreshold , Ġand Ġ$\ math bb Ġ{ I } Ġ\ l brace Ġmax (\ vec { S _ i }) Ġ> Ġs _ c \ r brace Ġ$ Ġis Ġan Ġindicator Ġfunction Ġthat Ġequals Ġ1 Ġif Ġ$ max (\ vec { S _ i }) Ġ> Ġs _ c $ Ġor Ġequals Ġ0 Ġif Ġnot . ĠIn Ġequation ĠDIS PLAY _ FORM 14 Ġwe Ġdefine Ġa Ġmetric Ġthat Ġmeasures ĠâĢ ľ information Ġcoverage âĢ Ŀ Ġof Ġthe Ġcandidate Ġterms Ġwith Ġrespect Ġto Ġa Ġtagged Ġentity . ĠIf Ġthe Ġconstituent Ġwords Ġof Ġa Ġcandidate Ġterm Ġare Ġrelevant Ġto Ġthe Ġconstituent Ġwords Ġin Ġthe Ġtagged Ġentity , Ġthen Ġthe Ġcandidate Ġterm Ġoffers Ġmore Ġinformation Ġcoverage . ĠInt uitive ly , Ġthe Ġmore Ġrelevant Ġwords Ġpresent Ġin Ġthe Ġcandidate Ġterm , Ġthe Ġmore Ġrelevant Ġthe Ġcandidate Ġis Ġto Ġthe Ġtagged Ġentity . ĠThe Ġpurpose Ġof Ġthe Ġcutoff , Ġ$ s _ c $, Ġis Ġto Ġscreen Ġthe Ġ$( C _ i , E _ j ) $ Ġword Ġpairs Ġthat Ġare Ġdiss imilar , Ġso Ġthat Ġthey Ġdo Ġnot Ġcontribute Ġto Ġinformation Ġcoverage . ĠOne Ġcan Ġadjust Ġthe Ġstrict ness Ġof Ġthe Ġentity Ġ- Ġterminology Ġmatching Ġby Ġadjusting Ġ$ s _ c $. ĠThe Ġhigher Ġwe Ġset Ġ$ s _ c $, Ġthe Ġfewer Ġcandidate Ġterms Ġwill Ġbe Ġselected Ġfor Ġa Ġtagged Ġentity . ĠA Ġnormal ization Ġfactor , Ġ$\ frac { 1 }{ m } $, Ġis Ġadded Ġto Ġgive Ġpreference Ġto Ġmore Ġconcise Ġcandidate Ġterms Ġgiven Ġthe Ġsame Ġamount Ġof Ġinformation Ġcoverage . </s> We Ġneed Ġto Ġcreate Ġan Ġextra Ġstop word Ġlist Ġto Ġinclude Ġwords Ġsuch Ġas ĠâĢ ľ config uration âĢ Ŀ Ġand ĠâĢ ľ color âĢ Ŀ , Ġand Ġexclude Ġthese Ġwords Ġfrom Ġthe Ġword Ġcount Ġfor Ġa Ġcandidate Ġterm . ĠThis Ġis Ġbecause Ġthe Ġterms Ġassociated Ġwith Ġthe Ġdescription Ġof Ġcolor Ġor Ġconfiguration Ġusually Ġhave Ġthe Ġword ĠâĢ ľ color âĢ Ŀ Ġor ĠâĢ ľ config uration âĢ Ŀ Ġin Ġthem . ĠOn Ġthe Ġother Ġhand , Ġa Ġuser Ġquery Ġnormally Ġdoes Ġnot Ġcontain Ġsuch Ġwords . ĠFor Ġexample , Ġa Ġtagged Ġentity Ġin Ġa Ġuser Ġquery Ġcould Ġbe ĠâĢ ľ round Ġyellow Ġpatches âĢ Ŀ , Ġfor Ġwhich Ġthe Ġrelevant Ġtermin ologies Ġinclude ĠâĢ ľ round Ġconfiguration âĢ Ŀ Ġand ĠâĢ ľ yellow Ġcolor âĢ Ŀ . ĠSince Ġwe Ġapplied Ġa Ġnormal ization Ġfactor , Ġ$\ frac { 1 }{ m } $, Ġto Ġthe Ġrelevance Ġscore , Ġthe Ġword ĠâĢ ľ color âĢ Ŀ Ġand ĠâĢ ľ config uration âĢ Ŀ Ġwould Ġlower Ġthe Ġrelevance Ġscore Ġbecause Ġthey Ġdo Ġnot Ġhave Ġa Ġcounterpart Ġin Ġthe Ġtagged Ġentity . ĠTherefore , Ġwe Ġneed Ġto Ġexclude Ġthem Ġfrom Ġword Ġcount . ĠOnce Ġthe Ġprocess Ġis Ġcomplete , Ġcalculate Ġ$ s ( C , E ) $ Ġfor Ġall Ġcandidate Ġterms Ġand Ġthen Ġwe Ġcan Ġapply Ġa Ġthreshold Ġon Ġall Ġ$ s ( C , E ) $ Ġto Ġignore Ġcandidate Ġterms Ġwith Ġlow Ġinformation Ġcoverage . ĠFinally , Ġrank Ġthe Ġterms Ġby Ġtheir Ġ$ s ( C , E ) $ Ġand Ġreturn Ġthe Ġranked Ġlist Ġas Ġthe Ġresults . </s> Exper iments Ġ: :: ĠData </s> Despite Ġthe Ġgreater Ġsimilarity Ġbetween Ġour Ġtask Ġand Ġthe Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 , Ġwe Ġuse Ġthe Ġclinical Ġnotes Ġfrom Ġthe ĠCE Ġtask Ġin Ġ2010 Ġi 2 b 2 / VA Ġon Ġaccount Ġof Ġ1 ) Ġthe Ġdata Ġfrom Ġ2010 Ġi 2 b 2 / VA Ġbeing Ġeasier Ġto Ġaccess Ġand Ġparse , Ġ2 ) Ġ2013 ĠSh AR e / CLE F Ġcontaining Ġdis j oint Ġentities Ġand Ġhence Ġrequiring Ġmore Ġcomplicated Ġtagging Ġschemes . ĠThe Ġsynthes ized Ġuser Ġqueries Ġare Ġgenerated Ġusing Ġthe Ġaforementioned Ġdermat ology Ġgloss ary . ĠT agged Ġsentences Ġare Ġextracted Ġfrom Ġthe Ġclinical Ġnotes . ĠSent ences Ġwith Ġno Ġclinical Ġentity Ġpresent Ġare Ġignored . Ġ22 , 489 Ġtagged Ġsentences Ġare Ġextracted Ġfrom Ġthe Ġclinical Ġnotes . ĠWe Ġwill Ġrefer Ġto Ġthese Ġtagged Ġsentences Ġinterchange ably Ġas Ġthe Ġi 2 b 2 Ġdata . ĠThe Ġsentences Ġare Ġshuff led Ġand Ġsplit Ġinto Ġtrain / dev / test Ġset Ġwith Ġa Ġratio Ġof Ġ7 : 2 : 1 . ĠThe Ġsynthes ized Ġuser Ġqueries Ġare Ġcomposed Ġby Ġrandomly Ġselecting Ġseveral Ġclinical Ġterms Ġfrom Ġthe Ġdermat ology Ġgloss ary Ġand Ġthen Ġcombining Ġthem Ġin Ġno Ġparticular Ġorder . ĠWhen Ġcombining Ġthe Ġclinical Ġterms , Ġwe Ġattach Ġthe ĠB IO Ġtags Ġto Ġtheir Ġconstituent Ġwords . ĠThe Ġsynthes ized Ġuser Ġqueries Ġ( 13 , 697 Ġin Ġtotal ) Ġare Ġthen Ġsplit Ġinto Ġtrain / dev / test Ġset Ġwith Ġthe Ġsame Ġratio . ĠNext , Ġeach Ġset Ġin Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġcorresponding Ġset Ġin Ġthe Ġsynthes ized Ġquery Ġdata Ġare Ġcombined Ġto Ġform Ġa Ġhybrid Ġtrain / dev / test Ġset , Ġrespectively . ĠThis Ġway Ġwe Ġensure Ġthat Ġin Ġeach Ġhybrid Ġtrain / dev / test Ġset , Ġthe Ġratio Ġbetween Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġsynthes ized Ġquery Ġdata Ġis Ġthe Ġsame . </s> The Ġreason Ġfor Ġcombining Ġthe Ġtwo Ġdata Ġis Ġtheir Ġdrastic Ġstructural Ġdifference Ġ( See Ġfigure ĠFIG REF 16 Ġfor Ġan Ġexample ). ĠPreviously , Ġwhen Ġtrained Ġon Ġthe Ġi 2 b 2 Ġdata Ġonly , Ġthe ĠBi L ST M - CR F Ġmodel Ġwas Ġnot Ġable Ġto Ġsegment Ġclinical Ġentities Ġat Ġthe Ġcorrect Ġboundary . ĠIt Ġwould Ġfail Ġto Ġrecognize Ġthe Ġuser Ġquery Ġin ĠFigure ĠFIG REF 16 ( a ) Ġas Ġfour Ġseparate Ġentities . ĠOn Ġthe Ġother Ġhand , Ġif Ġthe Ġmodel Ġwas Ġtrained Ġsolely Ġon Ġthe Ġsynthes ized Ġuser Ġqueries , Ġwe Ġcould Ġimagine Ġthat Ġit Ġwould Ġfail Ġmiser ably Ġon Ġany Ġqueries Ġthat Ġresemble Ġthe Ġsentence Ġin ĠFigure ĠFIG REF 16 ( b ) Ġbecause Ġthe Ġmodel Ġwould Ġhave Ġnever Ġseen Ġan ĠâĢ ľ O âĢ Ŀ Ġtag Ġin Ġthe Ġtraining Ġdata . ĠTherefore , Ġit Ġis Ġnecessary Ġto Ġuse Ġthe Ġhybrid Ġtraining Ġdata Ġcontaining Ġboth Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġsynthes ized Ġuser Ġqueries . </s> To Ġmake Ġthe Ġhybrid Ġtraining Ġdata , Ġwe Ġneed Ġto Ġun ify Ġthe Ġtags . ĠRecall Ġthat Ġin ĠSection ĠSEC REF 1 Ġwe Ġpoint Ġout Ġthat Ġthe Ġtags Ġare Ġdifferent Ġfor Ġthe Ġdifferent Ġtasks Ġand Ġdatasets . ĠSince Ġwe Ġuse Ġcustom Ġtags Ġfor Ġdermat ology Ġgloss ary Ġin Ġour Ġproblem , Ġwe Ġwould Ġneed Ġto Ġconvert Ġthe Ġtags Ġused Ġin Ġ2010 Ġi 2 b 2 / VA . ĠBut Ġthis Ġwould Ġbe Ġan Ġinf eas ible Ġjob Ġas Ġwe Ġneed Ġexperts Ġto Ġmanually Ġdo Ġthat . ĠAn Ġalternative Ġis Ġto Ġavoid Ġdistinguishing Ġthe Ġtag Ġtypes Ġand Ġlabel Ġall Ġtags Ġunder Ġthe Ġgeneric ĠB IO Ġtags . </s> Exper iments Ġ: :: ĠSetup </s> To Ġshow Ġthe Ġeffects Ġof Ġusing Ġthe Ġhybrid Ġtraining Ġdata , Ġwe Ġtrained Ġtwo Ġmodels Ġof Ġthe Ġsame Ġarchitecture Ġand Ġhyper param eters . ĠOne Ġmodel Ġwas Ġtrained Ġon Ġthe Ġhybrid Ġdata Ġand Ġwill Ġbe Ġreferred Ġto Ġas Ġhybrid ĠN ER Ġmodel . ĠThe Ġother Ġmodel Ġwas Ġtrained Ġon Ġclinical Ġnotes Ġonly Ġand Ġwill Ġbe Ġreferred Ġto Ġas Ġi 2 b 2 ĠN ER Ġmodel . ĠWe Ġevaluated Ġthe Ġperformance Ġof Ġthe ĠN ER Ġmodels Ġby Ġmicro - F 1 Ġscore Ġon Ġthe Ġtest Ġset Ġof Ġboth Ġthe Ġsynthes ized Ġqueries Ġand Ġthe Ġi 2 b 2 Ġdata . </s> We Ġused Ġthe ĠBi L ST M - CR F Ġimplementation Ġprovided Ġby Ġthe Ġflair Ġpackage ĠB IB REF 16 . ĠWe Ġset Ġthe Ġhidden Ġsize Ġvalue Ġto Ġbe Ġ256 Ġin Ġthe ĠL ST M Ġstructure Ġand Ġleft Ġeverything Ġelse Ġat Ġdefault Ġvalues Ġfor Ġthe ĠSequence T agger Ġmodel Ġon Ġflair . ĠFor Ġword Ġembed d ings , Ġwe Ġused Ġthe ĠEL Mo Ġembed d ings Ġfine - tun ed Ġon ĠPubMed Ġarticles Ġand Ġflair Ġembed d ings ĠB IB REF 13 Ġtrained Ġon Ġ$ 5 \ % $ Ġof ĠPubMed Ġabstract s Ġ, Ġrespectively . ĠWe Ġtrained Ġmodels Ġfor Ġ10 Ġepoch s Ġand Ġexperimented Ġwith Ġdifferent Ġlearning Ġrate , Ġmini Ġbatch Ġsize , Ġand Ġdrop outs . ĠWe Ġran Ġhyper param eter Ġoptimization Ġtests Ġto Ġfind Ġthe Ġbest Ġcombination . Ġ$ S _ c $ Ġis Ġset Ġto Ġbe Ġ0 . 6 Ġin Ġour Ġexperiment . </s> Exper iments Ġ: :: ĠHyper param eter ĠTun ing </s> We Ġdefined Ġthe Ġfollowing Ġhyper param eter Ġsearch Ġspace : </s> embed d ings : Ġ[ âĢ ľ EL Mo Ġon Ġpub med âĢ Ŀ , ĠâĢ ľ st acked Ġflair Ġon Ġpub med âĢ Ŀ ], </s> hidden _ size : Ġ[ 128 , Ġ256 ], </s> learning _ rate : Ġ[ 0 . 05 , Ġ0 . 1 ], </s> mini _ batch _ size : Ġ[ 32 , Ġ64 , Ġ128 ]. </s> The Ġhyper param eter Ġoptimization Ġwas Ġperformed Ġusing ĠHyper opt Ġ. ĠThree Ġevaluations Ġwere Ġrun Ġfor Ġeach Ġcombination Ġof Ġhyper param eters . ĠEach Ġran Ġfor Ġ10 Ġepoch s . ĠThen Ġthe Ġresults Ġwere Ġaveraged Ġto Ġgive Ġthe Ġperformance Ġfor Ġthat Ġparticular Ġcombination Ġof Ġhyper param eters . </s> Exper iments Ġ: :: ĠResults </s> From Ġthe Ġhyper param eter Ġtuning Ġwe Ġfound Ġthat Ġthe Ġbest Ġcombination Ġwas </s> embed d ings : ĠâĢ ľ EL Mo Ġon Ġpub med âĢ Ŀ , </s> hidden _ size : Ġ256 , </s> learning _ rate : Ġ0 . 05 , </s> mini _ batch _ size : Ġ32 . </s> With Ġthe Ġabove Ġhyper param eter Ġsetting , Ġthe Ġhybrid ĠN ER Ġmodel Ġachieved Ġa ĠF 1 Ġscore Ġof Ġ$ 0 . 995 $ Ġon Ġsynthes ized Ġqueries Ġand Ġ$ 0 . 9 48 $ Ġon Ġclinical Ġnotes Ġwhile Ġthe Ġi 2 b 2 ĠN ER Ġmodel Ġachieved Ġa ĠF 1 Ġscore Ġof Ġ$ 0 . 441 $ Ġon Ġsynthes ized Ġqueries Ġand Ġ$ 0 . 9 27 $ Ġon Ġclinical Ġnotes Ġ( See ĠTable ĠT AB REF 23 ). </s> Since Ġthere Ġwas Ġno Ġground Ġtruth Ġavailable Ġfor Ġthe Ġretrieved Ġterms , Ġwe Ġrandomly Ġpicked Ġa Ġfew Ġsamples Ġto Ġassess Ġits Ġperformance . ĠSome Ġexample Ġoutputs Ġof Ġour Ġcomplete Ġframework Ġon Ġreal Ġuser Ġqueries Ġare Ġshown Ġin ĠFigure ĠFIG REF 24 . ĠFor Ġexample , Ġfrom Ġthe Ġfigure Ġwe Ġsee Ġthat Ġthe Ġquery Ġ" child Ġfever Ġdouble Ġvision Ġdizz y " Ġwas Ġcorrectly Ġtagged Ġwith Ġfour Ġentities : Ġ" child ", Ġ" f ever ", Ġ" double Ġvision ", Ġand Ġ" d izzy ". ĠA Ġlist Ġof Ġterms Ġfrom Ġour Ġgloss ary Ġwas Ġmatched Ġto Ġeach Ġentity . ĠIn Ġreal Ġworld Ġapplication , Ġthe Ġlists Ġof Ġterms Ġwill Ġbe Ġpresented Ġto Ġthe Ġuser Ġas Ġthe Ġretrieval Ġresults Ġto Ġtheir Ġqueries . </s> Discussion </s> In Ġmost Ġreal Ġuser Ġqueries Ġwe Ġsampled , Ġthe Ġentities Ġwere Ġtagged Ġat Ġthe Ġcorrect Ġboundary Ġand Ġthe Ġtagging Ġwas Ġcomplete Ġ( such Ġas Ġthe Ġones Ġshown Ġin ĠFigure ĠFIG REF 24 ). ĠOnly Ġon Ġa Ġfew Ġuser Ġqueries Ġthe Ġtagging Ġwas Ġcontroversial . ĠFor Ġexample , Ġthe Ġquery ĠâĢ ľ E ry the mat ous Ġbl anch ing Ġround , Ġoval Ġpatches Ġon Ġtorso , Ġextrem ities âĢ Ŀ Ġwas Ġtagged Ġas ĠâĢ ľ E ry the mat ous Ġbl anch ing âĢ Ŀ Ġand ĠâĢ ľ oval Ġpatches Ġon Ġtorso âĢ Ŀ . ĠThe Ġentity ĠâĢ ľ ext rem ities âĢ Ŀ Ġwas Ġmissing . ĠThe Ġsegment ation Ġwas Ġalso Ġnot Ġcorrect . ĠA Ġmore Ġappropriate Ġtagging Ġwould Ġbe ĠâĢ ľ E ry the mat ous Ġbl anch ing Ġround , Ġoval Ġpatches âĢ Ŀ , ĠâĢ ľ tor so âĢ Ŀ , Ġand ĠâĢ ľ ext rem ities âĢ Ŀ . ĠThe Ġtagging Ġcould Ġbe Ġfurther Ġimproved Ġby Ġsynthes izing Ġmore Ġrealistic Ġuser Ġqueries . ĠRecall Ġthat Ġthe Ġsynthes ized Ġuser Ġqueries Ġwere Ġcreated Ġby Ġrandomly Ġcombining Ġtermin ologies Ġfrom Ġthe Ġdermat ology Ġgloss ary , Ġwhich , Ġwhile Ġproviding Ġdata Ġthat Ġhelped Ġthe Ġmodel Ġlearn Ġentity Ġsegment ation , Ġdid Ġnot Ġreflect Ġthe Ġco - occ urrence Ġinformation Ġin Ġreal Ġuser Ġqueries . ĠFor Ġexample , Ġthere Ġcould Ġbe Ġtwo Ġclinical Ġentities Ġthat Ġoften Ġco - occ ur Ġor Ġnever Ġco - occ ur Ġin Ġa Ġuser Ġquery . ĠBut Ġsince Ġthe Ġsynthes ized Ġuser Ġqueries Ġwe Ġused Ġcombined Ġterms Ġrandomly , Ġthe Ġco - occ urrence Ġinformation Ġwas Ġthus Ġmissing . </s> The Ġfinal Ġretrieval Ġresults Ġof Ġour Ġframework Ġwere Ġnot Ġevaluated Ġquant itatively Ġin Ġterms Ġof Ġrecall Ġand Ġprecision , Ġdue Ġthe Ġthe Ġlack Ġof Ġground Ġtruth . ĠWhen Ġground Ġtruth Ġbecomes Ġavailable , Ġwe Ġwill Ġbe Ġable Ġto Ġevaluate Ġour Ġframework Ġmore Ġthoroughly . ĠRecently , Ġa Ġfine - tun ed ĠB ERT Ġmodel Ġin Ġthe Ġmedical Ġdomain Ġcalled ĠBio BER T ĠB IB REF 17 Ġhas Ġattracted Ġsome Ġattention Ġin Ġthe Ġmedical ĠN LP Ġdomain . ĠWe Ġcould Ġexperiment Ġwith ĠBio BER T Ġembed d ings Ġin Ġthe Ġfuture . ĠWe Ġcould Ġalso Ġinclude Ġquery Ġexpansion Ġtechnique Ġfor Ġterm Ġmatching . ĠWhen Ġfinding Ġcandidate Ġterms Ġfor Ġan Ġentity , Ġour Ġfirst Ġstep Ġwas Ġstill Ġbased Ġon Ġstring Ġmatching . ĠGiven Ġthat Ġthere Ġmight Ġbe Ġmultiple Ġentities Ġthat Ġcould Ġbe Ġmatched Ġto Ġthe Ġsame Ġterm , Ġit Ġcould Ġbe Ġhard Ġto Ġinclude Ġall Ġthese Ġentities Ġin Ġthe Ġgloss ary Ġand Ġhard Ġto Ġmatch Ġterms Ġto Ġthese Ġentities . </s> Conclusion </s> In Ġthis Ġproject , Ġwe Ġtackle Ġthe Ġproblem Ġof Ġextracting Ġclinical Ġconcepts Ġfrom Ġuser Ġqueries Ġon Ġmedical Ġsearch Ġengines . ĠBy Ġtraining Ġa ĠBi L ST M - CR F Ġmodel Ġon Ġa Ġhybrid Ġdata Ġconsisting Ġof Ġsynthes ized Ġuser Ġqueries Ġand Ġsentences Ġfrom Ġclinical Ġnote , Ġwe Ġadopt Ġa ĠCE Ġframework Ġfor Ġclinical Ġuser Ġqueries Ġwith Ġminimal Ġeffort Ġspent Ġon Ġannot ating Ġuser Ġqueries . ĠWe Ġfind Ġthat Ġthe Ġhybrid Ġdata Ġenables Ġthe ĠN ER Ġmodel Ġperform Ġbetter Ġon Ġboth Ġtagging Ġthe Ġuser Ġqueries Ġand Ġthe Ġclinical Ġnote Ġsentences . ĠFurthermore , Ġour Ġframework Ġis Ġbuilt Ġon Ġan Ġeasy - to - use Ġdeep Ġlearning ĠN LP ĠPython Ġlibrary , Ġwhich Ġlends Ġit Ġmore Ġprospective Ġvalue Ġto Ġvarious Ġonline Ġmedical Ġapplications Ġthat Ġemploy Ġmedical Ġsearch Ġengines . </s> Acknowled gment </s> This Ġpaper Ġresults Ġfrom Ġa Ġtechnical Ġreport Ġof Ġa Ġproject Ġthe Ġauthors Ġhave Ġworked Ġon Ġwith Ġvisual D x , Ġa Ġhealthcare Ġinform atics Ġcompany Ġthat Ġprovides Ġweb - based Ġclinical Ġdecision Ġsupport Ġsystem . ĠThe Ġauthors Ġwould Ġlike Ġto Ġthank Ġvisual D x Ġfor Ġproviding Ġthem Ġthe Ġopportunity Ġto Ġwork Ġon Ġsuch Ġan Ġexciting Ġproject . ĠIn Ġparticular , Ġthe Ġauthors Ġwould Ġlike Ġto Ġthank ĠRoy ĠRobinson , Ġthe ĠVice ĠPresident Ġof ĠTechnology Ġand ĠMedical ĠIn format ics Ġat Ġvisual D x , Ġfor Ġproviding Ġthe Ġsynthes ized Ġuser Ġqueries , Ġas Ġwell Ġas Ġpreliminary Ġfeedback Ġon Ġthe Ġperformance Ġof Ġour Ġframework .
Introduction Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases. A typical online diagnosis system, for instance, relies on a medical search engine. The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem. One challenge medical search engines face is the segmentation of individual clinical entities. When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities. For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”. But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example. Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list. But this could also result in retrieving inaccurate or partial results and lead to poor user experience. We want to improve the medical search engine so that it can accurately retrieve all the relevant clinical concepts mentioned in a user query, where relevant clinical concepts are defined with respect to the terminologies the search engine uses. The problem of extracting clinical concept mentions from a user query can be seen as a variant of the Concept Extraction (CE) task in the frequently-cited NLP challenges in healthcare, such as 2010 i2b2/VA BIBREF0 and 2013 ShARe/CLEF Task 1 BIBREF1. Both CE tasks in 2010 i2b2/VA and 2013 ShARe/CLEF Task 1 ask the participants to design an algorithm to tag a set of predefined entities of interest in clinical notes. These entity tagging tasks are also known as clinical Named Entity Recognition (NER). For example, the CE task in 2010 i2b2/VA defines three types of entities: “problem”, “treatment”, and “test”. The CE task in 2013 ShARe/CLEF defines various types of disorder such as “injury or poisoning”, "disease or syndrome”, etc. In addition to tagging, the CE task in 2013 ShARe/CLEF has an encoding component which requires selecting one and only one Concept Unique Identifier (CUI) from Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT) for each disorder entity tagged. Our problem, similar to the CE task in 2013 ShARe/CLEF, also contains two sub-problems: tagging mentions of entities of interest (entity tagging), and selecting appropriate terms from a glossary to match the mentions (term matching). However, several major differences exist. First, compared to clinical notes, the user queries are much shorter, less technical, and often less coherent. Second, instead of encoding, we are dealing with term matching where we rank a few best terms that match an entity, instead of selecting only one. This is because the users who type the queries may not have a clear idea about what they are looking for, or could be laymen who know little terminology, it may be more helpful to provide a set of likely results and let the users choose. Third, the types of entities are different. Each medical search engine may have its own types of entities to tag. There is also one minor difference in the tagging scheme between our problem and the CE task in 2013 ShARe/CLEF - We limit our scope to dealing with entities of consecutive words and not disjoint entities . We use only Beginning, Inside, Outside (BIO) tags. Given the differences listed above, we need to customize a framework consisting of an entity tagging and term matching component for our CE problem. Related Work An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section. Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al. BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model. Lample et al. BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF. Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7. Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks. One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$. More sophisticated word embedding technique emerged since Word2vec. It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10. Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks. In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain. We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results. Tang et al. BIBREF14 provided straightforward algorithm for term matching. The algorithm starts with finding candidate terms that contain ALL the entity words, with term frequency - inverse document frequency (tf-idf) weighting. Then the candidates are ranked based on the pairwise cosine distance between the word embeddings of the candidates and the entity. Framework We adopt the tagging - encoding pipeline framework from the CE task in 2013 ShARe/CLEF. We first tag the clinical entities in the user query and then select relevant terms from a glossary in dermatology to match the entities. Framework ::: Entity Tagging We use the same BiLSTM-CRF model proposed by Huang et al. BIBREF2. An illustration of the architecture is shown in Figure FIGREF6 . Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,..., w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation. We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$. The notation and the procedure described here can be adopted for Flair embeddings or other embeddings. Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2, ..., x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM. To simplify notation, we will omit the $\theta $ and write $P$. Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag. Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints. The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence. For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag. A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$. Given a sequence of tags, $Y=(y_1,y_2,...,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by The probability of the sequence of tags is then calculated by a softmax, where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences. During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$. Framework ::: Term Matching The term matching algorithm of Tang et al. BIBREF14 is adopted with some major modifications. First, to identify candidate terms, we use a much looser string search algorithm where we stem the entity words with snowball stemmer and then find candidate terms which contain ANY non-stopword words in the entity. The stemming is mainly used to represent a word in its stem. For example, “legs” becomes “leg”, and “jammed” becomes “jam”. Thus, stemming can provide more tolerance when finding candidates. Similarly, finding candidates using the condition ANY (instead of ALL) also increases the tolerance. However, if the tagged entity contains stopwords such as “in”, “on”, etc., the pool of candidates will naturally grow very large and increase the computation cost for later part, because even completely irrelevant terms may contain stopwords. Therefore, we match based on non-stopwords only. To illustrate the points above, suppose a query is tagged with the entity Ex. 3.1 “severe burns on legs”, and one relevant term is “leg burn”. After stemming, “burns” and “legs” in Ex.UNKREF12 become “burn” and “leg”, respectively, allowing "leg burn" to be considered as a candidate. Although the word “severe” is not in the term “leg burn”, the term is still considered a candidate because we selected using ANY. The stopword “on” is ignored when finding candidate terms so that not every term that contains the word “on” is added to the candidate pool. When a candidate term, $C$, is found in this manner for the tagged entity, $E$, we calculate the semantic similarity score, $s$, between $C$ and $E$ in two steps. In the first step, calculate the maximum similarity score for each word in $C$ as shown in Figure FIGREF10. Given a word in the candidate term, $C_i$ ($1 \le i \le m$, $m$ is the number of words in the candidate term) and a word in the tagged entity, $E_j$.Their similarity score, $s_{ij}$ (shown as the element in the boxed matrix in Figure FIGREF10), is given by where $ELMo(C_i|C)$ and $ELMo(E_j|E)$ are the ELMo embeddings for the word $C_i$ and $E_j$, respectively. The ELMo embeddings have the same dimension for all words when using the same fine-tuned ELMo model. Thus, we can use a distance function (e.g., the cosine distance), denoted $d(\cdot )$ in equation DISPLAY_FORM13, to compute the semantic similarity between words. In step 2, we calculate the candidate-entity relevance score (similarity) using the formula where $s_c$ is a score threshold, and $\mathbb {I} \lbrace max(\vec{S_i}) > s_c\rbrace $ is an indicator function that equals 1 if $max(\vec{S_i}) > s_c$ or equals 0 if not. In equation DISPLAY_FORM14 we define a metric that measures “information coverage” of the candidate terms with respect to a tagged entity. If the constituent words of a candidate term are relevant to the constituent words in the tagged entity, then the candidate term offers more information coverage. Intuitively, the more relevant words present in the candidate term, the more relevant the candidate is to the tagged entity. The purpose of the cutoff, $s_c$, is to screen the $(C_i,E_j)$ word pairs that are dissimilar, so that they do not contribute to information coverage. One can adjust the strictness of the entity - terminology matching by adjusting $s_c$. The higher we set $s_c$, the fewer candidate terms will be selected for a tagged entity. A normalization factor, $\frac{1}{m}$, is added to give preference to more concise candidate terms given the same amount of information coverage. We need to create an extra stopword list to include words such as “configuration” and “color”, and exclude these words from the word count for a candidate term. This is because the terms associated with the description of color or configuration usually have the word “color” or “configuration” in them. On the other hand, a user query normally does not contain such words. For example, a tagged entity in a user query could be “round yellow patches”, for which the relevant terminologies include “round configuration” and “yellow color”. Since we applied a normalization factor, $\frac{1}{m}$, to the relevance score, the word “color” and “configuration” would lower the relevance score because they do not have a counterpart in the tagged entity. Therefore, we need to exclude them from word count. Once the process is complete, calculate $s(C,E)$ for all candidate terms and then we can apply a threshold on all $s(C,E)$ to ignore candidate terms with low information coverage. Finally, rank the terms by their $s(C,E)$ and return the ranked list as the results. Experiments ::: Data Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes. The synthesized user queries are generated using the aforementioned dermatology glossary. Tagged sentences are extracted from the clinical notes. Sentences with no clinical entity present are ignored. 22,489 tagged sentences are extracted from the clinical notes. We will refer to these tagged sentences interchangeably as the i2b2 data. The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1. The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order. When combining the clinical terms, we attach the BIO tags to their constituent words. The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio. Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively. This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same. The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example). Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary. It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities. On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data. Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries. To make the hybrid training data, we need to unify the tags. Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets. Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA. But this would be an infeasible job as we need experts to manually do that. An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags. Experiments ::: Setup To show the effects of using the hybrid training data, we trained two models of the same architecture and hyperparameters. One model was trained on the hybrid data and will be referred to as hybrid NER model. The other model was trained on clinical notes only and will be referred to as i2b2 NER model. We evaluated the performance of the NER models by micro-F1 score on the test set of both the synthesized queries and the i2b2 data. We used the BiLSTM-CRF implementation provided by the flair package BIBREF16. We set the hidden size value to be 256 in the LSTM structure and left everything else at default values for the SequenceTagger model on flair. For word embeddings, we used the ELMo embeddings fine-tuned on PubMed articles and flair embeddings BIBREF13 trained on $5\%$ of PubMed abstracts , respectively. We trained models for 10 epochs and experimented with different learning rate, mini batch size, and dropouts. We ran hyperparameter optimization tests to find the best combination. $S_c$ is set to be 0.6 in our experiment. Experiments ::: Hyperparameter Tuning We defined the following hyperparameter search space: embeddings: [“ELMo on pubmed”, “stacked flair on pubmed”], hidden_size: [128, 256], learning_rate: [0.05, 0.1], mini_batch_size: [32, 64, 128]. The hyperparameter optimization was performed using Hyperopt . Three evaluations were run for each combination of hyperparameters. Each ran for 10 epochs. Then the results were averaged to give the performance for that particular combination of hyperparameters. Experiments ::: Results From the hyperparameter tuning we found that the best combination was embeddings: “ELMo on pubmed”, hidden_size: 256, learning_rate: 0.05, mini_batch_size: 32. With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0.995$ on synthesized queries and $0.948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0.441$ on synthesized queries and $0.927$ on clinical notes (See Table TABREF23). Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance. Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24. For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy". A list of terms from our glossary was matched to each entity. In real world application, the lists of terms will be presented to the user as the retrieval results to their queries. Discussion In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24). Only on a few user queries the tagging was controversial. For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”. The entity “extremities” was missing. The segmentation was also not correct. A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”. The tagging could be further improved by synthesizing more realistic user queries. Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries. For example, there could be two clinical entities that often co-occur or never co-occur in a user query. But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing. The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth. When ground truth becomes available, we will be able to evaluate our framework more thoroughly. Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain. We could experiment with BioBERT embeddings in the future. We could also include query expansion technique for term matching. When finding candidate terms for an entity, our first step was still based on string matching. Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities. Conclusion In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines. By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries. We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences. Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines. Acknowledgment This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system. The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project. In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework.
where did they obtain the annotated clinical notes from?
4477bb513d56e57732fba126944073d414d1f75f
where Ġdid Ġthey Ġobtain Ġthe Ġannot ated Ġclinical Ġnotes Ġfrom ?
clinical Ġnotes Ġfrom Ġthe ĠCE Ġtask Ġin Ġ2010 Ġi 2 b 2 / VA
<s> where Ġdid Ġthey Ġobtain Ġthe Ġannot ated Ġclinical Ġnotes Ġfrom ? </s> Introduction </s> Medical Ġsearch Ġengines Ġare Ġan Ġessential Ġcomponent Ġfor Ġmany Ġonline Ġmedical Ġapplications , Ġsuch Ġas Ġonline Ġdiagnosis Ġsystems Ġand Ġmedical Ġdocument Ġdatabases . ĠA Ġtypical Ġonline Ġdiagnosis Ġsystem , Ġfor Ġinstance , Ġrelies Ġon Ġa Ġmedical Ġsearch Ġengine . ĠThe Ġsearch Ġengine Ġtakes Ġas Ġinput Ġa Ġuser Ġquery Ġthat Ġdescribes Ġsome Ġsymptoms Ġand Ġthen Ġoutputs Ġclinical Ġconcept Ġentries Ġthat Ġprovide Ġrelevant Ġinformation Ġto Ġassist Ġin Ġdiagn osing Ġthe Ġproblem . ĠOne Ġchallenge Ġmedical Ġsearch Ġengines Ġface Ġis Ġthe Ġsegment ation Ġof Ġindividual Ġclinical Ġentities . ĠWhen Ġa Ġuser Ġquery Ġconsists Ġof Ġmultiple Ġclinical Ġentities , Ġa Ġsearch Ġengine Ġwould Ġoften Ġfail Ġto Ġrecognize Ġthem Ġas Ġseparate Ġentities . ĠFor Ġexample , Ġthe Ġuser Ġquery ĠâĢ ľ f ever Ġjoint Ġpain Ġweight Ġloss Ġheadache âĢ Ŀ Ġcontains Ġfour Ġseparate Ġclinical Ġentities : ĠâĢ ľ f ever âĢ Ŀ , ĠâĢ ľ j oint Ġpain âĢ Ŀ , ĠâĢ ľ weight Ġloss âĢ Ŀ , Ġand ĠâĢ ľ head ache âĢ Ŀ . ĠBut Ġwhen Ġthe Ġsearch Ġengine Ġdoes Ġnot Ġrecognize Ġthem Ġas Ġseparate Ġentities Ġand Ġproceeds Ġto Ġretrieve Ġresults Ġfor Ġeach Ġword Ġin Ġthe Ġquery , Ġit Ġmay Ġfind Ġ" pain " Ġin Ġbody Ġlocations Ġother Ġthan Ġ" j oint Ġpain ", Ġor Ġit Ġmay Ġmiss Ġ" head ache " Ġaltogether , Ġfor Ġexample . ĠSome Ġsearch Ġengines Ġallow Ġthe Ġusers Ġto Ġenter Ġa Ġsingle Ġclinical Ġconcept Ġby Ġselecting Ġfrom Ġan Ġauto - com pletion Ġpick Ġlist . ĠBut Ġthis Ġcould Ġalso Ġresult Ġin Ġretrieving Ġinaccurate Ġor Ġpartial Ġresults Ġand Ġlead Ġto Ġpoor Ġuser Ġexperience . </s> We Ġwant Ġto Ġimprove Ġthe Ġmedical Ġsearch Ġengine Ġso Ġthat Ġit Ġcan Ġaccurately Ġretrieve Ġall Ġthe Ġrelevant Ġclinical Ġconcepts Ġmentioned Ġin Ġa Ġuser Ġquery , Ġwhere Ġrelevant Ġclinical Ġconcepts Ġare Ġdefined Ġwith Ġrespect Ġto Ġthe Ġtermin ologies Ġthe Ġsearch Ġengine Ġuses . ĠThe Ġproblem Ġof Ġextracting Ġclinical Ġconcept Ġmentions Ġfrom Ġa Ġuser Ġquery Ġcan Ġbe Ġseen Ġas Ġa Ġvariant Ġof Ġthe ĠConcept ĠExt raction Ġ( CE ) Ġtask Ġin Ġthe Ġfrequently - c ited ĠN LP Ġchallenges Ġin Ġhealthcare , Ġsuch Ġas Ġ2010 Ġi 2 b 2 / VA ĠB IB REF 0 Ġand Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 ĠB IB REF 1 . ĠBoth ĠCE Ġtasks Ġin Ġ2010 Ġi 2 b 2 / VA Ġand Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 Ġask Ġthe Ġparticipants Ġto Ġdesign Ġan Ġalgorithm Ġto Ġtag Ġa Ġset Ġof Ġpred efined Ġentities Ġof Ġinterest Ġin Ġclinical Ġnotes . ĠThese Ġentity Ġtagging Ġtasks Ġare Ġalso Ġknown Ġas Ġclinical ĠNamed ĠEntity ĠRecogn ition Ġ( NER ). ĠFor Ġexample , Ġthe ĠCE Ġtask Ġin Ġ2010 Ġi 2 b 2 / VA Ġdefines Ġthree Ġtypes Ġof Ġentities : ĠâĢ ľ problem âĢ Ŀ , ĠâĢ ľ treatment âĢ Ŀ , Ġand ĠâĢ ľ test âĢ Ŀ . ĠThe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġdefines Ġvarious Ġtypes Ġof Ġdisorder Ġsuch Ġas ĠâĢ ľ in jury Ġor Ġpoisoning âĢ Ŀ , Ġ" d ise ase Ġor Ġsyndrome âĢ Ŀ , Ġetc . ĠIn Ġaddition Ġto Ġtagging , Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġhas Ġan Ġencoding Ġcomponent Ġwhich Ġrequires Ġselecting Ġone Ġand Ġonly Ġone ĠConcept ĠUnique ĠIdent ifier Ġ( C UI ) Ġfrom ĠSystem at ized ĠN omen cl ature ĠOf ĠMedicine ĠClinical ĠTerms Ġ( SN OM ED - CT ) Ġfor Ġeach Ġdisorder Ġentity Ġtagged . ĠOur Ġproblem , Ġsimilar Ġto Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F , Ġalso Ġcontains Ġtwo Ġsub - pro blems : Ġtagging Ġmentions Ġof Ġentities Ġof Ġinterest Ġ( entity Ġtagging ), Ġand Ġselecting Ġappropriate Ġterms Ġfrom Ġa Ġgloss ary Ġto Ġmatch Ġthe Ġmentions Ġ( term Ġmatching ). ĠHowever , Ġseveral Ġmajor Ġdifferences Ġexist . ĠFirst , Ġcompared Ġto Ġclinical Ġnotes , Ġthe Ġuser Ġqueries Ġare Ġmuch Ġshorter , Ġless Ġtechnical , Ġand Ġoften Ġless Ġcoherent . ĠSecond , Ġinstead Ġof Ġencoding , Ġwe Ġare Ġdealing Ġwith Ġterm Ġmatching Ġwhere Ġwe Ġrank Ġa Ġfew Ġbest Ġterms Ġthat Ġmatch Ġan Ġentity , Ġinstead Ġof Ġselecting Ġonly Ġone . ĠThis Ġis Ġbecause Ġthe Ġusers Ġwho Ġtype Ġthe Ġqueries Ġmay Ġnot Ġhave Ġa Ġclear Ġidea Ġabout Ġwhat Ġthey Ġare Ġlooking Ġfor , Ġor Ġcould Ġbe Ġlay men Ġwho Ġknow Ġlittle Ġterminology , Ġit Ġmay Ġbe Ġmore Ġhelpful Ġto Ġprovide Ġa Ġset Ġof Ġlikely Ġresults Ġand Ġlet Ġthe Ġusers Ġchoose . ĠThird , Ġthe Ġtypes Ġof Ġentities Ġare Ġdifferent . ĠEach Ġmedical Ġsearch Ġengine Ġmay Ġhave Ġits Ġown Ġtypes Ġof Ġentities Ġto Ġtag . ĠThere Ġis Ġalso Ġone Ġminor Ġdifference Ġin Ġthe Ġtagging Ġscheme Ġbetween Ġour Ġproblem Ġand Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F Ġ- ĠWe Ġlimit Ġour Ġscope Ġto Ġdealing Ġwith Ġentities Ġof Ġconsecutive Ġwords Ġand Ġnot Ġdis j oint Ġentities Ġ. ĠWe Ġuse Ġonly ĠBeginning , ĠInside , ĠOutside Ġ( BI O ) Ġtags . ĠGiven Ġthe Ġdifferences Ġlisted Ġabove , Ġwe Ġneed Ġto Ġcustomize Ġa Ġframework Ġconsisting Ġof Ġan Ġentity Ġtagging Ġand Ġterm Ġmatching Ġcomponent Ġfor Ġour ĠCE Ġproblem . </s> Related ĠWork </s> An Ġeffective Ġmodel Ġthat Ġhas Ġbeen Ġcommonly Ġused Ġfor ĠN ER Ġproblem Ġis Ġa ĠBi - direction al ĠL ST M Ġwith Ġa ĠCond itional ĠRandom ĠField Ġ( CR F ) Ġon Ġthe Ġtop Ġlayer Ġ( Bi L ST M - CR F ), Ġwhich Ġis Ġdescribed Ġin Ġthe Ġnext Ġsection . ĠComb ining ĠL ST M âĢ Ļ s Ġpower Ġof Ġrepresenting Ġrelations Ġbetween Ġwords Ġand ĠCR F âĢ Ļ s Ġcapability Ġof Ġaccounting Ġfor Ġtag Ġsequence Ġconstraints , ĠHuang Ġet Ġal . ĠB IB REF 2 Ġproposed Ġthe ĠBi L ST M - CR F Ġmodel Ġand Ġused Ġhand crafted Ġword Ġfeatures Ġas Ġthe Ġinput Ġto Ġthe Ġmodel . ĠL ample Ġet Ġal . ĠB IB REF 3 Ġused Ġa Ġcombination Ġof Ġcharacter - level Ġand Ġword - level Ġword Ġembed d ings Ġas Ġthe Ġinput Ġto ĠBi L ST M - CR F . ĠSince Ġthen , Ġsimilar Ġmodels Ġwith Ġvariation Ġin Ġtypes Ġof Ġword Ġembed d ings Ġhave Ġbeen Ġused Ġextensively Ġfor Ġclinical ĠCE Ġtasks Ġand Ġproduced Ġstate - of - the - art Ġresults ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 . ĠWord Ġembed d ings Ġhave Ġbecome Ġthe Ġcornerstone Ġof Ġthe Ġneural Ġmodels Ġin ĠN LP Ġsince Ġthe Ġfamous ĠWord 2 vec ĠB IB REF 8 Ġmodel Ġdemonstrated Ġits Ġpower Ġin Ġword Ġanalogy Ġtasks . ĠOne Ġwell - known Ġexample Ġis Ġthat Ġafter Ġtraining ĠWord 2 vec Ġon Ġa Ġlarge Ġamount Ġof Ġnews Ġdata , Ġwe Ġcan Ġget Ġword Ġrelations Ġsuch Ġas Ġ$ vector ( ^ {\ prime Ġ} king ^ {\ prime Ġ}) Ġ- Ġvector ( ^ {\ prime Ġ} que en ^ {\ prime Ġ}) Ġ+ Ġvector ( ^ {\ prime Ġ} woman ^ {\ prime Ġ}) Ġ\ app rox Ġvector ( ^ {\ prime Ġ} man ^ {\ prime Ġ}) $. ĠMore Ġsophisticated Ġword Ġembed ding Ġtechnique Ġemerged Ġsince ĠWord 2 vec . ĠIt Ġhas Ġbeen Ġshown Ġempir ically Ġthat Ġbetter Ġquality Ġin Ġword Ġembed d ings Ġleads Ġto Ġbetter Ġperformance Ġin Ġmany Ġdownstream ĠN LP Ġincluding Ġentity Ġtagging ĠB IB REF 9 , ĠB IB REF 10 . ĠRecently , Ġcontextual ized Ġword Ġembed d ings Ġgenerated Ġby Ġdeep Ġlearning Ġmodels , Ġsuch Ġas ĠEL Mo ĠB IB REF 11 , ĠB ERT ĠB IB REF 12 , Ġand ĠFl air ĠB IB REF 13 , Ġhave Ġbeen Ġshown Ġto Ġbe Ġmore Ġeffective Ġin Ġvarious ĠN LP Ġtasks . ĠIn Ġour Ġproject , Ġwe Ġmake Ġuse Ġof Ġa Ġfine - tun ed ĠEL Mo Ġmodel Ġand Ġa Ġfine - tun ed ĠFl air Ġmodel Ġin Ġthe Ġmedical Ġdomain . ĠWe Ġexperiment Ġwith Ġthe Ġword Ġembed d ings Ġfrom Ġthe Ġtwo Ġfine - tun ed Ġmodels Ġas Ġthe Ġinput Ġto Ġthe ĠBi L ST M - CR F Ġmodel Ġseparately Ġand Ġcompare Ġthe Ġresults . </s> Tang Ġet Ġal . ĠB IB REF 14 Ġprovided Ġstraightforward Ġalgorithm Ġfor Ġterm Ġmatching . ĠThe Ġalgorithm Ġstarts Ġwith Ġfinding Ġcandidate Ġterms Ġthat Ġcontain ĠALL Ġthe Ġentity Ġwords , Ġwith Ġterm Ġfrequency Ġ- Ġinverse Ġdocument Ġfrequency Ġ( tf - id f ) Ġweight ing . ĠThen Ġthe Ġcandidates Ġare Ġranked Ġbased Ġon Ġthe Ġpair wise Ġcos ine Ġdistance Ġbetween Ġthe Ġword Ġembed d ings Ġof Ġthe Ġcandidates Ġand Ġthe Ġentity . </s> Fram ework </s> We Ġadopt Ġthe Ġtagging Ġ- Ġencoding Ġpipeline Ġframework Ġfrom Ġthe ĠCE Ġtask Ġin Ġ2013 ĠSh AR e / CLE F . ĠWe Ġfirst Ġtag Ġthe Ġclinical Ġentities Ġin Ġthe Ġuser Ġquery Ġand Ġthen Ġselect Ġrelevant Ġterms Ġfrom Ġa Ġgloss ary Ġin Ġdermat ology Ġto Ġmatch Ġthe Ġentities . </s> Fram ework Ġ: :: ĠEntity ĠT agging </s> We Ġuse Ġthe Ġsame ĠBi L ST M - CR F Ġmodel Ġproposed Ġby ĠHuang Ġet Ġal . ĠB IB REF 2 . ĠAn Ġillustration Ġof Ġthe Ġarchitecture Ġis Ġshown Ġin ĠFigure ĠFIG REF 6 Ġ. ĠGiven Ġa Ġsequence Ġ( or Ġsentence ) Ġof Ġn Ġtokens , Ġ$ r Ġ= Ġ( w _ 1 , Ġw _ 2 ,... , Ġw _ n ) $, Ġwe Ġuse Ġa Ġfine - tun ed ĠEL Mo Ġmodel Ġto Ġgenerate Ġcontextual Ġword Ġembed d ings Ġfor Ġall Ġthe Ġtokens Ġin Ġthe Ġsentence , Ġwhere Ġa Ġtoken Ġrefers Ġto Ġa Ġword Ġor Ġpunct uation . ĠWe Ġdenote Ġthe ĠEL Mo Ġembed ding , Ġ$ x $, Ġfor Ġa Ġtoken Ġ$ w $ Ġin Ġthe Ġsentence Ġ$ r $ Ġby Ġ$ x Ġ= ĠEL Mo ( w | r ) $. ĠThe Ġnotation Ġand Ġthe Ġprocedure Ġdescribed Ġhere Ġcan Ġbe Ġadopted Ġfor ĠFl air Ġembed d ings Ġor Ġother Ġembed d ings . ĠNow , Ġgiven Ġa Ġsequence Ġof Ġtokens Ġin ĠEL Mo Ġembed d ings , Ġ$ X Ġ= ( x _ 1 , Ġx _ 2 , Ġ... , Ġx _ n ) $, Ġthe ĠBi L ST M Ġlayer Ġgenerates Ġa Ġmatrix Ġof Ġscores , Ġ$ P (\ the ta Ġ) $ Ġof Ġsize Ġ$ n Ġ\ times Ġk $, Ġwhere Ġ$ k $ Ġis Ġthe Ġnumber Ġof Ġtag Ġtypes , Ġand Ġ$\ the ta Ġ$ Ġis Ġthe Ġparameters Ġof Ġthe ĠBi L ST M . ĠTo Ġsimplify Ġnotation , Ġwe Ġwill Ġomit Ġthe Ġ$\ the ta Ġ$ Ġand Ġwrite Ġ$ P $. ĠThen , Ġ$ P _{ i , j } $ Ġdenotes Ġthe Ġscore Ġof Ġthe Ġtoken , Ġ$ x _ i $, Ġbeing Ġassigned Ġto Ġthe Ġ$ j $ th Ġtag . ĠSince Ġcertain Ġconstraints Ġmay Ġexist Ġin Ġthe Ġtransition Ġbetween Ġtags , Ġan Ġ" O " Ġtag Ġshould Ġnot Ġbe Ġfollowed Ġby Ġan Ġ" I " Ġtag , Ġfor Ġexample , Ġa Ġtransition Ġmatrix , Ġ$ A $, Ġof Ġdimension Ġ$( k + 2 )\ times Ġ( k + 2 ) $, Ġis Ġinitialized Ġto Ġmodel Ġthe Ġconstraints . ĠThe Ġlearn able Ġparameters , Ġ$ A _{ i , j } $, Ġrepresent Ġthe Ġprobability Ġof Ġthe Ġ$ j $ th Ġtag Ġfollows Ġthe Ġ$ i $ th Ġtag Ġin Ġa Ġsequence . ĠFor Ġexample , Ġif Ġwe Ġindex Ġthe Ġtags Ġby : Ġ1 : âĢ ľ B âĢ Ŀ , Ġ2 : âĢ ľ I âĢ Ŀ , Ġand Ġ3 : âĢ ľ O âĢ Ŀ , Ġthen Ġ$ A _{ 1 , 3 } $ Ġwould Ġbe Ġthe Ġprobability Ġthat Ġan ĠâĢ ľ O âĢ Ŀ Ġtag Ġfollows Ġa ĠâĢ ľ B âĢ Ŀ Ġtag . ĠA Ġbeginning Ġtransition Ġand Ġan Ġend Ġtransition Ġare Ġinserted Ġin Ġ$ A $ Ġand Ġhence Ġ$ A $ Ġis Ġof Ġdimension Ġ$( k + 2 )\ times Ġ( k + 2 ) $. </s> Given Ġa Ġsequence Ġof Ġtags , Ġ$ Y =( y _ 1 , y _ 2 ,... , y _ n ) $, Ġwhere Ġeach Ġ$ y _ i $, Ġ$ 1 \ le Ġi Ġ\ le Ġn $, Ġcorresponds Ġto Ġan Ġindex Ġof Ġthe Ġtags , Ġthe Ġscore Ġof Ġthe Ġsequence Ġis Ġthen Ġgiven Ġby </s> The Ġprobability Ġof Ġthe Ġsequence Ġof Ġtags Ġis Ġthen Ġcalculated Ġby Ġa Ġsoft max , </s> where Ġ$\ l brace ĠY _ x \ r brace Ġ$ Ġdenotes Ġthe Ġset Ġof Ġall Ġpossible Ġtag Ġsequences . ĠDuring Ġtraining , Ġthe Ġobjective Ġfunction Ġis Ġto Ġmaximize Ġ$\ log Ġ( P ( Y | X )) $ Ġby Ġadjusting Ġ$ A $ Ġand Ġ$ P $. </s> Fram ework Ġ: :: ĠTerm ĠMatch ing </s> The Ġterm Ġmatching Ġalgorithm Ġof ĠTang Ġet Ġal . ĠB IB REF 14 Ġis Ġadopted Ġwith Ġsome Ġmajor Ġmodifications . ĠFirst , Ġto Ġidentify Ġcandidate Ġterms , Ġwe Ġuse Ġa Ġmuch Ġlo oser Ġstring Ġsearch Ġalgorithm Ġwhere Ġwe Ġstem Ġthe Ġentity Ġwords Ġwith Ġsnowball Ġstem mer Ġand Ġthen Ġfind Ġcandidate Ġterms Ġwhich Ġcontain ĠANY Ġnon - stop word Ġwords Ġin Ġthe Ġentity . ĠThe Ġstemming Ġis Ġmainly Ġused Ġto Ġrepresent Ġa Ġword Ġin Ġits Ġstem . ĠFor Ġexample , ĠâĢ ľ leg s âĢ Ŀ Ġbecomes ĠâĢ ľ leg âĢ Ŀ , Ġand ĠâĢ ľ j ammed âĢ Ŀ Ġbecomes ĠâĢ ľ jam âĢ Ŀ . ĠThus , Ġstemming Ġcan Ġprovide Ġmore Ġtolerance Ġwhen Ġfinding Ġcandidates . ĠSimilarly , Ġfinding Ġcandidates Ġusing Ġthe Ġcondition ĠANY Ġ( instead Ġof ĠALL ) Ġalso Ġincreases Ġthe Ġtolerance . ĠHowever , Ġif Ġthe Ġtagged Ġentity Ġcontains Ġstop words Ġsuch Ġas ĠâĢ ľ in âĢ Ŀ , ĠâĢ ľ on âĢ Ŀ , Ġetc ., Ġthe Ġpool Ġof Ġcandidates Ġwill Ġnaturally Ġgrow Ġvery Ġlarge Ġand Ġincrease Ġthe Ġcomputation Ġcost Ġfor Ġlater Ġpart , Ġbecause Ġeven Ġcompletely Ġirrelevant Ġterms Ġmay Ġcontain Ġstop words . ĠTherefore , Ġwe Ġmatch Ġbased Ġon Ġnon - stop words Ġonly . ĠTo Ġillustrate Ġthe Ġpoints Ġabove , Ġsuppose Ġa Ġquery Ġis Ġtagged Ġwith Ġthe Ġentity </s> Ex . Ġ3 . 1 ĠâĢ ľ severe Ġburns Ġon Ġlegs âĢ Ŀ , </s> and Ġone Ġrelevant Ġterm Ġis ĠâĢ ľ leg Ġburn âĢ Ŀ . ĠAfter Ġstemming , ĠâĢ ľ burn s âĢ Ŀ Ġand ĠâĢ ľ leg s âĢ Ŀ Ġin ĠEx . UN K REF 12 Ġbecome ĠâĢ ľ burn âĢ Ŀ Ġand ĠâĢ ľ leg âĢ Ŀ , Ġrespectively , Ġallowing Ġ" leg Ġburn " Ġto Ġbe Ġconsidered Ġas Ġa Ġcandidate . ĠAlthough Ġthe Ġword ĠâĢ ľ severe âĢ Ŀ Ġis Ġnot Ġin Ġthe Ġterm ĠâĢ ľ leg Ġburn âĢ Ŀ , Ġthe Ġterm Ġis Ġstill Ġconsidered Ġa Ġcandidate Ġbecause Ġwe Ġselected Ġusing ĠANY . ĠThe Ġstop word ĠâĢ ľ on âĢ Ŀ Ġis Ġignored Ġwhen Ġfinding Ġcandidate Ġterms Ġso Ġthat Ġnot Ġevery Ġterm Ġthat Ġcontains Ġthe Ġword ĠâĢ ľ on âĢ Ŀ Ġis Ġadded Ġto Ġthe Ġcandidate Ġpool . ĠWhen Ġa Ġcandidate Ġterm , Ġ$ C $, Ġis Ġfound Ġin Ġthis Ġmanner Ġfor Ġthe Ġtagged Ġentity , Ġ$ E $, Ġwe Ġcalculate Ġthe Ġsemantic Ġsimilarity Ġscore , Ġ$ s $, Ġbetween Ġ$ C $ Ġand Ġ$ E $ Ġin Ġtwo Ġsteps . ĠIn Ġthe Ġfirst Ġstep , Ġcalculate Ġthe Ġmaximum Ġsimilarity Ġscore Ġfor Ġeach Ġword Ġin Ġ$ C $ Ġas Ġshown Ġin ĠFigure ĠFIG REF 10 . ĠGiven Ġa Ġword Ġin Ġthe Ġcandidate Ġterm , Ġ$ C _ i $ Ġ($ 1 Ġ\ le Ġi Ġ\ le Ġm $, Ġ$ m $ Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġcandidate Ġterm ) Ġand Ġa Ġword Ġin Ġthe Ġtagged Ġentity , Ġ$ E _ j $. Their Ġsimilarity Ġscore , Ġ$ s _{ ij } $ Ġ( shown Ġas Ġthe Ġelement Ġin Ġthe Ġboxed Ġmatrix Ġin ĠFigure ĠFIG REF 10 ), Ġis Ġgiven Ġby </s> where Ġ$ EL Mo ( C _ i | C ) $ Ġand Ġ$ EL Mo ( E _ j | E ) $ Ġare Ġthe ĠEL Mo Ġembed d ings Ġfor Ġthe Ġword Ġ$ C _ i $ Ġand Ġ$ E _ j $, Ġrespectively . ĠThe ĠEL Mo Ġembed d ings Ġhave Ġthe Ġsame Ġdimension Ġfor Ġall Ġwords Ġwhen Ġusing Ġthe Ġsame Ġfine - tun ed ĠEL Mo Ġmodel . ĠThus , Ġwe Ġcan Ġuse Ġa Ġdistance Ġfunction Ġ( e . g ., Ġthe Ġcos ine Ġdistance ), Ġden oted Ġ$ d (\ cd ot Ġ) $ Ġin Ġequation ĠDIS PLAY _ FORM 13 , Ġto Ġcompute Ġthe Ġsemantic Ġsimilarity Ġbetween Ġwords . ĠIn Ġstep Ġ2 , Ġwe Ġcalculate Ġthe Ġcandidate - entity Ġrelevance Ġscore Ġ( similar ity ) Ġusing Ġthe Ġformula </s> where Ġ$ s _ c $ Ġis Ġa Ġscore Ġthreshold , Ġand Ġ$\ math bb Ġ{ I } Ġ\ l brace Ġmax (\ vec { S _ i }) Ġ> Ġs _ c \ r brace Ġ$ Ġis Ġan Ġindicator Ġfunction Ġthat Ġequals Ġ1 Ġif Ġ$ max (\ vec { S _ i }) Ġ> Ġs _ c $ Ġor Ġequals Ġ0 Ġif Ġnot . ĠIn Ġequation ĠDIS PLAY _ FORM 14 Ġwe Ġdefine Ġa Ġmetric Ġthat Ġmeasures ĠâĢ ľ information Ġcoverage âĢ Ŀ Ġof Ġthe Ġcandidate Ġterms Ġwith Ġrespect Ġto Ġa Ġtagged Ġentity . ĠIf Ġthe Ġconstituent Ġwords Ġof Ġa Ġcandidate Ġterm Ġare Ġrelevant Ġto Ġthe Ġconstituent Ġwords Ġin Ġthe Ġtagged Ġentity , Ġthen Ġthe Ġcandidate Ġterm Ġoffers Ġmore Ġinformation Ġcoverage . ĠInt uitive ly , Ġthe Ġmore Ġrelevant Ġwords Ġpresent Ġin Ġthe Ġcandidate Ġterm , Ġthe Ġmore Ġrelevant Ġthe Ġcandidate Ġis Ġto Ġthe Ġtagged Ġentity . ĠThe Ġpurpose Ġof Ġthe Ġcutoff , Ġ$ s _ c $, Ġis Ġto Ġscreen Ġthe Ġ$( C _ i , E _ j ) $ Ġword Ġpairs Ġthat Ġare Ġdiss imilar , Ġso Ġthat Ġthey Ġdo Ġnot Ġcontribute Ġto Ġinformation Ġcoverage . ĠOne Ġcan Ġadjust Ġthe Ġstrict ness Ġof Ġthe Ġentity Ġ- Ġterminology Ġmatching Ġby Ġadjusting Ġ$ s _ c $. ĠThe Ġhigher Ġwe Ġset Ġ$ s _ c $, Ġthe Ġfewer Ġcandidate Ġterms Ġwill Ġbe Ġselected Ġfor Ġa Ġtagged Ġentity . ĠA Ġnormal ization Ġfactor , Ġ$\ frac { 1 }{ m } $, Ġis Ġadded Ġto Ġgive Ġpreference Ġto Ġmore Ġconcise Ġcandidate Ġterms Ġgiven Ġthe Ġsame Ġamount Ġof Ġinformation Ġcoverage . </s> We Ġneed Ġto Ġcreate Ġan Ġextra Ġstop word Ġlist Ġto Ġinclude Ġwords Ġsuch Ġas ĠâĢ ľ config uration âĢ Ŀ Ġand ĠâĢ ľ color âĢ Ŀ , Ġand Ġexclude Ġthese Ġwords Ġfrom Ġthe Ġword Ġcount Ġfor Ġa Ġcandidate Ġterm . ĠThis Ġis Ġbecause Ġthe Ġterms Ġassociated Ġwith Ġthe Ġdescription Ġof Ġcolor Ġor Ġconfiguration Ġusually Ġhave Ġthe Ġword ĠâĢ ľ color âĢ Ŀ Ġor ĠâĢ ľ config uration âĢ Ŀ Ġin Ġthem . ĠOn Ġthe Ġother Ġhand , Ġa Ġuser Ġquery Ġnormally Ġdoes Ġnot Ġcontain Ġsuch Ġwords . ĠFor Ġexample , Ġa Ġtagged Ġentity Ġin Ġa Ġuser Ġquery Ġcould Ġbe ĠâĢ ľ round Ġyellow Ġpatches âĢ Ŀ , Ġfor Ġwhich Ġthe Ġrelevant Ġtermin ologies Ġinclude ĠâĢ ľ round Ġconfiguration âĢ Ŀ Ġand ĠâĢ ľ yellow Ġcolor âĢ Ŀ . ĠSince Ġwe Ġapplied Ġa Ġnormal ization Ġfactor , Ġ$\ frac { 1 }{ m } $, Ġto Ġthe Ġrelevance Ġscore , Ġthe Ġword ĠâĢ ľ color âĢ Ŀ Ġand ĠâĢ ľ config uration âĢ Ŀ Ġwould Ġlower Ġthe Ġrelevance Ġscore Ġbecause Ġthey Ġdo Ġnot Ġhave Ġa Ġcounterpart Ġin Ġthe Ġtagged Ġentity . ĠTherefore , Ġwe Ġneed Ġto Ġexclude Ġthem Ġfrom Ġword Ġcount . ĠOnce Ġthe Ġprocess Ġis Ġcomplete , Ġcalculate Ġ$ s ( C , E ) $ Ġfor Ġall Ġcandidate Ġterms Ġand Ġthen Ġwe Ġcan Ġapply Ġa Ġthreshold Ġon Ġall Ġ$ s ( C , E ) $ Ġto Ġignore Ġcandidate Ġterms Ġwith Ġlow Ġinformation Ġcoverage . ĠFinally , Ġrank Ġthe Ġterms Ġby Ġtheir Ġ$ s ( C , E ) $ Ġand Ġreturn Ġthe Ġranked Ġlist Ġas Ġthe Ġresults . </s> Exper iments Ġ: :: ĠData </s> Despite Ġthe Ġgreater Ġsimilarity Ġbetween Ġour Ġtask Ġand Ġthe Ġ2013 ĠSh AR e / CLE F ĠTask Ġ1 , Ġwe Ġuse Ġthe Ġclinical Ġnotes Ġfrom Ġthe ĠCE Ġtask Ġin Ġ2010 Ġi 2 b 2 / VA Ġon Ġaccount Ġof Ġ1 ) Ġthe Ġdata Ġfrom Ġ2010 Ġi 2 b 2 / VA Ġbeing Ġeasier Ġto Ġaccess Ġand Ġparse , Ġ2 ) Ġ2013 ĠSh AR e / CLE F Ġcontaining Ġdis j oint Ġentities Ġand Ġhence Ġrequiring Ġmore Ġcomplicated Ġtagging Ġschemes . ĠThe Ġsynthes ized Ġuser Ġqueries Ġare Ġgenerated Ġusing Ġthe Ġaforementioned Ġdermat ology Ġgloss ary . ĠT agged Ġsentences Ġare Ġextracted Ġfrom Ġthe Ġclinical Ġnotes . ĠSent ences Ġwith Ġno Ġclinical Ġentity Ġpresent Ġare Ġignored . Ġ22 , 489 Ġtagged Ġsentences Ġare Ġextracted Ġfrom Ġthe Ġclinical Ġnotes . ĠWe Ġwill Ġrefer Ġto Ġthese Ġtagged Ġsentences Ġinterchange ably Ġas Ġthe Ġi 2 b 2 Ġdata . ĠThe Ġsentences Ġare Ġshuff led Ġand Ġsplit Ġinto Ġtrain / dev / test Ġset Ġwith Ġa Ġratio Ġof Ġ7 : 2 : 1 . ĠThe Ġsynthes ized Ġuser Ġqueries Ġare Ġcomposed Ġby Ġrandomly Ġselecting Ġseveral Ġclinical Ġterms Ġfrom Ġthe Ġdermat ology Ġgloss ary Ġand Ġthen Ġcombining Ġthem Ġin Ġno Ġparticular Ġorder . ĠWhen Ġcombining Ġthe Ġclinical Ġterms , Ġwe Ġattach Ġthe ĠB IO Ġtags Ġto Ġtheir Ġconstituent Ġwords . ĠThe Ġsynthes ized Ġuser Ġqueries Ġ( 13 , 697 Ġin Ġtotal ) Ġare Ġthen Ġsplit Ġinto Ġtrain / dev / test Ġset Ġwith Ġthe Ġsame Ġratio . ĠNext , Ġeach Ġset Ġin Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġcorresponding Ġset Ġin Ġthe Ġsynthes ized Ġquery Ġdata Ġare Ġcombined Ġto Ġform Ġa Ġhybrid Ġtrain / dev / test Ġset , Ġrespectively . ĠThis Ġway Ġwe Ġensure Ġthat Ġin Ġeach Ġhybrid Ġtrain / dev / test Ġset , Ġthe Ġratio Ġbetween Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġsynthes ized Ġquery Ġdata Ġis Ġthe Ġsame . </s> The Ġreason Ġfor Ġcombining Ġthe Ġtwo Ġdata Ġis Ġtheir Ġdrastic Ġstructural Ġdifference Ġ( See Ġfigure ĠFIG REF 16 Ġfor Ġan Ġexample ). ĠPreviously , Ġwhen Ġtrained Ġon Ġthe Ġi 2 b 2 Ġdata Ġonly , Ġthe ĠBi L ST M - CR F Ġmodel Ġwas Ġnot Ġable Ġto Ġsegment Ġclinical Ġentities Ġat Ġthe Ġcorrect Ġboundary . ĠIt Ġwould Ġfail Ġto Ġrecognize Ġthe Ġuser Ġquery Ġin ĠFigure ĠFIG REF 16 ( a ) Ġas Ġfour Ġseparate Ġentities . ĠOn Ġthe Ġother Ġhand , Ġif Ġthe Ġmodel Ġwas Ġtrained Ġsolely Ġon Ġthe Ġsynthes ized Ġuser Ġqueries , Ġwe Ġcould Ġimagine Ġthat Ġit Ġwould Ġfail Ġmiser ably Ġon Ġany Ġqueries Ġthat Ġresemble Ġthe Ġsentence Ġin ĠFigure ĠFIG REF 16 ( b ) Ġbecause Ġthe Ġmodel Ġwould Ġhave Ġnever Ġseen Ġan ĠâĢ ľ O âĢ Ŀ Ġtag Ġin Ġthe Ġtraining Ġdata . ĠTherefore , Ġit Ġis Ġnecessary Ġto Ġuse Ġthe Ġhybrid Ġtraining Ġdata Ġcontaining Ġboth Ġthe Ġi 2 b 2 Ġdata Ġand Ġthe Ġsynthes ized Ġuser Ġqueries . </s> To Ġmake Ġthe Ġhybrid Ġtraining Ġdata , Ġwe Ġneed Ġto Ġun ify Ġthe Ġtags . ĠRecall Ġthat Ġin ĠSection ĠSEC REF 1 Ġwe Ġpoint Ġout Ġthat Ġthe Ġtags Ġare Ġdifferent Ġfor Ġthe Ġdifferent Ġtasks Ġand Ġdatasets . ĠSince Ġwe Ġuse Ġcustom Ġtags Ġfor Ġdermat ology Ġgloss ary Ġin Ġour Ġproblem , Ġwe Ġwould Ġneed Ġto Ġconvert Ġthe Ġtags Ġused Ġin Ġ2010 Ġi 2 b 2 / VA . ĠBut Ġthis Ġwould Ġbe Ġan Ġinf eas ible Ġjob Ġas Ġwe Ġneed Ġexperts Ġto Ġmanually Ġdo Ġthat . ĠAn Ġalternative Ġis Ġto Ġavoid Ġdistinguishing Ġthe Ġtag Ġtypes Ġand Ġlabel Ġall Ġtags Ġunder Ġthe Ġgeneric ĠB IO Ġtags . </s> Exper iments Ġ: :: ĠSetup </s> To Ġshow Ġthe Ġeffects Ġof Ġusing Ġthe Ġhybrid Ġtraining Ġdata , Ġwe Ġtrained Ġtwo Ġmodels Ġof Ġthe Ġsame Ġarchitecture Ġand Ġhyper param eters . ĠOne Ġmodel Ġwas Ġtrained Ġon Ġthe Ġhybrid Ġdata Ġand Ġwill Ġbe Ġreferred Ġto Ġas Ġhybrid ĠN ER Ġmodel . ĠThe Ġother Ġmodel Ġwas Ġtrained Ġon Ġclinical Ġnotes Ġonly Ġand Ġwill Ġbe Ġreferred Ġto Ġas Ġi 2 b 2 ĠN ER Ġmodel . ĠWe Ġevaluated Ġthe Ġperformance Ġof Ġthe ĠN ER Ġmodels Ġby Ġmicro - F 1 Ġscore Ġon Ġthe Ġtest Ġset Ġof Ġboth Ġthe Ġsynthes ized Ġqueries Ġand Ġthe Ġi 2 b 2 Ġdata . </s> We Ġused Ġthe ĠBi L ST M - CR F Ġimplementation Ġprovided Ġby Ġthe Ġflair Ġpackage ĠB IB REF 16 . ĠWe Ġset Ġthe Ġhidden Ġsize Ġvalue Ġto Ġbe Ġ256 Ġin Ġthe ĠL ST M Ġstructure Ġand Ġleft Ġeverything Ġelse Ġat Ġdefault Ġvalues Ġfor Ġthe ĠSequence T agger Ġmodel Ġon Ġflair . ĠFor Ġword Ġembed d ings , Ġwe Ġused Ġthe ĠEL Mo Ġembed d ings Ġfine - tun ed Ġon ĠPubMed Ġarticles Ġand Ġflair Ġembed d ings ĠB IB REF 13 Ġtrained Ġon Ġ$ 5 \ % $ Ġof ĠPubMed Ġabstract s Ġ, Ġrespectively . ĠWe Ġtrained Ġmodels Ġfor Ġ10 Ġepoch s Ġand Ġexperimented Ġwith Ġdifferent Ġlearning Ġrate , Ġmini Ġbatch Ġsize , Ġand Ġdrop outs . ĠWe Ġran Ġhyper param eter Ġoptimization Ġtests Ġto Ġfind Ġthe Ġbest Ġcombination . Ġ$ S _ c $ Ġis Ġset Ġto Ġbe Ġ0 . 6 Ġin Ġour Ġexperiment . </s> Exper iments Ġ: :: ĠHyper param eter ĠTun ing </s> We Ġdefined Ġthe Ġfollowing Ġhyper param eter Ġsearch Ġspace : </s> embed d ings : Ġ[ âĢ ľ EL Mo Ġon Ġpub med âĢ Ŀ , ĠâĢ ľ st acked Ġflair Ġon Ġpub med âĢ Ŀ ], </s> hidden _ size : Ġ[ 128 , Ġ256 ], </s> learning _ rate : Ġ[ 0 . 05 , Ġ0 . 1 ], </s> mini _ batch _ size : Ġ[ 32 , Ġ64 , Ġ128 ]. </s> The Ġhyper param eter Ġoptimization Ġwas Ġperformed Ġusing ĠHyper opt Ġ. ĠThree Ġevaluations Ġwere Ġrun Ġfor Ġeach Ġcombination Ġof Ġhyper param eters . ĠEach Ġran Ġfor Ġ10 Ġepoch s . ĠThen Ġthe Ġresults Ġwere Ġaveraged Ġto Ġgive Ġthe Ġperformance Ġfor Ġthat Ġparticular Ġcombination Ġof Ġhyper param eters . </s> Exper iments Ġ: :: ĠResults </s> From Ġthe Ġhyper param eter Ġtuning Ġwe Ġfound Ġthat Ġthe Ġbest Ġcombination Ġwas </s> embed d ings : ĠâĢ ľ EL Mo Ġon Ġpub med âĢ Ŀ , </s> hidden _ size : Ġ256 , </s> learning _ rate : Ġ0 . 05 , </s> mini _ batch _ size : Ġ32 . </s> With Ġthe Ġabove Ġhyper param eter Ġsetting , Ġthe Ġhybrid ĠN ER Ġmodel Ġachieved Ġa ĠF 1 Ġscore Ġof Ġ$ 0 . 995 $ Ġon Ġsynthes ized Ġqueries Ġand Ġ$ 0 . 9 48 $ Ġon Ġclinical Ġnotes Ġwhile Ġthe Ġi 2 b 2 ĠN ER Ġmodel Ġachieved Ġa ĠF 1 Ġscore Ġof Ġ$ 0 . 441 $ Ġon Ġsynthes ized Ġqueries Ġand Ġ$ 0 . 9 27 $ Ġon Ġclinical Ġnotes Ġ( See ĠTable ĠT AB REF 23 ). </s> Since Ġthere Ġwas Ġno Ġground Ġtruth Ġavailable Ġfor Ġthe Ġretrieved Ġterms , Ġwe Ġrandomly Ġpicked Ġa Ġfew Ġsamples Ġto Ġassess Ġits Ġperformance . ĠSome Ġexample Ġoutputs Ġof Ġour Ġcomplete Ġframework Ġon Ġreal Ġuser Ġqueries Ġare Ġshown Ġin ĠFigure ĠFIG REF 24 . ĠFor Ġexample , Ġfrom Ġthe Ġfigure Ġwe Ġsee Ġthat Ġthe Ġquery Ġ" child Ġfever Ġdouble Ġvision Ġdizz y " Ġwas Ġcorrectly Ġtagged Ġwith Ġfour Ġentities : Ġ" child ", Ġ" f ever ", Ġ" double Ġvision ", Ġand Ġ" d izzy ". ĠA Ġlist Ġof Ġterms Ġfrom Ġour Ġgloss ary Ġwas Ġmatched Ġto Ġeach Ġentity . ĠIn Ġreal Ġworld Ġapplication , Ġthe Ġlists Ġof Ġterms Ġwill Ġbe Ġpresented Ġto Ġthe Ġuser Ġas Ġthe Ġretrieval Ġresults Ġto Ġtheir Ġqueries . </s> Discussion </s> In Ġmost Ġreal Ġuser Ġqueries Ġwe Ġsampled , Ġthe Ġentities Ġwere Ġtagged Ġat Ġthe Ġcorrect Ġboundary Ġand Ġthe Ġtagging Ġwas Ġcomplete Ġ( such Ġas Ġthe Ġones Ġshown Ġin ĠFigure ĠFIG REF 24 ). ĠOnly Ġon Ġa Ġfew Ġuser Ġqueries Ġthe Ġtagging Ġwas Ġcontroversial . ĠFor Ġexample , Ġthe Ġquery ĠâĢ ľ E ry the mat ous Ġbl anch ing Ġround , Ġoval Ġpatches Ġon Ġtorso , Ġextrem ities âĢ Ŀ Ġwas Ġtagged Ġas ĠâĢ ľ E ry the mat ous Ġbl anch ing âĢ Ŀ Ġand ĠâĢ ľ oval Ġpatches Ġon Ġtorso âĢ Ŀ . ĠThe Ġentity ĠâĢ ľ ext rem ities âĢ Ŀ Ġwas Ġmissing . ĠThe Ġsegment ation Ġwas Ġalso Ġnot Ġcorrect . ĠA Ġmore Ġappropriate Ġtagging Ġwould Ġbe ĠâĢ ľ E ry the mat ous Ġbl anch ing Ġround , Ġoval Ġpatches âĢ Ŀ , ĠâĢ ľ tor so âĢ Ŀ , Ġand ĠâĢ ľ ext rem ities âĢ Ŀ . ĠThe Ġtagging Ġcould Ġbe Ġfurther Ġimproved Ġby Ġsynthes izing Ġmore Ġrealistic Ġuser Ġqueries . ĠRecall Ġthat Ġthe Ġsynthes ized Ġuser Ġqueries Ġwere Ġcreated Ġby Ġrandomly Ġcombining Ġtermin ologies Ġfrom Ġthe Ġdermat ology Ġgloss ary , Ġwhich , Ġwhile Ġproviding Ġdata Ġthat Ġhelped Ġthe Ġmodel Ġlearn Ġentity Ġsegment ation , Ġdid Ġnot Ġreflect Ġthe Ġco - occ urrence Ġinformation Ġin Ġreal Ġuser Ġqueries . ĠFor Ġexample , Ġthere Ġcould Ġbe Ġtwo Ġclinical Ġentities Ġthat Ġoften Ġco - occ ur Ġor Ġnever Ġco - occ ur Ġin Ġa Ġuser Ġquery . ĠBut Ġsince Ġthe Ġsynthes ized Ġuser Ġqueries Ġwe Ġused Ġcombined Ġterms Ġrandomly , Ġthe Ġco - occ urrence Ġinformation Ġwas Ġthus Ġmissing . </s> The Ġfinal Ġretrieval Ġresults Ġof Ġour Ġframework Ġwere Ġnot Ġevaluated Ġquant itatively Ġin Ġterms Ġof Ġrecall Ġand Ġprecision , Ġdue Ġthe Ġthe Ġlack Ġof Ġground Ġtruth . ĠWhen Ġground Ġtruth Ġbecomes Ġavailable , Ġwe Ġwill Ġbe Ġable Ġto Ġevaluate Ġour Ġframework Ġmore Ġthoroughly . ĠRecently , Ġa Ġfine - tun ed ĠB ERT Ġmodel Ġin Ġthe Ġmedical Ġdomain Ġcalled ĠBio BER T ĠB IB REF 17 Ġhas Ġattracted Ġsome Ġattention Ġin Ġthe Ġmedical ĠN LP Ġdomain . ĠWe Ġcould Ġexperiment Ġwith ĠBio BER T Ġembed d ings Ġin Ġthe Ġfuture . ĠWe Ġcould Ġalso Ġinclude Ġquery Ġexpansion Ġtechnique Ġfor Ġterm Ġmatching . ĠWhen Ġfinding Ġcandidate Ġterms Ġfor Ġan Ġentity , Ġour Ġfirst Ġstep Ġwas Ġstill Ġbased Ġon Ġstring Ġmatching . ĠGiven Ġthat Ġthere Ġmight Ġbe Ġmultiple Ġentities Ġthat Ġcould Ġbe Ġmatched Ġto Ġthe Ġsame Ġterm , Ġit Ġcould Ġbe Ġhard Ġto Ġinclude Ġall Ġthese Ġentities Ġin Ġthe Ġgloss ary Ġand Ġhard Ġto Ġmatch Ġterms Ġto Ġthese Ġentities . </s> Conclusion </s> In Ġthis Ġproject , Ġwe Ġtackle Ġthe Ġproblem Ġof Ġextracting Ġclinical Ġconcepts Ġfrom Ġuser Ġqueries Ġon Ġmedical Ġsearch Ġengines . ĠBy Ġtraining Ġa ĠBi L ST M - CR F Ġmodel Ġon Ġa Ġhybrid Ġdata Ġconsisting Ġof Ġsynthes ized Ġuser Ġqueries Ġand Ġsentences Ġfrom Ġclinical Ġnote , Ġwe Ġadopt Ġa ĠCE Ġframework Ġfor Ġclinical Ġuser Ġqueries Ġwith Ġminimal Ġeffort Ġspent Ġon Ġannot ating Ġuser Ġqueries . ĠWe Ġfind Ġthat Ġthe Ġhybrid Ġdata Ġenables Ġthe ĠN ER Ġmodel Ġperform Ġbetter Ġon Ġboth Ġtagging Ġthe Ġuser Ġqueries Ġand Ġthe Ġclinical Ġnote Ġsentences . ĠFurthermore , Ġour Ġframework Ġis Ġbuilt Ġon Ġan Ġeasy - to - use Ġdeep Ġlearning ĠN LP ĠPython Ġlibrary , Ġwhich Ġlends Ġit Ġmore Ġprospective Ġvalue Ġto Ġvarious Ġonline Ġmedical Ġapplications Ġthat Ġemploy Ġmedical Ġsearch Ġengines . </s> Acknowled gment </s> This Ġpaper Ġresults Ġfrom Ġa Ġtechnical Ġreport Ġof Ġa Ġproject Ġthe Ġauthors Ġhave Ġworked Ġon Ġwith Ġvisual D x , Ġa Ġhealthcare Ġinform atics Ġcompany Ġthat Ġprovides Ġweb - based Ġclinical Ġdecision Ġsupport Ġsystem . ĠThe Ġauthors Ġwould Ġlike Ġto Ġthank Ġvisual D x Ġfor Ġproviding Ġthem Ġthe Ġopportunity Ġto Ġwork Ġon Ġsuch Ġan Ġexciting Ġproject . ĠIn Ġparticular , Ġthe Ġauthors Ġwould Ġlike Ġto Ġthank ĠRoy ĠRobinson , Ġthe ĠVice ĠPresident Ġof ĠTechnology Ġand ĠMedical ĠIn format ics Ġat Ġvisual D x , Ġfor Ġproviding Ġthe Ġsynthes ized Ġuser Ġqueries , Ġas Ġwell Ġas Ġpreliminary Ġfeedback Ġon Ġthe Ġperformance Ġof Ġour Ġframework .