Experienced medical doctors used a linguistic annotation tool to annotate entities.
ABSTRACTIVE
WebAnno
EXTRACTIVE
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports. The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. Their quality varied widely throughout the corpus. The blank version was preferred by the annotators. We distribute the corpus in BioC JSON format. BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain. It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner. BioC is based on character offsets of annotations and allows the stacking of different layers.
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. Their quality varied widely throughout the corpus. The blank version was preferred by the annotators. We distribute the corpus in BioC JSON format. BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain. It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner. BioC is based on character offsets of annotations and allows the stacking of different layers.
2003.13032
Introduction </s> The Ġautomatic Ġprocessing Ġof Ġmedical Ġtexts Ġand Ġdocuments Ġplays Ġan Ġincreasingly Ġimportant Ġrole Ġin Ġthe Ġrecent Ġdevelopment Ġof Ġthe Ġdigital Ġhealth Ġarea . ĠTo Ġenable Ġdedicated ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġthat Ġis Ġhighly Ġaccurate Ġwith Ġrespect Ġto Ġmedically Ġrelevant Ġcategories , Ġmanually Ġannot ated Ġdata Ġfrom Ġthis Ġdomain Ġis Ġneeded . ĠOne Ġcategory Ġof Ġhigh Ġinterest Ġand Ġrelevance Ġare Ġmedical Ġentities . ĠOnly Ġvery Ġfew Ġannot ated Ġcorpor a Ġin Ġthe Ġmedical Ġdomain Ġexist . ĠMany Ġof Ġthem Ġfocus Ġon Ġthe Ġrelation Ġbetween Ġchemicals Ġand Ġdiseases Ġor Ġproteins Ġand Ġdiseases , Ġsuch Ġas Ġthe ĠBC 5 C DR Ġcorpus ĠB IB REF 0 , Ġthe ĠCompar ative ĠToxic ogen omics ĠDatabase ĠB IB REF 1 , Ġthe ĠF SU ĠPR ote in ĠGE ne Ġcorpus ĠB IB REF 2 Ġor Ġthe ĠAD E Ġ( ad verse Ġdrug Ġeffect ) Ġcorpus ĠB IB REF 3 . ĠThe ĠNC BI ĠDisease ĠCorpus ĠB IB REF 4 Ġcontains Ġcondition Ġmention Ġannotations Ġalong Ġwith Ġannotations Ġof Ġsymptoms . ĠSeveral Ġnew Ġcorpor a Ġof Ġannot ated Ġcase Ġreports Ġwere Ġmade Ġavailable Ġrecently . Ġgrou in - et al - 2019 - clinical Ġpresented Ġa Ġcorpus Ġwith Ġmedical Ġentity Ġannotations Ġof Ġclinical Ġcases Ġwritten Ġin ĠFrench , Ġcop d P hen otype Ġpresented Ġa Ġcorpus Ġfocusing Ġon Ġphen otyp ic Ġinformation Ġfor Ġchronic Ġobstruct ive Ġpulmonary Ġdisease Ġwhile Ġ10 . 10 93 / database / bay 143 Ġpresented Ġa Ġcorpus Ġfocusing Ġon Ġidentifying Ġmain Ġfinding Ġsentences Ġin Ġcase Ġreports . </s> The Ġcorpus Ġmost Ġcomparable Ġto Ġours Ġis Ġthe ĠFrench Ġcorpus Ġof Ġclinical Ġcase Ġreports Ġby Ġgrou in - et al - 2019 - clinical . ĠTheir Ġannotations Ġare Ġbased Ġon ĠU ML S Ġsemantic Ġtypes . ĠEven Ġthough Ġthere Ġis Ġan Ġoverlap Ġin Ġannot ated Ġentities , Ġsemantic Ġclasses Ġare Ġnot Ġthe Ġsame . ĠLab Ġresults Ġare Ġsubs umed Ġunder Ġfindings Ġin Ġour Ġcorpus Ġand Ġare Ġnot Ġannot ated Ġas Ġtheir Ġown Ġclass . ĠFactors Ġextend Ġbeyond Ġgender Ġand Ġage Ġand Ġdescribe Ġany Ġkind Ġof Ġrisk Ġfactor Ġthat Ġcontributes Ġto Ġa Ġhigher Ġprobability Ġof Ġhaving Ġa Ġcertain Ġdisease . ĠOur Ġcorpus Ġincludes Ġadditional Ġentity Ġtypes . ĠWe Ġannot ate Ġconditions , Ġfindings Ġ( including Ġmedical Ġfindings Ġsuch Ġas Ġblood Ġvalues ), Ġfactors , Ġand Ġalso Ġmodifiers Ġwhich Ġindicate Ġthe Ġneg ation Ġof Ġother Ġentities Ġas Ġwell Ġas Ġcase Ġentities , Ġi . Ġe ., Ġentities Ġspecific Ġto Ġone Ġcase Ġreport . ĠAn Ġoverview Ġis Ġavailable Ġin ĠTable ĠT AB REF 3 . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation Ġtasks </s> Case Ġreports Ġare Ġstandardized Ġin Ġthe ĠCARE Ġguidelines ĠB IB REF 5 . ĠThey Ġrepresent Ġa Ġdetailed Ġdescription Ġof Ġthe Ġsymptoms , Ġsigns , Ġdiagnosis , Ġtreatment , Ġand Ġfollow - up Ġof Ġan Ġindividual Ġpatient . ĠWe Ġfocus Ġon Ġdocuments Ġfreely Ġavailable Ġthrough ĠPubMed ĠCentral Ġ( PM C ). ĠThe Ġpresentation Ġof Ġthe Ġpatient 's Ġcase Ġcan Ġusually Ġbe Ġfound Ġin Ġa Ġdedicated Ġsection Ġor Ġthe Ġabstract . ĠWe Ġperform Ġa Ġmanual Ġannotation Ġof Ġall Ġmentions Ġof Ġcase Ġentities , Ġconditions , Ġfindings , Ġfactors Ġand Ġmodifiers . ĠThe Ġscope Ġof Ġour Ġmanual Ġannotation Ġis Ġlimited Ġto Ġthe Ġpresentation Ġof Ġa Ġpatient 's Ġsigns Ġand Ġsymptoms . ĠIn Ġaddition , Ġwe Ġannot ate Ġthe Ġtitle Ġof Ġthe Ġcase Ġreport . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation ĠGuidelines </s> We Ġannot ate Ġthe Ġfollowing Ġentities : </s> case Ġentity Ġmarks Ġthe Ġmention Ġof Ġa Ġpatient . ĠA Ġcase Ġreport Ġcan Ġcontain Ġmore Ġthan Ġone Ġcase Ġdescription . ĠTherefore , Ġall Ġthe Ġfindings , Ġfactors Ġand Ġconditions Ġrelated Ġto Ġone Ġpatient Ġare Ġlinked Ġto Ġthe Ġrespective Ġcase Ġentity . ĠWithin Ġthe Ġtext , Ġthis Ġentity Ġis Ġoften Ġrepresented Ġby Ġthe Ġfirst Ġmention Ġof Ġthe Ġpatient Ġand Ġoverl aps Ġwith Ġthe Ġfactor Ġannotations Ġwhich Ġcan , Ġe . Ġg ., Ġmark Ġsex Ġand Ġage Ġ( cf . ĠFigure ĠFIG REF 12 ). </s> condition Ġmarks Ġa Ġmedical Ġdisease Ġsuch Ġas Ġpneum oth or ax Ġor Ġdisl ocation Ġof Ġthe Ġshoulder . </s> factor Ġmarks Ġa Ġfeature Ġof Ġa Ġpatient Ġwhich Ġmight Ġinfluence Ġthe Ġprobability Ġfor Ġa Ġspecific Ġdiagnosis . ĠIt Ġcan Ġbe Ġimmutable Ġ( e . Ġg ., Ġsex Ġand Ġage ), Ġdescribe Ġa Ġspecific Ġmedical Ġhistory Ġ( e . Ġg ., Ġdiabetes Ġmell itus ) Ġor Ġa Ġbehaviour Ġ( e . Ġg ., Ġsmoking ). </s> finding Ġmarks Ġa Ġsign Ġor Ġsymptom Ġa Ġpatient Ġshows . ĠThis Ġcan Ġbe Ġvisible Ġ( e . Ġg ., Ġrash ), Ġdescribed Ġby Ġa Ġpatient Ġ( e . Ġg ., Ġheadache ) Ġor Ġmeasurable Ġ( e . Ġg ., Ġdecreased Ġblood Ġglucose Ġlevel ). </s> neg ation Ġmodifier Ġexplicitly Ġnegate Ġthe Ġpresence Ġof Ġa Ġcertain Ġfinding Ġusually Ġsetting Ġthe Ġcase Ġapart Ġfrom Ġcommon Ġcases . </s> We Ġalso Ġannot ate Ġrelations Ġbetween Ġthese Ġentities , Ġwhere Ġapplicable . ĠSince Ġwe Ġwork Ġon Ġcase Ġdescriptions , Ġthe Ġanchor Ġpoint Ġof Ġthese Ġrelations Ġis Ġthe Ġcase Ġthat Ġis Ġdescribed . ĠThe Ġfollowing Ġrelations Ġare Ġannot ated : </s> has Ġrelations Ġexist Ġbetween Ġa Ġcase Ġentity Ġand Ġfactor , Ġfinding Ġor Ġcondition Ġentities . </s> mod ifies Ġrelations Ġexist Ġbetween Ġneg ation Ġmodifiers Ġand Ġfindings . </s> ca uses Ġrelations Ġexist Ġbetween Ġconditions Ġand Ġfindings . </s> Example Ġannotations Ġare Ġshown Ġin ĠFigure ĠFIG REF 16 . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn not ators </s> We Ġasked Ġmedical Ġdoctors Ġexperienced Ġin Ġextracting Ġknowledge Ġrelated Ġto Ġmedical Ġentities Ġfrom Ġtexts Ġto Ġannot ate Ġthe Ġentities Ġdescribed Ġabove . ĠInitially , Ġwe Ġasked Ġfour Ġannot ators Ġto Ġtest Ġour Ġguidelines Ġon Ġtwo Ġtexts . ĠSub sequently , Ġidentified Ġissues Ġwere Ġdiscussed Ġand Ġresolved . ĠFollowing Ġthis Ġpilot Ġannotation Ġphase , Ġwe Ġasked Ġtwo Ġdifferent Ġannot ators Ġto Ġannot ate Ġtwo Ġcase Ġreports Ġaccording Ġto Ġour Ġguidelines . ĠThe Ġsame Ġannot ators Ġannot ated Ġan Ġoverall Ġcollection Ġof Ġ53 Ġcase Ġreports . </s> Inter - annot ator Ġagreement Ġis Ġcalculated Ġbased Ġon Ġtwo Ġcase Ġreports . ĠWe Ġreach Ġa ĠCohen 's Ġk appa ĠB IB REF 6 Ġof Ġ0 . 68 . ĠDis ag reements Ġmainly Ġappear Ġfor Ġfindings Ġthat Ġare Ġrather Ġun specific Ġsuch Ġas ĠShe Ġno Ġlonger Ġeats Ġout Ġwith Ġfriends Ġwhich Ġcan Ġbe Ġseen Ġas Ġa Ġfinding Ġreferring Ġto ĠâĢ ľ avoid ance Ġbehaviour âĢ Ŀ . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation ĠTools Ġand ĠFormat </s> The Ġannotation Ġwas Ġperformed Ġusing ĠWeb An no ĠB IB REF 7 , Ġa Ġweb - based Ġtool Ġfor Ġlinguistic Ġannotation . ĠThe Ġannot ators Ġcould Ġchoose Ġbetween Ġa Ġpre - annot ated Ġversion Ġor Ġa Ġblank Ġversion Ġof Ġeach Ġtext . ĠThe Ġpre - annot ated Ġversions Ġcontained Ġsuggested Ġentity Ġspans Ġbased Ġon Ġstring Ġmatches Ġfrom Ġlists Ġof Ġconditions Ġand Ġfindings Ġsyn onym Ġlists . ĠTheir Ġquality Ġvaried Ġwidely Ġthroughout Ġthe Ġcorpus . ĠThe Ġblank Ġversion Ġwas Ġpreferred Ġby Ġthe Ġannot ators . ĠWe Ġdistribute Ġthe Ġcorpus Ġin ĠBio C ĠJSON Ġformat . ĠBio C Ġwas Ġchosen Ġas Ġit Ġallows Ġus Ġto Ġcapture Ġthe Ġcomplexities Ġof Ġthe Ġannotations Ġin Ġthe Ġbiomedical Ġdomain . ĠIt Ġrepresented Ġeach Ġdocuments Ġproperties Ġranging Ġfrom Ġfull Ġtext , Ġindividual Ġpassages / sent ences Ġalong Ġwith Ġcaptured Ġannotations Ġand Ġrelationships Ġin Ġan Ġorganized Ġmanner . ĠBio C Ġis Ġbased Ġon Ġcharacter Ġoffsets Ġof Ġannotations Ġand Ġallows Ġthe Ġstacking Ġof Ġdifferent Ġlayers . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠCorpus ĠOverview </s> The Ġcorpus Ġconsists Ġof Ġ53 Ġdocuments , Ġwhich Ġcontain Ġan Ġaverage Ġnumber Ġof Ġ156 . 1 Ġsentences Ġper Ġdocument , Ġeach Ġwith Ġ19 . 55 Ġtokens Ġon Ġaverage . ĠThe Ġcorpus Ġcomprises Ġ8 , 275 Ġsentences Ġand Ġ167 , 7 39 Ġwords Ġin Ġtotal . ĠHowever , Ġas Ġmentioned Ġabove , Ġonly Ġcase Ġpresentation Ġsections , Ġhead ings Ġand Ġabstract s Ġare Ġannot ated . ĠThe Ġnumbers Ġof Ġannot ated Ġentities Ġare Ġsummarized Ġin ĠTable ĠT AB REF 24 . </s> Find ings Ġare Ġthe Ġmost Ġfrequently Ġannot ated Ġtype Ġof Ġentity . ĠThis Ġmakes Ġsense Ġgiven Ġthat Ġfindings Ġpaint Ġa Ġclinical Ġpicture Ġof Ġthe Ġpatient 's Ġcondition . ĠThe Ġnumber Ġof Ġtokens Ġper Ġentity Ġranges Ġfrom Ġone Ġtoken Ġfor Ġall Ġtypes Ġto Ġ5 Ġtokens Ġfor Ġcases Ġ( average Ġlength Ġ3 . 1 ), Ġnine Ġtokens Ġfor Ġconditions Ġ( average Ġlength Ġ2 . 0 ), Ġ16 Ġtokens Ġfor Ġfactors Ġ( average Ġlength Ġ2 . 5 ), Ġ25 Ġtokens Ġfor Ġfindings Ġ( average Ġlength Ġ2 . 6 ) Ġand Ġ18 Ġtokens Ġfor Ġmodifiers Ġ( average Ġlength Ġ1 . 4 ) Ġ( cf . ĠTable ĠT AB REF 24 ). ĠExamples Ġof Ġrather Ġlong Ġentities Ġare Ġgiven Ġin ĠTable ĠT AB REF 25 . </s> Ent ities Ġcan Ġappear Ġin Ġa Ġdiscontin uous Ġway . ĠWe Ġmodel Ġthis Ġas Ġa Ġrelation Ġbetween Ġtwo Ġspans Ġwhich Ġwe Ġcall ĠâĢ ľ disc ont in uous âĢ Ŀ Ġ( cf . ĠFigure ĠFIG REF 26 ). ĠEspecially Ġfindings Ġoften Ġappear Ġas Ġdiscontin uous Ġentities , Ġwe Ġfound Ġ5 43 Ġdiscontin uous Ġfinding Ġrelations . ĠThe Ġnumbers Ġfor Ġconditions Ġand Ġfactors Ġare Ġlower Ġwith Ġseven Ġand Ġtwo , Ġrespectively . ĠEnt ities Ġcan Ġalso Ġbe Ġnested Ġwithin Ġone Ġanother . ĠThis Ġhappens Ġeither Ġwhen Ġthe Ġspan Ġof Ġone Ġannotation Ġis Ġcompletely Ġembedded Ġin Ġthe Ġspan Ġof Ġanother Ġannotation Ġ( fully - n ested ; Ġcf . ĠFigure ĠFIG REF 12 ), Ġor Ġwhen Ġthere Ġis Ġa Ġpartial Ġoverlapping Ġbetween Ġthe Ġspans Ġof Ġtwo Ġdifferent Ġentities Ġ( part ially - n ested ; Ġcf . ĠFigure ĠFIG REF 12 ). ĠThere Ġis Ġa Ġhigh Ġnumber Ġof Ġinter - sent ential Ġrelations Ġin Ġthe Ġcorpus Ġ( cf . ĠTable ĠT AB REF 27 ). ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthe Ġcase Ġentity Ġoccurs Ġearly Ġin Ġeach Ġdocument ; Ġfurthermore , Ġit Ġis Ġrelated Ġto Ġfinding Ġand Ġfactor Ġannotations Ġthat Ġare Ġdistributed Ġacross Ġdifferent Ġsentences . </s> The Ġmost Ġfrequently Ġannot ated Ġrelation Ġin Ġour Ġcorpus Ġis Ġthe Ġhas - relation Ġbetween Ġa Ġcase Ġentity Ġand Ġthe Ġfindings Ġrelated Ġto Ġthat Ġcase . ĠThis Ġcorrelates Ġwith Ġthe Ġhigh Ġnumber Ġof Ġfinding Ġentities . ĠThe Ġrelations Ġcontained Ġin Ġour Ġcorpus Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports </s> We Ġevaluate Ġthe Ġcorpus Ġusing ĠNamed ĠEntity ĠRecogn ition Ġ( NER ), Ġi . Ġe ., Ġthe Ġtask Ġof Ġfinding Ġmentions Ġof Ġconcepts Ġof Ġinterest Ġin Ġun struct ured Ġtext . ĠWe Ġfocus Ġon Ġdetecting Ġcases , Ġconditions , Ġfactors , Ġfindings Ġand Ġmodifiers Ġin Ġcase Ġreports Ġ( cf . ĠSection ĠSEC REF 6 ). ĠWe Ġapproach Ġthis Ġas Ġa Ġsequence Ġlabeling Ġproblem . ĠFour Ġsystems Ġwere Ġdeveloped Ġto Ġoffer Ġcomparable Ġrobust Ġbas elines . </s> The Ġoriginal Ġdocuments Ġare Ġpre - process ed Ġ( sent ence Ġsplitting Ġand Ġtoken ization Ġwith ĠSc isp a Cy ). ĠWe Ġdo Ġnot Ġperform Ġstop Ġword Ġremoval Ġor Ġlower - c asing Ġof Ġthe Ġtokens . ĠThe ĠB IO Ġlabeling Ġscheme Ġis Ġused Ġto Ġcapture Ġthe Ġorder Ġof Ġtokens Ġbelonging Ġto Ġthe Ġsame Ġentity Ġtype Ġand Ġenable Ġspan - level Ġdetection Ġof Ġentities . ĠDetection Ġof Ġnested Ġand / or Ġdiscontin uous Ġentities Ġis Ġnot Ġsupported . ĠThe Ġannot ated Ġcorpus Ġis Ġrandomized Ġand Ġsplit Ġin Ġfive Ġfolds Ġusing Ġsc ik it - learn ĠB IB REF 9 . ĠEach Ġfold Ġhas Ġa Ġtrain , Ġtest Ġand Ġdev Ġsplit Ġwith Ġthe Ġtest Ġsplit Ġdefined Ġas Ġ. 15 % Ġof Ġthe Ġtrain Ġsplit . ĠThis Ġensures Ġcompar ability Ġbetween Ġthe Ġpresented Ġsystems . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠCond itional ĠRandom ĠFields </s> Cond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 10 Ġare Ġa Ġstandard Ġapproach Ġwhen Ġdealing Ġwith Ġsequential Ġdata Ġin Ġthe Ġcontext Ġof Ġsequence Ġlabeling . ĠWe Ġuse Ġa Ġcombination Ġof Ġlinguistic Ġand Ġsemantic Ġfeatures , Ġwith Ġa Ġcontext Ġwindow Ġof Ġsize Ġfive , Ġto Ġdescribe Ġeach Ġof Ġthe Ġtokens Ġand Ġthe Ġdependencies Ġbetween Ġthem . ĠHyper - param eter Ġoptimization Ġis Ġperformed Ġusing Ġrandomized Ġsearch Ġand Ġcross Ġvalidation . ĠSpan - based ĠF 1 Ġscore Ġis Ġused Ġas Ġthe Ġoptimization Ġmetric . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠBi L ST M - CR F </s> Prior Ġto Ġthe Ġemergence Ġof Ġdeep Ġneural Ġlanguage Ġmodels , ĠBi L ST M - CR F Ġmodels ĠB IB REF 11 Ġhad Ġachieved Ġstate - of - the - art Ġresults Ġfor Ġthe Ġtask Ġof Ġsequence Ġlabeling . ĠWe Ġuse Ġa ĠBi L ST M - CR F Ġmodel Ġwith Ġboth Ġword - level Ġand Ġcharacter - level Ġinput . ĠBio Word V ec ĠB IB REF 12 Ġpre - trained Ġword Ġembed d ings Ġare Ġused Ġin Ġthe Ġembed ding Ġlayer Ġfor Ġthe Ġinput Ġrepresentation . ĠA Ġbid irection al ĠL ST M Ġlayer Ġis Ġapplied Ġto Ġa Ġmultiplication Ġof Ġthe Ġtwo Ġinput Ġrepresentations . ĠFinally , Ġa ĠCR F Ġlayer Ġis Ġapplied Ġto Ġpredict Ġthe Ġsequence Ġof Ġlabels . ĠDrop out Ġand ĠL 1 / L 2 Ġregular ization Ġis Ġused Ġwhere Ġapplicable . ĠHe Ġ( un iform ) Ġinitialization ĠB IB REF 13 Ġis Ġused Ġto Ġinitialize Ġthe Ġkernels Ġof Ġthe Ġindividual Ġlayers . ĠAs Ġthe Ġloss Ġmetric , ĠCR F - based Ġloss Ġis Ġused , Ġwhile Ġoptimizing Ġthe Ġmodel Ġbased Ġon Ġthe ĠCR F ĠV iter bi Ġaccuracy . ĠAdditionally , Ġspan - based ĠF 1 Ġscore Ġis Ġused Ġto Ġserial ize Ġthe Ġbest Ġperforming Ġmodel . ĠWe Ġtrain Ġfor Ġa Ġmaximum Ġof Ġ100 Ġepoch s , Ġor Ġuntil Ġan Ġearly Ġstopping Ġcriterion Ġis Ġreached Ġ( no Ġchange Ġin Ġvalidation Ġloss Ġvalue Ġgr ater Ġthan Ġ0 . 01 Ġfor Ġten Ġconsecutive Ġepoch s ). ĠFurthermore , ĠAdam ĠB IB REF 14 Ġis Ġused Ġas Ġthe Ġoptim izer . ĠThe Ġlearning Ġrate Ġis Ġreduced Ġby Ġa Ġfactor Ġof Ġ0 . 3 Ġin Ġcase Ġno Ġsignificant Ġincrease Ġof Ġthe Ġoptimization Ġmetric Ġis Ġachieved Ġin Ġthree Ġconsecutive Ġepoch s . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠMulti - Task ĠLearning </s> Multi - Task ĠLearning Ġ( MT L ) ĠB IB REF 15 Ġhas Ġbecome Ġpopular Ġwith Ġthe Ġprogress Ġin Ġdeep Ġlearning . ĠThis Ġmodel Ġfamily Ġis Ġcharacterized Ġby Ġsimultaneous Ġoptimization Ġof Ġmultiple Ġloss Ġfunctions Ġand Ġtransfer Ġof Ġknowledge Ġachieved Ġthis Ġway . ĠThe Ġknowledge Ġis Ġtransferred Ġthrough Ġthe Ġuse Ġof Ġone Ġor Ġmultiple Ġshared Ġlayers . ĠThrough Ġfinding Ġsupporting Ġpatterns Ġin Ġrelated Ġtasks , ĠM TL Ġprovides Ġbetter Ġgeneral ization Ġon Ġunseen Ġcases Ġand Ġthe Ġmain Ġtasks Ġwe Ġare Ġtrying Ġto Ġsolve . </s> We Ġrely Ġon Ġthe Ġmodel Ġpresented Ġby Ġbe k oul is 2018 j oint Ġand Ġreuse Ġthe Ġimplementation Ġprovided Ġby Ġthe Ġauthors . ĠThe Ġmodel Ġjointly Ġtrains Ġtwo Ġobjectives Ġsupported Ġby Ġthe Ġdataset : Ġthe Ġmain Ġtask Ġof ĠN ER Ġand Ġa Ġsupporting Ġtask Ġof ĠRel ation ĠExt raction Ġ( RE ). ĠTwo Ġseparate Ġmodels Ġare Ġdeveloped Ġfor Ġeach Ġof Ġthe Ġtasks . ĠThe ĠN ER Ġtask Ġis Ġsolved Ġwith Ġthe Ġhelp Ġof Ġa ĠBi L ST M - CR F Ġmodel , Ġsimilar Ġto Ġthe Ġone Ġpresented Ġin ĠSection ĠSEC REF 32 ĠThe ĠRE Ġtask Ġis Ġsolved Ġby Ġusing Ġa Ġmulti - head Ġselection Ġapproach , Ġwhere Ġeach Ġtoken Ġcan Ġhave Ġnone Ġor Ġmore Ġrelationships Ġto Ġin - sent ence Ġtokens . ĠAdditionally , Ġthis Ġmodel Ġalso Ġlever ages Ġthe Ġoutput Ġof Ġthe ĠN ER Ġbranch Ġmodel Ġ( the ĠCR F Ġprediction ) Ġto Ġlearn Ġlabel Ġembed d ings . ĠShared Ġlayers Ġconsist Ġof Ġa Ġconc aten ation Ġof Ġword Ġand Ġcharacter Ġembed d ings Ġfollowed Ġby Ġtwo Ġbid irection al ĠL ST M Ġlayers . ĠWe Ġkeep Ġmost Ġof Ġthe Ġparameters Ġsuggested Ġby Ġthe Ġauthors Ġand Ġchange Ġ( 1 ) Ġthe Ġnumber Ġof Ġtraining Ġepoch s Ġto Ġ100 Ġto Ġallow Ġthe Ġcomparison Ġto Ġother Ġdeep Ġlearning Ġapproaches Ġin Ġthis Ġwork , Ġ( 2 ) Ġuse Ġlabel Ġembed d ings Ġof Ġsize Ġ64 , Ġ( 3 ) Ġallow Ġgradient Ġclipping Ġand Ġ( 4 ) Ġuse Ġ$ d = 0 . 8 $ Ġas Ġthe Ġpre - trained Ġword Ġembed ding Ġdrop out Ġand Ġ$ d = 0 . 5 $ Ġfor Ġall Ġother Ġdrop outs . Ġ$\ eta Ġ= 1 ^{ - 3 } $ Ġis Ġused Ġas Ġthe Ġlearning Ġrate Ġwith Ġthe ĠAdam Ġoptim izer Ġand Ġtan h Ġactivation Ġfunctions Ġacross Ġlayers . ĠAlthough Ġit Ġis Ġpossible Ġto Ġuse Ġadvers arial Ġtraining ĠB IB REF 16 , Ġwe Ġomit Ġfrom Ġusing Ġit . ĠWe Ġalso Ġomit Ġthe Ġpublication Ġof Ġresults Ġfor Ġthe Ġtask Ġof ĠRE Ġas Ġwe Ġconsider Ġit Ġto Ġbe Ġa Ġsupporting Ġtask Ġand Ġno Ġother Ġcompeting Ġapproaches Ġhave Ġbeen Ġdeveloped . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠBio BER T </s> Deep Ġneural Ġlanguage Ġmodels Ġhave Ġrecently Ġevolved Ġto Ġa Ġsuccessful Ġmethod Ġfor Ġrepresenting Ġtext . ĠIn Ġparticular , ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) Ġoutper formed Ġprevious Ġstate - of - the - art Ġmethods Ġby Ġa Ġlarge Ġmargin Ġon Ġvarious ĠN LP Ġtasks ĠB IB REF 17 . ĠFor Ġour Ġexperiments , Ġwe Ġuse ĠBio BER T , Ġan Ġadaptation Ġof ĠB ERT Ġfor Ġthe Ġbiomedical Ġdomain , Ġpre - trained Ġon ĠPubMed Ġabstract s Ġand ĠPM C Ġfull - text Ġarticles ĠB IB REF 18 . ĠThe ĠB ERT Ġarchitecture Ġfor Ġder iving Ġtext Ġrepresentations Ġuses Ġ12 Ġhidden Ġlayers , Ġconsisting Ġof Ġ768 Ġunits Ġeach . ĠFor ĠN ER , Ġtoken Ġlevel ĠB IO - tag Ġprobabilities Ġare Ġcomputed Ġwith Ġa Ġsingle Ġoutput Ġlayer Ġbased Ġon Ġthe Ġrepresentations Ġfrom Ġthe Ġlast Ġlayer Ġof ĠB ERT . ĠWe Ġfine - t une Ġthe Ġmodel Ġon Ġthe Ġentity Ġrecognition Ġtask Ġduring Ġfour Ġtraining Ġepoch s Ġwith Ġbatch Ġsize Ġ$ b = 32 $, Ġdrop out Ġprobability Ġ$ d = 0 . 1 $ Ġand Ġlearning Ġrate Ġ$\ eta Ġ= 2 ^{ - 5 } $. ĠThese Ġhyper - param eters Ġare Ġproposed Ġby ĠDev lin 2018 Ġfor ĠB ERT Ġfine - tun ing . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠEvaluation </s> To Ġevaluate Ġthe Ġperformance Ġof Ġthe Ġfour Ġsystems , Ġwe Ġcalculate Ġthe Ġspan - level Ġprecision Ġ( P ), Ġrecall Ġ( R ) Ġand ĠF 1 Ġscores , Ġalong Ġwith Ġcorresponding Ġmicro Ġand Ġmacro Ġscores . ĠThe Ġreported Ġvalues Ġare Ġshown Ġin ĠTable ĠT AB REF 29 Ġand Ġare Ġaveraged Ġover Ġfive Ġfolds , Ġutil ising Ġthe Ġseq eval Ġframework . </s> With Ġa Ġmacro Ġavg . ĠF 1 - score Ġof Ġ0 . 59 , ĠM TL Ġachieves Ġthe Ġbest Ġresult Ġwith Ġa Ġsignificant Ġmargin Ġcompared Ġto ĠCR F , ĠBi L ST M - CR F Ġand ĠB ERT . ĠThis Ġconfirms Ġthe Ġusefulness Ġof Ġjointly Ġtraining Ġmultiple Ġobjectives Ġ( min im izing Ġmultiple Ġloss Ġfunctions ), Ġand Ġenabling Ġknowledge Ġtransfer , Ġespecially Ġin Ġa Ġsetting Ġwith Ġlimited Ġdata Ġ( which Ġis Ġusually Ġthe Ġcase Ġin Ġthe Ġbiomedical ĠN LP Ġdomain ). ĠThis Ġresult Ġalso Ġsuggest Ġthe Ġusefulness Ġof ĠBio BER T Ġfor Ġother Ġbiomedical Ġdatasets Ġas Ġreported Ġby ĠLee 2019 . ĠDespite Ġbeing Ġa Ġrather Ġstandard Ġapproach , ĠCR F Ġoutper forms Ġthe Ġmore Ġelaborated ĠBi L ST M - CR F , Ġpresumably Ġdue Ġto Ġdata Ġscarcity Ġand Ġclass Ġimbalance . ĠWe Ġhypothes ize Ġthat Ġan Ġincrease Ġin Ġtraining Ġdata Ġwould Ġyield Ġbetter Ġresults Ġfor ĠBi L ST M - CR F Ġbut Ġnot Ġoutper form Ġtransfer Ġlearning Ġapproach Ġof ĠM TL Ġ( or Ġeven ĠBio BER T ). ĠIn Ġcontrast Ġto Ġother Ġcommon ĠN ER Ġcorpor a , Ġlike ĠCo N LL Ġ2003 , Ġeven Ġthe Ġbest Ġbaseline Ġsystem Ġonly Ġachieves Ġrelatively Ġlow Ġscores . ĠThis Ġoutcome Ġis Ġdue Ġto Ġthe Ġinherent Ġdifficulty Ġof Ġthe Ġtask Ġ( annot ators Ġare Ġexperienced Ġmedical Ġdoctors ) Ġand Ġthe Ġsmall Ġnumber Ġof Ġtraining Ġsamples . </s> Conclusion </s> We Ġpresent Ġa Ġnew Ġcorpus , Ġdeveloped Ġto Ġfacilitate Ġthe Ġprocessing Ġof Ġcase Ġreports . ĠThe Ġcorpus Ġfocuses Ġon Ġfive Ġdistinct Ġentity Ġtypes : Ġcases , Ġconditions , Ġfactors , Ġfindings Ġand Ġmodifiers . ĠWhere Ġapplicable , Ġrelationships Ġbetween Ġentities Ġare Ġalso Ġannot ated . ĠAdditionally , Ġwe Ġannot ate Ġdiscontin uous Ġentities Ġwith Ġa Ġspecial Ġrelationship Ġtype Ġ( disc ont in uous ). ĠThe Ġcorpus Ġpresented Ġin Ġthis Ġpaper Ġis Ġthe Ġvery Ġfirst Ġof Ġits Ġkind Ġand Ġa Ġvaluable Ġaddition Ġto Ġthe Ġscarce Ġnumber Ġof Ġcorpor a Ġavailable Ġin Ġthe Ġfield Ġof Ġbiomedical ĠN LP . ĠIts Ġcomplexity , Ġgiven Ġthe Ġdiscontin uous Ġnature Ġof Ġentities Ġand Ġa Ġhigh Ġnumber Ġof Ġnested Ġand Ġmulti - label Ġentities , Ġposes Ġnew Ġchallenges Ġfor ĠN LP Ġmethods Ġapplied Ġfor ĠN ER Ġand Ġcan , Ġhence , Ġbe Ġa Ġvaluable Ġsource Ġfor Ġinsights Ġinto Ġwhat Ġentities ĠâĢ ľ look Ġlike Ġin Ġthe Ġwild âĢ Ŀ . ĠMoreover , Ġit Ġcan Ġserve Ġas Ġa Ġplayground Ġfor Ġnew Ġmodelling Ġtechniques Ġsuch Ġas Ġthe Ġresolution Ġof Ġdiscontin uous Ġentities Ġas Ġwell Ġas Ġmulti - task Ġlearning Ġgiven Ġthe Ġcombination Ġof Ġentities Ġand Ġtheir Ġrelations . ĠWe Ġprovide Ġan Ġevaluation Ġof Ġfour Ġdistinct ĠN ER Ġsystems Ġthat Ġwill Ġserve Ġas Ġrobust Ġbas elines Ġfor Ġfuture Ġwork Ġbut Ġwhich Ġare , Ġas Ġof Ġyet , Ġunable Ġto Ġsolve Ġall Ġthe Ġcomplex Ġchallenges Ġthis Ġdataset Ġholds . ĠA Ġfunctional Ġservice Ġbased Ġon Ġthe Ġpresented Ġcorpus Ġis Ġcurrently Ġbeing Ġintegrated , Ġas Ġa ĠN ER Ġservice , Ġin Ġthe ĠQ UR ATOR Ġplatform ĠB IB REF 20 . </s> Acknowled gments </s> The Ġresearch Ġpresented Ġin Ġthis Ġarticle Ġis Ġfunded Ġby Ġthe ĠGerman ĠFederal ĠMinistry Ġof ĠEducation Ġand ĠResearch Ġ( B MB F ) Ġthrough Ġthe Ġproject ĠQ UR ATOR Ġ( Un ter ne h men ĠRegion , ĠW ach st ums k ern , Ġgrant Ġno . Ġ03 W K DA 1 A ), Ġsee Ġhttp :// q ur ator . ai . ĠWe Ġwant Ġto Ġthank Ġour Ġmedical Ġexperts Ġfor Ġtheir Ġhelp Ġannot ating Ġthe Ġdata Ġset , Ġespecially ĠAsh lee ĠFin ck h Ġand ĠSophie ĠKl op f enstein .
Introduction The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area. To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed. One category of high interest and relevance are medical entities. Only very few annotated corpora in the medical domain exist. Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3. The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms. Several new corpora of annotated case reports were made available recently. grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10.1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports. The corpus most comparable to ours is the French corpus of clinical case reports by grouin-etal-2019-clinical. Their annotations are based on UMLS semantic types. Even though there is an overlap in annotated entities, semantic classes are not the same. Lab results are subsumed under findings in our corpus and are not annotated as their own class. Factors extend beyond gender and age and describe any kind of risk factor that contributes to a higher probability of having a certain disease. Our corpus includes additional entity types. We annotate conditions, findings (including medical findings such as blood values), factors, and also modifiers which indicate the negation of other entities as well as case entities, i. e., entities specific to one case report. An overview is available in Table TABREF3. A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation tasks Case reports are standardized in the CARE guidelines BIBREF5. They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient. We focus on documents freely available through PubMed Central (PMC). The presentation of the patient's case can usually be found in a dedicated section or the abstract. We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers. The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms. In addition, we annotate the title of the case report. A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Guidelines We annotate the following entities: case entity marks the mention of a patient. A case report can contain more than one case description. Therefore, all the findings, factors and conditions related to one patient are linked to the respective case entity. Within the text, this entity is often represented by the first mention of the patient and overlaps with the factor annotations which can, e. g., mark sex and age (cf. Figure FIGREF12). condition marks a medical disease such as pneumothorax or dislocation of the shoulder. factor marks a feature of a patient which might influence the probability for a specific diagnosis. It can be immutable (e. g., sex and age), describe a specific medical history (e. g., diabetes mellitus) or a behaviour (e. g., smoking). finding marks a sign or symptom a patient shows. This can be visible (e. g., rash), described by a patient (e. g., headache) or measurable (e. g., decreased blood glucose level). negation modifier explicitly negate the presence of a certain finding usually setting the case apart from common cases. We also annotate relations between these entities, where applicable. Since we work on case descriptions, the anchor point of these relations is the case that is described. The following relations are annotated: has relations exist between a case entity and factor, finding or condition entities. modifies relations exist between negation modifiers and findings. causes relations exist between conditions and findings. Example annotations are shown in Figure FIGREF16. A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotators We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above. Initially, we asked four annotators to test our guidelines on two texts. Subsequently, identified issues were discussed and resolved. Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines. The same annotators annotated an overall collection of 53 case reports. Inter-annotator agreement is calculated based on two case reports. We reach a Cohen's kappa BIBREF6 of 0.68. Disagreements mainly appear for findings that are rather unspecific such as She no longer eats out with friends which can be seen as a finding referring to “avoidance behaviour”. A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Tools and Format The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation. The annotators could choose between a pre-annotated version or a blank version of each text. The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists. Their quality varied widely throughout the corpus. The blank version was preferred by the annotators. We distribute the corpus in BioC JSON format. BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain. It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner. BioC is based on character offsets of annotations and allows the stacking of different layers. A Corpus of Medical Case Reports with Medical Entity Annotation ::: Corpus Overview The corpus consists of 53 documents, which contain an average number of 156.1 sentences per document, each with 19.55 tokens on average. The corpus comprises 8,275 sentences and 167,739 words in total. However, as mentioned above, only case presentation sections, headings and abstracts are annotated. The numbers of annotated entities are summarized in Table TABREF24. Findings are the most frequently annotated type of entity. This makes sense given that findings paint a clinical picture of the patient's condition. The number of tokens per entity ranges from one token for all types to 5 tokens for cases (average length 3.1), nine tokens for conditions (average length 2.0), 16 tokens for factors (average length 2.5), 25 tokens for findings (average length 2.6) and 18 tokens for modifiers (average length 1.4) (cf. Table TABREF24). Examples of rather long entities are given in Table TABREF25. Entities can appear in a discontinuous way. We model this as a relation between two spans which we call “discontinuous” (cf. Figure FIGREF26). Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations. The numbers for conditions and factors are lower with seven and two, respectively. Entities can also be nested within one another. This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf. Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf. Figure FIGREF12). There is a high number of inter-sentential relations in the corpus (cf. Table TABREF27). This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences. The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case. This correlates with the high number of finding entities. The relations contained in our corpus are summarized in Table TABREF27. Baseline systems for Named Entity Recognition in medical case reports We evaluate the corpus using Named Entity Recognition (NER), i. e., the task of finding mentions of concepts of interest in unstructured text. We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf. Section SECREF6). We approach this as a sequence labeling problem. Four systems were developed to offer comparable robust baselines. The original documents are pre-processed (sentence splitting and tokenization with ScispaCy). We do not perform stop word removal or lower-casing of the tokens. The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities. Detection of nested and/or discontinuous entities is not supported. The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9. Each fold has a train, test and dev split with the test split defined as .15% of the train split. This ensures comparability between the presented systems. Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling. We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them. Hyper-parameter optimization is performed using randomized search and cross validation. Span-based F1 score is used as the optimization metric. Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling. We use a BiLSTM-CRF model with both word-level and character-level input. BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation. A bidirectional LSTM layer is applied to a multiplication of the two input representations. Finally, a CRF layer is applied to predict the sequence of labels. Dropout and L1/L2 regularization is used where applicable. He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers. As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy. Additionally, span-based F1 score is used to serialize the best performing model. We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0.01 for ten consecutive epochs). Furthermore, Adam BIBREF14 is used as the optimizer. The learning rate is reduced by a factor of 0.3 in case no significant increase of the optimization metric is achieved in three consecutive epochs. Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning. This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way. The knowledge is transferred through the use of one or multiple shared layers. Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve. We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors. The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE). Two separate models are developed for each of the tasks. The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens. Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings. Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers. We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0.8$ as the pre-trained word embedding dropout and $d=0.5$ for all other dropouts. $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers. Although it is possible to use adversarial training BIBREF16, we omit from using it. We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed. Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT Deep neural language models have recently evolved to a successful method for representing text. In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17. For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18. The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each. For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT. We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0.1$ and learning rate $\eta =2^{-5}$. These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning. Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores. The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework. With a macro avg. F1-score of 0.59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT. This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain). This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019. Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance. We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT). In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores. This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples. Conclusion We present a new corpus, developed to facilitate the processing of case reports. The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers. Where applicable, relationships between entities are also annotated. Additionally, we annotate discontinuous entities with a special relationship type (discontinuous). The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP. Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”. Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations. We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds. A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20. Acknowledgments The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no. 03WKDA1A), see http://qurator.ai. We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein.
How was annotation performed?
f2e8497aa16327aa297a7f9f7d156e485fe33945
How Ġwas Ġannotation Ġperformed ?
Exper ienced Ġmedical Ġdoctors Ġused Ġa Ġlinguistic Ġannotation Ġtool Ġto Ġannot ate Ġentities .
<s> How Ġwas Ġannotation Ġperformed ? </s> Introduction </s> The Ġautomatic Ġprocessing Ġof Ġmedical Ġtexts Ġand Ġdocuments Ġplays Ġan Ġincreasingly Ġimportant Ġrole Ġin Ġthe Ġrecent Ġdevelopment Ġof Ġthe Ġdigital Ġhealth Ġarea . ĠTo Ġenable Ġdedicated ĠNatural ĠLanguage ĠProcessing Ġ( N LP ) Ġthat Ġis Ġhighly Ġaccurate Ġwith Ġrespect Ġto Ġmedically Ġrelevant Ġcategories , Ġmanually Ġannot ated Ġdata Ġfrom Ġthis Ġdomain Ġis Ġneeded . ĠOne Ġcategory Ġof Ġhigh Ġinterest Ġand Ġrelevance Ġare Ġmedical Ġentities . ĠOnly Ġvery Ġfew Ġannot ated Ġcorpor a Ġin Ġthe Ġmedical Ġdomain Ġexist . ĠMany Ġof Ġthem Ġfocus Ġon Ġthe Ġrelation Ġbetween Ġchemicals Ġand Ġdiseases Ġor Ġproteins Ġand Ġdiseases , Ġsuch Ġas Ġthe ĠBC 5 C DR Ġcorpus ĠB IB REF 0 , Ġthe ĠCompar ative ĠToxic ogen omics ĠDatabase ĠB IB REF 1 , Ġthe ĠF SU ĠPR ote in ĠGE ne Ġcorpus ĠB IB REF 2 Ġor Ġthe ĠAD E Ġ( ad verse Ġdrug Ġeffect ) Ġcorpus ĠB IB REF 3 . ĠThe ĠNC BI ĠDisease ĠCorpus ĠB IB REF 4 Ġcontains Ġcondition Ġmention Ġannotations Ġalong Ġwith Ġannotations Ġof Ġsymptoms . ĠSeveral Ġnew Ġcorpor a Ġof Ġannot ated Ġcase Ġreports Ġwere Ġmade Ġavailable Ġrecently . Ġgrou in - et al - 2019 - clinical Ġpresented Ġa Ġcorpus Ġwith Ġmedical Ġentity Ġannotations Ġof Ġclinical Ġcases Ġwritten Ġin ĠFrench , Ġcop d P hen otype Ġpresented Ġa Ġcorpus Ġfocusing Ġon Ġphen otyp ic Ġinformation Ġfor Ġchronic Ġobstruct ive Ġpulmonary Ġdisease Ġwhile Ġ10 . 10 93 / database / bay 143 Ġpresented Ġa Ġcorpus Ġfocusing Ġon Ġidentifying Ġmain Ġfinding Ġsentences Ġin Ġcase Ġreports . </s> The Ġcorpus Ġmost Ġcomparable Ġto Ġours Ġis Ġthe ĠFrench Ġcorpus Ġof Ġclinical Ġcase Ġreports Ġby Ġgrou in - et al - 2019 - clinical . ĠTheir Ġannotations Ġare Ġbased Ġon ĠU ML S Ġsemantic Ġtypes . ĠEven Ġthough Ġthere Ġis Ġan Ġoverlap Ġin Ġannot ated Ġentities , Ġsemantic Ġclasses Ġare Ġnot Ġthe Ġsame . ĠLab Ġresults Ġare Ġsubs umed Ġunder Ġfindings Ġin Ġour Ġcorpus Ġand Ġare Ġnot Ġannot ated Ġas Ġtheir Ġown Ġclass . ĠFactors Ġextend Ġbeyond Ġgender Ġand Ġage Ġand Ġdescribe Ġany Ġkind Ġof Ġrisk Ġfactor Ġthat Ġcontributes Ġto Ġa Ġhigher Ġprobability Ġof Ġhaving Ġa Ġcertain Ġdisease . ĠOur Ġcorpus Ġincludes Ġadditional Ġentity Ġtypes . ĠWe Ġannot ate Ġconditions , Ġfindings Ġ( including Ġmedical Ġfindings Ġsuch Ġas Ġblood Ġvalues ), Ġfactors , Ġand Ġalso Ġmodifiers Ġwhich Ġindicate Ġthe Ġneg ation Ġof Ġother Ġentities Ġas Ġwell Ġas Ġcase Ġentities , Ġi . Ġe ., Ġentities Ġspecific Ġto Ġone Ġcase Ġreport . ĠAn Ġoverview Ġis Ġavailable Ġin ĠTable ĠT AB REF 3 . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation Ġtasks </s> Case Ġreports Ġare Ġstandardized Ġin Ġthe ĠCARE Ġguidelines ĠB IB REF 5 . ĠThey Ġrepresent Ġa Ġdetailed Ġdescription Ġof Ġthe Ġsymptoms , Ġsigns , Ġdiagnosis , Ġtreatment , Ġand Ġfollow - up Ġof Ġan Ġindividual Ġpatient . ĠWe Ġfocus Ġon Ġdocuments Ġfreely Ġavailable Ġthrough ĠPubMed ĠCentral Ġ( PM C ). ĠThe Ġpresentation Ġof Ġthe Ġpatient 's Ġcase Ġcan Ġusually Ġbe Ġfound Ġin Ġa Ġdedicated Ġsection Ġor Ġthe Ġabstract . ĠWe Ġperform Ġa Ġmanual Ġannotation Ġof Ġall Ġmentions Ġof Ġcase Ġentities , Ġconditions , Ġfindings , Ġfactors Ġand Ġmodifiers . ĠThe Ġscope Ġof Ġour Ġmanual Ġannotation Ġis Ġlimited Ġto Ġthe Ġpresentation Ġof Ġa Ġpatient 's Ġsigns Ġand Ġsymptoms . ĠIn Ġaddition , Ġwe Ġannot ate Ġthe Ġtitle Ġof Ġthe Ġcase Ġreport . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation ĠGuidelines </s> We Ġannot ate Ġthe Ġfollowing Ġentities : </s> case Ġentity Ġmarks Ġthe Ġmention Ġof Ġa Ġpatient . ĠA Ġcase Ġreport Ġcan Ġcontain Ġmore Ġthan Ġone Ġcase Ġdescription . ĠTherefore , Ġall Ġthe Ġfindings , Ġfactors Ġand Ġconditions Ġrelated Ġto Ġone Ġpatient Ġare Ġlinked Ġto Ġthe Ġrespective Ġcase Ġentity . ĠWithin Ġthe Ġtext , Ġthis Ġentity Ġis Ġoften Ġrepresented Ġby Ġthe Ġfirst Ġmention Ġof Ġthe Ġpatient Ġand Ġoverl aps Ġwith Ġthe Ġfactor Ġannotations Ġwhich Ġcan , Ġe . Ġg ., Ġmark Ġsex Ġand Ġage Ġ( cf . ĠFigure ĠFIG REF 12 ). </s> condition Ġmarks Ġa Ġmedical Ġdisease Ġsuch Ġas Ġpneum oth or ax Ġor Ġdisl ocation Ġof Ġthe Ġshoulder . </s> factor Ġmarks Ġa Ġfeature Ġof Ġa Ġpatient Ġwhich Ġmight Ġinfluence Ġthe Ġprobability Ġfor Ġa Ġspecific Ġdiagnosis . ĠIt Ġcan Ġbe Ġimmutable Ġ( e . Ġg ., Ġsex Ġand Ġage ), Ġdescribe Ġa Ġspecific Ġmedical Ġhistory Ġ( e . Ġg ., Ġdiabetes Ġmell itus ) Ġor Ġa Ġbehaviour Ġ( e . Ġg ., Ġsmoking ). </s> finding Ġmarks Ġa Ġsign Ġor Ġsymptom Ġa Ġpatient Ġshows . ĠThis Ġcan Ġbe Ġvisible Ġ( e . Ġg ., Ġrash ), Ġdescribed Ġby Ġa Ġpatient Ġ( e . Ġg ., Ġheadache ) Ġor Ġmeasurable Ġ( e . Ġg ., Ġdecreased Ġblood Ġglucose Ġlevel ). </s> neg ation Ġmodifier Ġexplicitly Ġnegate Ġthe Ġpresence Ġof Ġa Ġcertain Ġfinding Ġusually Ġsetting Ġthe Ġcase Ġapart Ġfrom Ġcommon Ġcases . </s> We Ġalso Ġannot ate Ġrelations Ġbetween Ġthese Ġentities , Ġwhere Ġapplicable . ĠSince Ġwe Ġwork Ġon Ġcase Ġdescriptions , Ġthe Ġanchor Ġpoint Ġof Ġthese Ġrelations Ġis Ġthe Ġcase Ġthat Ġis Ġdescribed . ĠThe Ġfollowing Ġrelations Ġare Ġannot ated : </s> has Ġrelations Ġexist Ġbetween Ġa Ġcase Ġentity Ġand Ġfactor , Ġfinding Ġor Ġcondition Ġentities . </s> mod ifies Ġrelations Ġexist Ġbetween Ġneg ation Ġmodifiers Ġand Ġfindings . </s> ca uses Ġrelations Ġexist Ġbetween Ġconditions Ġand Ġfindings . </s> Example Ġannotations Ġare Ġshown Ġin ĠFigure ĠFIG REF 16 . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn not ators </s> We Ġasked Ġmedical Ġdoctors Ġexperienced Ġin Ġextracting Ġknowledge Ġrelated Ġto Ġmedical Ġentities Ġfrom Ġtexts Ġto Ġannot ate Ġthe Ġentities Ġdescribed Ġabove . ĠInitially , Ġwe Ġasked Ġfour Ġannot ators Ġto Ġtest Ġour Ġguidelines Ġon Ġtwo Ġtexts . ĠSub sequently , Ġidentified Ġissues Ġwere Ġdiscussed Ġand Ġresolved . ĠFollowing Ġthis Ġpilot Ġannotation Ġphase , Ġwe Ġasked Ġtwo Ġdifferent Ġannot ators Ġto Ġannot ate Ġtwo Ġcase Ġreports Ġaccording Ġto Ġour Ġguidelines . ĠThe Ġsame Ġannot ators Ġannot ated Ġan Ġoverall Ġcollection Ġof Ġ53 Ġcase Ġreports . </s> Inter - annot ator Ġagreement Ġis Ġcalculated Ġbased Ġon Ġtwo Ġcase Ġreports . ĠWe Ġreach Ġa ĠCohen 's Ġk appa ĠB IB REF 6 Ġof Ġ0 . 68 . ĠDis ag reements Ġmainly Ġappear Ġfor Ġfindings Ġthat Ġare Ġrather Ġun specific Ġsuch Ġas ĠShe Ġno Ġlonger Ġeats Ġout Ġwith Ġfriends Ġwhich Ġcan Ġbe Ġseen Ġas Ġa Ġfinding Ġreferring Ġto ĠâĢ ľ avoid ance Ġbehaviour âĢ Ŀ . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠAn notation ĠTools Ġand ĠFormat </s> The Ġannotation Ġwas Ġperformed Ġusing ĠWeb An no ĠB IB REF 7 , Ġa Ġweb - based Ġtool Ġfor Ġlinguistic Ġannotation . ĠThe Ġannot ators Ġcould Ġchoose Ġbetween Ġa Ġpre - annot ated Ġversion Ġor Ġa Ġblank Ġversion Ġof Ġeach Ġtext . ĠThe Ġpre - annot ated Ġversions Ġcontained Ġsuggested Ġentity Ġspans Ġbased Ġon Ġstring Ġmatches Ġfrom Ġlists Ġof Ġconditions Ġand Ġfindings Ġsyn onym Ġlists . ĠTheir Ġquality Ġvaried Ġwidely Ġthroughout Ġthe Ġcorpus . ĠThe Ġblank Ġversion Ġwas Ġpreferred Ġby Ġthe Ġannot ators . ĠWe Ġdistribute Ġthe Ġcorpus Ġin ĠBio C ĠJSON Ġformat . ĠBio C Ġwas Ġchosen Ġas Ġit Ġallows Ġus Ġto Ġcapture Ġthe Ġcomplexities Ġof Ġthe Ġannotations Ġin Ġthe Ġbiomedical Ġdomain . ĠIt Ġrepresented Ġeach Ġdocuments Ġproperties Ġranging Ġfrom Ġfull Ġtext , Ġindividual Ġpassages / sent ences Ġalong Ġwith Ġcaptured Ġannotations Ġand Ġrelationships Ġin Ġan Ġorganized Ġmanner . ĠBio C Ġis Ġbased Ġon Ġcharacter Ġoffsets Ġof Ġannotations Ġand Ġallows Ġthe Ġstacking Ġof Ġdifferent Ġlayers . </s> A ĠCorpus Ġof ĠMedical ĠCase ĠReports Ġwith ĠMedical ĠEntity ĠAn notation Ġ: :: ĠCorpus ĠOverview </s> The Ġcorpus Ġconsists Ġof Ġ53 Ġdocuments , Ġwhich Ġcontain Ġan Ġaverage Ġnumber Ġof Ġ156 . 1 Ġsentences Ġper Ġdocument , Ġeach Ġwith Ġ19 . 55 Ġtokens Ġon Ġaverage . ĠThe Ġcorpus Ġcomprises Ġ8 , 275 Ġsentences Ġand Ġ167 , 7 39 Ġwords Ġin Ġtotal . ĠHowever , Ġas Ġmentioned Ġabove , Ġonly Ġcase Ġpresentation Ġsections , Ġhead ings Ġand Ġabstract s Ġare Ġannot ated . ĠThe Ġnumbers Ġof Ġannot ated Ġentities Ġare Ġsummarized Ġin ĠTable ĠT AB REF 24 . </s> Find ings Ġare Ġthe Ġmost Ġfrequently Ġannot ated Ġtype Ġof Ġentity . ĠThis Ġmakes Ġsense Ġgiven Ġthat Ġfindings Ġpaint Ġa Ġclinical Ġpicture Ġof Ġthe Ġpatient 's Ġcondition . ĠThe Ġnumber Ġof Ġtokens Ġper Ġentity Ġranges Ġfrom Ġone Ġtoken Ġfor Ġall Ġtypes Ġto Ġ5 Ġtokens Ġfor Ġcases Ġ( average Ġlength Ġ3 . 1 ), Ġnine Ġtokens Ġfor Ġconditions Ġ( average Ġlength Ġ2 . 0 ), Ġ16 Ġtokens Ġfor Ġfactors Ġ( average Ġlength Ġ2 . 5 ), Ġ25 Ġtokens Ġfor Ġfindings Ġ( average Ġlength Ġ2 . 6 ) Ġand Ġ18 Ġtokens Ġfor Ġmodifiers Ġ( average Ġlength Ġ1 . 4 ) Ġ( cf . ĠTable ĠT AB REF 24 ). ĠExamples Ġof Ġrather Ġlong Ġentities Ġare Ġgiven Ġin ĠTable ĠT AB REF 25 . </s> Ent ities Ġcan Ġappear Ġin Ġa Ġdiscontin uous Ġway . ĠWe Ġmodel Ġthis Ġas Ġa Ġrelation Ġbetween Ġtwo Ġspans Ġwhich Ġwe Ġcall ĠâĢ ľ disc ont in uous âĢ Ŀ Ġ( cf . ĠFigure ĠFIG REF 26 ). ĠEspecially Ġfindings Ġoften Ġappear Ġas Ġdiscontin uous Ġentities , Ġwe Ġfound Ġ5 43 Ġdiscontin uous Ġfinding Ġrelations . ĠThe Ġnumbers Ġfor Ġconditions Ġand Ġfactors Ġare Ġlower Ġwith Ġseven Ġand Ġtwo , Ġrespectively . ĠEnt ities Ġcan Ġalso Ġbe Ġnested Ġwithin Ġone Ġanother . ĠThis Ġhappens Ġeither Ġwhen Ġthe Ġspan Ġof Ġone Ġannotation Ġis Ġcompletely Ġembedded Ġin Ġthe Ġspan Ġof Ġanother Ġannotation Ġ( fully - n ested ; Ġcf . ĠFigure ĠFIG REF 12 ), Ġor Ġwhen Ġthere Ġis Ġa Ġpartial Ġoverlapping Ġbetween Ġthe Ġspans Ġof Ġtwo Ġdifferent Ġentities Ġ( part ially - n ested ; Ġcf . ĠFigure ĠFIG REF 12 ). ĠThere Ġis Ġa Ġhigh Ġnumber Ġof Ġinter - sent ential Ġrelations Ġin Ġthe Ġcorpus Ġ( cf . ĠTable ĠT AB REF 27 ). ĠThis Ġcan Ġbe Ġexplained Ġby Ġthe Ġfact Ġthat Ġthe Ġcase Ġentity Ġoccurs Ġearly Ġin Ġeach Ġdocument ; Ġfurthermore , Ġit Ġis Ġrelated Ġto Ġfinding Ġand Ġfactor Ġannotations Ġthat Ġare Ġdistributed Ġacross Ġdifferent Ġsentences . </s> The Ġmost Ġfrequently Ġannot ated Ġrelation Ġin Ġour Ġcorpus Ġis Ġthe Ġhas - relation Ġbetween Ġa Ġcase Ġentity Ġand Ġthe Ġfindings Ġrelated Ġto Ġthat Ġcase . ĠThis Ġcorrelates Ġwith Ġthe Ġhigh Ġnumber Ġof Ġfinding Ġentities . ĠThe Ġrelations Ġcontained Ġin Ġour Ġcorpus Ġare Ġsummarized Ġin ĠTable ĠT AB REF 27 . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports </s> We Ġevaluate Ġthe Ġcorpus Ġusing ĠNamed ĠEntity ĠRecogn ition Ġ( NER ), Ġi . Ġe ., Ġthe Ġtask Ġof Ġfinding Ġmentions Ġof Ġconcepts Ġof Ġinterest Ġin Ġun struct ured Ġtext . ĠWe Ġfocus Ġon Ġdetecting Ġcases , Ġconditions , Ġfactors , Ġfindings Ġand Ġmodifiers Ġin Ġcase Ġreports Ġ( cf . ĠSection ĠSEC REF 6 ). ĠWe Ġapproach Ġthis Ġas Ġa Ġsequence Ġlabeling Ġproblem . ĠFour Ġsystems Ġwere Ġdeveloped Ġto Ġoffer Ġcomparable Ġrobust Ġbas elines . </s> The Ġoriginal Ġdocuments Ġare Ġpre - process ed Ġ( sent ence Ġsplitting Ġand Ġtoken ization Ġwith ĠSc isp a Cy ). ĠWe Ġdo Ġnot Ġperform Ġstop Ġword Ġremoval Ġor Ġlower - c asing Ġof Ġthe Ġtokens . ĠThe ĠB IO Ġlabeling Ġscheme Ġis Ġused Ġto Ġcapture Ġthe Ġorder Ġof Ġtokens Ġbelonging Ġto Ġthe Ġsame Ġentity Ġtype Ġand Ġenable Ġspan - level Ġdetection Ġof Ġentities . ĠDetection Ġof Ġnested Ġand / or Ġdiscontin uous Ġentities Ġis Ġnot Ġsupported . ĠThe Ġannot ated Ġcorpus Ġis Ġrandomized Ġand Ġsplit Ġin Ġfive Ġfolds Ġusing Ġsc ik it - learn ĠB IB REF 9 . ĠEach Ġfold Ġhas Ġa Ġtrain , Ġtest Ġand Ġdev Ġsplit Ġwith Ġthe Ġtest Ġsplit Ġdefined Ġas Ġ. 15 % Ġof Ġthe Ġtrain Ġsplit . ĠThis Ġensures Ġcompar ability Ġbetween Ġthe Ġpresented Ġsystems . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠCond itional ĠRandom ĠFields </s> Cond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 10 Ġare Ġa Ġstandard Ġapproach Ġwhen Ġdealing Ġwith Ġsequential Ġdata Ġin Ġthe Ġcontext Ġof Ġsequence Ġlabeling . ĠWe Ġuse Ġa Ġcombination Ġof Ġlinguistic Ġand Ġsemantic Ġfeatures , Ġwith Ġa Ġcontext Ġwindow Ġof Ġsize Ġfive , Ġto Ġdescribe Ġeach Ġof Ġthe Ġtokens Ġand Ġthe Ġdependencies Ġbetween Ġthem . ĠHyper - param eter Ġoptimization Ġis Ġperformed Ġusing Ġrandomized Ġsearch Ġand Ġcross Ġvalidation . ĠSpan - based ĠF 1 Ġscore Ġis Ġused Ġas Ġthe Ġoptimization Ġmetric . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠBi L ST M - CR F </s> Prior Ġto Ġthe Ġemergence Ġof Ġdeep Ġneural Ġlanguage Ġmodels , ĠBi L ST M - CR F Ġmodels ĠB IB REF 11 Ġhad Ġachieved Ġstate - of - the - art Ġresults Ġfor Ġthe Ġtask Ġof Ġsequence Ġlabeling . ĠWe Ġuse Ġa ĠBi L ST M - CR F Ġmodel Ġwith Ġboth Ġword - level Ġand Ġcharacter - level Ġinput . ĠBio Word V ec ĠB IB REF 12 Ġpre - trained Ġword Ġembed d ings Ġare Ġused Ġin Ġthe Ġembed ding Ġlayer Ġfor Ġthe Ġinput Ġrepresentation . ĠA Ġbid irection al ĠL ST M Ġlayer Ġis Ġapplied Ġto Ġa Ġmultiplication Ġof Ġthe Ġtwo Ġinput Ġrepresentations . ĠFinally , Ġa ĠCR F Ġlayer Ġis Ġapplied Ġto Ġpredict Ġthe Ġsequence Ġof Ġlabels . ĠDrop out Ġand ĠL 1 / L 2 Ġregular ization Ġis Ġused Ġwhere Ġapplicable . ĠHe Ġ( un iform ) Ġinitialization ĠB IB REF 13 Ġis Ġused Ġto Ġinitialize Ġthe Ġkernels Ġof Ġthe Ġindividual Ġlayers . ĠAs Ġthe Ġloss Ġmetric , ĠCR F - based Ġloss Ġis Ġused , Ġwhile Ġoptimizing Ġthe Ġmodel Ġbased Ġon Ġthe ĠCR F ĠV iter bi Ġaccuracy . ĠAdditionally , Ġspan - based ĠF 1 Ġscore Ġis Ġused Ġto Ġserial ize Ġthe Ġbest Ġperforming Ġmodel . ĠWe Ġtrain Ġfor Ġa Ġmaximum Ġof Ġ100 Ġepoch s , Ġor Ġuntil Ġan Ġearly Ġstopping Ġcriterion Ġis Ġreached Ġ( no Ġchange Ġin Ġvalidation Ġloss Ġvalue Ġgr ater Ġthan Ġ0 . 01 Ġfor Ġten Ġconsecutive Ġepoch s ). ĠFurthermore , ĠAdam ĠB IB REF 14 Ġis Ġused Ġas Ġthe Ġoptim izer . ĠThe Ġlearning Ġrate Ġis Ġreduced Ġby Ġa Ġfactor Ġof Ġ0 . 3 Ġin Ġcase Ġno Ġsignificant Ġincrease Ġof Ġthe Ġoptimization Ġmetric Ġis Ġachieved Ġin Ġthree Ġconsecutive Ġepoch s . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠMulti - Task ĠLearning </s> Multi - Task ĠLearning Ġ( MT L ) ĠB IB REF 15 Ġhas Ġbecome Ġpopular Ġwith Ġthe Ġprogress Ġin Ġdeep Ġlearning . ĠThis Ġmodel Ġfamily Ġis Ġcharacterized Ġby Ġsimultaneous Ġoptimization Ġof Ġmultiple Ġloss Ġfunctions Ġand Ġtransfer Ġof Ġknowledge Ġachieved Ġthis Ġway . ĠThe Ġknowledge Ġis Ġtransferred Ġthrough Ġthe Ġuse Ġof Ġone Ġor Ġmultiple Ġshared Ġlayers . ĠThrough Ġfinding Ġsupporting Ġpatterns Ġin Ġrelated Ġtasks , ĠM TL Ġprovides Ġbetter Ġgeneral ization Ġon Ġunseen Ġcases Ġand Ġthe Ġmain Ġtasks Ġwe Ġare Ġtrying Ġto Ġsolve . </s> We Ġrely Ġon Ġthe Ġmodel Ġpresented Ġby Ġbe k oul is 2018 j oint Ġand Ġreuse Ġthe Ġimplementation Ġprovided Ġby Ġthe Ġauthors . ĠThe Ġmodel Ġjointly Ġtrains Ġtwo Ġobjectives Ġsupported Ġby Ġthe Ġdataset : Ġthe Ġmain Ġtask Ġof ĠN ER Ġand Ġa Ġsupporting Ġtask Ġof ĠRel ation ĠExt raction Ġ( RE ). ĠTwo Ġseparate Ġmodels Ġare Ġdeveloped Ġfor Ġeach Ġof Ġthe Ġtasks . ĠThe ĠN ER Ġtask Ġis Ġsolved Ġwith Ġthe Ġhelp Ġof Ġa ĠBi L ST M - CR F Ġmodel , Ġsimilar Ġto Ġthe Ġone Ġpresented Ġin ĠSection ĠSEC REF 32 ĠThe ĠRE Ġtask Ġis Ġsolved Ġby Ġusing Ġa Ġmulti - head Ġselection Ġapproach , Ġwhere Ġeach Ġtoken Ġcan Ġhave Ġnone Ġor Ġmore Ġrelationships Ġto Ġin - sent ence Ġtokens . ĠAdditionally , Ġthis Ġmodel Ġalso Ġlever ages Ġthe Ġoutput Ġof Ġthe ĠN ER Ġbranch Ġmodel Ġ( the ĠCR F Ġprediction ) Ġto Ġlearn Ġlabel Ġembed d ings . ĠShared Ġlayers Ġconsist Ġof Ġa Ġconc aten ation Ġof Ġword Ġand Ġcharacter Ġembed d ings Ġfollowed Ġby Ġtwo Ġbid irection al ĠL ST M Ġlayers . ĠWe Ġkeep Ġmost Ġof Ġthe Ġparameters Ġsuggested Ġby Ġthe Ġauthors Ġand Ġchange Ġ( 1 ) Ġthe Ġnumber Ġof Ġtraining Ġepoch s Ġto Ġ100 Ġto Ġallow Ġthe Ġcomparison Ġto Ġother Ġdeep Ġlearning Ġapproaches Ġin Ġthis Ġwork , Ġ( 2 ) Ġuse Ġlabel Ġembed d ings Ġof Ġsize Ġ64 , Ġ( 3 ) Ġallow Ġgradient Ġclipping Ġand Ġ( 4 ) Ġuse Ġ$ d = 0 . 8 $ Ġas Ġthe Ġpre - trained Ġword Ġembed ding Ġdrop out Ġand Ġ$ d = 0 . 5 $ Ġfor Ġall Ġother Ġdrop outs . Ġ$\ eta Ġ= 1 ^{ - 3 } $ Ġis Ġused Ġas Ġthe Ġlearning Ġrate Ġwith Ġthe ĠAdam Ġoptim izer Ġand Ġtan h Ġactivation Ġfunctions Ġacross Ġlayers . ĠAlthough Ġit Ġis Ġpossible Ġto Ġuse Ġadvers arial Ġtraining ĠB IB REF 16 , Ġwe Ġomit Ġfrom Ġusing Ġit . ĠWe Ġalso Ġomit Ġthe Ġpublication Ġof Ġresults Ġfor Ġthe Ġtask Ġof ĠRE Ġas Ġwe Ġconsider Ġit Ġto Ġbe Ġa Ġsupporting Ġtask Ġand Ġno Ġother Ġcompeting Ġapproaches Ġhave Ġbeen Ġdeveloped . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠBio BER T </s> Deep Ġneural Ġlanguage Ġmodels Ġhave Ġrecently Ġevolved Ġto Ġa Ġsuccessful Ġmethod Ġfor Ġrepresenting Ġtext . ĠIn Ġparticular , ĠBid irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTransformers Ġ( BER T ) Ġoutper formed Ġprevious Ġstate - of - the - art Ġmethods Ġby Ġa Ġlarge Ġmargin Ġon Ġvarious ĠN LP Ġtasks ĠB IB REF 17 . ĠFor Ġour Ġexperiments , Ġwe Ġuse ĠBio BER T , Ġan Ġadaptation Ġof ĠB ERT Ġfor Ġthe Ġbiomedical Ġdomain , Ġpre - trained Ġon ĠPubMed Ġabstract s Ġand ĠPM C Ġfull - text Ġarticles ĠB IB REF 18 . ĠThe ĠB ERT Ġarchitecture Ġfor Ġder iving Ġtext Ġrepresentations Ġuses Ġ12 Ġhidden Ġlayers , Ġconsisting Ġof Ġ768 Ġunits Ġeach . ĠFor ĠN ER , Ġtoken Ġlevel ĠB IO - tag Ġprobabilities Ġare Ġcomputed Ġwith Ġa Ġsingle Ġoutput Ġlayer Ġbased Ġon Ġthe Ġrepresentations Ġfrom Ġthe Ġlast Ġlayer Ġof ĠB ERT . ĠWe Ġfine - t une Ġthe Ġmodel Ġon Ġthe Ġentity Ġrecognition Ġtask Ġduring Ġfour Ġtraining Ġepoch s Ġwith Ġbatch Ġsize Ġ$ b = 32 $, Ġdrop out Ġprobability Ġ$ d = 0 . 1 $ Ġand Ġlearning Ġrate Ġ$\ eta Ġ= 2 ^{ - 5 } $. ĠThese Ġhyper - param eters Ġare Ġproposed Ġby ĠDev lin 2018 Ġfor ĠB ERT Ġfine - tun ing . </s> Bas eline Ġsystems Ġfor ĠNamed ĠEntity ĠRecogn ition Ġin Ġmedical Ġcase Ġreports Ġ: :: ĠEvaluation </s> To Ġevaluate Ġthe Ġperformance Ġof Ġthe Ġfour Ġsystems , Ġwe Ġcalculate Ġthe Ġspan - level Ġprecision Ġ( P ), Ġrecall Ġ( R ) Ġand ĠF 1 Ġscores , Ġalong Ġwith Ġcorresponding Ġmicro Ġand Ġmacro Ġscores . ĠThe Ġreported Ġvalues Ġare Ġshown Ġin ĠTable ĠT AB REF 29 Ġand Ġare Ġaveraged Ġover Ġfive Ġfolds , Ġutil ising Ġthe Ġseq eval Ġframework . </s> With Ġa Ġmacro Ġavg . ĠF 1 - score Ġof Ġ0 . 59 , ĠM TL Ġachieves Ġthe Ġbest Ġresult Ġwith Ġa Ġsignificant Ġmargin Ġcompared Ġto ĠCR F , ĠBi L ST M - CR F Ġand ĠB ERT . ĠThis Ġconfirms Ġthe Ġusefulness Ġof Ġjointly Ġtraining Ġmultiple Ġobjectives Ġ( min im izing Ġmultiple Ġloss Ġfunctions ), Ġand Ġenabling Ġknowledge Ġtransfer , Ġespecially Ġin Ġa Ġsetting Ġwith Ġlimited Ġdata Ġ( which Ġis Ġusually Ġthe Ġcase Ġin Ġthe Ġbiomedical ĠN LP Ġdomain ). ĠThis Ġresult Ġalso Ġsuggest Ġthe Ġusefulness Ġof ĠBio BER T Ġfor Ġother Ġbiomedical Ġdatasets Ġas Ġreported Ġby ĠLee 2019 . ĠDespite Ġbeing Ġa Ġrather Ġstandard Ġapproach , ĠCR F Ġoutper forms Ġthe Ġmore Ġelaborated ĠBi L ST M - CR F , Ġpresumably Ġdue Ġto Ġdata Ġscarcity Ġand Ġclass Ġimbalance . ĠWe Ġhypothes ize Ġthat Ġan Ġincrease Ġin Ġtraining Ġdata Ġwould Ġyield Ġbetter Ġresults Ġfor ĠBi L ST M - CR F Ġbut Ġnot Ġoutper form Ġtransfer Ġlearning Ġapproach Ġof ĠM TL Ġ( or Ġeven ĠBio BER T ). ĠIn Ġcontrast Ġto Ġother Ġcommon ĠN ER Ġcorpor a , Ġlike ĠCo N LL Ġ2003 , Ġeven Ġthe Ġbest Ġbaseline Ġsystem Ġonly Ġachieves Ġrelatively Ġlow Ġscores . ĠThis Ġoutcome Ġis Ġdue Ġto Ġthe Ġinherent Ġdifficulty Ġof Ġthe Ġtask Ġ( annot ators Ġare Ġexperienced Ġmedical Ġdoctors ) Ġand Ġthe Ġsmall Ġnumber Ġof Ġtraining Ġsamples . </s> Conclusion </s> We Ġpresent Ġa Ġnew Ġcorpus , Ġdeveloped Ġto Ġfacilitate Ġthe Ġprocessing Ġof Ġcase Ġreports . ĠThe Ġcorpus Ġfocuses Ġon Ġfive Ġdistinct Ġentity Ġtypes : Ġcases , Ġconditions , Ġfactors , Ġfindings Ġand Ġmodifiers . ĠWhere Ġapplicable , Ġrelationships Ġbetween Ġentities Ġare Ġalso Ġannot ated . ĠAdditionally , Ġwe Ġannot ate Ġdiscontin uous Ġentities Ġwith Ġa Ġspecial Ġrelationship Ġtype Ġ( disc ont in uous ). ĠThe Ġcorpus Ġpresented Ġin Ġthis Ġpaper Ġis Ġthe Ġvery Ġfirst Ġof Ġits Ġkind Ġand Ġa Ġvaluable Ġaddition Ġto Ġthe Ġscarce Ġnumber Ġof Ġcorpor a Ġavailable Ġin Ġthe Ġfield Ġof Ġbiomedical ĠN LP . ĠIts Ġcomplexity , Ġgiven Ġthe Ġdiscontin uous Ġnature Ġof Ġentities Ġand Ġa Ġhigh Ġnumber Ġof Ġnested Ġand Ġmulti - label Ġentities , Ġposes Ġnew Ġchallenges Ġfor ĠN LP Ġmethods Ġapplied Ġfor ĠN ER Ġand Ġcan , Ġhence , Ġbe Ġa Ġvaluable Ġsource Ġfor Ġinsights Ġinto Ġwhat Ġentities ĠâĢ ľ look Ġlike Ġin Ġthe Ġwild âĢ Ŀ . ĠMoreover , Ġit Ġcan Ġserve Ġas Ġa Ġplayground Ġfor Ġnew Ġmodelling Ġtechniques Ġsuch Ġas Ġthe Ġresolution Ġof Ġdiscontin uous Ġentities Ġas Ġwell Ġas Ġmulti - task Ġlearning Ġgiven Ġthe Ġcombination Ġof Ġentities Ġand Ġtheir Ġrelations . ĠWe Ġprovide Ġan Ġevaluation Ġof Ġfour Ġdistinct ĠN ER Ġsystems Ġthat Ġwill Ġserve Ġas Ġrobust Ġbas elines Ġfor Ġfuture Ġwork Ġbut Ġwhich Ġare , Ġas Ġof Ġyet , Ġunable Ġto Ġsolve Ġall Ġthe Ġcomplex Ġchallenges Ġthis Ġdataset Ġholds . ĠA Ġfunctional Ġservice Ġbased Ġon Ġthe Ġpresented Ġcorpus Ġis Ġcurrently Ġbeing Ġintegrated , Ġas Ġa ĠN ER Ġservice , Ġin Ġthe ĠQ UR ATOR Ġplatform ĠB IB REF 20 . </s> Acknowled gments </s> The Ġresearch Ġpresented Ġin Ġthis Ġarticle Ġis Ġfunded Ġby Ġthe ĠGerman ĠFederal ĠMinistry Ġof ĠEducation Ġand ĠResearch Ġ( B MB F ) Ġthrough Ġthe Ġproject ĠQ UR ATOR Ġ( Un ter ne h men ĠRegion , ĠW ach st ums k ern , Ġgrant Ġno . Ġ03 W K DA 1 A ), Ġsee Ġhttp :// q ur ator . ai . ĠWe Ġwant Ġto Ġthank Ġour Ġmedical Ġexperts Ġfor Ġtheir Ġhelp Ġannot ating Ġthe Ġdata Ġset , Ġespecially ĠAsh lee ĠFin ck h Ġand ĠSophie ĠKl op f enstein .