MIMIC-III
EXTRACTIVE
MIMIC-III
EXTRACTIVE
MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient's time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression.
MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient's time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression.
1810.12085
Introduction </s> Sum mar ization Ġof Ġpatient Ġinformation Ġis Ġessential Ġto Ġthe Ġpractice Ġof Ġmedicine . ĠClin icians Ġmust Ġsynthes ize Ġinformation Ġfrom Ġdiverse Ġdata Ġsources Ġto Ġcommunicate Ġwith Ġcolleagues Ġand Ġprovide Ġcoordinated Ġcare . ĠExamples Ġof Ġclinical Ġsummar ization Ġare Ġabundant Ġin Ġpractice ; Ġpatient Ġhand off Ġsumm aries Ġfacilitate Ġprovider Ġshift Ġchange , Ġprogress Ġnotes Ġprovide Ġa Ġdaily Ġstatus Ġupdate Ġfor Ġa Ġpatient , Ġoral Ġcase Ġpresentations Ġenable Ġtransfer Ġof Ġinformation Ġfrom Ġovernight Ġadmission Ġto Ġthe Ġcare Ġteam Ġand Ġattending , Ġand Ġdischarge Ġsumm aries Ġprovide Ġinformation Ġabout Ġa Ġpatient 's Ġhospital Ġvisit Ġto Ġtheir Ġprimary Ġcare Ġphysician Ġand Ġother Ġoutpatient Ġproviders ĠB IB REF 0 Ġ. </s> In form al , Ġun struct ured , Ġor Ġpoor Ġquality Ġsumm aries Ġcan Ġlead Ġto Ġcommunication Ġfailures Ġand Ġeven Ġmedical Ġerrors , Ġyet Ġclinical Ġinstruction Ġon Ġhow Ġto Ġformulate Ġclinical Ġsumm aries Ġis Ġad Ġhoc Ġand Ġinformal . ĠNon - existent Ġor Ġlimited Ġsearch Ġfunctionality , Ġfragmented Ġdata Ġsources , Ġand Ġlimited Ġvisual izations Ġin Ġelectronic Ġhealth Ġrecords Ġ( E HR s ) Ġmake Ġsummar ization Ġchallenging Ġfor Ġproviders ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFurthermore , Ġwhile Ġdict ation Ġof ĠE HR Ġnotes Ġallows Ġclinicians Ġto Ġmore Ġefficiently Ġdocument Ġinformation Ġat Ġthe Ġpoint Ġof Ġcare , Ġthe Ġstream Ġof Ġconsciousness - like Ġwriting Ġcan Ġhinder Ġthe Ġread ability Ġof Ġnotes . ĠK rip al ani Ġet Ġal . Ġshow Ġthat Ġdischarge Ġsumm aries Ġare Ġoften Ġlacking Ġkey Ġinformation , Ġincluding Ġtreatment Ġprogression Ġand Ġfollow - up Ġprotocols , Ġwhich Ġcan Ġhinder Ġcommunication Ġbetween Ġhospital Ġand Ġcommunity Ġbased Ġclinicians ĠB IB REF 4 Ġ. ĠRecently , ĠSt . ĠThomas ĠHospital Ġin ĠNashville , ĠTN Ġstip ulated Ġthat Ġdischarge Ġnotes Ġbe Ġwritten Ġwithin Ġ48 Ġhours Ġof Ġdischarge Ġfollowing Ġinc idences Ġwhere Ġimproper Ġcare Ġwas Ġgiven Ġto Ġread mitted Ġpatients Ġbecause Ġthe Ġdischarge Ġsummary Ġfor Ġthe Ġprevious Ġadmission Ġwas Ġnot Ġcompleted ĠB IB REF 5 Ġ. </s> Autom ated Ġsummary Ġgeneration Ġhas Ġthe Ġpotential Ġto Ġsave Ġclin ician Ġtime , Ġavoid Ġmedical Ġerrors , Ġand Ġaid Ġclinical Ġdecision Ġmaking . ĠBy Ġorganizing Ġand Ġsynthes izing Ġa Ġpatient 's Ġsalient Ġmedical Ġhistory , Ġalgorithms Ġfor Ġpatient Ġsummar ization Ġcan Ġenable Ġbetter Ġcommunication Ġand Ġcare , Ġparticularly Ġfor Ġchronically Ġill Ġpatients , Ġwhose Ġmedical Ġrecords Ġoften Ġcontain Ġhundreds Ġof Ġnotes . ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġthe Ġautomatic Ġsummar ization Ġof Ġdischarge Ġsummary Ġnotes , Ġwhich Ġare Ġcritical Ġto Ġensuring Ġcontinuity Ġof Ġcare Ġafter Ġhospital ization . ĠWe Ġ( 1 ) Ġprovide Ġan Ġupper Ġbound Ġon Ġextract ive Ġsummar ization Ġby Ġassessing Ġhow Ġmuch Ġinformation Ġin Ġthe Ġdischarge Ġnote Ġcan Ġbe Ġfound Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's ĠE HR Ġnotes Ġand Ġ( 2 ) Ġdevelop Ġa Ġclass ifier Ġfor Ġlabeling Ġthe Ġtopics Ġof Ġhistory Ġof Ġpresent Ġillness Ġnotes , Ġa Ġnarrative Ġsection Ġin Ġthe Ġdischarge Ġsummary Ġthat Ġdescribes Ġthe Ġpatient 's Ġprior Ġhistory Ġand Ġcurrent Ġsymptoms . ĠSuch Ġa Ġclass ifier Ġcan Ġbe Ġused Ġto Ġcreate Ġtopic Ġspecific Ġevaluation Ġsets Ġfor Ġmethods Ġthat Ġperform Ġextract ive Ġsummar ization . ĠThese Ġaims Ġare Ġcritical Ġsteps Ġin Ġultimately Ġdeveloping Ġmethods Ġthat Ġcan Ġautomate Ġdischarge Ġsummary Ġcreation . </s> Related ĠWork </s> In Ġthe Ġbroader Ġfield Ġof Ġsummar ization , Ġautom ization Ġwas Ġmeant Ġto Ġstandard ize Ġoutput Ġwhile Ġalso Ġsaving Ġtime Ġand Ġeffort . ĠPione ering Ġstrategies Ġin Ġsummar ization Ġstarted Ġby Ġextracting Ġ" significant " Ġsentences Ġin Ġthe Ġwhole Ġcorpus Ġto Ġbuild Ġan Ġabstract Ġwhere Ġ" significant " Ġsentences Ġwere Ġdefined Ġby Ġthe Ġnumber Ġof Ġfrequently Ġoccurring Ġwords ĠB IB REF 6 Ġ. ĠThese Ġinitial Ġmethods Ġdid Ġnot Ġconsider Ġword Ġmeaning Ġor Ġsyntax Ġat Ġeither Ġthe Ġsentence Ġor Ġparagraph Ġlevel , Ġwhich Ġmade Ġthem Ġcrude Ġat Ġbest . ĠMore Ġadvanced Ġextract ive Ġhe ur istics Ġlike Ġtopic Ġmodeling ĠB IB REF 7 Ġ, Ġcue Ġword Ġdictionary Ġapproaches ĠB IB REF 8 Ġ, Ġand Ġtitle Ġmethods ĠB IB REF 9 Ġfor Ġscoring Ġcontent Ġin Ġa Ġsentence Ġfollowed Ġsoon Ġafter . ĠFor Ġexample , Ġtopic Ġmodeling Ġextends Ġinitial Ġfrequency Ġmethods Ġby Ġassigning Ġtopics Ġscores Ġby Ġfrequency Ġof Ġtopic Ġsignatures , Ġclust ering Ġsentences Ġwith Ġsimilar Ġtopics , Ġand Ġfinally Ġextracting Ġthe Ġcent roid Ġsentence , Ġwhich Ġis Ġconsidered Ġthe Ġmost Ġrepresentative Ġsentence ĠB IB REF 10 Ġ. ĠRecently , Ġabstract ive Ġsummar ization Ġapproaches Ġusing Ġsequence - to - sequence Ġmethods Ġhave Ġbeen Ġdeveloped Ġto Ġgenerate Ġnew Ġtext Ġthat Ġsynthes izes Ġoriginal Ġtext ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ; Ġhowever , Ġthe Ġfield Ġof Ġabstract ive Ġsummar ization Ġis Ġquite Ġyoung . </s> Ex isting Ġapproaches Ġwithin Ġthe Ġfield Ġof Ġelectronic Ġhealth Ġrecord Ġsummar ization Ġhave Ġlargely Ġbeen Ġextract ive Ġand Ġindicative , Ġmeaning Ġthat Ġsumm aries Ġpoint Ġto Ġimportant Ġpieces Ġin Ġthe Ġoriginal Ġtext Ġrather Ġthan Ġreplacing Ġthe Ġoriginal Ġtext Ġaltogether . ĠFew Ġapproaches Ġhave Ġbeen Ġdeployed Ġin Ġpractice , Ġand Ġeven Ġfewer Ġhave Ġdemonstrated Ġimpact Ġon Ġquality Ġof Ġcare Ġand Ġoutcomes ĠB IB REF 15 Ġ. ĠSum mar ization Ġstrategies Ġhave Ġranged Ġfrom Ġextraction Ġof ĠâĢ ľ relevant âĢ Ŀ Ġsentences Ġfrom Ġthe Ġoriginal Ġtext Ġto Ġform Ġthe Ġsummary ĠB IB REF 16 Ġ, Ġtopic Ġmodeling Ġof ĠE HR Ġnotes Ġusing ĠLat ent ĠDir ich let Ġallocation Ġ( L DA ) Ġor Ġbay esian Ġnetworks ĠB IB REF 15 Ġ, Ġand Ġknowledge Ġbased Ġhe uristic Ġsystems ĠB IB REF 17 Ġ. ĠTo Ġour Ġknowledge , Ġthere Ġis Ġno Ġliterature Ġto Ġdate Ġon Ġextract ive Ġor Ġabstract ive ĠE HR Ġsummar ization Ġusing Ġneural Ġnetworks . </s> Data </s> M IM IC - III Ġis Ġa Ġfreely Ġavailable , Ġde identified Ġdatabase Ġcontaining Ġelectronic Ġhealth Ġrecords Ġof Ġpatients Ġadmitted Ġto Ġan ĠInt ensive ĠCare ĠUnit Ġ( IC U ) Ġat ĠBeth ĠIsrael ĠDe acon ess ĠMedical ĠCenter Ġbetween Ġ2001 Ġand Ġ2012 . ĠThe Ġdatabase Ġcontains Ġall Ġof Ġthe Ġnotes Ġassociated Ġwith Ġeach Ġpatient 's Ġtime Ġspent Ġin Ġthe ĠIC U Ġas Ġwell Ġas Ġ55 , 177 Ġdischarge Ġreports Ġand Ġ4 , 475 Ġdischarge Ġadd end ums Ġfor Ġ41 , 127 Ġdistinct Ġpatients . ĠOnly Ġthe Ġoriginal Ġdischarge Ġreports Ġwere Ġincluded Ġin Ġour Ġanalyses . ĠEach Ġdischarge Ġsummary Ġwas Ġdivided Ġinto Ġsections Ġ( Date Ġof ĠBirth , ĠSex , ĠChief ĠCompl aint , ĠMajor ĠS urgical Ġor ĠInv asive ĠProcedure , ĠHistory Ġof ĠPresent ĠIll ness , Ġetc .) Ġusing Ġa Ġregular Ġexpression . </s> U pper ĠBound Ġon ĠSum mar ization </s> Ext ractive Ġsummar ization Ġof Ġdischarge Ġsumm aries Ġrelies Ġon Ġthe Ġassumption Ġthat Ġthe Ġinformation Ġin Ġthe Ġdischarge Ġsummary Ġis Ġdocumented Ġelsewhere Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's Ġnotes . ĠHowever , Ġsometimes Ġclinicians Ġwill Ġdocument Ġinformation Ġin Ġthe Ġdischarge Ġsummary Ġthat Ġmay Ġhave Ġbeen Ġdiscussed Ġthroughout Ġthe Ġhospital Ġvisit , Ġbut Ġwas Ġnever Ġdocumented Ġin Ġthe ĠE HR . ĠThus , Ġour Ġfirst Ġaim Ġwas Ġto Ġdetermine Ġthe Ġupper Ġbound Ġof Ġextract ive Ġsummar ization . </s> For Ġeach Ġpatient , Ġwe Ġcompared Ġthe Ġtext Ġof Ġthe Ġdischarge Ġsummary Ġto Ġthe Ġremaining Ġnotes Ġfor Ġthe Ġpatient 's Ġcurrent Ġadmission Ġas Ġwell Ġas Ġtheir Ġentire Ġmedical Ġrecord . ĠConcept ĠUnique ĠIdent ifiers Ġ( CU Is ) Ġfrom Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem Ġ( U ML S ) Ġwere Ġcompared Ġin Ġorder Ġto Ġassess Ġwhether Ġclinically Ġrelevant Ġconcepts Ġin Ġthe Ġdischarge Ġsummary Ġcould Ġbe Ġlocated Ġin Ġthe Ġremaining Ġnotes ĠB IB REF 18 Ġ. ĠCU Is Ġwere Ġextracted Ġusing ĠApache Ġc TA K ES ĠB IB REF 19 Ġand Ġfiltered Ġby Ġremoving Ġthe ĠCU Is Ġthat Ġare Ġalready Ġsubs umed Ġby Ġa Ġlonger Ġspanning ĠC UI . ĠFor Ġexample , ĠCU Is Ġfor Ġ" head " Ġand Ġ" ache " Ġwere Ġremoved Ġif Ġa ĠC UI Ġexisted Ġfor Ġ" head Ġac he " Ġin Ġorder Ġto Ġextract Ġthe Ġmost Ġclinically Ġrelevant ĠCU Is . </s> In Ġorder Ġto Ġunderstand Ġwhich Ġsections Ġof Ġthe Ġdischarge Ġsumm aries Ġwould Ġbe Ġthe Ġeasiest Ġor Ġmost Ġdifficult Ġto Ġsummarize , Ġwe Ġperformed Ġthe Ġsame ĠC UI Ġoverlap Ġcomparison Ġfor Ġthe Ġchief Ġcomplaint , Ġmajor Ġsurgical Ġor Ġinvasive Ġprocedure , Ġdischarge Ġmedication , Ġand Ġhistory Ġof Ġpresent Ġillness Ġsections Ġof Ġthe Ġdischarge Ġnote Ġseparately . ĠWe Ġcalculated Ġwhich Ġfraction Ġof Ġthe ĠCU Is Ġin Ġeach Ġsection Ġwere Ġlocated Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's Ġnote Ġfor Ġa Ġspecific Ġhospital Ġstay . ĠWe Ġalso Ġcalculated Ġwhat Ġpercent Ġof Ġthe Ġgenders Ġrecorded Ġin Ġthe Ġdischarge Ġsummary Ġwere Ġalso Ġrecorded Ġin Ġthe Ġstructured Ġdata Ġfor Ġthe Ġpatient . </s> For Ġeach Ġof Ġthe Ġ55 , 177 Ġdischarge Ġsummary Ġreports Ġin Ġthe ĠM IM IC Ġdatabase , Ġwe Ġcalculated Ġwhat Ġfraction Ġof Ġthe ĠCU Is Ġin Ġthe Ġdischarge Ġsummary Ġcould Ġbe Ġfound Ġin Ġthe Ġremaining Ġnotes Ġfor Ġthe Ġpatient 's Ġcurrent Ġadmission Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġin Ġtheir Ġentire Ġlongitudinal Ġmedical Ġrecord Ġ( ĠIN LINE FORM 1 Ġ). ĠTable ĠT AB REF 13 Ġshows Ġthe ĠC UI Ġrecall Ġaveraged Ġacross Ġall Ġdischarge Ġsumm aries Ġby Ġboth Ġsubject _ id Ġand Ġhad m _ id . ĠThe Ġlow Ġrecall Ġsuggests Ġthat Ġclinicians Ġmay Ġincorporate Ġinformation Ġin Ġthe Ġdischarge Ġnote Ġthat Ġhad Ġnot Ġbeen Ġpreviously Ġdocumented Ġin Ġthe ĠE HR . ĠFigure ĠFIG REF 11 Ġplots Ġthe Ġrelationship Ġbetween Ġthe Ġnumber Ġof Ġnon - dis charge Ġnotes Ġfor Ġeach Ġpatient Ġand Ġthe ĠC UI Ġrecall Ġ( top ) Ġand Ġthe Ġnumber Ġof Ġtotal ĠCU Is Ġin Ġnon - dis charge Ġnotes Ġand Ġthe ĠC UI Ġrecall Ġ( middle ). ĠThe Ġnumber Ġof ĠCU Is Ġis Ġa Ġproxy Ġfor Ġthe Ġlength Ġof Ġthe Ġnotes , Ġand Ġas Ġexpected , Ġthe ĠC UI Ġrecall Ġtends Ġto Ġbe Ġhigher Ġin Ġpatients Ġwith Ġmore Ġand Ġlonger Ġnotes . ĠThe Ġbottom Ġpanel Ġin ĠFigure ĠFIG REF 11 Ġdemonstrates Ġthat Ġrecall Ġis Ġnot Ġcorrelated Ġwith Ġthe Ġpatient 's Ġlength Ġof Ġstay Ġoutside Ġthe ĠIC U , Ġwhich Ġindicates Ġthat Ġour Ġupper Ġbound Ġcalculation Ġis Ġnot Ġseverely Ġimpacted Ġby Ġonly Ġhaving Ġaccess Ġto Ġthe Ġpatient 's Ġnotes Ġfrom Ġtheir Ġstay Ġin Ġthe ĠIC U . </s> Finally , ĠTable ĠT AB REF 14 Ġshows Ġthe Ġrecall Ġfor Ġthe Ġsex , Ġchief Ġcomplaint , Ġprocedure , Ġdischarge Ġmedication , Ġand ĠHP I Ġdischarge Ġsummary Ġsections Ġaveraged Ġacross Ġall Ġthe Ġdischarge Ġsumm aries . ĠThe Ġprocedure Ġsection Ġhas Ġthe Ġhighest Ġrecall Ġof Ġ0 . 807 , Ġwhich Ġis Ġunderstandable Ġbecause Ġprocedures Ġundergone Ġduring Ġan Ġin patient Ġstay Ġare Ġmost Ġlikely Ġto Ġbe Ġdocumented Ġin Ġan ĠE HR . ĠThe Ġrecall Ġfor Ġeach Ġof Ġthese Ġfive Ġsections Ġis Ġmuch Ġhigher Ġthan Ġthe Ġoverall Ġrecall Ġin ĠTable ĠT AB REF 13 Ġ, Ġsuggesting Ġthat Ġextract ive Ġsummar ization Ġmay Ġbe Ġeasier Ġfor Ġsome Ġsections Ġof Ġthe Ġdischarge Ġnote . </s> Overall , Ġthis Ġupper Ġbound Ġanalysis Ġsuggests Ġthat Ġwe Ġmay Ġnot Ġbe Ġable Ġto Ġrecreate Ġa Ġdischarge Ġsummary Ġwith Ġextract ive Ġsummar ization Ġalone . ĠWhile ĠC UI Ġcomparison Ġallows Ġfor Ġcomparing Ġmedically Ġrelevant Ġconcepts , Ġc TA K ES 's ĠC UI Ġlab elling Ġprocess Ġis Ġnot Ġperfect , Ġand Ġfurther Ġwork , Ġperhaps Ġthrough Ġsophisticated Ġregular Ġexpressions , Ġis Ġneeded Ġto Ġdefine Ġthe Ġlimits Ġof Ġextract ive Ġsummar ization . </s> Label ing ĠHistory Ġof ĠPresent ĠIll ness ĠNotes </s> We Ġdeveloped Ġa Ġclass ifier Ġto Ġlabel Ġtopics Ġin Ġthe Ġhistory Ġof Ġpresent Ġillness Ġ( H PI ) Ġnotes , Ġincluding Ġdemographics , Ġdiagnosis Ġhistory , Ġand Ġsymptoms / sign s , Ġamong Ġothers . ĠA Ġrandom Ġsample Ġof Ġ5 15 Ġhistory Ġof Ġpresent Ġillness Ġnotes Ġwas Ġtaken , Ġand Ġeach Ġof Ġthe Ġnotes Ġwas Ġmanually Ġannot ated Ġby Ġone Ġof Ġeight Ġannot ators Ġusing Ġthe Ġsoftware ĠMulti - document ĠAn notation ĠEnvironment Ġ( MA E ) ĠB IB REF 20 Ġ. ĠMA E Ġprovides Ġan Ġinteractive ĠGUI Ġfor Ġannot ators Ġand Ġexports Ġthe Ġresults Ġof Ġeach Ġannotation Ġas Ġan ĠXML Ġfile Ġwith Ġtext Ġspans Ġand Ġtheir Ġassociated Ġlabels Ġfor Ġadditional Ġprocessing . Ġ40 % Ġof Ġthe ĠHP I Ġnotes Ġwere Ġlabeled Ġby Ġclinicians Ġand Ġ60 % Ġby Ġnon - cl in icians . ĠTable ĠT AB REF 5 Ġshows Ġthe Ġinstructions Ġgiven Ġto Ġthe Ġannot ators Ġfor Ġeach Ġof Ġthe Ġ10 Ġlabels . ĠThe Ġentire ĠHP I Ġnote Ġwas Ġlabeled Ġwith Ġone Ġof Ġthe Ġlabels , Ġand Ġinstructions Ġwere Ġgiven Ġto Ġlabel Ġeach Ġclause Ġin Ġa Ġsentence Ġwith Ġthe Ġsame Ġlabel Ġwhen Ġpossible . </s> Our ĠL ST M Ġmodel Ġwas Ġadopted Ġfrom Ġprior Ġwork Ġby ĠD ern on court Ġet Ġal ĠB IB REF 21 Ġ. ĠWhereas Ġthe ĠD ern on court Ġmodel Ġjointly Ġclassified Ġeach Ġsentence Ġin Ġa Ġmedical Ġabstract , Ġhere Ġwe Ġjointly Ġclassify Ġeach Ġword Ġin Ġthe ĠHP I Ġsummary . ĠOur Ġmodel Ġconsists Ġof Ġfour Ġlayers : Ġa Ġtoken Ġembed ding Ġlayer , Ġa Ġword Ġcontextual Ġrepresentation Ġlayer , Ġa Ġlabel Ġscoring Ġlayer , Ġand Ġa Ġlabel Ġsequence Ġoptimization Ġlayer Ġ( Figure ĠFIG REF 9 Ġ). </s> In Ġthe Ġfollowing Ġdescriptions , Ġlower case Ġital ics Ġis Ġused Ġto Ġdenote Ġscal ars , Ġlower case Ġbold Ġis Ġused Ġto Ġdenote Ġvectors , Ġand Ġu pperc ase Ġital ics Ġis Ġused Ġto Ġdenote Ġmat rices . </s> Token ĠEmb edd ing ĠLayer : ĠIn Ġthe Ġtoken Ġembed ding Ġlayer , Ġpret rained Ġword Ġembed d ings Ġare Ġcombined Ġwith Ġlearned Ġcharacter Ġembed d ings Ġto Ġcreate Ġa Ġhybrid Ġtoken Ġembed ding Ġfor Ġeach Ġword Ġin Ġthe ĠHP I Ġnote . ĠThe Ġword Ġembed d ings , Ġwhich Ġare Ġdirect Ġm appings Ġfrom Ġword ĠIN LINE FORM 0 Ġto Ġvector ĠIN LINE FORM 1 Ġ, Ġwere Ġpret rained Ġusing Ġword 2 vec ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġon Ġall Ġof Ġthe Ġnotes Ġin ĠM IM IC Ġ( v 30 ) Ġand Ġonly Ġthe Ġdischarge Ġnotes . ĠBoth Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġand Ġskip Ġgram Ġmodels Ġwere Ġexplored . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġsequence Ġof Ġcharacters Ġcomprising Ġthe Ġword ĠIN LINE FORM 1 Ġ. ĠEach Ġcharacter Ġis Ġmapped Ġto Ġits Ġembed ding ĠIN LINE FORM 2 Ġ, Ġand Ġall Ġembed d ings Ġare Ġinput Ġinto Ġa Ġbid irection al ĠL ST M , Ġwhich Ġultimately Ġoutputs ĠIN LINE FORM 3 Ġ, Ġthe Ġcharacter Ġembed ding Ġof Ġthe Ġword ĠIN LINE FORM 4 Ġ. </s> The Ġoutput Ġof Ġthe Ġtoken Ġembed ding Ġlayer Ġis Ġthe Ġvector Ġe , Ġwhich Ġis Ġthe Ġresult Ġof Ġconc aten ation Ġof Ġthe Ġword Ġembed ding , Ġt , Ġand Ġthe Ġcharacter Ġembed ding , Ġc . </s> Context ual ĠRepresent ation ĠLayer : ĠThe Ġcontextual Ġrepresentation Ġlayer Ġtakes Ġas Ġinput Ġthe Ġsequence Ġof Ġword Ġembed d ings , ĠIN LINE FORM 0 Ġ, Ġand Ġoutputs Ġan Ġembed ding Ġof Ġthe Ġcontextual Ġrepresentation Ġfor Ġeach Ġword Ġin Ġthe ĠHP I Ġnote . ĠThe Ġword Ġembed d ings Ġare Ġfed Ġinto Ġa Ġbi - direction al ĠL ST M , Ġwhich Ġoutputs ĠIN LINE FORM 1 Ġ, Ġa Ġconc aten ation Ġof Ġthe Ġhidden Ġstates Ġof Ġthe Ġtwo ĠL ST Ms Ġfor Ġeach Ġword . </s> Label ĠSc oring ĠLayer : ĠAt Ġthis Ġpoint , Ġeach Ġword ĠIN LINE FORM 0 Ġis Ġassociated Ġwith Ġa Ġhidden Ġrepresentation Ġof Ġthe Ġword , ĠIN LINE FORM 1 Ġ. ĠIn Ġthe Ġlabel Ġscoring Ġlayer , Ġwe Ġuse Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġwith Ġone Ġhidden Ġlayer Ġto Ġoutput Ġa Ġscore Ġassociated Ġwith Ġeach Ġof Ġthe Ġ10 Ġcategories Ġfor Ġeach Ġword . ĠLet ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠWe Ġcan Ġcompute Ġa Ġvector Ġof Ġscores Ġs Ġ= ĠIN LINE FORM 4 Ġwhere Ġthe Ġit h Ġcomponent Ġof Ġs Ġis Ġthe Ġscore Ġof Ġclass Ġi Ġfor Ġa Ġgiven Ġword . </s> Label ĠSequence ĠOptim ization ĠLayer : ĠThe ĠLabel ĠSequence ĠOptim ization ĠLayer Ġcomp utes Ġthe Ġprobability Ġof Ġa Ġlabeling Ġsequence Ġand Ġfinds Ġthe Ġsequence Ġwith Ġthe Ġhighest Ġprobability . ĠIn Ġorder Ġto Ġcondition Ġthe Ġlabel Ġfor Ġeach Ġword Ġon Ġthe Ġlabels Ġof Ġits Ġneighbors , Ġwe Ġemploy Ġa Ġlinear Ġchain Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġto Ġdefine Ġa Ġglobal Ġscore , ĠIN LINE FORM 0 Ġ, Ġfor Ġa Ġsequence Ġof Ġwords Ġand Ġtheir Ġassociated Ġscores ĠIN LINE FORM 1 Ġand Ġlabels , ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> where ĠT Ġis Ġa Ġtransition Ġmatrix ĠIN LINE FORM 0 ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġvectors Ġof Ġscores Ġthat Ġdescribe Ġthe Ġcost Ġof Ġbeginning Ġor Ġending Ġwith Ġa Ġlabel . </s> The Ġprobability Ġof Ġa Ġsequence Ġof Ġlabels Ġis Ġcalculated Ġby Ġapplying Ġa Ġsoft max Ġlayer Ġto Ġobtain Ġa Ġprobability Ġof Ġa Ġsequence Ġof Ġlabels : ĠDIS PLAY FORM 0 </s> Cross - ent ropy Ġloss , ĠIN LINE FORM 0 Ġ, Ġis Ġused Ġas Ġthe Ġobjective Ġfunction Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġcorrect Ġsequence Ġof Ġlabels Ġand Ġthe Ġprobability ĠIN LINE FORM 2 Ġis Ġcalculated Ġaccording Ġto Ġthe ĠCR F . </s> We Ġevaluated Ġour Ġmodel Ġon Ġthe Ġ5 15 Ġannot ated Ġhistory Ġof Ġpresent Ġillness Ġnotes , Ġwhich Ġwere Ġsplit Ġin Ġa Ġ70 % Ġtrain Ġset , Ġ15 % Ġdevelopment Ġset , Ġand Ġa Ġ15 % Ġtest Ġset . ĠThe Ġmodel Ġis Ġtrained Ġusing Ġthe ĠAdam Ġalgorithm Ġfor Ġgradient - based Ġoptimization ĠB IB REF 25 Ġwith Ġan Ġinitial Ġlearning Ġrate Ġ= Ġ0 . 001 Ġand Ġdecay Ġ= Ġ0 . 9 . ĠA Ġdrop out Ġrate Ġof Ġ0 . 5 Ġwas Ġapplied Ġfor Ġregular ization , Ġand Ġeach Ġbatch Ġsize Ġ= Ġ20 . ĠThe Ġmodel Ġran Ġfor Ġ20 Ġepoch s Ġand Ġwas Ġhalted Ġearly Ġif Ġthere Ġwas Ġno Ġimprovement Ġafter Ġ3 Ġepoch s . </s> We Ġevaluated Ġthe Ġimpact Ġof Ġcharacter Ġembed d ings , Ġthe Ġchoice Ġof Ġpret rained Ġw 2 v Ġembed d ings , Ġand Ġthe Ġaddition Ġof Ġlearned Ġword Ġembed d ings Ġon Ġmodel Ġperformance Ġon Ġthe Ġdev Ġset . ĠWe Ġreport Ġperformance Ġof Ġthe Ġbest Ġperforming Ġmodel Ġon Ġthe Ġtest Ġset . </s> Table ĠT AB REF 16 Ġcompares Ġdev Ġset Ġperformance Ġof Ġthe Ġmodel Ġusing Ġvarious Ġpret rained Ġword Ġembed d ings , Ġwith Ġand Ġwithout Ġcharacter Ġembed d ings , Ġand Ġwith Ġpret rained Ġversus Ġlearned Ġword Ġembed d ings . ĠThe Ġfirst Ġrow Ġin Ġeach Ġsection Ġis Ġthe Ġperformance Ġof Ġthe Ġmodel Ġarchitecture Ġdescribed Ġin Ġthe Ġmethods Ġsection Ġfor Ġcomparison . ĠModels Ġusing Ġword Ġembed d ings Ġtrained Ġon Ġthe Ġdischarge Ġsumm aries Ġperformed Ġbetter Ġthan Ġword Ġembed d ings Ġtrained Ġon Ġall ĠM IM IC Ġnotes , Ġlikely Ġbecause Ġthe Ġdischarge Ġsummary Ġword Ġembed d ings Ġbetter Ġcaptured Ġword Ġuse Ġin Ġdischarge Ġsumm aries Ġalone . ĠInterestingly , Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġembed d ings Ġoutper formed Ġskip Ġgram Ġembed d ings , Ġwhich Ġis Ġsurprising Ġbecause Ġthe Ġskip Ġgram Ġarchitecture Ġtypically Ġworks Ġbetter Ġfor Ġinf requent Ġwords ĠB IB REF 26 Ġ. ĠAs Ġexpected , Ġinclusion Ġof Ġcharacter Ġembed d ings Ġincreases Ġperformance Ġby Ġapproximately Ġ3 %. ĠThe Ġmodel Ġwith Ġword Ġembed d ings Ġlearned Ġin Ġthe Ġmodel Ġachieves Ġthe Ġhighest Ġperformance Ġon Ġthe Ġdev Ġset Ġ( 0 . 886 ), Ġwhich Ġmay Ġbe Ġbecause Ġthe Ġpret rained Ġworm Ġembed d ings Ġwere Ġtrained Ġon Ġa Ġprevious Ġversion Ġof ĠM IM IC . ĠAs Ġa Ġresult , Ġsome Ġwords Ġin Ġthe Ġdischarge Ġsumm aries , Ġsuch Ġas Ġmi - spe lled Ġwords Ġor Ġrare r Ġdiseases Ġand Ġmedications , Ġdid Ġnot Ġhave Ġassociated Ġword Ġembed d ings . ĠPer forming Ġa Ġsimple Ġspell Ġcorrection Ġon Ġout Ġof Ġvoc ab Ġwords Ġmay Ġimprove Ġperformance Ġwith Ġpret rained Ġword Ġembed d ings . </s> We Ġevaluated Ġthe Ġbest Ġperforming Ġmodel Ġon Ġthe Ġtest Ġset . ĠThe ĠLearned ĠWord ĠEmb edd ings Ġmodel Ġachieved Ġan Ġaccuracy Ġof Ġ0 . 88 Ġand Ġan ĠF 1 - Score Ġof Ġ0 . 8 76 Ġon Ġthe Ġtest Ġset . ĠTable ĠT AB REF 17 Ġshows Ġthe Ġprecision , Ġrecall , ĠF 1 Ġscore , Ġand Ġsupport Ġfor Ġeach Ġof Ġthe Ġten Ġlabels , Ġand ĠFigure ĠFIG REF 18 Ġshows Ġthe Ġconfusion Ġmatrix Ġillustrating Ġwhich Ġlabels Ġwere Ġfrequently Ġmis classified . ĠThe Ġdemographics Ġand Ġpatient Ġmovement Ġlabels Ġachieved Ġthe Ġhighest ĠF 1 Ġscores Ġ( 0 . 96 Ġand Ġ0 . 93 Ġrespectively ) Ġwhile Ġthe Ġv itals / l abs Ġand Ġmedication Ġhistory Ġlabels Ġhad Ġthe Ġlowest ĠF 1 Ġscores Ġ( 0 . 40 Ġand Ġ0 . 66 Ġrespectively ). ĠThe Ġdemographics Ġsection Ġconsistently Ġoccurs Ġat Ġthe Ġbeginning Ġof Ġthe ĠHP I Ġnote , Ġand Ġthe Ġpatient Ġmovement Ġsection Ġuses Ġa Ġlimited Ġvoc ab Ġ( trans ferred , Ġadmitted , Ġetc .), Ġwhich Ġmay Ġexplain Ġtheir Ġhigh ĠF 1 Ġscores . ĠOn Ġthe Ġother Ġhand , Ġthe Ġv itals / l abs Ġand Ġmedication Ġhistory Ġsections Ġhad Ġthe Ġlowest Ġsupport , Ġwhich Ġmay Ġexplain Ġwhy Ġthey Ġwere Ġmore Ġchallenging Ġto Ġlabel . </s> Words Ġthat Ġbelonged Ġto Ġthe Ġdiagnosis Ġhistory , Ġpatient Ġmovement , Ġand Ġprocedure / results Ġsections Ġwere Ġfrequently Ġlabeled Ġas Ġsymptoms / sign s Ġ( Figure ĠFIG REF 18 Ġ). ĠDiagn osis Ġhistory Ġsections Ġmay Ġbe Ġlabeled Ġfrequently Ġas Ġsymptoms / sign s Ġbecause Ġsymptoms / d ise ases Ġcan Ġeither Ġbe Ġdescribed Ġas Ġpart Ġof Ġthe Ġpatient 's Ġdiagnosis Ġhistory Ġor Ġcurrent Ġsymptoms Ġdepending Ġon Ġwhen Ġthe Ġsymptom / d ise ase Ġoccurred . ĠHowever , Ġmany Ġof Ġthe Ġmis class ification Ġerrors Ġmay Ġbe Ġdue Ġto Ġinconsistency Ġin Ġmanual Ġlab elling Ġamong Ġannot ators . ĠFor Ġexample , Ġsentences Ġdescribing Ġboth Ġpatient Ġmovement Ġand Ġpatient Ġsymptoms Ġ( e . g . Ġ" the Ġpatient Ġwas Ġtransferred Ġto Ġthe Ġhospital Ġfor Ġhis Ġhypertension ") Ġwere Ġlabeled Ġentirely Ġas Ġ' patient Ġmovement ' Ġby Ġsome Ġannot ators Ġwhile Ġother Ġannot ators Ġlabeled Ġthe Ġdifferent Ġclauses Ġof Ġthe Ġsentence Ġseparately Ġas Ġ' patient Ġmovement ' Ġand Ġ' sym ptoms / sign s .' ĠFurther Ġstandard ization Ġamong Ġannot ators Ġis Ġneeded Ġto Ġavoid Ġthese Ġmis class ifications . ĠFuture Ġwork Ġis Ġneeded Ġto Ġobtain Ġadditional Ġmanual Ġannotations Ġwhere Ġeach ĠHP I Ġnote Ġis Ġannot ated Ġby Ġmultiple Ġannot ators . ĠThis Ġwill Ġallow Ġfor Ġcalculation Ġof ĠCohen 's Ġk appa , Ġwhich Ġmeasures Ġinter - annot ator Ġagreement , Ġand Ġcomparison Ġof Ġclin ician Ġand Ġnon - cl in ician Ġannot ator Ġreliability . </s> Future Ġwork Ġis Ġalso Ġneeded Ġto Ġbetter Ġunderstand Ġcommonly Ġmis label ed Ġcategories Ġand Ġexplore Ġalternative Ġmodel Ġarchitectures . ĠHere Ġwe Ġperform Ġword Ġlevel Ġlabel Ġprediction , Ġwhich Ġcan Ġresult Ġin Ġphrases Ġthat Ġcontain Ġmultiple Ġlabels . ĠFor Ġexample , Ġthe Ġphrase Ġ" history Ġof Ġneck Ġpain " Ġcan Ġbe Ġlabeled Ġwith Ġboth Ġ' diagn osis Ġhistory ' Ġand Ġ' sym ptoms / sign s ' Ġlabels . ĠPost - processing Ġis Ġneeded Ġto Ġcreate Ġa Ġfinal Ġlabel Ġprediction Ġfor Ġeach Ġphrase . ĠWhile Ġphrase Ġlevel Ġprediction Ġmay Ġresolve Ġthese Ġchallenges , Ġit Ġis Ġdifficult Ġto Ġsegment Ġthe ĠHP I Ġnote Ġinto Ġphrases Ġfor Ġprediction , Ġas Ġa Ġsingle Ġphrase Ġmay Ġtruly Ġcontain Ġmultiple Ġlabels . ĠSe gment ation Ġof Ġsentences Ġby Ġpunct uation , Ġconj unctions , Ġand Ġpre pos itions Ġmay Ġyield Ġthe Ġbest Ġphrase Ġchunk er Ġfor Ġdischarge Ġsummary Ġtext . </s> Finally , Ġsupplement ing Ġthe Ġword Ġembed d ings Ġin Ġour ĠL ST M Ġmodel Ġwith ĠCU Is Ġmay Ġfurther Ġimprove Ġperformance . ĠWhile Ġword Ġembed d ings Ġdo Ġwell Ġin Ġlearning Ġthe Ġcontextual Ġcontext Ġof Ġwords , ĠCU Is Ġallow Ġfor Ġmore Ġexplicit Ġincorporation Ġof Ġmedical Ġdomain Ġexpertise . ĠBy Ġconc aten ating Ġthe ĠC UI Ġfor Ġeach Ġword Ġwith Ġits Ġhybrid Ġtoken Ġembed ding , Ġwe Ġmay Ġbe Ġable Ġto Ġleverage Ġboth Ġdata Ġdriven Ġand Ġont ology Ġdriven Ġapproaches . </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġdeveloped Ġa ĠC UI - based Ġupper Ġbound Ġon Ġextract ive Ġsummar ization Ġof Ġdischarge Ġsumm aries Ġand Ġpresented Ġa ĠN N Ġarchitecture Ġthat Ġjointly Ġclass ifies Ġwords Ġin Ġhistory Ġof Ġpresent Ġillness Ġnotes . ĠWe Ġdemonstrate Ġthat Ġour Ġmodel Ġcan Ġachieve Ġexcellent Ġperformance Ġon Ġa Ġsmall Ġdataset Ġwith Ġknown Ġheterogeneity Ġamong Ġannot ators . ĠThis Ġmodel Ġcan Ġbe Ġapplied Ġto Ġthe Ġ55 , 000 Ġdischarge Ġsumm aries Ġin ĠM IM IC Ġto Ġcreate Ġa Ġdataset Ġfor Ġevaluation Ġof Ġextract ive Ġsummar ization Ġmethods . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġour Ġannot ators , ĠAndrew ĠGoldberg , ĠLaurie ĠAl sent zer , ĠElaine ĠGoldberg , ĠAndy ĠAl sent zer , ĠGrace ĠLo , Ġand ĠJosh ĠDon is . ĠWe Ġwould Ġalso Ġlike Ġto Ġacknowledge ĠPete ĠSz ol ov its Ġfor Ġhis Ġguidance Ġand Ġfor Ġproviding Ġthe Ġpret rained Ġword Ġembed d ings Ġand ĠTr istan ĠNa umann Ġfor Ġproviding Ġthe ĠM IM IC ĠCU Is .
Introduction Summarization of patient information is essential to the practice of medicine. Clinicians must synthesize information from diverse data sources to communicate with colleagues and provide coordinated care. Examples of clinical summarization are abundant in practice; patient handoff summaries facilitate provider shift change, progress notes provide a daily status update for a patient, oral case presentations enable transfer of information from overnight admission to the care team and attending, and discharge summaries provide information about a patient's hospital visit to their primary care physician and other outpatient providers BIBREF0 . Informal, unstructured, or poor quality summaries can lead to communication failures and even medical errors, yet clinical instruction on how to formulate clinical summaries is ad hoc and informal. Non-existent or limited search functionality, fragmented data sources, and limited visualizations in electronic health records (EHRs) make summarization challenging for providers BIBREF1 , BIBREF2 , BIBREF3 . Furthermore, while dictation of EHR notes allows clinicians to more efficiently document information at the point of care, the stream of consciousness-like writing can hinder the readability of notes. Kripalani et al. show that discharge summaries are often lacking key information, including treatment progression and follow-up protocols, which can hinder communication between hospital and community based clinicians BIBREF4 . Recently, St. Thomas Hospital in Nashville, TN stipulated that discharge notes be written within 48 hours of discharge following incidences where improper care was given to readmitted patients because the discharge summary for the previous admission was not completed BIBREF5 . Automated summary generation has the potential to save clinician time, avoid medical errors, and aid clinical decision making. By organizing and synthesizing a patient's salient medical history, algorithms for patient summarization can enable better communication and care, particularly for chronically ill patients, whose medical records often contain hundreds of notes. In this work, we explore the automatic summarization of discharge summary notes, which are critical to ensuring continuity of care after hospitalization. We (1) provide an upper bound on extractive summarization by assessing how much information in the discharge note can be found in the rest of the patient's EHR notes and (2) develop a classifier for labeling the topics of history of present illness notes, a narrative section in the discharge summary that describes the patient's prior history and current symptoms. Such a classifier can be used to create topic specific evaluation sets for methods that perform extractive summarization. These aims are critical steps in ultimately developing methods that can automate discharge summary creation. Related Work In the broader field of summarization, automization was meant to standardize output while also saving time and effort. Pioneering strategies in summarization started by extracting "significant" sentences in the whole corpus to build an abstract where "significant" sentences were defined by the number of frequently occurring words BIBREF6 . These initial methods did not consider word meaning or syntax at either the sentence or paragraph level, which made them crude at best. More advanced extractive heuristics like topic modeling BIBREF7 , cue word dictionary approaches BIBREF8 , and title methods BIBREF9 for scoring content in a sentence followed soon after. For example, topic modeling extends initial frequency methods by assigning topics scores by frequency of topic signatures, clustering sentences with similar topics, and finally extracting the centroid sentence, which is considered the most representative sentence BIBREF10 . Recently, abstractive summarization approaches using sequence-to-sequence methods have been developed to generate new text that synthesizes original text BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ; however, the field of abstractive summarization is quite young. Existing approaches within the field of electronic health record summarization have largely been extractive and indicative, meaning that summaries point to important pieces in the original text rather than replacing the original text altogether. Few approaches have been deployed in practice, and even fewer have demonstrated impact on quality of care and outcomes BIBREF15 . Summarization strategies have ranged from extraction of “relevant” sentences from the original text to form the summary BIBREF16 , topic modeling of EHR notes using Latent Dirichlet allocation (LDA) or bayesian networks BIBREF15 , and knowledge based heuristic systems BIBREF17 . To our knowledge, there is no literature to date on extractive or abstractive EHR summarization using neural networks. Data MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient's time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression. Upper Bound on Summarization Extractive summarization of discharge summaries relies on the assumption that the information in the discharge summary is documented elsewhere in the rest of the patient's notes. However, sometimes clinicians will document information in the discharge summary that may have been discussed throughout the hospital visit, but was never documented in the EHR. Thus, our first aim was to determine the upper bound of extractive summarization. For each patient, we compared the text of the discharge summary to the remaining notes for the patient's current admission as well as their entire medical record. Concept Unique Identifiers (CUIs) from the Unified Medical Language System (UMLS) were compared in order to assess whether clinically relevant concepts in the discharge summary could be located in the remaining notes BIBREF18 . CUIs were extracted using Apache cTAKES BIBREF19 and filtered by removing the CUIs that are already subsumed by a longer spanning CUI. For example, CUIs for "head" and "ache" were removed if a CUI existed for "head ache" in order to extract the most clinically relevant CUIs. In order to understand which sections of the discharge summaries would be the easiest or most difficult to summarize, we performed the same CUI overlap comparison for the chief complaint, major surgical or invasive procedure, discharge medication, and history of present illness sections of the discharge note separately. We calculated which fraction of the CUIs in each section were located in the rest of the patient's note for a specific hospital stay. We also calculated what percent of the genders recorded in the discharge summary were also recorded in the structured data for the patient. For each of the 55,177 discharge summary reports in the MIMIC database, we calculated what fraction of the CUIs in the discharge summary could be found in the remaining notes for the patient's current admission ( INLINEFORM0 ) and in their entire longitudinal medical record ( INLINEFORM1 ). Table TABREF13 shows the CUI recall averaged across all discharge summaries by both subject_id and hadm_id. The low recall suggests that clinicians may incorporate information in the discharge note that had not been previously documented in the EHR. Figure FIGREF11 plots the relationship between the number of non-discharge notes for each patient and the CUI recall (top) and the number of total CUIs in non-discharge notes and the CUI recall (middle). The number of CUIs is a proxy for the length of the notes, and as expected, the CUI recall tends to be higher in patients with more and longer notes. The bottom panel in Figure FIGREF11 demonstrates that recall is not correlated with the patient's length of stay outside the ICU, which indicates that our upper bound calculation is not severely impacted by only having access to the patient's notes from their stay in the ICU. Finally, Table TABREF14 shows the recall for the sex, chief complaint, procedure, discharge medication, and HPI discharge summary sections averaged across all the discharge summaries. The procedure section has the highest recall of 0.807, which is understandable because procedures undergone during an inpatient stay are most likely to be documented in an EHR. The recall for each of these five sections is much higher than the overall recall in Table TABREF13 , suggesting that extractive summarization may be easier for some sections of the discharge note. Overall, this upper bound analysis suggests that we may not be able to recreate a discharge summary with extractive summarization alone. While CUI comparison allows for comparing medically relevant concepts, cTAKES's CUI labelling process is not perfect, and further work, perhaps through sophisticated regular expressions, is needed to define the limits of extractive summarization. Labeling History of Present Illness Notes We developed a classifier to label topics in the history of present illness (HPI) notes, including demographics, diagnosis history, and symptoms/signs, among others. A random sample of 515 history of present illness notes was taken, and each of the notes was manually annotated by one of eight annotators using the software Multi-document Annotation Environment (MAE) BIBREF20 . MAE provides an interactive GUI for annotators and exports the results of each annotation as an XML file with text spans and their associated labels for additional processing. 40% of the HPI notes were labeled by clinicians and 60% by non-clinicians. Table TABREF5 shows the instructions given to the annotators for each of the 10 labels. The entire HPI note was labeled with one of the labels, and instructions were given to label each clause in a sentence with the same label when possible. Our LSTM model was adopted from prior work by Dernoncourt et al BIBREF21 . Whereas the Dernoncourt model jointly classified each sentence in a medical abstract, here we jointly classify each word in the HPI summary. Our model consists of four layers: a token embedding layer, a word contextual representation layer, a label scoring layer, and a label sequence optimization layer (Figure FIGREF9 ). In the following descriptions, lowercase italics is used to denote scalars, lowercase bold is used to denote vectors, and uppercase italics is used to denote matrices. Token Embedding Layer: In the token embedding layer, pretrained word embeddings are combined with learned character embeddings to create a hybrid token embedding for each word in the HPI note. The word embeddings, which are direct mappings from word INLINEFORM0 to vector INLINEFORM1 , were pretrained using word2vec BIBREF22 , BIBREF23 , BIBREF24 on all of the notes in MIMIC (v30) and only the discharge notes. Both the continuous bag of words (CBOW) and skip gram models were explored. Let INLINEFORM0 be the sequence of characters comprising the word INLINEFORM1 . Each character is mapped to its embedding INLINEFORM2 , and all embeddings are input into a bidirectional LSTM, which ultimately outputs INLINEFORM3 , the character embedding of the word INLINEFORM4 . The output of the token embedding layer is the vector e, which is the result of concatenation of the word embedding, t, and the character embedding, c. Contextual Representation Layer: The contextual representation layer takes as input the sequence of word embeddings, INLINEFORM0 , and outputs an embedding of the contextual representation for each word in the HPI note. The word embeddings are fed into a bi-directional LSTM, which outputs INLINEFORM1 , a concatenation of the hidden states of the two LSTMs for each word. Label Scoring Layer: At this point, each word INLINEFORM0 is associated with a hidden representation of the word, INLINEFORM1 . In the label scoring layer, we use a fully connected neural network with one hidden layer to output a score associated with each of the 10 categories for each word. Let INLINEFORM2 and INLINEFORM3 . We can compute a vector of scores s = INLINEFORM4 where the ith component of s is the score of class i for a given word. Label Sequence Optimization Layer: The Label Sequence Optimization Layer computes the probability of a labeling sequence and finds the sequence with the highest probability. In order to condition the label for each word on the labels of its neighbors, we employ a linear chain conditional random field (CRF) to define a global score, INLINEFORM0 , for a sequence of words and their associated scores INLINEFORM1 and labels, INLINEFORM2 : DISPLAYFORM0 where T is a transition matrix INLINEFORM0 INLINEFORM1 and INLINEFORM2 are vectors of scores that describe the cost of beginning or ending with a label. The probability of a sequence of labels is calculated by applying a softmax layer to obtain a probability of a sequence of labels: DISPLAYFORM0 Cross-entropy loss, INLINEFORM0 , is used as the objective function where INLINEFORM1 is the correct sequence of labels and the probability INLINEFORM2 is calculated according to the CRF. We evaluated our model on the 515 annotated history of present illness notes, which were split in a 70% train set, 15% development set, and a 15% test set. The model is trained using the Adam algorithm for gradient-based optimization BIBREF25 with an initial learning rate = 0.001 and decay = 0.9. A dropout rate of 0.5 was applied for regularization, and each batch size = 20. The model ran for 20 epochs and was halted early if there was no improvement after 3 epochs. We evaluated the impact of character embeddings, the choice of pretrained w2v embeddings, and the addition of learned word embeddings on model performance on the dev set. We report performance of the best performing model on the test set. Table TABREF16 compares dev set performance of the model using various pretrained word embeddings, with and without character embeddings, and with pretrained versus learned word embeddings. The first row in each section is the performance of the model architecture described in the methods section for comparison. Models using word embeddings trained on the discharge summaries performed better than word embeddings trained on all MIMIC notes, likely because the discharge summary word embeddings better captured word use in discharge summaries alone. Interestingly, the continuous bag of words embeddings outperformed skip gram embeddings, which is surprising because the skip gram architecture typically works better for infrequent words BIBREF26 . As expected, inclusion of character embeddings increases performance by approximately 3%. The model with word embeddings learned in the model achieves the highest performance on the dev set (0.886), which may be because the pretrained worm embeddings were trained on a previous version of MIMIC. As a result, some words in the discharge summaries, such as mi-spelled words or rarer diseases and medications, did not have associated word embeddings. Performing a simple spell correction on out of vocab words may improve performance with pretrained word embeddings. We evaluated the best performing model on the test set. The Learned Word Embeddings model achieved an accuracy of 0.88 and an F1-Score of 0.876 on the test set. Table TABREF17 shows the precision, recall, F1 score, and support for each of the ten labels, and Figure FIGREF18 shows the confusion matrix illustrating which labels were frequently misclassified. The demographics and patient movement labels achieved the highest F1 scores (0.96 and 0.93 respectively) while the vitals/labs and medication history labels had the lowest F1 scores (0.40 and 0.66 respectively). The demographics section consistently occurs at the beginning of the HPI note, and the patient movement section uses a limited vocab (transferred, admitted, etc.), which may explain their high F1 scores. On the other hand, the vitals/labs and medication history sections had the lowest support, which may explain why they were more challenging to label. Words that belonged to the diagnosis history, patient movement, and procedure/results sections were frequently labeled as symptoms/signs (Figure FIGREF18 ). Diagnosis history sections may be labeled frequently as symptoms/signs because symptoms/diseases can either be described as part of the patient's diagnosis history or current symptoms depending on when the symptom/disease occurred. However, many of the misclassification errors may be due to inconsistency in manual labelling among annotators. For example, sentences describing both patient movement and patient symptoms (e.g. "the patient was transferred to the hospital for his hypertension") were labeled entirely as 'patient movement' by some annotators while other annotators labeled the different clauses of the sentence separately as 'patient movement' and 'symptoms/signs.' Further standardization among annotators is needed to avoid these misclassifications. Future work is needed to obtain additional manual annotations where each HPI note is annotated by multiple annotators. This will allow for calculation of Cohen's kappa, which measures inter-annotator agreement, and comparison of clinician and non-clinician annotator reliability. Future work is also needed to better understand commonly mislabeled categories and explore alternative model architectures. Here we perform word level label prediction, which can result in phrases that contain multiple labels. For example, the phrase "history of neck pain" can be labeled with both 'diagnosis history' and 'symptoms/signs' labels. Post-processing is needed to create a final label prediction for each phrase. While phrase level prediction may resolve these challenges, it is difficult to segment the HPI note into phrases for prediction, as a single phrase may truly contain multiple labels. Segmentation of sentences by punctuation, conjunctions, and prepositions may yield the best phrase chunker for discharge summary text. Finally, supplementing the word embeddings in our LSTM model with CUIs may further improve performance. While word embeddings do well in learning the contextual context of words, CUIs allow for more explicit incorporation of medical domain expertise. By concatenating the CUI for each word with its hybrid token embedding, we may be able to leverage both data driven and ontology driven approaches. Conclusion In this paper we developed a CUI-based upper bound on extractive summarization of discharge summaries and presented a NN architecture that jointly classifies words in history of present illness notes. We demonstrate that our model can achieve excellent performance on a small dataset with known heterogeneity among annotators. This model can be applied to the 55,000 discharge summaries in MIMIC to create a dataset for evaluation of extractive summarization methods. Acknowledgments We would like to thank our annotators, Andrew Goldberg, Laurie Alsentzer, Elaine Goldberg, Andy Alsentzer, Grace Lo, and Josh Donis. We would also like to acknowledge Pete Szolovits for his guidance and for providing the pretrained word embeddings and Tristan Naumann for providing the MIMIC CUIs.
what datasets were used?
c571deefe93f0a41b60f9886db119947648e967c
what Ġdatasets Ġwere Ġused ?
M IM IC - III
<s> what Ġdatasets Ġwere Ġused ? </s> Introduction </s> Sum mar ization Ġof Ġpatient Ġinformation Ġis Ġessential Ġto Ġthe Ġpractice Ġof Ġmedicine . ĠClin icians Ġmust Ġsynthes ize Ġinformation Ġfrom Ġdiverse Ġdata Ġsources Ġto Ġcommunicate Ġwith Ġcolleagues Ġand Ġprovide Ġcoordinated Ġcare . ĠExamples Ġof Ġclinical Ġsummar ization Ġare Ġabundant Ġin Ġpractice ; Ġpatient Ġhand off Ġsumm aries Ġfacilitate Ġprovider Ġshift Ġchange , Ġprogress Ġnotes Ġprovide Ġa Ġdaily Ġstatus Ġupdate Ġfor Ġa Ġpatient , Ġoral Ġcase Ġpresentations Ġenable Ġtransfer Ġof Ġinformation Ġfrom Ġovernight Ġadmission Ġto Ġthe Ġcare Ġteam Ġand Ġattending , Ġand Ġdischarge Ġsumm aries Ġprovide Ġinformation Ġabout Ġa Ġpatient 's Ġhospital Ġvisit Ġto Ġtheir Ġprimary Ġcare Ġphysician Ġand Ġother Ġoutpatient Ġproviders ĠB IB REF 0 Ġ. </s> In form al , Ġun struct ured , Ġor Ġpoor Ġquality Ġsumm aries Ġcan Ġlead Ġto Ġcommunication Ġfailures Ġand Ġeven Ġmedical Ġerrors , Ġyet Ġclinical Ġinstruction Ġon Ġhow Ġto Ġformulate Ġclinical Ġsumm aries Ġis Ġad Ġhoc Ġand Ġinformal . ĠNon - existent Ġor Ġlimited Ġsearch Ġfunctionality , Ġfragmented Ġdata Ġsources , Ġand Ġlimited Ġvisual izations Ġin Ġelectronic Ġhealth Ġrecords Ġ( E HR s ) Ġmake Ġsummar ization Ġchallenging Ġfor Ġproviders ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ. ĠFurthermore , Ġwhile Ġdict ation Ġof ĠE HR Ġnotes Ġallows Ġclinicians Ġto Ġmore Ġefficiently Ġdocument Ġinformation Ġat Ġthe Ġpoint Ġof Ġcare , Ġthe Ġstream Ġof Ġconsciousness - like Ġwriting Ġcan Ġhinder Ġthe Ġread ability Ġof Ġnotes . ĠK rip al ani Ġet Ġal . Ġshow Ġthat Ġdischarge Ġsumm aries Ġare Ġoften Ġlacking Ġkey Ġinformation , Ġincluding Ġtreatment Ġprogression Ġand Ġfollow - up Ġprotocols , Ġwhich Ġcan Ġhinder Ġcommunication Ġbetween Ġhospital Ġand Ġcommunity Ġbased Ġclinicians ĠB IB REF 4 Ġ. ĠRecently , ĠSt . ĠThomas ĠHospital Ġin ĠNashville , ĠTN Ġstip ulated Ġthat Ġdischarge Ġnotes Ġbe Ġwritten Ġwithin Ġ48 Ġhours Ġof Ġdischarge Ġfollowing Ġinc idences Ġwhere Ġimproper Ġcare Ġwas Ġgiven Ġto Ġread mitted Ġpatients Ġbecause Ġthe Ġdischarge Ġsummary Ġfor Ġthe Ġprevious Ġadmission Ġwas Ġnot Ġcompleted ĠB IB REF 5 Ġ. </s> Autom ated Ġsummary Ġgeneration Ġhas Ġthe Ġpotential Ġto Ġsave Ġclin ician Ġtime , Ġavoid Ġmedical Ġerrors , Ġand Ġaid Ġclinical Ġdecision Ġmaking . ĠBy Ġorganizing Ġand Ġsynthes izing Ġa Ġpatient 's Ġsalient Ġmedical Ġhistory , Ġalgorithms Ġfor Ġpatient Ġsummar ization Ġcan Ġenable Ġbetter Ġcommunication Ġand Ġcare , Ġparticularly Ġfor Ġchronically Ġill Ġpatients , Ġwhose Ġmedical Ġrecords Ġoften Ġcontain Ġhundreds Ġof Ġnotes . ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġthe Ġautomatic Ġsummar ization Ġof Ġdischarge Ġsummary Ġnotes , Ġwhich Ġare Ġcritical Ġto Ġensuring Ġcontinuity Ġof Ġcare Ġafter Ġhospital ization . ĠWe Ġ( 1 ) Ġprovide Ġan Ġupper Ġbound Ġon Ġextract ive Ġsummar ization Ġby Ġassessing Ġhow Ġmuch Ġinformation Ġin Ġthe Ġdischarge Ġnote Ġcan Ġbe Ġfound Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's ĠE HR Ġnotes Ġand Ġ( 2 ) Ġdevelop Ġa Ġclass ifier Ġfor Ġlabeling Ġthe Ġtopics Ġof Ġhistory Ġof Ġpresent Ġillness Ġnotes , Ġa Ġnarrative Ġsection Ġin Ġthe Ġdischarge Ġsummary Ġthat Ġdescribes Ġthe Ġpatient 's Ġprior Ġhistory Ġand Ġcurrent Ġsymptoms . ĠSuch Ġa Ġclass ifier Ġcan Ġbe Ġused Ġto Ġcreate Ġtopic Ġspecific Ġevaluation Ġsets Ġfor Ġmethods Ġthat Ġperform Ġextract ive Ġsummar ization . ĠThese Ġaims Ġare Ġcritical Ġsteps Ġin Ġultimately Ġdeveloping Ġmethods Ġthat Ġcan Ġautomate Ġdischarge Ġsummary Ġcreation . </s> Related ĠWork </s> In Ġthe Ġbroader Ġfield Ġof Ġsummar ization , Ġautom ization Ġwas Ġmeant Ġto Ġstandard ize Ġoutput Ġwhile Ġalso Ġsaving Ġtime Ġand Ġeffort . ĠPione ering Ġstrategies Ġin Ġsummar ization Ġstarted Ġby Ġextracting Ġ" significant " Ġsentences Ġin Ġthe Ġwhole Ġcorpus Ġto Ġbuild Ġan Ġabstract Ġwhere Ġ" significant " Ġsentences Ġwere Ġdefined Ġby Ġthe Ġnumber Ġof Ġfrequently Ġoccurring Ġwords ĠB IB REF 6 Ġ. ĠThese Ġinitial Ġmethods Ġdid Ġnot Ġconsider Ġword Ġmeaning Ġor Ġsyntax Ġat Ġeither Ġthe Ġsentence Ġor Ġparagraph Ġlevel , Ġwhich Ġmade Ġthem Ġcrude Ġat Ġbest . ĠMore Ġadvanced Ġextract ive Ġhe ur istics Ġlike Ġtopic Ġmodeling ĠB IB REF 7 Ġ, Ġcue Ġword Ġdictionary Ġapproaches ĠB IB REF 8 Ġ, Ġand Ġtitle Ġmethods ĠB IB REF 9 Ġfor Ġscoring Ġcontent Ġin Ġa Ġsentence Ġfollowed Ġsoon Ġafter . ĠFor Ġexample , Ġtopic Ġmodeling Ġextends Ġinitial Ġfrequency Ġmethods Ġby Ġassigning Ġtopics Ġscores Ġby Ġfrequency Ġof Ġtopic Ġsignatures , Ġclust ering Ġsentences Ġwith Ġsimilar Ġtopics , Ġand Ġfinally Ġextracting Ġthe Ġcent roid Ġsentence , Ġwhich Ġis Ġconsidered Ġthe Ġmost Ġrepresentative Ġsentence ĠB IB REF 10 Ġ. ĠRecently , Ġabstract ive Ġsummar ization Ġapproaches Ġusing Ġsequence - to - sequence Ġmethods Ġhave Ġbeen Ġdeveloped Ġto Ġgenerate Ġnew Ġtext Ġthat Ġsynthes izes Ġoriginal Ġtext ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ; Ġhowever , Ġthe Ġfield Ġof Ġabstract ive Ġsummar ization Ġis Ġquite Ġyoung . </s> Ex isting Ġapproaches Ġwithin Ġthe Ġfield Ġof Ġelectronic Ġhealth Ġrecord Ġsummar ization Ġhave Ġlargely Ġbeen Ġextract ive Ġand Ġindicative , Ġmeaning Ġthat Ġsumm aries Ġpoint Ġto Ġimportant Ġpieces Ġin Ġthe Ġoriginal Ġtext Ġrather Ġthan Ġreplacing Ġthe Ġoriginal Ġtext Ġaltogether . ĠFew Ġapproaches Ġhave Ġbeen Ġdeployed Ġin Ġpractice , Ġand Ġeven Ġfewer Ġhave Ġdemonstrated Ġimpact Ġon Ġquality Ġof Ġcare Ġand Ġoutcomes ĠB IB REF 15 Ġ. ĠSum mar ization Ġstrategies Ġhave Ġranged Ġfrom Ġextraction Ġof ĠâĢ ľ relevant âĢ Ŀ Ġsentences Ġfrom Ġthe Ġoriginal Ġtext Ġto Ġform Ġthe Ġsummary ĠB IB REF 16 Ġ, Ġtopic Ġmodeling Ġof ĠE HR Ġnotes Ġusing ĠLat ent ĠDir ich let Ġallocation Ġ( L DA ) Ġor Ġbay esian Ġnetworks ĠB IB REF 15 Ġ, Ġand Ġknowledge Ġbased Ġhe uristic Ġsystems ĠB IB REF 17 Ġ. ĠTo Ġour Ġknowledge , Ġthere Ġis Ġno Ġliterature Ġto Ġdate Ġon Ġextract ive Ġor Ġabstract ive ĠE HR Ġsummar ization Ġusing Ġneural Ġnetworks . </s> Data </s> M IM IC - III Ġis Ġa Ġfreely Ġavailable , Ġde identified Ġdatabase Ġcontaining Ġelectronic Ġhealth Ġrecords Ġof Ġpatients Ġadmitted Ġto Ġan ĠInt ensive ĠCare ĠUnit Ġ( IC U ) Ġat ĠBeth ĠIsrael ĠDe acon ess ĠMedical ĠCenter Ġbetween Ġ2001 Ġand Ġ2012 . ĠThe Ġdatabase Ġcontains Ġall Ġof Ġthe Ġnotes Ġassociated Ġwith Ġeach Ġpatient 's Ġtime Ġspent Ġin Ġthe ĠIC U Ġas Ġwell Ġas Ġ55 , 177 Ġdischarge Ġreports Ġand Ġ4 , 475 Ġdischarge Ġadd end ums Ġfor Ġ41 , 127 Ġdistinct Ġpatients . ĠOnly Ġthe Ġoriginal Ġdischarge Ġreports Ġwere Ġincluded Ġin Ġour Ġanalyses . ĠEach Ġdischarge Ġsummary Ġwas Ġdivided Ġinto Ġsections Ġ( Date Ġof ĠBirth , ĠSex , ĠChief ĠCompl aint , ĠMajor ĠS urgical Ġor ĠInv asive ĠProcedure , ĠHistory Ġof ĠPresent ĠIll ness , Ġetc .) Ġusing Ġa Ġregular Ġexpression . </s> U pper ĠBound Ġon ĠSum mar ization </s> Ext ractive Ġsummar ization Ġof Ġdischarge Ġsumm aries Ġrelies Ġon Ġthe Ġassumption Ġthat Ġthe Ġinformation Ġin Ġthe Ġdischarge Ġsummary Ġis Ġdocumented Ġelsewhere Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's Ġnotes . ĠHowever , Ġsometimes Ġclinicians Ġwill Ġdocument Ġinformation Ġin Ġthe Ġdischarge Ġsummary Ġthat Ġmay Ġhave Ġbeen Ġdiscussed Ġthroughout Ġthe Ġhospital Ġvisit , Ġbut Ġwas Ġnever Ġdocumented Ġin Ġthe ĠE HR . ĠThus , Ġour Ġfirst Ġaim Ġwas Ġto Ġdetermine Ġthe Ġupper Ġbound Ġof Ġextract ive Ġsummar ization . </s> For Ġeach Ġpatient , Ġwe Ġcompared Ġthe Ġtext Ġof Ġthe Ġdischarge Ġsummary Ġto Ġthe Ġremaining Ġnotes Ġfor Ġthe Ġpatient 's Ġcurrent Ġadmission Ġas Ġwell Ġas Ġtheir Ġentire Ġmedical Ġrecord . ĠConcept ĠUnique ĠIdent ifiers Ġ( CU Is ) Ġfrom Ġthe ĠUnified ĠMedical ĠLanguage ĠSystem Ġ( U ML S ) Ġwere Ġcompared Ġin Ġorder Ġto Ġassess Ġwhether Ġclinically Ġrelevant Ġconcepts Ġin Ġthe Ġdischarge Ġsummary Ġcould Ġbe Ġlocated Ġin Ġthe Ġremaining Ġnotes ĠB IB REF 18 Ġ. ĠCU Is Ġwere Ġextracted Ġusing ĠApache Ġc TA K ES ĠB IB REF 19 Ġand Ġfiltered Ġby Ġremoving Ġthe ĠCU Is Ġthat Ġare Ġalready Ġsubs umed Ġby Ġa Ġlonger Ġspanning ĠC UI . ĠFor Ġexample , ĠCU Is Ġfor Ġ" head " Ġand Ġ" ache " Ġwere Ġremoved Ġif Ġa ĠC UI Ġexisted Ġfor Ġ" head Ġac he " Ġin Ġorder Ġto Ġextract Ġthe Ġmost Ġclinically Ġrelevant ĠCU Is . </s> In Ġorder Ġto Ġunderstand Ġwhich Ġsections Ġof Ġthe Ġdischarge Ġsumm aries Ġwould Ġbe Ġthe Ġeasiest Ġor Ġmost Ġdifficult Ġto Ġsummarize , Ġwe Ġperformed Ġthe Ġsame ĠC UI Ġoverlap Ġcomparison Ġfor Ġthe Ġchief Ġcomplaint , Ġmajor Ġsurgical Ġor Ġinvasive Ġprocedure , Ġdischarge Ġmedication , Ġand Ġhistory Ġof Ġpresent Ġillness Ġsections Ġof Ġthe Ġdischarge Ġnote Ġseparately . ĠWe Ġcalculated Ġwhich Ġfraction Ġof Ġthe ĠCU Is Ġin Ġeach Ġsection Ġwere Ġlocated Ġin Ġthe Ġrest Ġof Ġthe Ġpatient 's Ġnote Ġfor Ġa Ġspecific Ġhospital Ġstay . ĠWe Ġalso Ġcalculated Ġwhat Ġpercent Ġof Ġthe Ġgenders Ġrecorded Ġin Ġthe Ġdischarge Ġsummary Ġwere Ġalso Ġrecorded Ġin Ġthe Ġstructured Ġdata Ġfor Ġthe Ġpatient . </s> For Ġeach Ġof Ġthe Ġ55 , 177 Ġdischarge Ġsummary Ġreports Ġin Ġthe ĠM IM IC Ġdatabase , Ġwe Ġcalculated Ġwhat Ġfraction Ġof Ġthe ĠCU Is Ġin Ġthe Ġdischarge Ġsummary Ġcould Ġbe Ġfound Ġin Ġthe Ġremaining Ġnotes Ġfor Ġthe Ġpatient 's Ġcurrent Ġadmission Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġin Ġtheir Ġentire Ġlongitudinal Ġmedical Ġrecord Ġ( ĠIN LINE FORM 1 Ġ). ĠTable ĠT AB REF 13 Ġshows Ġthe ĠC UI Ġrecall Ġaveraged Ġacross Ġall Ġdischarge Ġsumm aries Ġby Ġboth Ġsubject _ id Ġand Ġhad m _ id . ĠThe Ġlow Ġrecall Ġsuggests Ġthat Ġclinicians Ġmay Ġincorporate Ġinformation Ġin Ġthe Ġdischarge Ġnote Ġthat Ġhad Ġnot Ġbeen Ġpreviously Ġdocumented Ġin Ġthe ĠE HR . ĠFigure ĠFIG REF 11 Ġplots Ġthe Ġrelationship Ġbetween Ġthe Ġnumber Ġof Ġnon - dis charge Ġnotes Ġfor Ġeach Ġpatient Ġand Ġthe ĠC UI Ġrecall Ġ( top ) Ġand Ġthe Ġnumber Ġof Ġtotal ĠCU Is Ġin Ġnon - dis charge Ġnotes Ġand Ġthe ĠC UI Ġrecall Ġ( middle ). ĠThe Ġnumber Ġof ĠCU Is Ġis Ġa Ġproxy Ġfor Ġthe Ġlength Ġof Ġthe Ġnotes , Ġand Ġas Ġexpected , Ġthe ĠC UI Ġrecall Ġtends Ġto Ġbe Ġhigher Ġin Ġpatients Ġwith Ġmore Ġand Ġlonger Ġnotes . ĠThe Ġbottom Ġpanel Ġin ĠFigure ĠFIG REF 11 Ġdemonstrates Ġthat Ġrecall Ġis Ġnot Ġcorrelated Ġwith Ġthe Ġpatient 's Ġlength Ġof Ġstay Ġoutside Ġthe ĠIC U , Ġwhich Ġindicates Ġthat Ġour Ġupper Ġbound Ġcalculation Ġis Ġnot Ġseverely Ġimpacted Ġby Ġonly Ġhaving Ġaccess Ġto Ġthe Ġpatient 's Ġnotes Ġfrom Ġtheir Ġstay Ġin Ġthe ĠIC U . </s> Finally , ĠTable ĠT AB REF 14 Ġshows Ġthe Ġrecall Ġfor Ġthe Ġsex , Ġchief Ġcomplaint , Ġprocedure , Ġdischarge Ġmedication , Ġand ĠHP I Ġdischarge Ġsummary Ġsections Ġaveraged Ġacross Ġall Ġthe Ġdischarge Ġsumm aries . ĠThe Ġprocedure Ġsection Ġhas Ġthe Ġhighest Ġrecall Ġof Ġ0 . 807 , Ġwhich Ġis Ġunderstandable Ġbecause Ġprocedures Ġundergone Ġduring Ġan Ġin patient Ġstay Ġare Ġmost Ġlikely Ġto Ġbe Ġdocumented Ġin Ġan ĠE HR . ĠThe Ġrecall Ġfor Ġeach Ġof Ġthese Ġfive Ġsections Ġis Ġmuch Ġhigher Ġthan Ġthe Ġoverall Ġrecall Ġin ĠTable ĠT AB REF 13 Ġ, Ġsuggesting Ġthat Ġextract ive Ġsummar ization Ġmay Ġbe Ġeasier Ġfor Ġsome Ġsections Ġof Ġthe Ġdischarge Ġnote . </s> Overall , Ġthis Ġupper Ġbound Ġanalysis Ġsuggests Ġthat Ġwe Ġmay Ġnot Ġbe Ġable Ġto Ġrecreate Ġa Ġdischarge Ġsummary Ġwith Ġextract ive Ġsummar ization Ġalone . ĠWhile ĠC UI Ġcomparison Ġallows Ġfor Ġcomparing Ġmedically Ġrelevant Ġconcepts , Ġc TA K ES 's ĠC UI Ġlab elling Ġprocess Ġis Ġnot Ġperfect , Ġand Ġfurther Ġwork , Ġperhaps Ġthrough Ġsophisticated Ġregular Ġexpressions , Ġis Ġneeded Ġto Ġdefine Ġthe Ġlimits Ġof Ġextract ive Ġsummar ization . </s> Label ing ĠHistory Ġof ĠPresent ĠIll ness ĠNotes </s> We Ġdeveloped Ġa Ġclass ifier Ġto Ġlabel Ġtopics Ġin Ġthe Ġhistory Ġof Ġpresent Ġillness Ġ( H PI ) Ġnotes , Ġincluding Ġdemographics , Ġdiagnosis Ġhistory , Ġand Ġsymptoms / sign s , Ġamong Ġothers . ĠA Ġrandom Ġsample Ġof Ġ5 15 Ġhistory Ġof Ġpresent Ġillness Ġnotes Ġwas Ġtaken , Ġand Ġeach Ġof Ġthe Ġnotes Ġwas Ġmanually Ġannot ated Ġby Ġone Ġof Ġeight Ġannot ators Ġusing Ġthe Ġsoftware ĠMulti - document ĠAn notation ĠEnvironment Ġ( MA E ) ĠB IB REF 20 Ġ. ĠMA E Ġprovides Ġan Ġinteractive ĠGUI Ġfor Ġannot ators Ġand Ġexports Ġthe Ġresults Ġof Ġeach Ġannotation Ġas Ġan ĠXML Ġfile Ġwith Ġtext Ġspans Ġand Ġtheir Ġassociated Ġlabels Ġfor Ġadditional Ġprocessing . Ġ40 % Ġof Ġthe ĠHP I Ġnotes Ġwere Ġlabeled Ġby Ġclinicians Ġand Ġ60 % Ġby Ġnon - cl in icians . ĠTable ĠT AB REF 5 Ġshows Ġthe Ġinstructions Ġgiven Ġto Ġthe Ġannot ators Ġfor Ġeach Ġof Ġthe Ġ10 Ġlabels . ĠThe Ġentire ĠHP I Ġnote Ġwas Ġlabeled Ġwith Ġone Ġof Ġthe Ġlabels , Ġand Ġinstructions Ġwere Ġgiven Ġto Ġlabel Ġeach Ġclause Ġin Ġa Ġsentence Ġwith Ġthe Ġsame Ġlabel Ġwhen Ġpossible . </s> Our ĠL ST M Ġmodel Ġwas Ġadopted Ġfrom Ġprior Ġwork Ġby ĠD ern on court Ġet Ġal ĠB IB REF 21 Ġ. ĠWhereas Ġthe ĠD ern on court Ġmodel Ġjointly Ġclassified Ġeach Ġsentence Ġin Ġa Ġmedical Ġabstract , Ġhere Ġwe Ġjointly Ġclassify Ġeach Ġword Ġin Ġthe ĠHP I Ġsummary . ĠOur Ġmodel Ġconsists Ġof Ġfour Ġlayers : Ġa Ġtoken Ġembed ding Ġlayer , Ġa Ġword Ġcontextual Ġrepresentation Ġlayer , Ġa Ġlabel Ġscoring Ġlayer , Ġand Ġa Ġlabel Ġsequence Ġoptimization Ġlayer Ġ( Figure ĠFIG REF 9 Ġ). </s> In Ġthe Ġfollowing Ġdescriptions , Ġlower case Ġital ics Ġis Ġused Ġto Ġdenote Ġscal ars , Ġlower case Ġbold Ġis Ġused Ġto Ġdenote Ġvectors , Ġand Ġu pperc ase Ġital ics Ġis Ġused Ġto Ġdenote Ġmat rices . </s> Token ĠEmb edd ing ĠLayer : ĠIn Ġthe Ġtoken Ġembed ding Ġlayer , Ġpret rained Ġword Ġembed d ings Ġare Ġcombined Ġwith Ġlearned Ġcharacter Ġembed d ings Ġto Ġcreate Ġa Ġhybrid Ġtoken Ġembed ding Ġfor Ġeach Ġword Ġin Ġthe ĠHP I Ġnote . ĠThe Ġword Ġembed d ings , Ġwhich Ġare Ġdirect Ġm appings Ġfrom Ġword ĠIN LINE FORM 0 Ġto Ġvector ĠIN LINE FORM 1 Ġ, Ġwere Ġpret rained Ġusing Ġword 2 vec ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġon Ġall Ġof Ġthe Ġnotes Ġin ĠM IM IC Ġ( v 30 ) Ġand Ġonly Ġthe Ġdischarge Ġnotes . ĠBoth Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġ( CB OW ) Ġand Ġskip Ġgram Ġmodels Ġwere Ġexplored . </s> Let ĠIN LINE FORM 0 Ġbe Ġthe Ġsequence Ġof Ġcharacters Ġcomprising Ġthe Ġword ĠIN LINE FORM 1 Ġ. ĠEach Ġcharacter Ġis Ġmapped Ġto Ġits Ġembed ding ĠIN LINE FORM 2 Ġ, Ġand Ġall Ġembed d ings Ġare Ġinput Ġinto Ġa Ġbid irection al ĠL ST M , Ġwhich Ġultimately Ġoutputs ĠIN LINE FORM 3 Ġ, Ġthe Ġcharacter Ġembed ding Ġof Ġthe Ġword ĠIN LINE FORM 4 Ġ. </s> The Ġoutput Ġof Ġthe Ġtoken Ġembed ding Ġlayer Ġis Ġthe Ġvector Ġe , Ġwhich Ġis Ġthe Ġresult Ġof Ġconc aten ation Ġof Ġthe Ġword Ġembed ding , Ġt , Ġand Ġthe Ġcharacter Ġembed ding , Ġc . </s> Context ual ĠRepresent ation ĠLayer : ĠThe Ġcontextual Ġrepresentation Ġlayer Ġtakes Ġas Ġinput Ġthe Ġsequence Ġof Ġword Ġembed d ings , ĠIN LINE FORM 0 Ġ, Ġand Ġoutputs Ġan Ġembed ding Ġof Ġthe Ġcontextual Ġrepresentation Ġfor Ġeach Ġword Ġin Ġthe ĠHP I Ġnote . ĠThe Ġword Ġembed d ings Ġare Ġfed Ġinto Ġa Ġbi - direction al ĠL ST M , Ġwhich Ġoutputs ĠIN LINE FORM 1 Ġ, Ġa Ġconc aten ation Ġof Ġthe Ġhidden Ġstates Ġof Ġthe Ġtwo ĠL ST Ms Ġfor Ġeach Ġword . </s> Label ĠSc oring ĠLayer : ĠAt Ġthis Ġpoint , Ġeach Ġword ĠIN LINE FORM 0 Ġis Ġassociated Ġwith Ġa Ġhidden Ġrepresentation Ġof Ġthe Ġword , ĠIN LINE FORM 1 Ġ. ĠIn Ġthe Ġlabel Ġscoring Ġlayer , Ġwe Ġuse Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġwith Ġone Ġhidden Ġlayer Ġto Ġoutput Ġa Ġscore Ġassociated Ġwith Ġeach Ġof Ġthe Ġ10 Ġcategories Ġfor Ġeach Ġword . ĠLet ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġ. ĠWe Ġcan Ġcompute Ġa Ġvector Ġof Ġscores Ġs Ġ= ĠIN LINE FORM 4 Ġwhere Ġthe Ġit h Ġcomponent Ġof Ġs Ġis Ġthe Ġscore Ġof Ġclass Ġi Ġfor Ġa Ġgiven Ġword . </s> Label ĠSequence ĠOptim ization ĠLayer : ĠThe ĠLabel ĠSequence ĠOptim ization ĠLayer Ġcomp utes Ġthe Ġprobability Ġof Ġa Ġlabeling Ġsequence Ġand Ġfinds Ġthe Ġsequence Ġwith Ġthe Ġhighest Ġprobability . ĠIn Ġorder Ġto Ġcondition Ġthe Ġlabel Ġfor Ġeach Ġword Ġon Ġthe Ġlabels Ġof Ġits Ġneighbors , Ġwe Ġemploy Ġa Ġlinear Ġchain Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġto Ġdefine Ġa Ġglobal Ġscore , ĠIN LINE FORM 0 Ġ, Ġfor Ġa Ġsequence Ġof Ġwords Ġand Ġtheir Ġassociated Ġscores ĠIN LINE FORM 1 Ġand Ġlabels , ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> where ĠT Ġis Ġa Ġtransition Ġmatrix ĠIN LINE FORM 0 ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġvectors Ġof Ġscores Ġthat Ġdescribe Ġthe Ġcost Ġof Ġbeginning Ġor Ġending Ġwith Ġa Ġlabel . </s> The Ġprobability Ġof Ġa Ġsequence Ġof Ġlabels Ġis Ġcalculated Ġby Ġapplying Ġa Ġsoft max Ġlayer Ġto Ġobtain Ġa Ġprobability Ġof Ġa Ġsequence Ġof Ġlabels : ĠDIS PLAY FORM 0 </s> Cross - ent ropy Ġloss , ĠIN LINE FORM 0 Ġ, Ġis Ġused Ġas Ġthe Ġobjective Ġfunction Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġcorrect Ġsequence Ġof Ġlabels Ġand Ġthe Ġprobability ĠIN LINE FORM 2 Ġis Ġcalculated Ġaccording Ġto Ġthe ĠCR F . </s> We Ġevaluated Ġour Ġmodel Ġon Ġthe Ġ5 15 Ġannot ated Ġhistory Ġof Ġpresent Ġillness Ġnotes , Ġwhich Ġwere Ġsplit Ġin Ġa Ġ70 % Ġtrain Ġset , Ġ15 % Ġdevelopment Ġset , Ġand Ġa Ġ15 % Ġtest Ġset . ĠThe Ġmodel Ġis Ġtrained Ġusing Ġthe ĠAdam Ġalgorithm Ġfor Ġgradient - based Ġoptimization ĠB IB REF 25 Ġwith Ġan Ġinitial Ġlearning Ġrate Ġ= Ġ0 . 001 Ġand Ġdecay Ġ= Ġ0 . 9 . ĠA Ġdrop out Ġrate Ġof Ġ0 . 5 Ġwas Ġapplied Ġfor Ġregular ization , Ġand Ġeach Ġbatch Ġsize Ġ= Ġ20 . ĠThe Ġmodel Ġran Ġfor Ġ20 Ġepoch s Ġand Ġwas Ġhalted Ġearly Ġif Ġthere Ġwas Ġno Ġimprovement Ġafter Ġ3 Ġepoch s . </s> We Ġevaluated Ġthe Ġimpact Ġof Ġcharacter Ġembed d ings , Ġthe Ġchoice Ġof Ġpret rained Ġw 2 v Ġembed d ings , Ġand Ġthe Ġaddition Ġof Ġlearned Ġword Ġembed d ings Ġon Ġmodel Ġperformance Ġon Ġthe Ġdev Ġset . ĠWe Ġreport Ġperformance Ġof Ġthe Ġbest Ġperforming Ġmodel Ġon Ġthe Ġtest Ġset . </s> Table ĠT AB REF 16 Ġcompares Ġdev Ġset Ġperformance Ġof Ġthe Ġmodel Ġusing Ġvarious Ġpret rained Ġword Ġembed d ings , Ġwith Ġand Ġwithout Ġcharacter Ġembed d ings , Ġand Ġwith Ġpret rained Ġversus Ġlearned Ġword Ġembed d ings . ĠThe Ġfirst Ġrow Ġin Ġeach Ġsection Ġis Ġthe Ġperformance Ġof Ġthe Ġmodel Ġarchitecture Ġdescribed Ġin Ġthe Ġmethods Ġsection Ġfor Ġcomparison . ĠModels Ġusing Ġword Ġembed d ings Ġtrained Ġon Ġthe Ġdischarge Ġsumm aries Ġperformed Ġbetter Ġthan Ġword Ġembed d ings Ġtrained Ġon Ġall ĠM IM IC Ġnotes , Ġlikely Ġbecause Ġthe Ġdischarge Ġsummary Ġword Ġembed d ings Ġbetter Ġcaptured Ġword Ġuse Ġin Ġdischarge Ġsumm aries Ġalone . ĠInterestingly , Ġthe Ġcontinuous Ġbag Ġof Ġwords Ġembed d ings Ġoutper formed Ġskip Ġgram Ġembed d ings , Ġwhich Ġis Ġsurprising Ġbecause Ġthe Ġskip Ġgram Ġarchitecture Ġtypically Ġworks Ġbetter Ġfor Ġinf requent Ġwords ĠB IB REF 26 Ġ. ĠAs Ġexpected , Ġinclusion Ġof Ġcharacter Ġembed d ings Ġincreases Ġperformance Ġby Ġapproximately Ġ3 %. ĠThe Ġmodel Ġwith Ġword Ġembed d ings Ġlearned Ġin Ġthe Ġmodel Ġachieves Ġthe Ġhighest Ġperformance Ġon Ġthe Ġdev Ġset Ġ( 0 . 886 ), Ġwhich Ġmay Ġbe Ġbecause Ġthe Ġpret rained Ġworm Ġembed d ings Ġwere Ġtrained Ġon Ġa Ġprevious Ġversion Ġof ĠM IM IC . ĠAs Ġa Ġresult , Ġsome Ġwords Ġin Ġthe Ġdischarge Ġsumm aries , Ġsuch Ġas Ġmi - spe lled Ġwords Ġor Ġrare r Ġdiseases Ġand Ġmedications , Ġdid Ġnot Ġhave Ġassociated Ġword Ġembed d ings . ĠPer forming Ġa Ġsimple Ġspell Ġcorrection Ġon Ġout Ġof Ġvoc ab Ġwords Ġmay Ġimprove Ġperformance Ġwith Ġpret rained Ġword Ġembed d ings . </s> We Ġevaluated Ġthe Ġbest Ġperforming Ġmodel Ġon Ġthe Ġtest Ġset . ĠThe ĠLearned ĠWord ĠEmb edd ings Ġmodel Ġachieved Ġan Ġaccuracy Ġof Ġ0 . 88 Ġand Ġan ĠF 1 - Score Ġof Ġ0 . 8 76 Ġon Ġthe Ġtest Ġset . ĠTable ĠT AB REF 17 Ġshows Ġthe Ġprecision , Ġrecall , ĠF 1 Ġscore , Ġand Ġsupport Ġfor Ġeach Ġof Ġthe Ġten Ġlabels , Ġand ĠFigure ĠFIG REF 18 Ġshows Ġthe Ġconfusion Ġmatrix Ġillustrating Ġwhich Ġlabels Ġwere Ġfrequently Ġmis classified . ĠThe Ġdemographics Ġand Ġpatient Ġmovement Ġlabels Ġachieved Ġthe Ġhighest ĠF 1 Ġscores Ġ( 0 . 96 Ġand Ġ0 . 93 Ġrespectively ) Ġwhile Ġthe Ġv itals / l abs Ġand Ġmedication Ġhistory Ġlabels Ġhad Ġthe Ġlowest ĠF 1 Ġscores Ġ( 0 . 40 Ġand Ġ0 . 66 Ġrespectively ). ĠThe Ġdemographics Ġsection Ġconsistently Ġoccurs Ġat Ġthe Ġbeginning Ġof Ġthe ĠHP I Ġnote , Ġand Ġthe Ġpatient Ġmovement Ġsection Ġuses Ġa Ġlimited Ġvoc ab Ġ( trans ferred , Ġadmitted , Ġetc .), Ġwhich Ġmay Ġexplain Ġtheir Ġhigh ĠF 1 Ġscores . ĠOn Ġthe Ġother Ġhand , Ġthe Ġv itals / l abs Ġand Ġmedication Ġhistory Ġsections Ġhad Ġthe Ġlowest Ġsupport , Ġwhich Ġmay Ġexplain Ġwhy Ġthey Ġwere Ġmore Ġchallenging Ġto Ġlabel . </s> Words Ġthat Ġbelonged Ġto Ġthe Ġdiagnosis Ġhistory , Ġpatient Ġmovement , Ġand Ġprocedure / results Ġsections Ġwere Ġfrequently Ġlabeled Ġas Ġsymptoms / sign s Ġ( Figure ĠFIG REF 18 Ġ). ĠDiagn osis Ġhistory Ġsections Ġmay Ġbe Ġlabeled Ġfrequently Ġas Ġsymptoms / sign s Ġbecause Ġsymptoms / d ise ases Ġcan Ġeither Ġbe Ġdescribed Ġas Ġpart Ġof Ġthe Ġpatient 's Ġdiagnosis Ġhistory Ġor Ġcurrent Ġsymptoms Ġdepending Ġon Ġwhen Ġthe Ġsymptom / d ise ase Ġoccurred . ĠHowever , Ġmany Ġof Ġthe Ġmis class ification Ġerrors Ġmay Ġbe Ġdue Ġto Ġinconsistency Ġin Ġmanual Ġlab elling Ġamong Ġannot ators . ĠFor Ġexample , Ġsentences Ġdescribing Ġboth Ġpatient Ġmovement Ġand Ġpatient Ġsymptoms Ġ( e . g . Ġ" the Ġpatient Ġwas Ġtransferred Ġto Ġthe Ġhospital Ġfor Ġhis Ġhypertension ") Ġwere Ġlabeled Ġentirely Ġas Ġ' patient Ġmovement ' Ġby Ġsome Ġannot ators Ġwhile Ġother Ġannot ators Ġlabeled Ġthe Ġdifferent Ġclauses Ġof Ġthe Ġsentence Ġseparately Ġas Ġ' patient Ġmovement ' Ġand Ġ' sym ptoms / sign s .' ĠFurther Ġstandard ization Ġamong Ġannot ators Ġis Ġneeded Ġto Ġavoid Ġthese Ġmis class ifications . ĠFuture Ġwork Ġis Ġneeded Ġto Ġobtain Ġadditional Ġmanual Ġannotations Ġwhere Ġeach ĠHP I Ġnote Ġis Ġannot ated Ġby Ġmultiple Ġannot ators . ĠThis Ġwill Ġallow Ġfor Ġcalculation Ġof ĠCohen 's Ġk appa , Ġwhich Ġmeasures Ġinter - annot ator Ġagreement , Ġand Ġcomparison Ġof Ġclin ician Ġand Ġnon - cl in ician Ġannot ator Ġreliability . </s> Future Ġwork Ġis Ġalso Ġneeded Ġto Ġbetter Ġunderstand Ġcommonly Ġmis label ed Ġcategories Ġand Ġexplore Ġalternative Ġmodel Ġarchitectures . ĠHere Ġwe Ġperform Ġword Ġlevel Ġlabel Ġprediction , Ġwhich Ġcan Ġresult Ġin Ġphrases Ġthat Ġcontain Ġmultiple Ġlabels . ĠFor Ġexample , Ġthe Ġphrase Ġ" history Ġof Ġneck Ġpain " Ġcan Ġbe Ġlabeled Ġwith Ġboth Ġ' diagn osis Ġhistory ' Ġand Ġ' sym ptoms / sign s ' Ġlabels . ĠPost - processing Ġis Ġneeded Ġto Ġcreate Ġa Ġfinal Ġlabel Ġprediction Ġfor Ġeach Ġphrase . ĠWhile Ġphrase Ġlevel Ġprediction Ġmay Ġresolve Ġthese Ġchallenges , Ġit Ġis Ġdifficult Ġto Ġsegment Ġthe ĠHP I Ġnote Ġinto Ġphrases Ġfor Ġprediction , Ġas Ġa Ġsingle Ġphrase Ġmay Ġtruly Ġcontain Ġmultiple Ġlabels . ĠSe gment ation Ġof Ġsentences Ġby Ġpunct uation , Ġconj unctions , Ġand Ġpre pos itions Ġmay Ġyield Ġthe Ġbest Ġphrase Ġchunk er Ġfor Ġdischarge Ġsummary Ġtext . </s> Finally , Ġsupplement ing Ġthe Ġword Ġembed d ings Ġin Ġour ĠL ST M Ġmodel Ġwith ĠCU Is Ġmay Ġfurther Ġimprove Ġperformance . ĠWhile Ġword Ġembed d ings Ġdo Ġwell Ġin Ġlearning Ġthe Ġcontextual Ġcontext Ġof Ġwords , ĠCU Is Ġallow Ġfor Ġmore Ġexplicit Ġincorporation Ġof Ġmedical Ġdomain Ġexpertise . ĠBy Ġconc aten ating Ġthe ĠC UI Ġfor Ġeach Ġword Ġwith Ġits Ġhybrid Ġtoken Ġembed ding , Ġwe Ġmay Ġbe Ġable Ġto Ġleverage Ġboth Ġdata Ġdriven Ġand Ġont ology Ġdriven Ġapproaches . </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġdeveloped Ġa ĠC UI - based Ġupper Ġbound Ġon Ġextract ive Ġsummar ization Ġof Ġdischarge Ġsumm aries Ġand Ġpresented Ġa ĠN N Ġarchitecture Ġthat Ġjointly Ġclass ifies Ġwords Ġin Ġhistory Ġof Ġpresent Ġillness Ġnotes . ĠWe Ġdemonstrate Ġthat Ġour Ġmodel Ġcan Ġachieve Ġexcellent Ġperformance Ġon Ġa Ġsmall Ġdataset Ġwith Ġknown Ġheterogeneity Ġamong Ġannot ators . ĠThis Ġmodel Ġcan Ġbe Ġapplied Ġto Ġthe Ġ55 , 000 Ġdischarge Ġsumm aries Ġin ĠM IM IC Ġto Ġcreate Ġa Ġdataset Ġfor Ġevaluation Ġof Ġextract ive Ġsummar ization Ġmethods . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank Ġour Ġannot ators , ĠAndrew ĠGoldberg , ĠLaurie ĠAl sent zer , ĠElaine ĠGoldberg , ĠAndy ĠAl sent zer , ĠGrace ĠLo , Ġand ĠJosh ĠDon is . ĠWe Ġwould Ġalso Ġlike Ġto Ġacknowledge ĠPete ĠSz ol ov its Ġfor Ġhis Ġguidance Ġand Ġfor Ġproviding Ġthe Ġpret rained Ġword Ġembed d ings Ġand ĠTr istan ĠNa umann Ġfor Ġproviding Ġthe ĠM IM IC ĠCU Is .