 simple lookup table embeddings learned from scratch, using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13
EXTRACTIVE
ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13
EXTRACTIVE
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors). Refer to Table TABREF5 for the performance comparisons.
Lack of availability of a large volume of data is a typical challenge in healthcare. A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns). Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise. To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task. We further investigate the effects of training data size on our models.
1912.04961
Introduction </s> To Ġappear Ġin ĠProceedings Ġof ĠInternational ĠWorkshop Ġon ĠHealth ĠIntelligence Ġ( W 3 PH IA I ) Ġof Ġthe Ġ34 th ĠAAA I ĠConference Ġon ĠArtificial ĠIntelligence , Ġ2020 . </s> Phys ician Ġburn out Ġis Ġa Ġgrowing Ġconcern , Ġestimated Ġto Ġbe Ġexperienced Ġby Ġat Ġleast Ġ35 % Ġof Ġphysicians Ġin Ġthe Ġdeveloping Ġworld Ġand Ġ50 % Ġin Ġthe ĠUnited ĠStates ĠB IB REF 0 . ĠB IB REF 1 Ġfound Ġthat Ġfor Ġevery Ġhour Ġphysicians Ġprovide Ġdirect Ġclinical Ġfac etime Ġto Ġpatients , Ġnearly Ġtwo Ġadditional Ġhours Ġare Ġspent Ġon ĠE HR Ġ( Elect ronic ĠHealth ĠRecords ) Ġand Ġadministrative Ġor Ġdesk Ġwork . ĠAs Ġper Ġthe Ġstudy Ġconducted Ġby ĠMassachusetts ĠGeneral ĠPhysicians ĠOrganization Ġ( M PG O ) ĠB IB REF 2 Ġand Ġas Ġreported Ġby ĠB IB REF 3 , Ġthe Ġaverage Ġtime Ġspent Ġon Ġadministrative Ġtasks Ġincreased Ġfrom Ġ23 . 7 % Ġin Ġ2014 Ġto Ġ27 . 9 % Ġin Ġ2017 . ĠBoth Ġthe Ġsurveys Ġfound Ġthat Ġtime Ġspent Ġon Ġadministrative Ġtasks Ġwas Ġpositively Ġassociated Ġwith Ġhigher Ġlikelihood Ġof Ġburn out . ĠTop Ġreasons Ġunder Ġadministrative Ġburden Ġinclude Ġworking Ġon Ġthe Ġambul atory ĠE HR , Ġhandling Ġmedication Ġreconciliation Ġ( sometimes Ġdone Ġby Ġaids ), Ġmedication Ġrenew als , Ġand Ġmedical Ġbilling Ġand Ġcoding . ĠThe Ġmajority Ġof Ġthese Ġreasons Ġrev olve Ġaround Ġdocumentation Ġof Ġinformation Ġexchanged Ġbetween Ġdoctors Ġand Ġpatients Ġduring Ġthe Ġclinical Ġencounters . ĠAutom atically Ġextracting Ġsuch Ġclinical Ġinformation ĠB IB REF 4 , ĠB IB REF 5 Ġcan Ġnot Ġonly Ġhelp Ġalleviate Ġthe Ġdocumentation Ġburden Ġon Ġthe Ġphysician , Ġbut Ġalso Ġallow Ġthem Ġto Ġdedicate Ġmore Ġtime Ġdirectly Ġwith Ġpatients . </s> Among Ġall Ġthe Ġclinical Ġinformation Ġextraction Ġtasks , ĠMed ication ĠReg imen Ġ( Med ication , Ġdosage , Ġand Ġfrequency ) Ġextraction Ġis Ġparticularly Ġinteresting Ġdue Ġto Ġits Ġability Ġto Ġhelp Ġdoctors Ġwith Ġmedication Ġorders Ġcum Ġrenew als , Ġmedication Ġreconciliation , Ġpotentially Ġverifying Ġthe Ġreconc ili ations Ġfor Ġerrors , Ġand , Ġother Ġmedication - centered ĠE HR Ġdocumentation Ġtasks . ĠIn Ġaddition , Ġthe Ġsame Ġinformation Ġwhen Ġprovided Ġto Ġpatients Ġcan Ġhelp Ġthem Ġwith Ġbetter Ġrecall Ġof Ġdoctor Ġinstructions Ġwhich Ġmight Ġaid Ġin Ġcompliance Ġwith Ġthe Ġcare Ġplan . ĠThis Ġis Ġparticularly Ġimportant Ġgiven Ġthat Ġpatients Ġforget Ġor Ġwrongly Ġrecol lect Ġ40 - 80 % ĠB IB REF 6 Ġof Ġwhat Ġis Ġdiscussed Ġin Ġthe Ġclinic , Ġand Ġaccessing ĠE HR Ġdata Ġhas Ġits Ġown Ġchallenges . </s> Sp ont aneous Ġclinical Ġconversations Ġhappening Ġbetween Ġa Ġdoctor Ġand Ġa Ġpatient , Ġhave Ġseveral Ġdistinguishing Ġcharacteristics Ġfrom Ġa Ġnormal Ġmon ologue Ġor Ġprepared Ġspeech : Ġit Ġinvolves Ġmultiple Ġspeakers Ġwith Ġoverlapping Ġdialog ues , Ġcovers Ġa Ġvariety Ġof Ġspeech Ġpatterns , Ġand Ġthe Ġvocabulary Ġcan Ġrange Ġfrom Ġcoll oqu ial Ġto Ġcomplex Ġdomain - specific Ġlanguage . ĠWith Ġrecent Ġadvancements Ġin ĠConvers ational ĠSpeech ĠRecogn ition ĠB IB REF 7 Ġrendering Ġthe Ġsystems Ġless Ġprone Ġto Ġerrors , Ġthe Ġsubsequent Ġchallenge Ġof Ġunderstanding Ġand Ġextracting Ġrelevant Ġinformation Ġfrom Ġthe Ġconversations Ġis Ġreceiving Ġincreasing Ġresearch Ġfocus ĠB IB REF 4 , ĠB IB REF 8 . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġlocal Ġinformation Ġextraction Ġin Ġtransc ribed Ġclinical Ġconversations . ĠSpecifically , Ġwe Ġextract Ġdosage Ġ( e . g . Ġ5 mg ) Ġand Ġfrequency Ġ( e . g . Ġonce Ġa Ġday ) Ġfor Ġthe Ġmedications Ġ( e . g . Ġaspirin ) Ġfrom Ġthese Ġtranscripts , Ġcollectively Ġreferred Ġto Ġas ĠMed ication ĠReg imen Ġ( MR ) Ġextraction . ĠThe Ġinformation Ġextraction Ġis Ġlocal Ġas Ġwe Ġextract Ġthe Ġinformation Ġfrom Ġa Ġsegment Ġof Ġthe Ġtranscript Ġand Ġnot Ġthe Ġentire Ġtranscript Ġsince Ġdoing Ġthe Ġlatter Ġis Ġdifficult Ġowing Ġto Ġthe Ġlong Ġme andering Ġnature Ġof Ġthe Ġconversations Ġoften Ġwith Ġmultiple Ġmedication Ġreg imens Ġand Ġcare Ġplans Ġbeing Ġdiscussed . </s> The Ġchallenges Ġassociated Ġwith Ġthe ĠMed ication ĠReg imen Ġ( MR ) Ġextraction Ġtask Ġinclude Ġunderstanding Ġthe Ġspontaneous Ġdialog Ġwith Ġclinical Ġvocabulary Ġand Ġunderstanding Ġthe Ġrelationship Ġbetween Ġdifferent Ġentities Ġas Ġthe Ġdiscussion Ġcan Ġcontain Ġmultiple Ġmedications Ġand Ġdos ages Ġ( e . g . Ġdoctor Ġrev ising Ġa Ġdosage Ġor Ġreviewing Ġall Ġthe Ġcurrent Ġmedications ). </s> We Ġframe Ġthis Ġproblem Ġas Ġa ĠQuestion ĠAn sw ering Ġ( Q A ) Ġtask Ġby Ġgenerating Ġquestions Ġusing Ġtemplates . ĠWe Ġbase Ġthe ĠQ A Ġmodel Ġon Ġpointer - gener ator Ġnetworks ĠB IB REF 9 Ġaugmented Ġwith ĠCo - Att ent ions ĠB IB REF 10 . ĠIn Ġaddition , Ġwe Ġdevelop Ġmodels Ġcombining ĠQ A Ġand ĠInformation ĠExt raction Ġframeworks Ġusing Ġmulti - dec oder Ġ( one Ġeach Ġfor Ġdosage Ġand Ġfrequency ) Ġarchitecture . </s> L ack Ġof Ġavailability Ġof Ġa Ġlarge Ġvolume Ġof Ġdata Ġis Ġa Ġtypical Ġchallenge Ġin Ġhealthcare . ĠA Ġconversation Ġcorpus Ġby Ġitself Ġis Ġa Ġrare Ġcommodity Ġin Ġthe Ġhealthcare Ġdata Ġspace Ġbecause Ġof Ġthe Ġcost Ġand Ġdifficulty Ġin Ġhanding Ġ( because Ġof Ġdata Ġprivacy Ġconcerns ). ĠMoreover , Ġtransc ribing Ġand Ġlabeling Ġthe Ġconversations Ġis Ġa Ġcostly Ġprocess Ġas Ġit Ġrequires Ġdomain - specific Ġmedical Ġannotation Ġexpertise . ĠTo Ġaddress Ġdata Ġshortage Ġand Ġimprove Ġthe Ġmodel Ġperformance , Ġwe Ġinvestigate Ġdifferent Ġhigh - performance Ġcontextual Ġembed d ings Ġ( EL MO ĠB IB REF 11 , ĠB ERT ĠB IB REF 12 Ġand ĠClinical BER T ĠB IB REF 13 ), Ġand Ġpret rain Ġthe Ġmodels Ġon Ġa Ġclinical Ġsummar ization Ġtask . ĠWe Ġfurther Ġinvestigate Ġthe Ġeffects Ġof Ġtraining Ġdata Ġsize Ġon Ġour Ġmodels . </s> On Ġthe ĠMR Ġextraction Ġtask , ĠEL Mo Ġwith Ġenc oder Ġmulti - dec oder Ġarchitecture Ġand ĠB ERT Ġwith Ġenc oder - dec oder Ġwith Ġenc od ers Ġpret rained Ġon Ġthe Ġsummar ization Ġtask Ġperform Ġthe Ġbest . ĠThe Ġbest - performing Ġmodels Ġimprove Ġour Ġbaseline 's Ġdosage Ġand Ġfrequency Ġextract ions ĠR OU GE - 1 ĠF 1 Ġscores Ġfrom Ġ54 . 28 Ġand Ġ37 . 13 Ġto Ġ89 . 57 Ġand Ġ45 . 94 , Ġrespectively . </s> Using Ġour Ġmodels , Ġwe Ġpresent Ġthe Ġfirst Ġfully Ġautomated Ġsystem Ġto Ġextract ĠMR Ġtags Ġfrom Ġspontaneous Ġdoctor - patient Ġconversations . ĠWe Ġevaluate Ġthe Ġsystem Ġ( using Ġour Ġbest Ġperforming Ġmodels ) Ġon Ġthe Ġtranscripts Ġgenerated Ġfrom ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) ĠAPIs Ġoffered Ġby ĠGoogle Ġand ĠIBM . ĠIn ĠGoogle ĠAS R 's Ġtranscripts , Ġour Ġbest Ġmodel Ġobtained ĠR OU GE - 1 ĠF 1 Ġof Ġ71 . 75 Ġfor ĠDos age Ġextraction Ġ( which Ġin Ġthis Ġspecific Ġcase Ġequals Ġto Ġthe Ġpercentage Ġof Ġtimes Ġdosage Ġis Ġcorrect , Ġrefer ĠMet rics ĠSection Ġfor Ġmore Ġdetails ) Ġand Ġ40 . 13 Ġfor ĠFrequency Ġextraction Ġtasks . ĠOn Ġqualitative Ġevaluation , Ġwe Ġfind Ġthat Ġfor Ġ73 . 58 % Ġof Ġthe Ġmedications Ġthe Ġmodel Ġcan Ġfind Ġthe Ġcorrect Ġfrequency . ĠThese Ġresults Ġdemonstrate Ġthat Ġthe Ġresearch Ġon ĠN LP Ġcan Ġbe Ġused Ġeffectively Ġin Ġa Ġreal Ġclinical Ġsetting Ġto Ġbenefit Ġboth Ġdoctors Ġand Ġpatients </s> Data </s> Our Ġdataset Ġconsists Ġof Ġa Ġtotal Ġof Ġ6 , 693 Ġreal Ġdoctor - patient Ġconversations Ġrecorded Ġin Ġa Ġclinical Ġsetting Ġusing Ġdistant Ġmicrophones Ġof Ġvarying Ġquality . ĠThe Ġrecordings Ġhave Ġan Ġaverage Ġduration Ġof Ġ9 min Ġ28 s Ġand Ġhave Ġa Ġver bat im Ġtranscript Ġof Ġ1 , 500 Ġwords Ġon Ġaverage Ġ( written Ġby Ġthe Ġexperts ). ĠBoth Ġthe Ġaudio Ġand Ġthe Ġtranscript Ġare Ġde - identified Ġ( by Ġremoving Ġthe Ġidentifying Ġinformation ) Ġwith Ġdigital Ġz eros Ġand Ġ[ de - identified ] Ġtags , Ġrespectively . ĠThe Ġsentences Ġin Ġthe Ġtranscript Ġare Ġgrounded Ġto Ġthe Ġaudio Ġwith Ġthe Ġtim est amps Ġof Ġits Ġfirst Ġand Ġlast Ġword . </s> The Ġtranscript Ġof Ġthe Ġconversations Ġare Ġannot ated Ġwith Ġsumm aries Ġand ĠMed ication ĠReg imen Ġtags Ġ( MR Ġtags ), Ġboth Ġgrounded Ġusing Ġthe Ġtim est amps Ġof Ġthe Ġsentences Ġfrom Ġthe Ġtranscript Ġdeemed Ġrelevant Ġby Ġthe Ġexpert Ġannot ators , Ġrefer Ġto ĠTable ĠT AB REF 1 . ĠThe Ġtranscript Ġfor Ġa Ġtypical Ġconversation Ġcan Ġbe Ġquite Ġlong , Ġand Ġnot Ġeasy Ġfor Ġmany Ġof Ġthe Ġhigh Ġperforming Ġdeep Ġlearning Ġmodels Ġto Ġact Ġon . ĠMoreover , Ġthe Ġmedical Ġinformation Ġabout Ġa Ġconcept / condition / entity Ġcan Ġchange Ġduring Ġthe Ġconversation Ġafter Ġa Ġsignificant Ġtime Ġgap . ĠFor Ġexample , Ġdosage Ġof Ġa Ġmedication Ġcan Ġbe Ġdifferent Ġwhen Ġdiscussing Ġcurrent Ġmedication Ġthe Ġpatient Ġis Ġon Ġvs Ġwhen Ġthey Ġare Ġprescribed Ġa Ġdifferent Ġdosage . ĠHence , Ġwe Ġhave Ġannotations , Ġthat Ġare Ġgrounded Ġto Ġa Ġshort Ġsegment Ġof Ġthe Ġtranscript . </s> The Ġsumm aries Ġ(# words Ġ- Ġ$\ mu Ġ= Ġ9 . 7 ; Ġ\ s igma Ġ= Ġ10 . 1 $ ) Ġare Ġmedically Ġrelevant Ġand Ġlocal . ĠThe ĠMR Ġtags Ġare Ġalso Ġlocal Ġand Ġare Ġof Ġthe Ġform Ġ{ Med ication ĠName , ĠDos age , ĠFrequency }. ĠIf Ġdosage Ġ($ \ mu Ġ= Ġ2 . 0 ; Ġ\ s igma Ġ= Ġ0 $ ) Ġor Ġfrequency Ġ($ \ mu Ġ= Ġ2 . 1 ; Ġ\ s igma Ġ= Ġ1 . 07 $ ) Ġinformation Ġfor Ġa Ġmedication Ġis Ġnot Ġpresent Ġin Ġa Ġgrounded Ġsentence , Ġthe Ġcorresponding Ġfield Ġin Ġthe ĠMR Ġtag Ġwill Ġbe Ġmarked Ġas Ġ` none '. </s> In Ġthe ĠMR Ġtags , ĠMed ication ĠName Ġand ĠDos age Ġ( usually Ġa Ġquantity Ġfollowed Ġby Ġits Ġunits ) Ġcan Ġbe Ġrelatively Ġeasily Ġextracted Ġfrom Ġthe Ġtranscript Ġexcept Ġfor Ġthe Ġunits Ġof Ġthe Ġdosage Ġwhich Ġis Ġsometimes Ġinferred . ĠIn Ġcontrast , Ġdue Ġto Ġhigh Ġdegree Ġof Ġlinguistic Ġvariation Ġwith Ġwhich ĠFrequency Ġis Ġoften Ġexpressed , Ġextracting ĠFrequency Ġrequires Ġan Ġadditional Ġinference Ġstep . ĠFor Ġexample , Ġ` take Ġone Ġin Ġthe Ġmorning Ġand Ġat Ġnoon ' Ġfrom Ġthe Ġtranscript Ġis Ġtagged Ġas Ġ` tw ice Ġa Ġday ' Ġin Ġthe Ġfrequency Ġtag , Ġlikewise Ġ` take Ġit Ġbefore Ġsleeping ' Ġis Ġtagged Ġas Ġ` at Ġnight Ġtime '. </s> Out Ġof Ġoverall Ġ6 , 693 Ġfiles , Ġwe Ġset Ġaside Ġa Ġrandom Ġsample Ġof Ġ423 Ġfiles Ġ( den oted Ġas Ġ$\ math cal Ġ{ D } _{ test } $ ) Ġfor Ġfinal Ġevaluation . ĠThe Ġremaining Ġ6 , 270 Ġfiles Ġare Ġused Ġfor Ġtraining Ġwith Ġ80 % Ġtrain Ġ( 5 016 ), Ġ10 % Ġvalidation Ġ( 627 ), Ġand Ġ10 % Ġtest Ġ( 627 ) Ġsplit . ĠOverall , Ġthe Ġ6 , 270 Ġfiles Ġcontains Ġ156 , 186 Ġsumm aries Ġand Ġ32 , 000 ĠMR Ġtags Ġout Ġof Ġwhich Ġ8 , 654 ĠMR Ġtags Ġcontain Ġvalues Ġfor Ġat Ġleast Ġone Ġof Ġthe ĠDos age Ġor ĠFrequency , Ġwhich Ġwe Ġused Ġfor Ġtraining Ġto Ġavoid Ġover fitting Ġ( the Ġremaining ĠMR Ġtags Ġhave Ġboth ĠDos age Ġand ĠFrequency Ġas Ġ` none '). ĠNote Ġthat Ġwe Ġhave Ġtwo Ġtest Ġdatasets : Ġ` 10 % Ġtest ' Ġ- Ġused Ġto Ġevaluate Ġall Ġthe Ġmodels , Ġand Ġ$\ math cal Ġ{ D } _{ test } $ Ġ- Ġused Ġto Ġmeasure Ġthe Ġperformance Ġof Ġbest Ġperforming Ġmodels Ġon ĠAS R Ġtranscripts . </s> App roach </s> We Ġframe Ġthe ĠMed ication ĠReg imen Ġextraction Ġproblem Ġas Ġa ĠQuestion ĠAn sw ering Ġ( Q A ) Ġtask , Ġwhich Ġforms Ġthe Ġbasis Ġfor Ġour Ġfirst Ġapproach . ĠIt Ġcan Ġalso Ġbe Ġconsidered Ġas Ġa Ġspecific Ġinference Ġor Ġrelation Ġextract Ġtask , Ġsince Ġwe Ġextract Ġspecific Ġinformation Ġabout Ġan Ġentity Ġ( Med ication ĠName ), Ġhence Ġour Ġsecond Ġapproach Ġis Ġat Ġthe Ġintersection Ġof ĠQuestion ĠAn sw ering Ġ( Q A ) Ġand ĠInformation ĠExt raction Ġ( IE ) Ġdomains . ĠBoth Ġthe Ġapproaches Ġinvolve Ġusing Ġa Ġcontiguous Ġsegment Ġof Ġthe Ġtranscript Ġand Ġthe ĠMed ication ĠName Ġas Ġinput , Ġto Ġfind / in fer Ġthe Ġmedication 's ĠDos age Ġand ĠFrequency . ĠWhen Ġtesting Ġthe Ġapproaches Ġmim icking Ġreal - world Ġconditions , Ġwe Ġextract ĠMed ication ĠName Ġfrom Ġthe Ġtranscript Ġseparately Ġusing Ġont ology , Ġrefer Ġto ĠSEC REF 19 . </s> In Ġthe Ġfirst Ġapproach , Ġwe Ġframe Ġthe ĠMR Ġtask Ġas Ġa ĠQ A Ġtask Ġand Ġgenerate Ġquestions Ġusing Ġthe Ġtemplate : ĠâĢ ľ What Ġis Ġthe Ġ$ Ġ< $ dos age / frequency $ > $ Ġfor Ġ$ < $ Med ication ĠName $ > $ ". ĠHere , Ġwe Ġuse Ġan Ġabstract ive ĠQ A Ġmodel Ġbased Ġon Ġpointer - gener ator Ġnetworks ĠB IB REF 9 Ġaugmented Ġwith Ġco att ention Ġenc oder ĠB IB REF 10 Ġ( Q A - PG Net ). </s> In Ġthe Ġsecond Ġapproach , Ġwe Ġframe Ġthe Ġproblem Ġas Ġa Ġconditioned ĠIE Ġtask , Ġwhere Ġthe Ġinformation Ġextracted Ġdepends Ġon Ġan Ġentity Ġ( Med ication ĠName ). ĠHere , Ġwe Ġuse Ġa Ġmulti - dec oder Ġpointer - gener ator Ġnetwork Ġaugmented Ġwith Ġco att ention Ġenc oder Ġ( Multi - dec oder ĠQ A - PG Net ). ĠInstead Ġof Ġusing Ġtemplates Ġto Ġgenerate Ġquestions Ġand Ġusing Ġa Ġsingle Ġdec oder Ġto Ġextract Ġdifferent Ġtypes Ġof Ġinformation Ġas Ġin Ġthe ĠQ A Ġapproach Ġ( which Ġmight Ġlead Ġto Ġperformance Ġdegradation ), Ġhere Ġwe Ġconsider Ġseparate Ġdec od ers Ġfor Ġextracting Ġspecific Ġtypes Ġof Ġinformation Ġabout Ġan Ġentity Ġ$ E $ Ġ( Med ication ĠName ). </s> App roach Ġ: :: ĠPo inter - gener ator ĠNetwork Ġ( PG Net ) </s> The Ġnetwork Ġis Ġa Ġsequence - to - sequence Ġattention Ġmodel Ġthat Ġcan Ġboth Ġcopy Ġa Ġword Ġfrom Ġthe Ġinput Ġ$ I $ Ġcontaining Ġ$ P $ Ġword Ġtokens Ġor Ġgenerate Ġa Ġword Ġfrom Ġits Ġvocabulary Ġ$ voc ab $, Ġto Ġproduce Ġthe Ġoutput Ġsequence . </s> First , Ġthe Ġtokens Ġof Ġthe Ġ$ I $ Ġare Ġconverted Ġto Ġembed d ings Ġand Ġare Ġfed Ġone - by - one Ġto Ġthe Ġenc oder , Ġa Ġsingle Ġbi - L ST M Ġlayer , Ġwhich Ġenc odes Ġthe Ġtokens Ġin Ġ$ I $ Ġinto Ġa Ġsequence Ġof Ġhidden Ġstates Ġ- Ġ$ H = enc oder ( I ) $, Ġwhere Ġ$ ĠH =[ h _ 1 ... h _ P ] $. </s> For Ġeach Ġdec oder Ġtime Ġstep Ġ$ t $, Ġin Ġa Ġloop , Ġwe Ġcompute , Ġ1 ) Ġattention Ġ$ a _ t $ Ġ( using Ġthe Ġlast Ġdec oder Ġstate Ġ$ s _{ t - 1 } $ ), Ġover Ġthe Ġinput Ġtokens Ġ$ I $, Ġand Ġ2 ) Ġthe Ġdec oder Ġstate Ġ$ s _ t $ Ġusing Ġ$ a _ t $. ĠThen , Ġat Ġeach Ġtime Ġstep , Ġusing Ġboth Ġ$ a _ t $ Ġand Ġ$ s _ t $ Ġwe Ġcan Ġfind Ġthe Ġprobability Ġ$ P _ t ( w ) $, Ġof Ġproducing Ġa Ġword Ġ$ w $ Ġ( from Ġboth Ġ$ voc ab $ Ġand Ġ$ I $ ). ĠFor Ġconvenience , Ġwe Ġdenote Ġthe Ġattention Ġand Ġthe Ġdec oder Ġas Ġ$ dec oder _{ pg } ( H )= P ( w ) $, Ġwhere Ġ$ P ( w ) =[ P _ 1 ( w )... P _ T ( w )] $. ĠThe Ġoutput Ġcan Ġthen Ġbe Ġdec oded Ġfrom Ġ$ P ( w ) $, Ġwhich Ġis Ġdec oded Ġuntil Ġit Ġproduces Ġan Ġ` end Ġof Ġoutput Ġtoken ' Ġor Ġthe Ġnumber Ġof Ġsteps Ġreach Ġthe Ġmaximum Ġallowed Ġlimit . </s> App roach Ġ: :: ĠQ A ĠPG Net </s> We Ġfirst Ġencode Ġboth Ġthe Ġquestion Ġ- Ġ$ H _ Q Ġ= Ġenc oder ( Q ) $, Ġand Ġthe Ġinput Ġ- Ġ$ H _ I Ġ= Ġenc oder ( I ) $, Ġseparately Ġusing Ġenc od ers Ġ( with Ġshared Ġweights ). ĠThen , Ġto Ġcondition Ġ$ I $ Ġon Ġ$ Q $ Ġ( and Ġvice Ġversa ), Ġwe Ġuse Ġthe Ġco att ention Ġenc oder ĠB IB REF 10 Ġwhich Ġattends Ġto Ġboth Ġthe Ġ$ I $ Ġand Ġ$ Q $ Ġsimultaneously Ġto Ġgenerates Ġthe Ġco att ention Ġcontext Ġ- Ġ$ C _ D Ġ= Ġco att ( H _ I , ĠH _ Q ) $. ĠFinally , Ġusing Ġthe Ġpointer - gener ator Ġdec oder Ġwe Ġfind Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġoutput Ġsequence Ġ- Ġ$ P ( w ) Ġ= Ġdec oder _{ pg } ([ H _ I ; ĠC _ D ]) $, Ġwhich Ġis Ġthen Ġdec oded Ġto Ġgenerate Ġthe Ġanswer Ġsequence . </s> App roach Ġ: :: ĠMulti - dec oder Ġ( MD ) ĠQ A ĠPG Net </s> After Ġencoding Ġthe Ġinputs Ġinto Ġ$ H _ I $ Ġand Ġ$ H _ E $, Ġfor Ġextracting Ġ$ K $ Ġtypes Ġof Ġinformation Ġabout Ġan Ġentity Ġin Ġan ĠIE Ġfashion , Ġwe Ġuse Ġthe Ġfollowing Ġmulti - dec oder Ġ( MD ) Ġsetup : </s> Pred ictions Ġfor Ġeach Ġof Ġthe Ġ$ K $ Ġdec od ers Ġare Ġthen Ġdec oded Ġusing Ġ$ P ^ k ( w ) $. </s> All Ġthe Ġnetworks Ġdiscussed Ġabove Ġare Ġtrained Ġusing Ġa Ġnegative Ġlog - like lihood Ġloss Ġfor Ġthe Ġtarget Ġword Ġat Ġeach Ġtime Ġstep Ġand Ġsummed Ġover Ġall Ġthe Ġdec oder Ġtime Ġsteps . </s> Exper iments </s> We Ġinitialized ĠMR Ġextraction Ġmodels ' Ġvocabulary Ġfrom Ġthe Ġtraining Ġdataset Ġafter Ġremoving Ġwords Ġwith Ġa Ġfrequency Ġlower Ġthan Ġ30 Ġin Ġthe Ġdataset , Ġresulting Ġin Ġ4 56 Ġwords . ĠOur Ġvocabulary Ġis Ġsmall Ġbecause Ġof Ġthe Ġsize Ġof Ġthe Ġdataset , Ġhence Ġwe Ġrely Ġon Ġthe Ġmodel 's Ġability Ġto Ġcopy Ġwords Ġto Ġproduce Ġthe Ġoutput Ġeffectively . ĠIn Ġall Ġour Ġmodel Ġvariations , Ġthe Ġembed ding Ġand Ġthe Ġnetwork 's Ġhidden Ġdimension Ġare Ġset Ġto Ġbe Ġequal . ĠThe Ġnetworks Ġwere Ġtrained Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 00 15 , Ġdrop out Ġof Ġ0 . 5 Ġon Ġthe Ġembed ding Ġlayer , Ġnormal Ġgradient Ġclipping Ġset Ġat Ġ2 , Ġbatch Ġsize Ġof Ġ8 , Ġand Ġoptimized Ġwith ĠAd ag rad ĠB IB REF 15 Ġand Ġthe Ġtraining Ġwas Ġstopped Ġusing Ġthe Ġ$ 10 \ % $ Ġvalidation Ġdataset . </s> Exper iments Ġ: :: ĠData ĠProcessing </s> We Ġdid Ġthe Ġfollowing Ġbasic Ġpre processing Ġto Ġour Ġdata , Ġ1 ) Ġadded Ġ` none ' Ġto Ġthe Ġbeginning Ġof Ġthe Ġinput Ġutter ance , Ġso Ġthe Ġnetwork Ġcould Ġpoint Ġto Ġit Ġwhen Ġthere Ġwas Ġno Ġrelevant Ġinformation Ġin Ġthe Ġinput , Ġ2 ) Ġfiltered Ġoutl iers Ġwith Ġa Ġlarge Ġnumber Ġof Ġgrounded Ġtranscript Ġsentences Ġ($ > $ 150 Ġwords ), Ġand Ġ3 ) Ġconverted Ġall Ġtext Ġto Ġlower Ġcase . </s> To Ġimprove Ġperformance , Ġwe Ġ1 ) Ġstandardized Ġall Ġnumbers Ġ( both Ġdigits Ġand Ġwords ) Ġto Ġwords Ġconc aten ated Ġwith Ġa Ġhyp hen Ġ( e . g . Ġ110 Ġ- $ > $ Ġone - h undred - ten ), Ġin Ġboth Ġinput Ġand Ġoutput , Ġ2 ) Ġremoved Ġunits Ġfrom ĠDos age Ġas Ġsometimes Ġthe Ġunits Ġwere Ġnot Ġexplicitly Ġmentioned Ġin Ġthe Ġtranscript Ġsegment Ġbut Ġwere Ġwritten Ġby Ġthe Ġannot ators Ġusing Ġdomain Ġknowledge , Ġ3 ) Ġprep ended Ġall Ġmedication Ġmentions Ġwith Ġ` rx -' Ġtag , Ġas Ġthis Ġhelps Ġmodel 's Ġperformance Ġwhen Ġmultiple Ġmedications Ġare Ġdiscussed Ġin Ġa Ġsegment Ġ( in Ġboth Ġinput Ġand Ġoutput ), Ġand Ġ4 ) Ġwhen Ġa Ġtranscript Ġsegment Ġhas Ġmultiple Ġmedications Ġor Ġdos ages Ġbeing Ġdiscussed Ġwe Ġrandomly Ġshuffle Ġthem Ġ( in Ġboth Ġinput Ġand Ġoutput ) Ġand Ġcreate Ġa Ġnew Ġdata Ġpoint , Ġto Ġincreases Ġthe Ġnumber Ġof Ġtraining Ġdata Ġpoints . ĠRandom ly Ġshuff ling Ġthe Ġentities Ġincreases Ġthe Ġnumber Ġof Ġtraining ĠMR Ġtags Ġfrom Ġ8 , 654 Ġto Ġ11 , 5 21 . ĠBased Ġon Ġthe Ġdata Ġstatistics Ġafter Ġdata Ġprocessing , Ġwe Ġfixed Ġthe Ġmaximum Ġenc oder Ġsteps Ġto Ġ100 , Ġdosage Ġdec oder Ġsteps Ġto Ġ1 , Ġand Ġfrequency Ġdec oder Ġsteps Ġto Ġ3 Ġ( for Ġboth Ġthe ĠQ A Ġand ĠMulti - dec oder ĠQ A Ġmodels ). </s> Exper iments Ġ: :: ĠMet rics </s> For Ġthe ĠMR Ġextraction Ġtask , Ġwe Ġmeasure Ġthe ĠR OU GE - 1 Ġscores ĠB IB REF 14 Ġfor Ġboth Ġthe ĠDos age Ġand ĠFrequency Ġextraction Ġtasks . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġsince ĠDos age Ġis Ġa Ġsingle Ġword Ġtoken Ġ( after Ġprocessing ), Ġboth Ġthe Ġreference Ġand Ġhypothesis Ġare Ġa Ġsingle Ġtoken , Ġmaking Ġits ĠR OU GE - 1 ĠF 1 , ĠPrecision Ġand ĠRecall Ġscores Ġequal Ġand Ġbe Ġequal Ġto Ġpercentage Ġof Ġtimes Ġwe Ġfind Ġthe Ġcorrect Ġdosage Ġfor Ġthe Ġmedications . </s> In Ġour Ġannotations , ĠFrequency Ġhas Ġconflicting Ġtags Ġ( e . g . Ġ{ ` Once Ġa Ġday ', Ġ` tw ice Ġa Ġday ' } Ġand Ġ` daily '), Ġhence Ġmetrics Ġlike ĠEx act ĠMatch Ġwill Ġbe Ġerroneous . ĠTo Ġaddress Ġthis Ġissue , Ġwe Ġuse Ġthe ĠR OU GE Ġscores Ġto Ġcompare Ġdifferent Ġmodels Ġon Ġthe Ġ10 % Ġtest Ġdataset Ġand Ġwe Ġuse Ġqualitative Ġevaluation Ġto Ġmeasure Ġthe Ġtop - performing Ġmodels Ġon Ġ$\ math cal Ġ{ D } _{ test } $. </s> Exper iments Ġ: :: ĠModel Ġvariations </s> We Ġconsider ĠQ A ĠPG Net Ġand ĠMulti - dec oder ĠQ A ĠPG Net Ġwith Ġlookup Ġtable Ġembed ding Ġas Ġbaseline Ġmodels Ġand Ġimprove Ġon Ġthe Ġbas elines Ġwith Ġother Ġvariations Ġdescribed Ġbelow . </s> Apart Ġfrom Ġlearning - based Ġbas elines , Ġwe Ġalso Ġcreate Ġtwo Ġnaive Ġbas elines , Ġone Ġeach Ġfor Ġthe ĠDos age Ġand ĠFrequency Ġextraction Ġtasks . ĠFor ĠDos age Ġextraction , Ġthe Ġbaseline Ġwe Ġconsider Ġis Ġ` Ne arest ĠNumber ', Ġwhere Ġwe Ġtake Ġthe Ġnumber Ġnearest Ġto Ġthe ĠMed ication ĠName Ġas Ġthe Ġprediction , Ġand Ġ` none ' Ġif Ġno Ġnumber Ġis Ġmentioned Ġor Ġif Ġthe ĠMed ication ĠName Ġis Ġnot Ġdetected Ġin Ġthe Ġinput . ĠFor ĠFrequency Ġextraction , Ġthe Ġbaseline Ġwe Ġconsider Ġis Ġ` Random ĠTop - 3 ' Ġwhere Ġwe Ġpredict Ġa Ġrandom ĠFrequency Ġtag , Ġfrom Ġtop - 3 Ġmost Ġfrequent Ġones Ġfrom Ġour Ġdataset Ġ- Ġ{ ` none ', Ġ` daily ', Ġ` tw ice Ġa Ġday ' }. </s> Emb edd ing : ĠWe Ġdeveloped Ġdifferent Ġvariations Ġof Ġour Ġmodels Ġwith Ġa Ġsimple Ġlookup Ġtable Ġembed d ings Ġlearned Ġfrom Ġscratch Ġand Ġusing Ġhigh - performance Ġcontextual Ġembed d ings , Ġwhich Ġare ĠEL Mo ĠB IB REF 11 , ĠB ERT ĠB IB REF 16 Ġand ĠClinical BER T ĠB IB REF 13 Ġ( trained Ġand Ġprovided Ġby Ġthe Ġauthors ). ĠRefer Ġto ĠTable ĠT AB REF 5 Ġfor Ġthe Ġperformance Ġcomparisons . </s> We Ġderive Ġembed d ings Ġfrom ĠEL Mo Ġby Ġlearning Ġa Ġlinear Ġcombination Ġof Ġits Ġlast Ġthree Ġlayer 's Ġhidden Ġstates Ġ( task - specific Ġfine - tun ing ĠB IB REF 11 ). ĠSimilarly , Ġfor ĠB ERT - based Ġembed d ings , Ġwe Ġtake Ġa Ġlinear Ġcombination Ġof Ġthe Ġhidden Ġstates Ġfrom Ġits Ġlast Ġfour Ġlayers , Ġas Ġthis Ġcombination Ġperforms Ġbest Ġwithout Ġincreasing Ġthe Ġsize Ġof Ġthe Ġembed d ings ĠB IB REF 16 . ĠSince ĠB ERT Ġand ĠClinical BER T Ġuse Ġword - piece Ġvocabulary Ġand Ġcomp utes Ġsub - word Ġembed d ings , Ġwe Ġcompute Ġword - level Ġembed ding Ġby Ġaveraging Ġthe Ġcorresponding Ġsub - word Ġtokens . ĠEL Mo Ġand ĠB ERT Ġembed d ings Ġboth Ġhave Ġ1024 Ġdimensions , ĠClinical BER T Ġhave Ġ768 Ġas Ġit Ġis Ġbased Ġon ĠB ERT Ġbase Ġmodel , Ġand Ġthe Ġlookup Ġtable Ġhave Ġ128 ĠâĢĵ Ġhigher Ġdimension Ġmodels Ġleads Ġto Ġover fitting . </s> P ert aining ĠEnc oder : ĠWe Ġtrained Ġthe ĠPG Net Ġas Ġa Ġsummar ization Ġtask Ġusing Ġthe Ġclinical Ġsumm aries Ġand Ġused Ġthe Ġtrained Ġmodel Ġto Ġinitialize Ġthe Ġenc od ers Ġ( and Ġthe Ġembed d ings ) Ġof Ġthe Ġcorresponding ĠQ A Ġmodels . ĠWe Ġuse Ġa Ġvoc ab Ġsize Ġof Ġ40 73 Ġwords , Ġderived Ġfrom Ġthe Ġtraining Ġdataset Ġwith Ġa Ġfrequency Ġthreshold Ġof Ġ30 Ġfor Ġthe Ġtask . ĠWe Ġtrained Ġthe Ġmodels Ġusing ĠAd ag rad Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 015 , Ġnormal Ġgradient Ġclipping Ġset Ġat Ġ2 Ġand Ġtrained Ġfor Ġaround Ġ15 0000 Ġiterations Ġ( st opped Ġusing Ġvalidation Ġdataset ). ĠOn Ġthe Ġsummar ization Ġtask ĠPG Net Ġobtained ĠR OU GE - 1 ĠF 1 Ġscores Ġof Ġ41 . 42 Ġwith ĠEL Mo Ġand Ġ39 . 15 Ġwith ĠB ERT Ġembed d ings . ĠWe Ġcompare Ġthe Ġeffects Ġof Ġpret raining Ġthe Ġmodel Ġin ĠTable : ĠT AB REF 5 , Ġmodels Ġwith Ġ` pret rained Ġenc oder ' Ġhad Ġtheir Ġenc od ers Ġand Ġembed d ings Ġpret rained Ġwith Ġthe Ġsummar ization Ġtask . </s> Results Ġand ĠDiscussion Ġ: :: ĠDifference Ġin Ġnetworks Ġand Ġapproaches </s> Emb edd ings : ĠOn ĠDos age Ġextraction , Ġin Ġgeneral , ĠEL Mo Ġob tains Ġbetter Ġperformance Ġthan ĠB ERT , Ġrefer Ġto ĠTable ĠT AB REF 5 . ĠThis Ġcould Ġbe Ġbecause Ġwe Ġconc aten ated Ġthe Ġnumbers Ġwith Ġa Ġhyp hen , Ġand Ġas ĠEL Mo Ġuses Ġcharacter - level Ġtokens Ġit Ġcan Ġlearn Ġthe Ġtagging Ġbetter Ġthan ĠB ERT , Ġa Ġsimilar Ġobservation Ġis Ġalso Ġnoted Ġin ĠB IB REF 17 . ĠOn Ġthe Ġother Ġhand , Ġon ĠFrequency Ġextraction , Ġwithout Ġpret raining , ĠEL Mo 's Ġperformance Ġis Ġl agging Ġby Ġa Ġbig Ġmargin Ġof Ġ$\ sim Ġ$ 8 . 5 ĠR OU GE - 1 ĠF 1 Ġcompared Ġto ĠB ERT - based Ġembed d ings . </s> Although Ġin Ġcases Ġwithout Ġenc oder Ġpret raining , ĠClinical BER T Ġperformed Ġthe Ġbest Ġin Ġthe ĠFrequency Ġextraction Ġtask Ġ( by Ġa Ġsmall Ġmargin ), Ġin Ġgeneral , Ġit Ġdoes Ġnot Ġperform Ġas Ġwell Ġas ĠB ERT . ĠThis Ġcould Ġalso Ġbe Ġa Ġreflection Ġof Ġthe Ġfact Ġthat Ġthe Ġlanguage Ġand Ġstyle Ġof Ġwriting Ġused Ġin Ġclinical Ġnotes Ġis Ġvery Ġdifferent Ġfrom Ġthe Ġway Ġdoctors Ġcon verse Ġwith Ġpatients Ġand Ġthe Ġembed ding Ġdimension Ġdifference . ĠLook up Ġtable Ġembed ding Ġperformed Ġdec ently Ġin Ġthe Ġfrequency Ġextraction Ġtask Ġbut Ġl ags Ġbehind Ġin Ġthe ĠDos age Ġextraction Ġtask . </s> From Ġthe Ġmetrics Ġand Ġqualitative Ġinspection , Ġwe Ġfind Ġthat Ġthe ĠFrequency Ġextraction Ġis Ġan Ġeasier Ġtask Ġthan Ġthe ĠDos age Ġextraction . ĠThis Ġis Ġbecause , Ġin Ġthe Ġconversations , Ġfrequency Ġinformation Ġusually Ġoccurs Ġin Ġisolation Ġand Ġnear Ġthe Ġmedications , Ġbut Ġa Ġmedication 's Ġdosage Ġcan Ġoccur Ġ1 ) Ġnear Ġother Ġmedication 's Ġdos ages , Ġ2 ) Ġwith Ġprevious Ġdos ages Ġ( when Ġa Ġdosage Ġfor Ġa Ġmedication Ġis Ġrevised ), Ġand Ġ3 ) Ġafter Ġa Ġlarge Ġnumber Ġof Ġwords Ġfrom Ġthe Ġmedication . </s> Other ĠVari ations : ĠConsidering Ġvarious Ġmodels ' Ġperformance Ġ( without Ġpret raining ) Ġand Ġthe Ġresource Ġconstraint , Ġwe Ġchoose ĠEL Mo Ġand ĠB ERT Ġembed d ings Ġto Ġanalyze Ġthe Ġeffects Ġof Ġpret raining Ġthe Ġenc oder . ĠWhen Ġthe Ġnetwork 's Ġenc oder Ġ( and Ġembed ding ) Ġis Ġpret rained Ġwith Ġthe Ġsummar ization Ġtask , Ġwe Ġ1 ) Ġsee Ġa Ġsmall Ġdecrease Ġin Ġthe Ġaverage Ġnumber Ġof Ġiterations Ġrequired Ġfor Ġtraining , Ġ2 ) Ġimprovement Ġin Ġindividual Ġperformances Ġof Ġall Ġmodels Ġfor Ġboth Ġthe Ġsub - t asks , Ġand Ġ3 ) Ġget Ġbest Ġperformance Ġmetrics Ġacross Ġall Ġvariations , Ġrefer Ġto ĠTable ĠT AB REF 5 . ĠBoth Ġin Ġterms Ġof Ġperformance Ġand Ġthe Ġtraining Ġspeed , Ġthere Ġis Ġno Ġclear Ġwinner Ġbetween Ġshared Ġand Ġmulti - dec oder Ġapproaches . ĠMed ication Ġtagging Ġand Ġdata Ġaug mentation Ġincrease Ġthe Ġbest - performing Ġmodel 's ĠR OU GE - 1 ĠF 1 Ġscore Ġby Ġ$\ sim Ġ$ 1 . 5 Ġfor Ġthe ĠDos age Ġextraction Ġtask . </s> We Ġalso Ġmeasure Ġthe Ġperformance Ġof ĠMult it ask ĠQuestion ĠAn sw ering ĠNetwork Ġ( MQ AN ) ĠB IB REF 18 Ġa ĠQ A Ġmodel Ġtrained Ġby Ġthe Ġauthors Ġon Ġthe ĠDec athlon Ġmultit ask Ġchallenge . ĠSince ĠM Q AN Ġwas Ġnot Ġtrained Ġto Ġproduce Ġthe Ġoutput Ġsequence Ġin Ġour ĠMR Ġtags , Ġit Ġwould Ġnot Ġbe Ġfair Ġto Ġcompute ĠR OU GE Ġscores . ĠInstead , Ġwe Ġrandomly Ġsample Ġthe ĠM Q AN 's Ġpredictions Ġfrom Ġthe Ġ10 % Ġtest Ġdataset Ġand Ġqual itatively Ġevaluate Ġit . ĠFrom Ġthe Ġevaluations , Ġwe Ġfind Ġthat ĠM Q AN Ġcan Ġnot Ġdistinguish Ġbetween Ġfrequency Ġand Ġdosage , Ġand Ġmixed Ġthe Ġanswers . ĠM Q AN Ġcorrectly Ġpredicted Ġthe Ġdosage Ġfor Ġ29 . 73 % Ġand Ġfrequency Ġfor Ġ24 . 24 % Ġpercent Ġof Ġthe Ġmedications Ġcompared Ġto Ġ84 . 12 % Ġand Ġ76 . 34 % Ġfor Ġthe Ġenc oder Ġpret rained ĠB ERT ĠQ A ĠPG Net Ġmodel Ġtrained Ġon Ġour Ġdataset . ĠThis Ġcould Ġbe Ġbecause Ġof Ġthe Ġdifference Ġin Ġthe Ġtraining Ġdataset , Ġdomain Ġand Ġthe Ġtasks Ġin Ġthe ĠDec athlon Ġchallenge Ġcompared Ġto Ġours . </s> Almost Ġall Ġour Ġmodels Ġperform Ġbetter Ġthan Ġthe Ġnaive Ġbas elines Ġand Ġthe Ġones Ġusing Ġlookup Ġtable Ġembed d ings , Ġand Ġour Ġbest Ġperforming Ġmodels Ġoutper form Ġthem Ġsignificantly . ĠAmong Ġall Ġthe Ġvariations , Ġthe Ġbest Ġperforming Ġmodels Ġare ĠEL Mo Ġwith ĠMulti - dec oder Ġ( D os age Ġextraction ) Ġand ĠB ERT Ġwith Ġshared - dec oder ĠQ A ĠPG Net Ġarchitecture Ġwith Ġpret rained Ġenc oder Ġ( F requency Ġextraction ). ĠWe Ġchoose Ġthese Ġtwo Ġmodels Ġfor Ġour Ġsubsequent Ġanalysis . </s> Results Ġand ĠDiscussion Ġ: :: ĠBreak down Ġof ĠPerformance </s> We Ġcategor ize Ġthe Ġ10 % Ġtest Ġdataset Ġinto Ġdifferent Ġcategories Ġbased Ġon Ġthe Ġcomplexity Ġand Ġtype Ġof Ġthe Ġdata Ġand Ġanalyze Ġthe Ġbreakdown Ġof Ġthe Ġsystem 's Ġperformance Ġin ĠTable ĠT AB REF 11 . ĠWe Ġbreakdown Ġthe ĠFrequency Ġextraction Ġinto Ġtwo Ġcategories , Ġ1 ) ĠNone : Ġground Ġtruth ĠFrequency Ġtag Ġis Ġ` none ', Ġand Ġ2 ) ĠN N Ġ( Not ĠNone ): Ġground Ġtruth ĠFrequency Ġtag Ġis Ġnot Ġ` none '. ĠSimilarly , Ġthe ĠDos age Ġextraction Ġinto Ġ5 Ġcategories , Ġ1 ) ĠNone : Ġground Ġtruth Ġdosage Ġtag Ġis Ġ` none ', Ġ2 ) ĠMM Ġ( Multiple ĠMedicine ): Ġinput Ġsegment Ġhas Ġmore Ġthan Ġone ĠMed ication Ġmentioned , Ġ3 ) ĠMN Ġ( Multiple ĠNumbers ): Ġinput Ġsegment Ġhas Ġmore Ġthan Ġone Ġnumber Ġpresent , Ġand Ġ4 ) ĠN BM Ġ( Number Ġbetween Ġcorrect ĠDos age Ġand ĠMedicine ) Ġ: Ġbetween Ġthe ĠMed ication ĠName Ġand Ġthe Ġcorrect ĠDos age Ġin Ġthe Ġinput Ġsegment Ġthere Ġare Ġother Ġnumbers Ġpresent . ĠNote Ġthat Ġthe Ġcategories Ġof Ġthe ĠDos age Ġextraction Ġtask Ġare Ġnot Ġexhaustive , Ġand Ġone Ġtag Ġcan Ġbelong Ġto Ġmultiple Ġcategories . </s> From Ġthe Ġperformance Ġbreakdown Ġof ĠDos age Ġextraction Ġtask , Ġwe Ġsee Ġthat Ġ1 ) Ġthe Ġmodels Ġpredict Ġ` none ' Ġbetter Ġthan Ġother Ġcategories , Ġi . e ., Ġthe Ġmodels Ġare Ġcorrectly Ġable Ġto Ġidentify Ġwhen Ġa Ġmedication 's Ġdosage Ġis Ġabsent , Ġ2 ) Ġthere Ġis Ġperformance Ġdip Ġin Ġhard Ġcases Ġ( MM , ĠMN , Ġand ĠN BM ), Ġ3 ) Ġthe Ġmodels Ġare Ġable Ġto Ġfigure Ġout Ġthe Ġcorrect Ġdosage Ġ( dec ently ) Ġfor Ġa Ġmedication Ġeven Ġwhen Ġthere Ġare Ġmultiple Ġnumbers / dos age Ġpresent , Ġand Ġ4 ) Ġthe Ġmodel Ġstruggles Ġthe Ġmost Ġin Ġthe ĠN BM Ġcategory . ĠThe Ġmodels ' Ġlow Ġperformance Ġin ĠN BM Ġcould Ġbe Ġbecause Ġwe Ġhave Ġa Ġcomparatively Ġlower Ġnumber Ġof Ġexamples Ġto Ġtrain Ġin Ġthis Ġcategory . ĠThe ĠFrequency Ġextraction Ġtask Ġperforms Ġequally Ġwell Ġwhen Ġthe Ġtag Ġis Ġ` none ` Ġor Ġnot . ĠIn Ġmost Ġcategories , Ġwe Ġsee Ġan Ġincrease Ġin Ġperformance Ġwhen Ġusing Ġpret rained Ġenc od ers . </s> Results Ġand ĠDiscussion Ġ: :: ĠTraining ĠDat as et ĠSize </s> We Ġvary Ġthe Ġnumber Ġof ĠMR Ġtags Ġused Ġto Ġtrain Ġthe Ġmodel Ġand Ġanalyze Ġthe Ġmodel 's Ġperformance Ġwhen Ġtraining Ġthe Ġnetworks , Ġusing Ġpublicly Ġavailable Ġcontextual Ġembed d ings , Ġcompared Ġto Ġusing Ġpret rained Ġembed d ings Ġand Ġenc oder Ġ( pret rained Ġon Ġthe Ġsummar ization Ġtask ). ĠOut Ġof Ġthe Ġ5 , 016 Ġfiles Ġin Ġthe Ġ80 % Ġtrain Ġdataset Ġonly Ġ2 , 476 Ġhave Ġat le ast Ġone ĠMR Ġtag . ĠTherefore , Ġout Ġof Ġthe Ġ24 76 Ġfiles , Ġwe Ġrandomly Ġchoose Ġ100 , Ġ500 , Ġand Ġ1000 Ġfiles Ġand Ġtrained Ġthe Ġbest Ġperforming Ġmodel Ġvariations Ġto Ġobserve Ġthe Ġperformance Ġdifferences , Ġrefer Ġto ĠFigure ĠFIG REF 12 . ĠFor Ġall Ġthese Ġexperiments Ġwe Ġused Ġthe Ġsame Ġvocabulary Ġsize Ġ( 456 ), Ġthe Ġsame Ġhyper / training Ġparameters , Ġand Ġthe Ġsame Ġ10 % Ġtest Ġsplit Ġof Ġ6 27 Ġfiles . </s> As Ġexpected , Ġwe Ġsee Ġthat Ġthe Ġenc oder Ġpret rained Ġmodels Ġhave Ġhigher Ġperformance Ġon Ġall Ġthe Ġdifferent Ġtraining Ġdata Ġsizes , Ġi . e ., Ġthey Ġachieve Ġhigher Ġperformance Ġon Ġa Ġlower Ġnumber Ġof Ġdata Ġpoints , Ġrefer Ġto ĠFigure ĠFIG REF 12 . ĠThe Ġdifference , Ġas Ġexpected , Ġshr inks Ġas Ġthe Ġtraining Ġdata Ġsize Ġincreases . </s> Results Ġand ĠDiscussion Ġ: :: ĠEval uating Ġon ĠAS R Ġtranscripts </s> To Ġtest Ġthe Ġperformance Ġof Ġour Ġmodels Ġon Ġreal - world Ġconditions , Ġwe Ġuse Ġcommercially Ġavailable ĠAS R Ġservices Ġ( Google Ġand ĠIBM ) Ġto Ġtransc ribe Ġthe Ġ$\ math cal Ġ{ D } _{ test } $ Ġfiles Ġand Ġmeasure Ġthe Ġperformance Ġof Ġour Ġmodels Ġwithout Ġassuming Ġany Ġannotations Ġ( except Ġwhen Ġcalculating Ġthe Ġmetrics ). ĠIt Ġshould Ġbe Ġnoted Ġthat Ġthis Ġis Ġnot Ġthe Ġcase Ġin Ġour Ġprevious Ġevaluations Ġusing Ġ` 10 % Ġtest ' Ġdataset Ġwhere Ġwe Ġuse Ġthe Ġsegment ation Ġinformation . ĠFor Ġground Ġtruth Ġannotations Ġon ĠAS R Ġtranscripts , Ġwe Ġaligned Ġthe ĠMR Ġtags Ġfrom Ġhuman Ġwritten Ġtranscripts Ġto Ġthe ĠAS R Ġtranscript Ġusing Ġtheir Ġgrounded Ġtiming Ġinformation . ĠAdditionally , Ġsince ĠAS R Ġis Ġprone Ġto Ġerrors , Ġduring Ġthe Ġalignment , Ġif Ġa Ġmedication Ġfrom Ġan ĠMR Ġtag Ġis Ġnot Ġrecognized Ġcorrectly Ġin Ġthe ĠAS R Ġtranscript , Ġwe Ġremove Ġthe Ġcorresponding ĠMR Ġtag . </s> In Ġour Ġevaluations , Ġwe Ġuse ĠGoogle ĠCloud ĠSpeech - to - Text Ġ( G - ST T ) Ġand ĠIBM ĠWatson ĠSpeech Ġto ĠText Ġ( IB M - ST T ) Ġas Ġthese Ġwere Ġamong Ġthe Ġtop - performing ĠAS R ĠAPIs Ġon Ġmedical Ġspeech ĠB IB REF 19 Ġand Ġwere Ġreadily Ġavailable Ġto Ġus . ĠWe Ġused ĠG - ST T , Ġwith Ġthe Ġ` video Ġmodel ' Ġwith Ġpunct uation Ġsettings . ĠUnlike Ġour Ġhuman Ġwritten Ġtranscripts , Ġthe Ġtranscript Ġprovided Ġby ĠG - ST T Ġis Ġnot Ġver bat im Ġand Ġdoes Ġnot Ġhave Ġdis flu encies . ĠIBM - ST T , Ġon Ġthe Ġother Ġhand , Ġdoes Ġnot Ġgive Ġpunct uation Ġso Ġwe Ġused Ġthe Ġspeaker Ġchanges Ġto Ġadd Ġend - of - sent ence Ġpunct uation . </s> In Ġour Ġ$\ math cal Ġ{ D } _{ test } $ Ġdataset , Ġon Ġinitial Ġstudy Ġwe Ġsee Ġa ĠWord ĠError ĠRate Ġof Ġ$\ sim Ġ$ 50 % Ġfor Ġthe ĠAS R ĠAPIs Ġand Ġthis Ġnumber Ġis Ġnot Ġaccurate Ġbecause , Ġ1 ) Ġof Ġthe Ġde - ident ification , Ġ2 ) Ġdis flu encies Ġ( ver bat im ) Ġdifference Ġbetween Ġthe Ġhuman Ġwritten Ġand ĠAS R Ġtranscript , Ġand Ġ3 ) Ġminor Ġalignment Ġdifferences Ġbetween Ġthe Ġaudio Ġand Ġthe Ġground Ġtruth Ġtranscript . </s> During Ġthis Ġevaluation , Ġwe Ġfollowed Ġthe Ġsame Ġpre processing Ġmethods Ġwe Ġused Ġduring Ġtraining . ĠThen , Ġwe Ġauto Ġsegment Ġthe Ġtranscript Ġinto Ġsmall Ġcontiguous Ġsegments Ġsimilar Ġto Ġthe Ġgrounded Ġsentences Ġin Ġthe Ġannotations Ġfor Ġtags Ġextraction . ĠTo Ġsegment Ġthe Ġtranscript , Ġwe Ġfollow Ġa Ġsimple Ġprocedure . ĠFirst , Ġwe Ġdetected Ġall Ġthe Ġmedications Ġin Ġa Ġtranscript Ġusing ĠRx Norm ĠB IB REF 20 Ġvia Ġstring Ġmatching . ĠFor Ġall Ġthe Ġdetected Ġmedications , Ġwe Ġselected Ġ$ 2 Ġ\ le Ġx Ġ\ le Ġ5 $ Ġnearby Ġsentences Ġas Ġthe Ġinput Ġto Ġour Ġmodel . ĠWe Ġincreased Ġ$ x $ Ġiter atively Ġuntil Ġwe Ġencountered Ġa Ġquantity Ġentity ĠâĢĵ Ġdetected Ġusing Ġspa Cy 's Ġentity Ġrecogn izer , Ġand Ġwe Ġset Ġ$ x $ Ġas Ġ2 Ġif Ġwe Ġdid Ġnot Ġdetect Ġany Ġentities Ġin Ġthe Ġrange . </s> We Ġshow Ġthe Ġmodel 's Ġperformance Ġon ĠAS R Ġtranscripts Ġand Ġhuman Ġwritten Ġtranscripts Ġwith Ġautomatic Ġsegment ation , Ġand Ġhuman Ġwritten Ġtranscripts Ġwith Ġhuman Ġ( defined ) Ġsegment ation , Ġin ĠTable ĠT AB REF 18 . ĠThe Ġnumber Ġof Ġrecognized Ġmedications Ġin ĠIBM - ST T Ġis Ġonly Ġ95 Ġcompared Ġto Ġ7 25 Ġ( human Ġwritten ), Ġwe Ġmainly Ġconsider Ġthe Ġmodels ' Ġperformance Ġon ĠG - ST T 's Ġtranscripts Ġ( 343 ). </s> On Ġthe ĠMed ications Ġthat Ġwere Ġrecognized Ġcorrectly , Ġthe Ġmodels Ġcan Ġperform Ġdec ently Ġon ĠAS R Ġtranscripts Ġin Ġcomparison Ġto Ġhuman Ġtranscripts Ġ( within Ġ5 Ġpoints ĠR OU GE - 1 ĠF 1 Ġfor Ġboth Ġtasks , Ġrefer Ġto ĠTable ĠT AB REF 18 ). ĠThis Ġshows Ġthat Ġthe Ġmodels Ġare Ġrobust Ġto ĠAS R Ġvariations Ġdiscussed Ġabove . ĠThe Ġlower Ġperformance Ġcompared Ġto Ġhuman Ġtranscripts Ġis Ġmainly Ġdue Ġto Ġincorrect Ġrecognition Ġof ĠDos age Ġand Ġother Ġmedications Ġin Ġthe Ġsame Ġsegments Ġ( changing Ġthe Ġmeaning Ġof Ġthe Ġtext ). ĠBy Ġcomparing Ġthe Ġperformance Ġof Ġthe Ġmodel Ġon Ġthe Ġhuman Ġwritten Ġtranscripts Ġwith Ġhuman Ġ( defined ) Ġsegment ation Ġand Ġthe Ġsame Ġwith Ġauto Ġsegment ation , Ġwe Ġsee Ġa Ġ10 Ġpoint Ġdrop Ġin ĠDos age Ġand Ġ6 Ġpoint Ġdrop Ġin ĠFrequency Ġextraction Ġtasks . ĠThis Ġpoints Ġout Ġthe Ġneed Ġfor Ġmore Ġsophisticated Ġsegment ation Ġalgorithms . </s> With ĠG - ST T , Ġour Ġbest Ġmodel Ġobtained ĠR OU GE - 1 ĠF 1 Ġof Ġ71 . 75 Ġ( which Ġequals Ġto Ġpercentage Ġof Ġtimes Ġdosage Ġis Ġcorrect Ġin Ġthis Ġcase ) Ġfor ĠDos age Ġextraction Ġand Ġ40 . 13 Ġfor ĠFrequency Ġextraction Ġtasks . ĠTo Ġmeasure Ġthe Ġpercentage Ġof Ġtimes Ġthe Ġcorrect Ġfrequency Ġwas Ġextracted Ġby Ġthe Ġmodel , Ġwe Ġqual itatively Ġcompared Ġthe Ġextracted Ġand Ġpredicted Ġfrequency . ĠWe Ġfind Ġthat Ġfor Ġ73 . 58 % Ġof Ġthe Ġmedications Ġthe Ġmodel Ġcan Ġfind Ġthe Ġcorrect Ġfrequency Ġfrom Ġthe Ġtranscripts . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe ĠMed ication ĠReg imen Ġ( MR ) Ġextraction Ġtask Ġof Ġextracting Ġdosage Ġand Ġfrequency Ġfor Ġthe Ġmedications Ġmentioned Ġin Ġa Ġdoctor - patient Ġconversation Ġtranscript . ĠWe Ġexplore Ġdifferent Ġvariations Ġof Ġabstract ive ĠQ A Ġmodels Ġand Ġnew Ġarchitecture Ġin Ġthe Ġintersection Ġof ĠQ A Ġand ĠIE Ġframeworks Ġand Ġprovide Ġa Ġcomparative Ġperformance Ġanalysis Ġof Ġthe Ġmethods Ġalong Ġwith Ġother Ġtechniques Ġlike Ġpret raining Ġto Ġimprove Ġthe Ġoverall Ġperformance . ĠFinally , Ġwe Ġdemonstrate Ġthe Ġperformance Ġof Ġour Ġbest - performing Ġmodels Ġby Ġautomatically Ġextracting ĠMR Ġtags Ġfrom Ġspontaneous Ġdoctor - patient Ġconversations Ġ( using Ġcommercially Ġavailable ĠAS R ). ĠOur Ġbest Ġmodel Ġcan Ġcorrectly Ġextract Ġthe Ġdosage Ġfor Ġ71 . 75 % Ġ( interpret ation Ġof ĠR OU GE - 1 Ġscore ) Ġand Ġfrequency Ġfor Ġ73 . 58 % Ġ( on Ġqualitative Ġevaluation ) Ġof Ġthe Ġmedications Ġdiscussed Ġin Ġthe Ġtranscripts Ġgenerated Ġusing ĠGoogle ĠSpeech - To - Text . ĠIn Ġsummary , Ġwe Ġdemonstrate Ġthat Ġresearch Ġon ĠN LP Ġcan Ġbe Ġtranslated Ġinto Ġreal - world Ġclinical Ġsettings Ġto Ġrealize Ġits Ġbenefits Ġfor Ġboth Ġdoctors Ġand Ġpatients . </s> Using ĠAS R Ġtranscripts Ġin Ġour Ġtraining Ġprocess Ġto Ġimprove Ġour Ġperformance Ġon Ġboth Ġthe Ġtasks Ġand Ġextending Ġthe Ġmedication Ġregimen Ġextraction Ġnetwork Ġto Ġextract Ġother Ġimportant Ġmedical Ġinformation Ġcan Ġbe Ġinteresting Ġlines Ġof Ġfuture Ġwork . </s> Acknowled gements </s> We Ġthank : ĠUniversity Ġof ĠPittsburgh ĠMedical ĠCenter Ġ( U PM C ), Ġand ĠA bridge ĠAI ĠInc . Ġfor Ġproviding Ġaccess Ġto Ġthe Ġde - identified Ġdata Ġcorpus ; ĠDr . ĠShiv dev ĠRao , Ġa Ġfaculty Ġmember Ġand Ġpracticing Ġcard iologist Ġin ĠU PM C 's ĠHeart Ġand ĠV ascular ĠInstitute Ġand ĠProf . ĠFlor ian ĠMet ze , ĠAssociate ĠResearch ĠProfessor , ĠCarnegie ĠMellon ĠUniversity Ġfor Ġhelpful Ġdiscussions ; ĠBen ĠSch loss , ĠSteven ĠColeman , Ġand ĠDeborah ĠOs ak ue Ġfor Ġdata Ġbusiness Ġdevelopment Ġand Ġannotation Ġmanagement .
Introduction To appear in Proceedings of International Workshop on Health Intelligence (W3PHIAI) of the 34th AAAI Conference on Artificial Intelligence, 2020. Physician burnout is a growing concern, estimated to be experienced by at least 35% of physicians in the developing world and 50% in the United States BIBREF0. BIBREF1 found that for every hour physicians provide direct clinical facetime to patients, nearly two additional hours are spent on EHR (Electronic Health Records) and administrative or desk work. As per the study conducted by Massachusetts General Physicians Organization (MPGO) BIBREF2 and as reported by BIBREF3, the average time spent on administrative tasks increased from 23.7% in 2014 to 27.9% in 2017. Both the surveys found that time spent on administrative tasks was positively associated with higher likelihood of burnout. Top reasons under administrative burden include working on the ambulatory EHR, handling medication reconciliation (sometimes done by aids), medication renewals, and medical billing and coding. The majority of these reasons revolve around documentation of information exchanged between doctors and patients during the clinical encounters. Automatically extracting such clinical information BIBREF4, BIBREF5 can not only help alleviate the documentation burden on the physician, but also allow them to dedicate more time directly with patients. Among all the clinical information extraction tasks, Medication Regimen (Medication, dosage, and frequency) extraction is particularly interesting due to its ability to help doctors with medication orders cum renewals, medication reconciliation, potentially verifying the reconciliations for errors, and, other medication-centered EHR documentation tasks. In addition, the same information when provided to patients can help them with better recall of doctor instructions which might aid in compliance with the care plan. This is particularly important given that patients forget or wrongly recollect 40-80% BIBREF6 of what is discussed in the clinic, and accessing EHR data has its own challenges. Spontaneous clinical conversations happening between a doctor and a patient, have several distinguishing characteristics from a normal monologue or prepared speech: it involves multiple speakers with overlapping dialogues, covers a variety of speech patterns, and the vocabulary can range from colloquial to complex domain-specific language. With recent advancements in Conversational Speech Recognition BIBREF7 rendering the systems less prone to errors, the subsequent challenge of understanding and extracting relevant information from the conversations is receiving increasing research focus BIBREF4, BIBREF8. In this paper, we focus on local information extraction in transcribed clinical conversations. Specifically, we extract dosage (e.g. 5mg) and frequency (e.g. once a day) for the medications (e.g. aspirin) from these transcripts, collectively referred to as Medication Regimen (MR) extraction. The information extraction is local as we extract the information from a segment of the transcript and not the entire transcript since doing the latter is difficult owing to the long meandering nature of the conversations often with multiple medication regimens and care plans being discussed. The challenges associated with the Medication Regimen (MR) extraction task include understanding the spontaneous dialog with clinical vocabulary and understanding the relationship between different entities as the discussion can contain multiple medications and dosages (e.g. doctor revising a dosage or reviewing all the current medications). We frame this problem as a Question Answering (QA) task by generating questions using templates. We base the QA model on pointer-generator networks BIBREF9 augmented with Co-Attentions BIBREF10. In addition, we develop models combining QA and Information Extraction frameworks using multi-decoder (one each for dosage and frequency) architecture. Lack of availability of a large volume of data is a typical challenge in healthcare. A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns). Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise. To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task. We further investigate the effects of training data size on our models. On the MR extraction task, ELMo with encoder multi-decoder architecture and BERT with encoder-decoder with encoders pretrained on the summarization task perform the best. The best-performing models improve our baseline's dosage and frequency extractions ROUGE-1 F1 scores from 54.28 and 37.13 to 89.57 and 45.94, respectively. Using our models, we present the first fully automated system to extract MR tags from spontaneous doctor-patient conversations. We evaluate the system (using our best performing models) on the transcripts generated from Automatic Speech Recognition (ASR) APIs offered by Google and IBM. In Google ASR's transcripts, our best model obtained ROUGE-1 F1 of 71.75 for Dosage extraction (which in this specific case equals to the percentage of times dosage is correct, refer Metrics Section for more details) and 40.13 for Frequency extraction tasks. On qualitative evaluation, we find that for 73.58% of the medications the model can find the correct frequency. These results demonstrate that the research on NLP can be used effectively in a real clinical setting to benefit both doctors and patients Data Our dataset consists of a total of 6,693 real doctor-patient conversations recorded in a clinical setting using distant microphones of varying quality. The recordings have an average duration of 9min 28s and have a verbatim transcript of 1,500 words on average (written by the experts). Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively. The sentences in the transcript are grounded to the audio with the timestamps of its first and last word. The transcript of the conversations are annotated with summaries and Medication Regimen tags (MR tags), both grounded using the timestamps of the sentences from the transcript deemed relevant by the expert annotators, refer to Table TABREF1. The transcript for a typical conversation can be quite long, and not easy for many of the high performing deep learning models to act on. Moreover, the medical information about a concept/condition/entity can change during the conversation after a significant time gap. For example, dosage of a medication can be different when discussing current medication the patient is on vs when they are prescribed a different dosage. Hence, we have annotations, that are grounded to a short segment of the transcript. The summaries (#words - $\mu = 9.7; \sigma = 10.1$) are medically relevant and local. The MR tags are also local and are of the form {Medication Name, Dosage, Frequency}. If dosage ($\mu = 2.0; \sigma = 0$) or frequency ($\mu = 2.1; \sigma = 1.07$) information for a medication is not present in a grounded sentence, the corresponding field in the MR tag will be marked as `none'. In the MR tags, Medication Name and Dosage (usually a quantity followed by its units) can be relatively easily extracted from the transcript except for the units of the dosage which is sometimes inferred. In contrast, due to high degree of linguistic variation with which Frequency is often expressed, extracting Frequency requires an additional inference step. For example, `take one in the morning and at noon' from the transcript is tagged as `twice a day' in the frequency tag, likewise `take it before sleeping' is tagged as `at night time'. Out of overall 6,693 files, we set aside a random sample of 423 files (denoted as $\mathcal {D}_{test}$) for final evaluation. The remaining 6,270 files are used for training with 80% train (5016), 10% validation (627), and 10% test (627) split. Overall, the 6,270 files contains 156,186 summaries and 32,000 MR tags out of which 8,654 MR tags contain values for at least one of the Dosage or Frequency, which we used for training to avoid overfitting (the remaining MR tags have both Dosage and Frequency as `none'). Note that we have two test datasets: `10% test' - used to evaluate all the models, and $\mathcal {D}_{test}$ - used to measure the performance of best performing models on ASR transcripts. Approach We frame the Medication Regimen extraction problem as a Question Answering (QA) task, which forms the basis for our first approach. It can also be considered as a specific inference or relation extract task, since we extract specific information about an entity (Medication Name), hence our second approach is at the intersection of Question Answering (QA) and Information Extraction (IE) domains. Both the approaches involve using a contiguous segment of the transcript and the Medication Name as input, to find/infer the medication's Dosage and Frequency. When testing the approaches mimicking real-world conditions, we extract Medication Name from the transcript separately using ontology, refer to SECREF19. In the first approach, we frame the MR task as a QA task and generate questions using the template: “What is the $ <$dosage/frequency$>$ for $<$Medication Name$>$". Here, we use an abstractive QA model based on pointer-generator networks BIBREF9 augmented with coattention encoder BIBREF10 (QA-PGNet). In the second approach, we frame the problem as a conditioned IE task, where the information extracted depends on an entity (Medication Name). Here, we use a multi-decoder pointer-generator network augmented with coattention encoder (Multi-decoder QA-PGNet). Instead of using templates to generate questions and using a single decoder to extract different types of information as in the QA approach (which might lead to performance degradation), here we consider separate decoders for extracting specific types of information about an entity $E$ (Medication Name). Approach ::: Pointer-generator Network (PGNet) The network is a sequence-to-sequence attention model that can both copy a word from the input $I$ containing $P$ word tokens or generate a word from its vocabulary $vocab$, to produce the output sequence. First, the tokens of the $I$ are converted to embeddings and are fed one-by-one to the encoder, a single bi-LSTM layer, which encodes the tokens in $I$ into a sequence of hidden states - $H=encoder(I)$, where $ H=[h_1...h_P]$. For each decoder time step $t$, in a loop, we compute, 1) attention $a_t$ (using the last decoder state $s_{t-1}$), over the input tokens $I$, and 2) the decoder state $s_t$ using $a_t$. Then, at each time step, using both $a_t$ and $s_t$ we can find the probability $P_t(w)$, of producing a word $w$ (from both $vocab$ and $I$). For convenience, we denote the attention and the decoder as $decoder_{pg}(H)=P(w)$, where $P(w)=[P_1(w)...P_T(w)]$. The output can then be decoded from $P(w)$, which is decoded until it produces an `end of output token' or the number of steps reach the maximum allowed limit. Approach ::: QA PGNet We first encode both the question - $H_Q = encoder(Q)$, and the input - $H_I = encoder(I)$, separately using encoders (with shared weights). Then, to condition $I$ on $Q$ (and vice versa), we use the coattention encoder BIBREF10 which attends to both the $I$ and $Q$ simultaneously to generates the coattention context - $C_D = coatt(H_I, H_Q)$. Finally, using the pointer-generator decoder we find the probability distribution of the output sequence - $P(w) = decoder_{pg}([H_I; C_D])$, which is then decoded to generate the answer sequence. Approach ::: Multi-decoder (MD) QA PGNet After encoding the inputs into $H_I$ and $H_E$, for extracting $K$ types of information about an entity in an IE fashion, we use the following multi-decoder (MD) setup: Predictions for each of the $K$ decoders are then decoded using $P^k(w)$. All the networks discussed above are trained using a negative log-likelihood loss for the target word at each time step and summed over all the decoder time steps. Experiments We initialized MR extraction models' vocabulary from the training dataset after removing words with a frequency lower than 30 in the dataset, resulting in 456 words. Our vocabulary is small because of the size of the dataset, hence we rely on the model's ability to copy words to produce the output effectively. In all our model variations, the embedding and the network's hidden dimension are set to be equal. The networks were trained with a learning rate of 0.0015, dropout of 0.5 on the embedding layer, normal gradient clipping set at 2, batch size of 8, and optimized with Adagrad BIBREF15 and the training was stopped using the $10\%$ validation dataset. Experiments ::: Data Processing We did the following basic preprocessing to our data, 1) added `none' to the beginning of the input utterance, so the network could point to it when there was no relevant information in the input, 2) filtered outliers with a large number of grounded transcript sentences ($>$150 words), and 3) converted all text to lower case. To improve performance, we 1) standardized all numbers (both digits and words) to words concatenated with a hyphen (e.g. 110 -$>$ one-hundred-ten), in both input and output, 2) removed units from Dosage as sometimes the units were not explicitly mentioned in the transcript segment but were written by the annotators using domain knowledge, 3) prepended all medication mentions with `rx-' tag, as this helps model's performance when multiple medications are discussed in a segment (in both input and output), and 4) when a transcript segment has multiple medications or dosages being discussed we randomly shuffle them (in both input and output) and create a new data point, to increases the number of training data points. Randomly shuffling the entities increases the number of training MR tags from 8,654 to 11,521. Based on the data statistics after data processing, we fixed the maximum encoder steps to 100, dosage decoder steps to 1, and frequency decoder steps to 3 (for both the QA and Multi-decoder QA models). Experiments ::: Metrics For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks. It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications. In our annotations, Frequency has conflicting tags (e.g. {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous. To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$. Experiments ::: Model variations We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below. Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks. For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input. For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'}. Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors). Refer to Table TABREF5 for the performance comparisons. We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11). Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16. Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens. ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting. Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models. We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task. We trained the models using Adagrad optimizer with a learning rate of 0.015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset). On the summarization task PGNet obtained ROUGE-1 F1 scores of 41.42 with ELMo and 39.15 with BERT embeddings. We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task. Results and Discussion ::: Difference in networks and approaches Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5. This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17. On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8.5 ROUGE-1 F1 compared to BERT-based embeddings. Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT. This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference. Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task. From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction. This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication. Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder. When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5. Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches. Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1.5 for the Dosage extraction task. We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge. Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores. Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it. From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers. MQAN correctly predicted the dosage for 29.73% and frequency for 24.24% percent of the medications compared to 84.12% and 76.34% for the encoder pretrained BERT QA PGNet model trained on our dataset. This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours. Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly. Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction). We choose these two models for our subsequent analysis. Results and Discussion ::: Breakdown of Performance We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11. We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'. Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present. Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories. From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i.e., the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category. The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category. The Frequency extraction task performs equally well when the tag is `none` or not. In most categories, we see an increase in performance when using pretrained encoders. Results and Discussion ::: Training Dataset Size We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task). Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag. Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12. For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files. As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i.e., they achieve higher performance on a lower number of data points, refer to Figure FIGREF12. The difference, as expected, shrinks as the training data size increases. Results and Discussion ::: Evaluating on ASR transcripts To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics). It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information. For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information. Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag. In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us. We used G-STT, with the `video model' with punctuation settings. Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies. IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation. In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript. During this evaluation, we followed the same preprocessing methods we used during training. Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction. To segment the transcript, we follow a simple procedure. First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching. For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model. We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range. We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18. The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343). On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18). This shows that the models are robust to ASR variations discussed above. The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text). By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks. This points out the need for more sophisticated segmentation algorithms. With G-STT, our best model obtained ROUGE-1 F1 of 71.75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40.13 for Frequency extraction tasks. To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency. We find that for 73.58% of the medications the model can find the correct frequency from the transcripts. Conclusion In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript. We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance. Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR). Our best model can correctly extract the dosage for 71.75% (interpretation of ROUGE-1 score) and frequency for 73.58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text. In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients. Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work. Acknowledgements We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc. for providing access to the de-identified data corpus; Dr. Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof. Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management.
What embeddings are used?
f161e6d5aecf8fae3a26374dcb3e4e1b40530c95
What Ġembed d ings Ġare Ġused ?
Ġsimple Ġlookup Ġtable Ġembed d ings Ġlearned Ġfrom Ġscratch , Ġusing Ġhigh - performance Ġcontextual Ġembed d ings , Ġwhich Ġare ĠEL Mo ĠB IB REF 11 , ĠB ERT ĠB IB REF 16 Ġand ĠClinical BER T ĠB IB REF 13
<s> What Ġembed d ings Ġare Ġused ? </s> Introduction </s> To Ġappear Ġin ĠProceedings Ġof ĠInternational ĠWorkshop Ġon ĠHealth ĠIntelligence Ġ( W 3 PH IA I ) Ġof Ġthe Ġ34 th ĠAAA I ĠConference Ġon ĠArtificial ĠIntelligence , Ġ2020 . </s> Phys ician Ġburn out Ġis Ġa Ġgrowing Ġconcern , Ġestimated Ġto Ġbe Ġexperienced Ġby Ġat Ġleast Ġ35 % Ġof Ġphysicians Ġin Ġthe Ġdeveloping Ġworld Ġand Ġ50 % Ġin Ġthe ĠUnited ĠStates ĠB IB REF 0 . ĠB IB REF 1 Ġfound Ġthat Ġfor Ġevery Ġhour Ġphysicians Ġprovide Ġdirect Ġclinical Ġfac etime Ġto Ġpatients , Ġnearly Ġtwo Ġadditional Ġhours Ġare Ġspent Ġon ĠE HR Ġ( Elect ronic ĠHealth ĠRecords ) Ġand Ġadministrative Ġor Ġdesk Ġwork . ĠAs Ġper Ġthe Ġstudy Ġconducted Ġby ĠMassachusetts ĠGeneral ĠPhysicians ĠOrganization Ġ( M PG O ) ĠB IB REF 2 Ġand Ġas Ġreported Ġby ĠB IB REF 3 , Ġthe Ġaverage Ġtime Ġspent Ġon Ġadministrative Ġtasks Ġincreased Ġfrom Ġ23 . 7 % Ġin Ġ2014 Ġto Ġ27 . 9 % Ġin Ġ2017 . ĠBoth Ġthe Ġsurveys Ġfound Ġthat Ġtime Ġspent Ġon Ġadministrative Ġtasks Ġwas Ġpositively Ġassociated Ġwith Ġhigher Ġlikelihood Ġof Ġburn out . ĠTop Ġreasons Ġunder Ġadministrative Ġburden Ġinclude Ġworking Ġon Ġthe Ġambul atory ĠE HR , Ġhandling Ġmedication Ġreconciliation Ġ( sometimes Ġdone Ġby Ġaids ), Ġmedication Ġrenew als , Ġand Ġmedical Ġbilling Ġand Ġcoding . ĠThe Ġmajority Ġof Ġthese Ġreasons Ġrev olve Ġaround Ġdocumentation Ġof Ġinformation Ġexchanged Ġbetween Ġdoctors Ġand Ġpatients Ġduring Ġthe Ġclinical Ġencounters . ĠAutom atically Ġextracting Ġsuch Ġclinical Ġinformation ĠB IB REF 4 , ĠB IB REF 5 Ġcan Ġnot Ġonly Ġhelp Ġalleviate Ġthe Ġdocumentation Ġburden Ġon Ġthe Ġphysician , Ġbut Ġalso Ġallow Ġthem Ġto Ġdedicate Ġmore Ġtime Ġdirectly Ġwith Ġpatients . </s> Among Ġall Ġthe Ġclinical Ġinformation Ġextraction Ġtasks , ĠMed ication ĠReg imen Ġ( Med ication , Ġdosage , Ġand Ġfrequency ) Ġextraction Ġis Ġparticularly Ġinteresting Ġdue Ġto Ġits Ġability Ġto Ġhelp Ġdoctors Ġwith Ġmedication Ġorders Ġcum Ġrenew als , Ġmedication Ġreconciliation , Ġpotentially Ġverifying Ġthe Ġreconc ili ations Ġfor Ġerrors , Ġand , Ġother Ġmedication - centered ĠE HR Ġdocumentation Ġtasks . ĠIn Ġaddition , Ġthe Ġsame Ġinformation Ġwhen Ġprovided Ġto Ġpatients Ġcan Ġhelp Ġthem Ġwith Ġbetter Ġrecall Ġof Ġdoctor Ġinstructions Ġwhich Ġmight Ġaid Ġin Ġcompliance Ġwith Ġthe Ġcare Ġplan . ĠThis Ġis Ġparticularly Ġimportant Ġgiven Ġthat Ġpatients Ġforget Ġor Ġwrongly Ġrecol lect Ġ40 - 80 % ĠB IB REF 6 Ġof Ġwhat Ġis Ġdiscussed Ġin Ġthe Ġclinic , Ġand Ġaccessing ĠE HR Ġdata Ġhas Ġits Ġown Ġchallenges . </s> Sp ont aneous Ġclinical Ġconversations Ġhappening Ġbetween Ġa Ġdoctor Ġand Ġa Ġpatient , Ġhave Ġseveral Ġdistinguishing Ġcharacteristics Ġfrom Ġa Ġnormal Ġmon ologue Ġor Ġprepared Ġspeech : Ġit Ġinvolves Ġmultiple Ġspeakers Ġwith Ġoverlapping Ġdialog ues , Ġcovers Ġa Ġvariety Ġof Ġspeech Ġpatterns , Ġand Ġthe Ġvocabulary Ġcan Ġrange Ġfrom Ġcoll oqu ial Ġto Ġcomplex Ġdomain - specific Ġlanguage . ĠWith Ġrecent Ġadvancements Ġin ĠConvers ational ĠSpeech ĠRecogn ition ĠB IB REF 7 Ġrendering Ġthe Ġsystems Ġless Ġprone Ġto Ġerrors , Ġthe Ġsubsequent Ġchallenge Ġof Ġunderstanding Ġand Ġextracting Ġrelevant Ġinformation Ġfrom Ġthe Ġconversations Ġis Ġreceiving Ġincreasing Ġresearch Ġfocus ĠB IB REF 4 , ĠB IB REF 8 . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġlocal Ġinformation Ġextraction Ġin Ġtransc ribed Ġclinical Ġconversations . ĠSpecifically , Ġwe Ġextract Ġdosage Ġ( e . g . Ġ5 mg ) Ġand Ġfrequency Ġ( e . g . Ġonce Ġa Ġday ) Ġfor Ġthe Ġmedications Ġ( e . g . Ġaspirin ) Ġfrom Ġthese Ġtranscripts , Ġcollectively Ġreferred Ġto Ġas ĠMed ication ĠReg imen Ġ( MR ) Ġextraction . ĠThe Ġinformation Ġextraction Ġis Ġlocal Ġas Ġwe Ġextract Ġthe Ġinformation Ġfrom Ġa Ġsegment Ġof Ġthe Ġtranscript Ġand Ġnot Ġthe Ġentire Ġtranscript Ġsince Ġdoing Ġthe Ġlatter Ġis Ġdifficult Ġowing Ġto Ġthe Ġlong Ġme andering Ġnature Ġof Ġthe Ġconversations Ġoften Ġwith Ġmultiple Ġmedication Ġreg imens Ġand Ġcare Ġplans Ġbeing Ġdiscussed . </s> The Ġchallenges Ġassociated Ġwith Ġthe ĠMed ication ĠReg imen Ġ( MR ) Ġextraction Ġtask Ġinclude Ġunderstanding Ġthe Ġspontaneous Ġdialog Ġwith Ġclinical Ġvocabulary Ġand Ġunderstanding Ġthe Ġrelationship Ġbetween Ġdifferent Ġentities Ġas Ġthe Ġdiscussion Ġcan Ġcontain Ġmultiple Ġmedications Ġand Ġdos ages Ġ( e . g . Ġdoctor Ġrev ising Ġa Ġdosage Ġor Ġreviewing Ġall Ġthe Ġcurrent Ġmedications ). </s> We Ġframe Ġthis Ġproblem Ġas Ġa ĠQuestion ĠAn sw ering Ġ( Q A ) Ġtask Ġby Ġgenerating Ġquestions Ġusing Ġtemplates . ĠWe Ġbase Ġthe ĠQ A Ġmodel Ġon Ġpointer - gener ator Ġnetworks ĠB IB REF 9 Ġaugmented Ġwith ĠCo - Att ent ions ĠB IB REF 10 . ĠIn Ġaddition , Ġwe Ġdevelop Ġmodels Ġcombining ĠQ A Ġand ĠInformation ĠExt raction Ġframeworks Ġusing Ġmulti - dec oder Ġ( one Ġeach Ġfor Ġdosage Ġand Ġfrequency ) Ġarchitecture . </s> L ack Ġof Ġavailability Ġof Ġa Ġlarge Ġvolume Ġof Ġdata Ġis Ġa Ġtypical Ġchallenge Ġin Ġhealthcare . ĠA Ġconversation Ġcorpus Ġby Ġitself Ġis Ġa Ġrare Ġcommodity Ġin Ġthe Ġhealthcare Ġdata Ġspace Ġbecause Ġof Ġthe Ġcost Ġand Ġdifficulty Ġin Ġhanding Ġ( because Ġof Ġdata Ġprivacy Ġconcerns ). ĠMoreover , Ġtransc ribing Ġand Ġlabeling Ġthe Ġconversations Ġis Ġa Ġcostly Ġprocess Ġas Ġit Ġrequires Ġdomain - specific Ġmedical Ġannotation Ġexpertise . ĠTo Ġaddress Ġdata Ġshortage Ġand Ġimprove Ġthe Ġmodel Ġperformance , Ġwe Ġinvestigate Ġdifferent Ġhigh - performance Ġcontextual Ġembed d ings Ġ( EL MO ĠB IB REF 11 , ĠB ERT ĠB IB REF 12 Ġand ĠClinical BER T ĠB IB REF 13 ), Ġand Ġpret rain Ġthe Ġmodels Ġon Ġa Ġclinical Ġsummar ization Ġtask . ĠWe Ġfurther Ġinvestigate Ġthe Ġeffects Ġof Ġtraining Ġdata Ġsize Ġon Ġour Ġmodels . </s> On Ġthe ĠMR Ġextraction Ġtask , ĠEL Mo Ġwith Ġenc oder Ġmulti - dec oder Ġarchitecture Ġand ĠB ERT Ġwith Ġenc oder - dec oder Ġwith Ġenc od ers Ġpret rained Ġon Ġthe Ġsummar ization Ġtask Ġperform Ġthe Ġbest . ĠThe Ġbest - performing Ġmodels Ġimprove Ġour Ġbaseline 's Ġdosage Ġand Ġfrequency Ġextract ions ĠR OU GE - 1 ĠF 1 Ġscores Ġfrom Ġ54 . 28 Ġand Ġ37 . 13 Ġto Ġ89 . 57 Ġand Ġ45 . 94 , Ġrespectively . </s> Using Ġour Ġmodels , Ġwe Ġpresent Ġthe Ġfirst Ġfully Ġautomated Ġsystem Ġto Ġextract ĠMR Ġtags Ġfrom Ġspontaneous Ġdoctor - patient Ġconversations . ĠWe Ġevaluate Ġthe Ġsystem Ġ( using Ġour Ġbest Ġperforming Ġmodels ) Ġon Ġthe Ġtranscripts Ġgenerated Ġfrom ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) ĠAPIs Ġoffered Ġby ĠGoogle Ġand ĠIBM . ĠIn ĠGoogle ĠAS R 's Ġtranscripts , Ġour Ġbest Ġmodel Ġobtained ĠR OU GE - 1 ĠF 1 Ġof Ġ71 . 75 Ġfor ĠDos age Ġextraction Ġ( which Ġin Ġthis Ġspecific Ġcase Ġequals Ġto Ġthe Ġpercentage Ġof Ġtimes Ġdosage Ġis Ġcorrect , Ġrefer ĠMet rics ĠSection Ġfor Ġmore Ġdetails ) Ġand Ġ40 . 13 Ġfor ĠFrequency Ġextraction Ġtasks . ĠOn Ġqualitative Ġevaluation , Ġwe Ġfind Ġthat Ġfor Ġ73 . 58 % Ġof Ġthe Ġmedications Ġthe Ġmodel Ġcan Ġfind Ġthe Ġcorrect Ġfrequency . ĠThese Ġresults Ġdemonstrate Ġthat Ġthe Ġresearch Ġon ĠN LP Ġcan Ġbe Ġused Ġeffectively Ġin Ġa Ġreal Ġclinical Ġsetting Ġto Ġbenefit Ġboth Ġdoctors Ġand Ġpatients </s> Data </s> Our Ġdataset Ġconsists Ġof Ġa Ġtotal Ġof Ġ6 , 693 Ġreal Ġdoctor - patient Ġconversations Ġrecorded Ġin Ġa Ġclinical Ġsetting Ġusing Ġdistant Ġmicrophones Ġof Ġvarying Ġquality . ĠThe Ġrecordings Ġhave Ġan Ġaverage Ġduration Ġof Ġ9 min Ġ28 s Ġand Ġhave Ġa Ġver bat im Ġtranscript Ġof Ġ1 , 500 Ġwords Ġon Ġaverage Ġ( written Ġby Ġthe Ġexperts ). ĠBoth Ġthe Ġaudio Ġand Ġthe Ġtranscript Ġare Ġde - identified Ġ( by Ġremoving Ġthe Ġidentifying Ġinformation ) Ġwith Ġdigital Ġz eros Ġand Ġ[ de - identified ] Ġtags , Ġrespectively . ĠThe Ġsentences Ġin Ġthe Ġtranscript Ġare Ġgrounded Ġto Ġthe Ġaudio Ġwith Ġthe Ġtim est amps Ġof Ġits Ġfirst Ġand Ġlast Ġword . </s> The Ġtranscript Ġof Ġthe Ġconversations Ġare Ġannot ated Ġwith Ġsumm aries Ġand ĠMed ication ĠReg imen Ġtags Ġ( MR Ġtags ), Ġboth Ġgrounded Ġusing Ġthe Ġtim est amps Ġof Ġthe Ġsentences Ġfrom Ġthe Ġtranscript Ġdeemed Ġrelevant Ġby Ġthe Ġexpert Ġannot ators , Ġrefer Ġto ĠTable ĠT AB REF 1 . ĠThe Ġtranscript Ġfor Ġa Ġtypical Ġconversation Ġcan Ġbe Ġquite Ġlong , Ġand Ġnot Ġeasy Ġfor Ġmany Ġof Ġthe Ġhigh Ġperforming Ġdeep Ġlearning Ġmodels Ġto Ġact Ġon . ĠMoreover , Ġthe Ġmedical Ġinformation Ġabout Ġa Ġconcept / condition / entity Ġcan Ġchange Ġduring Ġthe Ġconversation Ġafter Ġa Ġsignificant Ġtime Ġgap . ĠFor Ġexample , Ġdosage Ġof Ġa Ġmedication Ġcan Ġbe Ġdifferent Ġwhen Ġdiscussing Ġcurrent Ġmedication Ġthe Ġpatient Ġis Ġon Ġvs Ġwhen Ġthey Ġare Ġprescribed Ġa Ġdifferent Ġdosage . ĠHence , Ġwe Ġhave Ġannotations , Ġthat Ġare Ġgrounded Ġto Ġa Ġshort Ġsegment Ġof Ġthe Ġtranscript . </s> The Ġsumm aries Ġ(# words Ġ- Ġ$\ mu Ġ= Ġ9 . 7 ; Ġ\ s igma Ġ= Ġ10 . 1 $ ) Ġare Ġmedically Ġrelevant Ġand Ġlocal . ĠThe ĠMR Ġtags Ġare Ġalso Ġlocal Ġand Ġare Ġof Ġthe Ġform Ġ{ Med ication ĠName , ĠDos age , ĠFrequency }. ĠIf Ġdosage Ġ($ \ mu Ġ= Ġ2 . 0 ; Ġ\ s igma Ġ= Ġ0 $ ) Ġor Ġfrequency Ġ($ \ mu Ġ= Ġ2 . 1 ; Ġ\ s igma Ġ= Ġ1 . 07 $ ) Ġinformation Ġfor Ġa Ġmedication Ġis Ġnot Ġpresent Ġin Ġa Ġgrounded Ġsentence , Ġthe Ġcorresponding Ġfield Ġin Ġthe ĠMR Ġtag Ġwill Ġbe Ġmarked Ġas Ġ` none '. </s> In Ġthe ĠMR Ġtags , ĠMed ication ĠName Ġand ĠDos age Ġ( usually Ġa Ġquantity Ġfollowed Ġby Ġits Ġunits ) Ġcan Ġbe Ġrelatively Ġeasily Ġextracted Ġfrom Ġthe Ġtranscript Ġexcept Ġfor Ġthe Ġunits Ġof Ġthe Ġdosage Ġwhich Ġis Ġsometimes Ġinferred . ĠIn Ġcontrast , Ġdue Ġto Ġhigh Ġdegree Ġof Ġlinguistic Ġvariation Ġwith Ġwhich ĠFrequency Ġis Ġoften Ġexpressed , Ġextracting ĠFrequency Ġrequires Ġan Ġadditional Ġinference Ġstep . ĠFor Ġexample , Ġ` take Ġone Ġin Ġthe Ġmorning Ġand Ġat Ġnoon ' Ġfrom Ġthe Ġtranscript Ġis Ġtagged Ġas Ġ` tw ice Ġa Ġday ' Ġin Ġthe Ġfrequency Ġtag , Ġlikewise Ġ` take Ġit Ġbefore Ġsleeping ' Ġis Ġtagged Ġas Ġ` at Ġnight Ġtime '. </s> Out Ġof Ġoverall Ġ6 , 693 Ġfiles , Ġwe Ġset Ġaside Ġa Ġrandom Ġsample Ġof Ġ423 Ġfiles Ġ( den oted Ġas Ġ$\ math cal Ġ{ D } _{ test } $ ) Ġfor Ġfinal Ġevaluation . ĠThe Ġremaining Ġ6 , 270 Ġfiles Ġare Ġused Ġfor Ġtraining Ġwith Ġ80 % Ġtrain Ġ( 5 016 ), Ġ10 % Ġvalidation Ġ( 627 ), Ġand Ġ10 % Ġtest Ġ( 627 ) Ġsplit . ĠOverall , Ġthe Ġ6 , 270 Ġfiles Ġcontains Ġ156 , 186 Ġsumm aries Ġand Ġ32 , 000 ĠMR Ġtags Ġout Ġof Ġwhich Ġ8 , 654 ĠMR Ġtags Ġcontain Ġvalues Ġfor Ġat Ġleast Ġone Ġof Ġthe ĠDos age Ġor ĠFrequency , Ġwhich Ġwe Ġused Ġfor Ġtraining Ġto Ġavoid Ġover fitting Ġ( the Ġremaining ĠMR Ġtags Ġhave Ġboth ĠDos age Ġand ĠFrequency Ġas Ġ` none '). ĠNote Ġthat Ġwe Ġhave Ġtwo Ġtest Ġdatasets : Ġ` 10 % Ġtest ' Ġ- Ġused Ġto Ġevaluate Ġall Ġthe Ġmodels , Ġand Ġ$\ math cal Ġ{ D } _{ test } $ Ġ- Ġused Ġto Ġmeasure Ġthe Ġperformance Ġof Ġbest Ġperforming Ġmodels Ġon ĠAS R Ġtranscripts . </s> App roach </s> We Ġframe Ġthe ĠMed ication ĠReg imen Ġextraction Ġproblem Ġas Ġa ĠQuestion ĠAn sw ering Ġ( Q A ) Ġtask , Ġwhich Ġforms Ġthe Ġbasis Ġfor Ġour Ġfirst Ġapproach . ĠIt Ġcan Ġalso Ġbe Ġconsidered Ġas Ġa Ġspecific Ġinference Ġor Ġrelation Ġextract Ġtask , Ġsince Ġwe Ġextract Ġspecific Ġinformation Ġabout Ġan Ġentity Ġ( Med ication ĠName ), Ġhence Ġour Ġsecond Ġapproach Ġis Ġat Ġthe Ġintersection Ġof ĠQuestion ĠAn sw ering Ġ( Q A ) Ġand ĠInformation ĠExt raction Ġ( IE ) Ġdomains . ĠBoth Ġthe Ġapproaches Ġinvolve Ġusing Ġa Ġcontiguous Ġsegment Ġof Ġthe Ġtranscript Ġand Ġthe ĠMed ication ĠName Ġas Ġinput , Ġto Ġfind / in fer Ġthe Ġmedication 's ĠDos age Ġand ĠFrequency . ĠWhen Ġtesting Ġthe Ġapproaches Ġmim icking Ġreal - world Ġconditions , Ġwe Ġextract ĠMed ication ĠName Ġfrom Ġthe Ġtranscript Ġseparately Ġusing Ġont ology , Ġrefer Ġto ĠSEC REF 19 . </s> In Ġthe Ġfirst Ġapproach , Ġwe Ġframe Ġthe ĠMR Ġtask Ġas Ġa ĠQ A Ġtask Ġand Ġgenerate Ġquestions Ġusing Ġthe Ġtemplate : ĠâĢ ľ What Ġis Ġthe Ġ$ Ġ< $ dos age / frequency $ > $ Ġfor Ġ$ < $ Med ication ĠName $ > $ ". ĠHere , Ġwe Ġuse Ġan Ġabstract ive ĠQ A Ġmodel Ġbased Ġon Ġpointer - gener ator Ġnetworks ĠB IB REF 9 Ġaugmented Ġwith Ġco att ention Ġenc oder ĠB IB REF 10 Ġ( Q A - PG Net ). </s> In Ġthe Ġsecond Ġapproach , Ġwe Ġframe Ġthe Ġproblem Ġas Ġa Ġconditioned ĠIE Ġtask , Ġwhere Ġthe Ġinformation Ġextracted Ġdepends Ġon Ġan Ġentity Ġ( Med ication ĠName ). ĠHere , Ġwe Ġuse Ġa Ġmulti - dec oder Ġpointer - gener ator Ġnetwork Ġaugmented Ġwith Ġco att ention Ġenc oder Ġ( Multi - dec oder ĠQ A - PG Net ). ĠInstead Ġof Ġusing Ġtemplates Ġto Ġgenerate Ġquestions Ġand Ġusing Ġa Ġsingle Ġdec oder Ġto Ġextract Ġdifferent Ġtypes Ġof Ġinformation Ġas Ġin Ġthe ĠQ A Ġapproach Ġ( which Ġmight Ġlead Ġto Ġperformance Ġdegradation ), Ġhere Ġwe Ġconsider Ġseparate Ġdec od ers Ġfor Ġextracting Ġspecific Ġtypes Ġof Ġinformation Ġabout Ġan Ġentity Ġ$ E $ Ġ( Med ication ĠName ). </s> App roach Ġ: :: ĠPo inter - gener ator ĠNetwork Ġ( PG Net ) </s> The Ġnetwork Ġis Ġa Ġsequence - to - sequence Ġattention Ġmodel Ġthat Ġcan Ġboth Ġcopy Ġa Ġword Ġfrom Ġthe Ġinput Ġ$ I $ Ġcontaining Ġ$ P $ Ġword Ġtokens Ġor Ġgenerate Ġa Ġword Ġfrom Ġits Ġvocabulary Ġ$ voc ab $, Ġto Ġproduce Ġthe Ġoutput Ġsequence . </s> First , Ġthe Ġtokens Ġof Ġthe Ġ$ I $ Ġare Ġconverted Ġto Ġembed d ings Ġand Ġare Ġfed Ġone - by - one Ġto Ġthe Ġenc oder , Ġa Ġsingle Ġbi - L ST M Ġlayer , Ġwhich Ġenc odes Ġthe Ġtokens Ġin Ġ$ I $ Ġinto Ġa Ġsequence Ġof Ġhidden Ġstates Ġ- Ġ$ H = enc oder ( I ) $, Ġwhere Ġ$ ĠH =[ h _ 1 ... h _ P ] $. </s> For Ġeach Ġdec oder Ġtime Ġstep Ġ$ t $, Ġin Ġa Ġloop , Ġwe Ġcompute , Ġ1 ) Ġattention Ġ$ a _ t $ Ġ( using Ġthe Ġlast Ġdec oder Ġstate Ġ$ s _{ t - 1 } $ ), Ġover Ġthe Ġinput Ġtokens Ġ$ I $, Ġand Ġ2 ) Ġthe Ġdec oder Ġstate Ġ$ s _ t $ Ġusing Ġ$ a _ t $. ĠThen , Ġat Ġeach Ġtime Ġstep , Ġusing Ġboth Ġ$ a _ t $ Ġand Ġ$ s _ t $ Ġwe Ġcan Ġfind Ġthe Ġprobability Ġ$ P _ t ( w ) $, Ġof Ġproducing Ġa Ġword Ġ$ w $ Ġ( from Ġboth Ġ$ voc ab $ Ġand Ġ$ I $ ). ĠFor Ġconvenience , Ġwe Ġdenote Ġthe Ġattention Ġand Ġthe Ġdec oder Ġas Ġ$ dec oder _{ pg } ( H )= P ( w ) $, Ġwhere Ġ$ P ( w ) =[ P _ 1 ( w )... P _ T ( w )] $. ĠThe Ġoutput Ġcan Ġthen Ġbe Ġdec oded Ġfrom Ġ$ P ( w ) $, Ġwhich Ġis Ġdec oded Ġuntil Ġit Ġproduces Ġan Ġ` end Ġof Ġoutput Ġtoken ' Ġor Ġthe Ġnumber Ġof Ġsteps Ġreach Ġthe Ġmaximum Ġallowed Ġlimit . </s> App roach Ġ: :: ĠQ A ĠPG Net </s> We Ġfirst Ġencode Ġboth Ġthe Ġquestion Ġ- Ġ$ H _ Q Ġ= Ġenc oder ( Q ) $, Ġand Ġthe Ġinput Ġ- Ġ$ H _ I Ġ= Ġenc oder ( I ) $, Ġseparately Ġusing Ġenc od ers Ġ( with Ġshared Ġweights ). ĠThen , Ġto Ġcondition Ġ$ I $ Ġon Ġ$ Q $ Ġ( and Ġvice Ġversa ), Ġwe Ġuse Ġthe Ġco att ention Ġenc oder ĠB IB REF 10 Ġwhich Ġattends Ġto Ġboth Ġthe Ġ$ I $ Ġand Ġ$ Q $ Ġsimultaneously Ġto Ġgenerates Ġthe Ġco att ention Ġcontext Ġ- Ġ$ C _ D Ġ= Ġco att ( H _ I , ĠH _ Q ) $. ĠFinally , Ġusing Ġthe Ġpointer - gener ator Ġdec oder Ġwe Ġfind Ġthe Ġprobability Ġdistribution Ġof Ġthe Ġoutput Ġsequence Ġ- Ġ$ P ( w ) Ġ= Ġdec oder _{ pg } ([ H _ I ; ĠC _ D ]) $, Ġwhich Ġis Ġthen Ġdec oded Ġto Ġgenerate Ġthe Ġanswer Ġsequence . </s> App roach Ġ: :: ĠMulti - dec oder Ġ( MD ) ĠQ A ĠPG Net </s> After Ġencoding Ġthe Ġinputs Ġinto Ġ$ H _ I $ Ġand Ġ$ H _ E $, Ġfor Ġextracting Ġ$ K $ Ġtypes Ġof Ġinformation Ġabout Ġan Ġentity Ġin Ġan ĠIE Ġfashion , Ġwe Ġuse Ġthe Ġfollowing Ġmulti - dec oder Ġ( MD ) Ġsetup : </s> Pred ictions Ġfor Ġeach Ġof Ġthe Ġ$ K $ Ġdec od ers Ġare Ġthen Ġdec oded Ġusing Ġ$ P ^ k ( w ) $. </s> All Ġthe Ġnetworks Ġdiscussed Ġabove Ġare Ġtrained Ġusing Ġa Ġnegative Ġlog - like lihood Ġloss Ġfor Ġthe Ġtarget Ġword Ġat Ġeach Ġtime Ġstep Ġand Ġsummed Ġover Ġall Ġthe Ġdec oder Ġtime Ġsteps . </s> Exper iments </s> We Ġinitialized ĠMR Ġextraction Ġmodels ' Ġvocabulary Ġfrom Ġthe Ġtraining Ġdataset Ġafter Ġremoving Ġwords Ġwith Ġa Ġfrequency Ġlower Ġthan Ġ30 Ġin Ġthe Ġdataset , Ġresulting Ġin Ġ4 56 Ġwords . ĠOur Ġvocabulary Ġis Ġsmall Ġbecause Ġof Ġthe Ġsize Ġof Ġthe Ġdataset , Ġhence Ġwe Ġrely Ġon Ġthe Ġmodel 's Ġability Ġto Ġcopy Ġwords Ġto Ġproduce Ġthe Ġoutput Ġeffectively . ĠIn Ġall Ġour Ġmodel Ġvariations , Ġthe Ġembed ding Ġand Ġthe Ġnetwork 's Ġhidden Ġdimension Ġare Ġset Ġto Ġbe Ġequal . ĠThe Ġnetworks Ġwere Ġtrained Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 00 15 , Ġdrop out Ġof Ġ0 . 5 Ġon Ġthe Ġembed ding Ġlayer , Ġnormal Ġgradient Ġclipping Ġset Ġat Ġ2 , Ġbatch Ġsize Ġof Ġ8 , Ġand Ġoptimized Ġwith ĠAd ag rad ĠB IB REF 15 Ġand Ġthe Ġtraining Ġwas Ġstopped Ġusing Ġthe Ġ$ 10 \ % $ Ġvalidation Ġdataset . </s> Exper iments Ġ: :: ĠData ĠProcessing </s> We Ġdid Ġthe Ġfollowing Ġbasic Ġpre processing Ġto Ġour Ġdata , Ġ1 ) Ġadded Ġ` none ' Ġto Ġthe Ġbeginning Ġof Ġthe Ġinput Ġutter ance , Ġso Ġthe Ġnetwork Ġcould Ġpoint Ġto Ġit Ġwhen Ġthere Ġwas Ġno Ġrelevant Ġinformation Ġin Ġthe Ġinput , Ġ2 ) Ġfiltered Ġoutl iers Ġwith Ġa Ġlarge Ġnumber Ġof Ġgrounded Ġtranscript Ġsentences Ġ($ > $ 150 Ġwords ), Ġand Ġ3 ) Ġconverted Ġall Ġtext Ġto Ġlower Ġcase . </s> To Ġimprove Ġperformance , Ġwe Ġ1 ) Ġstandardized Ġall Ġnumbers Ġ( both Ġdigits Ġand Ġwords ) Ġto Ġwords Ġconc aten ated Ġwith Ġa Ġhyp hen Ġ( e . g . Ġ110 Ġ- $ > $ Ġone - h undred - ten ), Ġin Ġboth Ġinput Ġand Ġoutput , Ġ2 ) Ġremoved Ġunits Ġfrom ĠDos age Ġas Ġsometimes Ġthe Ġunits Ġwere Ġnot Ġexplicitly Ġmentioned Ġin Ġthe Ġtranscript Ġsegment Ġbut Ġwere Ġwritten Ġby Ġthe Ġannot ators Ġusing Ġdomain Ġknowledge , Ġ3 ) Ġprep ended Ġall Ġmedication Ġmentions Ġwith Ġ` rx -' Ġtag , Ġas Ġthis Ġhelps Ġmodel 's Ġperformance Ġwhen Ġmultiple Ġmedications Ġare Ġdiscussed Ġin Ġa Ġsegment Ġ( in Ġboth Ġinput Ġand Ġoutput ), Ġand Ġ4 ) Ġwhen Ġa Ġtranscript Ġsegment Ġhas Ġmultiple Ġmedications Ġor Ġdos ages Ġbeing Ġdiscussed Ġwe Ġrandomly Ġshuffle Ġthem Ġ( in Ġboth Ġinput Ġand Ġoutput ) Ġand Ġcreate Ġa Ġnew Ġdata Ġpoint , Ġto Ġincreases Ġthe Ġnumber Ġof Ġtraining Ġdata Ġpoints . ĠRandom ly Ġshuff ling Ġthe Ġentities Ġincreases Ġthe Ġnumber Ġof Ġtraining ĠMR Ġtags Ġfrom Ġ8 , 654 Ġto Ġ11 , 5 21 . ĠBased Ġon Ġthe Ġdata Ġstatistics Ġafter Ġdata Ġprocessing , Ġwe Ġfixed Ġthe Ġmaximum Ġenc oder Ġsteps Ġto Ġ100 , Ġdosage Ġdec oder Ġsteps Ġto Ġ1 , Ġand Ġfrequency Ġdec oder Ġsteps Ġto Ġ3 Ġ( for Ġboth Ġthe ĠQ A Ġand ĠMulti - dec oder ĠQ A Ġmodels ). </s> Exper iments Ġ: :: ĠMet rics </s> For Ġthe ĠMR Ġextraction Ġtask , Ġwe Ġmeasure Ġthe ĠR OU GE - 1 Ġscores ĠB IB REF 14 Ġfor Ġboth Ġthe ĠDos age Ġand ĠFrequency Ġextraction Ġtasks . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġsince ĠDos age Ġis Ġa Ġsingle Ġword Ġtoken Ġ( after Ġprocessing ), Ġboth Ġthe Ġreference Ġand Ġhypothesis Ġare Ġa Ġsingle Ġtoken , Ġmaking Ġits ĠR OU GE - 1 ĠF 1 , ĠPrecision Ġand ĠRecall Ġscores Ġequal Ġand Ġbe Ġequal Ġto Ġpercentage Ġof Ġtimes Ġwe Ġfind Ġthe Ġcorrect Ġdosage Ġfor Ġthe Ġmedications . </s> In Ġour Ġannotations , ĠFrequency Ġhas Ġconflicting Ġtags Ġ( e . g . Ġ{ ` Once Ġa Ġday ', Ġ` tw ice Ġa Ġday ' } Ġand Ġ` daily '), Ġhence Ġmetrics Ġlike ĠEx act ĠMatch Ġwill Ġbe Ġerroneous . ĠTo Ġaddress Ġthis Ġissue , Ġwe Ġuse Ġthe ĠR OU GE Ġscores Ġto Ġcompare Ġdifferent Ġmodels Ġon Ġthe Ġ10 % Ġtest Ġdataset Ġand Ġwe Ġuse Ġqualitative Ġevaluation Ġto Ġmeasure Ġthe Ġtop - performing Ġmodels Ġon Ġ$\ math cal Ġ{ D } _{ test } $. </s> Exper iments Ġ: :: ĠModel Ġvariations </s> We Ġconsider ĠQ A ĠPG Net Ġand ĠMulti - dec oder ĠQ A ĠPG Net Ġwith Ġlookup Ġtable Ġembed ding Ġas Ġbaseline Ġmodels Ġand Ġimprove Ġon Ġthe Ġbas elines Ġwith Ġother Ġvariations Ġdescribed Ġbelow . </s> Apart Ġfrom Ġlearning - based Ġbas elines , Ġwe Ġalso Ġcreate Ġtwo Ġnaive Ġbas elines , Ġone Ġeach Ġfor Ġthe ĠDos age Ġand ĠFrequency Ġextraction Ġtasks . ĠFor ĠDos age Ġextraction , Ġthe Ġbaseline Ġwe Ġconsider Ġis Ġ` Ne arest ĠNumber ', Ġwhere Ġwe Ġtake Ġthe Ġnumber Ġnearest Ġto Ġthe ĠMed ication ĠName Ġas Ġthe Ġprediction , Ġand Ġ` none ' Ġif Ġno Ġnumber Ġis Ġmentioned Ġor Ġif Ġthe ĠMed ication ĠName Ġis Ġnot Ġdetected Ġin Ġthe Ġinput . ĠFor ĠFrequency Ġextraction , Ġthe Ġbaseline Ġwe Ġconsider Ġis Ġ` Random ĠTop - 3 ' Ġwhere Ġwe Ġpredict Ġa Ġrandom ĠFrequency Ġtag , Ġfrom Ġtop - 3 Ġmost Ġfrequent Ġones Ġfrom Ġour Ġdataset Ġ- Ġ{ ` none ', Ġ` daily ', Ġ` tw ice Ġa Ġday ' }. </s> Emb edd ing : ĠWe Ġdeveloped Ġdifferent Ġvariations Ġof Ġour Ġmodels Ġwith Ġa Ġsimple Ġlookup Ġtable Ġembed d ings Ġlearned Ġfrom Ġscratch Ġand Ġusing Ġhigh - performance Ġcontextual Ġembed d ings , Ġwhich Ġare ĠEL Mo ĠB IB REF 11 , ĠB ERT ĠB IB REF 16 Ġand ĠClinical BER T ĠB IB REF 13 Ġ( trained Ġand Ġprovided Ġby Ġthe Ġauthors ). ĠRefer Ġto ĠTable ĠT AB REF 5 Ġfor Ġthe Ġperformance Ġcomparisons . </s> We Ġderive Ġembed d ings Ġfrom ĠEL Mo Ġby Ġlearning Ġa Ġlinear Ġcombination Ġof Ġits Ġlast Ġthree Ġlayer 's Ġhidden Ġstates Ġ( task - specific Ġfine - tun ing ĠB IB REF 11 ). ĠSimilarly , Ġfor ĠB ERT - based Ġembed d ings , Ġwe Ġtake Ġa Ġlinear Ġcombination Ġof Ġthe Ġhidden Ġstates Ġfrom Ġits Ġlast Ġfour Ġlayers , Ġas Ġthis Ġcombination Ġperforms Ġbest Ġwithout Ġincreasing Ġthe Ġsize Ġof Ġthe Ġembed d ings ĠB IB REF 16 . ĠSince ĠB ERT Ġand ĠClinical BER T Ġuse Ġword - piece Ġvocabulary Ġand Ġcomp utes Ġsub - word Ġembed d ings , Ġwe Ġcompute Ġword - level Ġembed ding Ġby Ġaveraging Ġthe Ġcorresponding Ġsub - word Ġtokens . ĠEL Mo Ġand ĠB ERT Ġembed d ings Ġboth Ġhave Ġ1024 Ġdimensions , ĠClinical BER T Ġhave Ġ768 Ġas Ġit Ġis Ġbased Ġon ĠB ERT Ġbase Ġmodel , Ġand Ġthe Ġlookup Ġtable Ġhave Ġ128 ĠâĢĵ Ġhigher Ġdimension Ġmodels Ġleads Ġto Ġover fitting . </s> P ert aining ĠEnc oder : ĠWe Ġtrained Ġthe ĠPG Net Ġas Ġa Ġsummar ization Ġtask Ġusing Ġthe Ġclinical Ġsumm aries Ġand Ġused Ġthe Ġtrained Ġmodel Ġto Ġinitialize Ġthe Ġenc od ers Ġ( and Ġthe Ġembed d ings ) Ġof Ġthe Ġcorresponding ĠQ A Ġmodels . ĠWe Ġuse Ġa Ġvoc ab Ġsize Ġof Ġ40 73 Ġwords , Ġderived Ġfrom Ġthe Ġtraining Ġdataset Ġwith Ġa Ġfrequency Ġthreshold Ġof Ġ30 Ġfor Ġthe Ġtask . ĠWe Ġtrained Ġthe Ġmodels Ġusing ĠAd ag rad Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ0 . 015 , Ġnormal Ġgradient Ġclipping Ġset Ġat Ġ2 Ġand Ġtrained Ġfor Ġaround Ġ15 0000 Ġiterations Ġ( st opped Ġusing Ġvalidation Ġdataset ). ĠOn Ġthe Ġsummar ization Ġtask ĠPG Net Ġobtained ĠR OU GE - 1 ĠF 1 Ġscores Ġof Ġ41 . 42 Ġwith ĠEL Mo Ġand Ġ39 . 15 Ġwith ĠB ERT Ġembed d ings . ĠWe Ġcompare Ġthe Ġeffects Ġof Ġpret raining Ġthe Ġmodel Ġin ĠTable : ĠT AB REF 5 , Ġmodels Ġwith Ġ` pret rained Ġenc oder ' Ġhad Ġtheir Ġenc od ers Ġand Ġembed d ings Ġpret rained Ġwith Ġthe Ġsummar ization Ġtask . </s> Results Ġand ĠDiscussion Ġ: :: ĠDifference Ġin Ġnetworks Ġand Ġapproaches </s> Emb edd ings : ĠOn ĠDos age Ġextraction , Ġin Ġgeneral , ĠEL Mo Ġob tains Ġbetter Ġperformance Ġthan ĠB ERT , Ġrefer Ġto ĠTable ĠT AB REF 5 . ĠThis Ġcould Ġbe Ġbecause Ġwe Ġconc aten ated Ġthe Ġnumbers Ġwith Ġa Ġhyp hen , Ġand Ġas ĠEL Mo Ġuses Ġcharacter - level Ġtokens Ġit Ġcan Ġlearn Ġthe Ġtagging Ġbetter Ġthan ĠB ERT , Ġa Ġsimilar Ġobservation Ġis Ġalso Ġnoted Ġin ĠB IB REF 17 . ĠOn Ġthe Ġother Ġhand , Ġon ĠFrequency Ġextraction , Ġwithout Ġpret raining , ĠEL Mo 's Ġperformance Ġis Ġl agging Ġby Ġa Ġbig Ġmargin Ġof Ġ$\ sim Ġ$ 8 . 5 ĠR OU GE - 1 ĠF 1 Ġcompared Ġto ĠB ERT - based Ġembed d ings . </s> Although Ġin Ġcases Ġwithout Ġenc oder Ġpret raining , ĠClinical BER T Ġperformed Ġthe Ġbest Ġin Ġthe ĠFrequency Ġextraction Ġtask Ġ( by Ġa Ġsmall Ġmargin ), Ġin Ġgeneral , Ġit Ġdoes Ġnot Ġperform Ġas Ġwell Ġas ĠB ERT . ĠThis Ġcould Ġalso Ġbe Ġa Ġreflection Ġof Ġthe Ġfact Ġthat Ġthe Ġlanguage Ġand Ġstyle Ġof Ġwriting Ġused Ġin Ġclinical Ġnotes Ġis Ġvery Ġdifferent Ġfrom Ġthe Ġway Ġdoctors Ġcon verse Ġwith Ġpatients Ġand Ġthe Ġembed ding Ġdimension Ġdifference . ĠLook up Ġtable Ġembed ding Ġperformed Ġdec ently Ġin Ġthe Ġfrequency Ġextraction Ġtask Ġbut Ġl ags Ġbehind Ġin Ġthe ĠDos age Ġextraction Ġtask . </s> From Ġthe Ġmetrics Ġand Ġqualitative Ġinspection , Ġwe Ġfind Ġthat Ġthe ĠFrequency Ġextraction Ġis Ġan Ġeasier Ġtask Ġthan Ġthe ĠDos age Ġextraction . ĠThis Ġis Ġbecause , Ġin Ġthe Ġconversations , Ġfrequency Ġinformation Ġusually Ġoccurs Ġin Ġisolation Ġand Ġnear Ġthe Ġmedications , Ġbut Ġa Ġmedication 's Ġdosage Ġcan Ġoccur Ġ1 ) Ġnear Ġother Ġmedication 's Ġdos ages , Ġ2 ) Ġwith Ġprevious Ġdos ages Ġ( when Ġa Ġdosage Ġfor Ġa Ġmedication Ġis Ġrevised ), Ġand Ġ3 ) Ġafter Ġa Ġlarge Ġnumber Ġof Ġwords Ġfrom Ġthe Ġmedication . </s> Other ĠVari ations : ĠConsidering Ġvarious Ġmodels ' Ġperformance Ġ( without Ġpret raining ) Ġand Ġthe Ġresource Ġconstraint , Ġwe Ġchoose ĠEL Mo Ġand ĠB ERT Ġembed d ings Ġto Ġanalyze Ġthe Ġeffects Ġof Ġpret raining Ġthe Ġenc oder . ĠWhen Ġthe Ġnetwork 's Ġenc oder Ġ( and Ġembed ding ) Ġis Ġpret rained Ġwith Ġthe Ġsummar ization Ġtask , Ġwe Ġ1 ) Ġsee Ġa Ġsmall Ġdecrease Ġin Ġthe Ġaverage Ġnumber Ġof Ġiterations Ġrequired Ġfor Ġtraining , Ġ2 ) Ġimprovement Ġin Ġindividual Ġperformances Ġof Ġall Ġmodels Ġfor Ġboth Ġthe Ġsub - t asks , Ġand Ġ3 ) Ġget Ġbest Ġperformance Ġmetrics Ġacross Ġall Ġvariations , Ġrefer Ġto ĠTable ĠT AB REF 5 . ĠBoth Ġin Ġterms Ġof Ġperformance Ġand Ġthe Ġtraining Ġspeed , Ġthere Ġis Ġno Ġclear Ġwinner Ġbetween Ġshared Ġand Ġmulti - dec oder Ġapproaches . ĠMed ication Ġtagging Ġand Ġdata Ġaug mentation Ġincrease Ġthe Ġbest - performing Ġmodel 's ĠR OU GE - 1 ĠF 1 Ġscore Ġby Ġ$\ sim Ġ$ 1 . 5 Ġfor Ġthe ĠDos age Ġextraction Ġtask . </s> We Ġalso Ġmeasure Ġthe Ġperformance Ġof ĠMult it ask ĠQuestion ĠAn sw ering ĠNetwork Ġ( MQ AN ) ĠB IB REF 18 Ġa ĠQ A Ġmodel Ġtrained Ġby Ġthe Ġauthors Ġon Ġthe ĠDec athlon Ġmultit ask Ġchallenge . ĠSince ĠM Q AN Ġwas Ġnot Ġtrained Ġto Ġproduce Ġthe Ġoutput Ġsequence Ġin Ġour ĠMR Ġtags , Ġit Ġwould Ġnot Ġbe Ġfair Ġto Ġcompute ĠR OU GE Ġscores . ĠInstead , Ġwe Ġrandomly Ġsample Ġthe ĠM Q AN 's Ġpredictions Ġfrom Ġthe Ġ10 % Ġtest Ġdataset Ġand Ġqual itatively Ġevaluate Ġit . ĠFrom Ġthe Ġevaluations , Ġwe Ġfind Ġthat ĠM Q AN Ġcan Ġnot Ġdistinguish Ġbetween Ġfrequency Ġand Ġdosage , Ġand Ġmixed Ġthe Ġanswers . ĠM Q AN Ġcorrectly Ġpredicted Ġthe Ġdosage Ġfor Ġ29 . 73 % Ġand Ġfrequency Ġfor Ġ24 . 24 % Ġpercent Ġof Ġthe Ġmedications Ġcompared Ġto Ġ84 . 12 % Ġand Ġ76 . 34 % Ġfor Ġthe Ġenc oder Ġpret rained ĠB ERT ĠQ A ĠPG Net Ġmodel Ġtrained Ġon Ġour Ġdataset . ĠThis Ġcould Ġbe Ġbecause Ġof Ġthe Ġdifference Ġin Ġthe Ġtraining Ġdataset , Ġdomain Ġand Ġthe Ġtasks Ġin Ġthe ĠDec athlon Ġchallenge Ġcompared Ġto Ġours . </s> Almost Ġall Ġour Ġmodels Ġperform Ġbetter Ġthan Ġthe Ġnaive Ġbas elines Ġand Ġthe Ġones Ġusing Ġlookup Ġtable Ġembed d ings , Ġand Ġour Ġbest Ġperforming Ġmodels Ġoutper form Ġthem Ġsignificantly . ĠAmong Ġall Ġthe Ġvariations , Ġthe Ġbest Ġperforming Ġmodels Ġare ĠEL Mo Ġwith ĠMulti - dec oder Ġ( D os age Ġextraction ) Ġand ĠB ERT Ġwith Ġshared - dec oder ĠQ A ĠPG Net Ġarchitecture Ġwith Ġpret rained Ġenc oder Ġ( F requency Ġextraction ). ĠWe Ġchoose Ġthese Ġtwo Ġmodels Ġfor Ġour Ġsubsequent Ġanalysis . </s> Results Ġand ĠDiscussion Ġ: :: ĠBreak down Ġof ĠPerformance </s> We Ġcategor ize Ġthe Ġ10 % Ġtest Ġdataset Ġinto Ġdifferent Ġcategories Ġbased Ġon Ġthe Ġcomplexity Ġand Ġtype Ġof Ġthe Ġdata Ġand Ġanalyze Ġthe Ġbreakdown Ġof Ġthe Ġsystem 's Ġperformance Ġin ĠTable ĠT AB REF 11 . ĠWe Ġbreakdown Ġthe ĠFrequency Ġextraction Ġinto Ġtwo Ġcategories , Ġ1 ) ĠNone : Ġground Ġtruth ĠFrequency Ġtag Ġis Ġ` none ', Ġand Ġ2 ) ĠN N Ġ( Not ĠNone ): Ġground Ġtruth ĠFrequency Ġtag Ġis Ġnot Ġ` none '. ĠSimilarly , Ġthe ĠDos age Ġextraction Ġinto Ġ5 Ġcategories , Ġ1 ) ĠNone : Ġground Ġtruth Ġdosage Ġtag Ġis Ġ` none ', Ġ2 ) ĠMM Ġ( Multiple ĠMedicine ): Ġinput Ġsegment Ġhas Ġmore Ġthan Ġone ĠMed ication Ġmentioned , Ġ3 ) ĠMN Ġ( Multiple ĠNumbers ): Ġinput Ġsegment Ġhas Ġmore Ġthan Ġone Ġnumber Ġpresent , Ġand Ġ4 ) ĠN BM Ġ( Number Ġbetween Ġcorrect ĠDos age Ġand ĠMedicine ) Ġ: Ġbetween Ġthe ĠMed ication ĠName Ġand Ġthe Ġcorrect ĠDos age Ġin Ġthe Ġinput Ġsegment Ġthere Ġare Ġother Ġnumbers Ġpresent . ĠNote Ġthat Ġthe Ġcategories Ġof Ġthe ĠDos age Ġextraction Ġtask Ġare Ġnot Ġexhaustive , Ġand Ġone Ġtag Ġcan Ġbelong Ġto Ġmultiple Ġcategories . </s> From Ġthe Ġperformance Ġbreakdown Ġof ĠDos age Ġextraction Ġtask , Ġwe Ġsee Ġthat Ġ1 ) Ġthe Ġmodels Ġpredict Ġ` none ' Ġbetter Ġthan Ġother Ġcategories , Ġi . e ., Ġthe Ġmodels Ġare Ġcorrectly Ġable Ġto Ġidentify Ġwhen Ġa Ġmedication 's Ġdosage Ġis Ġabsent , Ġ2 ) Ġthere Ġis Ġperformance Ġdip Ġin Ġhard Ġcases Ġ( MM , ĠMN , Ġand ĠN BM ), Ġ3 ) Ġthe Ġmodels Ġare Ġable Ġto Ġfigure Ġout Ġthe Ġcorrect Ġdosage Ġ( dec ently ) Ġfor Ġa Ġmedication Ġeven Ġwhen Ġthere Ġare Ġmultiple Ġnumbers / dos age Ġpresent , Ġand Ġ4 ) Ġthe Ġmodel Ġstruggles Ġthe Ġmost Ġin Ġthe ĠN BM Ġcategory . ĠThe Ġmodels ' Ġlow Ġperformance Ġin ĠN BM Ġcould Ġbe Ġbecause Ġwe Ġhave Ġa Ġcomparatively Ġlower Ġnumber Ġof Ġexamples Ġto Ġtrain Ġin Ġthis Ġcategory . ĠThe ĠFrequency Ġextraction Ġtask Ġperforms Ġequally Ġwell Ġwhen Ġthe Ġtag Ġis Ġ` none ` Ġor Ġnot . ĠIn Ġmost Ġcategories , Ġwe Ġsee Ġan Ġincrease Ġin Ġperformance Ġwhen Ġusing Ġpret rained Ġenc od ers . </s> Results Ġand ĠDiscussion Ġ: :: ĠTraining ĠDat as et ĠSize </s> We Ġvary Ġthe Ġnumber Ġof ĠMR Ġtags Ġused Ġto Ġtrain Ġthe Ġmodel Ġand Ġanalyze Ġthe Ġmodel 's Ġperformance Ġwhen Ġtraining Ġthe Ġnetworks , Ġusing Ġpublicly Ġavailable Ġcontextual Ġembed d ings , Ġcompared Ġto Ġusing Ġpret rained Ġembed d ings Ġand Ġenc oder Ġ( pret rained Ġon Ġthe Ġsummar ization Ġtask ). ĠOut Ġof Ġthe Ġ5 , 016 Ġfiles Ġin Ġthe Ġ80 % Ġtrain Ġdataset Ġonly Ġ2 , 476 Ġhave Ġat le ast Ġone ĠMR Ġtag . ĠTherefore , Ġout Ġof Ġthe Ġ24 76 Ġfiles , Ġwe Ġrandomly Ġchoose Ġ100 , Ġ500 , Ġand Ġ1000 Ġfiles Ġand Ġtrained Ġthe Ġbest Ġperforming Ġmodel Ġvariations Ġto Ġobserve Ġthe Ġperformance Ġdifferences , Ġrefer Ġto ĠFigure ĠFIG REF 12 . ĠFor Ġall Ġthese Ġexperiments Ġwe Ġused Ġthe Ġsame Ġvocabulary Ġsize Ġ( 456 ), Ġthe Ġsame Ġhyper / training Ġparameters , Ġand Ġthe Ġsame Ġ10 % Ġtest Ġsplit Ġof Ġ6 27 Ġfiles . </s> As Ġexpected , Ġwe Ġsee Ġthat Ġthe Ġenc oder Ġpret rained Ġmodels Ġhave Ġhigher Ġperformance Ġon Ġall Ġthe Ġdifferent Ġtraining Ġdata Ġsizes , Ġi . e ., Ġthey Ġachieve Ġhigher Ġperformance Ġon Ġa Ġlower Ġnumber Ġof Ġdata Ġpoints , Ġrefer Ġto ĠFigure ĠFIG REF 12 . ĠThe Ġdifference , Ġas Ġexpected , Ġshr inks Ġas Ġthe Ġtraining Ġdata Ġsize Ġincreases . </s> Results Ġand ĠDiscussion Ġ: :: ĠEval uating Ġon ĠAS R Ġtranscripts </s> To Ġtest Ġthe Ġperformance Ġof Ġour Ġmodels Ġon Ġreal - world Ġconditions , Ġwe Ġuse Ġcommercially Ġavailable ĠAS R Ġservices Ġ( Google Ġand ĠIBM ) Ġto Ġtransc ribe Ġthe Ġ$\ math cal Ġ{ D } _{ test } $ Ġfiles Ġand Ġmeasure Ġthe Ġperformance Ġof Ġour Ġmodels Ġwithout Ġassuming Ġany Ġannotations Ġ( except Ġwhen Ġcalculating Ġthe Ġmetrics ). ĠIt Ġshould Ġbe Ġnoted Ġthat Ġthis Ġis Ġnot Ġthe Ġcase Ġin Ġour Ġprevious Ġevaluations Ġusing Ġ` 10 % Ġtest ' Ġdataset Ġwhere Ġwe Ġuse Ġthe Ġsegment ation Ġinformation . ĠFor Ġground Ġtruth Ġannotations Ġon ĠAS R Ġtranscripts , Ġwe Ġaligned Ġthe ĠMR Ġtags Ġfrom Ġhuman Ġwritten Ġtranscripts Ġto Ġthe ĠAS R Ġtranscript Ġusing Ġtheir Ġgrounded Ġtiming Ġinformation . ĠAdditionally , Ġsince ĠAS R Ġis Ġprone Ġto Ġerrors , Ġduring Ġthe Ġalignment , Ġif Ġa Ġmedication Ġfrom Ġan ĠMR Ġtag Ġis Ġnot Ġrecognized Ġcorrectly Ġin Ġthe ĠAS R Ġtranscript , Ġwe Ġremove Ġthe Ġcorresponding ĠMR Ġtag . </s> In Ġour Ġevaluations , Ġwe Ġuse ĠGoogle ĠCloud ĠSpeech - to - Text Ġ( G - ST T ) Ġand ĠIBM ĠWatson ĠSpeech Ġto ĠText Ġ( IB M - ST T ) Ġas Ġthese Ġwere Ġamong Ġthe Ġtop - performing ĠAS R ĠAPIs Ġon Ġmedical Ġspeech ĠB IB REF 19 Ġand Ġwere Ġreadily Ġavailable Ġto Ġus . ĠWe Ġused ĠG - ST T , Ġwith Ġthe Ġ` video Ġmodel ' Ġwith Ġpunct uation Ġsettings . ĠUnlike Ġour Ġhuman Ġwritten Ġtranscripts , Ġthe Ġtranscript Ġprovided Ġby ĠG - ST T Ġis Ġnot Ġver bat im Ġand Ġdoes Ġnot Ġhave Ġdis flu encies . ĠIBM - ST T , Ġon Ġthe Ġother Ġhand , Ġdoes Ġnot Ġgive Ġpunct uation Ġso Ġwe Ġused Ġthe Ġspeaker Ġchanges Ġto Ġadd Ġend - of - sent ence Ġpunct uation . </s> In Ġour Ġ$\ math cal Ġ{ D } _{ test } $ Ġdataset , Ġon Ġinitial Ġstudy Ġwe Ġsee Ġa ĠWord ĠError ĠRate Ġof Ġ$\ sim Ġ$ 50 % Ġfor Ġthe ĠAS R ĠAPIs Ġand Ġthis Ġnumber Ġis Ġnot Ġaccurate Ġbecause , Ġ1 ) Ġof Ġthe Ġde - ident ification , Ġ2 ) Ġdis flu encies Ġ( ver bat im ) Ġdifference Ġbetween Ġthe Ġhuman Ġwritten Ġand ĠAS R Ġtranscript , Ġand Ġ3 ) Ġminor Ġalignment Ġdifferences Ġbetween Ġthe Ġaudio Ġand Ġthe Ġground Ġtruth Ġtranscript . </s> During Ġthis Ġevaluation , Ġwe Ġfollowed Ġthe Ġsame Ġpre processing Ġmethods Ġwe Ġused Ġduring Ġtraining . ĠThen , Ġwe Ġauto Ġsegment Ġthe Ġtranscript Ġinto Ġsmall Ġcontiguous Ġsegments Ġsimilar Ġto Ġthe Ġgrounded Ġsentences Ġin Ġthe Ġannotations Ġfor Ġtags Ġextraction . ĠTo Ġsegment Ġthe Ġtranscript , Ġwe Ġfollow Ġa Ġsimple Ġprocedure . ĠFirst , Ġwe Ġdetected Ġall Ġthe Ġmedications Ġin Ġa Ġtranscript Ġusing ĠRx Norm ĠB IB REF 20 Ġvia Ġstring Ġmatching . ĠFor Ġall Ġthe Ġdetected Ġmedications , Ġwe Ġselected Ġ$ 2 Ġ\ le Ġx Ġ\ le Ġ5 $ Ġnearby Ġsentences Ġas Ġthe Ġinput Ġto Ġour Ġmodel . ĠWe Ġincreased Ġ$ x $ Ġiter atively Ġuntil Ġwe Ġencountered Ġa Ġquantity Ġentity ĠâĢĵ Ġdetected Ġusing Ġspa Cy 's Ġentity Ġrecogn izer , Ġand Ġwe Ġset Ġ$ x $ Ġas Ġ2 Ġif Ġwe Ġdid Ġnot Ġdetect Ġany Ġentities Ġin Ġthe Ġrange . </s> We Ġshow Ġthe Ġmodel 's Ġperformance Ġon ĠAS R Ġtranscripts Ġand Ġhuman Ġwritten Ġtranscripts Ġwith Ġautomatic Ġsegment ation , Ġand Ġhuman Ġwritten Ġtranscripts Ġwith Ġhuman Ġ( defined ) Ġsegment ation , Ġin ĠTable ĠT AB REF 18 . ĠThe Ġnumber Ġof Ġrecognized Ġmedications Ġin ĠIBM - ST T Ġis Ġonly Ġ95 Ġcompared Ġto Ġ7 25 Ġ( human Ġwritten ), Ġwe Ġmainly Ġconsider Ġthe Ġmodels ' Ġperformance Ġon ĠG - ST T 's Ġtranscripts Ġ( 343 ). </s> On Ġthe ĠMed ications Ġthat Ġwere Ġrecognized Ġcorrectly , Ġthe Ġmodels Ġcan Ġperform Ġdec ently Ġon ĠAS R Ġtranscripts Ġin Ġcomparison Ġto Ġhuman Ġtranscripts Ġ( within Ġ5 Ġpoints ĠR OU GE - 1 ĠF 1 Ġfor Ġboth Ġtasks , Ġrefer Ġto ĠTable ĠT AB REF 18 ). ĠThis Ġshows Ġthat Ġthe Ġmodels Ġare Ġrobust Ġto ĠAS R Ġvariations Ġdiscussed Ġabove . ĠThe Ġlower Ġperformance Ġcompared Ġto Ġhuman Ġtranscripts Ġis Ġmainly Ġdue Ġto Ġincorrect Ġrecognition Ġof ĠDos age Ġand Ġother Ġmedications Ġin Ġthe Ġsame Ġsegments Ġ( changing Ġthe Ġmeaning Ġof Ġthe Ġtext ). ĠBy Ġcomparing Ġthe Ġperformance Ġof Ġthe Ġmodel Ġon Ġthe Ġhuman Ġwritten Ġtranscripts Ġwith Ġhuman Ġ( defined ) Ġsegment ation Ġand Ġthe Ġsame Ġwith Ġauto Ġsegment ation , Ġwe Ġsee Ġa Ġ10 Ġpoint Ġdrop Ġin ĠDos age Ġand Ġ6 Ġpoint Ġdrop Ġin ĠFrequency Ġextraction Ġtasks . ĠThis Ġpoints Ġout Ġthe Ġneed Ġfor Ġmore Ġsophisticated Ġsegment ation Ġalgorithms . </s> With ĠG - ST T , Ġour Ġbest Ġmodel Ġobtained ĠR OU GE - 1 ĠF 1 Ġof Ġ71 . 75 Ġ( which Ġequals Ġto Ġpercentage Ġof Ġtimes Ġdosage Ġis Ġcorrect Ġin Ġthis Ġcase ) Ġfor ĠDos age Ġextraction Ġand Ġ40 . 13 Ġfor ĠFrequency Ġextraction Ġtasks . ĠTo Ġmeasure Ġthe Ġpercentage Ġof Ġtimes Ġthe Ġcorrect Ġfrequency Ġwas Ġextracted Ġby Ġthe Ġmodel , Ġwe Ġqual itatively Ġcompared Ġthe Ġextracted Ġand Ġpredicted Ġfrequency . ĠWe Ġfind Ġthat Ġfor Ġ73 . 58 % Ġof Ġthe Ġmedications Ġthe Ġmodel Ġcan Ġfind Ġthe Ġcorrect Ġfrequency Ġfrom Ġthe Ġtranscripts . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġexplore Ġthe ĠMed ication ĠReg imen Ġ( MR ) Ġextraction Ġtask Ġof Ġextracting Ġdosage Ġand Ġfrequency Ġfor Ġthe Ġmedications Ġmentioned Ġin Ġa Ġdoctor - patient Ġconversation Ġtranscript . ĠWe Ġexplore Ġdifferent Ġvariations Ġof Ġabstract ive ĠQ A Ġmodels Ġand Ġnew Ġarchitecture Ġin Ġthe Ġintersection Ġof ĠQ A Ġand ĠIE Ġframeworks Ġand Ġprovide Ġa Ġcomparative Ġperformance Ġanalysis Ġof Ġthe Ġmethods Ġalong Ġwith Ġother Ġtechniques Ġlike Ġpret raining Ġto Ġimprove Ġthe Ġoverall Ġperformance . ĠFinally , Ġwe Ġdemonstrate Ġthe Ġperformance Ġof Ġour Ġbest - performing Ġmodels Ġby Ġautomatically Ġextracting ĠMR Ġtags Ġfrom Ġspontaneous Ġdoctor - patient Ġconversations Ġ( using Ġcommercially Ġavailable ĠAS R ). ĠOur Ġbest Ġmodel Ġcan Ġcorrectly Ġextract Ġthe Ġdosage Ġfor Ġ71 . 75 % Ġ( interpret ation Ġof ĠR OU GE - 1 Ġscore ) Ġand Ġfrequency Ġfor Ġ73 . 58 % Ġ( on Ġqualitative Ġevaluation ) Ġof Ġthe Ġmedications Ġdiscussed Ġin Ġthe Ġtranscripts Ġgenerated Ġusing ĠGoogle ĠSpeech - To - Text . ĠIn Ġsummary , Ġwe Ġdemonstrate Ġthat Ġresearch Ġon ĠN LP Ġcan Ġbe Ġtranslated Ġinto Ġreal - world Ġclinical Ġsettings Ġto Ġrealize Ġits Ġbenefits Ġfor Ġboth Ġdoctors Ġand Ġpatients . </s> Using ĠAS R Ġtranscripts Ġin Ġour Ġtraining Ġprocess Ġto Ġimprove Ġour Ġperformance Ġon Ġboth Ġthe Ġtasks Ġand Ġextending Ġthe Ġmedication Ġregimen Ġextraction Ġnetwork Ġto Ġextract Ġother Ġimportant Ġmedical Ġinformation Ġcan Ġbe Ġinteresting Ġlines Ġof Ġfuture Ġwork . </s> Acknowled gements </s> We Ġthank : ĠUniversity Ġof ĠPittsburgh ĠMedical ĠCenter Ġ( U PM C ), Ġand ĠA bridge ĠAI ĠInc . Ġfor Ġproviding Ġaccess Ġto Ġthe Ġde - identified Ġdata Ġcorpus ; ĠDr . ĠShiv dev ĠRao , Ġa Ġfaculty Ġmember Ġand Ġpracticing Ġcard iologist Ġin ĠU PM C 's ĠHeart Ġand ĠV ascular ĠInstitute Ġand ĠProf . ĠFlor ian ĠMet ze , ĠAssociate ĠResearch ĠProfessor , ĠCarnegie ĠMellon ĠUniversity Ġfor Ġhelpful Ġdiscussions ; ĠBen ĠSch loss , ĠSteven ĠColeman , Ġand ĠDeborah ĠOs ak ue Ġfor Ġdata Ġbusiness Ġdevelopment Ġand Ġannotation Ġmanagement .