353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers), we build templates and expression pools using linguistic analysis
EXTRACTIVE
The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research. The speakers are 38 to 88 years old, equally distributed across gender, and comprise a range of ethnic groups (55% Chinese, 17% Malay, 14% Indian, 3% Eurasian, and 11% unspecified). The conversations cover 11 topics (e.g., medication compliance, symptom checking, education, greeting) and 9 symptoms (e.g., chest pain, cough) and amount to 41 hours. We divide the construction of data simulation into two stages. In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification. In Section SECREF20 , we present our proposed framework for generating simulated training data. The templates and framework are verified for logical correctness and clinical soundness.
1903.03530
Problem ĠStatement </s> Sp oken Ġconversations Ġstill Ġremain Ġthe Ġmost Ġnatural Ġand Ġeffort less Ġmeans Ġof Ġhuman Ġcommunication . ĠThus Ġa Ġlot Ġof Ġvaluable Ġinformation Ġis Ġconveyed Ġand Ġexchanged Ġin Ġsuch Ġan Ġun struct ured Ġform . ĠIn Ġtele health Ġsettings , Ġnurses Ġmight Ġcall Ġdischarged Ġpatients Ġwho Ġhave Ġreturned Ġhome Ġto Ġcontinue Ġto Ġmonitor Ġtheir Ġhealth Ġstatus . ĠHuman Ġlanguage Ġtechnology Ġthat Ġcan Ġefficiently Ġand Ġeffectively Ġextract Ġkey Ġinformation Ġfrom Ġsuch Ġconversations Ġis Ġclinically Ġuseful , Ġas Ġit Ġcan Ġhelp Ġstream line Ġworkflow Ġprocesses Ġand Ġdigitally Ġdocument Ġpatient Ġmedical Ġinformation Ġto Ġincrease Ġstaff Ġproductivity . ĠIn Ġthis Ġwork , Ġwe Ġdesign Ġand Ġprototype Ġa Ġdialogue Ġcomprehension Ġsystem Ġin Ġthe Ġquestion - ans w ering Ġmanner , Ġwhich Ġis Ġable Ġto Ġcomprehend Ġspoken Ġconversations Ġbetween Ġnurses Ġand Ġpatients Ġto Ġextract Ġclinical Ġinformation . </s> Mot ivation Ġof ĠApproach </s> Machine Ġcomprehension Ġof Ġwritten Ġpassages Ġhas Ġmade Ġtremendous Ġprogress Ġrecently . ĠLarge Ġquantities Ġof Ġsupervised Ġtraining Ġdata Ġfor Ġreading Ġcomprehension Ġ( e . g . ĠS Qu AD ĠB IB REF 0 Ġ), Ġthe Ġwide Ġadoption Ġand Ġintense Ġexperimentation Ġof Ġneural Ġmodeling ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġand Ġthe Ġadvancements Ġin Ġvector Ġrepresentations Ġof Ġword Ġembed d ings ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġall Ġcontribute Ġsignificantly Ġto Ġthe Ġachievements Ġobtained Ġso Ġfar . ĠThe Ġfirst Ġfactor , Ġthe Ġavailability Ġof Ġlarge Ġscale Ġdatasets , Ġem powers Ġthe Ġlatter Ġtwo Ġfactors . ĠTo Ġdate , Ġthere Ġis Ġstill Ġvery Ġlimited Ġwell - annot ated Ġlarge - scale Ġdata Ġsuitable Ġfor Ġmodeling Ġhuman - human Ġspoken Ġdialog ues . ĠTherefore , Ġit Ġis Ġnot Ġstraightforward Ġto Ġdirectly Ġport Ġover Ġthe Ġrecent Ġendeavors Ġin Ġreading Ġcomprehension Ġto Ġdialogue Ġcomprehension Ġtasks . </s> In Ġhealthcare , Ġconversation Ġdata Ġis Ġeven Ġscar cer Ġdue Ġto Ġprivacy Ġissues . ĠCrowd - s ourcing Ġis Ġan Ġefficient Ġway Ġto Ġannot ate Ġlarge Ġquantities Ġof Ġdata , Ġbut Ġless Ġsuitable Ġfor Ġhealthcare Ġscenarios , Ġwhere Ġdomain Ġknowledge Ġis Ġrequired Ġto Ġguarantee Ġdata Ġquality . ĠTo Ġdemonstrate Ġthe Ġfeasibility Ġof Ġa Ġdialogue Ġcomprehension Ġsystem Ġused Ġfor Ġextracting Ġkey Ġclinical Ġinformation Ġfrom Ġsymptom Ġmonitoring Ġconversations , Ġwe Ġdeveloped Ġa Ġframework Ġto Ġconstruct Ġa Ġsimulated Ġhuman - human Ġdialogue Ġdataset Ġto Ġboot strap Ġsuch Ġa Ġprototype . ĠSimilar Ġefforts Ġhave Ġbeen Ġconducted Ġfor Ġhuman - machine Ġdialog ues Ġfor Ġrestaurant Ġor Ġmovie Ġreservations ĠB IB REF 5 Ġ. ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġone Ġto Ġdate Ġhas Ġdone Ġso Ġfor Ġhuman - human Ġconversations Ġin Ġhealthcare . </s> Human - human ĠSp oken ĠConvers ations </s> Human - human Ġspoken Ġconversations Ġare Ġa Ġdynamic Ġand Ġinteractive Ġflow Ġof Ġinformation Ġexchange . ĠWhile Ġdeveloping Ġtechnology Ġto Ġcomprehend Ġsuch Ġspoken Ġconversations Ġpresents Ġsimilar Ġtechnical Ġchallenges Ġas Ġmachine Ġcomprehension Ġof Ġwritten Ġpassages ĠB IB REF 6 Ġ, Ġthe Ġchallenges Ġare Ġfurther Ġcomplicated Ġby Ġthe Ġinteractive Ġnature Ġof Ġhuman - human Ġspoken Ġconversations : </s> ( 1 ) ĠZero Ġan aph ora Ġis Ġmore Ġcommon : ĠCo - reference Ġresolution Ġof Ġspoken Ġutter ances Ġfrom Ġmultiple Ġspeakers Ġis Ġneeded . ĠFor Ġexample , Ġin ĠFigure ĠFIG REF 5 Ġ( a ) Ġheadaches , Ġthe Ġpain , Ġit , Ġhead Ġbul ging Ġall Ġrefer Ġto Ġthe Ġpatient 's Ġheadache Ġsymptom , Ġbut Ġthey Ġwere Ġuttered Ġby Ġdifferent Ġspeakers Ġand Ġacross Ġmultiple Ġutter ances Ġand Ġturns . ĠIn Ġaddition , Ġan aph ors Ġare Ġmore Ġlikely Ġto Ġbe Ġomitted Ġ( see ĠFigure ĠFIG REF 5 Ġ( a ) ĠA 4 ) Ġas Ġthis Ġdoes Ġnot Ġaffect Ġthe Ġhuman Ġlistener âĢ Ļ s Ġunderstanding , Ġbut Ġit Ġmight Ġbe Ġchallenging Ġfor Ġcomputational Ġmodels . </s> ( 2 ) ĠThinking Ġaloud Ġmore Ġcommonly Ġoccurs : ĠSince Ġit Ġis Ġmore Ġeffort less Ġto Ġspeak Ġthan Ġto Ġtype , Ġone Ġis Ġmore Ġlikely Ġto Ġreveal Ġher Ġrunning Ġthoughts Ġwhen Ġtalking . ĠIn Ġaddition , Ġone Ġcannot Ġretract Ġwhat Ġhas Ġbeen Ġuttered , Ġwhile Ġin Ġtext Ġcommunications , Ġone Ġis Ġmore Ġlikely Ġto Ġconfirm Ġthe Ġaccuracy Ġof Ġthe Ġinformation Ġin Ġa Ġwritten Ġresponse Ġand Ġrevise Ġif Ġnecessary Ġbefore Ġsending Ġit Ġout . ĠThinking Ġaloud Ġcan Ġlead Ġto Ġself - cont rad iction , Ġrequiring Ġmore Ġcontext Ġto Ġfully Ġunderstand Ġthe Ġdialogue ; Ġe . g ., Ġin ĠA 6 Ġin ĠFigure ĠFIG REF 5 Ġ( a ), Ġthe Ġpatient Ġat Ġfirst Ġsays Ġhe Ġhas Ġnone Ġof Ġthe Ġsymptoms Ġasked , Ġbut Ġlater Ġrev ises Ġhis Ġresponse Ġsaying Ġthat Ġhe Ġdoes Ġget Ġdizz y Ġafter Ġrunning . </s> ( 3 ) ĠTopic Ġdrift Ġis Ġmore Ġcommon Ġand Ġharder Ġto Ġdetect Ġin Ġspoken Ġconversations : ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 5 Ġ( a ) Ġin ĠA 3 , Ġwhere ĠNo Ġis Ġactually Ġreferring Ġto Ġcough Ġin Ġthe Ġprevious Ġquestion , Ġand Ġthen Ġthe Ġtopic Ġis Ġshifted Ġto Ġheadache . ĠIn Ġspoken Ġconversations , Ġutter ances Ġare Ġoften Ġincomplete Ġsentences Ġso Ġtraditional Ġlinguistic Ġfeatures Ġused Ġin Ġwritten Ġpassages Ġsuch Ġas Ġpunct uation Ġmarks Ġindicating Ġsynt actic Ġboundaries Ġor Ġconjunction Ġwords Ġsuggesting Ġdiscourse Ġrelations Ġmight Ġno Ġlonger Ġexist . </s> Dialogue ĠComp rehens ion ĠTask </s> Figure ĠFIG REF 5 Ġ( b ) Ġillustrates Ġthe Ġproposed Ġdialogue Ġcomprehension Ġtask Ġusing Ġa Ġquestion Ġanswering Ġ( Q A ) Ġmodel . ĠThe Ġinput Ġare Ġa Ġmulti - turn Ġsymptom Ġchecking Ġdialogue ĠIN LINE FORM 0 Ġand Ġa Ġquery ĠIN LINE FORM 1 Ġspecifying Ġa Ġsymptom Ġwith Ġone Ġof Ġits Ġattributes ; Ġthe Ġoutput Ġis Ġthe Ġextracted Ġanswer ĠIN LINE FORM 2 Ġfrom Ġthe Ġgiven Ġdialogue . ĠA Ġtraining Ġor Ġtest Ġsample Ġis Ġdefined Ġas ĠIN LINE FORM 3 Ġ. ĠFive Ġattributes , Ġspecifying Ġcertain Ġdetails Ġof Ġclinical Ġsignificance , Ġare Ġdefined Ġto Ġcharacterize Ġthe Ġanswer Ġtypes Ġof ĠIN LINE FORM 4 Ġ: Ġ( 1 ) Ġthe Ġtime Ġthe Ġpatient Ġhas Ġbeen Ġexperiencing Ġthe Ġsymptom , Ġ( 2 ) Ġactivities Ġthat Ġtrigger Ġthe Ġsymptom Ġ( to Ġoccur Ġor Ġworsen ), Ġ( 3 ) Ġthe Ġextent Ġof Ġseriousness , Ġ( 4 ) Ġthe Ġfrequency Ġoccurrence Ġof Ġthe Ġsymptom , Ġand Ġ( 5 ) Ġthe Ġlocation Ġof Ġsymptom . ĠFor Ġeach Ġsymptom / attribute , Ġit Ġcan Ġtake Ġon Ġdifferent Ġlinguistic Ġexpressions , Ġdefined Ġas Ġentities . ĠNote Ġthat Ġif Ġthe Ġquer ied Ġsymptom Ġor Ġattribute Ġis Ġnot Ġmentioned Ġin Ġthe Ġdialogue , Ġthe Ġground truth Ġoutput Ġis ĠâĢ ľ No ĠAnswer âĢ Ŀ , Ġas Ġin ĠB IB REF 6 Ġ. </s> Reading ĠComp rehens ion </s> Large - scale Ġreading Ġcomprehension Ġtasks Ġlike ĠS Qu AD ĠB IB REF 0 Ġand ĠMAR CO ĠB IB REF 7 Ġprovide Ġquestion - answer Ġpairs Ġfrom Ġa Ġvast Ġrange Ġof Ġwritten Ġpassages , Ġcovering Ġdifferent Ġkinds Ġof Ġfactual Ġanswers Ġinvolving Ġentities Ġsuch Ġas Ġlocation Ġand Ġnumerical Ġvalues . ĠFurthermore , ĠHot pot Q A ĠB IB REF 8 Ġrequires Ġmulti - step Ġinference Ġand Ġprovides Ġnumerous Ġanswer Ġtypes . ĠCo Q A ĠB IB REF 9 Ġand ĠQu AC ĠB IB REF 10 Ġare Ġdesigned Ġto Ġmimic Ġmulti - turn Ġinformation - seeking Ġdiscussions Ġof Ġthe Ġgiven Ġmaterial . ĠIn Ġthese Ġtasks , Ġcontextual Ġreasoning Ġlike Ġcore ference Ġresolution Ġis Ġnecessary Ġto Ġgrasp Ġrich Ġlinguistic Ġpatterns , Ġencouraging Ġsemantic Ġmodeling Ġbeyond Ġnaive Ġlex ical Ġmatching . ĠNeural Ġnetworks Ġcontribute Ġto Ġimpressive Ġprogress Ġin Ġsemantic Ġmodeling : Ġdistribution al Ġsemantic Ġword Ġembed d ings ĠB IB REF 3 Ġ, Ġcontextual Ġsequence Ġencoding ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġand Ġthe Ġattention Ġmechanism ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġare Ġwidely Ġadopted Ġin Ġstate - of - the - art Ġcomprehension Ġmodels ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. </s> While Ġlanguage Ġunderstanding Ġtasks Ġin Ġdialogue Ġsuch Ġas Ġdomain Ġidentification ĠB IB REF 15 Ġ, Ġslot Ġfilling ĠB IB REF 16 Ġand Ġuser Ġintent Ġdetection ĠB IB REF 17 Ġhave Ġattracted Ġmuch Ġresearch Ġinterest , Ġwork Ġin Ġdialogue Ġcomprehension Ġis Ġstill Ġlimited , Ġif Ġany . ĠIt Ġis Ġlabor - intensive Ġand Ġtime - consuming Ġto Ġobtain Ġa Ġcritical Ġmass Ġof Ġannot ated Ġconversation Ġdata Ġfor Ġcomputational Ġmodeling . ĠSome Ġpropose Ġto Ġcollect Ġtext Ġdata Ġfrom Ġhuman - machine Ġor Ġmachine - machine Ġdialog ues ĠB IB REF 18 Ġ, ĠB IB REF 5 Ġ. ĠIn Ġsuch Ġcases , Ġas Ġhuman Ġspeakers Ġare Ġaware Ġof Ġcurrent Ġlimitations Ġof Ġdialogue Ġsystems Ġor Ġdue Ġto Ġpre - defined Ġassumptions Ġof Ġuser Ġsim ulators , Ġthere Ġare Ġfewer Ġcases Ġof Ġzero Ġan aph ora , Ġthinking Ġaloud , Ġand Ġtopic Ġdrift , Ġwhich Ġoccur Ġmore Ġoften Ġin Ġhuman - human Ġspoken Ġinteractions . </s> N LP Ġfor ĠHealthcare </s> There Ġis Ġemerging Ġinterest Ġin Ġresearch Ġand Ġdevelopment Ġactivities Ġat Ġthe Ġintersection Ġof Ġmachine Ġlearning Ġand Ġhealthcare Ġ, Ġof Ġwhich Ġmuch Ġof Ġthe ĠN LP Ġrelated Ġwork Ġare Ġcentered Ġaround Ġsocial Ġmedia Ġor Ġonline Ġforums Ġ( e . g ., ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ), Ġpartially Ġdue Ġto Ġthe Ġworld Ġwide Ġweb Ġas Ġa Ġreadily Ġavailable Ġsource Ġof Ġinformation . ĠOther Ġwork Ġin Ġthis Ġarea Ġuses Ġpublic Ġdata Ġsources Ġsuch Ġas ĠM IM IC Ġin Ġelectronic Ġhealth Ġrecords : Ġtext Ġclassification Ġapproaches Ġhave Ġbeen Ġapplied Ġto Ġanalyze Ġun struct ured Ġclinical Ġnotes Ġfor ĠI CD Ġcode Ġassignment ĠB IB REF 21 Ġand Ġautomatic Ġintensive Ġemergency Ġprediction ĠB IB REF 22 Ġ. ĠSequence - to - sequence Ġtextual Ġgeneration Ġhas Ġbeen Ġused Ġfor Ġreadable Ġnotes Ġbased Ġon Ġmedical Ġand Ġdemographic Ġrecordings ĠB IB REF 23 Ġ. ĠFor Ġmental Ġhealth , Ġthere Ġhas Ġbeen Ġmore Ġfocus Ġon Ġanalyzing Ġdialog ues . ĠFor Ġexample , Ġsequential Ġmodeling Ġof Ġaudio Ġand Ġtext Ġhave Ġhelped Ġdetect Ġdepression Ġfrom Ġhuman - machine Ġinterviews ĠB IB REF 24 Ġ. ĠHowever , Ġfew Ġstudies Ġhave Ġexamined Ġhuman - human Ġspoken Ġconversations Ġin Ġhealthcare Ġsettings . </s> Data ĠPrepar ation </s> We Ġused Ġrecordings Ġof Ġnurse - in iti ated Ġtelephone Ġconversations Ġfor Ġcongest ive Ġheart Ġfailure Ġpatients Ġundergoing Ġtele monitor ing , Ġpost - dis charge Ġfrom Ġthe Ġhospital . ĠThe Ġclinical Ġdata Ġwas Ġacquired Ġby Ġthe ĠHealth ĠManagement ĠUnit Ġat ĠChang i ĠGeneral ĠHospital . ĠThis Ġresearch Ġstudy Ġwas Ġapproved Ġby Ġthe ĠSing Health ĠCentral ised ĠInst itutional ĠReview ĠBoard Ġ( Prot ocol Ġ155 656 15 15 ). ĠThe Ġpatients Ġwere Ġrecruited Ġduring Ġ2014 - 2016 Ġas Ġpart Ġof Ġtheir Ġroutine Ġcare Ġdelivery , Ġand Ġenrolled Ġinto Ġthe Ġtele monitor ing Ġhealth Ġmanagement Ġprogram Ġwith Ġconsent Ġfor Ġuse Ġof Ġanonym ized Ġversions Ġof Ġtheir Ġdata Ġfor Ġresearch . </s> The Ġdataset Ġcomprises Ġa Ġtotal Ġof Ġ353 Ġconversations Ġfrom Ġ40 Ġspeakers Ġ( 11 Ġnurses , Ġ16 Ġpatients , Ġand Ġ13 Ġcaregivers ) Ġwith Ġconsent Ġto Ġthe Ġuse Ġof Ġanonym ized Ġdata Ġfor Ġresearch . ĠThe Ġspeakers Ġare Ġ38 Ġto Ġ88 Ġyears Ġold , Ġequally Ġdistributed Ġacross Ġgender , Ġand Ġcomprise Ġa Ġrange Ġof Ġethnic Ġgroups Ġ( 55 % ĠChinese , Ġ17 % ĠMal ay , Ġ14 % ĠIndian , Ġ3 % ĠEuras ian , Ġand Ġ11 % Ġunspecified ). ĠThe Ġconversations Ġcover Ġ11 Ġtopics Ġ( e . g ., Ġmedication Ġcompliance , Ġsymptom Ġchecking , Ġeducation , Ġgreeting ) Ġand Ġ9 Ġsymptoms Ġ( e . g ., Ġchest Ġpain , Ġcough ) Ġand Ġamount Ġto Ġ41 Ġhours . </s> Data Ġpre processing Ġand Ġanonym ization Ġwere Ġperformed Ġby Ġa Ġdata Ġpreparation Ġteam , Ġseparate Ġfrom Ġthe Ġdata Ġanalysis Ġteam Ġto Ġmaintain Ġdata Ġconfidentiality . ĠThe Ġdata Ġpreparation Ġteam Ġfollowed Ġstandard Ġspeech Ġrecognition Ġtranscription Ġguidelines , Ġwhere Ġwords Ġare Ġtransc ribed Ġver bat im Ġto Ġinclude Ġfalse Ġstarts , Ġdis flu encies , Ġmis pron unci ations , Ġand Ġprivate Ġself - talk . ĠConf idential Ġinformation Ġwere Ġmarked Ġand Ġclipped Ġoff Ġfrom Ġthe Ġaudio Ġand Ġtransc ribed Ġwith Ġpred efined Ġtags Ġin Ġthe Ġannotation . ĠConversation Ġtopics Ġand Ġclinical Ġsymptoms Ġwere Ġalso Ġannot ated Ġand Ġclinically Ġvalidated Ġby Ġcertified Ġtele health Ġnurses . </s> L ingu istic ĠCharacter ization Ġon ĠSeed ĠData </s> To Ġanalyze Ġthe Ġlinguistic Ġstructure Ġof Ġthe Ġinquiry - response Ġpairs Ġin Ġthe Ġentire Ġ41 - hour Ġdataset , Ġwe Ġrandomly Ġsampled Ġa Ġseed Ġdataset Ġconsisting Ġof Ġ1 , 200 Ġturns Ġand Ġmanually Ġcategorized Ġthem Ġto Ġdifferent Ġtypes , Ġwhich Ġare Ġsummarized Ġin ĠTable ĠT AB REF 14 Ġalong Ġwith Ġthe Ġcorresponding Ġoccurrence Ġfrequency Ġstatistics . ĠNote Ġthat Ġeach Ġgiven Ġutter ance Ġcould Ġbe Ġcategorized Ġto Ġmore Ġthan Ġone Ġtype . ĠWe Ġelaborate Ġon Ġeach Ġutter ance Ġtype Ġbelow . </s> Open - ended ĠInquiry : ĠInqu iries Ġabout Ġgeneral Ġwell - being Ġor Ġa Ġparticular Ġsymptom ; Ġe . g ., ĠâĢ ľ How Ġare Ġyou Ġfeeling ? âĢ Ŀ Ġand ĠâĢ ľ Do Ġyou Ġcough ? âĢ Ŀ </s> Detailed ĠInquiry : ĠInqu iries Ġwith Ġspecific Ġdetails Ġthat Ġprompt Ġyes / no Ġanswers Ġor Ġclar ifications ; Ġe . g ., ĠâĢ ľ Do Ġyou Ġcough Ġat Ġnight ? âĢ Ŀ </s> Multi - Int ent ĠInquiry : ĠInqu iring Ġmore Ġthan Ġone Ġsymptom Ġin Ġa Ġquestion ; Ġe . g ., ĠâĢ ľ Any Ġcough , Ġchest Ġpain , Ġor Ġheadache ? âĢ Ŀ </s> Rec on f irmation ĠInquiry : ĠThe Ġnurse Ġrecon f irms Ġparticular Ġdetails ; Ġe . g ., ĠâĢ ľ Really ? ĠAt Ġnight ? âĢ Ŀ Ġand ĠâĢ ľ Ser ious Ġor Ġmild ? âĢ Ŀ . ĠThis Ġcase Ġis Ġusually Ġrelated Ġto Ġexplicit Ġor Ġimplicit Ġcore f eren cing . </s> In qu iry Ġwith ĠTrans itional ĠCla uses : ĠDuring Ġspoken Ġconversations , Ġone Ġmight Ġrepeat Ġwhat Ġthe Ġother Ġparty Ġsaid , Ġbut Ġit Ġis Ġunrelated Ġto Ġthe Ġmain Ġclause Ġof Ġthe Ġquestion . ĠThis Ġis Ġusually Ġdue Ġto Ġprivate Ġself - talk Ġwhile Ġthinking Ġaloud , Ġand Ġsuch Ġutter ances Ġform Ġa Ġtransitional Ġclause Ġbefore Ġthe Ġspeaker Ġstarts Ġa Ġnew Ġtopic ; Ġe . g ., ĠâĢ ľ Chest Ġpain ... Ġno Ġchest Ġpain , ĠI Ġsee ... Ġany Ġcough ? âĢ Ŀ . </s> Yes / No ĠResponse : ĠYes / No Ġresponses Ġseem Ġstraightforward , Ġbut Ġsometimes Ġlead Ġto Ġmisunderstanding Ġif Ġone Ġdoes Ġnot Ġinterpret Ġthe Ġcontext Ġappropriately . ĠOne Ġcase Ġis Ġtag Ġquestions : ĠA : âĢ ľ You Ġdon 't Ġcough Ġat Ġnight , Ġdo Ġyou ? âĢ Ŀ ĠB : ` Yes , Ġyes âĢ Ŀ ĠA : âĢ ľ c ough Ġat Ġnight ? âĢ Ŀ ĠB : âĢ ľ No , Ġno Ġcough âĢ Ŀ . ĠUsually Ġwhen Ġthe Ġanswer Ġis Ġunclear , Ġclar ifying Ġinquiries Ġwill Ġbe Ġasked Ġfor Ġrecon f irmation Ġpurposes . </s> Detailed ĠResponse : ĠRespons es Ġthat Ġcontain Ġspecific Ġinformation Ġof Ġone Ġsymptom , Ġlike ĠâĢ ľ I Ġfelt Ġtight ness Ġin Ġmy Ġchest âĢ Ŀ . </s> Response Ġwith ĠRevision : ĠRevision Ġis Ġinf requent Ġbut Ġcan Ġaffect Ġcomprehension Ġsignificantly . ĠOne Ġcause Ġis Ġthinking Ġaloud Ġso Ġa Ġlater Ġresponse Ġoverr ules Ġthe Ġprevious Ġone ; Ġe . g ., ĠâĢ ľ No Ġdizz iness , Ġoh Ġwait ... Ġlast Ġweek ĠI Ġfelt Ġa Ġbit Ġdizz y Ġwhen Ġbiking âĢ Ŀ . </s> Response Ġwith ĠTopic ĠDrift : ĠWhen Ġa Ġsymptom / topic Ġlike Ġheadache Ġis Ġinquired , Ġthe Ġresponse Ġmight Ġbe : ĠâĢ ľ Only Ġsome Ġchest Ġpain Ġat Ġnight âĢ Ŀ , Ġnot Ġreferring Ġto Ġthe Ġoriginal Ġsymptom Ġ( head ache ) Ġat Ġall . </s> Response Ġwith ĠTrans itional ĠCla uses : ĠRepe ating Ġsome Ġof Ġthe Ġprevious Ġcontent , Ġbut Ġoften Ġunrelated Ġto Ġcritical Ġclinical Ġinformation Ġand Ġusually Ġfollowed Ġby Ġtopic Ġdrift . ĠFor Ġexample , ĠâĢ ľ S well ing ... Ġswelling ... ĠI Ġdon 't Ġcough Ġat Ġnight âĢ Ŀ . </s> Sim ulating ĠSym pt om ĠMonitoring ĠDat as et Ġfor ĠTraining </s> We Ġdivide Ġthe Ġconstruction Ġof Ġdata Ġsimulation Ġinto Ġtwo Ġstages . ĠIn ĠSection ĠSEC REF 16 Ġ, Ġwe Ġbuild Ġtemplates Ġand Ġexpression Ġpools Ġusing Ġlinguistic Ġanalysis Ġfollowed Ġby Ġmanual Ġverification . ĠIn ĠSection ĠSEC REF 20 Ġ, Ġwe Ġpresent Ġour Ġproposed Ġframework Ġfor Ġgenerating Ġsimulated Ġtraining Ġdata . ĠThe Ġtemplates Ġand Ġframework Ġare Ġverified Ġfor Ġlogical Ġcorrectness Ġand Ġclinical Ġsound ness . </s> Template ĠConstruction </s> Each Ġutter ance Ġin Ġthe Ġseed Ġdata Ġis Ġcategorized Ġaccording Ġto ĠTable ĠT AB REF 14 Ġand Ġthen Ġabstract ed Ġinto Ġtemplates Ġby Ġreplacing Ġentity Ġphrases Ġlike Ġcough Ġand Ġoften Ġwith Ġrespective Ġplace holders ĠâĢ ľ # sym pt om # âĢ Ŀ Ġand ĠâĢ ľ # frequency # âĢ Ŀ . ĠThe Ġtemplates Ġare Ġrefined Ġthrough Ġverifying Ġlogical Ġcorrectness Ġand Ġinjecting Ġexpression Ġdiversity Ġby Ġlingu istically Ġtrained Ġresearchers . ĠAs Ġthese Ġreplacements Ġdo Ġnot Ġalter Ġthe Ġsynt actic Ġstructure , Ġwe Ġinterchange Ġsuch Ġplace holders Ġwith Ġvarious Ġverbal Ġexpressions Ġto Ġenlarge Ġthe Ġsimulated Ġtraining Ġset Ġin ĠSection ĠSEC REF 20 Ġ. ĠClinical Ġvalidation Ġwas Ġalso Ġconducted Ġby Ġcertified Ġtele health Ġnurses . </s> For Ġthe Ġ9 Ġsymptoms Ġ( e . g . Ġchest Ġpain , Ġcough ) Ġand Ġ5 Ġattributes Ġ( e . g ., Ġextent , Ġfrequency ), Ġwe Ġcollect Ġvarious Ġexpressions Ġfrom Ġthe Ġseed Ġdata , Ġand Ġexpand Ġthem Ġthrough Ġsyn onym Ġreplacement . ĠSome Ġattributes Ġare Ġunique Ġto Ġa Ġparticular Ġsymptom ; Ġe . g ., ĠâĢ ľ left Ġleg âĢ Ŀ Ġin Ġ# location # Ġis Ġonly Ġsuitable Ġto Ġdescribe Ġthe Ġsymptom Ġswelling , Ġbut Ġnot Ġthe Ġsymptom Ġheadache . ĠTherefore , Ġwe Ġonly Ġreuse Ġgeneral Ġexpressions Ġlike ĠâĢ ľ s light âĢ Ŀ Ġin Ġ# ext ent # Ġacross Ġdifferent Ġsymptoms Ġto Ġdivers ify Ġlinguistic Ġexpressions . </s> Two Ġlingu istically Ġtrained Ġresearchers Ġconstructed Ġexpression Ġpools Ġfor Ġeach Ġsymptom Ġand Ġeach Ġattribute Ġto Ġaccount Ġfor Ġdifferent Ġtypes Ġof Ġparaph r asing Ġand Ġdescriptions . ĠThese Ġexpression Ġpools Ġare Ġused Ġin ĠSection ĠSEC REF 20 Ġ( c ). </s> Sim ulated ĠData ĠGeneration ĠFramework </s> Figure ĠFIG REF 15 Ġshows Ġthe Ġfive Ġsteps Ġwe Ġuse Ġto Ġgenerate Ġmulti - turn Ġsymptom Ġmonitoring Ġdialogue Ġsamples . </s> ( a ) ĠTopic ĠSelection : ĠWhile Ġnurses Ġmight Ġprefer Ġto Ġinquire Ġthe Ġsymptoms Ġin Ġdifferent Ġorders Ġdepending Ġon Ġthe Ġpatient 's Ġhistory , Ġour Ġpreliminary Ġanalysis Ġshows Ġthat Ġmodeling Ġresults Ġdo Ġnot Ġdiffer Ġnoticeably Ġif Ġtopics Ġare Ġof Ġequal Ġprior Ġprobabilities . ĠThus Ġwe Ġadopt Ġthis Ġassumption Ġfor Ġsimplicity . </s> ( b ) ĠTemplate ĠSelection : ĠFor Ġeach Ġselected Ġtopic , Ġone Ġinquiry Ġtemplate Ġand Ġone Ġresponse Ġtemplate Ġare Ġrandomly Ġchosen Ġto Ġcompose Ġa Ġturn . ĠTo Ġminimize Ġadverse Ġeffects Ġof Ġunder fitting , Ġwe Ġredistributed Ġthe Ġfrequency Ġdistribution Ġin ĠTable ĠT AB REF 14 Ġ: ĠFor Ġutter ance Ġtypes Ġthat Ġare Ġbelow Ġ15 %, Ġwe Ġboosted Ġthem Ġto Ġ15 %, Ġand Ġthe Ġoverall Ġrelative Ġdistribution Ġranking Ġis Ġbalanced Ġand Ġconsistent Ġwith ĠTable ĠT AB REF 14 Ġ. </s> ( c ) ĠEn rich ing ĠL ingu istic ĠExpress ions : ĠThe Ġplace holders Ġin Ġthe Ġselected Ġtemplates Ġare Ġsubstituted Ġwith Ġdiverse Ġexpressions Ġfrom Ġthe Ġexpression Ġpools Ġin ĠSection ĠUID 19 Ġto Ġcharacterize Ġthe Ġsymptoms Ġand Ġtheir Ġcorresponding Ġattributes . </s> ( d ) ĠMulti - Turn ĠDialogue ĠState ĠTracking : ĠA Ġgreedy Ġalgorithm Ġis Ġapplied Ġto Ġcomplete Ġconversations . ĠA ĠâĢ ľ com pleted Ġsymptoms âĢ Ŀ Ġlist Ġand Ġa ĠâĢ ľ to - do Ġsymptoms âĢ Ŀ Ġlist Ġare Ġused Ġfor Ġsymptom Ġtopic Ġtracking . ĠWe Ġalso Ġtrack Ġthe ĠâĢ ľ com pleted Ġattributes " Ġand ĠâĢ ľ to - do Ġattributes ". ĠFor Ġeach Ġsymptom , Ġall Ġrelated Ġattributes Ġare Ġiter ated . ĠA Ġdialogue Ġends Ġonly Ġwhen Ġall Ġpossible Ġentities Ġare Ġexhausted , Ġgenerating Ġa Ġmulti - turn Ġdialogue Ġsample , Ġwhich Ġencourages Ġthe Ġmodel Ġto Ġlearn Ġfrom Ġthe Ġentire Ġdiscussion Ġflow Ġrather Ġthan Ġa Ġsingle Ġturn Ġto Ġcomprehend Ġcontextual Ġdependency . ĠThe Ġaverage Ġlength Ġof Ġa Ġsimulated Ġdialogue Ġis Ġ184 Ġwords , Ġwhich Ġhappens Ġto Ġbe Ġtwice Ġas Ġlong Ġas Ġan Ġaverage Ġdialogue Ġfrom Ġthe Ġreal - world Ġevaluation Ġset . ĠMoreover , Ġto Ġmodel Ġthe Ġroles Ġof Ġthe Ġrespondents , Ġwe Ġset Ġthe Ġratio Ġbetween Ġpatients Ġand Ġcaregivers Ġto Ġ2 : 1 ; Ġthis Ġstatistic Ġis Ġinspired Ġby Ġthe Ġreal Ġscenarios Ġin Ġthe Ġseed Ġdataset . ĠFor Ġboth Ġthe Ġcaregivers Ġand Ġpatients , Ġwe Ġassume Ġequal Ġprobability Ġof Ġboth Ġgenders . ĠThe Ġcorresponding Ġpronouns Ġin Ġthe Ġconversations Ġare Ġthus Ġdetermined Ġby Ġthe Ġrole Ġand Ġgender Ġof Ġthese Ġsettings . </s> ( e ) ĠMulti - Turn ĠSample ĠAn notation : ĠFor Ġeach Ġmulti - turn Ġdialogue , Ġa Ġquery Ġis Ġspecified Ġby Ġa Ġsymptom Ġand Ġan Ġattribute . ĠThe Ġground truth Ġoutput Ġof Ġthe ĠQ A Ġsystem Ġis Ġautomatically Ġlabeled Ġbased Ġon Ġthe Ġtemplate Ġgeneration Ġrules , Ġbut Ġalso Ġmanually Ġverified Ġto Ġensure Ġannotation Ġquality . ĠMoreover , Ġwe Ġadopt Ġthe Ġun answer able Ġdesign Ġin ĠB IB REF 6 Ġ: Ġwhen Ġthe Ġpatient Ġdoes Ġnot Ġmention Ġa Ġparticular Ġsymptom , Ġthe Ġanswer Ġis Ġdefined Ġas ĠâĢ ľ No ĠAnswer âĢ Ŀ . ĠThis Ġprocess Ġis Ġrepeated Ġuntil Ġall Ġlogical Ġperm utations Ġof Ġsymptoms Ġand Ġattributes Ġare Ġexhausted . </s> Exper iments </s> Model ĠDesign </s> We Ġimplemented Ġan Ġestablished Ġmodel Ġin Ġreading Ġcomprehension , Ġa Ġbi - direction al Ġattention Ġpointer Ġnetwork ĠB IB REF 1 Ġ, Ġand Ġequipped Ġit Ġwith Ġan Ġanswer able Ġclass ifier , Ġas Ġdepicted Ġin ĠFigure ĠFIG REF 21 Ġ. ĠFirst , Ġtokens Ġin Ġthe Ġgiven Ġdialogue ĠIN LINE FORM 0 Ġand Ġquery ĠIN LINE FORM 1 Ġare Ġconverted Ġinto Ġembed ding Ġvectors . ĠThen Ġthe Ġdialogue Ġembed d ings Ġare Ġfed Ġto Ġa Ġbi - direction al ĠL ST M Ġencoding Ġlayer , Ġgenerating Ġa Ġsequence Ġof Ġcontextual Ġhidden Ġstates . ĠNext , Ġthe Ġhidden Ġstates Ġand Ġquery Ġembed d ings Ġare Ġprocessed Ġby Ġa Ġbi - direction al Ġattention Ġlayer , Ġf using Ġattention Ġinformation Ġin Ġboth Ġcontext - to - query Ġand Ġquery - to - context Ġdirections . ĠThe Ġfollowing Ġtwo Ġbi - direction al ĠL ST M Ġmodeling Ġlayers Ġread Ġthe Ġcontextual Ġsequence Ġwith Ġattention . ĠFinally , Ġtwo Ġrespective Ġlinear Ġlayers Ġwith Ġsoft max Ġfunctions Ġare Ġused Ġto Ġestimate Ġtoken ĠIN LINE FORM 2 Ġ' s ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġprobability Ġof Ġthe Ġanswer Ġspan ĠIN LINE FORM 5 Ġ. </s> In Ġaddition , Ġwe Ġadd Ġa Ġspecial Ġtag ĠâĢ ľ [ SE Q ] âĢ Ŀ Ġat Ġthe Ġhead Ġof ĠIN LINE FORM 0 Ġto Ġaccount Ġfor Ġthe Ġcase Ġof ĠâĢ ľ No Ġanswer âĢ Ŀ ĠB IB REF 4 Ġand Ġadopt Ġan Ġanswer able Ġclass ifier Ġas Ġin ĠB IB REF 25 Ġ. ĠMore Ġspecifically , Ġwhen Ġthe Ġquer ied Ġsymptom Ġor Ġattribute Ġis Ġnot Ġmentioned Ġin Ġthe Ġdialogue , Ġthe Ġanswer Ġspan Ġshould Ġpoint Ġto Ġthe Ġtag ĠâĢ ľ [ SE Q ] âĢ Ŀ Ġand Ġanswer able Ġprobability Ġshould Ġbe Ġpredicted Ġas Ġ0 . </s> Im plementation ĠDetails </s> The Ġmodel Ġwas Ġtrained Ġvia Ġgradient Ġback prop ag ation Ġwith Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġof Ġanswer Ġspan Ġprediction Ġand Ġanswer able Ġclassification , Ġoptimized Ġby ĠAdam Ġalgorithm ĠB IB REF 26 Ġwith Ġinitial Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġ. ĠPre - trained ĠGl o Ve ĠB IB REF 3 Ġembed d ings Ġ( size ĠIN LINE FORM 1 Ġ) Ġwere Ġused . ĠWe Ġre - sh uff led Ġtraining Ġsamples Ġat Ġeach Ġepoch Ġ( batch Ġsize ĠIN LINE FORM 2 Ġ). ĠOut - of - voc abulary Ġwords Ġ( ĠIN LINE FORM 3 Ġ) Ġwere Ġreplaced Ġwith Ġa Ġfixed Ġrandom Ġvector . ĠL 2 Ġregular ization Ġand Ġdrop out Ġ( rate ĠIN LINE FORM 4 Ġ) Ġwere Ġused Ġto Ġalleviate Ġover fitting ĠB IB REF 27 Ġ. </s> E val uation ĠSetup </s> To Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġlingu istically - inspired Ġsimulation Ġapproach , Ġthe Ġmodel Ġis Ġtrained Ġon Ġthe Ġsimulated Ġdata Ġ( see ĠSection ĠSEC REF 20 Ġ). ĠWe Ġdesigned Ġ3 Ġevaluation Ġsets : Ġ( 1 ) ĠBase ĠSet Ġ( 1 , 264 Ġsamples ) Ġheld Ġout Ġfrom Ġthe Ġsimulated Ġdata . Ġ( 2 ) ĠAug mented ĠSet Ġ( 1 , 280 Ġsamples ) Ġbuilt Ġby Ġadding Ġtwo Ġout - of - dist ribution Ġsymptoms , Ġwith Ġcorresponding Ġdialogue Ġcontents Ġand Ġqueries , Ġto Ġthe ĠBase ĠSet Ġ( âĢ ľ ble eding âĢ Ŀ Ġand ĠâĢ ľ cold âĢ Ŀ , Ġwhich Ġnever Ġappeared Ġin Ġtraining Ġdata ). Ġ( 3 ) ĠReal - World ĠSet Ġ( 9 44 Ġsamples ) Ġmanually Ġdeline ated Ġfrom Ġthe Ġthe Ġsymptom Ġchecking Ġportions Ġ( approximately Ġ4 Ġhours ) Ġof Ġreal - world Ġdialog ues , Ġand Ġannot ated Ġas Ġevaluation Ġsamples . </s> Results </s> E val uation Ġresults Ġare Ġin ĠTable ĠT AB REF 25 Ġwith Ġexact Ġmatch Ġ( EM ) Ġand ĠF 1 Ġscore Ġin ĠB IB REF 0 Ġmetrics . ĠTo Ġdistinguish Ġthe Ġcorrect Ġanswer Ġspan Ġfrom Ġthe Ġplausible Ġones Ġwhich Ġcontain Ġthe Ġsame Ġwords , Ġwe Ġmeasure Ġthe Ġscores Ġon Ġthe Ġposition Ġindices Ġof Ġtokens . ĠOur Ġresults Ġshow Ġthat Ġboth ĠEM Ġand ĠF 1 Ġscore Ġincrease Ġwith Ġtraining Ġsample Ġsize Ġgrowing Ġand Ġthe Ġoptimal Ġsize Ġin Ġour Ġsetting Ġis Ġ100 k . ĠThe Ġbest - trained Ġmodel Ġperforms Ġwell Ġon Ġboth Ġthe ĠBase ĠSet Ġand Ġthe ĠAug mented ĠSet , Ġindicating Ġthat Ġout - of - dist ribution Ġsymptoms Ġdo Ġnot Ġaffect Ġthe Ġcomprehension Ġof Ġexisting Ġsymptoms Ġand Ġoutputs Ġreasonable Ġanswers Ġfor Ġboth Ġin - Ġand Ġout - of - dist ribution Ġsymptoms . ĠOn Ġthe ĠReal - World ĠSet , Ġwe Ġobtained Ġ78 . 23 ĠEM Ġscore Ġand Ġ80 . 18 ĠF 1 Ġscore Ġrespectively . </s> Error Ġanalysis Ġsuggests Ġthe Ġperformance Ġdrop Ġfrom Ġthe Ġsimulated Ġtest Ġsets Ġis Ġdue Ġto Ġthe Ġfollowing : Ġ1 ) Ġsp arsity Ġissues Ġresulting Ġfrom Ġthe Ġexpression Ġpools Ġexcluding Ġvarious Ġvalid Ġbut Ġsporadic Ġexpressions . Ġ2 ) Ġnurses Ġand Ġpatients Ġoccasionally Ġch it - chat Ġin Ġthe ĠReal - World ĠSet , Ġwhich Ġis Ġnot Ġsimulated Ġin Ġthe Ġtraining Ġset . ĠAt Ġtimes , Ġthese Ġch it - ch ats Ġmake Ġthe Ġconversations Ġoverly Ġlengthy , Ġcausing Ġthe Ġinformation Ġdensity Ġto Ġbe Ġlower . ĠThese Ġissues Ġcould Ġpotentially Ġdistract Ġand Ġconfuse Ġthe Ġcomprehension Ġmodel . Ġ3 ) Ġan Ġinteresting Ġtype Ġof Ġinf requent Ġerror Ġsource , Ġcaused Ġby Ġpatients Ġelabor ating Ġon Ġpossible Ġcausal Ġrelations Ġof Ġtwo Ġsymptoms . ĠFor Ġexample , Ġa Ġpatient Ġmight Ġsay ĠâĢ ľ My Ġg idd iness Ġmay Ġbe Ġdue Ġto Ġall Ġthis Ġcough âĢ Ŀ . ĠWe Ġare Ġcurrently Ġinvestigating Ġhow Ġto Ġclose Ġthis Ġperformance Ġgap Ġefficiently . </s> A bl ation ĠAnalysis </s> To Ġassess Ġthe Ġeffectiveness Ġof Ġbi - direction al Ġattention , Ġwe Ġbypass ed Ġthe Ġbi - att ention Ġlayer Ġby Ġdirectly Ġfeeding Ġthe Ġcontextual Ġhidden Ġstates Ġand Ġquery Ġembed d ings Ġto Ġthe Ġmodeling Ġlayer . ĠTo Ġevaluate Ġthe Ġpre - trained ĠGl o Ve Ġembed d ings , Ġwe Ġrandomly Ġinitialized Ġand Ġtrained Ġthe Ġembed d ings Ġfrom Ġscratch . ĠThese Ġtwo Ġprocedures Ġlead Ġto Ġ10 % Ġand Ġ18 % Ġperformance Ġdegradation Ġon Ġthe ĠAug mented ĠSet Ġand ĠReal - World ĠSet , Ġrespectively Ġ( see ĠTable ĠT AB REF 27 Ġ). </s> Conclusion </s> We Ġformulated Ġa Ġdialogue Ġcomprehension Ġtask Ġmotivated Ġby Ġthe Ġneed Ġin Ġtele health Ġsettings Ġto Ġextract Ġkey Ġclinical Ġinformation Ġfrom Ġspoken Ġconversations Ġbetween Ġnurses Ġand Ġpatients . ĠWe Ġanalyzed Ġlinguistic Ġcharacteristics Ġof Ġreal - world Ġhuman - human Ġsymptom Ġchecking Ġdialog ues , Ġconstructed Ġa Ġsimulated Ġdataset Ġbased Ġon Ġlingu istically Ġinspired Ġand Ġclinically Ġvalidated Ġtemplates , Ġand Ġprototyp ed Ġa ĠQ A Ġsystem . ĠThe Ġmodel Ġworks Ġeffectively Ġon Ġa Ġsimulated Ġtest Ġset Ġusing Ġsymptoms Ġexcluded Ġduring Ġtraining Ġand Ġon Ġreal - world Ġconversations Ġbetween Ġnurses Ġand Ġpatients . ĠWe Ġare Ġcurrently Ġimproving Ġthe Ġmodel 's Ġdialogue Ġcomprehension Ġcapability Ġin Ġcomplex Ġreasoning Ġand Ġcontext Ġunderstanding Ġand Ġalso Ġapplying Ġthe ĠQ A Ġmodel Ġto Ġsummar ization Ġand Ġvirtual Ġnurse Ġapplications . </s> Acknowled gements </s> Research Ġefforts Ġwere Ġsupported Ġby Ġfunding Ġfor ĠDigital ĠHealth Ġand ĠDeep ĠLearning ĠI 2 R Ġ( DL 2 ĠSS F ĠProject ĠNo : ĠA 17 18 g 00 45 ) Ġand Ġthe ĠScience Ġand ĠEngineering ĠResearch ĠCouncil Ġ( SER C ĠProject ĠNo : ĠA 18 18 g 00 44 ), ĠA * STAR , ĠSingapore . ĠIn Ġaddition , Ġthis Ġwork Ġwas Ġconducted Ġusing Ġresources Ġand Ġinfrastructure Ġprovided Ġby Ġthe ĠHuman ĠLanguage ĠTechnology Ġunit Ġat ĠI 2 R . ĠThe Ġtele health Ġdata Ġacquisition Ġwas Ġfunded Ġby Ġthe ĠEconomic ĠDevelopment ĠBoard Ġ( ED B ), ĠSingapore ĠLiving ĠLab ĠFund Ġand ĠPhilips ĠElectronics ĠâĢĵ ĠHospital Ġto ĠHome ĠPilot ĠProject Ġ( ED B Ġgrant Ġreference Ġnumber : ĠS 14 - 10 35 - RF - LL F ĠH Ġand ĠW ). </s> We Ġacknowledge Ġvaluable Ġsupport Ġand Ġassistance Ġfrom ĠY ul ong ĠLin , ĠYuan x in ĠXiang , Ġand ĠAi ĠTi ĠAw Ġat Ġthe ĠInstitute Ġfor ĠInf oc omm ĠResearch Ġ( I 2 R ); ĠWe ili ang ĠHuang Ġat Ġthe ĠChang i ĠGeneral ĠHospital Ġ( C GH ) ĠDepartment Ġof ĠCard iology , ĠHong ĠCh oon ĠOh Ġat ĠC GH ĠHealth ĠServices ĠResearch , Ġand ĠEd ris ĠAt ik ah ĠB te ĠAhmad , ĠCh iu ĠYan ĠAng , Ġand ĠMei ĠF oon ĠY ap Ġof Ġthe ĠC GH ĠIntegrated ĠCare ĠDepartment . </s> We Ġalso Ġthank ĠEdu ard ĠH ovy Ġand ĠBonnie ĠWeb ber Ġfor Ġinsightful Ġdiscussions Ġand Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġprecious Ġfeedback Ġto Ġhelp Ġimprove Ġand Ġextend Ġthis Ġpiece Ġof Ġwork .
Problem Statement Spoken conversations still remain the most natural and effortless means of human communication. Thus a lot of valuable information is conveyed and exchanged in such an unstructured form. In telehealth settings, nurses might call discharged patients who have returned home to continue to monitor their health status. Human language technology that can efficiently and effectively extract key information from such conversations is clinically useful, as it can help streamline workflow processes and digitally document patient medical information to increase staff productivity. In this work, we design and prototype a dialogue comprehension system in the question-answering manner, which is able to comprehend spoken conversations between nurses and patients to extract clinical information. Motivation of Approach Machine comprehension of written passages has made tremendous progress recently. Large quantities of supervised training data for reading comprehension (e.g. SQuAD BIBREF0 ), the wide adoption and intense experimentation of neural modeling BIBREF1 , BIBREF2 , and the advancements in vector representations of word embeddings BIBREF3 , BIBREF4 all contribute significantly to the achievements obtained so far. The first factor, the availability of large scale datasets, empowers the latter two factors. To date, there is still very limited well-annotated large-scale data suitable for modeling human-human spoken dialogues. Therefore, it is not straightforward to directly port over the recent endeavors in reading comprehension to dialogue comprehension tasks. In healthcare, conversation data is even scarcer due to privacy issues. Crowd-sourcing is an efficient way to annotate large quantities of data, but less suitable for healthcare scenarios, where domain knowledge is required to guarantee data quality. To demonstrate the feasibility of a dialogue comprehension system used for extracting key clinical information from symptom monitoring conversations, we developed a framework to construct a simulated human-human dialogue dataset to bootstrap such a prototype. Similar efforts have been conducted for human-machine dialogues for restaurant or movie reservations BIBREF5 . To the best of our knowledge, no one to date has done so for human-human conversations in healthcare. Human-human Spoken Conversations Human-human spoken conversations are a dynamic and interactive flow of information exchange. While developing technology to comprehend such spoken conversations presents similar technical challenges as machine comprehension of written passages BIBREF6 , the challenges are further complicated by the interactive nature of human-human spoken conversations: (1) Zero anaphora is more common: Co-reference resolution of spoken utterances from multiple speakers is needed. For example, in Figure FIGREF5 (a) headaches, the pain, it, head bulging all refer to the patient's headache symptom, but they were uttered by different speakers and across multiple utterances and turns. In addition, anaphors are more likely to be omitted (see Figure FIGREF5 (a) A4) as this does not affect the human listener’s understanding, but it might be challenging for computational models. (2) Thinking aloud more commonly occurs: Since it is more effortless to speak than to type, one is more likely to reveal her running thoughts when talking. In addition, one cannot retract what has been uttered, while in text communications, one is more likely to confirm the accuracy of the information in a written response and revise if necessary before sending it out. Thinking aloud can lead to self-contradiction, requiring more context to fully understand the dialogue; e.g., in A6 in Figure FIGREF5 (a), the patient at first says he has none of the symptoms asked, but later revises his response saying that he does get dizzy after running. (3) Topic drift is more common and harder to detect in spoken conversations: An example is shown in Figure FIGREF5 (a) in A3, where No is actually referring to cough in the previous question, and then the topic is shifted to headache. In spoken conversations, utterances are often incomplete sentences so traditional linguistic features used in written passages such as punctuation marks indicating syntactic boundaries or conjunction words suggesting discourse relations might no longer exist. Dialogue Comprehension Task Figure FIGREF5 (b) illustrates the proposed dialogue comprehension task using a question answering (QA) model. The input are a multi-turn symptom checking dialogue INLINEFORM0 and a query INLINEFORM1 specifying a symptom with one of its attributes; the output is the extracted answer INLINEFORM2 from the given dialogue. A training or test sample is defined as INLINEFORM3 . Five attributes, specifying certain details of clinical significance, are defined to characterize the answer types of INLINEFORM4 : (1) the time the patient has been experiencing the symptom, (2) activities that trigger the symptom (to occur or worsen), (3) the extent of seriousness, (4) the frequency occurrence of the symptom, and (5) the location of symptom. For each symptom/attribute, it can take on different linguistic expressions, defined as entities. Note that if the queried symptom or attribute is not mentioned in the dialogue, the groundtruth output is “No Answer”, as in BIBREF6 . Reading Comprehension Large-scale reading comprehension tasks like SQuAD BIBREF0 and MARCO BIBREF7 provide question-answer pairs from a vast range of written passages, covering different kinds of factual answers involving entities such as location and numerical values. Furthermore, HotpotQA BIBREF8 requires multi-step inference and provides numerous answer types. CoQA BIBREF9 and QuAC BIBREF10 are designed to mimic multi-turn information-seeking discussions of the given material. In these tasks, contextual reasoning like coreference resolution is necessary to grasp rich linguistic patterns, encouraging semantic modeling beyond naive lexical matching. Neural networks contribute to impressive progress in semantic modeling: distributional semantic word embeddings BIBREF3 , contextual sequence encoding BIBREF11 , BIBREF12 and the attention mechanism BIBREF13 , BIBREF14 are widely adopted in state-of-the-art comprehension models BIBREF1 , BIBREF2 , BIBREF4 . While language understanding tasks in dialogue such as domain identification BIBREF15 , slot filling BIBREF16 and user intent detection BIBREF17 have attracted much research interest, work in dialogue comprehension is still limited, if any. It is labor-intensive and time-consuming to obtain a critical mass of annotated conversation data for computational modeling. Some propose to collect text data from human-machine or machine-machine dialogues BIBREF18 , BIBREF5 . In such cases, as human speakers are aware of current limitations of dialogue systems or due to pre-defined assumptions of user simulators, there are fewer cases of zero anaphora, thinking aloud, and topic drift, which occur more often in human-human spoken interactions. NLP for Healthcare There is emerging interest in research and development activities at the intersection of machine learning and healthcare , of which much of the NLP related work are centered around social media or online forums (e.g., BIBREF19 , BIBREF20 ), partially due to the world wide web as a readily available source of information. Other work in this area uses public data sources such as MIMIC in electronic health records: text classification approaches have been applied to analyze unstructured clinical notes for ICD code assignment BIBREF21 and automatic intensive emergency prediction BIBREF22 . Sequence-to-sequence textual generation has been used for readable notes based on medical and demographic recordings BIBREF23 . For mental health, there has been more focus on analyzing dialogues. For example, sequential modeling of audio and text have helped detect depression from human-machine interviews BIBREF24 . However, few studies have examined human-human spoken conversations in healthcare settings. Data Preparation We used recordings of nurse-initiated telephone conversations for congestive heart failure patients undergoing telemonitoring, post-discharge from the hospital. The clinical data was acquired by the Health Management Unit at Changi General Hospital. This research study was approved by the SingHealth Centralised Institutional Review Board (Protocol 1556561515). The patients were recruited during 2014-2016 as part of their routine care delivery, and enrolled into the telemonitoring health management program with consent for use of anonymized versions of their data for research. The dataset comprises a total of 353 conversations from 40 speakers (11 nurses, 16 patients, and 13 caregivers) with consent to the use of anonymized data for research. The speakers are 38 to 88 years old, equally distributed across gender, and comprise a range of ethnic groups (55% Chinese, 17% Malay, 14% Indian, 3% Eurasian, and 11% unspecified). The conversations cover 11 topics (e.g., medication compliance, symptom checking, education, greeting) and 9 symptoms (e.g., chest pain, cough) and amount to 41 hours. Data preprocessing and anonymization were performed by a data preparation team, separate from the data analysis team to maintain data confidentiality. The data preparation team followed standard speech recognition transcription guidelines, where words are transcribed verbatim to include false starts, disfluencies, mispronunciations, and private self-talk. Confidential information were marked and clipped off from the audio and transcribed with predefined tags in the annotation. Conversation topics and clinical symptoms were also annotated and clinically validated by certified telehealth nurses. Linguistic Characterization on Seed Data To analyze the linguistic structure of the inquiry-response pairs in the entire 41-hour dataset, we randomly sampled a seed dataset consisting of 1,200 turns and manually categorized them to different types, which are summarized in Table TABREF14 along with the corresponding occurrence frequency statistics. Note that each given utterance could be categorized to more than one type. We elaborate on each utterance type below. Open-ended Inquiry: Inquiries about general well-being or a particular symptom; e.g., “How are you feeling?” and “Do you cough?” Detailed Inquiry: Inquiries with specific details that prompt yes/no answers or clarifications; e.g., “Do you cough at night?” Multi-Intent Inquiry: Inquiring more than one symptom in a question; e.g., “Any cough, chest pain, or headache?” Reconfirmation Inquiry: The nurse reconfirms particular details; e.g., “Really? At night?” and “Serious or mild?”. This case is usually related to explicit or implicit coreferencing. Inquiry with Transitional Clauses: During spoken conversations, one might repeat what the other party said, but it is unrelated to the main clause of the question. This is usually due to private self-talk while thinking aloud, and such utterances form a transitional clause before the speaker starts a new topic; e.g., “Chest pain... no chest pain, I see... any cough?”. Yes/No Response: Yes/No responses seem straightforward, but sometimes lead to misunderstanding if one does not interpret the context appropriately. One case is tag questions: A:“You don't cough at night, do you?” B:`Yes, yes” A:“cough at night?” B:“No, no cough”. Usually when the answer is unclear, clarifying inquiries will be asked for reconfirmation purposes. Detailed Response: Responses that contain specific information of one symptom, like “I felt tightness in my chest”. Response with Revision: Revision is infrequent but can affect comprehension significantly. One cause is thinking aloud so a later response overrules the previous one; e.g., “No dizziness, oh wait... last week I felt a bit dizzy when biking”. Response with Topic Drift: When a symptom/topic like headache is inquired, the response might be: “Only some chest pain at night”, not referring to the original symptom (headache) at all. Response with Transitional Clauses: Repeating some of the previous content, but often unrelated to critical clinical information and usually followed by topic drift. For example, “Swelling... swelling... I don't cough at night”. Simulating Symptom Monitoring Dataset for Training We divide the construction of data simulation into two stages. In Section SECREF16 , we build templates and expression pools using linguistic analysis followed by manual verification. In Section SECREF20 , we present our proposed framework for generating simulated training data. The templates and framework are verified for logical correctness and clinical soundness. Template Construction Each utterance in the seed data is categorized according to Table TABREF14 and then abstracted into templates by replacing entity phrases like cough and often with respective placeholders “#symptom#” and “#frequency#”. The templates are refined through verifying logical correctness and injecting expression diversity by linguistically trained researchers. As these replacements do not alter the syntactic structure, we interchange such placeholders with various verbal expressions to enlarge the simulated training set in Section SECREF20 . Clinical validation was also conducted by certified telehealth nurses. For the 9 symptoms (e.g. chest pain, cough) and 5 attributes (e.g., extent, frequency), we collect various expressions from the seed data, and expand them through synonym replacement. Some attributes are unique to a particular symptom; e.g., “left leg” in #location# is only suitable to describe the symptom swelling, but not the symptom headache. Therefore, we only reuse general expressions like “slight” in #extent# across different symptoms to diversify linguistic expressions. Two linguistically trained researchers constructed expression pools for each symptom and each attribute to account for different types of paraphrasing and descriptions. These expression pools are used in Section SECREF20 (c). Simulated Data Generation Framework Figure FIGREF15 shows the five steps we use to generate multi-turn symptom monitoring dialogue samples. (a) Topic Selection: While nurses might prefer to inquire the symptoms in different orders depending on the patient's history, our preliminary analysis shows that modeling results do not differ noticeably if topics are of equal prior probabilities. Thus we adopt this assumption for simplicity. (b) Template Selection: For each selected topic, one inquiry template and one response template are randomly chosen to compose a turn. To minimize adverse effects of underfitting, we redistributed the frequency distribution in Table TABREF14 : For utterance types that are below 15%, we boosted them to 15%, and the overall relative distribution ranking is balanced and consistent with Table TABREF14 . (c) Enriching Linguistic Expressions: The placeholders in the selected templates are substituted with diverse expressions from the expression pools in Section UID19 to characterize the symptoms and their corresponding attributes. (d) Multi-Turn Dialogue State Tracking: A greedy algorithm is applied to complete conversations. A “completed symptoms” list and a “to-do symptoms” list are used for symptom topic tracking. We also track the “completed attributes" and “to-do attributes". For each symptom, all related attributes are iterated. A dialogue ends only when all possible entities are exhausted, generating a multi-turn dialogue sample, which encourages the model to learn from the entire discussion flow rather than a single turn to comprehend contextual dependency. The average length of a simulated dialogue is 184 words, which happens to be twice as long as an average dialogue from the real-world evaluation set. Moreover, to model the roles of the respondents, we set the ratio between patients and caregivers to 2:1; this statistic is inspired by the real scenarios in the seed dataset. For both the caregivers and patients, we assume equal probability of both genders. The corresponding pronouns in the conversations are thus determined by the role and gender of these settings. (e) Multi-Turn Sample Annotation: For each multi-turn dialogue, a query is specified by a symptom and an attribute. The groundtruth output of the QA system is automatically labeled based on the template generation rules, but also manually verified to ensure annotation quality. Moreover, we adopt the unanswerable design in BIBREF6 : when the patient does not mention a particular symptom, the answer is defined as “No Answer”. This process is repeated until all logical permutations of symptoms and attributes are exhausted. Experiments Model Design We implemented an established model in reading comprehension, a bi-directional attention pointer network BIBREF1 , and equipped it with an answerable classifier, as depicted in Figure FIGREF21 . First, tokens in the given dialogue INLINEFORM0 and query INLINEFORM1 are converted into embedding vectors. Then the dialogue embeddings are fed to a bi-directional LSTM encoding layer, generating a sequence of contextual hidden states. Next, the hidden states and query embeddings are processed by a bi-directional attention layer, fusing attention information in both context-to-query and query-to-context directions. The following two bi-directional LSTM modeling layers read the contextual sequence with attention. Finally, two respective linear layers with softmax functions are used to estimate token INLINEFORM2 's INLINEFORM3 and INLINEFORM4 probability of the answer span INLINEFORM5 . In addition, we add a special tag “[SEQ]” at the head of INLINEFORM0 to account for the case of “No answer” BIBREF4 and adopt an answerable classifier as in BIBREF25 . More specifically, when the queried symptom or attribute is not mentioned in the dialogue, the answer span should point to the tag “[SEQ]” and answerable probability should be predicted as 0. Implementation Details The model was trained via gradient backpropagation with the cross-entropy loss function of answer span prediction and answerable classification, optimized by Adam algorithm BIBREF26 with initial learning rate of INLINEFORM0 . Pre-trained GloVe BIBREF3 embeddings (size INLINEFORM1 ) were used. We re-shuffled training samples at each epoch (batch size INLINEFORM2 ). Out-of-vocabulary words ( INLINEFORM3 ) were replaced with a fixed random vector. L2 regularization and dropout (rate INLINEFORM4 ) were used to alleviate overfitting BIBREF27 . Evaluation Setup To evaluate the effectiveness of our linguistically-inspired simulation approach, the model is trained on the simulated data (see Section SECREF20 ). We designed 3 evaluation sets: (1) Base Set (1,264 samples) held out from the simulated data. (2) Augmented Set (1,280 samples) built by adding two out-of-distribution symptoms, with corresponding dialogue contents and queries, to the Base Set (“bleeding” and “cold”, which never appeared in training data). (3) Real-World Set (944 samples) manually delineated from the the symptom checking portions (approximately 4 hours) of real-world dialogues, and annotated as evaluation samples. Results Evaluation results are in Table TABREF25 with exact match (EM) and F1 score in BIBREF0 metrics. To distinguish the correct answer span from the plausible ones which contain the same words, we measure the scores on the position indices of tokens. Our results show that both EM and F1 score increase with training sample size growing and the optimal size in our setting is 100k. The best-trained model performs well on both the Base Set and the Augmented Set, indicating that out-of-distribution symptoms do not affect the comprehension of existing symptoms and outputs reasonable answers for both in- and out-of-distribution symptoms. On the Real-World Set, we obtained 78.23 EM score and 80.18 F1 score respectively. Error analysis suggests the performance drop from the simulated test sets is due to the following: 1) sparsity issues resulting from the expression pools excluding various valid but sporadic expressions. 2) nurses and patients occasionally chit-chat in the Real-World Set, which is not simulated in the training set. At times, these chit-chats make the conversations overly lengthy, causing the information density to be lower. These issues could potentially distract and confuse the comprehension model. 3) an interesting type of infrequent error source, caused by patients elaborating on possible causal relations of two symptoms. For example, a patient might say “My giddiness may be due to all this cough”. We are currently investigating how to close this performance gap efficiently. Ablation Analysis To assess the effectiveness of bi-directional attention, we bypassed the bi-attention layer by directly feeding the contextual hidden states and query embeddings to the modeling layer. To evaluate the pre-trained GloVe embeddings, we randomly initialized and trained the embeddings from scratch. These two procedures lead to 10% and 18% performance degradation on the Augmented Set and Real-World Set, respectively (see Table TABREF27 ). Conclusion We formulated a dialogue comprehension task motivated by the need in telehealth settings to extract key clinical information from spoken conversations between nurses and patients. We analyzed linguistic characteristics of real-world human-human symptom checking dialogues, constructed a simulated dataset based on linguistically inspired and clinically validated templates, and prototyped a QA system. The model works effectively on a simulated test set using symptoms excluded during training and on real-world conversations between nurses and patients. We are currently improving the model's dialogue comprehension capability in complex reasoning and context understanding and also applying the QA model to summarization and virtual nurse applications. Acknowledgements Research efforts were supported by funding for Digital Health and Deep Learning I2R (DL2 SSF Project No: A1718g0045) and the Science and Engineering Research Council (SERC Project No: A1818g0044), A*STAR, Singapore. In addition, this work was conducted using resources and infrastructure provided by the Human Language Technology unit at I2R. The telehealth data acquisition was funded by the Economic Development Board (EDB), Singapore Living Lab Fund and Philips Electronics – Hospital to Home Pilot Project (EDB grant reference number: S14-1035-RF-LLF H and W). We acknowledge valuable support and assistance from Yulong Lin, Yuanxin Xiang, and Ai Ti Aw at the Institute for Infocomm Research (I2R); Weiliang Huang at the Changi General Hospital (CGH) Department of Cardiology, Hong Choon Oh at CGH Health Services Research, and Edris Atikah Bte Ahmad, Chiu Yan Ang, and Mei Foon Yap of the CGH Integrated Care Department. We also thank Eduard Hovy and Bonnie Webber for insightful discussions and the anonymous reviewers for their precious feedback to help improve and extend this piece of work.
How big is their created dataset?
1b1a30e9e68a9ae76af467e60cefb180d135e285
How Ġbig Ġis Ġtheir Ġcreated Ġdataset ?
353 Ġconversations Ġfrom Ġ40 Ġspeakers Ġ( 11 Ġnurses , Ġ16 Ġpatients , Ġand Ġ13 Ġcaregivers ), Ġwe Ġbuild Ġtemplates Ġand Ġexpression Ġpools Ġusing Ġlinguistic Ġanalysis
<s> How Ġbig Ġis Ġtheir Ġcreated Ġdataset ? </s> Problem ĠStatement </s> Sp oken Ġconversations Ġstill Ġremain Ġthe Ġmost Ġnatural Ġand Ġeffort less Ġmeans Ġof Ġhuman Ġcommunication . ĠThus Ġa Ġlot Ġof Ġvaluable Ġinformation Ġis Ġconveyed Ġand Ġexchanged Ġin Ġsuch Ġan Ġun struct ured Ġform . ĠIn Ġtele health Ġsettings , Ġnurses Ġmight Ġcall Ġdischarged Ġpatients Ġwho Ġhave Ġreturned Ġhome Ġto Ġcontinue Ġto Ġmonitor Ġtheir Ġhealth Ġstatus . ĠHuman Ġlanguage Ġtechnology Ġthat Ġcan Ġefficiently Ġand Ġeffectively Ġextract Ġkey Ġinformation Ġfrom Ġsuch Ġconversations Ġis Ġclinically Ġuseful , Ġas Ġit Ġcan Ġhelp Ġstream line Ġworkflow Ġprocesses Ġand Ġdigitally Ġdocument Ġpatient Ġmedical Ġinformation Ġto Ġincrease Ġstaff Ġproductivity . ĠIn Ġthis Ġwork , Ġwe Ġdesign Ġand Ġprototype Ġa Ġdialogue Ġcomprehension Ġsystem Ġin Ġthe Ġquestion - ans w ering Ġmanner , Ġwhich Ġis Ġable Ġto Ġcomprehend Ġspoken Ġconversations Ġbetween Ġnurses Ġand Ġpatients Ġto Ġextract Ġclinical Ġinformation . </s> Mot ivation Ġof ĠApproach </s> Machine Ġcomprehension Ġof Ġwritten Ġpassages Ġhas Ġmade Ġtremendous Ġprogress Ġrecently . ĠLarge Ġquantities Ġof Ġsupervised Ġtraining Ġdata Ġfor Ġreading Ġcomprehension Ġ( e . g . ĠS Qu AD ĠB IB REF 0 Ġ), Ġthe Ġwide Ġadoption Ġand Ġintense Ġexperimentation Ġof Ġneural Ġmodeling ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġand Ġthe Ġadvancements Ġin Ġvector Ġrepresentations Ġof Ġword Ġembed d ings ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġall Ġcontribute Ġsignificantly Ġto Ġthe Ġachievements Ġobtained Ġso Ġfar . ĠThe Ġfirst Ġfactor , Ġthe Ġavailability Ġof Ġlarge Ġscale Ġdatasets , Ġem powers Ġthe Ġlatter Ġtwo Ġfactors . ĠTo Ġdate , Ġthere Ġis Ġstill Ġvery Ġlimited Ġwell - annot ated Ġlarge - scale Ġdata Ġsuitable Ġfor Ġmodeling Ġhuman - human Ġspoken Ġdialog ues . ĠTherefore , Ġit Ġis Ġnot Ġstraightforward Ġto Ġdirectly Ġport Ġover Ġthe Ġrecent Ġendeavors Ġin Ġreading Ġcomprehension Ġto Ġdialogue Ġcomprehension Ġtasks . </s> In Ġhealthcare , Ġconversation Ġdata Ġis Ġeven Ġscar cer Ġdue Ġto Ġprivacy Ġissues . ĠCrowd - s ourcing Ġis Ġan Ġefficient Ġway Ġto Ġannot ate Ġlarge Ġquantities Ġof Ġdata , Ġbut Ġless Ġsuitable Ġfor Ġhealthcare Ġscenarios , Ġwhere Ġdomain Ġknowledge Ġis Ġrequired Ġto Ġguarantee Ġdata Ġquality . ĠTo Ġdemonstrate Ġthe Ġfeasibility Ġof Ġa Ġdialogue Ġcomprehension Ġsystem Ġused Ġfor Ġextracting Ġkey Ġclinical Ġinformation Ġfrom Ġsymptom Ġmonitoring Ġconversations , Ġwe Ġdeveloped Ġa Ġframework Ġto Ġconstruct Ġa Ġsimulated Ġhuman - human Ġdialogue Ġdataset Ġto Ġboot strap Ġsuch Ġa Ġprototype . ĠSimilar Ġefforts Ġhave Ġbeen Ġconducted Ġfor Ġhuman - machine Ġdialog ues Ġfor Ġrestaurant Ġor Ġmovie Ġreservations ĠB IB REF 5 Ġ. ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġno Ġone Ġto Ġdate Ġhas Ġdone Ġso Ġfor Ġhuman - human Ġconversations Ġin Ġhealthcare . </s> Human - human ĠSp oken ĠConvers ations </s> Human - human Ġspoken Ġconversations Ġare Ġa Ġdynamic Ġand Ġinteractive Ġflow Ġof Ġinformation Ġexchange . ĠWhile Ġdeveloping Ġtechnology Ġto Ġcomprehend Ġsuch Ġspoken Ġconversations Ġpresents Ġsimilar Ġtechnical Ġchallenges Ġas Ġmachine Ġcomprehension Ġof Ġwritten Ġpassages ĠB IB REF 6 Ġ, Ġthe Ġchallenges Ġare Ġfurther Ġcomplicated Ġby Ġthe Ġinteractive Ġnature Ġof Ġhuman - human Ġspoken Ġconversations : </s> ( 1 ) ĠZero Ġan aph ora Ġis Ġmore Ġcommon : ĠCo - reference Ġresolution Ġof Ġspoken Ġutter ances Ġfrom Ġmultiple Ġspeakers Ġis Ġneeded . ĠFor Ġexample , Ġin ĠFigure ĠFIG REF 5 Ġ( a ) Ġheadaches , Ġthe Ġpain , Ġit , Ġhead Ġbul ging Ġall Ġrefer Ġto Ġthe Ġpatient 's Ġheadache Ġsymptom , Ġbut Ġthey Ġwere Ġuttered Ġby Ġdifferent Ġspeakers Ġand Ġacross Ġmultiple Ġutter ances Ġand Ġturns . ĠIn Ġaddition , Ġan aph ors Ġare Ġmore Ġlikely Ġto Ġbe Ġomitted Ġ( see ĠFigure ĠFIG REF 5 Ġ( a ) ĠA 4 ) Ġas Ġthis Ġdoes Ġnot Ġaffect Ġthe Ġhuman Ġlistener âĢ Ļ s Ġunderstanding , Ġbut Ġit Ġmight Ġbe Ġchallenging Ġfor Ġcomputational Ġmodels . </s> ( 2 ) ĠThinking Ġaloud Ġmore Ġcommonly Ġoccurs : ĠSince Ġit Ġis Ġmore Ġeffort less Ġto Ġspeak Ġthan Ġto Ġtype , Ġone Ġis Ġmore Ġlikely Ġto Ġreveal Ġher Ġrunning Ġthoughts Ġwhen Ġtalking . ĠIn Ġaddition , Ġone Ġcannot Ġretract Ġwhat Ġhas Ġbeen Ġuttered , Ġwhile Ġin Ġtext Ġcommunications , Ġone Ġis Ġmore Ġlikely Ġto Ġconfirm Ġthe Ġaccuracy Ġof Ġthe Ġinformation Ġin Ġa Ġwritten Ġresponse Ġand Ġrevise Ġif Ġnecessary Ġbefore Ġsending Ġit Ġout . ĠThinking Ġaloud Ġcan Ġlead Ġto Ġself - cont rad iction , Ġrequiring Ġmore Ġcontext Ġto Ġfully Ġunderstand Ġthe Ġdialogue ; Ġe . g ., Ġin ĠA 6 Ġin ĠFigure ĠFIG REF 5 Ġ( a ), Ġthe Ġpatient Ġat Ġfirst Ġsays Ġhe Ġhas Ġnone Ġof Ġthe Ġsymptoms Ġasked , Ġbut Ġlater Ġrev ises Ġhis Ġresponse Ġsaying Ġthat Ġhe Ġdoes Ġget Ġdizz y Ġafter Ġrunning . </s> ( 3 ) ĠTopic Ġdrift Ġis Ġmore Ġcommon Ġand Ġharder Ġto Ġdetect Ġin Ġspoken Ġconversations : ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 5 Ġ( a ) Ġin ĠA 3 , Ġwhere ĠNo Ġis Ġactually Ġreferring Ġto Ġcough Ġin Ġthe Ġprevious Ġquestion , Ġand Ġthen Ġthe Ġtopic Ġis Ġshifted Ġto Ġheadache . ĠIn Ġspoken Ġconversations , Ġutter ances Ġare Ġoften Ġincomplete Ġsentences Ġso Ġtraditional Ġlinguistic Ġfeatures Ġused Ġin Ġwritten Ġpassages Ġsuch Ġas Ġpunct uation Ġmarks Ġindicating Ġsynt actic Ġboundaries Ġor Ġconjunction Ġwords Ġsuggesting Ġdiscourse Ġrelations Ġmight Ġno Ġlonger Ġexist . </s> Dialogue ĠComp rehens ion ĠTask </s> Figure ĠFIG REF 5 Ġ( b ) Ġillustrates Ġthe Ġproposed Ġdialogue Ġcomprehension Ġtask Ġusing Ġa Ġquestion Ġanswering Ġ( Q A ) Ġmodel . ĠThe Ġinput Ġare Ġa Ġmulti - turn Ġsymptom Ġchecking Ġdialogue ĠIN LINE FORM 0 Ġand Ġa Ġquery ĠIN LINE FORM 1 Ġspecifying Ġa Ġsymptom Ġwith Ġone Ġof Ġits Ġattributes ; Ġthe Ġoutput Ġis Ġthe Ġextracted Ġanswer ĠIN LINE FORM 2 Ġfrom Ġthe Ġgiven Ġdialogue . ĠA Ġtraining Ġor Ġtest Ġsample Ġis Ġdefined Ġas ĠIN LINE FORM 3 Ġ. ĠFive Ġattributes , Ġspecifying Ġcertain Ġdetails Ġof Ġclinical Ġsignificance , Ġare Ġdefined Ġto Ġcharacterize Ġthe Ġanswer Ġtypes Ġof ĠIN LINE FORM 4 Ġ: Ġ( 1 ) Ġthe Ġtime Ġthe Ġpatient Ġhas Ġbeen Ġexperiencing Ġthe Ġsymptom , Ġ( 2 ) Ġactivities Ġthat Ġtrigger Ġthe Ġsymptom Ġ( to Ġoccur Ġor Ġworsen ), Ġ( 3 ) Ġthe Ġextent Ġof Ġseriousness , Ġ( 4 ) Ġthe Ġfrequency Ġoccurrence Ġof Ġthe Ġsymptom , Ġand Ġ( 5 ) Ġthe Ġlocation Ġof Ġsymptom . ĠFor Ġeach Ġsymptom / attribute , Ġit Ġcan Ġtake Ġon Ġdifferent Ġlinguistic Ġexpressions , Ġdefined Ġas Ġentities . ĠNote Ġthat Ġif Ġthe Ġquer ied Ġsymptom Ġor Ġattribute Ġis Ġnot Ġmentioned Ġin Ġthe Ġdialogue , Ġthe Ġground truth Ġoutput Ġis ĠâĢ ľ No ĠAnswer âĢ Ŀ , Ġas Ġin ĠB IB REF 6 Ġ. </s> Reading ĠComp rehens ion </s> Large - scale Ġreading Ġcomprehension Ġtasks Ġlike ĠS Qu AD ĠB IB REF 0 Ġand ĠMAR CO ĠB IB REF 7 Ġprovide Ġquestion - answer Ġpairs Ġfrom Ġa Ġvast Ġrange Ġof Ġwritten Ġpassages , Ġcovering Ġdifferent Ġkinds Ġof Ġfactual Ġanswers Ġinvolving Ġentities Ġsuch Ġas Ġlocation Ġand Ġnumerical Ġvalues . ĠFurthermore , ĠHot pot Q A ĠB IB REF 8 Ġrequires Ġmulti - step Ġinference Ġand Ġprovides Ġnumerous Ġanswer Ġtypes . ĠCo Q A ĠB IB REF 9 Ġand ĠQu AC ĠB IB REF 10 Ġare Ġdesigned Ġto Ġmimic Ġmulti - turn Ġinformation - seeking Ġdiscussions Ġof Ġthe Ġgiven Ġmaterial . ĠIn Ġthese Ġtasks , Ġcontextual Ġreasoning Ġlike Ġcore ference Ġresolution Ġis Ġnecessary Ġto Ġgrasp Ġrich Ġlinguistic Ġpatterns , Ġencouraging Ġsemantic Ġmodeling Ġbeyond Ġnaive Ġlex ical Ġmatching . ĠNeural Ġnetworks Ġcontribute Ġto Ġimpressive Ġprogress Ġin Ġsemantic Ġmodeling : Ġdistribution al Ġsemantic Ġword Ġembed d ings ĠB IB REF 3 Ġ, Ġcontextual Ġsequence Ġencoding ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġand Ġthe Ġattention Ġmechanism ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġare Ġwidely Ġadopted Ġin Ġstate - of - the - art Ġcomprehension Ġmodels ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ. </s> While Ġlanguage Ġunderstanding Ġtasks Ġin Ġdialogue Ġsuch Ġas Ġdomain Ġidentification ĠB IB REF 15 Ġ, Ġslot Ġfilling ĠB IB REF 16 Ġand Ġuser Ġintent Ġdetection ĠB IB REF 17 Ġhave Ġattracted Ġmuch Ġresearch Ġinterest , Ġwork Ġin Ġdialogue Ġcomprehension Ġis Ġstill Ġlimited , Ġif Ġany . ĠIt Ġis Ġlabor - intensive Ġand Ġtime - consuming Ġto Ġobtain Ġa Ġcritical Ġmass Ġof Ġannot ated Ġconversation Ġdata Ġfor Ġcomputational Ġmodeling . ĠSome Ġpropose Ġto Ġcollect Ġtext Ġdata Ġfrom Ġhuman - machine Ġor Ġmachine - machine Ġdialog ues ĠB IB REF 18 Ġ, ĠB IB REF 5 Ġ. ĠIn Ġsuch Ġcases , Ġas Ġhuman Ġspeakers Ġare Ġaware Ġof Ġcurrent Ġlimitations Ġof Ġdialogue Ġsystems Ġor Ġdue Ġto Ġpre - defined Ġassumptions Ġof Ġuser Ġsim ulators , Ġthere Ġare Ġfewer Ġcases Ġof Ġzero Ġan aph ora , Ġthinking Ġaloud , Ġand Ġtopic Ġdrift , Ġwhich Ġoccur Ġmore Ġoften Ġin Ġhuman - human Ġspoken Ġinteractions . </s> N LP Ġfor ĠHealthcare </s> There Ġis Ġemerging Ġinterest Ġin Ġresearch Ġand Ġdevelopment Ġactivities Ġat Ġthe Ġintersection Ġof Ġmachine Ġlearning Ġand Ġhealthcare Ġ, Ġof Ġwhich Ġmuch Ġof Ġthe ĠN LP Ġrelated Ġwork Ġare Ġcentered Ġaround Ġsocial Ġmedia Ġor Ġonline Ġforums Ġ( e . g ., ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ), Ġpartially Ġdue Ġto Ġthe Ġworld Ġwide Ġweb Ġas Ġa Ġreadily Ġavailable Ġsource Ġof Ġinformation . ĠOther Ġwork Ġin Ġthis Ġarea Ġuses Ġpublic Ġdata Ġsources Ġsuch Ġas ĠM IM IC Ġin Ġelectronic Ġhealth Ġrecords : Ġtext Ġclassification Ġapproaches Ġhave Ġbeen Ġapplied Ġto Ġanalyze Ġun struct ured Ġclinical Ġnotes Ġfor ĠI CD Ġcode Ġassignment ĠB IB REF 21 Ġand Ġautomatic Ġintensive Ġemergency Ġprediction ĠB IB REF 22 Ġ. ĠSequence - to - sequence Ġtextual Ġgeneration Ġhas Ġbeen Ġused Ġfor Ġreadable Ġnotes Ġbased Ġon Ġmedical Ġand Ġdemographic Ġrecordings ĠB IB REF 23 Ġ. ĠFor Ġmental Ġhealth , Ġthere Ġhas Ġbeen Ġmore Ġfocus Ġon Ġanalyzing Ġdialog ues . ĠFor Ġexample , Ġsequential Ġmodeling Ġof Ġaudio Ġand Ġtext Ġhave Ġhelped Ġdetect Ġdepression Ġfrom Ġhuman - machine Ġinterviews ĠB IB REF 24 Ġ. ĠHowever , Ġfew Ġstudies Ġhave Ġexamined Ġhuman - human Ġspoken Ġconversations Ġin Ġhealthcare Ġsettings . </s> Data ĠPrepar ation </s> We Ġused Ġrecordings Ġof Ġnurse - in iti ated Ġtelephone Ġconversations Ġfor Ġcongest ive Ġheart Ġfailure Ġpatients Ġundergoing Ġtele monitor ing , Ġpost - dis charge Ġfrom Ġthe Ġhospital . ĠThe Ġclinical Ġdata Ġwas Ġacquired Ġby Ġthe ĠHealth ĠManagement ĠUnit Ġat ĠChang i ĠGeneral ĠHospital . ĠThis Ġresearch Ġstudy Ġwas Ġapproved Ġby Ġthe ĠSing Health ĠCentral ised ĠInst itutional ĠReview ĠBoard Ġ( Prot ocol Ġ155 656 15 15 ). ĠThe Ġpatients Ġwere Ġrecruited Ġduring Ġ2014 - 2016 Ġas Ġpart Ġof Ġtheir Ġroutine Ġcare Ġdelivery , Ġand Ġenrolled Ġinto Ġthe Ġtele monitor ing Ġhealth Ġmanagement Ġprogram Ġwith Ġconsent Ġfor Ġuse Ġof Ġanonym ized Ġversions Ġof Ġtheir Ġdata Ġfor Ġresearch . </s> The Ġdataset Ġcomprises Ġa Ġtotal Ġof Ġ353 Ġconversations Ġfrom Ġ40 Ġspeakers Ġ( 11 Ġnurses , Ġ16 Ġpatients , Ġand Ġ13 Ġcaregivers ) Ġwith Ġconsent Ġto Ġthe Ġuse Ġof Ġanonym ized Ġdata Ġfor Ġresearch . ĠThe Ġspeakers Ġare Ġ38 Ġto Ġ88 Ġyears Ġold , Ġequally Ġdistributed Ġacross Ġgender , Ġand Ġcomprise Ġa Ġrange Ġof Ġethnic Ġgroups Ġ( 55 % ĠChinese , Ġ17 % ĠMal ay , Ġ14 % ĠIndian , Ġ3 % ĠEuras ian , Ġand Ġ11 % Ġunspecified ). ĠThe Ġconversations Ġcover Ġ11 Ġtopics Ġ( e . g ., Ġmedication Ġcompliance , Ġsymptom Ġchecking , Ġeducation , Ġgreeting ) Ġand Ġ9 Ġsymptoms Ġ( e . g ., Ġchest Ġpain , Ġcough ) Ġand Ġamount Ġto Ġ41 Ġhours . </s> Data Ġpre processing Ġand Ġanonym ization Ġwere Ġperformed Ġby Ġa Ġdata Ġpreparation Ġteam , Ġseparate Ġfrom Ġthe Ġdata Ġanalysis Ġteam Ġto Ġmaintain Ġdata Ġconfidentiality . ĠThe Ġdata Ġpreparation Ġteam Ġfollowed Ġstandard Ġspeech Ġrecognition Ġtranscription Ġguidelines , Ġwhere Ġwords Ġare Ġtransc ribed Ġver bat im Ġto Ġinclude Ġfalse Ġstarts , Ġdis flu encies , Ġmis pron unci ations , Ġand Ġprivate Ġself - talk . ĠConf idential Ġinformation Ġwere Ġmarked Ġand Ġclipped Ġoff Ġfrom Ġthe Ġaudio Ġand Ġtransc ribed Ġwith Ġpred efined Ġtags Ġin Ġthe Ġannotation . ĠConversation Ġtopics Ġand Ġclinical Ġsymptoms Ġwere Ġalso Ġannot ated Ġand Ġclinically Ġvalidated Ġby Ġcertified Ġtele health Ġnurses . </s> L ingu istic ĠCharacter ization Ġon ĠSeed ĠData </s> To Ġanalyze Ġthe Ġlinguistic Ġstructure Ġof Ġthe Ġinquiry - response Ġpairs Ġin Ġthe Ġentire Ġ41 - hour Ġdataset , Ġwe Ġrandomly Ġsampled Ġa Ġseed Ġdataset Ġconsisting Ġof Ġ1 , 200 Ġturns Ġand Ġmanually Ġcategorized Ġthem Ġto Ġdifferent Ġtypes , Ġwhich Ġare Ġsummarized Ġin ĠTable ĠT AB REF 14 Ġalong Ġwith Ġthe Ġcorresponding Ġoccurrence Ġfrequency Ġstatistics . ĠNote Ġthat Ġeach Ġgiven Ġutter ance Ġcould Ġbe Ġcategorized Ġto Ġmore Ġthan Ġone Ġtype . ĠWe Ġelaborate Ġon Ġeach Ġutter ance Ġtype Ġbelow . </s> Open - ended ĠInquiry : ĠInqu iries Ġabout Ġgeneral Ġwell - being Ġor Ġa Ġparticular Ġsymptom ; Ġe . g ., ĠâĢ ľ How Ġare Ġyou Ġfeeling ? âĢ Ŀ Ġand ĠâĢ ľ Do Ġyou Ġcough ? âĢ Ŀ </s> Detailed ĠInquiry : ĠInqu iries Ġwith Ġspecific Ġdetails Ġthat Ġprompt Ġyes / no Ġanswers Ġor Ġclar ifications ; Ġe . g ., ĠâĢ ľ Do Ġyou Ġcough Ġat Ġnight ? âĢ Ŀ </s> Multi - Int ent ĠInquiry : ĠInqu iring Ġmore Ġthan Ġone Ġsymptom Ġin Ġa Ġquestion ; Ġe . g ., ĠâĢ ľ Any Ġcough , Ġchest Ġpain , Ġor Ġheadache ? âĢ Ŀ </s> Rec on f irmation ĠInquiry : ĠThe Ġnurse Ġrecon f irms Ġparticular Ġdetails ; Ġe . g ., ĠâĢ ľ Really ? ĠAt Ġnight ? âĢ Ŀ Ġand ĠâĢ ľ Ser ious Ġor Ġmild ? âĢ Ŀ . ĠThis Ġcase Ġis Ġusually Ġrelated Ġto Ġexplicit Ġor Ġimplicit Ġcore f eren cing . </s> In qu iry Ġwith ĠTrans itional ĠCla uses : ĠDuring Ġspoken Ġconversations , Ġone Ġmight Ġrepeat Ġwhat Ġthe Ġother Ġparty Ġsaid , Ġbut Ġit Ġis Ġunrelated Ġto Ġthe Ġmain Ġclause Ġof Ġthe Ġquestion . ĠThis Ġis Ġusually Ġdue Ġto Ġprivate Ġself - talk Ġwhile Ġthinking Ġaloud , Ġand Ġsuch Ġutter ances Ġform Ġa Ġtransitional Ġclause Ġbefore Ġthe Ġspeaker Ġstarts Ġa Ġnew Ġtopic ; Ġe . g ., ĠâĢ ľ Chest Ġpain ... Ġno Ġchest Ġpain , ĠI Ġsee ... Ġany Ġcough ? âĢ Ŀ . </s> Yes / No ĠResponse : ĠYes / No Ġresponses Ġseem Ġstraightforward , Ġbut Ġsometimes Ġlead Ġto Ġmisunderstanding Ġif Ġone Ġdoes Ġnot Ġinterpret Ġthe Ġcontext Ġappropriately . ĠOne Ġcase Ġis Ġtag Ġquestions : ĠA : âĢ ľ You Ġdon 't Ġcough Ġat Ġnight , Ġdo Ġyou ? âĢ Ŀ ĠB : ` Yes , Ġyes âĢ Ŀ ĠA : âĢ ľ c ough Ġat Ġnight ? âĢ Ŀ ĠB : âĢ ľ No , Ġno Ġcough âĢ Ŀ . ĠUsually Ġwhen Ġthe Ġanswer Ġis Ġunclear , Ġclar ifying Ġinquiries Ġwill Ġbe Ġasked Ġfor Ġrecon f irmation Ġpurposes . </s> Detailed ĠResponse : ĠRespons es Ġthat Ġcontain Ġspecific Ġinformation Ġof Ġone Ġsymptom , Ġlike ĠâĢ ľ I Ġfelt Ġtight ness Ġin Ġmy Ġchest âĢ Ŀ . </s> Response Ġwith ĠRevision : ĠRevision Ġis Ġinf requent Ġbut Ġcan Ġaffect Ġcomprehension Ġsignificantly . ĠOne Ġcause Ġis Ġthinking Ġaloud Ġso Ġa Ġlater Ġresponse Ġoverr ules Ġthe Ġprevious Ġone ; Ġe . g ., ĠâĢ ľ No Ġdizz iness , Ġoh Ġwait ... Ġlast Ġweek ĠI Ġfelt Ġa Ġbit Ġdizz y Ġwhen Ġbiking âĢ Ŀ . </s> Response Ġwith ĠTopic ĠDrift : ĠWhen Ġa Ġsymptom / topic Ġlike Ġheadache Ġis Ġinquired , Ġthe Ġresponse Ġmight Ġbe : ĠâĢ ľ Only Ġsome Ġchest Ġpain Ġat Ġnight âĢ Ŀ , Ġnot Ġreferring Ġto Ġthe Ġoriginal Ġsymptom Ġ( head ache ) Ġat Ġall . </s> Response Ġwith ĠTrans itional ĠCla uses : ĠRepe ating Ġsome Ġof Ġthe Ġprevious Ġcontent , Ġbut Ġoften Ġunrelated Ġto Ġcritical Ġclinical Ġinformation Ġand Ġusually Ġfollowed Ġby Ġtopic Ġdrift . ĠFor Ġexample , ĠâĢ ľ S well ing ... Ġswelling ... ĠI Ġdon 't Ġcough Ġat Ġnight âĢ Ŀ . </s> Sim ulating ĠSym pt om ĠMonitoring ĠDat as et Ġfor ĠTraining </s> We Ġdivide Ġthe Ġconstruction Ġof Ġdata Ġsimulation Ġinto Ġtwo Ġstages . ĠIn ĠSection ĠSEC REF 16 Ġ, Ġwe Ġbuild Ġtemplates Ġand Ġexpression Ġpools Ġusing Ġlinguistic Ġanalysis Ġfollowed Ġby Ġmanual Ġverification . ĠIn ĠSection ĠSEC REF 20 Ġ, Ġwe Ġpresent Ġour Ġproposed Ġframework Ġfor Ġgenerating Ġsimulated Ġtraining Ġdata . ĠThe Ġtemplates Ġand Ġframework Ġare Ġverified Ġfor Ġlogical Ġcorrectness Ġand Ġclinical Ġsound ness . </s> Template ĠConstruction </s> Each Ġutter ance Ġin Ġthe Ġseed Ġdata Ġis Ġcategorized Ġaccording Ġto ĠTable ĠT AB REF 14 Ġand Ġthen Ġabstract ed Ġinto Ġtemplates Ġby Ġreplacing Ġentity Ġphrases Ġlike Ġcough Ġand Ġoften Ġwith Ġrespective Ġplace holders ĠâĢ ľ # sym pt om # âĢ Ŀ Ġand ĠâĢ ľ # frequency # âĢ Ŀ . ĠThe Ġtemplates Ġare Ġrefined Ġthrough Ġverifying Ġlogical Ġcorrectness Ġand Ġinjecting Ġexpression Ġdiversity Ġby Ġlingu istically Ġtrained Ġresearchers . ĠAs Ġthese Ġreplacements Ġdo Ġnot Ġalter Ġthe Ġsynt actic Ġstructure , Ġwe Ġinterchange Ġsuch Ġplace holders Ġwith Ġvarious Ġverbal Ġexpressions Ġto Ġenlarge Ġthe Ġsimulated Ġtraining Ġset Ġin ĠSection ĠSEC REF 20 Ġ. ĠClinical Ġvalidation Ġwas Ġalso Ġconducted Ġby Ġcertified Ġtele health Ġnurses . </s> For Ġthe Ġ9 Ġsymptoms Ġ( e . g . Ġchest Ġpain , Ġcough ) Ġand Ġ5 Ġattributes Ġ( e . g ., Ġextent , Ġfrequency ), Ġwe Ġcollect Ġvarious Ġexpressions Ġfrom Ġthe Ġseed Ġdata , Ġand Ġexpand Ġthem Ġthrough Ġsyn onym Ġreplacement . ĠSome Ġattributes Ġare Ġunique Ġto Ġa Ġparticular Ġsymptom ; Ġe . g ., ĠâĢ ľ left Ġleg âĢ Ŀ Ġin Ġ# location # Ġis Ġonly Ġsuitable Ġto Ġdescribe Ġthe Ġsymptom Ġswelling , Ġbut Ġnot Ġthe Ġsymptom Ġheadache . ĠTherefore , Ġwe Ġonly Ġreuse Ġgeneral Ġexpressions Ġlike ĠâĢ ľ s light âĢ Ŀ Ġin Ġ# ext ent # Ġacross Ġdifferent Ġsymptoms Ġto Ġdivers ify Ġlinguistic Ġexpressions . </s> Two Ġlingu istically Ġtrained Ġresearchers Ġconstructed Ġexpression Ġpools Ġfor Ġeach Ġsymptom Ġand Ġeach Ġattribute Ġto Ġaccount Ġfor Ġdifferent Ġtypes Ġof Ġparaph r asing Ġand Ġdescriptions . ĠThese Ġexpression Ġpools Ġare Ġused Ġin ĠSection ĠSEC REF 20 Ġ( c ). </s> Sim ulated ĠData ĠGeneration ĠFramework </s> Figure ĠFIG REF 15 Ġshows Ġthe Ġfive Ġsteps Ġwe Ġuse Ġto Ġgenerate Ġmulti - turn Ġsymptom Ġmonitoring Ġdialogue Ġsamples . </s> ( a ) ĠTopic ĠSelection : ĠWhile Ġnurses Ġmight Ġprefer Ġto Ġinquire Ġthe Ġsymptoms Ġin Ġdifferent Ġorders Ġdepending Ġon Ġthe Ġpatient 's Ġhistory , Ġour Ġpreliminary Ġanalysis Ġshows Ġthat Ġmodeling Ġresults Ġdo Ġnot Ġdiffer Ġnoticeably Ġif Ġtopics Ġare Ġof Ġequal Ġprior Ġprobabilities . ĠThus Ġwe Ġadopt Ġthis Ġassumption Ġfor Ġsimplicity . </s> ( b ) ĠTemplate ĠSelection : ĠFor Ġeach Ġselected Ġtopic , Ġone Ġinquiry Ġtemplate Ġand Ġone Ġresponse Ġtemplate Ġare Ġrandomly Ġchosen Ġto Ġcompose Ġa Ġturn . ĠTo Ġminimize Ġadverse Ġeffects Ġof Ġunder fitting , Ġwe Ġredistributed Ġthe Ġfrequency Ġdistribution Ġin ĠTable ĠT AB REF 14 Ġ: ĠFor Ġutter ance Ġtypes Ġthat Ġare Ġbelow Ġ15 %, Ġwe Ġboosted Ġthem Ġto Ġ15 %, Ġand Ġthe Ġoverall Ġrelative Ġdistribution Ġranking Ġis Ġbalanced Ġand Ġconsistent Ġwith ĠTable ĠT AB REF 14 Ġ. </s> ( c ) ĠEn rich ing ĠL ingu istic ĠExpress ions : ĠThe Ġplace holders Ġin Ġthe Ġselected Ġtemplates Ġare Ġsubstituted Ġwith Ġdiverse Ġexpressions Ġfrom Ġthe Ġexpression Ġpools Ġin ĠSection ĠUID 19 Ġto Ġcharacterize Ġthe Ġsymptoms Ġand Ġtheir Ġcorresponding Ġattributes . </s> ( d ) ĠMulti - Turn ĠDialogue ĠState ĠTracking : ĠA Ġgreedy Ġalgorithm Ġis Ġapplied Ġto Ġcomplete Ġconversations . ĠA ĠâĢ ľ com pleted Ġsymptoms âĢ Ŀ Ġlist Ġand Ġa ĠâĢ ľ to - do Ġsymptoms âĢ Ŀ Ġlist Ġare Ġused Ġfor Ġsymptom Ġtopic Ġtracking . ĠWe Ġalso Ġtrack Ġthe ĠâĢ ľ com pleted Ġattributes " Ġand ĠâĢ ľ to - do Ġattributes ". ĠFor Ġeach Ġsymptom , Ġall Ġrelated Ġattributes Ġare Ġiter ated . ĠA Ġdialogue Ġends Ġonly Ġwhen Ġall Ġpossible Ġentities Ġare Ġexhausted , Ġgenerating Ġa Ġmulti - turn Ġdialogue Ġsample , Ġwhich Ġencourages Ġthe Ġmodel Ġto Ġlearn Ġfrom Ġthe Ġentire Ġdiscussion Ġflow Ġrather Ġthan Ġa Ġsingle Ġturn Ġto Ġcomprehend Ġcontextual Ġdependency . ĠThe Ġaverage Ġlength Ġof Ġa Ġsimulated Ġdialogue Ġis Ġ184 Ġwords , Ġwhich Ġhappens Ġto Ġbe Ġtwice Ġas Ġlong Ġas Ġan Ġaverage Ġdialogue Ġfrom Ġthe Ġreal - world Ġevaluation Ġset . ĠMoreover , Ġto Ġmodel Ġthe Ġroles Ġof Ġthe Ġrespondents , Ġwe Ġset Ġthe Ġratio Ġbetween Ġpatients Ġand Ġcaregivers Ġto Ġ2 : 1 ; Ġthis Ġstatistic Ġis Ġinspired Ġby Ġthe Ġreal Ġscenarios Ġin Ġthe Ġseed Ġdataset . ĠFor Ġboth Ġthe Ġcaregivers Ġand Ġpatients , Ġwe Ġassume Ġequal Ġprobability Ġof Ġboth Ġgenders . ĠThe Ġcorresponding Ġpronouns Ġin Ġthe Ġconversations Ġare Ġthus Ġdetermined Ġby Ġthe Ġrole Ġand Ġgender Ġof Ġthese Ġsettings . </s> ( e ) ĠMulti - Turn ĠSample ĠAn notation : ĠFor Ġeach Ġmulti - turn Ġdialogue , Ġa Ġquery Ġis Ġspecified Ġby Ġa Ġsymptom Ġand Ġan Ġattribute . ĠThe Ġground truth Ġoutput Ġof Ġthe ĠQ A Ġsystem Ġis Ġautomatically Ġlabeled Ġbased Ġon Ġthe Ġtemplate Ġgeneration Ġrules , Ġbut Ġalso Ġmanually Ġverified Ġto Ġensure Ġannotation Ġquality . ĠMoreover , Ġwe Ġadopt Ġthe Ġun answer able Ġdesign Ġin ĠB IB REF 6 Ġ: Ġwhen Ġthe Ġpatient Ġdoes Ġnot Ġmention Ġa Ġparticular Ġsymptom , Ġthe Ġanswer Ġis Ġdefined Ġas ĠâĢ ľ No ĠAnswer âĢ Ŀ . ĠThis Ġprocess Ġis Ġrepeated Ġuntil Ġall Ġlogical Ġperm utations Ġof Ġsymptoms Ġand Ġattributes Ġare Ġexhausted . </s> Exper iments </s> Model ĠDesign </s> We Ġimplemented Ġan Ġestablished Ġmodel Ġin Ġreading Ġcomprehension , Ġa Ġbi - direction al Ġattention Ġpointer Ġnetwork ĠB IB REF 1 Ġ, Ġand Ġequipped Ġit Ġwith Ġan Ġanswer able Ġclass ifier , Ġas Ġdepicted Ġin ĠFigure ĠFIG REF 21 Ġ. ĠFirst , Ġtokens Ġin Ġthe Ġgiven Ġdialogue ĠIN LINE FORM 0 Ġand Ġquery ĠIN LINE FORM 1 Ġare Ġconverted Ġinto Ġembed ding Ġvectors . ĠThen Ġthe Ġdialogue Ġembed d ings Ġare Ġfed Ġto Ġa Ġbi - direction al ĠL ST M Ġencoding Ġlayer , Ġgenerating Ġa Ġsequence Ġof Ġcontextual Ġhidden Ġstates . ĠNext , Ġthe Ġhidden Ġstates Ġand Ġquery Ġembed d ings Ġare Ġprocessed Ġby Ġa Ġbi - direction al Ġattention Ġlayer , Ġf using Ġattention Ġinformation Ġin Ġboth Ġcontext - to - query Ġand Ġquery - to - context Ġdirections . ĠThe Ġfollowing Ġtwo Ġbi - direction al ĠL ST M Ġmodeling Ġlayers Ġread Ġthe Ġcontextual Ġsequence Ġwith Ġattention . ĠFinally , Ġtwo Ġrespective Ġlinear Ġlayers Ġwith Ġsoft max Ġfunctions Ġare Ġused Ġto Ġestimate Ġtoken ĠIN LINE FORM 2 Ġ' s ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġprobability Ġof Ġthe Ġanswer Ġspan ĠIN LINE FORM 5 Ġ. </s> In Ġaddition , Ġwe Ġadd Ġa Ġspecial Ġtag ĠâĢ ľ [ SE Q ] âĢ Ŀ Ġat Ġthe Ġhead Ġof ĠIN LINE FORM 0 Ġto Ġaccount Ġfor Ġthe Ġcase Ġof ĠâĢ ľ No Ġanswer âĢ Ŀ ĠB IB REF 4 Ġand Ġadopt Ġan Ġanswer able Ġclass ifier Ġas Ġin ĠB IB REF 25 Ġ. ĠMore Ġspecifically , Ġwhen Ġthe Ġquer ied Ġsymptom Ġor Ġattribute Ġis Ġnot Ġmentioned Ġin Ġthe Ġdialogue , Ġthe Ġanswer Ġspan Ġshould Ġpoint Ġto Ġthe Ġtag ĠâĢ ľ [ SE Q ] âĢ Ŀ Ġand Ġanswer able Ġprobability Ġshould Ġbe Ġpredicted Ġas Ġ0 . </s> Im plementation ĠDetails </s> The Ġmodel Ġwas Ġtrained Ġvia Ġgradient Ġback prop ag ation Ġwith Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġof Ġanswer Ġspan Ġprediction Ġand Ġanswer able Ġclassification , Ġoptimized Ġby ĠAdam Ġalgorithm ĠB IB REF 26 Ġwith Ġinitial Ġlearning Ġrate Ġof ĠIN LINE FORM 0 Ġ. ĠPre - trained ĠGl o Ve ĠB IB REF 3 Ġembed d ings Ġ( size ĠIN LINE FORM 1 Ġ) Ġwere Ġused . ĠWe Ġre - sh uff led Ġtraining Ġsamples Ġat Ġeach Ġepoch Ġ( batch Ġsize ĠIN LINE FORM 2 Ġ). ĠOut - of - voc abulary Ġwords Ġ( ĠIN LINE FORM 3 Ġ) Ġwere Ġreplaced Ġwith Ġa Ġfixed Ġrandom Ġvector . ĠL 2 Ġregular ization Ġand Ġdrop out Ġ( rate ĠIN LINE FORM 4 Ġ) Ġwere Ġused Ġto Ġalleviate Ġover fitting ĠB IB REF 27 Ġ. </s> E val uation ĠSetup </s> To Ġevaluate Ġthe Ġeffectiveness Ġof Ġour Ġlingu istically - inspired Ġsimulation Ġapproach , Ġthe Ġmodel Ġis Ġtrained Ġon Ġthe Ġsimulated Ġdata Ġ( see ĠSection ĠSEC REF 20 Ġ). ĠWe Ġdesigned Ġ3 Ġevaluation Ġsets : Ġ( 1 ) ĠBase ĠSet Ġ( 1 , 264 Ġsamples ) Ġheld Ġout Ġfrom Ġthe Ġsimulated Ġdata . Ġ( 2 ) ĠAug mented ĠSet Ġ( 1 , 280 Ġsamples ) Ġbuilt Ġby Ġadding Ġtwo Ġout - of - dist ribution Ġsymptoms , Ġwith Ġcorresponding Ġdialogue Ġcontents Ġand Ġqueries , Ġto Ġthe ĠBase ĠSet Ġ( âĢ ľ ble eding âĢ Ŀ Ġand ĠâĢ ľ cold âĢ Ŀ , Ġwhich Ġnever Ġappeared Ġin Ġtraining Ġdata ). Ġ( 3 ) ĠReal - World ĠSet Ġ( 9 44 Ġsamples ) Ġmanually Ġdeline ated Ġfrom Ġthe Ġthe Ġsymptom Ġchecking Ġportions Ġ( approximately Ġ4 Ġhours ) Ġof Ġreal - world Ġdialog ues , Ġand Ġannot ated Ġas Ġevaluation Ġsamples . </s> Results </s> E val uation Ġresults Ġare Ġin ĠTable ĠT AB REF 25 Ġwith Ġexact Ġmatch Ġ( EM ) Ġand ĠF 1 Ġscore Ġin ĠB IB REF 0 Ġmetrics . ĠTo Ġdistinguish Ġthe Ġcorrect Ġanswer Ġspan Ġfrom Ġthe Ġplausible Ġones Ġwhich Ġcontain Ġthe Ġsame Ġwords , Ġwe Ġmeasure Ġthe Ġscores Ġon Ġthe Ġposition Ġindices Ġof Ġtokens . ĠOur Ġresults Ġshow Ġthat Ġboth ĠEM Ġand ĠF 1 Ġscore Ġincrease Ġwith Ġtraining Ġsample Ġsize Ġgrowing Ġand Ġthe Ġoptimal Ġsize Ġin Ġour Ġsetting Ġis Ġ100 k . ĠThe Ġbest - trained Ġmodel Ġperforms Ġwell Ġon Ġboth Ġthe ĠBase ĠSet Ġand Ġthe ĠAug mented ĠSet , Ġindicating Ġthat Ġout - of - dist ribution Ġsymptoms Ġdo Ġnot Ġaffect Ġthe Ġcomprehension Ġof Ġexisting Ġsymptoms Ġand Ġoutputs Ġreasonable Ġanswers Ġfor Ġboth Ġin - Ġand Ġout - of - dist ribution Ġsymptoms . ĠOn Ġthe ĠReal - World ĠSet , Ġwe Ġobtained Ġ78 . 23 ĠEM Ġscore Ġand Ġ80 . 18 ĠF 1 Ġscore Ġrespectively . </s> Error Ġanalysis Ġsuggests Ġthe Ġperformance Ġdrop Ġfrom Ġthe Ġsimulated Ġtest Ġsets Ġis Ġdue Ġto Ġthe Ġfollowing : Ġ1 ) Ġsp arsity Ġissues Ġresulting Ġfrom Ġthe Ġexpression Ġpools Ġexcluding Ġvarious Ġvalid Ġbut Ġsporadic Ġexpressions . Ġ2 ) Ġnurses Ġand Ġpatients Ġoccasionally Ġch it - chat Ġin Ġthe ĠReal - World ĠSet , Ġwhich Ġis Ġnot Ġsimulated Ġin Ġthe Ġtraining Ġset . ĠAt Ġtimes , Ġthese Ġch it - ch ats Ġmake Ġthe Ġconversations Ġoverly Ġlengthy , Ġcausing Ġthe Ġinformation Ġdensity Ġto Ġbe Ġlower . ĠThese Ġissues Ġcould Ġpotentially Ġdistract Ġand Ġconfuse Ġthe Ġcomprehension Ġmodel . Ġ3 ) Ġan Ġinteresting Ġtype Ġof Ġinf requent Ġerror Ġsource , Ġcaused Ġby Ġpatients Ġelabor ating Ġon Ġpossible Ġcausal Ġrelations Ġof Ġtwo Ġsymptoms . ĠFor Ġexample , Ġa Ġpatient Ġmight Ġsay ĠâĢ ľ My Ġg idd iness Ġmay Ġbe Ġdue Ġto Ġall Ġthis Ġcough âĢ Ŀ . ĠWe Ġare Ġcurrently Ġinvestigating Ġhow Ġto Ġclose Ġthis Ġperformance Ġgap Ġefficiently . </s> A bl ation ĠAnalysis </s> To Ġassess Ġthe Ġeffectiveness Ġof Ġbi - direction al Ġattention , Ġwe Ġbypass ed Ġthe Ġbi - att ention Ġlayer Ġby Ġdirectly Ġfeeding Ġthe Ġcontextual Ġhidden Ġstates Ġand Ġquery Ġembed d ings Ġto Ġthe Ġmodeling Ġlayer . ĠTo Ġevaluate Ġthe Ġpre - trained ĠGl o Ve Ġembed d ings , Ġwe Ġrandomly Ġinitialized Ġand Ġtrained Ġthe Ġembed d ings Ġfrom Ġscratch . ĠThese Ġtwo Ġprocedures Ġlead Ġto Ġ10 % Ġand Ġ18 % Ġperformance Ġdegradation Ġon Ġthe ĠAug mented ĠSet Ġand ĠReal - World ĠSet , Ġrespectively Ġ( see ĠTable ĠT AB REF 27 Ġ). </s> Conclusion </s> We Ġformulated Ġa Ġdialogue Ġcomprehension Ġtask Ġmotivated Ġby Ġthe Ġneed Ġin Ġtele health Ġsettings Ġto Ġextract Ġkey Ġclinical Ġinformation Ġfrom Ġspoken Ġconversations Ġbetween Ġnurses Ġand Ġpatients . ĠWe Ġanalyzed Ġlinguistic Ġcharacteristics Ġof Ġreal - world Ġhuman - human Ġsymptom Ġchecking Ġdialog ues , Ġconstructed Ġa Ġsimulated Ġdataset Ġbased Ġon Ġlingu istically Ġinspired Ġand Ġclinically Ġvalidated Ġtemplates , Ġand Ġprototyp ed Ġa ĠQ A Ġsystem . ĠThe Ġmodel Ġworks Ġeffectively Ġon Ġa Ġsimulated Ġtest Ġset Ġusing Ġsymptoms Ġexcluded Ġduring Ġtraining Ġand Ġon Ġreal - world Ġconversations Ġbetween Ġnurses Ġand Ġpatients . ĠWe Ġare Ġcurrently Ġimproving Ġthe Ġmodel 's Ġdialogue Ġcomprehension Ġcapability Ġin Ġcomplex Ġreasoning Ġand Ġcontext Ġunderstanding Ġand Ġalso Ġapplying Ġthe ĠQ A Ġmodel Ġto Ġsummar ization Ġand Ġvirtual Ġnurse Ġapplications . </s> Acknowled gements </s> Research Ġefforts Ġwere Ġsupported Ġby Ġfunding Ġfor ĠDigital ĠHealth Ġand ĠDeep ĠLearning ĠI 2 R Ġ( DL 2 ĠSS F ĠProject ĠNo : ĠA 17 18 g 00 45 ) Ġand Ġthe ĠScience Ġand ĠEngineering ĠResearch ĠCouncil Ġ( SER C ĠProject ĠNo : ĠA 18 18 g 00 44 ), ĠA * STAR , ĠSingapore . ĠIn Ġaddition , Ġthis Ġwork Ġwas Ġconducted Ġusing Ġresources Ġand Ġinfrastructure Ġprovided Ġby Ġthe ĠHuman ĠLanguage ĠTechnology Ġunit Ġat ĠI 2 R . ĠThe Ġtele health Ġdata Ġacquisition Ġwas Ġfunded Ġby Ġthe ĠEconomic ĠDevelopment ĠBoard Ġ( ED B ), ĠSingapore ĠLiving ĠLab ĠFund Ġand ĠPhilips ĠElectronics ĠâĢĵ ĠHospital Ġto ĠHome ĠPilot ĠProject Ġ( ED B Ġgrant Ġreference Ġnumber : ĠS 14 - 10 35 - RF - LL F ĠH Ġand ĠW ). </s> We Ġacknowledge Ġvaluable Ġsupport Ġand Ġassistance Ġfrom ĠY ul ong ĠLin , ĠYuan x in ĠXiang , Ġand ĠAi ĠTi ĠAw Ġat Ġthe ĠInstitute Ġfor ĠInf oc omm ĠResearch Ġ( I 2 R ); ĠWe ili ang ĠHuang Ġat Ġthe ĠChang i ĠGeneral ĠHospital Ġ( C GH ) ĠDepartment Ġof ĠCard iology , ĠHong ĠCh oon ĠOh Ġat ĠC GH ĠHealth ĠServices ĠResearch , Ġand ĠEd ris ĠAt ik ah ĠB te ĠAhmad , ĠCh iu ĠYan ĠAng , Ġand ĠMei ĠF oon ĠY ap Ġof Ġthe ĠC GH ĠIntegrated ĠCare ĠDepartment . </s> We Ġalso Ġthank ĠEdu ard ĠH ovy Ġand ĠBonnie ĠWeb ber Ġfor Ġinsightful Ġdiscussions Ġand Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġprecious Ġfeedback Ġto Ġhelp Ġimprove Ġand Ġextend Ġthis Ġpiece Ġof Ġwork .