There is nothing specific about the approach that depends on medical recommendations. The approach combines graph data and text data into a single embedding.
ABSTRACTIVE
It learns a representation of medical records. The learned representation (embeddings) can be used for other predictive tasks involving information from electronic health records.
ABSTRACTIVE
FLOAT SELECTED: Figure 2: The framework of G-BERT. It consists of three main parts: ontology embedding, BERT and fine-tuned classifier. Firstly, we derive ontology embedding for medical code laid in leaf nodes by cooperating ancestors information by Eq. 1 and 2 based on graph attention networks (Eq. 3, 4). Then we input set of diagnosis and medication ontology embedding separately to shared weight BERT which is pretrained using Eq. 6, 7, 8. Finally, we concatenate the mean of all previous visit embeddings and the last visit embedding as input and fine-tune the prediction layers using Eq. 10 for medication recommendation tasks.
In this paper we proposed a pre-training model named G-BERT for medical code representation and medication recommendation. To our best knowledge, G-BERT is the first that utilizes language model pre-training techniques in healthcare domain. It adapted BERT to the EHR data and integrated medical ontology information using graph neural networks. By additional pre-training on the EHR from patients who only have one hospital visit which are generally discarded before model training, G-BERT outperforms all baselines in prediction accuracy on medication recommendation task. One direction for the future work is to add more auxiliary and structural tasks to improve the ability of code representaion. Another direction may be to adapt our model to be suitable for even larger datasets with more heterogeneous modalities.
1906.00346
Introduction </s> The Ġavailability Ġof Ġmassive Ġelectronic Ġhealth Ġrecords Ġ( E HR ) Ġdata Ġand Ġthe Ġadvances Ġof Ġdeep Ġlearning Ġtechnologies Ġhave Ġprovided Ġunprecedented Ġresource Ġand Ġopportunity Ġfor Ġpredictive Ġhealthcare , Ġincluding Ġthe Ġcomputational Ġmedication Ġrecommendation Ġtask . ĠA Ġnumber Ġof Ġdeep Ġlearning Ġmodels Ġwere Ġproposed Ġto Ġassist Ġdoctors Ġin Ġmaking Ġmedication Ġrecommendation ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠThey Ġoften Ġlearn Ġrepresentations Ġfor Ġmedical Ġentities Ġ( e . g ., Ġpatients , Ġdiagnosis , Ġmedications ) Ġfrom Ġpatient ĠE HR Ġdata , Ġand Ġthen Ġuse Ġthe Ġlearned Ġrepresentations Ġto Ġpredict Ġmedications Ġthat Ġare Ġsuited Ġto Ġthe Ġpatient 's Ġhealth Ġcondition . </s> To Ġprovide Ġeffective Ġmedication Ġrecommendation , Ġit Ġis Ġimportant Ġto Ġlearn Ġaccurate Ġrepresentation Ġof Ġmedical Ġcodes . ĠDespite Ġthat Ġvarious Ġconsiderations Ġwere Ġhandled Ġin Ġprevious Ġworks Ġfor Ġimproving Ġmedical Ġcode Ġrepresentations ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthere Ġare Ġtwo Ġlimitations Ġwith Ġthe Ġexisting Ġwork : </s> To Ġmitigate Ġthe Ġaforementioned Ġlimitations , Ġwe Ġpropose ĠG - BER T Ġthat Ġcombines Ġthe Ġpre - training Ġtechniques Ġand Ġgraph Ġneural Ġnetworks Ġfor Ġbetter Ġmedical Ġcode Ġrepresentation Ġand Ġmedication Ġrecommendation . ĠG - BER T Ġis Ġenabled Ġand Ġdemonstrated Ġby Ġthe Ġfollowing Ġtechnical Ġcontributions : </s> Related ĠWork </s> Med ication ĠRecommend ation ĠMed ication ĠRecommend ation Ġcan Ġbe Ġcategorized Ġinto Ġinstance - based Ġand Ġlongitudinal Ġrecommendation Ġmethods ĠB IB REF 1 Ġ. ĠInst ance - based Ġmethods Ġfocus Ġon Ġcurrent Ġhealth Ġconditions . ĠAmong Ġthem , ĠLeap ĠB IB REF 9 Ġform ulates Ġa Ġmulti - instance Ġmulti - label Ġlearning Ġframework Ġand Ġproposes Ġa Ġvariant Ġof Ġsequence - to - sequence Ġmodel Ġbased Ġon Ġcontent - att ention Ġmechanism Ġto Ġpredict Ġcombination Ġof Ġmedicines Ġgiven Ġpatient 's Ġdiagnoses . ĠLong itudinal - based Ġmethods Ġleverage Ġthe Ġtemporal Ġdependencies Ġamong Ġclinical Ġevents , Ġsee ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠAmong Ġthem , ĠR ETA IN ĠB IB REF 10 Ġuses Ġa Ġtwo - level Ġneural Ġattention Ġmodel Ġto Ġdetect Ġinfluential Ġpast Ġvisits Ġand Ġsignificant Ġclinical Ġvariables Ġwithin Ġthose Ġvisits Ġfor Ġimproved Ġmedication Ġrecommendation . </s> Pre - training ĠTechniques ĠThe Ġgoal Ġof Ġpre - training Ġtechniques Ġis Ġto Ġprovide Ġmodel Ġtraining Ġwith Ġgood Ġinitial izations . ĠPre - training Ġhas Ġbeen Ġshown Ġextremely Ġeffective Ġin Ġvarious Ġareas Ġsuch Ġas Ġimage Ġclassification ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġmachine Ġtranslation ĠB IB REF 15 Ġ. ĠThe Ġun super vised Ġpre - training Ġcan Ġbe Ġconsidered Ġas Ġa Ġregular izer Ġthat Ġsupports Ġbetter Ġgeneral ization Ġfrom Ġthe Ġtraining Ġdataset ĠB IB REF 16 Ġ. ĠRecently , Ġlanguage Ġmodel Ġpre - training Ġtechniques Ġsuch Ġas ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġhave Ġshown Ġto Ġlargely Ġimprove Ġthe Ġperformance Ġon Ġmultiple ĠN LP Ġtasks . ĠAs Ġthe Ġmost Ġwidely Ġused Ġone , ĠB ERT ĠB IB REF 7 Ġbuilds Ġon Ġthe ĠTrans former ĠB IB REF 8 Ġarchitecture Ġand Ġimproves Ġthe Ġpre - training Ġusing Ġa Ġmasked Ġlanguage Ġmodel Ġfor Ġbid irection al Ġrepresentation . ĠIn Ġthis Ġpaper , Ġwe Ġadapt Ġthe Ġframework Ġof ĠB ERT Ġand Ġpre - train Ġour Ġmodel Ġon Ġeach Ġvisit Ġof Ġthe ĠE HR Ġdata Ġto Ġleverage Ġthe Ġsingle - vis it Ġdata Ġthat Ġwere Ġnot Ġfit Ġfor Ġmodel Ġtraining Ġin Ġother Ġmedication Ġrecommendation Ġmodels . </s> Graph ĠNeural ĠNetworks Ġ( G NN ) ĠG NN s Ġare Ġneural Ġnetworks Ġthat Ġlearn Ġnode Ġor Ġgraph Ġrepresentations Ġfrom Ġgraph - struct ured Ġdata . ĠVarious Ġgraph Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġto Ġencode Ġthe Ġgraph - st ructure Ġinformation , Ġincluding Ġgraph Ġconv olution al Ġneural Ġnetworks Ġ( GC N ) ĠB IB REF 17 Ġ, Ġmessage Ġpassing Ġnetworks Ġ( MP NN ) ĠB IB REF 18 Ġ, Ġgraph Ġattention Ġnetworks Ġ( G AT ) ĠB IB REF 19 Ġ. ĠG NN s Ġhave Ġalready Ġbeen Ġdemonstrated Ġuseful Ġon ĠE HR Ġmodeling ĠB IB REF 20 Ġ, ĠB IB REF 1 Ġ. ĠGR AM ĠB IB REF 20 Ġrepresented Ġa Ġmedical Ġconcept Ġas Ġa Ġcombination Ġof Ġits Ġancestors Ġin Ġthe Ġmedical Ġont ology Ġusing Ġan Ġattention Ġmechanism . ĠIt 's Ġdifferent Ġfrom ĠG - BER T Ġfrom Ġtwo Ġaspects Ġas Ġdescribed Ġin ĠSection Ġ" Input ĠRepresent ation " Ġ. ĠAnother Ġwork Ġworth Ġmentioning Ġis ĠGAM EN et ĠB IB REF 1 Ġ, Ġwhich Ġalso Ġused Ġgraph Ġneural Ġnetwork Ġto Ġassist Ġthe Ġmedication Ġrecommendation Ġtask . ĠHowever , ĠGAM EN et Ġhas Ġa Ġdifferent Ġmotivation Ġwhich Ġresults Ġin Ġusing Ġgraph Ġneural Ġnetworks Ġon Ġdrug - drug - inter action Ġgraphs Ġinstead Ġof Ġmedical Ġont ology . </s> Problem ĠForm al ization </s> Definition Ġ1 Ġ( Long itudinal ĠPatient ĠRecords ) ĠIn Ġlongitudinal ĠE HR Ġdata , Ġeach Ġpatient Ġcan Ġbe Ġrepresented Ġas Ġa Ġsequence Ġof Ġmult ivariate Ġobservations : Ġ$ Ġ\ math cal Ġ{ X } ^{ ( n )} Ġ= Ġ\ l brace Ġ Ġ\ math cal Ġ{ X } _ 1 ^{ ( n ) }, Ġ\ math cal Ġ{ X } _ 2 ^{ ( n ) }, Ġ\ cd ots Ġ, Ġ\ math cal Ġ{ X } _{ T ^{ ( n ) }} ^{ ( n )} Ġ\ r brace Ġ Ġ$ Ġwhere Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ Ġ, Ġ$ N $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġpatients ; Ġ$ T ^{ ( n )} $ Ġis Ġthe Ġnumber Ġof Ġvisits Ġof Ġthe Ġ$ n ^{ th } $ Ġpatient . ĠHere Ġwe Ġchoose Ġtwo Ġmain Ġmedical Ġcode Ġto Ġrepresent Ġeach Ġvisit Ġ$\ math cal Ġ{ X } _ t Ġ= Ġ\ math cal Ġ{ C } _ d ^ t Ġ\ cup Ġ\ math cal Ġ{ C } _ m ^ t $ Ġof Ġa Ġpatient Ġwhich Ġis Ġa Ġunion Ġset Ġof Ġcorresponding Ġdiagnoses Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t Ġ\ sub set Ġ\ math cal Ġ{ C } _ d $ Ġand Ġmedications Ġcodes Ġ$\ math cal Ġ{ C } _ m ^ t Ġ\ sub set Ġ\ math cal Ġ{ C } _ m $ Ġ. ĠFor Ġsimplicity , Ġwe Ġuse Ġ$\ math cal Ġ{ C } _ \ ast Ġ^ t $ Ġto Ġindicate Ġthe Ġunified Ġdefinition Ġfor Ġdifferent Ġtype Ġof Ġmedical Ġcodes Ġand Ġdrop Ġthe Ġsupers cript Ġ$( n ) $ Ġfor Ġa Ġsingle Ġpatient Ġwhenever Ġit Ġis Ġunamb iguous . Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 0 Ġdenotes Ġthe Ġmedical Ġcode Ġset Ġand Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 1 Ġthe Ġsize Ġof Ġthe Ġcode Ġset . Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 2 Ġis Ġthe Ġmedical Ġcode . </s> Definition Ġ2 Ġ( Medical ĠOnt ology ) ĠMedical Ġcodes Ġare Ġusually Ġcategorized Ġaccording Ġto Ġa Ġtree - struct ured Ġclassification Ġsystem Ġsuch Ġas ĠI CD - 9 Ġont ol oy Ġfor Ġdiagnosis Ġand ĠA TC Ġont ology Ġfor Ġmedication . ĠWe Ġuse Ġ$\ math cal Ġ{ O } _ d , Ġ\ math cal Ġ{ O } _ m $ Ġto Ġdenote Ġthe Ġont ology Ġfor Ġdiagnosis Ġand Ġmedication . ĠSimilarly , Ġwe Ġuse Ġ$\ math cal Ġ{ O } _ \ ast Ġ$ Ġto Ġindicate Ġthe Ġunified Ġdefinition Ġfor Ġdifferent Ġtype Ġof Ġmedical Ġcodes . ĠIn Ġdet ial , Ġ$\ math cal Ġ{ O } _ \ ast Ġ= Ġ\ over line {\ math cal Ġ{ C } _ \ ast Ġ} Ġ\ cup Ġ\ math cal Ġ{ C } _ \ ast Ġ$ Ġwhere Ġ$\ over line {\ math cal Ġ{ C } _ \ ast Ġ} $ Ġdenotes Ġthe Ġcodes Ġexcluding Ġleaf Ġcodes . ĠFor Ġsimplicity , Ġwe Ġdefine Ġtwo Ġfunction Ġ$ pa (\ cd ot Ġ), Ġch (\ cd ot Ġ) $ Ġwhich Ġaccept Ġtarget Ġmedical Ġcode Ġand Ġreturn Ġancestors ' Ġcode Ġset Ġand Ġdirect Ġchild Ġcode Ġset . </s> Problem ĠDefinition Ġ( Med ication ĠRecommend ation ) ĠGiven Ġdiagnosis Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t $ Ġof Ġthe Ġvisit Ġat Ġtime Ġ$ t $ Ġ, Ġpatient Ġhistory Ġ$\ math cal Ġ{ X } _{ 1 : t } Ġ= Ġ\ l brace Ġ\ math cal Ġ{ X } _ 1 , Ġ\ math cal Ġ{ X } _ 2 , Ġ\ cd ots Ġ, Ġ\ math cal Ġ{ X } _{ t - 1 }\ r brace Ġ$ Ġ, Ġwe Ġwant Ġto Ġrecommend Ġmultiple Ġmedications Ġby Ġgenerating Ġmulti - label Ġoutput Ġ$\ hat {\ math bf Ġ{ y }} _ t Ġ\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ^ { | \ math cal Ġ{ C } _ m | } $ Ġ. </s> Method </s> The Ġoverall Ġframework Ġof ĠG - BER T Ġis Ġdescribed Ġin ĠFigure Ġ2 Ġ. ĠG - BER T Ġfirst Ġderives Ġthe Ġinitial Ġembed ding Ġof Ġmedical Ġcodes Ġfrom Ġmedical Ġont ology Ġusing Ġgraph Ġneural Ġnetworks . ĠThen , Ġin Ġorder Ġto Ġfully Ġutilize Ġthe Ġrich ĠE HR Ġdata , ĠG - BER T Ġconstructs Ġan Ġadaptive ĠB ERT Ġmodel Ġon Ġthe Ġdiscarded Ġsingle - vis it Ġdata Ġfor Ġvisit Ġrepresentation . ĠFinally Ġwe Ġadd Ġa Ġprediction Ġlayer Ġand Ġfine - t une Ġthe Ġmodel Ġin Ġthe Ġmedication Ġrecommendation Ġtask . ĠIn Ġthe Ġfollowing Ġwe Ġwill Ġdescribe ĠG - BER T Ġin Ġdetail . ĠBut Ġfirst ly , Ġwe Ġgive Ġa Ġbrief Ġbackground Ġof ĠB ERT Ġespecially Ġfor Ġthe Ġtwo Ġpre - training Ġobjectives Ġwhich Ġwill Ġbe Ġlater Ġadapted Ġto ĠE HR Ġdata Ġin ĠSection Ġ" Pre - training " Ġ. </s> Background Ġof ĠB ERT ĠBased Ġon Ġa Ġmulti - layer ĠTrans former Ġenc oder ĠB IB REF 8 Ġ( The Ġtransformer Ġarchitecture Ġhas Ġbeen Ġubiqu it ously Ġused Ġin Ġmany Ġsequence Ġmodeling Ġtasks Ġrecently , Ġso Ġwe Ġwill Ġnot Ġintroduce Ġthe Ġdetails Ġhere ), ĠB ERT Ġis Ġpre - trained Ġusing Ġtwo Ġun super vised Ġtasks : </s> A Ġtypical Ġinput Ġto ĠB ERT Ġis Ġas Ġfollows Ġ( ĠB IB REF 7 Ġ): </s> Input Ġ= Ġ[ CL S ] Ġthe Ġman Ġwent Ġto Ġ[ MAS K ] Ġstore Ġ[ SE P ] Ġhe Ġbought Ġa Ġgallon Ġ[ MAS K ] Ġmilk Ġ[ SE P ] </s> Label Ġ= ĠIs Next </s> where Ġ[ CL S ] Ġis Ġthe Ġfirst Ġtoken Ġof Ġeach Ġsentence Ġpair Ġto Ġrepresent Ġthe Ġspecial Ġclassification Ġembed ding , Ġi . e . Ġthe Ġfinal Ġstate Ġof Ġthis Ġtoken Ġis Ġused Ġas Ġthe Ġaggreg ated Ġsequence Ġrepresentation Ġfor Ġclassification Ġtasks ; Ġ[ SE P ] Ġis Ġused Ġto Ġseparate Ġtwo Ġsentences ; Ġ[ MAS K ] Ġis Ġused Ġto Ġmask Ġout Ġthe Ġpredicted Ġwords Ġin Ġthe Ġmasked Ġlanguage Ġmodel . ĠUsing Ġthis Ġform , Ġthese Ġinputs Ġfacilitate Ġthe Ġtwo Ġtasks Ġdescribed Ġabove , Ġand Ġthey Ġwill Ġalso Ġbe Ġused Ġin Ġour Ġmethod Ġdescription Ġin Ġthe Ġfollowing Ġsection . </s> Input ĠRepresent ation </s> The ĠG - BER T Ġmodel Ġtakes Ġmedical Ġcodes ' Ġont ology Ġembed d ings Ġas Ġinput , Ġand Ġob tains Ġintermediate Ġrepresentations Ġfrom Ġa ĠTrans former Ġenc oder Ġas Ġthe Ġvisit Ġembed d ings . ĠIt Ġis Ġthen Ġpre - trained Ġon ĠE HR Ġfrom Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisit . ĠThe Ġderived Ġenc oder Ġand Ġvisit Ġembed ding Ġwill Ġbe Ġfed Ġinto Ġa Ġclass ifier Ġand Ġfine - tun ed Ġto Ġmake Ġpredictions . </s> Ont ology ĠEmb edd ing ĠWe Ġconstructed Ġont ology Ġembed ding Ġfrom Ġdiagnosis Ġont ology Ġ$\ math cal Ġ{ O } _ d $ Ġand Ġmedication Ġont ology Ġ$\ math cal Ġ{ O } _ m $ Ġ. ĠSince Ġthe Ġmedical Ġcodes Ġin Ġraw ĠE HR Ġdata Ġcan Ġbe Ġconsidered Ġas Ġleaf Ġnodes Ġin Ġthese Ġont ology Ġtrees , Ġwe Ġcan Ġenhance Ġthe Ġmedical Ġcode Ġembed ding Ġusing Ġgraph Ġneural Ġnetworks Ġ( G NN s ) Ġto Ġintegrate Ġthe Ġancestors ' Ġinformation Ġof Ġthese Ġcodes . ĠHere Ġwe Ġperform Ġa Ġtwo - stage Ġprocedure Ġwith Ġa Ġspecially Ġdesigned ĠG NN Ġfor Ġont ology Ġembed ding . </s> To Ġstart , Ġwe Ġassign Ġan Ġinitial Ġembed ding Ġvector Ġto Ġevery Ġmedical Ġcode Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ O } _ \ ast Ġ$ Ġwith Ġa Ġlearn able Ġembed ding Ġmatrix Ġ$\ math bf Ġ{ W } _ e Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ O } _ \ ast Ġ| Ġ\ times Ġd } $ Ġwhere Ġ$ d $ Ġis Ġthe Ġembed ding Ġdimension . </s> Stage Ġ1 . ĠFor Ġeach Ġnon - leaf Ġnode Ġ$ c _ \ ast Ġ\ in Ġ\ over line {\ math cal Ġ{ C } _ \ ast Ġ} $ Ġ, Ġwe Ġobtain Ġits Ġenhanced Ġmedical Ġembed ding Ġ$\ math bf Ġ{ h } _{ c _ \ ast Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġas Ġfollows : </s> $$ \ math bf Ġ{ h } _{ c _ \ ast Ġ} Ġ= Ġg ( c _ \ ast Ġ, Ġch ( c _ \ ast Ġ), Ġ\ math bf Ġ{ W } _ e ) $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġis Ġan Ġaggregation Ġfunction Ġwhich Ġaccepts Ġthe Ġtarget Ġmedical Ġcode Ġ$ c _ \ ast Ġ$ Ġ, Ġits Ġdirect Ġchild Ġcodes Ġ$ ch ( c _ \ ast Ġ) $ Ġand Ġinitial Ġembed ding Ġmatrix . ĠInt uitive ly , Ġthe Ġaggregation Ġfunction Ġcan Ġpass Ġand Ġfuse Ġinformation Ġin Ġtarget Ġnode Ġfrom Ġits Ġdirect Ġchildren Ġwhich Ġresult Ġin Ġthe Ġmore Ġrelated Ġembed ding Ġof Ġancestor ' Ġcode Ġto Ġchild Ġcodes ' Ġembed ding . </s> Stage Ġ2 . ĠAfter Ġobtaining Ġenhanced Ġembed d ings , Ġwe Ġpass Ġthe Ġenhance Ġembed ding Ġmatrix Ġ$\ math bf Ġ{ H } _ e Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ O } _ \ ast Ġ| Ġ\ times Ġd } $ Ġback Ġto Ġget Ġont ology Ġembed ding Ġfor Ġleaf Ġcodes Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ$ Ġas Ġfollows : </s> $$ \ math bf Ġ{ o } _{ c _ \ ast Ġ} Ġ= Ġg ( c _ \ ast Ġ, Ġpa ( c _ \ ast Ġ), Ġ\ math bf Ġ{ H } _ e ) $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġaccepts Ġancestor Ġcodes Ġof Ġtarget Ġmedical Ġcode Ġ$ c _ \ ast Ġ$ Ġ. ĠHere , Ġwe Ġuse Ġ$ pa ( c _ \ ast Ġ) $ Ġinstead Ġof Ġ$ ch ( c _ \ ast Ġ) $ Ġ, Ġsince Ġutilizing Ġthe Ġancestors ' Ġembed ding Ġcan Ġindirectly Ġassociate Ġall Ġmedical Ġcodes Ġinstead Ġof Ġtaking Ġeach Ġleaf Ġcode Ġas Ġindependent Ġinput . </s> The Ġoption Ġfor Ġthe Ġaggregation Ġfunction Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġis Ġflexible , Ġincluding Ġsum , Ġmean . ĠHere Ġwe Ġchoose Ġthe Ġone Ġfrom Ġgraph Ġattention Ġnetworks Ġ( G AT ) ĠB IB REF 19 Ġ, Ġwhich Ġhas Ġshown Ġefficient Ġembed ding Ġlearning Ġability Ġon Ġgraph - struct ured Ġtasks , Ġe . g ., Ġnode Ġclassification Ġand Ġlink Ġprediction . ĠIn Ġparticular , Ġwe Ġimplement Ġthe Ġaggregation Ġfunction Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġas Ġfollows : </s> $$ g Ġ( c _ \ ast Ġ, Ġp ( c _ \ ast Ġ), Ġ\ math bf Ġ{ H } _ e ) Ġ= Ġ\ oper at orn am ew ith limits {\ scale box Ġ{ 1 } [ 1 . 5 ] {\ par allel Ġ}} _{ k = 1 } ^ K Ġ\ s igma Ġ\ left ( Ġ\ sum Ġ_ { j Ġ\ in Ġ\ l brace Ġc _ \ ast Ġ\ r brace Ġ Ġ\ cup Ġpa ( c _ \ ast Ġ) } Ġ\ alpha Ġ_ { i , j } ^ k Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ j Ġ\ right ) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> where Ġ$\ oper at orn am ew ith limits {\ scale box Ġ{ 1 } [ 1 . 5 ] {\ par allel Ġ}} $ Ġrepresents Ġconc aten ation Ġwhich Ġenables Ġthe Ġmulti - head Ġattention Ġmechanism , Ġ$\ s igma Ġ$ Ġis Ġa Ġnon linear Ġactivation Ġfunction , Ġ$\ math bf Ġ{ W } ^ k Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġd } $ Ġis Ġthe Ġweight Ġmatrix Ġfor Ġinput Ġtransformation , Ġand Ġ$\ alpha Ġ_ { i , j } ^ k $ Ġare Ġthe Ġcorresponding Ġ$ k $ Ġ- th Ġnormalized Ġattention Ġcoefficients Ġcomputed Ġas Ġfollows : </s> $$ \ alpha Ġ_ { i , j } ^ k Ġ= Ġ\ frac { e ^ {\ left (\ text { Le aky Re LU } (\ math bf Ġ{ a } ^ \ inter cal Ġ[ \ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ i Ġ|| Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ j ]) \ right ) }} {\ sum Ġ_ { k Ġ\ in Ġ\ math cal Ġ{ N } _ i } Ġe ^ {\ left (\ text { Le aky Re LU } (\ math bf Ġ{ a } ^ \ inter cal Ġ[ \ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ i Ġ|| Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ k ]) \ right ) }} $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math bf Ġ{ a } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m } $ Ġis Ġa Ġlearn able Ġweight Ġvector Ġand ĠLe aky Re LU Ġis Ġa Ġnon linear Ġfunction . Ġ( we Ġassume Ġ$ m Ġ= Ġd / K $ Ġ). </s> As Ġshown Ġin ĠFigure Ġ2 Ġ, Ġwe Ġconstruct ĠI CD - 9 Ġtree Ġfor Ġdiagnosis Ġand ĠA TC Ġtree Ġfor Ġmedication Ġusing Ġthe Ġsame Ġstructure . ĠHere Ġthe Ġdirection Ġof Ġarrow Ġshows Ġthe Ġinformation Ġflow Ġwhere Ġancestor Ġnodes Ġcan Ġget Ġinformation Ġfrom Ġtheir Ġdirect Ġchildren Ġ( in Ġstage Ġ1 ) Ġand Ġsimilarly Ġleaf Ġnodes Ġcan Ġget Ġinformation Ġfrom Ġtheir Ġconnected Ġancestors Ġ( in Ġstage Ġ2 ). </s> It Ġis Ġworth Ġmentioning Ġthat Ġour Ġgraph Ġembed ding Ġmethod Ġon Ġmedical Ġont ology Ġis Ġdifferent Ġfrom ĠGR AM ĠB IB REF 20 Ġfrom Ġthe Ġfollowing Ġtwo Ġaspects : </s> Initial ization : Ġwe Ġinitialize Ġall Ġthe Ġnode Ġembed d ings Ġfrom Ġa Ġlearn able Ġembed ding Ġmatrix , Ġwhile ĠGR AM Ġlearns Ġthem Ġusing ĠGl ove Ġfrom Ġthe Ġco - occ urrence Ġinformation . </s> Up dating : Ġwe Ġdevelop Ġa Ġtwo - step Ġupdating Ġfunction Ġfor Ġboth Ġleaf Ġnodes Ġand Ġancestor Ġnodes ; Ġwhile Ġin ĠGR AM , Ġonly Ġthe Ġleaf Ġnodes Ġare Ġupdated Ġ( as Ġa Ġcombination Ġof Ġtheir Ġancestor Ġnodes Ġand Ġthemselves ). </s> Visit ĠEmb edd ing ĠSimilar Ġto ĠB ERT , Ġwe Ġuse Ġa Ġmulti - layer ĠTrans former Ġarchitecture ĠB IB REF 8 Ġas Ġour Ġvisit Ġenc oder . ĠThe Ġmodel Ġtakes Ġthe Ġont ology Ġembed ding Ġas Ġinput Ġand Ġderive Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ \ ast Ġ^ t Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġfor Ġa Ġpatient Ġat Ġ$ t $ Ġ- th Ġvisit : </s> $$ \ math bf Ġ{ v } _ \ ast Ġ^ t Ġ= Ġ\ math rm Ġ{ Trans former } (\ l brace Ġ\ text it Ġ{ [ CL S ] }\ r brace Ġ Ġ\ cup Ġ\ l brace Ġ\ math bf Ġ{ o } _{ c _ \ ast Ġ} ^ t | Ġc _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ t \ r brace Ġ) [ 0 ] $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġ[ CL S ] Ġis Ġa Ġspecial Ġtoken Ġas Ġin ĠB ERT . ĠIt Ġis Ġput Ġin Ġthe Ġfirst Ġposition Ġof Ġeach Ġvisit Ġof Ġtype Ġ$\ ast Ġ$ Ġand Ġits Ġfinal Ġstate Ġcan Ġbe Ġused Ġas Ġthe Ġrepresentation Ġof Ġthe Ġvisit . ĠInt uitive ly , Ġit Ġis Ġmore Ġreasonable Ġto Ġuse ĠTransformers Ġas Ġenc od ers Ġ( multi - head Ġattention Ġbased Ġarchitecture ) Ġthan ĠR NN Ġor Ġmean / sum Ġto Ġaggregate Ġmultiple Ġmedical Ġembed ding Ġfor Ġvisit Ġembed ding Ġsince Ġthe Ġset Ġof Ġmedical Ġcodes Ġwithin Ġone Ġvisit Ġis Ġnot Ġordered . </s> It Ġis Ġworth Ġnoting Ġthat Ġour ĠTrans former Ġenc oder Ġis Ġdifferent Ġfrom Ġthe Ġoriginal Ġone Ġin Ġthe Ġposition Ġembed ding Ġpart . ĠPosition Ġembed ding , Ġas Ġan Ġimportant Ġcomponent Ġin ĠTransformers Ġand ĠB ERT , Ġis Ġused Ġto Ġencode Ġthe Ġposition Ġand Ġorder Ġinformation Ġof Ġeach Ġtoken Ġin Ġa Ġsequence . ĠHowever , Ġone Ġbig Ġdifference Ġbetween Ġlanguage Ġsentences Ġand ĠE HR Ġsequences Ġis Ġthat Ġthe Ġmedical Ġcodes Ġwithin Ġthe Ġsame Ġvisit Ġdo Ġnot Ġgenerally Ġhave Ġan Ġorder , Ġso Ġwe Ġremove Ġthe Ġposition Ġembed ding Ġin Ġour Ġmodel . </s> Pre - training </s> We Ġadapted Ġthe Ġoriginal ĠB ERT Ġmodel Ġto Ġbe Ġmore Ġsuitable Ġfor Ġour Ġdata Ġand Ġtask . ĠIn Ġparticular , Ġwe Ġpre - train Ġthe Ġmodel Ġon Ġeach ĠE HR Ġvisit Ġ( within Ġboth Ġsingle - vis it ĠE HR Ġsequences Ġand Ġmulti - vis it ĠE HR Ġsequences ). ĠWe Ġmodified Ġthe Ġinput Ġand Ġpre - training Ġobjectives Ġof Ġthe ĠB ERT Ġmodel : Ġ( 1 ) ĠFor Ġthe Ġinput , Ġwe Ġbuilt Ġthe ĠTrans former Ġenc oder Ġon Ġthe ĠG NN Ġoutputs , Ġi . e . Ġont ology Ġembed d ings , Ġfor Ġvisit Ġembed ding . ĠFor Ġthe Ġoriginal ĠE HR Ġsequence , Ġit Ġmeans Ġessentially Ġwe Ġcombine Ġthe ĠG NN Ġmodel Ġwith Ġa ĠTrans former Ġto Ġbecome Ġa Ġnew Ġintegrated Ġenc oder . ĠIn Ġaddition , Ġwe Ġremoved Ġthe Ġposition Ġembed ding Ġas Ġwe Ġexplained Ġbefore . Ġ( 2 ) ĠAs Ġfor Ġthe Ġpre - training Ġprocedures , Ġwe Ġmodified Ġthe Ġoriginal Ġpre - training Ġtasks Ġi . e ., ĠMask ed ĠLM Ġ( language Ġmodel ) Ġtask Ġand ĠNext ĠSent ence Ġprediction Ġtask Ġto Ġself - pred iction Ġtask Ġand Ġdual - pred iction Ġtask . ĠThe Ġidea Ġto Ġconduct Ġthese Ġtasks Ġis Ġto Ġmake Ġthe Ġvisit Ġembed ding Ġabsorb Ġenough Ġinformation Ġabout Ġwhat Ġit Ġis Ġmade Ġof Ġand Ġwhat Ġit Ġis Ġable Ġto Ġpredict . </s> Thus , Ġfor Ġthe Ġself - pred iction Ġtask , Ġwe Ġwant Ġthe Ġvisit Ġembed ding Ġ$ v ^ 1 _ \ ast Ġ$ Ġto Ġrecover Ġwhat Ġit Ġis Ġmade Ġof , Ġi . e ., Ġthe Ġinput Ġmedical Ġcodes Ġ$\ math cal Ġ{ C } _ \ ast Ġ^ t $ Ġfor Ġeach Ġvisit Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ& \ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } ^ 1 _ \ ast Ġ, Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 ) Ġ= Ġ- \ log Ġp (\ math cal Ġ{ C } _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ\\ Ġ& Ġ= Ġ- Ġ\ sum Ġ_ { c Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 }\ log Ġp ( c _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ+ Ġ\ sum Ġ_ { c Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ\ set minus Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 }\ log Ġp ( c _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> we Ġminimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġ$\ math cal Ġ{ L } _ s $ Ġ, Ġand Ġin Ġpractise , Ġ$\ text { S igm oid } ( f (\ math bf Ġ{ v } _ \ ast Ġ)) $ Ġshould Ġbe Ġtransformed Ġby Ġapplying Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġ$ f (\ cd ot Ġ) $ Ġwith Ġone Ġhidden Ġlayer . ĠWith Ġan Ġanalogy Ġto Ġthe ĠMask ed ĠLM Ġtask Ġin ĠB ERT , Ġwe Ġalso Ġused Ġspecific Ġsymbol Ġ[ MAS K ] Ġto Ġrandomly Ġreplace Ġthe Ġoriginal Ġmedical Ġcode Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 $ Ġ. ĠSo Ġthere Ġare Ġ$ 15 \ % $ Ġcodes Ġin Ġ$\ math cal Ġ{ C } _ \ ast Ġ$ Ġwhich Ġwill Ġbe Ġreplaced Ġrandomly Ġand Ġthe Ġmodel Ġshould Ġhave Ġthe Ġability Ġto Ġpredict Ġthe Ġmasked Ġcode Ġbased Ġon Ġothers . </s> Likewise , Ġfor Ġthe Ġdual - pred iction Ġtask , Ġsince Ġthe Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ \ ast Ġ$ Ġcarries Ġthe Ġinformation Ġof Ġmedical Ġcodes Ġof Ġtype Ġ$\ ast Ġ$ Ġ, Ġwe Ġcan Ġfurther Ġexpect Ġit Ġhas Ġthe Ġability Ġto Ġdo Ġmore Ġtask - specific Ġprediction Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ math cal Ġ{ L } _{ du } Ġ= Ġ- \ log Ġp (\ math cal Ġ{ C } _ d ^ 1 Ġ| Ġ\ math bf Ġ{ v } _ m ^ 1 ) Ġ- Ġ\ log Ġp (\ math cal Ġ{ C } _ m ^ 1 Ġ| Ġ\ math bf Ġ{ v } _ d ^ 1 ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġwe Ġuse Ġthe Ġsame Ġtransformation Ġfunction Ġ$\ text { S igm oid } ( f _ 1 (\ math bf Ġ{ v } _ m ^ 1 )) $ Ġ, Ġ$\ text { S igm oid } ( f _ 2 (\ math bf Ġ{ v } _ d ^ 1 )) $ Ġwith Ġdifferent Ġweight Ġmatrix Ġto Ġtransform Ġthe Ġvisit Ġembed ding Ġand Ġoptimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġ$\ math cal Ġ{ L } _{ du } $ Ġexpanded Ġsame Ġas Ġ$\ math cal Ġ{ L } _{ se } $ Ġin ĠE q . Ġ19 Ġ. ĠThis Ġis Ġa Ġdirect Ġadaptation Ġof Ġthe Ġnext Ġsentence Ġprediction Ġtask . ĠIn ĠB ERT , Ġthe Ġnext Ġsentence Ġprediction Ġtask Ġfacilitates Ġthe Ġprediction Ġof Ġsentence Ġrelations , Ġwhich Ġis Ġa Ġcommon Ġtask Ġin ĠN LP . ĠHowever , Ġin Ġhealthcare , Ġmost Ġpredictive Ġtasks Ġdo Ġnot Ġhave Ġa Ġsequence Ġpair Ġto Ġclassify . ĠInstead , Ġwe Ġare Ġoften Ġinterested Ġin Ġpredicting Ġunknown Ġdisease Ġor Ġmedication Ġcodes Ġof Ġthe Ġsequence . ĠFor Ġexample , Ġin Ġmedication Ġrecommendation , Ġwe Ġwant Ġto Ġpredict Ġmultiple Ġmedications Ġgiven Ġonly Ġthe Ġdiagnosis Ġcodes . ĠIn versely , Ġwe Ġcan Ġalso Ġpredict Ġunknown Ġdiagnosis Ġgiven Ġthe Ġmedication Ġcodes . </s> Thus , Ġour Ġfinal Ġpre - training Ġoptimization Ġobjective Ġcan Ġsimply Ġbe Ġthe Ġcombination Ġof Ġthe Ġaforementioned Ġlosses , Ġas Ġshown Ġin ĠE q . Ġ21 Ġ. ĠIt Ġis Ġused Ġto Ġtrain Ġon ĠE HR Ġdata Ġfrom Ġall Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisits .. </s> $$ \ begin { aligned } Ġ\ math cal Ġ{ L } _{ pr } Ġ= Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { n = 1 } ^ N Ġ(( Ġ& \ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } _ d ^{ 1 , Ġ( n ) }, Ġ\ math cal Ġ{ C } _ d ^{ 1 , Ġ( n ) }) Ġ+ Ġ\ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } _ m ^{ 1 , Ġ( n ) }, Ġ\ math cal Ġ{ C } _ m ^{ 1 , Ġ( n ) }) \\ Ġ& Ġ+ Ġ\ math cal Ġ{ L } _{ du } ^{ ( n ) }) Ġ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> Fine - tun ing </s> After Ġobtaining Ġpre - trained Ġvisit Ġrepresentation Ġfor Ġeach Ġvisit , Ġfor Ġa Ġprediction Ġtask Ġon Ġa Ġmulti - vis it Ġsequence Ġdata , Ġwe Ġaggregate Ġall Ġthe Ġvisit Ġembed ding Ġand Ġadd Ġa Ġprediction Ġlayer Ġfor Ġthe Ġmedication Ġrecommendation Ġtask . ĠTo Ġbe Ġspecific , Ġfrom Ġpre - training Ġon Ġall Ġvisits , Ġwe Ġhave Ġa Ġpre - trained ĠTrans former Ġenc oder , Ġwhich Ġcan Ġthen Ġbe Ġused Ġto Ġget Ġthe Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ * ^ \ t au Ġ$ Ġat Ġtime Ġ$\ t au Ġ$ Ġ. ĠThe Ġknown Ġdiagnosis Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t $ Ġat Ġthe Ġprediction Ġtime Ġ$ t $ Ġis Ġalso Ġrepresented Ġusing Ġthe Ġsame Ġmodel Ġas Ġ$\ math bf Ġ{ v } _ * ^ t $ Ġ. ĠCon cat en ating Ġthe Ġmean Ġof Ġprevious Ġdiagnoses Ġvisit Ġembed d ings Ġand Ġmedication Ġvisit Ġembed d ings , Ġalso Ġthe Ġlast Ġdiagnoses Ġvisit Ġembed ding , Ġwe Ġbuilt Ġan ĠML P Ġbased Ġprediction Ġlayer Ġto Ġpredict Ġthe Ġrecommended Ġmedication Ġcodes Ġas Ġin ĠEqu ation Ġ23 Ġ. </s> $$ \ math bf Ġ{ y } _ t Ġ= Ġ\ math rm Ġ{ S igm oid } (\ math bf Ġ{ W } _ 1 Ġ[ (\ frac { 1 }{ t }\ sum Ġ_ {\ t au Ġ< Ġt } Ġ\ math bf Ġ{ v } _ d ^ \ t au Ġ) Ġ|| Ġ( \ frac { 1 }{ t }\ sum Ġ_ {\ t au Ġ< Ġt } Ġ\ math bf Ġ{ v } _ m ^ \ t au Ġ) Ġ|| \ math bf Ġ{ v } _ d ^ t ] Ġ+ Ġb ) $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ math bf Ġ{ W } _ 1 Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ C } _ m | Ġ\ times Ġ3 d } Ġ$ Ġis Ġa Ġlearn able Ġtransformation Ġmatrix . </s> Given Ġthe Ġtrue Ġlabels Ġ$\ hat {\ math bf Ġ{ y }} _ t $ Ġat Ġeach Ġtime Ġstamp Ġ$ t $ Ġ, Ġthe Ġloss Ġfunction Ġfor Ġthe Ġwhole ĠE HR Ġsequence Ġ( i . e . Ġa Ġpatient ) Ġis </s> $$ \ math cal Ġ{ L } Ġ= Ġ- Ġ\ frac { 1 }{ T - 1 } Ġ\ sum Ġ_ { t = 2 } ^ T Ġ( \ math bf Ġ{ y } _ t ^ \ inter cal Ġ\ log Ġ( \ hat {\ math bf Ġ{ y }} _ t ) Ġ+ Ġ( 1 Ġ- Ġ\ math bf Ġ{ y } _ t ^ \ inter cal Ġ) Ġ\ log Ġ( 1 Ġ- Ġ\ hat {\ math bf Ġ{ y }} _ t )) $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Exper iment </s> Data ĠWe Ġused ĠE HR Ġdata Ġfrom ĠM IM IC - III ĠB IB REF 21 Ġand Ġconducted Ġall Ġour Ġexperiments Ġon Ġa Ġcohort Ġwhere Ġpatients Ġhave Ġmore Ġthan Ġone Ġvisit . ĠWe Ġutilize Ġdata Ġfrom Ġpatients Ġwith Ġboth Ġsingle Ġvisit Ġand Ġmultiple Ġvisits Ġin Ġthe Ġtraining Ġdataset Ġas Ġpre - training Ġdata Ġsource Ġ( multi - vis it Ġdata Ġare Ġsplit Ġinto Ġvisit Ġslices ). ĠIn Ġthis Ġwork , Ġwe Ġtransform Ġthe Ġdrug Ġcoding Ġfrom ĠN DC Ġto ĠA TC ĠThird ĠLevel Ġfor Ġusing Ġthe Ġont ology Ġinformation . ĠThe Ġstatistics Ġof Ġthe Ġdatasets Ġare Ġsummarized Ġin ĠTable Ġ2 Ġ. </s> Bas eline ĠWe Ġcompared ĠG - BER T Ġwith Ġthe Ġfollowing Ġbas elines . ĠAll Ġmethods Ġare Ġimplemented Ġin ĠPy Tor ch ĠB IB REF 22 Ġand Ġtrained Ġon Ġan ĠUbuntu Ġ16 . 04 Ġwith Ġ8 GB Ġmemory Ġand ĠNvidia Ġ1080 ĠGPU . </s> We Ġalso Ġevaluated Ġthree ĠG - BER T Ġvariants Ġfor Ġmodel Ġab lation . </s> Met rics ĠTo Ġmeasure Ġthe Ġprediction Ġaccuracy , Ġwe Ġused ĠJ acc ard ĠSimilar ity ĠScore Ġ( J acc ard ), ĠAverage ĠF 1 Ġ( F 1 ) Ġand ĠPrecision ĠRecall ĠA UC Ġ( PR - A UC ). ĠJ acc ard Ġis Ġdefined Ġas Ġthe Ġsize Ġof Ġthe Ġintersection Ġdivided Ġby Ġthe Ġsize Ġof Ġthe Ġunion Ġof Ġground Ġtruth Ġset Ġ$ Y _ t ^{ ( k )} $ Ġand Ġpredicted Ġset Ġ$\ hat { Y } _ t ^{ ( k )} $ Ġ. </s> $$ \ text { J acc ard } Ġ= Ġ\ frac { 1 }{ \ sum Ġ_ k ^ N Ġ\ sum Ġ_ t ^{ T _ k } Ġ1 }\ sum Ġ_ k ^ N Ġ\ sum Ġ_ t ^{ T _ k } Ġ\ frac { | Y _ t ^{ ( k )} Ġ\ cap Ġ\ hat { Y } _ t ^{ ( k )} | }{ | Y _ t ^{ ( k )} Ġ\ cup Ġ\ hat { Y } _ t ^{ ( k )} | }\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ36 ) </s> where Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġpatients Ġin Ġtest Ġset Ġand Ġ$ T _ k $ Ġis Ġthe Ġnumber Ġof Ġvisits Ġof Ġthe Ġ$ k ^{ th } $ Ġpatient . </s> Im plementation ĠDetails ĠWe Ġrandomly Ġdivide Ġthe Ġdataset Ġinto Ġtraining , Ġvalidation Ġand Ġtesting Ġset Ġin Ġa Ġ$ 0 . 6 Ġ: Ġ0 . 2 Ġ: Ġ0 . 2 $ Ġratio . ĠFor ĠG - BER T , Ġthe Ġhyper param eters Ġare Ġadjusted Ġon Ġevaluation Ġset : Ġ( 1 ) ĠG AT Ġpart : Ġinput Ġembed ding Ġdimension Ġas Ġ75 , Ġnumber Ġof Ġattention Ġheads Ġas Ġ4 ; Ġ( 2 ) ĠB ERT Ġpart : Ġhidden Ġdimension Ġas Ġ300 , Ġdimension Ġof Ġposition - wise Ġfeed - forward Ġnetworks Ġas Ġ300 , Ġ2 Ġhidden Ġlayers Ġwith Ġ4 Ġattention Ġheads Ġfor Ġeach Ġlayer . ĠS pecially , Ġwe Ġaltern ated Ġthe Ġpre - training Ġwith Ġ5 Ġepoch s Ġand Ġfine - tun ing Ġprocedure Ġwith Ġ5 Ġepoch s Ġfor Ġ15 Ġtimes Ġto Ġstabilize Ġthe Ġtraining Ġprocedure . </s> For ĠLR , Ġwe Ġuse Ġthe Ġgrid Ġsearch Ġover Ġtypical Ġrange Ġof Ġhyper - param eter Ġto Ġsearch Ġthe Ġbest Ġhyper param eter Ġvalues Ġwhich Ġresult Ġin ĠL 1 Ġnorm Ġpenalty Ġwith Ġweight Ġas Ġ$ 1 . 1 $ Ġ. ĠFor Ġdeep Ġlearning Ġmodels , Ġwe Ġimplemented ĠR NN Ġusing Ġa Ġg ated Ġrecurrent Ġunit Ġ( GR U ) ĠB IB REF 24 Ġand Ġutilize Ġdrop out Ġwith Ġa Ġprobability Ġof Ġ0 . 4 Ġon Ġthe Ġoutput Ġof Ġembed ding . ĠWe Ġtest Ġseveral Ġembed ding Ġchoice Ġfor Ġbaseline Ġmethods Ġand Ġdetermine Ġthe Ġdimension Ġfor Ġmedical Ġembed ding Ġas Ġ300 Ġand Ġther sh old Ġfor Ġfinal Ġprediction Ġas Ġ0 . 3 Ġfor Ġbetter Ġperformance . ĠTraining Ġis Ġdone Ġthrough ĠAdam ĠB IB REF 25 Ġat Ġlearning Ġrate Ġ5 e - 4 . ĠWe Ġfix Ġthe Ġbest Ġmodel Ġon Ġevaluation Ġset Ġwithin Ġ100 Ġepoch s Ġand Ġreport Ġthe Ġperformance Ġin Ġtest Ġset . </s> Results </s> Exper imental ĠResults ĠTable . Ġ3 Ġcompares Ġthe Ġperformance Ġon Ġthe Ġmedication Ġrecommendation Ġtask . ĠFor Ġvariants Ġof ĠG - BER T , Ġ$\ text tt Ġ{ G - BER T } _{ G ^ -, P ^ - } $ Ġperforms Ġworse Ġcompared Ġwith Ġ$\ text tt Ġ{ G - BER T } _{ G ^ - } $ Ġand Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġwhich Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġusing Ġont ology Ġinformation Ġto Ġget Ġenhanced Ġmedical Ġembed ding Ġas Ġinput Ġand Ġemploy Ġan Ġun super vised Ġpre - training Ġprocedure Ġon Ġlarger Ġabundant Ġdata . ĠInc orpor ating Ġboth Ġhierarchical Ġont ology Ġinformation Ġand Ġpre - training Ġprocedure , Ġthe Ġend - to - end Ġmodel ĠG - BER T Ġhas Ġmore Ġcapacity Ġand Ġachieve Ġcomparable Ġresults Ġwith Ġothers . </s> As Ġfor Ġbaseline Ġmodels , ĠLR Ġand ĠLeap Ġare Ġworse Ġthan Ġour Ġmost Ġbasic Ġmodel Ġ( Ġ$\ text tt Ġ{ G - BER T } _{ G ^ -, P ^ - } $ Ġ) Ġin Ġterms Ġof Ġmost Ġmetrics . ĠComp aring Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġand ĠGR AM , Ġwhich Ġboth Ġused Ġmedical Ġont ology Ġinformation Ġwithout Ġpre - training , Ġthe Ġscores Ġof Ġour Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġis Ġslightly Ġhigher Ġin Ġall Ġmetrics . ĠThis Ġcan Ġdemonstrate Ġthe Ġvalid ness Ġof Ġusing ĠTrans former Ġenc od ers Ġand Ġthe Ġspecific Ġprediction Ġlayer Ġfor Ġmedication Ġrecommendation . ĠOur Ġfinal Ġmodel ĠG - BER T Ġis Ġalso Ġbetter Ġthan Ġthe Ġattention Ġbased Ġmodel , ĠR ETA IN , Ġand Ġthe Ġrecently Ġpublished Ġstate - of - the - art Ġmodel , ĠGAM EN et . ĠSpecifically , Ġeven Ġadding Ġthe Ġextra Ġinformation Ġof ĠD DI Ġknowledge Ġand Ġprocedure Ġcodes , ĠGAM EN et Ġstill Ġperforms Ġworse Ġthan ĠG - BER T . </s> In Ġaddition , Ġwe Ġvisual ized Ġthe Ġpre - training Ġmedical Ġcode Ġembed d ings Ġof Ġ$\ text tt Ġ{ G - BER T } _{ G ^ - } $ Ġand ĠG - BER T Ġto Ġshow Ġthe Ġeffectiveness Ġof Ġont ology Ġembed ding Ġusing Ġonline Ġembed ding Ġprojector Ġshown Ġin Ġ( https :// raw . github usercontent . com / j sh ang 123 / G - B ert / master / s aved / ts ne . png / ). </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġproposed Ġa Ġpre - training Ġmodel Ġnamed ĠG - BER T Ġfor Ġmedical Ġcode Ġrepresentation Ġand Ġmedication Ġrecommendation . ĠTo Ġour Ġbest Ġknowledge , ĠG - BER T Ġis Ġthe Ġfirst Ġthat Ġutilizes Ġlanguage Ġmodel Ġpre - training Ġtechniques Ġin Ġhealthcare Ġdomain . ĠIt Ġadapted ĠB ERT Ġto Ġthe ĠE HR Ġdata Ġand Ġintegrated Ġmedical Ġont ology Ġinformation Ġusing Ġgraph Ġneural Ġnetworks . ĠBy Ġadditional Ġpre - training Ġon Ġthe ĠE HR Ġfrom Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisit Ġwhich Ġare Ġgenerally Ġdiscarded Ġbefore Ġmodel Ġtraining , ĠG - BER T Ġoutper forms Ġall Ġbas elines Ġin Ġprediction Ġaccuracy Ġon Ġmedication Ġrecommendation Ġtask . ĠOne Ġdirection Ġfor Ġthe Ġfuture Ġwork Ġis Ġto Ġadd Ġmore Ġauxiliary Ġand Ġstructural Ġtasks Ġto Ġimprove Ġthe Ġability Ġof Ġcode Ġrepresent a ion . ĠAnother Ġdirection Ġmay Ġbe Ġto Ġadapt Ġour Ġmodel Ġto Ġbe Ġsuitable Ġfor Ġeven Ġlarger Ġdatasets Ġwith Ġmore Ġheter ogeneous Ġmod alities . </s> Acknowled gment </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠScience ĠFoundation Ġaward ĠI IS - 14 185 11 , ĠCC F - 15 33 768 Ġand ĠI IS - 18 380 42 , Ġthe ĠNational ĠInstitute Ġof ĠHealth Ġaward Ġ1 R 01 MD 01 16 82 - 01 Ġand ĠR 56 HL 13 84 15 .
Introduction The availability of massive electronic health records (EHR) data and the advances of deep learning technologies have provided unprecedented resource and opportunity for predictive healthcare, including the computational medication recommendation task. A number of deep learning models were proposed to assist doctors in making medication recommendation BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . They often learn representations for medical entities (e.g., patients, diagnosis, medications) from patient EHR data, and then use the learned representations to predict medications that are suited to the patient's health condition. To provide effective medication recommendation, it is important to learn accurate representation of medical codes. Despite that various considerations were handled in previous works for improving medical code representations BIBREF4 , BIBREF2 , BIBREF3 , there are two limitations with the existing work: To mitigate the aforementioned limitations, we propose G-BERT that combines the pre-training techniques and graph neural networks for better medical code representation and medication recommendation. G-BERT is enabled and demonstrated by the following technical contributions: Related Work Medication Recommendation Medication Recommendation can be categorized into instance-based and longitudinal recommendation methods BIBREF1 . Instance-based methods focus on current health conditions. Among them, Leap BIBREF9 formulates a multi-instance multi-label learning framework and proposes a variant of sequence-to-sequence model based on content-attention mechanism to predict combination of medicines given patient's diagnoses. Longitudinal-based methods leverage the temporal dependencies among clinical events, see BIBREF10 , BIBREF11 , BIBREF12 . Among them, RETAIN BIBREF10 uses a two-level neural attention model to detect influential past visits and significant clinical variables within those visits for improved medication recommendation. Pre-training Techniques The goal of pre-training techniques is to provide model training with good initializations. Pre-training has been shown extremely effective in various areas such as image classification BIBREF13 , BIBREF14 and machine translation BIBREF15 . The unsupervised pre-training can be considered as a regularizer that supports better generalization from the training dataset BIBREF16 . Recently, language model pre-training techniques such as BIBREF5 , BIBREF6 , BIBREF7 have shown to largely improve the performance on multiple NLP tasks. As the most widely used one, BERT BIBREF7 builds on the Transformer BIBREF8 architecture and improves the pre-training using a masked language model for bidirectional representation. In this paper, we adapt the framework of BERT and pre-train our model on each visit of the EHR data to leverage the single-visit data that were not fit for model training in other medication recommendation models. Graph Neural Networks (GNN) GNNs are neural networks that learn node or graph representations from graph-structured data. Various graph neural networks have been proposed to encode the graph-structure information, including graph convolutional neural networks (GCN) BIBREF17 , message passing networks (MPNN) BIBREF18 , graph attention networks (GAT) BIBREF19 . GNNs have already been demonstrated useful on EHR modeling BIBREF20 , BIBREF1 . GRAM BIBREF20 represented a medical concept as a combination of its ancestors in the medical ontology using an attention mechanism. It's different from G-BERT from two aspects as described in Section "Input Representation" . Another work worth mentioning is GAMENet BIBREF1 , which also used graph neural network to assist the medication recommendation task. However, GAMENet has a different motivation which results in using graph neural networks on drug-drug-interaction graphs instead of medical ontology. Problem Formalization Definition 1 (Longitudinal Patient Records) In longitudinal EHR data, each patient can be represented as a sequence of multivariate observations: $ \mathcal {X}^{(n)} = \lbrace  \mathcal {X}_1^{(n)}, \mathcal {X}_2^{(n)}, \cdots , \mathcal {X}_{T^{(n)}}^{(n)} \rbrace  $ where $n\in \lbrace 1,2,\ldots , N\rbrace $ , $N$ is the total number of patients; $T^{(n)}$ is the number of visits of the $n^{th}$ patient. Here we choose two main medical code to represent each visit $\mathcal {X}_t = \mathcal {C}_d^t \cup \mathcal {C}_m^t$ of a patient which is a union set of corresponding diagnoses codes $\mathcal {C}_d^t \subset \mathcal {C}_d$ and medications codes $\mathcal {C}_m^t \subset \mathcal {C}_m$ . For simplicity, we use $\mathcal {C}_\ast ^t$ to indicate the unified definition for different type of medical codes and drop the superscript $(n)$ for a single patient whenever it is unambiguous. $n\in \lbrace 1,2,\ldots , N\rbrace $0 denotes the medical code set and $n\in \lbrace 1,2,\ldots , N\rbrace $1 the size of the code set. $n\in \lbrace 1,2,\ldots , N\rbrace $2 is the medical code. Definition 2 (Medical Ontology) Medical codes are usually categorized according to a tree-structured classification system such as ICD-9 ontoloy for diagnosis and ATC ontology for medication. We use $\mathcal {O}_d, \mathcal {O}_m$ to denote the ontology for diagnosis and medication. Similarly, we use $\mathcal {O}_\ast $ to indicate the unified definition for different type of medical codes. In detial, $\mathcal {O}_\ast = \overline{\mathcal {C}_\ast } \cup \mathcal {C}_\ast $ where $\overline{\mathcal {C}_\ast }$ denotes the codes excluding leaf codes. For simplicity, we define two function $pa(\cdot ), ch(\cdot )$ which accept target medical code and return ancestors' code set and direct child code set. Problem Definition (Medication Recommendation) Given diagnosis codes $\mathcal {C}_d^t$ of the visit at time $t$ , patient history $\mathcal {X}_{1:t} = \lbrace \mathcal {X}_1, \mathcal {X}_2, \cdots , \mathcal {X}_{t-1}\rbrace $ , we want to recommend multiple medications by generating multi-label output $\hat{\mathbf {y}}_t \in \lbrace 0,1\rbrace ^{|\mathcal {C}_m|}$ . Method The overall framework of G-BERT is described in Figure 2 . G-BERT first derives the initial embedding of medical codes from medical ontology using graph neural networks. Then, in order to fully utilize the rich EHR data, G-BERT constructs an adaptive BERT model on the discarded single-visit data for visit representation. Finally we add a prediction layer and fine-tune the model in the medication recommendation task. In the following we will describe G-BERT in detail. But firstly, we give a brief background of BERT especially for the two pre-training objectives which will be later adapted to EHR data in Section "Pre-training" . Background of BERT Based on a multi-layer Transformer encoder BIBREF8 (The transformer architecture has been ubiquitously used in many sequence modeling tasks recently, so we will not introduce the details here), BERT is pre-trained using two unsupervised tasks: A typical input to BERT is as follows ( BIBREF7 ): Input = [CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP] Label = IsNext where [CLS] is the first token of each sentence pair to represent the special classification embedding, i.e. the final state of this token is used as the aggregated sequence representation for classification tasks; [SEP] is used to separate two sentences; [MASK] is used to mask out the predicted words in the masked language model. Using this form, these inputs facilitate the two tasks described above, and they will also be used in our method description in the following section. Input Representation The G-BERT model takes medical codes' ontology embeddings as input, and obtains intermediate representations from a Transformer encoder as the visit embeddings. It is then pre-trained on EHR from patients who only have one hospital visit. The derived encoder and visit embedding will be fed into a classifier and fine-tuned to make predictions. Ontology Embedding We constructed ontology embedding from diagnosis ontology $\mathcal {O}_d$ and medication ontology $\mathcal {O}_m$ . Since the medical codes in raw EHR data can be considered as leaf nodes in these ontology trees, we can enhance the medical code embedding using graph neural networks (GNNs) to integrate the ancestors' information of these codes. Here we perform a two-stage procedure with a specially designed GNN for ontology embedding. To start, we assign an initial embedding vector to every medical code $c_\ast \in \mathcal {O}_\ast $ with a learnable embedding matrix $\mathbf {W}_e \in \mathbb {R}^{|\mathcal {O}_\ast | \times d}$ where $d$ is the embedding dimension. Stage 1. For each non-leaf node $c_\ast \in \overline{\mathcal {C}_\ast }$ , we obtain its enhanced medical embedding $\mathbf {h}_{c_\ast } \in \mathbb {R}^{d}$ as follows: $$\mathbf {h}_{c_\ast } = g(c_\ast , ch(c_\ast ), \mathbf {W}_e)$$   (Eq. 11) where $g(\cdot ,\cdot ,\cdot )$ is an aggregation function which accepts the target medical code $c_\ast $ , its direct child codes $ch(c_\ast )$ and initial embedding matrix. Intuitively, the aggregation function can pass and fuse information in target node from its direct children which result in the more related embedding of ancestor' code to child codes' embedding. Stage 2. After obtaining enhanced embeddings, we pass the enhance embedding matrix $\mathbf {H}_e \in \mathbb {R}^{|\mathcal {O}_\ast | \times d}$ back to get ontology embedding for leaf codes $c_\ast \in \mathcal {C}_\ast $ as follows: $$\mathbf {o}_{c_\ast } = g(c_\ast , pa(c_\ast ), \mathbf {H}_e)$$   (Eq. 12) where $g(\cdot ,\cdot ,\cdot )$ accepts ancestor codes of target medical code $c_\ast $ . Here, we use $pa(c_\ast )$ instead of $ch(c_\ast )$ , since utilizing the ancestors' embedding can indirectly associate all medical codes instead of taking each leaf code as independent input. The option for the aggregation function $g(\cdot ,\cdot ,\cdot )$ is flexible, including sum, mean. Here we choose the one from graph attention networks (GAT) BIBREF19 , which has shown efficient embedding learning ability on graph-structured tasks, e.g., node classification and link prediction. In particular, we implement the aggregation function $g(\cdot ,\cdot ,\cdot )$ as follows: $$g (c_\ast , p(c_\ast ), \mathbf {H}_e) = \operatornamewithlimits{\scalebox {1}[1.5]{\parallel }}_{k=1}^K \sigma \left( \sum _{j \in \lbrace c_\ast \rbrace  \cup pa(c_\ast )} \alpha _{i,j}^k \mathbf {W}^k \mathbf {h}_j \right)$$   (Eq. 13) where $\operatornamewithlimits{\scalebox {1}[1.5]{\parallel }}$ represents concatenation which enables the multi-head attention mechanism, $\sigma $ is a nonlinear activation function, $\mathbf {W}^k \in \mathbb {R}^{m \times d}$ is the weight matrix for input transformation, and $\alpha _{i,j}^k$ are the corresponding $k$ -th normalized attention coefficients computed as follows: $$\alpha _{i,j}^k = \frac{e^{\left(\text{LeakyReLU}(\mathbf {a}^\intercal [\mathbf {W}^k \mathbf {h}_i || \mathbf {W}^k \mathbf {h}_j])\right)}}{\sum _{k \in \mathcal {N}_i} e^{\left(\text{LeakyReLU}(\mathbf {a}^\intercal [\mathbf {W}^k \mathbf {h}_i || \mathbf {W}^k \mathbf {h}_k])\right)}}$$   (Eq. 14) where $\mathbf {a} \in \mathbb {R}^{2m}$ is a learnable weight vector and LeakyReLU is a nonlinear function. (we assume $m = d/K$ ). As shown in Figure 2 , we construct ICD-9 tree for diagnosis and ATC tree for medication using the same structure. Here the direction of arrow shows the information flow where ancestor nodes can get information from their direct children (in stage 1) and similarly leaf nodes can get information from their connected ancestors (in stage 2). It is worth mentioning that our graph embedding method on medical ontology is different from GRAM BIBREF20 from the following two aspects: Initialization: we initialize all the node embeddings from a learnable embedding matrix, while GRAM learns them using Glove from the co-occurrence information. Updating: we develop a two-step updating function for both leaf nodes and ancestor nodes; while in GRAM, only the leaf nodes are updated (as a combination of their ancestor nodes and themselves). Visit Embedding Similar to BERT, we use a multi-layer Transformer architecture BIBREF8 as our visit encoder. The model takes the ontology embedding as input and derive visit embedding $\mathbf {v}_\ast ^t \in \mathbb {R}^d$ for a patient at $t$ -th visit: $$\mathbf {v}_\ast ^t = \mathrm {Transformer}(\lbrace \textit {[CLS]}\rbrace  \cup \lbrace \mathbf {o}_{c_\ast }^t| c_\ast \in \mathcal {C}_\ast ^t\rbrace )[0]$$   (Eq. 17) where [CLS] is a special token as in BERT. It is put in the first position of each visit of type $\ast $ and its final state can be used as the representation of the visit. Intuitively, it is more reasonable to use Transformers as encoders (multi-head attention based architecture) than RNN or mean/sum to aggregate multiple medical embedding for visit embedding since the set of medical codes within one visit is not ordered. It is worth noting that our Transformer encoder is different from the original one in the position embedding part. Position embedding, as an important component in Transformers and BERT, is used to encode the position and order information of each token in a sequence. However, one big difference between language sentences and EHR sequences is that the medical codes within the same visit do not generally have an order, so we remove the position embedding in our model. Pre-training We adapted the original BERT model to be more suitable for our data and task. In particular, we pre-train the model on each EHR visit (within both single-visit EHR sequences and multi-visit EHR sequences). We modified the input and pre-training objectives of the BERT model: (1) For the input, we built the Transformer encoder on the GNN outputs, i.e. ontology embeddings, for visit embedding. For the original EHR sequence, it means essentially we combine the GNN model with a Transformer to become a new integrated encoder. In addition, we removed the position embedding as we explained before. (2) As for the pre-training procedures, we modified the original pre-training tasks i.e., Masked LM (language model) task and Next Sentence prediction task to self-prediction task and dual-prediction task. The idea to conduct these tasks is to make the visit embedding absorb enough information about what it is made of and what it is able to predict. Thus, for the self-prediction task, we want the visit embedding $v^1_\ast $ to recover what it is made of, i.e., the input medical codes $\mathcal {C}_\ast ^t$ for each visit as follows: $$\begin{aligned} &\mathcal {L}_{se}(\mathbf {v}^1_\ast , \mathcal {C}_\ast ^1) = -\log p(\mathcal {C}_\ast ^1 | \mathbf {v}^1_\ast ) \\ & = - \sum _{c \in \mathcal {C}_\ast ^1}\log p(c_\ast ^1 | \mathbf {v}^1_\ast ) + \sum _{c \in \mathcal {C}_\ast \setminus \mathcal {C}_\ast ^1}\log p(c_\ast ^1 | \mathbf {v}^1_\ast ) \end{aligned}$$   (Eq. 19) we minimize the binary cross entropy loss $\mathcal {L}_s$ , and in practise, $\text{Sigmoid}(f(\mathbf {v}_\ast ))$ should be transformed by applying a fully connected neural network $f(\cdot )$ with one hidden layer. With an analogy to the Masked LM task in BERT, we also used specific symbol [MASK] to randomly replace the original medical code $c_\ast \in \mathcal {C}_\ast ^1$ . So there are $15\%$ codes in $\mathcal {C}_\ast $ which will be replaced randomly and the model should have the ability to predict the masked code based on others. Likewise, for the dual-prediction task, since the visit embedding $\mathbf {v}_\ast $ carries the information of medical codes of type $\ast $ , we can further expect it has the ability to do more task-specific prediction as follows: $$\begin{aligned} \mathcal {L}_{du} = -\log p(\mathcal {C}_d^1 | \mathbf {v}_m^1) - \log p(\mathcal {C}_m^1 | \mathbf {v}_d^1) \end{aligned}$$   (Eq. 20) where we use the same transformation function $\text{Sigmoid}(f_1(\mathbf {v}_m^1))$ , $\text{Sigmoid}(f_2(\mathbf {v}_d^1))$ with different weight matrix to transform the visit embedding and optimize the binary cross entropy loss $\mathcal {L}_{du}$ expanded same as $\mathcal {L}_{se}$ in Eq. 19 . This is a direct adaptation of the next sentence prediction task. In BERT, the next sentence prediction task facilitates the prediction of sentence relations, which is a common task in NLP. However, in healthcare, most predictive tasks do not have a sequence pair to classify. Instead, we are often interested in predicting unknown disease or medication codes of the sequence. For example, in medication recommendation, we want to predict multiple medications given only the diagnosis codes. Inversely, we can also predict unknown diagnosis given the medication codes. Thus, our final pre-training optimization objective can simply be the combination of the aforementioned losses, as shown in Eq. 21 . It is used to train on EHR data from all patients who only have one hospital visits.. $$\begin{aligned} \mathcal {L}_{pr} = \frac{1}{N} \sum _{n=1}^N (( &\mathcal {L}_{se}(\mathbf {v}_d^{1, (n)}, \mathcal {C}_d^{1, (n)}) + \mathcal {L}_{se}(\mathbf {v}_m^{1, (n)}, \mathcal {C}_m^{1, (n)})\\ & + \mathcal {L}_{du}^{(n)}) ) \end{aligned}$$   (Eq. 21) Fine-tuning After obtaining pre-trained visit representation for each visit, for a prediction task on a multi-visit sequence data, we aggregate all the visit embedding and add a prediction layer for the medication recommendation task. To be specific, from pre-training on all visits, we have a pre-trained Transformer encoder, which can then be used to get the visit embedding $\mathbf {v}_*^\tau $ at time $\tau $ . The known diagnosis codes $\mathcal {C}_d^t$ at the prediction time $t$ is also represented using the same model as $\mathbf {v}_*^t$ . Concatenating the mean of previous diagnoses visit embeddings and medication visit embeddings, also the last diagnoses visit embedding, we built an MLP based prediction layer to predict the recommended medication codes as in Equation 23 . $$\mathbf {y}_t = \mathrm {Sigmoid}(\mathbf {W}_1 [(\frac{1}{t}\sum _{\tau < t} \mathbf {v}_d^\tau ) || (\frac{1}{t}\sum _{\tau < t} \mathbf {v}_m^\tau ) ||\mathbf {v}_d^t] + b)$$   (Eq. 23) where $\mathbf {W}_1 \in \mathbb {R}^{|\mathcal {C}_m| \times 3d} $ is a learnable transformation matrix. Given the true labels $\hat{\mathbf {y}}_t$ at each time stamp $t$ , the loss function for the whole EHR sequence (i.e. a patient) is $$\mathcal {L} = - \frac{1}{T-1} \sum _{t=2}^T (\mathbf {y}_t^\intercal \log (\hat{\mathbf {y}}_t) + (1 - \mathbf {y}_t^\intercal ) \log (1 - \hat{\mathbf {y}}_t))$$   (Eq. 24) Experiment Data We used EHR data from MIMIC-III BIBREF21 and conducted all our experiments on a cohort where patients have more than one visit. We utilize data from patients with both single visit and multiple visits in the training dataset as pre-training data source (multi-visit data are split into visit slices). In this work, we transform the drug coding from NDC to ATC Third Level for using the ontology information. The statistics of the datasets are summarized in Table 2 . Baseline We compared G-BERT with the following baselines. All methods are implemented in PyTorch BIBREF22 and trained on an Ubuntu 16.04 with 8GB memory and Nvidia 1080 GPU. We also evaluated three G-BERT variants for model ablation. Metrics To measure the prediction accuracy, we used Jaccard Similarity Score (Jaccard), Average F1 (F1) and Precision Recall AUC (PR-AUC). Jaccard is defined as the size of the intersection divided by the size of the union of ground truth set $Y_t^{(k)}$ and predicted set $\hat{Y}_t^{(k)}$ . $$\text{Jaccard} = \frac{1}{\sum _k^N \sum _t^{T_k} 1}\sum _k^N \sum _t^{T_k} \frac{|Y_t^{(k)} \cap \hat{Y}_t^{(k)}|}{|Y_t^{(k)} \cup \hat{Y}_t^{(k)}|}\nonumber $$   (Eq. 36) where $N$ is the number of patients in test set and $T_k$ is the number of visits of the $k^{th}$ patient. Implementation Details We randomly divide the dataset into training, validation and testing set in a $0.6 : 0.2 : 0.2$ ratio. For G-BERT, the hyperparameters are adjusted on evaluation set: (1) GAT part: input embedding dimension as 75, number of attention heads as 4; (2) BERT part: hidden dimension as 300, dimension of position-wise feed-forward networks as 300, 2 hidden layers with 4 attention heads for each layer. Specially, we alternated the pre-training with 5 epochs and fine-tuning procedure with 5 epochs for 15 times to stabilize the training procedure. For LR, we use the grid search over typical range of hyper-parameter to search the best hyperparameter values which result in L1 norm penalty with weight as $1.1$ . For deep learning models, we implemented RNN using a gated recurrent unit (GRU) BIBREF24 and utilize dropout with a probability of 0.4 on the output of embedding. We test several embedding choice for baseline methods and determine the dimension for medical embedding as 300 and thershold for final prediction as 0.3 for better performance. Training is done through Adam BIBREF25 at learning rate 5e-4. We fix the best model on evaluation set within 100 epochs and report the performance in test set. Results Experimental Results Table. 3 compares the performance on the medication recommendation task. For variants of G-BERT, $\texttt {G-BERT}_{G^-,P^-}$ performs worse compared with $\texttt {G-BERT}_{G^-}$ and $\texttt {G-BERT}_{P^-}$ which demonstrate the effectiveness of using ontology information to get enhanced medical embedding as input and employ an unsupervised pre-training procedure on larger abundant data. Incorporating both hierarchical ontology information and pre-training procedure, the end-to-end model G-BERT has more capacity and achieve comparable results with others. As for baseline models, LR and Leap are worse than our most basic model ( $\texttt {G-BERT}_{G^-,P^-}$ ) in terms of most metrics. Comparing $\texttt {G-BERT}_{P^-}$ and GRAM, which both used medical ontology information without pre-training, the scores of our $\texttt {G-BERT}_{P^-}$ is slightly higher in all metrics. This can demonstrate the validness of using Transformer encoders and the specific prediction layer for medication recommendation. Our final model G-BERT is also better than the attention based model, RETAIN, and the recently published state-of-the-art model, GAMENet. Specifically, even adding the extra information of DDI knowledge and procedure codes, GAMENet still performs worse than G-BERT. In addition, we visualized the pre-training medical code embeddings of $\texttt {G-BERT}_{G^-}$ and G-BERT to show the effectiveness of ontology embedding using online embedding projector shown in (https://raw.githubusercontent.com/jshang123/G-Bert/master/saved/tsne.png/). Conclusion In this paper we proposed a pre-training model named G-BERT for medical code representation and medication recommendation. To our best knowledge, G-BERT is the first that utilizes language model pre-training techniques in healthcare domain. It adapted BERT to the EHR data and integrated medical ontology information using graph neural networks. By additional pre-training on the EHR from patients who only have one hospital visit which are generally discarded before model training, G-BERT outperforms all baselines in prediction accuracy on medication recommendation task. One direction for the future work is to add more auxiliary and structural tasks to improve the ability of code representaion. Another direction may be to adapt our model to be suitable for even larger datasets with more heterogeneous modalities. Acknowledgment This work was supported by the National Science Foundation award IIS-1418511, CCF-1533768 and IIS-1838042, the National Institute of Health award 1R01MD011682-01 and R56HL138415.
Is the G-BERT model useful beyond the task considered?
c4628d965983934d7a2a9797a2de6a411629d5bc
Is Ġthe ĠG - BER T Ġmodel Ġuseful Ġbeyond Ġthe Ġtask Ġconsidered ?
There Ġis Ġnothing Ġspecific Ġabout Ġthe Ġapproach Ġthat Ġdepends Ġon Ġmedical Ġrecommendations . ĠThe Ġapproach Ġcombines Ġgraph Ġdata Ġand Ġtext Ġdata Ġinto Ġa Ġsingle Ġembed ding .
<s> Is Ġthe ĠG - BER T Ġmodel Ġuseful Ġbeyond Ġthe Ġtask Ġconsidered ? </s> Introduction </s> The Ġavailability Ġof Ġmassive Ġelectronic Ġhealth Ġrecords Ġ( E HR ) Ġdata Ġand Ġthe Ġadvances Ġof Ġdeep Ġlearning Ġtechnologies Ġhave Ġprovided Ġunprecedented Ġresource Ġand Ġopportunity Ġfor Ġpredictive Ġhealthcare , Ġincluding Ġthe Ġcomputational Ġmedication Ġrecommendation Ġtask . ĠA Ġnumber Ġof Ġdeep Ġlearning Ġmodels Ġwere Ġproposed Ġto Ġassist Ġdoctors Ġin Ġmaking Ġmedication Ġrecommendation ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠThey Ġoften Ġlearn Ġrepresentations Ġfor Ġmedical Ġentities Ġ( e . g ., Ġpatients , Ġdiagnosis , Ġmedications ) Ġfrom Ġpatient ĠE HR Ġdata , Ġand Ġthen Ġuse Ġthe Ġlearned Ġrepresentations Ġto Ġpredict Ġmedications Ġthat Ġare Ġsuited Ġto Ġthe Ġpatient 's Ġhealth Ġcondition . </s> To Ġprovide Ġeffective Ġmedication Ġrecommendation , Ġit Ġis Ġimportant Ġto Ġlearn Ġaccurate Ġrepresentation Ġof Ġmedical Ġcodes . ĠDespite Ġthat Ġvarious Ġconsiderations Ġwere Ġhandled Ġin Ġprevious Ġworks Ġfor Ġimproving Ġmedical Ġcode Ġrepresentations ĠB IB REF 4 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġthere Ġare Ġtwo Ġlimitations Ġwith Ġthe Ġexisting Ġwork : </s> To Ġmitigate Ġthe Ġaforementioned Ġlimitations , Ġwe Ġpropose ĠG - BER T Ġthat Ġcombines Ġthe Ġpre - training Ġtechniques Ġand Ġgraph Ġneural Ġnetworks Ġfor Ġbetter Ġmedical Ġcode Ġrepresentation Ġand Ġmedication Ġrecommendation . ĠG - BER T Ġis Ġenabled Ġand Ġdemonstrated Ġby Ġthe Ġfollowing Ġtechnical Ġcontributions : </s> Related ĠWork </s> Med ication ĠRecommend ation ĠMed ication ĠRecommend ation Ġcan Ġbe Ġcategorized Ġinto Ġinstance - based Ġand Ġlongitudinal Ġrecommendation Ġmethods ĠB IB REF 1 Ġ. ĠInst ance - based Ġmethods Ġfocus Ġon Ġcurrent Ġhealth Ġconditions . ĠAmong Ġthem , ĠLeap ĠB IB REF 9 Ġform ulates Ġa Ġmulti - instance Ġmulti - label Ġlearning Ġframework Ġand Ġproposes Ġa Ġvariant Ġof Ġsequence - to - sequence Ġmodel Ġbased Ġon Ġcontent - att ention Ġmechanism Ġto Ġpredict Ġcombination Ġof Ġmedicines Ġgiven Ġpatient 's Ġdiagnoses . ĠLong itudinal - based Ġmethods Ġleverage Ġthe Ġtemporal Ġdependencies Ġamong Ġclinical Ġevents , Ġsee ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠAmong Ġthem , ĠR ETA IN ĠB IB REF 10 Ġuses Ġa Ġtwo - level Ġneural Ġattention Ġmodel Ġto Ġdetect Ġinfluential Ġpast Ġvisits Ġand Ġsignificant Ġclinical Ġvariables Ġwithin Ġthose Ġvisits Ġfor Ġimproved Ġmedication Ġrecommendation . </s> Pre - training ĠTechniques ĠThe Ġgoal Ġof Ġpre - training Ġtechniques Ġis Ġto Ġprovide Ġmodel Ġtraining Ġwith Ġgood Ġinitial izations . ĠPre - training Ġhas Ġbeen Ġshown Ġextremely Ġeffective Ġin Ġvarious Ġareas Ġsuch Ġas Ġimage Ġclassification ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġand Ġmachine Ġtranslation ĠB IB REF 15 Ġ. ĠThe Ġun super vised Ġpre - training Ġcan Ġbe Ġconsidered Ġas Ġa Ġregular izer Ġthat Ġsupports Ġbetter Ġgeneral ization Ġfrom Ġthe Ġtraining Ġdataset ĠB IB REF 16 Ġ. ĠRecently , Ġlanguage Ġmodel Ġpre - training Ġtechniques Ġsuch Ġas ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġhave Ġshown Ġto Ġlargely Ġimprove Ġthe Ġperformance Ġon Ġmultiple ĠN LP Ġtasks . ĠAs Ġthe Ġmost Ġwidely Ġused Ġone , ĠB ERT ĠB IB REF 7 Ġbuilds Ġon Ġthe ĠTrans former ĠB IB REF 8 Ġarchitecture Ġand Ġimproves Ġthe Ġpre - training Ġusing Ġa Ġmasked Ġlanguage Ġmodel Ġfor Ġbid irection al Ġrepresentation . ĠIn Ġthis Ġpaper , Ġwe Ġadapt Ġthe Ġframework Ġof ĠB ERT Ġand Ġpre - train Ġour Ġmodel Ġon Ġeach Ġvisit Ġof Ġthe ĠE HR Ġdata Ġto Ġleverage Ġthe Ġsingle - vis it Ġdata Ġthat Ġwere Ġnot Ġfit Ġfor Ġmodel Ġtraining Ġin Ġother Ġmedication Ġrecommendation Ġmodels . </s> Graph ĠNeural ĠNetworks Ġ( G NN ) ĠG NN s Ġare Ġneural Ġnetworks Ġthat Ġlearn Ġnode Ġor Ġgraph Ġrepresentations Ġfrom Ġgraph - struct ured Ġdata . ĠVarious Ġgraph Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġto Ġencode Ġthe Ġgraph - st ructure Ġinformation , Ġincluding Ġgraph Ġconv olution al Ġneural Ġnetworks Ġ( GC N ) ĠB IB REF 17 Ġ, Ġmessage Ġpassing Ġnetworks Ġ( MP NN ) ĠB IB REF 18 Ġ, Ġgraph Ġattention Ġnetworks Ġ( G AT ) ĠB IB REF 19 Ġ. ĠG NN s Ġhave Ġalready Ġbeen Ġdemonstrated Ġuseful Ġon ĠE HR Ġmodeling ĠB IB REF 20 Ġ, ĠB IB REF 1 Ġ. ĠGR AM ĠB IB REF 20 Ġrepresented Ġa Ġmedical Ġconcept Ġas Ġa Ġcombination Ġof Ġits Ġancestors Ġin Ġthe Ġmedical Ġont ology Ġusing Ġan Ġattention Ġmechanism . ĠIt 's Ġdifferent Ġfrom ĠG - BER T Ġfrom Ġtwo Ġaspects Ġas Ġdescribed Ġin ĠSection Ġ" Input ĠRepresent ation " Ġ. ĠAnother Ġwork Ġworth Ġmentioning Ġis ĠGAM EN et ĠB IB REF 1 Ġ, Ġwhich Ġalso Ġused Ġgraph Ġneural Ġnetwork Ġto Ġassist Ġthe Ġmedication Ġrecommendation Ġtask . ĠHowever , ĠGAM EN et Ġhas Ġa Ġdifferent Ġmotivation Ġwhich Ġresults Ġin Ġusing Ġgraph Ġneural Ġnetworks Ġon Ġdrug - drug - inter action Ġgraphs Ġinstead Ġof Ġmedical Ġont ology . </s> Problem ĠForm al ization </s> Definition Ġ1 Ġ( Long itudinal ĠPatient ĠRecords ) ĠIn Ġlongitudinal ĠE HR Ġdata , Ġeach Ġpatient Ġcan Ġbe Ġrepresented Ġas Ġa Ġsequence Ġof Ġmult ivariate Ġobservations : Ġ$ Ġ\ math cal Ġ{ X } ^{ ( n )} Ġ= Ġ\ l brace Ġ Ġ\ math cal Ġ{ X } _ 1 ^{ ( n ) }, Ġ\ math cal Ġ{ X } _ 2 ^{ ( n ) }, Ġ\ cd ots Ġ, Ġ\ math cal Ġ{ X } _{ T ^{ ( n ) }} ^{ ( n )} Ġ\ r brace Ġ Ġ$ Ġwhere Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ Ġ, Ġ$ N $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġpatients ; Ġ$ T ^{ ( n )} $ Ġis Ġthe Ġnumber Ġof Ġvisits Ġof Ġthe Ġ$ n ^{ th } $ Ġpatient . ĠHere Ġwe Ġchoose Ġtwo Ġmain Ġmedical Ġcode Ġto Ġrepresent Ġeach Ġvisit Ġ$\ math cal Ġ{ X } _ t Ġ= Ġ\ math cal Ġ{ C } _ d ^ t Ġ\ cup Ġ\ math cal Ġ{ C } _ m ^ t $ Ġof Ġa Ġpatient Ġwhich Ġis Ġa Ġunion Ġset Ġof Ġcorresponding Ġdiagnoses Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t Ġ\ sub set Ġ\ math cal Ġ{ C } _ d $ Ġand Ġmedications Ġcodes Ġ$\ math cal Ġ{ C } _ m ^ t Ġ\ sub set Ġ\ math cal Ġ{ C } _ m $ Ġ. ĠFor Ġsimplicity , Ġwe Ġuse Ġ$\ math cal Ġ{ C } _ \ ast Ġ^ t $ Ġto Ġindicate Ġthe Ġunified Ġdefinition Ġfor Ġdifferent Ġtype Ġof Ġmedical Ġcodes Ġand Ġdrop Ġthe Ġsupers cript Ġ$( n ) $ Ġfor Ġa Ġsingle Ġpatient Ġwhenever Ġit Ġis Ġunamb iguous . Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 0 Ġdenotes Ġthe Ġmedical Ġcode Ġset Ġand Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 1 Ġthe Ġsize Ġof Ġthe Ġcode Ġset . Ġ$ n \ in Ġ\ l brace Ġ1 , 2 , \ ld ots Ġ, ĠN \ r brace Ġ$ 2 Ġis Ġthe Ġmedical Ġcode . </s> Definition Ġ2 Ġ( Medical ĠOnt ology ) ĠMedical Ġcodes Ġare Ġusually Ġcategorized Ġaccording Ġto Ġa Ġtree - struct ured Ġclassification Ġsystem Ġsuch Ġas ĠI CD - 9 Ġont ol oy Ġfor Ġdiagnosis Ġand ĠA TC Ġont ology Ġfor Ġmedication . ĠWe Ġuse Ġ$\ math cal Ġ{ O } _ d , Ġ\ math cal Ġ{ O } _ m $ Ġto Ġdenote Ġthe Ġont ology Ġfor Ġdiagnosis Ġand Ġmedication . ĠSimilarly , Ġwe Ġuse Ġ$\ math cal Ġ{ O } _ \ ast Ġ$ Ġto Ġindicate Ġthe Ġunified Ġdefinition Ġfor Ġdifferent Ġtype Ġof Ġmedical Ġcodes . ĠIn Ġdet ial , Ġ$\ math cal Ġ{ O } _ \ ast Ġ= Ġ\ over line {\ math cal Ġ{ C } _ \ ast Ġ} Ġ\ cup Ġ\ math cal Ġ{ C } _ \ ast Ġ$ Ġwhere Ġ$\ over line {\ math cal Ġ{ C } _ \ ast Ġ} $ Ġdenotes Ġthe Ġcodes Ġexcluding Ġleaf Ġcodes . ĠFor Ġsimplicity , Ġwe Ġdefine Ġtwo Ġfunction Ġ$ pa (\ cd ot Ġ), Ġch (\ cd ot Ġ) $ Ġwhich Ġaccept Ġtarget Ġmedical Ġcode Ġand Ġreturn Ġancestors ' Ġcode Ġset Ġand Ġdirect Ġchild Ġcode Ġset . </s> Problem ĠDefinition Ġ( Med ication ĠRecommend ation ) ĠGiven Ġdiagnosis Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t $ Ġof Ġthe Ġvisit Ġat Ġtime Ġ$ t $ Ġ, Ġpatient Ġhistory Ġ$\ math cal Ġ{ X } _{ 1 : t } Ġ= Ġ\ l brace Ġ\ math cal Ġ{ X } _ 1 , Ġ\ math cal Ġ{ X } _ 2 , Ġ\ cd ots Ġ, Ġ\ math cal Ġ{ X } _{ t - 1 }\ r brace Ġ$ Ġ, Ġwe Ġwant Ġto Ġrecommend Ġmultiple Ġmedications Ġby Ġgenerating Ġmulti - label Ġoutput Ġ$\ hat {\ math bf Ġ{ y }} _ t Ġ\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ^ { | \ math cal Ġ{ C } _ m | } $ Ġ. </s> Method </s> The Ġoverall Ġframework Ġof ĠG - BER T Ġis Ġdescribed Ġin ĠFigure Ġ2 Ġ. ĠG - BER T Ġfirst Ġderives Ġthe Ġinitial Ġembed ding Ġof Ġmedical Ġcodes Ġfrom Ġmedical Ġont ology Ġusing Ġgraph Ġneural Ġnetworks . ĠThen , Ġin Ġorder Ġto Ġfully Ġutilize Ġthe Ġrich ĠE HR Ġdata , ĠG - BER T Ġconstructs Ġan Ġadaptive ĠB ERT Ġmodel Ġon Ġthe Ġdiscarded Ġsingle - vis it Ġdata Ġfor Ġvisit Ġrepresentation . ĠFinally Ġwe Ġadd Ġa Ġprediction Ġlayer Ġand Ġfine - t une Ġthe Ġmodel Ġin Ġthe Ġmedication Ġrecommendation Ġtask . ĠIn Ġthe Ġfollowing Ġwe Ġwill Ġdescribe ĠG - BER T Ġin Ġdetail . ĠBut Ġfirst ly , Ġwe Ġgive Ġa Ġbrief Ġbackground Ġof ĠB ERT Ġespecially Ġfor Ġthe Ġtwo Ġpre - training Ġobjectives Ġwhich Ġwill Ġbe Ġlater Ġadapted Ġto ĠE HR Ġdata Ġin ĠSection Ġ" Pre - training " Ġ. </s> Background Ġof ĠB ERT ĠBased Ġon Ġa Ġmulti - layer ĠTrans former Ġenc oder ĠB IB REF 8 Ġ( The Ġtransformer Ġarchitecture Ġhas Ġbeen Ġubiqu it ously Ġused Ġin Ġmany Ġsequence Ġmodeling Ġtasks Ġrecently , Ġso Ġwe Ġwill Ġnot Ġintroduce Ġthe Ġdetails Ġhere ), ĠB ERT Ġis Ġpre - trained Ġusing Ġtwo Ġun super vised Ġtasks : </s> A Ġtypical Ġinput Ġto ĠB ERT Ġis Ġas Ġfollows Ġ( ĠB IB REF 7 Ġ): </s> Input Ġ= Ġ[ CL S ] Ġthe Ġman Ġwent Ġto Ġ[ MAS K ] Ġstore Ġ[ SE P ] Ġhe Ġbought Ġa Ġgallon Ġ[ MAS K ] Ġmilk Ġ[ SE P ] </s> Label Ġ= ĠIs Next </s> where Ġ[ CL S ] Ġis Ġthe Ġfirst Ġtoken Ġof Ġeach Ġsentence Ġpair Ġto Ġrepresent Ġthe Ġspecial Ġclassification Ġembed ding , Ġi . e . Ġthe Ġfinal Ġstate Ġof Ġthis Ġtoken Ġis Ġused Ġas Ġthe Ġaggreg ated Ġsequence Ġrepresentation Ġfor Ġclassification Ġtasks ; Ġ[ SE P ] Ġis Ġused Ġto Ġseparate Ġtwo Ġsentences ; Ġ[ MAS K ] Ġis Ġused Ġto Ġmask Ġout Ġthe Ġpredicted Ġwords Ġin Ġthe Ġmasked Ġlanguage Ġmodel . ĠUsing Ġthis Ġform , Ġthese Ġinputs Ġfacilitate Ġthe Ġtwo Ġtasks Ġdescribed Ġabove , Ġand Ġthey Ġwill Ġalso Ġbe Ġused Ġin Ġour Ġmethod Ġdescription Ġin Ġthe Ġfollowing Ġsection . </s> Input ĠRepresent ation </s> The ĠG - BER T Ġmodel Ġtakes Ġmedical Ġcodes ' Ġont ology Ġembed d ings Ġas Ġinput , Ġand Ġob tains Ġintermediate Ġrepresentations Ġfrom Ġa ĠTrans former Ġenc oder Ġas Ġthe Ġvisit Ġembed d ings . ĠIt Ġis Ġthen Ġpre - trained Ġon ĠE HR Ġfrom Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisit . ĠThe Ġderived Ġenc oder Ġand Ġvisit Ġembed ding Ġwill Ġbe Ġfed Ġinto Ġa Ġclass ifier Ġand Ġfine - tun ed Ġto Ġmake Ġpredictions . </s> Ont ology ĠEmb edd ing ĠWe Ġconstructed Ġont ology Ġembed ding Ġfrom Ġdiagnosis Ġont ology Ġ$\ math cal Ġ{ O } _ d $ Ġand Ġmedication Ġont ology Ġ$\ math cal Ġ{ O } _ m $ Ġ. ĠSince Ġthe Ġmedical Ġcodes Ġin Ġraw ĠE HR Ġdata Ġcan Ġbe Ġconsidered Ġas Ġleaf Ġnodes Ġin Ġthese Ġont ology Ġtrees , Ġwe Ġcan Ġenhance Ġthe Ġmedical Ġcode Ġembed ding Ġusing Ġgraph Ġneural Ġnetworks Ġ( G NN s ) Ġto Ġintegrate Ġthe Ġancestors ' Ġinformation Ġof Ġthese Ġcodes . ĠHere Ġwe Ġperform Ġa Ġtwo - stage Ġprocedure Ġwith Ġa Ġspecially Ġdesigned ĠG NN Ġfor Ġont ology Ġembed ding . </s> To Ġstart , Ġwe Ġassign Ġan Ġinitial Ġembed ding Ġvector Ġto Ġevery Ġmedical Ġcode Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ O } _ \ ast Ġ$ Ġwith Ġa Ġlearn able Ġembed ding Ġmatrix Ġ$\ math bf Ġ{ W } _ e Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ O } _ \ ast Ġ| Ġ\ times Ġd } $ Ġwhere Ġ$ d $ Ġis Ġthe Ġembed ding Ġdimension . </s> Stage Ġ1 . ĠFor Ġeach Ġnon - leaf Ġnode Ġ$ c _ \ ast Ġ\ in Ġ\ over line {\ math cal Ġ{ C } _ \ ast Ġ} $ Ġ, Ġwe Ġobtain Ġits Ġenhanced Ġmedical Ġembed ding Ġ$\ math bf Ġ{ h } _{ c _ \ ast Ġ} Ġ\ in Ġ\ math bb Ġ{ R } ^{ d } $ Ġas Ġfollows : </s> $$ \ math bf Ġ{ h } _{ c _ \ ast Ġ} Ġ= Ġg ( c _ \ ast Ġ, Ġch ( c _ \ ast Ġ), Ġ\ math bf Ġ{ W } _ e ) $$ Ġ Ġ Ġ( E q . Ġ11 ) </s> where Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġis Ġan Ġaggregation Ġfunction Ġwhich Ġaccepts Ġthe Ġtarget Ġmedical Ġcode Ġ$ c _ \ ast Ġ$ Ġ, Ġits Ġdirect Ġchild Ġcodes Ġ$ ch ( c _ \ ast Ġ) $ Ġand Ġinitial Ġembed ding Ġmatrix . ĠInt uitive ly , Ġthe Ġaggregation Ġfunction Ġcan Ġpass Ġand Ġfuse Ġinformation Ġin Ġtarget Ġnode Ġfrom Ġits Ġdirect Ġchildren Ġwhich Ġresult Ġin Ġthe Ġmore Ġrelated Ġembed ding Ġof Ġancestor ' Ġcode Ġto Ġchild Ġcodes ' Ġembed ding . </s> Stage Ġ2 . ĠAfter Ġobtaining Ġenhanced Ġembed d ings , Ġwe Ġpass Ġthe Ġenhance Ġembed ding Ġmatrix Ġ$\ math bf Ġ{ H } _ e Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ O } _ \ ast Ġ| Ġ\ times Ġd } $ Ġback Ġto Ġget Ġont ology Ġembed ding Ġfor Ġleaf Ġcodes Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ$ Ġas Ġfollows : </s> $$ \ math bf Ġ{ o } _{ c _ \ ast Ġ} Ġ= Ġg ( c _ \ ast Ġ, Ġpa ( c _ \ ast Ġ), Ġ\ math bf Ġ{ H } _ e ) $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> where Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġaccepts Ġancestor Ġcodes Ġof Ġtarget Ġmedical Ġcode Ġ$ c _ \ ast Ġ$ Ġ. ĠHere , Ġwe Ġuse Ġ$ pa ( c _ \ ast Ġ) $ Ġinstead Ġof Ġ$ ch ( c _ \ ast Ġ) $ Ġ, Ġsince Ġutilizing Ġthe Ġancestors ' Ġembed ding Ġcan Ġindirectly Ġassociate Ġall Ġmedical Ġcodes Ġinstead Ġof Ġtaking Ġeach Ġleaf Ġcode Ġas Ġindependent Ġinput . </s> The Ġoption Ġfor Ġthe Ġaggregation Ġfunction Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġis Ġflexible , Ġincluding Ġsum , Ġmean . ĠHere Ġwe Ġchoose Ġthe Ġone Ġfrom Ġgraph Ġattention Ġnetworks Ġ( G AT ) ĠB IB REF 19 Ġ, Ġwhich Ġhas Ġshown Ġefficient Ġembed ding Ġlearning Ġability Ġon Ġgraph - struct ured Ġtasks , Ġe . g ., Ġnode Ġclassification Ġand Ġlink Ġprediction . ĠIn Ġparticular , Ġwe Ġimplement Ġthe Ġaggregation Ġfunction Ġ$ g (\ cd ot Ġ, \ cd ot Ġ, \ cd ot Ġ) $ Ġas Ġfollows : </s> $$ g Ġ( c _ \ ast Ġ, Ġp ( c _ \ ast Ġ), Ġ\ math bf Ġ{ H } _ e ) Ġ= Ġ\ oper at orn am ew ith limits {\ scale box Ġ{ 1 } [ 1 . 5 ] {\ par allel Ġ}} _{ k = 1 } ^ K Ġ\ s igma Ġ\ left ( Ġ\ sum Ġ_ { j Ġ\ in Ġ\ l brace Ġc _ \ ast Ġ\ r brace Ġ Ġ\ cup Ġpa ( c _ \ ast Ġ) } Ġ\ alpha Ġ_ { i , j } ^ k Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ j Ġ\ right ) $$ Ġ Ġ Ġ( E q . Ġ13 ) </s> where Ġ$\ oper at orn am ew ith limits {\ scale box Ġ{ 1 } [ 1 . 5 ] {\ par allel Ġ}} $ Ġrepresents Ġconc aten ation Ġwhich Ġenables Ġthe Ġmulti - head Ġattention Ġmechanism , Ġ$\ s igma Ġ$ Ġis Ġa Ġnon linear Ġactivation Ġfunction , Ġ$\ math bf Ġ{ W } ^ k Ġ\ in Ġ\ math bb Ġ{ R } ^{ m Ġ\ times Ġd } $ Ġis Ġthe Ġweight Ġmatrix Ġfor Ġinput Ġtransformation , Ġand Ġ$\ alpha Ġ_ { i , j } ^ k $ Ġare Ġthe Ġcorresponding Ġ$ k $ Ġ- th Ġnormalized Ġattention Ġcoefficients Ġcomputed Ġas Ġfollows : </s> $$ \ alpha Ġ_ { i , j } ^ k Ġ= Ġ\ frac { e ^ {\ left (\ text { Le aky Re LU } (\ math bf Ġ{ a } ^ \ inter cal Ġ[ \ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ i Ġ|| Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ j ]) \ right ) }} {\ sum Ġ_ { k Ġ\ in Ġ\ math cal Ġ{ N } _ i } Ġe ^ {\ left (\ text { Le aky Re LU } (\ math bf Ġ{ a } ^ \ inter cal Ġ[ \ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ i Ġ|| Ġ\ math bf Ġ{ W } ^ k Ġ\ math bf Ġ{ h } _ k ]) \ right ) }} $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> where Ġ$\ math bf Ġ{ a } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 m } $ Ġis Ġa Ġlearn able Ġweight Ġvector Ġand ĠLe aky Re LU Ġis Ġa Ġnon linear Ġfunction . Ġ( we Ġassume Ġ$ m Ġ= Ġd / K $ Ġ). </s> As Ġshown Ġin ĠFigure Ġ2 Ġ, Ġwe Ġconstruct ĠI CD - 9 Ġtree Ġfor Ġdiagnosis Ġand ĠA TC Ġtree Ġfor Ġmedication Ġusing Ġthe Ġsame Ġstructure . ĠHere Ġthe Ġdirection Ġof Ġarrow Ġshows Ġthe Ġinformation Ġflow Ġwhere Ġancestor Ġnodes Ġcan Ġget Ġinformation Ġfrom Ġtheir Ġdirect Ġchildren Ġ( in Ġstage Ġ1 ) Ġand Ġsimilarly Ġleaf Ġnodes Ġcan Ġget Ġinformation Ġfrom Ġtheir Ġconnected Ġancestors Ġ( in Ġstage Ġ2 ). </s> It Ġis Ġworth Ġmentioning Ġthat Ġour Ġgraph Ġembed ding Ġmethod Ġon Ġmedical Ġont ology Ġis Ġdifferent Ġfrom ĠGR AM ĠB IB REF 20 Ġfrom Ġthe Ġfollowing Ġtwo Ġaspects : </s> Initial ization : Ġwe Ġinitialize Ġall Ġthe Ġnode Ġembed d ings Ġfrom Ġa Ġlearn able Ġembed ding Ġmatrix , Ġwhile ĠGR AM Ġlearns Ġthem Ġusing ĠGl ove Ġfrom Ġthe Ġco - occ urrence Ġinformation . </s> Up dating : Ġwe Ġdevelop Ġa Ġtwo - step Ġupdating Ġfunction Ġfor Ġboth Ġleaf Ġnodes Ġand Ġancestor Ġnodes ; Ġwhile Ġin ĠGR AM , Ġonly Ġthe Ġleaf Ġnodes Ġare Ġupdated Ġ( as Ġa Ġcombination Ġof Ġtheir Ġancestor Ġnodes Ġand Ġthemselves ). </s> Visit ĠEmb edd ing ĠSimilar Ġto ĠB ERT , Ġwe Ġuse Ġa Ġmulti - layer ĠTrans former Ġarchitecture ĠB IB REF 8 Ġas Ġour Ġvisit Ġenc oder . ĠThe Ġmodel Ġtakes Ġthe Ġont ology Ġembed ding Ġas Ġinput Ġand Ġderive Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ \ ast Ġ^ t Ġ\ in Ġ\ math bb Ġ{ R } ^ d $ Ġfor Ġa Ġpatient Ġat Ġ$ t $ Ġ- th Ġvisit : </s> $$ \ math bf Ġ{ v } _ \ ast Ġ^ t Ġ= Ġ\ math rm Ġ{ Trans former } (\ l brace Ġ\ text it Ġ{ [ CL S ] }\ r brace Ġ Ġ\ cup Ġ\ l brace Ġ\ math bf Ġ{ o } _{ c _ \ ast Ġ} ^ t | Ġc _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ t \ r brace Ġ) [ 0 ] $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġ[ CL S ] Ġis Ġa Ġspecial Ġtoken Ġas Ġin ĠB ERT . ĠIt Ġis Ġput Ġin Ġthe Ġfirst Ġposition Ġof Ġeach Ġvisit Ġof Ġtype Ġ$\ ast Ġ$ Ġand Ġits Ġfinal Ġstate Ġcan Ġbe Ġused Ġas Ġthe Ġrepresentation Ġof Ġthe Ġvisit . ĠInt uitive ly , Ġit Ġis Ġmore Ġreasonable Ġto Ġuse ĠTransformers Ġas Ġenc od ers Ġ( multi - head Ġattention Ġbased Ġarchitecture ) Ġthan ĠR NN Ġor Ġmean / sum Ġto Ġaggregate Ġmultiple Ġmedical Ġembed ding Ġfor Ġvisit Ġembed ding Ġsince Ġthe Ġset Ġof Ġmedical Ġcodes Ġwithin Ġone Ġvisit Ġis Ġnot Ġordered . </s> It Ġis Ġworth Ġnoting Ġthat Ġour ĠTrans former Ġenc oder Ġis Ġdifferent Ġfrom Ġthe Ġoriginal Ġone Ġin Ġthe Ġposition Ġembed ding Ġpart . ĠPosition Ġembed ding , Ġas Ġan Ġimportant Ġcomponent Ġin ĠTransformers Ġand ĠB ERT , Ġis Ġused Ġto Ġencode Ġthe Ġposition Ġand Ġorder Ġinformation Ġof Ġeach Ġtoken Ġin Ġa Ġsequence . ĠHowever , Ġone Ġbig Ġdifference Ġbetween Ġlanguage Ġsentences Ġand ĠE HR Ġsequences Ġis Ġthat Ġthe Ġmedical Ġcodes Ġwithin Ġthe Ġsame Ġvisit Ġdo Ġnot Ġgenerally Ġhave Ġan Ġorder , Ġso Ġwe Ġremove Ġthe Ġposition Ġembed ding Ġin Ġour Ġmodel . </s> Pre - training </s> We Ġadapted Ġthe Ġoriginal ĠB ERT Ġmodel Ġto Ġbe Ġmore Ġsuitable Ġfor Ġour Ġdata Ġand Ġtask . ĠIn Ġparticular , Ġwe Ġpre - train Ġthe Ġmodel Ġon Ġeach ĠE HR Ġvisit Ġ( within Ġboth Ġsingle - vis it ĠE HR Ġsequences Ġand Ġmulti - vis it ĠE HR Ġsequences ). ĠWe Ġmodified Ġthe Ġinput Ġand Ġpre - training Ġobjectives Ġof Ġthe ĠB ERT Ġmodel : Ġ( 1 ) ĠFor Ġthe Ġinput , Ġwe Ġbuilt Ġthe ĠTrans former Ġenc oder Ġon Ġthe ĠG NN Ġoutputs , Ġi . e . Ġont ology Ġembed d ings , Ġfor Ġvisit Ġembed ding . ĠFor Ġthe Ġoriginal ĠE HR Ġsequence , Ġit Ġmeans Ġessentially Ġwe Ġcombine Ġthe ĠG NN Ġmodel Ġwith Ġa ĠTrans former Ġto Ġbecome Ġa Ġnew Ġintegrated Ġenc oder . ĠIn Ġaddition , Ġwe Ġremoved Ġthe Ġposition Ġembed ding Ġas Ġwe Ġexplained Ġbefore . Ġ( 2 ) ĠAs Ġfor Ġthe Ġpre - training Ġprocedures , Ġwe Ġmodified Ġthe Ġoriginal Ġpre - training Ġtasks Ġi . e ., ĠMask ed ĠLM Ġ( language Ġmodel ) Ġtask Ġand ĠNext ĠSent ence Ġprediction Ġtask Ġto Ġself - pred iction Ġtask Ġand Ġdual - pred iction Ġtask . ĠThe Ġidea Ġto Ġconduct Ġthese Ġtasks Ġis Ġto Ġmake Ġthe Ġvisit Ġembed ding Ġabsorb Ġenough Ġinformation Ġabout Ġwhat Ġit Ġis Ġmade Ġof Ġand Ġwhat Ġit Ġis Ġable Ġto Ġpredict . </s> Thus , Ġfor Ġthe Ġself - pred iction Ġtask , Ġwe Ġwant Ġthe Ġvisit Ġembed ding Ġ$ v ^ 1 _ \ ast Ġ$ Ġto Ġrecover Ġwhat Ġit Ġis Ġmade Ġof , Ġi . e ., Ġthe Ġinput Ġmedical Ġcodes Ġ$\ math cal Ġ{ C } _ \ ast Ġ^ t $ Ġfor Ġeach Ġvisit Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ& \ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } ^ 1 _ \ ast Ġ, Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 ) Ġ= Ġ- \ log Ġp (\ math cal Ġ{ C } _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ\\ Ġ& Ġ= Ġ- Ġ\ sum Ġ_ { c Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 }\ log Ġp ( c _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ+ Ġ\ sum Ġ_ { c Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ\ set minus Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 }\ log Ġp ( c _ \ ast Ġ^ 1 Ġ| Ġ\ math bf Ġ{ v } ^ 1 _ \ ast Ġ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> we Ġminimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġ$\ math cal Ġ{ L } _ s $ Ġ, Ġand Ġin Ġpractise , Ġ$\ text { S igm oid } ( f (\ math bf Ġ{ v } _ \ ast Ġ)) $ Ġshould Ġbe Ġtransformed Ġby Ġapplying Ġa Ġfully Ġconnected Ġneural Ġnetwork Ġ$ f (\ cd ot Ġ) $ Ġwith Ġone Ġhidden Ġlayer . ĠWith Ġan Ġanalogy Ġto Ġthe ĠMask ed ĠLM Ġtask Ġin ĠB ERT , Ġwe Ġalso Ġused Ġspecific Ġsymbol Ġ[ MAS K ] Ġto Ġrandomly Ġreplace Ġthe Ġoriginal Ġmedical Ġcode Ġ$ c _ \ ast Ġ\ in Ġ\ math cal Ġ{ C } _ \ ast Ġ^ 1 $ Ġ. ĠSo Ġthere Ġare Ġ$ 15 \ % $ Ġcodes Ġin Ġ$\ math cal Ġ{ C } _ \ ast Ġ$ Ġwhich Ġwill Ġbe Ġreplaced Ġrandomly Ġand Ġthe Ġmodel Ġshould Ġhave Ġthe Ġability Ġto Ġpredict Ġthe Ġmasked Ġcode Ġbased Ġon Ġothers . </s> Likewise , Ġfor Ġthe Ġdual - pred iction Ġtask , Ġsince Ġthe Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ \ ast Ġ$ Ġcarries Ġthe Ġinformation Ġof Ġmedical Ġcodes Ġof Ġtype Ġ$\ ast Ġ$ Ġ, Ġwe Ġcan Ġfurther Ġexpect Ġit Ġhas Ġthe Ġability Ġto Ġdo Ġmore Ġtask - specific Ġprediction Ġas Ġfollows : </s> $$ \ begin { aligned } Ġ\ math cal Ġ{ L } _{ du } Ġ= Ġ- \ log Ġp (\ math cal Ġ{ C } _ d ^ 1 Ġ| Ġ\ math bf Ġ{ v } _ m ^ 1 ) Ġ- Ġ\ log Ġp (\ math cal Ġ{ C } _ m ^ 1 Ġ| Ġ\ math bf Ġ{ v } _ d ^ 1 ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ20 ) </s> where Ġwe Ġuse Ġthe Ġsame Ġtransformation Ġfunction Ġ$\ text { S igm oid } ( f _ 1 (\ math bf Ġ{ v } _ m ^ 1 )) $ Ġ, Ġ$\ text { S igm oid } ( f _ 2 (\ math bf Ġ{ v } _ d ^ 1 )) $ Ġwith Ġdifferent Ġweight Ġmatrix Ġto Ġtransform Ġthe Ġvisit Ġembed ding Ġand Ġoptimize Ġthe Ġbinary Ġcross Ġentropy Ġloss Ġ$\ math cal Ġ{ L } _{ du } $ Ġexpanded Ġsame Ġas Ġ$\ math cal Ġ{ L } _{ se } $ Ġin ĠE q . Ġ19 Ġ. ĠThis Ġis Ġa Ġdirect Ġadaptation Ġof Ġthe Ġnext Ġsentence Ġprediction Ġtask . ĠIn ĠB ERT , Ġthe Ġnext Ġsentence Ġprediction Ġtask Ġfacilitates Ġthe Ġprediction Ġof Ġsentence Ġrelations , Ġwhich Ġis Ġa Ġcommon Ġtask Ġin ĠN LP . ĠHowever , Ġin Ġhealthcare , Ġmost Ġpredictive Ġtasks Ġdo Ġnot Ġhave Ġa Ġsequence Ġpair Ġto Ġclassify . ĠInstead , Ġwe Ġare Ġoften Ġinterested Ġin Ġpredicting Ġunknown Ġdisease Ġor Ġmedication Ġcodes Ġof Ġthe Ġsequence . ĠFor Ġexample , Ġin Ġmedication Ġrecommendation , Ġwe Ġwant Ġto Ġpredict Ġmultiple Ġmedications Ġgiven Ġonly Ġthe Ġdiagnosis Ġcodes . ĠIn versely , Ġwe Ġcan Ġalso Ġpredict Ġunknown Ġdiagnosis Ġgiven Ġthe Ġmedication Ġcodes . </s> Thus , Ġour Ġfinal Ġpre - training Ġoptimization Ġobjective Ġcan Ġsimply Ġbe Ġthe Ġcombination Ġof Ġthe Ġaforementioned Ġlosses , Ġas Ġshown Ġin ĠE q . Ġ21 Ġ. ĠIt Ġis Ġused Ġto Ġtrain Ġon ĠE HR Ġdata Ġfrom Ġall Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisits .. </s> $$ \ begin { aligned } Ġ\ math cal Ġ{ L } _{ pr } Ġ= Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { n = 1 } ^ N Ġ(( Ġ& \ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } _ d ^{ 1 , Ġ( n ) }, Ġ\ math cal Ġ{ C } _ d ^{ 1 , Ġ( n ) }) Ġ+ Ġ\ math cal Ġ{ L } _{ se } (\ math bf Ġ{ v } _ m ^{ 1 , Ġ( n ) }, Ġ\ math cal Ġ{ C } _ m ^{ 1 , Ġ( n ) }) \\ Ġ& Ġ+ Ġ\ math cal Ġ{ L } _{ du } ^{ ( n ) }) Ġ) Ġ\ end { aligned } $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> Fine - tun ing </s> After Ġobtaining Ġpre - trained Ġvisit Ġrepresentation Ġfor Ġeach Ġvisit , Ġfor Ġa Ġprediction Ġtask Ġon Ġa Ġmulti - vis it Ġsequence Ġdata , Ġwe Ġaggregate Ġall Ġthe Ġvisit Ġembed ding Ġand Ġadd Ġa Ġprediction Ġlayer Ġfor Ġthe Ġmedication Ġrecommendation Ġtask . ĠTo Ġbe Ġspecific , Ġfrom Ġpre - training Ġon Ġall Ġvisits , Ġwe Ġhave Ġa Ġpre - trained ĠTrans former Ġenc oder , Ġwhich Ġcan Ġthen Ġbe Ġused Ġto Ġget Ġthe Ġvisit Ġembed ding Ġ$\ math bf Ġ{ v } _ * ^ \ t au Ġ$ Ġat Ġtime Ġ$\ t au Ġ$ Ġ. ĠThe Ġknown Ġdiagnosis Ġcodes Ġ$\ math cal Ġ{ C } _ d ^ t $ Ġat Ġthe Ġprediction Ġtime Ġ$ t $ Ġis Ġalso Ġrepresented Ġusing Ġthe Ġsame Ġmodel Ġas Ġ$\ math bf Ġ{ v } _ * ^ t $ Ġ. ĠCon cat en ating Ġthe Ġmean Ġof Ġprevious Ġdiagnoses Ġvisit Ġembed d ings Ġand Ġmedication Ġvisit Ġembed d ings , Ġalso Ġthe Ġlast Ġdiagnoses Ġvisit Ġembed ding , Ġwe Ġbuilt Ġan ĠML P Ġbased Ġprediction Ġlayer Ġto Ġpredict Ġthe Ġrecommended Ġmedication Ġcodes Ġas Ġin ĠEqu ation Ġ23 Ġ. </s> $$ \ math bf Ġ{ y } _ t Ġ= Ġ\ math rm Ġ{ S igm oid } (\ math bf Ġ{ W } _ 1 Ġ[ (\ frac { 1 }{ t }\ sum Ġ_ {\ t au Ġ< Ġt } Ġ\ math bf Ġ{ v } _ d ^ \ t au Ġ) Ġ|| Ġ( \ frac { 1 }{ t }\ sum Ġ_ {\ t au Ġ< Ġt } Ġ\ math bf Ġ{ v } _ m ^ \ t au Ġ) Ġ|| \ math bf Ġ{ v } _ d ^ t ] Ġ+ Ġb ) $$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ math bf Ġ{ W } _ 1 Ġ\ in Ġ\ math bb Ġ{ R } ^{ | \ math cal Ġ{ C } _ m | Ġ\ times Ġ3 d } Ġ$ Ġis Ġa Ġlearn able Ġtransformation Ġmatrix . </s> Given Ġthe Ġtrue Ġlabels Ġ$\ hat {\ math bf Ġ{ y }} _ t $ Ġat Ġeach Ġtime Ġstamp Ġ$ t $ Ġ, Ġthe Ġloss Ġfunction Ġfor Ġthe Ġwhole ĠE HR Ġsequence Ġ( i . e . Ġa Ġpatient ) Ġis </s> $$ \ math cal Ġ{ L } Ġ= Ġ- Ġ\ frac { 1 }{ T - 1 } Ġ\ sum Ġ_ { t = 2 } ^ T Ġ( \ math bf Ġ{ y } _ t ^ \ inter cal Ġ\ log Ġ( \ hat {\ math bf Ġ{ y }} _ t ) Ġ+ Ġ( 1 Ġ- Ġ\ math bf Ġ{ y } _ t ^ \ inter cal Ġ) Ġ\ log Ġ( 1 Ġ- Ġ\ hat {\ math bf Ġ{ y }} _ t )) $$ Ġ Ġ Ġ( E q . Ġ24 ) </s> Exper iment </s> Data ĠWe Ġused ĠE HR Ġdata Ġfrom ĠM IM IC - III ĠB IB REF 21 Ġand Ġconducted Ġall Ġour Ġexperiments Ġon Ġa Ġcohort Ġwhere Ġpatients Ġhave Ġmore Ġthan Ġone Ġvisit . ĠWe Ġutilize Ġdata Ġfrom Ġpatients Ġwith Ġboth Ġsingle Ġvisit Ġand Ġmultiple Ġvisits Ġin Ġthe Ġtraining Ġdataset Ġas Ġpre - training Ġdata Ġsource Ġ( multi - vis it Ġdata Ġare Ġsplit Ġinto Ġvisit Ġslices ). ĠIn Ġthis Ġwork , Ġwe Ġtransform Ġthe Ġdrug Ġcoding Ġfrom ĠN DC Ġto ĠA TC ĠThird ĠLevel Ġfor Ġusing Ġthe Ġont ology Ġinformation . ĠThe Ġstatistics Ġof Ġthe Ġdatasets Ġare Ġsummarized Ġin ĠTable Ġ2 Ġ. </s> Bas eline ĠWe Ġcompared ĠG - BER T Ġwith Ġthe Ġfollowing Ġbas elines . ĠAll Ġmethods Ġare Ġimplemented Ġin ĠPy Tor ch ĠB IB REF 22 Ġand Ġtrained Ġon Ġan ĠUbuntu Ġ16 . 04 Ġwith Ġ8 GB Ġmemory Ġand ĠNvidia Ġ1080 ĠGPU . </s> We Ġalso Ġevaluated Ġthree ĠG - BER T Ġvariants Ġfor Ġmodel Ġab lation . </s> Met rics ĠTo Ġmeasure Ġthe Ġprediction Ġaccuracy , Ġwe Ġused ĠJ acc ard ĠSimilar ity ĠScore Ġ( J acc ard ), ĠAverage ĠF 1 Ġ( F 1 ) Ġand ĠPrecision ĠRecall ĠA UC Ġ( PR - A UC ). ĠJ acc ard Ġis Ġdefined Ġas Ġthe Ġsize Ġof Ġthe Ġintersection Ġdivided Ġby Ġthe Ġsize Ġof Ġthe Ġunion Ġof Ġground Ġtruth Ġset Ġ$ Y _ t ^{ ( k )} $ Ġand Ġpredicted Ġset Ġ$\ hat { Y } _ t ^{ ( k )} $ Ġ. </s> $$ \ text { J acc ard } Ġ= Ġ\ frac { 1 }{ \ sum Ġ_ k ^ N Ġ\ sum Ġ_ t ^{ T _ k } Ġ1 }\ sum Ġ_ k ^ N Ġ\ sum Ġ_ t ^{ T _ k } Ġ\ frac { | Y _ t ^{ ( k )} Ġ\ cap Ġ\ hat { Y } _ t ^{ ( k )} | }{ | Y _ t ^{ ( k )} Ġ\ cup Ġ\ hat { Y } _ t ^{ ( k )} | }\ non umber Ġ$$ Ġ Ġ Ġ( E q . Ġ36 ) </s> where Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġpatients Ġin Ġtest Ġset Ġand Ġ$ T _ k $ Ġis Ġthe Ġnumber Ġof Ġvisits Ġof Ġthe Ġ$ k ^{ th } $ Ġpatient . </s> Im plementation ĠDetails ĠWe Ġrandomly Ġdivide Ġthe Ġdataset Ġinto Ġtraining , Ġvalidation Ġand Ġtesting Ġset Ġin Ġa Ġ$ 0 . 6 Ġ: Ġ0 . 2 Ġ: Ġ0 . 2 $ Ġratio . ĠFor ĠG - BER T , Ġthe Ġhyper param eters Ġare Ġadjusted Ġon Ġevaluation Ġset : Ġ( 1 ) ĠG AT Ġpart : Ġinput Ġembed ding Ġdimension Ġas Ġ75 , Ġnumber Ġof Ġattention Ġheads Ġas Ġ4 ; Ġ( 2 ) ĠB ERT Ġpart : Ġhidden Ġdimension Ġas Ġ300 , Ġdimension Ġof Ġposition - wise Ġfeed - forward Ġnetworks Ġas Ġ300 , Ġ2 Ġhidden Ġlayers Ġwith Ġ4 Ġattention Ġheads Ġfor Ġeach Ġlayer . ĠS pecially , Ġwe Ġaltern ated Ġthe Ġpre - training Ġwith Ġ5 Ġepoch s Ġand Ġfine - tun ing Ġprocedure Ġwith Ġ5 Ġepoch s Ġfor Ġ15 Ġtimes Ġto Ġstabilize Ġthe Ġtraining Ġprocedure . </s> For ĠLR , Ġwe Ġuse Ġthe Ġgrid Ġsearch Ġover Ġtypical Ġrange Ġof Ġhyper - param eter Ġto Ġsearch Ġthe Ġbest Ġhyper param eter Ġvalues Ġwhich Ġresult Ġin ĠL 1 Ġnorm Ġpenalty Ġwith Ġweight Ġas Ġ$ 1 . 1 $ Ġ. ĠFor Ġdeep Ġlearning Ġmodels , Ġwe Ġimplemented ĠR NN Ġusing Ġa Ġg ated Ġrecurrent Ġunit Ġ( GR U ) ĠB IB REF 24 Ġand Ġutilize Ġdrop out Ġwith Ġa Ġprobability Ġof Ġ0 . 4 Ġon Ġthe Ġoutput Ġof Ġembed ding . ĠWe Ġtest Ġseveral Ġembed ding Ġchoice Ġfor Ġbaseline Ġmethods Ġand Ġdetermine Ġthe Ġdimension Ġfor Ġmedical Ġembed ding Ġas Ġ300 Ġand Ġther sh old Ġfor Ġfinal Ġprediction Ġas Ġ0 . 3 Ġfor Ġbetter Ġperformance . ĠTraining Ġis Ġdone Ġthrough ĠAdam ĠB IB REF 25 Ġat Ġlearning Ġrate Ġ5 e - 4 . ĠWe Ġfix Ġthe Ġbest Ġmodel Ġon Ġevaluation Ġset Ġwithin Ġ100 Ġepoch s Ġand Ġreport Ġthe Ġperformance Ġin Ġtest Ġset . </s> Results </s> Exper imental ĠResults ĠTable . Ġ3 Ġcompares Ġthe Ġperformance Ġon Ġthe Ġmedication Ġrecommendation Ġtask . ĠFor Ġvariants Ġof ĠG - BER T , Ġ$\ text tt Ġ{ G - BER T } _{ G ^ -, P ^ - } $ Ġperforms Ġworse Ġcompared Ġwith Ġ$\ text tt Ġ{ G - BER T } _{ G ^ - } $ Ġand Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġwhich Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġusing Ġont ology Ġinformation Ġto Ġget Ġenhanced Ġmedical Ġembed ding Ġas Ġinput Ġand Ġemploy Ġan Ġun super vised Ġpre - training Ġprocedure Ġon Ġlarger Ġabundant Ġdata . ĠInc orpor ating Ġboth Ġhierarchical Ġont ology Ġinformation Ġand Ġpre - training Ġprocedure , Ġthe Ġend - to - end Ġmodel ĠG - BER T Ġhas Ġmore Ġcapacity Ġand Ġachieve Ġcomparable Ġresults Ġwith Ġothers . </s> As Ġfor Ġbaseline Ġmodels , ĠLR Ġand ĠLeap Ġare Ġworse Ġthan Ġour Ġmost Ġbasic Ġmodel Ġ( Ġ$\ text tt Ġ{ G - BER T } _{ G ^ -, P ^ - } $ Ġ) Ġin Ġterms Ġof Ġmost Ġmetrics . ĠComp aring Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġand ĠGR AM , Ġwhich Ġboth Ġused Ġmedical Ġont ology Ġinformation Ġwithout Ġpre - training , Ġthe Ġscores Ġof Ġour Ġ$\ text tt Ġ{ G - BER T } _{ P ^ - } $ Ġis Ġslightly Ġhigher Ġin Ġall Ġmetrics . ĠThis Ġcan Ġdemonstrate Ġthe Ġvalid ness Ġof Ġusing ĠTrans former Ġenc od ers Ġand Ġthe Ġspecific Ġprediction Ġlayer Ġfor Ġmedication Ġrecommendation . ĠOur Ġfinal Ġmodel ĠG - BER T Ġis Ġalso Ġbetter Ġthan Ġthe Ġattention Ġbased Ġmodel , ĠR ETA IN , Ġand Ġthe Ġrecently Ġpublished Ġstate - of - the - art Ġmodel , ĠGAM EN et . ĠSpecifically , Ġeven Ġadding Ġthe Ġextra Ġinformation Ġof ĠD DI Ġknowledge Ġand Ġprocedure Ġcodes , ĠGAM EN et Ġstill Ġperforms Ġworse Ġthan ĠG - BER T . </s> In Ġaddition , Ġwe Ġvisual ized Ġthe Ġpre - training Ġmedical Ġcode Ġembed d ings Ġof Ġ$\ text tt Ġ{ G - BER T } _{ G ^ - } $ Ġand ĠG - BER T Ġto Ġshow Ġthe Ġeffectiveness Ġof Ġont ology Ġembed ding Ġusing Ġonline Ġembed ding Ġprojector Ġshown Ġin Ġ( https :// raw . github usercontent . com / j sh ang 123 / G - B ert / master / s aved / ts ne . png / ). </s> Conclusion </s> In Ġthis Ġpaper Ġwe Ġproposed Ġa Ġpre - training Ġmodel Ġnamed ĠG - BER T Ġfor Ġmedical Ġcode Ġrepresentation Ġand Ġmedication Ġrecommendation . ĠTo Ġour Ġbest Ġknowledge , ĠG - BER T Ġis Ġthe Ġfirst Ġthat Ġutilizes Ġlanguage Ġmodel Ġpre - training Ġtechniques Ġin Ġhealthcare Ġdomain . ĠIt Ġadapted ĠB ERT Ġto Ġthe ĠE HR Ġdata Ġand Ġintegrated Ġmedical Ġont ology Ġinformation Ġusing Ġgraph Ġneural Ġnetworks . ĠBy Ġadditional Ġpre - training Ġon Ġthe ĠE HR Ġfrom Ġpatients Ġwho Ġonly Ġhave Ġone Ġhospital Ġvisit Ġwhich Ġare Ġgenerally Ġdiscarded Ġbefore Ġmodel Ġtraining , ĠG - BER T Ġoutper forms Ġall Ġbas elines Ġin Ġprediction Ġaccuracy Ġon Ġmedication Ġrecommendation Ġtask . ĠOne Ġdirection Ġfor Ġthe Ġfuture Ġwork Ġis Ġto Ġadd Ġmore Ġauxiliary Ġand Ġstructural Ġtasks Ġto Ġimprove Ġthe Ġability Ġof Ġcode Ġrepresent a ion . ĠAnother Ġdirection Ġmay Ġbe Ġto Ġadapt Ġour Ġmodel Ġto Ġbe Ġsuitable Ġfor Ġeven Ġlarger Ġdatasets Ġwith Ġmore Ġheter ogeneous Ġmod alities . </s> Acknowled gment </s> This Ġwork Ġwas Ġsupported Ġby Ġthe ĠNational ĠScience ĠFoundation Ġaward ĠI IS - 14 185 11 , ĠCC F - 15 33 768 Ġand ĠI IS - 18 380 42 , Ġthe ĠNational ĠInstitute Ġof ĠHealth Ġaward Ġ1 R 01 MD 01 16 82 - 01 Ġand ĠR 56 HL 13 84 15 .