Time Aware Transformer based Network for Clinical Notes Series Prediction
A patient's clinical notes correspond to a sequence of free form text documents generated by healthcare professionals over time. Rich and unique information in clinical notes is useful for clinical decision making. In this work, we propose a time aware transformerbased hierarchical architecture, which we call Flexible Time aware LSTM Transformer (FTL Trans) for classifying a patient's health state based on her series of clinical notes. FTL Trans addresses the problem that current transformer based architectures cannot handle, which is the multi level structure inherent in clinical note series where a note contains a sequence of chucks and a chuck contains further a sequence of words. At the bottom layer, FTL Trans encodes equal length subsequences of a patient's clinical notes "chunks" into content embeddings using a pre trained ClinicalBERT model. Unlike ClinicalBERT, however, FTL Trans merges each content embedding and sequential information into a new position enhanced chunk representation in the second layer by an augmented multi level position embedding. Next, the time aware layer tackles the irregularity in the spacing of notes in the note series by learning a flexible time decay function and utilizing the time decay function to incorporate both the position enhanced chunk embedding and time information into a patient representation. This patient representation is then fed into the top layer for classification. Together, this hierarchical design of FTL Trans successfully captures the multi level sequential structure of the note series. Our extensive experimental evaluation conducted using multiple patient cohorts extracted from the MIMIC dataset illustrates that, while addressing the aforementioned issues, FTL Trans consistently outperforms the state of the art transformer based architectures up to 5% in AUROC and 6% in Accuracy. c (c) 2020 D. Zhang, J. Thadajarassiri, C. Sen E. Rundensteiner. Time Aware Transformer based Architecture
MLHC