Computer Science Medicine
Computer Science-2021.gz:143309
Computer Science
32a0c1bf48c6fc713db90ec139a4929aa1903e05
Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning

Early prediction of mortality and length of stay(LOS) of a patient is vital for saving a patient's life and management of hospital resources. Availability of electronic health records(EHR) makes a huge impact on the healthcare domain and there has seen several works on predicting clinical problems. However, many studies did not benefit from the clinical notes because of the sparse, and high dimensional nature. In this work, we extract medical entities from clinical notes and use them as additional features besides time-series features to improve our predictions. We propose a convolution based multimodal architecture, which not only learns effectively combining medical entities and time-series ICU signals of patients, but also allows us to compare the effect of different embedding techniques such as Word2vec, FastText on medical entities. In the experiments, our proposed method robustly outperforms all other baseline models including different multimodal architectures for all clinical tasks. The code for the proposed method is available at https://github.com/tanlab/ConvolutionMedicalNer.


Introduction
Electronic Health Record (EHR) data collected from patients who have been admitted into hospitals or intensive care units (ICU) offer a detailed overview of patients consisting of but not limited to demographics, insurance, laboratory test results and medical notes. With the EHR data becoming available for researchers, there has been increasing interest in using it with deep learning algorithms. Besides rapid progress in deep learning area, after Medical Information Mart for Intensive Care(MIMIC-III) [1], today's most popular public EHR database, was released, 1 arXiv:2011.12349v2 [cs.LG] 26 Nov 2020 numerous studies have achieved successful results using this data set and deep learning models to predict different clinical outcomes [2,3,4].
Understanding the health condition of the patient by observing the clinical measurements, laboratory test results and predicting the condition of patients during their ICU stay is a vital problem. In this paper, we focus on two different common risk prediction tasks, mortality (in-hospital & in-ICU) and length of ICU stay (LOS). Both are very important clinical outcomes for determining treatment methods, planning hospital resources and ultimately saving lives. Previous studies primarily focused on predicting clinical events using only the structured data of patient such as historical patient diagnosis (ICD codes) [5,6], lab results and patient ICU measurements [7,8,9] and did not benefit from the unstructured data in EHR. The EHR data which consists of clinical notes written by doctors, nurses, or radiology, discharge notes and many other sources, contains quite detailed information about patients, projecting the knowledge and inference of doctors and even critical details about patient health status for many cases. As per the importance of the clinical notes, researchers want to take advantage of the rich content in clinical notes. Moreover, the recent developments in Natural Language Processing (NLP), there has been increasing interest in using clinical notes to make clinical model predictions [10,11].
Although it may be possible to leverage clinical notes to make more accurate predictions, these notes may consist of long written free-text with an unusual grammatical structure and may contain redundant information. As it may be hard to process raw clinical notes, because of their high-dimensional and sparse nature, extracting medical entities is required to unlock the medical information trapped in the clinical notes and to feed them into prediction models.
Named Entity Recognition (NER) is a fundamental task in NLP that focuses on information extraction aiming to extract entities in a text and classify them into predefined classes. These classes can be locations, people, or organizations in general NER algorithms [12,13]. There can be various NER models for different domains like cybersecurity [14] or medicine [15]. Recently, several deep learning algorithms were applied to clinical texts to train clinical named entity recognition models. These clinical NER models generally try to extract medical information such as disease, drugs, dosage, frequency.
In this paper, we argue that the integration of structured data in EHR and medical entities positively affects the prediction of mortality and LOS. We also investigate the effect of different word representations such as Word2Vec [16], FastText [17], and concatenation of both representations on medical entities. To evaluate the success of our proposed multimodal architecture, we first train models separately with structured and medical entity features. Then we apply 2  In the next section, we summarize the similar studies that work on clinical domain especially predict mortality and length of stay at ICU. Following that, we discuss our data set, problem definitions, and deep learning models used in this study. Finally, we report experimental results and conclude the paper by our findings and conclusion.

Related Work
With the rapid development of deep learning algorithms in the last decade, the number of deep learning models increased substantially for various clinical predictions. Several studies have explored EHRs to solve clinical problems, e.g., [18] used 13 different vital measurements to classify 128 diagnoses using Long Short Term Memory (LSTM) and DoctorAI [5] used Gated Recurrent Unit (GRU) to predict multi-label diagnosis for the next visit. [19] proposed early heart failure detection using Recurrent Neural Networks (RNNs). Forecasting the LOS and mortality have been a popular clinical problem for healthcare researchers in recent years. In earlier studies [20,21,22] on mortality prediction, hand crafted features are selected and used simple machine learning models like logistic regression with different severity scores such as APACHE [23], SAPS-II [24], and SOFA [25]. Nowadays with the progress on deep learning, different architectures have been applied on EHR data to predict this kind of problems. [26] used ensemble learning to make an early mortality prediction and [27] proposed a method to predict mortality using 12 features extracted from the vital signals in the first hour of ICU admission. Darabi et al. [28] used Convolutional neural network to predict long-term mortality risk on the MIMIC-III dataset. More recent work [8] includes attention to their deep learning model to improve models' success. Another work [29] try to predict LOS for acute coronary syndrome patients. There is a comprehensive survey on mortality prediction and LOS [30]. Despite these studies and developments, one of the major problems that the healthcare researchers experienced, the researches on the literature are short of standardized preprocessing steps such as unit conversion, handling outlier and missing values, and transforming raw structured data into usable hourly time series data. In order to solve this problem, [31,32,33] carried out a comprehensive benchmark on MIMIC-III for various tasks such as mortality, LOS, readmission, phenotyping and make their code publicly available.
Purushotham et. al. [33]  We also use medical entities which are extracted from clinical notes to improve our model predictions. Clinical natural language processing and information extraction has been widely studied in recent years on clinical notes. [34,35] proposed a deep learning based multitask learning to make clinical predictions from clinical notes. [11] compared different embedding approaches such as Bag of Words (BoW), Word2Vec and LSTM on clinical note representation by evaluating the prediction performance on diagnosis prediction and mortality risk estimation.
More recently, transformer-based architectures such as BERT [36], XLNET [37] gave state-of-theart performance on different NLP tasks. These models are pre-trained on medical data, which is then fine-tuned on clinical text [38,39]. However, clinicians generally use medical jargon and shorthands when they take these clinical notes which makes hard to process directly. There are a number of studies in the field of clinical NLP which try to extract medical entities in clinical notes [40,41,42]. In this work, we use med7 [15] which is developed for free-text electronic health record. Then, we combine these medical entities with structured data to benefit from multimodal approach. For a detailed overview on deep learning for natural language processing in the clinical domain, readers can refer to [43].
Multimodal learning is a key research area that uses multiple sources to predict unique tasks [44]. This approach has shown success in image captioning tasks [45], visual question answering [46] and speech recognition [47]. In the healthcare research domain, [48] combines unstructured clinical notes and structural time-series data for predicting in-hospital mortality, decompensation, and LOS. Similarly, [49] made unified mortality prediction and try to explore how physiological time series data and clinical notes can be integrated. The study by Jin. et al [50] is the closest to our work in terms of motivation. They made hospital mortality prediction by combining clinical notes and time series data. Clinical notes are represented with Doc2VecC [51] algorithm in two different ways. First, they directly combine clinical notes with time series data, second, they use neural network based clinical NER service to extract five types of medical entities and identify negated entities from clinical notes. After this pre-processing, they use the same representation with the first model and reported a 2% increase in the Area Under ther Curve (AUC).
The difference of our paper from [50] and the main contributions of this work can be summarized as follows.
• We work with four different clinical outcome such as in-hospital mortality, in-ICU mortality, LOS>3 and, LOS>7 rather than just in-hospital mortality.
• We compare different types of word embedding methods (Word2Vec, FastText, Concatenation), and discuss the effect these methods on medical entities.
• We propose a convolutional based deep learning model for combining clinical NER features with time series ICU features. We compare our proposed model with several benchmarks.

Materials and Methods
In this section, we begin by describing our dataset. The details of baselines and clinical NER model are explained next and finally we propose our multimodal deep learning models.

Data
We use the publicly-available MIMIC-III dataset which contains de-identified EHR data of 58

Baseline Models
In this subsection, we discuss our time-series baseline modal that we evaluate on each of our four benchmark tasks. Further, we explain clinical NER model, embedding approaches to represent medical entities and the multimodal baselines used in this study .

Time Series Model
We employ both Long Short Term Memory (LSTM) [52] and Gated Recurrent Units (GRU) [53] networks to capture the temporal information between the patient features. As a result of time-series baseline experiments, GRU has shown a better AUC and AUPRC performance than  LSTM up to %0.5 -%1, while using a simpler architecture. Therefore, we use GRU for all of the multimodal architectures. In general, GRU cell has two gates, a reset gate r and an update gate z. With these gates, GRU can handle the vanishing gradient problem.
We can iterate the mathematical formulation of GRU modal as follows: where z t and r t respectively represent the update gate and the reset gate,ĥ t the candidate activation unit, h t the current activation, and • represents element-wise multiplication. For predicting the mortality and LOS, a sigmoid classifier is stacked on top of the one layer GRU with 256 hidden units.

Multimodal Approaches
In this work, besides time series features, we also use information from clinical notes to improve clinical task prediction performance. Instead of working directly with clinical notes, we first aim to extract medical related keywords. Recently, there are some notable works in the clinical domain that made their pre-trained clinical NER models publicly available [54,55,15]. We use a pre-trained clinical NER model, med7 [15], which uses the same dataset that we use in our  approaches are shown in Figure 1 and the statistics of the extracted medical entities by med7 in MIMIC-III dataset for selected patients are shown in Table 2. Word2Vec [16] is a two-layer neural network that learns the representations of words in the given text with two ways: as a continuous bag-of-words (CBOW) and as a skip-gram. FastText [17] is an extension of the skip-gram model implemented by Facebook's AI Research (FAIR) lab which can handle out-of-vocabulary (OOV) words, and can learn better representations for rare words using several n-grams for words. We use pre-trained word2vec (w i ∈ R 100 ) and fastText embeddings (f i ∈ R 100 ) which was trained on 2.8 billion words from MIMIC-III clinical notes as shown in [38]. Lastly, we design an experimental embedding approach which concatanates the Word2Vec

Proposed Model
where ⊗ is the concatenation operator and e refers to the representation of the medical entity and k is the number of entity. We use a 1D-CNN model similar [58] to extract features from medical entities. We stack three consecutive 1D convolutional layers with filter size 32, 64, and 96.
The kernel size is same for three convolutional layer. The output of the last convolutional layer is followed by the max-pooling layer. The final features of the max-pooling layers are concatenated with the features from one layer GRU with 256 hidden units and fed through one fully-connected layer with 512 hidden units.

Experimental Results
In this section, we report the results of our baseline and multimodel experiments, the metrics we used for the evaluation and details about our development platform.

Setting
Training. For all tasks, we use the patient's first 24 hours ICU measurements. For multimodal architectures, we use 0.2 dropout rate at the end of the fully connected layer. A ReLU activation function is used for nonlinearity and L 2 norm for sparsity regularization is selected with the 0.01 scale factor. For the optimization, we use ADAM [59] algorithm with a learning rate of 0.001.
All models are trained to minimize the binary crossentropy loss and we independently tune the hyperparameters -number of hidden layers, hidden units, convolutional filters, filter-size, learning rate, dropout rates and regularization parameters on the validation set. Each model is trained for 50 epochs and early stopping is used on the validation loss. We train each model 10 times with different initialization seed and report the average performance.
Evaluation metrics. The clinical problems that we work on suffer from class imbalance problem.
We use three different metrics which are Area Under the Receiver Operating Characteristics (AUROC), Area Under Precision-Recall (AUPRC) and F1. AUROC is a popular robust metric for imbalanced datasets [60]. The second metric AUPRC does not include the true negatives in calculation and this approach makes it useful for data with many true negatives as our dataset.
F1 is the final metric which calculates the harmonic mean of precision and recall.
Implementation Details. The aforementioned deep learning algorithms are implemented using Keras [61], which runs Tensorflow [62] on its backend. med7 is used for extracting clinical related   Table 4: Proposed model performance comparison with best baseline model. We select the highest score for each metric and each clinical task from baseline methods.

Baseline Modal Results
We predict four different clinical tasks with the patient's first 24 hours ICU measurements and medical entities.

Proposed Modal Results
In this section, we compare the result of our proposed model against the best scores taken from baseline models. All results for the proposed model against best baseline scores are provided in Table 4. As shown in Table 3, multimodal approach improves the performance of predictions tasks over the time-series, however we try to use medical entities more efficiently to improve the prediction of our models. Except the F1 score of LOS > 7 clinical task, our proposed multimodal architecture robustly outperforms all other baseline models for each task. Table 3 shows that the use of medical entity features improve the prediction performance on all clinical tasks. As shown in Table 3, multimodal baseline modals increase all metrics performance which indicates the benefit of using medical entities for predicting mortality and LOS. These experiments also provide an opportunity to compare the medical entity representation methods.

Discussion
Although there is no certain winner for all tasks, in the baseline models, the results show us for mortality prediction tasks, representing the medical entities with averaging method gives better results. For LOS prediction tasks, representing all medical entities together with Doc2Vec is also successful as averaging method. Furthermore, both scores on Table 3 and Table 4 gives us a chance to compare the word embedding approaches. We do not observe a significant change in performance between word embedding techniques, however pretrained Word2Vec model generally achieves slightly higher scores (around %0.5) than FastText and experimental concatenated embeddings. Apart from these experiments and comparisons, our main motivation is finding an efficient way to combine time-series features with medical entities. Even though both baseline multimodals improve the prediction results compared to timeseries baseline, to make better feature extraction on medical entities, we want to take the advantage of 1D CNN. In the literature, there have been several studies that use 1D CNN in NLP. We stack three 1D convolution operation to extract the features, and then apply 1D max pooling operation over the time-step to obtain a fixed-length vector. By analyzing the results between the proposed and baseline multimodals, we see that 1D CNN based multimodal approach give better results than the averaging and document based embedding methods. Addition to these trials, we also make experiments by 14 using only medical entity features as another baseline. However, only medical entity baseline give poor results (around less than %10 for all tasks) compared to the timeseries and multimodal, so we do not report these results.

Conclusion
Over the past decade, there has been increased attention to improve mortality and LOS prediction performance. Predicting any complications and saving patient's life is an important task for healthcare system which motivates us to work on mortality prediction. LOS is another important clinical problem to improve hospital performance and better healthcare resource utilisation. In this work, we present 1D-CNN based multimodal deep learning architecture that use time-series features and medical entities together and this model outperforms several baselines. Our proposed model performance gain over multimodal baselines is around %1 -%1.5 AUPRC, and the improvement over time-series baseline is around %2.5 -%3 AUPRC. We also make experiments to investigate the effect of different word embedding algorithms to solve our clinical problems and report the results. This work can be extended in multiple directions. First, we can involve more features associated with patient such as prescription data and diagnosis codes to improve the prediction performance. Second, using different word embedding especially transformer based techniques can be used for learning the entity representations. Another thing we may consider in the future is to use more advanced deep learning architectures with attention based will be useful for clinical tasks.