Effective Representations of Clinical Notes

Abstract: Clinical notes are a rich source of information about patient state. However, using them to predict clinical events with machine learning models is challenging. They are very high dimensional, sparse and have complex structure. Furthermore, training data is often scarce because it is expensive to obtain reliable labels for many clinical events. These difficulties have traditionally been addressed by manual feature engineering encoding task specific domain knowledge. We explored the use of neural networks and transfer learning to learn representations of clinical notes that are useful for predicting future clinical events of interest, such as all causes mortality, inpatient admissions, and emergency room visits. Our data comprised 2.7 million notes and 115 thousand patients at Stanford Hospital. We used the learned representations, along with commonly used bag of words and topic model representations, as features for predictive models of clinical events. We evaluated the effectiveness of these representations with respect to the performance of the models trained on small datasets. Models using the neural network derived representations performed significantly better than models using the baseline representations with small ($N < 1000$) training datasets. The learned representations offer significant performance gains over commonly used baseline representations for a range of predictive modeling tasks and cohort sizes, offering an effective alternative to task specific feature engineering when plentiful labeled training data is not available.

\section{Introduction}
Our goal is to devise a representation of clinical notes that allows us to make accurate predictions of a wide variety of future events, such as all causes mortality, inpatient admissions, and emergency room visits.  Clinical notes record important details about patients that cannot be found elsewhere in electronic health records (EHRs), but they present challenges to health care researchers, including very high dimensionality, sparsity, and complex linguistic and temporal structure.  Furthermore, reliable labels for many clinical outcomes are scarce relative to the complexity of the data because the gold standard for such labels remains manual review of patient records by clinicians.  We are thus specifically interested in representations of clinical notes that are \emph{efficient}, i.e., they admit good performance even labeled data is scarce.  Traditionally, these challenges are met by extensive feature engineering, guided by task specific domain knowledge.  We want to reduce the need for such feature engineering by automatically learning how to summarize a sequence of clinical notes about a given patient in a data driven manner.  The learned patient-level representation of the clinical notes should directly mitigate or admit simple strategies for overcoming the above challenges.  Specifically, it should support accurate models learned from limited training data, and  be useful across a wide range of clinical prediction tasks and research questions with minimal or no task-specific customization.

We take a transfer learning approach to learning this representation.  Transfer learning has been used to achieve state of the art results in computer vision and Natural Language Processing (NLP) on tasks with limited labeled training data.  Such tasks can be challenging for very flexible models such as deep neural nets.  Transfer learning addresses this problem by first training a large, complex model on a \emph{source task} with a large dataset that has abundant labels.  The resulting model, in the course of learning how to perform the source task, learns a transformation of the input data into a useful form, i.e., \emph{a representation}, that allows it to solve the source task well.  We then use this model on data for a \emph{target task} with more limited data.  We hope that the learned representation is more effective for solving the target task than the original, raw input.

We apply this paradigm to learning efficient patient-level representations of clinical notes.  \emph{Efficient} in this study means specifically that the representations allow us to perform well on the target task even when we have very limited training data.  \emph{Patient-level} means that our method must summarize the variable length sequence of clinical notes for a given patient into a fixed length vector suitable for use by simple models such as logistic regression.  We take two approaches, both based on commonly used techniques in Natural Language Processing (NLP).  The first approach, \emph{embed-and-aggregate}, uses GloVe \citep{pennington2014glove} to learn vector space representations, or embeddings, for biomedical concepts mentioned in clinical text.  We combine the embeddings for the concepts in each note into note representations using simple arithmetic operations.  These note representations are then similarly combined into patient representations.  In the second approach, we use recurrent neural nets (RNNs) \citep{elman1990finding} to learn representations of concepts, notes, and patients simultaneously.  In this approach, we must provide some form of supervision for the source task; we use contemporaneous data from another data modality, i.e., discrete diagnoses codes, as labels for each patient.  The input for both methods is a large corpus of clinical notes from Stanford Hospital.

The resulting representations are evaluated by using them as features for predictive models for complex clinical outcomes -- all causes mortality, inpatient admissions, and emergency room (ER) visits.  For each representation, we estimate \emph{learning curves}, i.e., the relationship between model performance and training set size, because we are particularly interested in the utility of these features in situations where labeled training data is scarce.  We find that the learned representations performed significantly better than common baseline representations based on bag of words (BOW) and topic models for small labeled training sets ($N < 1000$ patients).

\section{Background}
Let $\mathcal{P}$ be the set of all patients, indexed by $i$.  For each patient, we have a sequence of notes, $\boldsymbol\Pi^{(i)} = \{ \Note \}_{j = 1}^{T^{(i)}} $, with each note $\boldsymbol\eta_j^{(i)}$ indexed by $j$ in order from oldest to most recent, and with $T^{(i)}$ the number of notes for patient $i$.  Ultimately, our goal is to learn a function that estimates the probability of some clinical event of interest, such as all causes mortality, given the sequence of notes for a patient:

\[ F\left( \boldsymbol\Pi^{(i)} \right) \approx Pr( \text{event of interest} \mid \boldsymbol\Pi^{(i)}) \]

However, $\boldsymbol\Pi^{(i)}$ is quite complex.  Each patient has a variable number of notes, so the size of the input to $F$ is not uniform.  Furthermore, each note, $\boldsymbol\eta_j^{(i)}$, is itself a variable length sequence of words from a large vocabulary $\mathcal{V}$.  And finally, in the clinical domain we often have very limited labeled datasets for supervised learning, i.e., we have
$\left\{ ( \boldsymbol\Pi^{(i)}, y^{(i)} ) \right\}_{i \in \mathcal{P}_T}$, with $\setsize{\mathcal{P}_T} \ll \setsize{\mathcal{P}}$.
In such situations, it is hard to learn $F$ directly as a mapping from the raw notes to probabilities.  Instead we can devise a transformation of $\boldsymbol\Pi^{(i)}$ into an easier-to-use form, e.g., a fixed length vector of real numbers suitable for use as features for off the shelf models such as logistic regression or gradient boosted trees, i.e.,
\[ G: \boldsymbol\Pi^{(i)} \rightarrow \mathbb{R}^d \]
We refer to the transformed data as the patient representation, $\textbf{p}^{(i)} = G\left( \boldsymbol\Pi^{(i)} \right), \textbf{p}^{(i)} \in \mathbb{R}^d$, and we generally aim to have $d \ll \setsize{\mathcal{V}}$ to achieve dimensionality reduction in order to make the learning problem more tractable \footnote{Note that the transformation can be explicitly specified, i.e., $G$ can be the result of manual feature engineering.  However, in this work we focus on transformations that are learned from the data instead.}.  We can now try to learn:
\[ F(\textbf{p}^{(i)}) = F\left( G\left( \boldsymbol\Pi^{(i)} \right) \right) \approx Pr\left( \text{event of interest} \mid \boldsymbol\Pi^{(i)}\right) \]
We have decomposed the problem into two steps: estimating $G$ using plentiful data, and then estimating $F$ using our limited, labeled data (\autoref{fig:problem_decomposition}).  The key insight in transfer learning is that even when $\mathcal{P}_T$ is small, we can sometimes learn an excellent $G$ using the much larger dataset $\mathcal{P}$ \footnote{Note that in order to map $\boldsymbol\Pi^{(i)}$ into $\mathbb{R}^d$ with $d \ll \setsize{\mathcal{V}}$, we need to aggregate information across both within and between notes.}.

\begin{figure}[h]
\centering \includegraphics[width=0.6\textwidth]{figures/G_and_F.png}
\caption{Transfer learning decomposes our learning problem into two steps.  First, we we use unlabeled data to learn a function $G$ that maps variable length sequences of clinical notes into a fixed sized vector space.  Then we learn the function $F$ for our target task in that vector space using labeled data.}
\label{fig:problem_decomposition}
\end{figure}

\noindent In this section, we review the most commonly used methods for devising the transformation $G$, and briefly review their shortcomings.  We then introduce our proposed methods.

\subsection{Bag of Words Representations and their shortcomings}
Our starting point is the \emph{one-hot} representation of each word $w$ in our vocabulary $\mathcal{V}$.  Each $w$ is associated with a unique index $i_w \in \{ 1, 2, \dots , \setsize{\mathcal{V}} \}$, and we represent each $w \in \mathcal{V}$ with a vector $\textbf{o}_w \in \{0, 1\}^{\setsize{\mathcal{V}}}$, with a $1$ at position $i_w$ and $0$'s elsewhere.  The \emph{bag of words} representation of a note is simply the sum of the one-hot vectors of the words appearing in that note -- for the $j$-th note for patient $i$:
\[ \textbf{n}_j^{(i)} = \sum_{v \in \text{ note j }} \textbf{o}_w \]
We now define our baseline patient representation to be the patient-level bag of words:
\[ \PatientRepr_{BoW} =  \sum_{j = 1}^{T^{(i)}}  \textbf{n}_j^{(i)} \]

Bag of words representations are simple and transparent but have shortcomings inherited from one-hot representation of words.  They retain the high dimensionality and sparseness of one-hot word vectors.  More critically, they violate the intuition that semantically similar notes should have similar representations.  Since we are representing words and notes as vectors, we would like similar words and notes to be close together in that space. For instance, the one hot representations for \emph{non-small cell lung cancer} and \emph{lung adenocarcinoma} are as unrelated to each other as they are to \emph{hypertension}.  And since our baseline note level representation is the sum of the one-hot vectors for the concepts appearing in the note: $\textbf{n}_j^{(i)} = \sum_{v \text{in note}} \textbf{o}_w$, it inherits these problems -- \emph{Tarceva} is the same drug as \emph{erlotinib}, but our bag of words representations of \emph{Received Tarceva for non-small cell lung cancer} is as distant from \emph{lung adenocarcinoma was treated with erlotinib} as it is from \emph{the quick brown fox}.  Our baseline patient representation inherits these problems, albeit possibly to a lesser extent since they aggregate information across many notes.  These disadvantages are exacerbated when we attempt supervised learning with small datasets because some variants of semantically similar words may not appear in the training data at all.

\subsection{Improving on the Bag of Words Representation}
Bag of words representations are simple and can be quite effective when we have enough training data.  However, $\PatientRepr_{BoW}$ is still very high dimensional and sparse.  Many researchers thus reduce the dimensionality of $\PatientRepr_{BoW}$ to arrive at a final $\textbf{p}^{(i)}$.  We describe here two simple, widely used methods.  Latent Semantic Analysis (LSA), a classic method from information retrieval, uses a low rank approximation of the matrix $\textbf{X}_{BoW} \in \mathbb{R}^{ \setsize{\mathcal{V}} \times \setsize{\mathcal{P}} }$, in which the $i$-th row is $\PatientRepr_{BoW}$ as defined above.  Specifically, it calculates a truncated singular value decomposition of $\textbf{X}_{BoW}$:
\[ \textbf{X}_{BoW} \approx \textbf{U} \Sigma \textbf{V}^T  \]
$\textbf{V} \in \mathbb{R}^{\setsize{\mathcal{V}} \times K}$ is an orthogonal, rank $K$ matrix that can use used obtain a $K$ dimensional patient representation, $\PatientRepr_{LSA} = \textbf{V}^T \PatientRepr_{BoW} \in \mathbb{R}^K$.  In practice, practitioners often use some transformation of $\textbf{X}_{BoW}$, such as some variation of the \emph{Term Frequency / Inverse Document Frequency} transformation, that weights each word in an unsupervised manner.

Topic models such as Latent Dirichlet Allocation \citep{blei2003latent} are often presented as a probabilistic version of LSA that models each note as a categorical distribution over $K$ latent factors, interpreted as \emph{topics}.  Thus, each note is transformed:
\[ \boldsymbol\eta_j^{(i)} \rightarrow \textbf{t}_j^{(i)} \in [\theta_{jk}^{(i)}]_{k=1}^K \text{ where } \sum_{k = 1}^K \theta_{jk}^{(i)} = 1 \text{ and } \theta_{jk}^{(i)} \ge 0 \text{ }  \forall \text{ } k \]
In order to arrive at a patient level representation, we aggregate the topic distribution for each of a patient's notes.  In \emph{DeepPatient} \citep{miotto2016deep}, this was done by simply averaging the topics over each patient's notes:
\[ \PatientRepr_{LDA} = \frac{1}{T^{(i)}}\sum_{j=1}^{T^{(i)}} \textbf{t}_j^{(i)} \]
Both of these approaches reduce the dimensionality of our patient representations to $K \ll \setsize{\mathcal{V}}$, and are \emph{unsupervised} -- we do not need patient labels so we can use $\mathcal{P}$ to learn these transformations.

\subsection{Word embeddings for clinical concepts}
Many of the weaknesses of bag of words representations are inherited from the implicit one-hot encoding of words.  There are two main approaches to overcoming these shortcomings.  Some researchers use explicit domain knowledge, e.g., hierarchies of concepts encoded in biomedical ontologies.  Alternatively, using techniques developed in NLP, we can learn better concept representations in a data driven manner.  Specifically, for each $w \in \mathcal{V}$ we learn \emph{vector space embeddings} in which each $w$ is represented by a fixed length vector of real numbers, $\textbf{e}_w \in \mathbb{R}^d$, with $d \ll \setsize{\mathcal{V}}$.  These embeddings are learned from large, unlabeled corpora such that semantically similar words have similar $\textbf{e}_w$.  The precise meaning of \emph{similar} varies for different algorithms for learning these embeddings \citep{pennington2014glove,mikolov2013w2v}, but the essential idea is to learn embeddings that embody the distributional hypothesis (Harris, 1954) -- similar words appear in similar contexts.  In NLP, it has been found that simply averaging the embeddings for words appearing in a document is an excellent document level representation for many tasks \citep{Pagliardini2017, Wieting2015, Shen2018, Arora2017iclr}.  We thus hypothesize that a similar approach, applied over concepts appearing in a note, and then over the series of notes for a patient, may be an effective patient representation.  For instance, we may take:
\[ \PatientRepr_{EA_{mean}} = \frac{1}{T^{(i)}} \sum_{j = 1}^{T^{(i)}} \frac{1}{\text{num words in note}} \sum_{ w \text{ in note } j} \textbf{e}_w \]
i.e., we average the embeddings in each note and then across the notes for a given patient.  Note that we could have chosen a different aggregation function, for instance, an element-wise max:
\[ \PatientRepr_{EA_{max}} =  \underset{j = 1 \dots T^{(i)}}{max} \underset{ w \text{ in note } j}{max} \textbf{e}_w \]
In our experiments we focus on the \emph{min}, \emph{mean} and \emph{max} functions applied element-wise to word and note level representations.  Details for this approach, which we refer to as \emph{embed-and-aggregate}, in provided Materials and Methods.

\subsection{Transfer learning with Recurrent Neural Nets}
Recurrent Neural Nets (RNNs) are widely used to model variable length sequences, and have been widely applied to both general EHR and ICU time series \citep{choi2016learning,lipton2015learning,rajkomar2018}.  However, these prior works directly train models for the target tasks of interest, and we are primarily interested in target tasks for which labeled training data is very limited.  We thus take a transfer learning approach in which we train an RNN on a \emph{source task} devised from $\mathcal{P}$.  In order to solve the source task, the RNN learns to transform variable length input sequences, each element of which is a bag of words corresponding to the notes about a given patient on a particular day, into a fixed length vector, $\PatientRepr_{RNN} \in \mathbb{R}^d$, suitable for the source task. Our transformation function $G$ is embodied by the RNN.  We then use $G$ on patients in $\mathcal{P}_T$, which has labels for the \emph{target task} we are actually interested in, to learn $F$.  We provide details for this approach in Materials and Methods.

\section{Related Work}

\input{sections/related}

\section{Methods and Materials}

\subsection{Data}
Our data set comprises EHR data collected over a five year period (2009 through 2014) at Stanford Hospital.  These records were extracted into the Stanford Translational Research Integrated Database Environment (STRIDE) \citep{lowe2009stride}, a clinical data warehouse, where they were de-identified before being made available for research use.  The Stanford School of Medicine IRB determined that the use of this data was non-human subjects research.  Each patient's data comprised a timestamped series of encounters with care providers over the five year period.  Encounters generated both structured and unstructured data.  The structured data typically consists of discrete codes from controlled vocabularies for diagnoses, medications, and procedures, along with quantitative measurements such as vitals and laboratory measurements.  The unstructured data consists of the free text clinical notes written by care providers documenting each encounter and other aspects of patient care.  Our study focuses on the use of the clinical notes to predict clinical events.

\subsubsection{Text Processing}
The text of the clinical notes was processed using the methods described in \citep{LePendu2013CPT}.  Briefly, we find occurrences of terms in an expansive vocabulary of biomedical terms compiled from a collection of controlled terminologies and biomedical ontologies.  Each term mention with is also tagged with negation (e.g., \emph{atrial fibrillation was ruled out}) and family history (e.g., \emph{The patient's father had hypertension}).  Each note is thus reduced to a bag of words representation, and any terms that are not recognized as biomedical concepts are discarded, along with the sequence of words appearing in a note.  Each term is mapped to unique biomedical concepts \citep{LePendu2013CPT} using a mapping based on the the Unified Medical Language System (UMLS) Metathesaurus, which provides a mapping of strings to Concept Unique Identifiers (CUIs) \citep{Bodenreider2004}.  We retained negated concepts as distinct elements in our vocabulary because negative findings may be informative in clinical medicine.  Finally, we removed concepts that appeared in fewer than 50 or more than 10 million notes.  This resulted in a final vocabulary of 107,388 concepts.  For the remainder of this paper, we refer to concepts as "words" for convenience.  We emphasize that as a result of this text processing, all of our representations are on an equal footing in that they are all starting from a bag of words occurring in each note.  We discuss the implications of this choice, along with other choices in how the text is handled, in the discussion section.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/F1.png}
\caption{Schematic for how we constructed a prediction problem from retrospective data.  The figure shows three patients, numbered 1-3.  Each patient's data consists of longitudinal observations (blue lines) comprising both structured (e.g., diagnosis codes) and unstructured data (free text of clinical notes, shown as gray rectangles on each timeline).  For each patient, we define the prediction time as six months prior to the end of their records in the EHR, which may correspond to the last observed clinical encounter or mortality. Note that patient timelines are not synchronized; each may have a different start and end, and thus prediction time.  We define the twelve months prior to each patient's prediction time the observation era (shown in blue on each patient timeline); this is "the past" from which we draw clinical notes to use to predict clinical events (red dots) in the follow-up era (shown in pink on each patient timeline), which is "the future" with respect to the prediction time. }
\label{fig:prediction_times}
\end{figure}

\subsubsection{Prediction times and data splits}
For simplicity, we set the prediction time for each patient as six months prior to the end of their record, where the end of a patient's record is determined by a recorded clinical encounter or mortality event as ascertained from the Social Security Death Index, hospital records, or state death register records.  We refer to the 12 months prior to the prediction time as the \emph{observation era}, and the 6 months after the prediction time as the \emph{follow-up era}; we use clinical notes from each patient's observation era to make predictions about whether or not clinical events occur in the patient's follow-up era (\autoref{fig:prediction_times}) Note that this choice of prediction time likely induces some bias in the learning problem as in actual usage we of course do not make predictions relative to some uncertain future event (i.e., a clinical encounter of mortality). However, the focus of our study is not on absolute performance but the relative performance of different strategies for representing clinical notes.

After filtering out patients who do not have any clinical notes with recognized biomedical concepts in their observation era, we were left with 115,232 patients, with 2,735,647 notes.  These patients were split randomly into training, validation and test sets with 69,417, 11,290, and 34,525 patients, respectively.  The validation set was used in preliminary experiments to tune the representation learning methods; all final evaluations were conducted on the hold out test set.

\subsubsection{Target tasks}
We focus on clinical events that are complex, noisy functions of the input data: six month mortality, emergency department visits, and inpatient admissions.  These labels are ascertained from patient demographic and encounter data from the EHR. Note that these targets are low prevalence, so these problems exhibit significant class imbalance -- the prevelances of the outcomes are 2.8\%, 8.5\%, and 5.1\% respectively.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/expt_setup.png}
\caption{Experimental Setup.  For each $G$ that we devise to transform variable length sequences of clinical notes into a patient level representation, we perform the following experiment 20 times: for each target task and subsample size (n = 125, 250, 500, 1000, 2000, and 4000), we subsample the \emph{source task training set} patients to get a \emph{target task training set} for the target task, balanced to 20\% positive case prevalence.  Then we fit a predictive model on the subsample and evaluate it on the test set (with the natural positive case prevalence).  }
\label{fig:expt_setup}
\end{figure}

\subsection{Experimental setup}
We are interested in how our representations do when we have only a small set of labeled examples for the target task, along with a much larger dataset of unlabeled examples. Our experimental setup is thus as follows \autoref{fig:expt_setup}.  First, we define the \emph{source task training set} as the data from the observation era of the 69,417 training set patients, i.e., $\mathcal{P}$ as defined above.  We use this data to learn patient level representations (i.e., the transformation function $G$).  Note that no information from the follow-up eras of these patients is used to learn $G$.  Next we define the \emph{target task training set}, i.e., $\mathcal{P}_T$ as defined above, to be a random subset of the source task training set, plus labels derived from clinical events in those patients' follow-up eras. We use this data, transformed by $G$ into patient level representations, to learn $F$.  We vary the size of $\mathcal{P}_T$ from $N = 125$ to $N = 4000$.  Because we expect that results will be quite noisy for smaller $\setsize{\mathcal{P}_T}$, we subsample 20 times for each combination of target task training set size, patient representation, and target task.  We evaluate the effectiveness of the patient level representations using the 34,525 test set patients, with inputs and labels derived from their observation and follow-up eras respectively.  Note that we use stratified sampling to construct the $\mathcal{P}_T$ with a fixed prevalence of 20\% because the target task positive cases are quite rare.  However, the evaluation is conducted using the natural prevalence of the target task positive cases found in the test set patients. Note that this design does not estimate variance in the performance estimates due to sampling of the test set.

\subsection{Learning Patient Level Representations of Clinical Text}
In this section, we present two simple methods for representing clinical notes that use the training data to learn efficient patient level representations from clinical notes.  These methods can be framed as \emph{transfer} learning approaches, in which we use a \emph{source task} that uses a large dataset, $\mathcal{P}$, to learn how to transform raw patient data into a form that is useful for the source task, i.e., $G: \boldsymbol\Pi^{(i)} \rightarrow \textbf{p}^{(i)} \in \mathbb{R}^d$, and then use $\textbf{p}^{(i)}$ for  target tasks we care about but for which labeled training data is limited.

\subsubsection{Embed-and-aggregate}
In this approach, we construct $G$ in two phases.  First, as described above, we learn an embedding $e_w \in \mathbb{R}^d$ for each word $w \in \mathcal{V}$.  We then apply arithmetic functions -- mean, max or min -- element-wise to these vectors to aggregate the words in each note to arrive at a note level representation, and  likewise aggregate the note level representations to arrive at a patient level representation:
\[ \PatientRepr_{EA} =  \underset{j = 1 \dots T^{(i)}}{Aggr} \underset{ w \text{ in note } j}{Aggr} \textbf{e}_w \]
where $Aggr$ is an element-wise min, max or mean. Note that both phases are unsupervised; that is, no external labels are needed.  In this work, we use the GloVe method for learning these embeddings from the clinical notes from the observation eras of the training set patients.  Embeddings learned by methods such as GloVe and word2vec capture the intuition that semantically similar words occur in similar contexts.  In text, context may simply mean distance in the sequence of words.  However, we are starting from a bag of words representation of each note, and the relative ordering of words within notes is lost.  We thus use an approach used for learning embeddings of medical codes \citep{choi2015doctor, choi2016learning}, which are naturally unordered, and represent each note in the training corpus as 2 random permutations of its words.  We ran GloVe on this corpus with a neighborhood size of 10 for 25 iterations.

To test whether embeddings learned on different corpora differ in their utility, we also evaluated embeddings learned from 350,000 biomedical research article abstract \citep{de2014medical}.  Terms in the abstracts were mapped to UMLS CUIs, and embeddings were learned using the skipgram word2vec model with a neighborhood size of 5 and dimensionality of 200.  Embeddings for 52,000 CUIs are publicly available,\footnote{\url{https://github.com/clinicalml/embeddings}} and approximately 28,000 of those are in our vocabulary.  We refer to these embeddings as MCEMJ (Medical Concept Embeddings from Medical Journals).  Finally, we must specify the dimensionality of the embeddings; we present experiments using 300 and 500 dimensional embeddings.

\subsubsection{Recurrent Neural Nets}
In this approach, we use recurrent neural nets whose input is a sequence of bag of words vectors representing the clinical notes written about a given patient in that patient's observation era \citep{choi2015doctor}.  These vectors are input in temporal order to the RNN, and notes written on the same day are merged into a single input vector.  The final hidden state is used as the patient level representation for the following source task: predict the diagnosis codes assigned to the patient during their observation era.  The diagnosis codes are derived from the structured data in each patient's observation era by grouping ICD-9 codes according into Clinical Classification Software (CCS) groups \footnote{https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp}.  CCS maps 12,856 diagnosis codes into 254 broad disease categories such as \emph{lung cancer} or \emph{cardiac dysrhythmias}, and have previously been used as supervision labels for the training of recurrent neural nets \citep{choi2015doctor} because it dramatically reduces the label space.  The source task is thus a multi-task sequence classification problem.

We used \texttt{tanh} activations, dropout with drop probability 0.2, and a dense layer with binary cross-entropies for the multi-task supervision. Networks were trained with RMSProp \citep{tieleman2012lecture} for 100 epochs using a batch size of 32 and maximum sequence length of 50, with longer sequences truncated to keep the most recent notes.  The models were implemented in Keras 1.2.0 \citep{chollet2015keras} with a Tensorflow 0.12.1 backend \citep{tensorflow} on an NVIDIA GeForce GTX TITAN X GPU.

\subsubsection{Baseline Representations}
We compared the learned representations against the TF-IDF transformed bag of words (TF-IDF BOW), Latent Semantic Analysis (LSA), and Latent Dirichlet Allocation (LDA) representations presented above.  For TF-IDF BOW, we limited the vocabulary to the 15,000 most frequent words in each training subsample.  Word counts were then summed over each patient's input notes.  The final representation consisted of TF-IDF transformed counts of words for each patient, where the term and inverse document frequencies were computed over the entire training corpus (i.e., all notes from the observation eras of the training set patients).  For LSA, we summed the bag of words vectors for all the notes in each training set patient's observation era, and performed a truncated SVD on the resulting \emph{term-patient} matrix [Ref irlba package].  We truncated the SVD to 600 dimensions.  Finally, for LDA, we fit a topic model on the training corpus using Gensim \citep{gensim}. 300 topics were fit over two passes through the training notes.  The model was then used to estimate topic distributions for validation and test set notes.  The topic distributions for each note in a given patient's observation era were aggregated using mean and max pooling.

\subsection{Target Task Models}
The target task models were L2 regularized logistic regression models using the various patient level representations as fixed length inputs.  These models were tuned via 5-fold cross validation on the training subsamples using the glmnet package \citep{glmnet2010} in R \footnote{http://www.R-project.org}.

\section{Results}
We now present the results of our experiments, in which we compare the target task performance of our learned representations against each other and our baseline representations.  We first focus on results for broad classes of representations before discussing the overall comparison in order to simplify the latter.  In all experiments, we show three \emph{learning curves} -- estimates of the relationship between generalization performance and target task training set size -- for each of the three target tasks of interest and target task training set sizes of ranging from $N = 125$ to $N = 4000$.  The error bars are the standard error of the mean for each estimate across 20 random subsamples of target task training patients as described above.

\subsection{Performance of the baseline representations}
\autoref{fig:results_baselines} shows the performance of the baseline representations -- bag of words with TF-IDF weighting, LSA, and LDA with mean and max aggregation across notes -- across the target tasks and training set sizes. The bag of words with TF-IDF weighting representation dominates the others at larger sample sizes.  For very small sample size, the LSA representation can improve on TF-IDF weighted bag of words.  Note that mean aggregation for the LDA representation significantly underperforms the other baselines with the exception of the bag of words representation with very small training set sizes, and it appears that max pooling of the topic distributions performs much better.  Generally, it appears that LSA ($\PatientRepr_{LSA}$) does well at very small training set sizes but are quickly overtaken by the simple $\PatientRepr_{BoW}$ representation as training data is added.  We thus focus only on the $\PatientRepr_{BoW}$ and $\PatientRepr_{LSA}$ as baselines representations; it appears that these are both better choices than the simple topic modeling applied to summarize clinical text in Miotto et al \citep{miotto2016deep}.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/Baselines.png}
\caption{Performance of the baseline representations across the three target tasks and a range of training set sizes.       }
\label{fig:results_baselines}
\end{figure}

\subsection{Embed-and-aggregate}
Embed-and-aggregate method uses simple functions to form a patient level representation from the embeddings of the words found in the patient's notes:
\[ \PatientRepr_{EA} =  \underset{j = 1 \dots T^{(i)}}{Aggr} \underset{ w \text{ in note } j}{Aggr} \textbf{e}_w \]
In order to apply this method, we must make the following decisions.  First, what should we use for $Aggr$?  Prior work has suggested that the common choice of mean aggregation may not be optimal.  Second, what should the dimensionality of the embeddings be?  We are interested in this because more aggressive dimensionality reduction often benefits tasks with very small training sets.  And finally, what text should we use to learn the embeddings from?  The most obvious choice is to learn them from the corpus of clinical notes in our dataset, but more useful embeddings might be learned from less noisy text such as the abstracts of biomedical journal articles.

\subsubsection{Embed-and-aggregate - how should we aggregate?}
 \autoref{fig:EA_which_aggr} shows results for aggregating embeddings using $min$, $max$, and $mean$, along with concatenating the patient level embeddings for each of these choices into a single, larger patient level representation.  The patient level representation resulting from concatenating the \emph{min}, \emph{max} and \emph{mean} aggregated patient level representations dominate all other choices for all targets and over all target task training set sizes.  Interestingly, this holds even for the smallest target task training set size of $N = 125$ despite the much higher dimensionality of the concatenated representation ($d = 1500$ for the latter vs $d = 500$ for the other choices).

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/EA_which_aggr.png}
\caption{Performance of Embed-and-aggregate using $d=500$ embeddings learned from clinical notes and element-wise \emph{min}, \emph{max}, and \emph{mean} as aggregation functions.  We also test concatenation of the resulting patient level representations for all of these choices.}
\label{fig:EA_which_aggr}
\end{figure}

\subsubsection{Embed-and-aggregate - Dimensionality}
In this experiment, we assess the impact of the dimensionality of the embeddings learned from the clinical text on target task performance.  \autoref{fig:EA_dims} suggests that there is little penalty incurred by using the higher dimensional embeddings for the three target tasks, even for very small target task training set sizes ($N = 125)$.  In fact, for the Mortality target task, the higher dimensional representation dominates the lower dimensional representation across the range of target task training set sizes.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/EA_dims.png}
\caption{Performance of Embed-and-aggregate using $d=300$ vs $d=500$ embeddings learned from the text of notes from the observation eras of training set patients. }
\label{fig:EA_dims}
\end{figure}

\subsubsection{Embed-and-aggregate - which text should we learn from?}
While we have a large corpus of clinical notes from which to learn embeddings, it is not obvious that this is the best corpus from which to learn embeddings.  Clinical notes are noisy, and our text processing pipeline, which reduces each note into a bag of words, may be losing valuable information.  We therefore assess the relative utility of embeddings learned from an alternative source of text -- the abstracts of 350,000 biomedical journal articles.  In \citep{de2014medical}, the word2vec skipgram model is used to learn $d=200$ dimensional embeddings for 58,000 biomedical concepts (CUIs).  Of these, 28,000 are in our vocabulary.  \autoref{fig:EA_which_text} shows the performance of Embed-and-aggregate on the target tasks using embeddings learned from the clinical text vs journal article abstracts ("MCEMJ").  In both cases, we use the the concatenation of \emph{min}, \emph{max}, and \emph{mean} aggregated embeddings.

We also consider embeddings that are the concatenation of those learned from the clinical text and research article abstracts.  Here, there is no clear winner: for the most part, the MCEMJ embeddings perform somewhat worse than the other choices, but this may reflect the lower dimensionality (recall that higher dimensional embeddings learned from clinical text perform better than lower dimensional embeddings learned from the same corpus even for very small target task training sets).  Note that the combined embeddings generally fare no worse than the other options, particularly for smaller target task training sets, suggesting that the two corpora provide somewhat complementary information.  However, differences in the methods used to learn the embeddings, along with the different sized corpora and dimensionality of the embeddings render these results difficult to interpret.  We therefore focus on the concatenation of \emph{min}, \emph{max}, and \emph{mean} aggregated $d=500$ dimensional embeddings learned from clinical text, and use \emph{Embed-and-aggregate} or $\PatientRepr_{EA}$ to refer to this specific configuration.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/EA_which_text.png}
\caption{Performance of Embed-and-aggregate using embeddings learned from the text of clinical notes using GloVe vs the abstracts of biomedical research articles (MCEMJ). }
\label{fig:EA_which_text}
\end{figure}

\subsection{RNNs - preliminary tuning}
We performed preliminary experiments using the validation split in order to assess various technical aspects of the RNN models.   Below we briefly discuss the main conclusions.

\begin{itemize}
\item We found that providing supervision to the RNN after the final timestep was more effective than providing supervision at each time step.

\item Typically, the first layer of RNNs with bag of words inputs multiply an embedding matrix by the input vector, effectively summing the embeddings for the inputs.  However, we found that applying an element-wise maximum to the embeddings at each time-step significantly improved performance in the target tasks.

\item We explored initializing the embedding layer of our recurrent nets using word embeddings learned from the clinical notes or from from research article abstracts.  We found that random initialization consistently outperformed initialization with pre-trained embeddings.

\item Finally, we explored using Long Short-Term Memory (LSTM) \citep{hochreiter1997long} and Gated Recurrent Units (GRU) \citep{cho2014learning} as recurrent cells.  We found that, consistent with \citep{choi2015doctor}, GRUs consistently outperformed LSTMs.  Thus, our final experiments used GRUs with max aggregation of word embeddings.
\end{itemize}

\subsection{RNNs - Dimensionality}
Recurrent neural nets have a vast array of hyperparameters that can have a significant impact on their performance in various tasks.  However, tuning RNNs is computationally expensive, and at any rate we are interested in using RNNs as fixed feature extractors after fitting them on the source task.  It is not clear that incrementally better performance on the source task will lead to better performance on target tasks with severely limited training data.  Reflecting these realities, we did not conduct an extensive hyperparameter search to optimize the performance of our RNNs on the source task; rather, we focused on broad choices such as where LSTMs or GRUs work better as the recurrent units, and whether mean vs max aggregation works better, as described above.  We then turned to the dimensionality of the learned representation, evaluating models with $d = 300$ and $d = 600$.  \autoref{fig:RNN_dims} shows the results of that evaluation.  For one of the targets, future ER visits, the higher dimensional representation works better even for $N = 125$.  For the other two targets, however, we observe the expected pattern -- the lower dimensional ($d=300$) representation is somewhat better with very small target task training sets, but as the target task training set grows the higher dimensional representation catches up and does well. Because it is not clear which representation is best, we use both in our overall evaluation below.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/RNN_dims.png}
\caption{RNN derived representations with $d = 300$ vs $d = 600$.}
\label{fig:RNN_dims}
\end{figure}

\subsection{Overall}
Of course we must ask how well our learned representations fare against the baselines of bag of words with TF-IDF weighting and LSA.  \autoref{fig:overall_results} shows the results for the best overall \emph{embed-and-aggregate} representation, along with our two RNN based representations.  As expected, the learned representations are significantly better with smaller target task training sets, outperforming the baselines by substantial margins.  The RNN representations generally beat the \emph{embed-and-aggregate} representation, though the latter comes quite close to the two RNN representations, especially as the target task training set grows.  However, the most notable pattern is that with sufficiently large target task training sets, the humble bag of words representation catches up to, and in the case of Mortality eventually surpasses, the performance of the best learned representations.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/overall.png}
\caption{Overall comparison of the best baseline models vs the best learned representations. }
\label{fig:overall_results}
\end{figure}

\subsection{Wide and deep representations}
The strong performance of the learned representations with small $\mathcal{P}_T$, along with the strong performance of the simple bag of words representation with large $\mathcal{P}_T$, begs the question of whether it is possible to combine their benefits.  One simple approach for attempting this is to concatenate the two representations so that the models for the target tasks have access to both the reduced dimension summary provided by the learned representations and the fine grained information of the bag of words representation.  This is roughly analogous to the so-called \emph{wide and deep} representations that have been used for problems with very high dimensional and sparse inputs \citep{widedeep}.  \autoref{fig:wide_and_deep} shows the results when we combine the $d=600$ RNN representation with the bag of words.  Interestingly, adding the summary information provided by the learned representations to the bag of words representation strictly improves performance across the range of target task training set sizes, with the wide and deep representations out-performing the bag of words representations by significant margins at most training set sizes (the one exception is $N = 125$ for the Mortality task). The gain in performance with small target task training sets is not great enough to catch up to or surpass the learned representations; the extra information is still difficult to exploit due to the curse of dimensionality.  Nevertheless, these results suggest that this approach may yield significant improvements over BOW and the learned representations alike when training data is plentiful.  Of course, with enough training data we might be best off learning models for the target task directly instead of resorting to transfer learning.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/wide_deep.png}
\caption{ Performance of a \emph{wide and deep} representation. }
\label{fig:wide_and_deep}
\end{figure}

\section{Discussion}
We have demonstrated that learned representations of clinical text can be effective for predicting clinical events in scenarios where labeled data for supervised learning is scarce.  Our learned representations range from a very simple \emph{embed-and-aggregate} method to a more computationally intensive method that uses recurrent neural nets.  We find that embed-and-aggregate is nearly as effective as the RNN based method and is much simpler to compute and share.  In comparison to various commonly used baselines, we find that our learned representations admit significant more accurate predictive models with very small target task training set sizes.  However, as the training set size available for supervised learning increases, we find that the simple bag of words with TF-IDF weighting can close the performance gap, or sometimes even surpass the performance of the learned representations.  In such settings, it may be possible to do even better than bag of words and the learned representations by simply concatenating the two representations together.  Of course, with sufficient training data it may be possible to learn a complex model for the target task directly.

We acknowledge several important limitations of this study.  First, we note that we have started all of our methods from the same bag of words representation of the clinical notes, since that is the output of our text processing pipeline.  It is of course possible that there is much to be gained by learning representations directly from the full text of the notes.  Second, we note that the way we have chosen the prediction time for each patient does not correspond to the way that predictions would be made at run time since we have picked the prediction time in reference to an event that in the future relative to the prediction time.  This likely induces a somewhat optimistic bias in the performance estimates.  However, we believe that our conclusions regarding the \emph{comparative efffectiveness} of the different patient level representations are still valid, though of course follow up work with more careful problem formulation is warranted.  Third, we must note that we have constructed our target tasks to be experimentally convenient but at the cost of realism in important respects.  For each of the target tasks, we actually have ample training data since the positive cases are not too rare and are easily ascertained from EHR data.  In addition, in a real world setting with a truly limited labeled dataset for the target task of interest, our estimates of generalization error would be much noisier because the evaluation sets would likely be much smaller than that used in our study.  This implies that we would be hard pressed to accurately select the representation that truly admits the best generalization performance.  These limitations notwithstanding, our results suggest that relatively simple methods drawn from NLP can yield substantial performance gains over commonly used baseline representations of clinical text.

\section*{References}
\bibliographystyle{plainnat}

\section{Submission of papers to NIPS 2013}

NIPS requires electronic submissions.  The electronic submission site is
\begin{center}
   \url{http://papers.nips.cc}
\end{center}

Please read carefully the
instructions below, and follow them faithfully.
\subsection{Style}

Papers to be submitted to NIPS 2013 must be prepared according to the
instructions presented here. Papers may be only up to eight pages long,
including figures. Since 2009 an additional ninth page \textit{containing only
cited references} is allowed. Papers that exceed nine pages will not be
reviewed, or in any other way considered for presentation at the conference.

Please note that this year we have introduced automatic line number generation
into the style file (for \LaTeXe and Word versions). This is to help reviewers
refer to specific lines of the paper when they make their comments. Please do
NOT refer to these line numbers in your paper as they will be removed from the
style file for the final version of accepted papers.

The margins in 2013 are the same as since 2007, which allow for $\approx 15\%$
more words in the paper compared to earlier years. We are also again using
double-blind reviewing. Both of these require the use of new style files.

Authors are required to use the NIPS \LaTeX{} style files obtainable at the
NIPS website as indicated below. Please make sure you use the current files and
not previous versions. Tweaking the style files may be grounds for rejection.

\subsection{Retrieval of style files}

The style files for NIPS and other conference information are available on the World Wide Web at
\begin{center}
   \url{http://www.nips.cc/}
\end{center}
The file \verb+nips2013.pdf+ contains these
instructions and illustrates the
various formatting requirements your NIPS paper must satisfy. \LaTeX{}
users can choose between two style files:
\verb+nips11submit_09.sty+ (to be used with \LaTeX{} version 2.09) and
\verb+nips11submit_e.sty+ (to be used with \LaTeX{}2e). The file
\verb+nips2013.tex+ may be used as a ``shell'' for writing your paper. All you
have to do is replace the author, title, abstract, and text of the paper with
your own. The file
\verb+nips2013.rtf+ is provided as a shell for MS Word users.

The formatting instructions contained in these style files are summarized in
sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.

\section{General formatting instructions}
\label{gen_inst}

The text must be confined within a rectangle 5.5~inches (33~picas) wide and
9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).
Use 10~point type with a vertical spacing of 11~points. Times New Roman is the
preferred typeface throughout. Paragraphs are separated by 1/2~line space,
with no indentation.

Paper title is 17~point, initial caps/lower case, bold, centered between
2~horizontal rules. Top rule is 4~points thick and bottom rule is 1~point
thick. Allow 1/4~inch space above and below title to rules. All pages should
start at 1~inch (6~picas) from the top of the page.

For the final version, authors' names are
set in boldface, and each name is centered above the corresponding
address. The lead author's name is to be listed first (left-most), and
the co-authors' names (if different address) are set to follow. If
there is only one co-author, list both author and co-author side by side.

Please pay special attention to the instructions in section \ref{others}
regarding figures, tables, acknowledgments, and references.

\section{Headings: first level}
\label{headings}

First level headings are lower case (except for first word and proper nouns),
flush left, bold and in point size 12. One line space before the first level
heading and 1/2~line space after the first level heading.

\subsection{Headings: second level}

Second level headings are lower case (except for first word and proper nouns),
flush left, bold and in point size 10. One line space before the second level
heading and 1/2~line space after the second level heading.

\subsubsection{Headings: third level}

Third level headings are lower case (except for first word and proper nouns),
flush left, bold and in point size 10. One line space before the third level
heading and 1/2~line space after the third level heading.

\section{Citations, figures, tables, references}
\label{others}

These instructions apply to everyone, regardless of the formatter being used.

\subsection{Citations within the text}

Citations within the text should be numbered consecutively. The corresponding
number is to appear enclosed in square brackets, such as [1] or [2]-[5]. The
corresponding references are to be listed in the same order at the end of the
paper, in the \textbf{References} section. (Note: the standard
\textsc{Bib\TeX} style \texttt{unsrt} produces this.) As to the format of the
references themselves, any style is acceptable as long as it is used
consistently.

As submission is double blind, refer to your own published work in the
third person. That is, use ``In the previous work of Jones et al.\ [4]'',
not ``In our previous work [4]''. If you cite your other papers that
are not widely available (e.g.\ a journal paper under review), use
anonymous author names in the citation, e.g.\ an author of the
form ``A.\ Anonymous''.

\subsection{Footnotes}

Indicate footnotes with a number\footnote{Sample of the first footnote} in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2~inches
(12~picas).\footnote{Sample of the second footnote}

\subsection{Figures}

All artwork must be neat, clean, and legible. Lines should be dark
enough for purposes of reproduction; art work should not be
hand-drawn. The figure number and caption always appear after the
figure. Place one line space before the figure caption, and one line
space after the figure. The figure caption is lower case (except for
first word and proper nouns); figures are numbered consecutively.

Make sure the figure caption does not get separated from the figure.
Leave sufficient space to avoid splitting the figure and figure caption.

You may use color figures.
However, it is best for the
figure captions and the paper body to make sense if the paper is printed
either in black/white or in color.
\begin{figure}[h]
\begin{center}
\fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
\end{center}
\caption{Sample figure caption.}
\end{figure}

\subsection{Tables}

All tables must be centered, neat, clean and legible. Do not use hand-drawn
tables. The table number and title always appear before the table. See
Table~\ref{sample-table}.

Place one line space before the table title, one line space after the table
title, and one line space after the table. The table title must be lower case
(except for first word and proper nouns); tables are numbered consecutively.

\begin{table}[t]
\caption{Sample table title}
\label{sample-table}
\begin{center}
\begin{tabular}{ll}
\multicolumn{1}{c}{\bf PART}  &\multicolumn{1}{c}{\bf DESCRIPTION}
\\ \hline \\
Dendrite         &Input terminal \\
Axon             &Output terminal \\
Soma             &Cell body (contains cell nucleus) \\
\end{tabular}
\end{center}
\end{table}

\section{Final instructions}
Do not change any aspects of the formatting parameters in the style files.
In particular, do not modify the width or length of the rectangle the text
should fit into, and do not change font sizes (except perhaps in the
\textbf{References} section; see below). Please note that pages should be
numbered.

\section{Preparing PostScript or PDF files}

Please prepare PostScript or PDF files with paper size ``US Letter'', and
not, for example, ``A4''. The -t
letter option on dvips will produce US Letter files.

Fonts were the main cause of problems in the past years. Your PDF file must
only contain Type 1 or Embedded TrueType fonts. Here are a few instructions
to achieve this.

\begin{itemize}

\item You can check which fonts a PDF files uses.  In Acrobat Reader,
select the menu Files$>$Document Properties$>$Fonts and select Show All Fonts. You can
also use the program \verb+pdffonts+ which comes with \verb+xpdf+ and is
available out-of-the-box on most Linux machines.

\item The IEEE has recommendations for generating PDF files whose fonts
are also acceptable for NIPS. Please see
\url{http://www.emfield.org/icuwb2010/downloads/IEEE-PDF-SpecV32.pdf}

\item LaTeX users:

\begin{itemize}

\item Consider directly generating PDF files using \verb+pdflatex+
(especially if you are a MiKTeX user).
PDF figures must be substituted for EPS figures, however.

\item Otherwise, please generate your PostScript and PDF files with the following commands:
\begin{verbatim}
dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps
ps2pdf mypaper.ps mypaper.pdf
\end{verbatim}

Check that the PDF files only contains Type 1 fonts.

\item xfig "patterned" shapes are implemented with
bitmap fonts.  Use "solid" shapes instead.
\item The \verb+\bbold+ package almost always uses bitmap
fonts.  You can try the equivalent AMS Fonts with command
\begin{verbatim}
\usepackage[psamsfonts]{amssymb}
\end{verbatim}
 or use the following workaround for reals, natural and complex:
\begin{verbatim}
\newcommand{\RR}{I\!\!R}
\newcommand{\Nat}{I\!\!N}
\newcommand{\CC}{I\!\!\!\!C}
\end{verbatim}

\item Sometimes the problematic fonts are used in figures
included in LaTeX files. The ghostscript program \verb+eps2eps+ is the simplest
way to clean such figures. For black and white figures, slightly better
results can be achieved with program \verb+potrace+.
\end{itemize}
\item MSWord and Windows users (via PDF file):
\begin{itemize}
\item Install the Microsoft Save as PDF Office 2007 Add-in from
\url{http://www.microsoft.com/downloads/details.aspx?displaylang=en\&familyid=4d951911-3e7e-4ae6-b059-a2e79ed87041}
\item Select ``Save or Publish to PDF'' from the Office or File menu
\end{itemize}
\item MSWord and Mac OS X users (via PDF file):
\begin{itemize}
\item From the print menu, click the PDF drop-down box, and select ``Save
as PDF...''
\end{itemize}
\item MSWord and Windows users (via PS file):
\begin{itemize}
\item To create a new printer
on your computer, install the AdobePS printer driver and the Adobe Distiller PPD file from
\url{http://www.adobe.com/support/downloads/detail.jsp?ftpID=204} {\it Note:} You must reboot your PC after installing the
AdobePS driver for it to take effect.
\item To produce the ps file, select ``Print'' from the MS app, choose
the installed AdobePS printer, click on ``Properties'', click on ``Advanced.''
\item Set ``TrueType Font'' to be ``Download as Softfont''
\item Open the ``PostScript Options'' folder
\item Select ``PostScript Output Option'' to be ``Optimize for Portability''
\item Select ``TrueType Font Download Option'' to be ``Outline''
\item Select ``Send PostScript Error Handler'' to be ``No''
\item Click ``OK'' three times, print your file.
\item Now, use Adobe Acrobat Distiller or ps2pdf to create a PDF file from
the PS file. In Acrobat, check the option ``Embed all fonts'' if
applicable.
\end{itemize}

\end{itemize}
If your file contains Type 3 fonts or non embedded TrueType fonts, we will
ask you to fix it.

\subsection{Margins in LaTeX}

Most of the margin problems come from figures positioned by hand using
\verb+\special+ or other commands. We suggest using the command
\verb+\includegraphics+
from the graphicx package. Always specify the figure width as a multiple of
the line width as in the example below using .eps graphics
\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.eps}
\end{verbatim}
or
\begin{verbatim}
   \usepackage[pdftex]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.pdf}
\end{verbatim}
for .pdf graphics.
See section 4.4 in the graphics bundle documentation (\url{http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps})

A number of width problems arise when LaTeX cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \verb+\-+ command.

\subsection*{Embed and Aggregate}
In this approach, we break up the source task into two phases.  First,
we learn word embeddings using methods such as GloVe or word2vec.
These embeddings are then aggregated into patient level embeddings
using simple functions \textemdash min, mean, and max \textemdash
applied element-wise to the embedding dimensions.  Note that this
aggregation is over two levels \textemdash word embeddings are first
combined into note representations, and then into patient level
representations.  Neither phase uses external labels, and are
unsupervised.  We explored word embeddings learned from both clinical
text and from biomedical research article abstracts as detailed below.

\subsubsection*{GloVe embeddings learned from clinical notes}
We used GloVe to learn word embeddings from clinical text using a range
of dimensionality and context sizes.
In order to overcome the loss of the ordering of words in our input corpus, we
adopted a technique used for learning embeddings of
medical codes \citep{choi2015doctor, choi2016learning}, which are
naturally unordered, and represent each note in the training corpus using 2 random orderings
of its words.  We ran GloVe for 25 iterations.

\subsubsection*{Word embeddings from research article abstracts}
We also evaluated word embeddings learned from 350,000 biomedical
journal article abstracts in \citep{de2014medical}.  Terms in the
abstract text were mapped to UMLS CUIs, and the embeddings were
learned using the word2vec skip-gram model with a window size of 5 and
dimensionality of 200.  Embeddings for 52,000 CUIs are publicly
available,\footnote{\url{https://github.com/clinicalml/embeddings}}
and approximately 28,000 of those are in our vocabulary.  We refer to
these embeddings as MCEMJ (Medical Concept Embeddings from Medical
Journals).

\subsubsection*{Aggregation of word embeddings}
We constructed fixed length representations of the patients using
simple functions -- min, mean, and max -- applied element-wise to the
word embeddings within each note.  We explored a range of embedding
sizes, along with concatenation of embeddings resulting from different
aggregation methods, and concatenation of embeddings learned from
different data sources.

\subsection*{Recurrent neural nets}
In this approach, we use recurrent neural nets whose inputs are a
sequence of bag of words vectors representing the clinical notes from a
given patient's input era.  This approach jointly learns word-, note-,
and patient-level embeddings (figure \ref{fig:rnn_architecture}).  Models are trained using input era data
only and are then used as fixed feature extractors for the target
tasks.

\subsubsection*{Neural Net Supervision}
Supervision for the recurrent neural net source task was provided by
Clinical Classification Software
(CCS)\footnote{https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp}
codes derived from structured diagnosis codes from each patient's input era.  CCS maps
12,856 diagnosis codes into 254 broad disease categories such as
\emph{lung cancer} or \emph{cardiac dysrhythmias}, and have previously
been used as supervision labels for the training of recurrent neural
nets \citep{choi2015doctor} because it dramatically reduces the label
space.  We formulated learning as a multi-task sequence
classification problem in which the final hidden state of the net
is used as a representation for a multi-task problem with
CCS labels aggregated over the patient's input era.  In preliminary
experiments, this approach consistently outperformed
providing supervision at each step using only that encounter's codes.

\subsubsection*{Architecture}

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/rnn.png}
\caption{Model architecture for the recurrent neural net.  The input at each timestep is a many-hot vector of the terms mentioned at that timestep, corresponding to a clinical note.  The model jointly learns an embedding for each term and a patient level representation for the sequence of inputs.  Supervision for the model is provided by diagnosis codes contemporaneous with the input notes.}
\label{fig:rnn_architecture}
\end{figure}

As in other work using recurrent nets to model patient data, the input
to our recurrent nets were many-hot vectors representing the unordered
bag of words for each note \citep{choi2015doctor}.  Typically, the
first layer of the network multiplies an embedding matrix by this
vector, effectively summing the embeddings to arrive at a note level
representation that is input to recurrent cells.  However, we found
that applying an element-wise maximum to the embeddings at each
time-step significantly improved performance in the target tasks.

Finally, we explored using Long Short-Term Memory (LSTM)
\citep{hochreiter1997long} and Gated Recurrent Units (GRU)
\citep{cho2014learning} as recurrent cells.  We found that,
consistent with \citep{choi2015doctor}, GRUs consistently outperformed
LSTMs.  Thus, our final experiments used GRUs with max aggregation of word embeddings.

\subsubsection*{Learning details}
We used \texttt{tanh} activations, dropout with drop probability 0.2,
and a dense layer with binary cross-entropies for the multi-task
supervision. Networks were trained with RMSProp
\citep{tieleman2012lecture} for 100 epochs using a batch size of 32 and
maximum sequence length of 50, with longer sequences truncated to keep
the most recent notes.  The models were implemented in Keras 1.2.0
\citep{chollet2015keras} with a Tensorflow 0.12.1 backend \citep{tensorflow}
on an NVIDIA GeForce GTX TITAN X GPU.

We explored initializing the embedding layer of our recurrent nets
using word embeddings learned from the clinical notes or from from
research article abstracts.  We found that random initialization
consistently outperformed initialization with pre-trained embeddings.

\subsection*{Baselines - Bag of words and topic models}
We compared the learned representations against BOW and topic model representations of words and notes respectively.  For the BOW experiments, we limited the vocabulary to the 15,000 most frequent
words in each training subsample.  Word counts were then summed over each patient's input notes.  The final representation consisted of TF-IDF transformed counts of words for each patient. Latent Dirichlet Allocation was used to fit a topic model on training notes using Gensim \citep{gensim}. 300 topics were fit over two passes through the training notes.  The model was then used to estimate topic distributions for validation and test set notes.  The topic distributions for each note were aggregated into patient-level representations using mean and max pooling.

\subsubsection*{Text processing}
The text of the clinical notes was first processed using the National Center for Biomedical Ontology (NCBO)
Annotator \citep{lependu2013pharmacovigilance}, which
extracts occurrences of terms in an expansive vocabulary of biomedical
terms compiled from a collection of controlled terminologies and
biomedical ontologies.
The NCBO Annotator further flags each mention for negation
(e.g., ``atrial fibrillation was ruled out'') and family history
(e.g., ``The patient’s father had hypertension'').
Following annotation, each note is converted into a BOW representation and any terms that are not recognized as biomedical concepts are discarded.

Finally, terms are normalized to unique biomedical concepts using the Unified Medical Language System MetaThesaurus, which provides a mapping of strings to Concept Unique Identifiers (CUIs) \citep{Bodenreider2004}.
We retained negated concepts as distinct elements in our vocabulary because negative findings are
often informative.  We removed concepts that appeared in fewer than 50 or more than 10 million notes.  This resulted in a final vocabulary of 107,388 CUIs.  For the remainder of this paper, we will refer to CUIs as ``words'' for convenience.

\subsubsection*{Data splits}
We evaluate our learned representations on the basis of their utility for predicting clinical events of interest occurring in the future with respect to the input clinical notes, using datasets as small as 125 patients.  Thus, in addition to the traditional split of patients into training, validation and test sets used for training, tuning of model hyper-parameters and estimation of generalization error, we also split each patient's longitudinal record into two ``eras''.  The label era comprises the last six months of each patient's record; the structured data from this era provides labels, e.g., mortality, for the target tasks.  The input era comprises the 12 months prior to the label era.  Clinical notes from this era are the inputs for both the source and target tasks.  Additionally, any labels required for source tasks are derived from the structured data in the input era.  The use of data in each of the train and test splits, along with the input and label eras, is shown in figure \ref{fig:data_use}.

\begin{figure}[h]
\centering \includegraphics[width=0.9\textwidth]{figures/data_use_figure.jpeg}
\caption{Patient data consists of longitudinal series of records, with each record consisting of both structured (e.g., diagnosis codes) and unstructured data (clinical notes).  The source task is learned exclusively using data from the input era of the training set patients. Target tasks inputs and labels exclusively from the input and label eras of the test set patients.  Data from the label era of the training patients is unused.  The validation split used for tuning model hyperparameters for the source task is omitted for clarity.}
\label{fig:data_use}
\end{figure}

We retained only those patients with usable labels and one or more non-empty annotated clinical notes in their input era, yielding a final data set of 115,232 patients and 2,735,6487 notes.
This final set was then split into training, validation, and test sets with 69,417, 11,290, and 34,525
patients and 1,644,841, 275,684, and 815,122 notes, respectively.

\subsection*{Intrinsic evaluation of word embeddings}
\label{sec:embd-eval}
In order to shed light on the effectiveness of our learned representations, we asked whether the geometric properties that have been used to evaluate the quality of word embeddings are correlated with the utility of representations for our target tasks.
Word embeddings are typically evaluated using analogy-like tasks that
test whether or not the geometry of the word embeddings recapitulates
known relationships between words.  We evaluated the word level
embeddings learned from clinical text, research article abstracts, and
through the recurrent net models using the Medical Relatedness
Property framework developed in \citep{choi2016learning}.  In this
framework, we test whether word embeddings express known \emph{may
  treat} and \emph{may prevent} relationships between drugs and
indications using vector arithmetic.  For instance, if $d_1, d_2, m_1,
\textnormal{ and } m_2 $ denote two drugs and indications
respectively, and the relation $r$ holds for $(d_1, m_1)$ and $(d_2,
m_2)$, we check whether or not the following holds for their
respective embeddings:

\begin{align*}
e_{d_1} - e_{m_1} + e_{m_2} \approx e_{d_2}
\end{align*}

More precisely, the quality metric is the ratio of concepts for which
at least one of the top-40 neighbors of $e_{d_1} - e_{m_1} + e_{m_2}$
satisfies the relation $r$ with $m_2$.  Results are shown in
\autoref{tab:intrinsic}.

\begin{table}[h]
\centering
\begin{tabular}{|l|c|c|}
\hline
\textbf{Embedding method} & \textbf{May-Treat  (\%)} & \textbf{May-Prevent (\%)}    \\ \hline
GloVe            & 9.23           & 10.21          \\ \hline
MCEMJ                        & 8.25           & 6.81           \\ \hline
GRU        & 1.26           & 0.43           \\ \hline
\end{tabular}
\caption{Evaluation of the Medical Relatedness Property for two
  relations: May-Treat and May-Prevent.  }
\label{tab:intrinsic}
\end{table}

Interestingly, we found that GloVe embeddings performed better than
the MCEMJ embeddings, despite randomized word orders and a
significantly larger vocabulary.  This suggests that, with respect to both this intrinsic evaluation and as a building block for patient level representations, it not necessarily advantageous to have access to full text.  In addition, embeddings from
randomly initialized neural models perform poorly at this task despite
their effectiveness in the patient level prediction tasks, suggesting
that word level evaluations are not particularly predictive of utility
in such tasks.

\subsubsection*{Results with N = 500 patients}

\input{sections/intrinsic_eval}

\section{Full Results}
\label{app:fullresults}
\input{sections/full_results}

\subsection{Presentation}\label{sec:pres}
We use patient data collected at Stanford Hospital in the Stanford Translation Research Integrated Database Environment (STRIDE) from 2009 through 2014.  This dataset comprises
Throughout this work, we use the STRIDE dataset, which is an anonymized medical data extract used by the \textit{Stanford Shah Lab} (BMIR\footnote{Stanford Center for Biomedical Informatics}.). This dataset is extracted from the much larger \textit{Stanford Translational Research Integrated Database Environment} (\cite{lowe2009stride}), a clinical data warehouse.

This extract contains about 27 million notes, corresponding to 1,2 million patients, most of them being between 1998 and 2014. There are about 49 million visits, each them having multiple procedures and diagnoses.

\begin{figure}[h]
\centering
\includegraphics[width=0.25\textwidth]{notes.png}
\caption{Notes are collections of terms, ordered in a patient's history}
\end{figure}

The clinical notes in this particular extract are processed and annotated using the Open Biomedical Annotator (\cite{lependu2013pharmacovigilance}), which is a pretty standard practice. This process anonymizes the notes (rendering the words un-ordered), and converts each term to concept IDs (CUI), which we will sill refer to as words in the remaining of the paper. The annotated notes have the following structure:

\begin{itemize}
 \item each patient has a number of clinical notes, written sequentially and timestamped
 \item each note is a (un-ordered) collection of terms
 \item terms have been processed, such that it is annotated whether they are negated and/or refer to the patient's family history
 \item terms can be mapped (in a many-to-one relation) to medical concepts
\end{itemize}

Each notes can also be associated with a \textit{visit}, during which various diagnoses are made (through ICD9 codes). Those diagnoses can be mapped to more general diseases categories through the \textit{Clinical Classification Software} (CCS)\footnote{https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp}. Thus, for each note, we can obtain a collection of CCS codes diagnosed at the same time as the note.

\subsection{Setup and pre-processing}

\subsubsection*{Vocabulary}
While it is common practice to discard all negated words (such as in \cite{miotto2016deep}), we decided to include them as extra words in the vocabulary, as we believe negations have a particular meaning in clinical notes. This means that each word in our vocabulary also has its negated counterpart.

This gave us about 265k words, with an extremely long tail, so we restricted our vocabulary to keep relevant words only. We removed words appearing in fewer than 50 notes, and those appearing in more than 10 million notes. This led us to a vocabulary size of 107,388.

\subsubsection*{Patient eras}
As most of our extrinsic evaluations consist of prediction on patients, we split each patient's records in two eras: the input era, and the target era. Records in the input era will be used to train our embeddings as well as inputs in evaluation tasks, whereas records in the target era will be used as prediction targets in evaluation tasks.

For each patient, we isolated a 6-months span of interest for the target era, and we used the previous year as input era. For instance, a year of medical notes and their embeddings could be used to predict the development of various diseases for a patient in the 6 months following this year of data.

\subsubsection*{Data split}
Our dataset consists of 69,416 patients for training, 11,289 for validation and 34,524 for testing. These patients were filtered from the STRIDE database to satisfy practical constraints such as having at least a couple visits during the input / target eras and having at least one clinical note containing a word form our restricted vocabulary.

\subsection*{Data}
We use patient data collected at Stanford Hospital from 2009 through 2014 in the Stanford Translational Research Integrated Database Environment (STRIDE).  Each patient’s data comprises a timestamped series of encounters with care providers at Stanford Hospital.  Encounters may generate both structured and unstructured data.  The structured data typically consists of discrete codes from controlled vocabularies for diagnoses, medications, and procedures, along with quantitative measurements such as vitals and laboratory measurements.  The unstructured data consists of the free text of clinical notes written by care providers documenting each encounter and other aspects of patient care.  In all, the initial dataset comprises over 49 million encounters and 27 million notes for 2 million patients.  However, the effective size of the dataset is much smaller after it has been filtered for usable data as describe below.
\section{Conclusion}

Limitations:
\begin{itemize}
\item we didn't have raw notes with actual sequential structure -- what are implications for each model? RNN could conceivably make use of that info, don't think it would affect embeddings much
\item how would these be impacted if we removed standard clinical preprocessing pipeline? Would probably degrade a little but how much? Also, if preprocessing is available, we should use it!
\item evaluations were all concerned with prediction -- how would these work for traditional clinical studies?
\item are our randomly sampled small data sets representative of clinical study cohorts (meeting narrowly defined criteria)?
\item how would these perform for clinician-assigned labels vs. common outcomes or "derived" labels
\item transfer to data from other hospitals?
\end{itemize}

Nitpicks (maybe don't discuss):
\begin{itemize}
\item larger data sets: use 10-fold cross validation to do lambda.1se instead of lambda.min, may disadvantage bag of words
\end{itemize}

Future work:
\begin{itemize}
\item learning common embedding space for other data types + notes
\item train RNN as autoencoder
\item interpretability?
\end{itemize}

We showed in this work that is possible to learn embeddings from patients' clinical notes, which can easily be transfered to considerably smaller populations for specific prediction and classification problems, using a particularly relevant and thorough set of evaluation tasks. We showed that such learned representations not only render those smaller problems computationally more tractable, they also have the ability to improve the accuracy of such predictions, by capturing more of the semantic information contained in the notes than baseline models.

Furthermore, we proved that recurrent neural network can learn representations that yield particularly good improvements on these evaluation tasks, by leveraging the sequential arrangements of the notes in a patient's history, provided the correct design choices are made. However, it is interesting to note that while our \textit{"embed and aggregate"} methods perform less well than our sequence models, they are way quicker to train, demand less tweaking, and can be transfered across institutions and sub-problems in a more flexible manner. It is also interesting to note that models that perform the best on our extrinsic tasks (which are realistically the most important) are not necessarily the ones yielding the best word embeddings according to our word-level evaluation.

There are future developments that could be made to this model. First, it would be interesting to develop a new expression of the \texttt{GloVe} or \texttt{word2vec} objective function that takes into account the specific structure of these notes (sample negated words during negative sampling, sample windows from the set of words at each iteration, etc.). Secondly, it would be worth exploring other options for the RNN supervision. Ideally, the labels would be less sparse and would only use the notes' content; so perfect labels would be some very high-level aggregation of the concepts present in the note.
\section{Details on note-level evaluation}\label{sec:app-i2b2}
The publicly available dataset contains approximately 1,230 clinical notes (730 for training and 500 for testing) and 15 targets. The targets are phenotypes such as \textit{Asthma, Obesity}, etc, and they all have 4 possible classes: Present, Absent, Questionable and Unmentioned.

The notes were processed with the Shah Lab annotator so that it had the exact same format as the notes in the STRIDE database. From the original training set, we randomly subsampled a validation set containing 20\% of the data. We report only the classification models that yielded the highest accuracies: a Random Forest with 10 trees was used for the baseline representation (bag-of-words) and a $k$-Nearest Neighbors for all other inputs ($k$ was chosen by cross-validation on the remaining training data).

\begin{table}[h]
\centering

\begin{tabular}{|l|c|c|c|}
\hline
\textbf{Embedding method}                     & \textbf{Aggregation} & \textbf{Textual}   & \textbf{Intuitive}  \\ \hline
Baseline                       & all           & \textbf{0,807} & \textbf{0,803}  \\ \hline
GloVe-300-W7-R2                    & max         & 0,784          & \textbf{0,785}  \\ \hline
GloVe-300-W7-R2                 & min         & 0,772          & 0,774           \\ \hline
GloVe-300-W7-R2                   & mean        & 0,754          & 0,759           \\ \hline
GloVe-300-W7-R2                   & sum         & 0,763          & 0,765           \\ \hline
GloVe-500-W7-R2                    & max         & \textbf{0,788} & 0,776           \\ \hline
GloVe-300-W10-R2                   & max         & 0,749          & 0,758           \\ \hline
GloVe-300-W10-R1                         & max         & 0,77           & 0,764           \\ \hline
GloVe-300-W4-R3                         & max         & 0,769          & 0,783           \\ \hline
MCEMJ                         & max         & 0,775          & 0,761           \\ \hline
Cross-channel            & max         & 0,775          & 0,759           \\ \hline
MaxGRU300                & max         & 0,780          & 0,783           \\ \hline
MaxGRU200-MCEMJ & max         & 0,773          & 0,769           \\ \hline
\end{tabular}
\caption{Micro-averaged F1 scores on the \textit{validation} set for several word embeddings}
\label{table:i2b2-details}
\end{table}

\section{Patient-level evaluation with 2,000 patients}
\label{app:2000}

We report, for reference, the results of some of our models when the evaluation task is trained on 2,000 patients instead of 500. As stated earlier, the baseline could not converge in this setting. What's reported is still the average AUROC across 20 training groups, as well as the standard error.

\begin{table}[H]
\centering
\begin{tabular}{|l|c|c|c|}
\hline
\textbf{Method}                   & \multicolumn{1}{l|}{\textbf{Death}} & \multicolumn{1}{l|}{\textbf{Admission}} & \multicolumn{1}{l|}{\textbf{ER Visit}} \\ \hline
\textbf{Baseline}                 & NA                                  & NA                                      & NA                                     \\ \hline
\textbf{GloVe-300-W7-R2  + MCEMJ} & 0.864 (0.001)                       & 0.783 (0.002)                           & \textbf{0.78 (0.001)}                  \\ \hline
\textbf{MaxGRU300}                & \textbf{0.866 (0.001)}              & \textbf{0.791 (0.001)}                  & 0.768 (0.002)                          \\ \hline
\textbf{MaxGRU600}                & 0.864 (0.001)                       & 0.789 (0.001)                           & 0.775 (0.001)                          \\ \hline
\textbf{MaxGRU500-MCEMJ}          & 0.849 (0.001)                       & 0.775 (0.002)                           & 0.766 (0.001)                          \\ \hline
\textbf{MaxGRU200-MCEMJ}          & 0.854 (0.001)                       & 0.782 (0.002)                           & 0.767 (0.001)                          \\ \hline
\end{tabular}
\caption{Evaluation on prediction of mortality and hospital utilization.}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|l|c|c|c|c|c|c|}
\hline
\textbf{Method}                   & \textbf{Thyroid}       & \textbf{Dysrhythmias}  & \textbf{Diabetes}      & \textbf{Spondylosis}   & \textbf{LMD}           & \textbf{Hypertension}  \\ \hline
\textbf{Baseline}                 & NA                     & NA                     & NA                     & NA                     & NA                     & NA                     \\ \hline
\textbf{GloVe  + MCEMJ} & 0.704 (0.003)          & 0.728 (0.002)          & 0.735 (0.002)          & 0.73 (0.001)           & 0.706 (0.002)          & 0.719 (0.002)          \\ \hline
\textbf{MaxGRU300}                & 0.744 (0.002)          & 0.738 (0.013)          & 0.77 (0.002)           & 0.761 (0.001)          & 0.749 (0.001)          & 0.768 (0.001)          \\ \hline
\textbf{MaxGRU600}                & \textbf{0.759 (0.002)} & 0.738 (0.013)          & \textbf{0.785 (0.002)} & \textbf{0.766 (0.001)} & \textbf{0.752 (0.001)} & \textbf{0.778 (0.002)} \\ \hline
\textbf{MaxGRU500-MCEMJ}          & 0.685 (0.003)          & 0.737 (0.003)          & 0.757 (0.001)          & 0.745 (0.001)          & 0.737 (0.001)          & 0.768 (0.001)          \\ \hline
\textbf{MaxGRU200-MCEMJ}          & 0.706 (0.002)          & \textbf{0.741 (0.001)} & 0.761 (0.001)          & 0.755 (0.001)          & 0.743 (0.002)          & 0.768 (0.001)          \\ \hline
\end{tabular}
\caption{Evaluation on prediction of future diseases.}
\end{table}
\subsection{Word-level evaluation}\label{sec:embd-eval}

We evaluate our word embeddings through the \textit{Medical Relatedness Property} developed in \cite{choi2016learning} \footnote{We used their code to compute this metrics, which is available at \url{https://github.com/clinicalml/embeddings}}.

\begin{table}[h]
\centering
\begin{tabular}{|l|c|c|}
\hline
\textbf{Embedding method}             & \textbf{May-Treat  (\%)}    & \textbf{May-Prevent (\%)}    \\ \hline
GloVe300-W10-R1           & 7.83           & 8.51           \\ \hline
GloVe-100-W7-R2            & 6.01           & 08.09          \\ \hline
GloVe-300-W4-R2            & 8.81           & \textbf{10.64} \\ \hline
GloVe-300-W7-R2            & 8.25           & 10.21          \\ \hline
GloVe-500-W7-R2            & 9.23           & 10.21          \\ \hline
GloVe-300-W10-R2           & \textbf{10.49} & 9.79           \\ \hline
GloVe-300-W4-R3            & 6.57           & 7.23           \\ \hline
MCEMJ                        & 8.25           & 6.81           \\ \hline
Cross-channel                & 5.45           & 2.55           \\ \hline
MaxGRU200-MCEMJ & 7.27           & 5.53           \\ \hline
MaxGRU300        & 1.26           & 0.43           \\ \hline
\end{tabular}
\caption{Evaluation of the Medical Relatedness Property for two relations: May-Treat and May-Prevent. "GloVe-300-W10-R2" corresponds to GloVe embeddings trained on our clinical notes, with length 300, window size 10, and clinical notes repeated twice in the corpus. All GloVe models were run for 25 iterations. "MaxGRU200-MCEMJ" is a RNN with GRU cells, 200 cells, initialized with MCEMJ. This evaluation was computed with the code provided by \cite{choi2016learning}}
\label{my-label}
\end{table}

Interestingly, we notice that many of our GloVe embeddings perform better than the MCEMJ ones (learned from abstracts of medical journals), despite randomized word orders and a vocabulary twice as big.
In addition, embeddings from randomly initialized neural models perform worse, especially the RNN.

\subsection{Note-level evaluation}
\label{sec:note-eval}

To evaluate our note representations, we used the \texttt{i2b2} (Informatics for Integrating Biology to the Bedside) 2008 Obesity Challenge \cite{uzuner2009recognizing}.
The publicly available dataset contains approximately 1,230 clinical notes and 15 phenotype targets such as \textit{Asthma} and \textit{Obesity}.  The dataset is divided in two: the \textit{textual task} for which labels are attributed from explicit mention in the notes; for the \textit{intuitive task} however, classification was done by using doctors' intuition and judgment.

To align with this competition, we consider the micro-averaged F1\footnote{Which corresponds to summing counts (true positives, \dots) across classes, see \cite{uzuner2009recognizing} for details.} as our main metric. For simplicity, we will only report the average performance across the 15 targets. More details about this dataset and our approach is provided in Appendix \ref{sec:app-i2b2}. We used a raw bag-of-words encoding of the notes as baseline.

\begin{table}[h]
\centering
\begin{tabular}{|l|c|c|}
\hline
\textbf{Embedding method}               & \textbf{Textual}        & \textbf{Intuitive}      \\ \hline
Baseline                       & \textbf{0,822} & \textbf{0,802} \\ \hline
GloVe-300-W7-R2         & 0,784          & 0,773          \\ \hline
GloVe-500-W7-R2        & 0,779          & 0,774          \\ \hline
MCEMJ                         & 0,776          & 0,765          \\ \hline
MaxGRU300     & \textbf{0,793} & \textbf{0,778} \\ \hline
MaxGru200-MCEMJ & 0,781          & 0,769          \\ \hline
\end{tabular}
\caption{Micro-averaged F1 scores averaged over the 15 targets of the i2b2 Obesity Challenge test set. All embeddings where aggregated among the notes by component-wise maximum.}
\label{table:i2b2-test}
\end{table}

Overall, we did not notice much disparity in the accuracies yielded by the different representations, although the baseline performed a bit better than all our dense features. We also observed that in general, aggregating the word representation in a note with the \texttt{max} operator yielded the best results (compared to \texttt{mean, min, sum}). Finally, we notice that the baseline had a larger drop in performance between the \textit{textual} task and the \textit{intuitive} one, compared to the models based on the word embeddings. This could mean that word embeddings become more robust when the target gets more complex.

\subsection{Patient-level evaluation}

The final evaluation - at the patient level - consists of various real-world applications in which such embeddings could be used, and thus it is our extrinsic evaluation.
We look at the performance of various prediction tasks, using only the learned embeddings as input. Moreover, to replicate the common drawback of prediction tasks in healthcare in actual settings, where sample sizes for actual problems of interest are often relatively small, we only use small training sets (respectively 500 and 2,000 patients) to evaluate those prediction tasks (the results are then averaged over 20 trials).

The tasks we use are particularly relevant to bioinformatics scientists and healthcare providers. We use the embeddings to predict various events in the patient's "target era": mortality. future admission, future emergency visits (ER), future development of the 6 densest diseases in our training set (thyroid disorder, cardiac dysrhythmias, diabetes, spondylosis, disorder of lipid metabolism, and essential hypertension).

All of our evaluation models were trained using Ridge logistic regression, cross-validated over 5 folds (using \texttt{glmnet}), on a per-training group basis. The results were further averaged over 20 sampled training groups.
All GloVe methods reported here use a window size of 7 and 2 resampling of the notes. In addition all aggregations of word embeddings (GloVe and MCEMJ) are concatenations of aggregations by maximum, minimum and average since this always improved the performance.

The baseline is a Ridge logistic regression fit on a raw bag-of-words encoding of the patient history of notes, where the vocabulary is restricted to the most frequent words appearing in the small training set\footnote{This is the only way to have this model converge in a majority of cases.}. We only kept the results when the baselines converged, but this does not happen all the time, and would even more rarely with longer eras and more patients.

\subsubsection*{Results with 500 patients}
For each model, we report the resulting average and standard error of Area Under Receiver-Operator Curve (AUROC):

\begin{table}[H]
\centering
\captionsetup{justification=centering}
\setlength{\tabcolsep}{10pt}
\begin{tabular}{|l|c|c|c|}
\hline
\textbf{Method}             & \multicolumn{1}{l|}{\textbf{Death}} & \multicolumn{1}{l|}{\textbf{Admission}} & \multicolumn{1}{l|}{\textbf{ER Visit}} \\ \hline
Baseline  &
0.801 (0.019)&	0.711 (0.019)&	0.698 (0.005)                                        \\ \hline
GloVe-300 + MCEMJ  & 0.811 (0.005)                       & 0.758 (0.002)                           & 0.756 (0.002)                          \\ \hline
GloVe-500  + MCEMJ & 0.809 (0.006)                       & 0.757 (0.003)                           & 0.755 (0.003)                          \\ \hline
MCEMJ              & 0.795 (0.004)                       & 0.74 (0.003)                            & 0.729 (0.004)                          \\ \hline
GloVe-300          & 0.797 (0.005)                       & 0.752 (0.003)                           & 0.752 (0.002)                          \\ \hline
GloVe-500          & 0.805 (0.005)                       & 0.752 (0.004)                           & 0.752 (0.003)                          \\ \hline
MaxGRU300          & \textbf{0.842 (0.003)}              & 0.723 (0.022)                           & 0.745 (0.005)                          \\ \hline
MaxGRU600          & 0.829 (0.017)                       & \textbf{0.765 (0.004)}                  & \textbf{0.759 (0.002)}                 \\ \hline
MaxGRU500-MCEMJ    & 0.812 (0.017)                       & 0.746 (0.006)                           & 0.747 (0.004)                          \\ \hline
MaxGRU200-MCEMJ    & 0.817 (0.017)                       & 0.75 (0.006)                            & 0.754 (0.002)                          \\ \hline
\end{tabular}
\caption{Evaluation on mortality and utilization prediction tasks. Reported metrics are mean and standard error of AUROC and across 20 small evaluations.}
\end{table}

\begin{table}[H]
\centering
\captionsetup{justification=centering}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\textbf{Methods}  & \textbf{Thyroid} & \textbf{Dysrhythmias} & \textbf{Diabetes}      & \textbf{Spondylosis}   & \textbf{LMD\footnote{Lipid metabolism disorder.}} & \textbf{Hypertension} \\ \hline
Baseline                  &
0.610 (0.009) &	0.687 (0.01)	& 0.68 (0.005)	& 0.667 (0.017)	& 0.635 (0.011) & 	0.664 (0.009)
\\ \hline
GloVe-300 + MCEMJ & 0.632 (0.009)              & 0.69 (0.004)                  & 0.689 (0.004)          & 0.696 (0.004)          & 0.648 (0.009)                          & 0.679 (0.005)                   \\ \hline
GloVe-500 + MCEMJ & 0.648 (0.006)              & 0.689 (0.005)                 & 0.691 (0.003)          & 0.701 (0.004)          & 0.649 (0.009)                          & 0.667 (0.01)                    \\ \hline
MCEMJ             & 0.642 (0.01)               & 0.688 (0.004)                 & 0.69 (0.004)           & 0.688 (0.003)          & 0.651 (0.005)                          & 0.666 (0.01)                    \\ \hline
Glove-300         & 0.618 (0.008)              & 0.683 (0.005)                 & 0.683 (0.004)          & 0.689 (0.003)          & 0.649 (0.005)                          & 0.672 (0.005)                   \\ \hline
Glove-500         & 0.618 (0.01)               & 0.686 (0.004)                 & 0.683 (0.004)          & 0.693 (0.003)          & 0.642 (0.008)                          & 0.67 (0.004)                    \\ \hline
MaxGRU300         & 0.701 (0.011)              & \textbf{0.709 (0.017)}        & 0.737 (0.013)          & 0.732 (0.012)          & 0.7 (0.015)                            & \textbf{0.745 (0.003)}          \\ \hline
MaxGRU600         & \textbf{0.721 (0.004)}     & 0.673 (0.023)                 & \textbf{0.744 (0.013)} & \textbf{0.738 (0.013)} & 0.702 (0.016)                          & 0.735 (0.013)                   \\ \hline
MaxGRU500-MCEMJ   & 0.623 (0.013)              & 0.673 (0.02)                  & 0.718 (0.012)          & 0.725 (0.004)          & \textbf{0.706 (0.004)}                 & 0.739 (0.005)                   \\ \hline
MaxGRU200-MCEMJ   & 0.662 (0.01)               & 0.697 (0.016)                 & 0.705 (0.02)           & 0.728 (0.012)          & 0.703 (0.011)                          & 0.741 (0.003)                   \\ \hline
\end{tabular}
\caption{Evaluation on prediction of future diseases. Reported metrics are mean and standard error of AUROC and across 20 small evaluations. }
\end{table}

We can observe most models beat the baseline quite easily (which often did not converge when fitting these evaluation models), except on mortality prediction where it appears to perform particularly well. The sequence models achieve the best results, however it is unclear whether it is due to the training supervision or to the fact it captures the sequential semantic information in the notes.

\subsubsection*{Results with 2,000 patients}
The results with 2,000 patients are reported in the appendix \ref{app:2000}. The baseline is not included since it could never converge (2,000 patients per training groups means more word features, at a level where \texttt{glmnet} could not converge).

\section{Methods}

We explore three types of methods, which we explain here. They are, respectively, learning word-level embeddings and aggregating them to a fixed-length representation, fitting a sequence model to patients' notes to learn all embeddings jointly, and  learning a cross-channel encoder from medical notes to diagnoses.

\subsection{Embed and aggregate}
We first propose a method to learn word embeddings from clinical notes, and then discuss how these can be aggregated to represent a note or a patient state.

\subsubsection{Learning word embeddings from clinical notes}

It is natural to think about common methods in Natural Language Processing to learn word embeddings, such as \texttt{GloVe}\cite{pennington2014glove} and \texttt{Word2Vec}\cite{mikolov2013distributed}. However, as explained above, the Open Biomedical Annotator removes the order of words in the notes as part of the de-indentification process. We decided to tackle this challenge by simply re-generating notes with a random word order. Such approach was already proposed (e.g. in \cite{choi2015doctor} and \cite{choi2016learning}) as an easy trick to deal with the unordered medical codes and did not seem to alter the models' quality. However since clinical notes can be quite long, we suspect that it might be too hard for our models to capture the context of a word. Therefore, we also propose to juxtapose a couple times duplicates of a same note, but with different random orders.

This makes our data look exactly like text data, so that we can learn embeddings with  an efficient C implementation of GloVe\footnote{\url{https://github.com/stanfordnlp/GloVe}}.

\subsubsection{Learning word embeddings from external data (MCEMJ)}
\label{sec:mcemj}

In \cite{de2014medical} the authors learn embeddings from 350,000 medical journal abstracts in the OHSUMED dataset. They first map the free text to UMLS\footnote{Unifieid Medical Language System, the same as the one we use.} concept ID's and then learn the word embeddings using Word2Vec’s skip-gram model with hierarchical softmax. Their best results were achieved with embeddings of dimension 200 and a window size of 5. 52,100 such embeddings were shared publicly by the authors \footnote{\url{https://github.com/clinicalml/embeddings}} and approximately 28,000 of them belonged to our vocabulary. We will refer to them as MCEMJ (Medical Concept Embeddings from Medical Journal).

\subsubsection{Fixed-length aggregation at the note and patient level}
Although it seems to be a simple approach, it is still very common in medical applications to aggregate a patient's data over time by summing or averaging counts of medical codes. Similarly, we propose to aggregate (mean, max, min, or sum) the word embeddings per note and then per patient. We explored these various ways of pooling the embeddings both on the note and the patient level.

In addition, we investigate the utility of concatenating multiple of these  patient-level aggregations, especially some derived by different sources of embeddings (e.g. randomly ordered notes and medical journals).

\subsection{Recurrent neural network}

Our methods so far do not rely on the sequential nature of the notes in a patient's history. For a number of prediction tasks, it is important to capture the progressive evolution of a patient's state in its representation. Thus, it is a natural idea to use recurrent neural networks to learn those patient-level embeddings.

\subsubsection{Supervision}

We supervise our network with diagnoses associated with the notes (from the corresponding \textit{visits}, as explained in \ref{sec:pres}). Since these descriptors are extremely high-dimensional, we map them to more general disease categories using the Clinical Classification Software (CCS)\footnote{https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp}, leaving a label space of dimension 254.

At this point we have a design choice, training the network as a multi-task sequence classification problem (aggregating the CCS labels on a per-patient basis), or as a sequence labeling problem (supervising each timestep with its corresponding CCS codes). Experiments showed that the former gave considerably higher scores on the evaluation task, which is a probable consequence of the labels' sparsity (the latter providing very scarce supervision signal).

\subsubsection{Architecture}

\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{rnn.png}
\caption{Model architecture}
\end{figure}

\textbf{Pooled embedding layer}

Since this un-ordered structure in the input is quite uncommon, we use an unusual layer as input to our neural network. This layer computes trainable embeddings from the collections of words in an input note, and pools them before feeding them to the rest of the network. More precisely, we use max-pooling aggregation, so the input to our network for a note $x$ is actually given by:

$$ \phi_L(x) = \text{max}_1(x \odot  L) $$

where $x=[0,0,1, ... ,0,1,0]$ is the note bag-of-words encoding of shape $(V,1)$, $L$ is the word embedding matrix of shape $(V, D_x)$ ($V$ being the vocabulary size and $D_x$ the word embedding size), and $\text{max}_1$ performing a max over the first dimension.

We also explored with aggregating the word embeddings by taking their average, but this performed extremely poorly on the evaluation tasks. It makes sense to use particularly activated features in the learned word representations rather than their average, as those activated features will capture specific semantic indicators from the note.

\textbf{RNN cells}

We explored both using Long Short-Term Memory (\cite{hochreiter1997long}) and Gated Recurrent Units (\cite{cho2014learning}) as cell structure for our recurrent network. We found that LSTM learned a poor representation of the input, leaving a number of columns at 0 and performing poorly on evaluation tasks, which confirms the findings of \cite{choi2015doctor}. Thus, most of our final experiments were conducted with GRUs, and this is what we strongly recommend for researchers working with similar problems. \\

\textbf{Other details}

We used \texttt{tanh} activations, dropout (with drop probability 0.1 or 0.2 depending on models), and a dense layer with binary cross-entropies for the multi-task supervision. We trained the networks for 100 epochs, using RMSProp (\cite{tieleman2012lecture}) for loss optimization. The models were implemented in Keras (\cite{chollet2015keras}) and trained on a NVIDIA GeForce GTX TITAN X GPU.

\subsubsection{Embeddings initializations}

We explored initializing our embeddings in various ways. In particular, we tried initializing them randomly, with word embeddings pre-trained on our notes, and with word embeddings trained on journal abstract (MCEMJ) (see \ref{sec:mcemj} for details). While the results for random initialization and MCEMJ initialization are reported below, we found that initializing the weights with embeddings pre-trained on the clinical notes themselves gave very poor results on the evaluation tasks.

\subsection{Note-level cross-channel autoencoder}

Finally, we experimented with a flat version of the sequential model presented in the previous section, \textit{ie.} a two-hidden-layer neural network where the first layer is a randomly initialized embedding layer with max-pooling. For this model, the inputs are the clinical notes and the non-exclusive targets are CCS labels derived from the corresponding visit data.

This model seems appealing since it leverages our data to provide supervision in order to learn word embeddings. In addition, it is simpler to implement and deploy than a sequential model. However we found that in practice it mostly combined the drawbacks of both previous models: it does not provide a clear patient-level representation, it requires a lot of training data (and labels!), and is also very slow to run.

Machine Learning (stat.ML); Machine Learning (cs.LG)