Disease progression models are instrumental in predicting individual-level health trajectories and understanding disease dynamics. Existing models are capable of providing either accurate predictions of patients prognoses or clinically interpretable representations of disease pathophysiology, but not both. In this paper, we develop the phased attentive state space (PASS) model of disease progression, a deep probabilistic model that captures complex representations for disease progression while maintaining clinical interpretability. Unlike Markovian state space models which assume memoryless dynamics, PASS uses an attention mechanism to induce "memoryful" state transitions, whereby repeatedly updated attention weights are used to focus on past state realizations that best predict future states. This gives rise to complex, non-stationary state dynamics that remain interpretable through the generated attention weights, which designate the relationships between the realized state variables for individual patients. PASS uses phased LSTM units (with time gates controlled by parametrized oscillations) to generate the attention weights in continuous time, which enables handling irregularly-sampled and potentially missing medical observations. Experiments on data from a realworld cohort of patients show that PASS successfully balances the tradeoff between accuracy and interpretability: it demonstrates superior predictive accuracy and learns insightful individual-level representations of disease progression.
There have been various attempts to create interpretable RNN-based predictive models using attention. The models in @cite_27 and @cite_4 use the reverse-time attention mechanism to learn visit-level and variable-level attention weights that explain the prediction of a target label through measures of variable importance. The phased attention mechanism proposed in Section is a generalization of the reverse-time attention mechanism in @cite_27 that can operate in continuous-time, and update the attention weights at irregularly-spaced and potentially incomplete observations. The main difference between the way attention is used in PASS and the way it is used in models like @cite_27 can be summarized as follows. PASS applies attention to the latent state space , whereas applies attention to the observable sample space . Hence, the attention mechanism gives different types of explanations in the two models. In PASS, the phased attention mechanism interprets the hidden disease dynamics, and hence it provides an explanation for the mechanisms underlying disease progression. On the contrary, uses attention to measure feature importance, and hence it only explains predictions, but does not explain the disease progression mechanisms.
@cite_27 @cite_4
2952042565 2754586843 2962889000 2086210198
The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction. Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02 on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets. Abstract Recurrent Neural Networks (RNNs) have been widely used in natural language processing and computer vision. Amongst them, the Hierarchical Multi-scale RNN (HM-RNN), a recently proposed multi-scale hierarchical RNN, can automatically learn the hierarchical temporal structure from data. In this paper, we extend the work to solve the computer vision task of action recognition. However, in sequence-to-sequence models like RNN, it is normally very hard to discover the relationships between inputs and outputs given static inputs. As a solution, the attention mechanism can be applied to extract the relevant information from the inputs thus facilitating the modeling of the input–output relationships. Based on these considerations, we propose a novel attention network, namely Hierarchical Multi-scale Attention Network (HM-AN), by incorporating the attention mechanism into the HM-RNN and applying it to action recognition. A newly proposed gradient estimation method for stochastic neurons, namely Gumbel-softmax, is exploited to implement the temporal boundary detectors and the stochastic hard attention mechanism. To reduce the negative effect of the temperature sensitivity of the Gumbel-softmax, an adaptive temperature training method is applied to improve the system performance. The experimental results demonstrate the improved effect of HM-AN over LSTM with attention on the vision task. Through visualization of what has been learnt by the network, it can be observed that both the attention regions of the images and the hierarchical temporal structure can be captured by a HM-AN. The vigilance decrement has been described as a slowing in reaction times or an increase in error rates as an effect of time-on-task during tedious monitoring tasks. This decrement has been alternatively ascribed to either withdrawal of the supervisory attentional system, due to underarousal caused by the insufficient workload, or to a decreased attentional capacity and thus the impossibility to sustain mental effort. Furthermore, it has previously been reported that controlled processing is the locus of the vigilance decrement. This study aimed at answering three questions, to better define sustained attention. First, is endogenous attention more vulnerable to time-on-task than exogenous attention? Second, do measures of autonomic arousal provide evidence to support the underload vs overload hypothesis? And third, do these measures show a different effect for endogenous and exogenous attention? We applied a cued (valid vs invalid) conjunction search task, and ECG and respiration recordings were used to compute sympathetic (normalized low frequency power) and parasympathetic tone (respiratory sinus arrhythmia, RSA). Behavioural results showed a dual effect of time-on-task: the usually described vigilance decrement, expressed as increased reaction times (RTs) after 30 min for both conditions; and a higher cost in RTs after invalid cues for the endogenous condition only, appearing after 60 min. Physiological results clearly support the underload hypothesis to subtend the vigilance decrement, since heart period and RSA increased over time-on-task. There was no physiological difference between the endogenous and exogenous conditions. Subjective experience of participants was more compatible with boredom than with high mental effort.