Traditional RNN models are widely used for learning with sequential data. However, they are ineffective when dealing with long sequential data due to the vanishing gradient issue and computation cost of recurrent operations. Temporal convolutional network (TCN), e.g., WaveNet [1]}, can capture long-range temporal dependencies via dilated causal convolutions. A more recent work suggests that TCN outperforms RNN in various prediction problems based on sequential data, particularly when the input sequences are long [2]}. However, TCN models rely on deep hierarchy to ensure the causal convolutions and thus achieve large receptive fields. Deep hierarchy, namely a large stack of layers, incurs significant computation cost for inference at run time. Efficient attention models adapted from Transformer [3]} have been proposed recently for learning representations from long sequential data, which mainly focus on replacing the quadratic dot-product attention calculation with more efficient operations [4]}, [5]}. In this work, SEHM builds on previous insights and introduces a hierarchical model that integrates kernelized local attention and RNN. Kernelized local attention captures important local patterns and reduces the size of intermediate representation, while the higher-level RNN model learns long-term dynamics. As a result, SEHM can achieve better predictive performance and computational efficiency when learning and inferring from long multivariate intraoperative time series.
