Recurrent neural network (RNN) is a neural network used to process sequence data.
Theoretically, RNN can store long-term memory and update the previous state according to the current input at any time, but in fact, it is very difficult. In another word, RNN is difficult to solve the problem of long-term dependence [1]}.
LSTM [2]} is a special RNN, which is mainly to solve the problems of gradient disappearance and gradient explosion in the process of long sequence training. Compared with ordinary RNN, LSTM has better performance in long sequences, but LSTM can only maintain a long-term dependence within about 50 time steps.
Phased LSTM [3]} can solve the problem that LSTM can not process irregular input sequences. By integrating different sampling frequencies or irregularly sampled data on phase gate, Phase LSTM can remember signals with different periods, and the state can propagate for a long time. When the processing sequence reaches thousands of steps, LSTM is almost unavailable, while Phased LSTM performs well. But Phased LSTM is not suitable for modeling the complex event sequence with thousands of event types.
HE-LSTM [4]} is proposed to deal with heterogeneous temporal events in long-term dependence, but it can only extract event types while the features relationship of events can not be obtained.
Transformer [5]} is a powerful architecture that can achieve excellent performance on a variety of sequential learning tasks, which does not perform recursion on the sequence, but processes the feedforward model of the whole sequence simultaneously.
Recent research shows that transformer has the potential to improve the prediction ability [6]}. However, transformer has some serious problems that make it unable to be directly applied to multimodal irregular time-series data, such as quadratic time complexity, high memory utilization and the inherent limitations of encoder-decoder architecture [7]}.
In addition to the above problems, the biggest problem of Transformer is that the model contains no recurrence and no convolution, which results in the input tensor can not contain the time relationship of the input sequence effectively[8]}.
