Recent robust prediction models rely on the LSTMs or transformers architecture to provide robustness in their prediction. Even though LSTMs are designed to obtain long-term dependencies, their capacity to pay different degrees of attention to sub-window features within large time steps is inadequate [1]}. As an example, Khandelwa et al. [2]} showed that even though the LSTM model can have an effective sequence size of 200 observations, they are only able to sharply distinguish the 50 closest observations. This indicates that even LSTMs struggle to capture long-term dependencies.
On the other hand, conventional transformers
suffer from quadratic computation and memory requirements, which limits their ability to process long input sequences.
Even though such memory bottlenecks have been improved by using sparse-attention algorithms [3]}, their performance improvement is not significant compared to a full-attention mechanism for real-world datasets [4]}. Given that extreme events and anomalies are rare and can appear at very long distances from each other, it is computationally infeasible to increase the input sequence to provide attention to all previously seen anomalies and extreme events.
