As an alternative, Li et al. [1]} considered the use of transformers for time series prediction. Transformers use a self-attention mechanism that allows each observation in the feature sequence to attend independently to every other feature in the sequence. However, they have considerable computational and memory requirements that grow quadratically with respect to sequence length, making it computationally rigorous to train large-scale data [1]}. Such deficiency makes them computationally unsuitable for extreme events that often appear in longer sequences than the transformer's inputs. Moreover, it was not even clear from the design itself that transformers can be as effective as RNNs, whereas Zaheer et al. [3]} reported that the attention mechanism in transformers does not even obey the sequence order of time steps which is essential for the time series domain. What is more, non-transformer architectures (i.e., MLP) when designed and trained properly, can perform competitively with transformers [4]}.
