Li et al. proposed LogSparse Transformers to overcome memory challenges, thus making Transformers more feasible for time-series with long-term dependencies [1]}. LogSparse Transformers allow each time-step to attend to previous time-steps, selected using an exponential step size, reducing the memory utilization from \(\mathcal {O}(L^2)\)  to \(\mathcal {O}(L\log _2 L)\)  in each self-attention layer. Fig REF  shows the various ways in which LogSparse self-attention can be used for time-series analysis. The canonical self-attention mechanism used for neighboring time-steps in a specific range allows for gathering more information. Beyond that range, the LogSparse self-attention mechanism is applied. Another way is to restart LogSparse step size after a particular range of time-steps.
<FIGURE>