More approaches to improving attention include Non-Autoregressive Spatial-Temporal Transformer (NAST) [1]} and Informer [2]}. The architecture proposed in NAST is referred to as the Spatial-Temporal Attention Block. This approach combines a spatial attention block, which makes predictions across space, with a temporal attention block which makes predictions across time. The result improves learning in both spatial and temporal domains. The Informer architecture replaces the canonical self-attention with the ProbSparse self-attention mechanism, which is more efficient in time complexity and memory usage. This Transformer also uses a self-attention distilling function to decrease space complexity. These two mechanisms make the Informer highly efficient at processing exceedingly large input data sequences.
