Lastly we will discuss three architectures that improve upon attention, LogSparse Transformers [1]}, TFT [2]}, and YFormer [3]}. LogSparse Transformers introduce convolutional self-attention blocks, consisting of a convolutional layer prior to the attention mechanism for creating the queries and keys. A temporal self-attention decoder is used in TFT for learning the long-term dependencies in the data. The YFormer architecture proposes a sparse attention mechanism combined with a downsampling decoder. This architecture allows the model to better detect long-range effects in the data.
