Recently, Zhou et al. proposed Informer, which uses a ProbSparse self-attention mechanism to optimize the computational complexity and memory usage of the standard Transformer architecture [1]}. The authors also introduced the self-attention distilling operation, which considerably reduces the total space complexity of the model.
