To remedy this issue, several models with hybrid sparse attention mechanisms are proposed to handle long sequences, such as Longformer [1]} (up to 16384 tokens in a sequence) and ViL [2]} (up to \(96\times 96\)  patches in an image). They hybridize two variants of the attention mechanism and perform the computation of a local window attention and a task-motivated global attention. Such a hybrid sparse attention mechanism successfully reduce the complexity of attention to a linear level. This linear complexity significantly alleviates the memory burden, making it possible to train models with long sequences. However, since the hybrid sparse attention mechanism is not directly supported by the highly optimized GEMM kernels on CPU and GPU, the inference speed is limited, requiring efficient hardware acceleration to achieve a higher speed.
