Existing attention accelerators [1]}, [2]}, [3]}, [4]} meet performance bottlenecks given long input sequences. FTRANS [1]} compresses weights of transformers while computing the full attention without sparsity. SpAtten [2]} applies coarse-grained pruning to remove tokens and heads with a relatively low pruning ratio, which cannot effectively shorten the sequence length. \(A^3\)  [3]} and Sanger [4]} accelerate the attention by utilizing dynamic sparse patterns, which can incur large additional memory/computation overhead in the case of long sequences.
To this end, we propose SALO to efficiently accelerate the attention in transformers for tasks with long input sequences. The main contributions are summarized as follows:
