There are also many other ways to accelerate Transformers [1]}, [2]}.
For example, Reformer [1]} uses hashing techniques to cluster input embeddings into different buckets based on their similarities, and then chunks the buckets using a certain length. The tokens only attend to same bucket in their own chunk and previous chunk.
Linformer [2]} assumes that the self-attention matrix is low-rank, and it approximates the self-attention mechanism by using low-rank attention key and value projected by separate linear transformations.
Linear Transformer [5]} uses kernel functions to approximate the self-attention mechanism.
It derives a kernel-based formulation of self-attention based on the matrix multiplication associative property and designs a simple kernel function to approximate the computation.
However, these methods do not fully consider the characteristics of natural language and may be suboptimal in text understanding.
Different from existing efficient Transformers, Smart Bird uses learnable sparse attention to capture important interactions between tokens, which can achieve both efficient and effective text modeling.
