An intuitive way to improve the efficiency of Transformer is computing a sparse self-attention matrix to reduce the number of tokens to be attended [1]}.
For example, [2]} proposed Longformer, which computes sliding window attention to capture local contexts and global attention at a few positions to capture global contexts.
[3]} proposed BidBird, which further incorporates random attention that models the interactions between each token with a certain number of randomly selected tokens.
However, attending to the tokens that are randomly sampled or selected by certain rules may not be actually helpful for context modeling.
<FIGURE>