Recently, there has been extensive research to find good token mixers; see e.g., [1]} and references therein. The original self-attention imposes graph structures, and uses the similarity among the tokens to capture the long-range dependencies [2]}, [3]} . It is parameter efficient and adaptive, but suffers from a quadratic complexity in the sequence size. To achieve efficient mixing with linear complexity, several approximations have been introduced for self-attention; see Section . These approximations typically compromise accuracy for the sake of efficiency. For instance, long-short (LS) transformer aggregates a long-range attention with dynamic projection to model distant correlations and a short-term attention to capture local correlations [4]}. Long range dependencies are modeled in low dimensions, which can limit expressiveness.
<FIGURE><TABLE>