Transformer architectures [1]}
have achieved tremendous success on a variety of sequence modeling
tasks [2]}, [3]}, [4]}, [5]}, [6]}.
Under the hood, the key component is attention [7]}, which models pairwise interactions of the inputs,
regardless of their distances from each other.
This comes with quadratic time and memory costs,
making the transformers computationally expensive, especially for long sequences.
A large body of research has been devoted to
improving their time and memory efficiency [8]}.
Although better asymptotic complexity
and prominent gains for long sequences have been achieved [9]}, [10]}, [11]},
in practice, many existing approaches are less well-suited
for moderate-length ones:
the additional computation steps required by some approaches can overshadow
the time and memory they save [12]}, [13]}, [14]}.
