One common motivation across the following studies,
that is shared by this work and the research we have already discussed, is to scale
transformers to long sequences.
Note that there are plenty orthogonal choices for improving efficiency such as weight sharing [1]}, quantization [2]}, knowledge distillation [3]}, and adapters [4]}.
For a detailed overview we refer the reader to [5]}.
