Rfa and its gated variant (§)
can be used as a drop-in substitute for the canonical softmax attention,
and increase the number of parameters by less than 0.1%.
We explore its applications in transformers
on language modeling, machine translation, and long text classification (§).
Our experiments
show that Rfa achieves comparable performance to vanilla transformer baselines
in all tasks, while outperforming a recent related approach [1]}.
The gating mechanism proves
particularly useful in language modeling: the gated variant of Rfa outperforms the transformer baseline on WikiText-103.
Rfa shines in decoding, even for shorter sequences.
In our head-to-head comparison on machine translation benchmarks,
Rfa decodes around \(2\times \)  faster than a transformer baseline,
without accuracy loss.
Comparisons to several recent efficient transformer variants on three
long text classification datasets show that
Rfa is competitive in terms of both accuracy and efficiency.

Our analysis (§) shows that more significant time and memory efficiency
improvements
can be achieved for longer sequences: 12\(\times \)  decoding speedup with less than 10% of the memory for 2,048-length outputs.
