Notes on decoding speed.
With a lower memory overhead, Rfa can use
a larger batch size than the baseline.
As noted by [1]} and [2]},
if we had used mini-batches as large as the hardware allows,
Rfa could have achieved a more significant speed gain.
Nonetheless, we control for batch size
even though it is not the most favorable setting for Rfa,
since the conclusion
translates better to common applications
where one generates a single sequence at a time (e.g., instantaneous machine translation).
For the softmax attention baseline, we follow [3]} and cache previously computed query/key/value representations,
which significantly improves its decoding speed (over not caching).
