This work proposes random feature
attention (Rfa),
an efficient attention variant
that scales linearly in sequence length
in terms of time and space,
and achieves practical gains for both long and moderate length sequences.
Rfa builds on a kernel perspective of softmax [1]}.
Using the well-established random feature maps ([2]}, [3]}; §),
Rfa approximates the dot-then-exponentiate function
with a kernel trick [4]}: \({\exp ({\mathbf {x}}\cdot {\mathbf {y}})\approx \phi ({\mathbf {x}})\cdot \phi ({\mathbf {y}})}\) .
Inspired by its connections to
gated recurrent neural networks [5]}, [6]}
and fast weights [7]},
we further augment Rfa with an optional gating mechanism,
offering a straightforward way of learning with recency bias
when locality is desired.
