Compressed context.
[1]} compress the
context along the timesteps
so that the effective sequence length for attention
computation is reduced.
Another line of work aims to store
past context into a memory module with limited size [2]}, [3]}, [4]},
so that accessing longer history only moderately increases the overhead.
Reminiscent of RNN language models,
Rfa attends beyond a fixed context window
through a stateful computation, without increasing
time or memory overhead.
