Sparse attention patterns.
The idea behind these methods
is to limit
the reception field of attention computation.
It motivates earlier attempts in
improving attention's efficiency,
and still receives lots of interest.
The sparse patterns can be set a priori [1]}, [2]}, [3]}, [4]}
or learned from data [5]}, [6]}.
For most of these approaches,
it is yet to be empirically verified that
they are suitable for large-scale sequence-to-sequence learning;
few of them have recorded decoding speed benefits.
