Instead of attending to the heuristically or randomly selected tokens, attending to those potentially important tokens may help build higher-quality sparse attention for text modeling.
Fortunately, these potentially important tokens can be efficiently and effectively identified by a tiny Transformer model with very low dimensions.
Fig. REF  show the attention heatmaps learned by a tiny Transformer with 4 hidden dimensions and a standard 256-dim Transformer.
We find the attention heatmap produced by the low-dimensional Transformer is very similar to the heatmap learned by a Transformer with a much larger size.
In fact, on the Amazon dataset [1]}, the average Pearson correlation coefficient between the attention matrices learned by Transformers with 256-dim or 4-dim attention heads is 0.67, while the correlation coefficient between attention matrices learned by the same Transformer in two independent experiments with different random seeds is 0.76.
Thus, the attention heatmaps learned by the low-dimensional Transformer have the potential to guide the sparse attention mechanism by providing useful clues on recognizing important interactions among tokens to empower context modeling.
