We compare Smart Bird with the vanilla Transformer model and its several efficient variants based on sparse attention mechanism, including:
(1) Sparse Transformer [1]}, a sparse attention based Transformer model that uses a combination of local attention and stride attention;
(2) Longformer [2]}, a Transformer variant based on sliding window attention and global attention at a few positions;
(3) Big Bird [3]}, a Transformer variant that integrates local attention, global attention and random attention mechanisms.
In addition, on the MIND datasets, we compare two additional SOTA news recommendation methods, i.e., NRMS [4]} and FIM [5]}, to provide benchmarks for the comparison.
The performance of different methods on different datasets are compared in Tables REF , REF  and REF .
We have several observations from the results.
First, the performance of efficient Transformer baselines including Sparse Transformer, Longformer and Big Bird outperform the vanilla Transformer in long document modeling (e.g., classification tasks on Amazon, IMDB and MIND).
This is because the input text length of the vanilla Transformer is limited by the computing resources, and many useful contexts cannot be exploited when using a relatively short text truncation length.
Second, the baseline methods based on sparse attention are inferior to the vanilla Transformer in short sequence modeling (i.e., classification on AG and recommendation on MIND).
This is because these methods cannot fully capture the important interactions between different tokens.
Third, Smart Bird can achieve better performance than other compared methods on all datasets in different tasks.
This is because Smart Bird incorporates learnable sparse attention to better capture token interactions that may be important for context modeling.
These results demonstrate the effectiveness and generality of Smart Bird.
