In our experiments, we use Glove [1]} embeddings to initialize the token embeddings of the high-dimensional Transformer, and we use principal component analysis (PCA) to align with the dimension of the tiny Transformer.
The hidden dimensions of the tiny Transformer and the standard sparse attention are 4 and 256, respectively.
We use 2 layers for all models in classification tasks and 4 layers for both encoder and decoder in summarization tasks.
In the news recommendation task, following many prior works [2]} we use news titles to model news content.
We use different Transformers to replace the word-level and news-level self-attention networks in NRMS [3]}.
The text truncation length is 30 and the user clicked news sequence truncation length is 50.
On the AG dataset, the input truncation length is set to 256.
On the other datasets with longer text lengths, the truncation length of the vanilla Transformer and other efficient variants are 512 and 4096, respectively.The maximum text length of Transformer is limited by the GPU memory.
The sampling token number \(K\)  is set to 20.
We use Adam [4]} as the model optimizer with a learning rate of 1e-4.
Since the classes in some datasets (e.g., Amazon) are imbalanced, We use both accuracy and macro-Fscore as the metrics.
On the news recommendation task, we use AUC, MRR, nDCG@5 and nDCG@10 to evaluate recommendation performance.
On the text summarization tasks, we use the ROUGE-1, ROUGE-2 and ROUGE-L scores (shorten as R-1, R-2 and R-L) as metrics.
We report the average scores of 5 independent experiments.
<TABLE><TABLE><TABLE><TABLE>