To handle the issues of only local relations captured by CNN/RNN/GNN models, the Transformer [13] proposed in the year 2017 learns the long-range feature representations well. The core operation is the self-attention mechanism which transforms the input tokens into query, key, and value features, and outputs the long-range features by multiplying the similarity matrix (obtained via product between the query and key features) with the value features. The Transformer architecture first swept the NLP community with the help of _pre-training and fine-tuning_ paradigm [14], such as BERT [15], ERNIE [16], BART [17], GPT [18]. Then, other communities are also boosted with such networks, for example, the ViT [19] and Swin-Transformer [20] released in computer vision. Many researchers also exploit the hybrid network architectures by combining Transformer and other networks, or adapting the Transformer for multi-modal research problems [21, 22]. In the current stage, large foundation models are emerging, and Parameter-Efficient Fine-Tuning (PEFT) strategies [23] also have been greatly developed. However, the current Transformer-based models still require high-end graphics cards with larger memory for training and testing/deployment, which greatly limits their wider application.