RetNet [127], Mega [128], H3 [129]. Specifically, Acceptance Weighted Key Value (short for RWKV) [125] is a kind of RNN architecture and is developed based on attention-free Transformer [130] for natural language processing. It simultaneously features in the efficient parallelizable training of transformers and the efficient inference of RNNs. Duan et al. further adapt this framework to the computer vision tasks and propose the Vision-RWKV (VRWKV) [126] model. Their results demonstrate that it beats the ViT in the image classification task and also has significant advantages in speed, and memory usage (when processing the high-resolution inputs). The RWKV architecture is also widely used in many other tasks, such as time series related task [131], online action detection [132], diffusion model [133]. RetNet [127] is short for Retentive Network, which also targets building a large language model that achieves training parallelism, low-cost inference, and high performance, simultaneously. It supports parallel, recurrent, and chunkwise recurrent computation paradigms.