\(\bullet\)**Pre-trained big models using SSMs architecture.** In the pre-trained big model era, the scaling of deep neural networks is an important step for general artificial intelligence. Current big models are built based on CNN or Transformer networks, and seldom of them adopt the SSMs architecture. Recently, Jamba [136] released by AI21Labs is a novel large language model built by fusing the Transformer, Mamba, and MoE (Mixture-of-Experts). It supports the input of context length up to 256K tokens and also achieves comparable performance with Mixtral-8x7B [266] and Llama-2 7OB [225]. The study on building pure Mamba or hybrid architectures will be a promising direction for pre-trained big models.