**Model Architecture.** Due to the scalability and effectiveness, Transformer has become the de facto architecture for building LLMs. Various strategies have been proposed to improve the performance of this architecture, such as neural network configuration and scalable parallel training (see discussions in Section 4.2.2). However, Transformer still suffers from high training costs and slow inference rates. More efforts [251, 252] are still in need to develop improved model architectures for large-scale pre-training. Specially, system-level or hardware-level optimization (_e.g._, FlashAttention [284]) is worth more exploration to improve the efficiency of Transformer architectures. In addition, as an important basic capacity, existing LLMs typically maintain a long context window. For example, the most recent GPT-4 Turbo enables a long context of 128K tokens, and Claude 2.1 also supports the input up to 200K tokens. Although many efforts have been made to enhance the long context modeling ability of LLMs [264, 291], the resulting models still can't well process the information in the context window [299]. To address this issue, specific architecture adaptations or algorithms might be needed to enhance the modeling and utilization of long context information. Another worrying concern is that existing work mostly focuses on training LLMs with decoder-only Transformers. Despite the effectiveness, it severely limits the more wide, diverse explorations on alternative model architectures.