**Model Training.** For pre-training, it is essential to establish a data-centric infrastructure and training procedure for LLM optimization, which can effectively support a systematic process of data collection, data cleaning, data mixture, and data curriculum. Furthermore, it also calls for more flexible mechanisms of hardware support or resource schedule, so as to better organize and utilize the resources in a computing cluster. In practice, it is very challenging to pre-train capable LLMs, due to the huge compute consumption and the sensitivity to data quality and training tricks [78, 93]. Thus, it becomes particularly important to develop systemic, economical pre-training approaches for optimizing LLMs, _e.g._, predictable scaling [46] and proxy model training [59]. More training recipes or principles should be investigated and shared to reduce the potential risk of degradation or failure in large-scale model optimization. Although increasingly more model checkpoints and cleaned datasets have been released, there still lacks reproducible work on pre-training data preparation (_e.g._, detailed cleaning strategies) and data scheduling (_e.g._, data mixture and curriculum). Since it is very costly to pre-train a LLM from scratch, it is important to design suitable mechanisms for continually pre-training or fine-tuning the LLM based on publicly available model checkpoints (_e.g._, LLaMA [57] and Flan-T5 [69]). For this purpose, a number of technical issues have to be resolved, _e.g._, catastrophic forgetting and task specialization. Furthermore, it is also useful to develop effective tuning strategies that effectively inject or edit specific knowledge [672], _e.g._, correcting the outdated facts.