**Models with Tens of Billions of Parameters**. Most of the models in this category have a parameter scale ranging from 10B to 20B, except LLaMA [57] and LLaMA2 [99] (containing 70B parameters in the largest version), NLLB [91] (containing 54.5B parameters in the largest version), and Falcon [135] (containing 40B parameters in the largest version). Other models within this range include mT5 [83], PanGu-\(\alpha\)[84], T0 [28], GPT-NeoX-20B [87], CodeGen [86], UL2 [89], Flan-T5 [69], and mT0 [94]. Among them, Flan-T5 (11B version) can serve as a premier model for research on instruction tuning, since it explores the instruction tuning from three aspects [69]: increasing the number of tasks, scaling the model size, and fine-tuning with chain-of-thought prompting data. Besides, CodeGen (11B version), as an autoregressive language model designed for generating code, can be considered as a good candidate for exploring the code generation ability. It also introduces a new benchmark MTPB [86] specially for multi-turn program synthesis, which is composed by 115 expert-generated problems. To solve these problems, it requires LLMs to acquire sufficient programming knowledge (_e.g.,_ math, array operations, and algorithms). More recently, CodeGen2 [97] has been released to explore the impact of choices in model architecture, learning algorithms, and data distributions on the model. As another LLM specialized in coding abilities, StarCoder [98] has also achieved excellent results. As for multilingual tasks, mT0 (13B version) might be a good candidate model, which has been fine-tuned on multilingual tasks with multilingual prompts. Furthermore, PanGu-\(\alpha\)[84] shows good performance in Chinese downstream tasks in zero-shot or few-shot settings, which is developed based on the deep learning framework MindSpore [136]. Note that PanGu-\(\alpha\)[84] holds multiple versions of models (up to 200B parameters), while the largest public version has 13B parameters. As a popular LLM, LLaMA (65B version) [57], which contains approximately five times as many parameters as other models, has exhibited superior performance in tasks related to instruction following. Compared to LLaMA, LLaMA2 [99] has made more explorations in reinforcement learning from human feedback (RLHF) and developed a chat-oriented version called _LLaMA-chat_, which generally outperforms existing open-source models across a range of helpfulness and safety benchmarks. Due to the openness and effectiveness, LLaMA has attracted significant attention from the research community, and many efforts [137, 138, 139, 140] have been devoted to fine-tuning or continually pre-training its different model versions for implementing new models or tools. More recently, Falcon [135], as another open-source LLM, has also achieved very excellent performance on open benchmarks. It is featured by a more careful data cleaning process to prepare the pre-training data (with a publicly shared dataset _RefinedWeb [141]_). Typically, pre-training models at this scale require hundreds or even thousands of GPUs or TPUs. For instance, GPT-NeoX-20B uses 12 supermicro servers, each equipped with 8 NVIDIA A100-SXM4-40GB GPUs, while LLaMA utilizes 2,048 A100-80G GPUs as reported in their original publications. To accurately estimate the computation resources needed, it is suggested to use the metrics measuring the number of involved computations such as _FLOPS_ (_i.e.,_ FLoating point number Operations Per Second) [30].