* First release on March 31, 2023: the initial version. * Update on April 9, 2023: add the affiliation information, revise Figure 3 and Table 1 and clarify the corresponding selection criterion for LLMs, improve the writing, and correct some minor errors. * Update on April 11, 2023: correct the errors for library resources. * Update on April 12, 2023: revise Figure 3 and Table 1, and clarify the release date of LLMs.
* Update on April 16, 2023: add a new Section 2.2 about the technical evolution of GPT-series models. * Update on April 24, 2023: add the discussion about scaling laws and add some explanations about the model sizes for emergent abilities (Section 2.1); add an illustrative figure for the attention patterns for different architectures in Figure 9, and add the detailed formulas in Table 6. * Update on April 25, 2023: revise some copy errors in figures and tables. * Update on April 27, 2023: add efficient tuning in Section 5.3.
* Update on April 28, 2023: revise Section 5.3.
* Update on May 7, 2023: revise Table 1, Table 2, and some minor points. * Update on June 29, 2023 (major revision):
* Section 1: add Figure 1 for the trends of published LLM papers in arXiv;
* Section 2: add Figure 4 for GPT's evolution and the corresponding discussion;
* Section 3: add Figure 5 for LLMA family and the corresponding discussion;
* Section 5: add latest discussion about the synthetic data formatting of instruction tuning in Section 5.1.1, the empirical analysis for instruction tuning in Section 5.1.4, parameter-efficient model adaptation in Section 5.3 and memory-efficient adaptation in Section 5.4;
* Section 6: add latest discussion about the underlying mechanism of LCL 6.2.3, planning for complex task solving in Section 6.4;
* Section 7: update Table 14 for representative datasets for evaluating advanced abilities of LLMs, and empirical ability evaluation in Section 7.4;
* Section 6.1.1: add prompt design;
* Section 8: add the discussions on applications of LLMs in finance and scientific research domains;
* Update on September 10, 2023 (major revision):
* Claim the copyrights of the figures and tables in this paper. * Add latest LLMs, techniques and their descriptions in Section 3, Section 4, Section 5, Section 6 and Section 7;
* Section 4: add latest discussion about the decoding strategy in Section 4.2.5;
* Section 5: add latest discussion about the practical tricks for instruction tuning in Section 5.1.2, the empirical analysis on LLMA (13B) for instruction tuning in Section 5.1.4, practical strategies for RLHF in Section 5.2.3, alignment without RLHF in Section 5.2.4 and remarks on SFT and RLHF in Section 5.2.5;
* Section 6: update the content about the planning for complex task solving in Section 6.4;
* Section 7: add discussions about evaluation approaches in Section 7.3.2, Table 15 for the category of existing evaluation work, and update empirical ability evaluation in Section 7.4 and the results on Table 16;
* Section 6.1.1: add new prompt examples in Table 12;
* Update on November 23, 2023 (this version):
* Section 1: add Figure 2 for the evolution process of four generations of language models;
* Section 2: add more discussion about scaling laws and how emergent abilities relate to scaling laws;
* Section 3: add latest LLMs in Figure 3 and Table 1, latest APIs in Section 3.1, commonly used datasets
for instruction tuning and alignment tuning in Section 3.3, and several libraries in Section 3.4;
* Section 4: add latest discussion about the data scheduling, including data mixtures and data curriculum in Section 4.1.3; add summary of data preparation in Section 4.1.4; add discussion about modeling long context in Section 4.2.4; add discussion about decoding efficiency issues and add latest decoding strategies in Section 4.2.5;
* Section 5: add latest discussion about instance construction and tuning strategies in Section 5.1; add latest discussion about process-supervised RLHF in Section 5.2.3, and the empirical study on quantized LLaMA models (7B and 13B) in Section 5.4.3;
* Section 6: add latest discussion about prompt optimization in Section 6.1.2, and update the content about chain-of-thought prompting in Section 6.3;
* Section 8: add latest discussion about LLM for research directions in Section 8.1;
* Section 9: revise the content in the several aspects.