\(\bullet\)_Data collection_. It is suggested to include diverse data sources in the pre-training data. Although Falcon [141] shows that webpages alone can be employed to train powerful LLMs, a more typical approach is to also incorporate diverse high-quality text like code, books, scientific papers, _etc._ If a LLM is specialized with a certain skill, the proportion of corresponding data source should be increased accordingly. For example, Gopher [64] and Chinchilla [34] are trained with approximately 40% of data from books. PaLM [44] and LaMDA [68] use approximately 50% conversational data.