**Books.** BookCorpus [153] is a commonly used dataset in previous small-scale models (_e.g._, GPT [122] and GPT-2 [26]), consisting of over 11,000 books covering a wide range of topics and genres (_e.g._, novels and biographies). Another large-scale book corpus is Project Gutenberg [154], consisting of over 70,000 literary books including novels, essays, poetry, drama, history, science, philosophy, and other types of works in the public domain. It is currently one of the largest open-source book collections, which is used in training of MT-NLG [113] and LLaMA [57]. As for Books1 [55] and Books2 [55] used in GPT-3 [55], they are much larger than BookCorpus but have not been publicly released so far.