**Language Modeling.** As the most fundamental ability of LLMs, _language modeling_ aims to predict the next token based on the previous tokens [1], which mainly focuses on the capacity of basic language understanding and generation. For evaluating such an ability, typical language modeling datasets that existing work uses include Penn Treebank [538], WikiText-103 [539], and the Pile [161], where the metric of _perplexity_ is commonly used for evaluating the model performance under the zero-shot setting. Empirical studies [93, 55] show that LLMs bring substantial performance gains over the previous state-of-the-art methods on these evaluation datasets. To better test the modeling capacity of long-range dependencies in text, the LAMBADA dataset [233] has been introduced, where LLMs are required to predict the last word of sentences based on a paragraph of context. Then, the accuracy and perplexity of the predicted last words are employed to evaluate LLMs. As shown in existing work, the performance on the language modeling tasks typically follows the scaling law [30], which means that scaling language models would improve the accuracy and reduce the perplexity.