Scaling of language modes (LMs) on both model and data sizes has been shown to be effective for improving the performance on a wide range of tasks [33; 3; 16; 5; 53; 44; 28], leading to the widespread adoption of LM applications, e.g., ChatGPT. The success of such scaling is guided by scaling laws [15; 21; 7; 16], which study the predictability of pre-training loss given the model and data sizes.