where \(c\), \(k\), and \(\alpha\) are parameters to fit and \(x\) can be model sizes, numbers of training data, training steps2, and the amount of computation. Previous experience (Alabdulmohsin et al., 2022; OpenAI, 2023; Bi et al., 2024; Su et al., 2024) highlights the impressive predictability of scaling laws. Specifically, Eqn. 3 fitted on a collection of small models, training data, or computation can extrapolate to precisely predict the test loss of larger cases over orders of magnitudes. This enables practitioners to estimate the performance of a pretrained large language model without actually finishing the expensive runs. Recent development further shows various functional relationships between the performance of language models and a broader range of factors, including transfer learning (Hernandez et al., 2021), sparse architectures (Frantar et al., 2023), and repeated data (Muennighoff et al., 2024), consolidating the predictability of language model performance.