Neural Scaling Laws for Language ModelsNeural scaling laws relate the optimal number of model parameters and amount of training data for a fixed amount of compute. Hoffmann et al. (2022) presented the Chinchilla scaling laws for language models demonstrating that there was a linear relationship between the size of the model and the amount of training data needed. Their findings indicated that prior models such as Gopher (Rae et al., 2021) are severely undertrained. Recently, models such as Llama (Touvron et al., 2023) are trained with much more data. These scaling laws were drawn for the paradigm of single-epoch training. Recently, Muennighoff et al. (2023) showed that the marginal utility of repeated data rapidly diminishes when training for more than 4 epochs, and formulated scaling laws under repeated data. Concurrently, Xue et al. (2023) showed that repeating even small fractions of the pre-training data can lead to overfitting and reduce model performance.