Generally, machine learning experiments are not precisely predictable — complex models trained on complex data typically yield noisy or variable results [1]}, [2]}.For example, [2]} documents strong run-to-run irreproducibility in reinforcement learning on Atari games when only changing the initial random seed. This suggests that differences between algorithms may be difficult to measure rigorously due to such intrinsic noise. Though individual experiments may be unpredictable, the general performance of large generative models tends to exhibit smooth and predictable growth as a function of scale — larger systems tend to do increasingly better on a broad range of tasks. This was first noticed by [4]} who observed that capabilities such as machine translation and speech recognition increased in a smooth, predictable manner as the size of the model increased. Subsequent work formalized and experimentally validated a quantitative relationship between scale (in terms of both model size and training data size) and model generalization error [5]}. Furthermore, [6]} demonstrated that test loss performance on language modeling tasks scales as a predictable function of model size, dataset size, and duration of training. These three factors are like ingredients in a chemical reaction, such that if all are scaled up in tandem, the test loss improves proportionally. However, if there is too little of one ingredient, gains are limited by this ingredient. The trends are remarkably consistent, with only tiny deviations from a simple fit to the dataMore precisely, the relationship is a straight line on a log-log plot, equivalent to a power law., covering dozens of data points and several orders of magnitude (Figure REF ). Subsequent work has shown that similar scaling laws exist in generative models for other modalities (e.g., images, video, math, etc.) [7]}, audition [8]}, transfer from text to programming [9]}, few-shot adaptation of vision models [10]}, and more.
