1. The model follows the standard Transformer architecture [17] without obvious bottlenecks 4, which has been the de facto architecture in large language models such as GPT [1], Llama [13] and Gemini [14]23 Footnote 4: _e.g._, a limited amount of attention heads or inadequate layer normalization, which we find to make destabilize the estimation process of scaling laws. 2. We assume access to an extensive set of training data and the training process never re-uses the data. This reflects the typical scenario in model pre-training since the proliferation of online platforms and digital content has contributed to the abundance of available data
3. The training data is uniformly distributed across the training steps. This is also a reasonable assumption since the training data is often randomly shuffled in the pre-training stage, unless special tricks such as curriculum learning [1] is applied.