Footnote 1: There is a rich literature comparing how pretrained models perform on benchmark tasks. Most comparisons are for different model families trained over different data: if LLaMA-70B is better than Mistral-7B, does the gain come from its choice of pretrain data, or the architecture difference, or really the size of the model? Some comparisons are among the same architecture, such as LLaMA-70B scores 63.6% on the world knowledge benchmark while LLaMA-7B scores only 48.9% [33]; does this mean increasing model size by 10x increases its capacity only to 130% = 63.6/48.9? Thus, it is highly important to use a more principled framework to study scaling laws in a controlled setting.