While scaling laws focus on the pre-training loss, the scaling effect on the performance of downstream tasks has thus far less studied. Emergent abilities [47] are defined as abilities that present in larger LMs but not present in smaller one. The existence of such abilities is recently challenged for two reasons. First, small LMs trained on a sufficient number of tokens can outperform large models on tasks with claimed emergent abilities [44; 45; 19]. For example, LLaMA-13B with less compute [44] can outperform GPT-3 (175B) on MMLU [14]. Second, [37] claim that emergent abilities appear due to the nonlinear or discontinuous metrics selected to evaluate certain datasets, rather than from a fundamental change in larger models.