Comparisons across models trained and evaluated in different settings are difficult to untangle, as many externalities may influence the 1 987results (e.g., numerical precision of training vs inference, prompts used). We distinguish three levels of comparisons: (1) internal comparisons, with models trained and evaluated within our codebase, for which only the pretraining datasets differ; (2) benchmark-level comparisons, with models trained with a different codebase but evaluated with the Eleuther AI harness, taking results from Scao et al. (2022); Black et al. (2022); Aleph Alpha (2023); Dey et al. (2023), thereafter flagged with a \(*\); (3) external comparisons with Brown et al. (2020); Chowdhery et al. (2022), thereafter flagged with a \(\dagger\). For further details on evaluation, see Appendix F.1.