A few datapoints stand out. There is a column of points with training FLOP around \(10^{19}\) where some of the larger models obtain high loss values (up to 70% higher compared to models with similar compute budgets). These correspond to the lowest ratio of training tokens to parameters (the top five have a ratio \(<0.4\)). It is unclear why these training runs yielded such high losses. In our analysis, we drop the five points in this column. In Appendix A.1, we show that these would not materially change our conclusions about the poor fit of Hoffmann et al.'s estimated Chinchilla scaling law.