\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multicolumn{3}{c}{**Massive web datasets**} & \multicolumn{2}{c}{**Curated**} & **Ours** & \\ \hline  & OSCAR-21.09 & OSCAR-22.01 & C4 & \(\blacktriangledown\) The Pile & RW-Raw & RW-Filtered & \(\blacklozenge\) RefinedWeb \\ \hline
**1B@27GT** & 55.0\% & 52.7\% & 55.7\% & 53.4\% & 52.7\% & 54.3\% & **56.2\%** \\
**3B@60GT** & 59.1\% & 55.9\% & 59.6\% & 57.9\% & 57.4\% & 58.2\% & **59.8\%** \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Curation is not a silver bullet for zero-shot generalization: small-scale models trained on \(\blacklozenge\) RefinedWeb outperform models trained on web data (C4, OSCAR), and on curated corpora (\(\blacktriangledown\) The Pile). Average accuracy in zero-shot on the small-agg aggregate. All models trained with identical architectures and pretraining hyperparameters. We find that OSCAR-22.01 underperforms other datasets signficantly, perhaps because deduplication is only optional. C4 is a strong baseline, with OSCAR-21.09 lagging slightly behind, but we find that RefinedWeb outperforms both web datasets and the most popular curated dataset, The Pile. Both filtering and deduplication contribute significantly to improving zero-shot performance.**the BigScience Architecture and Scaling Pile model (Scao et al., 2022), PaLM-8B (Chowdhery et al., 2022), Aleph Alpha Luminous 13B (Aleph Alpha, 2023), the Pythia series (Biderman et al., 2023), and the Cerebras-GPT series (Dey et al., 2023). For GPT-3, we distinguish between results obtained through the API (babbage and curie) with the the EleutherAI LM evaluation harness (Gao et al., 2021) (*), and results reported in their paper, with a different evaluation setup (\(\dagger\)). Note that for PaLM and OPT, results were also obtained with a different evaluation suite (\(\dagger\)), while for other models they were obtained with the evaluation harness as well (*), allowing for more direct comparisons.