\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model size** & **1B** & & **7B** \\
**Dataset** & The Pile & RW & RW \\ \hline wiki-bpb \(\downarrow\) & 0.64 & 0.66 & 0.60 \\ \hline \hline \end{tabular}
\end{table}
Table 9: **Models trained on RefinedWeb achieve performance close to models trained on The Pile on Wikitext, despite not having seen any content from Wikipedia.