In a realistic scenario, tools like LLaMA2 can rewrite pretrain data like we did in \(\mathsf{bioR}\). Rewriting data 40 times can produce 40 distinct English paragraphs, sometimes with (different) hallucinated contents. Does this require the model to be 40x larger? No, our comparison between \(\mathsf{bioS}\) and \(\mathsf{bioR}\) shows that, if trained for the same duration (40 rewrites each for 25 passes), the model's capacity ratio remains nearly the same, slightly lower due to irrelevant data introduced by LLaMA2.