tasks. However, when evaluated on Pile perplexity, we observe significant degradation in perplexity across many sub-domains in Figure 3. This is likely because synthetic data is very clean containing few special characters and being highly structured. In contrast several sub-domains of the Pile such as OWT, and Hackernews have such special tokens. On domains such as Philipapers and Gutenberg, we observe that dropping real C4 text from the pre-training data, and training on synthetic documents alone drops performance significantly (increase in perplexity). This is once again attributed to the fact that synthetic data does not contain certain 'tags' and'styles' that are prevalent in real data scrapes, and emphasized how **WRAP** is a better strategy than pre-training on synthetic data alone. In terms of performance on zero-shot tasks, we once again note that the presence of real data helps improve zero-shot performance in Tables 3,4. Since zero-shot tasks contain well-written Q/A pairs, this effect is not as evident as that for perplexities on real data.