In this work, we propose **Web** Rephrase **A**ugmented Pre-training (**WRAP**)--that attempts to bridge three important challenges stemming from the ambiguity around data curation--(**i**) what data should you pre-train on? (ii) how can you pre-train with limited data? (iii) how can you pre-train computationally efficiently? In particular, we show that re-phrasing documents on the web using an off-the-shelf medium size LLM allows models to learn much more efficiently than learning from raw text on the web, and accounts for performance gains on out of distribution datasets that _can not_ be offset with additional web data. Our proposed method involves using a pre-trained off-the-shelf LLM to re-phrase documents from a web corpus into different styles. An overview of our approach is shown in Figure 0(a).