In our work, we tackle two important challenges faced during synthetic data curation in the works of Gunasekar et al. (2023)--generation cost, and data bias--by rephrasing articles on the web. (i) **WRAP** allows for using an open source, and much smaller LLM (1.8B/7B v/s GPT3.5) to rephrase unstructured and poorly phrased documents in different styles, since it does not rely on the LLM as a knowledge bank. (ii) Thanks to the information maintaining nature of rephrasing, we are able to leverage the natural diversity of the web, rather than relying on an LLM for information which may be prone to factual errors, and/or data biases. Our work shows that the "style" alone can result in large gains in downstream performance.