Generating Synthetic DataNow, we detail how we utilize an instruction-tuned language model to rephrase texts from web-crawled datasets such as C4 (Raffel et al., 2020) (which we use for all our experiments). In particular, we use a frozen Mistral-7B instruction-tuned model (Jiang et al., 2023) (see Ablations in Section 6 for other models). To generate synthetic data in "medium" style, the Mistral model is prompted using the following instruction: "_For the following paragraph give me a paraphrase of the same in high-quality English language as in sentences on Wikipedia"_. The prompt was created using iterative human feedback by comparing outputs of "medium" sized LLMs with those of GPT-4. We use the model output to create a parallel corpus of "high-quality" synthetic data corresponding to the original noisy web data. Each example has a maximum of 300 tokens, which was decided based on our empirical observation that asking an LLM to rephrase more than 300 tokens, often led to loss of information. Discussions on data quality can be found in Section C.