RQ3: How important is to have a high-quality re-phraser?To answer this, we use data from four distinct re-phrasing models (T5-base (Raffel et al., 2020), Owen-L8B-chat (Bai et al., 2023a), Mistral-7B-chat (Jiang et al., 2023), and Vicuna-13B-chat-v1.3 (Chiang et al., 2023)) and train a 345M model for 30B tokens. We generate data from all models using the same prompt. In case of the T5-base model, we finetune the model for 1 epoch on re-phrase pairs from the Vicuna-13b-chat model. We find that pre-training on data generated by smaller re-phrase models like Qwen-1.8B and Mistral-7B achieve lower perplexity than Vicuna 13B (Figure 5). At the same time, our fine-tuned T5-base model performs significantly worse than the rest. Even then, all rephrase models reduce perplexity over only real C4 data. It remains an open question to test the limits of how small can we train a paraphrase model that can generate high quality synthetic data to further scale the applicability of **WRAP**.