Using **WRAP** on the C4, we evaluate model performance on 13 different zero-shot tasks, and 21 different language modeling domains of the Pile, and find that pre-training LLMs with synthetic data allows us to train equivalent models with 5x lesser data, or 3x lesser compute. In fact, our synthetic data trained models, also outperform the recent TinyLLama models that were trained for 3 trillion tokens (10x data and compute) across several zero-shot Q/A tasks. We further observe a reduction in perplexity by \(\sim\) 50% on the Pile, and note that our 350M parameter model trained on combinations of real and synthetic rephrases on just 15% of the entire C4 corpus, outperforms pre-training a 1.3B parameter on the entire C4. Finally, we conduct an analysis on the potential of data leakage, properties of synthetic data styles, and how to combine synthetic data for improving **WRAP** based LLM pre-training.