Figure 6: **Is re-phrasing same as any augmentation?** We compare perplexity on the Pile for different augmemntation strategies. 350M parameter models are trained for a total of 15B tokens. **WRAP** (Medium + C4) performs significantly better than traditional augmentations.