Solving for equation 2 does not minimize the risk over C4-only, and hence it is unfair to compare \(\theta_{\mathcal{C}4}\) and \(\theta_{\mathbf{WRAP}}\) on the C4. For meaningfully comparing models trained on the C4 and on its synthetic rephrases, we evaluate their generalization capability on 21 different domains of the Pile (Gao et al., 2020). Results for each domain are presented in Figure 2.