\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \hline Dataset & \(L_{\text{test-regen}}\) & \(L_{\text{test-ref}}\) & \(L_{\text{train}}\) & \(L_{\text{train-regen}}\) & \(\Delta_{1}\) & \(\Delta_{2}\) \\ \hline GSM8K & 0.52 & 0.50 & 0.11 & 0.33 & 0.02 & 0.19 \\ MATH & 0.59 & 0.58 & 0.23 & 0.39 & 0.01 & 0.20 \\ \hline \hline \end{tabular}
\end{table}
Table 6: To validate benchmark data leakage during data generation, we compare LM loss on: 1) a training subset, which is a small subset with 256 samples from all synthetic data; 2) a regenerated training subset, where we maintain the original questions from the training subset and use GPT-4 Turbo to rewrite answers; 3) a regenerated test set, where we keep the questions from the evaluation metrics unchanged and use GPT-4 Turbo to rewrite answers; 4) a reference test set, where we use the test set as seed to generate new questions and answers via GPT-4 Turbo. Referring to Skywork, we also report two key metrics: \(\Delta_{1}=L_{\text{test-regen}}-L_{\text{test-ref}},\Delta_{2}=L_{\text{test- regen}}-L_{\text{train-regen}}\), As \(\Delta_{1}\) is close to 0 and \(\Delta_{2}\) is significantly greater than 0 in two benchmarks, we believe that there is no leakage during the process of data synthesis.