Training with synthetic data makes evaluation decontamination harder.The use of synthetic data in model training poses significant challenges to fair evaluation. Evaluation benchmarks are often created by referring to public text sources, such as coursework websites or forums. Consequently, it is arguable that all publicly available benchmark test cases might occasionally be included in the pre-training data of LLMs (Gao et al., 2021; Hoffmann et al., 2022). The use of synthetic data exacerbates this issue rather than mitigating it. Although the community has proposed several techniques to detect such evaluation contamination, such as _min-k9 prob_(Shi et al., 2023), which checks the probabilities of \(k\) long-tail tokens, these token-level decontamination methods are inadequate whenthe model is trained with synthetic data. Synthetic data might include rephrased versions of the benchmark data (Mattern et al., 2023; Oren et al., 2023), rendering token-level decontamination ineffective. In addition to developing more advanced evaluation contamination detection techniques, we recommend that model developers invest in creating and maintaining in-house and protected evaluation benchmarks. These proprietary benchmarks should be carefully safeguarded to prevent leakage and ensure the integrity of the evaluation process.