However, it is important to acknowledge that low-fidelity synthetic human preference data might be limited in accurately reflecting nuanced human judgment (Argyle et al., 2023). Consequently, the resulting models may be less robust under "jail-breaking attacks" (Deshpande et al., 2023; Huang et al., 2023a), and may reveal strategically deceptive behavior even through safety training (Everitt et al., 2021; Pan et al., 2022; Steinhardt, 2022). To mitigate these risks, researchers must continuously refine and improve the quality and diversity of synthetic data, incorporating more complex and comprehensive scenarios that better capture the intricacies of human values and preferences. Additionally, combining synthetic data with real-world data, and creating synthetic data in an interactive environment which can be synced with the real world, are promising remedies. As the need for effective AI governance and regulation grows, synthetic data will play an increasingly vital role in enabling scalable oversight mechanisms that promote trust, accountability, and the development of AI technologies that are aligned with human values and societal expectations.