Recent studies have proposed a mixture of synthetic data and real human data to train more robust reward models (Gao et al., 2023). Constitutional AI (Bai et al., 2022) proposes to use a small set of principles to steer the AI generated critiques and feedback, and use such synthetic data to replace the real human data in the typical RLHF pipeline. The model trained with this RLAIF (i.e., reinforcement learning from AI feedback) method shows similar strong performance as RLHF baselines. In general, synthetic data offers a powerful solution for human values and preferences alignment by allowing researchers to generate large-scale, diverse, and controlled training datasets in a low-cost way (Cui et al., 2023; Ganguli et al., 2022). By simulating a wide range of scenarios involving ethical dilemmas (Perez et al., 2022), social interactions (Liu et al., 2023c), and cultural norms (Ziems et al., 2023), synthetic data enables comprehensive and systematic testing of AI models' alignment with human values (Askell et al., 2021). This approach helps identify and mitigate issues related to bias (Liu et al., 2021; Ntoutsi et al., 2020), fairness (Landers and Behrend, 2023; Zhao et al., 2018), and unintended consequences before AI systems are deployed in real-world settings (Ye et al., 2024).