Synthetic data might cause ambiguity in AI alignment.The increasing use of synthetic data in aligning AI models (e.g., Constitutional AI (Bai et al., 2022)) can introduce significant ambiguity and uncertainty. The goal of AI alignment is to ensure that AI systems behave in ways that are aligned with human values and intentions. However, synthetic data, which is artificially generated rather than collected from real-world sources, may not accurately represent the nuances and complexities of human values and preferences (Zhou et al., 2024). This discrepancy can lead to AI models learning from data that is biased (Feng et al., 2023; Liu et al., 2021), ungrounded (Liu et al., 2022; Patel and Pavlick, 2022), or misrepresentative of real-world scenarios (Ji et al., 2023; Weidinger et al., 2021). As a result, AI systems trained on synthetic data may exhibit behaviors that are misaligned with human expectations, potentially leading to unintended consequences or even harmful actions (Anderljung et al., 2023; Zou et al., 2023). Moreover, the ambiguity introduced by synthetic data can make it challenging to interpret and understand the decision-making processes of AI models (Lightman et al., 2023), further complicating the task of ensuring alignment. To mitigate these risks, it is crucial for researchers to carefully consider the limitations and potential drawbacks of using synthetic data in alignment research and to develop robust methods for validating and testing AI models trained on such data.