Many language models are supervised finetuned to learn how to follow instructions, but in learning this behavior, they may inadvertently also learn to be _sycophantic_(Perez et al., 2023), tailoring their responses to follow a user's viewpoint, even if that viewpoint is not objectively correct (Wei et al., 2023b). Sharma et al. (2024) find evidence that the preference models (i.e., the reward model used for RLHF training) and even humans prefer sycophantic responses sometimes. On this front, Wei et al. (2023b) generates synthetic data to encourage models to be robust to user opinions and adds these data in a finetuning step to reduce sycophantic behavior on held-out prompts.