Thus, in most practical scenarios, the reference model is simply set as the base SFT model Tunstall et al. (2023); Intel (2023); Ivison et al. (2023), which is a much weaker alternative with potentially misaligned preferences. This reference model acts as _a lower bound_ in DPO, _i.e.,_ the target model is optimized to be at least as aligned as the reference model. Thus, we argue that a reference model that is already more aligned will serve as a better lower bound for DPO training, which would be beneficial for the alignment tuning. One option would be to utilize the plethora of open source models Tunstall et al. (2023); Ivison et al. (2023) that have already undergone alignment tuning.