To gauge the importance of using a well-aligned reference model in DPO, we perform preliminary experiments of DPO training with the Ultrafeedback dataset [13] on Mistral-7B-OpenOrca [12] and OpenHermes-2.5-Mistral-7B [14] as the SFT base model, owing to their excellent performance and small size. We compare the following reference models: i) the SFT base model itself, same as the conventional DPO setup; ii) SOLAR-0-70B [15], a larger and much more performant model; and iii) Intel-7B-DPO [13], an already aligned reference model. The results are summarized in Tab. 1.