As the table shows, using Intel-7B-DPO as the reference model results in the best performance, even better than using SOLAR-0-70B, which is a much larger model that was trained with more data. Thus, whether the reference model is pre-aligned or not plays an important role in the resulting aligned model's performance. Unfortunately, it is not always possible to simply use a open sourced pre-aligned model as the reference model due to technical and safety concerns, _i.e.,_ such a model may not exist yet or can be susceptible to various domain-specific harmfulness and fairness criteria. To remedy the above, we propose sDPO, which uses more aligned reference models as a part of the training framework.