Adopting open source models as reference models could be dangerous.We also show mean \(\gamma_{\pi_{ref}}\) of \(M_{2}\), the aligned model from step 2 of sDPO. Unlike \(S\) and \(M_{1}\), \(M_{2}\) is trained on the Ultrafeedback Cleaned dataset, _i.e.,_\(M_{2}\) is used as a reference model on data that was already used to train it. Note that such a case could happen commonly when adopting various open source models as reference models. This is because the datasets that were used in training those models are often unclear and could overlap with the preference datasets unintentionally. Mean \(\gamma_{\pi_{ref}}\) of \(M_{2}\) is \(84.35\), which is staggeringly higher than either \(S\) or \(M_{1}\). The strikingly high value for \(M_{2}\) likely points to overfitting of \(M_{2}\) to the Ultrafeedback Cleaned dataset. This result highlights the potential danger of merely adopting open source models as reference models instead of using sDPO.