Effectiveness of sDPO in terms of alignment tuning.In Sec. 2.2, we explain that the reference models in sDPO are more aligned, resulting in higher \(\gamma_{\pi_{ref}}\), _i.e.,_ a stricter lower bound. We verify the above empirically in Fig. 2 by comparing the mean \(\gamma_{\pi_{ref}}\) on the Ultrafeedback Cleaned dataset for the reference models in steps 1 and 2 of sDPO, _i.e.,_\(S\) and \(M_{1}\). Note that these two models have not been trained on the aforementioned dataset. Using the SFT base model \(S\) as the reference model, the mean of \(\gamma_{\pi_{ref}}\) is \(-38.60\). On the other hand, using the aligned model \(M_{1}\) from step 1 of sDPO as the reference model, the mean of \(\gamma_{\pi_{ref}}\) is \(-25.10\), an increase of \(13.50\) in _log scale_. Thus, a single step of sDPO greatly increases \(\gamma_{\pi_{ref}}\), which results in a more performant aligned model as seen in Tab. 2.