\(\gamma_{\pi_{\theta}}>\gamma_{\pi_{ref}}\) (on average). Thus, \(\gamma_{\pi_{ref}}\) can be understood as a lower bound defined by the reference model, of which the target model is trained such that \(\gamma_{\pi_{\theta}}>\gamma_{\pi_{ref}}\). In sDPO, \(\gamma_{\pi_{ref}}\) increases as the steps progress because the reference model that defines it is more and more aligned. Hence, \(\gamma_{\pi_{ref}}\) becomes a stricter lower bound as the steps pass, inducing a curriculum learning from easy to hard optimization tasks.