where \(\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\) is the reward implicitly defined by the language model \(\pi_{\theta}\) and reference model \(\pi_{\text{ref}}\) (more in Section 5). Intuitively, the gradient of the loss function \(\mathcal{L}_{\text{DPO}}\) increases the likelihood of the preferred completions \(y_{w}\) and decreases the likelihood of dispreferred completions \(y_{l}\). Importantly, the examples are weighed by how much higher the implicit reward model \(\hat{r}_{\theta}\) rates the dispreferred completions, scaled by \(\beta\), i.e, how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naive version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3).