where \(D\) is the preference dataset, \(x\) is the question, \(y_{w}\) and \(y_{l}\) are the chosen and rejected answers respectively, \(\theta\) is the learnable parameters of the model, and \(\gamma_{\pi}(x,y_{w},y_{l})=\log\frac{\pi(y_{w}|x)}{\pi(y_{l}|x)}\), _i.e.,_ the logratio of the chosen and rejected samples w.r.t. the policy \(\pi\).