This is the same objective optimized in prior works [49; 38; 1; 26] using the DPO-equivalent reward for the reward class of \(r_{\phi}\). In this setting, we can interpret the normalization term in \(f(r_{\phi},\pi_{\text{ref}},\beta)\) as the soft value function of the reference policy \(\pi_{\text{ref}}\). While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable. We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using a human completion baseline, essentially a single sample Monte-Carlo estimate of the normalizing term. In contrast the DPO reparameterization yields a reward function that does not require any baselines.