DPO is able to bypass both fitting an explicit reward and performing RL to learn the policy using a single maximum likelihood objective. Note the optimization objective Eq. 5 is equivalent to a Bradley-Terry model with a reward parameterization \(r^{*}(x,y)=\beta\log\frac{\pi_{\theta}^{*}(y|x)}{\pi_{\text{ref}}(y|x)}\) and we optimize our parametric model \(\pi_{\theta}\), equivalently to the reward model optimization in Eq. 2 under the change of variables. In this section we will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin with by defining an equivalence relation between reward functions.