We can apply this reparameterization to the ground-truth reward \(r^{*}\) and corresponding optimal model \(\pi^{*}\). Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e., \(p^{*}(y_{1}\succ y_{2}\mid x)=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2}))\). Substituting the reparameterization in Eq. 5 for \(r^{*}(x,y)\) into the preference model Eq. 1, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy \(\pi^{*}\) and reference policy \(\pi_{\text{ref}}\).