This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply \(\pi_{\theta}\). Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terrymodel, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution [4]. In Section 5, we further discuss theoretical properties of DPO in relation to other works.