Assuming access to a static dataset of comparisons \(\mathcal{D}=\left\{x^{(}),y_{w}^{(i)},y_{l}^{(i)}\right\}_{i=1}^{N}\) sampled from \(p^{*}\), we can parametrize a reward model \(r_{\phi}(x,y)\) and estimate the parameters via maximum likelihood.