The contrastive preference optimization is originally defined as minimizing \(\mathcal{L}(\pi_{\theta};U)\) under the constraint of minimizing the difference between preferred data distribution and outputs of the learnable policy: