**Theorem 1**.: _When \(\pi_{\text{ref}}\) is defined as \(\pi_{w}\), an ideal policy that precisely aligns with the true data distribution of preferred data, the DPO loss \(\mathcal{L}(\pi_{\theta};\pi_{w})+C\) is upper bounded by \(\mathcal{L}(\pi_{\theta};U)\), where \(C\) is a constant._