**CPO Loss Components** The CPO loss function consists of two components: \(\mathcal{L}_{\text{prefer}}\) for preference learning, and \(\mathcal{L}_{\text{NLL}}\), which ensures the model does not deviate significantly from the preferred data distribution. To illustrate the significance of each term, we re-train the model exclusively with one of the components. It is important to note that training solely with \(\mathcal{L}_{\text{NLL}}\) equates to the baseline scenario of SFT on preferred data. As depicted in the left of Figure 4, the inclusion of both terms yields the optimal performance, while the absence of either leads to a decrease in performance.