Several recent studies have explored the optimization of human preference without relying on learning a reward function. For instance, Direct Preference Optimization (DPO) Rafailov et al. (2023) refines the policy through a loss function constructed using the Bradley-Terry reward model. Identity Policy Optimization (IPO) Azar et al. (2023) suggests a direct optimization of pairwise human preferences using preference data. Unlike DPO, IPO does not assume a reward model. Kahneman-Tversky Optimization (KTO) Ethayarajh et al. (2024) proposes utilizing solely whether a given output is desirable or undesirable for a given input to align the model with human preferences.