Kahneman & Tversky’sprospect theorytells us that humans perceive random variables in a biased but well-defined manner; for example, humans are famously loss-averse.
We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases—the success of these objectives (e.g., DPO)  over cross-entropy minimization can partly be ascribed to them beinghuman-aware loss functions(HALOs) .
However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature.
Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do.
We call this approach Kahneman-Tversky Optimization (KTO) , and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B.
Crucially, KTO does not need preferences—only a binary signal of whether an output is desirable or undesirable for a given input.
This makes it far easier to use in the real world, where preference data is scarce and expensive.