Models of human preference for learning reward functions
The utility of reinforcement learning is limited by the alignment of reward
functions with the interests of human stakeholders. One promising method for
alignment is to learn the reward function from human-generated preferences
between pairs of trajectory segments. These human preferences are typically
assumed to be informed solely by partial return, the sum of rewards along each
segment. We find this assumption to be flawed and propose modeling preferences
instead as arising from a different statistic: each segment's regret, a measure
of a segment's deviation from optimal decision-making. Given infinitely many
preferences generated according to regret, we prove that we can identify a
reward function equivalent to the reward function that generated those
preferences. We also prove that the previous partial return model lacks this
identifiability property without preference noise that reveals rewards'
relative proportions, and we empirically show that our proposed regret
preference model outperforms it with finite training data in otherwise the same
setting. Additionally, our proposed regret preference model better predicts
real human preferences and also learns reward functions from these preferences
that lead to policies that are better human-aligned. Overall, this work
establishes that the choice of preference model is impactful, and our proposed
regret preference model provides an improvement upon a core assumption of
recent research.