@inproceedings{
eisenstein2024reward,
title={Reward Model Underspecification in Language Model Alignment},
author={Jacob Eisenstein and Jonathan Berant and Chirag Nagpal and Alekh Agarwal and Ahmad Beirami and Alexander Nicholas D'Amour and Krishnamurthy Dj Dvijotham and Katherine A Heller and Stephen Robert Pfohl and Deepak Ramachandran},
booktitle={NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models},
year={2024},
url={https://openreview.net/forum?id=ecufIfDNn0}
}
Reward models play a key role in aligning language model applications towards human preferences. However, this setup can create a dynamic in which the policy model has the incentive to exploit errors in the reward model to achieve high reward. This means that the success of reward-based alignment depends on the ability of reward models to transfer to new distributions created by the aligned policy model. We show that reward models are \emph{underspecified}, in the sense that models that perform similarly in-distribution can yield very different rewards on policy model outputs. These differences propagate to the aligned policies, which we show to be heavily influenced by the random seed used during \emph{pretraining} of the reward model. We show that even a simple alignment strategy --- best-of-$n$ reranking --- creates a semi-adversarial dynamic between the policy and reward models, promoting outputs on which the reward models are more likely to disagree. Finally, we show that a simple ensembling strategy can help to address this issue.
Reward Model Underspecification in Language Model Alignment
Jacob Eisenstein Jonathan Berant Chirag Nagpal Alekh Agarwal Ahmad Beirami Alexander Nicholas D'Amour Krishnamurthy Dj Dvijotham Katherine A Heller Stephen Robert Pfohl Deepak Ramachandran