@inproceedings{
zhang2024mitigating,
title={Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation},
author={Xiaoying Zhang and Jean-Francois Ton and Wei Shen and Hongning Wang and Yang Liu},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=kYio3xH6eb}
}
Reinforcement Learning from Human Feedback (RLHF) has been pivotal in aligning Large Language Models with human values but often suffers from overoptimization due to its reliance on a proxy reward model. To mitigate this limitation, we first propose a lightweight uncertainty quantification method that assesses the reliability of the proxy reward using only the last layer embeddings of the reward model. Enabled by this efficient uncertainty quantification method, we formulate AdvPO, a distributionally robust optimization procedure to tackle the reward overoptimization problem in RLHF. Through extensive experiments on the Anthropic HH and TL;DR summarization datasets, we verify the effectiveness of AdvPO in mitigating the overoptimization problem, resulting in enhanced RLHF performance as evaluated through human-assisted evaluation.
Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation
Xiaoying Zhang Jean-Francois Ton Wei Shen Hongning Wang Yang Liu