@inproceedings{
bukharin2024robust,
title={Robust Reinforcement Learning from Corrupted Human Feedback},
author={Alexander Bukharin and Ilgee Hong and Haoming Jiang and Zichong Li and Qingru Zhang and Zixuan Zhang and Tuo Zhao},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=cR2QDzdpEv}
}
Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels.  To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an $\ell_1$-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, $R^3M$ can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that $R^3M$ is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R^3M$ improves robustness of the reward  against several types of perturbations to the preference data.
Robust Reinforcement Learning from Corrupted Human Feedback
Alexander Bukharin Ilgee Hong Haoming Jiang Zichong Li Qingru Zhang Zixuan Zhang Tuo Zhao