@inproceedings{
huang2024oneshot,
title={One-Shot Safety Alignment for Large Language Models via Optimal Dualization},
author={Xinmeng Huang and Shuo Li and Edgar Dobriban and Osbert Bastani and Hamed Hassani and Dongsheng Ding},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=dA7hUm4css}
}
The growing safety concerns surrounding large language models raise an urgent need to align them with diverse human preferences to simultaneously enhance their helpfulness and safety. A promising approach is to enforce safety constraints through Reinforcement Learning from Human Feedback (RLHF). For such constrained RLHF, typical Lagrangian-based primal-dual policy optimization methods are computationally expensive and often unstable. This paper presents a perspective of dualization that  reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. This shortcut eliminates the need for cumbersome primal-dual policy iterations, greatly reducing the computational burden and improving training stability. Our strategy leads to two practical algorithms in model-based and preference-based settings (MoCAN and PeCAN, respectively). A broad range of experiments demonstrate the effectiveness and merits of our algorithms.
One-Shot Safety Alignment for Large Language Models via Optimal Dualization
Xinmeng Huang Shuo Li Edgar Dobriban Osbert Bastani Hamed Hassani Dongsheng Ding