Reinforcement learning from human feedback (RLHF)  aligns large language models (LLMs)  by encouraging their generations to have high rewards, using a reward model trained on human preferences.
To prevent the forgetting of pre-trained knowledge, RLHF usually incorporates aregularization; this forces the policy to remain close to its supervised fine-tuned initialization, though it hinders the reward optimization.
To tackle the trade-off betweenand reward, in this paper we introduce a novel alignment strategy named Weight Averaged Rewarded Policies (WARP) .WARPmerges policies in the weight space at three distinct stages.
First, it uses the exponential moving average of the policy as a dynamic anchor in theregularization.
Second, it applies spherical interpolation to merge independently fine-tuned policies into a new enhanced one.
Third, it linearly interpolates between this merged model and the initialization, to recover features from pre-training.
This procedure is then applied iteratively, with each iteration’s final model used as an advanced initialization for the next, progressively refining the-reward Pareto front, achieving superior rewards at fixed.
Experiments with Gemma policies validate thatWARPimproves their quality and alignment, outperforming other open-source LLMs.