\(\bullet\)_Effective RL training_. As the RL training process tends to be unstable and hyper-parameter sensitive, it is suggested that the language model should be well supervised fine-tuned before RL training, so as to reaching a good model capacity. A commonly-used way is to fine-tune the LLM on its best outputs of the prompts (referred to as _rejection sampling_ or _best-of-\(N\)_) from the alignment dataset until convergence before RL. Given a prompt, the LLM would first produce \(N\) outputs via the sampling algorithm, and then the best candidate from the model will be selected by the reward model for learning. After fine-tuning the LLM on the best samples until convergence, the RL process will be performed to further improve the performance. LLMaA 2 [99] has successively trained five versions of RLHF models, where the LLM has been progressively improved with the improvement of the reward models. In this way, the collected prompts and annotations of human preference data can better reflect the issues of the current model checkpoint, thus making special tuning to address these issues. In addition, LLMaA 2 also adds samples from prior iterations into the subsequent ones, to alleviate the possible capacity regression issue during iterative optimization.