\(\bullet\)_Effective reward model training_. Despite that InstructGPT used a small reward model (6B GPT model), increasing work [99] has shown it is often more effective to use a large reward model (_e.g._, equal or greater than the original model size), since large reward models generally perform better in judging the quality of the LLM generated outputs. In LLMaMa 2 [99], pretrained chat model checkpoints are used to initialize the reward model, they argue that such an approach can effectively reduce the information mismatch between the model to be aligned and the reward model by sharing the same pre-training knowledge. Whereas, it is common to encounter the overfitting problem when training large-scale reward models. As a simple yet effective solution, existing work [374, 375] has introduced the LM loss on the preferred response of the input prompt from the human-annotated alignment dataset as a regularizer, which alleviates the overfitting of the reward model on the binary classification task. In addition, as there are multiple criteria for alignment (_e.g._, helpfulness and honesty), it is often difficult to train a single reward model that can satisfy all the alignment criteria. Therefore, it is useful to train multiple reward models that focus on different alignment criteria [99], and compute the final reward based on the produced ones from them via special combination strategies (_e.g._, mean pooling and weighted sum). Such a way enables more flexible rules or standards on multiple criteria, _e.g._, relaxing the requirement on helpfulness while posing more strict limits on harmfulness.