Footnote 31: In RLHF, it seems to be also important that reward models should be aware of the knowledge or ability of a LLM to be aligned. For example, LLAMA 2 adopts pre-trained chat model checkpoints to initialize reward models [99].