**Overall Comparison with RL Formulation**. Following the discussion in Section 5.2.3 (the part related to RL training), the text generation problem can be formulated as a decision-making process based on RL. Taking a prompt as input, the task of a LLM is to generate a text completion that appropriately responds to the prompt. This task would be completed step by step. At each step, an agent (_i.e.,_ LLM) will perform an action (_i.e.,_ generating a token) according to the policy (_i.e.,_ the generative probability distribution of LLM) conditioned on the current state (currently generated token sequence and other available context information). It is expected that a high-quality output text would be produced by the LLM, which can earn a large reward score based on the entire response. Overall, RLHF and SFT can be considered as two different training approaches to optimizing the above decision making process for LLMs. Specially, RLHF firstly learns the reward model, and then employs it to improve the LLM with RL training (_e.g.,_ PPO). As a comparison, SFT adopts a teacher-forcing approach, which directly optimizes the likelihood of a demonstration output. Such a token-level training way essentially does _behavior cloning_ (a special algorithm of imitation learning [393]): it utilizes the expert's action (_i.e.,_ the target token at each step) as the supervision label and directly learns to imitate the demonstrations from experts without specifying a reward model as in typical RL algorithms. To learn the desired policies, SFT adopts a "local" optimization way (_i.e.,_ token-level loss) based on demonstration data, while RLHF takes a "global" optimization way (_i.e.,_ text-level loss) by involving human preference. More theoretical analysis about imitation learning and reinforcement learning can be referred to the related RL literature [393, 394].