**Pros and Cons of SFT**. SFT has been shown to be an effective approach to boosting the performance of LLMs on various benchmarks [67, 69, 137, 138], which can largely enhance the task generalization ability and flexibly endow specific functions (_e.g.,_ establishing the chatbot's identity). More discussions about the usefulness of SFT can be found in Section 5.1.3. It has been widely recognized that SFT mainly _unlocks_ the abilities but not _inject_ new abilities into LLMs. Thus, it might become problematic when one tries to stimulate the non-endogenous abilities of LLMs via SFT. As a concrete scenario, it would potentially advocate the hallucination behaviors when demonstration data is beyond the knowledge or ability scope of LLMs, _e.g.,_ training a LLM to answer questions about its unknown facts. An interesting viewpoint from John Schulman's talk on RLHF [395] is that distilling superior models to train less capable models (_e.g.,_ prompting GPT-4 to generate the response as fine-tuning data) might increase the possibilities of generating the hallucinated texts, thus likely affecting the factual accuracy of LLMs. Furthermore, as a behavior cloning method, SFT aims to imitate the behaviors (without explorations) of the experts who construct the demonstration data. However, there often exist variations among different annotators on the writing styles, quality, and preferences of demonstration data, which tends to affect the learning performance of SFT. Thus, high-quality instruction data (but not the quantity) is the primary factor for effective training of LLMs during the SFT stage [99].