Large language models (LLMs)  can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses.
Since Chain-of-Thought,
many suchSystem 2techniques have been proposed such as Rephrase and Respond, System 2 Attentionand Branch-Solve-Merge. In this work we investigate self-supervised methods to “compile” (distill)  higher quality outputs from System 2 techniques back into LLM generationswithoutintermediate reasoning token sequences, as this reasoning has been distilled intoSystem 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities
on the reasoning tasks that they cannot yet do well.