We use a student network that imitates the teacher and this goes in the direction of knowledge distillation [1]}, [2]}. [3]} demonstrated the usefulness of providing instruction that can assist the models in achieving better reasoning abilities. Similar to our hypothesis, [4]} argued that question-answering systems should not only focus on the final answer but also rationales that justify their reasoning to help reason better. We go beyond this in our work and work on not just the question-answering system but also on what questions need to be asked at each step that can assist in learning that reasoning step better. Finally, similar to our hypothesis of injecting reasoning capabilities in smaller models, [5]} used CoT-like reasoning from LLMs to train smaller models on a joint task of generating the solution and explaining the solution generated. We, on the other hand, use the LLM to generate subquestions and solution pairs and use them together to inject reasoning capabilities into smaller models.
