@inproceedings{sun-etal-2024-enhancing-code,
    title = "Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of {LLM}s",
    author = "Sun, Zhihong  and
      Lyu, Chen  and
      Li, Bolun  and
      Wan, Yao  and
      Zhang, Hongyu  and
      Li, Ge  and
      Jin, Zhi",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.521",
    pages = "5878--5895",
    abstract = "Large Language Models (LLMs) have recently made significant advances in code generation through the {`}Chain-of-Thought{'} prompting technique. This technique empowers the model to autonomously devise {``}solution plans{''} to tackle intricate programming challenges, thereby improving its performance in code generation. Nevertheless, smaller models have been struggling to keep up with LLMs in deducing these plans, adversely affecting their code generation capabilities. Given the considerable size and associated deployment costs, along with concerns about data security, many teams opt for deploying smaller models for code generation. Consequently, there arises a compelling need for transferring LLMs{'} code generation reasoning abilities to the smaller models. In this paper, we propose the CodePLAN framework, which aims to transfer LLMs{'} reasoning capabilities to smaller models through distillation. We adopt a multi-task learning approach, jointly undertaking code generation and solution plan generation tasks, to enhance the code generation capabilities of smaller model. To ensure the superior quality of the solution plans, we advocate for the utilization of backward reasoning and plan sampling strategies. Our experiments show that in comparison to the conventional fine-tuning approach, our approach improves the smaller model{'}s code generation performance (measured in pass@1 metric) by over 130{\%} on the challenging APPS benchmark.",
}

Large Language Models (LLMs) have recently made significant advances in code generation through the {`}Chain-of-Thought{'} prompting technique. This technique empowers the model to autonomously devise {``}solution plans{''} to tackle intricate programming challenges, thereby improving its performance in code generation. Nevertheless, smaller models have been struggling to keep up with LLMs in deducing these plans, adversely affecting their code generation capabilities. Given the considerable size and associated deployment costs, along with concerns about data security, many teams opt for deploying smaller models for code generation. Consequently, there arises a compelling need for transferring LLMs{'} code generation reasoning abilities to the smaller models. In this paper, we propose the CodePLAN framework, which aims to transfer LLMs{'} reasoning capabilities to smaller models through distillation. We adopt a multi-task learning approach, jointly undertaking code generation and solution plan generation tasks, to enhance the code generation capabilities of smaller model. To ensure the superior quality of the solution plans, we advocate for the utilization of backward reasoning and plan sampling strategies. Our experiments show that in comparison to the conventional fine-tuning approach, our approach improves the smaller model{'}s code generation performance (measured in pass@1 metric) by over 130{\%} on the challenging APPS benchmark.
Sun, Zhihong Lyu, Chen Li, Bolun Wan, Yao Zhang, Hongyu Li, Ge Jin, Zhi
Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs