We study how Decompositional Distillation can assist smaller models to learn better reasoning on a multi-hop math word problems dataset, GSM8Khttps://github.com/openai/grade-school-math [1]}. GSM8K consists of 8.5K grade school math word problems with each problem requiring 2 to 8 reasoning steps to solve. The solutions primarily involve a sequence of elementary calculations using basic arithmetic operations (\(+\)  \(-\)  \(*\)  \(/\) ). The dataset is partitioned into 7.5K training problems and 1K test problems. Since the GSM8K dataset has problem solutions in a step-by-step format leading to the final answer, we used the ground truth as a proof of concept for our experiments.
However, for other datasets with just the final answer, the step-by-step decomposition will be provided by the teacher model.
