**Rationale Distillation.** For computation efficiency or task performance, recent literature has explored methods to improve small LMs' reasoning abilities. Li et al. (2023); Shridhar et al. (2023); Ma et al. (2023) have experimented with rationale distillation, where a small student LM learns from a large teacher LM to generate CoT rationales. While these studies have mainly concentrated on comparing its performance in downstream tasks against that of large LMs, there has been limited investigation into addressing errors in the generated rationales that might have been inherited from the teacher model.