our framework can be flexibly applied to a wide range of LMs, we use FLAN-T5 (Longpre et al., 2023) models in this work because they are open-source and instruction-tuned on both QA and CoT data, enabling us to use it off-the-shelf without additional training or prompt engineering.1 Our experiment results show that LM-guided CoT prompting outperforms both the standard prompting and the original CoT prompting. More precisely, we find that (1) LM-guided CoT with KD and self-consistency (SC) decoding strategy (Wang et al., 2022b) maximizes the performance gain; (2) RL contributes to a slight increase in overall rationale quality and task performance; (3) choosing the highest-quality rationales for the large LM does not always guarantee improved task performance. This work presents a unique alternative to the direct optimization of the large LM through fine-tuning the comparatively smaller LM. Moreover, the clear separation of two fundamental sub-tasks within CoT reasoning grants practitioners greater control over each task.