**LM-Guided CoT** is a novel framework that decomposes a conventional CoT prompting into two steps using two models: (1) rationale generation and (2) answer prediction. This includes distilling the reasoning ability from a large LM to a small LM and further optimizing it with RL. The results reveal that our method outperforms all baselines, highlighting its potential to serve as an effective and resource-efficient approach to tackle challenges within the CoT prompting paradigm. Meanwhile, we also find that selecting top-quality rationales for answer prediction may not consistently boost task performance. This prompts the need to explore a more harmonious balance between LM-generated rationale utilities and overall task performance.