**RL for Rationale Refinement.** For RL, we utilized the Huggingface's TRL4 library that provides a set of tools to train transformer LMs with RL. Instead of using the same examples used for the rationale distillation, we selectively chose 5000 examples that FLAN-T5 XXL failed to answer the question correctly. The reason behind this choice was to increase the model's exposure to challenging questions, thus increasing the likelihood of receiving more learning signals. We used 90% of data for training and the remaining 10% for validation. Training hyperparameters are as follows: 1.4e-5 for learning rates, 1 epoch, 16 batch size. For generation configurations, we set top_k as 0.0, top_p as 1.0, and enabled sampling.