**Training & Evaluation Setup.** We use a randomly sampled subset of training data from two datasets (15K samples per data) for model training. After filtering unqualified reasoning, it results in 23K samples. All training-related hyperparameters can be found in SS11.3. According to our preliminary experiments, the impact of CoT prompting appeared to be diminished due to two potential factors: (1) questions being overly simplistic, obscuring the significance of intermediate reasoning processes; (2) LMs already possessing pertinent background information (e.g., Flan-T5 is fine-tuned on various question-answering datasets). To prevent models from answering based on parametric memory, we attempt to make the existing evaluation data more challenging, similar to the approaches taken by Ye and Durrett (2022) and Zhao et al. (2023).