**Annotation for Rationale Quality Measurement.** Inspired by previous text and rationale generation evaluation metrics (Golovneva et al., 2022; Fu et al., 2023), we attempt to quantify 8 linguistic aspects (factuality, relevance, logicality, consistency, coherence, fluency, naturalness, readability) of rationales generated by \(M^{*}\) in SS3.1. Let us denote \(r^{*}\) as generated reasoning. Since there is no ground truth rationale \(r\) available for comparison, our metrics are reference-free, utilizing a pair of \(r^{*}\) and one of existing inputs (\(q\) or \(c\)). Table 1 describes each aspect type and input combinations. As we intend to use these metrics for reward scoring in RL, it is critical to have an accurate metric. Hence, we obtain a small set (n=100) of gold labels for all aspect types through human annotation. A detailed description of the annotation process and results is reported in SS11.1.