**Automatic Measurement for Rationale Quality.** Manual annotation offers substantial value, but within the context of RL, it becomes notably challenging due to frequent reward scoring. Therefore, we probe several ways to automate this process. Ye and Durrett (2022) introduced the simple yet effective approach for assessing factuality and relevance through token-level lexical overlap. We follow their method for factuality and relevance measurement. For the remaining 6 categories, two methods are considered. The first approach is to harness a large LM to function as reference-free NLG evaluators, inspired by recent works (_e.g._, Liu et al. (2023), Wang et al. (2023)). The second approach, in contrast, involves training a simple machine learning classifier using human-annotated data. Both approaches are comprehensively described in SS11.2. Due to inference time efficiency and higher alignment scores to human annotators (see Table 5), we resort to the second method for all our experiments.