**RL for Rationale Refinement.** We next detail how to utilize established evaluation metrics as reward signals to update the knowledge-distilled \(M^{*}\) with Proximal Policy Optimization (PPO) (Schulman et al., 2017). Given each input (\(q\), \(c\))-output (\(a\)) pair from training data, we first prompt \(M^{*}\) to generate a corresponding rationale \(r^{*}\). During the generation process, an aspect-specific reward (denoted as \(R_{aspect}\)) is measured by aggregating all values returned from automatic evaluation metrics.2 We then pass \(r^{*}\) to \(M^{L}\) to retrieve the answer prediction \(a^{*}\) and compute a task-specific reward (denoted as \(R_{taskAcc}\)).