* In the first stage, we jointly optimize (1) and (2). Ranking by relevance and setting \(k=15\) achieve the lowest average Brier score. * In the second stage, we optimize (3) the reasoning prompt. We identify the top 3 prompts out of 15 candidates to elicit 3 predictions from our base model in our system; see Figure 15 for the best one.