* Arithmetic Reasoning. Similar to Wang et al. (2023b); Fu et al. (2023); Du et al. (2023), we select the GSM8K (Cobbe et al., 2021a) as one of the test sets. Additionally, we select the more challenging MATH dataset (Hendrycks et al., 2021b), which is used by Wu et al. (2023). * General Reasoning. Similar to Du et al. (2023); Jiang et al. (2023), we select the MMLU (Hendrycks et al., 2021a). Additionally, we select the dataset from the chess state tracking task (Chess) 1, which is used by Du et al. (2023); Zhang et al. (2023). Footnote 1: Chess State Tracking
* Code Generation. Similar to Liu et al. (2023), we select the HumanEval (Chen et al., 2021). To implement our method, we compute the BLEU score (Papineni et al., 2002) among all pairs of generated candidate answers. The answer with the highest cumulative BLEU score is then selected as the final output.