**Training data and settings.** Our training data consists of diverse instruction-following input-output pairs. In particular, we sample instances from Open-Instruct processed data (Wang et al., 2023) and knowledge-intensive datasets (Petroni et al., 2021; Stelmakh et al., 2022; Mihaylov et al., 2018). In total, we use 150k instruction-output pairs. We use Llama 7B and 13B (Touvron et al., 2023) as our generator base LM, and we use Llama2 7B as our base critic LM. For the retriever model \(\mathcal{R}\), we use off-the-shelf Contriever-MS MARCO (Izacard et al., 2022) by default and retrieve up to ten documents for each input. More training details are in the Appendix Section B.1.