**Training overview.** Self-Rag enables an arbitrary LM to generate text with reflection tokens by unifying them as next token predictions from the expanded model vocabulary (i.e., the original vocabulary plus reflection tokens). Specifically, we train the generator model \(\mathcal{M}\) on a curated corpus with interleaving passages retrieved by a _retriever_\(\mathcal{R}\) and reflection tokens predicted by a _critic_ model \(\mathcal{C}\) (summarized in Appendix Algorithm 2). We train \(\mathcal{C}\) to generate reflection tokens for evaluating retrieved passages and the quality of a given task output (Section 3.2.1). Using the critic model, we update the training corpus by inserting reflection tokens into task outputs offline. Subsequently, we train the final generator model (\(\mathcal{M}\)) using the conventional LM objective (Section 3.2.2) to enable \(\mathcal{M}\) to generate reflection tokens by itself without relying on the critic at inference time.