**Inference overview.** Figure 1 and Algorithm 1 present an overview of Self-Rag at inference. For every \(x\) and preceding generation \(y_{<t}\), the model decodes a retrieval token to evaluate the utility of retrieval. If retrieval is not required, the model predicts the next output segment, as it does in a standard LM. If retrieval is needed, the model generates: a critique token to evaluate the retrieved passage's relevance, the next response segment, and a critique token to evaluate if the information in the response segment is supported by the passage. Finally, a new critique token evaluates the overall utility of the response.4 To generate each segment, Self-Rag processes multiple passages in parallel and uses its own generated reflection tokens to enforce soft constraints (Section 3.3) or hard control (Algorithm 1) over the generated task output. For instance, in Figure 1 (right), the retrieved passages \(d_{1}\) is selected at the first time step since \(d_{2}\) does not provide direct evidence (\(\boxed{\texttt{IsNet}}\) is Irrelevant) and \(d_{3}\) output is only partially supported while \(d_{1}\) are fully supported.