Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Algorithm 1: Speculative RAG

Data: (Q, D = {d_1^n}) is the question and n retrieved documents; m subsets, each containing k documents, are sampled from D; k also corresponds to the number of clusters during clustering.
Result: Â is the predicted answer to the question.
1  Function Speculative RAG  (Q, D, m, k):
2    {c_1, c_2,..., c_k} ⟵ KMeans C(d_1,...., d_n|Q)  ▷ Cluster the documents into k groups using an embedding model C.
3    Δ ⟵ {}
4    repeat
5      δ_j ⟵ {}
6      for ci ∈ {c_1,..., c_k} do
7          δ_j  = δ_j ∪ {random.sample(ci)} ▷ Sample one document from each cluster c_i into subset δ_j .
8      end
9      Δ  = Δ ∪ {δ_j}
10    until |Δ| = m ▷ Repeat the sampling until there are m unique subsets in total.
11    for δ_j ∈ Δ do in parallel ▷ Process m subsets in parallel.
12       α_j, β_j ⟵ Mdrafter.generate(Q, δ_j) ▷ Generate the draft α and rationale β with Mdrafter.
13       ρ_j ⟵ Mverifier.score(α_j,β_j) ▷ Compute the confidence score ρ with Mverifier.
14    end
15   Â ← arg maxα_j ρ_j ▷ Select the one with the highest score as the final answer.
16  return Â


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Explanation

Specifically, as shown in Algorithm 1, we first cluster the retrieved documents with regard to their relation to the posed question, where each cluster represents one perspective in the retrieval results (Line 2). We then sample one document from each cluster into a subset so the documents in this subset covers the multiple perspectives in the retrieval results. We aim at minimizing redundancy and increase the diversity of the documents (Line 5 to 8). We denote one subset as δ_j ⊆ D that contains retrieved documents with diverse contents and multiple perspectives in the retrieval results. Then, we distribute each subset δ to a RAG drafter endpoint Mdrafter with the posed question Q to generate the answer draft α and the rationale β in parallel (Line 12). The RAG drafter is instruction-tuned to be a specialist in understanding the retrieved documents and produce rationales that are faithful to the input documents. It is smaller than generalist LMs, and its parallel processing further ensures high efficiency. For each draft-rationale pair (α_j, β_j) from Mdrafter, we compute a confidence score with the generalist LM Mverifier based on the question Q and corresponding rationale β (Line 13). It is worth mentioning that Mverifier does not need to be instruction-tuned since we leverage its language modeling ability already learned during pre-training. Meanwhile, Mverifier can verify the drafts based on the informative rationale provided by Mdrafter instead of processing tedious or possibly redundant retrieved documents. Finally, we select the answer draft with the highest confidence score as the final answer and integrate it into the generation results of the generalist LM (Line 15).


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Specialist RAG Drafter

Instead of tuning a large generalist LM for the RAG scenario, we leverage a smaller specialist LM, Mdrafter, to understand retrieved documents. Mdrafter is specialized in answering the given question based on the supporting documents and not expected to cope with general problems. It serves as a RAG module for the generalist LMs when solving knowledge-intensive tasks. We train Mdrafter to generate both the answer draft and the rationale to better understand the contextual documents.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Instruction Tuning

Instruction Tuning  Given a triplet (Q, A, D), where Q is a general query, A is the response, and D is a retrieved supporting document, we augment it with the rationale of the response A based on the document D. We denote the rationale as E which extracts essential information from the document and explains why the response is reasonable to the query concisely (Hsieh et al., 2023) so it is of


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Multi-Perspective Sampling

For each knowledge-intensive question, we retrieve a set of documents from the database using the posed question as the retrieval query. These documents may contain diverse content due to the ambiguity inherent in the query. To minimize redundancy and enhance diversity of the document subsets used for generating answer drafts, we employ a multi-perspective sampling strategy. We first cluster the documents into a few topics using an instruction-aware embedding model (Peng et al., 2024) and the K-Means clustering algorithm (Jin & Han, 2011).\n\n$$emb(d_1), ..., emb(d_n) = \epsilon(d_1, ..., d_n|Q)$$\n\n$$\{c_1, ..., c_k\} = K-Means(emb(d_1), ..., emb(d_n))$$\n\n$$\delta = \{random.sample(c) | c \in \{c_1\}^k_1\}$$\n\nwhere $\epsilon$ is an instruction-aware embedding model which embeds a string with regard to a provided instruction (the posed question Q); $emb(d_i)$ is the embedding for the retrieved document $d_i$; $c_i$ is a cluster of retrieved documents with similar topics and contents; $k$ is a hyper-parameter that controls the number of clusters. We sample one document from each cluster into a document subset $\delta$ so each subset contains $k$ documents of diverse contents. In total, we construct $m$ subsets for parallel inference with the RAG drafter.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:RAG Drafting

We run $M_{Drafter}$ over the m document subsets and produce corresponding answer drafts. Refer to Appendix B for detailed processing. We incorporate each document subset into the prompt and query LM for responses. We obtain m drafts as the answer candidates and each draft is grounded based on the multiple perspectives in the retrieval results. Specifically, given a document subset $$\tilde{\delta} = \{d_{i_1}, ..., d_{i_s}\},$$ we query $M_{Drafter}$ in parallel with the following prompt for the answer draft and rationale: $$Q, d_{i_1}, ..., d_{i_s} \rightarrow \alpha_j, \beta_j$$ where the prompt contains the posed question Q along with the document subset; the generation result contains the answer draft $\alpha$ and the rationale $\beta$. We denote the conditional generation probability as $P_{Draft} = P(\beta|Q, d_{i_1}, ..., d_{i_s}) + P(\alpha|Q, d_{i_1}, ..., d_{i_s}, \beta),$ which measures the reliability of generating rationales and the confidence in producing answer drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Generalist RAG Verifier

After generating drafts and the rationale from the RAG drafter $M_{Drafter}$, we evaluate them by a generalist LM $M_{Verifier}$ to filter out the less reliable drafts and select the best answer. The generalist LM can be any off-the-shelf pre-trained LM. We only consider the draft-rationale pair $(\alpha, \beta)$ and skip the tedious and redundant retrieval results. We resort to the language modeling ability of the generalist LM to rank and select the draft-rationale pairs.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Evaluation Scores

First, we calculate the self-consistency score by determining the conditional probability of generating a draft-rationale pair given the question, $P_{self-consist} = P(\alpha, \beta|Q)$. This score helps assess whether the draft and rationale are self-consistent in the context of the question. Given the characteristics of language modeling, a self-consistent draft-rationale pair is expected to yield a higher probability. Furthermore, we incorporate a self-reflection statement R that prompts $M_{Verifier}$ to assess the reliability of an answer draft (e.g. “Do you think the rationale supports the answer, yes or no?”). We define the self-reflection score as $P_{self-reflect} = P(\"Yes\"|Q, \alpha, \beta, R)$ where we compute the conditional probability of the positive answer (\"Yes\") to the self-reflection statement.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Computation Method

We can efficiently compute the self-consistency and self-reflection scores within one forward pass of $M_{Verifier}$. Given a question Q and a draft-rationale pair $(\alpha, \beta)$, we construct a prompt $[Q, \alpha, \beta, R, \"Yes\"]$, where R is the self-reflection statement. We encode the prompt with $M_{Verifier}$ and acquire the probability of each token conditioned on the previous tokens $P(t_i|t_{<i})$. We leverage this auto-regressive feature and aggregate the probability of the relevant condition.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Experiments

We evaluate our proposed SPECULATIVE RAG on four public retrieval augmented generation benchmarks: TriviaQA (unfiltered) (Joshi et al., 2017), MuSiQue (Trivedi et al., 2022), PubHealth (Zhang et al., 2023b), and ARC-Challenge (Clark et al., 2018). TriviaQA and MuSiQue are challenging open-domain question answering datasets where RAG systems are required to answer questions on factual knowledge. TriviaQA typically requires one accurate piece of evidence from the documents, whereas MuSiQue demands multiple documents to construct a multi-hop reasoning chain. Following previous works (Guu et al., 2020; Asai et al., 2023; Yan et al., 2024), we evaluate performance of the free-form generation based on whether gold answers are contained within the generated response or not. PubHealth and ARC-Challenge are closed-set generation datasets. PubHealth is a dataset of medical claims spanning a variety of biomedical subjects and it requires the RAG system to verify a given claim based on the retrieved documents. ARC-Challenge introduces a multi-choice question answering dataset, composed of science exam questions from grade 3 to grade 9. For closed-set generation tasks, we use accuracy metrics to evaluate whether the generated answers match the ground truth.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Baselines

Standard RAG. For standard RAG, we incorporate all the retrieved documents into the prompt as contextual information. Refer to Appendix C for detailed prompts. We run standard RAG experiments on off-the-shelf LLMs including Mistral79, Mistral-Instruct79 (Jiang et al., 2023a), Mixtralt78, Mixtral-Instructt78 (Jiang et al., 2024), and Alpaca78 (Dubois et al., 2024). We also include the performance of Toolformer (Schick et al., 2024) and SAIL (Luo et al., 2023a) which are originally reported from Asai et al. (2023). Toolformer79 is an LM instruction-tuned to use tools including a search engine, and SAIL79 is an LM instruction-tuned on the Alpaca instruction tuning set augmented with search results from difference sources such as DuckDuckGo and Wikipedia.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Self-Reflective RAG and Corrective RAG

SELF-REFLECTION RAG (Self-RAG) (Asai et al., 2023) and Corrective RAG (CRAG) (Yan et al., 2024) are more advanced RAG systems that enhance the quality of contextual information in the retrieval results. CRAG introduces an external evaluator to assess the quality of the retrieval results, and to refine them before the generation process. Self-RAG instruction-tunes an LM to generate special self-reflection tags. These tags guides the LM to dynamically retrieve documents when necessary, critique the retrieved documents relevance before generating responses. Self-CRAG is to apply the Self-RAG approach on the refined documents of CRAG. We adopt the same backbone LLMs across all methods as our proposed SPECULATIVE RAG.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Experiment Settings

In our experiments, we utilize Mistralt78 (v0.1) as our base LM for the RAG drafter. For RAG verifier, we employ either Mistral78 (v0.1) or Mixtralt78 (v0.1) without any fine-tuning, denoted as MVerifie:78 or MVerifie-x78. We pre-compute embeddings of retrieved documents using a lightweight instruction-aware embedding model InBedderate (Peng et al., 2024) as part of the retrieval process. Inference is conducted using the vLLM framework (Kwon et al., 2023) with greedy decoding (temperature 0). We adopt the same experiment settings from Asai et al. (2023) and include a more challenging benchmark, MuSiQue (Trivedi et al., 2022). Our focus is on RAG reasoning rather than evidence elicitation, so we omit the other two long-form generation benchmarks, Biography (Mishra et al., 2023) and ALCE-ASQA (Gao et al., 2023a). On TriviaQA, PubHealth, and ARC-Challenge, we retrieve top 10 documents and generate 5 drafts per query (m = 5), with each draft based on a subset of 2 documents (k = 2). For the MuSiQue dataset, we retrieve top 15 documents and generate 10


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Research Paper Content Extraction

Table 1: Retrieval augmentation generation results on TriviaQA, MusiQue, PubHealth, and ARC-Challenge (ARC-C). (\ We use the RAG drafter's generation probability \( p_{gen} \) as the confidence score for selecting drafts when we use it alone; † indicates numbers reported in Asai et al. (2023); — denotes numbers that are not reported by the original papers or are not applicable; ‡ we use Mistral\textsubscript{7B} or Mixtral\textsubscript{13B} as the RAG verifier, and denote them as \( \text{Verifier-7B} \) or \( \text{Verifier-13B} \).)

```
| RAG Method               | Free-form     | Closed-set   |
|--------------------------|---------------|--------------|
|                          | TriviaQA      | MusiQue      | PubHealth | ARC-C |
| Standard RAG             |               |              |           |       |
| Mistral\textsubscript{7B} (Jiang et al., 2023a) | 54.15         | 16.71        | 34.85     | 42.75 |
| Mixtral\textsubscript{13B} (Jiang et al., 2024) | 59.85         | 19.16        | 37.08     | 48.72 |
| Mixtral-Instruct\textsubscript{13B} (Jiang et al., 2023a) | 67.11         | 17.99        | 42.15     | 47.70 |
| Mixtral-Instruct\textsubscript{13B} (Jiang et al., 2024) | 73.91         | 29.42        | 63.63     | 78.41 |
| Alpaca\textsubscript{7B} (Dubois et al., 2024) | 64.1          |               | 40.2      | 48.1  |
| Toolformers (Schick et al., 2024)\textdagger | 48.8          |               |           |       |
| SaIL-1.0 (Luo et al., 2023)\textdagger |               |               | 69.2      | 48.4  |
|                          |               |               |           |       |
| Self-Reflective RAG & Corrective RAG |               |               |           |       |
| CRAG\textsubscript{Verifier-7B} (Yan et al., 2024) |               |               | 59.04     | 74.87 |
| Self-PRAG\textsubscript{13B} (Vasal et al., 2023) | 64.84         | 21.72        | 72.44     | 74.91 |
| Self-CRAG\textsubscript{Verifier-7B} (Yan et al., 2024) |               |               | 72.85     | 75.26 |
|                          |               |               |           |       |
| Our Speculative RAG     |               |               |           |       |
| \( M_{Drafter-7B} \)      | 71.11         | 27.89        | 75.58     | 74.49 |
| \( \text{Verifier-7B} + M_{Drafter-7B} \) | 73.91         | 31.03        | 75.79     | 76.19 |
| \( \text{Verifier-13B} + M_{Drafter-7B} \) | 74.24         | 31.57        | 76.60     | 80.55 |
```

- Drafts for each query (\( m = 10 \)), each using a subset of 6 documents due to more complex reasoning. Further details regarding instruction-tuning can be found in Appendix E.

4.3 Main Results

We compare SPECULATIVE RAG with standard RAG approaches, as well as the more advanced Self-Reflective RAG and Corrective RAG on four datasets: TriviaQA, MusiQue, PubHealth, and ARC-Challenge. We report the performance of \( M_{Drafter-7B} \) when used alone or paired with the RAG verifier (e.g., \( \text{Verifier-7B}, \text{Verifier-13B} \)). Following prior work (Asai et al., 2023; Yan et al., 2024), we report accuracy as the performance metric.

Superior Performance over Baselines Table 1 demonstrates that SPECULATIVE RAG consistently outperforms all baselines across all four benchmarks. Particularly, \( \text{Verifier-13B} + M_{Drafter-7B} \) surpasses the most competitive standard RAG model, Mixtral-Instruct\textsubscript{13B}, by 0.33% on TriviaQA, 2.15% on MusiQue, 12.97% on PubHealth, and 2.14% on ARC-Challenge. With a comparable number of instruction-tuned parameters, \( \text{Verifier-7B} + M_{Drafter-7B} \) outperforms all Self-Reflective RAG and Corrective RAG methods, and \( M_{Drafter-7B} \) alone can surpass these baselines in most settings.

Effective Instruction Tuning for RAG Drafter Our instruction tuning is effective in enhancing the reasoning ability of the drafter model (Hsieh et al., 2023), as we observe a remarkable performance improvement comparing Mistral\textsubscript{7B} and \( M_{Drafter-7B} \). Moreover, the performance of Mixtral\textsubscript{7B} significantly improves when paired with the instruction-tuned RAG drafter \( M_{Drafter-7B} \), showing gains of 14.39% on TriviaQA, 12.41% on MusiQue, 39.52% on PubHealth, and 31.38% on ARC-Challenge. Similar improvements are observed with Mistral as well. For Mistral\textsubscript{7B}, we observed improvements of 19.76% on TriviaQA, 14.32% on MusiQue, 40.94% on PubHealth, and 33.44% on ARC-Challenge. We attribute these improvements to the superior reasoning capabilities of the RAG drafter over the retrieved documents in SPECULATIVE RAG. By minimizing the redundancy in the sampled documents, the RAG drafter generates higher quality answers that base on diverse perspectives from the retrieval results.

Reliable Scoring by RAG Verifier The reliable draft verification by the generative LM also contributes to the enhanced performance. The performance improves remarkably comparing \( M_{Drafter-7B} \) and \( \text{Verifier-7B} + M_{Drafter-7B} \). The instruction-tuned RAG drafter is specialized in generating answer drafts based on the retrieved documents while the language modeling capabilities of generative LMs are leveraged to validate each draft in light of its rationale. This method is both effective and easy to implement, showcasing the effectiveness of this verification approach.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Latency Analysis

We analyze the latency of Standard RAG and our SPECULATIVE RAG on TriviaQA, MusiQue, PubHealth, and ARC-Challenge. We randomly sample 100 cases from each dataset and report the average time cost for each case, as shown in Figure 2. To simulate real-world application scenarios, we process cases individually without batching. As representative example, we run M_{Verific-xB7B + MDrafter-7B for SPECULATIVE RAG and Mixtral-Instruct-xB7B for Standard RAG, as these demonstrate the highest performance among competitive baselines (see Table 1). We launch 5 endpoints of MDrafter-7B for parallel drafting on TriviaQA, PubHealth, and ARC-Challenge. We launch 10 endpoints for MusiQue due to more drafts. We use tensor parallelism to fit Mixtral-Instruct-xB7B into the GPU memory. We report the latency of Mixtral-Instruct-xB7B under tensor parallelism sizes of 4, 8, and 16. Increasing tensor parallelism does not improve efficiency due to overheads in tensor parallel draft generation, consistently achieves the lowest latency across all datasets. Particularly, it reduces latency by up to 23.41% on TriviaQA, 17.28% on MusiQue, 51.25% on PubHealth, and 26.73% on ARC-Challenge. This highlights the advantage of our approach in reducing processing time while maintaining high performance.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Ablation Studies

We conduct ablation studies on the key components of SPECULATIVE RAG during both drafting and verification stages on TriviaQA and PubHealth in Table 2. We use M_{Verific-xB7B + MDrafter-7B as a running configuration. Same as the main results, we report the accuracy as performance metrics.\nDiversity and reduced redundancy in retrieval improves draft quality significantly. In the first set of experiments, we evaluate the impact of multi-perspective sampling during the drafting. Recall that SPECULATIVE RAG clusters retrieved documents into distinct perspectives and sample one document from each cluster to reduce redundancy for the draft generation. We compare this against two alternative sampling strategies: (1) Random sampling without clustering, where we randomly select a document subset as context, and (2) Sampling from the same cluster, where we select all documents from a single cluster. Our results indicate that our proposed sampling method yields the best performance thanks to its ability to leverage diverse context. Particularly, it improves the accuracy by up to 1.88% on TriviaQA and 2.32% on PubHealth. While random sampling without clustering introduces diversity, it is prone to including redundant documents, degrading draft quality. Sampling from the same cluster significantly underperforms due to a lack of diverse perspectives.\nScoring method on self-consistency and self-reflection refines draft quality effectively. In the second set of experiments, we examine the scoring method during verification. We remove each of the specific confidence scores, \text{P}_{self-contain or \text{P}_{self-reflect in turn. Performance drops are observed when any score is removed. Particularly, removing P_{Draft leads to a minimal decline, 0.19% on TriviaQA and 1.12% on PubHealth, likely due to the limited verification capability of the smaller RAG drafter. Removing either P_{self-contain or \text{P}_{self-reflect results in similar performance decreases, around 2.0% on TriviaQA and around 0.8% on PubHealth, indicating that both self-containment and self-reflection capture different key aspects of reasoning and are crucial during verification. Random


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Ablation Study and Performance Analysis

selection without verification leads to substantial underperformance, resulting in a performance decline of 5.69% on TriviaQA and 5.37% on PubHealth.

Table 2: Ablation study of SPECULATIVE RAG in the drafting and verification stages on TriviaQA and PubHealth.

Drafting Stage:
- Random sampling w/o clustering: 73.02 (-1.22) on TriviaQA, 75.38 (-1.22) on PubHealth
- Sampling from the same cluster: 72.36 (-1.8) on TriviaQA, 74.37 (-2.23) on PubHealth

Verification Stage:
- W/o P\textsc{Draft} (ρ = P\textsc{Self-contain} - P\textsc{Self-reflect}): 74.05 (-0.19) on TriviaQA, 75.48 (-1.12) on PubHealth
- W/o P\textsc{Self-contain} (ρ = P\textsc{Draft} - P\textsc{Self-reflect}): 72.04 (-2.20) on TriviaQA, 75.89 (-0.71) on PubHealth
- W/o P\textsc{Self-reflect} (ρ = P\textsc{Draft} - P\textsc{Self-contain}): 72.36 (-1.88) on TriviaQA, 75.68 (-0.92) on PubHealth
- Random selection w/o verification: 68.55 (-5.69) on TriviaQA, 71.23 (-5.37) on PubHealth

Figure 3: Performance analysis of SPECULATIVE RAG with (a) different numbers of drafts, and (b) different supporting document subset size on TriviaQA and PubHealth.

(a) We include 5, 10, 15, 20 drafts and sample 2 supporting documents for each draft.
(b) We sample 1, 2, 4, 6 supporting documents for each draft and we generate 10 answer drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:4.6 Effects of Generated Rationale for Verification

In SPECULATIVE RAG, we utilize the generated rationale β from the RAG drafter as an indicator of the trustworthiness of answer drafts α. The rationales highlight relevant contents, omit redundant information, and bridge logical gaps between drafts and their supporting documents. To evaluate the effectiveness of the rationales, we examine two alternative scoring methods: (a) replacing rationale with retrieved documents (ρ = \textsc{Score(α|Q, δ)}), or (b) adding retrieved documents to rationale (ρ = \textsc{Score(α|Q, β, δ)}). We compare these alternatives to the scoring method used in SPECULATIVE RAGG (ρ = \textsc{Score(α|Q, β)}) in Table 3. The results show that incorporating longer retrieved documents does not consistently improve performance and tends to increase latency. This suggests that the generated rationale is already of high quality and serves as an effective bridge between the supporting documents and the generated answer drafts. By leveraging this rationale, we can efficiently verify drafts using a generic LLM, leading to accurate final results.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:4.7 Effects of Draft Number and Document Subset Size

We investigate the performance of SPECULATIVE RAG under varying numbers of drafts. Using \textsc{M\textsc{Verifier-7B + M\textsc{Drafter-7B}}} with 5, 10, 15, 20 drafts on TriviaQA and PubHealth. We sample two documents as context per draft. The results are illustrated in Figure 3(a). Since we retrieve top 10 documents in total, we sample up to 20 drafts in these experiments. The results indicate that incorporating more drafts can further improve performance, likely thanks to higher coverage of diverse perspective of documents. Importantly, in SPECULATIVE RAG, we can launch multiple RAG drafter instances to generate drafts in parallel without additional latency.

We also examine the effect of document subset size. By varying the number of documents (1, 2, 4, or 6) sampled for draft generation on TriviaQA and PubHealth (Figure 3(b)), we find that including more documents in the context does not always lead to consistent performance improvement. While TriviaQA queries may benefit from more supporting documents due to their complexity, \textsc{M\textsc{Verifier-7B + M\textsc{Drafter-7B}}} can surpass Mistral-Instruct\textsubscript{13B} even with a single supporting document per draft. Furthermore, with two or more documents per draft, \textsc{M\textsc{Verifier-7B + M\textsc{Drafter-7B}}} can even surpass Mixtral-Instruct\textsubscript{8b}. This further demonstrates the effectiveness of our drafting design.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Performance and Latency Analysis

Table 3: Performance and latency analysis of SPECULATIVE RAG on TriviaQA and PubHealth using M_{
{| retrieval |}} and M_{
{| draft |}}. We add the original document subset δ to the context or replace the generated rationale β with the original retrieved document subset δ during verification, i.e. we compute the self-containment score as P(n, δ|Q) or self-containment P(n, δ|Q), and compute the self-reflection score as P("Yes"|Q, α, δ, R) or Self-reflect = P("Yes"|Q, α, δ, β, R), where Q is the query; α is the answer draft; R is the self-reflection statement.\n\n\n\n| M_{
{| retrieval, 8 |}} + M_{
{| draft, 8 |}} | TriviaQA | PubHealth |
|-----------------|------------|----------|
| Score = P(n, Q, δ) | Accuracy (%) | Latency (s) | Accuracy (%) | Latency (s) |
| ρ = \text{Score}(\alpha|Q, β) | 74.24 | 1.93 | 76.60 | 1.17 |
| ρ = \text{Score}(α|Q, δ) | 74.08 (-0.16) | 2.13 (+0.3676) | 76.09 (-0.51) | 1.31 (+11.97%) |
| ρ = \text{Score}(α|Q, β, δ) | 74.32 (+0.08) | 2.17 (+12.48%) | 76.29 (+0.31) | 1.33 (+13.68%) |


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:5 Conclusion

Our proposed SPECULATIVE RAG decomposes RAG tasks into two separate steps of drafting followed by verification. SPECULATIVE RAG delegates the heavy lifting of drafting to a small specialized RAG drafter, while verification is done using a large generalist LM. The parallel generation of multiple drafts from diverse document subsets provides high quality answer candidates while reducing input token counts and the potential risk of position-bias-over-long-context, resulting in substantial improvements in both the quality and speed of the final output generation. We demonstrate the effectiveness of SPECULATIVE RAG with accuracy gains up to 12.97% while reducing latency by 51% compared to conventional RAG systems. SPECULATIVE RAG sheds new light on the potential of collaborative architectures for enhancing RAG performance through task decomposition.\n\nLimitations\nIn this paper, we demonstrate that a smaller, specialized RAG drafter can work independently of a larger general-purpose LM for knowledge-intensive tasks. While SPECULATIVE RAG enhances both accuracy and efficiency, it does require training an additional drafter model. Although this step is computationally inexpensive, it adds a layer of complexity compared to using a vanilla instruction-tuned RAG model.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:References

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.\nAsai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023.\nBaek, J., Jeong, S., Kang, M., Park, J. C., and Hwang, S. Knowledge-augmented language model verification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1720–1736, 2023.\nBrown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.\nCai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10704, 2024.\nChen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023a.\nChen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., and Yu, D. Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06648, 2023b.\nChen, W., Liu, Y., Wang, W., Bakker, E. M., Georgiou, T., Fieguth, P., Liu, L., and Law, M. S. Deep learning for instance retrieval: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\nClark, P., Cowhey, E., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1805.05457, 2018.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Research Paper Citations and References

Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. 2


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:AlpacaFarm Citation

Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Bai, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. AlpacaFarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024. 6, 7


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Knowledge Card Citation

Feng, S., Shi, W., Bai, Y., Balachandran, V., He, T., and Tsvetkov, Y. Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models. In The Twelfth International Conference on Learning Representations, 2023. 2


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Survey on LLMs

Gao, T., Yen, H., Yu, J., and Chen, D. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465–6488, 2023a. 6


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Retrieval-Augmented Generation Survey

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023b. 1, 3


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Retrieval Augmented Language Model Pre-training

Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020. 6


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Step-by-step Distillation

Hsieh, C.-Y., Li, C.-Y., Keh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distilling step-by-step: outperforming larger language models with less training data and smaller model sizes. arXiv, 2023. 4, 7


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Survey on Hallucination in LLMs

Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. 1


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Leveraging Passage Retrieval

Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In Merlo, P., Tiedemann, J., and Tsarfaty, R. (eds.) Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880, Sevilla, Spain, April 19–23, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.74. 3


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Unsupervised Dense Retrieval

Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. 17


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Mixture of Experts for Retrieval-Augmented Generation

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. L., Bressan, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. 6, 7


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Research on Retrieval-Augmented Generators

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mistral of experts. arXiv preprint arXiv:2410.04808, 2024. 6, 7


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Active Learning for Language Models

Jiang, Z., Vu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In Bouamor, H., Pino, J., and Bailk, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969–7999, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.495. 3


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:K-means Clustering

Jin, X. and Han, J. K-means clustering. Encyclopedia of machine learning, pp. 563–564, 2011. 5


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Supervised Challenge Dataset for Reading Comprehension

Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, 2017. 6


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Open-Domain QA in Era of LLMs

Kamalloo, E., Dziri, N., Clarke, C., and Rafiei, D. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5591–5606, 2023. 1


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Generalization Through Memorization

Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR), 2020. 3


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Summarizing Retrievals Using Answer Credibility

Kim, J., Nam, J., Mo, S., Park, J., Lee, S.-W., Seo, M., Ha, J.-W., and Shin, J. Suer: Summarizing retrievals using answer credibility and novel fusions of dan. arXiv preprint arXiv:2404:10881, 2024. 3


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Zero-Shot Reasoners

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 2022. 1


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:References

1. Kwon, W. L., Z. Zhuang, S. Sheng, Y. Zheng, Lu, Y. C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.

2. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.

3. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

4. Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context LMs struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024.

5. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.

6. Luo, H., Chuang, Y.-S., Gong, Y., Zhang, T., Kim, Y., Wu, X., Fox, D., Meng, H., and Glass, J. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023a.

7. Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08474, 2023b.

8. Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303–5315, 2023.

9. Ma, X., Yang, X., Xiong, W., Chen, B., Yu, L., Zhang, H., May, J., Zettlemoyer, L., Levy, O., and Zhou, C. Megalodon: Efficient lm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801, 2024.

10. Miao, X., Oliar, G., Zhang, Z., Cheng, W., Xue, Z., Zhang, Z., Wong, R. Y. Y., Zhu, Q., Ang, L., Shi, X., et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 9th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol. 3, pp. 932–949, 2024.

11. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suite of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, 2018.

12. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factscore: Fine-grained automatic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076–12100, 2023.

13. Peng, L., Zhang, Y., Wang, Z., Srinivasa, J., Liu, G., Wang, Z., and Shang, J. Answer is all you need: Instruction-following text embedding via answering the question. arXiv preprint arXiv:2402.09642, 2024.

14. Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. Kilt: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523–2544, 2021.

15. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505–3506, 2020.

16. Reid, M., Savinov, N., Telpash, N., Lepikhin, D., Lilligrap, T., Alayrac, J.-B., Sorić, R., Lazaridou, A., Firat, O., Schwetitzer, J., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05350, 2024.

17. Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., and Manning, C. D. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprint arXiv:2401.18059, 2024.

18. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Schwitter, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Reference List

Stelmakh, L., Luan, Y., Dhingra, B., and Chang, M.-W. Asqa: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273–8288, 2022. 17

Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018. 3

Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Sori-cat, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 17

Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. 6

Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go. Exploring the state of instruction tuning on open resources, 2023a. 17

Wang, Z., Araki, J., Jiang, Z., Parvez, M. R., and Neubig, G. Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377, 2023b. 3

Wolff, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. 17

Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909–3925, 2023. 3

Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851, 2024. 2, 3

Xie, J., Zhang, K., Chen, J., Lou, R., and Su, Y. Adaptive chameleon on stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, 2023. 2

Xu, F., Shi, W., and Choi, E. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408, 2023. 3

Xu, Z., Jain, S., and Kankan-halli, M. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024. 1

Yan, S.-Q., Gu, J.-C., Zhu, Y., and Ling, Z.-H. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024. 2, 3, 6, 7, 17

Yoran, O., Wolfson, T., Ram, O., and Berant, J. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558, 2023. 3

Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210, 2023. 3

Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023a. 3

Zhang, T., Luo, H., Chuang, Y.-S., Fang, W., Gaitskell, L., Hartvigsen, T., Wu, X., Fox, D., Meng, H., and Glass, J. Interpretable unified language checking. arXiv preprint arXiv:2304.03728, 2023b. 6


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:A Prompt of Rationale Generation

===================================== Prompt =====================================

# Memorize this piece of evidence in mind and use it as if you already know it.
# Evidence: State religion
Despite enjoying considerable popular support, Christianity was still not the official state religion in Rome, although it was in some neighboring states such as Armenia, Iberia, and Aksum. Roman Religion (Neoplatonic Hellenism) was restored for a time by the Emperor Julian from 361 to 363. Julian does not appear to have reinstated the persecutions of the earlier Roman emperors. Catholic Christianity, as opposed to Arianism and other ideologies deemed heretical, was declared to be the state religion of the Roman Empire on 27 February 380 by the decree "De fide catolica".

# Instruction: When did Christianity become the official religion of Rome?

# Response: 380 AD

# Rationale: (Complete the following sentence with details from the evidence; you can only use the information from the evidence)

===================================== Completion =====================================
Catholic Christianity was declared the official state religion of the Roman Empire on February 27th, 380, through the decree "De fide catolica".

Figure 4: Prompt of Rationale Generation for Gemini-Ultra


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:B Prompt of RAG Drafting

===================================== Prompt =====================================
Response to the instruction. Also provide rationale for your response.
# Instruction: In Buddhism, what is the state of blissful repose or absolute existence by someone relieved of the necessity of rebirth?
# Evidence:
[1] Buddhism:
Nirvana literally means "blowing out, quenching, becoming extinguished". In early Buddhist texts, it is the state of restraint and self-control that leads to the "blowing out" and the ending of the cycles of sufferings associated with rebirths and redeaths. Many later Buddhist texts describe nirvana as identical with "anatta" with complete "emptiness, nothingness". In some texts, the state is described with greater detail, such as passing through the gate of emptiness ("sunyata") realizing that there"
[2] Salvation:
It includes a variety of disciplines, such as yoga and meditation. Nirvana is the profound peace of mind that is acquired with moksha (Liberation). In Buddhism and Jainism, it is the state of being free from suffering. In Hindu philosophy, it is union with the Brahman (Supreme Being). The word literally means "blown out" (as in a candle) and refers, in the Buddhist context, to the blowing out of the fires of desire, aversion, and delusion, and the imperturbable stillness of mind acquired thereafter. In Theravada Buddhism the emphasis is on one’s

===================================== Completion =====================================
# Rationale: Nirvana literally means "blowing out, quenching, becoming extinguished". It is described as a state of "restraint and self-control" that leads to the "blowing out" and the ending of the cycles of sufferings associated with rebirths and redeaths.
# Response: In Buddhism, the state of blissful repose or absolute existence by someone relieved of the necessity of rebirth is called Nirvana.

Figure 5: Prompt of RAG Drafting


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Prompt of Standard RAG

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Evidence:
[1] Britain (place name)
Britain, after which "Britain" became the more commonplace name for the island called Great Britain. After the Anglo-Saxon period, "Britain" was used as a historical term only.
Geoffrey of Monmouth in his pseudohistorical "Historia Regum Britanniae" ...

[2] Great Britain
The peoples of these islands of "Pretannike" were called the "Priteni" or "Pretani". "Priteni" is the source of the Welsh language term Prydain, "Britain", which has the same source as the Goidelic term Cruithne used to refer to the early Brythonic-speaking inhabitants of Ireland. The latter were later called Picts or Caledonians. ...

[10] Albion
Albion is an alternative name for Great Britain. The oldest attestation of the toponym comes from the Greek language. It is sometimes used poetically and generally to refer to the island, but is less common than "Britain" today. The name for Scotland in most of the Celtic languages is related to Albion: "Alba" in Scottish Gaelic, "Albain" ...

### Instruction: What was Britain called – before it was Britain?

### Response:


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Prompt of Standard RAG for Instruction-tuned LM

[INST] Below is an instruction that describes a task. Write a response for it and state your explanation supporting your response.

### Instruction: What was Britain called – before it was Britain?
### Evidence:
[1] Britain (place name)
Britain, after which "Britain" became the more commonplace name for the island called Great Britain. After the Anglo-Saxon period, "Britain" was used as a historical term only.
Geoffrey of Monmouth in his pseudohistorical "Historia Regum Britanniae" ...

[2] Great Britain
The peoples of these islands of "Pretannike" were called the "Priteni" or "Pretani". "Priteni" is the source of the Welsh language term Prydain, "Britain", which has the same source as the Goidelic term Cruithne used to refer to the early Brythonic-speaking inhabitants of Ireland. The latter were later called Picts or Caledonians. ...

[10] Albion
Albion is an alternative name for Great Britain. The oldest attestation of the toponym comes from the Greek language. It is sometimes used poetically and generally to refer to the island, but is less common than "Britain" today. The name for Scotland in most of the Celtic languages is related to Albion: "Alba" in Scottish Gaelic, "Albain" ...
[/INST] The response is:


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:D Case Study

Figure 8 shows two drafts generated for the same question. We observe that our RAG drafter can well understand the multiple perspectives in the retrieval results and generate high-quality drafts. Our RAG verifier can also help filter out the unreliable drafts as we observe a relatively low scores in the first draft in Figure 8.

================================== Prompt of Drafter ==================================
Response to the instruction. Also provide rationale for your response.
## Instruction: Which actress/singer starred as Doralee Rhodes in the 1980 film, "Nine to Five"?
## Evidence:
[1] Diana DeGarmo
Sheila until the show’s closure on June 27, 2010. DeGarmo played the role of Doralee Rhodes in the national tour of "9 to 5", which was launched in Nashville on September 21, 2010. She ended her run as Doralee after the July 2011 Minneapolis tour stop. DeGarmo appeared on musical theatre composer Scott Alan’s third album, ...
[2] 9 to 5 (musical)
Pippa Winslow as Violet Newstead, Louise Olley as Doralee Rhodes and Leo Sene as Franklin Hart Jr., with Samantha Giffard as Roz, Matthew Chase as Joe and Mark Houston, Rachel Ivy, and Blair Anderson. "9 to 5" will play in the West End at the Savoy Theatre from January 29 to August 31, 2019. The production stars Amber Davies (Judy), ...

================================ Completion =============================================
## Rationale: Diana DeGarmo played the role of Doralee Rhodes in the national tour of "9 to 5", which began in September 2010.
## Response: Diana DeGarmo

================================ Verification ============================================
draft score: 0.6594
self-confidence score: 0.3417
self-reflection score: 0.5238

================================== Prompt of Drafter ==================================
Response to the instruction. Also provide rationale for your response.
## Instruction: Which actress/singer starred as Doralee Rhodes in the 1980 film, "Nine to Five"?
## Evidence:
[1] Cynthia Rhodes
Rhodes garnered her first non-dance related role in Michael Crichton’s 1984 science fiction thriller "Runaway" with Tom Selleck, Kirstie Alley and Gene Simmons. Her most notable role was as dance instructor Penny Johnson in the hit 1987 motion picture "Dirty Dancing" with Jennifer Grey and Patrick Swayze. Rhodes’ final motion picture role was the character of Vicki Phillips opposite actor Jameson Parker in the sleeper action-adventure movie ...
[2] Dolly Parton
Parton’s feature film "9 to 5" (1980). The musical ran at the Ahmanson Theatre, Los Angeles in late 2008. It opened on Broadway at the Marquis Theatre in New York City, on April 30, 2009, to mixed reviews. The title track of her 2008 "Backwoods Barbie" album was written for the musical’s character Doralee. Although her score (as well as the musical debut of actress Allison Janney) were praised, ...

================================ Completion =============================================
## Rationale: Dolly Parton starred in the 1980 film "9 to 5" as Doralee Rhodes. Additionally, Parton’s 2008 musical "9 to 5" was based on the film, with Parton reprising her role as Doralee.
## Response: Dolly Parton starred as Doralee Rhodes in the 1980 film, "Nine to Five".

================================ Verification ============================================
draft score: 0.71
self-confidence score: 0.4346
self-reflection score: 0.7449

Figure 8: Case study of Speculative RAG from TriviaQA where Dolly Parton is the correct answer.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Instruction-Tuning Settings

We construct our training dataset for the RAG drafter from diverse instruction-following pairs. We sample instances from Open-Instruct processed data (Wang et al., 2023a) and knowledge-intensive datasets (Petroni et al., 2021; Stelmach et al., 2022; Mihaylov et al., 2018). We augment the instruction-following pairs with retrieved documents and generated rationale. We use the off-the-shelf dense retriever Contriever-MS MARCO (Izacard et al., 2021) to retrieve up to 10 documents for each pair and use Gemini-Ultra (Team et al., 2023) to generate rationale. In total, we acquire a dataset of 40k instances. We use Mistral
treq{b v0.1} as our base LM for the RAG drafter. We reproduce the performance of Self-RAG (Asai et al., 2023) and CRAG (Yan et al., 2024) with Mistral
treq{b v0.1} for a fair comparison. We implement the training scripts using the Transformers library from Hugging Face (Wolf et al., 2019). We employ DeepSpeed (Rasley et al., 2020) to accelerate the training process. All experiments are conducted on a Linux server equipped with 16 Nvidia A100-SXM4-40GB GPUs.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Effects of Self-Reflection Statement

We use “Do you think the explanation supports the answers? (Yes or No)” as the self-reflection statement in our main results. In this study, we replace it with other alternatives to see how the self-reflection statement affects the accuracy. The results are reported in Table 4. We observe that the performance does not change a lot given different self-reflection statements, which shows the stable verification capability of the generalist LMs by language modeling objective.

Table 4: Performance analysis of SPECULATIVE RAG with different self-reflection statements R when computing the self-reflection score P ("Yes"|Q, α, β, γ), where Q is the query, α, β, γ are the generated answer draft and rationale.

| Reflection Statement                                       | TriviaQA | PubHealth |
|-----------------------------------------------------------|----------|-----------|
| Do you think the explanation supports the answers? (Yes or No)     | 74.24    | 76.60     |
| Does the rationale support the answer? (Yes or No)                 | 74.22    | 76.09     |
| What do you think about the rationale? A good one? (Yes or No)     | 74.25    | 75.79     |
| Is the rationale good enough to support the answer? (Yes or No)    | 74.39    | 76.29     |


Searching for Best Practices in Retrieval-Augmented Generation:Abstract

Retrieval-augmented generation (RAG) techniques have proven to be effective in integrating up-to-date information, mitigating hallucinations, and enhancing response quality, particularly in specialized domains. While many RAG approaches have been proposed to enhance large language models through query-dependent retrievals, these approaches still suffer from their complex implementation and prolonged response times. Typically, a RAG workflow involves multiple processing steps, each of which can be executed in various ways. Here, we investigate existing RAG approaches and their potential combinations to identify optimal RAG practices. Through extensive experiments, we suggest several strategies for deploying RAG that balance both performance and efficiency. Moreover, we demonstrate that multimodal retrieval techniques can significantly enhance question-answering capabilities about visual inputs and accelerate the generation of multimodal content using a “retrieval as generation” strategy. Resources are available at https://github.com/FudanNLP/RAG.


Searching for Best Practices in Retrieval-Augmented Generation:Introduction

Generative large language models are prone to producing outdated information or fabricating facts, although they were aligned with human preferences by reinforcement learning [1] or lightweight alternatives [2–5]. Retrieval-augmented generation (RAG) techniques address these issues by combining the strengths of pretraining and retrieval-based models, thereby providing a robust framework for enhancing model performance [6]. Furthermore, RAG enables rapid deployment of applications for specific organizations and domains without necessitating updates to the model parameters, as long as query-related documents are provided.

Many RAG approaches have been proposed to enhance large language models (LLMs) through query-dependent retrievals [6–8]. A typical RAG workflow usually contains multiple intervening processing steps: query classification (determining whether retrieval is necessary for a given input query), retrieval (efficiently obtaining relevant documents for the query), reranking (refining the order of retrieved documents based on their relevance to the query), repacking (organizing the retrieved documents into a structured one for better generation), summarization (extracting key information for response generation from the repacked document and eliminating redundancies) modules. Implementing RAG also requires decisions on the ways to properly split documents into chunks, the types of embeddings to use for semantically representing these chunks, the choice of


Searching for Best Practices in Retrieval-Augmented Generation:Figure and Description

Figure 1: Retrieval-augmented generation workflow. This study investigates the contribution of each component and provides insights into optimal RAG practices through extensive experimentation. The optional methods considered for each component are indicated in bold fonts, while the methods underlined indicate the default choice for individual modules. The methods indicated in blue font denote the best-performing selections identified empirically.


Searching for Best Practices in Retrieval-Augmented Generation:Content

vector databases to efficiently store feature representations, and the methods for effectively fine-tuning LLMs (see Figure 1).

What adds complexity and challenge is the variability in implementing each processing step. For example, in retrieving relevant documents for an input query, various methods can be employed. One approach involves rewriting the query first and using the rewritten queries for retrieval. Alternatively, pseudo-responses to the query can be generated first, and the similarity between these pseudo-responses and the backend documents can be compared for retrieval. Another option is to directly employ embedding models, typically trained in a contrastive manner using positive and negative query-response pairs. The techniques chosen for each step and their combinations significantly impact both the effectiveness and efficiency of RAG systems. To the best of our knowledge, there has been no systematic effort to pursue the optimal implementation of RAG, particularly for the entire RAG workflow.

In this study, we aim to identify the best practices for RAG through extensive experimentation. Given the infeasibility of testing all possible combinations of these methods, we adopt a three-step approach to identify optimal RAG practices. First, we compare representative methods for each RAG step (or module) and select up to three of the best-performing methods. Next, we evaluate the impact of each method on the overall RAG performance by testing one method at a time for an individual step, while keeping the other RAG modules unchanged. This allows us to determine the most effective method for each step based on its contribution and interaction with other modules during response generation. Once the best method is chosen for a module, it is used in subsequent experiments. Finally, we empirically explore a few promising combinations suitable for different application scenarios where efficiency might be prioritized over performance, or vice versa. Based on these findings, we suggest several strategies for deploying RAG that balance both performance and efficiency.

The contributions of this study are three-fold:

• Through extensive experimentation, we thoroughly investigated existing RAG approaches and their combinations to identify and recommend optimal RAG practices.


Searching for Best Practices in Retrieval-Augmented Generation:Related Work

Ensuring the accuracy of responses generated by Large Language Models (LLMs) such as ChatGPT [13] and LLaMA [14] is essential. However, simply enlarging model size does not fundamentally address the issue of hallucinations [15, 16], especially in knowledge-intensive tasks and specialized domains. Retrieval-augmented generation (RAG) addresses these challenges by retrieving relevant documents from external knowledge bases, providing accurate, real-time, domain-specific context to LLMs [6]. Previous works have optimized the RAG pipeline through query and retrieval transformations, enhancing retriever performance, and fine-tuning both the retriever and generator. These optimizations improve the interaction between input queries, retrieval mechanisms, and generation processes, ensuring the accuracy and relevance of responses.

2.1 Query and Retrieval Transformation
Effective retrieval requires queries accurate, clear, and detailed. Even when converted into embeddings, semantic differences between queries and relevant documents can persist. Previous works have explored methods to enhance query information through query transformation, thereby improving retrieval performance. For instance, Query2Doc [17] and HyDE [10] generate pseudo-documents from original queries to enhance retrieval, while TOC [18] decomposes queries into subqueries, aiding the retrieval of content for final results.

Other studies have focused on transforming retrieval source documents. LlamaIndex [19] provides an interface to generate pseudo-queries for retrieval documents, improving matching with real queries. Some works employ contrastive learning to bring query and document embeddings closer in semantic space [12, 20, 21]. Post-processing retrieved documents is another method to enhance generated output, with techniques like hierarchical prompt summarization [22] and using abstractive and extractive compressors [23] to reduce content length and remove redundancy [24].

2.2 Retrieval Enhancement Strategy
Document chunking and embedding methods significantly impact retrieval performance. Common chunking strategies divide documents into chunks, but determining optimal chunk length can be challenging. Small chunks may fragment sentences, while large chunks might include irrelevant context. LlamaknIndex [19] optimizes the chunking method like Small2Big and sliding window. Retrieved chunks can be irrelevant and numbers can be large, so reranking is necessary to filter irrelevant documents. A common reranking approach employs deep language models such as BERT [25], T5 [26], or LLaMA [27], which requires slow inference steps during reranking but grants better performance. TILDE [28, 29] achieves efficiency by precomputing and storing the likelihood of query terms, ranking documents based on their sum.

2.3 Retriever and Generator Fine-tuning
Fine-tuning within the RAG framework is crucial for optimizing both retrievers and generators. Some research focuses on fine-tuning the generator to better utilize retriever context [30–32], ensuring faithful and robust generated content. Others fine-tune the retriever to learn to retrieve beneficial passages for the generator[33–35]. Holistic approaches treat RAG as an integrated system; fine-tuning both retriever and generator together to enhance overall performance[36–38], despite increased complexity and integration challenges.

Several surveys have extensively discussed current RAG systems, covering aspects like text generation [7, 8], integration with LLMs [6, 39], multimodal [40], and AI-generated content [41]. While these surveys provide comprehensive overviews of existing RAG methodologies, selecting the appro-


Searching for Best Practices in Retrieval-Augmented Generation:RAG Workflow

In this section, we detail the components of the RAG workflow. For each module, we review commonly used approaches and select the default and alternative methods for our final pipeline. Section 4 will discuss best practices. Figure 1 presents the workflow and methods for each module. Detailed experimental setups, including datasets, hyperparameters, and results are provided in Appendix A.


Searching for Best Practices in Retrieval-Augmented Generation:Query Classification

Not all queries require retrieval-augmented due to the inherent capabilities of LLMs. While RAG can enhance information accuracy and reduce hallucinations, frequent retrieval can increase response time. Therefore, we begin by classifying queries to determine the necessity of retrieval. Queries requiring retrieval proceed through the RAG modules; others are handled directly by LLMs.

Retrieval is generally recommended when knowledge beyond the model’s parameters is needed. However, the necessity of retrieval varies by task. For instance, an LLM trained up to 2023 can handle a translation request for "Sora was developed by OpenAI" without retrieval. Conversely, an introduction request for the same topic would require retrieval to provide relevant information.

Therefore, we propose classifying tasks by type to determine if a query needs retrieval. We categorize 15 tasks based on whether they provide sufficient information, with specific tasks and examples illustrated in Figure 2. For tasks entirely based on user-given information, we denote as "sufficient", which need not retrieval; otherwise, we denote as “insufficient”, and retrieval may be necessary. We train a classifier to automate this decision-making process. Experimental details are presented in Appendix A.1. Section 4 explores the impact of query classification on the workflow, comparing scenarios with and without classification.


Searching for Best Practices in Retrieval-Augmented Generation:Embedding Model Table

Embedding Model | namespace-Pt/msmarco | MRR @ 1 | MRR @ 10 | MRR @ 100 | R @ 1 | R @ 10 | R @ 100
BAAI/LLM-Embedder [20] | 21.74 | 37.58 | 38.62 | 24.07 | 66.45 | 90.75
BAAI/bge-base-en-v1.5 [12] | 23.34 | 35.80 | 36.94 | 22.63 | 64.12 | 90.13
BAAI/bge-small-en-v1.5 [12] | 23.27 | 35.78 | 36.89 | 22.65 | 63.92 | 89.80
BAAI/bge-large-en-v1.5 [12] | 24.63 | 37.48 | 38.59 | 23.91 | 65.57 | 90.60
BAAI/bge-large-en [12] | 24.84 | 37.66 | 38.73 | 24.13 | 66.03 | 90.64
BAAI/bge-small-en [12] | 23.28 | 35.79 | 36.91 | 22.62 | 63.96 | 89.67
BAAI/bge-base-en [12] | 23.47 | 35.94 | 37.07 | 22.73 | 64.17 | 90.14
Alibaba-NLP/bge-large-en-v1.5 [21] | 8.93 | 15.60 | 16.71 | 8.67 | 28.32 | 60.36
thenlper/gte-base [21] | 7.42 | 13.23 | 14.30 | 7.21 | 22.87 | 56.20
thenlper/gte-small [21] | 7.97 | 14.81 | 15.95 | 7.71 | 32.07 | 61.08
jinai/jina-embeddings-v2-small-en [42] | 8.07 | 15.02 | 16.12 | 7.87 | 32.55 | 60.36
intfloat/e5-small-v2 [11] | 10.04 | 18.23 | 19.41 | 9.74 | 38.92 | 68.42
intfloat/e5-large-v2 [11] | 9.58 | 17.94 | 19.03 | 9.35 | 39.09 | 66.11
sentence-transformers/all-mpnet-base-v2 | 5.80 | 11.26 | 12.26 | 5.66 | 25.57 | 50.94

Table 2: Results for different embedding models on namespace-Pt/msmarco.


Searching for Best Practices in Retrieval-Augmented Generation:3.2 Chunking

Chunking documents into smaller segments is crucial for enhancing retrieval precision and avoiding length issues in LLMs. This process can be applied at various levels of granularity, such as token, sentence, and semantic levels.
- Token-level Chunking is straightforward but may split sentences, affecting retrieval quality.
- Sentence-level Chunking uses LLMs to determine breakpoints, content-preserving but time-consuming.
- Semantic-level Chunking balances preserving context with maintaining efficiency.

In this study, we use sentence-level chunking, balancing simplicity and semantic preservation. We examine chunking from four dimensions.


Searching for Best Practices in Retrieval-Augmented Generation:3.2.1 Chunk Size

Chunk size significantly impacts performance. Larger chunks provide more context, enhancing comprehension but increasing processing time. Smaller chunks improve retrieval recall and reduce time but may lack sufficient context.

Finding the optimal chunk size involves a balance between some metrics such as faithfulness, and relevancy. Faithfulness measures whether the response is hallucinated or matches the retrieved texts. Relevancy measures whether the retrieved texts and responses match queries. We use the evaluation module of Llamalndex [43] to calculate the metrics above. For embedding, we use the text-embedding-ada-002 model, which supports long input length. We choose zephyr-7b-alpha3 and gpt-3.5-turbo4 as generation model and evaluation model respectively. The size of the chunk overlap is 20 tokens. First sixty pages of the document lyft_20215 were used as corpus, then prompting LLMs to generate about one hundred and seventy queries according to chosen corpus. The impact of different chunk sizes is shown in Table 3.


Searching for Best Practices in Retrieval-Augmented Generation:Comparison of different chunk sizes

Chunk Size | lyft_2021
| Average Faithfulness | Average Relevancy
2048 | 80.37 | 91.11
1024 | 94.26 | 95.56
512 | 97.59 | 97.41
256 | 97.22 | 97.78
128 | 95.74 | 97.22

Table 3: Comparison of different chunk sizes.


Searching for Best Practices in Retrieval-Augmented Generation:3.2.2 Chunking Techniques

Advanced techniques such as small-to-big and sliding window improve retrieval quality by organizing chunk block relationships. Small-sized chunks are used to match queries, and larger blocks that include the small ones along with contextual information are returned. To demonstrate the effectiveness of advanced chunking techniques, we use the LLM-Embedder [20] model as an embedding model. The smaller chunk size is 175 tokens, the larger chunk size is 512 tokens and the chunk overlap is 20 tokens. Techniques like small-to-big and sliding window improve retrieval quality by maintaining context and ensuring relevant information is retrieved. Detailed results are shown in Table 4.


Searching for Best Practices in Retrieval-Augmented Generation:3.2.3 Embedding Model Selection

Choosing the right embedding model is crucial for effective semantic matching of queries and chunk blocks. We use the evaluation module of FlagEmbedding[6] which uses the dataset namespace-Pt/msmarco[7] as queries and dataset namespace-Pt/msmarco-corpus[8] as corpus to choose the appropriate open source embedding model. As shown in Table 2, LLM-Embedder [20] achieves comparable results with BAAI/bge-large-en [12], however, the size of the former is three times smaller than that of the latter. Thus, we select the LLM-Embedder [20] for its balance of performance and size. \n\nTable 4: Comparison of different chunk skills. \n\nChunk Skill | Average Faithfulness | Average Relevancy\nOriginal | 95.74 | 95.37\nsmall2big | 96.67 | 95.37\nsliding window | 97.41 | 96.85


Searching for Best Practices in Retrieval-Augmented Generation:3.2.4 Metadata Addition

Enhancing chunk blocks with metadata like titles, keywords, and hypothetical questions can improve retrieval, provide more ways to post-process retrieved texts, and help LLMs better understand retrieved information. A detailed study on metadata inclusion will be addressed in future work.


Searching for Best Practices in Retrieval-Augmented Generation:3.3 Vector Databases

Vector databases store embedding vectors with their metadata, enabling efficient retrieval of documents relevant to queries through various indexing and approximate nearest neighbor (ANN) methods. To select an appropriate vector database for our research, we evaluated several options based on four key criteria: multiple index types, billion-scale vector support, hybrid search, and cloud-native capabilities. These criteria were chosen for their impact on flexibility, scalability, and ease of deployment in modern, cloud-based infrastructures. Multiple index types provide the flexibility to optimize searches based on different data characteristics and use cases. Billion-scale vector support is crucial for handling large datasets in LLM applications. Hybrid search combines vector search with traditional keyword search, enhancing retrieval accuracy. Finally, cloud-native capabilities ensure seamless integration, scalability, and management in cloud environments. Table 5 presents a detailed comparison of five open-source vector databases: Weaviate, Faiss, Chroma, Qdrant, and Milvus.\n\nTable 5: Comparison of Various Vector Databases\nDatabase | Multiple Index Type | Billion-Scale Support | Hybrid Search | Cloud-Native\nWeaviate | ✓ | X | X | ✓\nFaiss | ✓ | ✓ | X | X\nChroma | X | ✓ | ✓ | ✓\nQdrant | X | X | ✓ | ✓\nMilvus | ✓ | ✓ | ✓ | ✓\n\nOur evaluation indicates that Milvus stands out as the most comprehensive solution among the databases evaluated, meeting all the essential criteria and outperforming other open-source options.


Searching for Best Practices in Retrieval-Augmented Generation:Results for Different Retrieval Methods

Table 6: Results for different retrieval methods on TREC DL19/20. The best result for each method is made bold and the second is underlined.

| Configuration            | TREC DL19                                                                                | TREC DL20                                                                                |
|--------------------------|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
|                          | mAP      | nDCG@10 | R@50 | R@1k | Latency | mAP      | nDCG@10 | R@50 | R@1k | Latency |
|--------------------------|----------|---------|------|------|---------|----------|---------|------|------|---------|
| unsupervised             |                                                                                |                                                                                |
| BM25                     | 30.13    | 50.58   | 38.52| 75.01| 0.072   | 28.56    | 47.96   | 46.18| 75.83| 0.29    |
| Contriever               | 23.99    | 44.54   | 37.54| 74.59| 3.06    | 23.98    | 42.13   | 43.81| 75.39| 0.98    |
| supervised               |                                                                                |                                                                                |
| LLM-Embedder             | 44.66    | 70.20   | 49.06| 84.48| 2.61    | 45.60    | 68.76   | 61.36| 84.41| 0.71    |
| + Query Rewriting        | 44.56    | 67.89   | 51.45| 85.35| 7.80    | 45.16    | 65.65   | 59.63| 83.45| 2.06    |
| + Query Decomposition    | 41.93    | 66.10   | 48.66| 82.76| 7.48    | 43.50    | 64.95   | 57.74| 83.14| 3.32    |
| + HyDE                   | 50.87    | 75.64   | 54.33| 83.76| 7.21    | 50.94    | 73.94   | 63.30| 88.03| 2.14    |
| + Hybrid Search          | 47.14    | 72.50   | 51.13| 80.89| 3.20    | 47.72    | 69.80   | 64.32| 88.04| 0.77    |
| + HyDE + Hybrid Search   | 52.13    | 73.34   | 55.38| 90.42| 11.16   | 53.13    | 72.72   | 66.14| 90.67| 2.95    |

Table 7: HyDE with different concatenation of hypothetical documents and queries.

| Configuration               | TREC DL19                                                                                | TREC DL20                                                                                |
|-----------------------------|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
|                             | mAP      | nDCG@10 | R@50 | R@1k | Latency | mAP      | nDCG@10 | R@50 | R@1k | Latency |
|-----------------------------|----------|---------|------|------|---------|----------|---------|------|------|---------|
| HyDE                        |                                                                                |                                                                                |
| w / 1 pseudo-doc            | 48.77    | 72.49   | 53.20| 87.73| 8.08    | 51.31    | 70.37   | 63.28| 87.81| 2.09    |
| w / pseudo-doc + query      | 50.87    | 75.64   | 54.33| 83.76| 7.21    | 50.94    | 73.94   | 63.30| 88.03| 2.14    |
| w / pseudo-doc + query + 2  | 51.64    | 75.12   | 54.51| 89.17| 14.15   | 53.14    | 73.65   | 65.79| 88.67| 3.44    |


Searching for Best Practices in Retrieval-Augmented Generation:Retrieval Methods

Given a user query, the retrieval module selects the top-k relevant documents from a pre-built corpus based on the similarity between the query and the documents. The generation model then uses these documents to formulate a relevant response to the query. However, original queries often underperform due to poor expression and lack of semantic information [6], negatively impacting the retrieval process. To address these issues, we evaluated three fuzzy transformer methods using the LLM-Embedder recommended in Section 3.2 as the query and document encoder:

- Query Rewriting: Query rewriting refines queries to better match relevant documents. Inspired by the Rewrite-Retrieve-Read framework [9], we prompt an LLM to rewrite queries to enhance performance.

- Query Decomposition: This approach involves retrieving documents based on sub-questions derived from the original query, which is more complex to comprehend and handle.

- Pseudo-documents Generation: This approach generates a hypothetical document based on the user query and uses the embedding of hypothetical answers to retrieve similar documents. One notable implement is HyDE [10].

Recent studies, such as [44], indicate that combining lexical-based search with vector search significantly enhances performance. In this study, we use BM25 for sparse retrieval and Contriever [45], an unsupervised contrastive encoder, for dense retrieval, serving as two robust baselines based on Thakur et al. [46].

3.4.1 Results for different retrieval methods
We evaluated the performance of different search methods on the TREC DL 2019 and 2020 passage ranking datasets. The results presented in Table 6 show that supervised methods significantly outperformed unsupervised methods. Combining with HyDE and hybrid search, LLM-Embedder achieves the highest scores. However, query rewriting and query decomposition did not enhance retrieval performance as effectively. Considering the best performance and tolerated latency, we recommend Hybrid Search with HyDE as the default retrieval method. Taking efficiency into consideration, Hybrid Search combines sparse retrieval (BM25) and dense retrieval (Original embedding) and achieves notable performance with relatively low latency.

3.4.2 HyDE with Different Concatenation of Documents and Query
Table 7 shows the impact of different concatenation strategies for hypothetical documents and queries using HyDE. Concatenating multiple pseudo-documents with the original query can significantly improve retrieval performance.


Searching for Best Practices in Retrieval-Augmented Generation:Table 8

Results of hybrid search with different alpha values.

Figures and Tables:
- Table 8 shows the results of hybrid search with varying α values, which determine the weight between sparse and dense retrieval components. It includes metrics such as mAP, nDCG@10, R@50, R@1k, and latency for datasets TREC DL'19 and TREC DL'20.
- The best performance is noted when α = 0.3, suggesting a balance in retrieval effectiveness.


Searching for Best Practices in Retrieval-Augmented Generation:Table 9

Results of different reranking methods on the dev set of the MS MARCO Passage ranking dataset.

Figures and Tables:
- Table 9 compares different reranking methods using MS MARCO Passage ranking dataset. Models compared include Random Ordering, BM25, DLM Reranking, and TILDE Reranking, across metrics like MRR@1, MRR@10, MRR@100, Hit Rate@10, and Latency.


Searching for Best Practices in Retrieval-Augmented Generation:3.4.3 Hybrid Search with Different Weight on Sparse Retrieval

Subsection describing the impact of α values in hybrid search.
- Equation S_h = α * S_s + S_d defines the relevance score, balancing sparse and dense retrieval components.
- Five α values are tested, with α = 0.3 found as most effective.
- Further details can be found in Appendix A.2.


Searching for Best Practices in Retrieval-Augmented Generation:3.5 Reranking Methods

Section discussing the reranking phase to improve retrieval.
- Two methods are used: DLM Reranking and TILDE Reranking.

DLM Reranking:
- Uses deep language models (DLMs) to classify document relevance.
- Documents are ranked based on the probability of being 'true'.

TILDE Reranking:
- Predicts token probabilities across model's vocabulary.
- Scores by summing predicted probabilities across the model's vocabulary.


Searching for Best Practices in Retrieval-Augmented Generation:Comparison of Summarization Methods

Table 10: Comparison between different summarization methods.

Method | NQ | TQA | HotPotQA | Avg. | Avg. Token
F1 #token F1 #token F1 #token
- Origin Prompt: 	27.07 124 33.61 152 33.92 141 31.53 139
- BM25: 	27.97 40 32.44 59 28.00 63 29.47 54
- Contriever: 	23.62 42 33.79 65 23.64 60 27.02 56
- Recomp (extractive): 	27.84 34 35.32 60 29.46 58 30.87 51
- SelectiveContext: 	25.05 65 34.25 70 34.43 66 31.24 67
- LongLLMingua: 	21.32 51 32.81 56 30.79 57 28.29 55
- Recomp (abstractive): 	33.68 59 35.87 61 29.01 57 32.85 59

The pre-calculated log probabilities of query tokens, allowing for rapid reranking at inference. TILDEv2 improves this by indexing only document-present tokens, using NCE loss, and expanding documents, thus enhancing efficiency and reducing index size.

Our experiments were conducted on the MS MARCO Passage ranking dataset [47], a large-scale dataset for machine reading comprehension. We follow and make modifications to the implementation provided by PyGaggle [26] and TILDE [28], using the models monoT5, monoBERT, RankLLaMA and TILDEv2. Reranking results are shown in Table 9. We recommend monoT5 as a comprehensive method balancing performance and efficiency. RankLLaMA is suitable for achieving the best performance, while TILDEv2 is ideal for the quickest experience on a fixed collection. Details on the experimental setup and results are presented in Appendix A.3.


Searching for Best Practices in Retrieval-Augmented Generation:Document Repacking and Summarization

3.6 Document Repacking

The performance of subsequent processes, such as LLM response generation, may be affected by the order documents are provided. To address this issue, we incorporate a compact repacking module into the workflow after reranking, featuring three repacking methods: “forward”, “reverse” and “sides”. The “forward” method repacks documents by descending relevancy scores from the reranking phase, whereas the “reverse” arranges them in ascending order. Inspired by Liu et al. [48], concluding that optimal performance is achieved when relevant information is placed at the head or tail of the input, we also include a “sides” option.

Since the repacking method primarily affects subsequent modules, we select the best repacking method in Section 4 by testing it in combination with other modules. In this section, we choose the “sides” method as the default repacking method.

3.7 Summarization

Retrieval results may contain redundant or unnecessary information, potentially preventing LLMs from generating accurate responses. Additionally, long prompts can slow down the inference process. Therefore, efficient methods to summarize retrieved documents are crucial in the RAG pipeline.

Summarization tasks can be extractive or abstractive. Extractive methods segment text into sentences, then score and rank them based on importance. Abstractive compressors synthesize information from multiple documents to rephrase and generate a cohesive summary. These tasks can be query-based or non-query-based. In this paper, as RAG retrieves information relevant to queries, we focus exclusively on query-based methods.

- Recomp: Recomp [23] has extractive and abstractive compressors. The extractive compressor selects useful sentences, while the abstractive compressor synthesizes information from multiple documents.
- LongLLMingua: LongLLMingua [49] improves LLMLingua by focusing on key information related to the query.
- Selective Context: Selective Context enhances LLM efficiency by identifying and removing redundant information in the input context. It evaluates the informativeness of lexical units using.


Searching for Best Practices in Retrieval-Augmented Generation:Generator Fine-tuning

In this section, we focus on fine-tuning the generator while leaving retriever fine-tuning for future exploration. We aim to investigate the impact of fine-tuning, particularly the influence of relevant or irrelevant contexts on the generator's performance.

Formally, we denote x as the query fed into the RAG system, and D as the contexts for this input. The fine-tuning loss of the generator is the negative log-likelihood of the ground-truth output y.

To explore the impact of fine-tuning, especially relevant and irrelevant contexts, we define d_gold as a context relevant to the query, and d_random as a randomly retrieved context. We train the model by varying the composition of D as follows:

- D_g: The augmented context consists of query-relevant documents, denoted as D_g = {d_gold}.
- D_r: The context contains one randomly sampled document, denoted as D_r = {d_random}.
- D_gr: The augmented context comprises a relevant document and a randomly-selected one, denoted as D_gr = {d_gold, d_random}.
- D_gg: The augmented context consists of two copies of a query-relevant document, denoted as D_gg = {d_gold, d_gold}.

We denote the base LM generator not fine-tuned as M_b, and the model fine-tuned under the corresponding D as M_g, M_r, M_gr, M_gg. We fine-tuned our model on several QA and reading comprehension datasets. Ground-truth coverage is used as our evaluation metric since QA task answers are relatively short. We select Llama2-7B (52) as the base model. Similar to training, we evaluate all trained models on validation sets with D_g, D_r, D_gr, and D_gg, where D_Φ indicates inference without retrieval. Figure 3 presents our main results. Models trained with a mix of relevant and random documents (M_gr) perform best when provided with either gold or mixed contexts. This suggests that mixing relevant and random contexts during training can enhance the generator’s robustness to irrelevant information while ensuring effective utilization of relevant contexts. Therefore, we identify the practice of augmenting with a few relevant and randomly-selected documents during training as the best approach. Detailed dataset information, hyperparameters and experimental results can be found in Appendix A.5.


Searching for Best Practices in Retrieval-Augmented Generation:Searching for Best RAG Practices

In the following section, we investigate the optimal practices for implementing RAG. To begin with, we used the default practice identified in Section 3 for each module. Following the workflow depicted in Figure 1, we sequentially optimized individual modules and selected the most effective option among alternatives. This iterative process continued until we determined the best method for implementing the final summarization module. Based on Section 3.8, we used the Llama2-7B-Chat model fine-tuned where each query was augmented by a few random-selected and relevant documents.


Searching for Best Practices in Retrieval-Augmented Generation:Comprehensive Evaluation

We conducted extensive experiments across various NLP tasks and datasets to assess the performance of RAG systems. Specifically: (I) Commonsense Reasoning; (II) Fact Checking; (III) Open-Domain QA; (IV) MultiHop QA; (V) Medical QA. For further details on the tasks and their corresponding datasets, please refer to Appendix A.6. Furthermore, we evaluated the RAG capabilities on subsets extracted from these datasets, employing the metrics recommended in RAGs [51], including Faithfulness, Context Relevancy, Answer Relevancy, and Answer Correctness. Additionally, we measured Retrieval Similarity by computing the cosine similarity between retrieved documents and gold documents.

We used accuracy as the evaluation metric for the tasks of Commonsense Reasoning, Fact Checking, and Medical QA. For Open-Domain QA and Multihop QA, we employed token-level F1 score and Exact Match (EM) score. The final RAG score was calculated by averaging the aforementioned five RAG capabilities. We followed Trivedi et al. [52] and sub-sampled up to 500 examples from each dataset.


Searching for Best Practices in Retrieval-Augmented Generation:Results and Analysis

Based on the experimental results presented in Table 11, the following key insights emerge:

- Query Classification Module: This module is referenced and contributes to both effectiveness and efficiency, leading to an average improvement in the overall score from 0.428 to 0.443 and a reduction in latency time from 16.41 to 11.58 seconds per query.

Table 11 shows the results of the search for optimal RAG practices. Modules enclosed in red boxes are under investigation to determine the best method. The underlined method represents the selected technique in the given task and dataset, which achieves the highest score. Optimal scores are highlighted in bold. The average latency is measured in seconds per query. The best scores are highlighted.


Searching for Best Practices in Retrieval-Augmented Generation:Discussion

5 Discussion

5.1 Best Practices for Implementing RAG

According to our experimental findings, we suggest two distinct recipes or practices for implementing RAG systems, each customized to address specific requirements: one focusing on maximizing performance, and the other on striking a balance between efficiency and efficacy.

- Best Performance Practice: To achieve the highest performance, it is recommended to incorporate query classification module, use the “Hybrid with HyDE” method for retrieval, employ monoT5 for rerranking, opt for Reverse for repacking, and leverage Recomp for summarization. This configuration yielded the highest average score of 0.483, albeit with a computationally-intensive process.

- Balanced Efficiency Practice: In order to achieve a balance between performance and efficiency, it is recommended to incorporate the query classification module, implement the Hybrid method for retrieval, use TILDEv2 for reranking, opt for Reverse for repacking, and employ Recomp for summarization. Given that the retrieval module accounts for the majority of processing time in the system, transitioning to the Hybrid method while keeping other modules unchanged can substantially reduce latency while preserving a comparable performance.

5.2 Multimodal Extension

We have extended RAG to multimodal applications. Specifically, we have incorporated text2image and image2text retrieval capabilities into the system with a substantial collection of paired image and textual descriptions as a retrieval source. As depicted in Figure 4, the text2image capability speeds up the image generation process when a user query aligns well with the textual descriptions of stored images (i.e., “retrieval as generation” strategy), while the image2text functionality comes into play when a user provides an image and engages in conversation about the input image. These multimodal RAG capabilities offer the following advantages:

- Groundedness: Retrieval methods provide information from verified multimodal materials, thereby ensuring authenticity and specificity. In contrast, on-the-fly generation relies on models to generate new content, which can occasionally result in factual errors or inaccuracies.

- Efficiency: Retrieval methods are typically more efficient, especially when the answer already exists in stored materials. Conversely, generation methods may require more computational resources to produce new content, particularly for images or lengthy texts.


Searching for Best Practices in Retrieval-Augmented Generation:Content Extraction

Text2Image Retrieval

Image2Text Retrieval

Figure 4: Workflow of multimodal retrieval. The upper section illustrates the text-to-image retrieval process. Initially, a text query is used to find images in the database with the highest similarity. If a high similarity is found, the image is returned directly. If not, an image generation model is employed to create and return an appropriate image. The lower section demonstrates the image-to-text retrieval process. Here, a user-provided image is matched with images in the database to find the highest similarity. If a high similarity is identified, the pre-stored caption of the matching image is returned. Otherwise, an image captioning model generates and returns a new caption.

- Maintainability: Generation models often necessitate careful fine-tuning to tailor them for new applications. In contrast, retrieval-based methods can be improved to address new demands by simply enlarging the size and enhancing the quality of retrieval sources.

We plan to broaden the application of this strategy to include other modalities, such as video and speech, while also exploring efficient and effective cross-modal retrieval techniques.


Searching for Best Practices in Retrieval-Augmented Generation:Conclusion

In this study, we aim to identify optimal practices for implementing retrieval-augmented generation in order to improve the quality and reliability of content produced by large language models. We systematically assessed a range of potential solutions for each module within the RAG framework and recommended the most effective approach for each module. Furthermore, we introduced a comprehensive evaluation benchmark for RAG systems and conducted extensive experiments to determine the best practices among various alternatives. Our findings not only contribute to a deeper understanding of retrieval-augmented generation systems but also establish a foundation for future research.


Searching for Best Practices in Retrieval-Augmented Generation:Limitations

We have evaluated the impact of various methods for fine-tuning LLM generators. Previous studies have demonstrated the feasibility of training both the retriever and generator jointly. We would like to explore this possibility in the future. In this study, we embraced the principle of modularity.


Searching for Best Practices in Retrieval-Augmented Generation:Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foundation of China (No. 62076068).