Searching for Best Practices in Retrieval-Augmented Generation:References

[1] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.

[2] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.

[3] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLICHF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.

[4] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.

[5] Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997, 2023.

[6] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.

[7] Huayang Li, Xivuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110, 2022.

[8] Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in retrieval-augmented text generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 3417–3419, 2022.

[9] Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283, 2023.

[10] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496, 2022.

[11] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.

[12] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.


[13] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303. 08774. URL https://doi.org/10.48550/arXiv.2303.08774.

[14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

[15] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.

[16] Xiaohua Wang, Yuliang Yan, Longtao Huang, Xiaoqing Zheng, and Xuan-Jing Huang. Hallucination detection for generative large language models by bayesian sequential estimation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13561–13571, 2023.

[17] Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678, 2023.

[18] Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. arXiv preprint arXiv:2310.14696, 2023.

[19] Jerry Liu. Llamalndex, 11 2022. URL https://github.com/jerryjliu/llama_index.

[20] Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to anywhere in language models. arXiv preprint arXiv:2310.07554, 2023.

[21] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023.

[22] Huigang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmniagua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023.

[23] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408, 2023.

[24] Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to fetch context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377, 2023.

[25] Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXiv:1910.14424, 2019.

[26] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713, 2020.

[27] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2310.08319, 2023.

[28] Shengyao Zhuang and Guido Zuccon. Tilde: Term independent likelihood model for passage re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1483–1492, 2021.

[29] Shengyao Zhuang and Guido Zuccon. Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108.08513, 2021.

[30] Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen M. Meng, and James R. Glass. Sail: Search-augmented instruction learning. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https: //api.semanticscholar.org/CorpusID:258865283.


[31] Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei A. Zaharia, Ion Stoica, and Joseph E. Gonzalez. Raft: Adapting language model to domain specific rag. ArXiv, abs/2403.11031, 2024.
[32] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyue Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. 2024. URL https://api.semanticscholar.org/CorpusID:267053153.
[33] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. ArXiv, abs/2208.03299, 2022.
[34] Lingxi Zhang, Yue Yu, Kuan Wang, and Chao Zhang. Arl2: Aligning retrievers for black-box large language models via self-guided adaptive relevance labeling. ArXiv, abs/2402.13542, 2024.
[35] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
[36] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. ArXiv, abs/2002.08909, 2020.
[37] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szélvásy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dt: Retrieval-augmented dual instruction tuning. ArXiv, abs/2310.01352, 2023.
[38] Hamed Zamani and Michael Bendersky. Stochastic rag: End-to-end retrieval-augmented generation through expected utility maximization. 2024. URL https://api.semanticscholar.
org/CorpusID:269654348.
[39] Yizheng Huang and Jimmy Huang. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024.
[40] Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for augmented generation: A survey. arXiv preprint arXiv:2303.10866, 2023.
[41] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhenpeng Wang, Yunteg Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.14973, 2024.
[42] Michael Günther, Jackmin Ong, Isabelle Mohr, Aladdine Abdessalem, Tanguy Abel, Mohamad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sutura, Bo Wang, et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923, 2023.
[43] LlamaIndex. Lamaindex website. https://www.llamaindex.com. Accessed: 2024-06-08.
[44] Kunal Sawarkar, Abhilasha Mangal, and Shivam Raj Solanki. Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers. arXiv preprint arXiv:2404.07220, 2024.
[45] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021.
[46] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.
[47] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016.


[48] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
[49] Huiciang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839, 2023.
[50] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Weiyun Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korneev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yiheng Liu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rasih Ubgta, Kalyan Saladi, Alon Schlecht, Ruan Silva, Eric Michael Smith, R. Subramanian, Raia H. Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhenzhu Xu, Ilayn Zarov, Derek Zhuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288, 2023.
[51] ES Shauhl, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In Conference of the European Chapter of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:263152733.
[52] Harsh Trivedi, Nirasan Balsubramamian, Thurau Khot, and Ashish Sabharwal. Whose language, it’s a single-hop question composition. Transactions of the Association for Computational Linguistics, page 559–574, Dec. 2023. URL http://dx.doi.org/10.1162/tacl_a_00475.
[53] Mike Conover, Matt Hayes, Ankith Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/bol/2023/04/12/free-dolly-first-open-commercially-viable-instruction-tuned-llm.
[54] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. Overview of the trec 2019 deep learning track. arXiv, abs/2003.07820, 2020. URL https://api.semanticscholar.org/CorpusID:25234683.
[55] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. Overview of the trec 2020 deep learning track. arXiv, abs/2102.07662, 2021. URL https://api.semanticscholar.org/CorpusID:212737158.
[56] Jimmy Lin, Xueguang Ma, Sheng-Cheih Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356–2362, 2021.
[57] Tom Kwiatkowski, Jennimara Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danieli Espetia, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoeriet, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.
[58] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemover. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv, abs/1705.03551, 2017.
[59] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.06906, 2018.


[60] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers. ArXiv, abs/2204.06092, 2022.

[61] Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018.

[62] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.

[63] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.

[64] J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Loar: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021.

[65] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Cornell University - arXiv, Sep 2020.

[66] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL https://api.semanticscholar.org/CorpusID:3928216.

[67] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Jan 2018. doi: 10.18653/v1/d18-1260. URL http://dx.doi.org/10.18653/v1/d18-1260.

[68] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. ArXiv, abs/1803.05355, 2018. URL https://api.semanticscholar.org/CorpusID:4711425.

[69] Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaiteskill, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen M. Meng, and James R. Glass. Interpretable unified language checking. ArXiv, abs/2304.03728, 2023. URL https://api.semanticscholar.org/CorpusID:258041307.

[70] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. Empirical Methods in Natural Language Processing, Empirical Methods in Natural Language Processing, Oct 2013.

[71] Xanh Ho, A. Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. ArXiv, abs/2011.01060, 2020. URL https://api.semanticscholar.org/CorpusID:226236746.

[72] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, NoahA. Smith, and Mike Lewis. Emergent and nonemergent compositionality gap in language models. Oct 2022.

[73] Qiao Jin, Bhuwan Dhingra, Zhenping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api.semanticscholar.org/CorpusID:220572622.

[74] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-ref: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2301.11511, 2023.


Searching for Best Practices in Retrieval-Augmented Generation:Experimental Details

In this section, we provide detailed experimental settings for each module, covering dataset specifics, training parameters, and any additional experimental results.


Searching for Best Practices in Retrieval-Augmented Generation:A.1 Query Classification

Datasets: We utilized a subset of the Databricks-Dolly-15K [53] and generated additional data using GPT-4. The prompt template for generating questions is shown in Table 14. Implementation Details: We choose BERT-base-multilingual-cased as our classifier, with a batch size of 16 and a learning rate of 1e-5. The evaluation of results is showcased in Table 1.


Searching for Best Practices in Retrieval-Augmented Generation:A.2 Experimental Details of Retrieval Methods

Implementation details of the comparative experiments of different retrieval methods are as below: Datasets: We use the TREC DL 2019 [54] and 2020 [55] passage ranking datasets to evaluate the performance of different retrieval methods. Metrics: Widely-used evaluation metrics for retrieval include mAP, nDCG@10, R@50 and R@1k. Both mAP and nDCG@10 are order-aware metrics that take the ranking of search results into account. In contrast, R@k is an order-unaware metric. We also report the average latency incurred by each method per query. Implementation Details: For sparse retrieval, we use the BM25 algorithm, which relies on the TF-IDF algorithm. For dense retrieval, we employ Contriever as our unsupervised contrastive text encoder. Based on our evaluation of embedding models, we implement our supervised dense retrieval using LLM-embedder. We use the default implementation of BM25 and Contriever from Pyserini [56]. The BM25 index is constructed using Lucene on MS MARCO collections, while the dense vector index is generated with Faiss employing Flat configuration on the same dataset. For query rewriting, we prompt Zephyr-7b-alpha9, a model trained to act as a helpful assistant, to rewrite the original query. For query decomposition, we employ GPT-3.5-turbo-0125 to break down the original query into multiple sub-queries. We closely follow the implementation from HyDE [10], utilizing the more advanced instruction-following language model, GPT-3.5-turbo-instruct, to generate hypothetical answers. The model infers with a default temperature of 0.7, sampling up to a maximum of 512 tokens. Retrieval experiments and evaluation are conducted using the Pyserini toolkit.


Searching for Best Practices in Retrieval-Augmented Generation:A.3 Experimental Details of Reranking Methods

Datasets: Our experiments utilize the MS MARCO Passage ranking dataset, a substantial corpus designed for machine reading comprehension tasks. This dataset comprises over 8.8 million passages and 1 million queries. The training set contains approximately 398M tuples of queries paired with corresponding positive and negative passages, while the development set comprises 6,980 queries, paired with their BM25 retrieval results, and preserves the top-1000 ranked candidate passages for each query. We evaluate the effectiveness of the methods on the development set, as the test set is not publicly available. Metrics: The evaluation metrics MRR@1, MRR@10, MRR@1k and Hit Rate@10 are used. MRR@10 is the official metric proposed by MS MARCO. Implementation Details: We follow and make modifications to the implementation provided by PyGaggle [26] and TILDEv2 [28]. For DLM-based reranking, we use monoT5 [26] based on T5-base, monoBERT [25] based on BERT-large and RankLLAMA [27] based on Llama-2-7b. For TILDE reranking, we use TILDEv2 [29] based on BERT-base. Typically, 50 documents are retrieved as input for the reranking module. The documents remaining after the reranking and repacking phase can be further concentrated by assigning a top-k value or a relevancy score threshold. Results: Reranking results are shown in Table 9. We compare our results with a randomly shuffled ordering and the BM25 retrieval baseline. All reranking methods demonstrate a notable improvement.


Searching for Best Practices in Retrieval-Augmented Generation:Table and Context Description

Table 12: Results of the model augmented with different contexts on various QA datasets.

Context:
- D_∅
- D_g
- D_r
- D_gr

Model:
- M_b, M_g, M_r, M_gr, M_∅, M_ggr

Datasets:
- NQ (Natural Questions)
- TriviaQA
- HotpotQA
- ASQA

Results are shown for different models with corresponding scores on different datasets, displaying trends across the context and average (Avg.).

Scores indicate performance metrics with comparisons across Contexts on the models like NQ, TriviaQA, HotpotQA, ASQA, and the average:
- M_b ranges from 16.49 to 44.78
- M_g ranges from 22.15 to 85.72
- M_r ranges from 21.08 to 36.92
- M_gr ranges from 21.08 to 87.63

Increase in performance across all metrics with observations on approximate equal performance from monoT5 and monoBERT, and RankLLaMa as ascending in latency. TILDEv2 is the fastest with approximately 10 to 20 milliseconds per query.


Searching for Best Practices in Retrieval-Augmented Generation:A.4 Experimental Details of Summarization Methods

Selective Context enhances LLM efficiency by identifying and removing redundant information in the input context. Evaluates informativeness using self-information by a base causal language model, non-query-based, allowing comparison between query-based and non-query-based approaches.

Datasets:
- Natural Questions (NQ)
- TriviaQA
- HotpotQA

Metrics:
- F1 score
- Number of tokens changed post-summarization

Implementation Details:
- Use Llama3-8B-Instruct as the generator model
- Summarization ratio of 0.4
- For extractive methods, importance scores decide sentence arrangements
- Abstractive methods control sentence length according to extractive matches.
- Experiments on NQ test set, TriviaQA test set, and HotpotQA development set.


Searching for Best Practices in Retrieval-Augmented Generation:A.5 Experimental Details of Generator Fine-tuning

Datasets used for fine-tuning AQA and reading comprehension include:
- ASQA
- HotpotQA
- NarrativeQA
- NQ
- SQuAD
- TriviaQA
- TruthfulQA

Utilizes training splits with varying amounts of data to enhance model capability on specific tasks, ensuring diversity and coverage during training.


Searching for Best Practices in Retrieval-Augmented Generation:Context

For example: 1."French. Washington played a crucial role in the American Revolutionary War, leading the Continental Army against the British." Please continue writing the above paragraph. 2."The discovery of the double helix structure of DNA by James Watson and Francis Crick revolutionized the field of genetics, laying the foundation for modern molecular biology and biotechnology." Please continue by discussing recent developments in genetic research, such as CRISPR gene editing, and their potential ethical implications.


Searching for Best Practices in Retrieval-Augmented Generation:Continuation

entries than others, we conducted a random sample). For evaluation, ASQA [60], HotpotQA [59], NQ [57], TriviaQA [58] are used. We evaluate our model on their validation splits or manually split a subset from the training set to avoid overlapping. The exact number of entries in each train and test set is detailed in Table 13.


Searching for Best Practices in Retrieval-Augmented Generation:Table 13

Number of examples in each Dataset used in the fine-tuning experiments. - Dataset / #Train / #Eval - ASQA / 2,900 / 483 - HotpotQA / 15,000 / 7,405 - TriviaQA / 9,000 / 6,358 - NQ / 15,000 / 8,006 - NarrativeQA / 7,000 / -- - SQuAD / 67,000 / -- - TruthfulQA / 817 / --


We use the dataset-provided documents as d_gold for each data entry. To obtain d_random, we sample the context of different entries within the same dataset, to make sure the distributions of d_random and d_gold are roughly similar.


Searching for Best Practices in Retrieval-Augmented Generation:Metrics

We use the ground-truth coverage as our evaluation metric, considering that the answers of QA tasks are relatively short, while the generation length of the model is sometimes hard to limit.


Searching for Best Practices in Retrieval-Augmented Generation:A.6 Experimental Details of Comprehensive Evaluation

Tasks and Datasets We conducted extensive experiments across various NLP tasks and datasets to assess the performance of RAG systems. Specifically: (1) Commonsense Reasoning: We evaluated on MUML [65], ARC-Challenge [66] and OpenbookQA [67] datasets. (2) Fact-Checking: Our evaluation encompassed the FEVER [68] and PubHealth [69] datasets. (3) Open-Domain QA: We assessed on NQ [57], TriviaQA [58], and WebQuestions [70] datasets. (4) Multihop QA: Our evaluation included the HotpotQA [59], 2WikiMultihopQA [71], and MuSiQue [52] datasets. For MuSiQue, we followed the approach outlined in [72] and focused solely on answerable 2-hop questions. (5) Medical QA: We also assessed on the PubMedQA [73] dataset. In each dataset, we randomly sub-sample 500 entries from the test set for our experiments. For datasets without test set, we use devset instead. To assess RAG capabilities, we evenly collect a total of 500 entries from NQ, TriviaQA, HotpotQA, 2WikiMultihopQA and MuSiQue. Each entry is a “question, gold document, gold answer” triple. Metrics We use token-level F1 score and EM score for Open-Domain QA and Multihop QA tasks, and accuracy for others. We use a more lenient EM score, which evaluates performance based on whether the model generations include gold answers instead of strictly exact matching [74]. Towards RAG capabilities evaluation, we adopt four metrics from RAGAs, including Faithfulness, Context Relevancy, Answer Relevancy, and Answer Correctness. Faithfulness measures how factually consistent the generated answer is with the retrieved context. An answer is considered faithful if all claims made can be directly inferred from the provided context. Context Relevancy evaluates how relevant the retrieved context is to the original query. Answer Relevancy assesses the


Searching for Best Practices in Retrieval-Augmented Generation:Pertinence of the Generated Answer

pertinence of the generated answer to the original query. Answer Correctness involves the accuracy of the generated answer when compared to the ground truth. For example, Context Relevancy is calculated from the proportion of sentences within the retrieved context that are relevant for answering the given question to all sentences:

context relevancy = |S| / |Total| (2)

where |S| denotes the number of relevant sentences, |Total| denotes the total number of sentences retrieved. All these metrics are evaluated using the RAGAs framework, with GPT-4 serving as the judge.

Additionally, we compute the cosine similarity between the retrieved document and the gold document as Retrieval Similarity. The retrieved document and gold document are fed into an embedding model, then the resulting embeddings are used to compute the cosine similarity.


Searching for Best Practices in Retrieval-Augmented Generation:Implementation Details

For Open-Domain QA and MultiHop QA datasets, we set the generation model’s maximum new token number to 100 tokens. For other datasets, we set it to 50 tokens. To deal with excessively long retrieved documents, we truncated the documents to 2048 words when evaluating RankLLAMa and LongLLMLingua.

For all datasets, we use greedy decoding during generation. To better compare the capabilities of different RAG modules, we adopt the 0-shot evaluation setting, i.e., no in-context examples are offered. In the multiple choice and fact checking tasks, answers generated by the model may take a variety of forms (e.g., “the answer is A” instead of “A”). Therefore, we preprocess the responses generated by the model, applying regular expression templates to match them with gold labels.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Abstract

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conducted a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG’s significantly lower cost remains a distinct advantage. Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. SELF-ROUTE significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Introduction

Retrieval augmented generation (RAG) has been shown to be a both effective and efficient approach for large language models (LLMs) to leverage external knowledge. RAG retrieves relevant information based on the query and then prompts an LLM to generate a response in the context of the retrieved information. This approach significantly expands LLM’s access to vast amounts of information at a minimal cost. However, recent LLMs like Gemini and GPT-4 have demonstrated exceptional capabilities in understanding long contexts directly. For example, Gemini 1.5 can process up to 1 million tokens. This prompts the need for a systematic comparison between long-context (LC) LLMs and RAG: on one hand, RAG conceptually acts as a prior, regularizing the attention of LLMs onto retrieved segments, thus avoiding the distraction of the irrelevant information and saving unnecessary attention computations; on the other hand, large-scale pretraining may enable LLMs to develop even stronger long-context capabilities. Therefore, we are motivated to compare RAG and LC, evaluating both their performance and efficiency. In this work, we systematically benchmark RAG and LC on various public datasets, gaining a comprehensive understanding of their pros and cons, and ultimately combining them to get the best of both worlds. Different from findings in previous work, we find that LC consistently outperform RAG in almost all settings (when resourced sufficiently). This demonstrates the superior progress of recent LLMs in long-context understanding. Despite the suboptimal performance, RAG remains relevant due to its significantly lower computational cost. In contrast to LC, RAG significantly decreases the input length to LLMs, leading to reduced costs, as LLM API pricing is typically based on the number of input tokens. Figure 1 illustrates a comparison between different setups.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Figure 1 Description

Figure 1: While long-context LLMs (LC) surpass RAG in long-context understanding, RAG is significantly more cost-efficient. Our approach, SELF-ROUTE, combining RAG and LC, achieves comparable performance to LC at a much lower cost.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Content

Our analysis reveals that the predictions from LC and RAG are identical for over 60% of queries. For these queries, RAG can reduce cost without sacrificing performance. Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes various queries to RAG or LC based on model self-reflection. With SELF-ROUTE, we significantly reduce the cost while achieving overall performance comparable to LC. For example, the cost is reduced by 65% for Gemini-1.5-Pro and 39% for GPT-4O. Fig. 1 shows the comparisons of LC, RAG and SELF-ROUTE using three recent LLMs: GPT-4O, GPT-3.5-Turbo and Gemini-1.5-Pro. In addition to quantitative evaluation, we provide a comprehensive analysis comparing RAG and LC, including common failure patterns of RAG, the trade-offs between cost and performance, and the results on additional synthetic datasets. Our analysis serves as a starting point, inspiring future improvements of RAG, and as a empirical guide for building long-context applications using RAG and LC.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:2 Related Work

Long-context LLMs: There has long been efforts for enabling LLMs to handle long contexts (Guo et al., 2022; Beltagy et al., 2020; Chen et al., 2023b). While recent LLMs like Gemini-1.5 (Reid et al., 2024), GPT-4 (Achiam et al., 2023), Claude-3 (Anthropic, 2024) achieve significantly larger context window size, long-context prompting is still expensive due to the quadratic computation cost of transformers regarding to the input token numbers. Recent work proposes methods to reduce cost by prompt compression (Jiang et al., 2023), model distillation (Hsieh et al., 2023), or LLM cascading (Chen et al., 2023a). Retrieval-augmented generation: Augmenting LLMs with relevant information retrieved from various sources (Lewis et al., 2020), i.e., RAG, has been successful in complementing LLMs with external knowledge. RAG achieves good performance on various of tasks like language modeling (Khandelwal et al., 2019; Shi et al., 2023) and QA (Guu et al., 2020; Izacard and Grave, 2020), with a significantly lower computation cost (Borgeaud et al., 2022). Related to but different from our work, recently works augment RAG with correction (Yan et al., 2024), critique (Asai et al., 2023), or verification (Li et al., 2023) to improve retrieval quality on knowledge-intensive tasks. Long-context evaluation: Evaluating long-context models is challenging due to the difficulty in collecting and analyzing long texts. Recent researchers propose both synthetic tests like needle-in-a-haystack (Greg Kamradt, 2023), Ruler (Hsieh et al., 2024), or Counting Stars (Song et al., 2024), and real datasets including LongBench (Bai et al., 2023), ∞Bench (Zhang et al., 2024), L-Eval (An et al., 2023), and others (Shaham et al., 2022; Yuan et al., 2024; Maharana et al., 2024). Evaluating on these datasets, recent works study the performance degradation over various context lengths (Levy et al., 2024; Hsieh et al., 2024), the lost-in-the-middle phenomenon (Liu et al., 2024), and explore solutions (Kuratov et al., 2024). Related to our work, Xu et al. (2023) compare RAG and long-context prompting and find that long-context models still lags behind RAG. This is different from our findings, possibly due to consideration of stronger LLMs and longer contexts in our work.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:3 Benchmarking RAG versus LC

3.1 Datasets and metrics: We evaluate on a subset of datasets from LongBench (Bai et al., 2023) and ∞Bench (Zhang et al., 2024), which are recent benchmarks containing a collection of new and existing datasets for LLM evaluation, covering both synthetic and real texts in multiple languages. LongBench contains a collection of 21 datasets, with an average context length of 7k words. ∞Bench consists of even longer contexts with an average length of 100k tokens. Among the datasets, we mainly focus on tasks that are (a) in English, (b) real, and (c) query-based (e.g. summarization tasks do not contain queries for retrieving relevant information). This results in 7 datasets from LongBench including NarrativeQA (Kocisky et al., 2018), Qasper (Dasigi et al., 2021), MultiFieldQA (Bai et al., 2023), HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), QMSum (Zhong et al., 2021); and 2 datasets from ∞Bench including En-QA and EN.MC. Please refer to Appendix A for more details. Additionally, in Sec. 5.4, we will provide an ablation on a synthetic dataset PassKey from ∞Bench. For evaluation metrics, we report F1 scores for the open-ended QA tasks, accuracy for the multi-choice QA tasks, and ROUGE score for the summarization tasks.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Models and Retrievers

Three latest LLMs are evaluated, including Gemini-1.5-Pro (Reid et al., 2024), GPT-4O (OpenAI, 2024a), and GPT-3.5-Turbo (OpenAI, 2023). Gemini-1.5-Pro is a recent long-context LLM from Google, supporting up to 1 million tokens. GPT-4O, the newest lightweight yet strong LLM from OpenAI, supports 128k tokens. GPT-3.5-Turbo supports 16k tokens. Two retrievers are used in our study: Contriever (Izacard et al., 2021), which is a contrastively trained dense retriever outperforming BM25 on BEIR datasets, and Dragon (Lin et al., 2023), which is a recent generalizable dense retriever achieving high performance in both supervised and zero-shot settings without complex late interaction. Following (Xu et al., 2023), we divide long contexts into chunks of 300 words, and select the top k chunks (default k = 5) based on the cosine similarity of the query embedding and the chunk embeddings. The chunks are ordered by the similarity scores, with the chunk index prepended at the beginning. Since black-box LLMs are pretrained on unknown datasets, the leakage of evaluation datasets may occur. Especially, some of the evaluation datasets are based on Wikipedia, which has likely been seen by LLMs during during. In some cases, we find that model may predict the correct answer using exactly the same words as the groundtruth (e.g. “meticulously”), even when they do not appear in the provided context. In our experiment, we try mitigating this issue by prompting the model to answer “based only on the provided passage” for both RAG and LC. It remains an open question how to address the data leakage issue in LLM evaluation.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Benchmarking Results

We benchmark the performance of LC and RAG across the nine datasets, using three recent LLMs: Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo. Tab. 1 presents the results using the Contriever retriever, where rows *-1 and rows *-2 present the benchmarking results for LC and RAG respectively. Results using the Dragon retriever will be discussed in Sec. 5.3 and Tab. 2. As shown in Tab. 1, LC consistently outperforms RAG for all the three models, with a significant margin. On average, LC surpasses RAG by 7.6% for Gemini-1.5-Pro, 13.1% for GPT-4O, and 3.6% for GPT-3.5-Turbo. Noticeably, the performance gap is more significant for the more recent models (GPT-4O and Gemini-1.5-Pro) compared to GPT-3.5-Turbo, highlighting the exceptional long-context understanding capacity of the latest LLMs. However, there is an exception observed on the two longer datasets from ocBench (i.e., EnQA and En.MC), where RAG achieves higher performance than LC for GPT-3.5-Turbo. This result deviates from the overall trend, likely due to the significantly longer context in these datasets (147k words on average) compared with the limited context window (16k) of GPT-3.5-Turbo. This finding highlights the effectiveness of RAG when the input text considerably exceeds the model’s context window size, emphasizing a specific use case of RAG.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Self-Route

4.1 Motivation As demonstrated in Sec. 3, RAG lags behind long-context LLMs in terms of performance. However, despite this performance gap, we surprisingly find a high degree of overlap in their predictions, as illustrated in Fig. 2. Fig. 2 displays the distribution of the differences between RAG prediction scores S_RAG and LC prediction scores S_LC, specifically S_RAG – S_LC (the scores are multiplied by 100 to be scaled to 1-100). These scores S represent the evaluation of model predictions against the groundtruth. Notably, for most queries, RAG scores and LC scores are highly similar. In fact, for 63% queries, the model predictions are exactly identical; and for 70% queries, the score difference is less than 10 (absolute value).


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:4.2 Self-Route

Based on the above motivation, we propose Self-Route, a simple yet effective method combining RAG and LC to reduce cost while maintaining a performance comparable to LC. Self-Route utilizes LLM itself to route queries based on self-reflection, under the assumption that LLMs are well-calibrated in predicting whether a query is answerable given provided context.

Concretely, our method consists of two steps: a RAG-and-Route step and a long-context prediction step. In the first step, we provide the query and the retrieved chunks to the LLM, and prompt it to predict whether the query is answerable and, if so, generate the answer. This is similar to standard RAG, with one key difference: the LLM is given the option to decline answering with the prompt "Write unanswerable if the query cannot be answered based on the provided text". For the queries deemed answerable, we accept the RAG prediction as the final answer. For the queries deemed unanswerable, we proceed to the second step, providing the full context to the long-context LLMs to obtain the final prediction (i.e., LC).

As our results will demonstrate, most queries can be solved by the first RAG-and-Route step (e.g., 82% for Gemini-1.5-Pro), with only a small portion requiring the following long-context prediction step. Since the RAG-and-Route step only needs the retrieved chunks (e.g., 1.5k tokens) as input, which is significantly shorter than the full contexts (e.g., 10k - 100k tokens), the overall computation cost is substantially reduced. Detailed token count analysis will be provided in the results.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:4.3 Results

Rows *3 to *5 in Tab. 1 present the results of our method, utilizing the three LLMs. Rows *3 report the performance. Rows *4 show the percentage of answerable queries, as predicted in the RAG-and-Route step. Rows *5 display the percentage of tokens used by our method, compared to that of LC. In terms of performance (rows *3), Self-Route significantly outperforms RAG, achieving results comparable to LC. Across all three models, Self-Route surpasses RAG (rows *2) by over 5%. Compared to LC (rows *1), there is a slight performance drop for GPT-4O (-0.2%) and Gemini-1.5-Pro (-2.2%), but an improvement for GPT-3.5-Turbo (+1.7%).

All three LLMs consistently route more than half of queries towards RAG, as shown in rows *4. For Gemini-1.5-Pro, the answerable percentage even reaches 81.74% (row 1-4). This indicates that RAG may answer most queries without the need for LC, confirming our initial motivation.

Due to the high answerable rate, the number of tokens required is significantly reduced (rows *5). For example, GPT-4O uses only 61% tokens compared to LC.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Table 1: Results of Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4O

Table 1 shows that LC consistently outperforms RAG, while Self-Route achieves performance comparable to LC using much less tokens.

Interestingly, the identical predictions are not necessarily correct, as shown by the varying colors representing the average score, i.e., (S_RAG + S_LC)/2. This observation suggests that RAG and LC tend to make not only the same correct predictions but also similar errors.

This finding motivates us to leverage RAG for the majority of queries, reserving computationally more expensive LC for a small subset of queries where it truly excels. By doing so, RAG can significantly reduce computational costs without sacrificing overall performance.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Analysis

5 Analysis
5.1 Ablations of k
Both RAG and SELF-ROUTE rely on the top-k retrieved text chunks. The larger k is, the longer context are fed into LLMs for RAG prediction as well as routing, resulting in different costs versus performances. To study the influence of k, in Fig. 3, we plot the performance and cost (i.e. input token percentage) curves when different ks are used.

Figure 3: Trade-off curves between (a) model performance and (b) token percentage as a function of k.

In terms of performance, for both RAG and SELF-ROUTE, a larger k leads to better performance. While k increases, more and more chunks are fed into the LLMs, thus the performance gradually improves to approach LC. As can be seen in from the curves, the advantage of SELF-ROUTE is the most significant for smaller k. For example, when k = 1, RAG gets from 20.24% while SELF-ROUTE gets 37.9%, while when k is larger than 50, all three methods get similar performance.

However, the trend of cost is not monotonous for SELF-ROUTE. As seen, the cost reaches its minimum at k = 5. This is because when k increases, the cost of RAG (and routing) increases, but more queries are routed to RAG from LC, thus the overall cost may decrease. The sweet point of k might be different for each dataset, e.g. on average, k = 5 has the lowest cost as shown in the curves, but on some datasets, especially ones that contain extractive questions which do not need multi-hop reasoning (like NarrativeQA and QMSum), k = 1 leads to the lowest cost. This indicates that the optimal k depends on the nature of the task, as well as the performance requirement. We encourage future researchers to look for different ks when applying our method to various applications.

5.2 Why does RAG fail?
To gain a better understanding of why RAG lags behind LC, we analyze the failure reasons for the examples that cannot be answered by RAG. We first manually check some examples for which our RAG-and-Route step predicts “unanswerable” and summarize four typical failure reasons, then prompt LLM to classify all the examples.

The four reasons include: (A) The query requires multi-step reasoning so the results of previous steps are needed to retrieve information for later steps, e.g. “What nationality is the performer of song XXX”. (B) The query is general, e.g. “What does the group think about XXX”, which is challenging for the retriever to formulate a good query. (C) The query is long and complex, which is challenging for the retriever to understand. However, answering this kind of questions is arguably, an advantage of LLMs. (D) The query is implicit, demanding a thorough understanding of the entire context. For instance, in a lengthy conversational narrative about a space voyage, a question like “What caused the shadow behind the spaceship?” requires readers to connect the dots and deduce the answer, as there is no explicit mention of the shadow when the cause is revealed.

Using these reasons, we prompt Gemini-1.5-Pro with few-shot in-context examples that we manually annotated, to classify all the unanswerable cases more effectively.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Results on synthetic data

In this study, we mainly focus on real datasets, with a consideration that results on synthetic data, which are artificially created by researchers, may subject to dataset artifacts. We notice some methods that researchers adopted to create synthetic long context datasets may unconsciously, but largely, influence the performance comparison between RAG and LC. For example, here we describe the results on the “PassKey” dataset in ocBench and its variations.

This “PassKey” dataset presents a needle-in-a-haystack test, where a sentence with a passkey (e.g., “the passkey is 123456”) is hidden within chunks of irrelevant text, and the model is asked to answer the question “what is the passkey?”. The task requires strong retrieval capability. On this dataset, RAG achieves 80.34% accuracy, outperforming LC, which gets 65.25% using Gemini-1.5-Pro. However, if the query is slightly modified as “What is the special token hidden inside the texts?”, RAG accuracy sharply drops to only 4.58%, while LC keeps roughly the same (69.32%). Another example: if the chunks contain two passkeys and the query is “Which passkey is larger? First or second?”, then RAG (47.63%) under-performs LC (64.24%) as well. These examples demonstrate that the evaluation results highly subject to artifacts in dataset construction, showing limitation of synthetic testing.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Conclusion

This paper presents a comprehensive comparison of RAG and LC, highlighting the trade-offs between performance and computational cost. While LC demonstrate superior performance in long-context understanding, RAG remains a viable option due to its lower cost and advantages when the input considerably exceeds the model’s context window size. Our proposed method, which dynamically routes queries based on model self-reflection, effectively combines the strengths of both RAG and LC, achieving comparable performance to LC at a significantly reduced cost. We believe our findings contribute valuable insights for the practical application of long-context LLMs and pave the way for future research in optimizing RAG techniques.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Different retrievers

The results using a retriever, Dragon, is shown in Tab. 2 based on Gemini-1.5-Pro. As can be seen, the results are consistent with Contriever, for all of LC, RAG, and Self-Router, showing that our findings are generalizable across retrievers.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Figure 4: Distribution of typical RAG failure reasons

examples into those four categories, plus an “other” option. Fig. 4 shows the distribution of failure reasons on the seven datasets in LongBench. Each dataset may contain different number of RAG failure cases, resulting in various bar heights. The distribution patterns are consistent with the nature of the datasets. For example, the three Wikipedia-based multi-hop reasoning datasets (HotpotQA, 2WikiMQA, MuSiQue) are challenging for RAG because of multi-step retrieval as shown in blue. For NarrativeQA, which are long stories containing a lot of dialogues, most failure cases are due to implicit queries that requires understanding the whole context (shown in green). For QMSum, which is a summarization dataset contains open-ended questions, failures are mostly due to general queries (shown in red). We manually checked the examples classified as “others” and find that most of them are actually multi-step questions, often with ambiguities, which poses challenges for answering.

We hope this failure analysis inspires future improvements of RAG. For example, engaging chain-of-thought (Wei et al., 2022) into RAG may help address the multi-step questions, and revisiting query understanding techniques like query expansion (Lv and Zhai, 2009; Zhai and Lafferty, 2001) may help with the general queries and complex queries. We are also glad to see recent efforts towards the direction (Chan et al., 2024; Ma et al., 2023).


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Table 2: Results for Gemini-1.5-Pro using Dragon retriever

|         | Avg  | Narr | Qasp  | Mult | Hotp | 2Wiki | Musi |  Sum  | En.QA | En.MC |
|--------|------|------|-------|------|------|-------|------|-------|-------|-------|
| 1 LC   | 49.70|  3.276| 47.83 | 52.33| 61.85| 62.96 | 40.22|  70.73| 43.08 |  85.57|
| 2 RAG  | 38.09|  2.191| 44.33 | 53.03| 51.61| 50.05 | 30.47|  19.93|  21.25|  50.22|
| 3 combine | 46.81|  2.580| 43.82 | 54.62| 56.58| 60.62 | 40.66|  20.07|  37.79|  78.60|
| Dragon | 4 RAG ratio | 77.88| 74.00| 84.00 | 97.33| 86.00| 77.00 | 66.00| 95.50| 61.25| 59.83|
| 5 Token ratio | 37.87| 19.31| 54.15 | 34.78| 32.64| 55.65 | 41.86| 16.64|  38.71|  40.83|


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamlal Amadat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.

Anthropic. 2024. Claude 3.5 sonnet. https://www. anthropic.com/news/claude-3-5-sonnet/.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2301.11511.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidao Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610.

Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023b. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15559.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011.

Google. 2024. Gemini pricing. https://ai.google. dev/pricing.

Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/ LLMTest_NeedleInAHaystack.

Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.00654.

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step: outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.05201.

Gautier Izacard, Matthieu Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.

Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.

Huigiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmniqua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. In search of needles in a 10m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Research Paper Entries

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Maman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Xiaoan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Llattrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07838.

Sheng-Chieh Lin, Akari Asai, Minhang Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Yuanhua Lv and ChengXiang Zhai. 2009. Adaptive relevance feedback in information retrieval. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 255–264.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283.

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of lm agents. arXiv preprint arXiv:2402.17753.

OpenAI. 2023. Gpt-3.5-turbo. https://platform.openai.com/docs/models/gpt-3-5-turbo.

OpenAI. 2024a. Gpt-4to. https://openai.com/index/hello-gpt-4o/.

OpenAI. 2024b. Openai-api pricing. https://platform.openai.com/docs/overview.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitriy Lepikhin, Timothy Lillcrapp, Jean-baptiste Alayrac, Raudo Scott, Angeliki Lazaridou, Orhan Firat, Julian Schritterwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Weinan Zhong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2301.03533.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Reimagined augmented black-box language models. arXiv preprint arXiv:2301.12656.

Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramaniam, Evelina Bakhturina, Mohammed Shoyeb, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2301.03025.

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.

Tao Yuan, Neufeir Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhang, Zheyao Tan, Zhuyay Yao, Dahua Lin, Boxun Li, et al. 2024. L-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05135.

Chengxiang Zhai and John Lafferty. 2001. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of the tenth international conference on Information and knowledge management, pages 403–410.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024. Infinity bench: Extending long context evaluation beyond 10k tokens. arXiv preprint arXiv:2402.13718.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multidomain meeting summarization. arXiv preprint arXiv:2104.05938.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Dataset details

We evaluate on 7 datasets from LongBench (Bai et al., 2023). NarrativeQA (Kočiský et al., 2018) is a question answering dataset, where the context is a long story like a novel or a movie script. Qasper (Dasigi et al., 2021) focuses on question answering over academic NLP papers and is annotated by NLP practitioners. MultiFieldQA, originally proposed in LongBench, contains human-annotated QA over documents and articles from multiple sources, including legal documents, government reports, encyclopedias, academic papers, etc. HotpotQA (Yang et al., 2018) contains two-hop questions written by native English speakers that requires reasoning over two related Wikipedia paragraphs in the long context. 2WikiMultihopQA (Ho et al., 2020) contains up to 5-hop questions that are synthesized through manually designed templates, ensuring that they cannot be solved through shortcuts. The questions in MuSiQue (Trivedi et al., 2022) are up to 4-hop, first constructed from single-hop question compositions, and then paraphrased by annotators for linguistic diversity. QMSum (Zhong et al., 2021) is a query-based summarization dataset over meeting scripts from multiple domains. \n\nWe evaluate on 2 datasets from \inftyBench (Zhang et al., 2024). En.QA contains human-annotated question-answer pairs for long novels, with key entity names manually replaced in order to avoid knowledge leakage due to model pretraining. EN.MC is annotated similarly to En.QA, but differs in that the model is presented with four challenging answer choices written by the annotators. \n\nTab. 3 shows the details of the datasets, including the number of queries in each evaluation dataset and the average context length (i.e. number of words).


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Dataset statistics

\n\n|                 | Num. Query | Avg. Length |\n|-----------------|------------|-------------|\n| NarrativeQA     | 200        | 18,395      |\n| Qasper          | 200        | 3,599       |\n| MultiFieldQA    | 150        | 4,539       |\n| HotpotQA        | 200        | 9,133       |\n| 2WikiMultihopQA | 200        | 4,873       |\n| MuSiQue         | 200        | 11,196      |\n| QMSum           | 200        | 10,533      |\n|\n| \inftyBench (Zhang et al., 2024) |||\n| En.QA           | 351        | 150,374     |\n| En.MC           | 229        | 142,622     |\n\nTable 3: Dataset statistics.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Ablations of k

Tab. 4 shows the performance and token ratio for different k, which corresponds to Fig. 3. The performance of LC, which serves as an upper bound, is 45.53. The token ratio is computed the token counts for RAG or Self-Route divided the number of tokens required by LC. \n\n| top-k | Performance         | token ratio           |\n|-------|--------------------|-----------------------|\n|       | RAG    | Self-Route | RAG     | Self-Route |\n| 1     | 20.24  | 41.35      | 5.26    | 39.64      |\n| 5     | 37.92  | 43.33      | 17.02   | 38.63      |\n| 10    | 41.20  | 44.38      | 42.42   | 53.66      |\n| 50    | 44.06  | 45.19      | 95.29   | 102.97     |\n| 100   | 44.12  | 45.23      | 100.32  | 106.59     |\n\nTable 4: Performance and token ratio for different k. This table corresponds to Fig. 3.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Prompts

Tab. 5 shows the prompts for each dataset in our study. The prompts are modified from the released prompts as in LongBench (Bai et al., 2023) and ∞Bench (Zhang et al., 2024). Tab. 6 shows the prompts used in the failure case study as in Sec. 5.2.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Prompts for Each Dataset

NarrativeQA: You are given a story, which can be either a novel or a movie script, and a question. Answer the question as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write "unanswerable". Story: {context} Now, answer the question based on the story as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

Qasper: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Article: {context} Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Question: {input} Answer:

MultiQA: Read the following text and answer briefly. {context} Now, answer the following question based on the above text, only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

HotpotQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

2WikiMQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

MusiQue: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

QMSum: You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences. If the question cannot be answered based on the information in the article, write "unanswerable". Transcript: {context} Now, answer the query based on the above meeting transcript in one or more sentences. If the question cannot be answered based on the information in the article, write "unanswerable". Query: {input} Answer:

EN.QA: Read the book and answer the question. Be very concise in your answer. If the question cannot be answered based on the information in the article, write "unanswerable". {context} Question: {input} Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Answer:

EN.MC: Read the book and answer the question. If the question cannot be answered based on the information in the article, write "unanswerable". {context} Question: {input} {all_classes} Only output the letter of the correct answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The letter of the correct answer is

Table 5: Prompts for each dataset.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Prompt for Failure Case Analysis

You are given some text chunks from an article, and a question. The text chunks are retrieved by an external retriever. Now:

(1) Tell whether the question can be answered based only on the provided text chunks.
(2) If the question can be answered, answer the question based on the texts as concisely as you can, using a single phrase if possible.
(3) If the question cannot be answered, choose the reason from the following:

A. The question needs multistep reasoning, thus it is hard to retrieve all the relevant chunks. For example, "What nationality is the performer of song You Can?" contains two steps: find the performer, then find the nationality of the performer. Other examples include "Where does the director of film Wine Of Morning work at?", "What is another notable work made by the author of Miss Sara Sampson?"

B. The question is a general query, thus it is hard to retrieve relevant chunks. For example, "What did the group think about Dave leaving?" is general because the group may include multiple persons, and they can have different thinkings.

C. The question is long and complex, which is hard for the retriever to encode it to retrieve relevant chunks. For example, "What did Julie Morgan elaborate on the online survey when talking about the evaluations on the legitimacy of the children’s rights, protection and demands?", "The Huskies football team were invited to the Alamo Bowl where they were defeated by a team coached by Art Briles and who played their home games at what stadium?"

D. The question is not explicit and requires comprehensive understanding of the whole story and cannot be solved using retrieval-augmented generation. For example, "What caused the shadow behind Koerber’s ship?" needs a comprehensive understanding of the whole story. Another example like "How many words are there in the article" also requires the complete article.

E. Others.

Keep the above reasons in mind, and choose the most possible reason if you think the question cannot be answered based on the text. Output the results in JSON format.

{in_context_examples}
Text: {context}
Question: {input}
Answer:

Table 6: Prompt for the failure case analysis.