[131] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: An Atlas of Machine Commonsense for IFTTT Reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. 3027–3035.

[132] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQa: Commonsense Reasoning about Social Interactions. arXiv:1904.09728 [cs.CL] https://arxiv.org/abs/1904.09728

[133] Apoorv Saxena, Aditya Tripathi, and Partha P. Talukdar. 2020. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. 4498–4507.

[134] Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Minataka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022. 1604–1619.

[135] Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. 2024. Graph Transformers: A Survey. arXiv:2407.09777 [cs.LG] https://arxiv.org/abs/2407.09777

[136] Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje F. Karlsson, Tingting Ma, Yuzhong Qu, and Chin-Yew Lin. 2022. TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases. arXiv:2210.12925 [cs.CL] https://arxiv.org/abs/2210.12925

[137] Saurabh Srivastava, Milind D Jain, Harshita Jain, Kritik Jaroli, VJ Mayank Patel, and L Khan. 2020. IOT monitoring bin for smart cities. In 3rd Smart Cities Symposium (SCS 2020), Vol. 2020. E3T, 533–536.

[138] Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web. 697–706.

[139] Haitian Sun, Tania Bedrax-Weiss, and William W. Cohen. 2019. PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. 2380–2390.

[140] Haitian Sun, Bhuvan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Cohen. 2018. Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 4231–4242.

[141] Hao Sun, Yang Li, Liwei Deng, Bowen Li, Binyuan Hui, Binhua Li, Yunshi Lan, Yan Zhang, and Yongbin Li. 2023. History Semantic Graph Enhanced Conversational KBQA with Temporal Information Modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. 3521–3533.

[142] Jiashuo Sun, Chengjin Xu, Lunnyuyam Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. arXiv:2307.07697 [cs.CL] https://arxiv.org/abs/2307.07697

[143] Lei Sun, Zhengwei Tao, Youdi Li, and Hiroshi Arakawa. 2024. ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs. arXiv:2404.07677 [cs.CL] https://arxiv.org/abs/2404.07677

[144] Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers). 641–651.

[145] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). 4149–4158.

[146] Dhaval Taunk, Lakshya Khanna, Siri Venkata Pavan Kumar Kandru, Vasudeva Varma, Charu Sharma, and Makarand Tapaswi. 2023. GrapeQA: Graph Augmentation and Pruning to Enhance Question-Answering. In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023. 1138–1144.

[147] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing Text for Joint Embedding of Text and Knowledge Bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. 1499–1509.

[148] Hugo Touvron, Louis Martin, and et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL] https://arxiv.org/abs/2307.09288

[149] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual


Graph Retrieval-Augmented Generation: A Survey:Research Papers and References

[150] Peter Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. arXiv:1710.10903 [stat.ML] https://arxiv.org/abs/1710.10903

[151] Denny Vrandecič and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM 57, 10 (2014), 78–85.

[152] Chaojie Wang, Yishi Xu, Zhong Peng, Chenxi Zhang, Bo Chen, Xinrun Wang, Lei Feng, and Bo An. 2023. kqeping: knowledge-based question answering is a nature chain-of-thought mentor of LLM. arXiv:2401.00426 [cs.CL] https://arxiv.org/abs/2401.00426

[153] Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochaung Han, and Yulia Tsvetkov. 2023. Can Language Models Solve Graph Problems in Natural Language?. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

[154] Jinqiang Wang, Huansheng Ning, Yi Peng, Qikai Wei, Daniel Tesfai, Wenwei Mao, Tao Zhu, and Runhe Huang. 2024. A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations. arXiv:2406.10303 [cs.CL] https://arxiv.org/abs/2406.10303

[155] Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023. Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering. arXiv:2308.13259 [cs.CL] https://arxiv.org/abs/2308.13259

[156] Ruijie Wang, Zheng Li, Danqing Zhang, Qingyu Yin, Tong Zhao, Bing Yin, and Tarek F. Abdelzaher. 2022. RETE: Retrieval-Enhanced Temporal Event Forecasting on Unified Query Product Evolutionary Graph. In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022. 462–472.

[157] Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joloen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024. Large Language Models for Education: A Survey and Outlook. arXiv:2403.18105 [cs.CL] https://arxiv.org/abs/2403.18105

[158] Xintao Wang, Qinanw Wang, Yongting Qib, Jiaqing Liang, Qianyu He, Zhounhong Gu, Yanghua Xiao, and Wei Wang. 2023. KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases. arXiv:2308.11761 [cs.CL] https://arxiv.org/abs/2308.11761

[159] Yuqi Wang, Boran Jiang, Yi Luo, Dawei He, Peng Cheng, and Liangcai Gao. 2024. Reasoning on Efficient Knowledge Graphs:Knowledge Graph Guides Large Language Model for Domain Question Answering. arXiv:2404.10384 [cs.CL] https://arxiv.org/abs/2404.10384

[160] Yu Wang, Nedim Lipka, Ryan A. Rossi, Alexa F. Siu, Ruij Zhang, and Tyler Derr. 2024. Knowledge Graph Prompting for Multi-Document Question Answering. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2024, February 20-27, 2024, Vancouver, Canada. 12906–12914.

[161] Xue Wang, Yun Zhu, Wenqiao Zhang, Yueting Zhuang, Yunfei Li, and Siliang Tang. 2024. Bridging Local Details and Global Context in Text-Attributed Graphs. arXiv:2406.12608 [cs.CL] https://arxiv.org/abs/2406.12608

[162] Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multimodal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM international conference on multimedia. 1437–1445.

[163] Yilin Wen, Zifeng Wang, and Jimeng Sun. 2024. MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models. arXiv:2308.09729 [cs.AI] https://arxiv.org/abs/2308.09729

[164] Sondre Wold, Lilja Ovrelid, and Erik Velldal. 2023. Text-To-KG Alignment: Comparing Current Methods on Classification Tasks. arXiv:2306.02871 [cs.CL] https://arxiv.org/abs/2306.02871

[165] Shangyu Wu, Xing Xiong, Yufei Cui, Haolun Wu, Can Chen, Ye Yuan, Lianming Huang, Xue Liu, Tei-Wei Kuo, Nan Guan, and Chun Jason Xue. 2024. Retrieval-Augmented Generation for Natural Language Processing: A Survey. arXiv:2407.13193 [cs.CL] https://arxiv.org/abs/2407.13193

[166] Shirley Wu, Shiya Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. 2024. STaRk: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases. arXiv:2404.13207 [cs.IR] https://arxiv.org/abs/2404.13207

[167] Taiqiang Wu, Xingyu Bai, Weigang Guo, Weijie Liu, Siheng Li, and Yujui Yang. 2023. Modelling Fine-grained Information via Knowledge-aware Hierarchical Graph for Zero-shot Entity Retrieval. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 - 3 March 2023. 1021–1029.

[168] Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, Anhuan Xie, and Wei Song. 2023. Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering. arXiv:2309.11206 [cs.CL] https://arxiv.org/abs/2309.11206


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

[169] Zhento Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, 2905–2909.
[170] An Yang, Basoong Yang, and et al. 2024. Qwen2 Technical Report. arXiv:2407.10671 [cs.CL] https://arxiv.org/abs/2407.10671
[171] Rui Yang, Haoran Liu, Edison Marese-Taylor, Qingcheng Zeng, Yu He Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Cavleere, Yutaka Matsuo, and Irene Li. 2024. KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques. arXiv:2403.05881 [cs.CL] https://arxiv.org/abs/2403.05881
[172] Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen tau Yih, and Xin Luna Dong. 2024. CRAG - Comprehensive RAG Benchmark. arXiv:2406.04744 [cs.CL] https://arxiv.org/abs/2406.04744
[173] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 2369–2380.
[174] Mohammad Yani and Adila Alfa Krisnadhi. 2021. Challenges, Techniques, and Trends of Simple Knowledge Graph Question Answering: A Survey. Inf. 12, 7 (2021), 271.
[175] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. 535–546.
[176] Ruosong Xie, Ciaji Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. 2024. Language is All a Graph Needs. arXiv:2308.07134 [cs.CL] https://arxiv.org/abs/2308.07134
[177] Xi Ge, Samuel Yaurez, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. arXiv preprint arXiv:2109.06876 (2021).
[178] Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The Value of Semantic Parse Labeling for Knowledge Base Question Answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers.
[179] Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang, Zhiguo Wang, and Bing Xiang. 2023. DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
[180] Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. Evaluation of Retrieval-Augmented Generation: A Survey. arXiv:2405.07437 [cs.CL] https://arxiv.org/abs/2405.07437
[181] Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. 5773–5784.
[182] Mengmei Zhang, Mingwei Sun, Peng Wang, Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. 2024. GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks. In Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024. 1003–1014.
[183] Qinggang Zhang, Junnan Dong, Hao Chen, Daochen Zha, Zailiany Yu, and Xiao Huang. 2024. KnowGPT: Knowledge Graph based Prompting for Large Language Models. arXiv:2312.06185 [cs.CL] https://arxiv.org/abs/2312.06185
[184] Xiukun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022. GreaseLM: Graph REasoning Enhanced Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
[185] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018. Variational Reasoning for Question Answering With Knowledge Graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-8, 2018. 6069–6076.
[186] Jiannan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michel Bronstein, Zhaocheng Zhu, and Jian Tang. 2023. GraphText: Graph Reasoning in Text Space. arXiv:2310.01089 [cs.CL] https://arxiv.org/abs/2310.01089


Graph Retrieval-Augmented Generation: A Survey:References

[187] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. 2024. Retrieval-Augmented Generation for AI-Generated Content: A Survey. arXiv:2402.19473 [cs.CV] https://arxiv.org/abs/2402.19473

[188] Yanxin Zheng, Wensheng Gan, Zefeng Chen, Zhenlian Qi, Qian Liang, and Philip S. Yu. 2024. Large Language Models for Medicine: A Survey. arXiv:2405.13055 [cs.CL] https://arxiv.org/abs/2405.13055

[189] Yun Zhu, Yaoke Wang, Haizhou Shi, and Siliang Tang. 2024. Efficient Tuning and Inference for Large Language Models on Textual Graphs. arXiv:2401.15569 [cs.CL] https://arxiv.org/abs/2401.15569


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation of Retrieval-Augmented Generation: A Survey

Authors: Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, Zhaofeng Liu

Affiliations:
1. Tencent Company
2. McGill University
3. State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
Emails: hao.yu2@mail.mcgill.ca, gar@mail.ustc.edu.cn, {shiweitong7,zhaofengliu}@tencent.com, {kkzhang08,qiliuql}@ustc.edu.cn

---

**Abstract**

Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct a Unified Evaluation Process of RAG (υεπρα) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.

---

**1 Introduction**

Retrieval-Augmented Generation (RAG) efficiently enhances the performance of generative language models through integrating information retrieval techniques. It addresses a critical challenge faced by standalone generative language models: the tendency to produce responses that, while plausible, may not be grounded in facts. By retrieving relevant information from external sources, RAG significantly reduces the incidence of hallucinations or factually incorrect outputs, thereby improving the content's reliability and richness. This fusion of retrieval and generation capabilities enables the creation of responses that are not only contextually appropriate but also informed by the most current and accurate information available, making RAG a development in the pursuit of more intelligent and versatile language models.

**Paper Homepage:** https://github.com/YHPeter/Awesome-RAG-Evaluation

---

Corresponding Author: [Hao Yu]


Evaluation of Retrieval-Augmented Generation: A Survey:Introduction to RAG Systems

Numerous studies of RAG systems have emerged from various perspectives since the advent of Large Language Models (LLMs) [55][45][59][42][41][69][16]. The RAG system comprises two primary components: Retrieval and Generation. The retrieval component aims to extract relevant information from various external knowledge sources. It involves two main phases, indexing and searching. Indexing organizes documents to facilitate efficient retrieval, using either inverted indexes for sparse retrieval or dense vector encoding for dense retrieval [16][12][38]. The searching component utilizes these indexes to fetch relevant documents on the user's query, often incorporating the optional rerankers [43][9][52] to refine the ranking of the retrieved documents. The generation component utilizes the retrieved content and question query to formulate coherent and contextually relevant responses with the prompting and inferencing phases. As the “Emerging” ability [59] of LLMs and the breakthrough in aligning human commands [42], LLMs are the best performance choices model for the generation stage. Prompting methods like Chain of Thought (CoT) [60], Tree of Thought [65], Rephrase and Respond (RaR) [8] guide better generation results. In the inferencing step, LLMs interpret the prompted input to generate accurate and in-depth responses that align with the query's intent and integrate the extracted information [35][9] without further finetuning, such as fully finetuning [16][17][68] or LoRA [21]. Appendix A details the complete RAG structure. Figure 1 illustrates the structure of the RAG systems as mentioned.

Fig. 1: The structure of the RAG system with retrieval and generation components and corresponding four phases: indexing, search, prompting, and inferencing. The pairs of “Evaluable Outputs” (EOs) and “Ground Truths” (GTs) are highlighted in red frame and green frame, with brown dashed arrows.


Evaluation of Retrieval-Augmented Generation: A Survey:Importance of Evaluating RAG

The importance of evaluating RAG is increasing in parallel with the advancement of RAG-specific methodologies. On the one hand, RAG is a complex system intricately tied to specific requirements and language models, resulting in various evaluation methods, indicators, and tools, particularly given the black-box LLM generation. Evaluating RAG systems involves specific components and the complexity of the overall system assessment. On the other hand, the complexity of RAG systems is further compounded.


by the external dynamic database and the various downstream tasks, such as content creation or open domain question answering [16, 70]. These challenges necessitate the development of comprehensive evaluation metrics that can effectively capture the interplay between retrieval accuracy and generative quality [217]. To clarify the elements further, we try to address the current gaps in the area, which differs from the prior RAG surveys [74, 16, 24] that predominantly collected specific RAG methods or data. We have compiled 12 distinct evaluation frameworks, encompassing a range of aspects of the RAG system. Following the procedure of making benchmarks, we analyze through targets, datasets and metrics mentioned in these benchmarks and summarize them into A Unified Evaluation Process of RAG (Auepora) as three corresponding phases.

For this paper, we contribute in the following aspects:

1. **Challenge of Evaluation:** This is the first work that summarizes and classifies the challenges in evaluating RAG systems through the structure of RAG systems, including three parts retrieval, generation, and the whole system.

2. **Analysis Framework:** In light of the challenges posed by RAG systems, we introduce an analytical framework, referred to as A Unified Evaluation Process of RAG (Auepora), which aims to elucidate the unique complexities inherent to RAG systems and guide for readers to comprehend the effectiveness of RAG benchmarks across various dimensions.

3. **RAG Benchmark Analysis:** With the help of Auepora, we comprehensively analyze existing RAG benchmarks, highlighting their strengths and limitations and proposing recommendations for future developments in RAG system evaluation.


Evaluation of Retrieval-Augmented Generation: A Survey:2 Challenges in Evaluating RAG Systems

Evaluating hybrid RAG systems entails evaluating retrieval, generation and the RAG system as a whole. These evaluations are multifaceted, requiring careful consideration and analysis. Each of them encompasses specific difficulties that complicate the development of a comprehensive evaluation framework and benchmarks for RAG systems.

**Retrieval** The retrieval component is critical for fetching relevant information that informs the generation process. One primary challenge is the dynamic and vast nature of potential knowledge bases, ranging from structured databases to the entire web. This vastness requires evaluation metrics that can effectively measure the precision, recall, and relevance of retrieved documents in the context of a given query [52, 32]. Moreover, the temporal aspect of information, where the relevance and accuracy of data can change over time, adds another layer of complexity to the evaluation process [6]. Additionally, the diversity of information sources and the possibility of retrieving misleading or low-quality information pose significant challenges in assessing the effectiveness of filtering and selecting the most pertinent information [39]. The traditional evaluation indicators for retrieval, such as Recall and Precision, cannot fully capture the nuances of RAG retrieval systems, necessitating the development of more nuanced and task-specific evaluation metrics [49].


Evaluation of Retrieval-Augmented Generation: A Survey:Generation

The generation component, powered by LLMs, produces coherent and contextually appropriate responses based on the retrieved content. The challenge here lies in evaluating the faithfulness and accuracy of the generated content to the input data. This involves not only assessing the factual correctness of responses but also their relevance to the original query and the coherence of the generated text [75,49]. The subjective nature of certain tasks, such as creative content generation or open-ended question answering, further complicates the evaluation, as it introduces variability in what constitutes a "correct" or "high-quality" response [48].


Evaluation of Retrieval-Augmented Generation: A Survey:RAG System as a Whole

Evaluating the whole RAG system introduces additional complexities. The interplay between the retrieval and generation components means that the entire system’s performance cannot be fully understood by evaluating each component in isolation [49,14]. The system needs to be assessed on its ability to leverage retrieved information effectively to improve response quality, which involves measuring the added value of the retrieval component to the generative process. Furthermore, practical considerations such as response latency and the ability to handle ambiguous or complex queries are also crucial for evaluating the system’s overall effectiveness and usability [396].


Evaluation of Retrieval-Augmented Generation: A Survey:Conclusion

Evaluating the target shift from traditional absolute numeric metrics to multi-source and multi-target generation evaluation, along with the intricate interplay between retrieval and generation components, poses significant challenges. [5,50] Searches in a dynamic database may lead to misleading results or contradict the facts. Diverse and comprehensive datasets that accurately reflect real-world scenarios are crucial. Challenges also arise in the realm of metrics, encompassing generative evaluation criteria for distinct downstream tasks, human preferences, and practical considerations within the RAG system. Most prior benchmarks predominantly tackle one or several aspects of the RAG assessment but lack a comprehensive, holistic analysis.


Evaluation of Retrieval-Augmented Generation: A Survey:A Unified Evaluation Process of RAG (Auepora)

To facilitate a deeper understanding of RAG benchmarks, we introduce A Unified Evaluation Process of RAG (Auepora), which focuses on three key questions of benchmarks: What to Evaluate? How to Evaluate? How to Measure? which correlated to Target, Dataset, and Metric respectively. We aim to provide a clear and accessible way for readers to comprehend the complexities and nuances of RAG benchmarking. The Target module is intended to determine the evaluation direction. The Dataset module facilitates the comparison of various data constructions in RAG benchmarks. The final module, Metrics, introduces new metrics that correspond to specific targets and datasets used during evaluation. Overall, it is designed to provide a systematic methodology for assessing the effectiveness of RAG systems across various aspects by covering all possible pairs at the beginning between the “Evaluable Outputs” (EOs) and “Ground Truths” (GTs). In the following section, we will explain thoroughly Auepora and utilize it for introducing and comparing the RAG benchmarks.


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Target

The combination of EOs and GTs in the RAG system can generate all possible targets, which is the fundamental concept of the Auepora (as shown in Figure 1). Once identified, these targets can be defined based on a specific pair of EOs or EO with GT, as illustrated in Figure 2 and used to analyze all aspects of current RAG benchmarks.

Retrieval The EOs are the relevant documents for evaluating the retrieval component depending on the query. Then we can construct two pairwise relationships for the retrieval component, which are Relevant Documents ↔ Query, Relevant Documents ↔ Documents Candidates.

- Relevance (Relevant Documents ↔ Query) evaluates how well the retrieved documents match the information needed expressed in the query. It measures the precision and specificity of the retrieval process.
- Accuracy (Relevant Documents ↔ Documents Candidates) assesses how accurate the retrieved documents are in comparison to a set of candidate documents. It is a measure of the system’s ability to identify and score relevant documents higher than less relevant or irrelevant ones.

Generation The similar pairwise relations for the generation components are listed below. The EOs are the generated text and phrased structured content. Then we need to compare these EOs with the provided GTs and labels.

- Relevance (Response ↔ Query) measures how well the generated response aligns with the intent and content of the initial query. It ensures that the response is related to the query topic and meets the query’s specific requirements.
- Faithfulness (Response ↔ Relevant Documents) evaluates if the generated response accurately reflects the information contained within the relevant documents and measures the consistency between generated content and the source documents.
- Correctness (Response ↔ Sample Response) Similar to the accuracy in the retrieval component, this measures the accuracy of the generated response against a sample response, which serves as a ground truth. It checks if the response is correct in terms of factual information and appropriate in the context of the query.


Evaluation of Retrieval-Augmented Generation: A Survey:Table: Evaluation Targets and Metrics

The targets of Retrieval and Generation components are introduced. Table 1 lists the relative work on improving and evaluating RAG and its benchmarks cut off in June.

Table 1: The evaluating targets and corresponding metrics across various frameworks for evaluating RAG systems. The presentation distinguishes between the core areas of Retrieval and Generation considered in the evaluation. The different aspects of the evaluation are set as different colours in the table: Relevance, Accuracy of Retrieval and Faithfulness, Correctness and Relevance of Generation. The consideration of the Additional Requirements beyond the retrieval and generation component is also collected. Noted that quite a few of the works employed multiple methods or evaluated multiple aspects simultaneously.

| Category        | Framework            | Time   | Raw Targets                                           | Retrieval                 | Generation                     |
|-----------------|----------------------|--------|-------------------------------------------------------|---------------------------|---------------------------------|
| Tool            | TruEra RAG Trial [54]| 2023.10| Context Relevance                                      | LLM as a Judge            | LLM as a Judge                  |
|                 |                      |        | Answer Relevance                                       |                           |                                 |
|                 |                      |        | Groundedness                                           |                           |                                 |
|                 |                      |        | Accuracy                                               |                           |                                 |
|                 |                      |        | Faithfulness                                           |                           |                                 |
|                 |                      |        | Execution Time                                         |                           |                                 |
|                 |                      |        | Embed. CosDistance                                     |                           |                                 |
|                 |                      |        | Correctness                                            |                           |                                 |
| Tool            | LangChain Bench. [32]| 2023.11|                                                       | Accuracy                  | LLM as a Judge                  |
| Tool            | Databricks Eval [38] | 2023.12| Readability                                           | -                         | LLM as a Judge                  |
|                 |                      |        | Comprehensiveness                                     |                           |                                 |
| Benchmark       | RAGAs [14]           | 2023.09| Context Relevance                                      | LLM as a Judge            | LLM Gen + CoSim                 |
|                 |                      |        | Answer Relevance                                       |                           | LLM as a Judge                  |
| Benchmark       | RECALL [38]          | 2023.11| Response Quality                                       | -                         | BLEU, ROUGE-L                   |
|                 |                      |        | Robustness                                             |                           |                                 |
| Benchmark       | ARES [49]            | 2023.11| Answer Faithfulness                                    | LLM + Classifier          | LLM + Classifier                |
|                 |                      |        | Answer Relevance                                       |                           |                                 |
|                 |                      |        | Information Integration                                |                           |                                 |
| Benchmark       | RGB [6]              | 2023.12| Noise Robustness                                       | -                         | Accuracy                        |
|                 |                      |        | Negative Rejection                                     |                           |                                 |
|                 |                      |        | Counterfactual Robustness                              |                           |                                 |
| Benchmark       | MultiHop-RAG [52]    | 2024.01| Retrieval Quality                                      | MAP, MRR, H@k            | LLM as a Judge                  |
|                 |                      |        | Response Correctness                                   |                           |                                 |
| Benchmark       | CRUD-RAG [39]        | 2024.02| CREATE, READ, UPDATE, DELETE                          | -                         | ROUGE, BLEU, RAGQuestEval       |
| Benchmark       | MedRAG [61]          | 2024.02| Accuracy                                               | -                         | Accuracy                        |
|                 |                      |        | Consistency                                            |                           |                                 |
|                 |                      |        | Correctness                                            |                           |                                 |
| Benchmark       | FeB4RAG [57]         | 2024.02| Clarity                                               | -                         | Human Evaluation                |
|                 |                      |        | Coverage                                               |                           |                                 |
| Benchmark       | CDQA [62]            | 2024.03| Accuracy                                               | -                         | F1                              |
| Benchmark       | DomainRAG [58]       | 2024.06| Correctness                                            | F1, Exact-Match          | Rouge-L                         |
|                 |                      |        | Faithfulness                                           |                           |                                 |
|                 |                      |        | Noise Robustness                                       |                           | LLM as a Judge                  |
|                 |                      |        | Structural Output                                      |                           |                                 |
| Benchmark       | ReEval [65]          | 2024.06| Hallucination                                          | -                         | LLM as a Judge                  |
|                 |                      |        |                                                       |                           | Human Evaluation                |
| Research        | FiD-Light [6]        | 2023.07| Latency                                               |                           |                                 |
| Research        | Diversity Reranker 6]| 2023.08| Diversity                                              | Cosine Distance           |                                 |

Notes:
- Noted that quite a few of the works employed multiple methods or evaluated multiple aspects simultaneously.


2024. Table 1 portrays this information, where each evaluation criterion is represented by a different colour. For example, FeB4RAG [57], the fourth from the last, has posited four standards based on [17] that comprise Consistency, Correctness, Clarity, and Coverage. Correctness is equivalent to accuracy in retrieval, and Consistency is tantamount to faithfulness in the generation component. While accuracy in retrieval gauges the correctness of the retrieved information, we posit that Coverage pertains to the coverage rate and is more associated with diversity. Therefore, we consider Coverage to be linked with diversity and an additional requirement in our proposed evaluation framework, which will be introduced subsequently. The remaining standard, Clarity, is also classified as an additional requirement in our proposed framework. The other tools and benchmarks are processed similarly.

Tools and benchmarks offer varying degrees of flexibility in evaluating datasets for RAG systems. Tools, which specify only evaluation targets, provide a versatile framework capable of constructing complete RAG applications and evaluation pipelines, as seen in works like [54 52 33]. Benchmarks, on the other hand, focus on different aspects of RAG evaluation with specific emphasis on either retrieval outputs or generation targets. For instance, RAGAs [14] and ARES [49] assess the relevance of retrieval documents, while RGB and MultiHop-RAG [6 52] prioritize accuracy, necessitating comparison with GTs. The [66] focuses on the Hallucination, which is a combination of faithfulness and correctness. All benchmarks consider generation targets due to their critical role in RAG systems, though their focus areas vary.

Additional Requirement
In addition to evaluating the two primary components outlined, a portion of the works also addressed some additional requirements of RAG (Black and Italics targets in Table 2). The requirements are as follows:

- Latency [20 32] measures how quickly the system can find information and respond, crucial for user experience.
- Diversity [4 32] checks if the system retrieves a variety of relevant documents and generates diverse responses.
- Noise Robustness [6] assesses how well the system handles irrelevant information without affecting response quality.
- Negative Rejection [6] gauges the system’s ability to refrain from providing a response when the available information is insufficient.
- Counterfactual Robustness [6] evaluates the system’s capacity to identify and disregard incorrect information, even when alerted about potential misinformation.
- More: For more human preferences considerations, there can be more additional requirements, such as readability [57 33], toxicity, perplexity [33], etc.

For the exception, CRUD-RAG [39] introduces a comprehensive benchmark addressing the broader spectrum of RAG applications beyond question-answering, categorized into Create, Read, Update, and Delete scenarios. This benchmark evaluates RAG systems across diverse tasks, including text continuation, question answering, hallucination modification, and multi-document summarization. It offers insights for optimizing RAG technology across different scenarios. DomainRAG [58] identifies six complex abilities for RAG systems: conversational, structural information, faithfulness,


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Dataset

In Table 2, distinct benchmarks employ varying strategies for dataset construction, ranging from leveraging existing resources to generating entirely new data tailored for specific evaluation aspects. Several benchmarks draw upon the part of KILT (Knowledge Intensive Language Tasks) benchmark [44] (Natural Questions [29], HotpotQA [63], and FEVER [53]) and other established datasets such as SuperGLUE [56] (MultiRC [10], and ReCoRD [71]) [49]. However, the drawback of using such datasets can’t solve the challenges in dynamic real-world scenarios. A similar situation can be observed in WikiEval, from Wikipedia pages post 2022, constructed by RAGAs [14].

The advent of powerful LLMs has revolutionized the process of dataset construction. With the ability to design queries and ground truths for specific evaluation targets using these frameworks, authors can now create datasets in the desired format with ease. Benchmarks like RGB, MultiHop-RAG, CRUD-RAG, and CDQA [6][52][39][62] have taken this approach further by building their own datasets using online news articles to test RAG systems’ ability to handle real-world information beyond the training data of LM frameworks. Most recently, DomainRAG [58] combines various types of QA datasets with single-doc, multi-doc, single-round, and multi-round. These datasets are generated from the yearly changed information from the college website for admission and enrollment, which forces the LLMs to use the provided and updated information.


Evaluation of Retrieval-Augmented Generation: A Survey:Table 2: Evaluation Datasets for Benchmarks

The evaluation datasets used for each benchmark. The dataset without citation was constructed by the benchmark itself.

| Benchmark | Dataset |
|---|---|
| RAGAs [14] | WikiEval |
| RECALL [38] | EventKG [19], U [22] |
| ARES [49] | FEVER [53], WoW [11] |
| RGB [6] | Generated (Source: News) |
| MultiHop-RAG [52] | Generated (Source: News) |
| CRUD-RAG [39] | Generated (Source: News) |
| MedRAG [61] | MIRAGE |
| FebRAG [57] | Feb4RAG, BEIR [26] |
| CDQA [62] | Generation (Source: News), Labeller |
| DomainRAG [58] | Generation (Source: College Admission Information) |
| ReEval [66] | RealTimeQA [27], NQ [15][29] |


In summary, the creation and selection of datasets are crucial for evaluating RAG systems. Datasets tailored for specific metrics or tasks improve evaluation accuracy and guide the development of adaptable RAG systems for real-world information needs.

3.3 Evaluation Metric (How to quantify?)

Navigating the intricate terrain of evaluating RAG systems necessitates a nuanced understanding of the metrics that can precisely quantify the evaluation targets. However, creating evaluative criteria that align with human preferences and address practical considerations is challenging. Each component within the RAG systems requires a tailored evaluative approach that reflects its distinct functionalities and objectives.

Retrieval Metrics Various targets can be evaluated with various metrics that correspond to the given datasets. This section will introduce several commonly used metrics for retrieval and generation targets. The metrics for additional requirements can also be found in these commonly used metrics. The more specifically designed metrics can be explored in the original paper via Table 1 as a reference.

For the retrieval evaluation, the focus is on metrics that can accurately capture the relevance, accuracy, diversity, and robustness of the information retrieved in response to queries. These metrics must not only reflect the system’s precision in fetching pertinent information but also its resilience in navigating the dynamic, vast, and sometimes misleading landscape of available data. The deployment of metrics like Misleading Rate, Mistake Reappearance Rate, and Error Detection Rate within the [38] benchmark underscores a heightened awareness of RAG systems’ inherent intricacies. The integration of MAP@K, MRR@K, and Tokenization with F1 into benchmarks like [52,62] mirrors a deepening comprehension of traditional retrieval’s multifaceted evaluation. While the [17] also emphasizes that this ranking-based evaluation methodology is not unsuitable for the RAG system, and should have more RAG-specific retrieval evaluation metrics. These metrics not only capture the precision and recall of retrieval systems but also account for the diversity and relevance of retrieved documents, aligning with the complex and dynamic nature of information needs in RAG systems. The introduction of LLMs as evaluative judges, as seen in [14], further underscores the adaptability and versatility of retrieval evaluation, offering a comprehensive and context-aware approach to assessing retrieval quality.

Non-Rank Based Metrics often assess binary outcomes—whether an item is relevant or not—without considering the position of the item in a ranked list. Notice, that the following formula is just one format of these metrics, the definition of each metric may vary by the different evaluating tasks.
- **Accuracy** is the proportion of true results (both true positives and true negatives) among the total number of cases examined.
- **Precision** is the fraction of relevant instances among the retrieved instances,

\[
\text{Precision} = \frac{TP}{TP + FP}
\]

where \(TP\) represents true positives and \(FP\) represents false positives.


Evaluation of Retrieval-Augmented Generation: A Survey:Rank-Based Metrics

- Recall at k (Recall@k) is the fraction of relevant instances that have been retrieved over the total amount of relevant cases, considering only the top k results. \[ \text{Recall@k} = \frac{|RD \cap Top_{pkd}|}{|RD|} \] where RD is the relevant documents, and Topk is the top-k retrieved documents.

- Mean Reciprocal Rank (MRR) is the average of the reciprocal ranks of the first correct answer for a set of queries. \[ MRR = \frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{rank_i} \] where |Q| is the number of queries and ranki is the rank position of the first relevant document for the i-th query.

- Mean Average Precision (MAP) is the mean of the average precision scores for each query. \[ MAP = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{\sum_{k=1}^{n}(P(k) \times rel(k))}{|\text{relevant documents}|_q} \] where P(k) is the precision at cutoff k in the list, rel(k) is an indicator function equaling 1 if the item at rank k is a relevant document, 0 otherwise, and n is the number of retrieved documents.


Evaluation of Retrieval-Augmented Generation: A Survey:Generation Metrics

In the realm of generation, evaluation transcends the mere accuracy of generated responses, venturing into the quality of text in terms of coherence, relevance, fluency, and alignment with human judgment. This necessitates metrics that can assess the nuanced aspects of language production, including factual correctness, readability, and user satisfaction with the generated content. The traditional metrics like BLEU, ROUGE, and F1 Score continue to play a crucial role, emphasizing the significance of precision and recall in determining response quality. Yet, the advent of metrics such as Misleading Rate, Mistake Reappearance Rate, and Error Detection Rate highlights an evolving understanding of RAG systems’ distinct challenges [38].

The evaluation done by humans is still a very significant standard to compare the performance of generation models with one another or with the ground truth. The approach of employing LLMs as evaluative judges [75] is a versatile and automatic method for quality assessment, catering to instances where traditional ground truths may be elusive [14]. This methodology benefits from employing prediction-powered inference (PPI) and context relevance scoring, offering a nuanced lens through which LLM output can be assessed [49]. The strategic use of detailed prompt templates ensures a guided assessment aligned with human preferences, effectively standardizing evaluations across various content dimensions [1]. This shift towards leveraging LLMs


as arbiters mark a significant progression towards automated and context-responsive evaluation frameworks, enriching the evaluation landscape with minimal reliance on reference comparisons.

- ROUGE Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [37] is a set of metrics designed to evaluate the quality of summaries by comparing them to human-generated reference summaries. ROUGE can be indicative of the content overlap between the generated text and the reference text. The variants of ROUGEs measure the overlap of n-grams (ROUGE-N, ROUGGE-W), word subsequences (ROUGE-L, ROUGGE-S), and word pairs between the system-generated summary and the reference summaries.

- BLEU Bilingual Evaluation Understudy (BLEU) [43] is a metric for evaluating the quality of machine-translated text against one or more reference translations. BLEU calculates the precision of n-grams in the generated text compared to the reference text and then applies a brevity penalty to discourage overly short translations. BLEU has limitations, such as not accounting for the fluency or grammaticality of the generated text.

- BertScore BertScore [72] leverages the contextual embedding from pre-trained transformers like BERT to evaluate the semantic similarity between generated text and reference text. BertScore computes token-level similarity using contextual embedding and produces precision, recall, and F1 scores. Unlike n-gram-based metrics, BertScore captures the meaning of words in context, making it more robust to paraphrasing and more sensitive to semantic equivalence.

- LLM as a Judge Using “LLM as a Judge” for evaluating generated text is a more recent approach. [75] In this method, LLMs are used to score the generated text based on criteria such as coherence, relevance, and fluency. The LLM can be optionally finetuned on human judgments to predict the quality of unseen text or used to generate evaluations in a zero-shot or few-shot setting. This approach leverages the LLM’s understanding of language and context to provide a more nuanced text quality assessment. For instance, [1] illustrates how providing LLM judges with detailed scoring guidelines, such as a scale from 1 to 5, can standardize the evaluation process. This methodology encompasses critical aspects of content assessment, including coherence, relevance, fluency, coverage, diversity, and detail - both in the context of answer evaluation and query formulation.

Additional Requirements These additional requirements, such as latency, diversity, noise robustness, negative rejection, and counterfactual robustness, are used to ensure the practical applicability of RAG systems in real-world scenarios aligned with human preference. This section delves into the metrics used for evaluating these additional requirements, highlighting their significance in the comprehensive assessment of RAG systems.

Latency measures the time taken by the RAG system to finish the response of one query. It is a critical factor for user experience, especially in interactive applications such as chatbots or search engines [20]. Single Query Latency: The mean time is taken to process a single query, including both retrieval and generating phases.


Evaluation of Retrieval-Augmented Generation: A Survey:Diversity and Robustness

Diversity evaluates the variety and breadth of information retrieved and generated by the RAG system. It ensures that the system can provide a wide range of perspectives and avoid redundancy in responses. Cosine Similarity / Cosine Distance: The cosine similarity/distance calculates embeddings of retrieved documents or generated responses. Lower cosine similarity scores indicate higher diversity, suggesting that the system can retrieve or generate a broader spectrum of information.

Noise Robustness measures the RAG system’s ability to handle irrelevant or misleading information without compromising the quality of the response. The metrics Misleading Rate and Mistake Reappearance Rate are described, providing detailed descriptions tailored to the specific dataset and experimental setup.

Negative Rejection evaluates the system’s capability to withhold responses when the available information is insufficient or too ambiguous to provide an accurate answer. Rejection Rate: The rate at which the system refrains from generating a response.

Counterfactual Robustness Counterfactual robustness assesses the system’s ability to identify and disregard incorrect or counterfactual information within the retrieved documents. Error Detection Rate: The ratio of counterfactual statements detected in retrieved information.


Evaluation of Retrieval-Augmented Generation: A Survey:4 Discussion

For RAG systems, traditional Question Answering (QA) datasets and metrics remain a common format for interaction. While these provide a basic verification of RAG’s capabilities, it becomes challenging to distinguish the impact of retrieval components when faced with strong Language Models (LLMs) capable of excelling in QA benchmarks. To comprehensively evaluate the performance of entire RAG systems, there is a need for diverse and RAG-specific benchmarks. Several papers offer guidance on improving QA format benchmarks, including variations in question types: from simple Wikipedia filling questions to multi-hop, multi-document questions and single-round to multi-round dialogue. For answers, aspects such as structural output, content moderation, and hallucination can be considered when evaluating relevance, faithfulness, and correctness. In addition to these, RAG systems require additional requirements such as robustness to noisy documents, language expression, latency, and result diversity. Furthermore, research is needed on performance changes involving intermediate outputs and retrieved documents, as well as the relationship and analysis between retrieval metrics and final generation outputs.

Regarding datasets, creating a universal dataset was challenging due to the target-specific nature of different RAG benchmarks. Tailored datasets are necessary for a thorough evaluation, but this approach increases the effort and resources required. Moreover, the diversity of datasets, from news articles to structured databases, reflects the adaptability required of RAG systems but also poses a barrier to streamlined evaluation. Recently, with the cutting-edge performance of LLMs, complex data processing and automatic QA pair generation can be automated to achieve dual or finer-grained time resolution, preventing LLMs from cheating and evaluating the robustness of RAG systems in rapidly changing data.


When it comes to metrics, the use of LLMs as automatic evaluative judges signifies a burgeoning trend, promising versatility and depth in generative outputs with reasoning on a large scale compared to human evaluation. However, using “LLMs as a Judge” for responses presents challenges in aligning with human judgment, establishing effective grading scales, and applying consistent evaluation across varied use cases. Determining correctness, clarity, and richness can differ between automated and human assessments. Moreover, the effectiveness of example-based scoring can vary, and there’s no universally applicable grading scale and prompting text, complicating the standardization of “LLM as a Judge”.

In addition to the challenges mentioned above, it is important to consider the resource-intensive nature of using Large Language Models (LLMs) for data generation and validation. RAG benchmarks must balance the need for thorough evaluation with the practical constraints of limited computational resources. As such, it is desirable to develop evaluation methodologies that can effectively assess RAG systems using smaller amounts of data while maintaining the validity and reliability of the results.

5 Conclusion

This survey systematically explores the complexities of evaluating RAG systems, highlighting the challenges in assessing their performance. Through the proposed A Unified Evaluation Process of RAG, we outline a structured approach to analyzing RAG evaluations, focusing on targets, datasets and measures. Our analysis emphasizes the need for targeted benchmarks that reflect the dynamic interplay between retrieval accuracy and generative quality and practical considerations for real-world applications. By identifying gaps in current methodologies and suggesting future research directions, we aim to contribute to more effective, and user-aligned benchmarks of RAG systems.


Evaluation of Retrieval-Augmented Generation: A Survey:References

1. Balauger, A., Benara, V., Cunha, R.L.d.F., Filho, R.d.M.E., Hendry, T., Holstein, D., Marsnam, J., Mecklenburg, N., Malvar, S., Nunes, L.O., Padilha, R., Sharp, M., Silva, B., Sharma, S., Aski, V., Chandra, R.: RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. Tech. rep. (Jan 2024). http://arxiv.org/abs/2401.008406 arXiv:2401.008406 [cs] type: article

2. Barnett, S., Kurunwan, S., Thudumu, S., Brannelly, Z., Abdrelazek, M.: Seven failure points when engineering a retrieval augmented generation system (Jan 2024). https://doi.org/10.48550/ARXIV.2401.05856

3. Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianniazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., Hoefler, T.: Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence 2024 (AAAI’24) (Aug 2023). https://doi.org/10.48550/ARXIV.2308.09687

4. Blagojevic, V.: Enhancing RAG Pipelines in Haystack: Introducing DiversityRanker and LostInTheMiddleRanker (Aug 2023). https://towardsdatascience.com/enhancing-rag-pipelines-in-haystack-45f1e42bc9f5

5. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15(3), 1–45 (2024)

6. Chen, J., Lin, H., Han, X., Sun, L.: Benchmarking large language models in retrieval-augmented generation (Sep 2023). https://doi.org/10.48550/ARXIV.2309.15217

7. Cuconasu, F., Trappollini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., Silvestri, F.: The power of noise: Redefining retrieval for rag systems (Jan 2024). https://doi.org/10.48550/ARXIV.2401.14887

8. Deng, Y., Zhang, W., Chen, Z., Gu, Q.: Rephrase and respond: Let large language models ask better questions for themselves (Nov 2023). https://doi.org/10.48550/ARXIV.2311.04205

9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423 https://aclanthology.org/N19-1423

10. DeYoung, J., Jain, S., Rajani, N.F., Lehman, E., Xiong, C., Socher, R., Wallace, B.C.: Eraser: A benchmark to evaluate rationalized nlp models

11. Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of Wikipedia: Knowledge-powered conversational agents. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)

12. Douze, M., Guiziva, A., Deng, C., Johnson, J., Zailysav, G., Mazaré, P.E., Lomeli, M., Hossenini, L., Jégou, H.: The fais library (2024)

13. DuckDuckGo: DuckDuckGo — Privacy, simplified. (2024), https://duckduckgo.com/hpme

14. Es, S., James, J., Espinosa-Anke, L., Schockaert, S.: Ragas: Automated evaluation of retrieval augmented generation (Sep 2023). https://doi.org/10.48550/ARXIV.2309.15217


Evaluation of Retrieval-Augmented Generation: A Survey:Page Content

15. Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D.: MRQA 2019 shared task: Evaluating generalization in reading comprehension. In: Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D. (eds.) Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-5801 https://aclanthology.org/D19-5801

16. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q., Wang, M., Wang, H.: Retrieval-Augmented Generation for Large Language Models: A Survey. Tech. rep. (Jan 2024). http://arxiv.org/abs/2312.10997 arXiv:2312.10997 [cs] type: article

17. Giannop, L., Seelci, H., Beckers, N., Bevendorff, J., Wang, S., Kiesel, J., Syed, S., Fröbe, M., Zuccon, G., Stein, B., Hagen, M., Potthast, M.: Evaluating Generative Ad Hoc Information Retrieval. Tech. rep. (Nov 2023). http://arxiv.org/abs/2311.04694 arXiv:2311.04694 [cs] type: article

18. Google: Programmable Search Engine | Google for Developers (2024). https://developers.google.com/custom-search

19. Gottschalk, S., Demidova, E.: Evek: A multilingual event-centric temporal knowledge graph (Apr 2018) https://doi.org/10.48550/ARXIV.1804.04526

20. Hofstätter, S., Chen, J., Raman, K., Zamani, H.: HiP-Linear: Efficient and Effective Retrieval-Augmented Text Generation. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1437–1447. SIGIR '23. Association for Computing Machinery, New York, NY, USA (Jul 2023). https://doi.org/10.1145/3539618.3591687 https://doi.org/10.1145/3539618.3591687

21. Hu, L.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-Rank Adaptation of Large Language Models. Tech. rep. (Oct 2021). https://doi.org/10.48550/arXiv.2106.09685 http://arxiv.org/abs/2106.09685 arXiv:2106.09685 [cs] type: article

22. Huang, J., Shao, H., Chang, K.C.C., Xiong, J., Hwu, W.M.: Understanding jargon: Combining extraction and generation for definition modeling. In: Proceedings of EMNLP (2022)

23. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions (Nov 2023) https://doi.org/10.48550/ARXIV.2311.05232

24. Huang, Y., Huang, J.: A survey on retrieval-augmented text generation for large language models (Apr 2024) https://doi.org/10.48550/ARXIV.2404.10981

25. Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3), 535–547 (2019)

26. Kamali, E., Thakur, N., Lassance, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beer: Reproducible reference models and an official leaderboard (2023)

27. Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R.L., Asai, A., Yu, X., Radev, D., Smith, N.A., Choi, Y., Inui, K.: Realtime qa: What’s the answer right now? (Jul 2022). https://doi.org/10.48550/ARXIV.2207.13332 https://arxiv.org/abs/2207.13332

28. Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via contextualized late interaction over bert (Apr 2020). https://doi.org/10.48550/ARXIV.2004.12832

29. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.W., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: A benchmark for question


Evaluation of Retrieval-Augmented Generation: A Survey:Reference List

answering research. Transactions of the Association for Computational Linguistics 7, 453–466 (2019). https://doi.org/10.1162/tacl_a_00276 https://doi.org/10.1162/tacl_a_00276
30. Lilhafairi, A.R., Permasaani, A.E., Setiavan, N.A.: Cosine similarity to determine similarity measure: Study case in online essay assessment. In: 2016 4th International Conference on Cyber and IT Service Management. pp. 1–6 (2016). https://doi.org/10.1109/CITSM.2016.7577578
31. Lanchantin, J., Toshniwal, S., Weston, J., Szlam, A., Sukhbaatar, S.: Learning to reason and memorize with self-notes (May 2023). https://doi.org/10.48550/ARXIV.2305.00833
32. LangChain: Evaluating rag architectures on benchmark tasks (Nov 2023). https://langchain-ai.github.io/langchain-benchmarks/notebooks/retrieval/langchain_docs_qa.html
33. Leng, Q., Ulhenhuth, K., Polyzotis, A.: Best Practices for LLM Evaluation of RAG Applications (Dec 2023). https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG
34. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. pp. 9459-9474. NIPS’20, Curran Associates Inc., Red Hook, NY, USA (Dec 2020)
35. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Tech. rep. (Apr 2021). http://arxiv.org/abs/2005.11401 arXiv:2005.11401 [cs] type: article
36. Liang, X., Song, S., Niu, S., Li, Z., Xiong, F., Tang, B., Wang, W., Z., He, D., Cheng, P., Wang, Z., Deng, H.: Ueberai: Benchmarking the hallucination of chinese large language models with unconstrained generation. arXiv preprint arXiv:2311.15296 (2023)
37. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74-81. Association for Computational Linguistics, Barcelona, Spain (Jul 2004). https://aclanthology.org/W04-1013
38. Liu, Y., Huang, L., Li, S., Chen, S., Zhou, H., Meng, F., Zhou, J., Sun, X.: Recall: A benchmark for llms robustness against external counterfactual knowledge (Nov 2023). https://doi.org/10.48550/ARXIV.2311.08174
39. Lyu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., Chen, E., Luo, Y., Cheng, P., Deng, H., Wang, Z., Liu, Z.: Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models (Jan 2024). https://doi.org/10.48550/ARXIV.2401.17043
40. Microsoft: Web Search API | Microsoft Bing, https://www.microsoft.com/en-us/bing/apis/bing-web-search-api
41. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anandkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Balestrc, P., Bao, H., Bavarian, M., Belgium, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdanoff, L., Boiko, O., Boyd, M., Brakman, A.L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Can, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, J., Chen, Y., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H.W., Cummings, D., Currier, J., Dai, Y., Decarreaux, C., Depey, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffekt, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S.P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Vogianni, T., ...


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation of Retrieval-Augmented Generation: A Survey

Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S.S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Hecaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschle, P., Houghton, D., Hsu, K., Hu, S., Hu, X., Huijnga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jon, B., Jung, H., Kaftan, T., Kamali, A., Kantschieder, I., Keskar, N.S., Khan, T., Kilpatrick, L., Kim, J.W., Kim, C., Kim, Y., Kirchner, J.H., Kiros, J., Knight, M., Kokotajlo, D., Kondracka, A., Konstantinidis, A., Kosic, M., Krueger, G., Kuo, L., Lampe, M., Lan, L., Lee, T., Leike, J., Leung, J., Levy, D., Li, C.M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lupa, E., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovskiy, V., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S.M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Mentek, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murtha, M., Murk, O., Mély, D., Nahir, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, P., Noh, H., Ouyang, L., O'Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parisi, J., Parraibieta, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., Peres, F.A.D.B., Petrov, M., Pinto, H.P.O.D., Michael, Pokorny, Pokrass, M., Pong, V.H., Powell, T., Power, A., Power, B., Proehl, E., Pu, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, E., Rimbach, K., Ross, C., Rosted, B., Rushe, R., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, R., Sherbakov, T., Shieh, J., Shoker, N., Shyam, P., Sidow, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, J., Sokolovsky, B., Song, Y., Stauderach, N., Such, F.P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M.B., Tillet, P., Totonchian, A., Teng, E., Tuggle, P., Turley, N., Tworek, J., Uzkurer, H., Vallone, A., Vijayaraghava, A., Voss, C., Wainwright, C., Wang, J.J., Wang, A., Wang, B., Ward, J., Wei, J., Weinman, C., Welchman, A., Weidner, P., Weng, J.W., Wen, Y., Willis, C., Weng, J., Zdane, J., Zheng, T., Zhuan, J., Zhub, W., Zoph, B.: GPT-4 Technical Report (Mar 2023). https://doi.org/10.48550/arXiv.2303.08774

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schlam, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askel, A., Weidner, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. Tech. Rep. (Mar 2022). https://doi.org/10.48550/arXiv.2203.02155http://arxiv.org/abs/2203.02155

arXiv:2203.02155 [cs] type: article

Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Isabellpe, P., Charniak, E., Lin, D. (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (Jul 2002). https://doi.org/10.3115/1073083.1073135https://aclanthology.org/P02-1040

Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktischkel, T., Riedel, S.: KILT: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2523–2544. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.200https://aclanthology.org/2021.naacl-main.200

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskver, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), (2019)

Ramos, J., et al.: Using tf-idf to determine word relevance in document queries. In: Proceedings of the first instructional conference on machine learning, vol. 242, pp. 29–48. Citeseer


47. Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval 3(4), 333–389 (2009).
48. Rosset, C., Chang, H.L., Qin, G., Hou, F.C., Feng, Z., Awadallah, A., Neville, J., Rao, N.: Researchy questions: A dataset of multi-perspective, decompositional questions for lm web agents (Feb 2024). https://doi.org/10.48550/ARXIV.2402.17896
49. Saad-Falcon, J., Khatta, O., Potts, C., Zaharia, M.: Ares: An automated evaluation framework for retrieval-augmented generation systems (Nov 2023). https://doi.org/10.48550/ARXIV.2311.09476
50. Sai, A.B., Mohankumar, A.K., Khapra, M.M.: A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR) 55(2), 1–39 (2022)
51. Shahabi, C., Kolahdouzan, M.R., Sharifzadeh, M.: A road network embedding technique for k-nearest neighbor search in moving object databases. In: Proceedings of the 10th ACM international symposium on advances in geographic information systems, pp. 94–100 (2002)
52. Tang, Y., Yang, Y.: Mulltihop-rag: Benchmarking retrieval-augmented generation for multihop queries (Jan 2024). https://doi.org/10.48550/ARXIV.2401.15391
53. Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a large-scale dataset for fact extraction and VERification. In: NAACL-HLT (2018)
54. TruLens: TruLens (2023). https://www.trulens.org/trulens_eval/getting_started/quickstarts/quickstart/
55. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (Jun 2017). https://doi.org/10.48550/arXiv.1706.03762
56. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537 (2019)
57. Wang, S., Kharmsanova, E., Zhuang, S., Zuccon, G.: Febr4ag: Evaluating federated search in the context of retrieval augmented generation (Feb 2024). https://doi.org/10.48550/ARXIV.2402.11981
58. Wang, S., Liu, J., Song, S., Cheng, J., Fu, Y., Guo, P., Fang, K., Zhu, Y., Dou, Z.: Domimanga: A chinese benchmark for evaluating domain-specific retrieval-augmented generation (Jun 2024). https://doi.org/10.48550/ARXIV.2406.05564
59. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., Fedus, W.: Emergent abilities of large language models (Jun 2022). https://doi.org/10.48550/ARXIV.2206.07682
60. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (Jan 2022). https://doi.org/10.48550/ARXIV.2201.11903
61. Xiong, G., Jin, Q., Lu, Z., Zhang, A.: Benchmarking retrieval-augmented generation for medicine (Feb 2024). https://doi.org/10.48550/ARXIV.2402.13178
62. Xu, Z., Li, Y., Ding, R., Wang, X., Chen, B., Jiang, Y., Zheng, H.T., Lu, W., Xie, P., Huang, F.: Let lms take on the latest challenges! a chinese dynamic question answering benchmark (Feb 2024). https://doi.org/10.48550/ARXIV.2402.19248
63. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)
64. Yao, J.Y., Ning, X.P., Liu, Z.H., Ning, M.N., Yau, L.: Lim lms: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023)
65. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of Thoughts: Deliberate problem solving with large language models (2023).


66. Yu, X., Cheng, H., Liu, X., Roth, D., Gao, J.: ReVeal: Automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 133–1351. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024). https://aclanthology.org/2024.findings-naacl.85

67. Zhang, K., Liu, Q., Qian, H., Xiang, B., Cui, Q., Zhou, J., Chen, E.: Eatn: An efficient adaptive network for aspect-level sentiment analysis. IEEE Transactions on Knowledge and Data Engineering 35(1), 377–389 (2021)

68. Zhang, K., Zhang, H., Liu, Q., Zhao, H., Zhu, H., Chen, E.: Interactive attention transfer network for cross-domain sentiment classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5773–5780 (2019)

69. Zhang, K., Zhang, K., Zhang, M., Zhao, H., Liu, Q., Wu, W., Chen, E.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. arXiv preprint arXiv:2203.16369 (2022)

70. Zhang, Q., Chen, S., Xu, D., Cao, Q., Chen, X., Cohn, T., Fang, M.: A Survey for Efficient Open Domain Question Answering. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14447–14465. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-long.808 https://aclanthology.org/2023.acl-long.808

71. Zhang, S., Liu, X., Liu, J., Gao, J., Du, K., van Durme, B.: Record: Bridging the gap between human and machine commonsense reading comprehension (Oct 2018). https://doi.org/10.48550/ARXIV.1810.12885

72. Zhang, T., Kishore, V., Wu, F., Weinberger, K., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020). https://openreview.net/forum?id=SkeHuCVFDr

73. Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., Wang, L.: Merging Generated and Retrieved Knowledge for Open-Domain QA. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4710–4728. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp-main.286 https://aclanthology.org/2023.emnlp-main.286

74. Zhao, H., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Cui, B.: Retrieval-augmented generation for ai-generated content: A survey (Feb 2024). https://doi.org/10.48550/ARXIV.2402.19473

75. Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging llms-as-judge with nlt-bench and chatbot arena (Jun 2023). https://doi.org/10.48550/ARXIV.2306.05685

76. Zhou, Y., Lin, X., Zhang, X., Wang, M., Jiang, G., Lo, A., Hu, W., Yu, Z., Zhang, K., Yang, Z., Wang, K., Sui, Y., Jia, F., Tang, Z., Zhao, Y., Zhang, H., Yang, T., Chen, W., Mao, Y., Li, Y., Bao, D., Li, Y., Liao, H., Liu, T., Liu, J., Guo, J., Zhao, X., WEI, Y., Qian, H., Liu, Q., Wang, X., Xin, W., Chan, Li, C., Li, Y., Yang, S., Yan, J., Mou, C., Han, S., Jin, W., Zhang, G., Zeng, X.: On the opportunities of green computing: A survey (Nov 2023)

77. Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering. Tech. rep. (May 2021), http://arxiv.org/abs/2101.00774 arXiv:2101.00774 [cs] type: article


Evaluation of Retrieval-Augmented Generation: A Survey:Structure of RAG System

A.1 Retrieval Component

The retrieval component of RAG systems in Figure 1 can be categorized into three types: sparse retrieval, dense retrieval [77], and web search engine. The standard for evaluation is the output of relevant documents with numerical scores or rankings.

Before the introduction of neural networks, sparse retrievals are widely used for retrieving relative text content. Methods like TF-IDF [46] and BM25 [47] rely on keyword matching and word frequency but may miss semantically relevant documents without keyword overlap.

By leveraging deep learning models such as BERT [9], dense retrieval can capture the semantic meaning of texts, which allows them to find relevant documents even when keyword overlap is minimal. This is crucial for complex queries that require a contextual understanding to retrieve accurate information. With advanced fusion structure for queries and documents [28] and the more efficient implementation of K-Nearest Neighbors (KNN) [51], Approximate Nearest Neighbor (ANN) [12 25] search techniques, dense retrieval methods have become practical for large-scale use.

Web search engine employs the complex online search engine to provide relevant documents, such as Google Search [18], Bing Search [40], DuckDuckGo [13]. RAG systems can traverse the web’s extensive information, potentially returning a more diverse and semantically relevant set of documents via the API of the search provider. The black box of the search engine and the expense of large-scale search are not affordable sometimes.

It is observed that dense retrieval techniques, particularly those leveraging embeddings, stand out as the preferred choice within the RAG ecosystem. These methods are frequently employed in tandem with sparse retrieval strategies, creating a hybrid approach that balances precision and breadth in information retrieval. Moreover, the adoption of sophisticated web search engines for benchmark assessment underscores their growing significance in enhancing the robustness and comprehensiveness of evaluations.

Indexing The indexing component processes and indexes document collections, such as HuggingFace datasets or Wikipedia pages. Chunking before indexing can improve retrieval by limiting similarity scores to individual chunks, as semantic embedding is less accurate for long articles, and desired content is often brief [32]. Index creation is designed for fast and efficient search. For example, the inverted index for sparse retrieval and the ANN index for dense retrieval.

Sparse Retrieval involves calculating IDF for each term and storing values in a database for quick look-up and scoring when queried.

Dense Retrieval encodes documents into dense vectors using a pre-trained language model like BERT. These vectors are then indexed using an Approximate Nearest Neighbor (ANN) search technique, like graph-based Hierarchical Navigable Small World (HNSW) or Inverted File Index (IVF) [12]. This process allows for the efficient retrieval of “closed” items by given predefined distance metrics.


Evaluation of Retrieval-Augmented Generation: A Survey:Search

This step is responsible for retrieving relevant documents based on a given query. Queries are submitted using the respective API to retrieve relevant documents for web search engine retrieval. For local resources, the query component is responsible for formatting the query in the format required by different sparse or dense retrieval methods. Then, the query is submitted to the retrieval system, which returns a set of relevant documents along with their scores.

In both local and web-based scenarios, an optional reranker can be employed to refine the ranking of retrieved documents further. The reranker usually comprises a more complex and larger model that considers additional features of the documents and the given query. These additional features often include the semantic relationship between the query and the document content, document importance or popularity, and other custom measures specific to the information need at hand.


Evaluation of Retrieval-Augmented Generation: A Survey:A.2 Generation Component

The evaluable output for the generation component is the response of LLMs and the structured or formatted output from the phrased response.


Evaluation of Retrieval-Augmented Generation: A Survey:Prompting

The generation process critically hinges on prompting, where a query, retrieval outcomes, and instructions converge into a single input for the language model. Research showcases various strategic prompting tactics such as the Chain of Thought (CoT) [60], Tree of Thought (ToT) [3], and Self-Note [31], each significantly shaping the model’s output. These methods, especially the step-by-step approach, are pivotal in augmenting LLMs for intricate tasks.

Prompting innovations have introduced methods like Rephrase and Respond (RaR) [8], enhancing LLMs by refining queries within prompts for better comprehension and response. This technique has proven to boost performance across diverse tasks. The latest RAG benchmarks [61,62] in the specific domains start to evaluate the robustness of various prompting engineering skills, including CoT, RaR, etc.


Evaluation of Retrieval-Augmented Generation: A Survey:Inference

The final input string prepared in the prompting step is then passed on to the LLMs as input, which generates the output. The inference stage is where the LLM operates on the input derived from the retrieval and the prompting stages in the pipeline to generate the final output. This is usually the answer to the initial query and is used for downstream tasks.

Depending on the specifics of the task or expected output structure, a post-processing step may be implemented here to format the generated output suitably or extract specific information from the response. For example, the classification problems (multiple choice questions) or if the task requires the extraction of specific information from the generated text, this step could involve additional named entity recognition or parsing operations.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

### Authors
Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, Tomas Pfister

1. University of California, San Diego
2. Google Cloud AI Research
3. Google DeepMind
4. Google Cloud AI

### Abstract
Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce SPECULATIVE RAG – a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that SPECULATIVE RAG achieves state-of-the-art performance with reduced latency on TriviaQA, MuSiQue, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 51% compared to conventional RAG systems on PubHealth.

### 1 Introduction
Large language models (LLMs) have demonstrated remarkable success in question answering tasks (Brown et al., 2020; Achiam et al., 2023; Team et al., 2023). Trained on massive datasets, LLMs leverage their extensive parametric memory to generate seemingly plausible responses to user queries (Kojima et al., 2022; Kamalloo et al., 2023). However, when faced with knowledge-intensive questions demanding up-to-date information or obscure facts (Petroni et al., 2022a; 2022b), LLMs can struggle with factual inaccuracies and produce hallucinated content (Huang et al., 2023; Xu et al., 2024).

Retrieval Augmented Generation (RAG) has emerged as a promising solution to mitigate these issues. By incorporating information retrieved from an external database into the context (Gao et al., 2023b), RAG effectively reduces factual errors in knowledge-intensive tasks. This approach not only enables easy and efficient access to vast databases but also facilitates timely and accurate knowledge integration Due to the inherent limitations in the precision of current dense retrievers and the vastness of knowledge required to answer complex questions (Chen et al., 2022), RAG systems typically retrieve multiple documents to ensure the inclusion of all necessary information in the context (Petroni et al., 2021). This practice inevitably increases the length of the input to the LLMs.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Our Speculative Retrieval-Augmented Generation (SPECULATIVE RAG)

Figure 1: Illustration of different RAG approaches. Given a knowledge-intensive query Q and retrieved documents, (a) Standard RAG incorporates all documents into the prompt, increasing input length and slowing inference; (b) Self-Reflective RAG (Asai et al., 2023) requires specialized instruction-tuning of the general-purpose language model (LM) to generate specific tags for self-reflection; (c) Corrective RAG (Yan et al., 2024) employs an external retrieval evaluator to refine document quality, focusing solely on contextual information without enhancing reasoning capabilities; (d) In contrast, our proposed SPECULATIVE RAG leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, specialized LM. Each draft is generated from a distinct subset of retrieved documents, providing diverse perspectives on the evidence while minimizing the number of input tokens per draft.

In this world, we introduce SPECULATIVE RAG, a RAG framework designed to offload computational burden to a smaller, specialist LM that serves as an efficient and robust RAG module for existing generalist LMs. Inspired by Speculative Decoding (Leviathan et al., 2023; Chen et al., 2023a; Xia et al., 2024), which accelerates auto-regressive LM inference by concurrently generating multiple draft tokens with a smaller model and verifying them in parallel with the base model, our approach adapts this concept to RAG.

In SPECULATIVE RAG, we partition retrieved documents into subsets for drafting answer candidates. We cluster the retrieved documents by content similarity and sample one document from each cluster to form a subset, minimizing redundancy and maximizing diversity. These document subsets are then fed to multiple instances of the RAG module, which generate draft answers with corresponding rationales in parallel. This smaller, specialized RAG module, excels at reasoning over retrieved documents and can rapidly produce accurate responses. Subsequently, the generalist LM bypasses the detailed review of potentially repetitive documents, focusing instead on validating the drafts presented significant challenges, particularly since encoding lengthy retrieved documents incurs additional latency and require more complex reasoning. Recent studies have explored ways to extend the context length limit of LLMs (Ding et al., 2023; Reid et al., 2024; Ma et al., 2024), yet achieving well-grounded reasoning over extended contexts remains an open question (Liu et al., 2024; Li et al., 2024). Consequently, striking a balance between efficiency and effectiveness in RAG has become a central research question in the literature. Existing work on RAG systems primarily concentrates on improving the quality of contextual information in retrieval outcomes, but often neglecting the latency issues associated with these systems (Ma et al., 2023; Baek et al., 2023; Yan et al., 2024; Xie et al., 2023; Asai et al., 2023; Feng et al., 2023). These methods typically rely on multiple refinement iterations or customized instruction-tuning for self-critique abilities. Integrating such enhancements into generic LMs requires additional training or increased latency, posing practical challenges in real-world applications.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Introduction

We introduce a novel RAG framework that employs a smaller specialist RAG drafter to generate high-quality draft answers. Each draft is derived from a distinct subset of retrieved documents, offering diverse perspectives while reducing input token counts per draft. The generalist LM, operating with the RAG drafter, requires no additional tuning. It simply verifies and integrates the most promising draft into the final answer. This approach enhances comprehension of each subset and mitigates potential lost-in-the-middle (Liu et al., 2024) phenomenon. Our method significantly accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single, unbiased verification pass over the drafts in parallel. Extensive experiments on 4 free-form question-answering and closed-set generation benchmarks demonstrate the superior effectiveness and efficiency of the method.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:2 Related Works

Retrieval Augmented Generation Retrieval Augmented Generation (RAG) enhances LLMs by retrieving relevant documents from external databases and incorporating them into the generation process (Gao et al., 2023b; Lewis et al., 2020; Khandelwal et al., 2020; Izacard & Grave, 2021; Luo et al., 2023a). Recent work has primarily focused on enabling LLMs to understand when and what to retrieve (Ma et al., 2023; Chen et al., 2023b; Jiang et al., 2023b; Schick et al., 2024), or designing approaches to better utilize contexts (Yu et al., 2023; Yoran et al., 2023; Wang et al., 2023b; Sarthi et al., 2024; Baek et al., 2023; Xu et al., 2023; Kim et al., 2024). Among them, SAIL (Luo et al., 2023a) fine-tunes a pre-trained LLM on web search data to filter irrelevant contents. Self-Reflective RAG (Asai et al., 2023) introduces reflection tokens to guide retrieval and annotation in instruction-tuning datasets. However, many RAG approaches require additional instruction of generic LLMs, which is resource-intensive and may lead to forgetting or over-fitting (Luo et al., 2023b). Furthermore, long context within retrieved contents can suffer from computational inefficiency and position bias (Liu et al., 2024). Corrective RAG (Yan et al., 2024) on the other hand proposes a lightweight retrieval evaluator, but it lacks the capability for high-level reasoning. In contrast, our proposed SPECULATIVE RAG addresses these limitations by leveraging a smaller RAG drafter model to efficiently understand diverse perspectives in retrieval results and generate drafts for the generalist LMs to verify and integrate.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Speculative Decoding

Speculative decoding (Stern et al., 2018; Xia et al., 2023; Chen et al., 2023a; Leviathan et al., 2023; Xia et al., 2024) aims to reduce auto-regressive decoding latency through a draft-then-verify paradigm. This involves drafting multiple future tokens with a small model and verifying them in parallel with the target model (Xia et al., 2024). The draft model is typically either an independent model from the same series (Leviathan et al., 2023; Chen et al., 2023a) or the target model itself (Zhang et al., 2023a; Cai et al., 2024). Our approach extends this concept from token-level drafting to answer-level drafting. In contrast to traditional verification criteria (Stern et al., 2018; Xia et al., 2023; Leviathan et al., 2023; Chen et al., 2023a; Miao et al., 2024), which accept or reject tokens based on their generation probabilities, we leverage language modeling objectives to directly assess the confidence of entire answer drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:3 Speculative Retrieval Augmented Generation through Drafting

Problem Formulation In knowledge intensive tasks, each entry can be represented as (Q, D, A), where Q is a question or statement that requires additional knowledge; D = {d1, ..., dn} is a set of n documents retrieved from the database; A is the expected answer. Particularly, in question answering tasks, Q and A are the question and the expected answer in natural language form; in the statement verification tasks, Q is a statement and A ∈ {True, False} is a Boolean value indicating the statement’s correctness; in the multiple choice tasks, Q is a question with a few options and A ∈ {A, B, C, ...} is the index of the correct answer. The objective of a RAG system is to generate a fluent response containing the expected answer or select the expected answer from the provided options based on the context provided by the retrieved supporting documents.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Overview

We introduce Speculative Retrieval Augmented Generation (Speculative RAG), as illustrated in Figure 1. We aim at enhancing the reasoning ability of LLMs over retrieved documents without compromising processing speed. Instead of relying on brute-force parameter scaling or instruction-tuning an entire LM to handle knowledge-intensive tasks, we propose a divide-and-conquer approach. We utilize a smaller specialist LM, the RAG drafter, to rapidly generate multiple answer drafts based on retrieved results. Then, a larger generalist LM, the RAG verifier, assesses these drafts, selects the best one based on its rationale, and integrates it into the generation results.