RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:References

[1] Al@Meta. Llama 3 model card. 2024.

[2] A. Asai, S. Min, Z. Zhong, and D. Chen. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41–46, 2023.

[3] H. Chase. LangChain. https://github.com/langchain-ai/langchain Oct. 2022.

[4] J. Chen, H. Lin, X. Han, and L. Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762, 2024.

[5] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023.

[6] J. Ferrara, Ethan-Tonic, and O. M. Outrix. The RAG Triad. January 2024. https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

[7] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.

[8] Y. Guo, Z. Li, X. Jin, Y. Liu, Y. Zeng, W. Liu, X. Li, P. Yang, L. Bai, J. Guo, et al. Retrieval-augmented code generation for universal information extraction. arXiv preprint arXiv:2311.02962, 2023.

[9] R. Han, P. Qi, Y. Zhang, L. Liu, J. Burger, W. Wang, Z. Huang, B. Xiang, and D. Roth. Robustqa: Benchmarking the robustness of domain adaptation for open-domain question answering. In ACL Findings 2023, 2023.

[10] X. He, R. Yuan, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630, 2024.

[11] X. Hu, D. Ru, L. Qiu, Q. Guo, T. Zhang, Y. Xu, Y. Luo, P. Liu, Y. Zhang, and Z. Zhang. Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. 2024.

[12] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023.

[13] Y. Huang and J. Huang. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024.

[14] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Matrixl of experts. arXiv preprint arXiv:2401.04088, 2024.

[15] V. Katrandis and G. Barany. FaaF: Facts as a function for the evaluation of rag systems. arXiv preprint arXiv:2403.03888, 2024.

[16] M. Komeili, K. Shouser, and J. Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566, 2021.

[17] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019.

[18] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karuparse name="benchmarking" hakkasghbgdxdfjuhgpryakhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

[19] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

[20] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.


[21] N. F. Liu, T. Zhang, and P. Liang. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025, 2023.

[22] Y. Liu, L. Huang, S. Li, S. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun. Recall: A benchmark for llms robustness against external counterfactual knowledge. arXiv preprint arXiv:2311.08147, 2023.

[23] K. Lo, L. L. Wang, M. Neumann, R. Kinney, and D. Weld. S2ORC: The semantic scholar open research corpus. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online, July 2020. Association for Computational Linguistics.

[24] A. Louis, G. van Dijck, and G. Spanakis. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275, 2024.

[25] Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, and E. Chen. Crud-rag: a comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043, 2024.

[26] M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the web conference 2018, pages 1941–1942, 2018.

[27] Y. Mao, P. He, X. Liu, Y. Shen, J. Gao, J. Han, and W. Chen. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553, 2020.

[28] A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tzezak, J. W. Kim, C. Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10095, 2022.

[29] OpenAI, J. A. Shidess, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. E. Aleman, D. Almeida, J. Altenschmid, S. Altman, S. Anakat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Balji, C. Basti, H. Bao, M. Bavarian, J. Belijo, J. Bellido, G. Bernadotte, N. Bhambhra, C. Berner, L. Bogdanoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brunagel, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, D. Chang, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Cheng, H. Chens, C. Cho, C. Chu, A. Hw, D. Chung, D. Cummings, J. Currier, D. Dai, C. Decaeeraux, T. Derg, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dovving, S. Dunn, P. Endale, A. Ertel, T. Etinudson, D. Farhi, E. Faulk, S. Felix, R. Fishman, J. Forte, J. Fulford, L. Gaof, E. Georges, G. Gibbon, Y. Goei, T. Guengo, N. Goh, R. Gomtlpo-Juoles, J. Gordon, M. Gratsiak, S. Gray, C. Greene, J. Gross, S. Gu, P. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Hoogstn, S. Hsu, S. Hux, H. Ju, J. Huizinga, S. Jain, S. Jain, J. Jiang, B. Jiang, R. Jiang, H. Jin, D. Jin, S. Jonston, B. Jonn, H. Jun, T. Kafant, L. Kaaszler, A. Kamali, I. Kantschieder, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondracki, A. Kondo, A. Konstantinidis, K. Kosic, G. Krueger, Y. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lopez, P. Lowe, A. Makanuj, K. Malaficin, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Mendez, L. Metz, A. Mishenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mely, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachock, A. Paino, J. Palermo, A. Panitunga, G. Parascalndo, J. Parish, B. Paparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belustre Perez, M. Petro, H. de P. Oliveira Pinto, Michael, Pohonym, A. Pokrass, V. Pony, R. Porta, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rostedt, H. Roussecx, N. Ryder, M. Salterell, T. Sanders, S. Santurkar, G. Sastry, R. Schaad, D. Schmid, N. Schulman, D. Selasam, K. Shephard, T. Sherbakov, J. Shuey, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, J. Solh, B. Sokolowsky, Y. Song, N. Stashower, E. F. Such, N. Suthersan, J. Sutskever, J. Tang, N. Tezak, M. Thompson, P. Tillett, A. Tootoonchian, E. Tseng, P. Tugade, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayaraghavan, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, D. Ward, J. Wei, C. Weinman, A. Weilindha, P. Weinder, J. Weng, L. Weng, M. Whetfield, D. Willner, C. Winter, C. Weissman, A. Welchinda, P. Weiel, T. Wiens, M. Wilger, L. Wiseman, N. Wod, S. Wu, A. Wu, Y. Xing, H. Xia, Y. Xu, P. Xu, N. Li, T. Yao, Y. Ye, C. Yenipencot, A. Yap, V. Zhang, D. Zhou, G. Zhoa, H. Zhong, C. Zhu, Z. Zhu, and Y. Zieman. Language models are few-shot learners. arXiv preprint arXiv:2203.15556, 2022.


[28] S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report, 2023.

[30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In P. Isabelle, E. Charniak, and D. Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.

[31] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.

[32] M. R. Parvez, W. U. Ahmadi, S. Chakraborty, B. Ray, and K.-W. Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601, 2021.

[33] S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2010.

[34] S. Rosenthal, A. Sli, R. Florian, and S. Roukos. Clapp: Cohesive long-form answers from passages in natural questions for rag systems, 2024.

[35] J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia. Ares: An automated evaluation framework for retrieval-augmented generation systems, 2023.

[36] K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States, July 2022. Association for Computational Linguistics.

[37] R. Shuster, S. Pörﬂ, M. Chen, D. Rekala, and J. Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.

[38] H. Tan, Q. Luo, L. Jiang, Z. Zhan, J. Li, H. Zhang, and Y. Zhang. Prompt-based code completion via multi-retrieval augmented generation. arXiv preprint arXiv:2405.07530, 2024.

[39] Y. Tang and Y. Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries. arXiv preprint arXiv:2401.15391, 2024.

[40] N. Thakur, L. Bonifacio, X. Zhang, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, B. Chen, M. Rezagholizadeh, et al. Nomirai: Knowing when you don’t know for robust multilingual retrieval-augmented generation. arXiv preprint arXiv:2312.11361, 2023.

[41] D. Thulke, N. Daheim, C. Dugast, and H. Ney. Efficient retrieval-augmented generation from unstructured knowledge for task-oriented dialog. arXiv preprint arXiv:2102.04463, 2021.

[42] S. Tonmoy, S. Zamlan, V. Jain, A. Rani, V. Rawte, A. Chadha, and A. Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024.

[43] G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16:1–28, 2015.

[44] E. M. Voorhees et al. The trec-8 question answering track report. In Trec, volume 99, pages 77–82, 1999.

[45] C. Wang, S. Cheng, Q. Guo, Y. Yue, B. Ding, Z. Xu, Y. Wang, X. Hu, Z. Zhang, and Y. Zhang. Evaluating open-qqa evaluation. Advances in Neural Information Processing Systems, 36, 2024.

[46] C. Wang, X. Liu, Y. Yue, X. Tang, T. Zheng, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521, 2023.

[47] C. Wang, R. Ning, B. Pan, T. Wu, Q. Guo, C. Deng, G. Bao, Q. Wang, and Y. Zhang. Novelqa: A benchmark for long-range novel question answering, 2024.

[48] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023.


[49] K. Wu, E. Wu, and J. Zou. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198, 2024.
[50] G. Xiong, Q. Jin, Z. Lu, and A. Zhang. Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178, 2024.
[51] F. Xu, K. Lo, L. Soldaini, B. Kuehl, E. Choi, and D. Wadden. Kiwi: A dataset of knowledge-intensive writing instructions for answering research questions, 2024.
[52] Z. Xu, Y. Li, R. Ding, X. Wang, B. Chen, Y. Jiang, X. Deng, J. Ma, H.-T. Zheng, W. Lu, et al. Let llms take on the latest challenges! a chinese dynamic question answering benchmark. arXiv preprint arXiv:2402.19285, 2024.
[53] H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu. Evaluation of retrieval-augmented generation: A survey. arXiv preprint arXiv:2405.07437, 2024.
[54] C. Zakka, R. Shad, A. Chaurasia, A. R. Dalal, J. L. Kim, M. Moor, R. Fong, C. Phillips, K. Alexander, E. Ashley, et al. Almanac—retrieval-augmented language models for clinical medicine. NEJM AI, 1(2):Alo2300068, 2024.
[55] B. Zhang, H. Yang, T. Zhou, M. Ali Babar, and X.-Y. Liu. Enhancing financial sentiment analysis via retrieval augmented large language models. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 349–356, 2023.
[56] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[57] S. Zhou, U. Alon, F. F. Xu, Z. Wang, Z. Jiang, and G. Neubig. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv:2207.05987, 2022.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Details for Benchmark Curation

In this section, we introduce the benchmark datasets and the curation process for RAG evaluation. This benchmark datasets are derived from existing open-domain question answering (ODQA) datasets, including RobustQA (9), KIWI (51), ClapNQ (34), and NovelQA (47). However, most of the ground truth answers in existing ODQA datasets are short answers, while the answers provided by modern LLM-based RAG systems tend to be long-form answers. Therefore, we repurpose the ODQA datasets by eliminating overly simple questions and converting the short answers into long-form answers to match the capabilities of current RAG systems. The statistics of the benchmark are summarized in Tab. 1. In the rest of this section, we describe the datasets we use and the curation process for each domain.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A.1 Data Sources

RobustQA: We choose 7 domains from RobustQA’s collection of datasets: Biomedical, Finance, Lifestyle, Recreation, Technology, Science, and Writing. For the Biomedical domain, following RobustQA, we employ the BioASQ (43) dataset, which contains human expert-written question-answer pairs and ground truth documents based on abstracts of articles from PubMed. We use the test sets for Task 8 from 2014 to 2023 and the corpus of v.2022 to construct the benchmark. We keep QA pairs whose answers are relatively long (more than 50 words), obtaining 511 QA pairs for the biomedical domain. The other 6 domains are sourced from FiQA (26) and LoTTE (36), each of their question is annotated with a list of short answers that are spans of ground truth passages. We convert the short answers to long-form answers using GPT-4 and only keep the generated answers with no hallucinations, as checked by RefChecker. Finally, we sample 500 examples for each domain.

ClapNQ: Derived from NaturalQuestions (NQ) (17), an ODQA dataset based on Wikipedia, ClapNQ has long-form answers annotated for a subset of NQ for evaluating RAG. We employ the dev set of ClapNQ in our benchmark and take the annotated long-form answers as the ground truth.

KIWI: Is constructed by asking LLMs research questions about a set of NLP papers and guiding the LLMs to reach satisfactory long-form answers. The authors validated the quality of the generated answers by rating them as “good”, “neutral”, or “bad”. We take the answers labeled “good” as the ground truth answers and query the full text of the papers from S2ORC (23) as the corpus. As a result, we obtain 71 QA pairs and 429 papers as the corpus.

NovelQA: Is a benchmark for question answering over long novels containing over 100K tokens on average. Originally designed for benchmarking long-context LLMs, we repurpose it for evaluating RAG. In contrast with the other domains, each question in NovelQA is associated with a single novel, so when we use this dataset for RAG, we constrain the retrieval to within the corresponding novel. We select 19 copyright-free novels and convert the corresponding short answers to long-form answers following the same process for RobustQA.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A.2 Long-form Answer Generation

We employ GPT-4 (gpt-4-turbo-2024-04-09) to convert the human annotated short answers to long-form answers in the dataset of RobustQA and NovelQA. For RobustQA, the short answers are spans of the annotated ground truth passages, we take all the annotated short answers and the corresponding passages in the prompt and ask GPT-4 to convert them to one single long-form answer. For NovelQA, we take the human written evidences as the ground truth passage content and the human written short answers for the long-form answer generation. The prompt is shown in Fig. 2.

For quality control, we ask GPT-4 to generate the passage IDs associated with the long-form answer. We use RefChecker to check whether all the claims of a long-form answer are entailed by these passages, and we only keep the long-form answers that meet this criteria. The RefChecker we used here are described in Appendix G.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A3 Corpus Downsampling for Science and Biomedical Domains

In addition to long-form answer generation, we also perform downsampling for the corpora of Science and Biomedical domains as they are much larger than the others, with over 1 million documents each. Building indexes for a dense retriever is very costly for large corpora, so we downsample these domains to lower the evaluation cost for the community. For the biomedical domain, we first use BM25 retriever to obtain top 400 documents for each question. The subsampled corpus is formed by combining all documents from the retriever with annotated relevant documents from the datasets. Based on our initial study, we observe that the BM25 retriever yield competitive performance against the dense retriever, so we decide to only use the BM25 retriever for downsampling purpose to save computation cost. For the science domain, we leverage both the BM25 retriever and e5-mistral-7b-instruct based dense retriever to obtain document candidates. Specifically, we retrieve the top 200 documents from both retrievers (400 documents in total before deduplication). Similarly, the combination of all documents from the retrievers and annotated relevant documents forms datasets forms the downsampled corpus.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A4 License of The Datasets

The annotations from RobustQA, ClapNQ and NovelQA are under Apache-2.0 License. The corpora of Finance and annotations of KIWI are under CC-BY-SA-4.0. BioASQ is under CC BY 2.5 license. The license for the corpora of LoTTE are not specified.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:B The complete formula for all metrics

Denote the model response as m, the ground truth answer as gt, and the retrieved chunks as {chunk_k}. Leveraging RefChecker, we decompose the text into a set of claims {ci} and assess whether a specific claim ci can entail (ε) or not entail (∉) a given reference text Ref, where Ref may represent m, gt, or {chunk_k}. We assign an entailment label to each ground-truth claim relative to a chunk, and subsequently classify these chunks into relevant chunks {r-chunk_j} and irrelevant chunks


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:B.1 Overall Metrics

Precision = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \in gt|}{|\mathcal{C}^{(m)}_i|}\\ Recall = \frac{|\mathcal{C}^{(gt)}_i \cap \mathcal{C}^{(gt)}_i \in m|}{|\mathcal{C}^{(gt)}_i|}


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:B.2 Retriever Metrics

Claim Recall = \frac{|\mathcal{C}^{(gt)}_i \cap \mathcal{C}^{(gt)}_i \in \{chunk_j\}|}{|\mathcal{C}^{(gt)}_i|}\\ Context Precision = \frac{|r-chunk_j|}{k}


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:B.3 Generator Metrics

Faithfulness = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \in \{chunk_j\}|}{|\mathcal{C}^{(m)}_i|}\\ Relevant Noise Sensitivity = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \not\in gt \text{ and } \mathcal{C}^{(m)}_i \in \{r-chunk_j\}|}{|\mathcal{C}^{(m)}_i|}\\ Irrelevant Noise Sensitivity = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \not\in gt \text{ and } \mathcal{C}^{(m)}_i \in \{irr-chunk_j\}|}{|\mathcal{C}^{(m)}_i|}\\ Hallucination = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \not\in gt \text{ and } \mathcal{C}^{(m)}_i \not\in \{chunk_j\}|}{|\mathcal{C}^{(m)}_i|}\\ Self-knowledge = \frac{|\mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \in gt \text{ and } \mathcal{C}^{(m)}_i \not\in \{chunk_j\}|}{|\mathcal{C}^{(gt)}_i|}\\ Context Utilization = \frac{|\mathcal{C}^{(gt)}_i \cap \mathcal{C}^{(gt)}_i \in \{chunk_j\} \text{ and } \mathcal{C}^{(gt)}_i \in m|}{|\mathcal{C}^{(gt)}_i \cap \mathcal{C}^{(gt)}_i \in \{chunk_j\}|}


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:C Details of Meta Evaluation

In the meta evaluation, we ask 10 annotators compare two responses from the RAG system for each instance in the meta evaluation dataset. Seven of the annotators are in-house annotators, and three of them are graduate students. We pay the students 15 USD per hour and totally cost 255 dollars.\\ Annotators are required to choose their preference from five options: significantly better, slightly better, tie, slightly worse, or significantly worse. The annotation is based on three metrics: correctness, completeness, and overall assessment. The annotation interface with instructions are shown in Fig. 3.\\ To make sure the human evaluation to be agnostic to specific evaluation metrics, we provide the annotators with a detailed annotation guideline which contains detailed instruction and 5 examples.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Human Annotation Interface

Figure 3: The human annotation interface and instructions of the meta evaluation dataset. In the UI of the annotation tool, each response is shown with critiques generated by GPT-4 and we ask the annotators to refer to the content of the response and the critiques for labeling. The critiques are generated by prompting GPT-4 to compare the response with ground truth answer to ease the annotation job. In addition, each of the example the guideline are shown with a human-written explanation for the labeling. The 10 metrics included in the meta evaluation are selected from Trulens [6], RAGAS [5], ARES [35] and CRUD-RAG [25] as explained in Sec. 4.2. Their descriptions are summarized in Tab. 4. As a supplement of Tab. 2, the full correlation results of meta evaluation is shown in Tab. 5. For a detailed comparison between RAGChecker and the strongest baseline metric, RAGAS Answer Similarity, we plot the prediction score distribution of two metrics in Fig. 4. From the prediction score distribution and the mean line (dashed line) of the plot, we can observe a stronger correlation of RAGChecker than RAGAS Answer Similarity.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Details of the Experiment Setup

Models in Baseline RAG Systems We use the version of e5-mistral-7b-instruct for the E5-Mistral retriever. For the generators, we use gpt-4-turbo-2024-04-09 version for GPT-4, Llama3-88-Instruct for Llama3-88 and Llama3-70B-Instruct for Llama3-70B, and Mixtral-8x7B-Instruct-v0.1 for Mixtral-8x7B. We adopt OpenSearch [4] as the tool to implement the inverted index for BM25 and the approximate KNN search for dense retrieval. We use a p5.48xlarge instance with 8 NVIDIA A100 GPUs on AWS for inference of open-source models. We split documents in the corpus to chunks of 300 tokens with an overlap ratio of 0.2 by default. We use the tokenizer of E5-Mistral for both retrievers to control the chunking. For each query, top-20 chunks ranked by retrievers are used as context for LLM generation. The default prompt for all generators is shown in Fig. 5. We set the generation temperature to 0.8 (deterministic) and the maximum generation length to 2,048 tokens when calling proprietary LLMs.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Summary of Metrics

Table 4: Summary of the metrics included in the meta evaluation.

| Baseline | Metric              | Description                                                                                |
|----------|---------------------|---------------------------------------------------------------------------------------------------------------------------|
| TruLens  | Groundedness        | Assesses the overlap between each statement in the response and the provided context using an LLM.                        |
|          | Answer Relevance    | Prompts an LLM to give a relevance score between the response and question.                                                |
|          | Faithfulness        | Measures the proportion of claims in the response that can be inferred from the context.                                   |
| RAGAS    | Answer Relevance    | Computes the mean cosine similarity between the original question and a series of LLM-generated questions derived from the response and context.                                            |
|          | Answer Similarity   | Measures the semantic similarity between the response and the ground truth answer based on text-embedding-ada-002.         |
|          | Answer Correctness  | Quantifies both the semantic similarity and the factual overlap between the response and the ground truth answer.           |
| ARES     | Answer Faithfulness | Prompts an LLM to determine whether the response is faithful to the context.                                                |
|          | Answer Relevance    | Prompts an LLM to measure whether the response addresses all aspects of the question and provides only correct information from the context.                   |
| CRUD-RAG | Recall              | Computes the ratio of all questions generated from ground truth answers that can be answered by response.                   |
|          | Precision           | Evaluates if the generated text is accurate and consistent with the ground truth answer.                                  |


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Correlation Results

Table 5: Full Correlation results with Human Evaluation of Correctness, Completeness, and Overall Assessment.

| Baseline     | Metric              | Correctness |            | Completeness |            | Overall Assessment |            |
|--------------|---------------------|-------------|------------|--------------|------------|--------------------|------------|
|              |                     | Pearson     | Spearman   | Pearson      | Spearman   | Pearson            | Spearman   |
| BLEU         | BLEU-avg            | 38.89       | 35.32      | 32.13        | 21.85      | 35.14              | 29.42      |
| ROUGE        | ROUGE-L             | 31.75       | 31.72      | 47.88        | 45.67      | 43.10              | 43.21      |
| BERTScore    | BERTScore           | 30.34       | 27.05      | 37.93        | 40.05      | 33.51              | 35.57      |
| TruLens      | Groundedness        | 21.11       | 18.21      | 14.01        | 6.02       | 19.45              | 14.42      |
|              | Answer Relevance    | 25.01       | 27.37      | 37.24        | 37.91      | 35.15              | 33.59      |
| ARES         | Answer Relevance    | 18.63       | 16.84      | 20.13        | 18.13      | 17.81              | 16.26      |
|              | Answer Faithfulness | 9.46        | 7.60       | 10.25        | 8.99       | 8.80               | 7.58       |
| RAGAS        | Faithfulness        | 8.22        | 7.53       | 4.90         | 1.19       | 7.83               | 5.55       |
| RAGAS        | Answer Correctness  | 39.11       | 36.30      | 36.42        | 36.04      | 38.01              | 37.14      |
| RAGAS        | Answer Similarity   | 41.07       | 43.21      | 53.16        | 61.35      | 48.31              | 57.23      |
| RAGAS        | Answer Relevance    | 11.59       | 8.19       | 9.39         | 15.35      | 10.27              | 11.83      |
| CRUD-RAG     | Precision           | 20.73       | 15.67      | 25.58        | 20.33      | 25.95              | 19.63      |
| CRUD-RAG     | Recall              | 30.93       | 27.13      | 45.11        | 43.76      | 41.25              | 39.71      |
| RAGChecker   | Same metrics as human | 49.66     | 46.95      | 60.67        | 58.11      | 61.95              | 60.90      |
| Human        | Annotator sanity check | 63.67    | 59.19      | 71.91        | 68.36      | 70.09              | 68.89      |


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Comparison of Prediction Score Distribution

Figure 4: Comparison of prediction score distribution between RAGCHECKER and RAGAS Answer Similarity. Each point in the plot represents an instance in the meta evaluation dataset, where the x-axis is the human preference label under corresponding aspect and y-axis is the prediction score of RAGCHECKER and RAGAS Answer Similarity. The distribution of prediction score is represented by the colored area and the dashed line is the mean line.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Diagnosis on RAG for Improvements

We modify hyper-parameters commonly tuned in RAG systems to observe performance variance under the metrics defined by RAGCHECKER. We focus on how RAGCHECKER explains this variance and provides tuning suggestions for improvements on certain aspects. In this section, we evaluate three RAG baselines (BM25-GPT-4, E5-Mistral_GPT-4, and E5-Mistral_Llama3-70B) across three domains with increasing difficulty: Writing, Finance, and KIWI. We use our default settings (Appendix D) in main experiments as controls. We experiment with different numbers of chunks selected as context k ∈ {5,10,20}, different chunk size {150,300,600}, different chunk overlap ratio {0.0,0.2,0.4}, and different generation prompts.

More Context Enhances Faithfulness Top-k selection and chunk size both balance the amount of noise and useful information presented to the generator, but in different manners. Corresponding results are demonstrated in Fig. 6 and Fig. 7. Increasing k adds more context that could be less relevant, while increasing chunk size provides more surrounding context of relevant facts. Thus context precision decreases with larger k but increases with larger chunk sizes. Despite this, they both lead to better claim recall in Retrieval.

Generators tend to be more faithful when provided with more context, though this trend is less pronounced for Llama3, which already exhibits high faithfulness. Context utilization generally worsens with more context due to increasing noise, leading to higher relevant noise sensitivity.

Overall, the end-to-end RAG performance is slightly better with more context, primarily due to improved recall. We recommend moderately increasing the two parameters for more faithful answers, noting that saturation occurs at high values as the amount of useful information is limited. Given a limited context length, a larger chunk size with a smaller k is preferred, especially for easier datasets (Finance, Writing). This is evident when comparing a chunk size of 150 with k=20 against a chunk size of 300 with k=10.

Explicit Requirements in Prompts Affect Generation Preferences To validate the effect of the generation prompt, we added more detailed requirements to guide the generation for better faithfulness, context utilization, and lower noise sensitivity. The optimized prompt is shown in Fig. 9.

As shown in Fig. 8, we observed a general improvement in context utilization. However, as a counterpart to context utilization, noise sensitivity generally worsened. It demonstrates the difficulty of meeting all prompt requirements when there are subtle tension between them.

For the two generators, GPT-4 generally shows improvements in metrics related to faithfulness (hallucination, self-knowledge, faithfulness), whereas Llama3 does not exhibit the same behavior. This aligns with our previous observation (Sec. 4.3) that Llama3 already performs well on faithfulness, self-knowledge, and hallucinations.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Diagnosis on Top-k Selection

while GPT-4 tends to rely on self-knowledge without explicit requirements. Consequently, there is a steady improvement in overall F1 for GPT-4 when switched to the optimized prompt, while the difference for Llama3 is negligible.

RAG builders can optimize prompts by combining performance on modular metrics provided by RAGCHECKER with user preferences and generator capabilities on different aspects.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Chunk Overlap Does Not Matter a Lot

Chunk overlap ratio between adjacent chunks is usually set to be non-zero to help the generator better utilize surrounding information and identify chunks with coherent logic, thus alleviating the impact of hard splits in significant semantics.

According to our results in Fig. 10, higher overlap ratios generally lead to improved context precision. However, this does not necessarily translate to an increase in the total amount of useful information retrieved. This phenomenon can be attributed to the retrieval of more chunks that contain the same segment of useful information. Consequently, we observed that overlap ratio adjustments do not have a significant impact on other performance metrics in a consistent and obvious manner. This suggests that the overlap ratio may not require extensive tuning in practice.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Performance Validation of RefChecker with Llama3 Extractor and Checker

G. Performance Validation of RefChecker with Llama3 Extractor and Checker

We use Llama3-70B-Instruct for the extractor and checker in RefChecker. To validate the effectiveness of this combination, we test its performance on the RefChecker benchmark. As shown in Tab. 16, Llama 3 based RefChecker outperforms the best purely open-sourced combinations reported in the RefChecker paper in all the three context settings.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:H Limitations

While RAGCHECKER provides a comprehensive evaluation framework for RAG systems, it has a few limitations that should be acknowledged and addressed in future research.

First, the diagnostic metrics for the retriever component are less insightful compared to those for the generator. The retrieval metrics primarily focus on the recall of ground truth claims and precision of retrieved context, but they may not fully capture the nuances and complexities of the retrieval process. Developing more sophisticated metrics that consider factors such as the information density, diversity and coherence of the retrieved context could provide deeper insights into the retriever’s performance.

Second, the metrics proposed in RAGCHECKER do not differentiate between Neutral and Contradiction checking results from RefChecker when evaluating the generated responses. These two types of results may have different impacts on the final response quality, and treating them equally could lead to an incomplete assessment. Future work should explore ways to incorporate the distinction between neutral and contradiction results into the evaluation metrics, potentially assigning different weights or penalties based on their severity.

Finally, the evaluation benchmark used in this study is curated based on existing text-only datasets and is limited to English queries and corpus. While this allows for a focused evaluation of RAG systems, it may not fully represent the diverse range of tasks and languages that RAG systems can be applied to. Expanding the benchmark to include datasets from different modalities (e.g., images, audio) and languages would provide a more comprehensive assessment of RAG systems’ capabilities and generalization. Additionally, creating benchmark datasets specifically designed for evaluating RAG systems, rather than repurposing existing ones, could help to better capture the unique challenges and requirements of this task.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Figures and Analysis

Figure 7: Diagnosis on Chunk Size
The figure shows spider charts for various tasks—Base GPT-4, Summar GPT-4, Expert Human, Writing, Finance, KWM—evaluating the effects of chunk size. The metrics such as CU, Faith, CR, and others are indicated around each spider chart, and evaluations are done for chunk sizes 150, 300, and 500. The charts are depicted in a grid layout with labels for each task at the bottom. The main observations likely relate to how different chunk sizes impact the performance across the different analytical metrics shown in the spider charts.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Figure 8: Diagnosis on Generation Prompts

By refining the diagnostic metrics, incorporating the impact of different checking results, and expanding the evaluation benchmark, researchers can gain an even more comprehensive understanding of RAG systems’ performance and identify targeted areas for improvement.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Potential Negative Societal Impacts

The RAGCHECKER evaluation framework, while beneficial for assessing RAG systems, could inadvertently lead to several negative societal impacts. There is a risk that developers may focus on optimizing for RAGCHECKER’s specific metrics to the detriment of broader utility and ethical considerations. The computational and financial requirements to meet RAGCHECKER standards could disadvantage smaller organizations, potentially centralizing innovation among well-resourced entities. Moreover, an overreliance on quantitative measures might neglect qualitative factors like user experience and ethical implications.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table 6: Evaluation results for different RAG systems on ClapNQ dataset

RAG Systems | Overall | Retriever | Generator
Proc. | Rec. | F1 | CRT | CP | CU | NSQl | NSQlU | Halluci. | S2I | Faith | #Claim
BM25_GPT-4 | 56.9 | 50.0 | 46.7 | 81.1 | 41.3 | 56.4 | 29.4 | 5.9 | 7.5 | 2.2 | 90.3 | 8
BM25_Llama2-7b | 49.6 | 48.6 | 42.2 | 81.1 | 41.3 | 55.2 | 31.9 | 7.5 | 10.2 | 2.0 | 97.2 | 10
BM25_Llama-13b | 56.9 | 48.7 | 45.1 | 81.1 | 41.3 | 55.7 | 30.1 | 7.1 | 5.9 | 1.6 | 92.4 | 7
BM25_Mistral-7b+ | 47.9 | 49.6 | 49.7 | 81.1 | 41.3 | 55.8 | 36.9 | 7.3 | 6.9 | 2.3 | 90.9 | 9
E5-Mistral_GPT-4 | 59.7 | 57.1 | 51.7 | 81.5 | 43.6 | 56.9 | 31.1 | – | 5.8 | – | 23.7 | 92.3 | –
E5-Mistral_Llama2-7b | 50.4 | 50.9 | 43.9 | 81.5 | 43.6 | 59.4 | 33.2 | 6.4 | 10.0 | 1.5 | 85.5 | 10
E5-Mistral_Llama3-13b | 57.2 | 52.8 | 48.1 | 81.5 | 43.6 | 61.4 | 32.0 | 5.1 | 3.4 | 2.1 | 97.3 | 8
E5-Mistral_Mistral-8b7b | 51.4 | 44.7 | 51.5 | 43.6 | 63.2 | 37.0 | 5.2 | 5.5 | 15 | 93.0 | 10


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Optimized Prompt for Response Generation

You are an accurate and reliable AI assistant capable of answering questions using external documents. Always be faithful to the provided documents and leverage relevant, accurate information from them as much as possible. Be aware that external documents might contain noisy or factually incorrect data. Apply critical reasoning to discern and use the correct information from these sources.

<context>
<content>
<chunk_1>
</content>
<content>
<chunk_2>
</content>
...
<content>
<chunk_k>
</content>
</context>

Question: {question}

Please answer the question and tag your answer with <answer></answer>.

Figure 9: The optimized prompt for response generation. In this prompt, we explicitly instruct the LLMs to be faithful to the context and identify relevant information as possible.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation Results on NovelQA Dataset

Table 7: Evaluation results for different RAG systems on NovelQA dataset

| RAG systems              | Overall         | Retriever          | Generator          |
|--------------------------|-----------------|--------------------|--------------------|
|                          | Prec. | Rec. | F1  | CR  | CP  | CU  | NS(J) | NS(U) | Hallu. | SKJ  | Faith. | #Claim |
| BM25_GPT-4               | 71.10 | 56.2 | 56.4 | 82.1 | 42.6 | 64.9 | 17.6 | 5.4  | 6.1   | 2.2  | 91.7   | 4      |
| BM25_Llama4-8b           | 60.2  | 47.8 | 45.9 | 82.1 | 42.6 | 55.2 | 23.1 | 7.1  | 9.6   | 15.8 | 88.8   | 3      |
| BM25_Llama3-70b          | 65.0  | 51.8 | 51.9 | 82.1 | 42.6 | 59.6 | 21.4 | 7.5  | 6.1   | 2.1  | 91.8   | 3      |
| BM25_Mistral8x7b         | 55.0  | 50.2 | 46.0 | 82.1 | 42.6 | 58.4 | 24.8 | 6.4  | 10.9  | 2.3  | 86.8   | 4      |
| E5-Mistral_GPT-4         | 69.4  | 56.2 | 55.7 | 82.7 | 45.1 | 66.0 | 19.4 | 6.1  | 5.1   | 1.7  | 92.3   | 4      |
| E5-Mistral_Llama4-8b     | 58.7  | 48.1 | 45.7 | 82.7 | 45.1 | 55.1 | 23.8 | 8.1  | 9.2   | 15.8 | 90.3   | 3      |
| E5-Mistral_Llama3-70b    | 60.5  | 50.8 | 49.6 | 82.7 | 45.1 | 56.9 | 23.7 | 5.6  | 6.0   | 1.5  | 92.4   | 3      |
| E5-Mistral_Mistral8x7b   | 54.2  | 48.3 | 43.6 | 82.7 | 45.1 | 54.7 | 29.6 | 6.9  | 7.5   | 1.6  | 90.9   | 4      |


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation Results on RobustQA - Writing Dataset

Table 8: Evaluation results for different RAG systems on RobustQA - Writing dataset

| RAG systems              | Overall         | Retriever          | Generator          |
|--------------------------|-----------------|--------------------|--------------------|
|                          | Prec. | Rec. | F1  | CR  | CP  | CU  | NS(J) | NS(U) | Hallu. | SKJ  | Faith. | #Claim |
| BM25_GPT-4               | 76.3  | 63.6 | 66.0 | 86.3 | 64.3 | 70.0 | 17.5 | 1.0  | 5.1   | 4.0  | 90.9   | 10     |
| BM25_Llama4-8b           | 65.0  | 59.7 | 57.7 | 86.3 | 64.3 | 66.1 | 26.0 | 1.8  | 6.2   | 23.2 | 91.4   | 10     |
| BM25_Llama3-70b          | 72.2  | 62.1 | 63.6 | 86.3 | 64.3 | 68.4 | 23.1 | 1.5  | 3.2   | 2.2  | 94.7   | 8      |
| BM25_Mistral8x7b         | 67.0  | 60.1 | 59.8 | 86.3 | 64.3 | 66.1 | 25.2 | 1.5  | 4.0   | 2.2  | 93.8   | 8      |
| E5-Mistral_GPT-4         | 71.1  | 65.0 | 65.9 | 91.7 | 66.3 | 69.0 | 17.9 | 1.2  | 3.9   | 1.5  | 95.0   | 10     |
| E5-Mistral_Llama4-8b     | 68.7  | 64.5 | 66.6 | 91.7 | 66.3 | 66.8 | 25.5 | 2.2  | 3.5   | 1.6  | 94.9   | 9      |
| E5-Mistral_Llama3-70b    | 71.3  | 65.7 | 66.2 | 91.7 | 66.3 | 70.1 | 23.5 | 1.9  | 1.5   | 0.5  | 95.9   | 9      |
| E5-Mistral_Mistral8x7b   | 66.4  | 62.2 | 61.3 | 91.7 | 66.3 | 66.4 | 26.3 | 2.0  | 3.2   | 0.4  | 96.4   | 9      |


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Figure 10: Diagnosis on Chunk Overlap Ratio

The page presents a series of radar plots (spider charts) showing various evaluation metrics such as Faith, CP, NS(DL), Hallu, SkJ, Faith+, CU, and CU+. The charts compare these metrics across three conditions for different categories of writing, finance, and KWI.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table 9: Evaluation results for different RAG systems on RobustQA - BioASQ dataset

This table displays the performance evaluation of different RAG systems on the BioASQ dataset. Various systems including B M 25 G P T 4, BM 25 Llama 1 3b, BM 25 Llama 7 0b, and others are compared across multiple metrics. The metrics include Pre c. (Precision), Re c. (Recall), F 1 (F1 score), CR t, CP r, CU +, NS(DL), NSII(b), Hallu, SkJ, Faith +, and #C la i m. Each system shows different levels of performance with B M 25 G P T 4 generally leading with better metrics in many categories.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table 10: Evaluation results for different RAG systems on RobustQA - Finance dataset

Similar to Table 9, this table provides a comparative analysis of various RAG systems on the Finance dataset. It outlines systems such as BM 25 G P T 4, BM 25 Llama 1 3b, BM 25 Llama 7 0b, and others, along with their respective evaluation metrics including Pre c., Re c., F 1, CR t, CP r, CU +, NS(DL), NSII(b), Hallu, SkJ, Faith +, and #C la i m. Here too, B M 25 G P T 4 generally exhibits superior performance across most metrics.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation Results

Table 11: Evaluation results for different RAG systems on RobustQA - Lifestyle dataset
----------------------------------------------------------------------
RAG systems | Prec. | Rec. | F1 | CRT | CPT | CPut | NS(N) | NS(NU) | Halluc. | SkJ | Faith.* | #Claims
----------------------------------------------------------------------
BM25_GF1_4 | 63.3 | 50.5 | 53.5 | 70.2 | 47.0 | 64.8 | 24.0 | 2.7 | 11.0 | 6.8 | 40.4 | 12
BM25_Llama1-8b | 49.7 | 44.5 | 43.8 | 70.2 | 47.0 | 59.2 | 33.2 | 6.1 | 11.1 | 8.2 | 36.8 | 12
BM25_Llama3-70b | 59.6 | 44.4 | 44.5 | 70.2 | 47.0 | 58.5 | 30.7 | 3.8 | 5.9 | 24.9 | 91.8 | 9
BM25_Miniv3-87b | 52.8 | 43.5 | 44.1 | 70.2 | 47.0 | 56.8 | 34.5 | 4.9 | 6.8 | 2.3 | 90.9 | 10
E5-Mistral_GF1-4 | -*- | -* | 57.6 | 56.9 | 89.7 | 64.0 | 56.7 | 26.6 | 2.2 | - | - | 16.4 | 15
E5-Mistral_Llama1-8b | 66.4 | 56.0 | 56.9 | 89.7 | 64.0 | 56.7 | 24.6 | 1.4 | 2.5 | 0.6 | 93.4 | 14
E5-Mistral_Llama3-70b | 54.6 | 56.7 | 57.3 | 89.7 | 64.0 | 61.7 | 39.3 | 3.3 | 2.6 | 0.7 | 96.7 | 12
E5-Mistral_Miniv3x87b | 56.2 | 51.8 | 50.5 | 89.7 | 64.0 | 56.5 | 34.8 | 4.6 | 3.0 | 0.8 | 96.2 | 11

Table 12: Evaluation results for different RAG systems on RobustQA - Recreation dataset
----------------------------------------------------------------------
RAG systems | Prec. | Rec. | F1 | CRT | CPT | CPut | NS(N) | NS(NU) | Halluc. | SkJ | Faith.* | #Claims
----------------------------------------------------------------------
BM25_GF1_4 | 62.9 | 51.9 | 53.1 | 70.4 | 67.2 | 65.7 | 21.4 | 4.6 | 11.1 | 5.2 | 83.7 | 11
BM25_Llama1-8b | 50.6 | 45.1 | 43.1 | 70.4 | 37.5 | 58.1 | 31.0 | 10.1 | 8.1 | 1.8 | 90.2 | 8
BM25_Llama3-70b | 51.1 | 45.0 | 44.5 | 70.4 | 37.6 | 60.2 | 30.4 | 6.3 | 5.9 | 5.1 | 91.8 | 9
BM25_Miniv3-87b | 50.5 | 45.1 | 44.1 | 70.4 | 37.5 | 56.6 | 33.0 | 6.3 | 3.9 | 7.8 | 91.5 | 8
E5-Mistral_GF1-4 | -*- | -* | 57.0 | 56.1 | 85.7 | 51.1 | 64.2 | 27.8 | - | - | - | 95.7 | 12
E5-Mistral_Llama1-8b | 55.5 | 47.3 | 45.4 | 85.1 | 51.1 | 59.1 | 34.1 | 3.8 | 3.5 | - | 94.0 | 13
E5-Mistral_Llama3-70b | 60.1 | 53.7 | 52.7 | 89.5 | 55.0 | 68.3 | 30.8 | 6.2 | 2.6 | 0.7 | 92.8 | 12
E5-Mistral_Miniv3x87b | 52.1 | 51.8 | 47.9 | 85.1 | 51.1 | 58.6 | 34.1 | 8.2 | 3.9 | 0.6 | 95.1 | 11

Table 13: Evaluation results for different RAG systems on RobustQA - Science dataset
----------------------------------------------------------------------
RAG systems | Prec. | Rec. | F1 | CRT | CPT | CPut | NS(N) | NS(NU) | Halluc. | SkJ | Faith.* | #Claims
----------------------------------------------------------------------
BM25_GF1_4 | 58.5 | 47.1 | 51.1 | 71.3 | 62.6 | 56.2 | 23.2 | 5.3 | 14.2 | 1.7 | 84.1 | 14
BM25_Llama1-8b | 47.9 | 45.2 | 41.7 | 71.3 | 62.6 | 58.2 | 32.0 | 5.3 | 14.2 | 1.7 | 84.1 | 14
BM25_Llama3-70b | 51.0 | 45.5 | 45.0 | 71.3 | 62.6 | 57.7 | 33.0 | 6.6 | 7.1 | 8.7 | 91.8 | 11
BM25_Miniv3-87b | 51.5 | 44.1 | 45.1 | 71.3 | 62.6 | 56.5 | 34.8 | 5.4 | 6.5 | 7.8 | 91.5 | 11
E5-Mistral_GF1-4 | -*- | -* | 57.5 | 55.0 | 85.0 | 71.8 | 56.5 | 31.5 | 2.3 | - | - | 91.7 | 15
E5-Mistral_Llama1-8b | 48.8 | 45.5 | 45.5 | 71.8 | 58.5 | 71.8 | 54.8 | 3.9 | 3.6 | - | 91.5 | 13
E5-Mistral_Llama3-70b | 57.7 | 51.1 | 48.5 | 71.8 | 55.0 | 71.8 | 57.7 | 36.1 | 3.9 | 3.2 | 96.5 | 12
E5-Mistral_Miniv3x87b | 54.5 | 49.2 | 47.4 | 85.0 | 71.8 | 55.3 | 37.1 | 3.7 | 4.4 | 0.6 | 96.0 | 11

Table 14: Evaluation results for different RAG systems on RobustQA - Technology dataset
----------------------------------------------------------------------
RAG systems | Prec. | Rec. | F1 | CRT | CPT | CPut | NS(N) | NS(NU) | Halluc. | SkJ | Faith.* | #Claims
----------------------------------------------------------------------
BM25_GF1_4 | 57.5 | 48.5 | 49.5 | 69.8 | 68.3 | 63.4 | 28.1 | 3.1 | 11.2 | 4.3 | 84.5 | 14
BM25_Llama1-8b | 47.2 | 44.8 | 43.1 | 69.5 | 63.8 | 61.2 | 36.3 | 3.4 | 10.1 | 1.7 | 84.5 | 14
BM25_Llama3-70b | 55.9 | 44.4 | 45.9 | 69.5 | 68.3 | 59.2 | 36.5 | 5.4 | 4.6 | 1.2 | 94.0 | 13
BM25_Miniv3-87b | 55.9 | 44.4 | 45.9 | 69.5 | 68.3 | 59.2 | 36.5 | 5.4 | 4.6 | 1.2 | 94.0 | 15
E5-Mistral_GF1-4 | -*- | -* | 57.9 | 55.0 | 85.3 | 69.3 | 56.5 | 31.5 | 2.3 | - | - | 92.7 | 15
E5-Mistral_Llama1-8b | 50.9 | 51.1 | 48.3 | 87.3 | 76.4 | 59.0 | 40.4 | 5.6 | 5.9 | 0.5 | 93.5 | 14
E5-Mistral_Llama3-70b | 56.9 | 50.3 | 50.7 | 87.3 | 76.4 | 62.1 | 36.7 | 3.8 | 2.6 | 0.3 | 97.1 | 13
E5-Mistral_Miniv3x87b | 52.6 | 51.5 | 47.9 | 83.7 | 76.4 | 59.6 | 40.5 | 3.4 | 3.2 | 0.2 | 96.6 | 12

Table 15: Evaluation results for different RAG systems on KIWI dataset
----------------------------------------------------------------------
RAG systems | Prec. | Rec. | F1 | CRT | CPT | CPut | NS(N) | NS(NU) | Halluc. | SkJ | Faith.* | #Claims
----------------------------------------------------------------------
BM25_GF1_4 | 42.8 | 30.0 | 32.4 | 57.8 | 72.5 | 49.1 | 45.0 | 6.2 | 6.0 | 0.7 | 93.3 | 18
BM25_Llama1-8b | 43.0 | 24.7 | 26.5 | 72.5 | 72.5 | 39.6 | 41.8 | 4.8 | 9.1 | 1.7 | 92.8 | 16
BM25_Llama3-70b | 27.7 | 21.0 | 31.0 | 58.8 | 72.5 | 33.5 | 45.2 | 6.8 | 5.7 | 0.9 | 91.6 | 8
BM25_Miniv3x87b | 42.0 | 21.6 | 23.8 | 57.8 | 72.5 | 38.5 | 51.0 | 5.8 | 3.1 | 0.3 | 96.0 | 13
E5-Mistral_GF1-4 | -*- | -* | 44.5 | 57.8 | 72.5 | 44.9 | 43.4 | 4.6 | - | - | - | 92.8 | 14
E5-Mistral_Llama1-8b | 45.7 | 27.4 | 30.2 | 57.8 | 72.5 | 50.3 | 40.1 | 9.7 | 6.8 | 0.9 | 91.8 | 13
E5-Mistral_Llama3-70b | 45.2 | 30.9 | 34.0 | 64.6 | 86.7 | 45.3 | 47.5 | 3.7 | 3.6 | 0.3 | 96.1 | 18
E5-Mistral_Miniv3x87b | 36.7 | 23.1 | 23.5 | 64.6 | 86.7 | 32.2 | 45.0 | 4.9 | 2.9 | 0.5 | 96.7 | 14

Page: 26


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Research Paper Content Extraction

**Table 16: Performance of RefChecker on the RefChecker benchmark using Llama 3 70B Instruct as both the extractor and checker.** We compare the results with the best performed purely open-sourced combinations reported in the RefChecker paper.

|                | Accuracy | Fact. F1 | Non-Fact. F1 | Pearson | Spearman |
|----------------|----------|---------|--------------|---------|---------|
| **Zero Context**          |          |               |        |         |
| Mistral-SFT + RepC | 89.38    | 80.43   | 92.72         | 77.14   | 76.74   |
| Llama3 + Llama3    | 91.89    | 83.06   | 94.67         | 81.77   | 80.83   |
| **Noisy Context**        |          |               |        |         |
| Mistral-SFT + NLI  | 70.82    | 75.12   | 64.72         | 52.21   | 45.61   |
| Llama3 + Llama3    | 71.75    | 76.69   | 64.15         | 57.67   | 50.31   |
| **Accurate Context**     |          |               |        |         |
| Mistral-SFT + AlignScore | 74.12    | 81.6    | 56.38         | 46.34   | 43.22   |
| Llama3 + Llama3    | 78.35    | 84.87   | 61.92         | 59.48   | 52.03   |


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

BOCI PENG*, School of Intelligence Science and Technology, Peking University, China
YUN ZHU*, College of Computer Science and Technology, Zhejiang University, China
YONGCHAO LIU, Ant Group, China
XIAOHE BO, Gaoling School of Artificial Intelligence, Renmin University of China, China
HAIZHOU SHI, Rutgers University, US
CHUNTAO HONG, Ant Group, China
YAN ZHANG†, School of Intelligence Science and Technology, Peking University, China
SILIANG TANG, College of Computer Science and Technology, Zhejiang University, China

Recently, Retrieval-Augmented Generation (RAG) has achieved remarkable success in addressing the challenges of Large Language Models (LLMs) without necessitating retraining. By referencing an external knowledge base, RAG refines LLM outputs, effectively mitigating issues such as “hallucination”, lack of domain-specific knowledge, and outdated information. However, the complex structure of relationships among different entities in databases presents challenges for RAG systems. In response, GraphRAG leverages structural information across entities to enable more precise and comprehensive retrieval, capturing relational knowledge and facilitating more accurate, context-aware responses. Given the novelty and potential of GraphRAG, a systematic review of current technologies is imperative. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation. We then outline the core technologies and training methods at each stage. Additionally, we examine downstream tasks, application domains, evaluation methodologies, and industrial use cases of GraphRAG. Finally, we explore future research directions to inspire further inquiries and advance progress in the field.

CCS Concepts: • Computing methodologies → Knowledge representation and reasoning; • Information systems → Information retrieval; Data mining.

Additional Key Words and Phrases: Large Language Models, Graph Retrieval-Augmented Generation, Knowledge Graphs, Graph Neural Networks

1 Introduction

The development of Large Language Models like GPT-4 [116], Qwen2 [170], and LLaMA [24] has sparked a revolution in the field of artificial intelligence, fundamentally altering the landscape of natural language processing. These models, built on Transformer [149] architectures and trained on diverse and extensive datasets, have demonstrated unprecedented capabilities in understanding, interpreting, and generating human language. The impact of these advancements is profound, stretching across various sectors including healthcare [93, 154, 188], finance [84, 114], and education [38, 157], where they facilitate more nuanced and efficient interactions between humans and machines.

*Both authors contributed equally to this research.
†Corresponding Author.

Authors’ Contact Information: Boci Peng, School of Intelligence Science and Technology, Peking University, Beijing, China, bcpeng@stu.pku.edu.cn; Yun Zhu, College of Computer Science and Technology, Zhejiang University, Hangzhou, China, zhuyun_dcd@zju.edu.cn; Yongchao Liu, Ant Group, Hangzhou, China, yongchao.ly@antgroup.com; Xiaohe Bo, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China, bellebhx@gmail.com; Haizhou Shi, Rutgers University, New Brunswick, New Jersey, US, haizhou.shi@rutgers.edu; Chuntao Hong, Ant Group, Hangzhou, China, chuntao.hct@antgroup.com; Yan Zhang, School of Intelligence Science and Technology, Peking University, Beijing, China, zhyyh001@pku.edu.cn; Siliang Tang, College of Computer Science and Technology, Zhejiang University, Hangzhou, China, siliang@zju.edu.cn.


Graph Retrieval-Augmented Generation: A Survey:Comparison between Direct LLM, RAG, and GraphRAG

Fig. 1. Comparison between Direct LLM, RAG, and GraphRAG. Given a user query, direct answering by LLMs may suffer from shallow responses or lack of specificity. RAG addresses this by retrieving relevant textual information, somewhat alleviating the issue. However, due to the text’s length and flexible natural language expressions of entity relationships, RAG struggles to emphasize “influence” relations, which is the core of the question. While, GraphRAG methods leverage explicit entity and relationship representations in graph data, enabling precise answers by retrieving relevant structured information.


Graph Retrieval-Augmented Generation: A Survey:Introduction to RAG and its Limitations

Despite their remarkable language comprehension and text generation capabilities, LLMs may exhibit limitations due to a lack of domain-specific knowledge, real-time updated information, and proprietary knowledge, which are outside LLMs’ pre-training corpus. These gaps can lead to a phenomenon known as “hallucination” where the model generates inaccurate or even fabricated information. Consequently, it is imperative to supplement LLMs with external knowledge to mitigate this problem. Retrieval-Augmented Generation (RAG) emerged as a significant evolution, which aims to enhance the quality and relevance of generated content by integrating a retrieval component within the generation process. The essence of RAG lies in its ability to dynamically query a large text corpus to incorporate relevant factual knowledge into the responses generated by the underlying language models. This integration not only enriches the contextual depth of the responses but also ensures a higher degree of factual accuracy and specificity. RAG has gained widespread attention due to its exceptional performance and broad applications, becoming a key focus within the field. Although RAG has achieved impressive results and has been widely applied across various domains, it faces limitations in real-world scenarios: (1) Neglecting Relationships: In practice, textual content is not isolated but interconnected. Traditional RAG fails to capture significant structured relational knowledge that cannot be represented through semantic similarity alone. For instance, in a citation network where papers are linked by citation relationships, traditional RAG methods focus on finding the relevant papers based on the query but overlook important citation relationships between papers. (2) Redundant Information: RAG often recounts content in the form of textual snippets when concatenated as prompts. This makes context become excessively lengthy, leading to the “lost in the middle” dilemma. (3) Lacking Global Information: RAG can only retrieve a...


Graph Retrieval-Augmented Generation (GraphRAG) emerges as an innovative solution to address these challenges. Unlike traditional RAG, GraphRAG retrieves graph elements containing relational knowledge pertinent to a given query from a pre-constructed graph database, as depicted in Figure 1. These elements may include nodes, triples, paths, or subgraphs, which are utilized to generate responses. GraphRAG considers the interconnections between texts, enabling a more accurate and comprehensive retrieval of relational information. Additionally, graph data, such as knowledge graphs, offer abstraction and summarization of textual data, thereby significantly shortening the length of the input text and mitigating concerns of verbosity. By retrieving subgraphs or graph communities, we can access comprehensive information to effectively address the QFS challenge by capturing the broader context and interconnections within the graph structure.

In this paper, we are the first to provide a systematic survey of GraphRAG. Specifically, we begin by introducing the GraphRAG workflow, along with the foundational background knowledge that underpins the field. Then, we categorize the literature according to the primary stages of the GraphRAG process: Graph-Based Indexing (G-Indexing), Graph-Guided Retrieval (G-Retrieval), and Graph-Enhanced Generation (G-Generation) in Section 5, Section 6 and Section 7 respectively, detailing the core technologies and training methods within each phase. Furthermore, we investigate downstream tasks, application domains, evaluation methodologies, and industrial use cases of GraphRAG. This exploration elucidates how GraphRAG is being utilized in practical settings and reflects its versatility and adaptability across various sectors. Finally, acknowledging that research in GraphRAG is still in its early stages, we delve into potential future research directions. This prognostic discussion aims to pave the way for forthcoming studies, inspire new lines of inquiry, and catalyze progress within the field, ultimately propelling GraphRAG toward more mature and innovative horizons.

Our contributions can be summarized as follows:

- We provide a comprehensive and systematic review of existing state-of-the-art GraphRAG methodologies. We offer a formal definition of GraphRAG, outlining its universal workflow which includes G-Indexing, G-Retrieval, and G-Generation.
- We discuss the core technologies underpinning existing GraphRAG systems, including G-Indexing, G-Retrieval, and G-Generation. For each component, we analyze the spectrum of model selection, methodological design, and enhancement strategies currently being explored. Additionally, we contrast the diverse training methodologies employed across these modules.
- We delineate the downstream tasks, benchmarks, application domains, evaluation metrics, current challenges, and future research directions pertinent to GraphRAG, discussing both the progress and prospects of this field. Furthermore, we compile an inventory of existing industry GraphRAG systems, providing insights into the translation of academic research into real-world industry solutions.

Organization. The rest of the survey is organized as follows: Section 2 compares related techniques, while Section 3 outlines the general process of GraphRAG. Sections 5 to 7 categorize the techniques associated with GraphRAG’s three stages: G-Indexing, G-Retrieval, and G-Generation. Section 8 introduces the training strategies of retrievers and generators. Section 9 summarizes GraphRAG’s downstream tasks, corresponding benchmarks, application domains, evaluation metrics, and industrial GraphRAG systems. Section 10 provides an outlook on future directions. Finally, Section 11 concludes the content of this survey.


Graph Retrieval-Augmented Generation: A Survey:Comparison with Related Techniques and Surveys

In this section, we compare Graph Retrieval-Augmented Generation (GraphRAG) with related techniques and corresponding surveys, including RAG, LLMs on graphs, and Knowledge Base Question Answering (KBQA).

2.1 RAG

RAG combines external knowledge with LLMs for improved task performance, integrating domain-specific information to ensure factuality and credibility. In the past two years, researchers have written many comprehensive surveys about RAG [27, 37, 51, 54, 165, 180, 187]. For example, Fan et al. [27] and Gao et al. [37] categorize RAG methods from the perspectives of retrieval, generation, and augmentation. Zhao et al. [187] review RAG methods for databases with different modalities. Yu et al. [180] systematically summarize the evaluation of RAG methods. These works provide a structured synthesis of current RAG methodologies, fostering a deeper understanding and suggesting future directions of the area.

From a broad perspective, GraphRAG can be seen as a branch of RAG, which retrieves relevant relational knowledge from graph databases instead of text corpus. However, compared to text-based RAG, GraphRAG takes into account the relationships between texts and incorporates the structural information as additional knowledge beyond text. Furthermore, during the construction of graph data, raw text data may undergo filtering and summarization processes, enhancing the refinement of information within the graph data. Although previous surveys on RAG have touched upon GraphRAG, they predominantly center on textual data integration. This paper diverges by placing a primary emphasis on the indexing, retrieval, and utilization of structured graph data, which represents a substantial departure from handling purely textual information and spurs the emergence of many new techniques.


Graph Retrieval-Augmented Generation: A Survey:Figure

Fig. 2. The overview of the GraphRAG framework for question answering task. In this survey, we divide GraphRAG into three stages: G-Indexing, G-Retrieval, and G-Generation. We categorize the retrieval sources into open-source knowledge graphs and self-constructed graph data. Various enhancing techniques like query enhancement and knowledge enhancement may be adopted to boost the relevance of the results. Unlike RAG, which uses retrieved text directly for generation, GraphRAG requires converting the retrieved graph information into patterns acceptable to generators to enhance the task performance.


Graph Retrieval-Augmented Generation: A Survey:LLMs on Graphs

LLMs are revolutionizing natural language processing due to their excellent text understanding, reasoning, and generation capabilities, along with their generalization and zero-shot transfer abilities. Although LLMs are primarily designed to process pure text and struggle with non-Euclidean data containing complex structural information, such as graphs [41, 153], numerous studies [13, 28, 65, 83, 92, 105, 119, 120, 161, 189] have been conducted in these fields. These papers primarily integrate LLMs with GNNs to enhance modeling capabilities for graph data, thereby improving performance on downstream tasks such as node classification, edge prediction, graph classification, and others. For example, Zhu et al. [189] propose an efficient fine-tuning method named ENGNN, which combines LLMs and GNNs through a side structure for enhancing graph representation. Different from these methods, GraphRAG focuses on retrieving relevant graph elements using queries from an external graph-structured database. In this paper, we provide a detailed introduction to the relevant technologies and applications of GraphRAG, which are not included in previous surveys of LLMs on Graphs.


Graph Retrieval-Augmented Generation: A Survey:KBQA

KBQA is a significant task in natural language processing, aiming to respond to user queries based on external knowledge bases [33, 76, 77, 174], thereby achieving goals such as fact verification, passage retrieval enhancement, and text understanding. Previous surveys typically categorize existing KBQA approaches into two main types: Information Retrieval (IR)-based methods and Semantic Parsing (SP)-based methods. Specifically, IR-based methods [60, 61, 102, 142, 155, 168, 181] retrieve information related to the query from the knowledge graph (KG) and use it to enhance the generation process. While SP-based methods [12, 15, 29, 40, 141, 177] generate a logical form (LF) for each query and execute it against knowledge bases to obtain the answer. GraphRAG and KBQA are closely related, with IR-based KBQA methods representing a subset of GraphRAG approaches focused on downstream applications. In this work, we extend the discussion beyond KBQA to include GraphRAG’s applications across various downstream tasks. Our survey provides a thorough and detailed exploration of GraphRAG technology, offering a comprehensive understanding of existing methods and potential improvements.


Graph Retrieval-Augmented Generation: A Survey:Preliminaries

In this section, we introduce background knowledge of GraphRAG for easier comprehension of our survey. First, we introduce Text-Attributed Graphs which is a universal and general format of graph data used in GraphRAG. Then, we provide formal definitions for two types of models that can be used in the retrieval and generation stages: Graph Neural Networks and Language Models.


Graph Retrieval-Augmented Generation: A Survey:Text-Attributed Graphs

The graph data used in Graph RAG can be represented uniformly as Text-Attributed Graphs (TAGs), where nodes and edges possess textual attributes. Formally, a text-attributed graph can be denoted as G = (V, E, A, {x_v}v∈V, {e_i,j}i,j∈E), where V is the set of nodes, E ⊆ V × V is the set of edges, A ∈ {0, 1}|V|×|V| is the adjacent matrix. Additionally, {x_v}v∈V and {e_i,j}i,j∈E are textual attributes of nodes and edges, respectively. One typical kind of TAGs is Knowledge Graphs (KGs), where nodes are entities, edges are relations among entities, and text attributes are the names of entities and relations.


Graph Retrieval-Augmented Generation: A Survey:Graph Neural Networks

Graph Neural Networks (GNNs) are a kind of deep learning framework to model graph data. Classical GNNs, e.g., GCN [74], GAT [150], GraphSAGE [44], adopt a message-passing manner to obtain node representations. Formally, each node representation \( h^{(l-1)}_i \) in the \( l \)-th layer is updated by aggregating the information from neighboring nodes and edges:

\[
    h^{(l)}_i = \text{UPD}(h^{(l-1)}_i , \text{AGG}_{j \in \mathcal{N}(i)} \text{MSG}(h^{(l-1)}_i , h^{(l-1)}_j , e^{(l-1)}_{i,j} )),
\]

where \( \mathcal{N}(i) \) represents the neighbors of node \( i \). MSG denotes the message function, which computes the message based on the node, its neighbor, and the edge between them. AGG refers to the aggregation function that combines the received messages using a permutation-invariant method, such as mean, sum, or max. UPD represents the update function, which updates each node’s attributes with the aggregated messages.

Subsequently, a readout function, e.g., mean, sum, or max pooling, can be applied to obtain the global-level representation:

\[
    h_G = \text{READOUT}_{i \in V_G}(h^{(L)}_i ).
\]

In GraphRAG, GNNs can be utilized to obtain representations of graph data for the retrieval phase, as well as to model the retrieved graph structures.


Graph Retrieval-Augmented Generation: A Survey:Language Models

Language models (LMs) excel in language understanding and are mainly classified into two types: discriminative and generative. Discriminative models, like BERT [22], RoBERTa [97] and Sentence-BERT [129], focus on estimating the conditional probability \( P(y|x) \) and are effective in tasks such as text classification and sentiment analysis. In contrast, generative models, including GPT-3 [10] and GPT-4 [116], aim to model the joint probability \( P(x, y) \) for tasks like machine translation and text generation. These generative pre-trained models have significantly advanced the field of natural language processing (NLP) by leveraging massive datasets and billions of parameters, contributing to the rise of Large Language Models (LLMs) with outstanding performance across various tasks.

In the early stages, RAG and GraphRAG focused on improving pre-training techniques for discriminative language models [22, 97, 129]. Recently, LLMs such as ChatGPT [117], LLaMA [24], and Qwen2 [170] have shown great potential in language understanding, demonstrating powerful in-context learning capabilities. Subsequently, research on RAG and GraphRAG shifted towards enhancing information retrieval for language models, addressing increasingly complex tasks and mitigating hallucinations, thereby driving rapid advancements in the field.


Graph Retrieval-Augmented Generation: A Survey:Overview of GraphRAG

GraphRAG is a framework that leverages external structured knowledge graphs to improve contextual understanding of LMs and generate more informed responses, as depicted in Figure 2. The goal of GraphRAG is to retrieve the most relevant knowledge from databases, thereby enhancing the answers of downstream tasks. The process can be defined as

\[
    a^* = \arg \max_{a \in A} p(a|q, \mathcal{G}),
\]

where \( a^* \) is the optimal answer of the query \( q \) given the TAG \( \mathcal{G} \), and \( A \) is the set of possible responses. After that, we jointly model the target distribution \( p(a|q, \mathcal{G}) \) with a graph retriever \( p_\theta(G|q, \mathcal{G}) \) and an answer generator \( p_\phi(a|q, G) \) where \( \theta, \phi \) are learnable parameters, and utilize the


total probability formula to decompose \(p(a|q, \mathcal{G})\), which can be formulated as

\[ p(a|q, \mathcal{G}) = \sum_{\mathcal{G} \subseteq \mathcal{G}} p_\phi(a|q, G)p_\theta(G|q, \mathcal{G}) \]

\[ \approx p_\phi(a|q, G^*)p_\theta(G^*|q, \mathcal{G}), \]

(4)

where \(G^*\) is the optimal subgraph. Because the number of candidate subgraphs can grow exponentially with the size of the graph, efficient approximation methods are necessary. The first line of Equation 4 is thus approximated by the second line. Specifically, a graph retriever is employed to extract the optimal subgraph \(G^*\), after which the generator produces the answer based on the retrieved subgraph.

Therefore, in this survey, we decompose the entire process of GraphRAG into three main stages: Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation. The overall workflow of GraphRAG is illustrated in Figure 2 and detailed introductions of each stage are as follows.

**Graph-Based Indexing (G-Indexing).** Graph-Based Indexing constitutes the initial phase of GraphRAG, aimed at identifying or constructing a graph database \(\mathcal{G}\) that aligns with downstream tasks and establishing indices on it. The graph database can originate from public knowledge graphs [2, 7, 91, 131, 138, 151], graph data [112], or be constructed based on proprietary data sources such as textual [25, 43, 80, 160] or other forms of data [169]. The indexing process typically includes mapping node and edge properties, establishing pointers between connected nodes, and organizing data to support fast traversal and retrieval operations. Indexing determines the granularity of the subsequent retrieval stage, playing a crucial role in enhancing query efficiency.

**Graph-Guided Retrieval (G-Retrieval).** Following graph-based indexing, the graph-guided retrieval phase focuses on extracting pertinent information from the graph database in response to user queries or input. Specifically, given a user query \(q\) which is expressed in natural language, the retrieval stage aims to extract the most relevant elements (e.g., entities, triplets, paths, subgraphs) from knowledge graphs, which can be formulated as

\[ G^* = G\text{-}Retriever(q, \mathcal{G}) \]

\[ = \arg\max_{G \subseteq \mathcal{G}^{R}(G)} p_\theta(G|q, \mathcal{G}) \]

\[ = \arg\max_{G \subseteq \mathcal{G}^{R}(G)} Sim(q, G), \]

(5)

where \(G^*\) is the optimal retrieved graph elements and \(Sim(\cdot, \cdot)\) is a function that measures the semantic similarity between user queries and the graph data. \(\mathcal{R}(\cdot)\) represents a function to narrow down the search range of subgraphs, considering the efficiency.

**Graph-Enhanced Generation (G-Generation).** The graph-enhanced generation phase involves synthesizing meaningful outputs or responses based on the retrieved graph data. This could encompass answering user queries, generating reports, etc. In this stage, a generator takes the query, retrieved graph elements, and an optional prompt as input to generate a response, which can be denoted as

\[ a^* = G\text{-}Generator(q, G^*) \]

\[ = \arg\max_{a \in \mathcal{A}} p_\phi(a|q, G^*) \]

\[ = \arg\max_{a \in \mathcal{A}} p_\phi(a|\mathcal{F}(q, G^*)), \]

(6)

where \(\mathcal{F}(\cdot, \cdot)\) is a function that converts graph data into a form the generator can process.


Graph Retrieval-Augmented Generation: A Survey:Graph-Based Indexing

The construction and indexing of graph databases form the foundation of GraphRAG, where the quality of the graph database directly impacts GraphRAG’s performance. In this section, we categorize and summarize the selection or construction of graph data and various indexing methods that have been employed.


Graph Retrieval-Augmented Generation: A Survey:5.1 Graph Data

Various types of graph data are utilized in GraphRAG for retrieval and generation. Here, we categorize these data into two categories based on their sources, including Open Knowledge Graphs and Self-Constructed Graph Data.


Graph Retrieval-Augmented Generation: A Survey:5.1.1 Open Knowledge Graphs

Open knowledge graphs refer to graph data sourced from publicly available repositories or databases [2, 7, 138, 151]. Using these knowledge graphs could dramatically reduce the time and resources required to develop and maintain. In this survey, we further classify them into two categories according to their scopes, i.e., General Knowledge Graphs and Domain Knowledge Graphs.

(1) General Knowledge Graphs. General knowledge graphs primarily store general, structured knowledge, and typically rely on collective input and updates from a global community, ensuring a comprehensive and continually refreshed repository of information.

Encyclopedic knowledge graphs are a typical type of general knowledge graph, which contains large-scale real-world knowledge collected from human experts and encyclopedias. For example, Wikidata [151] is a free and open knowledge base that stores structured data of its Wikimedia sister projects like Wikipedia, Wikivoyage, Wiktionary, and others. Freebase [7] is an extensive, collaboratively edited knowledge base that compiles data from various sources, including individual contributions and structured data from databases like Wikipedia. DBpedia [2] represents information about millions of entities, including people, places, and things, by leveraging the infoboxes and categories present in Wikipedia articles. YAGO [138] collects knowledge from Wikipedia, WordNet, and GeoNames.

Commonsense knowledge graphs are another type of general knowledge graph. They include abstract commonsense knowledge, such as semantic associations between concepts and causal relationships between events. Typical Commonsense Knowledge Graphs include: ConceptNet [91] is a semantic network built from nodes representing words or phrases connected by edges denoting semantic relationships. ATOMIC [56, 131] models the causal relationships between events.

(2) Domain Knowledge Graphs. As discussed in Section 1, domain-specific knowledge graphs are crucial for enhancing LLMs in addressing domain-specific questions. These KGs offer specialized knowledge in particular fields, aiding models in gaining deeper insights and a more comprehensive understanding of complex professional relationships. In the biomedical field, CMeKG encompasses a wide range of data, including diseases, symptoms, treatments, medications, and relationships between medical concepts. CPubMed-KG is a medical knowledge database in Chinese, building on the extensive repository of biomedical literature in PubMed. In the movie domain, Wiki-Movies [110]


extracts structured information from Wikipedia articles related to films, compiling data about movies, actors, directors, genres, and other relevant details into a structured format. Additionally, Jin et al. [66] construct a dataset named GR-Bench, which includes five domain knowledge graphs spanning academic, e-commerce, literature, healthcare, and legal fields. Furthermore, He et al. [47] convert triplet-format and JSON files from ExplaGraphs and SceneGraphs into a standard graph format and selects questions requiring 2-hop reasoning from WebQSP to create the universal graph-format dataset GraphQA for evaluating GraphRAG systems.

5.1.2 Self-Constructed Graph Data. Self-Constructed Graph Data facilitates the customization and integration of proprietary or domain-specific knowledge into the retrieval process. For downstream tasks that do not inherently involve graph data, researchers often propose constructing a graph from multiple sources (e.g., documents, tables, and other databases) and leveraging GraphRAG to enhance task performance. Generally, these self-constructed graphs are closely tied to the specific design of the method, distinguishing them from the open-domain graph data previously mentioned. To model the structural relationships between the documents, Munikoti et al. [113] propose to construct a heterogeneous document graph capturing multiple document-level relations, including co-citation, co-topic, co-venue, etc. Li et al. [87] and Wang et al. [160] establish relationships between passages according to shared keywords. To capture the relations between entities in documents, Delile et al. [20], Edge et al. [25], Gutiérrez et al. [43] and Li et al. [80] utilize the named entity recognition tools to extract entities from documents and language models to further extract relations between entities, where the retrieved entities and relations then form a knowledge graph. There are also some mapping methods for downstream tasks that need to be designed based on the characteristics of the task itself. For example, to solve the patent phrase similarity inference task, Peng and Yang [122] convert the patent database into a patent-phrase graph. Connections between patent nodes and phrase nodes are established if the phrases appear in the patents, while connections between patent nodes are based on citation relations. Targeting customer service technical support scenarios, Xu et al. [169] propose to model historical issues into a KG, which transforms the issues into tree representations to maintain the intra-issue relations, and utilize semantic similarities and a threshold to preserve inter-issue relations.

5.2 Indexing Graph-Based Indexing plays a crucial role in enhancing the efficiency and speed of query operations on graph databases, directly influencing subsequent retrieval methods and granularity. Common graph-based indexing methods include graph indexing, text indexing, and vector indexing.

5.2.1 Graph Indexing. Graph indexing represents the most commonly used approach, preserving the entire structure of the graph. This method ensures that for any given node, all its edges and neighboring nodes are easily accessible. During subsequent retrieval stages, classic graph search algorithms such as BFS and Shortest Path Algorithms can be employed to facilitate retrieval tasks [64, 66, 102, 142, 146, 175].

5.2.2 Text Indexing. Text indexing involves converting graph data into textual descriptions to optimize retrieval processes. These descriptions are stored in a text corpus, where various text-based retrieval techniques, such as sparse retrieval and dense retrieval, can be applied. Some approaches transform knowledge graphs into human-readable text using predefined rules or templates. For instance, Li et al. [81], Huang et al. [55] and Li et al. [86] use predefined templates to convert each triple in knowledge graphs into natural language, while Yu et al. [179] merge triplets with the same head entity into passages. Additionally, some methods convert subgraph-level information into.


Graph Retrieval-Augmented Generation: A Survey:Vector Indexing

Vector indexing transforms graph data into vector representations to enhance retrieval efficiency, facilitating rapid retrieval and effective query processing. For example, entity linking can be seamlessly applied through query embeddings, and efficient vector search algorithms such as Locality Sensitive Hashing (LSH) can be utilized. G-Retriever employs language models to encode textual information associated with each node and edge within the graph, while GRAG uses language models to convert k-hop ego networks into graph embeddings, thereby better preserving structural information.


Graph Retrieval-Augmented Generation: A Survey:Remark

These three indexing methods each offer distinct advantages: graph indexing facilitates easy access to structural information, text indexing simplifies retrieval of textual content, and vector indexing enables quick and efficient searches. Therefore, in practical applications, a hybrid approach combining these indexing methods is often preferred over relying solely on one.


Graph Retrieval-Augmented Generation: A Survey:Graph-Guided Retrieval

6. Graph-Guided Retrieval

In GraphRAG, the retrieval process is crucial for ensuring the quality and relevance of generated outputs by extracting pertinent and high-quality graph data from external graph databases. However, retrieving graph data presents two significant challenges: (1) Explosive Candidate Subgraphs: As the graph size increases, the number of candidate subgraphs grows exponentially, requiring heuristic search algorithms to efficiently explore and retrieve relevant subgraphs. (2) Insufficient Similarity Measurement: Accurately measuring similarity between textual queries and graph data necessitates the development of algorithms capable of understanding both textual and structural information.

Considerable efforts have previously been dedicated to optimizing the retrieval process to address the above challenges. This survey focuses on examining various aspects of the retrieval process within GraphRAG, including the selection of the retriever, retrieval paradigm, retrieval granularity, and effective enhancement techniques. The general architectures of Graph-Guided Retrieval are depicted in Figure 3.


Graph Retrieval-Augmented Generation: A Survey:Retriever

6.1 Retriever

In GraphRAG, various retrievers possess unique strengths for addressing different aspects of retrieval tasks. We categorize retrievers into three types based on their underlying models: Non-parametric Retriever, LM-based Retriever, and GNN-based Retriever. It is important to note that...


Graph Retrieval-Augmented Generation: A Survey:Non-parametric Retriever

Non-parametric retrievers, based on heuristic rules or traditional graph search algorithms, do not rely on deep-learning models, thereby achieving high retrieval efficiency. For instance, Yasunaga et al. [175] and Taunk et al. [146] retrieve k-hop paths containing the topic entities of each question-choice pair. G-Retriever [47] enhances the conventional Prize-Collecting Steiner Tree (PCST) algorithm by incorporating edge prices and optimizing relevant subgraph extraction. Delile et al. [20] and Mavromatis and Karypis [108] first extract entities mentioned in the query and then retrieve the shortest path related to these entities. These methods often involve an entity linking pre-processing step to identify nodes in the graph before retrieval.


Graph Retrieval-Augmented Generation: A Survey:LM-based Retriever

LMs serve as effective retrievers in GraphRAG due to their strong natural language understanding capabilities. These models excel in processing and interpreting diverse natural language queries, making them versatile for a wide range of retrieval tasks within graph-based frameworks. We primarily categorized LMs into two types: discriminative and generative language models. Subgraph Retriever [181] trains RoBERTa [97] as the retriever, which expands from the topic entity and retrieves the relevant paths in a sequential decision process. KG-GPT [71] adopts LLMs to generate the set of top-K relevant relations of the specific entity. Wold et al. [164] utilize fine-tuned GPT-2 to generate reasoning paths. StructGPT [58] utilizes LLMs to automatically invoke several pre-defined functions, by which relevant information can be retrieved and combined to assist further reasoning.


Graph Retrieval-Augmented Generation: A Survey:GNN-based Retriever

GNNs are adept at understanding and leveraging complex graph structures. GNN-based retrievers typically encode graph data and subsequently score different retrieval granularities based on their similarity to the query. For example, GNN-RAG [108] first encodes the graph, assigns a score to each entity, and retrieves entities relevant to the query based on a threshold. EtD [90] iterates multiple times to retrieve relevant paths. During each iteration, it first uses LLaMA2 [148] to select edges connecting the current node, then employs GNNs to obtain embeddings of the new layer of nodes for the next round of LLM selection.


During the retrieval process, non-parametric retrievers exhibit good retrieval efficiency, but they may suffer from inaccurate retrieval due to a lack of training on downstream tasks. Meanwhile, although LM-based retrievers and GNN-based retrievers offer higher retrieval accuracy, they require significant computational overhead. Considering this complementarity, many methods propose hybrid retrieval approaches to improve both retrieval efficiency and accuracy. Many approaches adopt a multi-stage retrieval strategy, employing different models at each stage. For example, RoG [102] first utilizes LLMs to generate planning paths and then extracts paths satisfying the planning paths from knowledge graphs. GenTKGQA [36] infers crucial relations and constraints from the query using LLMs and extracts triplets according to these constraints.


Graph Retrieval-Augmented Generation: A Survey:Retrieval Paradigm

Within GraphRAG, different retrieval paradigms, including once retrieval, iterative retrieval, and multi-stage retrieval, play crucial roles in improving the relevance and depth of the retrieved information. Once retrieval aims to gather all pertinent information in a single operation. Iterative retrieval conducts further searches based on previously retrieved information, progressively narrowing down to the most relevant results. Here we further divide iterative retrieval into adaptive retrieval and non-adaptive retrieval, with the only difference lying in whether the stopping of the retrieval is determined by the model. Another retrieval paradigm is multi-stage retrieval, where


Graph Retrieval-Augmented Generation: A Survey:Once Retrieval

Once retrieval aims to retrieve all the relevant information in a single query. One category of approaches [43, 50, 81] utilize embedding similarities to retrieve the most relevant pieces of information. Another category of methods design pre-defined rules or patterns to directly extract specific structured information such as triplets, paths or subgraphs from graph databases. For example, G-Retriever [47] utilizes an extended PCST algorithm to retrieve the most relevant subgraph. KagNet [88] extracts paths between all pairs of topic entities with lengths not exceeding k. Yasunaga et al. [175] and Taunk et al. [146] extract the subgraph that contains all topic entities along with their 2-hop neighbors.

Furthermore, in this subsection, we also include some multiple retrieval methods that involve decoupled and independent retrievals, allowing them to be computed in parallel and executed only once. For example, Luo et al. [102] and Cheng et al. [16] first instruct LLMs to generate multiple reasoning paths and then use a BFS retriever to sequentially search for subgraphs in the knowledge graphs that match each path. KG-GPT [71] decomposes the original query into several sub-queries, retrieving relevant information for each sub-query in a single retrieval process.


Graph Retrieval-Augmented Generation: A Survey:Iterative Retrieval

In iterative retrieval, multiple retrieval steps are employed, with subsequent searches depending on the results of prior retrievals. These methods aim to deepen the understanding or completeness of the retrieved information over successive iterations. In this survey, we further classify iterative retrieval into two categories: (1) non-adaptive and (2) adaptive retrieval. We provide a detailed summary of these two categories of methods below.

(1) Non-Adaptive Retrieval. Non-adaptive methods typically follow a fixed sequence of retrieval, and the termination of retrieval is determined by setting a maximum time or a threshold. For example, PullNet [139] retrieves problem-relevant subgraphs through T iterations. In each iteration, the paper designs a retrieval rule to select a subset of retrieved entities, and then expands these entities by searching relevant edges in the knowledge graph. In each iteration, KGP [160] first selects seed nodes based on the similarity between the context and the nodes in the graph. It then uses LLMs to summarize and update the context of the neighboring nodes of the seed nodes, which is utilized in the subsequent iteration.

(2) Adaptive Retrieval. One distinctive characteristic of adaptive retrieval is to let models autonomously determine the optimal moments to finish the retrieval activities. For instance, [42, 168] leverage an LM for hop prediction, which serves as an indicator to end the retrieval. There is also a group of researchers who utilize model-generated special tokens or texts as termination signals for the retrieval process. For example, ToG [142] prompts the LLM agent to explore the multiple possible reasoning paths until the LLM determines the question can be answered based on the current reasoning path. [181] trains a RoBERTa to expand a path from each topic entity. In the process, a virtual relation named as “[END]” is introduced to terminate the retrieval process.

Another common approach involves treating the large model as an agent, enabling it to directly generate answers to questions to signal the end of iteration. For instance, [58, 60, 66, 143, 158] propose LLM-based agents to reason on graphs. These agents could autonomously determine the information for retrieval, invoke the pre-defined retrieval tools, and cease the retrieval process based on the retrieved information.


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

6.2.3 Multi-Stage Retrieval. Multi-stage retrieval divides the retrieval process linearly into multiple stages, with additional steps such as retrieval enhancement, and even generation processes occurring between these stages. In multi-stage retrieval, different stages may employ various types of retrievers, which enables the system to incorporate various retrieval techniques tailored to different aspects of the query. For example, Wang et al. [159] first utilize a non-parametric retriever to extract n-hop paths of entities in the query’s reasoning chain, then after a pruning stage, it further retrieves the one-hop neighbors of the entities in the pruned subgraph. OpenCSR [45] divides the retrieval process into two stages. In the first stage, it retrieves all 1-hop neighbors of the topic entity. In the second stage, it compares the similarity between these neighbor nodes and other nodes, selecting the top-k nodes with the highest similarity for retrieval. GNN-RAG [108] first employs GNNs to retrieve the top-k nodes most likely to be the answer. Subsequently, it retrieves all shortest paths between query entities and answer entities pairwise.

Remark. In GraphRAG, once retrieval typically exhibits lower complexity and shorter response times, making it suitable for scenarios requiring real-time responsiveness. In contrast, iterative retrieval often involves higher time complexity, especially when employing LLMs as retrievers, potentially leading to longer processing times. However, this approach can yield higher retrieval accuracy by iteratively refining retrieved information and generating responses. Therefore, the choice of retrieval paradigm should balance accuracy and time complexity based on specific use cases and requirements.

6.3 Retrieval Granularity

According to different task scenarios and indexing types, researchers design distinct retrieval granularities (i.e., the form of related knowledge retrieved from graph data), which can be divided into nodes, triplets, paths, and subgraphs. Each retrieval granularity has its own advantages, making it suitable for different practical scenarios. We will introduce the details of these granularities in the following sections.

6.3.1 Nodes. Nodes allow for precise retrieval focused on individual elements within the graph, which is ideal for targeted queries and specific information extraction. In general, for knowledge graphs, nodes refer to entities. For other types of text attribute graphs, nodes may include textual information that describes the node’s attributes. By retrieving nodes within the graph, GraphRAG systems could provide detailed insights into their attributes, relationships, and contextual information. For example, Munikoti et al. [113], Li et al. [87] and Wang et al. [160] construct document graphs and retrieves relevant passage nodes. Liu et al. [90], Sun et al. [139] and Gutiérrez et al. [43] retrieve entities from constructed knowledge graphs.

6.3.2 Triplets. Generally, triplets consist of entities and their relationships in the form of subject-predicate-object tuples, providing a structured representation of relational data within a graph. The structured format of triplets allows for clear and organized data retrieval, making it advantageous in scenarios where understanding relationships and contextual relevance between entities is critical. Yang et al. [171] retrieve triplets containing topic entities as relevant information. Huang et al. [55], Li et al. [81] and Li et al. [86] first convert each triplet of graph data into textual sentences using predefined templates and subsequently adopt a text retriever to extract relevant triplets. However, directly retrieving triplets from graph data may still lack contextual breadth and depth, thus being unable to capture indirect relationships or reasoning chains. To address this challenge, Wang et al. [152] propose to generate the logical chains based on the original question, and retrieve the relevant triplets of each logical chain.