Adaptive Retrieval-Augmented Generation for Conversational Systems:Model Training and Evaluation Setups

We evaluate the performance of introducing RA-Gate according to its binary classification performance and the effectiveness of the resulting response generation. Specifically, we use the KETOD dataset (Chen et al., 2022), which has fully annotated 5,324 dialogues and 52,063 turns of conversations. In particular, it is associated with 33,761 knowledge snippets to be retrieved and augmented. In addition, KETOD was developed with human labels on turns of conversations (around 12.1% of turns) about the need for augmenting with retrieved knowledge snippets for a natural and informative system response. Hence, we use these human labels as natural ground truths when evaluating RAGate. It is worth indicating that many current knowledge-augmented conversational datasets often ground their conversations on the knowledge snippet, such as Wizard of Wikipedia (Dinan et al., 2018) and CMU_DoG (Zhou et al., 2018), which makes them not a natural fit to be investigated in this study.

Due to the limited computational resource availability, we explore the use of Llama-v2-7B and Llama-v2-13B to implement RAGate-prompt and fine-tune Llama-v2-7B for RAGate-PEFT. We implement QLoRA using the PEFT library (Mangrulkar et al., 2022) and set the lower rank to 16. As discussed in Section 3, we have various input features to be combined for performance optimisation. We begin with the use of context only, then concatenate the context with the real response (contx-resp), with the synthetic response and recognised entities (contx-syn-resp-ner) and further extend with the use of retrieved knowledge (contx-syn-resp-ner-know) or the source of knowledge (contx-syn-resp-ner-source). Specifically, we retrieve the relevant knowledge by exploring the use of TF-IDF and a learned BERT ranker. We evaluate their performance with the classic Recall@1 and Recall@3 on the test collection. We use a shallow cutoff because we only use top-relevant knowledge snippets for augmentation. Table 1 shows their retrieval performance. According to the leading performance of BERT-Ranker, we augment knowledge with its retrieved top 3 relevant knowledge snippets (i.e., k = 3). Regarding the development of RAGate-MHA, we explore the combinations of 2 to 8 layers, 2 or 4 heads and the embedding size in [64, 128, 256] for the best classification accuracy. We report the precision, recall, F1, Area Under Curve (AUC) and the False Discovery Rate (FDR) as the main measures to show the classification effectiveness.

Next, we further deploy the best-performing RAGate gate function to update the KETOD dialogue system (Chen et al., 2022), which uses GPT-2 (Radford et al., 2019) as the backbone model. To highlight the effect of various augmentation setups, we use the context with the gold action without extra prediction as input to KETOD. Then, we compare the resulting performance to the KETOD model without knowledge augmentation and augmenting every system response as baselines. To report the response generation effectiveness, we report how close the response is to the ground truth via BLEU, ROUGE-1/2/L and BERTScores and the confidence score calculated by the minimum probabilities of individual tokens that compose the response. As argued by Varshney et al. (2023), this calculated confidence score can highly correlate with a language model’s likelihood of generating hallucinated responses. We trained our models and conducted the evaluations on one machine with one NVIDIA 4090 GPU.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Results and Analysis

5.1 Augmentation Need Classification

First, we evaluate the classification accuracy of our developed RAGate gate methods for addressing the adaptive RAG to system responses. Table 2 presents the classification performance of RAGate baselines while evaluated on the test collection of the KETOD dataset, which includes rich human labels on the use of RAG for response generation. As discussed in Section 3, we explore the development of RAGate with three variants: the use of LLM prompting (RAGate-Prompt), parameter-efficient fine-tuned LLMs (RAGate-PEFT), and a neural classifier with Multi-Head Attention structure (RAGate-MHA).


Adaptive Retrieval-Augmented Generation for Conversational Systems:Table 1: Retrieval Performance Evaluation when using context as the query.

| Retrieval Models | Recall@1 | Recall@3 |
|------------------|----------|----------|
| TF-IDF           | 0.0227   | 0.0871   |
| BERT-Ranker      | 0.2475   | 0.4714   |


Adaptive Retrieval-Augmented Generation for Conversational Systems:RA-Gate Performance

The table presents classification accuracy on adaptive augmentation for system response. It uses context, initial system response, and retrieved knowledge snippets as input. The results show that RA-Gate-PEFT with context-only input significantly outperforms RA-Gate-Prompt. For instance, context-only input has a precision of 0.5203 compared to a maximum of 0.1220 in RA-Gate-Prompt. However, this improvement comes at the expense of lower recall (0.3359 to 0.2321).

"RA-Gate-MHA: Context with / without Knowledge Input" results show similar trends, where the interaction between context and retrieved knowledge snippets can achieve high recall but often face a trade-off between precision and recall.

RA-Gate-PEFT approaches were able to improve the precision scores significantly when using synthetic response and recognized named entities. However, including retrieved knowledge snippets led to a performance drop across evaluated aspects, likely due to complexity in the snippets.

There is a need to balance the precision and recall trade-off, and further evaluation is conducted in Appendix B to explore the potential contribution of retrieved snippets in predicting decisions for retrieval augmentation.

"5.2 Adaptive Augmentation Analysis" discusses how the choice between human workers and RA-Gate in augmenting specific turns affects classification accuracy.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Frequency Analysis of Adaptive Augmentations

Figure 3: Frequency analysis of adaptive augmentations about the position of a conversation. Specifically, we analyse the frequency of augmentation in different positions of conversations and different domains covered in the KETOD dataset. We use the RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis as representatives for comparison. Figure 3 presents the frequency in different positions. Due to the unequal number of conversational turns, we use the ratio to indicate the relative position. According to the reported results in Figure 3, most human augmentation selections happen at the beginning of a conversation. This trend is also effectively captured by both RAGate approaches, especially RAGate-MHA. This can be caused by the reason that a conversation is semantically coherent, and once sufficient additional information is provided at the early stage, the value of knowledge augmentation to the later turns is naturally lower.\nOn the other hand, Figure 4 presents the augmentation frequency over different domains. We observe that system responses about certain domains are selected more often by humans than other domains, such as travel, hotels, trains, flights, service and rental cars, which require access to additional information to assist the suggestion-making, and the domains, like movies, music, media, events that often include entities require enriched description. By looking into the performance of RAGate-PEFT and RAGate-MHA, RAGate-MHA can make aligned selections for humans. However, the RAGate-PEFT does not guarantee the identification of appropriate augmentation use and often presents fewer augmentations, apart from the travel domain. Hence, by considering both position and domain augmentation frequency, we conclude that RAGate-MHA can outperform RAGate-MHA and effectively capture the trend of augmentation needs.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Frequency Analysis of Adaptive Augmentations about Dialogue Domains

Figure 4: Frequency analysis of adaptive augmentations about dialogue domains. Table 3: Performance of applying RAGate and compared to the KETOD baseline on the KETOD dataset. Confidence is calculated by the average value over the lowest logit of each generation. \n\nVariants | # Aug | BLEU | ROUGE-L | BERTScore | Confidence \n- No-Aug | 0 | 9.38 | 0.3176 | 0.1805 | 9.3425 | + \nRAGate-PEFT | 250 | 10.45 | 0.3825 | 0.1844 | 9.3473 | -0.06%\nRAGate-MHA | 787 | 12.41 | 0.3882 | 0.1892 | 9.3084 | -0.30%\nRandom-Aug | 830 | 9.03 | 0.3764 | 0.1810 | 9.2864 | -0.47%\nRandom-Aug | 787 | 10.01 | 0.3796 | 0.1816 | 9.1577 | -1.65%\nHuman-label | 631 | 11.66 | 0.3856 | 0.1878 | 9.2650 | -0.45%\nAugAll-Aug | 4964 | 16.05 | 0.3944 | 0.1839 | 9.0555 | -2.29% \n\n5.3 RAGate for Response Generation To evaluate the effect of adaptive RAG for a conversational system, we use RAGate-PEFT (contx-(syn-resp)-ner) with the highest precision and RAGate-MHA (MHA(contx)) with the best overall performance in the above analysis, to support the adaptive retrieval augmented conversational response generation. Table 3 presents the results of applying RAGate to the KETOD model for adaptive knowledge augmentation when evaluated on the KETOD dataset. We include four types of adaptive augmentation, namely the use of RAGate and comparison to the random selection with equal numbers of se-.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Research Paper Page Content

At first, without adaptive knowledge augmentation, we compare the choice of response generation without augmentation and with "always" augmentation (i.e., No-Aug versus Aug-All). In Table 3, we observe that by augmenting a total of 4,964 system responses in the test collection, the conversational model can generate more informative and effective responses according to the reported scores of BLEU, ROUGE and BERTscore. This aligns with the reported effectiveness of RAG in many existing studies. However, we also identify a significant drop in the model’s generation confidence level. As denoted by Varshney et al. (2023), a lower confidence level can correlate with a higher chance of generating hallucinated responses, which could be caused by the unnecessary use of external knowledge. Hence, to investigate the effectiveness of adaptive knowledge augmentation, we examine the impact of using RAGate. According to the reported experimental results in Table 3, the adaptive augmented response generation with fewer knowledge snippets can indeed result in a higher confidence level than Aug-All.

Moreover, comparing the performance between RAGate and random selections shows that, considering equal numbers (230 or 787 according to the classification with RAGate) of system responses for augmentation, RAGate can further result in a higher quality of generated response. RAGate-MHA even enables results that are comparable to Aug-All’s response quality, with only 787 turn augmentations instead of all 4964 turns. Specifically, the use of RAGate-PEFT, which identifies 230 turns of system responses for knowledge augmentation, can even outperform the random baseline that augments 787 system response turns with improved response quality. Apart from the improved response quality, RAGate also enables the conversational model to maintain a high confidence level and ensure faithful responses. Indeed, using RAGate-MHA, which augments 787 system responses, only lowers the average confidence score by 0.36%, instead of the 1.65% when randomly selecting an equal number of turns to augment.

In addition, considering the use of different quality and amount of knowledge snippets for augmentation, we also include the use of the most relevant knowledge snippet according to BERT-ranker in Table 3. We observe that the use of different amounts of knowledge snippets in different relevance levels has a marginal effect on this learned dialogue system. However, we observe a significant difference in the confidence level. We observe that using only the most relevant knowledge snippet enables the Aug-All to suffer less from a lower confidence level. In particular, the application of RAGate can even increase the confidence level of the conversation system in response generation. This indicates that the confidence score can also correlate with the quality of the augmented knowledge snippets. This observation is further validated using knowledge snippets with fifth-ranking positions by BERT-ranker and the use of TF-IDF ranker. We include the full experimental results in Table 4 and attached in the Appendix. These observations indicate the value of adaptive system response augmentation via RAGate in generating high-quality outputs, ensuring faithful responses, and potentially saving retrieval costs. We also show the value of using confidence scores to reflect the contribution of RAG.

6 Conclusions

Our study investigates a core research question about whether retrieval-augmented generation is always useful to a conversational system. To answer this research question, we propose adaptive retrieval-augmented generation for conversational systems and introduce corresponding gate functions, RAGate, for explicit control. A comprehensive set of experiments and results show the RAGate approaches can effectively identify augmentation needs. In addition, RAGate can capture human preference by augmenting the beginning turns of conversations, and RAGate can further identify knowledge augmentation for assisting suggestion-making and enriching description. When applying RAGate to conversational systems, we observe that it can ensure comparable quality of generated responses and enable the system to increase generation confidence for faithful outputs, especially with the appropriate use of relevant knowledge snippets.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Limitations

There are three limitations of this study. At first, due to the main focus of examining the adaptive retrieval-augmented generation for a conversation system. We only consider a few examples of retrieval techniques (TF-IDF and BERT-ranker), which can be further extended to recent retrieval techniques, such as dense passage retrieval for additional insights. The second limitation is the missing use of larger language models, such as GPT-4, due to the shortage of computational resources. Including larger language models for conversational systems could introduce additional experimental insights. The third limitation is the shortage of appropriate conversational data for extensive evaluations. This is mainly caused by the recent development of the retrieval augmented generation technique and its application to conversational systems. Future research is encouraged to address this limitation.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Ethics Statement

All experiments in this study were conducted using publicly available datasets and open-released language models, which do not contain any private information that could raise ethical concerns.


Adaptive Retrieval-Augmented Generation for Conversational Systems:References

Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2022. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.

Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul A Crook, and William Yang Wang. 2022. Ketod: Knowledge-enriched task-oriented dialogue. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2581–2593.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2022. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.

Alex Graves and Alex Graves. 2012. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37–45.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Taesuk Hong, Junhee Cho, Haeun Yu, Youngjoong Ko, and Jungyun Seo. 2023. Knowledge-grounded dialogue modelling with dialogue-state tracking, domain tracking, and entity extraction. Computer Speech & Language, 78:101460.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Xinxian Huang, Huang He, Siqi Bao, Fan Wang, Hua Wu, and Haifeng Wang. 2021. Plato-kag: Unsupervised knowledge-grounded conversation via joint modeling. In Proc. of NLP4ConvAI.

Di Jin, Shuyang Gao, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Towards zero and few-shot knowledge-seeking turn detection in task-orientated dialogue systems. In 3rd Workshop on Natural Language Processing for Conversational AI, NLP4ConvAI 2021, pages 281–288.

Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. 2023. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proc. of EMNLP.

Atoosa Kasirzadeh and Hosan Gabriel. 2023. In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):27.

Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani-Tur. 2020. Beyond domain apis: Task-oriented conversational modeling with unstructured knowledge access. In Proc. of SIGDIAL.

Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460–8478.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Research Paper References

Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2022. Knowledge-grounded dialogue generation with a unified knowledge representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 206–218.

Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. In Proc. of IJCAI.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tune methods. https://github. com/huggingface/peft.

Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth M Daly, David Piorkowski, and John T Richards. 2024. Language models in dialogue: Conversational maxims for human-ai interactions. arXiv preprint arXiv:2403.15115.

Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learning based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):3505–3155.

Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What lms know about unseen entities. In Proc. of NAACL.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proc. of EMNLP.

Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. Preprint, arXiv:2404.13781.

Procheta Sen, Xi Wang, Ruiqing Xu, and Emine Yilmaz. 2023. Task2kb: A public task-oriented knowledge base. In Proceedings of the AAAI Conference on Artificial Intelligence.

Weizhou Shen, Yingqi Gao, Canbin Huang, Fangqi Wen, Xiaojun Quan, and Wei Bi. 2023. Retrieval-generation alignment for end-to-end task-oriented dialogue system. arXiv preprint arXiv:2310.08877.

Tianyuan Shi, Liangzhi Li, Zijian Lin, Tao Yang, Xiaojun Quan, and Qifan Wang. 2023. Dual-feedback knowledge retrieval for task-oriented dialogue systems. arXiv preprint arXiv:2310.14528.

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Proc. of EMNLP.

Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of lms by validating low-confidence generation. arXiv preprint arXiv:2307.03897.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Hongru Wang, Wenyu Huang, Yang Deng, Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z Pan, and Kam-Fai Wong. 2024. Unims-rag: A unified multi-source retrieval-augmented generation for personalized dialogue systems. arXiv preprint arXiv:2401.13256.

Shiquan Yang, Rui Zhang, and Sarah Erfani. 2020. Graphdialog: Integrating graph knowledge into end-to-end task-oriented dialogue systems. In Proc. of EMNLP.

Linhao Ye, Zhikai Lei, Jianqiao Yin, Qin Chen, Jie Zhou, and Liang He. 2024. Boosting conversational question answering with fine-grained retrieval-augmentation and self-check. arXiv preprint arXiv:2403.18243.

Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. Evaluation of retrieval-augmented generation: A survey. Preprint, arXiv:2405.07437.

Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge-grounded dialogue generation with pre-trained language models. In Proc. of EMNLP.

Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708–713.

### New Prompts

In this section, we list the used prompts for the RAGate-Prompt gate mechanism.

### Zero-Shot Prompt

Below is an instruction that describes a task. Please respond with ‘True’ or ‘False’ only that appropriately completes the request.

### Instruction:

Analyse the conversational context so far. Generate an appropriate response. Consider the involved entities. Estimate if augmenting the response with external knowledge is helpful with an output of ‘True’ or ‘False’ only.

### Response:

In-Context Learning Prompt:


Adaptive Retrieval-Augmented Generation for Conversational Systems:Augmentation Variants Table

Table 4: Performance of applying RAGate and compared to KETOD on the SGD dataset. Confidence is calculated by the average value over the lowest logit of each generation.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Instructions

Below is an instruction that describes a task. Please respond with 'True' or 'False' only that appropriately completes the request.

### Instruction: Analyse the conversational context so far. Generate an appropriate response. Consider the involved entities. Estimate if augmenting the response with external knowledge is helpful with an output of 'True' or 'False' only.

### Example 1: USER: I'm planning a trip, can you help me look for a flight? SYSTEM: Which day are you planning to return and from which city? USER: I want to go from NYC the day after tomorrow and return on the 13th of this month. SYSTEM: Where would you like to go? USER: I want to go to Vancouver, BC. Can you look for a Premium Economy class ticket. SYSTEM: I found 1 flight for you. It is a Delta Airlines flight that takes off at 6 am and returns at 2:50 am. The price is $505. USER: What is the departure airport, and how many stops does the flight have?

### Response: True

### Example 2: USER: Get me bus tickets to a Cher event on March 6th. SYSTEM: How many to buy? USER: only one, please.

### Response: False.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Impact of Retrieval Quality on Adaptive RAG

B. Impact of Retrieval Quality on Adaptive RAG

To have a successful conversation model with a retrieval-augmented system, two main criteria must be met. One is identifying insufficient context, and the other is the quality of retrieved information (Salemi and Zamani, 2024; Yu et al., 2024). A conversational model performs better when both criteria are satisfied. In our proposed approach, as shown in Table 2, we have already assessed whether our adaptive retrieval method can detect insufficient context. We further explored to determine whether our model can inherently estimate the quality of the retrieved snippets to address such insufficiency and, based on that, decide on the retrieval. Although we do not explicitly provide retrieved snippets to our model, retrieval comes with a corpus that includes potentially relevant knowledge snippets. Consequently, given a query and a retrieval collection, it can be estimated whether


Adaptive Retrieval-Augmented Generation for Conversational Systems:Additional experimental results about RAGate for Response Generation

useful information for the query exists in the corpus to address the insufficient context. To investigate by following this direction, we randomly selected 50 samples from instances where our proposed approach (RAGate-MHA, the best-performing gate model) predicted using retrieval augmentation. We asked domain experts (co-authors) to score whether they thought the retrieved snippets in those scenarios could be useful to response generation. Users rated the snippets on a scale of 0 – 4, with scores of 3 or 4 indicating ‘useful’ or ‘highly useful’. We found that in 54% of cases where the prediction was for augmentation, users also found the snippets useful. This indicates that our proposed approach can implicitly capture the potential for obtaining high-quality retrieval snippets.

In Table 4, we include the complete experimental results of applying RAGate for adaptive retrieval-augmented system response generation. Specifically, explore the use of retrieved knowledge snippets to different extents of relevance. We include top-3 knowledge snippets retrieved by BERTranker and TF-IDF. In addition, we also explore the use of knowledge snippets in different ranking positions (rank 1 and 5) according to the BERTranker retriever. The experimental result shows that precisely using a suitable amount of relevant knowledge can generate a response with higher confidence (i.e., less is more). In addition, this observation also indicates the potential use of confidence levels to evaluate the quality of the augmented knowledge.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Abstract

Enterprise chatbots, powered by generative AI, are rapidly emerging as the most explored initial applications of this technology in the industry, aimed at enhancing employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), Langchain/Llamadex types of LLM orchestration frameworks serve as key technological components in building generative-AI based chatbots. However, building successful enterprise chatbots is not easy. They require meticulous engineering of RAG pipelines. This includes fine-tuning semantic embeddings and LLMs, extracting relevant documents from vector databases, rephrasing queries, reranking results, preserving effective prompting, honoring document access controls, providing concise responses, including pertinent references, safeguarding personal information, and building systems to orchestrate all these activities. In this paper, we present a framework for building effective RAG-based chatbots based on our firsthand experience of building three chatbots at NVIDIA: chatbots for HR and IT benefits, company financial earnings, and general enterprise content. Our contributions in this paper are three-fold. First, we introduce our FACTS framework for building enterprise-grade RAG-based chatbots that address the challenges mentioned. FACTS mnemonic refers to the five dimensions that RAG-based chatbots must get right - namely content freshness (F), architectures(A), cost economics of LLMs (C), testing (T), and security (S). Second, we present fifteen control points of RAG pipelines and techniques for optimizing chatbots’ performance at each stage. Finally, we present empirical results from our enterprise data on the accuracy-efficiency tradeoffs between large LLMs vs small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots.


FACTS About Building Retrieval Augmented Generation-based Chatbots:1 Introduction

Chatbots are increasingly becoming an extension of search tools in companies for finding relevant information. Whether it’s HR benefits, IT help, sales queries, or engineering issues, enterprise chatbots are now go-to productivity tools. Before the debut of OpenAI’s Chat-GPT [2] in November 2022, companies relied on internally developed chatbots based on dialog flows. Such bots required extensive training for intent understanding and meticulous orchestration for response generation and yet could only provide extractive answers at best. These early bots, built on dialog management systems paired with information retrieval and question answering (IRQA) solutions were fragile and limited in capability. While previous generation language models and GPT models existed, they lacked the accuracy, robustness, and reliability needed for broad enterprise use [5]. Chat-GPT’s release, the convergence of vector databases, and the wide-spread use of retrieval augmented generation (RAGs) [8] marked the beginning of a new era in Chatbot domain. Now, LLMs can understand user intents with simple prompts in natural language, eliminating the need for complex intent variant training, synthesizing enterprise content coherently, thereby empowering chatbots with conversational capability way beyond simple intent recognition. While LLMs bring their generative capabilities to construct coherent, factual, and logical responses to user queries, vector databased-powered information retrieval (IR) systems augment LLMs ability to retrieve fresh content. Tools like LangChain [1] and Llamadex [6] facilitate chatbot construction, and orchestration of complex workflows including memory, agents, prompt templates, and overall flow. Together, vector-search based IR systems, LLMs, and LangChain-like frameworks form core components of a RAG pipeline and power driving generative AI chatbots in post Chat-GPT era. At NVIDIA, our main motivation was to improve our employee productivity by building enterprise chatbots. Our initial enthusiasm quickly met with the reality of addressing numerous challenges. We learned that crafting a successful enterprise chatbot, even in post Chat-GPT era, while promising, is not easy. The process demands meticulous engineering of RAG pipelines, fine-tuning LLMs, and engineering prompts, ensuring relevancy and accuracy of enterprise knowledge, honoring document access control permissions, providing concise responses, including pertinent references, and safeguarding personal information. All of these require careful design, skillful execution, and thorough evaluation demanding many iterations. Additionally, maintaining user engagement while optimizing for speed and cost-efficiency is essential. Through our journey, we learned that getting an enterprise conversational virtual assistant right is akin to achieving a perfect symphony where every note carries significance! In this paper, we share our experiences and strategies in building effective, secure, and cost-efficient chatbots. We answer the following questions from a practitioner perspective:


FACTS About Building Retrieval Augmented Generation-based Chatbots:1. Table

Table 1: A summary of the three chatbots and the current state of development.

| Chatbot | Domain | Data Sources | Data Types | Access Control | Sample Queries | State |
|---|---|---|---|---|---|---|
| NVInfo Bot | Enterprise Internal | SharePoint, GoogleDrive, Slack | Docs, HTML | Yes | Can I park overnight in HQ parking lots? | Early Access |
| VHelp IT Help Bot | Knowledge Articles for ITHelp | Confluence, ServiceNow, Jira, etc | PDFs, Slides | Yes | How to enroll in Employee Stock Purchase plan? | Testing |
| Scout Bot | Financial Earnings | Company news, blogs, SEC filings | HTML, PDFs Docs | No | What are NVIDIA revenues for the past 3 years? | Production |

* What are the key challenges to consider when building and deploying enterprise-grade generative AI-based chatbots? We present our findings from trying to deliver fresh content (F) with flexible architectures (A) that are cost-efficient (C), tested well (T), and secure (S): (FACTS).
* How to achieve user acceptable levels of quality with RAG systems in building chatbots? We present the fifteen control points of RAG pipelines and techniques for optimizing each control point and the overall RAG pipeline.


FACTS About Building Retrieval Augmented Generation-based Chatbots:2. Case Study

Our company’s content landscape includes both authoritative knowledge and unauthoritative content. Authoritative content encompasses IT help articles, HR resources in platforms like ServiceNow, and project documentation on Confluence, SharePoint, Google Drive, and engineering tools like NVBugs and GutiHub. Employee-generated content complements these sources on platforms such as Slack and MS Teams. In this paper, we present three bots that we have built internally using RAGs and LLMs. These bots are briefly introduced below. All three bots are built on our in-house built generative-AI chatbot platform called NVBot platform. Some of the specific content that our bots are capable of answering are shown in Table 1.
* NVInfo Bot answers questions about enterprise content (approx. 500M documents of size > 7 TB), complements intranet search. It manages diverse data formats and enforces document access controls. The tech stack includes LangChain, a vendor vector database for retrieval and to handle document access controls, LLM model (multiple LLM models can be selected), and a custom web-UI.
* NVHelp Bot focuses on IT help and HR benefits (approx. 2K multi-modal documents containing text, tables, images, pdfs, and html pages), using a similar tech stack to NVInfo bot with a smaller data volume.
* Scout Bot handles questions about financial earnings from public sources, managing structured and unstructured data (approx. 4K multi-modal documents containing text, tables, pdfs, and html pages). The tech stack includes an Open source Vector DB, LangChain, RAGs evaluation, selectable LLM models, and a custom web-UI.
In the remainder of the paper, we present our FACTS framework that summarizes the challenges experienced and the learnings gained in building the aforementioned three chatbots. We first start with the challenge of dealing with delivering fresh enterprise content in each of the chatbots.


FACTS About Building Retrieval Augmented Generation-based Chatbots:3. Ensuring Freshness of Enterprise Data in LLM-Powered Chatbots (F)

Ensuring the freshness of enterprise data in LLM-powered chatbots presents several challenges. Foundation models, although powerful, often fall short as they lack domain-specific and enterprise-specific knowledge. Once trained, these models are essentially frozen in time and may hallucinate, providing undesired or inaccurate information when used on enterprise content that they are not trained on.
Retrieved Augmented Generation (RAG) is a process where relevant information is retrieved from vector databases through semantic matching and then fed to LLMs for response generation. In a RAG pipeline, vector databases and LLMs collaboratively ensure the delivery of up-to-date enterprise knowledge. However, RAG pipelines have many control points, each of which when not tuned well can lead to lower accuracy, hallucinations, and irrelevant responses by Chatbots. Additionally, document access control performs a competitor use assessment regularly, requiring careful management to ensure data security and relevance. Furthermore, multi-modal content needs these carefully-pruned retrievers to handle structured, unstructured, and semi-structured data, including presentations, diagrams, videos, and meeting recordings. Addressing these challenges is critical for maintaining the accuracy and reliability of enterprise chatbots. In section 3.1, we identify fifteen control points of RAG from our case studies visualized in Figure 1. Each control point is labeled with a number. In the remainder of this section, we present our insights and learnings for addressing RAG control points.


FACTS About Building Retrieval Augmented Generation-based Chatbots:3.1 Learnings

In figure 4, we present a summary description of the fifteen control points of RAG pipelines, challenges associated with each control point, and our suggested approaches for optimizing each control point. Each control point is labeled as RAG-C[num] and RAG-Op[num] for RAG and RAGOps flows, respectively. Below, we present a few key learnings and insights to manage the fresh enterprise content.
Metadata Enrichment, Chunking, Query Rephrasal, Query Re-ranking: We noticed that metadata enrichment, chunking, query rephrasal and query re-ranking stages of RAG pipeline have the most impact on the quality of Chatbot responses. LLM response generation quality is highly dependent on retrieval relevancy. Retrieval relevancy is, in turn, highly dependent on document metadata enrichment, chunking, and query rephrasal. We implemented grid search-based auto-ML capabilities to find the right configurations of chunk token-sizes, experimented with various prompt variations, and explored different chunk re-ranking strategies to find optimal control point configurations.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Control Points in RAG Pipeline

Figure 1 illustrates the control points in a typical Retrieval-Augmented Generation (RAG) pipeline when building chatbots. The pipeline includes continuous data ingestion from multiple sources, followed by several processes including document parsing, sentence splitting, and embedding model integration. The GPU accelerated vector store is used for chunking, making it suitable for language model deployments. Additionally, various components such as chat workflows and chatbot operation management stages are highlighted.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Agent Architecture for Handling Complex Queries

Figure 2 presents an agent architecture designed to handle complex queries. Complex agents and multi-agent architectures are necessary for effectively managing these queries. The figure demonstrates how vector-based searches, combined with lexical searches, enhance retrieval relevancy and accuracy by supporting both query decomposition and orchestration.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Content

We have made significant improvements in retrieval relevancy and answer quality, yet further optimization of the RAG pipeline is needed. In our hybrid search approach, leveraging lexical search capabilities enhances the strengths of both lexical and vector-based searches. Comparing revenue data from NVIDIA and performing analytical commentary reveals key insights. However, current Information Retrieval (IR) systems and Large Language Models (LLMs) may not suffice for complex queries. Fine-tuning of LLMs requires delicate balancing with domain-specific customizations to ensure efficiency in data labeling, training, and evaluation. Handling multi-modal data is crucial for a robust RAG pipeline. Implementing section-level splitting and incorporating inline text helps improve retrieval relevancy.


FACTS About Building Retrieval Augmented Generation-based Chatbots:3. NYHelp answer quality and latency metrics comparison among different models

RAGOps: Effective health monitoring of RAG pipelines is essential once they are deployed. When answer quality is poor, a thorough error analysis is required to determine whether the issue lies in retrieval relevancy or LLM response generation. To debug retrieval relevancy, developers need detailed information on which chunks were stored in vector databases with their associated metadata, how queries were rephrased, which chunks were retrieved, and how those chunks were ranked. Similarly, if an LLM response is incorrect, it is crucial to review the final prompt used for answer generation. It ensures that citations, developers, and clients trace back to the correct document links and their corresponding chunks. RAGOps/LLMOps and evaluation frameworks, such as Ragas, are critical for providing the necessary automation to enable rapid iteration during discovery improvement in RAG pipelines. More details on each control point per model are described in Figure 4. In summary, while promising, implementing RAG systems for chatbots demands meticulous planning and continuous evaluation to ensure secure and accurate data retrieval.


FACTS About Building Retrieval Augmented Generation-based Chatbots:4. BUILDING FLEXIBLE ARCHITECTURES FOR GENERATIVE AI CHATBOTS (A)

Keeping up with rapid progress in AI is like navigating a fast-moving river. Every aspect, from vector databases and embedding models to LLMs, agent architectures, low-code/no-code platforms, RAG evaluation frameworks, and prompting techniques, is evolving rapidly. Concurrently, departments within companies are exploring generative AI by building their own chatbots and AI copilots. In this dynamic environment, building common, flexible, and adaptable platforms are crucial. At NVIDIA, our chatbot ecosystem has grown significantly, reflecting a trend likely seen in many companies. From building three initial chatbots, we realized the importance of a common platform to avoid duplicated efforts in security, guardrails, authentication, prompts, user interfaces, feedback mechanisms, usage reporting, monitoring, and evaluations. To address this, we developed the NVBot platform (Figure 7), a modular platform with a pluggable architecture. It allows developers to select LLMs, vector databases, embedding models, agents, and RAG evaluation frameworks that best suit their use case. It also provides common components for essential features like security, guardrails, authentication, authorization, user experience, and monitoring. Additionally, the platform supports citizen development, allowing multiple teams to contribute their tested prompts, workflows, guardrails, and fine-tuned models for collective use. As our ecosystem of bots expanded, we faced a critical question: should organizations build many domain-specific bots, a single enterprise bot, or go with a hybrid approach? Domain-specific chatbots excel in tailored environments, while enterprise-wide chatbots act as generalists, providing a centralized knowledge base for all employees. Through our experience, we realized that there is no need to choose one over the other. Novel architectural patterns are emerging where enterprise-wide chatbots act as 'switchboards', directing inquiries to specialized bots tuned with domain-specific data. This multibot architecture allows for the concurrent development of specialized chatbots while providing users with a unified interface. Our NVBot platform supports the coexistence and orchestration of multiple chatbots within an enterprise. The debate over a single bot or multiple specialized bots is ongoing. We envision a landscape where domain-specific bots coexist with a centralized information bot, supported by 'copilots'—generative AI capabilities integrated into workplace environments like programming IDEs and collaboration tools. At NVIDIA, we’re actively exploring all three chatbot variations—domain-specific, enterprise-wide, and copilot as generative AI reshapes workplace efficiency and information accessibility.


FACTS About Building Retrieval Augmented Generation-based Chatbots:5. COST ECONOMICS OF CHATBOT DEPLOYMENTS

Understanding the cost economics of generative AI-based chatbots involves several critical factors. The high costs of major and commercial LLMs can be unsustainable, with expenses adding up significantly across multiple use cases. Additionally, unseen expenses often accumulate as teams test various LLMs to meet specific needs. Moreover, when using commercial LLM vendor APIs, securing sensitive enterprise data requires guardrails to detect and prevent sensitive data leakage, as well as gateways for audit and legally permitted learning. There are also costs for wasted tokens and low cost per consumer, as large LLMs with long context lengths typically have slower response times, impacting overall efficiency. Bigger Vs. Smaller Models: Larger, commercial LLMs, smaller open source LLMs are increasingly becoming viable for many use cases, therefore offering cost-effective alternatives to companies. As open source models are catching up with larger, commercial models, they are increasingly offering close-comparable accuracy, as demonstrated in our NYHelp bot empirical evaluation in Figure 3, and generally have better latency performance compared to larger models. Additionally, GPU optimization of inference models can further speed up processing times. Open-source models optimized with NVIDIA's Tensor RT-LLM inference models, for instance, have shown faster performance than non-optimized models. These strategies help balance the need for cost-efficiency with maintaining high performance and security standards. LLM Gateway: If you must use a vendor LLM API, it is better to implement an internal company LLM Gateway for audit, subscription, and data management across the company. Implementing an internal company LLM Gateway can streamline LLM usage, subscriptions, and data tracking for security audits. This central hub


FACTS About Building Retrieval Augmented Generation-based Chatbots:RAG Control Points, Challenges, and Remediations

Figure 4 presents a table outlining various stages of the RAG (Retrieval-Augmented Generation) process. The table includes stages such as RAG-C1 (Data Ingestion), RAG-C2 (Data Transformation), RAG-C3 (Metadata Enrichment), RAG-C4 (Embedding Model), and so on. Each of these stages is associated with descriptions of their functions, challenges faced, and possible remediations. For instance, challenges like 'Scale' and 'Metadata enrichment' are addressed with solutions like 'ACL Support' and 'Opportunistic embedding'. The table covers aspects such as data security, scaling volumes, hybrid models, hallucinations, and the necessity for continuous validation among others.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Scout Bot: Multi-part Query

Figure 5 discusses the use of Scout Bot for managing multiple queries and simplifying resource allocation at NVIDIA IT. The bot helps in logging network payloads required for auditing and access control purposes, enabling efficient organization of large language models (LLM) API invocations. The text emphasizes the importance of a hybrid and balanced LLM strategy and states that using smaller LLMs for cost control allows exploration with larger LLMs.


FACTS About Building Retrieval Augmented Generation-based Chatbots:NVHelp Bot: Answering Questions on HR Benefits

Figure 6 showcases the NVHelp Bot used for responding to employee queries regarding HR benefits like Health Savings Accounts (HSA). The bot illustrates the company's contributions towards an employee's HSA and highlights the significance of LLM Gateway monitoring to safeguard sensitive data. It also discusses the delicate balance required between cost, accuracy, and latency to optimize the ROI of LLM infrastructure.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Testing RAG-Based Chatbots

Testing generative AI solutions can be a lengthy process due to the need for human response validation. LLMs are increasingly being employed using 'LLM-as-a-judge' approach. However, it is advisable to use caution when using LLMs as human proxies, as using LLMs as judges can lead to self-fulfilling prophecy type of scenarios reinforcing their inherent biases in evaluations as well.

- Security Testing: Automating security testing is critical for maintaining development velocity without compromising safety. A strong security framework and regression test datasets ensure that the chatbot remains resilient to potential threats. We are collaborating with our internal RED teams in security to prepare a set of datasets that can be tested with each major iteration of the chatbot.

- Prompt Change Testing: Generative AI models can be highly sensitive to prompt changes. To maintain accuracy, full regression testing is needed with each prompt alteration.

- Feedback Loops: Incorporating feedback gathered and the RLHF cycle is pivotal for continuous improvement. It allows LLM models to refine both our solutions and Language Models over time, ensuring that the chatbot becomes increasingly proficient. However, if the chosen foundational models don’t offer customization, then it becomes difficult to align the models to human feedback. If the feedback is significant and comes in an unstructured, then model customization may be an option. As of now, we have begun gathering user feedback but haven’t built our continuous learning pipelines using RLHF yet. Having tools to make this automated is critical to post-production life cycle management of these chatbots.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Learning

6.1 Learning Tips

Plan for Long Test Cycles: Effective testing of RAG-based chatbots requires anticipation of lengthy test cycles. Begin by focusing on automating tests and enhancing accuracy assessments to streamline this essential phase.

Build Representative Ground Truth Datasets: It is crucial to construct comprehensive ground truth datasets that reflect full spectrum of targeted solution strengths. This ensures that the chatbot is tested against scenarios that it will encounter in actual use.

Automate Evaluations: While leveraging LLMs as evaluators can provide scalable testing options, remember that the quality of human evaluations is unmatched. Automated tools should be used where feasible to supplement but not replace human oversight.

Incorporate Human Feedback and Continuous Learning: Establish mechanisms that allow for human feedback and systematic error analysis. Prioritize iterative improvements based on this feedback to continually refine chatbot performance and adaptability.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Securing RAG-Based Chatbots

7 Securing RAG-Based Chatbots

Building trust is paramount when deploying generative AI chatbots. To mitigate risks, guardrails for hallucinations, toxicity, fairness, transparency, and security are critical. Strong foundational models are increasingly getting better at these guardrails. However, there are still many possibilities of jail breaks, adversarial attacks, and other security issues. Apart from these security risks, generative AI-based chatbots are susceptible to derivative risks (explained below). Since our bots are all internal enterprise chatbots, our focus has been more on the enterprise content security and guardrailing for


FACTS About Building Retrieval Augmented Generation-based Chatbots:7.1 Learnings

Enterprise Content Access Control: Enterprise documents are protected by access controls, requiring RAG-based chatbots to comply with Access Control Lists (ACLs) during response generation. To ensure this compliance, we specifically selected an IR product known for its capability to honor these document ACLs effectively.
Derivative Risks with Generative AI: Chatbots might generate responses that lack context from their original data sources, potentially leading to misinterpretations. Additionally, enhanced search methods could inadvertently elevate the risk of exposing sensitive data if enterprise content is inappropriately secured. As part of our NVInfo bot journey, we implemented sensitive data guardrails in addition to leveraging existing data filtering and classification capabilities provided by the vector search solution we used to automatically filter out sensitive data during the retrieval.
Data Governance and Content Security: Efficient knowledge access can increase sensitive data leakage risks. Thus, it’s essential to prioritize data governance before deployment to safeguard against unauthorized access and data breaches. At NVIDIA, we embarked on an enterprise content security initiative for document sensitivity classification and exclusion of sensitive content from chatbots.
Enterprise API Governance: Implementing comprehensive generative AI responses with specific enterprise policies and rules is essential. These guardrails help mitigate risks by ensuring that Chatbot-generated content adheres to established norms and ethical guidelines, preventing potential legal and reputational damages. In NVInfo bots, we implemented many guardrails in LLM prompts initially. However, later realized that not all LLMs follow these prompts consistently. Therefore, we implemented these guardrails during pre and post processing of queries and responses respectively using Nemo Guardrails [13].


FACTS About Building Retrieval Augmented Generation-based Chatbots:8 RELATED WORK

Our work can be compared with RAG papers on various topics dealing with RAG, along all the FACTS dimensions we presented (freshness, architecture, costs, testing and security). Due to lack of space, we contrast our work with selective works. Barnett et al. [3] presented several failure points when engineering RAG systems. In their work, they highlight the challenges of getting retrieval augmented generation right by presenting their findings from having built three chatbots. Wenqi Glantz [6] elaborated 12 RAG pain points and presented solutions. We experienced similar challenges first-hand when building our chatbots. However, none of these works discuss the challenges with complex queries, testing, dealing with document security, and the need for flexible architectures. In our work, we not only build on failure/pain points of RAGs as mentioned above, but also present our 15 control points in RAG pipelines and offer specific solutions for each stage. Also, we extend our insights and present practical techniques for handling complex queries, testing, and security. We present a reference architecture for one of the implementations of agentic architectures for complex query handling, strategies for testing and evaluating subjectively query responses, and raised awareness for dealing with document ACLs and security. Furthermore, we present a reference architecture for a flexible generative-AI based Chatbot platform.
ChipMemo [10] presents evidence for using a domain adapted language model for improving RAG’s performance on domain specific questions. They finetune the 5.5M-unsupervised model with 3,000 domain specific auto-generated samples. We tried finetuning 5-slarge embeddings model in Scout Bot. Our results did not demonstrate significant improvements. We are presently collecting high quality human-annotated data to repeat the experiments. This could be an important direction to explore in the future for our work. Another interesting technique was presented by Setty et al. [15], in improving RAG performance using Hypothetical Document Embeddings (HYDE) technique. HyDe uses an LLM to generate a theoretical document when responding to a query and then does the similarity search with both the original question and hypothetical answer. This is a promising approach that might make the architecture complex.
Active Retrieval augmented generation (FLARE) [7] iteratively synthesizes a hypothetical next sentence. If the generated sentence contains low-probability tokens, FLARE would use the sentence as the new query for retrieval and regenerate the sentence. Malon et al. [12] reviews works for advanced augmented generation methods in language model. Self-refine [11] builds an agentic framework for initial usage of RAG through iterative feedback and refinement. ReAct [16] Agent is worked on building the agent knowledges in a recursive manner. On the evaluation front, RAGs [14] and ARES [14] utilize LLMs as judges and build automatic RAG benchmark to evaluate the RAG system. Zhu et al [17] overview the intensive usages of LLM in a RAG pipeline including retriever, reader, reformer, rewriter, and reranker. We believe that our work provides a unique perspective on building secure enterprise-grade chatbots via our FACTS framework.


FACTS About Building Retrieval Augmented Generation-based Chatbots:9 CONCLUSIONS

In this paper, we presented our approach to developing effective RAG-based chatbots, highlighting our experiences of building these chatbots at NVIDIA. We outlined our FACTS framework, emphasizing the importance of content freshness (F), architecture (A), LLM cost (C) management, planning for testing (T), and security (S) in creating robust, secure, and enterprise-grade chatbots. We also identified and elaborated on fifteen critical control points within RAG pipelines, providing strategies to enhance chatbot performance at each stage. Furthermore, our empirical analysis reveals the trade-offs between accuracy and latency when comparing large and small LLMs. This paper offers a holistic perspective on the essential factors and practical solutions for building secure and efficient enterprise-grade chatbots, making a unique contribution to the field. More work is needed in several areas to build effective RAG-based chatbots. This includes developing agentic architectures for handling complex, multi-part, and analytical queries; efficiently summarizing large volumes of frequently updated enterprise data; incorporating auto-ML capabilities to optimize various RAG control points automatically; and creating more robust evaluation frameworks for assessing subjective responses and conversations.


FACTS About Building Retrieval Augmented Generation-based Chatbots:References

[1] Touvron, H. GitHub: https://github.com/langchain-ai.
[2] Achlioptas, D., Adams, S., Agarwal, S., Alabdulkarim, A., Akkaya, I., Alemi, A. F., Bailey, J., Aitken, A., Armandpour, M., Ait-El-Hara, S., Anand, S. T., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).
[3] Barnett, S., Kothari, N. S., Turumella, S., Binnanyel, Z., and AbdelRazek, M. Seven failure points when engineering a retrieval augmented generation system. arXiv preprint arXiv:2305.06586 (2024).
[4] Es, S., Jain, J., Ferguson-Auken, L., and Schoenrkst, R. S. Ragar: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023).
[5] Gartzen, B. Developing enterprise chatbots. Springer, 2019.
[6] Glantz, W. R. Zag: pain points and proposed solutions.
[7] Jiang, X., Wu, F., Gao, L., Sun, Z., Lu, Q., Dvivedi-Yu, J., Yang, Y., Callan, J., and Rogers, A. C. Adaptive retrieval augmented generation. arXiv preprint arXiv:2205.06083 (2023).
[8] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kettenhaber, T., Lewis, M., Wu, L.-M., Rocktaschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems (2020), 4957–4970.
[9] Liu, J., and Jannach, D. Hugging face | hugging face: An introduction (2022).
[10] Liu, M., Sze, T.-D., Kurby, R., Cheng, C., Pinckney, N., Liang, R., Allen, J., Anand, H., Banerjee, S., Bayraktaroglu, I., et al. Chipmoe: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.01176 (2023).
[11] Madan, A., Tandon, N., Gupta, P., Heilman, S., Sago, M., Wingrave, J., Adami, D., Dean, M., Pandruvada, S., Yang, Y., T. et al. Self-reinforcing representations for task shift. Advances in Neural Information Processing Systems (2022).
[12] Maini, G., Desai, R., Loébl, M., Najaftorkaman, C., Pascutti, R., Jafariani, R., Rozgie, B., Schick, T., Dvivedi-Yu, J., Cynelva, A., et al. Augmented language models: a survey. arXiv preprint arXiv:2302.09527 (2023).
[13] Riguade, A., Todri, R., Schreiber, M., Paraszeh, C., and Cohen, J. Nemo guardrails: A toolkit for controllable and safe ai/ml applications with programmable rails. arXiv preprint arXiv:2306.18917 (2023).
[14] Salvo-Zuccala, J., Khattar, D., Potts, C., and Zakiaria, M. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09522, (2023).
[15] Street, S., Joo, C., Chung, E., and Vidra, N. Improving retrieval for rag-based question answering models on financial documents. arXiv preprint arXiv:2304.12270, (2023).
[16] Vano, A., Yao, D., Chu, D., and Sharifi, I., Narasiman, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2305.10268 (2022).
[17] Zhu, X., Li, Z., Zhong, S., Liu, J., Liu, J., Lee, W., Deng, C., Dod, Z., and Weis, J. R. Large language models for information retrieval: A survey. arXiv preprint arXiv:2314.07017 (2023).


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:RAGCHECKER: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

Authors: Dongyu Ru*, Lin Qiu*, Xiangkun Hu*, Tianhang Zhang*, Peng Shi¹, Shuaichen Chang¹, Cheng Jiayang¹, Cunxiang Wang¹, Shichao Sun², Huanyu Li², Zizhao Zhang¹, Binjie Wang¹, Jiarong Jiang¹, Tong He¹, Zhiguo Wang¹, Pengfei Liu², Yue Zhang³, Zheng Zhang¹
Affiliations: ¹Amazon AWS AI, ²Shanghai Jiaotong University, ³Westlake University

Abstract

Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGCHECKER, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGCHECKER has significantly better correlations with human judgments than other evaluation metrics. Using RAGCHECKER, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGCHECKER can guide researchers and practitioners in developing more effective RAG systems.

1 Introduction

Retrieval-Augmented Generation (RAG) systems [18][7] enhance Large Language Models (LLMs) by incorporating external knowledge bases, enabling more precise and contextually relevant responses [7][53][13]. As these systems become integral to a variety of applications [54][2][8], it's imperative to develop robust and comprehensive evaluation frameworks to assess their performance and identify areas for improvement. Evaluating RAG systems, however, presents several challenges:

(1) modular complexity: The modular nature of RAG systems, comprising both a retriever and a generator, complicates the design of effective evaluation metrics. It is crucial to establish metrics that can holistically assess the entire system as well as evaluate the individual modules and their interplay [53], allowing for fully understanding the sources of the errors and misses and how they are generated. (2) metric limitation: Existing metrics for evaluating RAG systems, which are often rule-based or coarse-grained, fall short in providing accurate and interpretable results. Specifically, traditional metrics like recall@k and MRR [44] for retrievers depend on annotated chunks and a rigid chunking approach, missing out on the full semantic scope of the knowledge base. For generators, typical measures such as n-gram-based (e.g., BLEU [30], ROUGE [19]), embedding-based (e.g., BERTScore [56]), and LLM-based methods [45] perform well with concise answers but fail to detect finer distinctions in longer responses. To bridge these gaps, it is essential to develop detailed, semantic-based evaluation metrics that effectively capture the intricacies and overall quality of both the retrieval and generation components in RAG systems. (3) metric reliability: The reliability

 *Shared first authorship.†Work done during internship at Amazon. This work has been open sourced at https://github.com/amazon-science/RAGChecker


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Introduction

...of existing metrics for RAG remains under-explored. Effective evaluation metrics must not only accurately reflect system performance but also align with human judgments to ensure their utility in real-world scenarios.

To overcome these challenges, we introduce RAGCHECKER, an innovative evaluation framework designed for detailed analysis of both retrieval and generation processes. RAGCHECKER is based on claim-level entailment checking which involves operations of extracting claims from the response and ground truth answer and checking them against other texts. This approach enables fine-grained evaluation instead of response-level assessment. RAGCHECKER processes the user query, retrieved context, response, and ground truth answer, producing a suite of metrics:

1. **Overall Metrics** to provide a holistic view of the system performance, assessing the overall quality of the generated responses.
2. **Diagnostic Retriever Metrics** to evaluate the effectiveness of the retriever, identifying its strengths and weaknesses in finding relevant information from the knowledge base.
3. **Diagnostic Generator Metrics** to assess the performance of the generator, diagnosing how well the generator utilizes the retrieved context, handles noisy information, and generates accurate and faithful responses.

Compared to existing evaluation frameworks, RAGCHECKER provides a more comprehensive assessment of RAG systems. While some frameworks offer fine-grained evaluation only on certain metrics (e.g., RAGAS [5], Trulens [6], ARES [35]) or evaluate specific aspects of RAG (e.g., RGB [4], RECALL [22], MoNRLACL [40]), RAGCHECKER’s metrics are all based on fine-grained claim-level checking and are designed to provide actionable insights into the sources of errors.

To ensure the reliability of RAGCHECKER, we annotate a human judgment dataset to assess the correlations between the proposed metrics and human judgments. This meta-evaluation validates the effectiveness of RAGCHECKER in capturing the quality and reliability of RAG systems from a human perspective. We demonstrate the effectiveness of RAGCHECKER through comprehensive experiments evaluating 8 state-of-the-art RAG systems on a benchmark repurposed from public datasets across 10 domains. In-depth analysis of the evaluation results reveals that RAGCHECKER provides insightful diagnostic signals (Sec. 4.3), pointing the directions for improvements of RAG systems (Sec. 4.4).

The main contributions of this paper are as follows:

- We propose RAGCHECKER, a novel RAG evaluation framework that offers fine-grained evaluation for both the retriever and generator components, introducing new diagnostic metrics to provide actionable insights into the sources of errors.
- We conduct meta evaluation and verified RAGCHECKER has significantly better correlations with human judgements than other evaluation metrics.
- We perform extensive experiments evaluating 8 RAG systems on our curated benchmark across 10 domains, and uncover valuable insights, such as the trade-off between retrieval improvement and noise introduction, and the tendency of faithful open-source models to blind trust on context.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:2.1 Retrieval Augmented Generation

Large Language Models (LLMs) demonstrate strong capabilities in generating text, but there are also obstacles such as outdated information and the potential to hallucinate [42, 46, 12]. To address these issues, RAG retrieves external knowledge to generate responses with improved accuracy and reliability [7, 53, 13]. Integrating external knowledge is especially crucial in fields like legal, medical and finance, where precision and reliability are essential [24, 50, 55].

RAG systems have shown impressive performance across a range of tasks, including open-domain question answering [27, 100, 18], code generation [32, 57, 38] and dialogue [67, 16, 41]. Additionally, real world products like Bing Search[4] and Langchain[3] have integrated applications based on RAG.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation of RAG

Existing evaluation practices for RAG systems can be categorized into two main approaches: evaluating essential capabilities of generators only and assessing end-to-end performance of RAG systems.

Within the two components of a RAG system, the retriever has been well studied in recent years; thus a line of recent work focused on evaluating essential generator capabilities. RGB [4] evaluated 4 fundamental abilities required for generators including Noise Robustness, Negative Rejection, Information Integration and Counterfactual Robustness by manual constructed test sets. RECALL [22] introduced manually created counterfactual contexts into QA and text generation datasets to evaluate the counterfactual robustness of LLMs. NoMIRACL [40] evaluated LLMs’ robustness against first-stage retrieval errors of RAG systems with manually judged relevant and non-relevant datasets. Wu et al. [49] quantified the tug-of-war between LLMs’ faithfulness and internal prior by introducing varying levels of perturbations on the provided contexts. FaaF [15] introduced a fine-grained fact verification formulation to improve previous prompting-based approaches in evaluating factuality of generators. However, we argue that above generator-only evaluation approaches with manually constructed datasets cannot serve as a general RAG evaluation framework to reveal the entanglement of between generation results and different retrieval behaviors, as shown in the analysis of Sec. 4.3.

Another line of work focused on assessing end-to-end quality scores of RAG systems. Trulen.6 [6] introduced the concept of RAG Triad, which decomposes the quality scores into three aspects: context relevance, groundedness and answer relevance, then predicted the score by prompting LLMs or using NLI models. RAGAS [5] and ARES [35] followed the RAG Triad concept and improved the score prediction approaches on different datasets. CRUD-RAG [26] referred to the CRUD (Create, Read, Update and Delete) actions between users and knowledge bases to develop corresponding datasets and evaluation metrics for RAG systems. We compare the above four evaluation frameworks with RAGCHECKER in the meta evaluation of Sec. 4.2.

Besides, the following work also provided good insight or high quality datasets for end-to-end RAG evaluation. Liu et al. [21] conducted human evaluation to audit four popular generation search engines in terms of fluency, perceived utility, and verifiability. MEDRAG [50] constructed a medical RAG benchmark from medical QA datasets and evaluated medical RAG systems such as Repligator. MultiHop-RAG [39] generated multi-hop queries from news articles and evaluated RAG systems with QA accuracy. CQOA [52] proposed a novel approach to generate dynamic QA questions which requires latest information to answer. However, the evaluation metrics used in the work mentioned above rely either on human evaluation or simple textual accuracy, making them incapable of complex RAG scenarios that require long answer evaluation. Therefore, we do not include them in the meta evaluation.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:RAGCHECKER Framework

Formulation Define a modular RAG system as RAG = {R, G}, where R is the retriever and G is the generator. Given a query q and documents D, it first retrieves top-k relevant context {chunk_k} = R(q, D, k), and then generates a model response m = G({chunk_k}, q). For simplicity, we can also represent the overall RAG generation process as m = RAG(q, D).

Design Principle Given the compositional nature of RAG, we observe there are two major personae using a RAG evaluation framework. The first persona is a user that cares about the overall performance of RAGs and might choose a system with the best performance. Such a persona prefers a single value metric to compare and rank among RAG systems against a benchmark. The second persona is a developer that focuses on improving a RAG system with the need to identify causes of mistakes and potential rooms for improvements. Causes of errors in response can be classified into 1) retrieval errors, where the retriever fails to return complete and relevant context, and 2) generator errors, where the generator struggles to identify and leverage relevant information from context.

Consequently, metrics that reveal error causes should be different from those for overall performance, in the sense that error causes are module-specific or even reflected only by a certain behavior of a module. To help both personae to assess RAG performance, we design RAGCHECKER, a evaluation framework of RAG systems that consists of a benchmark with rich annotations and a set of diversely-purposed fine-grained metrics.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Inputs to RAGCHECKER

We prepare each sample in our benchmark dataset in the format of a tuple ⟨q, D, gt⟩ representing query, documents, and ground-truth answer, where query is the input question to a RAG system, documents form the database providing possible context and are processed into chunks with the same number of tokens, and ground-truth answer is a complete and correct answer for the input question. Further information is provided in Sec. 4.1.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Fine-grained Evaluation with Claim Entailment

As illustrated in Fig. 1, a response generated by a RAG system might be a mixture of correct ( ◯ ) and incorrect claims ( × ), while also missing some in-ground-truth claims ( △ ). In this sense, evaluating responses at a finer granularity is crucial to comprehensively assess the quality of an answer. For this purpose, we introduce two components: 1) a text-to-claim extractor that decomposes a given text T into a set of claims {ci}, and 2) a claim-entailment checker to determine whether a given claim c is entailed (∈) in a reference text Ref or not (∉).


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:RAGCHECKER Metrics

With the annotation and claim-level entailment functions specified, we next define the metrics. For a RAG user, we design metrics to compare the performance among RAG systems, including a single-value F1 score as an overall metric. For a RAG developer, on the other hand, we propose two sets of modular metrics for the retriever and the generator in a RAG system respectively, that aim to decompose the system and diagnose the source of errors. In the rest of this section, we will first introduce the overall metrics and then go over modular metrics for retriever and generator separately. The formulas for each metric are summarized in Appendix B.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Figure 1

Illustration of the proposed metrics in RAGCHECKER. The upper Venn diagram depicts the comparison between a model response and the ground truth answer, showing possible correct ( ◯ ), incorrect ( × ), and missing claims ( △ ). The retrieved chunks are classified into two categories based on the type of claims they contain. Below, we define the overall, retriever, and generator metrics, illustrating how each component of the RAG system is evaluated for its performance.

- Overall Metrics:
  - precision: ◯ ∣ ◯ + ◯ ∣ × ∣ △
  - recall: ◯ ∣ ◯ + △

- Retriever Metrics:
  - context precision: □ ∣ □ + □ ∣ × ∣ ◼
  - claim recall: ◯ ∣ ◯ + △ ∣ ∈ △ ∣ ∉

- Generator Metrics:
  - context utilization: ◯ ∣ ◯ ∈ △ ∣ ∉
  - noise sensitivity: × ∣ ∈ ∪ ∉
  - hallucination: × ∣ ∈ ∪ ∉
  - self-knowledge: ◯ ∣ ◯ ∣ ×
  - faithfulness: ◯ ∣ ◯ ∈ △ ∣ ∉


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:3.3.1 Overall Metrics

To assess the overall response quality of a RAG system from a user’s perspective, we can compute the precision and recall at claim level for each model generated response against its paired ground-truth answer. Specifically, we first extract claims from a model response m and a ground-truth answer gt as \( c_i^{(m)} \) and \( c_i^{(gt)} \) respectively. Then, we define correct claims in the response as \( \{ c_i^{(m)} \mid c_i^{(m)} \in gt \} \), and correct claims in the ground-truth answer as \( \{ c_i^{(gt)} \mid c_i^{(gt)} \in m \} \). Two metrics can be computed directly: precision is the proportion of correct claims in all response claims, and recall is the proportion of correct claims in all ground-truth answer claims. Further, the harmonic average of precision and recall gives the F1 score, as the overall performance metric.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:3.3.2 Retriever Metrics

Ideally, a perfect retriever returns precisely all claims needed to generate the ground-truth answer. Completeness-wise, we can measure how many claims made in the ground-truth answer are covered by retrieved chunks. With retrieved chunks as the reference text, we compute claim recall as the proportion of \( \{ c_i^{(gt)} \mid c_i^{(gt)} \in \text{chunks} \} \).

Differently, we define the retriever precision at chunk-level instead of claim-level. A retrieved chunk is called relevant chunk (r-chunk), if any ground-truth claim is entailed in it. In other words, chunk \( c_i \) is a relevant chunk if \( \exists i, \text{s.t. } c_i^{(gt)} \in c_i^{\text{chunk}} \). The rest retrieved chunks are called irrelevant chunk (irr-chunk). The retriever’s context precision is defined as \( [r\text{-chunk}_j]/k \), where \( k \) is the number of all retrieved chunks.

Note that a chunk-level precision provides better interpretability than a claim-level one, because in practice RAG systems usually work with documents processed to be text chunks in a fixed size. That being said, it is likely that a chunk may contain relevant claims and irrelevant or misleading information at the same time. As a result, the best possible retriever can only achieve a claim-level precision score lower than 100%, and such an upper-bound varies depending on the actual text distribution in \( D \) and chunking strategy.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:3.3.3 Generator Metrics

Given \( k \) retrieved chunks (possibly mixing relevant and irrelevant information), a perfect generator would identify and include all ground-truth-relevant claims and ignore any that are not. Because the generator’s results have dependency on retrieved chunks, we provide in total six metrics characterizing different aspects of its performance.

Given a model response \( m \) and its claims \( \{ c_i^{(m)} \} \), we first compute the proportion of \( c_i^{(m)} \) that are entailed in retrieved chunks. This metric is faithfulness, as it describes how faithful the generator is to the provided context, thus the higher the better.

Next, we examine three types of incorrect response claims, i.e. \( \{ c_i^{(m)} \mid c_i^{(m)} \not \in gt \} \).

1. The first type includes incorrect claim that are entailed in a relevant chunk, then it indicates the generator is sensitive to noise coupled with useful information. The proportion of this type of claims to all \( \{ c_i^{(m)} \} \) is relevant noise sensitivity.
2. The second type includes incorrect claim that are entailed in an irrelevant chunk, then it indicates the generator is also sensitive to noise even in an irrelevant context. The proportion of these incorrect claims is irrelevant noise sensitivity.
3. Finally, the third type includes incorrect claims that are not entailed in any retrieved chunk, meaning all such claims are generated by the generator itself. Its proportion is hallucination.

Note that for simplicity we group the two noise sensitivities in Fig. 1, but later in Sec. 4.3 we can see that generators generally has different sensitivity to relevant and irrelevant noise.

Finally, we characterize how a generator uses information sources to produce correct claims. A correct claim not entailed by any chunk can only be based on generator’s self-knowledge, thus the proportion of these claims reflects how many correct claims are generated on its own. A lower self-knowledge score is better, when the generator is expected to fully depend on retrieved context only in a RAG system. On the other hand, we also check how much retrieved relevant information is captured.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Experiments

4.1 Experimental Setup

Baseline RAG Systems We apply RAGCHECKER to 8 customized RAG systems to demonstrate how these metrics reflect the properties and differences among them, and how they guide the refinement of these systems. The 8 RAG systems are combinations with 2 retrievers and 4 generators. For retrievers, we choose BM25 [33], a representative classic sparse retrieval framework, and E5-Mistral [48], the SOTA open-source dense retriever. Our four generators are GPT-4 [29], Mistral-8x7B [14], Llama3-8B, and Llama3-70B [8], covering open-source and proprietary LLMs in various sizes. Further details are deferred to Appendix D. We employ Llama3-70B as both the claim extractor and checker implemented by an open-sourced framework RefChecker. As a validation of its performance on the RefChecker’s hallucination detection benchmark, this setup outperforms the best purely open-sourced combinations reported in RefChecker’s paper (see Appendix G).

Benchmark Datasets For comprehensive evaluations, we curate a benchmark containing 4,162 queries across 10 domains. This benchmark is repurposed from public datasets of open domain question answering, spanning domains of Wikipedia, AI science, novel, biomedical, finance, lifestyle, recreation, science, technology and writing. We convert the short answers to long-form answers in the datasets to align with the current LLM-based RAG systems. Please refer to Appendix A for the details of the benchmark curation process. The statistics of the benchmark are shown in Tab. 1.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Meta Evaluation

4.2 Meta Evaluation

We first conduct the meta evaluation to verify the soundness of RAGCHECKER and compare with existing baseline RAG evaluation frameworks.

Baseline RAG Evaluation Frameworks We include a total of 10 metrics from Trulens [6], RAGAS [5], ARES [35] and CRUD-RAG [25] in the meta evaluation, as they are capable to evaluate end-to-end performance with long answers. Metrics selected for comparison along with their descriptions are summarized in Tab. 4 of Appendix C. To ensure a fair comparison, we use Llama3-70B-IntRust as the LLM backbone when applicable. Since models in the Llama3 family don’t provide an embedding model, baseline metrics requiring embedding capability still use their corresponding default LLM backbones. In addition to the 10 metrics detailed in the table, we also incorporate BLEU [31], ROUGE-L [20], and BERTScore [56] to assess the correlation between the generated responses and the ground truth answers.

Meta Evaluation Dataset All baseline metrics are designed with different aspects and functionalities to a certain degree, thus making an exact comparison over metric scores inapplicable. However, we argue that a good metric should reflect the relative human preference over different RAG systems. In this spirit, we construct the meta evaluation dataset with sampled instances from the generated responses of 8 baseline RAG systems introduced in Sec.4.1 on our benchmark. Each meta evaluation instance is a pair of responses from two baseline RAG systems given the same query. By considering all combinations over 10 domains and 28 baseline pairs, we end up with 280 instances for pairwise human preference labeling. For each instance, annotators compare a pair of responses based on correctness, completeness, and overall assessment. For each aspect, annotators measure their preferences as one of five relative choices, including significantly better, slightly better, tie, slightly worse and significantly worse. For quality control, each instance is annotated by two annotators, and their overall agreement and correlation are measured. To conclude, we build a meta evaluation dataset with 280 instances, each instance is labeled by two annotators with their preference in terms of correctness, completeness and overall assessment.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Statistics of the RAG Benchmark

This benchmark is repurposed from public datasets across 10 domains, containing 4,162 questions. For the domains of Finance, Lifestyle, Recreation, Technology, Science and Novel, the short answers are extended to long-form answers with GPT-4.

| Dataset    | Domain    | # Query | # Doc. | Source                | Example Query                                                             |
|------------|-----------|---------|--------|-----------------------|--------------------------------------------------------------------------|
| ClapNQ     | Wikipedia | 300     | 4,293  | ClapNQ                | Difference between russian blue and british blue cat                      |
| NovelQA    | Novel     | 280     | 19     | NovelQA               | When do the Ewell kids go to school?                                     |
| RobustQA Writing     | Writing   | 500     | 199,994| LoTTE, RobustQA  | What is the difference between online and internet?                      |
| RobustQA BioASQ      | Biomedical| 511     | 197,816| BioASQ, RobustQA  | What hand deformities do patients with Apert syndrome present with?       |
| RobustQA Finance     | Finance   | 500     | 57,638 | FioA, RobustQA     | Is it safer to send credit card number via unsecured website form or by e-mail? What safer options are there?|
| RobustQA Lifestyle   | Lifestyle | 500     | 119,461| LoTTE, RobustQA   | Can i eat a day old peanut butter sandwich?                              |
| RobustQA Recreation  | Recreation| 500     | 166,975| LoTTE, RobustQA   | Why are so many american (spy) movies set in europe?                      |
| RobustQA Science     | Science   | 500     | 125,368| LoTTE, RobustQA   | Where is the flaw in this proof that 1=2? (derivative of repeated addition)|
| RobustQA Technology  | Technology| 500     | 638,509| LoTTE, RobustQA   | Why not use larger cipher keys?                                           |
| KIWI       | AI Science| 71      | 429    | KIWI                 | What are the prior approaches proposed to improve faithfulness of the reasoning steps generated by LLMs and what tasks are they applied on?|

Meta Evaluation Process and Results
Based on the meta evaluation dataset, we perform the following evaluation process. Since the human preference labels can be seen as the score difference of a response pair, where $h_i = H(r^{B+}) − H(r^{B}) \in \{−2,−1,0,1,2\}$, with a baseline RAG evaluation model $E$, we compute a normalized score difference as $e_i = \frac{E(r_i^B+) − E(r_i^B+)}{\in \{−2,2\}$, where f is a linear normal function. Our meta evaluation is the correlation between $h_i$ and $e_i$; overall instances as reported in Tab. 2, together with the correlation between $h_i$ and $h_i'$ from two annotators as the upper-bound. In addition, we further compute human agreement rate as the proportion of instances satisfying $abs(h_i − h_i') \leq 1$, and the result is 90.95%.

Table 2: Correlation results with Human Evaluation of Correctness, Completeness, and Overall Assessment. We only show the metric with the best correlation for each baseline framework. Full results can be found in Tab. 5 of Appendix C.

| Baseline      | Metric                  | Correctness         | Completeness       | Overall Assessment |
|---------------|-------------------------|---------------------|--------------------|--------------------|
|               |                         | Pearson | Spearman | Pearson | Spearman | Pearson | Spearman |
| BLEU          | BLEU-avg                | 38.89   | 35.32    | 32.13   | 21.85   | 35.14   | 42.92   |
| ROUGE         | ROUGE-L                 | 31.75   | 31.72    | 47.88   | 45.67   | 43.10   | 43.21   |
| BERTScore     | BERTScore               | 30.34   | 27.05    | 37.93   | 40.05   | 33.51   | 35.57   |
| TuLENS        | Answer Relevance        | 31.20   | 27.37    | 37.24   | 37.91   | 35.11   | 35.59   |
| ARES          | Answer Relevance        | 18.63   | 16.84    | 20.13   | 18.13   | 18.51   | 16.26   |
| RAGAS         | Answer Similarity       | 41.07   | 43.21    | 53.16   | 61.35   | 43.81   | 57.23   |
| CRUD4RQ       | Recall                  | 30.93   | 27.13    | 45.11   | 43.76   | 41.25   | 39.71   |
| RAGChecker    | Same metric as human    | 49.66   | 46.95    | 60.07   | 58.11   | 61.93   | 60.90   |
| Human         | Annotator correlation   | 63.47   | 59.19    | 71.91   | 68.36   | 70.09   | 68.89   |

From the table, we can observe that RAGChecker has the strongest correlation with human preference in terms of three aspects. Among other baseline metrics, Answer Similarity of RAGAS, which is based on the stronger backbone model text-embedding-ada-002, shows the best performance. We also provide a detailed comparison between RAGChecker and this strongest baseline in Fig. 4.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Main Results

We present the averaged evaluation results for 8 RAG systems across 10 diverse domain datasets in Table 3. Additional results for all datasets are provided in Appendix H. The RAG system that exhibited the best performance in our experiments is E5-Mistral_GPT-4, owing to the strong retrieval capability of E5-Mistral coupled with the adept comprehension capabilities of GPT-4. Next, we provide a list of insights induced from Table 3, along with their interpretation and possible directions for improvements.

Table 3: The averaged evaluation results for different RAG systems across 10 datasets. The overall performance of the RAG system is quantified using precision (Prec.), recall (Rec.), and F1 scores. The retriever component is evaluated based on claim recall (CR) and context precision (CP), while the generator component is diagnosed through context utilization (CU), relevant noise sensitivity (NS(l)), irrelevant noise sensitivity (NS(ll)), hallucination (Hallu.), self-knowledge (SK), and faithfulness (Faith.). Additionally, the average number of response claims for each RAG system is provided.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Retriever Matters Consistently

The quality of retrieval is crucial, as evidenced by the notable differences in overall Precision, Recall and F1 scores when comparing BM25 with E5-Mistral with the generator fixed. This improvement is agnostic to the specific choice of generator, suggesting a consistent benefit from employing a better retriever.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Generator Model Size Brings All-Round Improvement

Paired to the same retriever, Llama3-8b consistently achieves better overall performance than Llama3-8b. More concretely, this superiority is supported by a better performance over every generator metric, such as improved context utilization, reduced noise sensitivity, and less hallucination.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Stable and Performant Context Utilization is Key

Among all generator metrics, we observe that context utilization strongly correlates to the overall F1 score, while such correlation is relatively weaker for other generator metrics. Also, generators’ context utilization are relatively stable between the two retrievers, meaning their overall recall can be improved with a better retriever. These observations indicate that the capability to fully utilize retrieved context is key, which is intuitive because the generator in a RAG system is expected to leverage context to surpass its self-knowledge.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Informative Context Improves Faithfulness and Reduces Hallucination

As E5-Mistral achieves better claim recall, we observe generators paired to it achieves better faithfulness, indicating generators are all capable to identify and leverage information in context. Similarly, hallucination and self-knowledge are both reduced as well.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Retriever Recall Trades-off with Generator Noise Sensitivity

Claim recall for a retriever characterizes the coverage of all information necessary to produce ground-truth answer. In practice, however, because of the fixed-size chunking strategy, retrieved relevant chunks may inevitably also carry over noise as part of the context. As retriever claim recall increases, all generators become more sensitive to such noise, which can be explained as their faithfulness to certain context is not discriminative enough. This observation shows that generators’ capability to precisely leverage relevant context is still a challenge.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Relevant Noise Sensitivity is More Challenging

For every baseline RAG system, there’s an apparent gap between its relevant and irrelevant noise sensitivity. In correlation to the last paragraph, it further enhance the point that generators demonstrate a chunk-level faithfulness. It means a relevant


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:4.4 Diagnosis on RAG Settings for Improvements

Guided by observations in Sec. 4.3, we modify settings commonly tuned in RAG systems that may lead to improvements, diagnose their working mechanisms with RAGCHECKER metrics, and provide suggestions for improvements on certain aspects. We experiment with different numbers of chunks, chunk sizes, chunk overlap ratios, and generation prompts. We highlight our main findings and suggestions as below, please refer to Appendix F for detailed analysis and results.

More Context Enhances Faithfulness. Increasing the number (k) and size of chunks improves the recall of more useful information (claim recall 61.5→77.6 with k 5→20, 70.3→77.6 with size 150→300). Consequently, this provides more context for the generators to be more faithful to it (faithfulness 88.1→92.2 with k 5→20, 91.2→92.2 with size 150→300), though at the same time they also become more sensitive to additional noise (noise sensitivity 34.0→35.4 with k 5→20, 34.5→35.4 with size 150→300). Improvements in the overall performance (F1 51.7→53.4 with k 5→20, 52.6→53.4 with size 150→300) indicates benefits from more context.

Explicit Requirements in Prompts Affect Generation Preferences. When prompts introduce explicit requirements for better faithfulness, context utilization, and lower noise sensitivity, generators show improvements in faithfulness (92.2→93.6), but struggle with the subtle tension between context utilization (52.9→63.7) and noise sensitivity (35.4→38.1).

Chunk Overlap Does Not Matter a Lot. The chunk overlap ratio is usually set to be non-zero to help generators better utilize surrounding information and identify chunks with coherent logic. However, it minimally affects generation performance, as retrieving more chunks sharing similar useful information (increased context precision 69.3→71.1) does not necessarily increase the total amount of retrieved useful information (comparable claim recall 77.8→78.1).

Suggestions to RAG Builders
Improving the retriever is an effective way to enhance overall performance. While a better embedding model leads to improvements in both precision and recall, moderately increasing the number and size of chunks improves recall and thus F1 with minimal efforts in practice. Note that the effect saturates as the total amount of relevant information is fixed, so they need not be too large for a balanced cost-performance. On the other hand, given a limited number of context, larger chunk sizes with fewer chunks are preferred for better context precision. However, when targeting better context utilization or reduced noise sensitivity, opposite adjustments should be made to alleviate the influence of noise.

When tuning the generator, the trilemma of context utilization, noise sensitivity, and faithfulness makes it difficult to improve all aspects simultaneously. RAG builders should prioritize certain aspects in the prompt based on their targets, user preferences and the generator's capability.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:5 Conclusion

This paper presents RAGCHECKER, a novel evaluation framework designed for RAG systems. We validate our comprehensive suite of metrics, both overall and modular, through rigorous human assessments, demonstrating a strong correlation with evaluations conducted by human annotators. We have undertaken a detailed evaluation of eight distinct RAG systems using these metrics, yielding pivotal insights into the behaviors of the retriever and generator components and the trade-offs inherent in RAG system designs. These findings not only deepen our understanding of RAG system architectures but also furnish critical guidance for future advancements in RAG applications.