@inproceedings{gonzalez-torres-etal-2024-automated,
    title = "Automated Question-Answer Generation for Evaluating {RAG}-based Chatbots",
    author = "Gonz{\'a}lez Torres, Juan Jos{\'e}  and
      B{\^\i}ndil{\u{a}}, Mihai Bogdan  and
      Hofstee, Sebastiaan  and
      Szondy, Daniel  and
      Nguyen, Quang-Hung  and
      Wang, Shenghui  and
      Englebienne, Gwenn",
    editor = "Demner-Fushman, Dina  and
      Ananiadou, Sophia  and
      Thompson, Paul  and
      Ondov, Brian",
    booktitle = "Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.cl4health-1.25",
    pages = "204--214",
    abstract = "In this research, we propose a framework to generate human-like question-answer pairs with long or factoid answers automatically and, based on them, automatically evaluate the quality of Retrieval-Augmented Generation (RAG). Our framework can also create datasets that assess hallucination levels of Large Language Models (LLMs) by simulating unanswerable questions. We then apply the framework to create a dataset of question-answer (QA) pairs based on more than 1,000 leaflets about the medical and administrative procedures of a hospital. The dataset was evaluated by hospital specialists, who confirmed that more than 50{\%} of the QA pairs are applicable. Finally, we show that our framework can be used to evaluate LLM performance by using Llama-2-13B fine-tuned in Dutch (Vanroy, 2023) with the generated dataset, and show the method{'}s use in testing models with regard to answering unanswerable and factoid questions appears promising.",
}

In this research, we propose a framework to generate human-like question-answer pairs with long or factoid answers automatically and, based on them, automatically evaluate the quality of Retrieval-Augmented Generation (RAG). Our framework can also create datasets that assess hallucination levels of Large Language Models (LLMs) by simulating unanswerable questions. We then apply the framework to create a dataset of question-answer (QA) pairs based on more than 1,000 leaflets about the medical and administrative procedures of a hospital. The dataset was evaluated by hospital specialists, who confirmed that more than 50{\%} of the QA pairs are applicable. Finally, we show that our framework can be used to evaluate LLM performance by using Llama-2-13B fine-tuned in Dutch (Vanroy, 2023) with the generated dataset, and show the method{'}s use in testing models with regard to answering unanswerable and factoid questions appears promising.
Gonz{\'a}lez Torres, Juan Jos{\'e} B{\^\i}ndil{\u{a}}, Mihai Bogdan Hofstee, Sebastiaan Szondy, Daniel Nguyen, Quang-Hung Wang, Shenghui Englebienne, Gwenn
Automated Question-Answer Generation for Evaluating RAG-based Chatbots