@inproceedings{antoine-etal-2024-extrinsic,
    title = "Extrinsic evaluation of question generation methods with user journey logs",
    author = "Antoine, Elie  and
      Besnehard, El{\'e}onore  and
      Bechet, Frederic  and
      Damnati, Geraldine  and
      Kergosien, Eric  and
      Laborderie, Arnaud",
    editor = "Balloccu, Simone  and
      Belz, Anya  and
      Huidrom, Rudali  and
      Reiter, Ehud  and
      Sedoc, Joao  and
      Thomson, Craig",
    booktitle = "Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.humeval-1.6",
    pages = "63--70",
    abstract = "There is often a significant disparity between the performance of Natural Language Processing (NLP) tools as evaluated on benchmark datasets using metrics like ROUGE or BLEU, and the actual user experience encountered when employing these tools in real-world scenarios. This highlights the critical necessity for user-oriented studies aimed at evaluating user experience concerning the effectiveness of developed methodologies. A primary challenge in such {``}ecological{''} user studies is their assessment of specific configurations of NLP tools, making replication under identical conditions impractical. Consequently, their utility is limited for the automated evaluation and comparison of different configurations of the same tool. The objective of this study is to conduct an {``}ecological{''} evaluation of a question generation within the context of an external task involving document linking. To do this we conducted an ''\textit{ecological}'' evaluation of a document linking tool in the context of the exploration of a Social Science archives and from this evaluation, we aim to derive a form of a {``}reference corpus{''} that can be used offline for the automated comparison of models and quantitative tool assessment. This corpus is available on the following link: https://gitlab.lis-lab.fr/archival-public/autogestion-qa-linking",
}

There is often a significant disparity between the performance of Natural Language Processing (NLP) tools as evaluated on benchmark datasets using metrics like ROUGE or BLEU, and the actual user experience encountered when employing these tools in real-world scenarios. This highlights the critical necessity for user-oriented studies aimed at evaluating user experience concerning the effectiveness of developed methodologies. A primary challenge in such {``}ecological{''} user studies is their assessment of specific configurations of NLP tools, making replication under identical conditions impractical. Consequently, their utility is limited for the automated evaluation and comparison of different configurations of the same tool. The objective of this study is to conduct an {``}ecological{''} evaluation of a question generation within the context of an external task involving document linking. To do this we conducted an ''\textit{ecological}'' evaluation of a document linking tool in the context of the exploration of a Social Science archives and from this evaluation, we aim to derive a form of a {``}reference corpus{''} that can be used offline for the automated comparison of models and quantitative tool assessment. This corpus is available on the following link: https://gitlab.lis-lab.fr/archival-public/autogestion-qa-linking
Antoine, Elie Besnehard, El{\'e}onore Bechet, Frederic Damnati, Geraldine Kergosien, Eric Laborderie, Arnaud
Extrinsic evaluation of question generation methods with user journey logs