@inproceedings{fresen-etal-2024-reprohum,
    title = "{R}epro{H}um{\#}0043: Human Evaluation Reproducing Language Model as an Annotator: Exploring Dialogue Summarization on {AMI} Dataset",
    author = "Fresen, Vivian  and
      Wu-Urbanek, Mei-Shin  and
      Eger, Steffen",
    editor = "Balloccu, Simone  and
      Belz, Anya  and
      Huidrom, Rudali  and
      Reiter, Ehud  and
      Sedoc, Joao  and
      Thomson, Craig",
    booktitle = "Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.humeval-1.17",
    pages = "199--209",
    abstract = "This study, conducted as part of the ReproHum project, aimed to replicate and evaluate the experiment presented in {``}Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization{''} by Feng et al. (2021). By employing DialoGPT, BART, and PGN models, the study assessed dialogue summarization{'}s informativeness. Based on the ReproHum project{'}s baselines, we conducted a human evaluation for the AIMI dataset, aiming to compare the results of the original study with our own experiments. Our objective is to contribute to the research on human evaluation and the reproducibility of the original study{'}s findings in the field of Natural Language Processing (NLP). Through this endeavor, we seek to enhance understanding and establish reliable benchmarks in human evaluation methodologies within the NLP domain.",
}

This study, conducted as part of the ReproHum project, aimed to replicate and evaluate the experiment presented in {``}Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization{''} by Feng et al. (2021). By employing DialoGPT, BART, and PGN models, the study assessed dialogue summarization{'}s informativeness. Based on the ReproHum project{'}s baselines, we conducted a human evaluation for the AIMI dataset, aiming to compare the results of the original study with our own experiments. Our objective is to contribute to the research on human evaluation and the reproducibility of the original study{'}s findings in the field of Natural Language Processing (NLP). Through this endeavor, we seek to enhance understanding and establish reliable benchmarks in human evaluation methodologies within the NLP domain.
Fresen, Vivian Wu-Urbanek, Mei-Shin Eger, Steffen
ReproHum#0043: Human Evaluation Reproducing Language Model as an Annotator: Exploring Dialogue Summarization on AMI Dataset