@inproceedings{belz-thomson-2024-2024,
    title = "The 2024 {R}epro{NLP} Shared Task on Reproducibility of Evaluations in {NLP}: Overview and Results",
    author = "Belz, Anya  and
      Thomson, Craig",
    editor = "Balloccu, Simone  and
      Belz, Anya  and
      Huidrom, Rudali  and
      Reiter, Ehud  and
      Sedoc, Joao  and
      Thomson, Craig",
    booktitle = "Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.humeval-1.9",
    pages = "91--105",
    abstract = "This paper presents an overview of, and the results from, the 2024 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP{'}24), following on from three previous shared tasks on reproducibility of evaluations in NLP, ReproNLP{'}23, ReproGen{'}22 and ReproGen{'}21. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of reproducibility across the two fields. We describe the ReproNLP{'}24 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.",
}

This paper presents an overview of, and the results from, the 2024 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP{'}24), following on from three previous shared tasks on reproducibility of evaluations in NLP, ReproNLP{'}23, ReproGen{'}22 and ReproGen{'}21. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of reproducibility across the two fields. We describe the ReproNLP{'}24 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.
Belz, Anya Thomson, Craig
The 2024 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results