@inproceedings{sasidharan-nair-etal-2024-exploring,
    title = "Exploring Reproducibility of Human-Labelled Data for Code-Mixed Sentiment Analysis",
    author = "Sasidharan Nair, Sachin  and
      Dinkar, Tanvi  and
      Abercrombie, Gavin",
    editor = "Balloccu, Simone  and
      Belz, Anya  and
      Huidrom, Rudali  and
      Reiter, Ehud  and
      Sedoc, Joao  and
      Thomson, Craig",
    booktitle = "Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.humeval-1.11",
    pages = "114--124",
    abstract = "Growing awareness of a {`}Reproducibility Crisis{'} in natural language processing (NLP) has focused on human evaluations of generative systems. While labelling for supervised classification tasks makes up a large part of human input to systems, the reproduction of such efforts has thus far not been been explored. In this paper, we re-implement a human data collection study for sentiment analysis of code-mixed Malayalam movie reviews, as well as automated classification experiments. We find that missing and under-specified information makes reproduction challenging, and we observe potentially consequential differences between the original labels and those we collect. Classification results indicate that the reliability of the labels is important for stable performance.",
}

Growing awareness of a {`}Reproducibility Crisis{'} in natural language processing (NLP) has focused on human evaluations of generative systems. While labelling for supervised classification tasks makes up a large part of human input to systems, the reproduction of such efforts has thus far not been been explored. In this paper, we re-implement a human data collection study for sentiment analysis of code-mixed Malayalam movie reviews, as well as automated classification experiments. We find that missing and under-specified information makes reproduction challenging, and we observe potentially consequential differences between the original labels and those we collect. Classification results indicate that the reliability of the labels is important for stable performance.
Sasidharan Nair, Sachin Dinkar, Tanvi Abercrombie, Gavin
Exploring Reproducibility of Human-Labelled Data for Code-Mixed Sentiment Analysis