@inproceedings{rizzi-etal-2024-soft,
    title = "Soft metrics for evaluation with disagreements: an assessment",
    author = "Rizzi, Giulia  and
      Leonardelli, Elisa  and
      Poesio, Massimo  and
      Uma, Alexandra  and
      Pavlovic, Maja  and
      Paun, Silviu  and
      Rosso, Paolo  and
      Fersini, Elisabetta",
    editor = "Abercrombie, Gavin  and
      Basile, Valerio  and
      Bernadi, Davide  and
      Dudy, Shiran  and
      Frenda, Simona  and
      Havens, Lucy  and
      Tonelli, Sara",
    booktitle = "Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.nlperspectives-1.9",
    pages = "84--94",
    abstract = "The move towards preserving judgement disagreements in NLP requires the identification of adequate evaluation metrics. We identify a set of key properties that such metrics should have, and assess the extent to which natural candidates for soft evaluation such as Cross Entropy satisfy such properties. We employ a theoretical framework, supported by a visual approach, by practical examples, and by the analysis of a real case scenario. Our results indicate that Cross Entropy can result in fairly paradoxical results in some cases, whereas other measures Manhattan distance and Euclidean distance exhibit a more intuitive behavior, at least for the case of binary classification.",
}

The move towards preserving judgement disagreements in NLP requires the identification of adequate evaluation metrics. We identify a set of key properties that such metrics should have, and assess the extent to which natural candidates for soft evaluation such as Cross Entropy satisfy such properties. We employ a theoretical framework, supported by a visual approach, by practical examples, and by the analysis of a real case scenario. Our results indicate that Cross Entropy can result in fairly paradoxical results in some cases, whereas other measures Manhattan distance and Euclidean distance exhibit a more intuitive behavior, at least for the case of binary classification.
Rizzi, Giulia Leonardelli, Elisa Poesio, Massimo Uma, Alex ra Pavlovic, Maja Paun, Silviu Rosso, Paolo Fersini, Elisabetta
Soft metrics for evaluation with disagreements: an assessment