@inproceedings{parrish-etal-2024-picture,
    title = "Is a picture of a bird a bird? A mixed-methods approach to understanding diverse human perspectives and ambiguity in machine vision models",
    author = "Parrish, Alicia  and
      Hao, Susan  and
      Laszlo, Sarah  and
      Aroyo, Lora",
    editor = "Abercrombie, Gavin  and
      Basile, Valerio  and
      Bernadi, Davide  and
      Dudy, Shiran  and
      Frenda, Simona  and
      Havens, Lucy  and
      Tonelli, Sara",
    booktitle = "Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.nlperspectives-1.1",
    pages = "1--18",
    abstract = "Human experiences are complex and subjective. This subjectivity is reflected in the way people label images for machine vision models. While annotation tasks are often assumed to deliver objective results, this assumption does not allow for the subjectivity of human experience. This paper examines the implications of subjective human judgments in the behavioral task of labeling images used to train machine vision models. We identify three primary sources of ambiguity: (1) depictions of labels in the images can be simply ambiguous, (2) raters{'} backgrounds and experiences can influence their judgments and (3) the way the labeling task is defined can also influence raters{'} judgments. By taking steps to address these sources of ambiguity, we can create more robust and reliable machine vision models.",
}

Human experiences are complex and subjective. This subjectivity is reflected in the way people label images for machine vision models. While annotation tasks are often assumed to deliver objective results, this assumption does not allow for the subjectivity of human experience. This paper examines the implications of subjective human judgments in the behavioral task of labeling images used to train machine vision models. We identify three primary sources of ambiguity: (1) depictions of labels in the images can be simply ambiguous, (2) raters{'} backgrounds and experiences can influence their judgments and (3) the way the labeling task is defined can also influence raters{'} judgments. By taking steps to address these sources of ambiguity, we can create more robust and reliable machine vision models.
Parrish, Alicia Hao, Susan Laszlo, Sarah Aroyo, Lora
Is a picture of a bird a bird? A mixed-methods approach to understanding diverse human perspectives and ambiguity in machine vision models