@inproceedings{aliady-poesio-2024-linguistic,
    title = "Linguistic Acceptability and Usability Enhancement: A Case Study of {GWAP} Evaluation and Redesign",
    author = "Aliady, Wateen Abdullah  and
      Poesio, Massimo",
    editor = "Madge, Chris  and
      Chamberlain, Jon  and
      Fort, Karen  and
      Kruschwitz, Udo  and
      Lukin, Stephanie",
    booktitle = "Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.games-1.10",
    pages = "85--96",
    abstract = "Collecting high-quality annotations for Natural Language Processing (NLP) tasks poses challenges. Gamified annotation systems, like Games-with-a-Purpose (GWAP), have become popular tools for data annotation. For GWAPs to be effective, they must be user-friendly and produce high-quality annotations to ensure the collected data{'}s usefulness. This paper investigates the effectiveness of a gamified approach through two specific studies on an existing GWAP designed for collecting NLP coreference judgments. The first study involved preliminary usability testing using the concurrent think-aloud method to gather open-ended feedback. This feedback was crucial in pinpointing design issues. Following this, we conducted semi-structured interviews with our participants, and the insights collected from these interviews were instrumental in crafting player personas, which informed design improvements aimed at enhancing user experience. The outcomes of our research have been generalized to benefit other GWAP implementations. The second study evaluated the linguistic acceptability and reliability of the data collected through our GWAP. Our findings indicate that our GWAP produced reliable corpora with 91.49{\%} accuracy and 0.787 Cohen{'}s kappa.",
}

Collecting high-quality annotations for Natural Language Processing (NLP) tasks poses challenges. Gamified annotation systems, like Games-with-a-Purpose (GWAP), have become popular tools for data annotation. For GWAPs to be effective, they must be user-friendly and produce high-quality annotations to ensure the collected data{'}s usefulness. This paper investigates the effectiveness of a gamified approach through two specific studies on an existing GWAP designed for collecting NLP coreference judgments. The first study involved preliminary usability testing using the concurrent think-aloud method to gather open-ended feedback. This feedback was crucial in pinpointing design issues. Following this, we conducted semi-structured interviews with our participants, and the insights collected from these interviews were instrumental in crafting player personas, which informed design improvements aimed at enhancing user experience. The outcomes of our research have been generalized to benefit other GWAP implementations. The second study evaluated the linguistic acceptability and reliability of the data collected through our GWAP. Our findings indicate that our GWAP produced reliable corpora with 91.49{\%} accuracy and 0.787 Cohen{'}s kappa.
Aliady, Wateen Abdullah Poesio, Massimo
Linguistic Acceptability and Usability Enhancement: A Case Study of GWAP Evaluation and Redesign