@inproceedings{fort-etal-2024-stereotypical,
    title = "Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts",
    author = "Fort, Karen  and
      Alonso Alemany, Laura  and
      Benotti, Luciana  and
      Bezan{\c{c}}on, Julien  and
      Borg, Claudia  and
      Borg, Marthese  and
      Chen, Yongjian  and
      Ducel, Fanny  and
      Dupont, Yoann  and
      Ivetta, Guido  and
      Li, Zhijian  and
      Mieskes, Margot  and
      Naguib, Marco  and
      Qian, Yuyan  and
      Radaelli, Matteo  and
      Schmeisser-Nieto, Wolfgang S.  and
      Raimundo Schulz, Emma  and
      Saci, Thiziri  and
      Saidi, Sarah  and
      Torroba Marchante, Javier  and
      Xie, Shilin  and
      Zanotto, Sergio E.  and
      N{\'e}v{\'e}ol, Aur{\'e}lie",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1545",
    pages = "17764--17769",
    abstract = "Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting The study of bias, fairness and social impact in Natural Language Processing (NLP) lacks resources in languages other than English. Our objective is to support the evaluation of bias in language models in a multilingual setting. We use stereotypes across nine types of biases to build a corpus containing contrasting sentence pairs, one sentence that presents a stereotype concerning an underadvantaged group and another minimally changed sentence, concerning a matching advantaged group. We build on the French CrowS-Pairs corpus and guidelines to provide translations of the existing material into seven additional languages. In total, we produce 11,139 new sentence pairs that cover stereotypes dealing with nine types of biases in seven cultural contexts. We use the final resource for the evaluation of relevant monolingual and multilingual masked language models. We find that language models in all languages favor sentences that express stereotypes in most bias categories. The process of creating a resource that covers a wide range of language types and cultural settings highlights the difficulty of bias evaluation, in particular comparability across languages and contexts.",
}

Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting The study of bias, fairness and social impact in Natural Language Processing (NLP) lacks resources in languages other than English. Our objective is to support the evaluation of bias in language models in a multilingual setting. We use stereotypes across nine types of biases to build a corpus containing contrasting sentence pairs, one sentence that presents a stereotype concerning an underadvantaged group and another minimally changed sentence, concerning a matching advantaged group. We build on the French CrowS-Pairs corpus and guidelines to provide translations of the existing material into seven additional languages. In total, we produce 11,139 new sentence pairs that cover stereotypes dealing with nine types of biases in seven cultural contexts. We use the final resource for the evaluation of relevant monolingual and multilingual masked language models. We find that language models in all languages favor sentences that express stereotypes in most bias categories. The process of creating a resource that covers a wide range of language types and cultural settings highlights the difficulty of bias evaluation, in particular comparability across languages and contexts.
Fort, Karen Alonso Alemany, Laura Benotti, Luciana Bezan{\c{c}}on, Julien Borg, Claudia Borg, Marthese Chen, Yongjian Ducel, Fanny Dupont, Yoann Ivetta, Guido Li, Zhijian Mieskes, Margot Naguib, Marco Qian, Yuyan Radaelli, Matteo Schmeisser-Nieto, Wolfgang S. Raimundo Schulz, Emma Saci, Thiziri Saidi, Sarah Torroba Marchante, Javier Xie, Shilin Zanotto, Sergio E. N{\'e}v{\'e}ol, Aur{\'e}lie
Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts