language:tr arxiv:1903.10561 region:us
---
language:
- tr
---
Translation of bias evaluation framework of May et al. (2019) from [this repository](https://github.com/W4ngatang/sent-bias) and [this paper](https://arxiv.org/abs/1903.10561) into Turkish. There is a total of 37 tests including tests addressing gender-bias as well as tests designed to evaluate the ethnic bias toward Kurdish people in Türkiye context.


Abstract of the paper:
While the growing size of pre-trained language models has led to large improvements in a variety of natural language processing tasks, the success of these models comes with a price: They are trained on drastic amounts of mostly Web-based data, which often contains social stereotypes and biases that the models might pick up. This can have negative consequences, as models can abuse these biases in downstream tasks or applications. An application exemplifying the embedded cultural stereotypes is statistical machine translation, a common natural language processing task. Translations to English from a gender-neutral language such as Turkish, which does not have any grammatical gender like the gendered pronouns 'he' or 'she' in English, lead to gender-stereotyped sentences. For instance, Google Translate converts these Turkish sentences with gender-neutral pronouns: 'O bir doktor. O bir hemşire.' to these English sentences: 'He is a doctor. She is a nurse.' The same behavior can be observed when translating these Turkish sentences into other languages with grammatical gender like Spanish, Russian, and German. The gender-neutral Turkish pronoun 'o' is converted into gender-stereotyped pronouns in the respective language. Mitigating different types of bias in LMs would have diverse implications: On the one hand, it would allow us to avoid amplifying these biases. On the other hand, by avoiding algorithms enforcing social biases against minorities one could shift the social balance in the long term.

Previous research has primarily focused on the English language, especially in the realm of gender bias in language models. However, the investigation of more languages with different linguistic elements than English, especially the ones like Turkish that are grammatically gender-neutral, can deepen our insights into the role of gender bias in LMs. The goal of this thesis was to address this research gap and to investigate the significance of gender-bias in Turkish language models. We used existing bias evaluation frameworks on Turkish models by both translating existing English datasets and creating new ones designed to measure gender-bias in the context of Türkiye. We also extended the testing framework to evaluate Turkish models for their embedded ethnic bias toward Kurdish people. Based on the test outcomes, we suggested possible relations of the picked up biases to different model characteristics such as the model size, their multilingualism, and the training corpora.