@inproceedings{omidi-shayegan-etal-2024-evaluation,
    title = "An Evaluation of Language Models for Hyperpartisan Ideology Detection in {P}ersian {T}witter",
    author = "Omidi Shayegan, Sahar  and
      Nejadgholi, Isar  and
      Pelrine, Kellin  and
      Yu, Hao  and
      Levy, Sacha  and
      Yang, Zachary  and
      Godbout, Jean-Fran{\c{c}}ois  and
      Rabbany, Reihaneh",
    editor = "Ojha, Atul Kr.  and
      Ahmadi, Sina  and
      Cinkov{\'a}, Silvie  and
      Fransen, Theodorus  and
      Liu, Chao-Hong  and
      McCrae, John P.",
    booktitle = "Proceedings of the 2nd Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.eurali-1.8",
    pages = "51--62",
    abstract = "Large Language Models (LLMs) have shown significant promise in various tasks, including identifying the political beliefs of English-speaking social media users from their posts. However, assessing LLMs for this task in non-English languages remains unexplored. In this work, we ask to what extent LLMs can predict the political ideologies of users in Persian social media. To answer this question, we first acknowledge that political parties are not well-defined among Persian users, and therefore, we simplify the task to a much simpler task of hyperpartisan ideology detection. We create a new benchmark and show the potential and limitations of both open-source and commercial LLMs in classifying the hyper-partisan ideologies of users. We compare these models with smaller fine-tuned models, both on the Persian language (ParsBERT) and translated data (RoBERTa), showing that they considerably outperform generative LLMs in this task. We further demonstrate that the performance of the generative LLMs degrades when classifying users based on their tweets instead of their bios and even when tweets are added as additional information, whereas the smaller fine-tuned models are robust and achieve similar performance for all classes. This study is a first step toward political ideology detection in Persian Twitter, with implications for future research to understand the dynamics of ideologies in Persian social media.",
}

Large Language Models (LLMs) have shown significant promise in various tasks, including identifying the political beliefs of English-speaking social media users from their posts. However, assessing LLMs for this task in non-English languages remains unexplored. In this work, we ask to what extent LLMs can predict the political ideologies of users in Persian social media. To answer this question, we first acknowledge that political parties are not well-defined among Persian users, and therefore, we simplify the task to a much simpler task of hyperpartisan ideology detection. We create a new benchmark and show the potential and limitations of both open-source and commercial LLMs in classifying the hyper-partisan ideologies of users. We compare these models with smaller fine-tuned models, both on the Persian language (ParsBERT) and translated data (RoBERTa), showing that they considerably outperform generative LLMs in this task. We further demonstrate that the performance of the generative LLMs degrades when classifying users based on their tweets instead of their bios and even when tweets are added as additional information, whereas the smaller fine-tuned models are robust and achieve similar performance for all classes. This study is a first step toward political ideology detection in Persian Twitter, with implications for future research to understand the dynamics of ideologies in Persian social media.
Omidi Shayegan, Sahar Nejadgholi, Isar Pelrine, Kellin Yu, Hao Levy, Sacha Yang, Zachary Godbout, Jean-Fran{\c{c}}ois Rabbany, Reihaneh
An Evaluation of Language Models for Hyperpartisan Ideology Detection in Persian Twitter