Model Card for roberta-base-namecalling

This is a roBERTa-base model fine-tuned on ~12K social media posts annotated for the presence or absence of namecalling.

How to Get Started with the Model

You can use this model directly with a pipeline for text classification:

>>> import transformers
>>> model_name = "civility-lab/roberta-base-namecalling"
>>> classifier = transformers.TextClassificationPipeline(
...     tokenizer=transformers.AutoTokenizer.from_pretrained(model_name),
...     model=transformers.AutoModelForSequenceClassification.from_pretrained(model_name))
>>> classifier("Be careful around those Democrats.")
[{'label': 'not-namecalling', 'score': 0.9995089769363403}]
>>> classifier("Be careful around those DemocRats.")
[{'label': 'namecalling', 'score': 0.996940016746521}]

Model Details

This is a 2023 update of the model built by Ozler et al. (2020) incorporating data from Rains et al. (2021) and using a more recent version of the transformers library.

Developed by: Steven Bethard, Kate Kenski, Steve Rains, Yotam Shmargad, Kevin Coe
Language: en
License: apache-2.0
Parent Model: roberta-base
Resources for more information:
- GitHub Repo
- Kadir Bulut Ozler; Kate Kenski; Steve Rains; Yotam Shmargad; Kevin Coe; and Steven Bethard. Fine-tuning for multi-domain and multi-label uncivil language detection. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 28–33, Online, November 2020. Association for Computational Linguistics
- Stephen A Rains; Yotam Shmargad; Kevin Coe; Kate Kenski; and Steven Bethard. Assessing the Russian Troll Efforts to Sow Discord on Twitter during the 2016 U.S. Election. Human Communication Research, 47(4): 477-486. 08 2021.
- Stephen A Rains; Jake Harwood; Yotam Shmargad; Kate Kenski; Kevin Coe; and Steven Bethard. Engagement with partisan Russian troll tweets during the 2016 U.S. presidential election: a social identity perspective. Journal of Communication, 73(1): 38-48. 02 2023.

Uses

The model is intended to be used for text classification, taking as input social media posts and predicting as output whether the post contains namecalling.

It is not intended to generate namecalling, and it should not be used as part of any incivility generation model.

Training Details

The model was trained on data from four sources: comments on the Arizona Daily Star website from 2011, Russian troll Tweets from 2012-2018, Tucson politician Tweets from 2018, and US presidential primary Tweets from 2019. Each dataset was annotated for the presence of namecalling following the approach of Coe et al. (2014) and split into training, development, and test partitions.

The roberta-base model was fine-tuned on the combined training partitions from all four datasets, with texts tokenized using the standard roberta-base tokenizer.

Evaluation

The model was evaluated on the test partition of each of the datasets. It achieves the following F1 scores:

0.58 F1 on Arizona Daily Star comments
0.71 F1 on Russian troll Tweets
0.71 F1 on Tucson politician Tweets
0.81 F1 on US presidential primary Tweets

Limitations and Biases

The human coders and their trainers were mostly Western, educated, industrialized, rich and democratic (WEIRD), which may have shaped how they evaluated incivility. The trained models will reflect such biases.

Environmental Impact

Hardware Type: Tesla V100S-PCIE-32GB
Hours used: 22
HPC Provider: https://hpc.arizona.edu/
Carbon Emitted: 2.85 kg CO2 (estimated by ML CO2 Impact)

Citation

@inproceedings{ozler-etal-2020-fine,
    title = "Fine-tuning for multi-domain and multi-label uncivil language detection",
    author = "Ozler, Kadir Bulut  and
      Kenski, Kate  and
      Rains, Steve  and
      Shmargad, Yotam  and
      Coe, Kevin  and
      Bethard, Steven",
    booktitle = "Proceedings of the Fourth Workshop on Online Abuse and Harms",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.alw-1.4",
    doi = "10.18653/v1/2020.alw-1.4",
    pages = "28--33",
}