task_categories:text-classification size_categories:10K<n<100K language:en license:cc-by-nc-sa-4.0 irony disaggregated metadata perspectivism region:us
---
license: cc-by-nc-sa-4.0
task_categories:
- text-classification
language:
- en
tags:
- irony
- disaggregated
- metadata
- perspectivism
pretty_name: EPIC
size_categories:
- 10K<n<100K
---
# Dataset Card for EPICorpus

## Dataset Description

- **Repository:**  https://github.com/simonasnow/MultilingualPerspectivistNLU/tree/main 
- **Paper:** https://aclanthology.org/2023.acl-long.774/ 

### Dataset Summary

EPIC (English Perspectivist Irony Corpus) is a disaggregated English corpus for irony detection, containing 3,000 pairs of short conversations (posts-replies) from Twitter and Reddit, along with the demographic information of each annotator (age, nationality, gender, and so on). 

### Supported Tasks and Leaderboards

Irony classification task using soft labels (i.e., distribution of annotations) or hard labels (i.e., aggregated labels). 

### Languages

The language of EPIC is English. It contains texts in different varieties of English: British, American, Irish, Australian, and Indian.

## Dataset Structure

### Data Instances

Size of downloaded dataset files: 6.48 MB
Total amount of instances: 14,172
Total number of annotators: 74

### Data Fields

EPIC is structured as follows:
in rows, the annotation of each annotator (identified with a “user” id)
in columns, the various information about the target text annotated by the user (id_original, parent_text, language_instance, and language_variety), and the metadata about annotators (age, sex, ethnicity, and so on).

### Data Splits

The corpus is not split in training and validation/test sets.

## Dataset Creation

### Source Data

#### Initial Data Collection and Normalization

Information about the creation of EPIC are available in the paper: https://aclanthology.org/2023.acl-long.774/ 

#### Who are the source language producers?

Reddit and Twitter users.

### Annotations

#### Annotation process

The annotation process has been performed on Prolific platform. More information: https://aclanthology.org/2023.acl-long.774/ 

#### Who are the annotators?

The annotators are only English-speakers coming from the United Kingdom, United States of America, Australia, India, and Ireland.

### Personal and Sensitive Information

All the personal information available about the annotators in EPIC are provided by Prolific platform and under their consensus.
In the corpus, any metadata about the user who generated the texts on Reddit and Twitter are not available.

## Considerations for Using the Data

### Social Impact of Dataset

EPIC has not a specific social impact, but the proposition of datasets released with disaggregated annotations is encouraging the community to develop more inclusive, and thus respectful of various perspectives, AI-based technologies.

### Discussion of Biases

The analysis proposed in our work shows that in case of aggregation of labels employing a majority voting strategy, some biases can be introduced in the dataset. However, we release the dataset in its disaggregated form, and for its annotation we took into account various annotators with different sociodemographic traits.

### Other Known Limitations

While we tried to maintain a fair balance in terms of demographic profile of the annotators, we limited the resource to five varieties of English tied to five countries, leaving out other potential locations (e.g., New Zealand or Nigeria) or even more nuanced distinctions among language varieties.
About the self-identified gender dimension, we are aware of the wider spectrum of genders. However, this information is provided by the annotators only in a binary form.
Another potential limitation is that, in the spirit of constructing a perspectivist corpus, we fully trusted the contributors. While the chosen crowdsourcing platform (Prolific) is known for a high quality standard obtained, and we added a layer of checks through attention test questions, random noise in the annotation may still be present and undetected.

## Additional Information

### Dataset Curators

Department of Computer Science at the University of Turin.

### Citation Information

```latex
@inproceedings{frenda-etal-2023-epic,
    title = "{EPIC}: Multi-Perspective Annotation of a Corpus of Irony",
    author = "Frenda, Simona  and
      Pedrani, Alessandro  and
      Basile, Valerio  and
      Lo, Soda Marem  and
      Cignarella, Alessandra Teresa  and
      Panizzon, Raffaella  and
      Marco, Cristina  and
      Scarlini, Bianca  and
      Patti, Viviana  and
      Bosco, Cristina  and
      Bernardi, Davide",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.774",
    doi = "10.18653/v1/2023.acl-long.774",
    pages = "13844--13857",
}
```

### Contributions

The creation of this dataset was partially funded by the Multilingual Perspective-Aware NLU project in partnership with Amazon Alexa.
