task_categories:text-classification size_categories:10K<n<100K source_datasets:multi_eurlex source_datasets:xsum source_datasets:csebuetnlp/xlsum source_datasets:mlsum source_datasets:amazon_polarity source_datasets:https://sinai.ujaen.es/investigacion/recursos/coah source_datasets:https://sinai.ujaen.es/investigacion/recursos/coar source_datasets:carblacac/twitter-sentiment-analysis source_datasets:cardiffnlp/tweet_sentiment_multilingual source_datasets:https://www.kaggle.com/datasets/ricardomoya/tweets-poltica-espaa source_datasets:wiki_lingua language:en language:es license:cc-by-nc-sa-4.0 arxiv:2309.11285 region:us
---
license: cc-by-nc-sa-4.0
task_categories:
- text-classification
language:
- en
- es
pretty_name: AuTexTification 2023
size_categories:
- 10K<n<100K
source_datasets:
- multi_eurlex
- xsum
- csebuetnlp/xlsum
- mlsum
- amazon_polarity
- https://sinai.ujaen.es/investigacion/recursos/coah
- https://sinai.ujaen.es/investigacion/recursos/coar
- carblacac/twitter-sentiment-analysis
- cardiffnlp/tweet_sentiment_multilingual
- https://www.kaggle.com/datasets/ricardomoya/tweets-poltica-espaa
- wiki_lingua
---

# Dataset Card for AuTexTification 2023

## Dataset Description

- **Homepage:** https://sites.google.com/view/autextification
- **Repository:** https://github.com/autextification/AuTexTification-Overview
- **Paper:** https://arxiv.org/abs/2309.11285

### Dataset Summary

AuTexTification 2023 @IberLEF2023 is a shared task focusing in Machine-Generated Text Detection and Model Attribution in English and Spanish.
The dataset includes human and generated text in 5 domains: tweets, reviews, how-to articles, news, and legal documents.
The generations are obtained using six language models: BLOOM-1B1, BLOOM-3B, BLOOM-7B1, Babbage, Curie, and text-davinci-003.
For more information, please refer to our overview paper: https://arxiv.org/abs/2309.11285

### Supported Tasks and Leaderboards

- Machine-Generated Text Detection
- Model Attribution

### Languages

English and Spanish

## Dataset Structure

### Data Instances

163k instances of labeled text in total. 

### Data Fields

For MGT Detection:
- id
- prompt
- text
- label
- model
- domain

For Model Attribution:
- id
- prompt
- text
- label
- domain

### Data Splits


- MGT Detection Data:
| Language | Split | Human  | Generated | Total  |
| -------- | ----- | ------ | --------- | ------ |
| English  | Train | 17.046 | 16.799    | 33.845 |
|          | Test  | 10.642 | 11.190    | 21.832 |
|          | Total | 27.688 | 27.989    |  |
| Spanish  | Train | 15.787 | 16.275    | 32.062 |
|          | Test  | 11.209 | 8.920     | 20.129 |
|          | Total | 26.996 | 25.195    |  |

- Model Attribution Data:
|          |       | BLOOM |       |       | GPT     |       |                  |        |
| -------- | ----- | ----- | ----- | ----- | ------- | ----- | ---------------- | ------ |
| Language | Split | 1B7   | 3B    | 7B    | babbage | curie | text-davinci-003 | Total  |
| English  | Train | 3.562 | 3.648 | 3.687 | 3.870   | 3.822 | 3.827            | 14.767 |
|          | Test  | 887   | 875   | 952   | 924     | 979   | 988              | 3.638  |
|          | Total | 4.449 | 4.523 | 4.639 | 4.794   | 4.801 | 4.815            |  |
| Spanish  | Train | 3.422 | 3.514 | 3.575 | 3.788   | 3.770 | 3.866            | 14.299 |
|          | Test  | 870   | 867   | 878   | 946     | 1.004 | 917              | 3.561  |
|          | Total | 4.292 | 4.381 | 4.453 | 4.734   | 4.774 | 4.783            |  |

## Dataset Creation

### Curation Rationale

Human data was gathered and used to prompt language models, obtaining generated data. 
Specific decisions were made to ensure the data gathering process was carried out in an unbiased manner, making the final human and generated texts probable continuations of a given prefix. 
For more detailed information, please refer to the overview paper: https://arxiv.org/abs/2309.11285

### Source Data
The following datasets were used as human text:
- multi_eurlex
- xsum
- csebuetnlp/xlsum
- mlsum
- amazon_polarity
- https://sinai.ujaen.es/investigacion/recursos/coah
- https://sinai.ujaen.es/investigacion/recursos/coar
- carblacac/twitter-sentiment-analysis
- cardiffnlp/tweet_sentiment_multilingual
- https://www.kaggle.com/datasets/ricardomoya/tweets-poltica-espaa
- wiki_lingua

These datasets were only used as sources of human text. The labels of the datasets were not employed in any manner.

### Licensing Information

CC-BY-NC-SA-4.0

### Citation Information

```
@inproceedings{autextification2023,
    title = "Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domains",
    author = "Sarvazyan, Areg Mikael and
      Gonz{\'a}lez, Jos{\'e} {\'A}ngel and
      Franco-Salvador, Marc and
      Rangel, Francisco and
      Chulvi, Berta and
      Rosso, Paolo",
    month = sep,
    year = "2023",
    address = "Jaén, Spain",
    booktitle = "Procesamiento del Lenguaje Natural",
}
```