language:sc language:ch language:zh license:cc-by-4.0 region:us
---
license: cc-by-4.0
language:
- sc
- ch
- zh
---
# Dataset Card for Dataset Name

## Dataset Description

- **Homepage:** 
- **Repository:https://github.com/tjunlp-lab/TGEA** 
- **Paper:Ge, H., Zhao, X., Liu, C., Zeng, Y., Liu, Q., & Xiong, D. (2022). TGEA 2.0: A Large-Scale Diagnostically Annotated Dataset with Benchmark Tasks for Text Generation of Pretrained Language Models. Advances in Neural Information Processing Systems, 35, 31612-31626.** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary

In order to diagnostically analyze and improve the capability of pretrained language models (PLMs) in text generation, we propose TGEA 2.0, to date the largest dataset built on machine-authored texts by PLMs with fine-grained semantic annotations on a wide variety of pathological generation errors. We collect 170K nominal, phrasal and sentential prompts from 6M natural sentences in 3 domains. These prompts are fed into 4 generative PLMs with their best decoding strategy to generate paragraphs. 
195,629 sentences are extracted from these generated paragraphs for manual 
annotation, where 36K erroneous sentences are detected, 42K erroneous spans are 
located and categorized into an error type defined in a two-level error taxonomy.
We define a Minimal Set of }rror-related 
Words (MiSEW) for each erroneous span, which not only provides 
error-associated words but also rationalizes the reasoning behind the error. 
Quality control with a pre-annotation and feedback loop is performed before and 
during the entire annotation process. With the diagnostically annotated dataset, 
we propose 5 diagnosis benchmark tasks (i.e., erroneous text detection, 
MiSEW extraction, erroneous span location and correction together with error type
classification) and 2 pathology mitigation benchmark tasks (pairwise comparison 
and word prediction). Experiment results on these benchmark tasks demonstrate th
at TGEA 2.0 is a challenging dataset that could facilitate further research on au
tomatic diagnosis and pathology mitigation over machine texts. 

### Languages

Chinese


### Cite

If you use the source codes here in your work, please cite the corresponding paper. The bibtex is listed below:

```
@inproceedings{DBLP:conf/nips/GeZLZ0X22,
  author       = {Huibin Ge and
                  Xiaohu Zhao and
                  Chuang Liu and
                  Yulong Zeng and
                  Qun Liu and
                  Deyi Xiong},
  title        = {{TGEA} 2.0: {A} Large-Scale Diagnostically Annotated Dataset with
                  Benchmark Tasks for Text Generation of Pretrained Language Models},
  booktitle    = {NeurIPS},
  year         = {2022},
  url          = {http://papers.nips.cc/paper\_files/paper/2022/hash/cd556f38dba3a6c367c42fa85fc0801c-Abstract-Datasets\_and\_Benchmarks.html},
  timestamp    = {Thu, 11 May 2023 17:08:22 +0200},
  biburl       = {https://dblp.org/rec/conf/nips/GeZLZ0X22.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}
```

### Data Splits

|  Train   | Dev  | Test  |
|  ----  | ----  | ----  |
| 156,502  | 19,563 |19,564 |

