language:en license:cc-by-nc-4.0 natural-language-inference news-articles-summarization arxiv:2305.11171 arxiv:2210.11416 region:us
---
license: cc-by-nc-4.0
language:
- en
tags:
- natural-language-inference
- news-articles-summarization
---

# **TrueTeacher**

## Dataset Summary

This is a large-scale synthetic dataset for training **Factual Consistency Evaluation** models, introduced in the [TrueTeacher paper (Gekhman et al, 2023)](https://arxiv.org/pdf/2305.11171.pdf).

## Dataset Details
The dataset contains model-generated summaries of articles from the train split of the **CNN/DailyMail** dataset [(Hermann et al., 2015)](https://proceedings.neurips.cc/paper_files/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf) 
which are annotated for factual consistency using **FLAN-PaLM 540B** [(Chung et al.,2022)](https://arxiv.org/pdf/2210.11416.pdf). 
Summaries were generated using summarization models with different capacities, which were created by fine-tuning **T5** [(Raffel et al., 2020)](https://jmlr.org/papers/volume21/20-074/20-074.pdf) on the **XSum** dataset [(Narayan et  al.,  2018)](https://aclanthology.org/D18-1206.pdf).
We used the following 5 capacities: T5-11B, T5-3B, T5-large, T5-base and T5-small.

## Data format

The data contains json lines with the following keys:

- `"summarization_model"` - The summarization model used to generate the summary.
- `"cnndm_id"` - The original id from the CNN/DailyMail dataset, this need to be used in order to retrieve the corresponding article from CNN/DailyMail (which was used as the grounding document).
- `"summary"` - The model-generated summary.
- `"label"` - A binary label ('1' - Factualy Consistent, '0' - Factualy Inconsistent).

Here is an example of a single data item:

```json
{
  "summarization_model": "T5-11B",
  "cnndm_id": "f72048a23154de8699c307e2f41157abbfcae261",
  "summary": "Children's brains are being damaged by prolonged internet access, a former children's television presenter has warned."
  "label": "1",
}
```

## Loading the dataset

To use the dataset, you need to fetch the relevant documents from the CNN/DailyMail dataset. The follwoing code can be used for that purpose:

```python
from datasets import load_dataset
from tqdm import tqdm

trueteacher_data = load_dataset("google/trueteacher", split='train')
cnn_dailymail_data = load_dataset("cnn_dailymail", version="3.0.0", split='train')
cnn_dailymail_articles_by_id = {example['id']: example['article'] for example in cnn_dailymail_data}
trueteacher_data_with_documents = []
for example in tqdm(trueteacher_data):
  example['document'] = cnn_dailymail_articles_by_id[example['cnndm_id']]
  trueteacher_data_with_documents.append(example)
```

## Intended Use

This dataset is intended for a research use (**non-commercial**) in English.

The recommended use case is training factual consistency evaluation models for summarization.

## Out-of-scope use 
Any use cases which violate the **cc-by-nc-4.0** license.

Usage in languages other than English. 

## Citation

If you use this dataset for a research publication, please cite the TrueTeacher paper (using the bibtex entry below), as well as the CNN/DailyMail, XSum, T5 and FLAN papers mentioned above.


```
@misc{gekhman2023trueteacher,
      title={TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models}, 
      author={Zorik Gekhman and Jonathan Herzig and Roee Aharoni and Chen Elkind and Idan Szpektor},
      year={2023},
      eprint={2305.11171},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```