task_categories:summarization size_categories:1K<n<10K language:en region:us
---
task_categories:
- summarization
language:
- en
pretty_name: Summarization with Human Feedback
size_categories:
- 1K<n<10K
---


# REALSumm: Re-evaluating EvALuation in Summarization

Dataset assembled from https://github.com/neulab/REALSumm with the conversion script:

```python
idx = [1017, 10586, 11343, 1521, 2736, 3789, 5025, 5272, 5576, 6564, 7174, 7770, 8334, 9325, 9781, 10231, 10595, 11351, 1573, 2748, 3906, 5075, 5334, 5626, 6714, 7397, 7823, 8565, 9393, 9825, 10325, 10680, 11355, 1890, 307, 4043, 5099, 5357, 5635, 6731, 7535, 7910, 8613, 9502, 10368, 10721, 1153, 19, 3152, 4303, 5231, 5420, 5912, 6774, 7547, 8001, 8815, 9555, 10537, 10824, 1173, 1944, 3172, 4315, 5243, 5476, 6048, 6784, 7584, 8054, 8997, 9590, 10542, 11049, 1273, 2065, 3583, 4637, 5244, 5524, 6094, 6976, 7626, 8306, 9086, 9605, 10563, 11264, 1492, 2292, 3621, 4725, 5257, 5558, 6329, 7058, 7670, 8312, 9221, 9709]

link = "https://github.com/neulab/REALSumm/raw/master/scores_dicts/abs.pkl"

x = requests.get(link)
data = pickle.loads(x.content)

with open("/home/manuel/Downloads/summeval/src.txt", "r") as f:
   src = f.readlines()
src_cleaned = [src[i] for i in idx]
del src

models = list(data[0]["system_summaries"].keys())
tot_df = pd.DataFrame()
ref_sums = [data[x]["ref_summ"] for x in range(100)]
for model in models:
    df = pd.DataFrame([data[x]["system_summaries"][model]["scores"] for x in range(100)])
    sums = [data[x]["system_summaries"][model]["system_summary"] for x in range(100)]²
    df["model"] = model
    df["model_summary"] = sums
    df["ref_summary"] = ref_sums
    df["source"] = src_cleaned
    tot_df = pd.concat([tot_df, df])
tot_df = tot_df.reset_index()
tot_df = tot_df.rename(columns={"index": "doc_id"})
tot_df.index.name = "index"
```

## Dataset Structure

```
DatasetDict({
    train: Dataset({
        features: ['index', 'doc_id', 'rouge_1_f_score', 'rouge_2_recall', 'rouge_l_recall', 'rouge_2_precision', 'rouge_2_f_score', 'rouge_1_precision', 'rouge_1_recall', 'rouge_l_precision', 'rouge_l_f_score', 'js-2', 'mover_score', 'bert_recall_score', 'bert_precision_score', 'bert_f_score', 'litepyramid_recall', 'model', 'model_summary', 'ref_summary', 'source'],
        num_rows: 1400
    })
})
```


```
@inproceedings{Bhandari-2020-reevaluating,
title = "Re-evaluating Evaluation in Text Summarization",
author = "Bhandari, Manik  and Narayan Gour, Pranav  and Ashfaq, Atabak  and  Liu, Pengfei and Neubig, Graham ",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
year = "2020"
}
```