license:mit region:us
NEWSROOM is a large dataset for training and evaluating summarization systems.
It contains 1.3 million articles and summaries written by authors and
editors in the newsrooms of 38 major publications.
Dataset features includes:
  - text: Input news text.
  - summary: Summary for the news.
This dataset can be downloaded upon requests. Unzip all the contents
"train.jsonl, dev.josnl, test.jsonl" to the tfds folder.
@inproceedings{N18-1065,
  author    = {Grusky, Max and Naaman, Mor and Artzi, Yoav},
  title     = {NEWSROOM: A Dataset of 1.3 Million Summaries
               with Diverse Extractive Strategies},
  booktitle = {Proceedings of the 2018 Conference of the
               North American Chapter of the Association for
               Computational Linguistics: Human Language Technologies},
  year      = {2018},
}
---
license: mit
---

Modification of the [newsroom](https://huggingface.co/datasets/newsroom) dataset in Hugging Face. The main goal is to reproduce the results on BART. 

References: https://huggingface.co/datasets/newsroom/blob/main/newsroom.py

Major changes: 
1. replace the "\n\n" with one whitespace in the text 
2. remove the html related tags, remove the latin1 coded characters in the summary.
3. remove unnessary data features.

The main motivation for doing such modifications is that with the original data, particularly the summary data, the prediction will be weird and would involve multiple lines, which then makes the number of predictions unmatched with the number of references. 