task_categories:summarization size_categories:100K<n<1M language:en region:us
---
dataset_info:
  features:
  - name: gem_id
    dtype: string
  - name: url
    dtype: string
  - name: title
    dtype: string
  - name: target
    dtype: string
  - name: references
    list: string
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 975957381
    num_examples: 306521
  - name: test
    num_bytes: 34940457
    num_examples: 11535
  - name: validation
    num_bytes: 35201391
    num_examples: 11535
  download_size: 655141522
  dataset_size: 1046099229
task_categories:
- summarization
language:
- en
size_categories:
- 100K<n<1M
---

# xlsum

- Source: https://huggingface.co/datasets/GEM/xlsum
- Num examples: 
  - 306,521 (train)
  - 11,535 (validation)
  - 11,535 (test)
- Language: English

```python
from datasets import load_dataset

load_dataset("tdtunlp/xlsum_en")
```

- Format for Summarization task

```python
def preprocess(sample):
    title = sample['title']
    article = sample['text']
    summary = sample['target']
    return {'text': f'<|startoftext|><|title|>{title}<|article|>{article}<|summary|>{summary}<|endoftext|>'}

"""
<|startoftext|><|title|>Weather alert issued for gale force winds in Wales<|article|>The Met Office has issued a yellow weather warning for wind covering Wales and England, starting from 21:00 GMT on Wednesday evening.

Travel and power are both likely to be disrupted, with the warning to remain in place until 15:00 on Thursday.

Gusts of 55mph (88kmh) are likely and could hit up to 70mph on coasts and hills, with heavy and blustery showers.

<|summary|>Winds could reach gale force in Wales with stormy weather set to hit the whole of the country this week.<|endoftext|>
"""
```