task_categories:summarization size_categories:10K<n<100K language:en LM region:us
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: dialogue
    dtype: string
  - name: summary
    dtype: string
  splits:
  - name: train
    num_bytes: 9479117
    num_examples: 14732
  - name: test
    num_bytes: 534480
    num_examples: 819
  - name: validation
    num_bytes: 516419
    num_examples: 818
  download_size: 6735713
  dataset_size: 10530016
task_categories:
- summarization
language:
- en
tags:
- LM
size_categories:
- 10K<n<100K
---
# SAMSum

- Source: https://huggingface.co/datasets/samsum
- Num examples: 
  - 14,732 (train)
  - 818 (validation)
  - 819 (test)
- Language: English

```python
from datasets import load_dataset

load_dataset("tdtunlp/samsum_en")
```

- Format for Dialog Summarization task

```python
def preprocess(
    sample,
    sep_key="<|endofprompt|>",
    end_key="<|endoftext|>",
):
    chat_transcript = sample['dialogue']
    completion = sample['summary']
    return {'text': """{chat_transcript}
Summarize the above conversation between 2 people.
{sep_key}
{completion}
{end_key}""".format(
    chat_transcript=chat_transcript,
    sep_key=sep_key,
    completion=completion,
    end_key=end_key,
)}

"""
Olivia: Who are you voting for in this election? 
Oliver: Liberals as always.
Olivia: Me too!!
Oliver: Great
Summarize the above conversation between 2 people.
<|endofprompt|>
Olivia and Olivier are voting for liberals in this election. 
<|endoftext|>
"""
```