task_categories:feature-extraction language:en license:mit region:us
---
language:
- en
license: mit
task_categories:
- feature-extraction
pretty_name: Dialog-Fact Encod
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
  - split: validation
    path: data/validation-*
dataset_info:
  features:
  - name: dialogue
    dtype: string
  - name: summary
    dtype: string
  - name: is_truncated
    dtype: bool
  - name: is_augmented
    dtype: bool
  splits:
  - name: train
    num_bytes: 225951776.22338164
    num_examples: 336975
  - name: test
    num_bytes: 25105976.423639305
    num_examples: 37442
  - name: validation
    num_bytes: 27895380.35297907
    num_examples: 41602
  download_size: 174858508
  dataset_size: 278953133.0
---
# Dataset Card for "dfe-stacked_samsum"

This custom dataset [julep-ai/dfe-stacked_samsum](https://huggingface.co/datasets/julep-ai/dfe-stacked_samsum) was created from [stacked-summaries/stacked-samsum-1024](https://huggingface.co/datasets/stacked-summaries/stacked-samsum-1024) by:
1. Extracting summaries for corresponding dialogs to emulate "facts"
2. Then truncating the dialogs to emulate "missing information"
3. And then augmenting the dialogs using LLMs to emulate "additional information"

It is used to train our [Dialog-Fact Encoder](https://huggingface.co/julep-ai/dfe-base-en) model.

> This dataset is permissively licensed under the MIT license.

## Notebooks

The data preparation process is documented in the [notebook](https://huggingface.co/datasets/julep-ai/dfe-stacked_samsum/blob/main/data_prep.ipynb) and you can also view the [rendered pdf](https://huggingface.co/datasets/julep-ai/dfe-stacked_samsum/blob/main/data_prep.pdf).