task_categories:conversational task_categories:question-answering task_categories:summarization task_categories:text-generation language:en license:apache-2.0 arxiv:2307.10172 region:us
@misc{zhang2023dialogstudio,
    title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI}, 
    author={Jianguo Zhang and Kun Qian and Zhiwei Liu and Shelby Heinecke and Rui Meng and Ye Liu and Zhou Yu and and Huan Wang and Silvio Savarese and Caiming Xiong},
    year={2023},
    eprint={2307.10172},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
---
extra_gated_heading: "Acknowledge to follow corresponding dataset licenses to access the repository"
extra_gated_button_content: "Agree and access repository"
license: apache-2.0
task_categories:
- conversational
- question-answering
- summarization
- text-generation
language:
- en
pretty_name: Dialog Studio
---


<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/logo.png"
alt="drawing" width="510"/>

# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
[Paper](https://arxiv.org/abs/2307.10172)|[Github](https://github.com/salesforce/DialogStudio)|[GDrive]



<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/DialogStudio_Stats.jpg"
alt="drawing" width="800"/>



**Follow the [DialogStudio](https://github.com/salesforce/DialogStudio) GitHub repository for latest information.**

### Datasets



### Load dataset
The datasets are split into several categories in HuggingFace 
```
Datasets/
├── Knowledge-Grounded-Dialogues
├── Natural-Language-Understanding
├── Open-Domain-Dialogues
├── Task-Oriented-Dialogues
├── Dialogue-Summarization
├── Conversational-Recommendation-Dialogs
```


You can load any dataset in the DialogStudio from the [HuggingFace hub](https://huggingface.co/datasets/Salesforce/dialogstudio) by claiming the `{dataset_name}`, which is exactly the dataset folder name. All available datasets are described in [dataset content](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv). For easier reference, [available dataset names](#Available Datasets) are also listed below.

Below is one example to load the [MULTIWOZ2_2](https://huggingface.co/datasets/Salesforce/dialogstudio/blob/main/task_oriented/MULTIWOZ2_2.zip) dataset under the [task-oriented-dialogues](https://huggingface.co/datasets/Salesforce/dialogstudio/tree/main/task_oriented) category:


Load the dataset
```python
from datasets import load_dataset

dataset = load_dataset('Salesforce/dialogstudio', 'MULTIWOZ2_2')
```
Here is the output structure of MultiWOZ 2.2
```python
DatasetDict({
    train: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 8437
    })
    validation: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 1000
    })
})
```


### Available Datasets

The ``data_name`` for  ``load_dataset("Salesforce/dialogstudio", data_name)`` can be found below. More detailed information for each dataset can be found in out [github](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv).
```python
    "natural_language_understanding": [
                        "ATIS", "ATIS-NER", "BANKING77", "BANKING77-OOS", "CLINC-Single-Domain-OOS-banking", 
                        "CLINC-Single-Domain-OOS-credit_cards", "CLINC150", "DSTC8-SGD", "HWU64", "MIT-Movie",
                        "MIT-Restaurant", "RESTAURANTS8K", "SNIPS", "SNIPS-NER", "TOP", "TOP-NER"
                    ],  
    "task_oriented": [
                        "ABCD", "AirDialogue", "BiTOD", "CaSiNo", "CraigslistBargains", 
                        "Disambiguation", "DSTC2-Clean", "FRAMES", "GECOR", "HDSA-Dialog", 
                        "KETOD", "KVRET", "MetaLWOZ", "MS-DC", "MuDoCo", 
                        "MulDoGO", "MultiWOZ_2.1", "MULTIWOZ2_2", "SGD", "SimJointGEN", 
                        "SimJointMovie", "SimJointRestaurant", "STAR", "Taskmaster1", "Taskmaster2", 
                        "Taskmaster3", "WOZ2_0"
                    ],
    "dialogue_summarization": [
                        "AMI", "CRD3", "DialogSum", "ECTSum", "ICSI", 
                        "MediaSum", "QMSum", "SAMSum", "TweetSumm", "ConvoSumm", 
                        "SummScreen_ForeverDreaming", "SummScreen_TVMegaSite"
                    ],
    "conversational_recommendation": [
                        "Redial", "DuRecDial-2.0", "OpenDialKG", "SalesBot",
                    ],
    "open_domain": [
                        "chitchat-dataset", "ConvAI2", "AntiScam", "Empathetic", "HH-RLHF", 
                        "PLACES3.5", "Prosocial", "SODA", "ShareGPT"
                    ],
    "knowledge_grounded": [
                        "CompWebQ", "CoQA", "CoSQL", "DART", "FeTaQA", 
                        "GrailQA", "HybridQA", "MTOP", "MultiModalQA", "SParC", 
                        "Spider", "SQA", "ToTTo", "WebQSP", "WikiSQL", 
                        "WikiTQ", "wizard_of_internet", "wizard_of_wikipedia"
                    ],  
```

# License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio: 
   - A portion of these datasets is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).
   - Some retain their original licenses even after modification.
   - For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.


# Citation
The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own.

```
@misc{zhang2023dialogstudio,
      title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI}, 
      author={Jianguo Zhang and Kun Qian and Zhiwei Liu and Shelby Heinecke and Rui Meng and Ye Liu and Zhou Yu and and Huan Wang and Silvio Savarese and Caiming Xiong},
      year={2023},
      eprint={2307.10172},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```