language:en region:us
---
language: en
dataset_info:
  features:
  - name: prompt
    dtype: string
  - name: response
    dtype: string
  - name: source
    dtype: string
  splits:
  - name: test
    num_bytes: 18266901
    num_examples: 6807
  - name: train
    num_bytes: 220790357
    num_examples: 56167
  download_size: 137475849
  dataset_size: 239057258
---
# MosaicML Instruct V3

This is an aggregate dataset, comprised of [Dolly HHRLHF](https://huggingface.co/datasets/mosaicml/dolly_hhrlhf) (derived from the [Databricks Dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) and the [Anthropic Helpful and Harmless (HH-RLHF)](https://huggingface.co/datasets/Anthropic/hh-rlhf) datasets), combined with [Competition Math](https://huggingface.co/datasets/competition_math), [Duorc](https://huggingface.co/datasets/duorc), [CoT GSM8k](https://huggingface.co/datasets/conceptofmind/cot_submix_original), [Qasper](https://huggingface.co/datasets/allenai/qasper), [Quality](https://huggingface.co/datasets/emozilla/quality), [Summ Screen FD](https://huggingface.co/datasets/tau/scrolls) and [Spider](https://huggingface.co/datasets/spider).
The intention was to create a permissively-licensed instruction-following dataset with a large number of longform samples.

## Data Processing

Some data was transformed during the creation of this dataset. This involved: formatting the data into the Alpaca format, filtering for length, filtering for duplicates, adding instructions (for summarization and QA datasets), and making the instructions more like human input (transforming case, adding typos, etc).

## Data Mix


| Data Source | Number of Samples | Proportion (By Count of Samples) | Number of Tokens in Source | Proportion (By Count of Tokens) |
|-------------|------------|------------|------------|------------|
| competition_math | 4,995 | 8.89% | 1.6 M | 3.66% |
| cot_gsm8k | 4,995 | 8.89% | 3.36 M | 7.67% |
| dialogsum | 400 | 0.71% | 0.1 M | 0.23% |
| dolly_hhrlhf | 34,333 | 61.13% | 5.89 M | 13.43% |
| duorc | 4,986 | 8.88% | 7.8 M | 17.80% |
| qasper | 1,998 | 3.56% | 8.72 M | 19.90% |
| quality | 1,963 | 3.49% | 11.29 M | 25.78% |
| scrolls/summ_screen_fd | 1,498 | 2.67% | 4.97 M | 11.33% |
| spider | 999 | 1.78% | 0.089 M | 0.20% |


## License/Attribution
<!--
**Copyright (2023) MosaicML, Inc.**
-->
This dataset was developed at MosaicML (https://www.mosaicml.com) and its use is subject to the CC BY-SA 3.0 license. 

Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license:

Wikipedia (various pages) - https://www.wikipedia.org/
Copyright © Wikipedia editors and contributors.

Dolly — Databricks (https://www.databricks.com)
Copyright © Databricks

When citing this dataset, please use the following:

```
@misc{mosaicml2023instruct-v3,
  author = {MosaicML},
  title = {MosaicML Instruct-v3 Dataset},
  year = {2023},
  publisher = {HuggingFace Datasets},
  howpublished = {https://huggingface.co/datasets/mosaicml/instruct-v3},
}
```
