task_categories:text-generation language:en license:mit arxiv:2309.10818 region:us
---
license: mit
task_categories:
- text-generation
language:
- en
pretty_name: SlimPajama-627B-divided
---

### Dataset Description:
This is a split version of [cerebras/SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B) that divides data based on its sources. 
The content of this dataset is the same as SlimPajama-627B.
We divide data from different sources based on the "redpajama_setname" and save them in different directories, which is convenient for future dataset combination related research.

This dataset consists of 15,967 jsonl files and is ~ 883G compressed.

### Primary Usage:

This dataset is used for our study: [SlimPajama-DC: Understanding Data Combinations for LLM Training](https://arxiv.org/abs/2309.10818).

For more details about the content in this dataset, please refer to the original [cerebras/SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B).

### License:
Please refer to the licenses of the data subsets you use.
- [Common Crawl Foundation Terms of Use](https://commoncrawl.org/terms-of-use/full/)
- [C4 license](https://huggingface.co/datasets/allenai/c4#license)
- GitHub was limited to MIT, BSD, or Apache licenses only
- Books: [the_pile_books3 license](https://huggingface.co/datasets/the_pile_books3#licensing-information) and [pg19 license](https://huggingface.co/datasets/pg19#licensing-information)
- [ArXiv Terms of Use](https://info.arxiv.org/help/api/tou.html)
- [Wikipedia License](https://huggingface.co/datasets/wikipedia#licensing-information)
- [StackExchange license on the Internet Archive](https://archive.org/details/stackexchange)