task_categories:text-generation size_categories:1M<n<10M language:en region:us
---
language:
- en
size_categories:
- 1M<n<10M
task_categories:
- text-generation
pretty_name: SlimPajama-6B
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: text
    dtype: string
  - name: meta
    struct:
    - name: redpajama_set_name
      dtype: string
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: train
    num_bytes: 23918118724
    num_examples: 5489000
  - name: validation
    num_bytes: 39109042
    num_examples: 9347
  - name: test
    num_bytes: 40114950
    num_examples: 9346
  download_size: 14048972121
  dataset_size: 23997342716
---
Sampled version of [cerebras/SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B).

[Since the original data was shuffled before chunking](https://huggingface.co/datasets/cerebras/SlimPajama-627B/discussions/4), I only downloaded train/chunk1 (of 10 total) and further sampled 10%. This should result in roughly 6B tokens, hence SlimPajama-6B.

The dataset is 24GBs in storage size when decompressed (original dataset is over 2TBs) and has 5489000 rows.

The validation set and test set were sampled as well.

---
#### Data source proportions for SlimPajama-627B and SlimPajama-6B
For sanity purpose, I caluclated the byte proportion of the sampled version.


| Data source   | SlimPajama-627B | SlimPajama-6B |
| ------------- | ---------- | --------- |
| Commoncrawl   | 52.2%      | 54.1%    |
| C4            | 26.7%      | 28.7%    |
| GitHub        | 5.2%       | 4.2%     |
| Books         | 4.2%       | 3.7%     |
| ArXiv         | 4.6%       | 3.4%     |
| Wikpedia      | 3.8%       | 3.1%     |
| StackExchange | 3.3%       | 2.8%     |


---
Please refer to the original dataset for other info.
```
@misc{cerebras2023slimpajama,
  author = {Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan},
  title = {{SlimPajama: A 627B token cleaned and deduplicated version of RedPajama}},
  month = June,
  year = 2023,
  howpublished = {\url{https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama}},
  url = {https://huggingface.co/datasets/cerebras/SlimPajama-627B},
}
```