arxiv:2305.07759 region:us
---
dataset_info:
  features:
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 1911420483
    num_examples: 2119719
  - name: validation
    num_bytes: 19306310
    num_examples: 21990
  download_size: 1000775442
  dataset_size: 1930726793
---
A description of this dataset can be found at https://arxiv.org/abs/2305.07759

Copied from roneneldan/TinyStories

Modified with:

```
import ftfy.bad_codecs
from datasets import Dataset, DatasetDict

train = open('./TinyStories-train.txt', 'r', encoding='sloppy-windows-1252').read()
train = train.split('<|endoftext|>')
train = [l.strip() for l in train]

valid = open('./TinyStories-valid.txt', 'r', encoding='sloppy-windows-1252').read()
valid = valid.split('<|endoftext|>')
valid = [l.strip() for l in valid]

dataset = DatasetDict({
    'train': Dataset.from_dict({'text': train }),
    'validation': Dataset.from_dict({'text': valid}),
})
dataset.save_to_disk('./TinyStories')
```