task_categories:text-generation size_categories:1B<n<10B region:us
---
task_categories:
- text-generation
pretty_name: Streamable Minipile Tokenized
size_categories:
- 1B<n<10B
---

Original Dataset [JeanKaddour/minipile](https://huggingface.co/datasets/JeanKaddour/minipile)


See the [Thought Tokens Repository](https://github.com/ZelaAI/thought-tokens) for demonstration of streaming usage of this dataset and specific implementation of how this dataset was prepared.

Tokenized with the GPTNeoX tokenizer, split into sequences of length 513, intended for 512 input and 512 target ids.