task_categories:text-generation size_categories:100K<n<1M language:en license:cc-by-nc-sa-4.0 arxiv:2309.05463 arxiv:2305.07759 region:us
---
task_categories:
- text-generation
language:
- en
pretty_name: Tiny Orca Textbooks
size_categories:
- 100K<n<1M
license: cc-by-nc-sa-4.0
---

# Textbook-like Dataset: A Comprehensive Resource for Text-Based Skills Development in Small Language Models

This dataset is a collection of **147k synthetic textbooks** designed to enhance the text-based skills of small language models. The curriculum is meticulously structured to progress from simple to complex tasks, ensuring a gradual and effective learning experience during pretraining or finetuning SLMs.

The inspiration for this dataset comes from the technical report paper, [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463). The source texts incorporated in this dataset are derived from the [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset, a well-known resource in the field.

Emphasizing text-based skills, this dataset serves as a practical reasoning for small language models to learn and do exercise, providing them with a diverse range of skills to learn and adapt from. The step-by-step progression mirrors the structure of a textbook, making it an ideal in-context learning sample.

### Disclaimer
While every effort has been made to ensure the accuracy of the information contained within this dataset, please note that it is provided 'as is' and without any warranties.

The use of the `textbook` field in this dataset is intended for research purposes only. You are advised to verify any information obtained from this dataset before acting upon it.

## Tiny Series

Explore the possibilities and limitations of building Small Language Models with these tiny gems of data!

- [TinyStories](https://arxiv.org/abs/2305.07759): The paper that sparked my interest in the journey of the tiny-* series.
- [tiny-codes](https://huggingface.co/datasets/nampdn-ai/tiny-codes): Collection of 1.6M short and clear code snippets that can help LLM models learn how to reason.
- [tiny-textbooks](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks): 420k "things of internet" synthetic textbooks. 
- [tiny-webtext](https://huggingface.co/datasets/nampdn-ai/tiny-webtext): A 6GB (4.5M records) variety of diverse webtext enriched with critical thinking methods to make unbiased English dataset.
- [tiny-lessons](https://huggingface.co/datasets/nampdn-ai/tiny-lessons): Subset of *tiny-textbooks* dataset, various lessons about "things of internet" augmented in a bite-sized textbook Markdown format.
- [tiny-bridgedict](https://huggingface.co/datasets/nampdn-ai/tiny-bridgedict): A dataset that links and transfers knowledge between English, Vietnamese, Chinese in a tiny multilingual models.

### Others small HQ datasets with textbook-like quality
- [devdocs.io](https://huggingface.co/datasets/nampdn-ai/devdocs.io): FreeCodeCamp has provided 189k comprehensive API documentation across a wide range of tech stacks and programming languages.
- [sciphi-python-textbook](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-python-textbook)
- [textbook_quality_programming](https://huggingface.co/datasets/vikp/textbook_quality_programming)
- [sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)
