task_categories:text-generation size_categories:100K<n<1M language:en license:cc-by-nc-sa-4.0 arxiv:2309.05463 arxiv:2306.01116 arxiv:2304.08442 arxiv:2305.07759 doi:10.57967/hf/1126 region:us
---
task_categories:
- text-generation
language:
- en
pretty_name: Tiny Textbooks
size_categories:
- 100K<n<1M
license: cc-by-nc-sa-4.0
---

# Textbook-like Dataset: A High-Quality Resource for Small Language Models

The idea is simply inspired by the [Textbooks Are All You Need II: phi-1.5 technical report](https://arxiv.org/abs/2309.05463) paper. The source texts in this dataset have been gathered and carefully select the best of the [falcon-refinedweb](https://arxiv.org/abs/2306.01116) and [minipile](https://arxiv.org/abs/2304.08442) datasets to ensure the diversity, quality while tiny in size. The dataset was synthesized using 4x3090 Ti cards over a period of 500 hours, thanks to [Nous-Hermes-Llama2-13b](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b) finetuned model.

Why settle for low-quality text when you can train on a high-quality, textbook-like dataset? Training language models on subpar text can lead to several issues:

1. **Noise**: Such text often contains typos, grammatical errors, and poorly structured sentences, which can confuse models and degrade performance.
2. **Misinformation**: Low-quality web text may contain incorrect or misleading information, leading to models propagating these inaccuracies.
3. **Lack of Depth**: Subpar text often lacks the depth and detail found in high-quality content, limiting a model's understanding of complex topics.

Conversely, training on my clean and high-quality dataset offers numerous advantages:

1. **Accuracy**: The theoretical concepts in my dataset provide near accurate and detailed information, akin to a well-written textbook. (Need more contribute for facts check)
2. **Context**: Practical examples demonstrate how these concepts apply in real-world situations, offering valuable context.
3. **Performance**: Models trained on high-quality data can generate more accurate, insightful, and human-like text.

A standout feature of this dataset is its volume. It boasts a whopping **420,000 textbook documents**. This extensive collection ensures a wide coverage of topics and concepts, providing your models with a comprehensive and diverse learning resource.

Moreover, this dataset is generated using an open-source language model, ensuring the data is open for every researcher to process. I love the openness and that's why I want to contribute this dataset for the community to push over the limit.

Quality over quantity is a principle that holds true even in machine learning. Training on a large amount of low-quality tokens can lead to models learning and propagating the noise, inaccuracies, and poor structures present in the bad text. This can result in models that generate less accurate and less coherent outputs.

On the other hand, training on a smaller amount of high-quality tokens, like those in this dataset, can yield significantly better results. High-quality tokens provide accurate, well-structured, and meaningful information from which models can learn effectively. This leads to models that can generate more accurate, insightful, and human-like text.

In essence, it's about making every token count. Each high-quality token that a model learns from is a step towards better performance. So why waste computational resources and learning capacity on bad tokens when you can focus on high-quality ones? It's a more efficient and effective approach to training language models.

Choosing high-quality dataset over low-quality web text is akin to opting for a reliable textbook over scattered internet articles. This choice can significantly enhance the performance and reliability of your causal language models.

I'm excited to present this unique blend of theoretical concepts and practical examples designed to supercharge your causal language models. This isn't just another dataset; it's a high-quality resource that can help your models learn more effectively and with better common sense.

I hope this dataset is an useful resource for ML researchers working with small causal language models. I eagerly await your feedback and suggestions as I continue to refine and expand the dataset. Together, let's push the boundaries of what's possible with a **tiny language models**!

## Visualization

[Nomic Atlas](https://atlas.nomic.ai/map/0348f3f7-9280-404f-b6d3-d0b5993a6693/846bcd82-fcc5-474d-b24b-82d1b791f80b) 230k data points visualized thanks to Nomic AI platform.

### Disclaimer
While every effort has been made to ensure the accuracy of the information contained within this dataset, please note that it is provided 'as is' and without any warranties.

The use of the `textbook` field in this dataset is intended for research purposes only. You are advised to verify any information obtained from this dataset before acting upon it.

## Tiny Series

Explore the possibilities and limitations of building Small Language Models with these tiny gems of data!

- [TinyStories](https://arxiv.org/abs/2305.07759): The paper that sparked my interest in the journey of the tiny-* series.
- [tiny-codes](https://huggingface.co/datasets/nampdn-ai/tiny-codes): Collection of 1.6M short and clear code snippets that can help LLM models learn how to reason.
- [tiny-orca-textbooks](https://huggingface.co/datasets/nampdn-ai/tiny-orca-textbooks): Synthetic textbook to help model learn in-context on how it should perform task the right way.
- [tiny-webtext](https://huggingface.co/datasets/nampdn-ai/tiny-webtext): A 6GB (4.5M records) variety of diverse webtext enriched with critical thinking methods to make unbiased English dataset.
- [tiny-lessons](https://huggingface.co/datasets/nampdn-ai/tiny-lessons): Subset of this dataset, various lessons about "things of internet" augmented in a bite-sized textbook Markdown format.
- [tiny-bridgedict](https://huggingface.co/datasets/nampdn-ai/tiny-bridgedict): A dataset that links and transfers knowledge between English, Vietnamese, Chinese in a tiny multilingual models.

### Others small HQ datasets with textbook-like quality
- [devdocs.io](https://huggingface.co/datasets/nampdn-ai/devdocs.io): FreeCodeCamp has provided 189k comprehensive API documentation across a wide range of tech stacks and programming languages.
- [sciphi-python-textbook](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-python-textbook)
- [textbook_quality_programming](https://huggingface.co/datasets/vikp/textbook_quality_programming)
- [sciphi-textbooks-are-all-you-need](https://huggingface.co/datasets/emrgnt-cmplxty/sciphi-textbooks-are-all-you-need)