Instruction tuning improves the reasoning abilities of large language models (LLMs) , with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million
naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1)  recalling relevant documents, (2)  extracting instruction-response pairs, and (3)  refining the extracted pairs using open-source LLMs.
Fine-tuning base LLMs on this dataset, we buildMAmmoTH2models, which significantly boost performance on reasoning benchmarks. Notably,MAmmoTH2-7B’s (Mistral)  performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further trainingMAmmoTH2on public instruction tuning datasets yieldsMAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks.
Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.