Large language models (LLMs) have exhibited remarkable capabilities across a wide array of natural language processing (NLP) tasks Brown et al. (2020); Ouyang et al. (2022); OpenAI (2023); Anil et al. (2023). In particular, LLMs can be trained for improved instruction-following through various methods, including fine-tuning on human-annotated data Touvron et al. (2023); Bai et al. (2022) or extracted knowledge from stronger LLMs Wang et al. (2022); Taori et al. (2023); Chiang et al. (2023); Peng et al. (2023). Recent progress in this area highlights the critical role of high-quality data in enhancing LLMs' instruction-following capabilities Zhou et al. (2023); Kopf et al. (2023); Chen et al. (2023). However, acquiring such data through human annotation remains cost-prohibitive and difficult to scale, hindering further progress.