**LLM as Encoder with Instruction Metadata.** We begin by encoding the given seed instructions \(\mathcal{D}_{s}=\{I_{i}\}_{i=1}^{n}\) into instruction _metadata_\(\mathcal{M}\), _i.e._, keywords that capture the underlying target instruction distribution. Inspired by the task pool by Wang et al. (2022) and the post-hoc analysis on skill distribution by Xu et al. (2023), we define the metadata as encompassing two key aspects: _use case_ and _skills_. Use case describes the intended task (_e.g._, question answering or creative writing), while Skills are the knowledge the LLM required to have to successfully respond to the given instruction (_e.g._, algorithms or communication). Skills are often generalizable to different use cases. Therefore, each instruction has a single use case and may involve multiple skills. To extract this metadata, we leverage the strong LLM \(f_{s}\) following the prompt template in Figure 7, Appendix A.9. While richer definitions are possible based on finer-grained instruction-following metrics Zhou et al. (2023), we prioritize use case and skills for their broad applicability across diverse instruction distributions. Future work can explore extending this metadata further.