We study the open-domain instruction following problem (Wang et al., 2022; Taori et al., 2023; Xu et al., 2023), where instructions vary in input format and tasks. Specifically, we consider two practical scenarios: (1) Starting with a given set of \(n\) seed instructions \(\mathcal{D}_{s}=\{I_{i}\}_{i=1}^{n}\), each drawn from some underlying distribution \(P_{I}\). For our experiments, we create a set of seed instructions using a held-out validation set. Practically, such instructions can be collected from the usage traffic of users. (2) In the absence of seed instructions, but with prior knowledge of downstream tasks, we directly start with a given set of instruction metadata \(\mathcal{M}\) (see Section 4.1 for definition). The latter scenario is especially useful for end users who lack existing instruction data but wish to jumpstart LLM tailored to specific applications, similar to the concept of GPTs (OpenAI, 2023).