**Exploration on Distribution Matching.** As shown by previous results, generating metadata extracted from the downstream instruction distribution indeed helps. However, in practice, the extracted or human-written metadata may not be able to precisely characterize the instruction distribution. Therefore, it is necessary to explore the performance of CodecLM when the distribution represented by instruction metadata does not fully match the test distribution. As the true test distribution is complicated and not known as a prior, we approximate various extent of distribution matching by random subsampling from the set of metadata \(\mathcal{M}\). To control the effect of data quantity, we keep the total number of instruction-response pairs the same for each case. For example, when subsampling \(20\%\) of \(\mathcal{M}\), we prompt the strong LLM to generate 5 times more instructions for each metadata accordingly. The result is shown in the upper part of Figure 4, and we did observe the trend that the better instruction metadata captures the underlying distribution, the better performance the target LLM can achieve. Moreover, when the metadata matching proportion is equal or greater than \(60\%\), we obtain close performance as the fully-matched result. This observation highlights CodecLM's robustness under potential instruction metadata mismatch.