Finally, as shown in Section 5.5, although CodecLM is robust to moderate distribution mismatch, its performance still depends on how well the metadata captures the underlying instruction distribution. In practice, our collected seed instruction might differ from the actual test instructions. Or in the case that we directly create metadata from user specification, the users might change their mind at test time to send the model out-of-distribution instructions beyond the original metadata. As a consequence, CodecLM may suffer performance degradation under distribution mismatch. As a remedy, we can constantly collect user instruction traffic or user feedback to update the generated data from CodecLM, and continuously update the target LLM.