**Data Generation for Instruction Tuning.** To address the high cost of human annotation for high-quality instruction-response pairs, several studies advocate for automating the data generation process Schick and Schutze (2021); Liu et al. (2022); Meng et al. (2023). Leveraging the in-context learning Brown et al. (2020) ability of LLMs, Wang et al. (2020), and et al. (2022); Honovich et al. (2022) prompt LLMs with seed instructions to generate synthetic ones. These are then fed to stronger LLMs, _e.g._, Chat-GPT, to generate responses for training the target (often smaller) LLM (Taori et al., 2023). As a representative work, WizardLM (Xu et al., 2023), designs a fixed set of human-crafted operations to increase complexity of instructions and control difficulty of generated data. Zhao et al. (2023); Zhou et al. (2023) further confirm the importance of instruction complexity for LLM alignment through empirical studies. Different from these works that rely on pre-defined rules without considering the downstream tasks, CodecLM enables automatically tailoring instructions for different downstream tasks and target LLMs. We also introduce Self-Rubrics and Contrastive Filtering to further identify the most effective instruction-response pairs.