**Implementation Details of CodecLM.** We split all benchmarks into 20% validation set and 80% evaluation set. We extract the instruction metadata from the validation set, see Appendix A.3 for more details. Depending on the specified total data size, we prompt the strong LLM to generate equal number of base instruction per metadata. We generate 500-8000 synthetic data throughout the experiments. We generate 4 rubrics and corresponding actions. At each iteration, we randomly choose 1 action for improving instruction. We run Self-Rubrics at most 4 iterations. For Contrastive Filtering, We set the scoring scale to 10 and the filtering threshold to 3 for all experiments. We align these configurations with Xu et al. (2023) and leave more detailed rationales of these configurations, additional hyperparameter settings, and training details in Appendix A.3-A.4.