We evaluate CodecLM on four widely-used open-domain instruction-following benchmarks with diverse instruction distributions to reduce evaluation bias. Our test benchmarks include Evol-Instruct (Xu et al., 2023), Vicuna (Chiang et al., 2023), Self-Instruct (Wang et al., 2022) and Koala (Geng et al., 2023). To complement the evaluation, we also evaluate on two standard NLPbenchmarks MMLU Hendrycks et al. (2020) and BBH Suzgun et al. (2022) in Appendix A.7. Please refer to Appendix A.1 for benchmark details.