For supervised fine-tuning, we employ the Adam optimizer with a cosine learning rate schedule spanning a total of 3 epochs of training. The maximum learning rate is set 2e-5 (except that 2e-6 for the Mistral-7b model) and there is a 4% linear warm-up. The maximum token length is set 2048, and the Vicuna-v1.1 Zheng et al. (2023) system prompt is used. All experiments are conducted on 8\(\times\)Nvidia H100 GPUs. Our most resource-intensive experiment, involving a 70B model and 960K data points, takes 1900 H100 GPU hours.