During pre-training (or mixed-training), we use a weight decay of 0.03 and select the best among three learning rates: 0.0005, 0.001, 0.002; we also employ 1000 steps of learning rate warmup. During finetuning (FT), we use a weight decay of 0.01, and select the best among two learning rates: 0.0003 or 0.0005; we do not use learning rate warmup. During mixed-training, we use \(\text{QA}_{r}=0.3\) which means 30% of the training tokens come from instruction finetune data.