Unlike pre-training, instruction tuning is often more efficient since only a moderate number of instances are used for training. Since instruction tuning can be considered as a supervised training process, its optimization is different from pre-training in several aspects [69], such as the training objective (_i.e.,_ sequence-to-sequence loss) and optimization configuration (_e.g.,_ smaller batch size and learning rate), which require special attention in practice.