**Combining Instruction Tuning and Pre-Training.** To make the tuning process more effective and stable, OPT-IML [95] incorporates pre-training data during instruction tuning, which can be regarded as regularization for model tuning. Further, instead of using a separate two-stage process (_pre-training_ then _instruction tuning_), some studies attempt to train a model from scratch with a mixture of pre-training data (_i.e.,_ plain texts) and instruction tuning data (_i.e.,_ formatted datasets) using multi-task learning [82]. Specifically, GLM-130B [93] and Galactica [35] integrate instruction-formatted datasets as a small proportion of the pre-training corpora to pre-train LLMs, which potentially achieves the advantages of pre-training and instruction tuning at the same time.