During the training process, we utilized the AdamW optimizer with \(\beta_{1}\) set to 0.9, \(\beta_{2}\) set to 0.95, weight decay set to 0.1, and performed gradient clipping to limit it to 1.0. For continual pre-training, we start with a base learning rate of 5e-5, followed by a warm-up phase of 2000 steps, and then gradually reduced the learning rate according to a cosine schedule until it reached 10% of the base learning rate. The model is pre-trained for 23063 steps. For supervised fine-tuning, we start with the learning rate 1e-5 and train the model for 3 epochs. The model is trained on 32 Tesla V100 32GB GPUs.