The models are trained on A40 GPUs (48GB VRAM) for one epoch, taking up to 48 GPUs depending on the model size. The parameter-efficient training with LoRA is performed with PEFT library5. We also utilize DeepSpeed (Rasley et al., 2020) to optimize memory efficiency during the training process. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with a peak learning rate of 2e-4 and 5% warm-up cosine scheduler. Additionally, we apply gradient clipping with a value of 1.0 to mitigate potential gradient explosion.