For our benchmark pre-trained LLM model, we select 1B and 6.9B parameter models from the Pythia suite [4]. The Pythia model suite offers a diverse array of model sizes, ranging from 70 million to 12 billion parameters. The continual pre-training configuration is tailored from Pythia's training setup [4]. Specifically, we set a learning rate of 1.2e-05 for FinPythia-6.9B and 3e-05 for _FinPythia-1B_, the smallest learning rates in their original schedules. We use small learning rates to mitigate catastrophic forgetting. We keep them constant throughout the course for efficient pre-training. We use the precision of fb16 rather than fp16 used in Pythia. We half the original batch size to 512.