We run the continual pre-training job on one P4d.24xlarge instance through AWS SageMaker. As the model size is moderate, we only use data parallelism via DeepSpeed ZeRO Stage 2 [20] with activation checkpointing enabled. It takes 18 days for FinPythia-6.9B to pre-train and 3 days for FinPythia-1B to pre-train on 24 billion tokens.