To monitor the pre-training process, we randomly sample 0.1% of our financial corpus as a financial test dataset. The model is also evaluated on the Pile test dataset. The loss trajectory for FinPythia-6.9B is reported in Figure 2. The training loss is smoothed using a moving average of 50 optimization steps. We observe a sharp decrease in Financial test (Fin test) loss during the early stage of continual pre-training, and the progress gradually becomes saturated, similar to the loss trajectory of training from scratch [31; 27]. The loss log suggests that domain-adaptive continual pre-trainingsucceeds in adopting Pythia to the financial domains at the expense of a mild increase in Pile loss (Pile test).