In this subsection, we establish the effect of increasing parameter count by an order of magnitude on the final performance of continual pre-training. To accomplish this we compare two continually pre-trained models to several baselines at 405M and 10B parameter model sizes in the _two dataset weak shift_ (Pile \(\rightarrow\) SlimPajama) and _two dataset strong shift_ (Pile \(\rightarrow\) German) settings.