In this subsection, we compare two continually pre-trained 405M parameter models to several baselines in the _two dataset weak shift_ (Pile \(\rightarrow\) SlimPajama) and _two dataset strong shift_ (Pile \(\rightarrow\) German) settings. Our main goal is to determine how the differences in distribution shift affect final performance.