Figure 3: Performance at each epoch on InvariantLAMA during continued pretraining in Main, Small, and Small-P1\(\rightarrow\)Small-P2 scenarios. Each marker indicates the result at each continual pretraining epoch.