In addition to the CKL benchmark, we also show in Table 8 the performance on the dev set of KILT (Petroni et al., 2021) after finetuning each of the continually pretrained models of Table 2. Since KILT is made from Wikipedia, which corresponds to the old pretraining corpus \(D_{0}\), the performance on KILT measures how continual pretraining on new corpus \(D_{1}\) affects the performance on the knowledge obtained from \(D_{0}\) if finetuning is done on behalf of the knowledge from \(D_{0}\).