However, for the other datasets that cannot be performed in a zero-shot manner, the intermediate process of continually pretraining on corpus \(D_{1}\) does not seem to be that harmful on the finetuning for the target tasks even though they are more related to the knowledge of \(D_{0}\). Even T5-Vanilla shows modest performance, sometimes with better results than some other CKL baselines. One hypothesis is that the models could have regained the original knowledge from corpus \(D_{0}\) through the finetuning process. Also, some of the knowledge could have been recovered through the test-train overlap (Lewis et al., 2020; Wang et al., 2021).