Footnote 4: Despite a simple case, our findings on two domains have practical applications to continue pretraining (Gururangan et al., 2020), where we aim to enhance a pretrained model on a given domain by training it on a mixture of the original pretraining data and upcoming domain data. Please see Sec. 5 for details.