New Text Corpus for Language ModelingFor LMs to renew their internal knowledge, they need to be continually pretrained on a new text corpus \(D_{1}\) which has the updated and new information. \(D_{1}\) should ideally be much smaller than \(D_{0}\), as a large \(D_{1}\) amounting to the size of \(D_{0}\) will result in massive computational costs similar to pretraining the LMs from scratch.