where \(\theta_{CP}\) represents the model parameters, \(\mathcal{D}_{\mathcal{CP}}\) stands for the data used in continual pre-training, S represents the length of the input token sequence, and \(x_{i}\) represents the token to be predicted, while \(x_{0},x_{1},...,x_{i-1}\) make up the context.