where, \(\Theta\) represents the model parameters, \(\mathcal{D}_{\text{PT}}\) is the pre-training dataset, \(x_{i}\) is the token to be predicted, and \(x_{0},x_{1},\ldots,x_{i-1}\) constitute the context.