The common goal of language modelling is to predict the next word (distribution) given the context of the previous tokens [1]}. The performance is measured by perplexity: the inverse of the geometric average of the whole sequence probability (lower is better). The input to the model is the text prefix and the output of the following word. The prefix is usually the object that is passed to the encoder.
