Language models generate responses by producing a series of tokens in immediate succession: thetoken is an outcome of manipulatinghidden vectors per layer, one vector per preceding token. What if instead we were to let the model manipulate say,hidden vectors, before it outputs thetoken? We operationalize this idea by performing
training and inference on language models with a (learnable) pausetoken, a sequence of which is appended to the input prefix. We then delay extracting the model’s outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluatepause-trainingon decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall. Our main finding is that inference-time delays show gains on our tasks when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on eight tasks, most prominently, a gain ofEM score on the QA task of SQuAD,on CommonSenseQA andaccuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.