We study models that take a sequence of tokens drawn from a vocabulary \(\mathcal{X}\) as input. Let \(\mathcal{P}\left(\mathcal{X}\right)\) denote the space of probability distributions over \(\mathcal{X}\). We study parameterized models \(f_{\theta}:\mathcal{X}^{N}\rightarrow\mathcal{P}\left(\mathcal{X}\right)\) that produce a probability distribution over the next output token, given an input sequence of \(N\) tokens.