Language models give the probability of the next token conditioned on the tokens that came before it, i.e. \(\mathbf{v}=p(x_{T+1}\mid x_{1},...,x_{T};\theta)\), where \(\mathbf{v}\in\Delta^{|\mathcal{V}|-1}\) gives the probability of each of \(|\mathcal{V}|\) possible next tokens. Generally these models have relatively large vocabulary sizes; the vocabulary \(\mathcal{V}\) may contain tens or hundreds of thousands of elements.