These models are usually composed of a neural network \(f_{\theta}\) that takes sequences of tokens \((y_{<i})\in[1,V]^{i-1}\) as inputs and produces a relatively low-dimensional contextual representation in \(\mathbb{R}^{d}\), where \(d\) is the _hidden dimension_ of the model. They then rely on a _language modeling head_ that produces logits for contextual token probabilities. A common choice for the language modeling head is a linear layer with parameter \(W\in\mathbb{R}^{V\times d}\), where \(V\) is the number of possible tokens.