In practice, a neural language model \(\phi_{\theta}\) produces contextual representations \(x_{i}=\phi_{\theta}(y_{<i})\) of dimension \(d\in\mathbb{N}^{*}\). The linear language modeling head \(W_{\theta}\in\mathcal{R}^{V\times d}\) is trained concurrently with \(\phi_{\theta}\) with the same objective as in Equation 1.