In order to empirically measure the effect of the rank of the linear head, we propose to train rank-constrained heads on pretrained contextual representations from highly-parameterized language models. In order to control the maximum rank \(r\), we consider heads of the form \(W=AB\in\mathbb{R}^{V\times d}\), where the coefficients of \(A\in\mathbb{R}^{V\times r}\) and \(B\in\mathbb{R}^{r\times d}\) are drawn from \(\mathcal{N}(0,1)\) (\(d\) being the hidden dimension of the model). The rank of such \(W\) matrices is limited by the parameter \(r\in[1,d]\), which we sweep over a wide range of values.