Almost all LLMs that have publicly available architecture details use LayerNorm (Ba et al., 2016) or RMSNorm (Zhang and Sennrich, 2019) just before applying the output projection \(\mathbf{W}\)(Biderman, 2024). LayerNorm begins with a centering step, which projects its input onto a \((h-1)\)-dimensional subspace (and RMSNorm does not). In theory, this could break our assumption that the rank of the matrix with columns \(g_{\theta}\left(p_{i}\right)\) (\(i=1,...,n\)) has rank \(h\) (Lemma 4.1). In practice, all LLMs we surveyed (Biderman, 2024) enabled the LayerNorm bias, which means the matrices had full rank \(h\) (besides GPT-2 Small: see Appendix A).