**Intuition.** Suppose we query a language model on a large number of different random prefixes. Even though each output logit vector is an \(l\)-dimensional vector, they all actually lie in a \(h\)-dimensional subspace because the embedding projection layer up-projects from \(h\)-dimensions. Therefore, by querying the model "enough" (more than \(h\) times) we will eventually observe new queries are linearly dependent of past queries. We can then compute the dimensionality of this subspace (e.g., with SVD) and report this as the hidden dimensionality of the model.