Contributions.We introduce an attack that can be applied to black-box language models, and allows us to recover the complete _embedding projection layer_ of a transformer language model. Our attack departs from prior approaches that reconstruct a model in a _bottom-up_ fashion, starting from the input layer. Instead, our attack operates _top-down_ and directly extracts the model's last layer. Specifically, we exploit the fact that the final layer of a language model projects from the hidden dimension to a (higher dimensional) logit vector. This final layer is thus low-rank, and by making targeted queries to a model's API, we can extract its embedding dimension or its final weight matrix.