We begin with a simple attack that allows an adversary to recover the size of the hidden dimension of a language model by making queries to the oracle \(\mathcal{O}\) (Algorithm 1). The techniques we use to perform this attack will be the foundation for attacks that we further develop to perform complete extraction of the final embedding projection matrix.