\[\mathcal{O}\left(p\right)\leftarrow\mathbf{W}\cdot g_{\theta}\left(p\right)\enspace.\]We develop new attack techniques that allow us to perform high-fidelity extraction of (a small part of) a transformer. Section 4.1 demonstrates how we can identify the hidden dimension \(h\) using the logits API and Section 4.2 presents an algorithm that can recover the matrix \(\mathbf{W}\).