**Experiments.** In order to visualize the intuition behind this attack, Figure 1 illustrates an attack against the Pythia-1.4b LLM. Here, we plot the magnitude of the singular values of \(\mathbf{Q}\) as we send an increasing number \(n\) of queries to the model.