[leftmargin=*,nosep]
Auxiliary dataset. We use \(\sim 1.8\times 10^5\)  samples from the train split of Wikitext as the auxiliary dataset, and \(1,000\)  samples from the test split for evaluating the attack. In addition, we introduce a weaker variant of our attack that only assumes knowledge of the marginal token distribution for the language model training data. Instead of using the WikiText train split as auxiliary data, we sample random tokens according to the marginal token distribution to generate pseudo-data for training the inversion model. We show that this variant, which we denote LTI-P, can even outperform LTI with in-distribution auxiliary data due to access to infinite training data.

Inversion model architecture.
We train a two-layer MLP with ReLU activation and first hidden-layer size 600 and second hidden-layer size \(1,000\) . The inversion model outputs \(L\)  probability vectors each with size equal to the vocabulary size (\(\sim 50,000\) ), and we train it using the cross-entropy loss to predict the \(L\)  tokens given the target model gradient.
We use feature hashing [1]} to reduce the target model gradient to \(10\%\)  of its original dimensions as input to the inversion model.

Training details. We use Adam [2]} to train the inversion model over 20 epochs with batch size 64. Learning rates are selected separately for each defense from \(\lbrace 10^{-3}, 10^{-4}, 10^{-5}\rbrace \) .

Computation cost. Our experiments are conducted using NVIDIA GeForce RTX 3090 GPUs and each training run takes about 3 hours.

