For evaluating LTI on a language task, we experiment with causal language model trainingWe follow the task setup and code in https://github.com/JonasGeiping/breaching for next-token prediction.
The language model \(f_{\mathbf {w}}\)  is a three-layer transformer [1]} with frozen token embedding layer. This is a common technique for language model fine-tuning [2]}, which also has privacy benefits since direct privacy leakage from the gradient magnitude of the token embedding layer can be prevented [3]}, [4]}. As a result, the trainable model contains about \(1.1\times 10^6\)  parameters.
We train the language model on WikiText [5]}, where each training sample is limited to \(L=16\)  tokens and the language model is trained to predict the next token \(\mathbf {x}_l\)  given \(\mathbf {x}_{:l-1}\)  for \(l=1,\ldots ,L\)  using the cross-entropy loss.
