For language tasks this optimization problem is considerably more complex since the samples \(\mathbf {x}_1,\ldots ,\mathbf {x}_B\)  are sequences of discrete tokens, and optimizing eq:optobjective amounts to solving a discrete optimization problem. To circumvent this difficulty, [1]} and [2]} instead optimize the token embeddings to match the observed gradient and then maps the recovered embeddings to their closest tokens in the embedding layer to recover the private text. In contrast, [3]} leveraged the insight that gradient of the token embedding layer can be used to recover exactly the set of tokens present in the training sample, and then uses beam search to optimize the ordering of tokens for fluency to recover the private text.
