Little has been done to adapt watermarking to identify model infringements in language tasks.
Although a number of defense techniques have been proposed to prevent the model extraction for computer vision, they are not applicable to language tasks with discrete tokens.
Among them, deep neural networks (DNN) watermarking [1]}, [2]} works by embedding a secret watermark (e.g., logo or signature) into the model exploiting the over-parameterization property of DNNs.
This procedure leverages a trigger set to stamp invisible watermarks on their commercial models before distributing them to customers.
When suspicion of model theft arises, model owners can conduct an official ownership claim with the aid of the trigger set.
However, these protections all focus on the image/audio tasks, since it is easy to modify the continuous data.
In addition, most watermarking methods are invasive and fragile. They cannot avoid tampering with the training procedure in order to embed the watermark. Besides, the watermarks are outliers of the task distribution so that the adversary may not carry the watermark through distillation.
