Experiments on four diverse tasks (POS Tagging/NER/SST-2/MRPC) verify that DRW detects extracted models with 100% mean average precision, yet with only a small drop (<5%) in original prediction performance.
Related Work
Model Extraction Attacks 
Model extraction attacks target the confidentiality of ML models and aim to imitate the function of a black-box victim model [1]}, [2]}, [3]}. First, adversaries collect or synthesize an initially unlabeled substitute dataset. Next, they exploit the ability to query the victim model APIs for label predictions to annotate the substitute dataset. Then, they can train a high-performance model utilizing the pseudo-labeled dataset. Recently, several works [4]}, [5]}, [6]} attempt to address the model extraction attacks on NLP models, e.g. BERT [7]} or Google Translate.
Knowledge Distillation 
Model extraction attacks are closely related to knowledge distillation (KD) [8]}, where the adversary acts as the student who approximates the behaviors of the teacher (victim) model. The student can learn from soft labels or hard labels. KD with soft labels has been widely applied due to the fact that soft labels can carry a lot of useful information [9]}, [10]}.
Watermarking 
A digital watermark is an undetected label embedded in a noise-tolerant signal, such as audio, video, or image data. It is designed to identify the owner of the signal's copyright. Some works [11]}, [12]}, [13]}, [14]} employ watermarks to prevent precise duplication of machine learning models. They insert watermarks into the parameters of the protected model or construct backdoor images that activate particular predictions. If an adversary exactly copies a protected model, a watermark can be used to verify ownership. However, safeguarding models from model extraction attacks is more difficult due to the fact that the parameters of the suspect model might be vastly different from those of the victim model, and the backdoor behavior may not be transferred to the suspect model either. Several works [15]}, [16]}, [17]}, [18]}, [19]} study how to identify extracted models that are distilled from the victim model. [17]} forces the protected model to acquire features for identifying data samples taken from authentic and watermarked data. [19]} conducts lexical modification as a watermarking method to protect language generation APIs. CosWM [18]} incorporates a watermark as a cosine signal into the output of the protected model. Since the cosine signal is difficult to eliminate, extracted models trained via distillation will continue to have a significant watermark signal. Nonetheless, CosWM only applies to image data and soft distillation. We design multiple new techniques to extend CosWM in handling the text data with discrete sequence and we provide a theoretical guarantee on the protected API accuracy for soft and hard distillations

Proposed Method: DRW
<FIGURE>Overview
Figure REF  presents an overview of distillation procedure, watermarking and detection. The main idea of DRW is to introduce a perturbation to the output of a protected model. This designed perturbation is transferred onto a suspect model distilled from a victim model that remains identifiable by probing the suspect model.
Problem Formulation  We consider a common real-world scenario that the adversary only has black-box access to the victim model's API \(\mathcal {V}\) . There exist two types of output from victim model API: soft (real-valued) labels (i.e. probabilities) and hard labels. The adversary employs an auxiliary unlabeled dataset to query \(\mathcal {V}\) . Once the adversary gains the predictions from the victim model, it can train a separate model \(\mathcal {S}\)  from scratch with the pseudo-labeled dataset. The adversary may either distill the victim model with hard labels by minimizing the cross-entropy loss
\(\mathcal {L}_{\text{CE}}=-\sum _{i=1}^{m} \hat{\mathbf {y}}_{i} \log \left(\hat{\mathbf {q}}_{i}\right),\) 
where \(\hat{\mathbf {q}}_i\)  is the prediction from the stealer's model and \(\hat{\mathbf {y}}\)  are the pseudo-labels from the victim model; or distill from soft labels by minimizing the Kullback–Leibler (KL) divergence loss
\(\mathcal {L}_{\text{KL}} = \sum _{i=1}^{m} \hat{\mathbf {y}}_{i} \log \left(\frac{\hat{\mathbf {y}}_{i}}{\hat{\mathbf {q}}_{i}}\right).\) 

Watermarking the Victim Models
DRW dynamically embeds a watermark in response to queries made by an API's end-user. We use a set of variables to represent key \(K = (c^*, f_w, \mathbf {v}_k, \mathbf {v}_s, \mathbf {M})\) , where \(c^*\in \lbrace 1, \dots , m\rbrace \)  is the target class to embed watermark; \(f_w \in \mathbb {R}\)  is the angular frequency; \(\mathbf {v}_k \in \mathbb {R}^n\)  is the phase vector; \( \mathbf {v}_s \in \mathbb {R}^n\)  is the selection vector; \(\mathbf {M}\in \mathbb {R}^{|D|\times n}\)  is the random token matrix. \(|D|\)  represents the vocabulary size, so that every token ID corresponds to vector \(\mathbf {M}_i \in \mathbb {R}^n\) . Following [18]}, we define a periodic signal function based on \(K\)  and the input \(x\) .
\(\mathbf {z}_{c}(x)=\left\lbrace \begin{array}{lr}\cos \left(f_{w} g(\mathbf {v}_k, x, \mathbf {M})\right), & c=c^{*} \\\cos \left(f_{w} g(\mathbf {v}_k, x, \mathbf {M})+\pi \right), & c\ne c^{*}\end{array}\right.\) 
for \(c \in \lbrace 1, \dots , m\rbrace \) , where \(g(\cdot )\in [0, 1)\)  is a hash function projecting a text representation to a scalar. Ideally, the scalar should uniformly distribute spanning multiple cycles.
Constructing the hash function
We project every input \(x\)  into the fixed scalar range to add the sinusoidal perturbation by the hash function \(g(\cdot )\) . We randomly generate the phase vector \(\mathbf {v}_k\) , selection vector \(\mathbf {v}_s\)  and the token matrix \(\mathbf {M}\) . Each element in \(\lbrace \mathbf {v}_k, \mathbf {v}_s\rbrace \)  is randomly sampled from a uniform distribution over \([0, 1)\) . Each element of the matrix \(\mathbf {M}\)  is randomly sampled from a standard normal distribution \(\mathbf {M}_{ij} \sim \mathcal {N}(0, 1)\) . Let \(\mathbf {M}_{i} \in \mathbb {R}^n\)  denote the \(i\) -th row of matrix \(\mathbf {M}\) , \(\mathbf {v}_k^\top \mathbf {M}_{i} \sim \mathcal {N}(0, \frac{n}{3})\)  and \(\mathbf {v}_s^\top \mathbf {M}_{i} \sim \mathcal {N}(0, \frac{n}{3})\)  (we prove it in Appendix REF ). Then we apply probability integral transformation to obtain the uniform distribution of the hash values, where \(g(\mathbf {v}_k, x, \mathbf {M})\sim \mathcal {U}(0, 1)\)  and \(g(\mathbf {v}_s, x, \mathbf {M})\sim \mathcal {U}(0, 1)\) . We set \(g(\mathbf {v}_s, x, \mathbf {M}) \le \tau \)  to select part of all samples, where \(\tau \)  is the data selection ratio. When implementing sequence labeling tasks, we use the token ID to fetch the vector in matrix \(\mathbf {M}\) . Similarly, when implementing sentence classification tasks, we use the ID of the second token in the sentence to obtain the vector.
Next we compute the periodic signal for the victim output
\(\small \hat{\mathbf {y}}_c =\left\lbrace \begin{array}{lr}\hat{\mathbf {p}}_{c}, & g(\mathbf {v}_s, x , \mathbf {M}) > \tau \\\frac{\hat{\mathbf {p}}_{c}+\varepsilon \left(1+\mathbf {z}_{c}(x)\right)}{1+2 \varepsilon }, & c=c^{*} \text{~and~} g(\mathbf {v}_s, x, \mathbf {M}) \le \tau \\ \frac{\hat{\mathbf {p}}_{c}+\frac{\varepsilon \left(1+\mathbf {z}_{c}(x)\right)}{m-1}}{1+2 \varepsilon }, & c\ne c^{*} \text{~and~} g(\mathbf {v}_s, x, \mathbf {M}) \le \tau \end{array}\right.\) 
where \(\varepsilon \)  is the watermark level for the periodic signal and \(\hat{\mathbf {p}}_{c}\)  is the victim model's prediction before watermarking. Since \(0 \le \hat{\mathbf {y}}_i\le 1\)  and \(\sum _{i=1}^{m} \hat{\mathbf {y}}_i = 1\)  (see proof in Appendix REF ), \(\hat{\mathbf {y}}\)  is a surrogate for softmax output.
In the soft label setting, the victim model generates output \(\hat{\mathbf {y}}\)  directly; while in the hard label setting, the victim model produces the sampling hard label, i.e. a one-hot label with probability \(\hat{\mathbf {y}}_i\)  for each class \(i\) . Intuitively, the hard-label sampled output retains the watermark because it is equal to \(\hat{\mathbf {y}}\)  in expectation. Further, we define the accuracy for soft label output, named “argmax soft”, which calculates the accuracy of the argmax of soft output compared with the true label. Similarly, we define “sampling hard” to describe the output of the victim model which is a one-hot vector.

Detecting Watermark from Suspect Models
We first create a probing dataset \(\mathcal {D}_p\) , for which the labels are not required. \(\mathcal {D}_p\)  can be drawn from the training data of the extracted model since the owner is able to store any query sent by a specific end-user. In our setting, we also allow \(\mathcal {D}_p\)  to be drawn from other distributions.
We employ the Lomb-Scargle periodogram method [24]} for detecting and characterizing periodic signals. The Lomb–Scargle periodogram yields an estimate of the Fourier power spectrum \(P(f)\)  at frequency \(f\)  in an unevenly sampled dataset. After getting the power spectrum, we evaluate the signal strength by calculating the signal-to-noise ratio
\(P_{\text{signal }}=\frac{1}{\delta } \int _{f_{w}-\frac{\delta }{2}}^{f_{w}+\frac{\delta }{2}} P(f) df \\\text{\footnotesize $P_{\text{noise }}=\frac{1}{F-\delta }\left[\int _{0}^{f_{w}-\frac{\delta }{2}} P(f) d f+\int _{f_{w}+\frac{\delta }{2}}^{F} P(f) d f\right]$} \\P_{\text{snr}}=P_{\text{signal }} / P_{\text{noise }},\) 
where \(\delta \)  controls the window width of \(\left[f_{w}-\frac{\delta }{2}, f_{w}+\frac{\delta }{2}\right]\) ; \(F\)  is the maximum frequency, and \(f_{w}\)  is the angular frequency embedded into the victim model.
A higher signal-to-noise ratio \(P_{\text{snr}}\)  indicates a higher peak in the frequency domain.

Theoretical Analysis
In this section, we provide theoretical guarantees for DRW for both argmax soft output and sampling hard output. The analysis assumes the victim is calibrated so its soft-predictions are informative. We also focus on the binary classification task, i.e., \(m=2\) . Generalization to \(m>2\)  is straightforward and omitted only to ensure a clean presentation.
Theorem 1 
Without loss of generality, set target class \(c^* = 1\) , so that \(\hat{p} = \hat{\mathbf {p}}_1(x), \hat{y} = \hat{\mathbf {y}}_1, z(x) = \mathbf {z}_1(x)\) .
Assume \(\hat{p}(x)\)  is calibrated, i.e., \(\mathbb {E}[y|\hat{p}(x)=a]=a, ~\forall 0\le a\le 1\) , the argmax soft label of the victim model is \(\hat{y}_s = \mathbb {1}\lbrace \frac{\hat{p}(x)+\varepsilon (1+z(x))}{1+2\varepsilon } > 0.5\rbrace \)  and the sampling hard label of the victim output is \(\hat{y}_h \sim \operatorname{Ber}(\frac{\hat{p}(x)+\varepsilon (1+z(x))}{1+2\varepsilon })\) . For a fixed \(\mathbf {v}_k\) , given that
\(z(x) = \cos \left(f_{w} g(\mathbf {v}_k, x, \mathbf {M})\right)\in [-1, 1]\)  and the data selection ratio is set to \(\tau \) , then
DRW argmax soft label and sampling hard label satisfy:
\(\mathbb {E}_{\mathbf {v}_k}&\left[ \text{Acc}(\text{Argmax Soft}) \right] \ge \text{Acc}(\text{Victim})\\& - \tau (0.5+\varepsilon )\mathbb {P}[0.5-\varepsilon \le \hat{p}\le 0.5+\varepsilon ],\\\mathbb {E}_{\mathbf {v}_k}&\left[\text{Acc}(\text{Sampling Hard})\right] \ge (1-\tau )\text{Acc}(\text{Victim})\\& + \frac{\tau }{1+2\varepsilon } \mathbb {E}\left[2\hat{p}^2 -2\hat{p} + 1\right].\) 

The proof is deferred to Appendix REF .
Equation (REF ) says that, in the soft label setting, DRW does not hurt the accuracy too much if the watermark level \(\varepsilon \)  is small. Note that only samples in which the victim model output lies around 0.5 (\(\pm \varepsilon \) ) might be affected by the watermarking. These are data points where the victim model is uncertain and inaccurate anyway.
Equation () lowerbounds the accuracy of the sampled hard labels, which is close to the vanilla victim model if \(\tau \)  is small. Observe that if \(\tau = 1\) , the accuracy may drop even if the watermark magnitude \(\varepsilon =0\)  due to the sampling of the output labelUnder the calibration assumption,
\(Acc(Victim)= \mathbb {E}\left[\hat{p}\mathbf {1}(\hat{p}\ge 0.5) + (1-\hat{p})\mathbf {1}(\hat{p}<0.5)\right]\) , which is strictly bigger than \(\mathbb {E}\left[2\hat{p}^2 -2\hat{p} + 1\right]\)  except when \(\hat{p}\)  is supported only at trivial points \(\lbrace 0,1,0.5\rbrace \). Our design of a second random projection \(\mathbf {v}_s\)  plays an important role here as it allows us to control the accuracy drop to any level we desire by adjusting \(\tau \) .

Experiments
<TABLE><TABLE>Tasks
We evaluate the performance of DRW on four different tasks. Two are sequence labeling tasks, Part-Of-Speech (POS) Tagging and Named Entity Recognition (NER); the other two are from GLUE [25]} text classification tasks, SST-2 and MRPC. We choose BERT [7]} as our model backbone and fine-tune it in different tasks.
Sequence labeling
We utilize the CoNLL-2003 dataset [27]} for POS Tagging and NER tasks. The CoNLL-2003 dataset consists of news articles from the Reuters
RCV1 corpus with POS and NER tags. We formulate POS Tagging and NER as token-level classification tasks following standard practice. Specifically, POS Tagging has 47 classes and NER has 9 classes. We take the token embedding of the last hidden layer of BERT [7]} as the input to a linear layer, which is then used as the classifier over the POS/NER label set. The token ID is set as the input \(x\)  for the hash function \(g(\cdot )\) . F1 score is hired for the evaluation metric.

Text classification
SST-2 is a binary single-sentence classification task consisting of movie reviews with corresponding sentiment [29]}. MRPC is a collection of sentence pairs from online news with labels suggesting whether the pair is semantically equivalent or not [30]}. We use the final hidden vector of the special [CLS] token of BERT as the input to a linear layer, which serves as the sentence classifier. The ID of the second token in the sentence is set as the input \(x\)  for the hash function \(g(\cdot )\) . Since GLUE does not include any test dataset, we use accuracy of the validation set as the evaluation metric.
For each task, we train the protected model to achieve the best performance on the validation set. As demonstrated in Table REF , the victim model has comparable performance to BERT [7]}. For soft and hard label distillation, we split the training data in each task into two parts and use the first half to query the victim model. Then the extracted model is trained for 20 epochs on the pseudo-labeled dataset. We choose the same key \(K = (c^*, f_w, \mathbf {v}_k, \mathbf {v}_s, \mathbf {M})\) , where frequency \(f_w = 16.0\) , watermark level \(\varepsilon = 0.2\)  and \(\lbrace \mathbf {v}_k, \mathbf {v}_s, \mathbf {M}\rbrace \)  are generated with different random seed. We set target class \(c^*=22\)  (“NNP” tag) for POS Tagging, \(c^*=2\)  (“I-PER” tag) for NER and \(c^*=0\)  (“negative” class) for SST-2/MRPC. We set data selection ratio \(\tau = 0.5\)  to add watermarks to half of the output data. More details for the experiment setting can be found in Appendix REF .
<FIGURE>
Baseline
We take the state-of-the-art method DeepJudge [32]} as a baseline against DRW. DeepJudge quantitatively tests the similarities between the victim model and suspect model, then determines whether the suspect model is a copy based on the testing metrics. Since DeepJudge is designed for continuous signals such as images and audio, we modify the method to apply it to texts. We consider the black-box setting for DeepJudge, and compute Jensen-Shanon Distance (\(JSD\) ) [33]} for the probing dataset of the victim model and the extracted model. \(JSD\)  measures the similarity of two probability distributions. We use the probing dataset to query both the victim model and the suspect model, and then calculate \(JSD\)  of the output layer as follows
\(\text{\small $JSD\left(\hat{\mathbf {y}}, \hat{\mathbf {q}}\right)=\frac{1}{2|\mathcal {D}_p|} \sum _{x \in \mathcal {D}_p} K\left(\hat{\mathbf {y}}(x), u\right)+K\left(\hat{\mathbf {q}}(x), u\right)$}\) 
where \(u=\left(\hat{\mathbf {y}}(x)+\hat{\mathbf {p}}(x)\right) / 2\)  and \(K(\cdot , \cdot )\)  is the Kullback-Leibler divergence. A small \(JSD\)  value implies similar output distribution of the two models, which further indicates that the suspect model may be distilled from the victim model.

Evaluation
We evaluate the performance of the victim model and the extracted model with accuracy/F1 score. In order to compare DRW with DeepJudge in detecting extracted models, we can reduce this binary classification problem to thresholding a particular test score. Since DRW and DeepJudge use different scores to detect the extracted model, we set up a series of ranking tasks to show the effect of these scores. For each task, we train 10 extracted models from the watermarked victim model with different random initialization as positive samples, 10 extracted models from the unwatermarked victim model with different random initialization, and 10 models from scratch with true labels as negative samples. For DRW, we use the watermark signal strength values \(P_{\text{snr}}\)  as the score for ranking (identifying whether it is an extracted model); for DeepJudge, we use \(JSD\)  as the score. Next, we compute the mean average precision (mAP) for the ranking tasks which assesses the model extraction detection performance. A higher mAP means the detecting method can distinguish the victim and the suspect model better.
We show the experiment results in the following subsections.

Effectiveness: Is DRW able to identify model infringements?
We evaluate our method in two settings, distillation with soft labels and distillation with hard labels. The results are displayed in Table REF . DeepJudge performs well on SST-2 and MRPC tasks but it can not effectively detect the extracted models in POS Tagging and NER tasks. In contrast, our method can successfully detect the extraction with 100% mAP across all tasks in both settings. We also present the range of \(JSD\)  and \(P_{\text{snr}}\)  in Table REF . Regarding the performance of DeepJudge on POS Tagging and NER tasks, the \(JSD\)  intervals for positive and negative samples overlap each other, resulting in the aforementioned lower mAP compared to DRW. A case in point is DeepJudge-\(JSD\) -Hard for NER task, where the ranges for negative suspect score and positive suspect score are \([0.009, 0.274]\)  and \([0.011, 0.052]\)  respectively. The overlapping intervals lead to the imperfect detection result, i.e., mAP \(=0.40\) . Whereas, DRW is able to perfectly distinguish between positive and negative suspects. Typically, \(P_{\text{snr}}\)  for the negative suspect is smaller than 5 while that for the positive suspect is larger than 15.

Fidelity: Does DRW decrease the performance of the model?
The results for the model performance at the watermark level \(\varepsilon =0.2\)  are displayed in Table REF . The perturbed API (victim model with argmax soft/sampling hard) only has a slight performance drop (within 5%) in comparison to the original one due to the trade-off between detection effectiveness and model performance. For the victim model API, argmax soft exhibits less performance drop than the sampling hard, since the argmax of the soft label remains unchanged with small perturbation. For the extracted model, distillation with soft label tends to have a better accuracy/F1 score than that with hard label. Additionally, the performances of extracted models are very close to those of victim models, a clear manifestation of the distillation success.

Case Study
We present how our method works on some examples in NER task. We fix the victim model and choose different settings for the suspect model across all the examples. For the watermarked ones, we set \(f_w = 16, \varepsilon = 0.2 \text{~and~} c^* = 2\) .
In Figure REF  (a), we show how DRW works on a suspect model that does not extract the victim model. We select a model trained from scratch with true labels as a negative example. There is no sinusoidal signal in the output of the suspect model hence a small \(P_\text{snr}\) .
In Figure REF  (b)(c), we illustrate the effect on soft distillation and hard distillation. We use the watermark key \(K\)  to extract the output of the victim model and suspect model. The extracted model clearly follows the victim model and there is a prominent peak at frequency \(f_w\) . Note that suspect model distillation with soft labels has a higher \(P_{\text{snr}}\)  than the one with hard labels. This is because the training process of extracted models can be more effective and faster with soft labels [9]}.
In Figure REF  (d), we validate the secrecy of our method. If the adversary does not have the secret key, it can not justify what the watermark is or whether there exist watermarks. The output of the victim model and extracted model are almost indiscernible when we use a wrong key to project them given the hash function \(g(\cdot )\) .
In Figure REF  (e)(f), we demonstrate the generality of our method. Watermarking algorithm should be independent of the dataset and the ML algorithms. In sub-figure (e), a different dataset is used to probe the suspect model. To be specific, we select the second half of the training data as the probing dataset, rather than the first half used in previous experiments. The results imply that DRW turns out to work well when we use unseen data to produce the probing dataset for the suspect model. In sub-figure (f), we choose a different backbone RoBERTa [35]} for the suspect model, in which the victim model continues to be the BERT model. The high peak in the power spectrum at frequency \(f_w\)  reveals that DRW is still able to detect the signal.

Ablation Study
Does watermark level impact detection?
<FIGURE>An important aspect of watermarking is how much perturbation we add to the output of the victim model. Theoretically, a smaller watermark level is associated with a higher accuracy/F1 score of the victim, yet it makes it harder to extract the signal from the probing results. We conduct two experiments to investigate the effect of the watermark level.
In the first experiment, we vary the watermark level in SST-2 task. According to the Theorem REF , the accuracy of the victim model output is bounded and a higher watermark level causes poorer performance. As shown in Figure REF , when the watermark level rises from 0 to 0.7, the performance drops by around 10 percent. It is worth noting that a big drop of the argmax soft emerges as \(\varepsilon \)  passes 0.5, which means the argmax of the output is highly likely to be changed in this case.
<FIGURE>In the second experiment, we design 10 sets of ranking tasks, and build up 10 positive samples together with 20 negative samples (similar to the setting in Section REF ) for each set in NER task. The watermark level is the only varied parameter across different tasks, ranging from 0.02 to 0.2. We plot the mAP of the detection against the watermark level in Figure REF . When the watermark level \(\varepsilon \)  is below 0.12, DRW can not generate perfect detection of positive and negative suspects, indicating that the adversary may not convey a strong sinusoidal signal at a low watermark level. In this case, DRW can not extract the watermark in frequency space and thus fails to detect it successfully.
These two experiments demonstrate the trade-off between the detection effectiveness and the victim model's performance after watermarking.

Do categories affect watermark protection?
<FIGURE>We vary the target class \(c^*\)  of the watermark key \(K\)  in POS Tagging task. We add watermarks to four different categories and then train the extracted model by soft distillation and hard distillation. The results of the signal-to-noise ratio \(P_\text{snr}\)  are visualized in Figure REF . The effect of the watermark will be more salient if the category involves more samples. Since "NNP" covers the most (14.16%) of all tokens, adding watermark to "NNP" produces the strongest signal. In contrast, the determiners ("DT") category only has a few number of types, such as "the" and "a". As a result, adding watermark to "DT" is ineffective as it is hard to add a periodic signal to a very discrete domain.

How much should be selected for watermarking?
<FIGURE>A critical design of our method is that we apply selection vector \(\mathbf {v}_s\)  to select a portion of the victim model output to be watermarked. We change the ratio of the watermarked data by tuning the data selection ratio \(\tau \)  in MRPC task. The results shown in Figure REF  indicate that the accuracy of the victim model output falls with a higher data selection ratio, yet it introduces a greater signal strength of the extracted model. This trade-off is similar to the one described in Section REF . 0.5 could be a reasonable selection ratio.

Conclusion
In this work, we propose Distillation-Resistant Watermarking (DRW), a novel and unified watermarking technique against model extraction attacks on NLP models. By injecting watermarks into the prediction output of the victim model, the model owner can detect the watermark if the adversary distills the protected model. We prove the theoretical guarantee of DRW and show remarkable empirical results on text classification and sequence labeling tasks.

Limitations
1) The watermark detection does not work well when the watermarked data covers only a small amount of the whole training data for the extracted model.
2) Our method may not work well when the adversary only makes a few queries to the victim model APIs and trains the extracted model with few-shot learning.
3) If the victim model outputs soft labels, even with watermarking, the adversary can take argmax operation to erase the watermark. So it is better to combine watermarks with hard label output in real-world applications.

Broader Impact
This work will alleviate ethical concerns of commercial NLP models.
This paper provides one promising solution to an important aspect of NLP: how to protect the intellectual property of trained NLP models. Companies with NLP web services can apply our method to protect their models from model extraction attacks.

Acknowledgements
XZ was supported by UCSB Chancellor’s Fellowship. The authors would like to thank Yang Gao for polishing up the draft and Dan Qiao for the helpful discussion.

Appendix
Proof for the Theorem 1
Theorem A.1 (Restate Theorem REF ) 
Without loss of generality, set target class \(c^* = 1\) , so that \(\hat{p} = \hat{\mathbf {p}}_1(x), \hat{y} = \hat{\mathbf {y}}_1, z(x) = \mathbf {z}_1(x)\) .
Assume \(\hat{p}(x)\)  is calibrated, i.e., \(\mathbb {E}[y|\hat{p}(x)=a]=a, ~\forall 0\le a\le 1\) , the argmax soft label of the victim model is \(\hat{y}_s = \mathbb {1}\lbrace \frac{\hat{p}(x)+\varepsilon (1+z(x))}{1+2\varepsilon } > 0.5\rbrace \)  and the sampling hard label of the victim output is \(\hat{y}_h \sim \operatorname{Ber}(\frac{\hat{p}(x)+\varepsilon (1+z(x))}{1+2\varepsilon })\) . For a fixed \(\mathbf {v}_k\) , given that
\(z(x) = \cos \left(f_{w} g(\mathbf {v}_k, x, \mathbf {M})\right)\in [-1, 1]\)  and the data selection ratio is set to \(\tau \) , then
DRW argmax soft label and sampling hard label satisfy:
\(\mathbb {E}_{\mathbf {v}_k}&\left[ \text{Acc}(\text{Argmax Soft}) \right] \ge \text{Acc}(\text{Victim})\\& - \tau (0.5+\varepsilon )\mathbb {P}[0.5-\varepsilon \le \hat{p}\le 0.5+\varepsilon ],\\\mathbb {E}_{\mathbf {v}_k}&\left[\text{Acc}(\text{Sampling Hard})\right] \ge (1-\tau )\text{Acc}(\text{Victim})\\& + \frac{\tau }{1+2\varepsilon } \mathbb {E}\left[2\hat{p}^2 -2\hat{p} + 1\right].\) 

We first prove the argmax soft label case with \(\tau =1\) .
\(&\mathbb {E}\left[\mathbb {1}(\hat{y}_s=y)\right] \\=& \mathbb {E}\left[\mathbb {P}(\hat{y}_s=y|x)\right]\\=& \mathbb {E}\left[\mathbb {P}(\hat{y}_s=1, y=1|x)+\mathbb {P}(\hat{y}_s=0, y=0|x)\right]\\=& \mathbb {E}\left[\mathbb {P}(\hat{y}_s=1|x)\mathbb {P}(y=1|x)\right.\\&+\left.\mathbb {P}(\hat{y}_s=0|x)\mathbb {P}(y=0|x)\right]\\=& \mathbb {E}\left[\mathbb {1}\lbrace \hat{p}+\varepsilon z(x)> 0.5\rbrace  \mathbb {P}(y=1|x)\right.\\&+\left.\mathbb {1}\lbrace \hat{p}+\varepsilon z(x)\le 0.5\rbrace \left(1-\mathbb {P}(y=1|x)\right)\right]\\=& \mathbb {E}\big [\mathbb {E}\left[\mathbb {1}\lbrace \hat{p}+\varepsilon z(x)> 0.5\rbrace  \mathbb {P}(y=1|x)\right.\\&+\left.\mathbb {1}\lbrace \hat{p}+\varepsilon z(x)\le 0.5\rbrace \left(1-\mathbb {P}(y=1|x)\right)|\hat{p}\right]\big ]\\\ge & \mathbb {E}\big [\mathbb {E}\left[\mathbb {1}\lbrace \hat{p}-\varepsilon > 0.5\rbrace \mathbb {P}(y=1|x)|\hat{p}\right]\\&+ \mathbb {E}\left[\mathbb {1}\lbrace \hat{p}+\varepsilon \le 0.5\rbrace  (1-\mathbb {P}(y=1|x))|\hat{p}\right]\big ]\\=& \mathbb {E}\big [\mathbb {1}\lbrace \hat{p}-\varepsilon > 0.5\rbrace \mathbb {E}\left[\mathbb {P}(y=1|x)|\hat{p}\right]\\&+ \mathbb {1}\lbrace \hat{p}+\varepsilon \le 0.5\rbrace \mathbb {E}\left[1-\mathbb {P}(y=1|x)|\hat{p}\right]\big ]\\=& \mathbb {E}\big [\mathbb {1}\lbrace \hat{p} > 0.5+\varepsilon \rbrace \hat{p} + \mathbb {1}\lbrace \hat{p} \le 0.5-\varepsilon \rbrace (1-\hat{p})\big ]\\=& \underbrace{\mathbb {E}\left[\mathbb {1}\lbrace \hat{p}>0.5\rbrace \hat{p}+\mathbb {1}\lbrace \hat{p}\le 0.5\rbrace (1-\hat{p})\right]}_{\text{Accuracy of victim model without watermark}}\\&-\mathbb {E}\left[\mathbb {1}\lbrace 0.5<\hat{p}\le 0.5+\varepsilon \rbrace \hat{p}\right]\\&+\mathbb {E}\left[\mathbb {1}\lbrace 0.5-\varepsilon \le \hat{p}\le 0.5\rbrace (1-\hat{p})\right]\\\ge & \text{Acc}(\text{Victim Model}) \\&- (0.5+\varepsilon )\mathbb {P}(0.5-\varepsilon \le \hat{p}\le 0.5+\varepsilon )\) 

where the first "\(\ge \) " follows from \(|z(x)| \le 1\) ; the third "\(=\) " follows from the conditional independence of \(\hat{y}_s\)  and \(y\)  given \(x\) ; the seventh "\(=\) " follows from the calibration assumption, i.e. \(\mathbb {E}\left[\mathbb {P}(y=1|x)|\hat{p}(x)\right] = \hat{p}(x)\) .
Notice that over the distribution of \(\mathbf {v}_s\)  selects every unique \(x\)  with probability \(\tau \)  independently to everything else, by exchanging the order of expectation, it is easy to prove that the expected accuracy is a convex combination of the accuracy of the victim model (with weight \(1-\tau \) ) and the case above (with weight \(\tau \) ). This completes the proof for argmax soft label.
We then start by analyzing the sampling hard label case with \(\tau =1\) .
\(&\mathbb {E}\left[\mathbb {1}(\hat{y}=y)\right] \\=& \mathbb {E}\left[\mathbb {P}(\hat{y}=y|x)\right]\\=& \mathbb {E}\left[\mathbb {P}(\hat{y}=1, y=1|x)+\mathbb {P}(\hat{y}=0, y=0|x)\right]\\=& \mathbb {E}\left[\mathbb {E}(\hat{y}|x)\mathbb {E}(y|x)+\mathbb {E}(1-\hat{y}|x)\mathbb {E}(1-y|x)\right]\\=& \mathbb {E}\left[\left(\frac{\hat{p}}{1+2\varepsilon }+\frac{\varepsilon (1+z(x))}{1+2\varepsilon }\right)\mathbb {E}(y|x)\right. \\&+\left. \left(\frac{1-\hat{p}}{1+2\varepsilon }+\frac{\varepsilon (1-z(x))}{1+2\varepsilon }\right)\mathbb {E}(1-y|x)\right]\\=& \text{\scriptsize $\frac{1}{1+2\varepsilon }\underbrace{\mathbb {E}\left[\hat{p}\mathbb {E}(y|x)+(1-\hat{p})\mathbb {E}(1-y|x)\right]}_{A}$}\\&+ \text{\scriptsize ${\frac{\varepsilon }{1+2\varepsilon } \underbrace{\mathbb {E}\left[(1+z(x))\mathbb {E}(y|x)+(1-z(x))\mathbb {E}(1-y|x)\right]}_{B}}$}\) 
\(A =& \mathbb {E}\left[\mathbb {E}\left[\hat{p}\mathbb {E}(y|x)+(1-\hat{p})\mathbb {E}(1-y|x)|\hat{p}\right]\right]\\=& \mathbb {E}\left[\hat{p}\mathbb {E}(y|\hat{p})+(1-\hat{p})\mathbb {E}(1-y|\hat{p})\right]\\=& \mathbb {E}\left[\hat{p}^2+(1-\hat{p})^2\right]\\=& \mathbb {E}\left[2\hat{p}^2-2\hat{p}+1\right]\) 

where the third line follows from the calibration assumption, i.e., \(\mathbb {E}[y|\hat{p}(x)=a]=a\) .
\(B = &\mathbb {E}\left[\mathbb {E}(y|x)+\mathbb {E}(y|x)z(x)+1-z(x)\right.\\&-\mathbb {E}(y|x) +\left.\mathbb {E}(y|x)z(x)\right]\\= & 1+\mathbb {E}\left[\left(2\mathbb {E}(y|x)-1\right)z(x)\right]\\\ge & 0\) 

where the last line follows from the facts that \(|z(x)|\le 1\)  and \(|2\mathbb {E}(y|x)-1| \le 1\) .
Finally, notice that for each \(x\)  the probability to be chosen to add watermark and to sample the output is \(\tau \)  independently, thus the expected accuracy is the convex combination of the accuracy of the victim model and that of the fully watermarked model.

Distribution Property
Lemma 1 Assume \(\mathbf {v} \sim \mathcal {U}(0,1),~ \mathbf {v}\in \mathbb {R}^n\)  and \(\mathbf {x} \sim \mathcal {N}(0,1),~ \mathbf {x}\in \mathbb {R}^n\) , where \(\mathbf {v}\)  and \(\mathbf {x}\)  are both \(i.i.d.\)  and independent of each other. Then we have:
\(\frac{1}{\sqrt{n}}\mathbf {v}\cdot \mathbf {x} \rightsquigarrow \mathcal {N}\left(0, \frac{1}{3}\right),~ n\rightarrow \infty \) 

Let \(u_i=\mathbf {v}_i\mathbf {x}_i,~ i\in {1,2,\ldots ,n}\) . By assumption, \(u_i\)  are \(i.i.d.\) . Clearly, the first and second moments are bounded, so the claim follows from the classical central limit theorem,
\(\sqrt{n}\bar{u}_n &= \frac{\sum _{i=1}^n u_i}{\sqrt{n}} \rightsquigarrow \mathcal {N}\left(\mu , \sigma ^2\right)~\text{ as }~n\rightarrow \infty \) 

where
\(\mu &= \mathbb {E}\left(u_i\right) = \mathbb {E}\left(\mathbf {v}_i\mathbf {x}_i\right) = \mathbb {E}\left(\mathbf {v}_i\right) \mathbb {E}\left(\mathbf {x}_i\right) \\&= 0 \\\sigma ^2 &= \operatorname{Var}(u_i) = \mathbb {E}\left(u_i^2\right) - \left(\mathbb {E}\left(u_i\right)\right)^2\\&= \mathbb {E}\left(u_i^2\right) = \mathbb {E}\left(\mathbf {v}_i^2\mathbf {x}_i^2\right) = \mathbb {E}\left(\mathbf {v}_i^2\right)\mathbb {E}\left(\mathbf {x}_i^2\right)\\&= \frac{1}{3}\) 

It follows that given large \(n\)
\(\frac{1}{\sqrt{n}}\mathbf {v}\cdot \mathbf {x} \rightsquigarrow \mathcal {N}\left(0, \frac{1}{3}\right)\) 

Modified Softmax Properties
Lemma 2 (Lemma 1 in [18]}) 
Let \(\hat{\mathbf {p}}\)  be the softmax output of a model \(\mathcal {V}\) , then the modified softmax \(\hat{\mathbf {y}}\) , as defined in Equation REF  satisfies \(0 \le \hat{\mathbf {y}}_i\le 1\)  and \(\sum _{i=1}^{m} \hat{\mathbf {y}}_i = 1\) .

Notice that in Equation REF , when \(g(\mathbf {v}_s, x, \mathbf {M}) > \tau \) , \(\hat{\mathbf {y}}\)  = \(\hat{\mathbf {p}}\) , so that it satisfies the property above.
By the definition of softmax, for all class \(c \in \lbrace 1, \dots , m\rbrace \)  we have
\(0 \le \hat{\mathbf {p}}_{c} \le 1,-1 \le \mathbf {z}_{c}(x) \le 1 .\) 
Therefore, when \(c=c^{*}\) , we have
\(0 \le \hat{\mathbf {p}}_{c}+\varepsilon \left(1+\mathbf {z}_{c}(x)\right) \le 1+2 \varepsilon ,\) 
and then
\(0 \le \frac{\hat{\mathbf {p}}_{c}+\varepsilon \left(1+\mathbf {z}_{c}(x)\right)}{1+2 \varepsilon } \le 1 .\) 
When \(c \ne c^{*}\) , since \(m \ge 2\) , we have
\(0 \le \hat{\mathbf {p}}_{c}+\frac{\varepsilon \left(1+\mathbf {z}_{c}(x)\right)}{m-1} \le 1+\frac{2 \varepsilon }{m-1} \le 1+2 \varepsilon \) 
and then
\(0 \le \frac{\hat{\mathbf {p}}_{c}+\frac{\varepsilon \left(1+\mathbf {z}_{c}(x)\right)}{m-1}}{1+2 \varepsilon } \le 1 .\) 
Thus, \(\hat{\mathbf {q}}\)  satisfies \(0 \le \hat{\mathbf {y}}_i\le 1\) .
To prove \(\sum _{i=1}^{m} \hat{\mathbf {y}}_i = 1\) , we use the fact that \(\mathbf {z}_{c^{*}}+\)  \(\mathbf {z}_{i \ne c^{*}}=0\)  and obtain
\(\sum _{i=1}^{c} \hat{\mathbf {y}}_i&=\frac{\hat{\mathbf {p}}_{c^{*}}+\varepsilon \left(1+\mathbf {z}_{c^{*}}\right)}{1+2 \varepsilon }+\sum _{i \ne c^{*}} \frac{\hat{\mathbf {p}}_{i}+\frac{\varepsilon \left(1+\mathbf {z}_{i}\right)}{m-1}}{1+2 \varepsilon } \\&=\sum _{i=1}^{m} \frac{\hat{\mathbf {p}}_{i}}{1+2 \varepsilon }+\sum _{i \ne c^{*}} \frac{\varepsilon \left(1+\mathbf {z}_{c^{*}}+1+\mathbf {z}_{i}\right)}{(m-1)(1+2 \varepsilon )} \\&=\frac{1}{1+2 \varepsilon }+\frac{2 \varepsilon }{1+2 \varepsilon } \\&=1\) 


Experiment Details
We provide more details for the experiments in this section.
We build our classification models upon bert-base-uncased from Hugging Facehttps://huggingface.co/. The model contains 110M parameters. We add a dropout layer before the last linear layer with a dropout rate of 0.5. We implement DRW in PyTorch 1.11.0 on a server with 4 NVIDIA TITAN-Xp GPUs. We set batch size to 8 for SST-2 and MRPC tasks, and 32 for POS Tagging and NER tasks.
We train the victim model using AdamW [37]} optimizer with learning rate 1e-5 and epsilon 1e-8. Each victim model is trained 40 epochs and the one with the best validation results is chosen.
Regarding the extracted model, we use half of the training data to query the victim model and obtain the labeled dataset. Then the extracted model is trained with Adam [38]} optimizer for 20 epochs with learning rate 5e-5. The average training time is 3 minutes for each epoch.
We show the results for RoBERTa model in Section REF . In this setting, we choose roberta-base from Hugging Face, which has 125M parameters.
