For each task, we train the protected model to achieve the best performance on the validation set. As demonstrated in Table REF , the victim model has comparable performance to BERT [1]}. For soft and hard label distillation, we split the training data in each task into two parts and use the first half to query the victim model. Then the extracted model is trained for 20 epochs on the pseudo-labeled dataset. We choose the same key \(K = (c^*, f_w, \mathbf {v}_k, \mathbf {v}_s, \mathbf {M})\) , where frequency \(f_w = 16.0\) , watermark level \(\varepsilon = 0.2\)  and \(\lbrace \mathbf {v}_k, \mathbf {v}_s, \mathbf {M}\rbrace \)  are generated with different random seed. We set target class \(c^*=22\)  (“NNP” tag) for POS Tagging, \(c^*=2\)  (“I-PER” tag) for NER and \(c^*=0\)  (“negative” class) for SST-2/MRPC. We set data selection ratio \(\tau = 0.5\)  to add watermarks to half of the output data. More details for the experiment setting can be found in Appendix REF .
<FIGURE>