Large-scale pre-trained neural models have shown great success in NLP tasks [1]}, [2]}.
Task-specific NLP models are often deployed as web services with pay-per-query APIs in business applications.
Protecting the intellectual property of these cloud deployed models is a critical issue in both research and practice.
Service providers often use authentication mechanism to authorize valid accesses.
However, while this prevents clients directly copying a victim model, it does not hinder clients from stealing it using distillation.
Emerging model extraction attacks have demonstrated convincingly that most functions of the victim API are likely to be stolen with carefully designed queries [3]}, [4]}, [5]}, [6]}.
A model extraction process is often imperceptible because it queries APIs in the same way as a normal user does [7]}.
In this paper, we study the problem of model protection for NLP against distillation stealing.
