In this paper we ask: _how much information can an adversary learn about a production language model by making queries to its API?_ This is the question studied by the field of _model stealing_(Tramer et al., 2016): the ability of an adversary to extract model weights by making queries its API.