We are motivated to study this problem not because we expect to be able to steal an entire production transformer model bit-for-bit, but because we hope to conclusively demonstrate that model stealing attacks are not just of academic concern but can be practically applied to the largest production models deployed today. We see a number of potential directions for improving on this attack.