**Inversion and Language Model Scale.** To understand how dependent inversion results are to language model size, we consider inverting different sizes of GPT-2 (Radford et al., 2019) and show results in Table 9 (Left). Interestingly, the reconstructions perform very similarly (within 1 point of BLEU score) regardless of the size of the language model inverted. The fact that output probabilities contain similar amounts of information even after going through vastly different amounts of processing (dependent on the varying model size) differ from the findings of Dosovitskiy and Brox (2016) who note that more layers of computation make inversion more difficult in CNNs.