Reducing input dimensionality.When stored on disk in 32-bit precision, 10 million probability vectors for a vocabulary of size of \(32,000\) take up 1.28 TB. Is it necessary to retain the full dimensionality of these input vectors? Surprisingly, Table 9 (Right) indicates that the majority of the probability vector is required to achieve good inversion performance. Even though the top 1000 predicted tokens contain \(98\%\) of the probability mass on average, training and evaluating with only the top \(1000\) tokens reduces performance by \(45\%\).