Parameterization.We consider one alternative model architecture, an encoder-decoder with projection as in Morris et al. (2023). This model performs quite poorly, indicating that projecting the probability vector down to a smaller rank discards significant information. We also test an identical parameterization that conditions on the raw outputs of the language model instead of log-normalized probabilities to determine if un-normalized outputs contain more usable information than the probabilities. This removal also makes a difference: without applying the softmax to the inputs, we observe over a \(20\%\) drop in BLEU.