**Architectural changes.** Instead of modifying the API, we could instead make changes to the model. Our attack only works because the hidden dimension \(h\) is less than the output dimension \(l\). This suggests a natural architectural defense: split the final layer into two layers, one that goes from \(h\to t\) and then \(t\to l\) where \(t>l\) and a nonlinearity was placed in between. This is not very efficient though, as the last linear layer is large (quadratic in the vocabulary size).