Intuitively, the proof in Appendix C.2 relied on pre-multiplying the input projection weight of layers by a matrix \(\mathbf{S}^{-1}\), so that this cancelled the rotation \(\mathbf{S}\) applied to the model's hidden state (called the'residual stream' in mechanistic interpretability literature (Elhage et al., 2021)). Formally, if we let the input projection layer be \(\mathbf{M}\), we were using the fact that \(\left(\mathbf{MS}^{-1}\right)\left(\mathbf{S}x\right)=\mathbf{M}x\).