In Sections C.3-C.6 we can no longer use general invertible matrices \(\mathbf{S}\) in our arguments, and must instead use orthogonal matrices, matrices \(\mathbf{U}\) such that \(\mathbf{U}^{\top}\mathbf{U}=I\). In models with LayerNorm, we specialise further, too (Appendix C.5).