More generally, if \(g_{\theta}\) is composed of residual layers but no normalization layers then \(g_{\theta}(p)=\sum_{i}L_{i}(p)\), where \(L_{i}(p)\) is the output of the \(i\)th residual layer in the model, ignoring the skip connection (Ehlage et al., 2021; Veit et al., 2016). Assume also that each \(L_{i}\) has a final layer that is a fully connected linear layer and a linear input layer (this assumption is true for both attention and MLP modules in transformers without normalization layers). Constructing \(\theta^{\prime}\) such that each \(L_{i}\) has input weights pre-multiplied by \(\mathbf{S}^{-1}\) and output FC weights multiplied by \(\mathbf{S}\), we have \(g_{\theta^{\prime}}(p)=\sum_{i}\mathbf{S}L_{i}(p)=\mathbf{S}\cdot g_{\theta}(p)\) by linearity.