We now extend to realistic models where the \(\varepsilon\) in the denominator of LayerNorm is not 0. We can do this because the only fact we used about \(x\mapsto n(x)\) was that \(x\mapsto n(\mathbf{U}x)\) was identical to \(x\mapsto\mathbf{U}n(x)\). In turn Lemma C.2 relied on \(||\mathbf{U}x||=||x||\) due to orthogonality. But adjusting \(n(x)\) to \(n^{\prime}(x):=x\bigg{/}\sqrt{\frac{1}{h}||x||^{2}+\varepsilon}\) (i.e. normalization with an epsilon), since \(||x||=||\mathbf{U}x||\), \(n^{\prime}\) commutes with \(\mathbf{U}\), and so the proofs in Appendix C.4 and Appendix C.5 still work when using \(n^{\prime}\) instead of \(n\).