The LayerNorm operation is the composition of a centering operation \(x\mapsto x-\bar{x}\) with RMSNorm (i.e. first centering is applied, then RMSNorm). Therefore the identical argument to Appendix C.4 goes through, besides the fact that we need \(\mathbf{U}\) to also commute with the centering operation. Since the centering operation fixes a \((h-1)\) dimensional subspace defined by \(\mathbf{1}^{T}x=0\) where \(\mathbf{1}\in\mathds{R}^{h}\) is the vector of ones, it is enough to impose an additional condition that \(\mathbf{U}\,\mathbf{1}\in\{-\mathbf{1},\mathbf{1}\}\).