which is identical to the original model. Applying this procedure to all layers added to the hidden state (using the different \(W\) diagonal matrices each time) gives us a model \(g_{\theta^{\prime}}(p)\) such that \(g_{\theta^{\prime}}(p)=\mathbf{U}g_{\theta^{\prime}}(p)\) so a different embedding projection matrix \(\mathbf{W}\mathbf{U}^{\top}\) will give identical outputs to the original model \(g_{\theta}(p)\) (with embedding projection \(\mathbf{W}\)).