The decomposition converts a linear projection of input \(W\mathbf {x}\)  into a series of (unconstrained) linear projections \(U \Sigma V^{\top } \mathbf {x}\) . However, this can cause numerical instability during optimization [1]}. We could perform the Singular Value Decomposition (SVD) over \(W\)  so that it performs rotation (\(V^{\top }\) ), stretch (\(\Sigma \) ), and rotation (\(U\) ) with orthogonal basis vectors in \(U\)  and \(V\) . Unfortunately, solving the full SVD has a computational complexity of \(\mathcal {O}(\max (M,N)^2)\)  [2]}. Here, we put an orthogonality constraint only on \(\Sigma \)  and perform projection (\(V^{\top }\) ), rotation (\(\Sigma \) ), and projection (\(U\) ) of input \(\mathbf {x}\) . In addition, we put \(V^{\top }\mathbf {x}\)  in a unit sphere (via \(\ell _2\) -normalization) before rotating it with \(\Sigma \) . This not only improves numerical stability, but also removes magnitude information in \(V^{\top }\mathbf {x}\)  and keeps angular information only, which has been shown to provide sample discriminative information [3]}. To impose the orthogonality constraint on \(\Sigma \) , we use the Padé approximation with a scale-squaring trick of [4]}. Intuitively, we linearly project \(\mathbf {x}\)  onto a unit sphere (\(V^{\top } \mathbf {x}\) ) and rotate it (\(\Sigma V^{\top } \mathbf {x}\) ) in each Transformer so that it captures the dynamics of each input distribution independently. We then project it to the shared space via \(U\) , capturing shared regularities across all three Transformers.
