The exploding and vanishing gradient issues are common in RNNs, and they could occur whenever the recurrent kernel does not have unit eigenvalues [1]}, [2]}, [3]}, [4]}, [5]}, i.e., the associated weight matrix is not orthogonal. Recent work proposes imposing orthogonality constraint on the weight matrix directly [1]}, [7]}:
\(\begin{split}\min _{\theta } {\mathcal {L}}{f_{\theta }({\mathbf {x}}),{\mathbf {y}}} \quad \operatorname{s.t.}{\mathbf {W}}_{hh}^{(\theta ){\mathbf {W}}_{hh}(\theta ) = {\mathbf {I}}, \det {\mathbf {W}}_{hh}(\theta )=1,}\end{split}where f_{\theta } is the RNN parameterized by \theta , and {\mathbf {W}}_{hh} is the recurrent kernel of the RNN (i.e., a subvector of \theta ). Since the constraints define the famous special orthogonal group which is a smooth manifold, manifold optimization methods can be developed to solve {eq: orthoRNN}~\cite {lezcano2019cheap,helfrich2018orthogonal}. But these methods entail heavy mathematics from differential geometry, and hence is impractical for non-experts to implement and build on (there could be additional constraints in real applications). Again, \texttt {PyGRANSO}\ \,is able to directly handle the nonlinear constraints, whether the user recognizes or not that the constraint forms the special orthogonal group.\) 
