In this experiment, we consider a distillation problem [1]}: we generate a random set of parameters \(\theta ^*\)  that gives the target teacher network \(\Phi _{\theta ^*}\) . Then, starting from a random parameter initialization, we try to learn the mapping \(\Phi _{\theta ^*}\)  with a student network with parameters \(\theta \) , by minimizing the loss \(\mathcal {L}(\theta )= \sum _{q=1}^Q\Vert \Phi _{\theta }(x^q) - \Phi _{\theta ^*}(x^q)\Vert ^2\) , where the \(x^q\)  are the training examples, drawn i.i.d. from a normal distribution.
We consider the optimization of the orthogonal weights \(W_1, \dots , W_L\)  with different methods.
We use stochastic Riemannian gradient descent using the exponential or Cayley retraction or the landing flow, with or without momentum for the latter.
We also consider trivializations [2]}, [3]}, where each matrix is parametrized as \(W_i = \exp (A_i)\)  with \(A_i\in \operatorname{Skew}_p\) , and the optimization is carried over \(A_i\)  (more details in app:triv). We use trivializations with SGD, SGD + momentum, Adam, and RMSProp.
We use matrices of size \(p=100\) , with a depth \(D=10\) . The learning rate is \(\eta =0.5\) , and the batch size is 256.
Orthogonalization and test error are displayed in fig:simplenetwork. Orthogonalization error is not displayed for trivialization methods, since they are exact.
The landing flow with momentum is the fastest method.
It also leads to smaller orthogonalization error than the other retraction methods, because it does not suffer from accumulation of numerical errors (see sec:numerrors).
<FIGURE>