Computational cost of Riemannian gradient descent
Riemannian gradient descent on \(\mathcal {O}_p\)  first computes the descent direction, using the Euclidean gradient of \(f\) , and then computes the next iterate using a retraction. Depending on the problem, the main computational bottleneck may come from either of the two steps.
For instance, when training a Recurrent Neural Network (RNN) with orthogonal weights [1]}, [2]}, [3]}, there is usually only one orthogonal matrix used in a large computational graph. Here, the cost of computing the Euclidean gradient with backpropagation is much higher than the cost of computing a retraction. Consequently, using the landing algorithm in this setting will only slightly reduce the cost of computations.
On the other hand, it is common to impose an orthogonality constraint on multilayer perceptron or convolutional neural networks [4]}, [5]}. It has been reported that this constraint allows faster training and better generalization. In this case, there are many orthogonal matrices, and the main bottleneck in training can be computing the retraction. Therefore, as we will see in the experiments, it is interesting to use the landing algorithm in this setting.
