LeNet on MNIST
We train a LeNet5 [1]} for classification on the MNIST dataset. The network has 3 convolutional layers, and we impose an orthogonal constraint on the square kernel matrices. We take a batch size of 4, and for each algorithm, we take the learning rate that gives the fastest convergence in \(4^{i}, i=-5\dots 0\) .
We compare the same algorithms as before.
fig:mnist displays the results of our experiment.
Here, the landing algorithm is about \(50\%\)  faster than retraction methods, and reaches a smaller orthogonalization error.
While trivialization methods with advanced optimizers like RMSprop or Adam allow to reach the smallest test error, these methods are an order of magnitude slower than traditional methods in this case. The main reason is that they backprop through a matrix exponential at each iteration, which is costly (more details in Appendix).
<FIGURE>