From the observations of Lezcano-Casado and Martínez-Rubio [1]}, we have 10 times less learning rate for the orthogonal parameters than that for the non-orthogonal parameters: we use 1e-5 for the former and 1e-4 for the latter.
