The above subsection described that TTUR using decaying learning rates can in theory be applied to local Nash equilibrium problems in GANs. Meanwhile, the numerical results in [1]}, [2]} showed that TTUR and deep learning optimizers using constant learning rates perform well. The motivation behind this work is thus to identify whether TTUR with constant learning rates can in theory be applied to local Nash equilibrium problems in GANs. In so doing, we may bridge the gap between theory and practice for TTUR.
