This result implies that subsequences of the whole sequence \(((\mathbf {\theta }_n, \mathbf {w}_n))_{n\in \mathbb {N}}\)  approximate a stationary local Nash equilibrium \((\mathbf {\theta }^\star , \mathbf {w}^\star )\)  satisfying \(\nabla _{\mathbf {\theta }} L_G (\mathbf {\theta }^\star , \mathbf {w}^\star ) = \mathbf {0}\)  and \(\nabla _{\mathbf {w}} L_D (\mathbf {\theta }^\star , \mathbf {w}^\star ) = \mathbf {0}\)  (see Problem REF ). Hence, TTUR using constant learning rates \(\alpha ^G\)  and \(\alpha ^D\)  should converge unstably. Figure REF  plots TTUR based on Adam with constant learning rates defined by \(\alpha ^G = 5 \times 10^{-4}\)  and \(\alpha ^D = 10^{-5}\)  [1]}, and a fixed batch size \(b = 2^6\)  for training a deep convolutional GAN (DCGAN) [2]} on the CIFAR-10 dataset. Although the values of the loss functions of the generator and discriminator fluctuate, we can see that there are subsequences of the whole sequence which converge to some point. Accordingly, our theoretical result (Theorem REF ) would be adequate from a practical viewpoint.
