Also, Adam-type algorithms are very effective in min-max optimization. As a direct and widely used application of min-max optimization, generative adversarial networks (GANs) are notorious for the training difficulty. Training by SGD will easily diverge to nowhere or converge to a limiting cycle, both of which will lead to an ill-performing solution, while Adam optimizer, as the default optimizer for GANs [1]}, can obtain better performance. The reason why these two optimizers have so much difference in GANs' training has long been an open problem. Traditionally, the training performance of min-max optimization is measured according to its first-order convergence, which means the norm of the gradient, but is it really true in GANs' training?
