This paper thus aims to explain this phenomenon by bridging the gap between theory and practice. On one hand, we understand under which conditions Adam-type optimization algorithms have provable convergence for min-max optimization. Towards this end, a recent work [1]} designs two algorithms, Optimistic Stochastic Gradient (OSG) and Optimistic AdaGrad (OAdaGrad) for solving a class of non-convex non-concave min-max problems and gives theoretical guarantee on their convergence. [1]} also proposes an open problem on the convergence proof of Adam-type algorithms, which is solved by this paper. On the other hand, we find that the MVI condition needed for our convergence proof does not practically hold for GANs. Instead, we propose the much milder one-sided MVI condition, which tends to hold practically and under which we provide the theoretical guarantee of the one-sided convergence of Adam-type algorithms in GANs' training.
