Despite some theoretical guarantee made on the convergence of Adam-type algorithms on convex concave or non-convex concave min-max optimization, in the non-convex non-concave setting which is most general, there is no theoretical guarantee on convergence. Comparatively speaking, proving the convergence of Adam-type algorithms is much more difficult since they use an empirical version of Momentum. Although it has been shown to perform well in practice, it is actually difficult to analyze theoretically. Even in the standard convex setting, proving the convergence of Adam-type algorithms [1]}, [2]} is much harder than other adaptive algorithms such as AdaGrad [3]}. Actually, the original version of Adam is known not to converge in convex settings. Therefore, to formally analyze the convergence of Adam-type algorithms in min-max optimization, we also consider a “theoretically correct” version of Adam, which is an analog of AMSGrad [1]}.
In this paper, there are three main contributions, which are listed as follows:
