On Scaled Methods for Saddle Point Problems
Methods with adaptive scaling of different features play a key role in
solving saddle point problems, primarily due to Adam's popularity for solving
adversarial machine learning problems, including GANS training. This paper
carries out a theoretical analysis of the following scaling techniques for
solving SPPs: the well-known Adam and RmsProp scaling and the newer AdaHessian
and OASIS based on Hutchison approximation. We use the Extra Gradient and its
improved version with negative momentum as the basic method. Experimental
studies on GANs show good applicability not only for Adam, but also for other
less popular methods.