A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning
Adaptive gradient methods have become popular in optimizing deep neural
networks; recent examples include AdaGrad and Adam. Although Adam usually
converges faster, variations of Adam, for instance, the AdaBelief algorithm,
have been proposed to enhance Adam's poor generalization ability compared to
the classical stochastic gradient method. This paper develops a generic
framework for adaptive gradient methods that solve non-convex optimization
problems. We first model the adaptive gradient methods in a state-space
framework, which allows us to present simpler convergence proofs of adaptive
optimizers such as AdaGrad, Adam, and AdaBelief. We then utilize the transfer
function paradigm from classical control theory to propose a new variant of
Adam, coined AdamSSM. We add an appropriate pole-zero pair in the transfer
function from squared gradients to the second moment estimate. We prove the
convergence of the proposed AdamSSM algorithm. Applications on benchmark
machine learning tasks of image classification using CNN architectures and
language modeling using LSTM architecture demonstrate that the AdamSSM
algorithm improves the gap between generalization accuracy and faster
convergence than the recent adaptive gradient methods.