A Unified Convergence Theorem for Stochastic Optimization Methods
In this work, we provide a fundamental unified convergence theorem used for
deriving expected and almost sure convergence results for a series of
stochastic optimization methods. Our unified theorem only requires to verify
several representative conditions and is not tailored to any specific
algorithm. As a direct application, we recover expected and almost sure
convergence results of the stochastic gradient method (SGD) and random
reshuffling (RR) under more general settings. Moreover, we establish new
expected and almost sure convergence results for the stochastic proximal
gradient method (prox-SGD) and stochastic model-based methods (SMM) for
nonsmooth nonconvex optimization problems. These applications reveal that our
unified theorem provides a plugin-type convergence analysis and strong
convergence guarantees for a wide class of stochastic optimization methods.