Adaptive gradient methods and the AdaGrad algorithm were proposed in [1]}. Soon afterwards followed other popular algorithms, most notable amongst them are Adam [2]}, RMSprop [3]}, and AdaDelta [4]}.
