where \(f:\mathcal {D}\rightarrow \mathbb {R}\)  is a continuously differentiable function. As one of the most dominant algorithms on the optimization problem above, Stochastic Gradient Descent (SGD) was originally proposed by [1]}, which has been both empirically and theoretically proved effective, especially when facing large datasets and complicated models. To further improve the performance of SGD, several adaptive variants of SGD have been proposed, such as RMSprop, Adam [2]}, AdaGrad [3]}, AMSGrad [4]} and AdaDelta [5]}. Distinguished from the vanilla gradient descent or its stochastic version SGD, adaptive gradient methods use a coordinate-wise scaling of the updating direction and each iteration relies on the history information of past gradients. In AdaGrad, we use arithmetic average when adopting history gradient information of each iteration while in Adam, RMSprop etc., we use exponential moving average instead because its believed that the more current gradient information is more important. Although adaptive gradient methods and momentum based methods are two different routes on optimization, they are combined perfectly in Adam. Now we introduce the family of adaptive gradient methods and Adam-type, and all of them have the following form:
\(\begin{aligned}m_{t+1}&=h_t \nabla f(x_t)+r_t\cdot m_t\\v_{t+1}&=p_t (\nabla f(x_t))^2 + q_t\cdot v_t\\x_{t+1}&=x_t-\lambda _t\cdot \frac{m_{t+1}}{\sqrt{v_{t+1}}+\varepsilon }.\end{aligned}[\mathrm {Adaptive}]\) 
