The Adam (short for Adaptive Moment Estimation) optimizer [1]} uses the following exponential smoothing of the gradients and the squared gradients of the objective function \(g_{\lambda }(\mathbf {w})\) . For each iteration \(l = 1,2, \dots ,\)  with \(\mathbf {w}^{(l-1)}\)  being the value of \(\mathbf {w}\)  obtained from the previous iteration, let
\(\mathbf {u}^{(l)} &= - (1 - \xi _1) \sum _{k = 0}^{l-1} \xi _1^{l -k -1} \nabla g_{\lambda }(\mathbf {w}^{(k)}), \quad (\text{momentum update})\\\mathbf {v}^{(l)} &= (1 - \xi _2)\sum _{k = 0}^{l-1} \xi _2^{l - k -1} \left(\nabla g_{\lambda }(\mathbf {w}^{(k)}) \odot \nabla g_{\lambda }(\mathbf {w}^{(k)})\right),\) 
