In their seminal paper on stochastic approximation, [1]} consider the situation where only noisy gradients are available \(G_n(\theta _{k})+e_k\)  with \(e_k\)  iid, mean-zero. Using the notation above, they propose the gradient-descent update :
\( \theta _{k+1}=\theta _{k}- \gamma _{k} [G(\theta _{k})+e_k]\) , and show under regularity conditions that \(\theta _k \overset{a.s.}{\rightarrow } \hat{\theta }_n\)  when \(\gamma _k > 0\)  satisfy
\(\text{(i)}\quad \sum _{k=1}^\infty \gamma _k= +\infty , \quad \quad \text{(ii)} \quad \sum _{k=1}^\infty \gamma _k^2<+\infty .\) 
