The first condition ensures that all possible solutions will be reached with high probability regardless of the \(\theta _0\)  while the second ensures convergence to the true value. [1]} extend the results to non-linear least squares with gradients computed by finite-differences. Feasible choices of learning rate schedules include \(\gamma _k = \gamma _0 k^{-\delta }\) , with \(\delta \in (1/2,1]\)  and \(\gamma _0>0\)  as choice parameters. Depending on \(\delta \) , convergence as measured by \(\mathbb {E}(\Vert \theta _k-\hat{\theta }_n\Vert ^2)\)  can occur at a \(1/k\)  rate or slower. [2]} and [3]} show that the averaged values \(\overline{\theta }_k = \frac{1}{k}\sum _{i=1}^k \theta _i\) , converge at the fastest \(1/k\)  rate for all choices of \(\delta \in (1/2,1]\) . This is commonly refered to as Polyak-Ruppert averaging. Building on these results, modern stochastic gradient-descent updates are given by
\(\theta _{k+1}&=&\theta _{k}- \gamma _k G_m(\theta _{k})\) 
