thus proving the theorem.

In conclusion, from Theorems
REF ,REF  it is thus observed that for the online optimization setting the regret bounds when using stochastic-learning-rates differ by \( \sqrt{\frac{c_2}{c_1}} \)  from the bounds of the original SGD algorithm. In specific, the guarantees in (REF ) and (REF ) are the same as in the online learning setting for the original SGD, except for a multiplying factor of \( \sqrt{\frac{{{c_2}}}{{{c_1}}}} \)  compared to previous works (e.g. [1]}).
