Convergence analysis in expectation but for deterministic learning rates has been studied in [1]}, [2]}, [3]}, [4]}, [5]}. This work provides convergence results in-expectation for SGD using stochastic learning rate as well as results under the online learning framework (introduced in [6]}). It shows that with stochastic-learning-rates optimization performance is significantly improved, while providing similar technical results to the deterministic-learning-rate case.
Convergence analysis in the almost-surely setting using stochastic learning rate schemes, and in specific MSLR, for algorithms including SGD has been done in [7]}.
