We note that some prior works have established the convergence in expectation. Although the explosion of failure probability in (b) is suppressed by the direct characterization of the expectation, the drawback (a) still remains. In addition, such results require stronger assumptions. For example, [1]} showed a similar recursion to (REF ) for \(\mathbb {E}[\Vert _t - {^\star }\Vert ]\) . However, that work assumed a bounded moments condition for iterates, and assumed each component \(f_i\)  in (REF ) to be strongly convex. Both conditions are not needed in high probability analyses. (Note that the majority of high probability analyses assume that each component \(f_i\)  is convex though not necessarily strongly convex [2]}, [3]}, while we do not impose any conditions on \(f_i\) , which significantly weakens the existing conditions in the literature). Furthermore, the stepsize in [1]} was prespecified without any adaptivity. [5]} introduced a non-monotone line search to address the adaptivity issue, while extra conditions such as the compactness of \(\lbrace _t\rbrace _t\)  were imposed for their expectation analysis.
