The main contribution of this paper is a theoretical analysis of TTUR using constant learning rates. We examine TTURs based on several adaptive methods (see Algorithm REF  and Table REF ), such as Adam [1]}, adaptive mean square gradient (AMSGrad) [2]}, and AdaBelief (short for adapting step sizes by the belief in observed gradients) [3]}. Let \(((\mathbf {\theta }_n, \mathbf {w}_n))_{n\in \mathbb {N}} \subset \mathbb {R}^\Theta \times \mathbb {R}^W\)  be the sequence generated by TTUR using constant learning rates \(\alpha ^G\)  and \(\alpha ^D\) . We show that, under certain assumptions, there exist accumulation points \((\mathbf {\theta }^*, \mathbf {w}^*)\)  and \((\mathbf {\theta }_*, \mathbf {w}_*)\)  of \(((\mathbf {\theta }_n, \mathbf {w}_n))_{n\in \mathbb {N}}\)  such that
\(&\mathbb {E}\left[ \Vert \nabla _{\mathbf {\theta }} L_G (\mathbf {\theta }^*, \mathbf {w}^*) \Vert ^2 \right]\le \frac{\alpha ^{G} (\sigma _G^2 b^{-1} + M_G^2)}{2\tilde{\beta }^G \tilde{\gamma }^{G^2} h_{0,*}^G}+ \sqrt{\Theta \mathrm {Dist}(\tilde{\mathbf {\theta }})\left( \frac{\sigma _G^2}{b} + M_G^2 \right)}\frac{\beta ^G}{\tilde{\beta }^G}, \\&\mathbb {E}\left[ \Vert \nabla _{\mathbf {w}} L_D (\mathbf {\theta }_*, \mathbf {w}_*) \Vert ^2 \right]\le \frac{\alpha ^{D} (\sigma _D^2 b^{-1} + M_D^2)}{2\tilde{\beta }^D \tilde{\gamma }^{D^2} h_{0,*}^D}+ \sqrt{W \mathrm {Dist}(\tilde{\mathbf {w}})\left( \frac{\sigma _D^2}{b} + M_D^2 \right)}\frac{\beta ^D}{\tilde{\beta }^D},\) 
