At the end of each iteration \(t\) , the critic then updates the value function via \(V^s_{t} = (1-\alpha _t)V^s_{t-1} + \alpha _t \rho _t^s\) , where \(\rho _t^s\)  is an estimation of \(x_t^{s^\top } Q^s_t y^s_t\)  such that \(|\rho ^s_t - x_t^{s^\top } Q^s_t y^s_t| \le \varepsilon \) .For simplicity, here we assume that the two players share the same estimator \(\rho _t^s\)  (and thus same \(V^s_t\)  and \(Q^s_t\) ). However, our analysis works even if they maintain different versions of \(\rho _t^s\) , as long as they are \(\varepsilon \) -close to \(x_t^{s^\top } Q^s_t y^s_t\)  with respect to their own \(Q^s_t\) .
To stabilize the game matrix, we require the learning rate \(\alpha _t\)  to decrease in \(t\)  and go to zero.
Most of our analysis is conducted under this general condition, and the final convergence rate depends on the concrete form of \(\alpha _t\) , which we set to \(\alpha _t=\frac{H+1}{H+t}\)  with \(H=\frac{2}{1-\gamma }\)  inspired by [1]} (there could be a different choice leading to a better convergence though).
