Proposition REF  (1) states that the operator \(F(w,x)\)  is Lipschitz in terms of \(w\) , which further implies affine growth rate of \(F(w,x)\)  in the sense that \(\Vert F(w,x)\Vert _2\le f(\gamma \zeta _\pi )(\Vert w\Vert _2+1)\)  for any \(w\in \mathbb {R}^d\)  and \(x\in \mathcal {X}\) . Proposition REF  (2) states that the auxiliary Markov chain \(\lbrace X_k\rbrace \)  also preserves the geometric mixing property, which is particularly useful for us to control the Markovian noise in the update equation (REF ). Proposition REF  (3) implies that using \(M(w-w_\pi )\)  as the Lyapunov function, both SA algorithm (REF ) and its associated ODE have a negative drift. This is the key property used in [1]} to establish the finite-sample convergence bounds. Now we are ready to apply [1]} to establish finite-sample bounds of Algorithm (REF ) (and hence Algorithm REF ). The details are presented in Appendix REF .
