By Corollary REF , under Assumption REF , player \(i\)  faces an MDP with state process \(\lbrace  \textbf {x}_t \rbrace _{t \ge 0}\) . Again by the softness of \(\mathbf {\pi }\)  and Assumption REF , each \((\textbf {s}, a^i ) \in \textbf {X}\times \mathbb {U}\)  is visited infinitely often almost surely. The convergence of Q-factors under these conditions is well-known; see for instance [1]}. The same analysis can be used to prove that \(\lim _{t \rightarrow \infty } \widehat{J}^i_t ( \textbf {s}) = J^i_{\mathbf {\pi }} (\textbf {s})\)  almost surely for each \(\textbf {s}\in \textbf {X}\) , proving the second part.
