(i)
The MDP with state space \(\varphi ^i ( \textbf {X})\)  described above is an instance of what [1]} calls an approximate belief MDP with memory length 0.
(ii)
Under Assumption REF , each soft policy \(\tilde{\mathbf {\pi }} \in \mathbf {\Gamma }_{S}\)  gives rise to a unique invariant measure \(\nu _{\tilde{\mathbf {\pi }}} \in \mathcal {P}( \textbf {X})\) , and we have that \({\rm Pr}^{\tilde{\mathbf {\pi }}}_{\nu } \left( \textbf {x}_t \in \cdot \right) \rightarrow \nu _{\tilde{\mathbf {\pi }}} ( \cdot )\)  in total variation as \(t \rightarrow \infty \) .
(iii)
The MDP described above, with state space \(\varphi ^i ( \textbf {X})\) , depends on the joint policy \(\mathbf {\pi }\)  through the invariant measure \(\nu _{\mathbf {\pi }}\) . In particular, this MDP depends on the entire joint policy \(( \pi ^i ,\mathbf {\pi }^{-i})\) , not merely on \(\mathbf {\pi }^{-i}\) . Crucially, this means that the limiting quantities \(\tilde{V}^{*i}_{\mathbf {\pi }}\)  and \(\tilde{W}^{*i}_{\mathbf {\pi }}\)  depend, in general, on the policy \(\pi ^i\)  used by player \(i\) . This is in contrast to the MDP settings (including the specific cases in Parts 2) and 3) of Theorem REF ), where the limiting values of Q-learning and value function learning are the same for different (soft/sufficiently exploratory) policies.
(iv)
In general, as player \(i\)  does not face an MDP with state process \(\lbrace  y^i_t \rbrace _{t \ge 0}\) , \(i\)  does not face this particular MDP on \(\varphi ^i ( \textbf {X}) \subset \mathbb {Y}\) . The limiting quantities \(\tilde{V}^{*i}_{\mathbf {\pi }}\)  and \(\tilde{W}^{*i}_{\mathbf {\pi }}\)  do not, in general, have any inherent relevance to player \(i\) 's objective function in the game \(\mathcal {G}\) . These quantities should instead be interpreted as the subjective beliefs of player \(i\) , which were arrived at through a naive independent learning process.

