Under mild conditions on the MDP \(M\) , the action value function \(Q^{*}_{M}\)  can be learned iteratively using the Q-learning algorithm [1]}. Similarly, for stationary policies \(\pi \in \Gamma _{S}\) , the value function \(J_{ \pi }\)  can be learned iteratively. Thus, an agent playing the MDP \(M\)  and using a stationary policy \(\pi \in \Gamma _{S}\)  may use an estimated surrogate of the inequality of (REF )—involving stochastic estimates of \(Q^{*}_{M}\)  and \(J_{\pi }\) —as a stopping condition when searching for an \(\epsilon \) -optimal policy. This idea will feature heavily in the subsequent sections; in particular, we will use an analogous condition for our definition of subjective best-responding.
