[of Theorem REF ]
The proof is largely based on the lines of proof used by [1]} and [2]}. We tailor their proofs based on the Q-learning framework into the context of this paper.
We introduce notation for some intermediate stage quantities that are the analogs to the intermediate stage Q- and V- functions. Let \(\Xi _{S, \pi }^{(q)}(H^{(q)}, a^{(q)}) = \phi ^{(q)}\lbrace S^{(q)}_{\pi }(\cdot \mid H^{(q)}, A^{(q)}= a^{(q)})\rbrace \)  and \(\Phi _{S, \pi }^{(q)}(H^{(q)}, a^{(q)}) = \phi ^{(q)}\lbrace S^{(q)}_{\pi }(\cdot \mid H^{(q)}, A^{(q)}= \pi ^{(q)}(H^{(q)})\rbrace \) . The dependency of \(\Xi \)  and \(\Phi \)  on the choice of \(\phi \)  is marked with the corresponding subscripts, e.g., \(\Phi _{\cdot , \mu }^{(q)}(\cdot ) = \phi _\mu ^{(q)}(\cdot )\) , or is suppressed, if it is clear by context.
