Remarks: In [1]}, the authors show that \(\tilde{W}^{*i}_{\mathbf {\pi }}\) , restricted to \( \lbrace  \varphi ^i ( \textbf {s}) : \textbf {s}\in \textbf {X}\rbrace  \times \mathbb {U}\) , is in fact the Q-function for some fully observed Markov decision problem with state space \( \varphi ^i ( \textbf {X}) := \lbrace  \varphi ^i (\textbf {s}) : \textbf {s}\in \textbf {X}\rbrace \) . The same argument used there can be used to establish that \(\tilde{V}^{*i}_{\mathbf {\pi }}\) , restricted to \(\varphi ^i (\textbf {X})\) , is the state value function for the same MDP. A complete specification of this MDP—with explicit characterization of its transition probabilities and its stage cost function—is possible, but is somewhat involved. We refer to [1]} for details, and instead offer the following important notes:
