Similarly, the reward function can be approximated by a parametric model \(\widehat{r}\)  if assumed unknown.
With \(\widehat{P}\)  and \(\widehat{r}\) , the true MDP \({\mathcal {M}}\)  can be approximated by \(\widehat{{\mathcal {M}}}= ({\mathbb {S}}, {\mathbb {A}}, \widehat{P}, \widehat{r}, \gamma , \mu _0)\) .
We further define \(d_{\pi ,\gamma }^{P^*}(s,a)\)  as the stationary state-action distribution induced by \(\pi \)  on \(P^*\)  (or MDP \({\mathcal {M}}\) ), and \(d_{\pi ,\gamma }^{\widehat{P}}(s,a)\)  as that on the learned dynamic \(\widehat{P}\)  (or MDP \(\widehat{{\mathcal {M}}}\) ).
We approximate \(d_{\pi _{\mathbf {\phi }},\gamma }^{P^*}\)  by simulating \(\pi _{\mathbf {\phi }}\)  on \(\widehat{{\mathcal {M}}}\)  for a short horizon \(h\)  starting from state \(s \in {\mathcal {D}}_{\mathrm {env}}\) , as in prior work [1]}, [2]}, [3]}, [4]}.
The resulting transitions are stored in a replay buffer \({\mathcal {D}}_{\mathrm {model}}\) , constructed similar to the off-policy RL [5]}, [6]}.
To better approximate \(d_{\pi _{\mathbf {\phi }},\gamma }^{P^*}\) , sampling from \({\mathcal {D}}_{\mathrm {env}}\)  in Eqs. (REF ) and (REF ) is commonly replaced by sampling from the augmented dataset \(\mathcal {D} = f {\mathcal {D}}_{\mathrm {env}}+ (1-f) {\mathcal {D}}_{\mathrm {model}}, f \in [0,1]\) , denoting sampling from \({\mathcal {D}}_{\mathrm {env}}\)  and \({\mathcal {D}}_{\mathrm {model}}\)  with probabilities \(f\)  and \(1 - f\) , respectively.
We follow [2]} to use \(f=0.5\) .
