We follow the OPE literature [1]}, [2]}, [3]} to assume that the initial-state distribution \(\mu _0\)  is known.
In the implementation, we augment \({\mathcal {D}}_{\mathrm {env}}\)  with \(10^5\)  samples from the ground-truth \(\mu _0\) .
To enhance training stability, we are motivated by the double Q-learning [4]} to use a target marginal importance weight network \(\omega ^{\prime }(s,a)\)  and a target critic network \(Q^{\prime }(s,a)\)  in the implementation of MIW training.
We employ the warm-start strategy so that each retraining of the marginal importance weight starts from the current \(\omega (s,a)\) .
On each run of the weight-retraining, \(Q^{\prime }\)  is initialized as the current critic target \(Q_{{\mathbf {\theta }}^{\prime }}\) .
Since the weights \(\omega (s,a)\) 's are normalized to have mean 1 in the weighted re-training of the dynamic model, to narrow the gap between the learning and the application of the weights, we add a constraint into the training of \(\omega \)  that its mean on each mini-batch is upper bounded by some constant \(g_{\mathrm {constraint}}\) .
Specifically, on each mini-batch gradient step, MIW is trained by the following constraint optimization
\(\leavevmode \xbox {resizebox}{\XMLaddatt {width}{397.10687pt}\begin{aligned}\arg \min _{\omega }& \; \left(\frac{1}{|{{\mathcal {B}}}|} \sum _{\left(s_i, a_i\right) \in {{\mathcal {B}}}} \omega \left(s_i, a_i\right) \cdot Q_{\mathbf {\theta }}\left(s_i, a_i\right) - y\right)^2, \\\mathrm {s.t.}~~~& \; \frac{1}{|{{\mathcal {B}}}|} \sum _{\left(s_i, a_i\right) \in {{\mathcal {B}}}} \omega \left(s_i, a_i\right) \le g_{\mathrm {constraint}}, \\\text{where } & \; y = \gamma \cdot \frac{1}{|{\mathcal {B}}|}\sum _{~{\left(s_i, a_i, s^{\prime }_i\right) \in {\mathcal {B}}}\atop {a^{\prime }_i\sim \pi _{\mathbf {\phi }}\left(\cdot \,|\,s^{\prime }_i\right)}} \omega ^{\prime }\left(s_i, a_i\right) \cdot Q^{\prime }\left(s^{\prime }_i, a^{\prime }_i\right) + (1-\gamma ) \cdot \frac{1}{|{\mathcal {B}}_{\mathrm {init}}|}\sum _{~{s_0 \in {\mathcal {B}}_{\mathrm {init}}}\atop {a_0\sim \pi _{\mathbf {\phi }}\left(\cdot \,|\,s_0\right)}} Q^{\prime }\left(s_0, a_0\right),\end{aligned}}\) 
