We start with assuming that the data is generated from an MDP (Figure REF ). We explain specific challenges using the example of Importance sampling (IS), which is often used to get an unbiased estimate of this estimand, when data is collected using policy \(\pi _b\) . Below we demonstrate the IS-OPE estimand [1]} and highlight assumptions made along the way that relate to and can be implied from assumptions on the data-generating mechanism.
\(\mathbb {E}_{{\mathbf {S}}, {\mathbf {Y}}}[\sum _{t=0}^T \gamma ^t Y_t{^{\color {OI-blue}\pi _e}} | \mathcal {H}_T{^{\color {OI-purple}\pi _b}}] &= \mathbb {E}_{{\mathbf {S}}, {\mathbf {Y}}, A_t \sim {\color {OI-blue}\pi _e}}[\sum _{t=0}^T \gamma ^t Y_t^{A_t} | \mathcal {H}_T{^{\color {OI-purple}\pi _b}}] \\&= \int \sum _{t=0}^T \gamma ^t Y_t^{A_t}\prod _{t=1}^T p(Y_t| A_t, S_t){\color {OI-blue}{\pi _{e}(A_t|S_t)}} {\color {OI-green}{p(S_{t+1}|A_t, S_{t})}} p(S_0) \\&= \int \sum _{t=0}^T \gamma ^t Y_t\prod _{t=1}^T p(Y_t| A_t, S_t){\color {OI-blue}{\pi _{e}(A_t|S_t)}} {\color {OI-green}{p(S_{t+1}|A_t, S_{t})}}  p(S_0) \\&= \int \sum _{t=0}^T \gamma ^t Y_t \prod _{t=1}^T p(Y_t| A_t, S_t){\color {OI-blue}{\pi _{e}(A_t|S_t)}} {\color {OI-purple}{\frac{\pi _{b}(A_t|S_t)}{\pi _{b}(A_t|S_t)}}} {\color {OI-green}{p(S_{t+1}|A_t, S_{t})}} p(S_0)\\&=\int \sum _{t=0}^T \gamma ^t Y_t \prod _{t=1}^T p(Y_t| A_t, S_t){\color {OI-purple}{\pi _{b}(A_t|S_t)}} \frac{{\color {OI-blue}{\pi _{e}(A_t|S_t)}}}{{\color {OI-purple}{\pi _{b}(A_t|S_t)}}} {\color {OI-green}{p(S_{t+1}|A_t, S_{t})}} p(S_0)\\&= \int \underbrace{\sum _{t=0}^T \gamma ^t Y_t \prod _{t=1}^T p(Y_t| A_t, S_t){\color {OI-purple}{\pi _{b}(A_t|S_t)}} {\color {OI-green}{p(S_{t+1}|A_t, S_{t})}}p(S_0)}_{\text{Trajectories under behavior policy}} \underbrace{\frac{{\color {OI-blue}{\pi _{e}(A_t|S_t)}}}{{\color {OI-purple}{\pi _{b}(A_t|S_t)}}}}_{\text{IS Re-weighting factor}} \) 
