where the last step is due to the d-separation.
Similarly to [1]}, we assume that distribution \(P(\mathbf {z}| \mathbf {x}_{t,a})\)  is known based on previous knowledge that was used to build the causal graph.
Then, by using a sample mean estimator (denoted by \(\hat{\mu }_{\mathbf {w}}(t)\) ) to estimate \(\mathbb {E}[R|\mathbf {w} ]\)  based on the observational data up to time \(t\) , the estimated reward mean is given by
\(\hat{\mu }_{\pi }(t) &=\mathbb {E}_{\mathbf {a}\sim \pi } \left[ \sum _{\mathbf {Z}} \hat{\mu }_{\mathbf {w}}(t) \cdot P(\mathbf {z}| \mathbf {x}_{t,a}) \right]\) 
