Estimation. Once the appropriate observational and experimental data required for identifiable OPE are established, we may either have sufficient observational data, or collect some additional experimental data to enable estimation. In practice, we often work with finite samples, which are in-turn used to estimate the appropriate intermediate quantities that provide the OPE estimate. For example, Equation , requires reasonable estimates of the dynamics model, and the behavior policy. In Example 2, we will obtain finite sample estimates of \(\mathbb {E}[Y^{A_t = a_t}]\)  from experimental data. Finite sample issues induce additional source(s) of uncertainty, in some cases requiring large amounts of observational/experimental data for accurate estimation. This is true irrespective of the type of estimand, and might put additional demands on data-collection needs. Since this uncertainty can be reduced by collecting additional data, this constitutes modeling or epistemic uncertainty.
Beyond structural assumptions about the data-generating process, parametric assumptions such as stationarity of MDP dynamics further add to the source of modeling or epistemic uncertainty. For instance, non-stationary exogenous variables often complicate model estimation, and can propagate uncertainty, thereby worsening how severe violations of assumptions like sufficient overlap [1]}, [2]} are in the estimation process. These assumptions are a form of parametric assumptions that leave some sources of uncertainty irreducible (by absorbing non-stationarity into irreducible noise). Finally, lack of sufficient data, or inappropriately accounting for confounding variables (i.e. state representations) can also introduce biases in the estimate depending on the severity of violations of assumptions of ignorability and overlap [3]}. Issues of finite samples have only recently received attention and is necessary toward making causal inference a practical reality in ML and statistics [4]}, [5]}
Given these insights, we discuss dominant sources of uncertainty for different OPE estimands through our examples. 


