Example 2: This example of ETT requires both observational data to estimate the marginal \(p(A_t)\) , and experimental data to estimate \(\mathbb {E}[Y_t^{A_t=a_t^{\prime }}]\)  each inducing different sources of modeling uncertainty depending on the amount and quality of available data. If only observational data is available, then this OPE estimand is not identifiable, and the irreducible or aleatoric uncertainty corresponds to that induced due to our uncertainty of the knowledge of \(\mathbb {E}[Y_t^{A_t=a_t^{\prime }}]\) .
We now discuss the implication of characterizing these sources of uncertainty.
First is the need to obtain additional domain knowledge to reduce uncertainty associated with identifiability. In traditional causality literature, the general focus of establishing counterfactual identifiability has been on graphical criteria with less emphasis on parametric assumptions on the corresponding functional relationships between covariates of interest. Of course there are several examples where making parametric assumptions can aid identifiability, even if non-parametric identifiability does not hold [1]}, [2]}, [3]}.
Second is to resort to collecting experimental data, which in and of itself defeats the primary goal of OPE. More specifically, we discuss how humans can play a critical role for OPE where the causal estimand formalizes the type of human expertise and experimentation is required for practical OPE. We elaborate on these implications in detail in the following section.
The Role of Humans in OPE
In this section, we illustrate how the causal estimand, and explicitly outlining the sources of uncertainty helps to precisely characterize how human feedback should be incorporated and what the role can fill in improving the reliability of OPE. In particular,
the role of incorporating human expertise can be useful for: i) providing modeling assumptions (and relaxations) under which we can describe the right causal OPE estimand,
ii) physical experimentation to aid identifiability, iii) mitigating sources of uncertainty to compensate for limited data, and iv) providing assumptions that enable to understand the conditions for the validity of the OPE estimand. In essence, most of these principled ways of incorporating human feedback can be chalked up to reducing different sources of uncertainty and improving the reliability of OPE estimates. We now elaborate on these distinct roles that human feedback can fulfill in OPE, while explicitly discussing the kind of benefit it may provide.
Parametric assumptions and modeling constraints.
In many situations, making additional modeling assumptions can be sufficient and valid to establish some form of identifiability, depending on the application. Human experts can assess the applicability of such assumptions to the domain, as well as provide any additional constraints. For example, assumptions such as consistency, monotonicity [4]}, or generalizations thereof are often used in epidemiological literature to estimate counterfactuals from observational data and have recently been explored in the context of OPE for model debugging [5]}. In Example 2, assuming treatments are binary is a form of simple parametric assumption that can reduce the aleatoric uncertainty by reducing the number of interventional experiments that need to be conducted. These assumptions aid identifiability by making fundamental assumptions about the nature of counterfactuals, reducing aleatoric uncertainty. Additional knowledge like linearity and other functional assumptions, like the availability of noisy proxies of confounders [6]}, [7]}, or availability of instrumental variables can help constrain the amount of online experimentation required for OPE [8]}. These assumptions can help reduce modeling uncertainty, during estimation as well as reduce aleatoric uncertainty by constraining counterfactual distributions to specific functional forms.
When populations are heterogeneous, human input can additionally provide such functional assumptions at a higher level of granularity by drawing from existing clinical knowledge. If the estimated dynamics in models are unreliable or may shift across domains, domain knowledge about the amount of shift can improve the robustness of OPE estimates in practice [9]}. In many cases, observational data may be insufficient, but experimentation is prohibitively costly or unethical. In such cases, domain experts can provide auxiliary model information like disease kinematics, or biophysical models collected from prior scientific experiments, as an alternative to simulating appropriate interventions [10]}, [11]}. This form of human or expert feedback can therefore reduce modeling uncertainty in the system. Most of these assumptions have been implicitly made to aid some form of identifiability in the most general sense and focused on population-level identifiability.

Physical experimentation for aiding identifiability.
Establishing non-identifiability of a causal estimand corresponding to an OPE task can help identify the physical experimentation necessary to enable OPE for all cases we outline in Figure REF . As we saw in Example 2,
OPE tasks requiring counterfactual reasoning, like ETT, might reveal the need for additional interventional/experimental data (or at the very least require domain expertise that further help establish identifiability). We demonstrated in Equation (REF ), that in this case, a combination of i) binary treatment and outcome (parametric assumption a human can provide and justify) as well as ii) interventional/experimental data is necessary when no additional assumptions like modeling assumptions can be justified. Without experimentation, there is irreducible uncertainty in the system. Further, consider the POMDP in Figure REF  where the prospective/retrospective OPE is unidentifiable (in expectation or for individual-level OPE). To aid identifiability, one primarily needs to collect unobserved confounders. Thus, physical experimentation need not directly imply expensive experiments like a randomized controlled trial, but could also help if additional confounding variables could be collected to establish identifiability. Data collection for specific variables can be less prohibitive than an active collection of interventional quantities and has been explored recently for (conditional) average treatment effect estimation [12]}, [13]}. Nonidentifiability exacerbated due to unobserved confounding and lack of the ability to experiment is the primary source of irreducible uncertainty or aleatoric uncertainty. Thus this form of human feedback can explicitly reduce aleatoric uncertainty. We argue that this is the main challenge where human input can further aid individual-level and retrospective OPE applications.

Mitigating estimation uncertainty from finite data.
In many cases, while observational data collected from behavior policy might be sufficient for identifiability, the amount of data might be insufficient, significantly violating overlap assumptions which manifest in the increased variance of the OPE estimate. For instance, in Example 1, we may have very few samples collected from behavior policy, or the state representation may be high-dimensional. In these cases, humans may be able to analyze the validity of off-policy evaluation estimates posthoc to, for instance, reduce our modeling uncertainty. To this end, [14]} propose using humans to assess the validity of OPE estimates by manually analyzing the influential observations whose removal increases the estimation error. Similar approaches consider using human expertise to learn shaped control variates for variance reduction in OPE [15]} or using human expertise to define admissible rewards for high confidence OPE [16]}. This is another form of reducing stochasticity due to insufficient data i.e. reducing modeling uncertainty. While most existing work in this domain is applicable for prospective OPE tasks (and specifically Example 1), focused on sub-population/population-level OPE, there is significant potential for human input to reduce modeling uncertainty for retrospective and individual-level OPE tasks.

Validating OPE estimates.
Validation of OPE is a significant but critical challenge for practical utility. The potential for human feedback is the least well explored especially for retrospective OPE. Technically, sensitivity analysis models are the main modeling frameworks that can allow the incorporation of domain knowledge that characterizes the severity of unobserved confounding in the system. For instance, it can help validate prospective as well as retrospective OPE tasks for complex settings such as the POMDP (Figure REF ). Currently, the most commonly used sensitivity model known as the marginal sensitivity model, which constrains deviations to the propensity of treatment in the presence of unobserved confounders, and has been widely explored for population and sub-population-level prospective OPE in reinforcement learning [17]}, [18]}. Model debugging of this kind has been explored for counterfactual OPE for discrete SCMs for managing sepsis among diabetic patients in [5]}.
<FIGURE>
A Roadmap for OPE in Practice
In this work, we motivate the need to generalize the purview of OPE in sequential decision-making, specifically the need to distinguish prospective versus retrospective OPE tasks conducted at the individual, sub-population, and population level. To operationalize the generalization, we argue that OPE tasks should be framed in the context of the data generating process and a corresponding causal estimand. We discuss how this operationalization can highlight and help mitigate fundamental OPE challenges related to identification, estimation, and principally outline the role of human input to make OPE tasks practical. In doing so we highlight many open questions. In this section, we conclude by providing a prescriptive roadmap for addressing OPE tasks in practice, and technical challenges to consider along the way. Our prescriptive roadmap is summarized in Figure REF .
Elicit goals of OPE using a causal estimand.
By eliciting the appropriate causal estimand, we can transparently identify whether an OPE task is prospective/retrospective and required at the level of individuals, sub-populations, and population levels. More importantly, this allows us to consider whether existing observational data collected from a behavior policy and additional assumptions can be justified to estimate the corresponding estimand. The corresponding assumptions in terms of the data-generating process (causal structure) and other assumptions (such as parametric assumptions) help provide precise specifications of the claim and validity of the OPE estimand.

Outline and justify nature of human input.
The primary goal of OPE is to enable the evaluation of new policies based on observational data from behavior policies without additional experimentation. However, we highlight situations human input may be necessary and justified to solve specific OPE tasks. Here the form of human input can range from the need to conduct complex physical experimentation such as randomized controlled trials or in the form of data collection. This form of human input is explicitly aimed at mitigating non-identifiability issues. In some cases, human input might suffice to establish partial OPE estimates (bounds on the OPE estimate as opposed to point estimates). Even if an estimand is identifiable, we have to consider whether sufficient observational data is available to estimate the corresponding off-policy estimand. Here human input can help mitigate modeling uncertainty associated with estimation challenges by providing additional domain knowledge and would help improve the statistical properties of the corresponding estimators.

Identify minimal validation requirements using causal desiderata.
The nature and type of validation required of OPE estimands are clear by focusing on the assumptions, the appropriate causal estimand, and the nature of human input incorporated to address identifiability and to obtain desirable estimation properties. In particular, these desiderata further identify the nature of validation, in the form of sensitivity analysis, specifically of the type that invalidates assumptions justified for the domain, the amount and nature of unobserved confounding, or structural assumptions to clearly identify granular conditions under which the OPE estimand is meaningful. Finally, to reliably deploy policies that are evaluated in this manner, it is imperative to consider additional human factors that will allow for appropriate interpretation of recommendations inferred from the OPE estimates under the specifications outlined by our desiderata.
Note that most generally, assumptions of the data-generating process, as we have assumed throughout the draft, is a form of human expertise that allows to incorporate the most fundamental set of conditions of the domain under which the OPE estimand can be formulated. Using observational data as a means to learn such structures reliably is a nascent area called causal discovery or causal structure learning [20]}.
In conclusion, these considerations should move us towards a more rigorous way of performing OPE in practice and create a better understanding of those factors essential for making OPE reliable for practice.
