In many cases, while observational data collected from behavior policy might be sufficient for identifiability, the amount of data might be insufficient, significantly violating overlap assumptions which manifest in the increased variance of the OPE estimate. For instance, in Example 1, we may have very few samples collected from behavior policy, or the state representation may be high-dimensional. In these cases, humans may be able to analyze the validity of off-policy evaluation estimates posthoc to, for instance, reduce our modeling uncertainty. To this end, [1]} propose using humans to assess the validity of OPE estimates by manually analyzing the influential observations whose removal increases the estimation error. Similar approaches consider using human expertise to learn shaped control variates for variance reduction in OPE [2]} or using human expertise to define admissible rewards for high confidence OPE [3]}. This is another form of reducing stochasticity due to insufficient data i.e. reducing modeling uncertainty. While most existing work in this domain is applicable for prospective OPE tasks (and specifically Example 1), focused on sub-population/population-level OPE, there is significant potential for human input to reduce modeling uncertainty for retrospective and individual-level OPE tasks.
