The problem formulation seems, at first glance, to only capture observed contextual factors. The next example shows how latent confounders can be modeled as well. It is helpful to compare this example to the randomization based inference framework as applied to RCTs, which dates back to [1]}. In that framework, one would not view the reward an arm generates as being a sample from some fixed population. Instead, potential outcomes (i.e. the samples one would observe if an arm were pulled that period) can follow some arbitrary pattern, and the objective is to measure an arm's average performance in hindsight throughout the time of the experiment. The experimenter's randomization serves as the “reasoned basis” for inference. This is similar to nonstochastic MAB models [2]}, where nature selects rewards adversarially and performance is assessed relative to the best fixed-arm in hindsight. The model below might be viewed as a Bayesian analogue where nature's behavior is stochastic but nonstationary. The practitioner's choice of a prior then governs the kinds of patterns the algorithm guards against.
