
Changing the learning target: Thompson sampling for contextual bandits usually samples an action according to the probability it maximizes the mean reward in the current context. In particular, one sets \(\mathbb {P}(I_t=i \mid H_t) = \mathbb {P}( i = \operatornamewithlimits{arg\,max}_{j \in [k]} \mu (\theta , j, X_t) \mid H_t)\) . DTS is instead based on sampling from the posterior distribution of the arm \(I^*\) , which is the arm that maximizes the average reward in the target population rather than in the current context. Sampling from \(\alpha _{t}\)  controls for confounders while directing exploration toward learning about the target arm of interest. Appendix  shows that the algorithm may fail without this change.

Resampling: Consider a problem without contexts. Then standard TS draws \(I_t\)  from \(\alpha _t\) , without resampling. This algorithm is designed to maximize the reward earned throughout the experiment, implicitly imagining that the experimentation process never ends. But it performs poorly if there is an interest also in being able to rapidly stop and commit confidently to a decision. To understand the issue, imagine that \(\alpha _{t,1}=.95\) , so the algorithm believes there is a 95% chance that arm 1 is optimal. Then TS plays arm 1 in roughly 19/20 periods, making it very slow to gather information about alternatives. TS would be very slow to reach 99% confidence as a result and this is exacerbated if even higher confidence is desired.
To overcome this issue, [1]} suggests a “top-two sampling” version of TS, which continues drawing arms from TS until two distinct options are drawn and then flips a biased coin to select among these two. To understand the resampling step,
imagine that \(\alpha _{t,1} \rightarrow 1\)  as \(t\rightarrow \infty \) . In this limit, the first sample from \(\alpha _t\)  is nearly always arm 1 and this is played with probability \(\beta _t\) . Otherwise, an arm is chosen by resampling, and the chance of picking arm \(j>1\)  is roughly \(\mathbb {P}(I_t= j \mid I_t \ne 1) \sim \frac{\alpha _{t,j}}{1-\alpha _{t,1}} = \mathbb {P}(I^*=j \mid I^*\ne 1)\) . Resampling shifts \(1-\beta _t\)  fraction of measurement effort away from arm 1 and assigns it to the strongest challengers. In particular, a challenger is sampled according to its conditional probability of being optimal.
It is worth noting that Proposition REF , which is about robustness to delay and non-stationary contexts, holds even if \(\beta _t=1\)  and there is no resampling. The resampling is only needed for the faster asymptotic rates in Section .

