Typically, \(B > 0 \)  for algorithms using variance reduction techniques, for example \(\chi ^2\) -UCRL and UCRL2B. For these algorithms, from Table REF , it is clear that it is always better to use active updating. Thus, we focus our discussion on this updating scheme. Neither \(\chi ^2\) -UCRL nor UCRL2B recover their regret bounds in the non-delayed feedback setting, due to bounding the empirical variance of the optimistic value function in each episode by \(H\) . This was necessary to avoid a multiplicative increase in regret due to the delays. Further, introducing the empirical variance comes at the expense of some lower-order terms: \(HS\log (H, S, A, T, \delta ^{\prime })/N_{kh}^{\prime }(s, a)\) . Lemma REF  shows that these lower-order terms can be problematic, as they now depend on the expected delay. In particular, they lead to a term of order \(H^2 S^2 A\mathbb {E}[\tau ]\)  in the regret bound. We note that the impact of these lower order terms when using non-standard counters has also been observed in settings such as differentially private reinforcement learning [1]}.
