Unfortunately, UCBVI [1]} does not fit into our framework. However, even if it did, we argue that it would not lead to improved results. Indeed, the algorithm makes use of Bernstein's inequality to handle the estimation error, resulting in the need to bound an additional term of the form: \(H^2 S/N_{kh}^{\prime }(s, a)\) . Lemma REF  shows that this term alone introduces the following dependency on the delay: \(H^3 S^2 A\mathbb {E}[\tau ]\) , which is worse than any of the other algorithms we consider in this paper. UBEV [2]}, for example, has a delay dependency of: \(H^2 S A\mathbb {E}[\tau ]\) ; an improvement of \(HS\) . Moreover, due to bounding the empirical variance, UCBVI and UBEV will have the same leading order term in their regret bound. Therefore, we do not believe that using UCBVI in the delayed setting will lead to an improvement over UBEV, an algorithm which does fit into our framework.
