In summary, we showed that the total regret of \(Q\) -learning with UCB-Hoeffding and Max-Optimal Initialization is upper bounded by \(O(\sqrt{H^2T\iota ^{\prime }})\) . This total regret bound is tighter than that of the \(Q\) -learning with UCB-Hoeffding algorithm, \(O(\sqrt{H^4SAT\iota })\)  [1]}. This result provides theoretical justification for applying \(Q\) -function reuse on the \(Q\) -function with UCB-Hoeffding algorithm. Although we make the strong assumption that all but one of the Q-values are optimal, we believe that this provides a solid starting point for future work to build upon.
