For our analysis, we assume the \(Q\) -function initialization is optimal for all but one value.
Although this is a rather strong assumption, we believe it provides useful insights and a promising starting point for future work.
In general, the \(Q\) -function initialization in a target MDP after \(Q\) -function reuse has some nearly optimal \(Q\) -values and some \(Q\) -values that are far from optimal.
Therefore, to address the general case of \(Q\) -function reuse, future work might include increasing the number of not optimal \(Q\) -values to more than one and relaxing the optimal \(Q\) -function initialization for the remaining \(Q\) -values to some notion of sub-optimality.
To the best of our knowledge, there has not been any theoretical work showing the total regret of \(Q\) -function reuse applied on a tabular, model-free algorithm.
We perform a regret analysis, showing that the \(Q\) -learning with the UCB-Hoeffding algorithm [1]}, along with our initialization assumptions achieves a total regret of only \(O(\sqrt{H^2T\iota ^{\prime }})\)  (independent of the state and action space), while regular \(Q\) -learning with UCB-Hoeffding suffers a regret of \(O(\sqrt{H^4SAT\iota })\)  [1]}.
Empirical results are presented to support these theoretical claims.
