From the above enumeration, points 1 and 3 have already been discussed in detail and are shown explicitly as Algorithm  and Algorithm 1 in [1]} respectively.
The goal of point 2 is to show that A2 is crucial for our theoretical regret bound (as seen in Theorem REF ).
Since A2 is removed in point 2, we expect the total regret to be greater than that of our algorithm.
An important detail is that the \(Q\) -learning with UCB-Hoeffding and Max-Optimal Initialization without A2 algorithm is the same as our algorithm except with A2 removed, but also with \(N_h(x, a)\)  initialized to 0 for all \((x, a, h) \in \mathcal {S} \times \mathcal {A} \times [H]\) . Since now the \(Q\) -function is updated for all states, actions, and steps, a visit count \(N_h(x, a)\)  must be kept for all \((x, a, h)\) .
