A simple way to identify the problem in the previous derivation is to consider the special case where \(\mathcal {E}_0=0\) , which corresponds to using \(\Phi =I_{|\mathcal {S}||\mathcal {A}|}\)  in our NAC algorithm (i.e., the tabular setting). In this case, since the RHS of Eq. (REF ) is infinity, the convergence bound in Eq. (REF ) and also Eq. () are meaningless. In our Theorem REF , when \(\Phi =I_{|\mathcal {S}||\mathcal {A}|}\)  and hence \(\xi =0\) , Theorem REF  still provides a meaningful finite-sample bounds. In fact, it coincides with the finite-sample bounds of tabular NAC provided in [1]} when the two truncation levels within the \(Q\) -trace algorithm are large enough. Therefore, the issue of trading off asymptotic error and convergence rate using AM-GM inequality is not present in our results.
