The term \(A_2\)  captures the error introduced to the system due to function approximation, and cannot be eliminated asymptotically. Moreover, known results in approximate policy iteration (API) literature suggest that the \(1/(1-\gamma )^2\)  coefficient inside the term \(A_2\)  is inevitable. Specifically, it is shown [1]}, [2]} that when \(\max _{\pi }\Vert V^\pi -\Phi w_\pi \Vert _\infty \le \xi \) , under the API algorithm \(\limsup _{k\rightarrow \infty }\Vert V^{\pi _k}-V^{\pi ^*}\Vert _\infty \le \frac{2\gamma \xi }{(1-\gamma )^2}\) , and an example is presented in [2]}, where the inequality is tight. Since NAC algorithm can be viewed as an API algorithm with a softmax policy update (which is also weighted by the current policy), it is natural to expect a similar function approximation bias.
Therefore, to improve the function approximation bias term \(A_2\) , one has to develop instance dependent bound, which is one of our future direction.
