In the existing literature of NAC algorithm with linear function approximation, the critic aims at finding the projection (onto \(\mathcal {Q}\) ) of the target \(Q\) -function \(Q^{\pi _t}\)  with respect to a suitable norm involving the state visitation distribution \(d^{\pi _t}\)  [1]}. More specifically, \(w_t\)  is an estimate of the minimizer of the optimization problem \(\mathbb {E}_{s\sim d^{\pi _t},a\sim \pi _t}[(Q^{\pi _t}(s,a)-\phi (s,a)^\top w)^2]\) . However, the distribution \(d^{\pi _t}\)  is unknown and also requires special sampling [1]}. Moreover, such sampling requires constant reset of the system which is necessary in the AC algorithms proposed in [1]}, [4]}, [5]}; see [6]} for a more detailed discussion.
In contrast, in our Algorithm REF , the critic aims at solving the projected Bellman equation (REF ), and uses a single sample trajectory. The projected Bellman equation was first introduced in the seminal work [7]} for analyzing on-policy TD with linear function approximation. Here we generalize the result of [7]} to the off-policy setting and we use it in the critic of NAC.
